found missing 33p tbs (tablebase stats) files

[jkominek]

Well I didn't find them, actually, I'm generating them.

Decompressing everything proved too inconvenient so instead I modified tbstat to read in either compressed or uncompressed files. I've done a few comparisons with existing tbs files and have seen no differences, so I'm convinced the basic processing is correct. It's only when max dtm is around 127 that extra care is required: the length of encoding (8-bit vs 16-bit) is not documented in the tablebase file itself and must be known externally.

I'm currently churning away on everything. Here's the first 10 of the missing 28. Interpret the numbers as max-dtm wtm/btm.

• 1. kbbkpp - 104/87
  2. kbnknp - 128/99
  3. knnkpp - 130/83
  4. kqbkbp - 58/58
  5. kqbknp - 63/45
  6. kqbkpp - 46/125
  7. kqbkrp - 105/90
  8. kqnkbp - 51/102
  9. kqnknp - 66/98
  10. kqnkpp - 63/128

Once finished I'll post the tbs files and invite cross comparison. And if Kirill is satisfied that they're okay, put them on the website.

john

[jkominek]

The 28 missing tbs files are ready. From what I can cross-check they look okay to me. I welcome other eyes (a Job for Guy?).

Kirill -- an email with these files attached bounced back to me. Perhaps your mail server is suspicious of zipped archives?

john

[Kirill Kryukov]

The 28 missing tbs files are ready. From what I can cross-check they look okay to me. I welcome other eyes (a Job for Guy?).

Kirill -- an email with these files attached bounced back to me. Perhaps your mail server is suspicious of zipped archives?

john

Hi John, thanks for this effort! I checked the files and it seems the format is slightly different from Nalimov tbs files. This part is not present in Nalimov files:

Code: Select all

wtm: positions won:             3096722420    21.260 %
wtm: positions lost:             755949359     5.190 %
wtm: positions drawn:          10713610907    73.551 %
wtm: positions legal:          14566282686    84.844 %
wtm: positions indexed:        17168385456

btm: positions won:             3009997689    18.587 %
btm: positions lost:             307257288     1.897 %
btm: positions drawn:          12877271523    79.516 %
btm: positions legal:          16194526500    98.411 %
btm: positions indexed:        16455976740

Can you create the output in exactly same format with Nailmov files? Or should I simply remove the extra stats above?

Also, when you produce the stats for an endgame with known tbs, do your statst match exactly?

By the way, If you want to provide any extra statistics (additional to the Nalimov format), we can do that too. Either as separate files, or substituting the Nalimov tbs. I just want to have stats in the same format for all files, so in such case we will have to re-generate the stats for all files. I can help that if it takes long time.

Also, are you going to release the stats generator program?

[Jaroslav]

These TBS files (most of them) are out very long time (2004).

[Kirill Kryukov]

Thanks, Jaroslav! Are they authentic stats from Nalimov (Hyatt's FTP)?

[Jaroslav]

Sorry, I don't remember yet I'm working with 6-men tablebases from 2002 (3+3)...

[guyhaw]

Not sure why .tbs files are being generating for missing .tbs files.

I think I posted all the .tbs files in a zipfile on this b'board in some thread or other. If not, I can provide the originals.
I'm certainly not volunteering to check 'new' statsfiles against the Nalimov ones - sorry.
g

[Kirill Kryukov]

Not sure why .tbs files are being generating for missing .tbs files.

I think I posted all the .tbs files in a zipfile on this b'board in some thread or other. If not, I can provide the originals.
I'm certainly not volunteering to check 'new' statsfiles against the Nalimov ones - sorry.
g

Hi Guy, yes, it would be great if you can post the 33p stat files (tbs). I must have misssed them when you posted them before!

[guyhaw]

There appear to be 61 rather than 65 .tbs files in this zipfile - I thought it was complete. I suspect I have the other four 'around' somewhere, and I'll look when I eventually discover which they are.
g

[Kirill Kryukov]

Thanks Guy!

Following is the list of files that
1. I was missing.
2. Contained in Jaroslav's archive
3. Contained in Guy's archive.
4. Identical in Guy's and Jaroslav's archives.

Code: Select all

kbbkpp.tbs
kbnknp.tbs
knnkpp.tbs
kqbkbp.tbs
kqbknp.tbs
kqbkpp.tbs
kqbkrp.tbs
kqnkbp.tbs
kqnknp.tbs
kqnkpp.tbs
kqnkqp.tbs
kqnkrp.tbs
kqpkbn.tbs
kqpkqp.tbs
kqpkrr.tbs
kqqkpp.tbs
kqrkbp.tbs
kqrknp.tbs
kqrkpp.tbs
krbkbp.tbs
krbknp.tbs
krnkbp.tbs
krpkbb.tbs
krrknp.tbs
krrkpp.tbs

I am adding them to the page. I will post what sets are still lacking the tbs later.

[jkominek]

This reminds me of a wise - or wry - comment about how the social interaction of newsgroups operates. If you want to get the answer to a question (find missing tbs files), don't ask politely, instead post a wrong answer (generate new files) and wait for the flames to fire back with the correction (missing tbs files). :)

Guy - Kirill mentioned that he thought someone had posted the missing files. But neither he nor I nor EnPlaisant could locate it. Got buried in the newgroup I'm afraid. Knowing now that you made the posting (should have guessed!) I could find it with a search. Alas, only 61/65 as you admit.

Kirill - yes, I can release my modification, if it would be welcomed. Currently I have two programs: one for 8-bit encoded files and one for 16-bit. A single program to handle both would be better, experience has shown, so that's a change to make.

For my generated files, I've compared about 20 by hand and found no discrepancies. Once I've generated them for all tablebases I'll write a script to compare new versus existing. I expect them to match but wouldn't propose adopting my files until the match is verified 100%. There are 12 more to go to complete the full 440.

john

P.S. I see you noticed my "summary stats" section. I added that as a useful extra.

[jkominek]

By the way, I just checked Hyatt's ftp server. He has 141 6-man tbs files available. Only 17 of them are for 33p. So his site doesn't have what we're after.

It appears that Hyatt is out of the tablebase "business." After I collected the complete Nalimov set I sent him an email asking if he'd like a copy. ("Send me the disks, I'll return them full.") He never replied. I can only infer that he's not interested any longer. Possibly the issue is the monthly hosting costs, a bill he's no longer willing to foot. This is an issue I've been trying to work out at my own university, which I can describe in a separate posting.

john

[Kirill Kryukov]

A bit more about the missing stats..

1. Guy's archive contained 4 tbs files which are different from those I already had. The difference is in number of draws and broken positions:

Code: Select all

Comparing files C:\TEMP\TBS FROM GUY\kbbknp.tbs and C:\TEMP\TBS IN EGTB PROJECT\KBBKNP.TBS
***** C:\TEMP\TBS FROM GUY\kbbknp.tbs
btm: Lost in 131:             1
btm: Draws:            769487238
btm: Mate in  86:             8
***** C:\TEMP\TBS IN EGTB PROJECT\KBBKNP.TBS
btm: Lost in 131:             1
btm: Draws:            5064454534
btm: Mate in  86:             8
*****

Code: Select all

Comparing files C:\TEMP\TBS FROM GUY\kbnkbp.tbs and C:\TEMP\TBS IN EGTB PROJECT\KBNKBP.TBS
***** C:\TEMP\TBS FROM GUY\kbnkbp.tbs
wtm: Lost in  96:             1
wtm: Draws:            2395943561
wtm: Mate in 105:           165
***** C:\TEMP\TBS IN EGTB PROJECT\KBNKBP.TBS
wtm: Lost in  96:             1
wtm: Draws:            10985878153
wtm: Mate in 105:           165
*****

***** C:\TEMP\TBS FROM GUY\kbnkbp.tbs
btm: Lost in 104:            50
btm: Draws:            3560501349
btm: Mate in 100:             2
***** C:\TEMP\TBS IN EGTB PROJECT\KBNKBP.TBS
btm: Lost in 104:            50
btm: Draws:            12150435941
btm: Mate in 100:             2
*****

Code: Select all

Comparing files C:\TEMP\TBS FROM GUY\kqbkqp.tbs and C:\TEMP\TBS IN EGTB PROJECT\KQBKQP.TBS
***** C:\TEMP\TBS FROM GUY\kqbkqp.tbs
wtm: Lost in 127:             1
wtm: Draws:            732514479
wtm: Mate in  63:             6
***** C:\TEMP\TBS IN EGTB PROJECT\KQBKQP.TBS
wtm: Lost in 127:             1
wtm: Draws:            5027481775
wtm: Mate in  63:             6
*****

***** C:\TEMP\TBS FROM GUY\kqbkqp.tbs
wtm: Mate in   1:      241009710
wtm: Broken positions: 1794708894
btm: Lost in   0:      33893960
***** C:\TEMP\TBS IN EGTB PROJECT\KQBKQP.TBS
wtm: Mate in   1:      241009710
wtm: Broken positions: 6089676190
btm: Lost in   0:      33893960
*****

***** C:\TEMP\TBS FROM GUY\kqbkqp.tbs
btm: Lost in  63:             3
btm: Draws:            2049130245
btm: Mate in 128:             1
***** C:\TEMP\TBS IN EGTB PROJECT\KQBKQP.TBS
btm: Lost in  63:             3
btm: Draws:            6344097541
btm: Mate in 128:             1
*****

Code: Select all

Comparing files C:\TEMP\TBS FROM GUY\krpknn.tbs and C:\TEMP\TBS IN EGTB PROJECT\KRPKNN.TBS
***** C:\TEMP\TBS FROM GUY\krpknn.tbs
btm: Lost in 251:             1
btm: Draws:            1313134857
btm: Mate in 115:            30
***** C:\TEMP\TBS IN EGTB PROJECT\KRPKNN.TBS
btm: Lost in 251:             1
btm: Draws:            5608102153
btm: Mate in 115:            30
*****

I don't know which ones are correct - my files or files from Guy. May be John's program can be used to check at least number of draws?

[Kirill Kryukov]

More about missing stats. Only 3 tbs files are now missing:

kqpkbb.tbs
kqqkrp.tbs
kqrkqp.tbs

Guy's archive contains all of them, and Jaroslav's archive contains kqrkqp.tbs which is different from Guy's. Given this and the discrepancy I posted above I am not sure if they are correct or not... Any additional information would be very much welcome.

[Kirill Kryukov]

This reminds me of a wise - or wry - comment about how the social interaction of newsgroups operates. If you want to get the answer to a question (find missing tbs files), don't ask politely, instead post a wrong answer (generate new files) and wait for the flames to fire back with the correction (missing tbs files).

I observed this many times but did not realize.

Guy - Kirill mentioned that he thought someone had posted the missing files. But neither he nor I nor EnPlaisant could locate it. Got buried in the newgroup I'm afraid. Knowing now that you made the posting (should have guessed!) I could find it with a search. Alas, only 61/65 as you admit.

62 33p stat files are now online at EGTB Online web page. 3 still "missing" (see above).

Kirill - yes, I can release my modification, if it would be welcomed. Currently I have two programs: one for 8-bit encoded files and one for 16-bit. A single program to handle both would be better, experience has shown, so that's a change to make.

Yes it would be welcomed! May be also to check correctness of tablebases (for those who does not trust MD5).

For my generated files, I've compared about 20 by hand and found no discrepancies. Once I've generated them for all tablebases I'll write a script to compare new versus existing. I expect them to match but wouldn't propose adopting my files until the match is verified 100%. There are 12 more to go to complete the full 440.

john

P.S. I see you noticed my "summary stats" section. I added that as a useful extra.

I expect nothing less than a byte-to-byte match with Nalimov's tbs. When it can produce byte-to-byte identical stats, then you can think about adding additional fancy info. My IMHO. And thanks for your efforts!

[jkominek]

I expect nothing less than a byte-to-byte match with Nalimov's tbs. When it can produce byte-to-byte identical stats, then you can think about adding additional fancy info.

Sorry, I don't agree. What matters is the logical information - the numbers. In particular, Nalimov made the formatting field width too small for the large numbers generated (number of draw and broken positions especially), and this ruins the right justification of the print out. I increased the field width to fix this inelegance. I'm not going to "unfix" it for the sake of byte-by-byte identiy.

Backward compatibility has its limits!

john

[Kirill Kryukov]

I expect nothing less than a byte-to-byte match with Nalimov's tbs. When it can produce byte-to-byte identical stats, then you can think about adding additional fancy info.

Sorry, I don't agree. What matters is the logical information - the numbers. In particular, Nalimov made the formatting field width too small for the large numbers generated (number of draw and broken positions especially), and this ruins the right justification of the print out. I increased the field width to fix this inelegance. I'm not going to "unfix" it for the sake of byte-by-byte identiy.

Backward compatibility has its limits!

john

I would be OK to change stats file format, but only if we can re-compute the stats in new format for all files, starting from 3-men. Then if the tool is available, anyone can download a file, compute stats and compare with the stats from web-page.

We can't change format for some files and leave it as it is for other files. So if I were you I would "unfix" it for byte-to-byte identity, release version 1, and then make version 2 which has fixed text format and additional output, and any other improvements you may think about.

(Or, if it is open-source, may be I can unfix it myself, or someone else here. )

[Jaroslav]

More about missing stats. Only 3 tbs files are now missing:

kqpkbb.tbs
kqqkrp.tbs
kqrkqp.tbs

Guy's archive contains all of them, and Jaroslav's archive contains kqrkqp.tbs which is different from Guy's. Given this and the discrepancy I posted above I am not sure if they are correct or not... Any additional information would be very much welcome.

I'm posting all 65 tbs files for 3+3p. I have identical differencies against Guy's tbs files as Kirill (in other words, I have same tbs files as Kirill).

Guy's tbs are wrong, because of 32-bit limitation in numbers, f.e. 5064454534 = 769487238 + 0x100000000 (difference in kbbknp.tbs).

[Kirill Kryukov]

More about missing stats. Only 3 tbs files are now missing:

kqpkbb.tbs
kqqkrp.tbs
kqrkqp.tbs

Guy's archive contains all of them, and Jaroslav's archive contains kqrkqp.tbs which is different from Guy's. Given this and the discrepancy I posted above I am not sure if they are correct or not... Any additional information would be very much welcome.

I'm posting all 65 tbs files for 3+3p. I have identical differencies against Guy's tbs files as Kirill (in other words, I have same tbs files as Kirill).

Guy's tbs are wrong, because of 32-bit limitation in numbers, f.e. 5064454534 = 769487238 + 0x100000000 (difference in kbbknp.tbs).

Thanks Jaroslav! Also great that you explained the difference. I will add the last 3 files from your archive to the web-page tomorrow.

[jkominek]

I would be OK to change stats file format, but only if we can re-compute the stats in new format for all files, starting from 3-men.

Yes, you're exactly right about that. They should be consistent, i.e. both in format and generated by the same program. It's worth remembering that technically, the existing tbs files do NOT live up to this standard -- because they were generated over the span of several years by a program that changed to accomodate larger files. That Guy has in his possession erroneous files is a result of this.

May be also to check correctness of tablebases (for those who does not trust MD5).

An excellent point. Establishing correctness is one of Guy's hobbyhorses ("special concerns" for those that don't know English idiom), and this would be a great second-stage verification beyond md5. Thus if we establish a set of reference tbs files, I can verify the tablebase counts againt what they should be.

I'll take this as a feature request. Give me a week or two to work this in and publish a software release. I use BSD-style licenses for my open source projects.

ohn

[Kirill Kryukov]

It's worth remembering that technically, the existing tbs files do NOT live up to this standard -- because they were generated over the span of several years by a program that changed to accomodate larger files. That Guy has in his possession erroneous files is a result of this.

I think this is one additional argument supporting the change of format.

May be also to check correctness of tablebases (for those who does not trust MD5).

An excellent point. Establishing correctness is one of Guy's hobbyhorses ("special concerns" for those that don't know English idiom), and this would be a great second-stage verification beyond md5. Thus if we establish a set of reference tbs files, I can verify the tablebase counts againt what they should be.

I'll take this as a feature request. Give me a week or two to work this in and publish a software release. I use BSD-style licenses for my open source projects.

Sounds very good! Yes, please take your time! I will look forward to see and use your program. Also I will add announcement and explanation to the web-page.

[guyhaw]

The four .tbs files previously missing from my 3-3p set were for KQPKNN, KQQKQP, KRRKBP and KRRKRP. I obviously had them at one time as I've decanted their stats into .xls's. I have them now - from Rob Hyatt's ftp site. I guess they escaped in the last laptop migration.

I'm not following the discussion here too closely, but there's talk of erronous stats. The only problem is that when the numbers got big, Nalimov would drop 1*, 2* or even 3* 2^32 positions because he was using 32-bit arithmetic. I discovered this by checking his total of stm win/drawn/lost/broken positions against the index range (which should have agreed).
I've corrected the numbers in my .xls's but not in Nalimov's original files. I didn't keep a record of which .tbs files suffered in this way, and I think Eugene eventually switched to 64-bit arithmetic in the stats program.
g

[guyhaw]

For some reason, Nalimov's KBPKN stats were originally incorrect. This was spotted when we put the DTC and DTZ stats alongside, and maybe sooner - I forget.
Nalimov computed new stats, but - again mysteriously - the incorrect stats persisted on Rob Hyatt's ftp site. So I caveat these stats.
Btw, the Nalimov-index ranges I checked stats-totals against were found in the access-code.
g

[Kirill Kryukov]

More about missing stats. Only 3 tbs files are now missing:

kqpkbb.tbs
kqqkrp.tbs
kqrkqp.tbs

Guy's archive contains all of them, and Jaroslav's archive contains kqrkqp.tbs which is different from Guy's. Given this and the discrepancy I posted above I am not sure if they are correct or not... Any additional information would be very much welcome.

I'm posting all 65 tbs files for 3+3p. I have identical differencies against Guy's tbs files as Kirill (in other words, I have same tbs files as Kirill).

Guy's tbs are wrong, because of 32-bit limitation in numbers, f.e. 5064454534 = 769487238 + 0x100000000 (difference in kbbknp.tbs).

Thanks Jaroslav! Also great that you explained the difference. I will add the last 3 files from your archive to the web-page tomorrow.

Added to the web-page now, so now we have all 65 33p tbs files online. Thanks Jaroslav, John and Guy!

[Pachnes]

Hi!

Thank you very much.