Tablebase version comparison

[mbourzut]

Comparing the Nalimov generated tablebases with those obtained through Johan de Koning's FEG we so far have 100% agreement. The .tbs statistics files are identical, and the compressed file sizes are (almost) all identical. md5 comparisons will also be done and I expect those to be identical as well.

The only exceptions are the kqpkrp and kqpkbp endings. Sometime between version 20.0 and 20.14 of crafty Eugene has decided to store these in a slightly different format (fewer but larger files) I have transformed the FEG generated files into the same format, and attach the corresponding md5sums. These should replace the md5sums for kqpkrp and kqpkbp I posted previously.

I will send Josh a script to convert those files on his end.

-Marc

[jshriver]

Thanks, received the zip. Going to start converting these hopefully this weekend.

I want the view of the group. I'm thinking about taking down my site temporarily till the conversion is done so that we are sure people are getting the final product that will "eventually" be used amongst all engines.

-Josh

[Kirill Kryukov]

Hi Josh, I thought that only kqpkrp and kqpkbp need to be converted? Which means we can continue distributing other sets? Please correct me if I am wrong.

[mbourzut]

Hi Josh, I thought that only kqpkrp and kqpkbp need to be converted? Which means we can continue distributing other sets? Please correct me if I am wrong.

To make absolutely sure, we should only distribute those sets where the md5sums have been matched as well (in addition to file sizes).

[Wulff]

Hi!

How is the verification process comming along ??

[guyhaw]

I understand from Nelson H and Vincent D that Vincent is d'loading the Nalimov-originated EGTs from Nelson, and that this will be finished on Monday.
Nelson has not MD5sum-checked the files (against Marc B's revised set of MD5sum data): Vincent will do so - but only when all files are d'loaded.
It's unfortunate that Eugene did not MD5sum-tag the files before they started their journey, but 'datacomp' and Marc B's independent data are guardians to the process.
I'm wondering what versions of EGT-access-code are 'out there' in engines, and which will line-up with Eugene's latest choices of 8-bit/16-bit bases for the EGTs. I guess the latest CRAFTY will access the 8-bit KQPKKBP and KQPKRP EGTs.
g

[jshriver]

Greetings everyone,

I have the data on my hard drive and going to let it crunch while I'm out of town.

I have 2 days off in a row, so driving back home to visit my family that are 6-7 hours away, I haven't seen them since Jan. Last chance I'll get since I go back to school in a week and work full time.

I hope to have good news when I return, and hope I haven't been to slow to respond.

Thanks for your patience.
-Josh

[guyhaw]

Josh - I don't think you are on the critical path now.
As Marc B has said, the best 'green light' to restart the promulgation of Nalimov-original or FEG-derived EN DTM EGTs for the 'last 16 endgames' is when someone confirms that the MD5sum data for the Nalimov-original files is identical to the MD5sum data for Marc's FEG-derived 'Nalimov' EGTs.
I haven't seen any such declaration yet - either in public or in my email.
g

[jshriver]

Josh - I don't think you are on the critical path now.
As Marc B has said, the best 'green light' to restart the promulgation of Nalimov-original or FEG-derived EN DTM EGTs for the 'last 16 endgames' is when someone confirms that the MD5sum data for the Nalimov-original files is identical to the MD5sum data for Marc's FEG-derived 'Nalimov' EGTs.
I haven't seen any such declaration yet - either in public or in my email.
g

Just came home. From my understanding the data is identical in regards to the actual database. The only differences are in the index size and how the files were split.

I'm going to finish the conversion, and generate md5sums. I'm not sure, but if Vincent can produce md5sum's of the nalimov release once he finished receiving them we can compare to make sure.

I'm not going to post any more files till we have it figured out and come to a common ground. Though if the conversion does yield identical sets then it wont take long to get these out to people.

-Josh

[guyhaw]

Josh,
My understanding is that:
1) Nelson H, on the basis of a successful telephone call, was able to receive from Eugene Nalimov himself the Nalimov-original 3-3p DTM EGTs for the 'last 16' endgames.
2) Vincent D in Holland has uploaded all these files to his machine in Holland.
3) A DATACOMP check by VD showed the need for re-uploading 3 files but eventually ok'd all these files. Note that this _only_ means that the files are as they were when compressed by Eugene Nalimov. It does not mean that the Nalimov-original files are chessicly correct.
4) The Nalimov-original .tbs stats files for these last 16 endgames agree 100% with the MB FEG.lof stats files (generated by JdKonig's independent technology FEG), and with MB's Nalimov-style .tbs files created from his FEG-derived Nalimov-style DTM EGTs.

The .tbs-file alignment is an encouraging sign that in fact the Nalimov-original set of files will prove to be chessicly correct and identical to MB's FEG-derived set of Nalimov files. I would not bet against it, but ...

The final, fully independent verification of this fact is when someone creates a file of MD5sum-evidence showing that the Nalimov-original EGT-files are all identical to MB's FEG-derived 'EN EGT' files.
The ideal way forward is that Vincent does this so that no further file-promulgation is required in advance of this evidence being produced. Vincent needs time to do this, and it hasn't been done yet. However, another 1-1 d'load [not to me] from Nelson H is in progress in case Vincent does not produce this MD5sum x-comparison evidence.

Footnote re 'history'. Note that normally, one would receive Nalimov-original EGT files, run DATACOMP, and accept that these files were chessicly correct because Eugene runs his own verification software. However, EN's verification software has an 80% overlap with his generation software and is not therefore independent of it.
But this time, we are looking for the extra MD5sum x-comparison evidence, not only because it is the obvious way of showing that the 'EN' and 'MB' files are the same, but because it completes a verification of the Nalimov-original EGTs by the fully independent FEG technology of JdKonig - a 'first'.

The team of people each doing their bit to achieve this 'first' is considerable, and evidence of the community-building power of p2p and of Kyrill's 'EGTs Online' project.
g

[kp1089]

Josh - I don't think you are on the critical path now.
As Marc B has said, the best 'green light' to restart the promulgation of Nalimov-original or FEG-derived EN DTM EGTs for the 'last 16 endgames' is when someone confirms that the MD5sum data for the Nalimov-original files is identical to the MD5sum data for Marc's FEG-derived 'Nalimov' EGTs.
I haven't seen any such declaration yet - either in public or in my email.
g

Just came home. From my understanding the data is identical in regards to the actual database. The only differences are in the index size and how the files were split.

I'm going to finish the conversion, and generate md5sums. I'm not sure, but if Vincent can produce md5sum's of the nalimov release once he finished receiving them we can compare to make sure.

I'm not going to post any more files till we have it figured out and come to a common ground. Though if the conversion does yield identical sets then it wont take long to get these out to people.

-Josh

I have completed a verification with Marc B.'s md5 files that the Eugene Nalimov generated egtbs are identical with the ones Marc created from the FEG egtbs. I have sent the results to Guy Haworth, who agrees they are identical. I informed you by email, but you did not seem to get the message, or have not replied, so I am announcing it here. All of the 16 missing sets are online, and if you want to double-check them for a match with a second verification, just do a search on the KAD network for a set. Check what is on there, and then share your files, and do another search after they finish hashing. If all of the files you have shared match up with the ones online, that will be a second method of proof of identicality. You should do the initial search first, so you can identify any spurious files before sharing yours, since there are a couple out there for kbnkpp already. That way you do not think the spurious files are yours. Let me know if you have any questions.

kp

[guyhaw]

That is the 'all clear' signal that we were waiting for - the confirmation by MD5sum-trial that the Nalimov-original and MB-FEG-derived EGTs are identical in every respect, and that therefore the two technologies validate each other's results.

If Josh MD5sum-checks his files in the usual way, he can become another with-integrity-supplier of them on the p2p-network, alongside Nelson and your good self.

The EGTs also went from Nelson to Vincent Diepeveen, but I've not hard of his MD5sum-check since. I do gather that vd will not be uploading on the p2p network, but rather driving his hard-disc around the neighbourhood to people who will.

Thank you very much - g

[guyhaw]

.