Tablebase version comparison

Endgame analysis using tablebases, EGTB generation, exchange, sharing, discussions, etc..

Tablebase version comparison

Postby mbourzut » Thu Aug 10, 2006 12:42 pm

Comparing the Nalimov generated tablebases with those obtained through Johan de Koning's FEG we so far have 100% agreement. The .tbs statistics files are identical, and the compressed file sizes are (almost) all identical. md5 comparisons will also be done and I expect those to be identical as well.

The only exceptions are the kqpkrp and kqpkbp endings. Sometime between version 20.0 and 20.14 of crafty Eugene has decided to store these in a slightly different format (fewer but larger files) I have transformed the FEG generated files into the same format, and attach the corresponding md5sums. These should replace the md5sums for kqpkrp and kqpkbp I posted previously.

I will send Josh a script to convert those files on his end.

-Marc
Attachments
kqpkbp_kqpkrp.zip
(848 Bytes) Downloaded 169 times
mbourzut
 
Posts: 30
Joined: Fri Mar 03, 2006 7:48 pm

Postby jshriver » Fri Aug 11, 2006 6:17 am

Thanks, received the zip. Going to start converting these hopefully this weekend.

I want the view of the group. I'm thinking about taking down my site temporarily till the conversion is done so that we are sure people are getting the final product that will "eventually" be used amongst all engines.

-Josh
User avatar
jshriver
 
Posts: 287
Joined: Tue Jan 31, 2006 5:59 am
Location: Toledo, OH, USA

Postby Kirill Kryukov » Fri Aug 11, 2006 8:27 am

Hi Josh, I thought that only kqpkrp and kqpkbp need to be converted? Which means we can continue distributing other sets? Please correct me if I am wrong. :-)
User avatar
Kirill Kryukov
Site Admin
 
Posts: 7380
Joined: Sun Dec 18, 2005 9:58 am
Location: Mishima, Japan

Postby mbourzut » Fri Aug 11, 2006 11:18 am

Kirill Kryukov wrote:Hi Josh, I thought that only kqpkrp and kqpkbp need to be converted? Which means we can continue distributing other sets? Please correct me if I am wrong. :-)


To make absolutely sure, we should only distribute those sets where the md5sums have been matched as well (in addition to file sizes).
mbourzut
 
Posts: 30
Joined: Fri Mar 03, 2006 7:48 pm

Verification process status

Postby Wulff » Sun Aug 13, 2006 5:58 am

Hi!

How is the verification process comming along ?? 8)
Wulff
 
Posts: 53
Joined: Thu Jan 05, 2006 4:08 pm

Promulgation and Verification progress ...

Postby guyhaw » Sun Aug 13, 2006 9:39 am

I understand from Nelson H and Vincent D that Vincent is d'loading the Nalimov-originated EGTs from Nelson, and that this will be finished on Monday.
Nelson has not MD5sum-checked the files (against Marc B's revised set of MD5sum data): Vincent will do so - but only when all files are d'loaded.
It's unfortunate that Eugene did not MD5sum-tag the files before they started their journey, but 'datacomp' and Marc B's independent data are guardians to the process.
I'm wondering what versions of EGT-access-code are 'out there' in engines, and which will line-up with Eugene's latest choices of 8-bit/16-bit bases for the EGTs. I guess the latest CRAFTY will access the 8-bit KQPKKBP and KQPKRP EGTs.
g
guyhaw
 
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Location: Reading, UK

Short vacation.

Postby jshriver » Tue Aug 15, 2006 4:54 am

Greetings everyone,

I have the data on my hard drive and going to let it crunch while I'm out of town.

I have 2 days off in a row, so driving back home to visit my family that are 6-7 hours away, I haven't seen them since Jan. Last chance I'll get since I go back to school in a week and work full time.

I hope to have good news when I return, and hope I haven't been to slow to respond.

Thanks for your patience.
-Josh
User avatar
jshriver
 
Posts: 287
Joined: Tue Jan 31, 2006 5:59 am
Location: Toledo, OH, USA

The critical path ...

Postby guyhaw » Tue Aug 15, 2006 9:50 pm

Josh - I don't think you are on the critical path now.
As Marc B has said, the best 'green light' to restart the promulgation of Nalimov-original or FEG-derived EN DTM EGTs for the 'last 16 endgames' is when someone confirms that the MD5sum data for the Nalimov-original files is identical to the MD5sum data for Marc's FEG-derived 'Nalimov' EGTs.
I haven't seen any such declaration yet - either in public or in my email.
g
guyhaw
 
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Location: Reading, UK

Re: The critical path ...

Postby jshriver » Thu Aug 17, 2006 4:52 am

guyhaw wrote:Josh - I don't think you are on the critical path now.
As Marc B has said, the best 'green light' to restart the promulgation of Nalimov-original or FEG-derived EN DTM EGTs for the 'last 16 endgames' is when someone confirms that the MD5sum data for the Nalimov-original files is identical to the MD5sum data for Marc's FEG-derived 'Nalimov' EGTs.
I haven't seen any such declaration yet - either in public or in my email.
g


Just came home. From my understanding the data is identical in regards to the actual database. The only differences are in the index size and how the files were split.

I'm going to finish the conversion, and generate md5sums. I'm not sure, but if Vincent can produce md5sum's of the nalimov release once he finished receiving them we can compare to make sure.

I'm not going to post any more files till we have it figured out and come to a common ground. Though if the conversion does yield identical sets then it wont take long to get these out to people.

-Josh
User avatar
jshriver
 
Posts: 287
Joined: Tue Jan 31, 2006 5:59 am
Location: Toledo, OH, USA

The x-comparison of 'EN' and 'MB' EGTs for the 'last 16'

Postby guyhaw » Thu Aug 17, 2006 5:43 am

Josh,
My understanding is that:
1) Nelson H, on the basis of a successful telephone call, was able to receive from Eugene Nalimov himself the Nalimov-original 3-3p DTM EGTs for the 'last 16' endgames.
2) Vincent D in Holland has uploaded all these files to his machine in Holland.
3) A DATACOMP check by VD showed the need for re-uploading 3 files but eventually ok'd all these files. Note that this _only_ means that the files are as they were when compressed by Eugene Nalimov. It does not mean that the Nalimov-original files are chessicly correct.
4) The Nalimov-original .tbs stats files for these last 16 endgames agree 100% with the MB FEG.lof stats files (generated by JdKonig's independent technology FEG), and with MB's Nalimov-style .tbs files created from his FEG-derived Nalimov-style DTM EGTs.

The .tbs-file alignment is an encouraging sign that in fact the Nalimov-original set of files will prove to be chessicly correct and identical to MB's FEG-derived set of Nalimov files. I would not bet against it, but ...

The final, fully independent verification of this fact is when someone creates a file of MD5sum-evidence showing that the Nalimov-original EGT-files are all identical to MB's FEG-derived 'EN EGT' files.
The ideal way forward is that Vincent does this so that no further file-promulgation is required in advance of this evidence being produced. Vincent needs time to do this, and it hasn't been done yet. However, another 1-1 d'load [not to me] from Nelson H is in progress in case Vincent does not produce this MD5sum x-comparison evidence.

Footnote re 'history'. Note that normally, one would receive Nalimov-original EGT files, run DATACOMP, and accept that these files were chessicly correct because Eugene runs his own verification software. However, EN's verification software has an 80% overlap with his generation software and is not therefore independent of it.
But this time, we are looking for the extra MD5sum x-comparison evidence, not only because it is the obvious way of showing that the 'EN' and 'MB' files are the same, but because it completes a verification of the Nalimov-original EGTs by the fully independent FEG technology of JdKonig - a 'first'.

The team of people each doing their bit to achieve this 'first' is considerable, and evidence of the community-building power of p2p and of Kyrill's 'EGTs Online' project.
g
guyhaw
 
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Location: Reading, UK

Re: The critical path ...

Postby kp1089 » Wed Aug 23, 2006 11:23 am

jshriver wrote:
guyhaw wrote:Josh - I don't think you are on the critical path now.
As Marc B has said, the best 'green light' to restart the promulgation of Nalimov-original or FEG-derived EN DTM EGTs for the 'last 16 endgames' is when someone confirms that the MD5sum data for the Nalimov-original files is identical to the MD5sum data for Marc's FEG-derived 'Nalimov' EGTs.
I haven't seen any such declaration yet - either in public or in my email.
g


Just came home. From my understanding the data is identical in regards to the actual database. The only differences are in the index size and how the files were split.

I'm going to finish the conversion, and generate md5sums. I'm not sure, but if Vincent can produce md5sum's of the nalimov release once he finished receiving them we can compare to make sure.

I'm not going to post any more files till we have it figured out and come to a common ground. Though if the conversion does yield identical sets then it wont take long to get these out to people.

-Josh



I have completed a verification with Marc B.'s md5 files that the Eugene Nalimov generated egtbs are identical with the ones Marc created from the FEG egtbs. I have sent the results to Guy Haworth, who agrees they are identical. I informed you by email, but you did not seem to get the message, or have not replied, so I am announcing it here. All of the 16 missing sets are online, and if you want to double-check them for a match with a second verification, just do a search on the KAD network for a set. Check what is on there, and then share your files, and do another search after they finish hashing. If all of the files you have shared match up with the ones online, that will be a second method of proof of identicality. You should do the initial search first, so you can identify any spurious files before sharing yours, since there are a couple out there for kbnkpp already. That way you do not think the spurious files are yours. Let me know if you have any questions.

kp
How small, of all that human hearts endure, that part that laws and kings can cause or cure. -- Samuel Johnson.
kp1089
 
Posts: 10
Joined: Tue Jan 24, 2006 7:54 am
Location: Skamania, WA

Nicely done, kp ...

Postby guyhaw » Wed Aug 23, 2006 7:25 pm

That is the 'all clear' signal that we were waiting for - the confirmation by MD5sum-trial that the Nalimov-original and MB-FEG-derived EGTs are identical in every respect, and that therefore the two technologies validate each other's results.

If Josh MD5sum-checks his files in the usual way, he can become another with-integrity-supplier of them on the p2p-network, alongside Nelson and your good self.

The EGTs also went from Nelson to Vincent Diepeveen, but I've not hard of his MD5sum-check since. I do gather that vd will not be uploading on the p2p network, but rather driving his hard-disc around the neighbourhood to people who will.

Thank you very much - g
guyhaw
 
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Location: Reading, UK

Important: promoting to p1 for now :-)

Postby guyhaw » Mon Feb 16, 2009 12:04 am

.
guyhaw
 
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Location: Reading, UK


Return to Endgame Tablebases

Who is online

Users browsing this forum: No registered users and 2 guests

cron