Page 1 of 2
The 16 "missing" Nalimov files
Posted: Mon Jul 17, 2006 5:56 pm
by mbourzut
I have created the 16 missing Nalimov tablebases for 33p endings by conversion from Johan de Koning's FEG data. The total compressed size is about 130GB, with kqpkrb being the largest at about 17GB, and kppkpp the smallest at about 1 GB. The original FEG data is about 90 GB, smaller partially because of larger block size (32k as opposed to 8k), and a more efficient compression algorithm.
I have run verification on a few, but not all the endings.
If no other source is forthcoming, I'd be willing to put these on a disk and send to someone in the US who has all the existing datasets and can run a complete verification before distributing more widely. Verification is reasonably fast, kqpkrb took about 1.5 days on a 3.8 Ghz machine, but the other endings will run a lot faster.
Neatly done ...
Posted: Tue Jul 18, 2006 12:25 am
by guyhaw
Fine job as ever, Marc. Excellent news: thank you.
I'd be happy to file your FEG-derived 'EN .tbs' stats alongside your FEG.lof stats if you care to leave a file on this CCRL board.
Maybe a disc on a round trip from Eiko or Renze in Europe would expedite the verification. I don't have the requisite EGTs myself.
g
Posted: Tue Jul 18, 2006 2:29 am
by Kirill Kryukov
Excellent news, thank you very much, Marc! I'd suggest to send it to Joshua Shriver, as he lives in US (AFAIK), and he has good connection, and because I know he will share the files online. (And he also volanteered in another thread).
... and the MD5sums please ...
Posted: Tue Jul 18, 2006 3:57 pm
by guyhaw
... which will help check the EGTs after whatever trip they do. g
Posted: Wed Jul 19, 2006 2:59 am
by jshriver
Kirill Kryukov wrote:Excellent news, thank you very much, Marc! I'd suggest to send it to Joshua Shriver, as he lives in US (AFAIK), and he has good connection, and because I know he will share the files online. (And he also volanteered in another thread).
Aye
I live in Ohio. What are the prereq sets for verification? Regardless I'd be willing to host them.
-Josh
Re: The 16 "missing" Nalimov files
Posted: Wed Jul 19, 2006 3:00 am
by jshriver
mbourzut wrote:I have created the 16 missing Nalimov tablebases for 33p endings by conversion from Johan de Koning's FEG data. The total compressed size is about 130GB, with kqpkrb being the largest at about 17GB, and kppkpp the smallest at about 1 GB. The original FEG data is about 90 GB, smaller partially because of larger block size (32k as opposed to 8k), and a more efficient compression algorithm.
I have run verification on a few, but not all the endings.
If no other source is forthcoming, I'd be willing to put these on a disk and send to someone in the US who has all the existing datasets and can run a complete verification before distributing more widely. Verification is reasonably fast, kqpkrb took about 1.5 days on a 3.8 Ghz machine, but the other endings will run a lot faster.
Does this include kppkpp? Either way congrats on the great work
-Josh
3-3p: the 'last 16 endgames' ....
Posted: Wed Jul 19, 2006 9:40 am
by guyhaw
The 'last 16' 3-3p endgames for which Nalimov did not publish the EGTs are:
KBNKPP, KBPKNP, KBPKPP, KNPKPP, KPPKPP, KQPKBP, KQPKNP. KQPKPP,
KQPKRB, KQPKRN, KQPKRP, KRBKPP, KRNKPP, KRPKBP, KRPKNP, KRPKPP
Marc is asking the (USA-)offload-receiver to verify these chessicly using Nalimov's EGT-verification software before sharing them on 'EGTs Online'. Therefore, any 6- and 5-man endgame these can convert to is needed as verification is not just an internal matter [and indeed, known mistakes have occurred at the conversion-point].
I agree: this is more great work from Marc, and its good to see FEG making a bigger contribution, and a contribution to the verification process.
g
Posted: Wed Jul 19, 2006 12:52 pm
by Kirill Kryukov
I can provide any of already shared 33p sets. I have all except two of them currently, the last two are downloading right now, should finish in less than a week. If Josh will do verification, I can set up an FTP to provide him with all necessary sets.
Posted: Thu Jul 20, 2006 1:14 am
by jshriver
Kirill Kryukov wrote:I can provide any of already shared 33p sets. I have all except two of them currently, the last two are downloading right now, should finish in less than a week. If Josh will do verification, I can set up an FTP to provide him with all necessary sets.
Sounds good to me
-Josh
Posted: Mon Jul 24, 2006 5:49 am
by vb4
Hi Mark,
Super job nice work! Listen when do you expect these 16 TB's to be verified for correctness? I have been studying the data from the TBS files and am very much interested in the results of the TBS files and not the actual tablebases. I guess until all 16 of these files are verified for correctness there is no sense in analyzing the TBS files at this point. Mark one other big question. Is there any chance if these 16 come out ok that you would be willing to generate the 5-1 tbalebases along with their TBS files?? It would be important to me for sure to support some of my research in EGTB's. I would think that except for the KPPPPK tablebase that the others should converge pretty quickly IMHO. Even the KPPPPK probably converges quickly also. Any chance that you could do them???
Thanks in advance,
Les
Posted: Tue Jul 25, 2006 3:35 am
by jshriver
vb4 wrote:Hi Mark,
Super job nice work! Listen when do you expect these 16 TB's to be verified for correctness? I have been studying the data from the TBS files and am very much interested in the results of the TBS files and not the actual tablebases. I guess until all 16 of these files are verified for correctness there is no sense in analyzing the TBS files at this point. Mark one other big question. Is there any chance if these 16 come out ok that you would be willing to generate the 5-1 tbalebases along with their TBS files?? It would be important to me for sure to support some of my research in EGTB's. I would think that except for the KPPPPK tablebase that the others should converge pretty quickly IMHO. Even the KPPPPK probably converges quickly also. Any chance that you could do them???
Thanks in advance,
Les
Hrm if I could get the generator working under 32-bit linux I'd start generating them
will add to queue.
-Josh
Posted: Tue Jul 25, 2006 3:48 am
by vb4
jshriver wrote:vb4 wrote:Hi Mark,
Super job nice work! Listen when do you expect these 16 TB's to be verified for correctness? I have been studying the data from the TBS files and am very much interested in the results of the TBS files and not the actual tablebases. I guess until all 16 of these files are verified for correctness there is no sense in analyzing the TBS files at this point. Mark one other big question. Is there any chance if these 16 come out ok that you would be willing to generate the 5-1 tbalebases along with their TBS files?? It would be important to me for sure to support some of my research in EGTB's. I would think that except for the KPPPPK tablebase that the others should converge pretty quickly IMHO. Even the KPPPPK probably converges quickly also. Any chance that you could do them???
Thanks in advance,
Les
Hrm if I could get the generator working under 32-bit linux I'd start generating them
will add to queue.
Hi Josh,
Wow that would be great !!!! I dont know much about Linux but perhaps someone out here might be able to assist you. Keeping my fingers crossed.
Thanks,
Les
-Josh
Posted: Tue Jul 25, 2006 7:07 am
by Lutz Nebe
vb4 wrote:Hi Mark,
...... Is there any chance if these 16 come out ok that you would be willing to generate the 5-1 tbalebases along with their TBS files?? It would be important to me for sure to support some of my research in EGTB's. I would think that except for the KPPPPK tablebase that the others should converge pretty quickly IMHO. Even the KPPPPK probably converges quickly also. Any chance that you could do them???
Thanks in advance,
Les
With the current version of the EGTB generator you can't create any of the 5+1 tablebases. It has to be modified. I will try to check the code during my holidays within the next three weeks. If anyone else has more information just let me know.
Posted: Tue Jul 25, 2006 2:10 pm
by vb4
Lutz Nebe wrote:vb4 wrote:Hi Mark,
...... Is there any chance if these 16 come out ok that you would be willing to generate the 5-1 tbalebases along with their TBS files?? It would be important to me for sure to support some of my research in EGTB's. I would think that except for the KPPPPK tablebase that the others should converge pretty quickly IMHO. Even the KPPPPK probably converges quickly also. Any chance that you could do them???
Thanks in advance,
Les
With the current version of the EGTB generator you can't create any of the 5+1 tablebases. It has to be modified. I will try to check the code during my holidays within the next three weeks. If anyone else has more information just let me know.
Hi Lutz,
I didnt know that Eugene's code would need to be modified!! I hope it isnt too difficult to do since in the past I have heard his code is rather complicating. Please keep me posted as to your progress when you have time to look at this or if anyone else out here is knowledgeable enough to make the necessary code changes.
Thanks,
Les
Re: The 16 "missing" Nalimov files
Posted: Thu Jul 27, 2006 12:31 am
by mbourzut
mbourzut wrote:I have created the 16 missing Nalimov tablebases for `3.8 Ghz machine, but the other endings will run a lot faster.
I have decided to run verification on all the endings to minimize the chances of repeated file exchanges. All the verifications were successful. Attached are the .tbs and md5sums for the lot. I have contacted Josh if he is willing to host the data files.
Re: The 16 "missing" Nalimov files
Posted: Thu Jul 27, 2006 1:38 am
by jshriver
mbourzut wrote:mbourzut wrote:I have created the 16 missing Nalimov tablebases for `3.8 Ghz machine, but the other endings will run a lot faster.
I have decided to run verification on all the endings to minimize the chances of repeated file exchanges. All the verifications were successful. Attached are the .tbs and md5sums for the lot. I have contacted Josh if he is willing to host the data files.
Sounds great
anxious to be part of this. I can host 30gigs from my website, and can try and host all 130 gig using emule from my home computer.
After receiving the data, it will take maybe a day or two to upload to the site.
-Josh
Posted: Thu Jul 27, 2006 1:41 am
by jshriver
In retrospect, to make things run even smoother. I can setup a hidden area on my site for people in this forum download. That way I can get the files to other key distributors before releasing it to the public.
What do you all think? This would take some strain off my server by allowing it to be more easily shared via emule, especially initially. Figure once these go public it's going to be insane traffic
will be fun watching the system stats though lol.
I could also send some to TapaniS.
-Josh
Re: The 16 "missing" Nalimov files
Posted: Thu Jul 27, 2006 2:48 am
by vb4
mbourzut wrote:mbourzut wrote:I have created the 16 missing Nalimov tablebases for `3.8 Ghz machine, but the other endings will run a lot faster.
I have decided to run verification on all the endings to minimize the chances of repeated file exchanges. All the verifications were successful. Attached are the .tbs and md5sums for the lot. I have contacted Josh if he is willing to host the data files.
Thank you so much for doing this! Now all I need are those darn 5-1 positions and I can kick some ... completing my study on the 3-4-5-6 piece egtb's. There have been some interesting findings so far which I will eventually summarize and share with everyone.
Thanks,
Les
Re: The 16 "missing" Nalimov files
Posted: Thu Jul 27, 2006 2:55 am
by jshriver
mbourzut wrote:I have created the 16 missing Nalimov tablebases for 33p endings by conversion from Johan de Koning's FEG data.
I was reading this page:
http://www.aarontay.per.sg/Winboard/egt ... masteredgb
It mentions that the Koning egtb only stores wins from the white point of view. Will this effect the quality of the nalimov conversion? Will we have the information for black's side?
On a side note this quote seems promising:
"The generator also makes generating 6-man with KWWWWB possible on a 32 bit computer possible , but that requires about 300MB (which isn't a lot by todays standards) of free RAM"
I'm interpreting this as being able to generate 5-1 egtbs
-Josh
Is this right ??
Posted: Thu Jul 27, 2006 2:58 am
by vb4
Hi Mark,
Listen I just took a peek inside one of your tbs files (KPPKPP) and see that it only lists the stats for wtm, win or lose but no stats for btm win or lose. I checked with the 4 piece EGTB KPKP and it has both sides to move stats?? Can you explain Mark? I would appreciate it.
Thanks,
Les
Posted: Thu Jul 27, 2006 3:18 am
by Kirill Kryukov
Thank you, Marc! I'm looking forward into completing the online collection!
jshriver wrote:In retrospect, to make things run even smoother. I can setup a hidden area on my site for people in this forum download. That way I can get the files to other key distributors before releasing it to the public.
What do you all think? This would take some strain off my server by allowing it to be more easily shared via emule, especially initially. Figure once these go public it's going to be insane traffic
will be fun watching the system stats though lol.
I could also send some to TapaniS.
-Josh
I think it's a good idea - to distribute the files to several key uploaders first, who will then share them by eMule.
Re: The 16 "missing" Nalimov files
Posted: Thu Jul 27, 2006 4:58 am
by jshriver
mbourzut wrote:I have created the 16 missing Nalimov tablebases for 33p endings by conversion from Johan de Koning's FEG data. The total compressed size is about 130GB, with kqpkrb being the largest at about 17GB, and kppkpp the smallest at about 1 GB. The original FEG data is about 90 GB, smaller partially because of larger block size (32k as opposed to 8k), and a more efficient compression algorithm.
I have run verification on a few, but not all the endings.
If no other source is forthcoming, I'd be willing to put these on a disk and send to someone in the US who has all the existing datasets and can run a complete verification before distributing more widely. Verification is reasonably fast, kqpkrb took about 1.5 days on a 3.8 Ghz machine, but the other endings will run a lot faster.
Would your code be able to convert 5-1 FEG data to nalimov? While in another thread it was mentioned that the tbgen doesnt support 5-1.. does the code which reads the data.
After a lot of digging I finally found feg on ubisofts ftp site. I'm generating the 5man knnnk right now and will start knnnnk when this finishes. Assuming it works
-Josh
Re: The 16 "missing" Nalimov files
Posted: Thu Jul 27, 2006 6:11 am
by vb4
jshriver wrote:mbourzut wrote:I have created the 16 missing Nalimov tablebases for 33p endings by conversion from Johan de Koning's FEG data.
I was reading this page:
http://www.aarontay.per.sg/Winboard/egt ... masteredgb
It mentions that the Koning egtb only stores wins from the white point of view. Will this effect the quality of the nalimov conversion? Will we have the information for black's side?
Hi Josh,
You must have been reading my mind regarding the btm stats. Do you or anyone reading this know if we will be able to get this missing data?? It seems as though all the rest of the files I had looked at had both WTM and BTM info. Any feedback regarding this would be most appreciated.
Take care,
Les
On a side note this quote seems promising:
"The generator also makes generating 6-man with KWWWWB possible on a 32 bit computer possible , but that requires about 300MB (which isn't a lot by todays standards) of free RAM"
I'm interpreting this as being able to generate 5-1 egtbs
-Josh
Few more questions with these 16 egtb's (Mark?)
Posted: Thu Jul 27, 2006 6:48 am
by vb4
I just finished looking at all 16 egtb's that Mark was generous enough to generate and have a few questions. As mentioned in anearlier post there is no BTM data for kppkpp and hope that someone here can resolve if we can get that info. Secondly the following 3 egtb's have all the data except the last line which is broken positions for BTM (I think or atleast I hope it was just clipped off the file when it was zipped) Can someone fill me in about these 3 egtb's krnkpp - krbkpp - kbnkpp. I know it has nothing to do with the functionality but for my research it is important.
Thanks in advance,
Les
Re: Few more questions with these 16 egtb's (Mark?)
Posted: Thu Jul 27, 2006 7:59 am
by Kirill Kryukov
vb4 wrote:I just finished looking at all 16 egtb's that Mark was generous enough to generate and have a few questions. As mentioned in anearlier post there is no BTM data for kppkpp and hope that someone here can resolve if we can get that info.
BTM statistics for symmetrical endgames is identical to WTM statistics. Also, in my collection of TBS files there are symmetrical sets with missing BTM info (eg. KRNKRN). If you need to have separate WTM and BTM info, you can simply duplicate the WTM stats.