5-1 EGTB generation revisited

[Martin Kreuzer]

Hi all,

since all 4-2 and 3-3 EGTB are now well shared, I have restarted
some attempts to compute the 5-1 EGTB. I have computed
the endgames

kqqqqk, krrrrk, kbbbbk, knnnnk, kqrbnk, kqqqrk, kqqqbk, and kqqqnk

using FEG. Currently I am computing kqqqpk to see if endgames with
pawn are feasible. I am using my laptop (Athlon 64 bit mobile, 1 GB memory). The earlier endgames took 1/2 day -- 1 day each.

If you are interested in joining this effort, there are plenty of things
that remain to be done:

1) Compute the other 61 endgames of type 5-1.
It would be nice to coordinate the computation, e.g. via this forum.

2) Share the results. Anybody with web space or ed2k capacities?

3) Transfer the results to Nalimov EGTB format.
Marc Bourzutschky has performed this type of computation before,
so it should be possible.

4) Adjust the code of tbgen (more precisely, tbindex.cpp) such that
chess programs are able to access these data bases.

The main problem with 3) and 4) is probably that one has to write
a c++ function which creates all positions of 4 pieces and
numbers them according to the scheme in the paper

http://www.is.reading.ac.uk/common/publ ... /02127.pdf

This looks certainly doable, but requires more c++ skills than I have.
Volunteers are welcome!

Greetings,
Martin Kreuzer

[Kirill Kryukov]

Hi Martin, this is very nice! I can probably help with step 2 (sharing). Do you have approximate estimate how much space the 5-1 will take?

[guido]

Hi Martin, this is very nice! I can probably help with step 2 (sharing). Do you have approximate estimate how much space the 5-1 will take?

With my indexing scheme the total space amounts at 616.7 Gb uncompressed. Probably something less (10-15 %) with Nalimov format.
CPU time and RAM could be a problem for endgames containing a pawn and three different pieces (e.g. krbnpk), whose dimensions are about 17.4 Gb uncompressed for each.

Good luck!
Guido

[Martin Kreuzer]

Dear Kirill, dear Guido,

the computation of the endgame kqqqpk is now finished, too.
It took about 1 day. The 9 endgames have the following sizes:

kqqqqk: 55.7 MB
krrrrk: 61.4 MB
kbbbbk: 129 MB
knnnnk: 156 MB
kqrbnk: 990 MB
kqqqrk: 106 MB
kqqqbk: 129 MB
kqqqnk: 130 MB
kqqqpk: 395 MB

So, the total space for the 70 endgames should be
around 30-40 GB. (This is a guesstimate.)

Kirill: Is it o.k. if I upload the data files of the nine endgames
to your ftp-area? Can I use my old login info?

Since there was some interest in earlier threads, I have attached
a file stats.zip which contains the statistics of the 9 endgames
I did so far. They are in the formats *.log, *.lof, and *.los,
as explained in feg.txt.

Best wishes,
Martin

[Martin Kreuzer]

Hi all,

just in case: if you want to start using feg.exe, you can download
it from

ftp://ftp.ubisoft.com/games/chessmaster/other/Feg.zip

The zip-file also contains the file feg.txt I mentioned in my previous post.

Greetings,
Martin Kreuzer

[Kirill Kryukov]

Kirill: Is it o.k. if I upload the data files of the nine endgames
to your ftp-area? Can I use my old login info?

Yes, please try to upload. I made new FTP account egtb51 (password: egtb51) at kd.lab.nig.ac.jp for this. I am afraid it may be slow, but please try!

Basically I am more curious about Nalimov format files and updated probing code, but I don't have time now to help with the code.

Thanks for your efforts!

[Martin Kreuzer]

Hi all,

I have computed five additional 5+1 egtb over the weekend:
kqrrrk, krrrbk, krrrnk, krrrpk, and kqbbbk.
The stats files are attached. I do not know whether I read them
correctly, but possibly there are some longer mates (>20 moves) in krrrpk.
I do not know how to find these positions in the database, though.

If anybody is interested in joining this effort, please let me know
so that we can distribute the computation.

Greetings,
Martin Kreuzer

[vb4]

Dear Kirill, dear Guido,

the computation of the endgame kqqqpk is now finished, too.
It took about 1 day. The 9 endgames have the following sizes:

kqqqqk: 55.7 MB
krrrrk: 61.4 MB
kbbbbk: 129 MB
knnnnk: 156 MB
kqrbnk: 990 MB
kqqqrk: 106 MB
kqqqbk: 129 MB
kqqqnk: 130 MB
kqqqpk: 395 MB

So, the total space for the 70 endgames should be
around 30-40 GB. (This is a guesstimate.)

Kirill: Is it o.k. if I upload the data files of the nine endgames
to your ftp-area? Can I use my old login info?

Since there was some interest in earlier threads, I have attached
a file stats.zip which contains the statistics of the 9 endgames
I did so far. They are in the formats *.log, *.lof, and *.los,
as explained in feg.txt.

Hi Martin,

I am not familiar with the above 3 formats! Can any of them provide me with simialr output as Eugenes stat files?? If so perhaps you can shed some light on this for me. I very much appreciate you generating these last remaining 5-1 files.

Thanks,

Les

Best wishes,
Martin

[vb4]

Hi all,

I have computed five additional 5+1 egtb over the weekend:
kqrrrk, krrrbk, krrrnk, krrrpk, and kqbbbk.
The stats files are attached. I do not know whether I read them
correctly, but possibly there are some longer mates (>20 moves) in krrrpk.

Hi Martin,

I assume the stat files for these like the previous set you generated are in one of those 3 file formats?? If so I will wait here patiently for your response to my other message.

Thanks,

Les
I do not know how to find these positions in the database, though.

If anybody is interested in joining this effort, please let me know
so that we can distribute the computation.

Greetings,
Martin Kreuzer

[Pachnes]

Hi Kirill,

please send me the link and the password to your ftp-site. I will start to share the files as soon, when they are checked and as soon you will have released them.

[Kirill Kryukov]

Hi Kirill,

please send me the link and the password to your ftp-site. I will start to share the files as soon, when they are checked and as soon you will have released them.

Hi Thomas!

Yes, please try to upload. I made new FTP account egtb51 (password: egtb51) at kd.lab.nig.ac.jp for this. I am afraid it may be slow, but please try!

Please note that the files in question are not in Nalimov format, so only Chessmaster can use them. That's why I did not make any announcements and did not add them to the project page.

Good to see you joining!

[Martin Kreuzer]

Hi Martin,

I am not familiar with the above 3 formats! Can any of them provide me with simialr output as Eugenes stat files?? If so perhaps you can shed some light on this for me. I very much appreciate you generating these last remaining 5-1 files.

Thanks,
Les

Hi Les,

the EGTB I am computing are in the "Chessmaster" format.
The meaning of the statistics files is explained in the file feg.txt
which is contained in the file feg.zip. The link to feg.zip
is contained in my earlier post.

The problem with generating these EGTB in Nalimov style
files is that -- as far as I know -- nobody has so far written the
code to generate the correct numbering for the positions
in the way described in the paper by Nalimov et al. (see the
link in my earlier post).

Greetings and best wishes,
Martin

P.S.: When you reply to a post, please quote only the original
post (or parts of it). Including your reply in the quote makes it hard
to read.

[guido]

Dear Kirill, dear Guido,

the computation of the endgame kqqqpk is now finished, too.
It took about 1 day. The 9 endgames have the following sizes:

kqqqqk: 55.7 MB
krrrrk: 61.4 MB
kbbbbk: 129 MB
knnnnk: 156 MB
kqrbnk: 990 MB
kqqqrk: 106 MB
kqqqbk: 129 MB
kqqqnk: 130 MB
kqqqpk: 395 MB

So, the total space for the 70 endgames should be
around 30-40 GB. (This is a guesstimate.)

Kirill: Is it o.k. if I upload the data files of the nine endgames
to your ftp-area? Can I use my old login info?

Since there was some interest in earlier threads, I have attached
a file stats.zip which contains the statistics of the 9 endgames
I did so far. They are in the formats *.log, *.lof, and *.los,
as explained in feg.txt.

Best wishes,
Martin

Hi Martin,

I generated all the above mentioned endgames except kqrbnk on an Athlon64 3800+ PC.
Compressing the files under Linux with gzip (using the default option -6), I obtain lesser values than yours except strangely for kqqqpk. The values are the sums of both the binary files (wtm and btm).
In the last column I put the elapsed time in seconds to generate the TB.

kqqqqk 55.7 MB 26.5 MB 2045 s
krrrrk 61.4 MB 27.5 MB 2257 s
kbbbbk 129 MB 93.8 MB 3727 s
knnnnk 156 MB 131.3 MB 3060 s
kqrbnk 990 MB not done
kqqqrk 106 MB 88.5 MB 9571 s
kqqqbk 129 MB 118.7 MB 9708 s
kqqqnk 130 MB 110.6 MB 10269 s
kqqqpk 395 MB 520.1 MB 32939 s

What was the elapsed time for kqrbnk?
Some differences in the dimensions of the compressed TBs are very strange to explain; the compression factor for certain cases is very high (about 20) so your prevision of only 30-40 GB could be likely.

About the endgame krrrpk, there are not positions with > 20 moves to mate. The maximum distance to mate are:

WTM : 2 positions with 8 moves
BTM : 9 positions with 14 moves

If you are interested I have no problem to extract the list of these cases in FEN format.

Ciao
Guido

[Martin Kreuzer]

Hi Guido,

thanks for your infos. I did not write down the times it took to
compute the endgames, but they were all significantly longer than your timings, more or less between 30 ksec and 55 ksec. The endgame
kqrbnk took about the same time. The hardest computation so far
was krrrpk which took about 3 days.

Thanks also for the clarification w.r.t. the depth of krrrpk; now I understand
that the last positions resolved in round 28 correspond to the ones which
are mate in 14 (you can see this info in KRRRPK.log).

Meanwhile I have computed four more EGTB:
krbbbk, kqnnnk, kbbbnk, and krnnnk.
The timings were all about 40-60 ksec.
The files are on Kirills ftp account, together with the stats files
and md5sums.

It would be nice to produce a conversion tool between your
egtb and the ones I am computing. In this way we could verify each
others results. Unfortunately I do not know the numbering scheme
for the positions used in the Chessmaster format.

Best wishes and kind regards,
Martin

[Martin Kreuzer]

Hi Guido,

I just discovered that the timings are indeed contained in the stats files I posted. The computations were mostly done on my Athlon64 Mobile Laptop
and some on my P4 3400 MHz.

1) KBBBBK 74773 sec
2) KBBBNK 55203 sec
3) KNNNNK 109580 sec
4) KQBBBK 46049 sec
5) KQNNNK 63944 sec
6) KQQQBK 34017 sec
7) KQQQNK 33869 sec
KQQQPK 138464 sec
9) KQQQQK 48754 sec
10) KQQQRK 32251 sec
11) KQRBNK 32369 sec
12) KQRRRK 39223 sec
13) KRBBBK 48568 sec
14) KRNNNK 69202 sec
15) KRRRBK 31972 sec
16) KRRRNK 31149 sec
17) KRRRPK 227658 sec
18) KRRRRK 53058 sec

Currently I am trying to compute KBBBPK. It seems to be quite hard.

Cordial greetings,
Martin

[guido]

Hi Guido,

I just discovered that the timings are indeed contained in the stats files I posted. The computations were mostly done on my Athlon64 Mobile Laptop
and some on my P4 3400 MHz.

1) KBBBBK 74773 sec
2) KBBBNK 55203 sec
3) KNNNNK 109580 sec
4) KQBBBK 46049 sec
5) KQNNNK 63944 sec
6) KQQQBK 34017 sec
7) KQQQNK 33869 sec
KQQQPK 138464 sec
9) KQQQQK 48754 sec
10) KQQQRK 32251 sec
11) KQRBNK 32369 sec
12) KQRRRK 39223 sec
13) KRBBBK 48568 sec
14) KRNNNK 69202 sec
15) KRRRBK 31972 sec
16) KRRRNK 31149 sec
17) KRRRPK 227658 sec
18) KRRRRK 53058 sec

Currently I am trying to compute KBBBPK. It seems to be quite hard.

Cordial greetings,
Martin

Hi Martin,

Your infos about the times spent to generate TBs are very interesting but not easy to be explained.
I have to premise that my program is not very fast in general (the time of about 5 days to generate all the 5-men TBs has not been considered short by some readers of my site).
My first impression is that the times of the FEG are much less dependent on the endgame than mine.
Consider for example KQQQQK where FEG spends 48754 s versus my 2045 s. I can explain easily why I
spend so little. The dimension of each of the two files is about 257 MB, so all are in RAM, and the program does only 4 cycles (from 0 to 3) of computation of which 3 are executed on one file only (this point is characteristic of all 5-1 TBs because the black never wins in these endgames).
On the contrary considering KQRBNK the two files are about 17 GB each and the number of cycles must be
greater because in the first move the black king can capture the white queen and therefore the endgame becomes KRBNK that requests much more than 4 cycles (30 cycles from 0 to 29). I think that my program will spend some days or more to compute this endgame. So it is very strange that the times for FEG are of the same order in these two extreme cases.
The only hypothesis is that FEG during computation considers both the endgames KQQQQK and KQRBNK as
having the same number of positions and only in a second phase reduces the positions in the first by a factor of 24 = 4!. In this case FEG would be very fast if compared with my program and this would be evident in the most complex endgames.

A comparison of my results with those of FEG and/or Nalimov would be very interesting but I don't know if FEG TBs are accessible by private programs or its indexing scheme is known.

My best
Guido

[Martin Kreuzer]

Hi Guido,

in an earlier thread Marc Bourzutschky wrote:

These results are quite reasonable, especially if there were many like pieces in the ending, as FEG does not implement permutation symmetry until the final compression stage.

-Marc

Therefore your conjecture

The only hypothesis is that FEG during computation considers both the endgames KQQQQK and KQRBNK as
having the same number of positions and only in a second phase reduces the positions in the first by a factor of 24 = 4!. In this case FEG would be very fast if compared with my program and this would be evident in the most complex endgames.

seems to be correct. I wonder if anybody here knows the
details of the format of the files generated by FEG?

Ciao, best wishes,
Martin

[Martin Kreuzer]

Hi all,

today I have uploaded to egtb51 four new EGTB which I computed:

kbnnnk, kqqrbk, kqqrnk and kqqrrk.

There are also stats files and md5sums.

Greetings,
Martin Kreuzer

[Martin Kreuzer]

Hi all,

I have computed and uploaded four new EGTB:
kqqbbk, kqqbnk, kqqnnk, and kqqrpk.
The total number is now 26, all of them with stats files
and md5sums.

Greetings,
Martin Kreuzer

[clocks]

I would be willing to commit to generating some and uploading the results that I get, I messed around with the FEG generator, more than easy enough.

If you want to post which ones would be good for me to do, just a handful at a time would be great, I could do a set, and then complete that set doing the one with a pawn, etc. Whatever would be helpful.

Right now I am limited on disk space until I get my usb enclosures in.

I am on about the same laptop it sounds like, athlon 64, 1GB memory. Been parting off my old system that died on me (Dual MP 2800 2GB memory), but I'm sure even a few of these could speed up the process.

Got some various bills I need to take care of, so hopefully have a better rig setup as far as a workstation goes by febuary, think quad AMD dual-core, really. So stashing money aside for that little project.

Upgraded the internet, upload capacity over emule should be about 350k/sec I can dedicate soon, hopefully in the next day or two. I have 150k/sec dedicated at the moment.

Let me know,

Derek

[Martin Kreuzer]

Hi Derek,

thank you very much for your offer to help with 5-1 EGTB generation.
Possible targets of your computation could be:

(1) kqrnnk, kqbnnk, and then (using the endgames kqqnnk and kqnnnk
which I already posted) also kqnnpk.

(2) krbnnk, and then krnnpk (using kqrnnk from (1) and the endgames
krrnnk [to be posted today] and krnnnk [already posted]).

These five computations should be doable in less than 1 week CPU time
(1/2 to 1 day for each).

If you want a "challenge", try kbbbpk. (I was not able to finish this one.
It takes several days. The necessary subendgames are already posted.)

Greetings, Martin

[Martin Kreuzer]

Hi all,
now I have computed and posted four new 5-1 EGTB:

kbbnnk, krrbbk, krrnnk and kqqbpk.

They are complete with stats files and md5 sums. The total count is 30, and halftime is approaching quickly!

Greetings, Martin Kreuzer

[clocks]

Hello,

I will start these as soon as I can just waiting on some items to arrive in the mail

Will let you know,

Derek

[vb4]

Hi Martin,

Listen I was just wondering if there is a way of working on your stat files in such a way that I could convert them into a format that would look like Eugenes stat file format? I am doing a very intense study on EGTB's but need the stat files to be in a format like Eugenes stat files. Hoping that there is a way to do this.

Looking forward to your response,

Les

[Martin Kreuzer]

Hi Les,

the information you are looking for is partly contained in the statistics files produced by FEG.EXE.
For your information, I have appended the file FEG.TXT. Please look at Section XV there ("Statistics Lines Explained").
It says that Column 5 contains the number of positions resolved by a certain pass, i.e. the number of positions with a certain number for the ply-to-win depth.
However, the ply-to-win depth may not be the ply-to-mate depth. It seems that the DTM (distance to mate) numbers are stored in the database, however. To get lists of all the positions won in certain number of plys you should use the command FEG -LL (I have not yet used this command, so I cannot comment on it.)

Hope this helps! Best wishes and good luck,
Martin