CCRL Discussion Board

Hi,

I've been out of computer chess for a few years, but I am trying to get back up to speed.
I am still thrilled about endgame tablebases, so this new format is of great interest to me.

I am no C programmer, nor familiar with linux, and/or gcc, so a simple question: Is there anybody who has windows 64-bit binaries of the generator, that I could get my hands on ?
I dont know if it is even possible to compile it into a windows executable, but until I am told otherwise, I am hopeful

Greetings

Dan Wulff

It's possible to compile windows executables, see the readme:
https://github.com/syzygy1/tb

The attached tb.zip file, if it works (maybe executables are blocked, but let's see), contains rtbgen.exe and rtbgenp.exe which should work on CPUs with popcount support. It also contains the run.pl script which requires a Perl installation. I've tested with ActiveState Perl.

See the readme for instructions. If you manage to generate 3-4-5-piece tables using the script, count on about 100x as long for the generation of all 6-piece tables. The script will skip the tables that have already been generated, so you can interrupt the generation halfway and continue later.

When you're done, check your files using fsum and these checksums:
http://kirill-kryukov.com/chess/tablebases-online/

It's probably a good idea to verify the 3-4-5 piece tables before starting 6.

You need 16 GB for generating the 6-piece tables. Use "--disk" if you have 16 GB, you can do without it if you have 24 GB or more. Using --disk will not take drastically longer (it will just save some data to disk between generation and compression).

Awesome!, thank you very much!!

Hi again,

I have made my own winodws batch file to make the tables.
That generator is FAST!

Verifying the generated tables is good... Only problem for me is that the package you attached only contains the generator, and the verifyer is a different program if I read the Perl script correctly.
Would you by chance have the verifier in a windows version, or am I just not reading the scirpt correctly ?

syzygy wrote:It's possible to compile windows executables, see the readme:
https://github.com/syzygy1/tb

The attached tb.zip file, if it works (maybe executables are blocked, but let's see), contains rtbgen.exe and rtbgenp.exe which should work on CPUs with popcount support. It also contains the run.pl script which requires a Perl installation. I've tested with ActiveState Perl.

See the readme for instructions. If you manage to generate 3-4-5-piece tables using the script, count on about 100x as long for the generation of all 6-piece tables. The script will skip the tables that have already been generated, so you can interrupt the generation halfway and continue later.

When you're done, check your files using fsum and these checksums:
http://kirill-kryukov.com/chess/tablebases-online/

It's probably a good idea to verify the 3-4-5 piece tables before starting 6.

You need 16 GB for generating the 6-piece tables. Use "--disk" if you have 16 GB, you can do without it if you have 24 GB or more. Using --disk will not take drastically longer (it will just save some data to disk between generation and compression).

Wulff wrote:Verifying the generated tables is good... Only problem for me is that the package you attached only contains the generator, and the verifyer is a different program if I read the Perl script correctly.

Correct, but it is not worth it to run the verifier. Just check the md5sums with fsum and the checksums on Kirill's site. If they don't match, something is wrong.

The verifier takes much longer to run. It was useful to convince myself that the generator was correct (even though it shares code and ideas with the generator, but as I said, it was about convincing myself ).

ok, I'll do that, then. Thanks.

The generator works like a charm, and I have all 3+4+5 man classes generated. Good work on your part.

I tried generating KQQVsKQQ on my 12 GB system, but the swapping immediately bogged my PC down, so I had to give up on that

A couple things, I've been thinking about:

From the generation output I saw that you seemed to generate the endings with pawns in them in stages, and saving the output in temporary files, then merging them at the end of the run.
Would it be possible, and feasible to expand on that method, and generate e.g. the 6-man pieces in steps of, say, one file per square of e.g. the white king, thus saving a lot of RAM, and possibly taking the black king into account too, and that way opening up for 7-man base generation without having a system with several hundred gigabytes of RAM ?
I am aware that generation would be slower, but as to how much, maybe you can elaborate on that, based on your experience with the disk-option in your currenct generator.

Not that I am saying you should do that, I am simply curious as to the possibility ...

Why do you have two separate generators? isnt most of the code the same, and it could be one program with two functions in it, instead of switching back and forth between the two?

One "feature" that could be real nice would be the one where it detects which bases are required for a given class, then check whether that class already exists, and if not generates those subclasses first.
On second thought, that would only work if it was one program, but then all 3+4+5 man classes could be generated with just two commands: rtbgen KPPVsKP and rtbgen KPPPVsK.

Well, I dont even know if you are planning to do anything more with the generator, so I hope you will forgive me, if you think I am overstepping my bounds.

Wulff wrote:From the generation output I saw that you seemed to generate the endings with pawns in them in stages, and saving the output in temporary files, then merging them at the end of the run.
Would it be possible, and feasible to expand on that method, and generate e.g. the 6-man pieces in steps of, say, one file per square of e.g. the white king, thus saving a lot of RAM, and possibly taking the black king into account too, and that way opening up for 7-man base generation without having a system with several hundred gigabytes of RAM ?

Pawn endings with the ("leading") pawn in different files are completely independent, so can be generated separately without any complications. Pawns cannot move from one file to another, unless by capture into an already generated table. With kings this is completely different, as kings can travel all over the board and back without any captures taking place.

So it is not possible to independently generate tables per "king square" and merge them together afterwards.

I am aware that generation would be slower, but as to how much, maybe you can elaborate on that, based on your experience with the disk-option in your currenct generator.

The disk option just frees up some RAM between generation and compression. The generation remains RAM-based.

Other algorithms may be more suitable for disk-based generation. Not to do disk-based generation was a fundamental design decision for my generator.

Why do you have two separate generators? isnt most of the code the same, and it could be one program with two functions in it, instead of switching back and forth between the two?

It's easier not to mix the generation code for non-pawn and pawn tables in one program. The generation routines are similar in structure, but not the same. It is not difficult to invoke the correct generator from a script, so there is little to gain by going through pains integrating both executables into a single executable.

One "feature" that could be real nice would be the one where it detects which bases are required for a given class, then check whether that class already exists, and if not generates those subclasses first.

That could also be done at a scripting level.

syzygy wrote:

Wulff wrote:From the generation output I saw that you seemed to generate the endings with pawns in them in stages, and saving the output in temporary files, then merging them at the end of the run.
Would it be possible, and feasible to expand on that method, and generate e.g. the 6-man pieces in steps of, say, one file per square of e.g. the white king, thus saving a lot of RAM, and possibly taking the black king into account too, and that way opening up for 7-man base generation without having a system with several hundred gigabytes of RAM ?

Pawn endings with the ("leading") pawn in different files are completely independent, so can be generated separately without any complications. Pawns cannot move from one file to another, unless by capture into an already generated table. With kings this is completely different, as kings can travel all over the board and back without any captures taking place.

So it is not possible to independently generate tables per "king square" and merge them together afterwards.

Can you estimate how much RAM and time is needed for 7-men tables generated?

EvgeniyZh wrote:Can you estimate how much RAM and time is needed for 7-men tables generated?

About 1 TB of RAM.
Generation time probably about 64x as long per table.

The current generator would need a few modifications to be able to generate 7-men tables. Mostly some internal limits, which is easy, but something needs to be done as well to handle very large DTZ values. That only affects a few tables, though. (My idea is to simply divide them by 2 for DTZ > 50. Then everything still fits.)