Generator of 3-4-5-6-7-chessman tablebases.

[Skipper_NORTON]

Chess EndGame Analyzer. Generator of 3-4-5-6-7-chessman tablebases.
http://generatorchess.com/
Format - DTM, 7-chessman tablebases can be generated using 4 GB of memory (maximum - 14 GB memory required).

[redpawn]

IS THIS A NEW VERSION OR IS IT THE SAME VERSION

Secondly: could you please elaborate more about this please

No Description what so ever is really not good

[Ed Trice]

From the enclosed documentation"

Retrospective_Chess_Generator, ChessDLL, RetroChess3 are 3 projects for chess game.
The programs and dll require OS Windows. Retrospective_Chess_Generator is a 64-bit application,
therefore it can work only in Windows 64-bit (e.g. Windows XP x64, Windows Vista x64, Windows
Server 2003 x64, etc.). ChessDLL and RetroChess3 are 32-bit.

Description of Retrospective_Chess_Generator3.

This application generates tablebases for chess endgames. By default generated bases are stored in
/Bases folder. When there are less than 8 chessmen on a chessboard, it's comparatively easy to
analyze every possible position to see, if the position is drown or one of the side will win and
how many steps it will take to win. In order to create a 7-men tablebase, the application would
require more than 2 GB of memory (RAM), however 32-bit application cannot allocate this amount of memory.
That is why Retrospective_Chess_Generator3 is a 64-bit program.

Once the tablebase is built, other applications can use the tablebase to analyze all the possible
moves from a given position and choose the best move. A program that uses such a tablebase, plays
faultlessly and it isn't possible to play stronger. It is proved, as mathematical theorem.
Retrospective_Chess_Generator can build tablebases of 3, 4, 5, 6, or 7 chessmen. To start with,
you need to select an endgame (e.g. "5_KRB-KR") and start analysis using a Build button. As a result,
a tablebase will be generated. The file has .tby extension, in our case it will be named 5_KRB-KR.tby.
It means all positions of "Rook and Bishop against Rook" will be analyzed. In it a endgame, superfluous
only Bishop, but white can win from many positions.

Designation: K - King, Q - Queen, R - Rook, B - Bishop, N - Knight, P - Pawn. RetroChess3 requires
at least all 3-4-man tablebases (40 MB). The program Retrospective_Chess_Generator3 is optimized for
the best "use of RAM / time of generation" ratio.

For example, many 7-man tablebases can be generated using 4 GB of RAM (for generation of any tablebases
will not need more than 14GB); 6-man tablebases need only 300 Mb, etc. This optimization is extremely
important, because creation of all 7-man tablebases will takes several years (if for generation use
many computers). (Although, 5- and 6-men tablebases are generation much faster). That is why
Retrospective_Chess_Generator3 features advanced optimizations. Ready tablebases are not compressed
as much as well known Nalimov's tablebases. The difference is about 20%. E.g., all 5-men tablebases
take 7.1GB in 'emd' format, ant 9.1GB in the format of this program ('.tby'). However, using WinRAR,
you can get 'tby' tablebases compressed even more than Nalimov's tablebases. The tablebases can be
converted from one format to the other comparatively quickly using ChessDLL.dll (the computer with very
considerable quantity of memory is necessary).

At the moment of writing of given article (2010) tablebases in other formats, including tablebases
of Nalimov, exist only for positions with 3-6 chessmen. If built 7-man tablebase, chess engines can
use them (Game will be stronger and in positions with 9-10 chessmen. Because of search singularities).
Also it is possible to create a site, for the analysis of different positions, and it is possible to
analyze the chess endgames, which represent the big theoretical interest.

All programs presuppose that endgame tablebases already exist, either in 'Bases' folder, or in the
directories listed in a configuration file "Bases\DirectoriesForEG.td". The directories paths should
look like this:
D:\path1\
D:\path2\
These are absolute paths. The correct format - each path on a separate line, no white spaces are allowed.
The length of a path cannot exceed 1000 chars, and there cannot be more than 100 paths. A path should
end with '\'. During tablebase generation, temporary files (with extensions ".tb _" and ".tb __" )
will be in the 'Bases' directory. The hard disk must have enough free space (The program will show at
a type selection for generation). It should be formatted to NTFS, since FAT/FAT32 don't allow files
of more than 4Gb (The such can be necessary). Minor end-games - can be moved anywhere.
The button "Stop and save" works only after 2->3 iteration. If your system doesn't have enough RAM,
it will take 10-100 times more to generate the tablebases due to work with swap file. The program
cannot load the CPU up to 100% because the bottleneck is HDD performance. That is why it doesn't make
sense to run 2 instances of this program on the same HDD. But if your CPU has 4 cores, it does make
sense to attach 4 HDDs and run 4 programs simultaneously. Also, generation will faster, if HDD unite
in RAID.At the moment of writing of given article (2010), there are HDD in volume to 3 ÒÁ
(and it is possible to unite to 5 HDD, having received necessary space), and it is possible to
deliver 16 GB memory in the normal personal computer. The recommended way to stop tablebase generation
is to click "Stop and save", or "Stop and delete". This will help to avoid possible failures
(but the probability of failure is small and without click "Stop and save"). In the course of generation,
make sure your system doesn't have hardware errors. A bad block on your HDD will make the tablebase
corrupt. You can use Windows System Events console to monitor hardware errors.


Description of ChessDLL.dll.
This project is created to make it possible to use generated tablebases for other programs. The dll has
a function 'GetEvaluateFromFEN' that accepts a position as a line, in FEN format. E.g.
"4k3/8/8/Pp6/8/8/8/4K1Q1 w - b6", or "4k3/8/8/Pp6/8/8/8/4K1Q1 w". If there is no designation of pawns
after "w" or "b" (white to move or black to move), the last should must be a symbol - 0, after 'w' or 'b'.
Correct values are: from -1999, up to 1999. If the library returns -2000 - the position is illegal, or
the tablebase doesn't exist. It is impossible to pass an illegal position to the dll.

Description of RetroChess3.
This is a chess program. Allows to analyze and search for chess positions in endgame if the corresponding
tablebase is available. The program plays from any position even if there is no tablebase, but it's not
strong, since it's been developed as a shell for endgames tablebases. Current version doesn't have a well
optimized evaluation function.

[redpawn]

Hi Ed:

I Have an older version of the program

That's why I've asked my questions

Best Regards,

[Ed Trice]

Chess EndGame Analyzer. Generator of 3-4-5-6-7-chessman tablebases.
http://generatorchess.com/
Format - DTM, 7-chessman tablebases can be generated using 4 GB of memory (maximum - 14 GB memory required).

I started running the program on my 3.9 GHz i7-860 (see below)



I only selected up to 5-pieces to test its speed. It seems to be solving rather quickly. It is working on 5_KBN_KQ after only 1 hour 5 minutes.

Maybe we could run the entire thing on a specialized 5.0 GHz Gulftown with several large drives to tackle the 7-piece?

We'd need to make some changes to the program first.

1) The interface is not user friendly. You have to click on every ending you wish to solve. The other generators, you specify only the last slice (with all pawns) and it automatically figures out which databases need to be solved, and those are solved first.

2) There is no automatic use of multiple cores that I can tell right now. The program should do other databases that have no dependence on one another, like 3 vs. 2 and 4 vs. 1. For 7 pieces, 4 vs. 3, 5 vs. 2, and 6 vs. 1 can all be solved at the same time.

Further parallelization is also possible, but this gets complicated. But, to save months or years of time, it is worth the additional effort.

If you guys want me to build a single machine with 12 cores (24 threads) with lots of RAM, overclocked to 4.8 GHz or faster, I'd be willing to do it. I would then make the endgames available to whoever wants them.

Talk it up, and decide among yourselves.

[koistinen]

I could not get download to work, but from the posted documentation it seems to be a program for windows that is limited mainly by ram size and disk speed.
If anyone has a link to the source I would like to have a look at it.
I don't understand you about the speed. Do you mean a five man ending takes about 1 hour to compute?

[Ed Trice]

I could not get download to work, but from the posted documentation it seems to be a program for windows that is limited mainly by ram size and disk speed.

I would not say "limited", I would say it makes better use of available RAM. I can tell as I watch it computing that it uses intelligent reverse move generation functions, since each iteration gets faster and faster, and it moves most quickly towards the end. Other db solvers iterate over all of the positions with each pass, this one does not. It uses only the subset of previously solved DTW positions to look at the positions that could play into those (via reversing the moves) and therefore it gets the job done fast.

I don't understand you about the speed. Do you mean a five man ending takes about 1 hour to compute?

No, I mean it solved all of the tablebase slices up to that point in only 65 minutes.

KNN vs. N
KNN vs. B
KNN vs. R
KNN vs. Q

KBN vs. N
KBN vs. B
KBN vs. R
KBN vs. Q

All of those 5-piece endings (don't forget each side has a king) were done in 65 minutes total.

Apparently it solves the pawn slices at the very end, and not as "soon as possible", otherwise it would have done the tablebase for KNN vs. P before KBN vs. N.

[koistinen]

Ah, so speed is about 8 min/ending. That means 7-man should take about 1 month/ending.
Managed to download after figuring out there was a captcha. No source code for generator as far as I can see.

[Ed Trice]

That means 7-man should take about 1 month/ending.

No, much faster than that. The tb5 speed was on a 3.9 GHz system, not parallelized. The 7-piece was being spec'd on a system with 12 processors, 24 threads, each at 5.0 GHz, and would take about 91 months. There's a great deal more than 91 endings in total.

[redpawn]

Could Any Verify:

That the HDD usage in this software is so much for generation

is it practical at all to use this software

The probing interface seems bad

In fact it's almost user hostile

[koistinen]

Ok, 4 years then. That is reasonably quick. I recommend testing the program on commodity hardware first too see how much the cpu gets used when computing a 7-man. The computation does not even need to finish to do this.
If the system you think of building has 12 processors, I'd recommend using motherboards with 8 sata ports each. That way you can attach 96 drives at once without extra io cards.

Does this look right?

Code: Select all

6-1: 126
5-2: 350
4-3: 525
3-4: 525
2-5: 350
7-man total: 1876
5-1: 70
4-2: 175
3-3: 225
2-4: 175
6-man total: 645
4-1: 35
3-2: 75
2-3: 75
5-man total: 185
3-1: 15
2-2: 25
4-man total: 40
2-1: 5
3-man total: 5

[Ed Trice]

The probing interface seems bad

In fact it's almost user hostile

Sometimes the smartest programmers make the worst graphical user interfaces

Maybe if they publish the format and the accessing code, someone can create a decent gui for it.

[redpawn]

I'd be Happy if this guy speaks ever again in this forum

He Announces & disappear

& That's not good

I don't know why but I think Mr. Balicora is still our best hope for 6 & 7 men EGTB

If you have what it takes ED why not try creating a GUI to manipulate GAVIOTA EGTBs

& by manipulating I mean generate, check, compress & even some FEN probing

[Ed Trice]

If you have what it takes ED why not try creating a GUI to manipulate GAVIOTA EGTBs
& by manipulating I mean generate, check, compress & even some FEN probing

Well heck that covers just about everything now, doesn't it

I did do some endgame database work with Marc Bourzutschky on the game of Gothic Chess (10x8 board with a Bishop/Knight piece and a Rook/Knight piece). He solved the 5-piece set in 2005, here are some of the more interesting wins:

http://www.gothicchess.com/javascript_endings.html

I did go back and re-index these endgames, with my own move generator and DTW logic, and I found a way to remove superfluous positions and improve the compression. I construct only legal positions, without the "broken" labeling that you see in the other .tbs files. It's tricky, but it has huge benefits. It does slow down the indexing function calls, but the benefit is much lower RAM overhead.

I guess I could produce (yet another) database format through 7 pieces, but you're talking about a huge project.

[guyhaw]

There seems little point in investing in cutting-edge processors to generate EGTs when the activity is essentially I/O limited.

Even less argument for 'pushing the envelope' in the old direction of single-processor speed when the cooling technology brings its own risks and the industry is moving the other way for more than 'green' reasons.Overclocking is relatively 'old school': the established way forward is multi-core, especially where 'embarrassing parallelism' is available.

The program speeds being quoted seem slow even for a Nalimov generator and certainly in comparison to the Konoval performance. Does anyone know if the program being offered 'out of the blue' has any real legs?

[Ed Trice]

There seems little point in investing in cutting-edge processors to generate EGTs when the activity is essentially I/O limited.

Hello Guy,

My experiments in hardware have shown that alleviating the CPU-bound bottleneck in the Gulftown allows for that amazing architecture to really kick in. Overclocked to 5.0 GHz, it performs more like a 5.4 to 5.6 GHz system, depending on whether or not 2200 MHz DDR3 RAM or 2500 MHz DDR3 RAM is used and how well the FSB is accelerated vs. the multiplier. If more of the overclocking is from FSB (FrontSide Bus) acceleration, you get better overall performance.

Even less argument for 'pushing the envelope' in the old direction of single-processor speed when the cooling technology brings its own risks and the industry is moving the other way for more than 'green' reasons.

Cooling adds longevity to the processor. Run a stock Gulftown at 3.33 GHz at full-tilt next to an overclocked 5.0 GHz model. At -40 degrees, the 5.0 GHz model will long outlive the other one.

The only "risk" is incurred by hobbyists who have not yet mastered the craft. We've all over-voltaged/smoked/burnt/blown up boards from time to time, for a variety of reasons (faulty censors giving out bogus readings, untuned controllers not reacting to changes in the heat load fast enough, unanticipated CPU usage spikes when running different benchmark apps for the first time, and so on).

The industry is not so humanitarian in their motivation. Computers first broke 3.0 GHz in 2002. So why is a stock Gulftown in 2010 only 3.33 GHz? We've hit a thermodynamic brick wall. A beam of light travels only 4 inches by the time one clock pulse ticks @ 3.0 Ghz. That's pretty darn fast, in other words. Pushing ever faster is problematic. The heat curve looks like "e to the x" for each 50 or 100 MHz faster. That's why laptops will always be slower than desktop systems. The tighter enclosures will always generate more heat, so they have to be dialed down slower to release even less heat than their desktop cousins.

Overclocking is relatively 'old school': the established way forward is multi-core, especially where 'embarrassing parallelism' is available.

Apples and Oranges. Overclocking allows systems to achieve what thermodynamics otherwise prohibits. We overclock 6 core boxes now, originally it was just single CPUs. There will always be a need for more speed.

Overclocking is not being done by billion dollar companies with Clean Room environments. Overclocking does not employ post-PhD fellows with electronic pioneering backgrounds. That's where the multi-core development is coming from. Overclocking does not drive the industry: it reacts to, and improves upon it, with mostly home-grown solutions.

It's more like the hot-rod car enthusiasts than anything else. You don't see car showrooms full of modified 1932 Deuce Coupe Vicky's with Cheater Slicks, killer paint jobs, and 500 horsepower engines. But every time you see one, at a car show or on the road, you say: "Wow, isn't that cool!" or "Wish I could have that!"

[guyhaw]

Ed: I think you are beginning to sound like an overclocking/fridge salesman rather than an endgame enthusiast - so maybe this is the wrong place for this duscussion.

Obviously, if you can turn the clock up, you will lessen the 'CPU bottleneck' if there is one. But if you are creating n-man EGTs and 'n' is big enough, you don't have a CPU-bottleneck but an HDD I/O bottleneck. I noticed that your checkers benchmarks are on 6-man checkers endgames - but 9-man EGTs have been generated and the frontier is now 10-man. So why not show benchmarks where the challenge is?

I'm a sysems engineer, not someone needing the 'fix' of the nominally-fastest CPU. If I wanted more CPU power, I'd buy more cores - because the sort of computations I want to do have enough parallelism to exploit multi-core.

The 'risk' I referred to is not in the overclocking but in the assembly of 'specialist components' necessary to achieve the cooling. These assemblies are produced by relatively small-scale companies and they are less tested than mainline I.T.

Some, or maybe all, of your physics may be correct - but that does not mean you have a good argument for fridge-freezing and overclocking.

I look forward to your contribution on chess EGT discussions but your argument, which sounds suspiciously like sales hype, for complex hardware is I fear wasted on at least me, especially in the context of EGTs and this board.

g

[Ed Trice]

Ed: I think you are beginning to sound like an overclocking/fridge salesman rather than an endgame enthusiast - so maybe this is the wrong place for this duscussion.

I'll leave that remark alone in the interest of peace.

I noticed that your checkers benchmarks are on 6-man checkers endgames - but 9-man EGTs have been generated and the frontier is now 10-man. So why not show benchmarks where the challenge is?

Dr. Jonathan Schaeffer solved checkers, so there is no frontier for that game, technically speaking. Ed Gilbert generated a 10-piece database for the game of checkers as well, but this was not Distance To Win, only win-loss-draw, or "Game Theoretical Value" as it is commonly called.

Only two people have generated Distance to Win databases for checkers: Gil Dodgen, the true pioneer for checkers "perfect play", and myself. We solved the 7-piece Distance to Win database together as a team in 2003. I've since done 8-, 9-, and parts of the 10-piece database. At 1.25 bytes per position, the 9- and 10-piece Distance To Win databases are a huge challenge. There would be no way to implement a benchmarking application for this, since very few people have access to the type of hardware I have to run the solver application.

The 6-piece checkers database benchmark is just that: a benchmark. It can be run in a few hours, and the longevity of the test eliminates people with "rigged" overclocks that are not stable or can only execute for short periods of time.

It serves its purpose in that many people can run it, needing less than 1 GB of RAM, and they can send in the report.txt file to let me know about how fast their particular hardware setup is.

I'm a sysems engineer, not someone needing the 'fix' of the nominally-fastest CPU. If I wanted more CPU power, I'd buy more cores - because the sort of computations I want to do have enough parallelism to exploit multi-core.

OK. To each his own. The computer industry can provide what you need. I like building faster versions of what the computer industry provides.

The 'risk' I referred to is not in the overclocking but in the assembly of 'specialist components' necessary to achieve the cooling. These assemblies are produced by relatively small-scale companies and they are less tested than mainline I.T.

These days, a computer company that generates about $100 million in annual revenue is considered "small", but they still need to manufacture reliable components. Now, if you are talking about some of the -200 Fahrenheit rigs out there, such as this one (from England)

http://www.xtremesystems.org/FORUMS/sho ... p?t=223974

...then yes, I would agree, this "one off" is fairly untested and a home built design is just that. But there are companies that produce various overclocking components in high volume. For instance:

CoolIt Systems of Canada, about $10,000,000 per year in sales
http://www.coolitsystems.com/

Frozen CPU, who has emerged as the "New Egg" of overclocking and cooling
http://www.frozencpu.com/

Koolance is another reliable supplier with a fairly large market
http://www.koolance.com/default.php

I think you were just unaware of companies like these, which is fine, a year ago, so was I.

Some, or maybe all, of your physics may be correct - but that does not mean you have a good argument for fridge-freezing and overclocking.

Well, I can modify existing hardware that exists today, and deliver performance that is about 8 years ahead of schedule if you apply linear regression analysis with the most contemporary models of processing gains. If you want to label this as a "bad argument", you are welcome to do so.

I look forward to your contribution on chess EGT discussions but your argument, which sounds suspiciously like sales hype,...

Hype?

Yankees baseball player Yogi Berra had a great saying: "It ain't braggin, if you can do it."

And I don't wish to brag, but sales of this 5.0 GHz Gulftown are doing fabulous. If you type 5.0 ghz gulftown into google, even without the quotes, not only is this system on page 1 of Google, every link on page 1 except for 2 or 3 are talking about this exact computer that I build.

I was talking about making a massively parallel version of it (something you should like) but there does not appear to be an endgame tablebase generator that could keep 24 cores busy. Is there a way right now?

If you go through some of my posts, I think the discussions were reaching the point of my involvement of writing such software, when my 10-piece checkers project winds down, and after I publish those results and release that source code.

[Kirill Kryukov]

I did go back and re-index these endgames, with my own move generator and DTW logic, and I found a way to remove superfluous positions and improve the compression. I construct only legal positions, without the "broken" labeling that you see in the other .tbs files. It's tricky, but it has huge benefits. It does slow down the indexing function calls, but the benefit is much lower RAM overhead.

I do the same in my recent solvers - only legal positions are stored, in fact only unique legal positions.

[Ed Trice]

I do the same in my recent solvers - only legal positions are stored, in fact only unique legal positions.

Kirill,

Have you come up with a generalized indexing function for all material distribution permutations? I can do it for all endgames except those with en passant captures. That throws a monkey wrench into my equations since you have 2 identical looking positions that are different because of the en passant flag status. I have to do all of the non-en passant positions first, get the "count" of every position in the tablebase that way, then use that as an offset to where the en passant region begins. It is not so elegant, and I am wondering if you have to do something similar, or if you write separate indexing functions for en passant tablebases.

[dann corbit]

What is the magic incantation to download it. The download does not work for me.

[Ed Trice]

What is the magic incantation to download it. The download does not work for me.

Hi Dan,

I uploaded it to my site. Link is:

http://www.liquidnitrogenoverclocking.com/chess.zip

It's about 38 MB compressed because it comes with up to tb4 pre-solved. I ran the 5-piece generator on my computer, it took a little less than 24 hours to solve the whole thing.

[Kirill Kryukov]

What is the magic incantation to download it. The download does not work for me.

There is a captcha, at first it got me too.

[Kirill Kryukov]

Kirill,

Have you come up with a generalized indexing function for all material distribution permutations? I can do it for all endgames except those with en passant captures. That throws a monkey wrench into my equations since you have 2 identical looking positions that are different because of the en passant flag status. I have to do all of the non-en passant positions first, get the "count" of every position in the tablebase that way, then use that as an offset to where the en passant region begins. It is not so elegant, and I am wondering if you have to do something similar, or if you write separate indexing functions for en passant tablebases.

I use a separate mini-table for all positions with en-passant or castling rights. Also such positions use their own indexing function. After the table is completed, of course it's possible to package them together with the main table using offset index, or keep separate, a matter of taste. I only briefly experimented with this though. The variants I study now don't have en-passant nor castling (small board chess and shogi).

[Ed Trice]

I use a separate mini-table for all positions with en-passant or castling rights.

I did not compute separate DTW for any positions where castling is legal. My querying program accepts castle flags as an input parameter, logically performs a castle, and checks the DTL (if it is still a loss) for the opponent's side. I then check "DTL + 1" against the DTW for not castling. If it is less than the uncastled DTW, I instruct the program to move both the king and the rook, and announce the winning information for the new position.

That way, I don't have to make the tablebase more complicated. All of the positions where you castle are contained within the database.

But, now that I think about it, I suppose there could be a line of play where moves leading up to a castling are needed to preserve a win or play better before the castling. Darn, time to re-compute!!