Generating 7 Pieces

[Vegan]

Today, after 7 hrs of work. I have now more or less designed a machine that can manage to do 7 pieces. I even researched component costs which are less this year then they were back in the summer of 2006 when Eugene finished the 6 piece tablebases.

Using rack mounted storage its not that bad compared to the cost of RAM. 8 GB DDR2-533 server RAM at IBM is a measly $6,500 a stick. The processor blades cost peanuts compared to the RAM prices. You can buy one of the storage servers for the price of a stick of this memory.

Using a blade system, a single rack can manage. The problem is the blades can not use more then 64 GB of RAM which is a nuisance. To deal with the problem, tablebases would have to be computed uncompressed before the CPU blades could compress the whole set for distribution.

I even got prices.

[Dhanish]

Is the generator available?

Are the system requirements definite? It should somehow be possible to run the generator in some distributed fashion, even if it takes more time?

[Vegan]

No, I have asked Eugene to update EGTB and TBGEN to 7 and 8 pieces to push tablebases forward a few more years.

As for the system requirements, that machine would be stressed hard to manage with 4 blades and 4 storage servers. The main problem is even quad cpu machines only have 8 memory slots.

Due to the high memory bandwidth requirements, distributed computing is not suitable that way. Instead blades can work on various parts from the 43 block and 52 which are mutually exclusive as far a dependency is concerned. For that reason the blade machine can use multiple blades working on the problem.

When the piece parts are done then the pawn parts which depend on the piece parts can be executed. At this point the storage servers would be stressed more aggressively.

When blades come to market that support DDR3 it will be possible (expensive) to use 16 GB memory modules (max for DDR3, DDR2 max is 8 GB).

Make my page you home page, there is a Google search making it usable. I could use the commissions to fund the 7 piece machine. Blades are over $60 grand each.

[jkominek]

Make my page you home page, there is a Google search making it usable. I could use the commissions to fund the 7 piece machine. Blades are over $60 grand each.

I've taken a look at your website and must express disapproval at the way you are conducting yourself. To begin with, it is improper to offer a product for sale (6 man Nalimov tablebases, $349) that you cannot now deliver ("Work on the 33p is ongoing. The 42p group is pending."). Further, your description is ambiguous about what you are actually doing. The impression is that you are generating the things -- possibly even writing the generator -- when in fact we know you are downloading them like everyone else. To even mention 7-man tables (with size requirements down-to-the-byte) in your list of product offerings is ridiculous.

Regarding your pleas and mechanisms for funding a server for 7 piece tablebase generation, I notice that you estimate total system cost at "almost $1 million". A heady amount for a presidential candidate fund raiser, let alone a grass roots teams-of-one chess initiative. Before casting lot with your plan, as a reader of ambitious proposals, I'd like to see evidence that you have experience in building and maintaining supercomputing clusters.

I suspect that you are young. For you exude - let us be charitable in our labels - "the enthusiasm of youth." Take care in how you express your enthusiasm.

john

[Vegan]

I am going to test TBGEN when I get the update from Eugene on a 64 CPU Sun mainframe and see how well it performs with the architecture. I doubt it will be very fast even though the machine has 64 GB of RAM. I have access to a Cray as well, but it does not have enough RAM for 7 pieces.

I am researching the proposed machine to see if I can find faster solutions. The main problem is TBGEN wants fast memory rather than CPU speed. This is due to extensive searching the tablebases for distances.

I have TBGEN and I am downloading some and generating others.

The blade machine on my site is barely adequate for the piece tablebases. It is intended to get some parts done until better blades can be procured. This is due to the limited amount of RAM available. Better is a machine with extreme memory bandwidth and about 256 GB.

Ass for the size, the exact amount is not hard to calculate. First you need the uncompressed file sizes of 42. Now for each part, we have 42 plus a p, plus a B, plus a N, plus a R, plus a Q. The p multiplies by 48x (no 1st or 8th rank), the B multiplies by 32x (color) and the rest by 64-n where n is the number of pieces on the board. No need to consider 33, the results are symmetric.

This is the basis I used for calculating the estimated size. Its close to exact as I did the math.

I did a lot of thinking about a distributed job, way to slow because of the extreme amount of searching.

[guyhaw]

I tend to agree with jk. vegan is making too many simple mistakes, which suggests some combination of naivety, lack of vision and carelessness.

'v' ignores the fact that there are 3x as many 7-man endgame-forces as there are for 6-man, that the trend with more men is for deeper endgames requiring 16-bit entries rather than 8-bit, and that these will take longer to compute. v also ignores the fact that EN has not explicitly worked on his code or even communicated about it for a long time now. Nor has not picked up published information to the effect that EN's code is iffy on wide-scale SMPs, presumably because of some untrapped race conditions.

The website offering Nalimov EGTs and implying that 7-man EGTs are going to be available some time soon is, I agree, irritating.

g

[Mark]

The website offering Nalimov EGTs and implying that 7-man EGTs are going to be available some time soon is, I agree, irritating.

Well, I thought offering the 7-man tablebases for $250,000 was entertaining and informative! Of course no one actually thinks they are available, and no one is grabbing for their credit card to buy these. It does, however, illustrate that generating the 7-man set is a monumental task and just how far we are from having the set available.

Regards,

Mark

[Kirill Kryukov]

I have to join the concerns raised by others. Vegan is ignorant and making basic mistakes.

Vegan, if you don't have the complete 6-men set, then you can't sell it on your site. If you keep doing this I'll post a warning on EGTB Project page. Same for 7-man bases. This is for your own sake. I would also not be surprised if someone contacted your hosting and warned them of a fraud you are commiting.

A few comments on a some quotes of yours.

I have now more or less designed a machine that can manage to do 7 pieces.

You can't do this without having any idea about the generator.

I have asked Eugene to update EGTB and TBGEN to 7 and 8 pieces to push tablebases forward a few more years.

You could as well ask Santa Claus. If you only knew how many people asked Eugene the same thing. Even getting his permission to use his existing EGTB code is almost impossible these days. Anyway, "asking" does not count as activity. When you have his answer, then you can tell us about it. But it's still long long way from having the generator and probing code.

Make my page you home page, there is a Google search making it usable.

It's not polite to suggest a home page to computer literate people.

I could use the commissions to fund the 7 piece machine.

A number of people already invested a lot in generation and sharing of tablebases. Most of us still share it for free. If you are not aware, there was a number of offers on this forum to share tablebases via HDD for the price of mailing the HDD. There are people who generated some 51 bases, who played with other generators, and who achieved at least something already. You are, for what we know, downloading the tablebases, and generating them using public generator. It's a common practice to show something you managed to achieve before asking money for an ambitious project.

For example, I did write a working generator from scratch (for a chess variant). Many others wrote generators too. I am sure that if I give $250K to John, or Josh, or Harm (or to myself), we will see a perfectly working, stable, open source generator and probing code, and some tablebases in less than 2 years. I don't have this confidence in you yet.

I did a lot of thinking about a distributed job, way to slow because of the extreme amount of searching.

A lot of thinking does not count for activity either.

Adding 51 is not really necessary, most engines can beat up on a lone king easy enough with 4 pieces to work with.

As others said already, 51 is necessary for completeness and for 52 (and 61).

I suggest the new TBGEN be open source, and free to use to encourage wide adoption.

I and others are "suggesting" it for as long as we can remember.

I am a skilled C++ programmer so I can talk.

Just FYI, a skilled C++ programmer reads this as "I am noob".

[Vegan]

Back before all the 6 pieces were done, I had tbgen running on an old single core AMD64 machine. I was a pain in the butt to migrate from the DEC Alpha processor but after months of toil I had it running fairly well using the AMD CPU as best i could.

The problem was that the compiler for the Alpha and for the AMD were different in the level of C/C++ supported. I had to change a lot of code to make it native 64-bit on the PC. The program was configured for 32-bit PC not 64-bit.

I am working on getting use of a supercomputer to work on some of the tablebases. If that pans out then I can use the supercomputer and save a wad on building a more mundane blade machine.

I offer tablebases as a courtesy off my web site for those who don't have bandwidth, or an evil ISP.

The problem with 51, EGTB.CPP does not support them. So they are useless. Beside what prog can't win with 4 pieces to the good against a lone king?

So moving to 7 pieces, the engine EGTB.CPP needs to be updated too.

[Kirill Kryukov]

Vegan. For continuously spreading misinformation, for selling the 7-men tablebases that you don't have, and for shamelessly trying to raise money selling the hard work of others without giving any credit, I ban your web-site from this forum. No mentions of your site are allowed here anymore.

[Vegan]

I am in no big rush for 7 pieces for obvious reasons, who has enough RAM. I have been downloading engines that are tablebase enabled and found a bunch over on Ridderick. While I have the source to Crafty so I know what the version of EGTB.CPP he has, I am not sure about other engines.

For the most part 5 pieces on my AMD X2 4200+ ran fine. The trouble started when I initially added 42 and noted some engines caused Arena to restart the engine. It seems the loading of the tablebases took too long. Some engines used a LOT more RAM than the tbcache value indicated.

So I built a list of engines, and I am now trying them a couple at a time to see how bad the situation has become. Generally I found engines lagged with 42 and/or 33 presented. It suggests that 5 minute blitz games may be unsuitable for ELO purposes.

Once I get a bigger disk, I can restore all the tablebases I have generated/downloaded and see how bad it gets. Extrapolating however suggests I may may need to buy more RAM to use 6 pieces in addition to a large disk for the tablebases.

I am still tying to gain access on a supercomputer to test out to obtain some idea as to how long the ETA is for generating 7 pieces. from experience, its going to be a very very long time.

The price for 7 pieces is an estimate for hardware plus RAM to generate them, it will be revises depending on the patience of the client. The main problem is 8 GB sticks are as big as DDR2 allows, DDR3 allows 16 GB max. IBM sells DDR2 8 GB sticks for $6,500 each.

Nobody else has DDR2 at 8 GB, Kingston only has DDR at that size. And that's what made anybody's blade expensive.

After the experiences with this dual-core machine, I estimate it to be minimal for playing chess with all the 6 pieces. The memory overhead is heavy as hell. This Asus board maxes out at 8 GB.

KRILL
I doubt I would find funding for 7 pieces here, I have some access at some universities where I am inquiring about using a supercomputer. Summer is slow, might get some reaction when universities go back in fall. You got disk space to post the parts P2P?

TBGEN as it is now has templates for a maximum of 4 pieces of black or white. So 43 and 44 can be added relatively easily. Its somewhat more work to make it work with 5 pieces on a side.

As for the size, I took uncompressed sizes from TBGEN, and added the P, B, N, R, Q which is 5x fanout, not 3x. And that is for each side, WTM and BTM. I then did the math to arrive at the estimate. I do not attempt to optimize for illegal positions etc.

The purpose was to illustrate the magnitude of 7 pieces, and how far away from practicality it is today. In 5 years disks may be bigger and RAM might be faster. Like it says, work in progress.

As for hard work, very hard. Chess is one of the hardest projects a programmer can embark on.