CCRL Discussion Board

Hi all. After half a year of sharing we still have less than 50 people participating. Very few new people are joining. Where are all thousands of chess players, why they don't want to improve their endgame analysis by perfect chess knowledge? I think still too few people know what are endgame tablebases, where to get them, how to use them. We need to do more talking to general chess people, to get them interested.

For that we need convincing examples - positions where EGTBs bring a point home (or half a point). Positions where winning (or drawing) move is very hard to find without 6-piece tablebases, hard for both human and computer. Positions from real games would be best.

If you have any such examples, please post here. After we have enough of them we can make a web-page. Ideally we should construct a epd testsuite which will test for every EGTB set, but any positions would be good for a start.

Please post here if you have any. Note that you can post diagrams in this forum. Just press "Diagram" button at the top, then enter the FEN, then press that button again. Thanks!

I read a while back, where a post analysis done by a GM used the 6men egtb to find a flaw in the his or his opponent's endgame. If I find the article I'll post it, since the game pgn would be a good example.

-Josh

At the http://www.chessbase.com site on 13.05.2006 the "star Grandmaster annotator" Mihail Marin is analysing some games from the MTel tournament. In one of the games (Kamsky-Bacrot), a Rook endgame is reached, which eventualy becames a KRP/KRP tablebase position. During the analyse another tablebase endgame - KRPP/KR - can be reached. Marin missanalyse that position, but has an excuse:
"I do not know whether Nalimov databases have been built for R + 2Ps vs. R and in any case do not have them! Besides, I find myself somewhere in the middle of the nature, together with my team mates, preparing for the Olympiad, which makes the task of finding "digital help" on this matter quite impossible. However, I hope that the "analogical" analysis presented above is accurate enough."
No, is not accurate at all.
If a well respected annotator needs the Nalimov tablebases, we average chess players need them even more...

Kirill Kryukov wrote:Hi all. After half a year of sharing we still have less than 50 people participating. Very few new people are joining. Where are all thousands of chess players, why they don't want to improve their endgame analysis by perfect chess knowledge?

Is it something to do with the huge hard drive storage requirements ?
Most people still have a single hard drive of less than 200MB in size, and if so 6-men are impossible no ?

Ray Banks wrote:

Kirill Kryukov wrote:Hi all. After half a year of sharing we still have less than 50 people participating. Very few new people are joining. Where are all thousands of chess players, why they don't want to improve their endgame analysis by perfect chess knowledge?

Is it something to do with the huge hard drive storage requirements ?
Most people still have a single hard drive of less than 200MB in size, and if so 6-men are impossible no ?

Not impossible just less likely. To store the complete egb's takes about 1.6 TB, from what I've read. That would take 8 200gig HD's, or a little over 6 250's. So you're looking at over $1000 in just HD for storing the data. So this is something a bit more reserved for the hobbiest or professional chess player. However with the ever increasing size of storage at cheaper prices I'm sure there will be a day when you can go down to Best Buy and pick up a 2TB device for $100. But by then 7-8men will probably be out requiring over a petabyte, so it's all relative.

-Josh

I think most of the chessplayers don't have the 3-5men tablebases either, so the problem is not only the big storage size. They don't know what are the endgame tablebases or how to get them.

I do not understand very well how a top grandmaster annotator can ignore what tablebases are, or where to find them (chessbase sell a little part of them.....) . The general win in kbbkn is known since more than twenty years... Can you imagine a top grandmaster with no knowledge of Averbach or Fine's books ?
You can find in the databases many 5 or 6 pieces endings you can analyse with the tablebases . 1 Tb hard drive is not obligatory as you can use servers (with Shredder Classic or CA)
Another major interest of tablebases is to analyse many studies ( and cook some) . Study composers , problem composers too probably , may use them. And the downloading player with 2600iccf pseudo probably has good reasons to download them.
Here joined you 'll find a little base of studies where QN against Q appears and a part of a listing of reciprocal zugzwangs ( half-points zugzwangs) in some aristocratic 6-pieces endings

And now the pgn

sorry

kalyanasundaram wrote: Another major interest of tablebases is to analyse many studies ( and cook some) . Study composers , problem composers too probably , may use them.

Yes, at least half of the QN/Q studies from your database are cooked.
As a study composer I often use the tablebases:
1.indirectly - analysing with an engine
2.directly - using Wilhelm for finding interesting positions which can be the starting points for composing a study, like the following positions:
8/Q2K3k/8/8/4q3/8/1P6/8
WTM
8/Q2K3k/8/8/8/5q2/1P6/8
WTM
Both are wins for white, they already can be considered as twin studies, but the main goal it would be a syntese of them: one study with both positions in the solution.
Advice:Try to solve them with tablebases!

EGTs are less valuable in chess than in checkers: the former game need not approach the endgame whereas the latter game always will.
Therefore, it is more likely that the chess-endgame enthusiast is interested in the intrinsic merits of the endgame, rather than in the fact that it is the end of the game.
'Interesting positions' come from the classes:
- endgame positions arising over the board
- didactic, study and game positions published via some medium
JN's trilogy on various 5- and now 6-man endgames is a standout. Marc Bourzutschky's piece on 7-man EGTs in 'EG Vol.XI' was for me the best item in that book. MB has also - and I wonder if I still have the file on this - analysed HvdH's database of studies (v2) with the EGTs he had at the time.
The routine data-mining exercises - lists of maxDTx positions, mzugs and maxDTx-mzugs - are rather held up by the lack of the last 16 EGTs from Eugene.
Curiously, the Chess Studies community - most woefully, many of the judges of tourneys - are way off the pace with regard to knowledge and use of EGTs. In that community, A.J.Roycroft (the editor of EG) is waging what is essentially a one-man crusade against the use of EGT'd positions, arguing - incorrectly in my opinion - that a position listed in a file is already 'published'. [However, it is one thing for a position to be in a computer file, and another before it is championed as 'interesting' by a human and ushered into collective human consciousness.]
I have, for the ICGA, been considering how its website at http://www.icga.org can organise and promulgate knowledge on the chess endgame. Your ideas are welcome.
g

I think tablebases will become very important once 7 and 8 piece tablebases become available, and even more important as tablebases become deeper. When 15 piece tablebases come out, I think chess will be sufficiently solved.

Great, thanks for many examples! game annotation and endgame studies are particularly convincing. Please keep them coming! Hopefully we will eventually make a web-page with those examples to illustrate how usefulf tablebases really are.

Arpad Rusz wrote:As a study composer I often use the tablebases:
1.indirectly - analysing with an engine
2.directly - using Wilhelm for finding interesting positions which can be the starting points for composing a study, like the following positions:

Examples of published endgame studies which are cooked will be also good to prove the point of tablebase usefulness. Especially contest winning studies.

Leto wrote:I think tablebases will become very important once 7 and 8 piece tablebases become available, and even more important as tablebases become deeper. When 15 piece tablebases come out, I think chess will be sufficiently solved.

6-men tablebases are already sufficiently important to me to buy a 1.6 TB RAID array. After I manage to collect the whole 6-men collection, I will start to test engines with 6-men tablebases, to estimate ELO advantage from using them (comparing to just 5-men set). (Games will be published as part of CCRL engine testing study).

Leto wrote:I think tablebases will become very important once 7 and 8 piece tablebases become available, and even more important as tablebases become deeper. When 15 piece tablebases come out, I think chess will be sufficiently solved.

I think that you overestimate the value of tablebases.
I do not expect them to be very important in the near future.

My guess is that you will also not see 15 piece tablebases in the next 50 years.

I also think that chess can be practically solved in 20 years without tablebases of more than 8 pieces and by practically solved I mean that comp-comp games between top programs will be usually drawn.

Uri

Greetings. I've just joined and am happy that this forum exists.
Using a simple exponential fit on the data sizes of the 3, 4, 5, 6-piece tablebases, an estimate of the total size of all 15-piece tablebases is 2.4*10^22 Terabytes. We would have to wait many generations for that!
This brings me to a good point about having this forum. If we want any chance of seeing very good size tablebases in our lifetime, we need to COOPERATE in generating the files. We cannot rely only on Eugene N. to generate everything for us because as expert as he is, he is only one person. We must pool our resources and have hundreds, if not thousands, of people generating together. If such mass-cooperation exists in the search of large Mersenne primes, why not for endgame tablebases?
Now if we do eventually get such cooperation, we may soon have to address the issue of allowing castling in our tablebases. At what point should we incorporate castling? And looking at the long distant future, if (when?) we finally get 32-man tablebases, or close to it, chess will certainly be no longer interesting (being proven a draw right from the start). So after how many pieces should we actually stop generating tablebases?

Hi andytoh, welcome to the club!

andytoh wrote:Greetings. I've just joined and am happy that this forum exists.
Using a simple exponential fit on the data sizes of the 2, 4, 5, 6-piece tablebases, an estimate of the total size of all 15-piece tablebases is 2.4*10^22 Terabytes. We would have to wait many generations for that!
This brings me to a good point about having this forum. If we want any chance of seeing very good size tablebases in our lifetime, we need to COOPERATE in generating the files. We cannot rely only on Eugene N. to generate everything for us because as expert as he is, he is only one person. We must pool our resources and have hundreds, if not thousands, of people generating together. If such mass-cooperation exists in the search large prime numbers, why not for endgame tablebases?

Because the resulting information, or "answer", is tiny for large prime numbers, so it can be exchanged between nodes, and collected on master node. When solving chess (or any game), the answer is huge so our main limitation is storage size, not only computation time. So far the common way was that everyone interested in EGTB is keeping his own set of tablebases on his computer locally. It works for anyone with 5-men EGTB, and for about 10 to 50 with 6-men ETGB, and for no one with 7-men EGTB. No one can afford to even keep all 7-men information in DTM (or DTZ or DTC) metric, let alone generate them.

Are you proposing to also have distributed storage for EGTBs? It may be possible, but as a new concept it will need some serious planning and discussion.

andytoh wrote: Now if we do eventually get such cooperation, we may soon have to address the issue of allowing casting in our tablebases. At what point should we incorporate castling?

I'd say we don't need castling (practically) for 7-men endgames. How about we solve 7-men chess first, and then come back to this question? BTW, if someone feels need to do that, it should be possible to construct a set of "supplimentary" tablebase files, covering only those positions where castling is possible. Those files will be tiny anyway (comparable to 4-men EGTB size for 6-men endings).

andytoh wrote: And looking at the long distant future, if (when?) we finally get 32-man tablebases, or close to it, chess will certainly be no longer interesting (being proven a draw right from the start). So after how many pieces should we actually stop generating tablebases?

This is simpe. We will try to generate tablebases for as many pieces as we practically can. Though I don't expect anything more than 8-men to be practical for about next 10 years.

If (when) we will get 32-men tablebases, we will simply switch to other challenges, like solving Shogi (where always 40 pieces are in game, never 5 or 6 or 39), or Go.

"Are you proposing to also have distributed storage for EGTBs? It may be possible, but as a new concept it will need some serious planning and discussion."
Yes. That is exactly what I mean! It is the only way to get results in the timeframe that we want. I buy the latest computer in the market every 2 years and have already a network of 4 computers at home (with the fastest being roughly the same as yours probably). I believe I qualify (as all of us probably do) to be a contributor.
Once the code for 7-man tablebases is developed, it must be passed on to all those who wish to cooperate in generating (instead of having the developer do all the generating alone). The team of generators should then develop lists in an organized fashion (and avoid wasting time with overlaps). Since there is no such list for 7-man EGTBs yets, I for one have chosen the following list:
1.kqrrkqr
2.kqrbkqr
3.kqrnkqr
I know this list is short, but I predict that it could take up to a year, and by then there could be over 100 other volunteers to do the rest. So, I alone will generate 3 (perhaps more). But this will mean very little if there are not many other teammembers. If we have 100+ member team, we could ideally generate all the 7-man EGTBs in about one year, and then less time to check the accuracy. At this rate, just maybe we can get the 15-man EGTBs in our lifetime.
As for distribution of the data, I figure it could only be done on hard copies (blu-ray discs perhaps, about 20-40 blu-ray discs per person). We would have to mail our generated data to one person (the captain?) who then, if he is willing, could make many copies of the entire collection (using some good technology) to redisribute back to the original contributors (in terms of mass production, I haven't thought that far, for I am not in this for financial gain).
So to reiterate, I reserve the generation of:
1.kqrrkqr
2.kqrbkqr
3.kqrnkqr
Anyone else wishing to join this team shall choose something else! If we do get a serious compilation of lists for this, we should post the information on a separate thread so that everyone can see it clearly and avoid generating that which is already in progress or done. I'm glad to join this forum, but would be more elated to join a generating team of this sort. Remember, if we do not form such a team, or a team that's not large enough, do not expect to see more than 8-piece tablebases in your lifetime!

In response largely to andytoh:

Dekker, JvdHerik and Herschberg (1989) produced EGTs including castling options but this is the only case I know of, and they don't survive afaik.
However, the suggestion that castling is included made me realise for the first time that this is a purely 'additive' process. One cannot move to a position with more castling options, so the depths DTx of current positions in EGTs cannot be changed by the belated recognition of positions featuring castling opportunities.
Since there are 5 options (e.p., w/b K-/Q- side castling), there are 2^5 zones of positions, of which EGTs recognise only 2 corresponding to the e.p. possibility. Castling rights fix the relevant Ks and Rs, and don't allow retro-moves by these pieces, so the other 30 zones are relatively small. At the extreme, KRRKRR features the only position in 3-to-6-man chess with all castling rights and it's a win in 1 for the stm. So this would be a neat 'add on' small-scale (but relatively complex) project for someone, and I would like to see future index-schemes incorporate all 32 zones of positions.

As for the future, the YK-MB work is showing that there are still untapped ideas which will speed up EGT-generation, but I will say - in the hope of being proved wrong - that future advances will depend on technology more than new ideas. I can see 7-man being completed by 2015 but 8-man by 2030 looks like a stretch. Storage, and even more so, distribution as we see at the moment with 6-man EGTs, is a big problem.

Computing chess EGTs is parallelisable but is constrained by the need for the subgame EGTs. However, if one is computing DTC or DTZ EGTs (and YK-MB are targetting DTZ), these subgame EGTs can be handed around in 'WDL' form indicating value-only, which ought to give some good (4?) saving in size after compression.
Chess EGT-generation is not as parallelisable as the GIMPS Mersenne Number trawl, distributed.net's 'RC5' number-sieve factorisation, SETI@home or the imminent STARDUST@home ... where the nature of the task permits fine-grain distribution of work. But certainly, one can 'EGT' all endgames with the same number of Pawns in parallel.
Main drawback is that a 'distributed.net' approach requires a funded 'hq' to organise the work and results, support the software etc. Capable volunteers would be required on a long-term basis.

An indexed distributed store of EGTs moves the p2p idea on a notch and would be good. It also creates a 'Data Assurance Challenge' to ensure the correctness of the data, as we can see from the latest flag about corrupted-EGTs being shared on eMule.

g

Here is a new example - a position from the 2nd game of the Topalov-Kramnik WCh match:

6k1/1p2r3/3K4/6N1/3P4/8/8/8 b - - 0 1


1...Re1? [1...Re3! was the only winning move 2.d5 Kf8 3.Kd7 b5 4.Ne6+ Kg8! Switchback 5.d6 b4 6.Ke7 b3 7.d7 Rd3 8.Nc5 Rxd7+! 9.Kxd7 b2-+] 2.d5 Kf8 3.Ne6+? [Only the natural 3.Kd7! draws b5 4.Ne6+ Kf7 5.Nd8+! Kf8 (5...Kf6 6.Nc6=) 6.Ne6+ perpetual check(6.Nc6? Re4-+) ] 3...Ke8 4.Nc7+ Kd8 5.Ne6+ Kc8 6.Ke7 Rh1!-+

Points for topicality - and I was just pondering the KRPKNP zugzwangs at the time g

For completeness, some URLs might be mentioned here:
http://www.chessbase.com/newsdetail.asp?newsid=3366 ... at the foot is a Nunn treatment of the KRPKNP position
http://www.chessbase.com/news/2006/elis ... nunn02.htm ... same text with playthrough facilities
http://www.k4it.de/index.php?topic=egtb&lang=en for Eiko B's EGT-query service
Two results for this community - John N and Eiko B both supplied by Nelson Hernandez ... g

Why to use EGTBs...
Creators of chess engines say that EGTBs are almost useless for programs:
1) If in EGTB, program cannot "play to win" in draw positions,
2) Slow HD access if program is "near" EGTB position,
3) EGTB position with complicated play (out of computers reach) is rare in practice.

Are there researches and general results on these matters?
Best, Andrey

Of course this is *somewhat* working in the opposite direction - but I have used lokasoft's 5-piece tablebase server a handful of times when I was still on dial up to analyze a few positions.

It would open up a new world for some if we offered the same sort of server, with all complete 6-piece files. At least for this, very little speed would be needed, and I'm sure people would be willing to wait a few seconds for a response anyways.

How much disk activity is truly used when accessing TB's? Maybe it is more practical, at least for some time that the 7-piece ones simply be accessable through a server. Not for distributing the files, but at least so they can be referenced.

If it not much transfer needed, I can even offer these (6-piece) once I have my complete set by VPN. Users could simply connect to me, and have a 'hard drive' on their computer they could use all current engines with like they were on their own system. Just need some way of someone not transfering the entire database off, or it could bottleneck everyone accessing

Just more thoughts,

Derek