CCRL Discussion Board

Introduction

I realized that we are still lacking reliable way to test whether an engine is using endgame tablebases or not. Often you install a new engine and wonder if it is accessing the tablebases. These days many engines use bitbases which are more hard to check.

We need a way to test if an engine is actually accessing the tablebases. So I propose to create a set of positions for testing each piece configuration (or as many as possible).

Theoretically such test suite would be also useful for a developer, working on an engine, a GUI, or a new EGTB format.

Two test suites are proposed: basic test suite and advanced test suite. Basic test suite will consist of positions that are contained in the endgame tables they are testing. Such positions test for capability to probe endgame table at the root of search.

Advanced test suite will consist of positions that are not contained in the endgame tables they are testing. Such positions will test for engine's ability to access the tablebase in search.

Requirements

Each position should meet these criteria:

1. Position should have only one winning move, or only one drawing move if there are no winning moves. This requirement is needed to make this test suite useful for bitbases too.
2. Position should be trivial to solve for an engine that probes the relevant table in its search. Finding correct move should take a few seconds for an engine accessing the endgame table being tested.
3. Position should be hard to solve without that particular table (or set of tables) it is testing. An engine that does not have access to the table should prefer wrong moves, even if it has all other tables (with the same number of pieces).
4. For the advanced test suite, position should be a few moves before the endgame it is testing (so some captures will follow to convert into the EGTB position). For the basic test suite, position can be within the table it is testing.
5. Solution should be not sensitive to the 50 moves rule. This means a win should be still a win if 50 moves rule is used. And a draw should not depend on 50 moves rule to be a draw. (This way the test will work the same way for both current Nalimov tables that ignore 50 moves rule and for future tables that will take it into account).
6. Position should be legal, if possible.

Submission

Please post your positions here or send via email to kkryukov ~ at ~ gmail.com with subject "EGTB Test Suite". Also you can post them on CCC or anywhere else you think I'll find them. Please use PGN, EPD or FEN format.

Please clearly indicate the author of all positions you submit. If you know, please also include where and when it was published.

If will help a lot if you will include your analysis together with your submitted position. What endgame table it is testing? What engines did you try? How long they take to find the correct move when they access the table in question? How long do they take and can they solve it at all without the table?

Note that all positions submitted without analysis will have to wait for my free time which occurs every once in a long while.

Tips and tricks for creating a good test position

Many endgame studies can serve as a good EGTB test.

You can try using my collection of longest checkmate positions as a starting point.

In case you are missing some EGTB files, you know where to get them.

If your position falls within a 6-piece table, you can check it with online EGTB interface.

Disable EGTB support in your GUI when you are testing an engine or working on a test position.

The best way to select a particular set of endgame tables is by moving the files around on your hard drive. Move the EGTB files you don't want to use into another directory, and reload the engine.

Wilhelm program can mine tablebases for specific types of positions, this can be useful to find good tests.

"Just an observation: some positions in which a pawn is promoted followed by a long series of moves to keep a draw in hand tend to be mishandled by computer analysis: the pawn poised to promote is scored too high, and so an unassisted computer program keeps it rather than reach the promotion case which doesn't score anything but a draw." - ath.

How you can help

- Create a good test position and submit with detailed analysis.

- Look through published endgame studies, find a good position, make sure it qualifies and submit with detailed analysis.

- Analyze any of the positions submitted without analysis and post your results.

- Spread information about this project to attract more people.

- If you have an study composer friend, ask him if he has any positions that may be a good tablebase test. (Or if he can create one).

- Anything else you can think of.

Final remarks

As you can see it is not a trivial task to construct a perfect test position for an endgame table. For a start, may be some positions for testing 4-men or 5-men tables can be made.

Eventually we should have a test position for each piece configuration, so running an engine through the test suite will reveal any missing tables and test whether that engine is able to use them. Note that this test would be useful not only for Nalimov tables, but for any other format as well.

Any questions, comments, ideas and suggestions are welcome!

Thanks!

Positions received so far

Tests for 6-men tables

• Advanced Test Suite for 6-men tables
• Basic Test Suite for 6-men tables

Tests for 5-men tables

• Advanced Test Suite for 5-men tables
• Basic Test Suite for 5-men tables

Tests for 4-men tables

• Advanced Test Suite for 4-men tables
• Basic Test Suite for 4-men tables

Other

• Submitted positions, that still have to be analyzed and classified

Kirill Kryukov wrote:As you can see it is not a trivial task to construct a perfect test position for an endgame table. If you can manage to create any positions, please post here! For a start, may be some positions for testing 4-men or 5-men tables can be made.

Just an observation: some positions in which a pawn is promoted followed by a long series of moves to keep a draw in hand tend to be mishandled by computer analysis: the pawn poised to promote is scored too high, and so an unassisted computer program keeps it rather than reach the promotion case which doesn't score anything but a draw.

ath wrote:Just an observation: some positions in which a pawn is promoted followed by a long series of moves to keep a draw in hand tend to be mishandled by computer analysis: the pawn poised to promote is scored too high, and so an unassisted computer program keeps it rather than reach the promotion case which doesn't score anything but a draw.

Thanks! If you can illustrate it with a good test position that would be even more great!

BTW test positions that I already received will be collected in the first post for now (when there are still few of them).

One more idea. You can try using my collection of longest checkmate positions as a starting point. Good luck!

This is a part of the EGTB Test Suite project. This post will be edited as new data is submitted or analyzed.

Submitted positions, that still have to be analyzed and classified

(Note: When I say some position is "trivial" I don't mean I can see a solution at one glance. I mean it's easy for a beancounter.)

From Dann Corbit:

1k6/6p1/1Pp1pp1p/2P5/7P/8/5PP1/6K1 w - - bm h5; id "J&S45.06"; - Need to test more.

8/8/4k3/6p1/2pPpP2/4P2P/6K1/8 w - - bm f5+; id "J&S45.09"; - Need to test more. Possibly useful, although many engines like 1.f5+ within 5 seconds.

8/2pp2pp/8/2PP1P2/1p5k/8/PP4p1/6K1 w - - bm f6; id "J&S45.11"; - Need to test more. This is a nice test, just have to check if tablebases have any noticeable effect here.

8/5p1p/8/6k1/8/6P1/5PP1/7K w - - bm Kh2; id "bote.6"; - Need to test more. Some engines are troubled so this might be useful.

k7/8/pp6/2p4K/8/PPP5/8/8 w - - bm c4; id "(Simon Finn on CCC) Grigoriev 1933"; pv c4 Kb7 a4 Kc6 Kg5 Kc7 Kf6 Kd7 a5 Kd6 Kf5 Kc7 Ke6 Kc6 Ke7 Kc7 Ke8 Kf8 axb6 Kb7 Kd7 Kxb6 Kc8 Kb6; - Need to test more. Looks like a great test, need to check if tablebases help here.

7k/6p1/p4p1p/2p5/1pPP4/1P3R1K/r7/B7 w - - bm dxc5; id "(ML) Gabdushev-Hasangatin, Bor 2000"; - Need to test more. Most engines take time to find 1.dxc5, so this could be a good test. Not sure yet it tablebases help here.

8/p7/K6p/1P4p1/2k5/8/6PP/8 w - - bm h3; c0 "Draws."; id "(JAFM) CCC#191202"; - Need to test more. Many engines are troubled, should see if tablebases help here.

8/5p2/7p/5pk1/8/5KPP/8/8 b - - bm Kf6 h5; c0 "h5 is a mate in 43 according to Yace"; id "(JAFM) CCC#191597"; - Need to test more. Does Kf6 win too? Do tablebases help?

From Edmund Moshammer:

2k4q/3R4/1P6/8/8/8/4P3/2K5 w - - bm Rd3; (KRPPKQ, only one drawing move) - Should be OK. Many engines find Rd3 in about a second. Troubled: Abrok 5.0, Amyan 1.597, Delfi 5.2, Fruit 2.1, Green Light Chess 3.00.

3k3q/2R5/1P6/8/8/8/4P3/2K5 w - - bm Rc4; (KRPPKQ, only one drawing move) - Should be OK. Troubled engines: Green Light Chess 3.00, Kiwi 0.6d, Ktulu 4.2, List 5.12, Naum 2.0, Nejmet 3.07.

From Tano-Urayoan Russi Roman: here.

From touron: here.

From Guy Haworth: here and here.

This is a part of the EGTB Test Suite project. This post will be edited as new data is submitted or analyzed.

Advanced test suits tests for EGTB access in search, so each test position is outside of the table it is testing.

• 1k6/8/6NP/5K2/4pR2/3P4/8/7q w - - bm dxe4; c0 "Wins."; - Probably a good test for KRNPKQ, but should check more. Particularly we should make sure it does not break the 50 moves rule. Posted by Mc Mad.
• 8/8/6p1/7R/b6K/5k2/3n4/2n5 w - - bm Rh6; c0 "Draws."; - Tests KBNPKR. Only 1.Rh6 draws, but some engines find it without tablebases as well. A study by Akobia+Becker 2005, posted by Mario Kamody.

This is a part of the EGTB Test Suite project. This post will be edited as new data is submitted or analyzed.

Basic test suits tests for EGTB access in current position, so only a few lookups are needed to solve each position.

(GUI support for tablebases should be disabled, unless you want to test the GUI itself.)

None so far.

This is a part of the EGTB Test Suite project. This post will be edited as new data is submitted or analyzed.

Advanced test suits tests for EGTB access in search, so each test position is outside of the table it is testing.

(You should have no 6-men tables installed for a meaningful 5-men test)

• k7/P6R/2K5/8/7P/1r6/8/8 w - - bm Rh8+; dm 46; - Tests KRPKR. Only one winning move! Unfortunately many engines can solve it without EGTB (Anaconda 2.0.1: 0 sec, Fruit 2.1: 0 sec, Glaurung 1.2.1: 0 sec, Arasan 10.0: 1 sec, Colossus 2007c: 1 sec, Pepito 1.59: 1 sec, Hamsters 0.6: 5 sec, Delfi 5.2: 7 sec). Posted by Robin Smith.

This is a part of the EGTB Test Suite project. This post will be edited as new data is submitted or analyzed.

Basic test suits tests for EGTB access in current position, so only a few lookups are needed to solve each position.

(You should have no 6-men tables installed for a meaningful 5-men test. Also, GUI support for tablebases should be disabled, unless you want to test the GUI itself.)

• 8/1k6/1p1r4/5K2/8/8/8/2R5 w - - bm Ke4; c0 "Draws."; - Posted by Tano-Urayoan Russi Roman.
• 8/8/8/8/5kp1/P7/8/1K1N4 w - - bm Kc2; dm 50; - Posted by Dhanish.

Removed positions:

• 7k/4K1n1/8/6P1/8/3B4/8/8 w - - bm Bg6; dm 18; - Tests KBPKN. Unfortunately many programs manage without EGTB in a few seconds. A study by Sachodjakin, posted by Mike Scheidl.
• 8/8/7p/5pk1/8/5K2/7P/8 w - - bm Ke2; - Tests KPPKP. Only one move draws. Unfortunately some programs find it without KPPKP as well. Vaganian - Sunye Neto, 1979, posted by Tano-Urayoan Russi Roman.

This is a part of the EGTB Test Suite project. This post will be edited as new data is submitted or analyzed.

Advanced test suits tests for EGTB access in search, so each test position is outside of the table it is testing.

(You should have no 5 or 6-men tables installed for a meaningful 4-men test)

• 8/k7/8/2R5/8/4q3/8/4B2K w - - bm Bg3; c0 "Draws."; - Tests KQKR table. By Vitali Halberstadt, 1950 (starting position is 8/k7/8/B1R5/8/q7/8/7K w - - bm Be1; - unfortunately it can be solved by engines without tables), Posted by Jouni Uski.
• 5Q1K/8/8/4B3/8/8/1rp5/1k6 w - - bm Bf4; dm 32; - Tests KQKQ, KQKR and KQKP tables. By Kirill Kryukov, 2008.
• 8/6Q1/8/8/3r4/2B5/k1p5/5K2 w - - bm Qg2; dm 48; - Tests KQKQ, KQKR and KQKP tables. By Kirill Kryukov, 2008.

Removed positions:

• 5K2/2p5/8/2k5/8/2P5/4P3/8 w - - bm Kg7; dm 25; - Tests 4-men tables, mainly KPKP. Many engines can solve it without tables, still some can't. By Mikhail Afanasjevich Zinar, Shakhmaty v SSSR 1981, special prize. Sent to me by Siegfried Hornecker.

This is a part of the EGTB Test Suite project. This post will be edited as new data is submitted or analyzed.

Basic test suits tests for EGTB access in current position, so only a few lookups are needed to solve each position.

(You should have no 5 or 6-men tables installed for a meaningful 4-men test. Also, GUI support for tablebases should be disabled, unless you want to test the GUI itself.)

None so far.

Here is a position for a basic test suite for KNPkp:

8/8/8/8/5kp1/P7/8/1K1N4 w - - 0 1

Nalimov: Kb1-c2 {#50}

Most engines seem to prefer Nf2, rather than Kc2, for example:

Glaurung 1.2.1:
---------------------------------------------------------------------------
23 04:39 129.103.536 462.814 +1.68 1.Kc1 g3 2.Kd2 g2 3.Nc3 g1N 4.Kd3 Ke5 5.Kc4 Kd6 6.Kb5 Nf3 7.a4 Nd4+ 8.Kb6 Nb3 9.Na2 Nd2 10.Nb4 Nc4+ 11.Ka6 Kc5 12.Nc2 Ne5 13.a5
23 04:39 129.103.536 462.814 +2.06 1.Kc2 g3 2.Kd3 g2 3.Nc3 g1N 4.Kc4 Ke5 5.Kc5 Nf3 6.a4 Nd2 7.Ne2 Ke6 8.Kb5 Kd7 9.a5 Kc7 10.a6 Ne4 11.Nf4 Nc3+ 12.Kc5 Kb8
23 04:39 129.103.536 462.841 +2.60 1.Nf2 g3 2.Nh3+ Ke5 3.Kc2 g2 4.Kb3 Kd6 5.Kb4 Kc6 6.Ka5 Kb7 7.Kb5 Ka7 8.Ng1 Kb7 9.a4 Ka7 10.a5 Kb7 11.Kc5 Ka6 12.Kb4 Ka7

Does any engine do better?

Dhanish wrote:Here is a position for a basic test suite for KNPkp:

8/8/8/8/5kp1/P7/8/1K1N4 w - - 0 1

Nalimov: Kb1-c2 {#50}

Most engines seem to prefer Nf2, rather than Kc2, for example:

Glaurung 1.2.1:
---------------------------------------------------------------------------
23 04:39 129.103.536 462.814 +1.68 1.Kc1 g3 2.Kd2 g2 3.Nc3 g1N 4.Kd3 Ke5 5.Kc4 Kd6 6.Kb5 Nf3 7.a4 Nd4+ 8.Kb6 Nb3 9.Na2 Nd2 10.Nb4 Nc4+ 11.Ka6 Kc5 12.Nc2 Ne5 13.a5
23 04:39 129.103.536 462.814 +2.06 1.Kc2 g3 2.Kd3 g2 3.Nc3 g1N 4.Kc4 Ke5 5.Kc5 Nf3 6.a4 Nd2 7.Ne2 Ke6 8.Kb5 Kd7 9.a5 Kc7 10.a6 Ne4 11.Nf4 Nc3+ 12.Kc5 Kb8
23 04:39 129.103.536 462.841 +2.60 1.Nf2 g3 2.Nh3+ Ke5 3.Kc2 g2 4.Kb3 Kd6 5.Kb4 Kc6 6.Ka5 Kb7 7.Kb5 Ka7 8.Ng1 Kb7 9.a4 Ka7 10.a5 Kb7 11.Kc5 Ka6 12.Kb4 Ka7

Does any engine do better?

Thanks, this looks like an excellent test!

I am not sure that the following stated constraint is necessary: "wins are also wins under the 50-move rule".
If one has a position that is a win without the 50-move rule, and a draw with it, then an easy test that one has the DTZ50 EGT in use is that the engine unequivocably and instantly says "0.00". With John Tamplin and Marc Bourzutschky, I have published such exemplar positions for all endgames for which the DTZ50 EGT has been computed.
If one does not have the DTZ50 EGT in use, it does not matter that the position is not a win under the 50-move rule.

I am reminded of the Simpson's script: "What about a 'Core explosion: repent your sins' message?" "Don't be silly: if there was a core explosion, there would be no power to light the sign."
g

I think this test suite may be useful, but not really for the reason cited. If not knowing whether an EGT is online or not, the engines will shortly make EGT-status available as WILHLEM does.

Ignoring the 50-move-rule clause (see above), and looking through my maxDTM exemplar positions, the (unique) winning moves in the following positions may be less than obvious to some engines:

Suggested format (as below, pace the 'scrambling' introduced by transcription):
1 'Endgame' is the object endgame the test is about
2 'Δ' is the number of plies before the object endgame is reached

Criteria:
1 stm secures result with an absolutely unique move
2 should be difficult for (most) engines to 'pick' without EGTs

Non-criteria:
1 The win can also be achieved in the context of the 50-move rule


# Endgame Res Δ Force Positions DTC DTM DTZ Move

001 KPKP 1-0 0 KPKP 2K5/k7/7p/8/8/8/6P1/8 w 014 033 007 1.Kc7"
002 KRKB 1-0 0 KRKB 8/8/8/8/8/2R5/8/3K1bk1 w 018 029 018 1.Ke1"
003 KRKN 1-0 0 KRKN 8/2R5/8/8/7k/3K4/8/4n3 w 026 040 026 1.Ke2"
004 KRKP 1-0 0 KRKP 8/8/K7/3p4/8/3k4/4R3/8 w 012 026 008 1.Ra2"

005 KBNKB 1-0 0 KBNKB 8/7B/8/4N3/8/8/3k4/K1b5 w 008 039 008 1.Kb1"
006 KBNKN 1-0 0 KBNKN 8/8/8/8/8/8/1n5B/2K1N2k w 077 107 077 1.Bg3"
007 KBNKQ 0-1 0 KBNKQ 8/8/8/8/3B4/1q6/3KN3/k7 b 042 053 042 1.Kb1"
008 KBPKP 1-0 0 KBPKP 8/7p/8/8/8/1B6/K6P/2k5 w 053 067 009 1.Ka3"
009 KNNKP 1-0 0 KNNKP 1N6/8/p7/8/8/8/2k1N3/K7 w 114 115 042 1.Ka2"
010 KRBKB 1-0 0 KRBKB 8/8/8/8/8/3R4/6k1/3K1bB1 w 008 030 080 1.Rc3"

g

guyhaw wrote:I am not sure that the following stated constraint is necessary: "wins are also wins under the 50-move rule".
If one has a position that is a win without the 50-move rule, and a draw with it, then an easy test that one has the DTZ50 EGT in use is that the engine unequivocably and instantly says "0.00". With John Tamplin and Marc Bourzutschky, I have published such exemplar positions for all endgames for which the DTZ50 EGT has been computed.
If one does not have the DTZ50 EGT in use, it does not matter that the position is not a win under the 50-move rule.

I am reminded of the Simpson's script: "What about a 'Core explosion: repent your sins' message?" "Don't be silly: if there was a core explosion, there would be no power to light the sign."
g

Hi Guy! This test suite is intended to test presence of any tablebases, including WDL type. Since there is no such test suite existing yet (which is reliable and checked with many engines), this is what has to be done first. The distinction between DTZ50 and DTM is more subtle and will have to be covered by another, separate test suite.

Also, there are no any available DTZ50 tables (and code) to speak of, so there is no practical way someone (other than you, John Tamplin or Marc Bourzutschky?) can create or work with a 50-moves sensitive test positions.

guyhaw wrote:I think this test suite may be useful, but not really for the reason cited. If not knowing whether an EGT is online or not, the engines will shortly make EGT-status available as WILHLEM does.

Not all engines report that they use tablebases.

guyhaw wrote:Ignoring the 50-move-rule clause (see above), and looking through my maxDTM exemplar positions, the (unique) winning moves in the following positions may be less than obvious to some engines:

Suggested format (as below, pace the 'scrambling' introduced by transcription):
1 'Endgame' is the object endgame the test is about
2 'Δ' is the number of plies before the object endgame is reached

Criteria:
1 stm secures result with an absolutely unique move
2 should be difficult for (most) engines to 'pick' without EGTs

Non-criteria:
1 The win can also be achieved in the context of the 50-move rule


# Endgame Res Δ Force Positions DTC DTM DTZ Move

001 KPKP 1-0 0 KPKP 2K5/k7/7p/8/8/8/6P1/8 w 014 033 007 1.Kc7"
002 KRKB 1-0 0 KRKB 8/8/8/8/8/2R5/8/3K1bk1 w 018 029 018 1.Ke1"
003 KRKN 1-0 0 KRKN 8/2R5/8/8/7k/3K4/8/4n3 w 026 040 026 1.Ke2"
004 KRKP 1-0 0 KRKP 8/8/K7/3p4/8/3k4/4R3/8 w 012 026 008 1.Ra2"

005 KBNKB 1-0 0 KBNKB 8/7B/8/4N3/8/8/3k4/K1b5 w 008 039 008 1.Kb1"
006 KBNKN 1-0 0 KBNKN 8/8/8/8/8/8/1n5B/2K1N2k w 077 107 077 1.Bg3"
007 KBNKQ 0-1 0 KBNKQ 8/8/8/8/3B4/1q6/3KN3/k7 b 042 053 042 1.Kb1"
008 KBPKP 1-0 0 KBPKP 8/7p/8/8/8/1B6/K6P/2k5 w 053 067 009 1.Ka3"
009 KNNKP 1-0 0 KNNKP 1N6/8/p7/8/8/8/2k1N3/K7 w 114 115 042 1.Ka2"
010 KRBKB 1-0 0 KRBKB 8/8/8/8/8/3R4/6k1/3K1bB1 w 008 030 080 1.Rc3"

g

Thanks for positions, I or someone will check when we have time. Please note that in this project the requirement is not only "should be difficult for (most) engines to 'pick' without EGTs", but a more strong "should be difficult for (most) engines to 'pick' with (N-1)-man EGTs".

Good point about the positions being difficult for (most) engines with the relevant sub-endgame EGTs. In that case, we should be looking at the maxDTC(onversion) positions rather than the maxDTM(ate) positions. Will take another look tomorrow.
g

This contribution supercedes my previous one which was maxDTM-oriented. This is largely based on maxDTC positions on file. There are probably plenty of Chess Studies which are technically correct and offer a suitable test.


# Endgame Res. Δ plies Force Position DTC DTM DTZ Move

1 KPKP 1-0 0 KPKP 2K5/k7/7p/8/8/8/6P1/8 w 014 033 007 Kc7"
2 KRKB 1-0 0 KRKB 8/8/8/8/8/2R5/8/3K1bk1 w 018 029 018 Ke1"
3 KRKN 1-0 0 KRKN 5R2/8/8/8/8/k7/4n3/2K5 w 027 039 027 Kd2"
4 KRKP 1-0 0 KRKP K7/8/7p/8/7k/R7/8/8 w 016 025 013 Ra7"

5 KBNKB 1-0 0 KBNKB 8/8/8/2B3N1/8/8/8/k1K1b3 w 013 038 013 Kc2"
6 KBNKN 1-0 0 KBNKN 8/8/8/8/8/8/1n5B/2K1N2k w 077 107 077 Bg3"
7 KBNKQ 0-1 0 KBNKQ 8/8/8/8/3B4/1q6/3KN3/k7 b 042 053 042 Kb1"
8 KBPKP 1-0 0 KBPKP 8/7p/8/8/8/1B6/K6P/2k5 w 053 067 009 Ka3"
9 KNNKP 1-0 0 KNNKP 1N6/8/p7/8/8/8/2k1N3/K7 w 114 115 042 Ka2"
10 KRBKB 1-0 0 KRBKB 8/8/8/b7/8/1R6/B1k5/K7 w 025 027 025 Rb2+

11 KBBKBN 1-0 0 KBBKBN 8/8/k7/8/8/4B3/6n1/1BK4b w 028 090 028 Bd3+"
12 KBBKNN 1-0 0 KBBKNN 8/8/7B/5B2/3K4/1k6/8/n6n w 038 106 038 Be6+"
13 KBNKNN 1-0 0 KBNKNN 8/8/N7/n1B5/8/8/k1K5/7n w 013 024 013 Nb4+"
14 KQBKQB 1-0 0 KQBKQB 7q/1B5b/8/8/7Q/3k4/8/3K4 w 046 058 046 Ba6+"
15 KQBKQN 1-0 0 KQBKQN qn6/8/8/8/7B/3K4/2Q5/6k1 w 035 035 035 Bf2+"
16 KQBKRB 1-0 0 KQBKRB 8/8/8/8/4b3/5k2/3K1r2/B1Q5 w 042 053 042 Ke1"
17 KQBKRN 1-0 0 KQBKRN 8/8/1r6/3B4/3n4/3k4/8/1K4Q1 w 027 053 027 Ka2"
18 KQBKRR 1-0 0 KQBKRR 3r4/6Q1/5B2/8/8/5k2/2Kr4/8 w 085 107 085 Kb3"
19 KQNKBN 1-0 0 KQNKBN 7N/8/8/8/3b4/1Q6/3kn3/K7 w 017 030 017 Kb1"
20 KQNKNN 1-0 0 KQNKNN 8/N7/3Q4/8/8/n4k2/4n3/2K5 w 016 026 016 Kd1"
21 KQNKQN 1-0 0 KQNKQN 2q5/4Q3/n3N3/8/6k1/3K4/8/8 w 029 037 029 Qg5+"
22 KQNKRB 1-0 0 KQNKRB 8/r7/8/6Q1/8/2b5/1k1K4/N7 w 026 040 026 Kd3"
23 KQNKRR 1-0 0 KQNKRR 3r2r1/8/k7/8/8/8/3K4/1Q4N1 w 152 174 152 Ke3"
24 KQQKQB 1-0 0 KQQKQB 8/8/1k6/1q6/1b4Q1/8/3K4/7Q w 062 071 062 Kc1"
25 KQQKQQ 1-0 0 KQQKQQ 8/7k/8/1q1q4/8/8/8/K4QQ1 w 044 050 044 Qh2+"
26 KQQKQR 1-0 0 KQQKQR 7k/8/8/8/8/r1K5/q2Q4/3Q4 w 048 069 048 Kd4"
27 KQRKNN 1-0 0 KQRKNN 8/8/3Q4/8/8/n1n2k2/8/RK6 w 011 018 011 Kc1"
28 KQRKQN 1-0 0 KQRKQN 8/4nq2/k7/6R1/8/4Q3/8/3K4 w 071 083 071 Qe2+"
29 KQRKRB 1-0 0 KQRKRB 7R/8/8/8/7b/r6k/8/K5Q1 w 021 027 021 Kb2"
30 KRBKBB 1-0 0 KRBKBB k7/8/8/8/1R6/3b4/8/1K2B2b w 083 087 083 Kb2"
31 KRBKBN 1-0 0 KRBKBN B7/8/8/4b3/8/1R6/3kn3/K7 w 098 112 098 Kb1"
32 KRNKNN 1-0 0 KRNKNN 8/8/8/8/2n2k2/2n5/5BR1/1K6 w 223 238 223 Kc1"
33 KRBKRB 1-0 0 KRBKRB 3r4/3B2R1/8/6b1/8/8/8/k2K4 w 017 040 017 Kc2"
34 KRBKRN 1-0 0 KRBKRN 7n/8/r7/8/4R3/8/2BK4/k7 w 021 041 021 Kc3"
35 KRNKBB 1-0 0 KRNKBB 4N1R1/8/2b5/4kb2/8/8/8/K7 w 140 152 140 Nc7"
36 KRNKBN 1-0 0 KRNKBN NbR5/8/n7/8/8/8/8/2K2k2 w 190 209 190 Kd1"
37 KRNKNN 1-0 0 KRNKNN 6k1/5n2/8/8/8/5n2/1RK5/1N6 w 243 262 243 Kd3"
38 KRRKBB 1-0 0 KRRKBB 8/8/1b6/3b4/4R3/8/5k2/K6R w 037 045 037 Rh2+"
39 KRRKBN 1-0 0 KRRKBN 8/4k1b1/3R4/8/4n3/8/8/1R1K4 w 026 035 026 Rg6"
40 KRRKNN 1-0 0 KRRKNN 8/5n2/4R3/3k4/8/7n/3K1R2/8 w 033 043 033 Rff6
41 KRRKRB 1-0 0 KRRKRB 5R2/8/8/8/1k1r3R/8/2K5/b7 w 054 064 054 Rb8+"
42 KRRKRN 1-0 0 KRRKRN r7/8/8/6n1/8/1k6/3RR3/2K5 w 073 089 073 Rd3+"
43 KRRKRR 1-0 0 KRRKRR 1R6/rR1r4/k7/8/8/8/2K5/8 w 018 040 018 Rb6+"

44 KBBBKB 1-0 0 KBBBKB 8/8/5b2/8/8/3k4/1K1B3B/7B w 020 024 020 Kc1"
45 KBBNKB 1-0 0 KBBNKB 8/8/B7/2N4B/3k4/8/8/1K1b4 w 036 053 036 Ne6+"
46 KBBNKN 1-0 0 KBBNKN 2N5/8/8/1k3n2/8/B7/1B6/2K5 w 031 050 031 Bc3"
47 KBNNKN 1-0 0 KBNNKN 7N/8/1n6/8/8/3N4/2k5/K1B5 w 028 044 028 Be3"
48 KBNNKQ 1-0 0 KBNNKQ 8/8/8/5B2/q3N3/2K5/8/2k1N3 w 012 034 012 Nd3+"
49 KBNNKR 1-0 0 KBNNKR 4B3/8/5r2/8/3N1N2/8/3K4/7k w 049 056 049 Ke3"
50 KQBBKQ 1-0 0 KQBBKQ 8/6k1/4B3/8/6B1/3q4/8/1K5Q w 093 096 093 Kb2"
51 KQBNKQ 1-0 0 KQBNKQ 7q/8/8/8/5B1N/2K4k/5Q2/8 w 065 084 065 Kb3"
52 KQRBKQ 1-0 0 KQRBKQ R4Q2/1k6/7B/8/8/8/8/2K3q1 w 049 052 049 Kc2"
53 KQRNKQ 1-0 0 KQRNKQ 8/8/2q5/6k1/8/2K5/N7/QR6 w 055 061 055 Kd3"
54 KQRRKQ 1-0 0 KQRRKQ 6Q1/8/8/8/4kq2/1R6/8/R1K5 w 041 045 041 Kb1"
55 KRBBKR 1-0 0 KRBBKR 8/8/2r2B2/4R3/8/6B1/2k5/K7 w 036 039 036 Bd8"
56 KRBNKQ 1-0 0 KRBNKQ R3B3/8/8/1N6/8/3K4/8/1k4q1 w 099 121 099 Na3+"
57 KRNNKB 1-0 0 KRNNKB 8/8/6b1/8/4R3/8/3k3N/1K4N1 w 012 018 012 Nf1+"
58 KRNNKN 1-0 0 KRNNKN 8/8/8/8/2k5/1n1R4/8/2K3NN w 013 017 013 Kc2"
59 KRRNKQ 1-0 0 KRNNKQ 5R1N/8/6R1/4q3/7k/8/8/2K5 w 100 107 100 Rgf6"

g

guyhaw wrote:This contribution supercedes my previous one which was maxDTM-oriented. This is largely based on maxDTC positions on file. There are probably plenty of Chess Studies which are technically correct and offer a suitable test.


# Endgame Res. Δ plies Force Position DTC DTM DTZ Move

1 KPKP 1-0 0 KPKP 2K5/k7/7p/8/8/8/6P1/8 w 014 033 007 Kc7"
2 KRKB 1-0 0 KRKB 8/8/8/8/8/2R5/8/3K1bk1 w 018 029 018 Ke1"
3 KRKN 1-0 0 KRKN 5R2/8/8/8/8/k7/4n3/2K5 w 027 039 027 Kd2"
4 KRKP 1-0 0 KRKP K7/8/7p/8/7k/R7/8/8 w 016 025 013 Ra7"

5 KBNKB 1-0 0 KBNKB 8/8/8/2B3N1/8/8/8/k1K1b3 w 013 038 013 Kc2"
6 KBNKN 1-0 0 KBNKN 8/8/8/8/8/8/1n5B/2K1N2k w 077 107 077 Bg3"
7 KBNKQ 0-1 0 KBNKQ 8/8/8/8/3B4/1q6/3KN3/k7 b 042 053 042 Kb1"
8 KBPKP 1-0 0 KBPKP 8/7p/8/8/8/1B6/K6P/2k5 w 053 067 009 Ka3"
9 KNNKP 1-0 0 KNNKP 1N6/8/p7/8/8/8/2k1N3/K7 w 114 115 042 Ka2"
10 KRBKB 1-0 0 KRBKB 8/8/8/b7/8/1R6/B1k5/K7 w 025 027 025 Rb2+

11 KBBKBN 1-0 0 KBBKBN 8/8/k7/8/8/4B3/6n1/1BK4b w 028 090 028 Bd3+"
12 KBBKNN 1-0 0 KBBKNN 8/8/7B/5B2/3K4/1k6/8/n6n w 038 106 038 Be6+"
13 KBNKNN 1-0 0 KBNKNN 8/8/N7/n1B5/8/8/k1K5/7n w 013 024 013 Nb4+"
14 KQBKQB 1-0 0 KQBKQB 7q/1B5b/8/8/7Q/3k4/8/3K4 w 046 058 046 Ba6+"
15 KQBKQN 1-0 0 KQBKQN qn6/8/8/8/7B/3K4/2Q5/6k1 w 035 035 035 Bf2+"
16 KQBKRB 1-0 0 KQBKRB 8/8/8/8/4b3/5k2/3K1r2/B1Q5 w 042 053 042 Ke1"
17 KQBKRN 1-0 0 KQBKRN 8/8/1r6/3B4/3n4/3k4/8/1K4Q1 w 027 053 027 Ka2"
18 KQBKRR 1-0 0 KQBKRR 3r4/6Q1/5B2/8/8/5k2/2Kr4/8 w 085 107 085 Kb3"
19 KQNKBN 1-0 0 KQNKBN 7N/8/8/8/3b4/1Q6/3kn3/K7 w 017 030 017 Kb1"
20 KQNKNN 1-0 0 KQNKNN 8/N7/3Q4/8/8/n4k2/4n3/2K5 w 016 026 016 Kd1"
21 KQNKQN 1-0 0 KQNKQN 2q5/4Q3/n3N3/8/6k1/3K4/8/8 w 029 037 029 Qg5+"
22 KQNKRB 1-0 0 KQNKRB 8/r7/8/6Q1/8/2b5/1k1K4/N7 w 026 040 026 Kd3"
23 KQNKRR 1-0 0 KQNKRR 3r2r1/8/k7/8/8/8/3K4/1Q4N1 w 152 174 152 Ke3"
24 KQQKQB 1-0 0 KQQKQB 8/8/1k6/1q6/1b4Q1/8/3K4/7Q w 062 071 062 Kc1"
25 KQQKQQ 1-0 0 KQQKQQ 8/7k/8/1q1q4/8/8/8/K4QQ1 w 044 050 044 Qh2+"
26 KQQKQR 1-0 0 KQQKQR 7k/8/8/8/8/r1K5/q2Q4/3Q4 w 048 069 048 Kd4"
27 KQRKNN 1-0 0 KQRKNN 8/8/3Q4/8/8/n1n2k2/8/RK6 w 011 018 011 Kc1"
28 KQRKQN 1-0 0 KQRKQN 8/4nq2/k7/6R1/8/4Q3/8/3K4 w 071 083 071 Qe2+"
29 KQRKRB 1-0 0 KQRKRB 7R/8/8/8/7b/r6k/8/K5Q1 w 021 027 021 Kb2"
30 KRBKBB 1-0 0 KRBKBB k7/8/8/8/1R6/3b4/8/1K2B2b w 083 087 083 Kb2"
31 KRBKBN 1-0 0 KRBKBN B7/8/8/4b3/8/1R6/3kn3/K7 w 098 112 098 Kb1"
32 KRNKNN 1-0 0 KRNKNN 8/8/8/8/2n2k2/2n5/5BR1/1K6 w 223 238 223 Kc1"
33 KRBKRB 1-0 0 KRBKRB 3r4/3B2R1/8/6b1/8/8/8/k2K4 w 017 040 017 Kc2"
34 KRBKRN 1-0 0 KRBKRN 7n/8/r7/8/4R3/8/2BK4/k7 w 021 041 021 Kc3"
35 KRNKBB 1-0 0 KRNKBB 4N1R1/8/2b5/4kb2/8/8/8/K7 w 140 152 140 Nc7"
36 KRNKBN 1-0 0 KRNKBN NbR5/8/n7/8/8/8/8/2K2k2 w 190 209 190 Kd1"
37 KRNKNN 1-0 0 KRNKNN 6k1/5n2/8/8/8/5n2/1RK5/1N6 w 243 262 243 Kd3"
38 KRRKBB 1-0 0 KRRKBB 8/8/1b6/3b4/4R3/8/5k2/K6R w 037 045 037 Rh2+"
39 KRRKBN 1-0 0 KRRKBN 8/4k1b1/3R4/8/4n3/8/8/1R1K4 w 026 035 026 Rg6"
40 KRRKNN 1-0 0 KRRKNN 8/5n2/4R3/3k4/8/7n/3K1R2/8 w 033 043 033 Rff6
41 KRRKRB 1-0 0 KRRKRB 5R2/8/8/8/1k1r3R/8/2K5/b7 w 054 064 054 Rb8+"
42 KRRKRN 1-0 0 KRRKRN r7/8/8/6n1/8/1k6/3RR3/2K5 w 073 089 073 Rd3+"
43 KRRKRR 1-0 0 KRRKRR 1R6/rR1r4/k7/8/8/8/2K5/8 w 018 040 018 Rb6+"

44 KBBBKB 1-0 0 KBBBKB 8/8/5b2/8/8/3k4/1K1B3B/7B w 020 024 020 Kc1"
45 KBBNKB 1-0 0 KBBNKB 8/8/B7/2N4B/3k4/8/8/1K1b4 w 036 053 036 Ne6+"
46 KBBNKN 1-0 0 KBBNKN 2N5/8/8/1k3n2/8/B7/1B6/2K5 w 031 050 031 Bc3"
47 KBNNKN 1-0 0 KBNNKN 7N/8/1n6/8/8/3N4/2k5/K1B5 w 028 044 028 Be3"
48 KBNNKQ 1-0 0 KBNNKQ 8/8/8/5B2/q3N3/2K5/8/2k1N3 w 012 034 012 Nd3+"
49 KBNNKR 1-0 0 KBNNKR 4B3/8/5r2/8/3N1N2/8/3K4/7k w 049 056 049 Ke3"
50 KQBBKQ 1-0 0 KQBBKQ 8/6k1/4B3/8/6B1/3q4/8/1K5Q w 093 096 093 Kb2"
51 KQBNKQ 1-0 0 KQBNKQ 7q/8/8/8/5B1N/2K4k/5Q2/8 w 065 084 065 Kb3"
52 KQRBKQ 1-0 0 KQRBKQ R4Q2/1k6/7B/8/8/8/8/2K3q1 w 049 052 049 Kc2"
53 KQRNKQ 1-0 0 KQRNKQ 8/8/2q5/6k1/8/2K5/N7/QR6 w 055 061 055 Kd3"
54 KQRRKQ 1-0 0 KQRRKQ 6Q1/8/8/8/4kq2/1R6/8/R1K5 w 041 045 041 Kb1"
55 KRBBKR 1-0 0 KRBBKR 8/8/2r2B2/4R3/8/6B1/2k5/K7 w 036 039 036 Bd8"
56 KRBNKQ 1-0 0 KRBNKQ R3B3/8/8/1N6/8/3K4/8/1k4q1 w 099 121 099 Na3+"
57 KRNNKB 1-0 0 KRNNKB 8/8/6b1/8/4R3/8/3k3N/1K4N1 w 012 018 012 Nf1+"
58 KRNNKN 1-0 0 KRNNKN 8/8/8/8/2k5/1n1R4/8/2K3NN w 013 017 013 Kc2"
59 KRRNKQ 1-0 0 KRNNKQ 5R1N/8/6R1/4q3/7k/8/8/2K5 w 100 107 100 Rgf6"

g

Tested the above positions on four engines at 5s per move on my old Pentium III 600 with the following results:

Movei00_8_438 could find the correct move without any tablebases for all but the following positions: 6, 8, 9, 14, 28, 32, 34, 35, 36, 42, 46, 55

Rybka v1.0 Beta.w32 could find the correct move without any tablebases for all but the following positions: 2, 4, 5, 6, 8, 9, 14, 23, 25, 28, 32, 35, 36, 42, 46, 49, 53, 55, 56, 59

Spike1.2 could find the correct move without any tablebases for all but the following positions: 4, 5, 9, 11, 18, 23, 24, 25, 28, 34, 35, 46, 59

Glaurung-32bit 1.2.1 could find the correct move without any tablebases for all but the following positions: 4, 6, 8, 11, 12, 23, 33, 34, 35, 36, 37, 42, 46, 53

Hence remaining candidate test positions: 28, 34, 35, 46

Added 27 Feb 2008:
Allowing one failure, remaining positions are: 4, 6, 8, 9, 23, 28, 34, 35, 36, 42, 46

Tested these remaining positions with Crafty 19.20 BH without any tablebases for maximum 90s and it finds correct move for 4 in 23 seconds, 8 in 2 seconds, 23 in 2 seconds.

Hence remaining candidate positions: 6, 9, 28, 34, 35, 36, 42, 46

Tested these remaining positions with Bright 0.2c without any tablebases for maximum 90s and it finds correct move for 9 in 2s.

Ktulu 4.2 finds correct move for 6 in 67s, 36 in 17s.

Shredder 10 solves 42 in 10s and Fritz 8 in 22s.

Hence remaining candidate positions: 28, 34, 35, 46

Update: 29 Feb 2008:

28 & 35 have DTZ>50, can we include in the Testsuite?

Position 34: Scorpio 2.0 chooses correct move after 3s, though the score remains low! (First failure)

Enthusiastic and interesting results from Dhanish, but I don't follow the logic of the "Hence". Maybe I am missing the point.
My understanding is that these positions are meant to help the engine-user understand whether a relevant EGT is accessible or not.
If the EGT is accessible, I would expect the engine to come back straight away with a "win in M". If the EGT is not accessible, these positions should be tough enough to cause enough of a delay before a decision is made. I'm not sure whether one should give one's engine 2 minutes to decide or a short time (as Dh does): should we not be forcing the engine to return a result quickly?

Chess Studies typically have White walking a tightrope to a result, and (for aesthetic reasons) should not feature immediate exchanges. So, if they are correct (and that's a big IF), they should provide a major challenge if the main line wanders into the 'EGT Zone'. I don't have any 6-man P-ful maxDTC positions.
g

guyhaw wrote:I don't follow the logic of the "Hence".
g

It appears you don't use engines much, otherwise of course you would have tested the positions yourself. Why do we need such a test suite?

I use engines frequently for analysis of my corr games, and I have discovered problems with tablebase access, the hard way. The Fritz GUI can itself access tablebases. But you have to restart the GUI after enabling it, otherwise the access does not happen sometimes. After changing the tablebases path in the Fritz GUI, the current engine is not able to access tablebases immediately. One has to start another engine and then back to the old engine. Sometimes you may enter the path wrongly, and you may not notice it while analysing. Some engines cannot access tablebases at all. Some engines can access only bitbases. For Shredder 10, to enable Shredderbases, one has to edit the bases.ini file and restart the engine and how does one verify whether it has been correctly done? A version of Toga claiming to access Scorpio bitbases, but actually unable to do so, was released to public! Further, there is always a possibility that a tablebase file has become corrupt during use.

So there is a necessity to test whether the engine is able to access the tablebase. And to automate this testing, we should be able to use the facility available in GUIs. Fritz GUI has an option: Process Test set. It will analyse a series of positions for a certain time or depth and if the best move chosen by the engine matches with the solution given in the test set, it is counted. Unfortunately, it does not consider the engine score, whether it is 0.00 or +32767. There is a similar facility available in Arena also. By analysing the test results, we should be able to say which tablebase or bitbases are being accessed successfully.

That is why the second sentence of requirement 2: "Finding correct move should take a few seconds for an engine accessing the endgame table being tested."

While making a test suite, we cannot ensure that engines developed in future will not be able to find the solution, but at least we should ensure that common engines are not able to do so.

I chose 5s, because allowing too short a time causes the GUI to crash more frequently. (Perhaps my system is too old and RAM is insufficient).

Further in case of bitbases, engines do not give "win in M".

Hence my "hence".

Dhanish

Thanks Dhanish, this is a good explanation, and exactly the reason (one the reasons) I started this initiative. Also thanks for analysis of the positions.

I usually use 1 min per position per engine, and I test with about 20 engines that access EGTB (testing with target EGTB both present and absent). That's why I take long time updating the test suite. Though I don't require a successful test to work for 100% of the engines. Often an engine or two will still solve it without tables. I appreciate any contributed analysis of the posted positions. (It's better to post the positions together with the convincing analysis actually).

By the way some engines are pretty good in solving test positions without EGTB. Movei 00.8.438 is one of the good ones in my experience.

Guy, thanks for positions! By the way, a good EGTB test does not necessarily have to be "Max something", an ordinary positions can be a good test if it manages to trick an engine (that has no table) into wrong direction. This is what current positions in the test are doing.

Seconded: thanks to Dhanish (and KK) for further explanation of the background for the test suite. Dh is about right in saying that I don't use engines much: I tend to use FRITZ or SHREDDER with 5-man EGTs on, and haven't had any problems knowing whether they are on or not. Having said that, I'm usually looking at individual positions or Chess Studies.
I understand that a position does not have to be a maxDTx position, but - as I had some maxDTC examples, I thought they'd have a reasonable chance of keeping the engines away from the subgame EGTs.
g

Glad you found the explanation satisfactory.

Kirill Kryukov wrote:I usually use 1 min per position per engine, and I test with about 20 engines

It was mentioned "a few seconds" in the requirements post, hence I chose 5 seconds. I feel 1 minute may be too high with good hardware, it will be difficult to get positions for the test set. With tablebases, 1 or 2 seconds are good enough to get the correct result. We only need to differentiate the result with this, isn't it? Maybe if tablebases are stored on USB sticks or other media, disk access time could be more. Eigenmann uses 60s per move, http://64.68.157.89/forum/viewtopic.php?t=14004 .

If testing has to be carried out by others also, we need to standardise it. Perhaps the Axon test can be used to standardise the hardware. Multiprocessor engines may have to be used as single processor ones, for equivalent times? And of course the test engine set. What are the engines that you try?

I have Fritz 8 and Shredder 10 but they work only in Fritz GUI. And Fritz GUI cannot use EPD files. Is there a way to convert EPD files to PGN files or vice versa? Arena can use EPD files as well as PGN files for Automatic Analysis of several engines one by one.

{Update on 28 Feb 2008: A search uncovered EPDt2diag at http://www.rebel.nl/epd2diag.htm . But unfortunately it is converting the best move field to a Black move tag. Then found another EPD to PGN utility at http://www.geocities.com/fhub55/chess/ . But this is also skipping the bm field!

Dhanish wrote: If testing has to be carried out by others also, we need to standardise it. Perhaps the Axon test can be used to standardise the hardware.

Axon benchmark uses 16-bit code and won't run under 64-bit Windows, so it's not a good choice
Crafty or some other benchmark perhaps