I realized that we are still lacking reliable way to test whether an engine is using endgame tablebases or not. Often you install a new engine and wonder if it is accessing the tablebases. These days many engines use bitbases which are more hard to check.
We need a way to test if an engine is actually accessing the tablebases. So I propose to create a set of positions for testing each piece configuration (or as many as possible).
Theoretically such test suite would be also useful for a developer, working on an engine, a GUI, or a new EGTB format.
Two test suites are proposed: basic test suite and advanced test suite. Basic test suite will consist of positions that are contained in the endgame tables they are testing. Such positions test for capability to probe endgame table at the root of search.
Advanced test suite will consist of positions that are not contained in the endgame tables they are testing. Such positions will test for engine's ability to access the tablebase in search.
Requirements
Each position should meet these criteria:
- Position should have only one winning move, or only one drawing move if there are no winning moves. This requirement is needed to make this test suite useful for bitbases too.
- Position should be trivial to solve for an engine that probes the relevant table in its search. Finding correct move should take a few seconds for an engine accessing the endgame table being tested.
- Position should be hard to solve without that particular table (or set of tables) it is testing. An engine that does not have access to the table should prefer wrong moves, even if it has all other tables (with the same number of pieces).
- For the advanced test suite, position should be a few moves before the endgame it is testing (so some captures will follow to convert into the EGTB position). For the basic test suite, position can be within the table it is testing.
- Solution should be not sensitive to the 50 moves rule. This means a win should be still a win if 50 moves rule is used. And a draw should not depend on 50 moves rule to be a draw. (This way the test will work the same way for both current Nalimov tables that ignore 50 moves rule and for future tables that will take it into account).
- Position should be legal, if possible.
Please post your positions here or send via email to kkryukov ~ at ~ gmail.com with subject "EGTB Test Suite". Also you can post them on CCC or anywhere else you think I'll find them. Please use PGN, EPD or FEN format.
Please clearly indicate the author of all positions you submit. If you know, please also include where and when it was published.
If will help a lot if you will include your analysis together with your submitted position. What endgame table it is testing? What engines did you try? How long they take to find the correct move when they access the table in question? How long do they take and can they solve it at all without the table?
Note that all positions submitted without analysis will have to wait for my free time which occurs every once in a long while.
Tips and tricks for creating a good test position
Many endgame studies can serve as a good EGTB test.
You can try using my collection of longest checkmate positions as a starting point.
In case you are missing some EGTB files, you know where to get them.
If your position falls within a 6-piece table, you can check it with online EGTB interface.
Disable EGTB support in your GUI when you are testing an engine or working on a test position.
The best way to select a particular set of endgame tables is by moving the files around on your hard drive. Move the EGTB files you don't want to use into another directory, and reload the engine.
Wilhelm program can mine tablebases for specific types of positions, this can be useful to find good tests.
"Just an observation: some positions in which a pawn is promoted followed by a long series of moves to keep a draw in hand tend to be mishandled by computer analysis: the pawn poised to promote is scored too high, and so an unassisted computer program keeps it rather than reach the promotion case which doesn't score anything but a draw." - ath.
How you can help
- Create a good test position and submit with detailed analysis.
- Look through published endgame studies, find a good position, make sure it qualifies and submit with detailed analysis.
- Analyze any of the positions submitted without analysis and post your results.
- Spread information about this project to attract more people.
- If you have an study composer friend, ask him if he has any positions that may be a good tablebase test. (Or if he can create one).
- Anything else you can think of.
Final remarks
As you can see it is not a trivial task to construct a perfect test position for an endgame table. For a start, may be some positions for testing 4-men or 5-men tables can be made.
Eventually we should have a test position for each piece configuration, so running an engine through the test suite will reveal any missing tables and test whether that engine is able to use them. Note that this test would be useful not only for Nalimov tables, but for any other format as well.
Any questions, comments, ideas and suggestions are welcome!
Thanks!
Positions received so far
Tests for 6-men tables
Tests for 5-men tables
Tests for 4-men tables
Other