7-men EGTB Bounty

[Lutz Neweklowsky]

Hallo friends of chessproblems ,
my mate in 530 is completely generated to DTM ( not only to DTC ) on my pc , and so there doesn`t exist a shorter solution. Have for example a look at my completely generated mate in 267 in wikipedia http://en.wikipedia.org/wiki/Chess_problem
kindly regards
Lutz Neweklowsky
Drais-1
76135 Karlsruhe
Germany
lutz.neweklowsky@gmx.de

[syzygy]

Most of tables will be generated in WDL or WDL50 first. Then, some important tables will be generated in DTZ or DTZ50, by those who want them. Having a mix of DTZ and WDL is the unique and strong point of our approach.

WDL files are easy to exchange and store. We won't be able to afford massive DTZ for a long time. So it makes perfect sense to incorporate a very fast WDL-only generator, in addition to the logical path DTZ->WDL.

I agree it won't be feasible to store and distribute DTZ or DTZ50 for all 7-piece tables for a long time. So let's suppose that it is feasible to store and distribute WDL or WDL50.

Do you agree that for practical use WDL50 is not very useful, because no human or engine will be able to play perfectly in respect of the 50-move rule without the DTZ50-information?

If WDL50 is available without DTZ50, then engines will find draws without being able to draw them. Suppose it takes 5 years before DTZ50 becomes feasible. During all that time engines will be losing drawn positions. (In my view a lot worse than not being able to win a won position. People will regard the tables as buggy if draws are lost.)

Yes, but WDL50 will allow easy creation of DTZ50, for example for kpppkpp ending. So it makes sense to create WDL50.

IMO it makes sense to create WDL50 at the same time as generating DTZ50. It will not take much time to derive a WDL50 table from a DTZ50 table. Furthermore, it is not very helpful to have kpppkpp in DTZ50 without having the subtables in DTZ50.

A more technical issue is that it is very hard (and likely impossible) to perform an internal consistency check on WDL50. A 1-ply search on a position in a WDL, DTZ or DTZ50-table will return the value stored for that position. This is not the case for a WDL50-table. One more reason to derive WDL50 from DTZ50.

[Kirill Kryukov]

I agree it won't be feasible to store and distribute DTZ or DTZ50 for all 7-piece tables for a long time. So let's suppose that it is feasible to store and distribute WDL or WDL50.

Do you agree that for practical use WDL50 is not very useful, because no human or engine will be able to play perfectly in respect of the 50-move rule without the DTZ50-information?

If WDL50 is available without DTZ50, then engines will find draws without being able to draw them. Suppose it takes 5 years before DTZ50 becomes feasible. During all that time engines will be losing drawn positions. (In my view a lot worse than not being able to win a won position. People will regard the tables as buggy if draws are lost.)

I am not sure that with WDL50 losing a drawn position will happpen more often than drawing a won position. And I also don't think that to lose a drawn position is worse than to draw a won position. In both cases you lose a well deserved 0.5 of a point.

IMO it makes sense to create WDL50 at the same time as generating DTZ50. It will not take much time to derive a WDL50 table from a DTZ50 table. Furthermore, it is not very helpful to have kpppkpp in DTZ50 without having the subtables in DTZ50.

A more technical issue is that it is very hard (and likely impossible) to perform an internal consistency check on WDL50. A 1-ply search on a position in a WDL, DTZ or DTZ50-table will return the value stored for that position. This is not the case for a WDL50-table. One more reason to derive WDL50 from DTZ50.

Yes it is totally fine to generate WDL50 at the same time as DTZ50. It is also fine to generate WDL50 from a computed DTZ50.

It became clear by now that both WDL+WDT camp and WDL50+DTZ50 camps are strongly represented in the community. (I still estimate the WDL50+DTZ50 camp to be larger, even though it is not obvious from reading this forum). This means that the new generator has to support all of these metrics. The exact method how it is done is up to the generator author. The choice if the metric to use and the order in which the tables are generated is up to the community. This is how I see it. Current requirements allow for any meaningful scenario to happen.

I want to stress that once we actually start the bounty project (which we are still discussing now), we won't be able to change the requirements. So let's see if we are not missing anything and then we can progress to the next step.

[kronsteen]

WDL or WDL50 tables can’t prevent KK’s scenario (engine starting from a won position but going in circles and finally conceding 50-move rule draw) from happening. This is due to lack of “winning line” info. This is the reason why DTZ or DTZ50 info is also needed for “not clear-cut” endings.

On the other hand, Syz’s pitfall (engine defending a genuine draw but converting it blindly into a “blessed loss” then finally losing it) can easily be avoided : it needs only to have WDL table (not WDL50) available, which identifies genuine draws. This points out that, should only WDL tables be available, WDL seems better suited than WDL50 for practical purposes (and WDL+WDL50 together would be even better). But it is only one argument among a lot of other considerations.

It comes down to this : we might not only write requirements, but also justifications of these requirements. Justifications help everybody to understand why current requirements, and not other ones, have been chosen, and will make this information traceable and not forgotten in the future. Justification of currently chosen metrics might be given in 3 domains : added value for practical play, added value for problem composing, and computing issues. Here is what I think of it (this is to be criticized / corrected / completed, of course) :

WDL :
Playing : Helps playing programs by fully avoiding theoretical blunders, i.e moves that convert a win into a draw (or loss), or a draw into a loss. Note : this is to be considered as good help but doesn’t lead to perfect play as it 1) doesn’t take 50 move rule into account and 2) won’t prevent a bad chess engine from getting astray by competent defence and finally conceding a 50-move rule draw in a theoretically won position
Composing : Allows full checking of all “chess studies”, i.e. problems in “white to play and win” or “white to play and draw” form. Can check consistency of initial position, intended composer’s line of play, and can detect duals.
Computing : Leads to very compact tables, far smaller than DTZ tables. Allows computation of WDL and DTZ “mother” endings.

WDL50 :
Playing : Used alone, helps playing programs in the same way as WDL but can bug in some rare cases (syz’s example). When used with WDL, never bugs, allows detection of winning/losing positions which can be forced to 50-move rule draw by inferior side, which means upgrading 3-state position assessment (won-drawn-lost) to 5-state (won-“hopeful”-drawn-“hazardous”-lost). Practical use of 5-state tables built upon WDL+WDL50 is very simple and similar as standard WDL tables : chess engines will only consider moves that lead to best possible resulting state.
Composing : no added value compared with WDL (same usage, but can bug due to 50-move rule interference which is irrelevant for chess problems)
Computing : very compact (like WDL is). Can be stored together with WDL with very little data size increase, as differential data between WDL and WDL50 is of negligible weight. Allows computation of WDL50 and DTZ50 mother endings.

DTZ :
Playing : shows practical lines to win in vast majority of cases (but not all, because might in some rare cases be defeated by 50-move rule). Shows best practical lines to try to win a position that can be theoretically forced to 50-move rule draw.
Composing : Allows (easier than with WDL) finding winning / drawing lines in chess studies.
Computing : is the most compact currently known metric which delivers “quantitative” information, i.e information required to build winning lines. More compact than DTC, and far more compact than DTM.

DTZ50 :
Playing : supports full practical analysis of every chess position : status, and winning lines. Does this according to 50-move rule, and can do it with taking into account the number of non-zeroing moves that have been played in order to reach a position (which is very useful for “dynamic” position and line of play assessment). Can make use of DTZ in order to detect drawn positions that could be won off 50-move rule (DTZ also gives best practical lines for these positions).
Composing : no added value (compared with DTZ)
Computing : same advantages as DTZ, and can be stored together with DTZ with very little data size increase because differential data between DTZ50 and DTZ is of negligible weight.


Uncovered terrain by WDL-WDL50-DTZ-DTZ50 :

Playing :
- Winning lines won’t be given for endings whose TBs are only in WDL / WDL50 form. This is why at the end it will be best to keep only “clear-cut” endings in WDL form (a rule of thumb is to consider as clear-cut endings whose delta in material between sides is 5 or more, and not clear-cut those whose delta is 1 or 3. Among the 875 endings of 43/43p/52/52p form, 631 are clearcut and 244 are not).
- DTZ / DTZ50 tables will sometimes advise unwise moves such as premature pawn pushes or unnecessary piece sacrifices, provided these don’t throw away the win. This will complicate the task of building, not only winning lines, but “good practical” winning lines (in a human sense).

Composing :
Tables are not suitable for checking problems in “white to play and mate in x” form. They might remove some unlikely big flaws, but are in general unable to check x, thematic variants, thematic tries, and duals. If x is not too big, conventional chess engines can do the job. If x is big, only DTM would be able to check these problems (but manmade miniature problems of this form will be few, if they only exist).

[notnale]

I don't understand what you mean by a DTZ50 sometimes leading to 'unwise' moves

It will never throw away a win or a draw. If you set it to minimize DTZ50, it will also avoid making silly moves, in other words moves that are obviously inefficient.

It is true that DTZ50 isn't sufficient to construct the winning line which minimizes the total number of moves required to mate, as it is conceivable that in some circumstances, zeroing the move counter earlier can lead to taking longer to mate in the resulting position. However, by definition, it is impossible to tell when DTZ50 makes an inefficient move because doing so would require the hypothetical complete tablebase, which could be up to 50 times as big as the DTZ50 tablebase.

In other words, DTZ50 will never blunder, and it will always make moves that look good to a casual observer or computer with similar disk space. Any tablebase that does better then DTZ50 would require an unfeasible amount of storage space.

[syzygy]

WDL or WDL50 tables can’t prevent KK’s scenario (engine starting from a won position but going in circles and finally conceding 50-move rule draw) from happening. This is due to lack of “winning line” info. This is the reason why DTZ or DTZ50 info is also needed for “not clear-cut” endings.

Yes. And I would like to add that without DTZ/DTZ50 tables available, there is probably no practical difference between a dtz=40 and a dtz=60 position. I.e. engines will not have better chances of winning a position with dtz=40 than of winning a position with dtz=60. But WDL50 makes a difference between these positions.

On the other hand, Syz’s pitfall (engine defending a genuine draw but converting it blindly into a “blessed loss” then finally losing it) can easily be avoided : it needs only to have WDL table (not WDL50) available, (...)

Good point, to be honest I did not even think of genuine draws converting into blessed losses. So WDL50 is very dangerous to use without a corresponding DTZ50-table. More dangerous than I thought.

And I also don't think that to lose a drawn position is worse than to draw a won position. In both cases you lose a well deserved 0.5 of a point.

I don't think you should look at the half point loss, but at what happens on the board. In one case the winning side struggles for 50 moves and fails to actually win a position that is still winnable (except for the 50-move rule). In the other case the side that thought and had announced it secured a draw ends up getting mated. The first case most people will understand; it's inevitable that some positions in a WDL-table can in fact not be won in under 50 moves if the opponent plays optimally. The second case seems much harder to explain.

I'm not against DTZ50-tables, btw. (My preference goes to DTZ50+ or at least DTZ50 with "cursed positions" marked as such.) I'm convinced now however that even the DTZ50-tables without any "extra" information will be larger in size than DTZ-tables, because accurate 50-move rule play places higher demands on the accuracy of the tablebase data. But I think it's worth it, since I suspect pure DTZ could easily mess up winning positions, especially in tables with pawns.

I don't understand what you mean by a DTZ50 sometimes leading to 'unwise' moves

It will never throw away a win or a draw. If you set it to minimize DTZ50, it will also avoid making silly moves, in other words moves that are obviously inefficient.

I agree it will never blunder, but it will make moves that no human would play. E.g. queen sacs just to convert to another won position as fast as possible. Also a win-preserving pawn push will get preference over a mate in 2. A winning side with a lot of pawns might promote all its pawns before mating the opponent king, which I guess would be regarded as an insult to the opponent when done by a human player. Some plies of searching would probably solve many of these problems.

I think that if a tablebase position is won but not in a trivial way, then a DTZ50-move will probably almost always look "good".

[guyhaw]

I like kronsteen's idea that the motivation for requirements should be clear. I also recommend that students use:
a) MoSCoW ranking for requirements - 'Must', 'Should', 'Could' and 'Would be Nice if' - which can translate into a 'view forward' with priorities, and
b) define a non-requirement zone of potential requirements which are not currently seen as 'in scope'.

While we have had interesting discussions about (DTR, DTZR), it should now be clear that these are non-trivial concepts and currently out of scope for this project. My own top requirements are for:

- assured and easily-verifiable correctness (whatever is produced) ... the EGTs effectively 'define' n-man endgame chess and can easily be corrupted,
- code-proving by creating 3- to 6-man EGTs which check out against the Nalimov EGTs,
- completeness in that e.p.-positions (but not necessarily positions with castling rights) are included
- 'per-phase modularity': it should be possible to 'call up' the EGT for, e.g., KQP(a5)KQ without getting the rest of the KQPKQ EGT
- proven WDL before WDL50 is derived from it, WDL before the DTZ EGT 'EZ' as it helps generate that EGT, and 'EZ'/WDL50 before the DTZ50 EGT 'EZ50'.

g

[notnale]

I agree it will never blunder, but it will make moves that no human would play. E.g. queen sacs just to convert to another won position as fast as possible.

I'm sure that if you gave a human a choice between say KQRRKQR or KRK, they would pick the latter every time. Most humans would trade or sacrifice in a heartbeat if it led to a known winning position. The difference is that the computer has a lot more 'known' won endgames.

Also a win-preserving pawn push will get preference over a mate in 2. A winning side with a lot of pawns might promote all its pawns before mating the opponent king, which I guess would be regarded as an insult to the opponent when done by a human player.

This actually is a valid problem that I hadn't thought of. Of course, most humans would have resigned long before then. (Remember no chance of the computer accidentally stalemating a human with its gazillion queens like an overeager novice might. If the chance of you drawing is comparable to the chance of the computer suddenly getting struck by lightning, then it's time to call it quits)

[kronsteen]

In other words, DTZ50 will never blunder, and it will always make moves that look good to a casual observer or computer with similar disk space. Any tablebase that does better then DTZ50 would require an unfeasible amount of storage space.

Take this example : Kc4, Rd1, Pd4 / Kd7, Bc7

d5 is the worst possible move : by blocking a vital manoeuvring square for the king, it forces superior side to aim at a difficult zugzwang to break down the defence. After d5, DTM is 36, when it is at most 24 after any other move. But d5 is the choice of any DTZ or DTZ50 table. And the defect of d5 may not be so obvious to a beginner. We can easily imagine 7-men positions where DTZ similarly advises a slack move whose defect might not be obvious even at high human level. This is the reason why DTZ is less effective than DTM to build practical rules and practical lines of play or humans. But as you correctly said, doing better would need TBs of unaffordable size for 7-men.

[Kirill Kryukov]

It comes down to this : we might not only write requirements, but also justifications of these requirements. Justifications help everybody to understand why current requirements, and not other ones, have been chosen, and will make this information traceable and not forgotten in the future. Justification of currently chosen metrics might be given in 3 domains : added value for practical play, added value for problem composing, and computing issues. Here is what I think of it (this is to be criticized / corrected / completed, of course) :

Agreed, and thanks for posting! I will copy the justification into the first post. I'll think about them and comment later when I have time.

[Kirill Kryukov]

And I also don't think that to lose a drawn position is worse than to draw a won position. In both cases you lose a well deserved 0.5 of a point.

I don't think you should look at the half point loss, but at what happens on the board. In one case the winning side struggles for 50 moves and fails to actually win a position that is still winnable (except for the 50-move rule). In the other case the side that thought and had announced it secured a draw ends up getting mated. The first case most people will understand; it's inevitable that some positions in a WDL-table can in fact not be won in under 50 moves if the opponent plays optimally. The second case seems much harder to explain.

I don't think there is a big difference between these two cases. The tablebase sais the position is a win, and you fail to win it, then the tablebase can be considered broken just the same. The reason why we are forgiving when a tablebase is broken in this way is because we are used to Nalimov tables that don't consider the 50-move rule.

I'm not against DTZ50-tables, btw. (My preference goes to DTZ50+ or at least DTZ50 with "cursed positions" marked as such.) I'm convinced now however that even the DTZ50-tables without any "extra" information will be larger in size than DTZ-tables, because accurate 50-move rule play places higher demands on the accuracy of the tablebase data. But I think it's worth it, since I suspect pure DTZ could easily mess up winning positions, especially in tables with pawns.

Yes it's possible that DTZ50 may be larger than DTZ, and WDL50 may be larger than WDL.

I don't understand what you mean by a DTZ50 sometimes leading to 'unwise' moves

It will never throw away a win or a draw. If you set it to minimize DTZ50, it will also avoid making silly moves, in other words moves that are obviously inefficient.

I agree it will never blunder, but it will make moves that no human would play. E.g. queen sacs just to convert to another won position as fast as possible. Also a win-preserving pawn push will get preference over a mate in 2. A winning side with a lot of pawns might promote all its pawns before mating the opponent king, which I guess would be regarded as an insult to the opponent when done by a human player. Some plies of searching would probably solve many of these problems.

I think that if a tablebase position is won but not in a trivial way, then a DTZ50-move will probably almost always look "good".

Same like with the current Nalimov format tablebases, an engine will be free to perform any additional search. When you are in the table, you don't have the time pressure anyway, so you can afford to search a little deeper to pick up the possible mates in 2 or 3, or to find a more optimal path. An engine does not have obligation to only search 1 ply. A tablebase provides priceless information, but what to do with this information is still up to the engine author.

I agree that seeing all pawns promoted by following DTZ50 will be funny. However, an engine author could easily add some code like "if we are two queens up, stop pushing the pawns and look for a mate". Or something more clever. The alternative t oDTZ50 is DTZ which does not guarantee you to win a won ending, and it is I think clearly worse of the two possibilities. Of course the "composers" want DTZ anyway, that's why it is in the requirement.

[Kirill Kryukov]

I like kronsteen's idea that the motivation for requirements should be clear. I also recommend that students use:
a) MoSCoW ranking for requirements - 'Must', 'Should', 'Could' and 'Would be Nice if' - which can translate into a 'view forward' with priorities, and

Did you check the current requirements in the first post of this thread? There are two parts: Required features, and optional features. Required features must be provided by every submitted solution. Optional features can be used to pick the best out of several solutions.

I think these two feature sets correspond to 'Must', 'Would be Nice if' in your terminology. Can you suggest in which way adding the 'Should', 'Could' can help in this project?

b) define a non-requirement zone of potential requirements which are not currently seen as 'in scope'.

What do you mean by "potential requirements"? Something that may become a requirement at some point in future? At which point and by who's decision? After we start receiving the donations, it won't be right to keep changing the requirements.

While we have had interesting discussions about (DTR, DTZR), it should now be clear that these are non-trivial concepts and currently out of scope for this project. My own top requirements are for:

- assured and easily-verifiable correctness (whatever is produced) ... the EGTs effectively 'define' n-man endgame chess and can easily be corrupted,

Suppose you receive a solution for evaluation, how can you determine whether it has "assured and easily-verifiable correctness" or not?

- code-proving by creating 3- to 6-man EGTs which check out against the Nalimov EGTs,

Ability to produce 3-to-6-men EGTB is in the requirement. I don't think checking against Nalimov's tables is a good idea for requirement, because: 1. It depends on a non-free code. 2. Only WDL and DTZ can be checked. (Not WDL50 and DTZ50). However anyone will be free to do such cross-check as he wishes, once the generator is released to public.

- completeness in that e.p.-positions (but not necessarily positions with castling rights) are included

EP-correctness is in the requirements. Castling is among the optional features. It would help if you read the requirements before suggesting improvements.

- 'per-phase modularity': it should be possible to 'call up' the EGT for, e.g., KQP(a5)KQ without getting the rest of the KQPKQ EGT

The access speed and compactness are in the optional features. The exact way how they are achieved is up to the developer to experiment. Also what exactly do you mean by "call up" ? Storing each pawn-splice in a separate file?

- proven WDL before WDL50 is derived from it, WDL before the DTZ EGT 'EZ' as it helps generate that EGT, and 'EZ'/WDL50 before the DTZ50 EGT 'EZ50'.

WDL50 is not derived from WDL. Basically you are talking about the generation order, which is up to the community after the generator is released. The ability to use WDL (WDL50) when generating DTZ (DTZ50) is in the requirement. Also, what do you mean by "proven"?

[notnale]

I agree that seeing all pawns promoted by following DTZ50 will be funny. However, an engine author could easily add some code like "if we are two queens up, stop pushing the pawns and look for a mate". Or something more clever. The alternative t oDTZ50 is DTZ which does not guarantee you to win a won ending, and it is I think clearly worse of the two possibilities. Of course the "composers" want DTZ anyway, that's why it is in the requirement.

It would be even funnier if the author of the chess engine set it to evaluate promotions in order of increasing material instead of decreasing. (e.g. Knight first, queen last)
Then, in a KPPPk position, the computer would promote all three pawns to knights!

[guyhaw]

There was a match between FRITZ and JUNIOR some years ago: I think it was set up as an 'eliminator' to decide who played (?) Kramnik.

In one game, a Rook and Pawns endgame was reached with I think JUNIOR being well on top. JUNIOR sacrificed a Rook in order to convert into a 5-man endgame in the 'EGT Zone'. Does anyone have the details?
g

[syzygy]

I'm sure that if you gave a human a choice between say KQRRKQR or KRK, they would pick the latter every time. Most humans would trade or sacrifice in a heartbeat if it led to a known winning position. The difference is that the computer has a lot more 'known' won endgames.

Yes, and a computer as white in KQBNK will typically force black to takes its queen as it prefers KBNK (in DTZ/DTZ50). This is certainly not a human move . Combined with the preference for pawn pushes you might see the computer first getting four queens, then sacrificing three of them before common sense returns.

Of course, most humans would have resigned long before then. (Remember no chance of the computer accidentally stalemating a human with its gazillion queens like an overeager novice might. If the chance of you drawing is comparable to the chance of the computer suddenly getting struck by lightning, then it's time to call it quits)

On the other hand, it might be wise for the human to test whether the engine is programmed properly and whether the tables were properly verified. Of course he shouldn't then complain about silly play by the computer.

And I also don't think that to lose a drawn position is worse than to draw a won position. In both cases you lose a well deserved 0.5 of a point.

I don't think you should look at the half point loss, but at what happens on the board. In one case the winning side struggles for 50 moves and fails to actually win a position that is still winnable (except for the 50-move rule). In the other case the side that thought and had announced it secured a draw ends up getting mated. The first case most people will understand; it's inevitable that some positions in a WDL-table can in fact not be won in under 50 moves if the opponent plays optimally. The second case seems much harder to explain.

I don't think there is a big difference between these two cases. The tablebase sais the position is a win, and you fail to win it, then the tablebase can be considered broken just the same. The reason why we are forgiving when a tablebase is broken in this way is because we are used to Nalimov tables that don't consider the 50-move rule.

I am curious what others think about this. I personally find the difference between
1. failing to win a won position because of the 50-move rule (as an obvious consequence of choosing the much smaller WDL over DTZ), and
2. getting mated after reaching a position on the board that is known to be a draw
very significant, especially for an engine that is using terabytes of "perfect knowledge".

I agree that seeing all pawns promoted by following DTZ50 will be funny. However, an engine author could easily add some code like "if we are two queens up, stop pushing the pawns and look for a mate".

Yes. Btw, my examples of DTZ/DTZ50 playing "unwise moves" were not meant as an argument against DTZ or DTZ50, but to illustrate kronsteen's point. Clearly it's something for the engine to solve (if it is a problem at all) and not for the tablebase generator.

[notnale]

Although with 4 queens, theres a good chance of having mate in one anyway.
Mate in two is practically guaranteed.

One partial solution might be to include a complete tablebase (DTM for every possible position and move count) should be included for 4, maybe 5 piece positions, since this shouldn't take much space anyway. One it's sacrificed a piece or two, it will stop playing idiotically and get down to buisness.
This could be combined with an automatic mate in 3 search for even better results

[kronsteen]

I am curious what others think about this. I personally find the difference between
1. failing to win a won position because of the 50-move rule (as an obvious consequence of choosing the much smaller WDL over DTZ), and
2. getting mated after reaching a position on the board that is known to be a draw
very significant, especially for an engine that is using terabytes of "perfect knowledge".

Chessically speaking, the loss is the same : 0.5 point, and the only relevant thing is how much you lose and how likely it is, and not how you lose it. Technically speaking, there is a big difference between the first case, which is a technical limitation (when constrained to use WDL and not DTZ due to space storage limitations, you simply have no way to access perfect 7-men play) and the second one, which is wrong use of knowledge (WDL50 when in fact WDL would be needed), and is not acceptable in general.

Feeling or not difference between both cases inherently depend on what point of view you’re from : chessical, or technical

[syzygy]

Technically speaking, there is a big difference between the first case, which is a technical limitation (when constrained to use WDL and not DTZ due to space storage limitations, you simply have no way to access perfect 7-men play) and the second one, which is wrong use of knowledge (WDL50 when in fact WDL would be needed), and is not acceptable in general.

Indeed, it is the wrong use of knowledge if a WDL50 draw value is interpreted as a draw (and DTZ50 is not available). The correct way to use a WDL50-table in that case is to ignore all draw values.

So with WDL an engine probing a position can cut the rest of the branch and return a win, draw or loss value.

With WDL50 an engine probing a position can cut the rest of the branch if the the result is a win or a loss. If the position is a draw, the engine must ignore it and act as if the position is not in the table (i.e. search deeper or apply the evaluation function).

It is true that if the position is a win (WDL or WDL50), the engine might not be able to find the winning line once the position is on the board, but if winning values are ignored as well then all benefit of WDL/WDL50 is gone. Using WDL/WDL50-only means you accept the rare 50-move rule draw.

[Kirill Kryukov]

WDL might have some advantages over WDL50, like rarely saved 0.5 point, or may be compactness. However one important advantage of WDL50 is that it allows to construct a DTZ50. So if I want kpppkpp in DTZ50 I have to generate all the necessary WDL50 tables first. This is the reason why I am going to contribute my CPU time towards WDL50 generation when the generator will be available.

[Kirill Kryukov]

I modified the requirements by moving WDL(k) and DTZ(k) into optional features.

[guyhaw]

I think there was some discussion of metrics driving a game in a chessicly bizarre way. This often happens.
A continuation of Carlsen's win today (Bilbao, Grand Prix Final, Rnd.1) is 8/8/4p2p/4P2P/8/6k1/4Kp2/8 b - - 0 60.
Here, with the DTM EGT for KPPKPP to hand, the top suggestion is 60...f1Q+ ! Well, it is mate in 22.
Enough said - g

[syzygy]

However one important advantage of WDL50 is that it allows to construct a DTZ50. So if I want kpppkpp in DTZ50 I have to generate all the necessary WDL50 tables first.

True, but having kpppkpp in DTZ50 is useless if you don't have the subtables in DTZ50. And if you have the subtables in DTZ50, it is much faster to generate WDL50 from those (compared to generating WDL50 and DTZ50 by separate programs).

Two hypothetical scenarios:
scenario 1:
2009: generation of WDL
2015: generation of DTZ50, derivation of WDL50 from DTZ50

scenario 2:
2009: generation of WDL50
2015: generation of DTZ50

In scenario 1, usable (and easy-to-use and explain) tables are available over the period 2009-2015.

In scenario 2, over the period 2009-2015 tables are available that are quite restricted in usability. Engine support has to be carefully implemented to avoid interpreting a draw value as a draw. It is hard to explain to people why a draw value is not really a draw (in fact, I am much in doubt that I have succeeded to explain this in this thread).

One small advantage of scenario 2 is that in 2015, WDL50 does not have to be derived from DTZ50. This saves a bit of computation time. However, deriving WDL50 from DTZ50 is probably at least 100x as fast as generating WDL50 from scratch on equal equipment. We may hope that computers in 2015 are at least 10x as fast as computers in 2009. So the "extra time" scenario 1 requires in 2015 is less than 0.1% of the time required to generate WDL in 2009. That's negligible if you ask me.

(Plus there's the problem of checking WDL50 for internal consistency.)

[notnale]

However one important advantage of WDL50 is that it allows to construct a DTZ50. So if I want kpppkpp in DTZ50 I have to generate all the necessary WDL50 tables first. This is the reason why I am going to contribute my CPU time towards WDL50 generation when the generator will be available.

Are you sure you didn't get this backwards?
You can easily generate WDL50 out of DTZ50, but I'm not sure how you intend to accomplish the reverse

[syzygy]

However one important advantage of WDL50 is that it allows to construct a DTZ50. So if I want kpppkpp in DTZ50 I have to generate all the necessary WDL50 tables first. This is the reason why I am going to contribute my CPU time towards WDL50 generation when the generator will be available.

Are you sure you didn't get this backwards?
You can easily generate WDL50 out of DTZ50, but I'm not sure how you intend to accomplish the reverse

I think the idea is that to generate kpppkpp in DTZ50, you only need all subtables in WDL50.

But I'm not sure how that helps much, as you'll need the subtables in DTZ50 if you want to use kpppkpp in DTZ50.

[kronsteen]

But I'm not sure how that helps much, as you'll need the subtables in DTZ50 if you want to use kpppkpp in DTZ50.

Wrong. DTZ(50) needs only WDL(50) for daughter endings, not DTZ(50). This is the main strength of the “dual metric” approach : as soon as you have 7-men full set in WDL(50) format, you can compute DTZ(50) table of any 7-men ending you want (even kpppkpp first if it is your wish), regardless of other DTZ(50) tables that already exist. Even if a future generator is designed to build DTZ(50) and WDL(50) tables at the same time (WDL(50) being a “sub product” of DTZ(50)), keeping only WDL(50) first due to space storage limitations (and waiting for hardware upgrade for DTZ) might be a sensible approach, as DTZ(50) tables are something like 5 times bigger, and their size can be hundreds of GBs.

The fact is DTZ(50) info will not necessarily be sustained everywhere from initial position to mate, but only in “interesting” phases. Taking kpppkpp as an example, if you succeed in converting to kqppkpp, maybe this ending will be kept in WDL(50) format as WDL info added to your playing ability will be enough to win this in general. But should this ending convert to kqppkqp, you may want DTZ(50) info back as this is a difficult ending. This is perfectly possible.

Should you want uninterrupted DTZ support from kpppkpp, it would require storing DTZ on every 43+43p endings, even endings such like kqqqkbn which are possible (even if unlikely) outcomes from kpppkpp. It doesn’t make much sense to store knowledge about if 1, 2, 3, 4, 5 or 6 plies are needed in kqqqkbn positions in order to either mate, capture a black piece, or blunder away a queen in such a way that black cannot decline the offer (which is precisely DTZ knowledge). According to my estimates, this rather useless knowledge would make final set grow up from 30 TB to 60 TB. Quite a difference.