DTR is no good!

Endgame analysis using tablebases, EGTB generation, exchange, sharing, discussions, etc..
Post Reply
h.g.muller
Posts: 223
Joined: Mon Feb 19, 2007 8:24 am
Sign-up code: 0
Location: Amsterdam
Contact:

DTR is no good!

Post by h.g.muller »

I changed my opinion on the DTR metric (as others apparently did before me, without me properly understanding the problem they were pointing out): for tablebases that tore DTx as a function of position, rather than full game state (which includes also things like reversible-move counter), it is a useless metric, in the same sense that DTM is a useless metric: playing by it makes you bungle won games. For DTM this is obvious, as it might set you on a path where one of the phases exceeds the 50-move rule (or whatever move rule is in force). For DTR (actualy {DTR, DTZR}) it is more subtle, and only occurs if the opponent does a "suboptimal" move:

Suppose that the current phase has a much larger DTx than the subsequent phases. This is a very common situation in pawn-les end-games, where you in general have to force the win by first gaining one of the opponent's pieces, after which his defense crumbles. A good example is KBBKN: it can take about 80 moves to gain the Knight, while KBBK takes at most 19 moves to win. So suppose that, you convert to KBBKN at a point where it takes you 49 moves to gain the Knight, and after that 12 moves to mate in KBBK. Under a 50-move rule this would be won, and the DTR would be 49.

As you play the DTR drops, and at some point it will hover at 12, while you continue playing by DTZR to make progress towards gaining the Knight. So suppose you arrive at the point that in KBBKN DTZR=5, but DTR is still 12, because it is dominated by the KBBK phase. Now suppose the opponent makes an "error", which allows you to reach a position that has DTR = DTZR = 7, which needs both 7 moves to gain the Knight, and 7 moves after that to mate in KBBK. So your DTM reduces from 17 to 14, and your DTR will drop from 12 to 7. The DTR metric would think this is a good deal.

Unfortunately, your reversible-move counter was already at 44. By the time you complete the current phase, it is at 51, and the game has ended in a draw. By needlessly reducing the duration of the next phase to make it winnable under a hypothetical 7-move rule rather than a hypothetical 10-move rule, while you knew a 50-move rule applied, you managed to draw out the current phase beyond 50 moves!

DTR would only work as a metric if you would build a separate EGT for each value of the reversible move counter, for in taking the maximum of the duration in all phases, you should add the current value of the 50-move counter to the number of moves you still need in the current phase, before taking that maximum.
syzygy
Posts: 166
Joined: Sun Aug 03, 2008 4:02 pm

Re: DTR is no good!

Post by syzygy »

Yes, this is one of the scenarios where DTR goes wrong.

Another scenario is when both sides are playing according to the DTR-table and the table leads to the winning side playing a move with lower DTR than the winning side can actually achieve given the current move-counter. The problem is (again) that the DTR-values in the table always assume move-counter = 0.

As kronsteen pointed out, what you need is a way to get a value that has meaning for the current value of move-counter. With DTZ50 this works fine: if move-counter + dtz50 <= 50, then dtz50 is the distance to the next zeroing move. If move-counter + dtz50 > 50, then the current position (given the value of move-counter) is a draw.

Btw, I wouldn't call DTM useless. If you choose DTM, you choose to ignore the 50-move rule. If you ignore the 50-move rule, then DTM is fine. Same for DTZ (not to be confused with DTZ50 which does take into account the 50-move rule).

Maybe you were thinking of something like "DTM50": DTM taking into account the 50-move rule. For the same reasons why DTR does not work, DTM50 does not work (you'd need separate tables for each value of the move-counter).

It might be interesting to look at this issue from a slightly more theoretical viewpoint. If you ignore the 50-move rule, the game state in chess is determined by the board position and side-to-move (ok, there's also castling and en passant and draw-by-repetition, but let's ignore those for now). A tablebase therefore needs to store values indexed by (board position, side-to-move). This is how we always do it, and it works fine for e.g. DTM.

But now let's take into account the 50-move rule. The game state becomes (board position, side-to-move, move-counter). A tablebase now needs to store values indexed by (board position, side-to-move, move-counter). Without any compression they become 50x as large (maybe even 100x).

Now the very unique thing about the DTZ50-metric is that you can derive the value for (board position, side-to-move, x) from the value for (board position, side-to-move, 0). Thanks to this small miracle, it is possible to build a table that correctly takes into account the 50-move rule without having to store 50x or 100x as many positions as the tables we are used to.
h.g.muller
Posts: 223
Joined: Mon Feb 19, 2007 8:24 am
Sign-up code: 0
Location: Amsterdam
Contact:

Re: DTR is no good!

Post by h.g.muller »

Note that DTM and DTR are actually the two extreme cases of a DnTM metric, where n is a numbr that describes how to combine the time spent in each phase to a total time. This sequence of the number of moves spent in each phase could be seen as a vector, and the total winning distance as the norm of this vector. A very common vector norm is the L_n norm, which is defined as |v| = (SUM v_k^n ) ^( 1/n), i.e. the power-n root of the sum of n'th powers of the components. For n=1 this becomes the simple sum of the components, which is why D1TM = DTM. In the limit of n -> infinity, only the largest component contributes, so the expression amounts to taking a maximum. Thus DinfTM = DTR.

The basic flaw in all these norms is that they weight every phase identically. I would prefer a metric that puts more emphasis on the later phases, as a computer opponent is more likely to be fallible in early phases than in late phases. In particular, the more men on the board, the less likely he is to have the correspondng EGT.

So basicllay, if some phases will have to last longer than 50 moves to secure a win against best defense, and this can be achieved in a number of ways, I would consider the one that exceeds 50-moves in a phase with fewer men (say N men) with only one move to have a shorter "distance to 50-move rule" than a solution that needs 60 moves in an earlier phase with more men but stays below 50 in any N-men phase.

So any move that would reduces the duration of a future phase that was above 50 would get precedence over a move that would lower DTZ in the current phase. This seems to invite the sme mistake as I described above for DTR, except that it would only be don for a good reason, namely to avoid a certain draw in a simple phase for which the opponent almost certinly has the tablebase.

This would make it less painful to ignore the move counter in the current phase, as even when you knew it, you would consciously take the decision to possibly overrun it, rather than take that risk later, and hope for fallibility of the opponent now. This seems a bad gamble if you are already very close to 50 moves, especially if the move counter is already at 49, and you have the choice now to convert to a positon that takes 51 moves, or not convert and face an immediate draw claim. But in practice that will hardly ever occur, as the 50-move counter can only run high if you are already in the current phase for a long time. So if the opponent was indeed fallible, you should already have caught up with the 50-move deficit long before you get in this situation. If he is not fallible, than you are battling a lost cause anyway.

So I would build a tablebase by first seeding with all true wins (< 50 moves in any later phase), and build normally from there. Say this fills DTx = 0-80. Then I would seed the TB from the positions with the smallest 50-move deficit in the immediate successors (which has DTx = 51 there). Positions that convert to that would get DTx=81 in the current TB, and I would build normally from there, say 81-125. Then I would take the DTx=52 from the successors (next-best deficit), and give positions that convert to there DTx=126. And so on, recursively.
notnale
Posts: 43
Joined: Sun Aug 17, 2008 6:36 am

Re: DTR is no good!

Post by notnale »

It's not that DTZ50 doesn't store all 100 values per state, it is just heavily compressed
Just think of it as using 100 bits and then applying run length compression
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

DTR is some good ...

Post by guyhaw »

As szyzygy correctly pointed out (for which I am most grateful), the DTR == {(DTR, DTZR)} EGT cannot be used as easily as I anticipated, even though DTR and DTZR are well-defined and can be generated for all positions.

The difficulty is that the EGT may offer a route to a sharper DTR-goal which is, nevertheless, one whose achievement can be made to take more moves than are available in the current phase under whatever k-move rule is in place. [I am just saying, in maybe a different way, what szyzygy says above.]

For example, suppose you jump into a new phase of the game with 50 moves to go, defending (as the attacker) DTR = 20 and DTZR = 20. Say every 5 moves you get offered a DTR one less, but DTZR = DTR again. After 35 moves (so, with 15 moves left) you are offered DTR=13 and DTZR=13. All is still ok: this is an objective you can accept. But 5 moves later, when you are trying to follow the '(13, 13)-path' to the end, 'the map' offers you a (12, 12) path with only 10 moves to go. This is not 'safe' to accept as you will have to abandon the DTR-minimaxing path for the DTZ-minimaxing path and then there is no knowing where DTR will finish up.

However, I have some fairly good news: all is not lost. The winnable wins may not all be secured by merely looking up the DTR EGT, but I still think the DTR EGT is the best aid. The extra-cost now discovered as necessary is that a degree of forward searching on the DTR EGT may be required.

a) in position P1(dr, dzr), if there is a move to P2(dr-1, dzr-1) or - not quite as good P3(dr, dzr-1) then the forward path is still 'in the DTR EGT', but
b) if not, clearly there is no point moving to drawing/losing positions, to P2(dr+, ~), to P3(dr, dzr+) or to P4(dr, dzr)
... this will narrow the width of the search-tree, hopefully quite a lot
c) if the search-tree features a position Pn(dr-, dzr-) which again is a feasible goal, given 'moves left', that search-tree node need not be expanded

That's the forward-search idea, one that h.g.m. has been championing elsewhere incidentally :-).

The backward-search idea is that, given that we are are position P1(dr, dzr), one takes the subgame positions with DTR = dr and retros back into the current endgame, not needing to retro further back than dzr moves (or 2*dzr plies). This is like generating the DTZ[dr] EGT at runtime except that we only need to know about the 'dr' bit of the DTZ[dr] EGT, and only to a certain depth. The runtime generation is assisted by having the relevant WDL EGT, and the DTR information for DTR < dr, and DTZR > dzr.

So, yes, I was trying to do the impossible - metaphorically representing many sheets of pasta in one sheet of pasta - but given that we only have one layer of pasta, it is still very usable.

g
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

What is an optimal move for the defender ...

Post by guyhaw »

hgm also described a scenario in which the defender dangled an unachievable DTR-objective in front of the attacker. As sz points out, it is not even necessary for the defender to do this.

I didn't notice hgm's scenario first time round as hgm talked about the defender making a 'sub-optimal move'. Well, it was in DTR terms, but it was not in DTZR terms (in which terms lay the trap for the attacker).

However, the flaw - had the attacker accepted this overambitious DTR-target - would not have been in the DTR EGT but in the use made of it by the attacker. It is too easy to assume that the win is easy to achieve (like picking low-hanging fruit) when there is an EGT around.


Incidentally, there are some shortcuts to generating DTR EGTs: take the KRBNKQN endgame for example, in which MB found a DTC/DTZ = 517m win for Black. If 'dz' > maxDTZ in any subgame, we know that 'dr' = 'dzr' = 'dz'. maxDTZ for 'Q-side' wins in subgames is 17 (KBNKQN), 40 (KRNKQN), 26 (KRBKQN), 29 (KRBNKQ) and 01 (KRBNKN). So we know that dz > 40 for a 0-1 win in KRBNKQN ==> dr = dzr = dz. So for dr and dzr in [41, 517], we have dr and dzr already - which saves a lot of retrograde steps in the generation of the DTR EGT. It also means that the 'DTR-path' is not going to get hidden until dr < 41 and dzr < 40.

g
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

DTR-related strategies in multiphase play

Post by guyhaw »

hgm's association of DTM with the L_1 norm and of DTR with the L_infinity norm is insightful - but unfortunately, I don't think it's particularly helpful, although it would be very elegant if it were.

More simply, we can say that DTM and DTR are 'about' the whole winning line, not just the current phase. We can also say that DTM assumes the minimaxing of the total number of moves, whereas DTR assumes minimaxing of the length of the longest phase which is relevant in the context of a k-move rule. My expectation is that in P-ful endgames, DTR helps the attacker avoid getting into overdeep water. For example, the ground can be prepared in KQP(a5)KQ before getting into the harder territory of KQP(a6)KQ and KQP(a7)KQ, and spare moves in a phase can be used beneficially against a fallible opponent.

It would be really interesting to generate the DTR EGTs to KQPKQ to find out which are the most difficult positions from which to finesse wins. Determined use of the DTZk EGT generator, and MB has one such, would do the job quickly enough.

Beyond minimaxing DTR, I find it difficult to argue about how best to lessen the length of one phase at the cost of greater length in another: everyone has their ceiling, and that's above mine at the moment. Maybe it's crossing a bridge too early: there is no experience of using DTR EGTs yet, and that's because they haven't been generated yet.

My assumption so far has been that the attacker should always 'guard' both Theoretical Value V and DTZ to avoid a draw-claim in the current phase, ensuring that DTZ is no greater than 'moves available' if possible and minimising DTZ if not. So, all move-filtering strategies should start SV+Zo (in my notation).

We also have the idea, see above, that the attacker should defend current DTR and DTZR (i.e. not concede in either dimension): strategies incorporating this idea start SV+ZoRdZRd ('d' standing for 'defend'). As Jansen suggested, a metric can be the subject of speculation, rather than simply 'defended'. However, as he did not demonstrate, there needs to be an informed model of the opponent's fallibility, and there is limited opportunity to gain information about the opponent and use it to speculate with.

What best happens after 'ZRd' depends on 'dr' and on 'dzr-versus-moves_available'. If dr =< k, there is no need to adopt sharper dr-goals: minimising drz might be more sensible. On the other hand, sticking with a route 'explicitly documented' in the DTR EGT rather than trying to find a hidden one is sensible, and this could involve aiming for a lower 'dr' at the cost of greater 'drz'. Potentially there could be an already-narrowed tree-search, perhaps meeting a focussed DTR EGT generation at runtime. Note that RYBKA is currently winning the ICGA's WCCC_20008 on a 40-node cluster (!?) of computers, so there's plenty of runtime power around.

Observations of 'endgame capability' so far have shown that the more men in the endgame, the more likely are the sides to concede depth if not armed with the right EGTs. For example, KBBKN-play typically sheds depth more readily than KQKR-play. Again this points to minimising DTR in the current phase, within the constraint of finishing the current phase 'in time'.

g
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

Small omission corrected ...

Post by guyhaw »

The 'ZRd' part of the strategies discussed above only applies when we are considering a move to a position with the same DTR as the current position. If the DTR of the considered position is higher, we should not be considering the position anyway. If the DTR of the considered position is lower, that path may be hiding the path we are looking for which defends the current DTR.

There are many cases where the attacker can reverse the effect of their last move while retaining the win - increasing such metrics as DTC, DTM, DTZ (and DTZR) by one rather than decreasing them. This is the sort of thing that needs to be prevented.

g
h.g.muller
Posts: 223
Joined: Mon Feb 19, 2007 8:24 am
Sign-up code: 0
Location: Amsterdam
Contact:

Re: DTR is no good!

Post by h.g.muller »

Only DTR needs the tie breaker in the form of DTZR. This because in the limit n -> infinity the DnTM norm (= (SUM pase_i^n) ^ (1/n) ) the (sup-)norm of the DTM vector gets completely independent of the duration of any shoter phase. So DTR = DinfTM can no longer be used to order positions with equal DTR, while for all finite values of N, the order is well defined, as the shorter phases always contribute something to the norm. So the limit of the ordering is well defined, (and indicated by DTRZ), but the ordering of the limit case (DTR) was not.
notnale
Posts: 43
Joined: Sun Aug 17, 2008 6:36 am

Re: DTR is no good!

Post by notnale »

I still don't see what the point of using anything other then DTZ50 is

DTZ50 is sufficient for perfect play, and it is also the smallest possible tablebase that has this property
syzygy
Posts: 166
Joined: Sun Aug 03, 2008 4:02 pm

Re: DTR is no good!

Post by syzygy »

notnale wrote:I still don't see what the point of using anything other then DTZ50 is

DTZ50 is sufficient for perfect play, and it is also the smallest possible tablebase that has this property
I fully agree that DTZ50 is all you need for perfect play.

When playing an imperfect opponent some extra information might help to win positions that are drawn under the 50-move rule. The DTZ50+-tables that were discussed a while ago are a good candidate. Such a table corresponds exactly to a DTZ50-table for positions that are won or lost under the 50-move rule and for positions that are drawn even without a 50-move rule. The other positions get a value that means "drawn, but in x moves some progress can be made". There are a couple of ways to define these values. In my view, the simpler the definition the better.
h.g.muller
Posts: 223
Joined: Mon Feb 19, 2007 8:24 am
Sign-up code: 0
Location: Amsterdam
Contact:

Re: DTR is no good!

Post by h.g.muller »

I think the metric I describe in the third post of this thread is the most useful choice for pawnles end-games.
notnale
Posts: 43
Joined: Sun Aug 17, 2008 6:36 am

Re: DTR is no good!

Post by notnale »

The problem is that there is no solid answer, because fallible opponents are fallible in different ways
Good luck teaching the computer to read minds

In fact, even with a single person, there is still huge variation. Some traps you'll see, some you won't

There are various heuristics you can make up, but there are no 'right' or 'wrong' ones. You are basically guessing which moves are most likely to make the human mess up. But ultimately, it all comes down to the human, not the tablebase.
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

Models of fallible players ...

Post by guyhaw »

Notnale has hit the - nail - squarely on the head. Actual chess players show fallibility in many different ways.

While the EZ50 EGT to the DTZ50 metric is necessary and sufficient to win from a position that should be won under the 50-move rule, it is specific to a 50-move rule, one which I don't regard as set in stone. Finessing a win that should result in a 50-move draw-claim [like JUNIOR-RYBKA threatened to do yesterday in the ICGA WCCC 2008] requires some insight into the opponent's fallibility, as notnale suggests.

Jansen, some years ago, suggested playing on the opponent's fallibility but did not suggest any models of fallibility. Around 2002-3, I suggested associating actual endgame play with 'agents' in a 'Reference Fallible Endgame Player' space - and then using what little information there was to assist in move choice, especially where there were two metrically equi-optimal moves. So far, the models do not include 'chessic insight' such as "Opponent is overly attracted to moves that check my King" but Jansen did suggest such inclusions and my models can be extended to include such ideas.

My anticipation, and evidence is hard to come by, is that human players play the endgames less competently if there are more men on the board, and that therefore, opportunities for speculating/risking metric-depth in the reasonable expectation of greater progress will be more frequent in endgames with more men.

hgm's L_n metric can be well-defined but has the two disadvantages of being (a) difficult to calculate compared to the retrograde methods we are used to, and (b) speculative as to its value.

It has the one advantage that it defines a metric which is (much) harder to calculate than DTR/DTZR. In fact, I am confident that the DTR/DTZR metric can be efficiently computed with an evolution of MB's gtbgen code which can calculate EZk EGTs to the DTZk metric, especially if WDL EGTs are available for those endgames.

Incidentally, I think I have now said enough to demonstrate - in response to szyzygy's insight - that the ER EGT DTR/DTZR can be used effectively to win winnable positions and finesse draw-claim-vulnerable positions ... to disprove hgm's claim that "DTR is no good!" The DTR metric is more complex as a concept than DTC/DTM/DTZ(k), so if anyone has any residual doubts about the benefit of the DTR/DTZR metric, they are welcome to air them here.

g
h.g.muller
Posts: 223
Joined: Mon Feb 19, 2007 8:24 am
Sign-up code: 0
Location: Amsterdam
Contact:

Re: DTR is no good!

Post by h.g.muller »

It is interesting you mention that it is likely to encounter more fallibility in the presence of more men. The philosophy on which DTR is based, (even if you make it work, by including the current value of the reversible-move counter in the index), s tht the fallibility in all phases is equal: if you can force a win in 53+53+53 moves (for the 5-men, 4-men and 3-men phase), it would prefer that method of play over a forced win in 54+49+49 moves. In other words, it attaches more value in reducing the number of moves for the current phase from 54 to 53, than it would attach to reducing that of the 4-men and 3-men phases from 53 to 49.

Now perhaps this example is a bit extreme, because in going from 53 to 49 you pas the all-important threshold of 50. But even if the number of moves to draw was not known in advance, I think the 54+49+49 solution would be highly preferable to the 53+53+53 one: only very little fallibility is needed to reduce the longest phase in the first, by reducing the 54 in the 5-men phase. For the second one the maximum phase duration can only go down if the opponent hows fallibility even in the 3-men phase. By the time he is fallible enough to reduce the duration of that phase by one move, he might be likely to reduce that of the 5-men phase by 3 moves, so that in practice you would win by (54-3)+(49-2)+(49-1) = 51+47+48. With the second the same fallibility would give you (53-3)+(53-2)+(53-1) = 50+51+52. And 51 < 52.
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

DTR ... and fallibility

Post by guyhaw »

a reply primarily to hgm ... notice that I am not lazily recycling the "DTR is no good!" thread title, which I consider inappropriate and poorly informed as well as over-theatrical :-)

DTR is not so much based on 'philosophy' but on the simple observation that, in the presence of a k-move rule, it is the maximum value of the move-count which will determine whether play is stopped by a justified k-move-rule draw-claim or not. The minimaxed maximum-phase-length is well-defined for decisive positions, and can fortunately be calculated by a variant of the retrograde 'dynamic programming' algorithm, established and familiar for EGTs.

There is no question that the ER EGT to the DTR/DTZR metric can be generated with reasonable efficiency, even if with a little more ingenuity, beyond the algorithm for creating EZk EGTs to the DTZk metric, is necessary. There are no issues about 'making it work'. For I hope the last time, I note that it is not necessary to include the move-count in the DTR EGT. The move-count is taken into consideration by the user of the DTR EGT, not by the EGT itself.

Like any EGT, the DTR EGT provides a necessarily-limited amount of information which can be used to inform strategies for choosing moves. Richer information would allow more sophisticated strategies for finessing wins, but that's 'above my ceiling' at the moment. Now that the method has been discovered, post szyzygy's astute observation, for dealing with the partial hiding of required DTR/DTZR-optimal paths in the DTR EGT, the usefulness of the DTR EGT is clear, at least to me. The proper use of the DTR EGT will win a winnable position under any k-move rule.

However, there is no 'optimal strategy' for finessing a win against a fallible opponent: strategies can be improved by improved models of the opponent's fallibility, improved use of spare moves in the current phase, and improved ways of forward-searching a tree of moves.


To return to your example, if in a 5-man endgame, one forced a win in 54+49+49 moves, you would never get to the '49+49' as there would have been a 50-move claim in the first-phase. If in fact DTR = 53 as determined by the first phase, you would be working (in the presumed context of a 50-move rule) as priority to close out the current phase in 50 moves or less. This would involve using first the EZ EGT to the DTZ metric, and when the opponent had conceded '3' in DTZ terms, you would be able to turn to the DTR metric to reduce the maximum length of subsequent phases.

I agree that if you had a way of 'hazarding' a Phase-1 length of 54, and gambling on more concessions by the opponent in Phase-1, in return for a later max-phase-length of 49, this might be more likely to land the win, but I don't see how the 54+49+49 opportunity would be either 'stored' or recognised.

Your hypothetical example could have been better chosen of course. maxDTR =< maxDTZ for the current and future endgames, so:
maxDTR (3-man) = 16 moves c/o KRK ... so EZ == EM suffices,
maxDTR (4-man) = 33 moves c/o KBNK ... where EZ == EM suffices again,
maxDTR (5-man) = 82 moves c/o KNNKP where the first discussions of trade-offs between phases have already been published, and
[ maxDTR (6m) = 243 moves c/o KRNKNN, though other endgames like KBBKNN are more interesting DTR-wise. ]

There is one way in which DTR can be generalised, but I haven't spent much time thinking about it because, in practice, I would expect the presumed fallible opponent to concede depth quickly enough for it not to be necessary: human play suggests as much. DTZ is in fact the minimaxed maximum phase-length when considering just the current phase, a 'DTR[1]' if you will. DTR is the minimaxed maximum phase-length when considering all phases, 'DTR[all]'. Just as one first focuses on not succumbing to a move-count draw-claim in the first phase, one might then give priority to not succumbing in the first two phases (over not succumbing in the first three) - so a 'DTR[2]' for the next two phases could be useful.

KQPKQ(d) provides perhaps the most interesting scenario: maxDTZ(KQP(dn)KQ) is 41(n=2), 53(3), 64(4), 45(5), 58(6) and 42(7) [data, ICCA J v9.1].
With the pawn on d2, a potential '53' for d3 is a more immediate issue than a potential 55 for d4 or 57 for d6.

g
syzygy
Posts: 166
Joined: Sun Aug 03, 2008 4:02 pm

Re: DTR is no good!

Post by syzygy »

notnale wrote:The problem is that there is no solid answer, because fallible opponents are fallible in different ways
Good luck teaching the computer to read minds
What you can do is to have the information required to win positions that can be won when ignoring the 50-move rule. Even if you don't care too much about optimizing for fallible play, this is interesting information to have.

Let's call a position a "cursed win" if it is drawn under the 50-move rule, but can be won when ignoring the 50-move rule. The meaning of the value v stored for a position p could e.g. be:
v = 0: position is a draw
1 <= v <= 100: position is won under the 50-move rule, v plies to a winning zeroing move
v > 100: position is a cursed win. v-100 plies either to a won position within the same tablebase or pawn slice, or to a zeroing move that moves into a cursed win.
Then -100 <= v <= 1 and v < -100 indicate corresponding losses.

Another possibility is hgm's scheme. His scheme might indeed give more winning chances against a fallible opponent, but will also be more expensive in terms of computation time and storage space. I also see some complications in comparing DTx > 50 (in moves) values of different subtables: DTx=70 for one subtable has a different meaning than DTx=70 for another subtable. This could probably be solved by taking the DTx ranges large enough and accepting that tables will have holes in their DTx range (which is not a problem since compression will take care of that).

I agree with hgm that DTR does not seem to be particularly well-suited for dealing with fallible play in relation to the 50-move rule. Apart from the problem that DTR is flawed unless the move-counter is included in the index, it fails to treat the 50-move rule as special over arbitrary k-move rules. If a table is just as useful for "40-move rule" play and "60-move rule" play as it is for "50-move rule" play, then it likely contains redundant information.
guyhaw wrote:There are no issues about 'making it work'. For I hope the last time, I note that it is not necessary to include the move-count in the DTR EGT.
In my view it has been established that DTR without the move-counter in the index does not work (and with the move-counter in the index the tables most likely simply get too big). DTR-values can still be used as upper bounds to guide a search, but that doesn't seem very practical and is not a very good use of computation time and storage space. I find it inaccurate to state that "proper use" of DTR will win a winnable position, if that "proper use" involves deep searches.
kronsteen
Posts: 88
Joined: Fri Aug 01, 2008 11:20 am
Sign-up code: 0
Location: France

Re: DTR is no good!

Post by kronsteen »

syzygy wrote:and with the move-counter in the index the tables most likely simply get too big
I agree that with the move counter in the index (which is, as I currently believe, necessary for metrics such as DTR, DTM50 -or DTMx-, DTC50 -or DTCx-) tables will grow a lot, but their size will certainly be still manageable provided the targeted TB set is not too ambitious, say 3-5 men first, followed if successful by a few interesting 6-men samples. I don’t expect such tables weighing more than 5-10x DTM Nalimov tables (see why below). This puts an upper limit for DTR or DTM50 tables at 75 GB for 5-men (immediately achievable) and 12 TB for 6-men (far more ambitious, but in similar range as WDL 7-men, finally).

For each position, DTR/DTM50/DTC50 metrics are not a single number, but a finite series of non-strictly growing numbers with move count (or, more accurately, ply count) as index. For DTM50/DTC50, values can become +inf if position becomes a draw when move count exceeds a certain value (they automatically do so at plycount = 100). For DTR, I think it can be easily demonstrated that for every position there is a move count value for which DTR becomes equal to movecount+DTZ (and sticks to that for every higher move count), so what is needed is storage of this move count value and DTR values for lower move counts.

Examples :
DTM50 = 50 if MC <=15, 60 if 16<=MC<=40, +inf if MC>=41
DTR=10 if MC<=5, 14 if 6<=MC<=12, MC+3 if MC>=13 (position has DTZ of 3).

I bet that for most positions DTR/DTM50/DTC50 series will contain only few different values. Many positions will have only 2 values (ex : DTM50=DTM if plycount<x and +inf instead), many more will probably be in the 3-10 value range, and positions with more than 10 DTR/DTM50 values will certainly be very exceptional. That’s why with a proper compression scheme these tables should remain of acceptable size, at least for 3-5 men, and maybe for some 6-men samples.

I would be very enthusiastic (as I am for 7-men :) ) to see sb launch a try on this.
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

DTR - in summary

Post by guyhaw »

I think I have explained as clearly as is possible:

- how the ER EGT to the DTR/DTZR metric is created: like other EGTs, it does not need to recognise move-count information,
- how the ER EGT should be used, given that minimaxing-paths to current-DTR-targets can be hidden by longer paths to lower DTR, and
- how the ER EGT is used to exploit the fallibility of a fallible opponent.

szyzygy noted the partial hiding of the minimaxing-path in the DTR EGT, which has been a step forward here. I have responsed to that, showing how DTR is used to narrow forward-search as it cannot always be used for direct lookup.

I don't think I have anything further to add, and am quite happy with the revised status of DTR and DTZR.

g
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

Further observations on KRBNKQN and DTR

Post by guyhaw »

I noted that, where DTZ >= any DTZ that can occur in a subgame, dr = dzr = dz.

For KRBNKQN, we have maxDTZ = 17 (for KBNKQN), 40 (KRNKQN), 26 (KRBKQN), 29 (KRBNKQ) and 01 (KRBNKN): successor-maxDTZ = 40. Thus, for dz >= 40, we know dr = drz = dz directly.

The most important effect is that, with positions with DTR = dr taking some '2*dr' cycles to recognise, we cut the number of retro-cycles from 267,806 to 1640. The 'deeper DTR' positions were only 6.25% of the btm won positions and 21.50% of the wtm lost positions [cf mb's stats elsewhere here] but the computational benefit is much greater than these percentages.

A move-filtering strategy starting SV+Zo and using DTR/DTZR from a deep position would be dominated by DTZ and the generousity of the opponent for a long while and it would be interesting to see how quickly KRBN would concede depth in DTZ terms. After all, only 6.25% of the wtm-lost positions have DTZ > 40. In fact, as the problem-phase (50-move rule-wise) is only the first phase, one could just drive this endgame out with the DTZ EGT 'EZ'.

DTR's value is where the following phases are longer than the current phase, and longer than the 'k' of the k-move rule prevailing - e.g. KBBKP, KNNKP and KQPKQ as previously discussed.

g
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

The KNNKP Challenge

Post by guyhaw »

Ignoring the 50-move rule, wtm can win 40,546,725 positions: with the 50-move rule, wtm should only win 29,861,757 of these.
So 10,684,968 positions are affected by the 50m-rule. Only 1,966,997 of these have DTZ > 50: for these, let's assume you focus on reducing DTZ.
That leaves 8,717,971 positions with DTZ =< 50 and DTR > 50 .. over 21% of the 'DTZ' wins.

Question 1 is ... what move-choosing strategy would you use to try to finesse a win in these positions?

Question 2, for alpha-player points is, how would you best try for a win in the 85,991,778 'DTZ' draws?

g
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

The KQPKQ Challenge

Post by guyhaw »

Analogous to, but on a much smaller scale than, the KNNKP challenge ...

Ignoring the 50-move rule, wtm can win 136,304,160 positions: with the 50-move rule, wtm should only win 136,275,692 of these.
So 28,468 of these positions are affected by the 50m-rule. Only 6,434 of these have DTZ > 50: for these, let's assume one focuses on reducing DTZ.
That leaves 22,034 positions with DTZ =< 50 and DTR > 50 ... only 0.02% of the 'DTZ wins'.

Same questions as before ...
Question 1 is ... what move-choosing strategy would you use to try to finesse a win in these positions?

Question 2, for alpha-player points is, how would you best try for a win in the 62,203,047 KQPKQ 'DTZ' draws?

g
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

The KBBKNN Challenge

Post by guyhaw »

Analogous to the KNNKP and KQPKQ Challenges, on a larger scale, but only involving two phases of play ...

Ignoring the 50-move rule, wtm can win 282,912,378 positions: with the 50-move rule, wtm should only win 141,038,155 of these.
So 141,874,223 of these positions are affected by the 50m-rule. None of these have DTZ > 50: maxDTZ = 38 and maxDTZ50 = 29, less!
All affected positions have DTZ =< 50 and DTR > 50 ... over 50% of the 'DTZ wins'.

Same questions as before ...
Question 1 is ... what move-choosing strategy would you use to try to finesse a win in these positions?

Question 2, for alpha-player points is, how would you best try for a win in the 885,809,752 KBBKNN 'DTZ' draws?

Answers to these questions would arguably convert KBBKNN from a 'likely draw' endgame to a 'likely won' endgame.
g
Post Reply