Thank you for your informative reply.
It is really good news that I can set aside my 'April fool' doubts as I hoped. I had just been through a sceptical phase after reading Chessbase's neat April Fool story about FIDE raising the K-factor of the ELO system to 60 (which, at the time of writing, is not true).
It is some time since I have been visiting this Discussion Board regularly, and I tended not to take much interest in items which majored on the 50-move rule as one cannot rely on FIDE not changing '50' to '60' or '40': they had tried to accommodate deeper endgames in the past. If they were to go for a round number that is higher than all known endgames, it would be around 600 today.
However, it is clear that you are combining a number of ideas that are newish or at least 'revivals' for me, so I attempt some sort of 'reprise' and review here for at least my benefit:
01) You are using value-only 'WDL-style' EGTs. I had advocated the creation and use of these and indeed, the use of WDL EGTs in the creation of DTx EGTs as this avoids examining potential forced-losses which are in fact draws.
02) The values-only EGT is to be used either on its own or in conjunction with a DTx EGT giving depth information. Stefan Meyer-Kahlen has introduced Shredder_Bases for sub-6-man (but unfortunately not sub-7-man) chess and users of SHREDDER get the benefit there as there is no need to 'Nalimov' some positions. Eugene Nalimov must know he has become part of the fabric of chess when his family name has become a verb!
03) Because your provide a values-only EGT, there is no need to provide for the DTx EGT being used on its own. Therefore, the DTx EGT does not have to carry information which can be inferred from the values-only EGT. Thus, you do not need to indicate whether a position is a win, draw or loss, removing the need for (a) a 'draw' value, and (b) a 'sign' on depth to indicate stm win or stm loss.
04) You are extending the idea of 'WDL' EGTs to 'WDL+' EGTs (or strictly, to a WDL50+ EGT) where I think you have five values, WIn/Draw/Loss (under the 50-move rule) plus:
- C ... 'Cursed Win', i.e. would be a Win without the 50-move rule but is not a win under the 50-move rule
- B ... Blessed Loss', i.e. would be a Loss without the 50-move rule but is not a Loss under the 50-move rule
05) It follows that before one can create a WDL50+/DTZ50+ pair of EGTs, one has to create a WDL/DTZ pair of EGTs.
06) As per point 03, the accompanying 'DTZ50+' EGT need not carry information which can be inferred from the WDL50+ EGT. Thus, a '7' in the DTZ50+ can mean, as indicated in the WDL50+ EGT:
- 7 winner's moves to either a DTZ50-win or a DTZ50-loss in the next phase of play (after a Pawn-move and/or capture),
- a draw (ignoring the 50-move rule) ... where '7' just happens to be the most convenient value to put there from compression considerations,
- (if WDL50+ has a 'C' or a 'B' against this position) a win/loss in absolute terms, but 7-moves from 'something' given the 50-move rule
I need to think more about what this 'something' is. I have a feeling there is some 'information loss' here but cannot resolve this entirely yet.
07) You are saving at least half the size of the EGT by only keeping the smaller stm-EGT, at the cost of requiring the chess-engine to do an extra one-ply search. Thus, if the EGT were remote, e.g. on the web, this would multiply the communication cost by 'm' where 'm' could be quite large. Ken Thompson did something similar I think with his original DTC EGTs.
08) You are using a compression-scheme which is not that of Andrew Kadatch as in the Nalimov EGTs. You say this is based on ideas of Larsson and Moffat,
http://www.larsson.dogma.net/dcc99.pdf, and has Huffman-coding as a follow-up to iteratively creating an alphabet of symbols. I guess this proves to be better than Kadatch's scheme. Compression/decompression-time seems not to be an important parameter compared to disc-I/O time. I think the recent MVL 7-man initiative uses LZW-compression but they have not expounded on that much yet.
09) However, and this is an issue with Nalimov EGTs as well, the size of the indexing table needs to be included in the effective size of the EGT, and maybe it has been in your case already. Initialising the Nalimov EGTs seems to take a long time.
10) As you say, it is the combination of space and time that determines the utility of one's EGT. I like your observation that 'all EGTs can be compressed to the size of the code that generates them'
I will dig out my sub-6-man DTZ50 EGTs. It may be that the size I quoted was only for DTZ50 EGTs that are different from the DTZ EGTs and maybe the article does not make this clear.
Congratulations on your achievements: this is certainly an original approach bringing new ideas to the 'state of the art'.
I would be interested in using your WDL/DTZ (rather than your WDL50+/DTZ50+) EGTs at some stage if that were convenient - as I am more interested in an 'absolute truth' not moderated by FIDE's current k-move rule.
I also have several large sets of sub-6-man positions, sorted by endgame, which I can share somehow if you care to evaluate them in DTZ terms for me.
Best wishes ... g