kqpkrb

Endgame analysis using tablebases, EGTB generation, exchange, sharing, discussions, etc..
mbourzut
Posts: 30
Joined: Fri Mar 03, 2006 7:48 pm
Sign-up code: 0

kqpkrb

Post by mbourzut »

I have created the "missing" Nalimov endgame kqpkrb by transforming the output from Johan de Koning's FEG program. This takes several hours and lots of disk space, but is still quite a bit faster than directly creating it with the Nalimov program. Also, the conversion has not been extensively tested, but so far seems to produce sensible results.

For those who want to compare with other computations I'm attaching the .tbs and .md5 files.

There is one unique full-point mutual zugzwang, where whoever is to move loses:

2K1k3/2P5/8/8/8/8/1r2b3/4Q3 w - - 0 1
2K1k3/2P5/8/8/8/8/1r2b3/4Q3 w - - 0 1

The longest mate is in 155:

8/1kb5/8/8/3r4/8/4P3/2K1Q3 w - - 0 1
8/1kb5/8/8/3r4/8/4P3/2K1Q3 w - - 0 1

[Event "W +155"]
[Date "????.??.??"]
[Site "?"]
[Round "?"]
[White "QP"]
[Black "RB"]
[Result "1-0"]
[SetUp "1"]
[FEN "8/1kb5/8/8/3r4/8/4P3/2K1Q3 w - - 0 1"]

1. Qh1+! Kb6 2. Qg1! Be5 3. Qg8! Rd6 4. Qb8+! Kc5 5. Qe8! Kd4 6. Kc2! Bf4 7.
Qa4+ Ke5 8. Qb5+ Rd5 9. Qb8+! Ke4 10. Qb4+ Ke5 11. Qe7+ Kd4 12. Qf6+ Ke4 13.
Qc6 Ke5 14. Kc3 Bd2+ 15. Kc4! Rd4+ 16. Kc5 Bb4+ 17. Kb5! Be7 18. Qe8 Ke6 19.
Qg6+ Ke5 20. Qg3+ Kd5 21. Kb6 Bd8+ 22. Kb7 Bf6 23. e3! Rb4+ 24. Kc8! Bc3 25.
Qg6 Kc4 26. Qe4+! Kc5 27. Qd3 Rc4 28. Kd7 Kb4 29. Qb1+! Ka5 30. Kd6 Bb4+ 31.
Kd5 Rc8 32. Qe4 Kb5 33. Qd3+! Kb6 34. Ke4 Re8+ 35. Kf3 Rf8+ 36. Ke2 Rc8 37.
Qd4+ Bc5 38. Qb2+ Kc6 39. Qa2 Rc7 40. Qa6+! Bb6 41. e4 Re7 42. Kf3 Re5 43.
Qa4+ Kb7 44. Qb4 Kc6 45. Qc3+ Rc5 46. Qf6+! Kb7 47. Qd6 Rc3+ 48. Kg4 Rc5 49.
Qe7+ Rc7 50. Qa3 Rc5 51. Qb2 Ka7 52. Qg7+ Kb8 53. Qh6 Ka7 54. Qf8 Kb7 55. Qd6
Rc2 56. Qd7+ Rc7 57. Qd5+ Ka7 58. Qa2+ Kb7 59. Qb2 Kc6 60. Kf5 Bc5 61. Qa2!
Kb5 62. Qd5 Ra7 63. Qd3+ Kb6 64. Qb3+! Kc6 65. Qe6+ Kb5 66. Qc8 Re7 67. Qd8!
Kc6 68. Kf6! Rc7 69. Qd5+ Kb5 70. Qd3+ Ka5 71. Qg3 Kb6 72. Qb3+! Kc6 73.
Qa4+ Kb6 74. Ke5 Be7 75. Ke6 Bf8 76. Qb3+ Kc6 77. Qc2+ Kb6 78. Qb2+ Kc6 79.
Qb3 Bc5 80. Qa4+ Kb6 81. Kd5 Be7 82. Qb3+ Ka6 83. Qb8 Rb7 84. Qc8 Kb6 85. Qe6+
Ka7 86. Qc6 Kb8 87. Qc3 Bd8 88. Qg3+ Kc8 89. Qa3 Kb8 90. Ke6 Rc7 91. Qb2+ Rb7
92. Qh2+ Kc8 93. Qc2+ Kb8 94. Qc5 Ra7 95. Qd4 Re7+ 96. Kf5 Kc8 97. Qc4+ Rc7
98. Qe6+ Kb8 99. Qd5 Be7 100. Ke6 Bb4 101. Qe5 Bc5 102. Qc3 Re7+ 103. Kd5 Rd7+
104. Kc4 Rc7 105. Qh8+ Kb7 106. Qb2+ Kc8 107. Kd5 Be7 108. Ke6 Bd8 109. Qa2
Re7+ 110. Kf5 Rb7 111. Qa8+ Kc7 112. Ke6 Rb6+ 113. Kf7 Rb7 114. Ke8 Bh4 115.
Qa5+ Kb8 116. Qe5+ Ka7 117. Qc5+ Kb8 118. Qd6+ Ka8 119. Qh6 Bf2 120. Qd2 Bb6
121. e5 Rb8+ 122. Kf7 Rb7+ 123. Ke6 Kb8 124. Qd6+ Kc8 125. Qc6+ Kb8 126. Qe8+
Ka7 127. Kd6 Be3 128. Qd8 Bd2 129. e6 Bc3 130. Kd5 Bb4 131. Kc6 Be7 132. Qc8
Rb6+ 133. Kd7 Ba3 134. e7 Bxe7 135. Kxe7! Rb5 136. Qh3 Ka6 137. Qa3+ Kb7 138.
Ke6 Kb8 139. Qa4 Rb2 140. Qa5 Kb7 141. Kd7 Rb6 142. Qa4 Rh6 143. Qe4+ Ka6 144.
Kc7 Ka5 145. Qf5+ Kb4 146. Qf4+ Kc5 147. Qxh6 Kd4 148. Qc1 Kd3 149. Qe1 Kc4
150. Kc6 Kb3 151. Kd5 Ka2 152. Qb4 Ka1 153. Kd4 Ka2 154. Kc3 Ka1 155. Qb2#
1-0
Attachments
kqpkrb.zip
(3.77 KiB) Downloaded 336 times
User avatar
Martin Kreuzer
Posts: 53
Joined: Thu Jun 08, 2006 9:02 am
Sign-up code: 0
Contact:

kqpkrb and kqpkrn

Post by Martin Kreuzer »

Hi Marc,

presently I am computing kqpkrn. I am using Nalimov's program.
My computer is now doing the 9th iteration, with 63 still to go.
The speed is currently a little below 1 iteration per day.

After that I had planned to do kqpkrb. Do you think it is worthwhile
to do this? Or are their better ways to check correctness?

Do you plan to do kqpkrb, too, so we can compare results?

How do you plan to share the endgame? (emule, ftp, ...)
The size of my two uncompressed files will be 16 and 17 GB.

Best wishes and kind regards,
Martin Kreuzer
User avatar
Kirill Kryukov
Site Admin
Posts: 7399
Joined: Sun Dec 18, 2005 9:58 am
Sign-up code: 0
Location: Mishima, Japan
Contact:

Post by Kirill Kryukov »

Hi Marc, Martin! It is great what you are doing, please keep it on! About distribution, I suggest using FTP to upload new files to 2 or 3 volanteers, who will then share them with eMule. I will be glad to help with this. :-)
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

FEG ---> EN conversion verification

Post by guyhaw »

Nice work, Marc.

I would have thought a verification of an 'EN DTM EGT' produced via FEG would be an independent confirmation that it is aok. This would be quicker than generating it again with tbgen.

I read somewhere that the .a/.b/.c etc files are produced by chopping the uncompressed EGT files into pieces of size 2^31 bytes. I guess you would then get exactly what Eugene produced last year.

g
mbourzut
Posts: 30
Joined: Fri Mar 03, 2006 7:48 pm
Sign-up code: 0

kqpkrb and kqpkrn

Post by mbourzut »

Hi Martin,

The good news is that the iterations on kqpkrn should speed up over time, but the bad news is that there are going to be 110 iterations. 69 is just the minimum necessary to resolve all promotions and captures. I'm attaching my results for kqpkrn.

Your program is very slow. The Nalimov algorithm requires 2 bits of memory per position, which in your case is about (17/4) = 4.25 GB. Unless you have ported the Nalimov code to true 64 bit there will be a lot of swapping. Endings with repeated pieces, like krnkpp and krbkpp require less memory, and should run much faster.

The easiest way for me to share results is to put them on a disk, and send them to someone (preferably in the US) who can then ftp. Creating all the missing Nalimov files will only take a few days. Testing can be done by running the verification option in the Nalimov code, which is a lot faster than creating. I could do this myself, but then I would have to convert all the relevant subgames as well. Instead, I will only test a few representative endings.

-Marc
Attachments
kqpkrn.zip
(2.78 KiB) Downloaded 308 times
Arpad Rusz
Posts: 93
Joined: Mon Mar 27, 2006 5:33 pm
Sign-up code: 0
Location: Romania/Hungary
Contact:

Post by Arpad Rusz »

Marc,
let's see if you can reproduce Eugene's .tbs for KPPKPP:

wtm: Lost in 0: 16424
wtm: Lost in 1: 507796
wtm: Lost in 2: 496720
wtm: Lost in 3: 1147055
wtm: Lost in 4: 3032634
wtm: Lost in 5: 10848451
wtm: Lost in 6: 18353722
wtm: Lost in 7: 23912416
wtm: Lost in 8: 28010956
wtm: Lost in 9: 34162876
wtm: Lost in 10: 46080764
wtm: Lost in 11: 60117848
wtm: Lost in 12: 63816352
wtm: Lost in 13: 52442596
wtm: Lost in 14: 37593507
wtm: Lost in 15: 26714573
wtm: Lost in 16: 19913391
wtm: Lost in 17: 15605294
wtm: Lost in 18: 12278349
wtm: Lost in 19: 9792049
wtm: Lost in 20: 7858326
wtm: Lost in 21: 6303174
wtm: Lost in 22: 5140237
wtm: Lost in 23: 4124535
wtm: Lost in 24: 3223929
wtm: Lost in 25: 2518362
wtm: Lost in 26: 2017911
wtm: Lost in 27: 1648888
wtm: Lost in 28: 1354585
wtm: Lost in 29: 1129779
wtm: Lost in 30: 960248
wtm: Lost in 31: 834200
wtm: Lost in 32: 737737
wtm: Lost in 33: 664019
wtm: Lost in 34: 608193
wtm: Lost in 35: 566743
wtm: Lost in 36: 534043
wtm: Lost in 37: 507318
wtm: Lost in 38: 487547
wtm: Lost in 39: 470243
wtm: Lost in 40: 455486
wtm: Lost in 41: 439773
wtm: Lost in 42: 423811
wtm: Lost in 43: 408598
wtm: Lost in 44: 392350
wtm: Lost in 45: 378712
wtm: Lost in 46: 362622
wtm: Lost in 47: 347759
wtm: Lost in 48: 337113
wtm: Lost in 49: 326931
wtm: Lost in 50: 317403
wtm: Lost in 51: 308784
wtm: Lost in 52: 302505
wtm: Lost in 53: 295227
wtm: Lost in 54: 284033
wtm: Lost in 55: 271835
wtm: Lost in 56: 259456
wtm: Lost in 57: 244961
wtm: Lost in 58: 229543
wtm: Lost in 59: 216907
wtm: Lost in 60: 202788
wtm: Lost in 61: 192666
wtm: Lost in 62: 180877
wtm: Lost in 63: 171961
wtm: Lost in 64: 164744
wtm: Lost in 65: 158696
wtm: Lost in 66: 151163
wtm: Lost in 67: 144098
wtm: Lost in 68: 138285
wtm: Lost in 69: 131179
wtm: Lost in 70: 123877
wtm: Lost in 71: 114716
wtm: Lost in 72: 106386
wtm: Lost in 73: 100586
wtm: Lost in 74: 93773
wtm: Lost in 75: 87242
wtm: Lost in 76: 82227
wtm: Lost in 77: 77005
wtm: Lost in 78: 70664
wtm: Lost in 79: 67068
wtm: Lost in 80: 62627
wtm: Lost in 81: 58868
wtm: Lost in 82: 54395
wtm: Lost in 83: 49542
wtm: Lost in 84: 44129
wtm: Lost in 85: 38394
wtm: Lost in 86: 33619
wtm: Lost in 87: 29035
wtm: Lost in 88: 24773
wtm: Lost in 89: 22230
wtm: Lost in 90: 20108
wtm: Lost in 91: 17785
wtm: Lost in 92: 15247
wtm: Lost in 93: 13572
wtm: Lost in 94: 12113
wtm: Lost in 95: 11140
wtm: Lost in 96: 9742
wtm: Lost in 97: 8671
wtm: Lost in 98: 7644
wtm: Lost in 99: 6872
wtm: Lost in 100: 5664
wtm: Lost in 101: 4854
wtm: Lost in 102: 3848
wtm: Lost in 103: 3064
wtm: Lost in 104: 2481
wtm: Lost in 105: 2067
wtm: Lost in 106: 1777
wtm: Lost in 107: 1443
wtm: Lost in 108: 1058
wtm: Lost in 109: 874
wtm: Lost in 110: 840
wtm: Lost in 111: 741
wtm: Lost in 112: 621
wtm: Lost in 113: 445
wtm: Lost in 114: 328
wtm: Lost in 115: 354
wtm: Lost in 116: 288
wtm: Lost in 117: 299
wtm: Lost in 118: 338
wtm: Lost in 119: 382
wtm: Lost in 120: 320
wtm: Lost in 121: 229
wtm: Lost in 122: 205
wtm: Lost in 123: 223
wtm: Lost in 124: 214
wtm: Lost in 125: 172
wtm: Lost in 126: 192
wtm: Lost in 127: 213
wtm: Lost in 128: 160
wtm: Lost in 129: 157
wtm: Lost in 130: 111
wtm: Lost in 131: 59
wtm: Lost in 132: 45
wtm: Lost in 133: 48
wtm: Lost in 134: 25
wtm: Lost in 135: 7
wtm: Lost in 137: 4
wtm: Lost in 138: 14
wtm: Lost in 139: 25
wtm: Lost in 140: 27
wtm: Lost in 141: 7
wtm: Draws: 297012778
wtm: Mate in 141: 7
wtm: Mate in 140: 28
wtm: Mate in 139: 28
wtm: Mate in 138: 9
wtm: Mate in 137: 1
wtm: Mate in 135: 11
wtm: Mate in 134: 27
wtm: Mate in 133: 32
wtm: Mate in 132: 45
wtm: Mate in 131: 62
wtm: Mate in 130: 113
wtm: Mate in 129: 139
wtm: Mate in 128: 192
wtm: Mate in 127: 273
wtm: Mate in 126: 283
wtm: Mate in 125: 305
wtm: Mate in 124: 326
wtm: Mate in 123: 354
wtm: Mate in 122: 315
wtm: Mate in 121: 382
wtm: Mate in 120: 461
wtm: Mate in 119: 512
wtm: Mate in 118: 464
wtm: Mate in 117: 441
wtm: Mate in 116: 391
wtm: Mate in 115: 509
wtm: Mate in 114: 617
wtm: Mate in 113: 915
wtm: Mate in 112: 1171
wtm: Mate in 111: 1298
wtm: Mate in 110: 1195
wtm: Mate in 109: 1176
wtm: Mate in 108: 1409
wtm: Mate in 107: 2145
wtm: Mate in 106: 2644
wtm: Mate in 105: 3071
wtm: Mate in 104: 3464
wtm: Mate in 103: 4409
wtm: Mate in 102: 5873
wtm: Mate in 101: 7380
wtm: Mate in 100: 8645
wtm: Mate in 99: 10319
wtm: Mate in 98: 11378
wtm: Mate in 97: 12913
wtm: Mate in 96: 14704
wtm: Mate in 95: 16015
wtm: Mate in 94: 18020
wtm: Mate in 93: 19928
wtm: Mate in 92: 22756
wtm: Mate in 91: 28529
wtm: Mate in 90: 32505
wtm: Mate in 89: 35267
wtm: Mate in 88: 40823
wtm: Mate in 87: 47239
wtm: Mate in 86: 54224
wtm: Mate in 85: 62531
wtm: Mate in 84: 71450
wtm: Mate in 83: 79169
wtm: Mate in 82: 86789
wtm: Mate in 81: 96171
wtm: Mate in 80: 103846
wtm: Mate in 79: 110683
wtm: Mate in 78: 118907
wtm: Mate in 77: 129230
wtm: Mate in 76: 135948
wtm: Mate in 75: 143869
wtm: Mate in 74: 154990
wtm: Mate in 73: 164795
wtm: Mate in 72: 177329
wtm: Mate in 71: 192077
wtm: Mate in 70: 207660
wtm: Mate in 69: 220711
wtm: Mate in 68: 230913
wtm: Mate in 67: 240413
wtm: Mate in 66: 250037
wtm: Mate in 65: 261800
wtm: Mate in 64: 271393
wtm: Mate in 63: 279941
wtm: Mate in 62: 292092
wtm: Mate in 61: 303678
wtm: Mate in 60: 319013
wtm: Mate in 59: 332781
wtm: Mate in 58: 349486
wtm: Mate in 57: 367817
wtm: Mate in 56: 383141
wtm: Mate in 55: 396887
wtm: Mate in 54: 414567
wtm: Mate in 53: 423300
wtm: Mate in 52: 426857
wtm: Mate in 51: 430127
wtm: Mate in 50: 434625
wtm: Mate in 49: 440806
wtm: Mate in 48: 443319
wtm: Mate in 47: 453108
wtm: Mate in 46: 466982
wtm: Mate in 45: 479632
wtm: Mate in 44: 488180
wtm: Mate in 43: 499585
wtm: Mate in 42: 507895
wtm: Mate in 41: 517556
wtm: Mate in 40: 526006
wtm: Mate in 39: 534418
wtm: Mate in 38: 543832
wtm: Mate in 37: 561484
wtm: Mate in 36: 583281
wtm: Mate in 35: 614403
wtm: Mate in 34: 664008
wtm: Mate in 33: 727485
wtm: Mate in 32: 821291
wtm: Mate in 31: 948836
wtm: Mate in 30: 1113892
wtm: Mate in 29: 1347670
wtm: Mate in 28: 1686010
wtm: Mate in 27: 2199282
wtm: Mate in 26: 2916618
wtm: Mate in 25: 4023291
wtm: Mate in 24: 5566593
wtm: Mate in 23: 7532153
wtm: Mate in 22: 9838521
wtm: Mate in 21: 12577005
wtm: Mate in 20: 15825836
wtm: Mate in 19: 19378606
wtm: Mate in 18: 23225341
wtm: Mate in 17: 28266856
wtm: Mate in 16: 35061095
wtm: Mate in 15: 45159623
wtm: Mate in 14: 60442039
wtm: Mate in 13: 81495138
wtm: Mate in 12: 101203334
wtm: Mate in 11: 107438442
wtm: Mate in 10: 95531810
wtm: Mate in 9: 78153768
wtm: Mate in 8: 66287726
wtm: Mate in 7: 55341878
wtm: Mate in 6: 42515081
wtm: Mate in 5: 28740655
wtm: Mate in 4: 12882535
wtm: Mate in 3: 4815089
wtm: Mate in 2: 3588606
wtm: Mate in 1: 2044993
wtm: Broken positions: 128354347

which boils down to:

16.60% draws
54.55% stm wins, average DTM-depth 12.62
28.85% sntm wins, average DTM-depth 13.43


I think that would be a good confirmation of corectness!
User avatar
jshriver
Posts: 298
Joined: Tue Jan 31, 2006 5:59 am
Sign-up code: 0
Location: Toledo, OH, USA
Contact:

Post by jshriver »

Kirill Kryukov wrote:Hi Marc, Martin! It is great what you are doing, please keep it on! About distribution, I suggest using FTP to upload new files to 2 or 3 volanteers, who will then share them with eMule. I will be glad to help with this. :-)
I'd also be willing to host them as well :) http and amule

-Josh
User avatar
jshriver
Posts: 298
Joined: Tue Jan 31, 2006 5:59 am
Sign-up code: 0
Location: Toledo, OH, USA
Contact:

Re: kqpkrb and kqpkrn

Post by jshriver »

Martin Kreuzer wrote:Hi Marc,

presently I am computing kqpkrn. I am using Nalimov's program.
My computer is now doing the 9th iteration, with 63 still to go.
The speed is currently a little below 1 iteration per day.

After that I had planned to do kqpkrb. Do you think it is worthwhile
to do this? Or are their better ways to check correctness?

Do you plan to do kqpkrb, too, so we can compare results?

How do you plan to share the endgame? (emule, ftp, ...)
The size of my two uncompressed files will be 16 and 17 GB.

Best wishes and kind regards,
Martin Kreuzer
Greetings Martin,

I sent you an email a while back after your initial post offering to email the source of the egtb gen.. is that offer still up? I'd really enjoy setting aside a couple of my cluster nodes to work on this. Just need to take them off the network.

-Josh
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

EN-FEG cross-comparison

Post by guyhaw »

[ Oops: now I see that Marc has already split the EN-via-FEG KQPRKB DTM EGT into its .a/.b etc parts. ]
KQPKRB is one of 4 endgames for which we have EN .tbs files but no EN DTM EGT files. The others are KBNKPP, (importantly) KPPKPP and KQPKRN.
Marc's provided .tbs stats, presumably from his 'EN-via-FEG' EGT, agree for each ply-level with EN's original KQPKRB .tbs stats.
Marc has confirmed that his 3-3p FEG .lof stats agree with EN's 3-3p FEG .tbs stats where the latter are available.
I have personally only checked agreement of maxDTMs and number of positions at maxDTM, plus per-ply-level-agreement of number of positions for a few endgames. I haven't found any discrepancies and don't expect to find any.
When Marc's 'EN-via-FEG' EGTs are verified (using EN's verifier) against the EN EGTs for subgames, the full verification will have been done. Again, I'd forgotten that MB might not have the EN EGTs for those subgames.
g
User avatar
Martin Kreuzer
Posts: 53
Joined: Thu Jun 08, 2006 9:02 am
Sign-up code: 0
Contact:

kqbkrn

Post by Martin Kreuzer »

Hi Marc, Josh,

did I understand correctly that you have also generated
kqpkrn already? I will let my computer finish the calculation anyway
because I would like to compare the result.

The source code of tbgen I compiled using g++ and -O3 is attached.
I did nothing but eliminate the error messages, so there is probably ample
room for improvement. The compiled program is true 64 bit,
but not multiprocessor. The size of the computation is 8.1 GB.

Thank you for sharing the "missing" endgames!
Cordial greetings,
Martin Kreuzer
Attachments
linux_src.zip
(115.44 KiB) Downloaded 365 times
User avatar
jshriver
Posts: 298
Joined: Tue Jan 31, 2006 5:59 am
Sign-up code: 0
Location: Toledo, OH, USA
Contact:

Re: kqbkrn

Post by jshriver »

Martin Kreuzer wrote:Hi Marc, Josh,

did I understand correctly that you have also generated
kqpkrn already? I will let my computer finish the calculation anyway
because I would like to compare the result.

The source code of tbgen I compiled using g++ and -O3 is attached.
I did nothing but eliminate the error messages, so there is probably ample
room for improvement. The compiled program is true 64 bit,
but not multiprocessor. The size of the computation is 8.1 GB.

Thank you for sharing the "missing" endgames!
Cordial greetings,
Martin Kreuzer
Since this is a CPU intensive program, have you considered or tried using Intel's C compiler? You can get free license for personal use. In almost everything I've compiled using icc I get 25-30% speed boast just by switching compilers. That could shave off days or possibly weeks on a serious egtb generation.

Also do you have a makefile or what did you do to compile this? I tried
gcc -O3 -c *.c
g++ -03 -c *.cpp and I get a lot of errors.

-Josh
User avatar
Martin Kreuzer
Posts: 53
Joined: Thu Jun 08, 2006 9:02 am
Sign-up code: 0
Contact:

compilation

Post by Martin Kreuzer »

Hi Josh,

since the computer I am using belongs to my research project at the university, I will probably not get the Intel compiler for free. If you suceed in compiling a faster executable, I'd be very interested. I used the command

g++ -O3 -o tbgen tbgen.cpp

Of course, structuring the code into partial files and creating a make file are on my "to do"-list (which is very long). You may have to adjust the compiler directives at the beginning of tbgen.cpp, eg. T_INDEX64 means 64 bit table base position index numbers. If you use it on a 32 bit system, the code seems to apply a certain "work around".

Good luck and best wishes,
Martin Kreuzer
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

FEG-derived EN EGTs: stats confirmations

Post by guyhaw »

Ok, I'm catching up with Marc's FEG-derived EN-style EGT stats ...
His EN-style .tbs stats for what I take to be his EN-style KQPKRB and KQPKRN EGTs as derived from his FEG EGTs ... agree for each ply-level with the .tbs stats provided by Nalimov himself a year ago. I expected nothing less.
So one would not bet against these EGTs being entirely correct, but a Nalimov-style EGT-verification, which involves KQ(Q/R/B/N)KRB etc, gives the full confirmation that the EGT is ok. This needs one of the 'core players' who have all the requisite Nalimov EGTs.
g
guido
Posts: 42
Joined: Sun Jun 18, 2006 8:55 pm
Sign-up code: 0
Location: Milan, Italy

Re: compilation

Post by guido »

Martin Kreuzer wrote:Hi Josh,

since the computer I am using belongs to my research project at the university, I will probably not get the Intel compiler for free. If you suceed in compiling a faster executable, I'd be very interested. I used the command

g++ -O3 -o tbgen tbgen.cpp

Of course, structuring the code into partial files and creating a make file are on my "to do"-list (which is very long). You may have to adjust the compiler directives at the beginning of tbgen.cpp, eg. T_INDEX64 means 64 bit table base position index numbers. If you use it on a 32 bit system, the code seems to apply a certain "work around".

Good luck and best wishes,
Martin Kreuzer

Hi Martin and all,

I read this thread with much interest and I have some comments to do:

- How can exist the Nalimov kppkpp files, in particular the kppkpp.tbs, reported by Arpad Rusz, if all_the_kxxkxx (x represents any man) endings were not previously generated? Or why are they not available?

- It would be interesting to know the algorithm applied by EN. Some years ago he explained on CCC that tbgen used the direct method but this is probably no more true in this version because the times of generation would be much higher.
For this reason I presume he uses the retrograde algorithm. This is much faster but requests much more accesses to the TBs, and therefore the program uses more I/O, as the 6-man TBs cannot be kept entirely in the RAM. Tests with my program indicate that in the ending kqrkqr (about 6 GB), certainly easier than kqpkrn, the time for I/O with the retrograde algorithm tends to be the main part of the total time; in fact the total time spent for one iteration was about 15000 s, of which 4800 s of cpu time and 10200 s of I/O time. This tendency increases over the time with the iterations because in the retrograde algorithm cpu time is proportional to the number of the opponent's victories, which are correspondently decreasing.

- I generally use "gcc -O2 -fomit-frame-pointer ... " I didn't find any difference between -O2 and -O3 and gcc and g++, while -fomit-frame-pointer accelerates the computation because it gives an extra register variable which can be important in small but frequently used functions.
What is the meaning of "I did nothing but eliminate the error messages, so there is probably ample room for improvement"? I don't think that error messages are time consuming, but improving EN program seems to me very hard. Do you think to convert into assembler some functions?
"The size of computation is 8.1 GB": does this mean that this is the minimum RAM for generating this TB? The needed RAM in kqpkrn ending is for my program 164 MB, 212 MB, 1179 MB for method 1,2,3 respectively, but method 1 is terribly slow. If you dispose of 8.1 GB RAM any preceding comparison with the times obtained by me is unreliable.

Thank you Martin for the Nalimov source. But I have errors in compiling with gcc or g++ as follows:

[guido@localhost tbgen]$ g++ -O3 -o tbgen tbgen.cpp
In file included from tbgen.cpp:1143:
tbindex.cpp:18:1: warning: "_LARGEFILE_SOURCE" redefined
In file included from /usr/include/string.h:26,
from tbgen.cpp:12:
/usr/include/features.h:220:1: warning: this is the location of the previous definition
In file included from tbgen.cpp:1143:
tbindex.cpp:4465: warning: `TB_CRC_CHECK' initialized and declared `extern'
tbgen.cpp: In static member function `static void BIT_algorithm<TB_T>::BIT_VCreateTable(int, int)':
tbgen.cpp:7792: internal compiler error: in uses_template_parms, at cp/pt.c:4860
Please submit a full bug report,
with preprocessed source if appropriate.
See <URL:http://bugzilla.redhat.com/bugzilla> for instructions.
Preprocessed source stored into /tmp/ccxMGBOi.out file, please attach this to your bugreport.

Ciao
Guido
Nulla dies sine linea
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

Some answers for guido

Post by guyhaw »

Eugene Nalimov sent me his kppkpp.tbs file, August 2005. He did not share the actual EGT files with anyone. I also have .tbs files for 3 other unpublished EGTs - see above in this CCRL thread. There are 16 unpublished 3-3p EGTs, the only EGTs missing from the 'wanted' 3-6-man set (as 5-1(p) is of little interest).
Marc B has computed all 3-3p EGTs with FEG, and his FEG.lof stats exactly match Nalimov's .tbs-stats where the latter exist. So, all tests possible at this stage indicate that EN's EGTs will be ok when they appear.
Further, MB is converting his FEG EGTs to Nalimov format. The results are satisfying the first obvious check - that the .tbs stats match up for the two of 'the four' EGTs (mentioned above) done so far.

Nalimov's code does repeated passes through all positions, seeing if it can now give a depth to those positions. So depths are 'backed up' in a retrograde way, but the outer algorithm is not retrograde itself.
Thompson, Wirth and now Konoval simply back-up the positions which have just been given a depth. This is a 'fully retrograde' algorithm and clearly quicker when the endgame is essentially drawn.
Yes, I think I/O performance and disc-use is a big determinant of runtime-performance. Having multiple drives, and maybe doing (de)compression (after)before transfer seems a good idea.
Also, if you have a DTM EGT and want to create an EGT to another metric, I guess you can use your first EGT to advantage to speed things up. That's an idea for the future though.
g
User avatar
Martin Kreuzer
Posts: 53
Joined: Thu Jun 08, 2006 9:02 am
Sign-up code: 0
Contact:

tbgen compiling

Post by Martin Kreuzer »

Dear Guido,

since I start a 3-week vacation tomorrow, let me reply briefly.
1) 8.1 MB is the current size (in RAM) of my computation of kqpkrn
There is no swapping going on, as I have 24 GB of RAM.
2) I am using version 3.4.5 of gcc. "Internal compiler error"
is probably not due to a problem with the code, but with the compiler.
My students told me that version 4.x of gcc could be not compatible,
at least there could be additional error messages. The warnings you
get were ignored by me.
3) In my experiments, the option -O3 yielded code about 10% faster than
compiling with -O2. I don't know the reason for this.
4) Thanks for the -fomit-frame-pointer hint. I will definitely try it.
My remark about "optimization" was meant exactly in this way:
I did not try many tricks to produce a faster executable.
My remark was not indended as a suggestion to rework the algorithm.
5) The current version of tbgen requires that the "dependent" egtb
are there in decompressed format. You can use the program
"datacomp" to compress and decompress egtb.

Kind regards, Martin Kreuzer
User avatar
Martin Kreuzer
Posts: 53
Joined: Thu Jun 08, 2006 9:02 am
Sign-up code: 0
Contact:

4 tbs files

Post by Martin Kreuzer »

Dear Guy,

would you be willing to post the four tbs fils of "missing" endgames here? I would be interested to check the result of my computation (if it finishes).

Cordially, Martin Kreuzer
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

Four EN DTM EGTs - with stats but unpublished

Post by guyhaw »

EN's DTM EGT .tbs stats x4 - pasted into one file - hopefully correctly - g
Attachments
EN DTM 3-3p EGT stats x4.txt
(43.09 KiB) Downloaded 322 times
guido
Posts: 42
Joined: Sun Jun 18, 2006 8:55 pm
Sign-up code: 0
Location: Milan, Italy

Re: Some answers for guido

Post by guido »

guyhaw wrote:Eugene Nalimov sent me his kppkpp.tbs file, August 2005. He did not share the actual EGT files with anyone. I also have .tbs files for 3 other unpublished EGTs - see above in this CCRL thread. There are 16 unpublished 3-3p EGTs, the only EGTs missing from the 'wanted' 3-6-man set (as 5-1(p) is of little interest).
Marc B has computed all 3-3p EGTs with FEG, and his FEG.lof stats exactly match Nalimov's .tbs-stats where the latter exist. So, all tests possible at this stage indicate that EN's EGTs will be ok when they appear.
Further, MB is converting his FEG EGTs to Nalimov format. The results are satisfying the first obvious check - that the .tbs stats match up for the two of 'the four' EGTs (mentioned above) done so far.

Nalimov's code does repeated passes through all positions, seeing if it can now give a depth to those positions. So depths are 'backed up' in a retrograde way, but the outer algorithm is not retrograde itself.
Thompson, Wirth and now Konoval simply back-up the positions which have just been given a depth. This is a 'fully retrograde' algorithm and clearly quicker when the endgame is essentially drawn.
Yes, I think I/O performance and disc-use is a big determinant of runtime-performance. Having multiple drives, and maybe doing (de)compression (after)before transfer seems a good idea.
Also, if you have a DTM EGT and want to create an EGT to another metric, I guess you can use your first EGT to advantage to speed things up. That's an idea for the future though.
g
Hi Guy,

if the statistics of EN and FEG are the same, it is very likely that also the TBs contain the same results.
I remember that once I wanted to do a check whit EN TBs, but I found that in ending without pawns my program eliminated some positions that EN tbgen computed. These positions were those in which the kings are both on the main diagonal (a1-h8) and the first piece is in the upper triangle (a1-h8-a8). EN explained that this elimination costed more than the calculation. So I cannot do an exact comparison of .tbs files with my statistics when there are only pieces. In any cases it is very interesting that this has been possible between EN and FEG. There is also the problem
in some endings of the ep capture that not all the generators consider.

About what you say of the algorithm used by EN, I find this strange, because some years ago I tried exactly the same idea but I abandoned it because I didn't find any sensible advantage. Therefore I deduce that his code and his I/O must be very optimized, if he obtains so short times or he found something that then I had not seen.

You are right about the reasons because the retrograde algorithm is faster than other, and this is true not only for draws but in general when the number of cycles is high. Moreover if for instance the white has no win in 50 moves, the black cannot lose in 50 moves, so it is possible to skip the correspondent black cycle. The difficulty in this algorithm is the capture and promotion moves which cannot be obtained by a back move, and for some years I thought that the algorithm was not feasible. But someone had did it and finally I found a solution for my fully
retrograde algorithm.

Once De Koenig said to me that he generated the TBs maintaining them always compressed.
I have an idea for reducing the I/O but it is not very easy to implement. I'll try when I'll have time.

Ciao
Guido
Nulla dies sine linea
guido
Posts: 42
Joined: Sun Jun 18, 2006 8:55 pm
Sign-up code: 0
Location: Milan, Italy

Re: tbgen compiling

Post by guido »

Martin Kreuzer wrote:Dear Guido,

since I start a 3-week vacation tomorrow, let me reply briefly.
1) 8.1 MB is the current size (in RAM) of my computation of kqpkrn
There is no swapping going on, as I have 24 GB of RAM.
2) I am using version 3.4.5 of gcc. "Internal compiler error"
is probably not due to a problem with the code, but with the compiler.
My students told me that version 4.x of gcc could be not compatible,
at least there could be additional error messages. The warnings you
get were ignored by me.
3) In my experiments, the option -O3 yielded code about 10% faster than
compiling with -O2. I don't know the reason for this.
4) Thanks for the -fomit-frame-pointer hint. I will definitely try it.
My remark about "optimization" was meant exactly in this way:
I did not try many tricks to produce a faster executable.
My remark was not indended as a suggestion to rework the algorithm.
5) The current version of tbgen requires that the "dependent" egtb
are there in decompressed format. You can use the program
"datacomp" to compress and decompress egtb.

Kind regards, Martin
Kreuzer
Hi Martin,

thank you for your explanations.
1) It is very strange that 8.1 GB are sufficient to avoid swapping in kqpkrn ending.
I found a RAM need of 58 GB, as one of the two kqpkrn files is about 18.7 GB in my indexing scheme and you have to add all the TBs by capture and promotion. Dimensions for EN files can be about 10-15 % lesser. So also 24 GB wouldn't be sufficient. A certain level of swapping seems to me necessary.
2) In fact I use the 3.4.3 20041212 version. This could be the reason of all.
3) My experiments on -O2 and -O3 were made on old gcc versions and therefore are no more reliable. I'll try with -O3 as I did in beginning.
4) With -fomit-frame-pointer I had, if I remember well, a gain of 10 %, but this depends on the functions of the program. Moreover it is probable that the small functions are made automatically inline by the optimization and this can influence the importance of the extra-register.
5) I would try tbgen in generating all 4-men endings, only to check the speed and compare it with my program, so the dependent endings would have been all decompressed.

Best regards
Guido
Nulla dies sine linea
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

Supplementary points on stats ...

Post by guyhaw »

An exchange between a 'Guy' and a 'Guido' must be a rare thing ...
When Pawns feature, and only the a-h symmetry applies, stats from various programs should agree - after (as with FEG.lof stats) dividing by 2, 4, 6 or 12 as necessitated by 'like pieces'. It is difficult to impossible to compare Nalimov and FEG stats for P-less endgames.
Without Pawns, the exact stats depend on the treatment of 'both Kings on a1-h8' positions. Wirth removed all functional-duplicates; Nalimov lives with (a likely) two positions which are functionally equivalent.
The convention has been to include 'en passant' situations, which was perhaps why none of Ken Thompson's EGTs featured Ps on both sides, but not to include consideration of castling. There are some studies where castling is done in the main line of the solution.
With all EGTs, it is possible for a highly visible position, e.g. a maximal, to be unreachable from the initial array. Because I don't have the means to filter out all such positions, the maxDTx figures quoted are prior to any such filtering.
Agreed: it would be a strange error indeed that produced .tbs=.lof stats-agreement against the backcloth of an incorrect EGT.
Eugene has said that his one and only programming goal for the EGT-generator has been performance. He can push the limits on the compiler because he writes compilers. I am inexpert in his code and can't comment further.
You are incorrect to say that "if White has no win in m moves, Black cannot lose in m moves": Black might have a forced conversion to loss in another endgame. An example of this extremely rare genre is invited. EGT-generation to the DTM metric requires a minimum-DTM to be calculated first so that all follow-on-endgame positions are backed up: this is not so for the DTC and DTZ metrics. DTC/Z EGTs, besides being smaller than DTM EGTs when compressed, can dispense with subgame-position-inheritance in the first one or two cycles.

EN's index-scheme avoids positions with unavoidable-stm-checks in, and therefore gives smaller index sizes.
For reference, EN's KQPKRN index has 16,048,887,360 wtm and 17,032,882,752 btm positions. At 1bit/position, that's 2GB right there.
g
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

'DTM-isolated' side-to-move losses

Post by guyhaw »

My previous note asked for a position where DTM = m, the side-to-move loses, and there are no subsequent wins with DTM = m.
Answering my own question, a quick perusal of 3-3p stats shows up:

KBNKPP: 1 btm loss in DTM=122 but no wtm wins in DTM=122 (or 121).

I wonder what the position is? Conjecture: quite likely to be a promotion, and because of the extreme depth, an underpromotion.
g
guido
Posts: 42
Joined: Sun Jun 18, 2006 8:55 pm
Sign-up code: 0
Location: Milan, Italy

Re: Supplementary points on stats ...

Post by guido »

guyhaw wrote:An exchange between a 'Guy' and a 'Guido' must be a rare thing ...
When Pawns feature, and only the a-h symmetry applies, stats from various programs should agree - after (as with FEG.lof stats) dividing by 2, 4, 6 or 12 as necessitated by 'like pieces'. It is difficult to impossible to compare Nalimov and FEG stats for P-less endgames.
Without Pawns, the exact stats depend on the treatment of 'both Kings on a1-h8' positions. Wirth removed all functional-duplicates; Nalimov lives with (a likely) two positions which are functionally equivalent.
The convention has been to include 'en passant' situations, which was perhaps why none of Ken Thompson's EGTs featured Ps on both sides, but not to include consideration of castling. There are some studies where castling is done in the main line of the solution.
With all EGTs, it is possible for a highly visible position, e.g. a maximal, to be unreachable from the initial array. Because I don't have the means to filter out all such positions, the maxDTx figures quoted are prior to any such filtering.
Agreed: it would be a strange error indeed that produced .tbs=.lof stats-agreement against the backcloth of an incorrect EGT.
Eugene has said that his one and only programming goal for the EGT-generator has been performance. He can push the limits on the compiler because he writes compilers. I am inexpert in his code and can't comment further.
You are incorrect to say that "if White has no win in m moves, Black cannot lose in m moves": Black might have a forced conversion to loss in another endgame. An example of this extremely rare genre is invited. EGT-generation to the DTM metric requires a minimum-DTM to be calculated first so that all follow-on-endgame positions are backed up: this is not so for the DTC and DTZ metrics. DTC/Z EGTs, besides being smaller than DTM EGTs when compressed, can dispense with subgame-position-inheritance in the first one or two cycles.

EN's index-scheme avoids positions with unavoidable-stm-checks in, and therefore gives smaller index sizes.
For reference, EN's KQPKRN index has 16,048,887,360 wtm and 17,032,882,752 btm positions. At 1bit/position, that's 2GB right there.
g
Hi my dear homonymous,

Also with pawns different programs couldn't always agree, because for example in my program there is an option which excludes some illegal positions because unreachable. The test is clearly rough and not exact, but equally interesting for analyzing such cases and is made doing one back move, so for instance in the simple kpk ending all the positions where the white pawn in the second row gives check to the black king are illegal even if the black has to move. In this specific case I found:
Legal positions without this check: 81664 (white moves) and 84012 (black moves)
Legal positions with this check: 81660 (white moves) and 83622 (black moves)
This option cannot be used with retrograde algorithm because would cause a crash, as it is easily foreseeable.
I don't use normally this option because it costs too much in time and more of the gain obtained avoiding the calculation of these positions.

For endings without pawns my program remove duplicates as Wirth does, while I don't consider ep and castling in the calculation of TBs. ep is considered only when the TBs are interrogated, but it is not always exactly the same, as was demonstrated to me some years ago by Bob Hyatt and others in CCC forum.

I too have difficulty to interpret EN code, but at first sight I imagine it is very efficient.

About my statement: "if White has no win in m moves, Black cannot lose in m moves" you are right but I'm right too!
I was speaking of the fact that I don't have to execute the cycle m of the black and this remains true, because during a cycle I examine only positions of the main TB, not of the derived TBs. The forced conversion of the black has been
computed before. But this is an easy case because the conversion is forced and this position is immediately solvable, while the main difficulty is when I have to compare results obtained by non-forced capture or promotion moves with results obtained from "normal" moves to choose the best result. Excuse me if I have not been clear in my last message.

For checking my program I run all 3 and 4-men TBs with the three different method, obtaining the same identical results (binary files and statistics) with all files in RAM. As a consequence my algorithms should be right in general even if I continue finding errors. There is still only an unknown error in some TBs calculated with the retrograde
algorithm if I use a 3 MB RAM, insufficient to keep the files in memory.

Your last sentence about DTM, DTC and DTZ is not completely clear to me, because I always considered only DTM, and the implications of DTC and DTZ are not known for me. The files of Nalimov are about 10-15% lesser than mine. For both the kqpkrn files the dimension of my (not yet generated) TB are 18,719,406,720 bytes, which is
inside the mentioned limits. But EN seems to paid this reduction with the non-reversibility of the indexing, while from my tables it is possible to extract any position with an assigned results or combined results.

Ciao
Guido

P.S. I can't be useful to you for the question in your new last message at least ... for now
Nulla dies sine linea
Wulff
Posts: 53
Joined: Thu Jan 05, 2006 4:08 pm
Sign-up code: 10159

Making the TB available

Post by Wulff »

Hi!

What is the time frame/plans to release the TB to us hungry TB nuts ??
8)
guyhaw
Posts: 489
Joined: Sat Jan 21, 2006 10:43 am
Sign-up code: 10159
Location: Reading, UK
Contact:

Various ...

Post by guyhaw »

[DW: EN may relay the EGTs via one of our colleagues shortly. Meanwhile, Marc may convert more of 'the 16' to EN form in his own time. Not ETAs. I've collated MB FEG.lof stats with EN.tbs stats with 100% alignment as expected.]

Knocking out 'unreachable' positions completely, afaik, is an unsolved problem. A position may be retro'd any number of plies before it is realised that it is unreachable, e.g. wBa1, wPb2. [It is not now possible to convert a P to a man of the other colour! ] Therefore, any subtractions on this basis are rather arbitrary.

Guido's [not 'homonymous' but sharing a linguistic root] point about "not computing btm loses in m because it was done before" is fair enough, but in the more efficient programs (which only need bitmaps in RAM), DTM wins in m are not backed up until all DTM losses in m are being looked at.
This 'deferral' is not, I imagine, particularly convenient, and the advantage of DTC and DTZ EGT-generation is that all backing up from subgames can be done immediately 'up front'. Nalimov's indexing-scheme is reversible but is relatively slow to compute as there are lots of integer divisions.
Post Reply