Can't reproduce analysis: CCRL 07/04/08 Toga II 1.4.1SE

Questions and comments related to CCRL testing study
Post Reply
Carl Mascott
Posts: 5
Joined: Sat Jul 12, 2008 3:25 pm

Can't reproduce analysis: CCRL 07/04/08 Toga II 1.4.1SE

Post by Carl Mascott »

I am using the Toga II 1.4.1SE 1-CPU binary from the CCRL website.
I am looking at Game 1.126.816, dated 2008/06/21, downloaded from CCRL: Toga vs Rybka 2.3.2a 64-bit, 0-1.
Move 15 in the actual game is as follows: 15. Rad1 {(a2-a3) +0.45/16 58s} Bc8 {(Ra8-c8) +0.02/15 36s}
My Toga's analysis of White, complete to depth 17, is shown below.
No line of analysis matches the +0.45/16 result in the (commented) game file.

My understanding is that when a chess engine breaks off evaluation at some time t it then uses the last analysis line produced by time t. Is this correct? If not then please correct me. If so then there seems to be a problem here. Any ideas?

Thanks!

10/32 0:01 +0.72 15.a3 Qc7 16.Ng5 Rac8 17.axb4 Nxb4 18.Qd2 a5 19.Nc3 (570.932) 492
11/32 0:02 +0.43 15.a3 Qc7 16.Ng5 bxa3 17.Rxa3 h6 18.Nf3 Nb4 19.Qc3 Qb7 20.e5 (1.084.108) 496
11/32 0:04 +0.48 15.Rae1 Rac8 16.Rd1 Qc7 17.Ned4 Nxd4 18.Bxd4 Nh5 19.Bxg7 Kxg7 20.f5 Nf6 (2.338.889) 495
12/32 0:07 +0.37 15.Rae1 Rac8 16.Rd1 Nb8 17.c4 Nc6 18.Rd2 Kh8 (3.981.191) 497
12/32 0:09 +0.57 15.a3 Qc7 16.f5 gxf5 17.exf5 Na5 18.Ba2 bxa3 19.b4 Nc6 20.b5 Nb4 (4.728.973) 500
13/41 0:16 +0.43 15.a3 Qc7 16.f5 gxf5 17.exf5 Na5 18.Ba2 bxa3 19.b4 Nc6 20.c3 Rab8 21.Kh1 (8.202.839) 502
14/41 0:36 +0.36 15.a3 Qc7 16.axb4 Nxb4 17.Qd2 a5 18.Nc3 Bc6 19.e5 Ne4 20.Nxe4 Bxe4 21.exd6 exd6 22.c3 Nd3 (18.595.072) 508
14/41 0:47 +0.40 15.Rad1 Rac8 16.g4 Nd8 17.a4 Bc6 18.e5 Ne4 19.Ned4 Bd5 20.Bxd5 Qxd5 21.exd6 exd6 (24.111.548) 510
15/41 1:30 +0.35 15.Rad1 Rac8 16.Rd2 Kh8 17.Ng5 Nd8 18.c3 bxc3 19.Nxc3 Ne6 20.Nxe6 Bxe6 21.Bxe6 fxe6 22.Bd4 Bh6 (46.544.534) 515
15/41 1:40 +0.61 15.a3 Qc7 16.axb4 Nxb4 17.Qd2 a5 18.Nc3 Bc6 19.e5 Ne4 20.Nxe4 Bxe4 21.c3 Nd3 22.Ng5 Qb7 23.exd6 exd6 (52.066.807) 516
16/41 2:22 +0.64 15.a3 Qc7 16.axb4 Nxb4 17.Qd2 a5 18.Nc3 Bc6 19.e5 Ne4 20.Nxe4 Bxe4 21.c3 Nd3 22.Ng5 Qc6 23.exd6 exd6 (73.810.616) 518
17/41 3:40 +0.34 15.a3 Qc7 16.e5 Bf5 17.Qa6 dxe5 18.fxe5 Nxe5 19.Nfd4 Be4 20.axb4 Nd5 21.Nb5 Qc6 22.Bc5 Rfd8 23.Qa5 (115.754.724) 525
17/41 5:06 +0.37 15.Rad1 Rac8 16.Rf2 Nd8 17.c3 Bc6 18.Nd2 Ne6 19.e5 Nd7 20.Bxe6 fxe6 21.exd6 exd6 22.Qxd6 Qxa2 23.Nd4 Ba4 (162.254.531) 529
best move: Ra1-d1 time: 9:44.422 min n/s: 530.783 CPU 100.0% n/s(1CPU): 530.783 nodes: 310.110.000
Harald Johnsen
Posts: 1
Joined: Sat Dec 15, 2007 9:32 am

Re: Can't reproduce analysis: CCRL 07/04/08 Toga II 1.4.1SE

Post by Harald Johnsen »

You will rarely have the same pv.

The search is influenced by the content of the hash table for move ordering for example, and some lines are cut because the positions stored in the hash table tell that the score can not go below xx or higher then xx. So the search will be a bit different if you use an empty or a full hash table.
If you want an analyse to match more precisely the game pv then what you can try is to analyse the previous 2 or 3 white moves to the same depth of the game pv. Now your hash table will be filled with something that is not so far from what was used during the game and you can try to analyse the position you are interested in.

Steps :
- clear hash table or restart engine
- go to move 10 (position after black move), push analyze and stop when needed
- go to move 12, analyze then stop
- go to move 14, analyze white move #15 then stop

HJ.
Carl Mascott
Posts: 5
Joined: Sat Jul 12, 2008 3:25 pm

Re: Can't reproduce analysis: CCRL 07/04/08 Toga II 1.4.1SE

Post by Carl Mascott »

You are correct about the state of the hash table affecting the analysis. I didn't know that. Thanks for the explanation. It was a real eye-opener for me that one can't just jump in and analyze move n and get the same result that the engine got in the course of a game. In this case, at move 15, not only are the PVs different: the best move at the same depth is often different.

The move in question is only the third non-book move in the game, so I just let Toga play (as White) the game starting at the first non-book move. I entered Black's moves manually. I set the time per move high enough so that Toga would analyze to the depth shown in the CCRL game and, when necessary, I stopped the analysis manually as soon as the engine started examining depth n+1. Move 13 (first non-book move) was the first move analyzed by the engine after a fresh start.

It seems that one problem remains: the PVs for moves 13 - 15 still don't match. I'm no longer sure that the PVs for moves 14 - 15 should match, but shouldn't the PV for the very first move match?

Results below.

[Event "CCRL 40/40"]
[Site "CCRL"]
[Date "2008.06.21"]
[Round "1.126.816"]
[White "Toga II 1.4.1SE"]
[Black "Rybka 2.3.2a 64-bit"]
[Result "0-1"]

12. Nf3 {+0.00/0 0s} b5 {Both last book move +0.00/0 0s}
13. Qd3 {+0.37/16 97s} b4 {(Nc6-b4) +0.15/16 71s}
14. Ne2 {+0.47/17 59s} Qa5 {+0.11/16 87s}
15. Rad1 {(a2-a3) +0.45/16 58s} Bc8 {(Ra8-c8) +0.02/15 36s}

CPU0: GenuineIntel x86 Family 6 Model 13 Stepping 8 1728 MHz
GUI: Tablebases with 4 pieces found! [Cache: 32 MB + internal 1.39 MB]
GUI: Shredderbases with 0 pieces found (0 MB)
Engine: Toga II 1.4.1SE [EGBB] (128 MB)
by Thomas Gaksch and Fabien Letouzey
11/30 0:01 +0.45 13.a3 Rac8 14.Qd3 a6 15.Ba2 b4 16.Ne2 bxa3 17.Ng3 Qa5 18.Bc4 Nb4 (838.133) 537
12/30 0:02 +0.44 13.a3 Rac8 14.Qd3 a6 15.Bb6 Bh6 16.Nd4 e6 17.Nxc6 Bxc6 (1.351.120) 530
13/32 0:06 +0.40 13.a3 Rac8 14.Qd3 Na5 15.e5 Nxb3 16.exf6 Nc5 17.fxg7 Nxd3 18.gxf8Q+ Kxf8 19.cxd3 Qf5 20.d4 Qc2 (3.507.041) 536
14/34 0:16 +0.45 13.a3 a6 14.Qd3 Na5 15.Ba2 Nc4 16.Bxc4 bxc4 17.Qxc4 Rfc8 18.Qd3 Rab8 19.Rab1 Qa5 20.f5 gxf5 (9.048.149) 546
15/34 0:35 +0.45 13.a3 a6 14.Qe2 Rac8 15.Rad1 Be6 16.Bxe6 fxe6 17.Qd3 Kh8 18.Ng5 e5 19.Ne6 Rf7 20.fxe5 Nxe5 (19.940.893) 555
16/38 1:18 +0.27 13.a3 a5 14.Qd3 a4 15.Bd5 Rab8 16.Bxc6 Bxc6 17.Nd4 Bd7 18.f5 Qh4 19.fxg6 fxg6 (43.861.765) 555
16/40 1:54 +0.45 13.Qd3 a5 14.a4 bxa4 15.Bxa4 Nb4 16.Qd2 Bxa4 17.Rxa4 d5 18.exd5 Nfxd5 19.Nxd5 Qxd5 20.Qxd5 Nxd5 21.Bd4 Bxd4+ 22.Nxd4 Rfd8 23.Re1 (63.426.438) 553
best move: Qd1-d3 time: 2:11.797 min n/s: 558.481 CPU 100.0% n/s(1CPU): 558.481 nodes: 73.510.000

13/37 0:01 +0.52 14.Ne2 Qa5 15.Rac1 Rac8 16.Rcd1 Kh8 17.Ng5 Nd8 18.Nd4 h6 19.e5 hxg5 20.exf6 Bxf6 21.fxg5 Bxg5 (765.553) 492
14/37 0:08 +0.58 14.Ne2 Qa5 15.Ng5 Qc7 16.a3 bxa3 17.Rxa3 h6 18.Nf3 Nb4 19.Qd2 Rfb8 20.Nc3 (4.444.725) 510
15/39 0:25 +0.42 14.Ne2 Qa5 15.a3 Qc7 16.Ng3 Na5 17.e5 Nxb3 18.exf6 Nc5 19.fxg7 Nxd3 20.gxf8Q+ Kxf8 21.cxd3 Qc2 22.axb4 Qxd3 23.Bd4 Bc6 (13.115.062) 517
16/39 0:43 +0.44 14.Ne2 Qa5 15.a3 Qc7 16.Ng3 Na5 17.e5 Nxb3 18.exf6 Nc5 19.fxg7 Nxd3 20.gxf8Q+ Kxf8 21.cxd3 Qc2 22.axb4 Qxd3 23.Bd4 Bc6 24.Rfd1 Qc2 (22.971.435) 526
17/39 1:23 +0.61 14.Ne2 Qa5 15.a3 Qc7 16.axb4 Nxb4 17.Qd2 a5 18.Nc3 Bc6 19.e5 Ne4 20.Nxe4 Bxe4 21.c3 Nd3 22.Ng5 Qb7 23.exd6 exd6 (44.975.789) 537
best move: Nc3-e2 time: 1:40.594 min n/s: 540.887 CPU 100.0% n/s(1CPU): 540.887 nodes: 54.410.000

12/31 0:01 +0.49 15.Rad1 Rac8 16.Bc4 Qc7 17.f5 Na5 18.Ba6 Qxc2 19.Bxc8 Qxd3 20.Rxd3 Rxc8 21.Ng3 Rc2 (712.449) 511
13/32 0:03 +0.37 15.Rad1 Rac8 16.Bc4 b3 17.a3 bxc2 18.Qxc2 Qc7 19.Rc1 Kh8 20.f5 gxf5 21.exf5 (1.835.004) 513
13/32 0:07 +0.39 15.Kh1 Qc7 16.Ng3 h5 17.Qd2 a5 18.a4 e6 19.Bc4 (4.018.669) 511
13/32 0:12 +0.61 15.a3 Qc7 16.axb4 Nxb4 17.Qd2 Qb7 18.Nc3 Nxe4 19.Nxe4 Qxe4 20.Ng5 Qb7 21.c3 Nc6 22.Qd5 Rab8 (6.252.835) 512
14/32 0:13 +0.37 15.a3 Qc7 16.axb4 Nxb4 17.Qd2 Qb7 18.Nc3 Nxe4 19.Nxe4 Qxe4 20.Ng5 Qb7 21.c3 Nc6 22.Qd5 Nd8 23.Qc4 (6.917.882) 512
15/34 0:51 +0.39 15.a3 Qc7 16.axb4 Nxb4 17.Qd2 Qb7 18.Nc3 Nxe4 19.Nxe4 Qxe4 20.Ng5 Qb7 21.c3 Nc6 22.Qd5 Nd8 23.Qc4 Ne6 (26.636.621) 520
15/37 1:13 +0.49 15.Rad1 Rac8 16.Bc4 Qc7 17.Ng3 Rcd8 18.Bb5 Bh6 19.Qc4 Qb7 20.a4 Kh8 (38.281.637) 520
16/37 2:00 +0.42 15.Rad1 Rac8 16.Bc4 Nb8 17.e5 Bf5 18.Qb3 Ne4 19.Rd5 Nc5 20.Bxc5 Rxc5 21.Ned4 Be4 22.Rxc5 Qxc5 (62.664.636) 521
best move: Ra1-d1 time: 3:16.421 min n/s: 526.817 CPU 100.0% n/s(1CPU): 526.817 nodes: 103.470.000
User avatar
Shaun
Posts: 6926
Joined: Sat May 13, 2006 3:24 pm
Sign-up code: 10159
Location: Brighton. UK

Re: Can't reproduce analysis: CCRL 07/04/08 Toga II 1.4.1SE

Post by Shaun »

Hi Carl,

sorry only just had a chance to look at this... and Harald thank you for your initial reply.

One thing to add the Haralds information - the hash size will also effect the pv. I can't check now but I suspect that the hash size was 256mb or it could possibly have been 128mb.

Looking at you analysis you matched the move and depth for the 3 moves you checked, it is only the score that is slightly different - that is probably all you can hope for. Certainly beyond the first move from the book you will not be able to recreate PVs exactly.

Shaun
Carl Mascott
Posts: 5
Joined: Sat Jul 12, 2008 3:25 pm

Re: Can't reproduce analysis: CCRL 07/04/08 Toga II 1.4.1SE

Post by Carl Mascott »

Harald and Shaun,

Thanks for your replies.
The only thing that still puzzles me is why the PVs for the first non-book move (move 13) don't match.
In both cases (AFAIK) it was the engine's first analysis after being started.
(The hash table must get cleared at the start of each game, mustn't it?)
Would a difference in hash table size account for the different PVs on the first non-book move?
User avatar
Shaun
Posts: 6926
Joined: Sat May 13, 2006 3:24 pm
Sign-up code: 10159
Location: Brighton. UK

Re: Can't reproduce analysis: CCRL 07/04/08 Toga II 1.4.1SE

Post by Shaun »

Carl Mascott wrote:Would a difference in hash table size account for the different PVs on the first non-book move?
It can, to see this try changing the hash size and I am fairly confident you will see different PV's

Shaun

for example

1.e4 c5 2.Nf3 Nc6 3.d4 cxd4 4.Nxd4 e5 5.Nb5 d6 6.N1c3 Nf6 7.Nd5 Nxd5 8.exd5 Ne7 9.c4 Nf5 10.Bd3 Be7 11.O-O O-O 12.Qc2

analysed with Toga II 1.4.1SE

64mb
info multipv 1 depth 12 seldepth 23 score cp 20 time 1262 nodes 686389 pv g7g6 c2b3 f8e8 a2a3 e7h4 d3f5 c8f5 b3b4 h4e7 c1e3 d8d7 e3a7 info multipv 1 depth 13 seldepth 29 score cp 17 time 5678 nodes 2963422 pv g7g6 f2f4 a7a6 b5c3 f7f6 c2f2 g8h8 c1e3 f5e3 f2e3 c8g4 f4e5 f6e5 f1f8 d8f8 a1f1 info multipv 1 depth 14 seldepth 33 score cp 8 time 14030 nodes 6771093 pv g7g6 f2f4 a7a6 b5c3 b7b5 c4b5 a6b5 f4e5 b5b4 e5d6 d8d6 c3e4 d6d5 c1d2 c8b7 a2a4 b4a3 info multipv 1 depth 15 seldepth 39 score cp 9 time 22021 nodes 10888449 pv g7g6 f2f4 a7a6 b5c3 b7b5 g1h1 b5c4 d3c4 f5d4 c2f2 d8c7 c4d3 c8f5 d3f5 d4f5 f4e5 d6e5 c1e3 info multipv 1 depth 16 seldepth 39 score cp 12 time 42371 nodes 21179534 pv g7g6 g1h1 a7a6 b5c3 f5d4 c2d1 h7h5 f2f4 c8g4 d1e1 e7h4 e1e3 h4f6 f4e5 f6e5 info multipv 1 depth 17 seldepth 39 score cp 0 time 58284 nodes 29125278 pv g7g6 g1h1 a7a6 b5c3 f5d4 c2d1 h7h5 c1e3 d4f5 d1d2 f5e3 f2e3 c8d7 c3e4 f7f5 e4c3 d8c7
128mb
info multipv 1 depth 12 seldepth 23 score cp 20 time 1262 nodes 686389 pv g7g6 c2b3 f8e8 a2a3 e7h4 d3f5 c8f5 b3b4 h4e7 c1e3 d8d7 e3a7 info multipv 1 depth 13 seldepth 29 score cp 17 time 5599 nodes 2963422 pv g7g6 f2f4 a7a6 b5c3 f7f6 c2f2 g8h8 c1e3 f5e3 f2e3 c8g4 f4e5 f6e5 f1f8 d8f8 a1f1 info multipv 1 depth 14 seldepth 33 score cp 8 time 12839 nodes 6770995 pv g7g6 f2f4 a7a6 b5c3 b7b5 c4b5 a6b5 f4e5 b5b4 e5d6 d8d6 c3e4 d6d5 c1d2 c8b7 a2a4 b4a3 info multipv 1 depth 15 seldepth 39 score cp 10 time 21702 nodes 11350646 pv g7g6 f2f4 a7a6 b5c3 f7f6 c2f2 c8d7 c1e3 f5e3 f2e3 d8c7 a1e1 g8h8 e3g3 info multipv 1 depth 16 seldepth 39 score cp 11 time 36223 nodes 19112671 pv g7g6 f2f4 a7a6 b5c3 f7f6 g1h1 d8b6 g2g4 f5e3 c1e3 b6e3 a1e1 e3h3 f4f5 g8h8 c2d2 g6f5 d3f5 c8f5 g4f5 info multipv 1 depth 17 seldepth 39 score cp 1 time 56221 nodes 29842543 pv g7g6 f2f4 a7a6 b5c3 f7f6 g1h1 d8b6 c2e2 f5d4 e2f2 c8d7 c1e3 a8e8 f4f5 g6g5 f1e1
256mb
info multipv 1 depth 12 seldepth 23 score cp 20 time 1422 nodes 686389 pv g7g6 c2b3 f8e8 a2a3 e7h4 d3f5 c8f5 b3b4 h4e7 c1e3 d8d7 e3a7 info multipv 1 depth 13 seldepth 29 score cp 17 time 5898 nodes 2963422 pv g7g6 f2f4 a7a6 b5c3 f7f6 c2f2 g8h8 c1e3 f5e3 f2e3 c8g4 f4e5 f6e5 f1f8 d8f8 a1f1 info multipv 1 depth 14 seldepth 33 score cp 8 time 13289 nodes 6770987 pv g7g6 f2f4 a7a6 b5c3 b7b5 c4b5 a6b5 f4e5 b5b4 e5d6 d8d6 c3e4 d6d5 c1d2 c8b7 a2a4 b4a3 info multipv 1 depth 15 seldepth 39 score cp 10 time 21571 nodes 10894620 pv g7g6 f2f4 a7a6 b5c3 f7f6 c2f2 c8d7 c1e3 f5e3 f2e3 d8c7 a1e1 g8h8 e3g3 info multipv 1 depth 16 seldepth 39 score cp 5 time 35251 nodes 18145286 pv g7g6 f2f4 a7a6 b5c3 b7b5 g1h1 b5c4 d3c4 d8b6 c2d3 c8d7 g2g4 f5d4 f4e5 d6e5 c1h6 d7g4 h6f8 a8f8 d3e4 info multipv 1 depth 17 seldepth 43 score cp 1 time 53537 nodes 27885558 pv g7g6 f2f4 a7a6 b5c3 b7b5 g1h1 b5c4 d3c4 f5d4 c2f2 d8c7 c4d3 c8f5 d3f5 d4f5 c1e3 e7h4 g2g3 f5e3 f2e3 info multipv 1 depth 18 seldepth 47 score cp -11 time 184055 nodes 94034768 pv g7g6 f2f4 a7a6 b5c3 b7b5 g1h1 b5c4 d3c4 a8b8 c2d3 f7f6 c3e4 g8h8 f4e5 f6e5 c4a6 c8a6 d3a6 b8b4 a6d3 f5d4 f1f8 d8f8 info multipv 1 depth 19 seldepth 47 score cp -13 time 310016 nodes 159595701 pv g7g6 f2f4 a7a6 b5c3 b7b5 g1h1 b5c4 d3c4 c8d7 c2d3 d8b6 g2g4 f5d4 f4e5 d6e5 c1h6 d7g4 d3e4 b6b2 e4g4 b2c3 h6f8 a8f8 a1c1 c3d2 c4a6 d2a2

512mb
info multipv 1 depth 12 seldepth 23 score cp 20 time 1472 nodes 686389 pv g7g6 c2b3 f8e8 a2a3 e7h4 d3f5 c8f5 b3b4 h4e7 c1e3 d8d7 e3a7 info multipv 1 depth 13 seldepth 29 score cp 17 time 5838 nodes 2963422 pv g7g6 f2f4 a7a6 b5c3 f7f6 c2f2 g8h8 c1e3 f5e3 f2e3 c8g4 f4e5 f6e5 f1f8 d8f8 a1f1 info multipv 1 depth 14 seldepth 33 score cp 8 time 13008 nodes 6770987 pv g7g6 f2f4 a7a6 b5c3 b7b5 c4b5 a6b5 f4e5 b5b4 e5d6 d8d6 c3e4 d6d5 c1d2 c8b7 a2a4 b4a3 info multipv 1 depth 15 seldepth 39 score cp 10 time 20860 nodes 10894083 pv g7g6 f2f4 a7a6 b5c3 f7f6 c2f2 c8d7 c1e3 f5e3 f2e3 d8c7 a1e1 g8h8 e3g3 info multipv 1 depth 16 seldepth 39 score cp 6 time 31725 nodes 16770779 pv g7g6 f2f4 a7a6 b5c3 f7f6 g1h1 c8d7 g2g4 f5d4 c2g2 d8c8 f4f5 g6f5 g4f5 g8h8 c1e3 d4f5 g2h3 info multipv 1 depth 17 seldepth 39 score cp 0 time 52455 nodes 27802633 pv g7g6 f2f4 a7a6 b5c3 f7f6 g1h1 c8d7 c2e2 d8c8 e2e4 g8h8 f4e5 f6e5 g2g4 f5d4 c1e3 c8c5 info multipv 1 depth 18 seldepth 43 score cp 1 time 118520 nodes 62823192 pv g7g6 f2f4 a7a6 b5c3 b7b5 g1h1 b5c4 d3c4 f5d4 c2f2 d8c7 c4d3 c8f5 d3f5 d4f5 c1e3 e7h4 g2g3 f5e3 f2e3 info multipv 1 depth 19 seldepth 50 score cp -6 time 212345 nodes 112721777 pv g7g6 f2f4 a7a6 b5c3 b7b5 g1h1 b5c4 d3c4 f5d4 c2f2 d8c7 c4d3 c8f5 d3f5 d4f5 f4e5 d6e5 g2g4 e7h4 f2g2 f5d4 c1h6
the pvs start the same and then begin to differ.
Carl Mascott
Posts: 5
Joined: Sat Jul 12, 2008 3:25 pm

Re: Can't reproduce analysis: CCRL 07/04/08 Toga II 1.4.1SE

Post by Carl Mascott »

Thanks, Shaun!
I am puzzled no more.
Post Reply