Mate in 15, How Long Does It Take To Solve?

[Ed Trice]

This mate in 15 was solved in 7:16 on my 3.9 GHz Intel Core i7-860 (the slowest computer I have). I am wondering, how long does it take to solve on other systems?

[dcorbit]

7r/kpr1b1pn/p3P3/q1p1N1p1/P1Qp2P1/1R5P/1PP1R1B1/6K1 w - - bm Rxb7+; c0 "Ed Trice"; dm 15;

[Kirill Kryukov]

Nice time I'm sure, however what I wonder is if this kind of test makes a reliable benchmark. In my experience, multi-threaded engines often show huge variation in time they take to solve the same position, in different runs (same position, same engine unloaded and loaded fresh each time).

If you repeat this test several times, how far are the time values one from another? I.e., sample size and standard deviation would be nice to know (before we invest time in benchmarking our much slower machines). Even small sample of 5 to 10 runs will help a lot.

[Ed Trice]

I think that is true for when you vary the hash table size, but once one thread reports a mate that is found, and updates the other threads, they should all "rush towards the goal" very quickly, unless the losing side finds a postponing move.

I will do a few runs as you suggest and report the times.

[Ed Trice]

Ok, I ran these tests:

1. Computer as is (browser launched, already on, then launched Shredder and loaded Houdini).
2. Restarted computer, cleared the hash table, Shredder would boot with Houdini as the engine.
3. Shut down the computer, let it cool down a few minutes, then repeated everything in test #2.

The results:

1. Saw the first mate @ 5:17, announced mate in 15 @ 9:43
2. Saw the first mate @ 4:58, announced mate in 15 @ 8:02
3. Saw the first mate @ 4:22, announced mate in 15 @ 7:43

So there was some variance depending on the "state of the computer" at the time of the test.

I would not have expected this with the hash table the same, but as you noted, this is probably "not deterministic" since the parallel searchers have slight changes from test to test.

[Kirill Kryukov]

Thanks for checking, Ed. Yeah, too large variation for a useful benchmark, as I expected. A reliable multi-threaded chess benchmark is something I am looknig for a long time. As it appears so far, it's better to use non-chess benchmarks for multi-core benchmarking.

[Ed Trice]

Well, look at the last two times. The difference is only 19 seconds.

I don't see how that is all that great of a difference in time for such a hard position.

I am wondering, how long does it take other computers to solve the mate? 20 minutes? 1 hour? Longer?

I will test it on the i7-2600K I will be building for someone next week. They asked for 5.4 GHz+, we'll see if I can deliver on that request.

[ernest]

So there was some variance depending on the "state of the computer" at the time of the test.

Hi Ed,
I don't understand
If those tests were done with MP, why do you relate the "variance" to the "state of the computer"?
Isn't that the same as saying that the variance is due to running 3 times around the table (indian style) between tests?

[Ed Trice]

Small changes matter.

In test #1, my system had been up and running almost 72 hours. It was hot, no doubt some of the RAM was fragmented, several program were open and had been running, and my browser had at least 15 tabs open. Therefore, as I quit some programs and launched Shredder, the allocation of RAM for the hash table and the program were not in what I would called an "optimized state". I also had to unload the current chess engine and load the Houdini engine, which again will move memory around.

The movement of memory will impact software that uses memory, even if only very slightly.

In test #2, I rebooted, Houdini was the engine of choice that came up (no memory movement for that operation), and I had a fresh RAM allocation.

In test #3, my CPU temps were almost 10 C cooler after the reboot, meaning my overclock was probably 0.02 Ghz = 20 MHz faster.

Each time, performance was slightly better as reported by the shorter time to discover the mate.

[Kirill Kryukov]

I tried to benchmark my Phenom II x4 905e (2.5 GHz), using 2 cores (other cores were busy). I stopped it after one hour, it was already showing mate in 16 at that time. For a benchmark use, I suggest: 1. Something that takes shorter time to compute. 2. Using some non-controversial free engine.

[ernest]

Small changes matter.

Yes, but have you done 3 successive tests, in conditions as identical as possible?
I am sure that you will then see the same large variance!

...simply because this is MP!

[Ed Trice]

The purpose of the test was to try and see if the time of 7:16 was "extraordinary" or not. I have been dealing with superfast systems for so long, my friends have told me that I "should not complain" that it "took so long" to solve that mate in 15.

After all, it's the slowest system I have, and I must admit, I have gotten used to all of this high speed computing (if a computer takes longer than 8 seconds to boot Windows I get impatient).

If other programs could solve it in 15-20 minutes, I would guess it was not that special of a result.

[ernest]

The purpose of the test was to try and see if the time of 7:16 was "extraordinary" or not.

OK Ed, I did the 3 consecutive tests, on the same Houdini 1.01 engine, but 32-bit and on my (lowly) Core 2 Duo @3GHz (2threads), 512 MB hash.
1st try: mate in 15 obtained in 26 minutes, at depth 31
2nd try: only mate in 62 reached after 12 and half hours, depth still only 29
3rd try: only mate in 29 reached after more than 6 hours, depth 30

Quite a good example of MP (basic) non-reproducibility...
And no, there was no thunder and lightning between tests!

New game
7r/kpr1b1pn/p3P3/q1p1N1p1/P1Qp2P1/1R5P/1PP1R1B1/6K1 w - - 0 1
Analysis by Houdini 1.01 w32 2_CPU:
1st try
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Bb7 Ra8 5.Bxa8 d3 6.Qxd3 Kxa5 7.Qxh7 c4 8.Qe4 Bc5+ 9.Kh2 g6 10.Qxg6 Bd6+ 11.Kg2 Be7 12.Qe8 Bf6 13.Rf2 Be5 14.Rf5 Rc5 15.Rxg5 Bxb2 16.Rxc5+ Kb4 17.e7 Ka3 18.Rxc4 Ka2 19.Qd7 a5 20.e8Q Kb1
+- (44.53) Depth: 23/71 00:11:10 2068mN
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Bb7 Ra8 5.Bxa8 d3 6.Qxd3 Kxa5 7.Qxh7 c4 8.Qe4 Bc5+ 9.Kh2 Bd6+ 10.Kg2 Be7 11.Qe5+ Rc5 12.Qxg7 Rc7 13.Bc6 Kb6 14.Bd7 Bd6 15.e7 Kb7 16.Qxg5 Rc5 17.Qxc5 Bxc5 18.e8Q Ka7
+- (#27) Depth: 24/71 00:12:41 2404mN
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Bb7 Ra8 5.Bxa8 d3 6.Qxd3 Kxa5 7.Qxh7 c4 8.Qe4 Bc5+ 9.Kh2 Bd6+ 10.Kg2 Be7 11.Qe5+ Rc5 12.Qxg7 Rc7 13.Bc6 Kb6 14.Bd7 Bd6 15.Qxg5 Rc5 16.Qd8+ Bc7 17.e7 Ka7 18.Qc8 Bd6 19.Qxc5+ Bxc5 20.e8Q Bd6 21.Qe3+ Kb7 22.Qe4+ Kb8 23.Qxc4 a5 24.Qc6 Ka7
+- (#28) Depth: 25/71 00:12:55 2452mN
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Bb7 Ra8 5.Bxa8 d3 6.Qxd3 c4 7.Qxh7 Kxa5 8.Qe4 Bc5+ 9.Kh2 g6 10.Qxg6 Bd6+ 11.Kg2 Re7 12.Qxg5+ Kb6 13.Qd5 Bb4 14.a5+ Ka7 15.Qxc4 Kxa8 16.Qxb4 Rb7 17.Qxb7+ Kxb7 18.e7 Kc6 19.e8Q+ Kd6 20.Qe7+ Kc6 21.Re6+ Kd5 22.Re5+ Kc6 23.Rc5#
+- (#23) Depth: 26/71 00:13:55 2665mN
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Bb7 Ra8 5.Bxa8 d3 6.Qxd3 c4 7.Qxh7 Kxa5 8.Qe4 c3 9.bxc3 Kb6 10.Qe5 Rc8 11.a5+ Ka7 12.Qxg7 Kxa8 13.Qxe7 Rb8 14.Qxg5 Rb1+ 15.Kg2 Rb8 16.e7 Re8 17.Qh5 Rc8 18.e8Q Kb7 19.Re7+ Kb8
+- (#21) Depth: 27/71 00:15:06 2921mN
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Bb7 Ra8 5.Bxa8 d3 6.Qxd3 Kxa5 7.Qc3+ Kb6 8.a5+ Ka7 9.Bg2 Kb8 10.Qb3+ Kc8 11.Qd5 Ra7 12.Qc6+ Kb8 13.Qb6+ Kc8 14.Qxa7 Bd6 15.Qb7+ Kd8 16.Qd7#
+- (#16) Depth: 28/71 00:17:27 3443mN
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Bb7 Ra8 5.Bxa8 d3 6.Qxd3 Kxa5 7.Qc3+ Kb6 8.a5+ Ka7 9.Bg2 Kb8 10.Qb3+ Kc8 11.Qd5 Ra7 12.Qc6+ Kb8 13.Qb6+ Kc8 14.Qxa7 Bd6 15.Qb7+ Kd8 16.Qd7#
+- (#16) Depth: 29/71 00:19:43 3956mN
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Bb7 Ra8 5.Bxa8 Nf6 6.Bb7 Rxb7 7.Qb3+ Kxa5 8.Qxb7 Nd5 9.Qxd5 d3 10.cxd3 Kb6 11.Qd7 Bf6 12.e7 Bd4+ 13.Kg2 g6 14.e8Q c4 15.Qxd4+ Kb7 16.Re7#
+- (#16) Depth: 30/71 00:22:21 4565mN
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Bb7 Ra8 5.Bxa8 Nf6 6.Nc6 Rxc6 7.a5+ Kc7 8.Qb3 c4 9.Qb7+ Kd8 10.Qxc6 d3 11.Qb6+ Ke8 12.Bc6+ Nd7 13.Bxd7+ Kf8 14.Qb8+ Bd8 15.Qxd8#
+- (#15) Depth: 31/71 00:26:33 5545mN 3481 kN/s
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Bb7 Ra8 5.Bxa8 Nf6 6.Nc6 Rxc6 7.a5+ Kc7 8.Qb3 c4 9.Qb7+ Kd8 10.Qxc6 d3 11.Qb6+ Ke8 12.Bc6+ Nd7 13.Bxd7+ Kf8 14.Qb8+ Bd8 15.Qxd8#
+- (#15) Depth: 32/71 00:30:19 6410mN

2nd try
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Nb3 Ka7 5.Nxd4 Rb8 6.Nc6+ Rxc6 7.Bxc6 Nf6 8.Qc3 Rc8 9.Bg2 Rd8 10.Rd2 Rxd2 11.Qxd2 Kb6 12.Qxg5 Kc7 13.Qxg7 Kd6 14.Qf7 Nd7 15.exd7 Kxd7 16.Qd5+ Bd6 17.Qb7+ Kd8 18.Qxa6 Kd7
+- (28.38) Depth: 27/122 10:33:58 124763mN
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Nb3 Ka7 5.Nxd4 Rb8 6.Nc6+ Rxc6 7.Bxc6 Nf6 8.Qc3 Rg8 9.Bg2 Rd8 10.Rd2 Rxd2 11.Qxd2 Kb6 12.Qxg5 Kc7 13.Qxg7 Kd6 14.Qf7 Nd7 15.exd7 Kxd7 16.Qf5+ Kc7 17.Qe4 Bg5 18.Qc6+ Kd8 19.Qd5+ Kc8 20.Qf5+
+- (#55) Depth: 28/122 12:25:29 151969mN
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Nb3 Ka7 5.Nxd4 Rb8 6.Nc6+ Rxc6 7.Bxc6 Nf6 8.Qc3 Rc8 9.Bg2 Rd8 10.Rd2 Rxd2 11.Qxd2 Kb6 12.Qxg5 Kc7 13.Qxg7 Kd6 14.Qf7 Nd7 15.exd7 Kxd7 16.Qf5+ Kc7 17.Qe4 Bh4 18.Qh7+ Kd6 19.Qxh4 Ke6
+- (#62) Depth: 29/122 12:33:47 153908mN 3403 kN/s

3rd try
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Nb3 Nf6 5.Nxd4 Rd8 6.Qb3+ Ka7 7.Nc6+ Rxc6 8.Bxc6 Rb8 9.Qc3 Rb4 10.a5 Kb8 11.Qe5+ Kc8 12.Qxg5 g6 13.c3 Rb8 14.Qxg6 Kc7 15.Qf7 Ng8 16.Be8 c4 17.Qxg8 Bc5+ 18.Kh2 Rc8
+- (35.15) Depth: 27/93 05:18:59 65302mN
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Nb3 Nf6 5.Nxd4 Rd8 6.Qb3+ Ka7 7.Nc6+ Rxc6 8.Bxc6 Rb8 9.Qc3 Rb4 10.a5 Kb8 11.Qe5+ Kc8 12.Qxg5 g6 13.Qxg6 Kc7 14.Qg7 Kd6 15.Rd2+ Rd4 16.Rxd4+ cxd4 17.g5 Ng8 18.Qxg8 Kxc6 19.Qe8+ Kd6 20.Qd7+ Ke5 21.Qxe7
+- (#30) Depth: 28/93 05:54:01 74411mN
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Nb3 Nf6 5.Nxd4 Rd8 6.Qb3+ Ka7 7.Nc6+ Rxc6 8.Bxc6 Rb8 9.Qc3 Rb4 10.a5 Kb8 11.Qe5+ Kc8 12.Qxg5 g6 13.Qxg6 Kc7 14.Qg7 Kd6 15.Rd2+ Rd4 16.Rxd4+ cxd4 17.g5 Ng8 18.Qxg8 Kc5 19.Qf7 Kxc6 20.Qxe7 d3
+- (#32) Depth: 29/93 05:55:16 74708mN
1.Rxb7+ Rxb7 2.Nc6+ Kb6 3.Nxa5 Rc7 4.Nb3 Nf6 5.Nxd4 Rd8 6.Qb3+ Ka7 7.Nc6+ Rxc6 8.Bxc6 Rb8 9.Qc3 Rb4 10.a5 Kb8 11.Qe5+ Kc8 12.Qxg5 g6 13.Qxg6 Kc7 14.Qg7 Kd6 15.Rd2+ Rd4 16.Rxd4+ cxd4 17.g5 Ng8 18.Qxg8 Kxc6 19.Qe8+ Kd6 20.Qd7+ Ke5 21.Qxe7
+- (#29) Depth: 30/93 06:17:19 80117mN 3539 kN/s

[Ed Trice]

The purpose of the test was to try and see if the time of 7:16 was "extraordinary" or not.

OK Ed, I did the 3 consecutive tests, on the same Houdini 1.01 engine, but 32-bit and on my (lowly) Core 2 Duo @3GHz (2threads), 512 MB hash.
1st try: mate in 15 obtained in 26 minutes, at depth 31
2nd try: only mate in 62 reached after 12 and half hours, depth still only 29
3rd try: only mate in 29 reached after more than 6 hours, depth 30

Quite a good example of MP (basic) non-reproducibility...
And no, there was no thunder and lightning between tests!

Wow, that is what I call a huge variation in reported times.

I take it the 26 minutes is a good result then. And mate in 62? That's 123 plies!