Mega 40/4

[BFG]

As mentioned in another thread, I am currently running a 75-engine 40/4 round robin tourney, using official CCRL testing protocols. Each round consists of 2,775 games, so it's going to take a while...

The primary motivation behind this is (theoretically) to reveal weaker engines with playing styles that cause problems for the stronger ones. Ordinarily, weaker engines are rarely pitted against the stronger ones; most tourneys only have a few engine entries and (for the most part) are of similar strengths.

I'll post updates here as they become available.

EDIT: Thought I should clarify I'm doing a "best version only" tourney, and am only using free, stable engines. I eventually plan to add even more engines to this tourney...I have another 60 ready to go, and eventually would like to include every "best version" XBoard or UCI engine ever released. So, in many ways, this tourney is a new 40/4 PURE list.

[BFG]

Well, we're 1900 games in, and so far the biggest surprise is that Stockfish 5 is only in 5th place, losing to Houdini 1.5, Strelka(!), Gull, and Komodo 5.1. Of course, the first round isn't finished yet so this could change.

BTW, I would prefer to test the latest versions (Houdini 4, Komodo 7, etc.). But currently I am limited to free engines. If anyone wants to help me out, let me know...

[BFG]

Getting close to wrapping things up, if anyone's interested...

[Graham Banks]

Getting close to wrapping things up, if anyone's interested...

Big undertaking.
You should post your results in a forum like Talkchess too, as there's a much wider audience there.
Of course there will be people interested.

[BFG]

Thanks for the suggestion! I probably will post there when I'm done.

In the meantime, here are the engines that have been disqualified due to instability:
BlackMamba 2.0
Conqueror 1.1b
Eia 0.3
Protej 0.58c

I've also dropped 4 engines out as newer versions were released while the tourney was running - so of the 75 engines I started with, 67 remain.

[BFG]

I have eliminated several engines from the tournament due to instability, or due to newer versions being released. I am now working my way back up to 75.
Here are the 71 engines that have been entered so far. Are there any engines not on this list that any of you would like to see entered?

Rules for entry are:
1. The engine must be the most recent free version available. Only one version per engine.
2. The engine must be capable of running on an older machine - SSE3 instructions or older. 64-bit preferred but 32-bit will be used if unavailable. The engine must come pre-compiled, and have no outside dependencies...I won't enter engines I must compile myself, or (for example) install a C++ library for.
3. The engine must be stable. If it crashes 3 or more times it is out of the tournament.
4. Every engine gets exactly 2 games against every other engine - once on white, and once on black.
5. Standard CCRL 40/4 rules are used, with the Perfect 2010 opening book and Gaviota 3-4-5 EGTBs. Games are replayed if fewer than 28ply.


Adachess 2.0
Adamant 1.7
ALChess1.5b
Andscacs x64
Anechka 0.08
AnMon 5.75
Belka 1.8.20
Bismark 1.4a
Bison 9.11 x64
Booot 5.2.0 x64
Bouquet 1.8 x64
Bright 0.5c
BugChess2 1.9 x64
Cheese 1.61 x64
Cheng 4.0.36c x64
Chess 3x
Cinnamon 1.2b x64
Coiled 0.2b
Counter 1.2
Critter 1.6a x64
DanaSah 5.07
Deuterium 14.2.33.276 x64
Djinn 1.021 x64
Dragon 4.6
Eeyore 1.52 x64
ExCE r167
EXchess 7.31 x64
Fire 3.0 x64
Fizbo 1-2 x64
Frenzee 3.5.19 x64
Fruit 2.3.1
Gaviota 1.0 x64
Gull 3 x64
Hannibal 1.4b x64
Hermann 2.8 x64
Houdini 1.5a x64
Ifrit m1.8
K2 0.57
Kaissa2 1.8a
Komodo 5.1 x64
Macromix 2.01
MadChess 1.4
Marginal 0.1
Maverick 0.51 x64
Murka 3.0 x64
Mustang 4.97
Nejmet 3.07
Nemo 1.01 Beta x64
Numpty DD
Octochess r5190
Patriot 2006
Pharaon 3.51
Protector 1.70 x64
Pulse 1.5.0
Quazar 0.4 x64
Redqueen 1.1.97 x64
Rodent 1.5 x64
Ruffian 1.0.5
Rybka 2.3.2a x64
Senpai 1.0 x64
SmarThink 1.70 x64
Spark 1.0
Spike 1.4
Stockfish 5 x64
Strelka 6.0
Texel 1.04 x64
Ufim 8.02
Uralochka 1.1b
VICE 1.1
WildCat 8.0
Zeus 1.29

[Graham Banks]

I wouldn't include Belka or Patriot as they are known clones.

[David Carteau]

Rules for entry are:
1. The engine must be the most recent free version available. Only one version per engine.
2. The engine must be capable of running on an older machine - SSE3 instructions or older. 64-bit preferred but 32-bit will be used if unavailable.
3. The engine must be stable. If it crashes 3 or more times it is out of the tournament.
4. Every engine gets exactly 2 games against every other engine - once on white, and once on black.
5. Standard CCRL 40/4 rules are used, with the Perfect 2010 opening book and Gaviota 3-4-5 EGTBs. Games are replayed if fewer than 28ply.

Dear BFG, it would be very nice if you could add Orion to your tournament !
Thanks

[Graham Banks]

Are there any engines not on this list that any of you would like to see entered?

I'm not an expert on sub-2000 ELO engines, but all of the engines in the CCRL 40/40 list are probably okay to test.
Adam and Gabor would probably be best placed to comment on them though.

[BFG]

Thanks for the suggestions! I'll remove Belka and Patriot - I didn't know Belka was a clone, and I'd forgotten Patriot is - and add Orion, Sjakk, Adam, and Gobor. That still leaves 2 positions if anyone wishes to request them.

EDIT: One major omission (in my opinion) is Crafty...I've entered them twice but they've released multiple new versions as I've been running the tournament So I thought I'd wait until development quiets down a little.

[BFG]

As mentioned above, I will remove Belka and Patriot since they are clones. Since editing them out of the PGN can take a while, and it's easier to remove several at the same time, can you tell me if any of the following are also clones? These are the only 3 entered in the tournament that I haven't found in CCRL's listings.

Chess 3x
ExCE r167
Kaissa2 1.8a

[Graham Banks]

As mentioned above, I will remove Belka and Patriot since they are clones. Since editing them out of the PGN can take a while, and it's easier to remove several at the same time, can you tell me if any of the following are also clones? These are the only 3 entered in the tournament that I haven't found in CCRL's listings.

Chess 3x
ExCE r167
Kaissa2 1.8a

Will need to do some searches.

[Graham Banks]

As mentioned above, I will remove Belka and Patriot since they are clones. Since editing them out of the PGN can take a while, and it's easier to remove several at the same time, can you tell me if any of the following are also clones? These are the only 3 entered in the tournament that I haven't found in CCRL's listings.

Chess 3x
ExCE r167
Kaissa2 1.8a

Will need to do some searches.

Kaissa = Crafty clone
Chess 3x = once commercial?
ExCE r167 = okay

[BFG]

Thanks for the research! I'll remove Kaissa as well then. I'm okay with keeping Chess 3x in even though it was commercial at one point.

[BFG]

Results so far (68 engines, 4556 games):

• Place Engine Score S-B 01 Stockfish 5 x64 126.0/134 8040.5 02 Houdini 1.5a x64 123.0/134 7611.2 03 Critter 1.6a x64 122.5/134 7596.5 04 Strelka 6.0 121.5/134 7482.0 05 Gull 3 x64 119.5/134 7295.5 06 Bouquet 1.8 x64 119.5/134 7292.0 07 Fire 3.0 x64 118.5/134 7152.2 08 Senpai 1.0 x64 112.5/134 6600.7 09 Protector 1.70 x64 112.5/134 6534.7 10 Komodo 5.1 x64 110.0/134 6409.7 11 Hannibal 1.4b x64 110.0/134 6373.7 12 Texel 1.04 x64 106.5/134 5963.7 13 Rybka 2.3.2a 64 bit 105.0/134 5868.2 14 Spike 1.4 99.5/134 5308.5 15 SmarThink 1.70 x64 99.0/134 5325.0 16 Murka 3.0 x64 99.0/134 5235.5 17 Spark 1.0 95.5/134 5087.5 18 Booot 5.2.0 x64 94.0/134 4904.7 19 Quazar 0.4 x64 93.0/134 4829.2 20 EXchess 7.31 x64 89.0/134 4445.7 21 Gaviota 1.0 x64 87.5/134 4367.2 22 Deuterium 14.2.33.276 x64 86.5/134 4169.5 23 Nemo 1.01 Beta x64 86.5/134 4164.0 24 Cheng 4.0.36c x64 86.0/134 4160.0 25 Fruit 2.3.1 84.5/134 4056.2 26 Octochess r5190 84.5/134 4054.0 27 Bright 0.5c 84.5/134 4039.0 28 Rodent 1.5 x64 84.0/134 4068.0 29 Fizbo 1-2 x64 82.0/134 3797.2 30 Bison 9.11 x64 81.0/134 3872.2 31 Frenzee 3.5.19 x64 80.5/134 3734.5 32 BugChess2 1.9 x64 80.0/134 3735.0 33 Andscacs x64 75.0/134 3287.2 34 WildCat 8.0 72.5/134 3112.5 35 Redqueen 1.1.97 x64 69.0/134 2911.7 36 Djinn 1.021 x64 67.5/134 2722.7 37 Pharaon 3.51 65.0/134 2494.5 38 Ruffian 1.0.5 65.0/134 2474.0 39 Ufim 8.02 62.5/134 2347.2 40 Cheese 1.61 x64 60.0/134 2214.2 41 Hermann 2.8 x64 59.0/134 2207.2 42 DanaSah 5.07 59.0/134 2174.7 43 AnMon 5.75 54.5/134 1786.2 44 Counter 1.2 50.0/134 1834.5 45 Ifrit m1.8 50.0/134 1603.2 46 Dragon 4.6 47.5/134 1373.5 47 Eeyore 1.52 x64 47.0/134 1437.7 48 Nejmet 3.07 45.5/134 1440.0 49 Maverick 0.51 x64 41.0/134 1045.2 50 Zeus 1.29 38.0/134 931.50 51 Uralochka 1.1b 35.5/134 766.25 52 Anechka 0.08 34.0/134 762.50 53 Chess 3x 31.5/134 669.00 54 K2 0.57 29.5/134 659.00 55 MadChess 1.4 29.0/134 532.00 56 Mustang 4.97 27.5/134 613.25 57 VICE 1.1 26.0/134 421.75 58 Cinnamon 1.2b x64 21.0/134 363.50 59 Bismark 1.4a 21.0/134 323.25 60 Macromix 2.01 14.5/134 247.75 61 Coiled 0.2b 14.5/134 169.00 62 Adachess 2.0 13.0/134 162.00 63 ALChess1.5b 12.5/134 256.25 64 Adamant 1.7 11.0/134 131.75 65 Pulse 1.5.0 11.0/134 129.75 66 Marginal 0.1 8.5/134 72.25 67 Numpty DD 4.0/134 49.25 68 ExCE r167 0.5/134 2.00 -- Crafty 24.0 x64 (updated) -- Delphil 3.1 x64 (updated) -- Protector 1.60 x64 (updated) -- Rodent 1.4 x64 (updated) DQ Belka 1.8.20 (clone) DQ BlackMamba 2.0 x64 (unstable) DQ Conqueror 1.1b (unstable) DQ Eia 0.3 (unstable) DQ Kaissa2 1.8a (clone) DQ Patriot 2006 (clone) DQ Protej 0.58c (unstable)

[BFG]

I'm trying to decide whether to remove Bismark from the tournament. It has crashed twice - meaning it only has one strike left - and today tried to commit an illegal move against Sjakk. That's the first attempted illegal move of the tournament, and I hadn't considered how to judge that. Incidentally, it was on move 11, which means the game would be tossed out due to CCRL's 14-move minimum.

Thoughts?

[Graham Banks]

I'm trying to decide whether to remove Bismark from the tournament. It has crashed twice - meaning it only has one strike left - and today tried to commit an illegal move against Sjakk. That's the first attempted illegal move of the tournament, and I hadn't considered how to judge that. Incidentally, it was on move 11, which means the game would be tossed out due to CCRL's 14-move minimum.

Thoughts?

I had no joy with Bismark.

[BFG]

Sorry, I don't understand what you mean. Did you mean you were unable to get Bismark to work? Or that you did not experience crashes/illegal moves with it? Or, perhaps something else?

[Graham Banks]

Sorry, I don't understand what you mean. Did you mean you were unable to get Bismark to work? Or that you did not experience crashes/illegal moves with it? Or, perhaps something else?

Couldn't get it to work properly.

[BFG]

Ah, okay. I'm using 1.4a; maybe there were problems with earlier versions? Anyway, other than the two crashes and the one illegal move, it's worked fine for me.

[BFG]

I just realized, after trying - and failing - to find a chess engine names Gabor, that you were talking about the people, not engines!
(Incidentally, there IS an engine named Adam.)

[Adam Hair]

I just realized, after trying - and failing - to find a chess engine names Gabor, that you were talking about the people, not engines!
(Incidentally, there IS an engine named Adam.)

Hi, this is Adam the person, not the engine

You could add either Komodo CCT or Komodo 5.1 MP to your tournament. Both are free downloads at http://komodochess.com/ . Komodo CCT is slightly stonger (it is the last version of the original Komodo), but Komodo 5.1 MP is multiprocessor (it is the first release of the rewritten Komodo).

Here are a few more engines you could possibly add:

DiscoCheck 5.2.1 - http://www.open-chess.org/viewtopic.php?f=7&t=2711#wrap (post #8)
iCE 2 v2240 - http://www.fam-petzke.de/cp_download_en.shtml
Dirty 20APR2013 - http://www.dirtychess.com/downloads.php
GreKo 12.1 - http://sourceforge.net/projects/greko/files/
Sjakk 2.2 - http://www.pvv.ntnu.no/~jacobbl/jacob/Sjakk/Sjakk.htm
picoSzachy 4.0 - http://www.kalisz.mm.pl/~pic/nanochess/
SamChess - http://www.3dkingdoms.com/chess/sam/samchess.html
Cmcchess 2.5 - http://people.oregonstate.edu/~machao/C ... index.html
Tarrasch Toy Engine 0.906 - http://www.triplehappy.com/downloads.html

[BFG]

Thanks for the suggestions! I'll give 'em a try.
I had intended to use the last "free" version of Komodo...I didn't realize there was a 5.1 MP separate from the regular 5.1. Whoops.