Mega 40/4
Mega 40/4
As mentioned in another thread, I am currently running a 75-engine 40/4 round robin tourney, using official CCRL testing protocols. Each round consists of 2,775 games, so it's going to take a while...
The primary motivation behind this is (theoretically) to reveal weaker engines with playing styles that cause problems for the stronger ones. Ordinarily, weaker engines are rarely pitted against the stronger ones; most tourneys only have a few engine entries and (for the most part) are of similar strengths.
I'll post updates here as they become available.
EDIT: Thought I should clarify I'm doing a "best version only" tourney, and am only using free, stable engines. I eventually plan to add even more engines to this tourney...I have another 60 ready to go, and eventually would like to include every "best version" XBoard or UCI engine ever released. So, in many ways, this tourney is a new 40/4 PURE list.
The primary motivation behind this is (theoretically) to reveal weaker engines with playing styles that cause problems for the stronger ones. Ordinarily, weaker engines are rarely pitted against the stronger ones; most tourneys only have a few engine entries and (for the most part) are of similar strengths.
I'll post updates here as they become available.
EDIT: Thought I should clarify I'm doing a "best version only" tourney, and am only using free, stable engines. I eventually plan to add even more engines to this tourney...I have another 60 ready to go, and eventually would like to include every "best version" XBoard or UCI engine ever released. So, in many ways, this tourney is a new 40/4 PURE list.
Last edited by BFG on Sat Sep 06, 2014 11:20 pm, edited 2 times in total.
Re: Mega 40/4
Well, we're 1900 games in, and so far the biggest surprise is that Stockfish 5 is only in 5th place, losing to Houdini 1.5, Strelka(!), Gull, and Komodo 5.1. Of course, the first round isn't finished yet so this could change.
BTW, I would prefer to test the latest versions (Houdini 4, Komodo 7, etc.). But currently I am limited to free engines. If anyone wants to help me out, let me know...
BTW, I would prefer to test the latest versions (Houdini 4, Komodo 7, etc.). But currently I am limited to free engines. If anyone wants to help me out, let me know...
Re: Mega 40/4
Getting close to wrapping things up, if anyone's interested...
- Graham Banks
- Posts: 27018
- Joined: Sun Dec 18, 2005 5:47 pm
- Sign-up code: 0
- Location: Auckland, NZ
Re: Mega 40/4
Big undertaking.BFG wrote:Getting close to wrapping things up, if anyone's interested...
You should post your results in a forum like Talkchess too, as there's a much wider audience there.
Of course there will be people interested.
Re: Mega 40/4
Thanks for the suggestion! I probably will post there when I'm done.
In the meantime, here are the engines that have been disqualified due to instability:
BlackMamba 2.0
Conqueror 1.1b
Eia 0.3
Protej 0.58c
I've also dropped 4 engines out as newer versions were released while the tourney was running - so of the 75 engines I started with, 67 remain.
In the meantime, here are the engines that have been disqualified due to instability:
BlackMamba 2.0
Conqueror 1.1b
Eia 0.3
Protej 0.58c
I've also dropped 4 engines out as newer versions were released while the tourney was running - so of the 75 engines I started with, 67 remain.
Re: Mega 40/4
I have eliminated several engines from the tournament due to instability, or due to newer versions being released. I am now working my way back up to 75.
Here are the 71 engines that have been entered so far. Are there any engines not on this list that any of you would like to see entered?
Rules for entry are:
1. The engine must be the most recent free version available. Only one version per engine.
2. The engine must be capable of running on an older machine - SSE3 instructions or older. 64-bit preferred but 32-bit will be used if unavailable. The engine must come pre-compiled, and have no outside dependencies...I won't enter engines I must compile myself, or (for example) install a C++ library for.
3. The engine must be stable. If it crashes 3 or more times it is out of the tournament.
4. Every engine gets exactly 2 games against every other engine - once on white, and once on black.
5. Standard CCRL 40/4 rules are used, with the Perfect 2010 opening book and Gaviota 3-4-5 EGTBs. Games are replayed if fewer than 28ply.
Adachess 2.0
Adamant 1.7
ALChess1.5b
Andscacs x64
Anechka 0.08
AnMon 5.75
Belka 1.8.20
Bismark 1.4a
Bison 9.11 x64
Booot 5.2.0 x64
Bouquet 1.8 x64
Bright 0.5c
BugChess2 1.9 x64
Cheese 1.61 x64
Cheng 4.0.36c x64
Chess 3x
Cinnamon 1.2b x64
Coiled 0.2b
Counter 1.2
Critter 1.6a x64
DanaSah 5.07
Deuterium 14.2.33.276 x64
Djinn 1.021 x64
Dragon 4.6
Eeyore 1.52 x64
ExCE r167
EXchess 7.31 x64
Fire 3.0 x64
Fizbo 1-2 x64
Frenzee 3.5.19 x64
Fruit 2.3.1
Gaviota 1.0 x64
Gull 3 x64
Hannibal 1.4b x64
Hermann 2.8 x64
Houdini 1.5a x64
Ifrit m1.8
K2 0.57
Kaissa2 1.8a
Komodo 5.1 x64
Macromix 2.01
MadChess 1.4
Marginal 0.1
Maverick 0.51 x64
Murka 3.0 x64
Mustang 4.97
Nejmet 3.07
Nemo 1.01 Beta x64
Numpty DD
Octochess r5190
Patriot 2006
Pharaon 3.51
Protector 1.70 x64
Pulse 1.5.0
Quazar 0.4 x64
Redqueen 1.1.97 x64
Rodent 1.5 x64
Ruffian 1.0.5
Rybka 2.3.2a x64
Senpai 1.0 x64
SmarThink 1.70 x64
Spark 1.0
Spike 1.4
Stockfish 5 x64
Strelka 6.0
Texel 1.04 x64
Ufim 8.02
Uralochka 1.1b
VICE 1.1
WildCat 8.0
Zeus 1.29
Here are the 71 engines that have been entered so far. Are there any engines not on this list that any of you would like to see entered?
Rules for entry are:
1. The engine must be the most recent free version available. Only one version per engine.
2. The engine must be capable of running on an older machine - SSE3 instructions or older. 64-bit preferred but 32-bit will be used if unavailable. The engine must come pre-compiled, and have no outside dependencies...I won't enter engines I must compile myself, or (for example) install a C++ library for.
3. The engine must be stable. If it crashes 3 or more times it is out of the tournament.
4. Every engine gets exactly 2 games against every other engine - once on white, and once on black.
5. Standard CCRL 40/4 rules are used, with the Perfect 2010 opening book and Gaviota 3-4-5 EGTBs. Games are replayed if fewer than 28ply.
Adachess 2.0
Adamant 1.7
ALChess1.5b
Andscacs x64
Anechka 0.08
AnMon 5.75
Belka 1.8.20
Bismark 1.4a
Bison 9.11 x64
Booot 5.2.0 x64
Bouquet 1.8 x64
Bright 0.5c
BugChess2 1.9 x64
Cheese 1.61 x64
Cheng 4.0.36c x64
Chess 3x
Cinnamon 1.2b x64
Coiled 0.2b
Counter 1.2
Critter 1.6a x64
DanaSah 5.07
Deuterium 14.2.33.276 x64
Djinn 1.021 x64
Dragon 4.6
Eeyore 1.52 x64
ExCE r167
EXchess 7.31 x64
Fire 3.0 x64
Fizbo 1-2 x64
Frenzee 3.5.19 x64
Fruit 2.3.1
Gaviota 1.0 x64
Gull 3 x64
Hannibal 1.4b x64
Hermann 2.8 x64
Houdini 1.5a x64
Ifrit m1.8
K2 0.57
Kaissa2 1.8a
Komodo 5.1 x64
Macromix 2.01
MadChess 1.4
Marginal 0.1
Maverick 0.51 x64
Murka 3.0 x64
Mustang 4.97
Nejmet 3.07
Nemo 1.01 Beta x64
Numpty DD
Octochess r5190
Patriot 2006
Pharaon 3.51
Protector 1.70 x64
Pulse 1.5.0
Quazar 0.4 x64
Redqueen 1.1.97 x64
Rodent 1.5 x64
Ruffian 1.0.5
Rybka 2.3.2a x64
Senpai 1.0 x64
SmarThink 1.70 x64
Spark 1.0
Spike 1.4
Stockfish 5 x64
Strelka 6.0
Texel 1.04 x64
Ufim 8.02
Uralochka 1.1b
VICE 1.1
WildCat 8.0
Zeus 1.29
Last edited by BFG on Wed Oct 08, 2014 11:55 am, edited 1 time in total.
- Graham Banks
- Posts: 27018
- Joined: Sun Dec 18, 2005 5:47 pm
- Sign-up code: 0
- Location: Auckland, NZ
Re: Mega 40/4
I wouldn't include Belka or Patriot as they are known clones.
-
- Posts: 13
- Joined: Tue May 20, 2014 5:38 am
- Sign-up code: 10159
Re: Mega 40/4
Dear BFG, it would be very nice if you could add Orion to your tournament !BFG wrote:
Rules for entry are:
1. The engine must be the most recent free version available. Only one version per engine.
2. The engine must be capable of running on an older machine - SSE3 instructions or older. 64-bit preferred but 32-bit will be used if unavailable.
3. The engine must be stable. If it crashes 3 or more times it is out of the tournament.
4. Every engine gets exactly 2 games against every other engine - once on white, and once on black.
5. Standard CCRL 40/4 rules are used, with the Perfect 2010 opening book and Gaviota 3-4-5 EGTBs. Games are replayed if fewer than 28ply.
Thanks
Download Orion :
https://www.orionchess.com/
https://www.orionchess.com/
- Graham Banks
- Posts: 27018
- Joined: Sun Dec 18, 2005 5:47 pm
- Sign-up code: 0
- Location: Auckland, NZ
Re: Mega 40/4
I'm not an expert on sub-2000 ELO engines, but all of the engines in the CCRL 40/40 list are probably okay to test.BFG wrote:Are there any engines not on this list that any of you would like to see entered?
Adam and Gabor would probably be best placed to comment on them though.
Re: Mega 40/4
Thanks for the suggestions! I'll remove Belka and Patriot - I didn't know Belka was a clone, and I'd forgotten Patriot is - and add Orion, Sjakk, Adam, and Gobor. That still leaves 2 positions if anyone wishes to request them.
EDIT: One major omission (in my opinion) is Crafty...I've entered them twice but they've released multiple new versions as I've been running the tournament So I thought I'd wait until development quiets down a little.
EDIT: One major omission (in my opinion) is Crafty...I've entered them twice but they've released multiple new versions as I've been running the tournament So I thought I'd wait until development quiets down a little.
Re: Mega 40/4
As mentioned above, I will remove Belka and Patriot since they are clones. Since editing them out of the PGN can take a while, and it's easier to remove several at the same time, can you tell me if any of the following are also clones? These are the only 3 entered in the tournament that I haven't found in CCRL's listings.
Chess 3x
ExCE r167
Kaissa2 1.8a
Chess 3x
ExCE r167
Kaissa2 1.8a
- Graham Banks
- Posts: 27018
- Joined: Sun Dec 18, 2005 5:47 pm
- Sign-up code: 0
- Location: Auckland, NZ
Re: Mega 40/4
Will need to do some searches.BFG wrote:As mentioned above, I will remove Belka and Patriot since they are clones. Since editing them out of the PGN can take a while, and it's easier to remove several at the same time, can you tell me if any of the following are also clones? These are the only 3 entered in the tournament that I haven't found in CCRL's listings.
Chess 3x
ExCE r167
Kaissa2 1.8a
- Graham Banks
- Posts: 27018
- Joined: Sun Dec 18, 2005 5:47 pm
- Sign-up code: 0
- Location: Auckland, NZ
Re: Mega 40/4
Kaissa = Crafty cloneGraham Banks wrote:Will need to do some searches.BFG wrote:As mentioned above, I will remove Belka and Patriot since they are clones. Since editing them out of the PGN can take a while, and it's easier to remove several at the same time, can you tell me if any of the following are also clones? These are the only 3 entered in the tournament that I haven't found in CCRL's listings.
Chess 3x
ExCE r167
Kaissa2 1.8a
Chess 3x = once commercial?
ExCE r167 = okay
Re: Mega 40/4
Thanks for the research! I'll remove Kaissa as well then. I'm okay with keeping Chess 3x in even though it was commercial at one point.
Re: Mega 40/4
Results so far (68 engines, 4556 games):
- Place Engine Score S-B 01 Stockfish 5 x64 126.0/134 8040.5 02 Houdini 1.5a x64 123.0/134 7611.2 03 Critter 1.6a x64 122.5/134 7596.5 04 Strelka 6.0 121.5/134 7482.0 05 Gull 3 x64 119.5/134 7295.5 06 Bouquet 1.8 x64 119.5/134 7292.0 07 Fire 3.0 x64 118.5/134 7152.2 08 Senpai 1.0 x64 112.5/134 6600.7 09 Protector 1.70 x64 112.5/134 6534.7 10 Komodo 5.1 x64 110.0/134 6409.7 11 Hannibal 1.4b x64 110.0/134 6373.7 12 Texel 1.04 x64 106.5/134 5963.7 13 Rybka 2.3.2a 64 bit 105.0/134 5868.2 14 Spike 1.4 99.5/134 5308.5 15 SmarThink 1.70 x64 99.0/134 5325.0 16 Murka 3.0 x64 99.0/134 5235.5 17 Spark 1.0 95.5/134 5087.5 18 Booot 5.2.0 x64 94.0/134 4904.7 19 Quazar 0.4 x64 93.0/134 4829.2 20 EXchess 7.31 x64 89.0/134 4445.7 21 Gaviota 1.0 x64 87.5/134 4367.2 22 Deuterium 14.2.33.276 x64 86.5/134 4169.5 23 Nemo 1.01 Beta x64 86.5/134 4164.0 24 Cheng 4.0.36c x64 86.0/134 4160.0 25 Fruit 2.3.1 84.5/134 4056.2 26 Octochess r5190 84.5/134 4054.0 27 Bright 0.5c 84.5/134 4039.0 28 Rodent 1.5 x64 84.0/134 4068.0 29 Fizbo 1-2 x64 82.0/134 3797.2 30 Bison 9.11 x64 81.0/134 3872.2 31 Frenzee 3.5.19 x64 80.5/134 3734.5 32 BugChess2 1.9 x64 80.0/134 3735.0 33 Andscacs x64 75.0/134 3287.2 34 WildCat 8.0 72.5/134 3112.5 35 Redqueen 1.1.97 x64 69.0/134 2911.7 36 Djinn 1.021 x64 67.5/134 2722.7 37 Pharaon 3.51 65.0/134 2494.5 38 Ruffian 1.0.5 65.0/134 2474.0 39 Ufim 8.02 62.5/134 2347.2 40 Cheese 1.61 x64 60.0/134 2214.2 41 Hermann 2.8 x64 59.0/134 2207.2 42 DanaSah 5.07 59.0/134 2174.7 43 AnMon 5.75 54.5/134 1786.2 44 Counter 1.2 50.0/134 1834.5 45 Ifrit m1.8 50.0/134 1603.2 46 Dragon 4.6 47.5/134 1373.5 47 Eeyore 1.52 x64 47.0/134 1437.7 48 Nejmet 3.07 45.5/134 1440.0 49 Maverick 0.51 x64 41.0/134 1045.2 50 Zeus 1.29 38.0/134 931.50 51 Uralochka 1.1b 35.5/134 766.25 52 Anechka 0.08 34.0/134 762.50 53 Chess 3x 31.5/134 669.00 54 K2 0.57 29.5/134 659.00 55 MadChess 1.4 29.0/134 532.00 56 Mustang 4.97 27.5/134 613.25 57 VICE 1.1 26.0/134 421.75 58 Cinnamon 1.2b x64 21.0/134 363.50 59 Bismark 1.4a 21.0/134 323.25 60 Macromix 2.01 14.5/134 247.75 61 Coiled 0.2b 14.5/134 169.00 62 Adachess 2.0 13.0/134 162.00 63 ALChess1.5b 12.5/134 256.25 64 Adamant 1.7 11.0/134 131.75 65 Pulse 1.5.0 11.0/134 129.75 66 Marginal 0.1 8.5/134 72.25 67 Numpty DD 4.0/134 49.25 68 ExCE r167 0.5/134 2.00 -- Crafty 24.0 x64 (updated) -- Delphil 3.1 x64 (updated) -- Protector 1.60 x64 (updated) -- Rodent 1.4 x64 (updated) DQ Belka 1.8.20 (clone) DQ BlackMamba 2.0 x64 (unstable) DQ Conqueror 1.1b (unstable) DQ Eia 0.3 (unstable) DQ Kaissa2 1.8a (clone) DQ Patriot 2006 (clone) DQ Protej 0.58c (unstable)
Re: Mega 40/4
I'm trying to decide whether to remove Bismark from the tournament. It has crashed twice - meaning it only has one strike left - and today tried to commit an illegal move against Sjakk. That's the first attempted illegal move of the tournament, and I hadn't considered how to judge that. Incidentally, it was on move 11, which means the game would be tossed out due to CCRL's 14-move minimum.
Thoughts?
Thoughts?
- Graham Banks
- Posts: 27018
- Joined: Sun Dec 18, 2005 5:47 pm
- Sign-up code: 0
- Location: Auckland, NZ
Re: Mega 40/4
I had no joy with Bismark.BFG wrote:I'm trying to decide whether to remove Bismark from the tournament. It has crashed twice - meaning it only has one strike left - and today tried to commit an illegal move against Sjakk. That's the first attempted illegal move of the tournament, and I hadn't considered how to judge that. Incidentally, it was on move 11, which means the game would be tossed out due to CCRL's 14-move minimum.
Thoughts?
Re: Mega 40/4
Sorry, I don't understand what you mean. Did you mean you were unable to get Bismark to work? Or that you did not experience crashes/illegal moves with it? Or, perhaps something else?
- Graham Banks
- Posts: 27018
- Joined: Sun Dec 18, 2005 5:47 pm
- Sign-up code: 0
- Location: Auckland, NZ
Re: Mega 40/4
Couldn't get it to work properly.BFG wrote:Sorry, I don't understand what you mean. Did you mean you were unable to get Bismark to work? Or that you did not experience crashes/illegal moves with it? Or, perhaps something else?
Re: Mega 40/4
Ah, okay. I'm using 1.4a; maybe there were problems with earlier versions? Anyway, other than the two crashes and the one illegal move, it's worked fine for me.
Re: Mega 40/4
I just realized, after trying - and failing - to find a chess engine names Gabor, that you were talking about the people, not engines!
(Incidentally, there IS an engine named Adam.)
(Incidentally, there IS an engine named Adam.)
- Adam Hair
- Posts: 1566
- Joined: Sun May 30, 2010 3:28 am
- Sign-up code: 10159
- Location: Fuquay-Varina, North Carolina, USA
Re: Mega 40/4
Hi, this is Adam the person, not the engineBFG wrote:I just realized, after trying - and failing - to find a chess engine names Gabor, that you were talking about the people, not engines!
(Incidentally, there IS an engine named Adam.)
You could add either Komodo CCT or Komodo 5.1 MP to your tournament. Both are free downloads at http://komodochess.com/ . Komodo CCT is slightly stonger (it is the last version of the original Komodo), but Komodo 5.1 MP is multiprocessor (it is the first release of the rewritten Komodo).
Here are a few more engines you could possibly add:
DiscoCheck 5.2.1 - http://www.open-chess.org/viewtopic.php?f=7&t=2711#wrap (post #8)
iCE 2 v2240 - http://www.fam-petzke.de/cp_download_en.shtml
Dirty 20APR2013 - http://www.dirtychess.com/downloads.php
GreKo 12.1 - http://sourceforge.net/projects/greko/files/
Sjakk 2.2 - http://www.pvv.ntnu.no/~jacobbl/jacob/Sjakk/Sjakk.htm
picoSzachy 4.0 - http://www.kalisz.mm.pl/~pic/nanochess/
SamChess - http://www.3dkingdoms.com/chess/sam/samchess.html
Cmcchess 2.5 - http://people.oregonstate.edu/~machao/C ... index.html
Tarrasch Toy Engine 0.906 - http://www.triplehappy.com/downloads.html
Re: Mega 40/4
Thanks for the suggestions! I'll give 'em a try.
I had intended to use the last "free" version of Komodo...I didn't realize there was a 5.1 MP separate from the regular 5.1. Whoops.
I had intended to use the last "free" version of Komodo...I didn't realize there was a 5.1 MP separate from the regular 5.1. Whoops.