New engines?

[BFG]

Hello,
I've watched CCRL for some time, but only recently registered so I could post this topic.

I'm interested in engines that aren't currently on the CCRL lists, to see if any of them are "hidden gems". Right now, I am using CCRL's 40/4 protocol to test 22 new engines - including the Eia engine suggested by someone else - against a strong benchmark (Stockfish 5 x64), weak (Mustang 4.97), and middle of the road (Frenzee 3.5.19 x64). I have disqualified two of the engines, Obender and Powder, because the former lacks proper time control, and the latter crashes when an opponent uses en passant.

What are the requirements for having an engine tested in the CCRL? Besides the stated requirements that the engine be stable and not a "beta", is interest all that is required? Should I post the names and locations of all 20 of these engines so CCRL can have a look?

[BFG]

If anyone's interested, here are the "new" engines I am testing right now.
(If any of these HAVE been tested by CCRL, please let me know. I originally said I had 20 but I only thought to check the 40/40 list...40/4 had several of them already.)

Reference:
HIGH- Stockfish 5 x64 oldCPU --- #1 3369elo
MEDIUM- Frenzee 3.5.19 x64 --- #52 2763elo
LOW- Mustang 4.97 --- #305 1941elo

Engines:
Belka 1.8.20
Chess 3X
EIA 0.3
ExCE 1.67
Gromit 3.0
Kaissa 1.8a
NGplay 9.85 x64
Patriot 2006
Woodpecker 2-x

Disqualified:
??? (may have misidentified, but perhaps Ares? Sent illegal moves repeatedly)
OBender - Not capable of proper time management for 40/4 tournament
Powder v? - Froze when opponent used en passant

[Graham Banks]

You'll probably find most of these engines in our 40/4 list, but only NGplay 9.85 64-bit is in our 40/40 list.

[BFG]

I wasn't able to find any of the shorter list in the 40/4 list...I'll have to look again.

(I originally posted a list of 20+ engines, then realized many of those were in the 40/4. Those have since been removed.)



EDIT: Are you certain NGPlay is on the 40/40 list? I cannot find it, so if it is, I must be blind...is it under a different spelling?

[Graham Banks]

I wasn't able to find any of the shorter list in the 40/4 list...I'll have to look again.

(I originally posted a list of 20+ engines, then realized many of those were in the 40/4. Those have since been removed.)

http://computerchess.org.uk/ccrl/404/cg ... ons_only=1

EDIT: Are you certain NGPlay is on the 40/40 list? I cannot find it, so if it is, I must be blind...is it under a different spelling?

http://computerchess.org.uk/ccrl/4040/c ... _85_64-bit

[BFG]

Aha! I did miss the NG-Play engine due to the dash in the name. Of the others, however, I only saw an older version of Eia (0.2), so I think they truly are new.

Interestingly, both Belka and Patriot are doing well right now. As expected, they both lost to Stockfish, but they beat Mustang and Frenzee in the first round.

[BFG]

Looks like Belka and Patriot are both worth further looks - after Round 2 they are both outscoring all engines except Stockfish. Stay tuned.

[Graham Banks]

Looks like Belka and Patriot are both worth further looks - after Round 2 they are both outscoring all engines except Stockfish. Stay tuned.

I doubt whether any of our testers are interested in those two engines.

[BFG]

Out of curiosity, why? Do those engines have some history I am unaware of? The whole point to the testing I'm doing is to find engines that might have been missed/overlooked by the main testers. If my results will not generate interest amongst the CCRL testers, then there's no reason for me to continue.

Of course, it is the tester's prerogative what engines to test. But Frenzee currently is rated at #52/2763elo in the 40/4, and these two engines are currently beating it...so I don't see why there would be no interest.

EDIT: I should have started by asking how I could assist the CCRL testers find new engines of interest, rather than just jumping in. If you have any suggestions for how I could help ID new engines, please let me know.

[BFG]

I just wanted to follow up on this. I am interested in helping CCRL testers find "best version" engines that have been missed. I thought the best way to do this would be to run 40/4s on unlisted engines and report back the results, but I may be wrong considering Graham doubts "any of our testers are interested".

Is there any way I can help identify new engines for testing that would be of interest? What will I need to do? I have a high-powered work laptop that sits unused 10 months of the year - a perfect candidate for tournaments.

[Graham Banks]

I just wanted to follow up on this. I am interested in helping CCRL testers find "best version" engines that have been missed. I thought the best way to do this would be to run 40/4s on unlisted engines and report back the results, but I may be wrong considering Graham doubts "any of our testers are interested".

Is there any way I can help identify new engines for testing that would be of interest? What will I need to do? I have a high-powered work laptop that sits unused 10 months of the year - a perfect candidate for tournaments.

Although I'm not in a position to pass judgement, both Patriot and Belka have been widely labelled as clones.

There are likely very few engines that have passed under our radar when it comes to 40/4 testing.

Speaking from a purely personal point of view, the best way you could help me would be to run matches involving newly released versions against a couple of suitable opponents to give a rough idea as to how much improvement has been made.
However, that may not interest you?

Graham.

[BFG]

Belka and Patriot are clones? Aww, what a disappointment. I'm not interested in stolen/cloned work either.

While your suggestion wouldn't be my first pick, I'd prefer to do something that would be useful...so, sure! Just let me know what you'd like me to test, and what benchmarks you'd like me to use (or I can come up with my own). One word of warning: my main/personal laptop is a Windows 7 x64 with Intel Core II Duo P8400. This means I can't run AVX, SSE42, POPCNT, or most of the "new instruction set" engine versions.

Oh, and I don't suppose CCRL has a list of prohibited/banned/stolen engines? I'll probably keep looking for engines that have been missed, if there are any, but would like to exclude any from that list or the 40/4.

[BFG]

Hmm, it looks like Belka and Patriot both dropped down while I was at work today anyway. In 232 of 550 games (42%):

1. Stockfish 5 x64, 42.0/42
2. Frenzee 3.5.19 x64, 34.5/42
3. Belka 1.8.20, 33.5/42
4. Patriot 2006, 31.5/43
5. Kaissa 1.8a, 24.5/42
6. Gromit 3.0, 22.0/42
7. Mustang 4.97, 13.0/42
8. Chess 3X, 11.0/42 SV 96.25
9. Eia 0.3, 11.0/42 SV 76.00
10. Woodpecker 2-x, 9.0/43
11. ExCE 1.67, 0.0/42

I should have used a weaker engine than Mustang for my "weak" benchmark

[Graham Banks]

Belka and Patriot are clones? Aww, what a disappointment. I'm not interested in stolen/cloned work either.

While your suggestion wouldn't be my first pick, I'd prefer to do something that would be useful...so, sure! Just let me know what you'd like me to test, and what benchmarks you'd like me to use (or I can come up with my own). One word of warning: my main/personal laptop is a Windows 7 x64 with Intel Core II Duo P8400. This means I can't run AVX, SSE42, POPCNT, or most of the "new instruction set" engine versions.

No problem. Not all my comps use popcount either.

Oh, and I don't suppose CCRL has a list of prohibited/banned/stolen engines? I'll probably keep looking for engines that have been missed, if there are any, but would like to exclude any from that list or the 40/4.

No, as we're not experts in this regard. Best look at the results of similarity tests posted in Talkchess.

[Graham Banks]

By the way, we're always on the lookout for new testers, so if you're interested, send me a private message or email.
No problem if you're not interested.

[BFG]

Message sent!

[BFG]

Final results, if anyone's interested:

1. Stockfish 5 x64, 98.0/100
2. Frenzee 3.5.19 x64, 81.5/100
3. Belka 1.8.20, 81.0/100
4. Patriot 2006, 66.0/100
5. Kaissa 1.8a, 62.5/100
6. Gromit 3.0, 49.5/100
7. Chess 3X, 33.5/100
8. Mustang 4.97, 28.0/100
9. Eia 0.3, 26.0/100
10. Woodpecker 2-x, 24.0/100
11. ExCE 1.67, 0.0/100

[Graham Banks]

Thanks for your efforts. Gromit is an oldie that I'll try to include at some stage.

[BFG]

Sounds good. In the meantime, I'll keep looking for other unlisted engines. And I would be happy to conduct formal CCRL testing per the private message I sent you. I've actually installed over 100 engines so far, but only these 8 appeared to be missing from CCRL.

[Graham Banks]

Sounds good. In the meantime, I'll keep looking for other unlisted engines. And I would be happy to conduct formal CCRL testing per the private message I sent you. I've actually installed over 100 engines so far, but only these 8 appeared to be missing from CCRL.

Hi - I didn't get the private message. Perhaps you could email me.

[BFG]

Strangely, my account has no record of it being sent either. I tried to resend via PM; if you still don't receive it, then I will email you. Thanks!

[Graham Banks]

Have you tried Conqueror 1.1? If so, I'd be keen to know whether or not you had any issues with it?

Graham.

[BFG]

Hi again! I thought I would publicly post our discussion.

I tried the latest build of Conqueror 1.1 versus a lineup of 10 engines representing ELOs 1000-3400. It does not appear ready for CCRL testing as it is unstable - so far it has crashed 5 times out of 47 games.
I'm still not sure exactly what causes the crash, but it appears due either to loss of all major material, or loss of queen plus rooks. Either way, it frequently is unable to finish the game. Oddly, 3 of the 5 crashes were against the ELO 2800 entry (Tornado 4.88 x64)...for some reason those two engines do not get along. Even with these crashes excluded, ELO is coming in around 1700, not the 2100 that SourceForge states.

EDIT: I just noticed that it crashed on rounds 1, 3, and 5 versus Tornado...meaning that it has crashed every time it has played White versus Tornado, but always finished when playing Black. How peculiar.

[Graham Banks]

Appreciate the feedback. I wonder whether GUI adjudication after material deficit -6.00 would avoid these crash losses?

[BFG]

Hmm...perhaps. I used the default -900 centipawns setting in my quick 40/3 gauntlet, and ended up experiencing 13 crashes out of 100 games. The Tornado pattern (crashing on white, not crashing on black) did not continue, and I also saw at least one pawn-only endgame it was able to complete successfully. The opponent in that game was also down to pawns only, so perhaps a significant deficit, or pawn promotion analysis, causes the crash?