CCRL 40/15 Testing Conditions (previously 40/40)

[Gabor Szots]

Finally, is there a way to pause/interrupt a tournament when I need to do other stuff on the computer and resume it when I'm done? (I figured out a rather roundabout way to pause it after the current game; that might be marginally acceptable in a 40/4 tournament, but not in a 40/40 tournament.)

Uh, I have just got to read this so probably you have already solved the issue.

Anyway, here's how it goes. There are very convenient ways to interrupt a tournament. While it is running, select the 'Tournament' tab and click on either 'Last game' or 'Last round'. The effect of the former is that the current game will be finished, then the tournament interrupted. The latter lets the whole round finish and interrupts the tournament only thereafter. The nice thing is that by either method you won't have interrupt a tournament in the middle of a game.
To resume the interrupted tournament you restart Arena, press F9 then the Resume button.

PS. I had problems using the Last round option. As I switched tabs, sometimes Arena forgot to interrupt the tournament. Maybe 3.0 does not have that bug any longer. Never had a problem with Last game, though.

Gabor

[XulChris]

Hi:

Quick question. I think the answer is obvious, but I just want to make sure.

When you use the tablebases, the GUI takes over for the engine, correct? Because stockfish does not have tablebase support built-in, while Houdini does and I was wondering if this could affect the ELO rating. If the tablebases are handled by the GUI, then it should not matter.

I checked ChessGUI and it has tablebase support, so I am assuming that the GUI takes over for the engine during the endgame. Is that how it works?

Thanks for clarifying this for me. I'm new to working with chess engines.

[Ray]

The GUI will adjudicate games as tablebase win or draw, if you configure it that way. You can usually tell the GUI not to do this if you prefer.

[Graham Banks]

Hi:

Quick question. I think the answer is obvious, but I just want to make sure.

When you use the tablebases, the GUI takes over for the engine, correct? Because stockfish does not have tablebase support built-in, while Houdini does and I was wondering if this could affect the ELO rating. If the tablebases are handled by the GUI, then it should not matter.

I checked ChessGUI and it has tablebase support, so I am assuming that the GUI takes over for the engine during the endgame. Is that how it works?

Thanks for clarifying this for me. I'm new to working with chess engines.

Engines that can access tablebases do so before there are 5 pieces left on the board. For those engines that can't access tablebases, the GUI will access them only once there are five pieces left on the board.
There has been much posted about the effect of tablebases on engine v engine testing. The general consensus seems to be that they have minimal effect on ratings, possibly 5-10 Elo.

[XulChris]

Hi:

Quick question. I think the answer is obvious, but I just want to make sure.

When you use the tablebases, the GUI takes over for the engine, correct? Because stockfish does not have tablebase support built-in, while Houdini does and I was wondering if this could affect the ELO rating. If the tablebases are handled by the GUI, then it should not matter.

I checked ChessGUI and it has tablebase support, so I am assuming that the GUI takes over for the engine during the endgame. Is that how it works?

Thanks for clarifying this for me. I'm new to working with chess engines.

Engines that can access tablebases do so before there are 5 pieces left on the board. For those engines that can't access tablebases, the GUI will access them only once there are five pieces left on the board.
There has been much posted about the effect of tablebases on engine v engine testing. The general consensus seems to be that they have minimal effect on ratings, possibly 5-10 Elo.

Graham:

Thanks for replying. That is very interesting. I've been doing some research on tablebases and compressing them, and even reducing the size of them down to only contain winning positions leaving out draws and losing positions. With a 32meg EGTB cache, you could possibly load in tablebases which are very similar to a current position in ram during an opponents move and compress them down to only winning position information. Then when calculating positions in a 6-piece end game, every time you reach a capture, you could then look it up in your EGTB cache (if its there) and see if that move is winning or not.

Also, one more quick question. What is the difference between the 40/4 and 40/4 FRC? It wasn't clear to me from the web site.

Thanks in advance.

[Adam Hair]

Hi:

Quick question. I think the answer is obvious, but I just want to make sure.

When you use the tablebases, the GUI takes over for the engine, correct? Because stockfish does not have tablebase support built-in, while Houdini does and I was wondering if this could affect the ELO rating. If the tablebases are handled by the GUI, then it should not matter.

I checked ChessGUI and it has tablebase support, so I am assuming that the GUI takes over for the engine during the endgame. Is that how it works?

Thanks for clarifying this for me. I'm new to working with chess engines.

Just to be certain your question was answered:

The GUI does not take over for the engine. If an engine does not support tablebases or is not configured to do so, then it will not use tablebases and is on its own in the endgame. As Ray said, the GUI, if configured to use tablebases, can use the tablebases to adjudicate games. For both engines and GUIs, you have to specify the location of the tablebases. And as Graham said, for engine matches tablebases do not contribute much Elo in most cases. But they can be important when analyzing positions.

[Adam Hair]

Hi:

Quick question. I think the answer is obvious, but I just want to make sure.

When you use the tablebases, the GUI takes over for the engine, correct? Because stockfish does not have tablebase support built-in, while Houdini does and I was wondering if this could affect the ELO rating. If the tablebases are handled by the GUI, then it should not matter.

I checked ChessGUI and it has tablebase support, so I am assuming that the GUI takes over for the engine during the endgame. Is that how it works?

Thanks for clarifying this for me. I'm new to working with chess engines.

Engines that can access tablebases do so before there are 5 pieces left on the board. For those engines that can't access tablebases, the GUI will access them only once there are five pieces left on the board.
There has been much posted about the effect of tablebases on engine v engine testing. The general consensus seems to be that they have minimal effect on ratings, possibly 5-10 Elo.

Graham:

Thanks for replying. That is very interesting. I've been doing some research on tablebases and compressing them, and even reducing the size of them down to only contain winning positions leaving out draws and losing positions. With a 32meg EGTB cache, you could possibly load in tablebases which are very similar to a current position in ram during an opponents move and compress them down to only winning position information. Then when calculating positions in a 6-piece end game, every time you reach a capture, you could then look it up in your EGTB cache (if its there) and see if that move is winning or not.

Also, one more quick question. What is the difference between the 40/4 and 40/4 FRC? It wasn't clear to me from the web site.

Thanks in advance.

FRC is Fischer Random Chess, also known as Chess960. The pieces on the back row start the game in a randomized position.

[BlueLotus]

I had no idea about the testing conditions and I checked in the website for any testing conditions are available or not, and then realized what they shared on the internet was an old one. I hope the one you have shared here is the updated one.




________________________________________________
Advertising removed - Shaun - Spam ?

[Dark_wizzie]

Do you figure that having a neutral opening book ups the reliability of the CCRL results? I'm doing some personal 10 + 15 Houdini 4 vs Stockfish tests and I'm thinking of switching over to Perfect2012t for openings.

[Graham Banks]

Do you figure that having a neutral opening book ups the reliability of the CCRL results? I'm doing some personal 10 + 15 Houdini 4 vs Stockfish tests and I'm thinking of switching over to Perfect2012t for openings.

The reason that we use neutral opening books is because we want to test engine strength without the influence of tailored opening books.

Some engines have much larger opening books than others, some are developed especially to suit the engine in question, others have tiny books and some have no books at all.

By using a neutral book with limited depth, we are taking this factor out of the equation.
Some agree with this approach whereas others don't, arguing that an engine comes as a package.

[EvgeniyZh]

It is easy to use single book when you are testing alone, but in a large team this will quickly become an issue. Everyone seems to have some preferences. So we now use only two requirements for a book: 1. Opening book must be general, which means not tuned to any particular engine. 2. Opening book must be limited to 12 moves maximum (24 plies). Personally I use 8 moves or shorter books.

Indeed - and no doubt Marc you've seen the book history page which shows what books we've used over time and which ones are most popular

http://www.computerchess.org.uk/ccrl/40 ... _book.html

Is it possible to sort books in history by number of uses during last month rather than in alphabetical order? That can help to see which books are used lately the most.

Is there a list with known "total elapsed time" for different PC architectures ?

Regards

Marc

PS I would like to know which is the fastest presently available monoprocessor architecture for 32 bits engines
Here I have :
HP core duo : TET = 38
Pentium M 2.0 : TET = 50
PIV 3.0 : TET = 68

We maintain an internal list of benchmark results on our machines. The list is unlikely to become public, but we may extract and publish some essense from it (Theoretically at least).

The fastest we have is 28 seconds on overclocked Core 2 Duo. Though sometimes we forget to add the machines to the list so someone may already have a faster one.

Just wondering, if the list still exists, what's the record now?

[RomainGoussault]

Hi,

I have a quick question concerning opening books, before I submit my engine.

Opening book: Any generic. Examples: remis.ctg, draw.ctg, 5moves.ctg, perfect.ctg etc. Book line length is limited to 12 moves per side maximum and book learning is off or the book set as read only. The same books are used for all engines in the match, tournament or gauntlet.

Does my engine have to probe the opening book or it's handled by the GUI/program used to play the tournaments?

Thanks,
Romain

[Graham Banks]

Hi,

I have a quick question concerning opening books, before I submit my engine.

Opening book: Any generic. Examples: remis.ctg, draw.ctg, 5moves.ctg, perfect.ctg etc. Book line length is limited to 12 moves per side maximum and book learning is off or the book set as read only. The same books are used for all engines in the match, tournament or gauntlet.

Does my engine have to probe the opening book or it's handled by the GUI/program used to play the tournaments?

Thanks,
Romain

Handled through the GUI.
If your engine has its own opening book, it would be useful to provide the option to disable it.

[Ulrich von Hehlen]

Hi,

are the hardware specifications listed under
http://www.computerchess.org.uk/ccrl/4040/about.html
still used in 2016?

Great stuff available here. There are some real diamonds hidden in these pgns.

Greetings, Ulrich

[Graham Banks]

Hi,

are the hardware specifications listed under
http://www.computerchess.org.uk/ccrl/4040/about.html
still used in 2016?

Great stuff available here. There are some real diamonds hidden in these pgns.

Greetings, Ulrich

None of us use that particular hardware.
Our computers are just benchmarked according to that.
Our 40/40 is now more equivalent to roughly 40/18 on modern computers.

[pistolero]

hello Graham,
In the ranking CCRL40/4 we can notice the engine Chiron 4 released in january hasn't been tested yet with 4 CPU.
is it a oversight or because you(the team) don't have had enough time ?
Best regards,
Alan

[Graham Banks]

hello Graham,
In the ranking CCRL40/4 we can notice the engine Chiron 4 released in january hasn't been tested yet with 4 CPU.
is it a oversight or because you(the team) don't have had enough time ?
Best regards,
Alan

Not an oversight. At present I'm the only one doing 40/4 4CPU testing, but I'm helping to get the 40/40 4CPU list in better shape before turning my attention back to the 40/4 4CPU list.

[Redshift]

Leela Chess Zero is an open-source neural-net based chess engine, attempting to replicate what DeepMind achieved with AlphaZero. It has been learning chess from scratch (starting as a random mover) entirely by self-play since late February 2018.

It has currently reached an ELO probably above 2000 and is certainly reaching the stage where it may be interesting for it to compete in CCRL versus standard chess engines. However a stumbling block is the fact that as a neural network it is optimised to run on a GPU rather than CPU. It can run as CPU-only and by using many threads currently can achieve performance comparable with a fast GPU, but this may not continue to be true when the neural network is enlarged in the course of training.

Do you feel it would be possible in the future to incorporate Leela into the CCRL? How would you deal with the hardware calibration issues? Do note that other neural net engines may be along shortly.

[dejan]

Lc0 is already on CCRL, but not in all sections - check the 404 section here: http://ccrl.chessdom.com/ccrl/404/rating_list_all.html

[Ray]

yes it is there quite clearly on the index page

http://ccrl.chessdom.com/ccrl/404/

and on the single CPU list

http://ccrl.chessdom.com/ccrl/404/cgi/c ... librate=no

[dejan]

Not an oversight. At present I'm the only one doing 40/4 4CPU testing, but I'm helping to get the 40/40 4CPU list in better shape before turning my attention back to the 40/4 4CPU list.

Could you then also add Amoeba 3.1 4CPU to the 40/40 list please?

[Graham Banks]

Not an oversight. At present I'm the only one doing 40/4 4CPU testing, but I'm helping to get the 40/40 4CPU list in better shape before turning my attention back to the 40/4 4CPU list.

Could you then also add Amoeba 3.1 4CPU to the 40/40 list please?

I can't get Amoeba to use more than 1 core on my non-popcount octal (2x4CPU Xeon).

[Ray]

I can't get Amoeba to use more than 1 core on my non-popcount octal (2x4CPU Xeon).

That may be a ChessGUI problem ? I saw another bug report about that.

https://github.com/abulmo/amoeba/issues

[dejan]

Amoeba, like for an example Stockfish, supports the "Threads" option. So if Stockfish works in your application, Amoeba should certainly work too.

Amoeba:

Code: Select all

uci
id name Amoeba 3.1.l64p-l
id author Richard Delorme
option name Ponder type check default false
option name Hash type spin default 64 min 1 max 65536
option name Threads type spin default 1 min 1 max 256
option name Affinity type string default 0:0
option name Log type check default false
option name MultiPV type spin default 1 min 1 max 256
option name UCI_AnalyseMode type check default false
uciok

Stockfish:

Code: Select all

uci
id name Stockfish 010220 64 POPCNT
id author T. Romstad, M. Costalba, J. Kiiski, G. Linscott

option name Debug Log File type string default 
option name Contempt type spin default 24 min -100 max 100
option name Analysis Contempt type combo default Both var Off var White var Black var Both
option name Threads type spin default 1 min 1 max 512
option name Hash type spin default 16 min 1 max 131072
option name Clear Hash type button
option name Ponder type check default false
option name MultiPV type spin default 1 min 1 max 500
option name Skill Level type spin default 20 min 0 max 20
option name Move Overhead type spin default 30 min 0 max 5000
option name Minimum Thinking Time type spin default 20 min 0 max 5000
option name Slow Mover type spin default 84 min 10 max 1000
option name nodestime type spin default 0 min 0 max 10000
option name UCI_Chess960 type check default false
option name UCI_AnalyseMode type check default false
option name UCI_LimitStrength type check default false
option name UCI_Elo type spin default 1350 min 1350 max 2850
option name SyzygyPath type string default <empty>
option name SyzygyProbeDepth type spin default 1 min 1 max 100
option name Syzygy50MoveRule type check default true
option name SyzygyProbeLimit type spin default 7 min 0 max 7
uciok

[Graham Banks]

Amoeba, like for an example Stockfish, supports the "Threads" option. So if Stockfish works in your application, Amoeba should certainly work too.

Amoeba:

Code: Select all

uci
id name Amoeba 3.1.l64p-l
id author Richard Delorme
option name Ponder type check default false
option name Hash type spin default 64 min 1 max 65536
option name Threads type spin default 1 min 1 max 256
option name Affinity type string default 0:0
option name Log type check default false
option name MultiPV type spin default 1 min 1 max 256
option name UCI_AnalyseMode type check default false
uciok

Stockfish:

Code: Select all

uci
id name Stockfish 010220 64 POPCNT
id author T. Romstad, M. Costalba, J. Kiiski, G. Linscott

option name Debug Log File type string default 
option name Contempt type spin default 24 min -100 max 100
option name Analysis Contempt type combo default Both var Off var White var Black var Both
option name Threads type spin default 1 min 1 max 512
option name Hash type spin default 16 min 1 max 131072
option name Clear Hash type button
option name Ponder type check default false
option name MultiPV type spin default 1 min 1 max 500
option name Skill Level type spin default 20 min 0 max 20
option name Move Overhead type spin default 30 min 0 max 5000
option name Minimum Thinking Time type spin default 20 min 0 max 5000
option name Slow Mover type spin default 84 min 10 max 1000
option name nodestime type spin default 0 min 0 max 10000
option name UCI_Chess960 type check default false
option name UCI_AnalyseMode type check default false
option name UCI_LimitStrength type check default false
option name UCI_Elo type spin default 1350 min 1350 max 2850
option name SyzygyPath type string default <empty>
option name SyzygyProbeDepth type spin default 1 min 1 max 100
option name Syzygy50MoveRule type check default true
option name SyzygyProbeLimit type spin default 7 min 0 max 7
uciok

Have you tried to run Amoeba 3.1 with 4 cores using ChessGUI, which is what I use for all of my testing?
You set threads to 4, but it still only uses 1.