CCRL 40/40 Testing Conditions

[Ray]

For information, our testing conditions are as follows:

CCRL 40/40 Testing Conditions

Time control
Our time control is equivalent to 40 moves in 40 minutes on AMD X2 4600+ at 2.4GHz.
We use Crafty 19.17 BH as a benchmark to determine the equivalent time control for particular machine.


CCRL Testing Conditions

Endgame tablebases: 4 or 5 piece tablebases.

Pondering: OFF

Tournament type: Any. Match, Tournament, Gauntlet etc.
Matches or Gauntlets are usually limited to 60 games maximum per engine pair. Our standard match length in 40/40 is 30 games.

Hash size: is set to the same value of either 128 or 256 MB for all engines in a match or tourney. There are three exeptions:
1) "deep" engines running on 2 CPUs should have double hash size, compared to single engines in the same tourney. 2) "deep" engines running on 4 CPUs should have four time hash size
3) If an engine has problems with particular hash size, it may play with smaller hash.

EGTB hash: 32 MB

Tournament Interface: Any. Examples: Winboard, Arena, Shredder, Chessbase, Chess Partner.

Opening book: Any generic. Examples: remis.ctg, draw.ctg, 5moves.ctg, perfect.ctg etc. Book line length is limited to 12 moves per side maximum and book learning is off or the book set as read only. The same books are used for all engines in the match, tournament or gauntlet.

Engines with their own books have them disabled (deleted or switched off in parameters). Engines which can't disable their own book can't participate in CCRL testing.

Book learning: Off for all engines.

Position learning: Off for all engines. Alternatively learning files are set to read-only or deleted after each match.


Choosing engines to test

Engine choice is up to ndividual testers, with just a few limitations:

1. We test only stable versions of chess engines. We generally test only publicly available engines, although private versions and betas are tested solely at our discretion on a case-by-case basis.

Note: If we test a beta version, and that beta version is then released as stable version, and if the engine author says there is no playing strength difference between release and beta, then those games of the beta version are re-named.

2. We try to avoid over-testing any particular pair of engines. The reason is to avoid rating distortions and have more reliable statistics. Normally we test with 30 games for each pair, and with 60 games for interesting pairs, but anything up to 100 is fine.

Tournament format is entirely up to individual testers. We run matches, roundrobin tournaments, gauntlet, swiss or knockout events.

New or old engines can be selected and studying the current rating list will show which engines are currently being tested and which engines may need additional games. Rating differences of estimated +-400 points should be avoided.

[Marc Lacrosse]

For information, our testing conditions are as follows:

CCRL 40/40 Testing Conditions

Time control
Our time control is equivalent to 40 moves in 40 minutes on Athlon 64 3800+ at 2.4GHz, or an AMD X2 4600+ also at 2.4GHz.
We use Crafty 19.17 BH as a benchmark to determine the equivalent time control for particular machine.

Hi Ray

How do I adjust the timings for a given machine according to the results of crafty 19.17 benchmark?
Is there a table or a formula for this ?

In which precise conditions do I do the benchmarking ?

Thanks

Marc

[Kirill Kryukov]

Hi Marc! Here is extract from our conditions (updated version at our internal wiki page).

Time control

We are doing our testing in three time controls: 40/40, 40/12 and 40/4, as measured on Athlon X2 4600+ machine. If you have different machine (actually even if you have the same machine) you need to benchmark it and adjust the time control according to the results. We use Crafty 19.17 BH as a benchmark to determine the equivalent time control for particular machine.

Crafty 19.17 BH benchmark can be downloaded here. (Version 19.17, Brian Hoffman compile, 32-bit, single-CPU). Please note that we should use the same version and compile because different versions may be slightly faster or slower and will give different benchmark time.

How to test: Reboot a machine (not required but preferrable), extract Crafty executable into a separate folder, make sure there are no other files in that folder, run crafty, type 'bench' <enter>, wait a while (don't use computer in that time), when the benchmark ends type 'quit' <enter> (to quit the Crafty). You will then see a new file "log.001" in the Crafty folder. Open that file and find a line "Total elapsed time: 96" near the end. (Your time may be different of course).

Time control for your machine is then computed based on Crafty 19.17 BH benchmark result as following:

CCRL 40/40: T minutes / 40 moves repeated, where T = 40 * <your elapsed seconds> / 48 = <your elapsed seconds> / 1.2
CCRL 40/12: T minutes / 40 moves repeated, where T = 40 * <your elapsed seconds> / 160 = <your elapsed seconds> / 4
CCRL 40/4: T minutes / 40 moves repeated, where T = 40 * <your elapsed seconds> / 480 = <your elapsed seconds> / 12

Example: Your machine runs Crafty 19.17 BH benchmark in 55 seconds and you want to run long time control games (CCRL 40/40). Compute T as 40 * 55 / 48 = 45.833333, rounding to the nearest integer we get T = 46. So, your time control for CCRL 40/40 is 40 moves in 46 minutes.

Note 1. It is totally your choice which of the three time controls to use. Either one is fine and will be good addition to the database and rating list. Just be sure to check the right coordination page, and submit the games to the appropriate update thread. (Best is to add '4040', '4012' and '404' to the file name to help avoiding the mistake).

Note 2. We use repeated time control. It means the, say, in 40/4 the engines have 4 minutes for the first 40 moves. Then they get another 4 minutes for the next 40 moves, and so on.

Note 3. (About benchmark hardware). Initially we used Athlon 64 3800+ as our standard hardware. In January 2007 we changed to Athlon X2 4600+ to better reflect the fact that we do a lot of multi-CPU testing. Those two platforms have the same Crafty benchmark result. Note that the benchmarking it always 32-bit single-CPU.

[Marc Lacrosse]

Hi Marc! Here is extract from our conditions (updated version at our internal wiki page).

Time control

We are doing our testing in three time controls: 40/40, 40/12 and 40/4, as measured on Athlon X2 4600+ machine. If you have different machine (actually even if you have the same machine) you need to benchmark it and adjust the time control according to the results. We use Crafty 19.17 BH as a benchmark to determine the equivalent time control for particular machine.

Crafty 19.17 BH benchmark can be downloaded here. (Version 19.17, Brian Hoffman compile, 32-bit, single-CPU). Please note that we should use the same version and compile because different versions may be slightly faster or slower and will give different benchmark time.

How to test: Reboot a machine (not required but preferrable), extract Crafty executable into a separate folder, make sure there are no other files in that folder, run crafty, type 'bench' <enter>, wait a while (don't use computer in that time), when the benchmark ends type 'quit' <enter> (to quit the Crafty). You will then see a new file "log.001" in the Crafty folder. Open that file and find a line "Total elapsed time: 96" near the end. (Your time may be different of course).

Time control for your machine is then computed based on Crafty 19.17 BH benchmark result as following:

CCRL 40/40: T minutes / 40 moves repeated, where T = 40 * <your elapsed seconds> / 48 = <your elapsed seconds> / 1.2
CCRL 40/12: T minutes / 40 moves repeated, where T = 40 * <your elapsed seconds> / 160 = <your elapsed seconds> / 4
CCRL 40/4: T minutes / 40 moves repeated, where T = 40 * <your elapsed seconds> / 480 = <your elapsed seconds> / 12

Example: Your machine runs Crafty 19.17 BH benchmark in 55 seconds and you want to run long time control games (CCRL 40/40). Compute T as 40 * 55 / 48 = 45.833333, rounding to the nearest integer we get T = 46. So, your time control for CCRL 40/40 is 40 moves in 46 minutes.

Note 1. It is totally your choice which of the three time controls to use. Either one is fine and will be good addition to the database and rating list. Just be sure to check the right coordination page, and submit the games to the appropriate update thread. (Best is to add '4040', '4012' and '404' to the file name to help avoiding the mistake).

Note 2. We use repeated time control. It means the, say, in 40/4 the engines have 4 minutes for the first 40 moves. Then they get another 4 minutes for the next 40 moves, and so on.

Note 3. (About benchmark hardware). Initially we used Athlon 64 3800+ as our standard hardware. In January 2007 we changed to Athlon X2 4600+ to better reflect the fact that we do a lot of multi-CPU testing. Those two platforms have the same Crafty benchmark result. Note that the benchmarking it always 32-bit single-CPU.

Hi Kiril !

Let me just say that I highly praise the extremely high standard of your tests and the excellent quality of your site (I myself am involved in some statistical analysis in my medical research professional activities).

Just a little question : how does your benchmark adjust on multiprocessors PCs ?
When I just type "bench" for crafty without any crafty.rc file : does it use multiple processors if they are present? How does it influence adjustment for engines with/without SMP ability ?

I am not yet quite sure that I will perfectly adjust to your protocol (I do not like "x moves in y time repeating", I do prefer pure Fischer timing; and also I do not like the variability of opening books in your tests) but I do intend to adjust the relative timings on my different PCs according to your kind of benchmarking.

Thanks a lot for all your good work !

You are my favorite ranking list !

Marc

[Kirill Kryukov]

Thanks for kind words, Marc!

Just a little question : how does your benchmark adjust on multiprocessors PCs ?
When I just type "bench" for crafty without any crafty.rc file : does it use multiple processors if they are present? How does it influence adjustment for engines with/without SMP ability ?

The exact Crafty compile I referred to in previous post uses 1 CPU (or core) by default, so it runs the same way on multi-CPU machine as it would on a single-CPU machine with the same CPU. This is why we can make meaningful benchmarking.

This benchmarking we use ensures that single-CPU engines search comparable number of nodes on different machines, no matter 1, 2, or 4 CPUs. When an engine can use multi-CPUs, it becomes its own advantage which is reflected in the rating.

... and also I do not like the variability of opening books in your tests

It is easy to use single book when you are testing alone, but in a large team this will quickly become an issue. Everyone seems to have some preferences. So we now use only two requirements for a book: 1. Opening book must be general, which means not tuned to any particular engine. 2. Opening book must be limited to 12 moves maximum (24 plies). Personally I use 8 moves or shorter books.

[Ray]

It is easy to use single book when you are testing alone, but in a large team this will quickly become an issue. Everyone seems to have some preferences. So we now use only two requirements for a book: 1. Opening book must be general, which means not tuned to any particular engine. 2. Opening book must be limited to 12 moves maximum (24 plies). Personally I use 8 moves or shorter books.

Indeed - and no doubt Marc you've seen the book history page which shows what books we've used over time and which ones are most popular

http://www.computerchess.org.uk/ccrl/40 ... _book.html

.

[M Lacrosse]

(...)
We use Crafty 19.17 BH as a benchmark to determine the equivalent time control for particular machine.
(...)
Open that file and find a line "Total elapsed time: 96" near the end.

Hi Kyril

Is there a list with known "total elapsed time" for different PC architectures ?

Regards

Marc

PS I would like to know which is the fastest presently available monoprocessor architecture for 32 bits engines
Here I have :
HP core duo : TET = 38
Pentium M 2.0 : TET = 50
PIV 3.0 : TET = 68

[Kirill Kryukov]

Is there a list with known "total elapsed time" for different PC architectures ?

Regards

Marc

PS I would like to know which is the fastest presently available monoprocessor architecture for 32 bits engines
Here I have :
HP core duo : TET = 38
Pentium M 2.0 : TET = 50
PIV 3.0 : TET = 68

We maintain an internal list of benchmark results on our machines. The list is unlikely to become public, but we may extract and publish some essense from it (Theoretically at least).

The fastest we have is 28 seconds on overclocked Core 2 Duo. Though sometimes we forget to add the machines to the list so someone may already have a faster one.

[Graham Banks]

Is there a list with known "total elapsed time" for different PC architectures ?

Regards

Marc

PS I would like to know which is the fastest presently available monoprocessor architecture for 32 bits engines
Here I have :
HP core duo : TET = 38
Pentium M 2.0 : TET = 50
PIV 3.0 : TET = 68

We maintain an internal list of benchmark results on our machines. The list is unlikely to become public, but we may extract and publish some essense from it (Theoretically at least).

The fastest we have is 28 seconds on overclocked Core 2 Duo. Though sometimes we forget to add the machines to the list so someone may already have a faster one.

From what I can gather, Marc would be interested to know the hardware that benchmarks at 28 seconds.

[Ray]

Nothing secretive about it - the Crafty benchmark is available for download and anyone can run it

28 seconds was - Core 2 Duo E6600 @ 3330 MHz

[Carl Mascott]

The download link in this thread for Crafty 19.17 BH no longer works.
I haven't been able to find the program elsewhere.
Could someone post the binary or a link?
Thanks!

[Graham Banks]

The download link in this thread for Crafty 19.17 BH no longer works.
I haven't been able to find the program elsewhere.
Could someone post the binary or a link?
Thanks!

viewtopic.php?f=7&t=3608

[Shaun]

Link fixed...

Shaun

[Laimonas123]

Link dead again, please fix it agian, because I have same problem.

[Shaun]

Link dead again, please fix it agian, because I have same problem.

Seems okay???

If you still have problems let us know, but in the meantime Grahams post above gives an alternative download

Shaun

[Kirill Kryukov]

"Note 2. We use repeated time control. It means the, say, in 40/4 the engines have 4 minutes for the first 40 moves. Then they get another 4 minutes for the next 40 moves, and so on."

Just to make this clear for my testing, you repeat the 40 moves/40 minutes *indefinitely* (not a set number of repetitions)? Do you call the game a draw after X moves?

Yes, indefinitely. Although some CCRL members allow a tournament GUI to adjudicate games when both engines evaluate position near 0 for a set number of moves, or when a game hits a drawn EGTB ending. Personally I don't use and don't support using such adjudication, but probably it can save a lot of time with 40/40.

[Ray]

The games of course eventually end with either checkmate, stalemate, 3-fold rep, or 50 move rule. (did I miss any ?)

Some winboard engines have config files where they can be told to resign when they consider the game lost. Otherwise as Kirill suggests, the GUI can be set to adjudicate the game before its natural conclusion

[Graham Banks]

Although some CCRL members allow a tournament GUI to adjudicate games when both engines evaluate position near 0 for a set number of moves, or when a game hits a drawn EGTB ending. Personally I don't use and don't support using such adjudication, but probably it can save a lot of time with 40/40.

Yep.

[Joe McCauley]

Has anyone created an Arena setup guide for CCRL Testing Conditions?

I'm a recent new user of Arena (and chess GUIs/engines in general, though my interest in electronic chess games and PC software goes back nearly 30 years), and I'm having a devil of a time finding some of the settings. In some respects, Arena is not exactly intuitive, the helpfile isn't very helpful, and while the website is nice in some respects, two of the more glaring omissions are an FAQ file and a discussion forum.

Specifics: I think I've got Ponder turned off but I'm not positive all engines are using the setting. I've added an opening book but apparently the engines aren't using it, since some of them still calculate from the first move (and I suspect the ones that don't are using their own books). I can't find any of the hash size settings, nor the book/position learning settings. And I haven't even gotten started yet on endgame tablebases (including where to download them from).

Or would it be simpler to use a different tournament Interface?

FYI, Crafty 19.17 BH returned a value of 35 on my laptop, which translates into 27 minutes for 40/40.

[Graham Banks]

Has anyone created an Arena setup guide for CCRL Testing Conditions?

I'm a recent new user of Arena (and chess GUIs/engines in general, though my interest in electronic chess games and PC software goes back nearly 30 years), and I'm having a devil of a time finding some of the settings. In some respects, Arena is not exactly intuitive, the helpfile isn't very helpful, and while the website is nice in some respects, two of the more glaring omissions are an FAQ file and a discussion forum.

Specifics: I think I've got Ponder turned off but I'm not positive all engines are using the setting. I've added an opening book but apparently the engines aren't using it, since some of them still calculate from the first move (and I suspect the ones that don't are using their own books). I can't find any of the hash size settings, nor the book/position learning settings. And I haven't even gotten started yet on endgame tablebases (including where to download them from).

Or would it be simpler to use a different tournament Interface?

FYI, Crafty 19.17 BH returned a value of 35 on my laptop, which translates into 27 minutes for 40/40.

I haven't used Arena for several years now because ChessGUI suits my testing needs better, but I believe that some are experiencing issues with the latest Arena version.
If you have an older version, you're probably safer using that for now.
From memory, when you've loaded an engine, you need to right click on the engine pane to access its parameters (same as in the Shredder GUI). I'm sure that an Arena user could confirm this?

Cheers,
Graham.

[Ray]

Arena is an excellent GUI, but does have some issues - more frequent time losses than other GUIs, and not giving UCI engines the correct hash as set in common UCI settings

I wouldn't recommend ChessGUI to beginners - even now I have no idea what most of the options in it are for or do.

[Adam Hair]

Has anyone created an Arena setup guide for CCRL Testing Conditions?

I'm a recent new user of Arena (and chess GUIs/engines in general, though my interest in electronic chess games and PC software goes back nearly 30 years), and I'm having a devil of a time finding some of the settings. In some respects, Arena is not exactly intuitive, the helpfile isn't very helpful, and while the website is nice in some respects, two of the more glaring omissions are an FAQ file and a discussion forum.

Specifics: I think I've got Ponder turned off but I'm not positive all engines are using the setting. I've added an opening book but apparently the engines aren't using it, since some of them still calculate from the first move (and I suspect the ones that don't are using their own books). I can't find any of the hash size settings, nor the book/position learning settings. And I haven't even gotten started yet on endgame tablebases (including where to download them from).

Or would it be simpler to use a different tournament Interface?

FYI, Crafty 19.17 BH returned a value of 35 on my laptop, which translates into 27 minutes for 40/40.

Hi Joe,

What version of Arena are you using? I can help you out, but I like to be able to refer to the same version of Arena
as you are using ( in case there are any subtle differences between versions ).

[Joe McCauley]

I'm using Arena 3.0.

Most of the problems I eventually solved. Current status - I think I have ponder off globally and I think I have all the engines using the same opening book, but I don't know what to look for to tell for sure. Got Nalimov 3-4-5 EGTBs installed. Don't know where the settings are for book learning, position learning, and EGTB hash size. Still couldn't get it to use the opening book I wanted it to (a .ctg file - can these be converted to .abk files?) so it's currently using a smaller 8-move book. And I need to figure out how to save the games in .pgn files.

Finally, is there a way to pause/interrupt a tournament when I need to do other stuff on the computer and resume it when I'm done? (I figured out a rather roundabout way to pause it after the current game; that might be marginally acceptable in a 40/4 tournament, but not in a 40/40 tournament.)

[Graham Banks]

I'm using Arena 3.0.

Most of the problems I eventually solved. Current status - I think I have ponder off globally and I think I have all the engines using the same opening book, but I don't know what to look for to tell for sure. Got Nalimov 3-4-5 EGTBs installed. Don't know where the settings are for book learning, position learning, and EGTB hash size. Still couldn't get it to use the opening book I wanted it to (a .ctg file - can these be converted to .abk files?) so it's currently using a smaller 8-move book. And I need to figure out how to save the games in .pgn files.

Finally, is there a way to pause/interrupt a tournament when I need to do other stuff on the computer and resume it when I'm done? (I figured out a rather roundabout way to pause it after the current game; that might be marginally acceptable in a 40/4 tournament, but not in a 40/40 tournament.)

I can answer two of your questions and the rest I'll leave for one of our Arena users.
You can't use books other than abk books with Arena. The ctg books used in the Fritz GUI can't be converted to abk format as far as I'm aware.
You should interrupt a tournament between games. Interrupting in the middle of a game just means that it will start that game again from the beginning when you resume your tournament.

[Adam Hair]

I'm using Arena 3.0.

Most of the problems I eventually solved. Current status - I think I have ponder off globally and I think I have all the engines using the same opening book, but I don't know what to look for to tell for sure. Got Nalimov 3-4-5 EGTBs installed. Don't know where the settings are for book learning, position learning, and EGTB hash size. Still couldn't get it to use the opening book I wanted it to (a .ctg file - can these be converted to .abk files?) so it's currently using a smaller 8-move book. And I need to figure out how to save the games in .pgn files.

Finally, is there a way to pause/interrupt a tournament when I need to do other stuff on the computer and resume it when I'm done? (I figured out a rather roundabout way to pause it after the current game; that might be marginally acceptable in a 40/4 tournament, but not in a 40/40 tournament.)

Joe,

Some of the settings have to be set individually for each engine. For UCI engines, start the UCI engine. Then click
on the "Engines" tab, move the cursor down to "Engine 1" or "Engine 2", then select "Configure". This will bring up
the UCI options (if available) for that engine. Also, if you click on "Engines", then "Manage", then "UCI", you are
able to access the global settings for UCI engines. For example, you can enter the path to the EGTB's and set the
EGTB hash for UCI engines. In Arena 3.0, there is also a tab for WB engines. You can set the global hash size
and EGTB path for winboard engines. But, settings for book learning and position learning are not global.
For UCI engines, you should be able to find them in the UCI options (see above) if they are available. For WB engines,
you must locate the file that contains the engine options ( normally that file ends in .ini, .rc, or possibly some
abbreviation of .config ). That file, if it exists, will be in the individual engine folder. Also for WB engines, it looks like
the EGTB hash must be set individually.

If you have set your main Arena book ("Book", "Manage", "Load") and if each engine is set to use the Arena main book
("Engines", "Manage", "Details", select an engine, "Books", check "use Arena main books with engine") then each
engine should use that book, though a few engines can be problematic. As Graham said, Arena only uses .abk books.
If you really want to use a .ctg book, there is a way. It involves using Aquarium as an adapter. If you are interested,
I will see if I can dig up the details.

When you set up a tournament, you are in the Engine Tournament box. Click on the files tab. Make sure to check the
"Save Games in PGN format" box, and set the folder where you want the PGN file to be saved to.

If I can help out more, let me know. If there is something I don't know, we can ask Gabor.

Adam