CCRL 40/40 Testing Conditions

Questions and comments related to CCRL testing study

Re: CCRL 40/40 Testing Conditions

Postby Gabor Szots » Tue Nov 29, 2011 2:04 pm

Joe McCauley wrote:Finally, is there a way to pause/interrupt a tournament when I need to do other stuff on the computer and resume it when I'm done? (I figured out a rather roundabout way to pause it after the current game; that might be marginally acceptable in a 40/4 tournament, but not in a 40/40 tournament.)


Uh, I have just got to read this so probably you have already solved the issue.

Anyway, here's how it goes. There are very convenient ways to interrupt a tournament. While it is running, select the 'Tournament' tab and click on either 'Last game' or 'Last round'. The effect of the former is that the current game will be finished, then the tournament interrupted. The latter lets the whole round finish and interrupts the tournament only thereafter. The nice thing is that by either method you won't have interrupt a tournament in the middle of a game.
To resume the interrupted tournament you restart Arena, press F9 then the Resume button.

PS. I had problems using the Last round option. As I switched tabs, sometimes Arena forgot to interrupt the tournament. Maybe 3.0 does not have that bug any longer. Never had a problem with Last game, though.

Gabor
User avatar
Gabor Szots
 
Posts: 5775
Joined: Sat Dec 09, 2006 6:30 am
Location: Szentendre, Hungary

Re: CCRL 40/40 Testing Conditions

Postby XulChris » Sat Aug 04, 2012 8:35 pm

Hi:

Quick question. I think the answer is obvious, but I just want to make sure.

When you use the tablebases, the GUI takes over for the engine, correct? Because stockfish does not have tablebase support built-in, while Houdini does and I was wondering if this could affect the ELO rating. If the tablebases are handled by the GUI, then it should not matter.

I checked ChessGUI and it has tablebase support, so I am assuming that the GUI takes over for the engine during the endgame. Is that how it works?

Thanks for clarifying this for me. I'm new to working with chess engines.
XulChris
 
Posts: 12
Joined: Sat Aug 04, 2012 6:05 pm

Re: CCRL 40/40 Testing Conditions

Postby Ray » Sat Aug 04, 2012 8:40 pm

The GUI will adjudicate games as tablebase win or draw, if you configure it that way. You can usually tell the GUI not to do this if you prefer.
Ray
 
Posts: 14506
Joined: Sun Dec 18, 2005 6:33 pm
Location: U.K.

Re: CCRL 40/40 Testing Conditions

Postby Graham Banks » Sat Aug 04, 2012 8:40 pm

XulChris wrote:Hi:

Quick question. I think the answer is obvious, but I just want to make sure.

When you use the tablebases, the GUI takes over for the engine, correct? Because stockfish does not have tablebase support built-in, while Houdini does and I was wondering if this could affect the ELO rating. If the tablebases are handled by the GUI, then it should not matter.

I checked ChessGUI and it has tablebase support, so I am assuming that the GUI takes over for the engine during the endgame. Is that how it works?

Thanks for clarifying this for me. I'm new to working with chess engines.


Engines that can access tablebases do so before there are 5 pieces left on the board. For those engines that can't access tablebases, the GUI will access them only once there are five pieces left on the board.
There has been much posted about the effect of tablebases on engine v engine testing. The general consensus seems to be that they have minimal effect on ratings, possibly 5-10 Elo.
User avatar
Graham Banks
 
Posts: 18191
Joined: Sun Dec 18, 2005 5:47 pm
Location: Auckland, NZ

Re: CCRL 40/40 Testing Conditions

Postby XulChris » Sun Aug 05, 2012 1:10 am

Graham Banks wrote:
XulChris wrote:Hi:

Quick question. I think the answer is obvious, but I just want to make sure.

When you use the tablebases, the GUI takes over for the engine, correct? Because stockfish does not have tablebase support built-in, while Houdini does and I was wondering if this could affect the ELO rating. If the tablebases are handled by the GUI, then it should not matter.

I checked ChessGUI and it has tablebase support, so I am assuming that the GUI takes over for the engine during the endgame. Is that how it works?

Thanks for clarifying this for me. I'm new to working with chess engines.


Engines that can access tablebases do so before there are 5 pieces left on the board. For those engines that can't access tablebases, the GUI will access them only once there are five pieces left on the board.
There has been much posted about the effect of tablebases on engine v engine testing. The general consensus seems to be that they have minimal effect on ratings, possibly 5-10 Elo.


Graham:

Thanks for replying. That is very interesting. I've been doing some research on tablebases and compressing them, and even reducing the size of them down to only contain winning positions leaving out draws and losing positions. With a 32meg EGTB cache, you could possibly load in tablebases which are very similar to a current position in ram during an opponents move and compress them down to only winning position information. Then when calculating positions in a 6-piece end game, every time you reach a capture, you could then look it up in your EGTB cache (if its there) and see if that move is winning or not.

Also, one more quick question. What is the difference between the 40/4 and 40/4 FRC? It wasn't clear to me from the web site.

Thanks in advance.
XulChris
 
Posts: 12
Joined: Sat Aug 04, 2012 6:05 pm

Re: CCRL 40/40 Testing Conditions

Postby Adam Hair » Sun Aug 05, 2012 1:22 am

XulChris wrote:Hi:

Quick question. I think the answer is obvious, but I just want to make sure.

When you use the tablebases, the GUI takes over for the engine, correct? Because stockfish does not have tablebase support built-in, while Houdini does and I was wondering if this could affect the ELO rating. If the tablebases are handled by the GUI, then it should not matter.

I checked ChessGUI and it has tablebase support, so I am assuming that the GUI takes over for the engine during the endgame. Is that how it works?

Thanks for clarifying this for me. I'm new to working with chess engines.


Just to be certain your question was answered:

The GUI does not take over for the engine. If an engine does not support tablebases or is not configured to do so, then it will not use tablebases and is on its own in the endgame. As Ray said, the GUI, if configured to use tablebases, can use the tablebases to adjudicate games. For both engines and GUIs, you have to specify the location of the tablebases. And as Graham said, for engine matches tablebases do not contribute much Elo in most cases. But they can be important when analyzing positions.
User avatar
Adam Hair
 
Posts: 1571
Joined: Sun May 30, 2010 3:28 am
Location: Fuquay-Varina, North Carolina, USA

Re: CCRL 40/40 Testing Conditions

Postby Adam Hair » Sun Aug 05, 2012 1:25 am

XulChris wrote:
Graham Banks wrote:
XulChris wrote:Hi:

Quick question. I think the answer is obvious, but I just want to make sure.

When you use the tablebases, the GUI takes over for the engine, correct? Because stockfish does not have tablebase support built-in, while Houdini does and I was wondering if this could affect the ELO rating. If the tablebases are handled by the GUI, then it should not matter.

I checked ChessGUI and it has tablebase support, so I am assuming that the GUI takes over for the engine during the endgame. Is that how it works?

Thanks for clarifying this for me. I'm new to working with chess engines.


Engines that can access tablebases do so before there are 5 pieces left on the board. For those engines that can't access tablebases, the GUI will access them only once there are five pieces left on the board.
There has been much posted about the effect of tablebases on engine v engine testing. The general consensus seems to be that they have minimal effect on ratings, possibly 5-10 Elo.


Graham:

Thanks for replying. That is very interesting. I've been doing some research on tablebases and compressing them, and even reducing the size of them down to only contain winning positions leaving out draws and losing positions. With a 32meg EGTB cache, you could possibly load in tablebases which are very similar to a current position in ram during an opponents move and compress them down to only winning position information. Then when calculating positions in a 6-piece end game, every time you reach a capture, you could then look it up in your EGTB cache (if its there) and see if that move is winning or not.

Also, one more quick question. What is the difference between the 40/4 and 40/4 FRC? It wasn't clear to me from the web site.

Thanks in advance.


FRC is Fischer Random Chess, also known as Chess960. The pieces on the back row start the game in a randomized position.
User avatar
Adam Hair
 
Posts: 1571
Joined: Sun May 30, 2010 3:28 am
Location: Fuquay-Varina, North Carolina, USA

Re: CCRL 40/40 Testing Conditions

Postby BlueLotus » Tue Nov 12, 2013 7:06 am

I had no idea about the testing conditions and I checked in the website for any testing conditions are available or not, and then realized what they shared on the internet was an old one. I hope the one you have shared here is the updated one.




________________________________________________
Advertising removed - Shaun - Spam ?
BlueLotus
 
Posts: 1
Joined: Mon Nov 11, 2013 11:46 am

Re: CCRL 40/40 Testing Conditions

Postby Dark_wizzie » Fri Dec 27, 2013 10:38 am

Do you figure that having a neutral opening book ups the reliability of the CCRL results? I'm doing some personal 10 + 15 Houdini 4 vs Stockfish tests and I'm thinking of switching over to Perfect2012t for openings.
Dark_wizzie
 
Posts: 3
Joined: Sun Sep 22, 2013 10:32 am

Re: CCRL 40/40 Testing Conditions

Postby Graham Banks » Fri Dec 27, 2013 6:48 pm

Dark_wizzie wrote:Do you figure that having a neutral opening book ups the reliability of the CCRL results? I'm doing some personal 10 + 15 Houdini 4 vs Stockfish tests and I'm thinking of switching over to Perfect2012t for openings.


The reason that we use neutral opening books is because we want to test engine strength without the influence of tailored opening books.

Some engines have much larger opening books than others, some are developed especially to suit the engine in question, others have tiny books and some have no books at all.

By using a neutral book with limited depth, we are taking this factor out of the equation.
Some agree with this approach whereas others don't, arguing that an engine comes as a package.
User avatar
Graham Banks
 
Posts: 18191
Joined: Sun Dec 18, 2005 5:47 pm
Location: Auckland, NZ

Re:

Postby EvgeniyZh » Wed May 28, 2014 2:42 pm

Ray wrote:
Kirill Kryukov wrote:It is easy to use single book when you are testing alone, but in a large team this will quickly become an issue. Everyone seems to have some preferences. So we now use only two requirements for a book: 1. Opening book must be general, which means not tuned to any particular engine. 2. Opening book must be limited to 12 moves maximum (24 plies). Personally I use 8 moves or shorter books.


Indeed - and no doubt Marc you've seen the book history page which shows what books we've used over time and which ones are most popular

http://www.computerchess.org.uk/ccrl/40 ... _book.html

Is it possible to sort books in history by number of uses during last month rather than in alphabetical order? That can help to see which books are used lately the most.

Kirill Kryukov wrote:
M Lacrosse wrote:Is there a list with known "total elapsed time" for different PC architectures ?

Regards

Marc

PS I would like to know which is the fastest presently available monoprocessor architecture for 32 bits engines
Here I have :
HP core duo : TET = 38
Pentium M 2.0 : TET = 50
PIV 3.0 : TET = 68

We maintain an internal list of benchmark results on our machines. The list is unlikely to become public, but we may extract and publish some essense from it (Theoretically at least).

The fastest we have is 28 seconds on overclocked Core 2 Duo. Though sometimes we forget to add the machines to the list so someone may already have a faster one.

Just wondering, if the list still exists, what's the record now?
The winner of the game is the player who makes the next-to-last mistake - Savielly Tartakower
User avatar
EvgeniyZh
 
Posts: 3
Joined: Wed May 28, 2014 4:56 am
Location: Israel

Re: CCRL 40/40 Testing Conditions

Postby RomainGoussault » Sun Oct 16, 2016 12:19 pm

Hi,

I have a quick question concerning opening books, before I submit my engine.

Opening book: Any generic. Examples: remis.ctg, draw.ctg, 5moves.ctg, perfect.ctg etc. Book line length is limited to 12 moves per side maximum and book learning is off or the book set as read only. The same books are used for all engines in the match, tournament or gauntlet.


Does my engine have to probe the opening book or it's handled by the GUI/program used to play the tournaments?

Thanks,
Romain
RomainGoussault
 
Posts: 4
Joined: Sun Oct 16, 2016 11:05 am

Re: CCRL 40/40 Testing Conditions

Postby Graham Banks » Mon Oct 17, 2016 5:54 am

RomainGoussault wrote:Hi,

I have a quick question concerning opening books, before I submit my engine.

Opening book: Any generic. Examples: remis.ctg, draw.ctg, 5moves.ctg, perfect.ctg etc. Book line length is limited to 12 moves per side maximum and book learning is off or the book set as read only. The same books are used for all engines in the match, tournament or gauntlet.


Does my engine have to probe the opening book or it's handled by the GUI/program used to play the tournaments?

Thanks,
Romain


Handled through the GUI.
If your engine has its own opening book, it would be useful to provide the option to disable it.
User avatar
Graham Banks
 
Posts: 18191
Joined: Sun Dec 18, 2005 5:47 pm
Location: Auckland, NZ

Re: CCRL 40/40 Testing Conditions

Postby Ulrich von Hehlen » Mon Oct 24, 2016 1:11 pm

Hi,

are the hardware specifications listed under
http://www.computerchess.org.uk/ccrl/4040/about.html
still used in 2016?

Great stuff available here. There are some real diamonds hidden in these pgns.

Greetings, Ulrich
Ulrich von Hehlen
 
Posts: 1
Joined: Tue Jul 19, 2016 5:54 am

Re: CCRL 40/40 Testing Conditions

Postby Graham Banks » Mon Oct 24, 2016 6:24 pm

Ulrich von Hehlen wrote:Hi,

are the hardware specifications listed under
http://www.computerchess.org.uk/ccrl/4040/about.html
still used in 2016?

Great stuff available here. There are some real diamonds hidden in these pgns.

Greetings, Ulrich


None of us use that particular hardware.
Our computers are just benchmarked according to that.
Our 40/40 is now more equivalent to roughly 40/18 on modern computers.
User avatar
Graham Banks
 
Posts: 18191
Joined: Sun Dec 18, 2005 5:47 pm
Location: Auckland, NZ

Re: CCRL 40/40 Testing Conditions

Postby pistolero » Fri Mar 17, 2017 7:29 am

hello Graham,
In the ranking CCRL40/4 we can notice the engine Chiron 4 released in january hasn't been tested yet with 4 CPU.
is it a oversight or because you(the team) don't have had enough time ?
Best regards,
Alan
pistolero
 
Posts: 5
Joined: Thu Jul 17, 2014 12:25 pm

Re: CCRL 40/40 Testing Conditions

Postby Graham Banks » Fri Mar 17, 2017 7:34 am

pistolero wrote:hello Graham,
In the ranking CCRL40/4 we can notice the engine Chiron 4 released in january hasn't been tested yet with 4 CPU.
is it a oversight or because you(the team) don't have had enough time ?
Best regards,
Alan


Not an oversight. At present I'm the only one doing 40/4 4CPU testing, but I'm helping to get the 40/40 4CPU list in better shape before turning my attention back to the 40/4 4CPU list.
User avatar
Graham Banks
 
Posts: 18191
Joined: Sun Dec 18, 2005 5:47 pm
Location: Auckland, NZ

Previous

Return to CCRL Public

Who is online

Users browsing this forum: Bing [Bot] and 1 guest