The latest CCRL update (9th December 2017)

Questions and comments related to CCRL testing study

The latest CCRL update (9th December 2017)

Postby Graham Banks » Sat Jan 26, 2008 10:42 pm

The purpose of our rating lists is provide engine authors and enthusiasts with a general comparison of engine strength. We also provide other data that could be of interest.
For various reasons, there may be some engines that do not appear on our lists, therefore it is useful to look at other rating lists as well as ours.
The inclusion or exclusion of engines in our lists should not be taken as our group making a statement about their legality or status.
We test chess engines for our own enjoyment and receive no payment for doing so.


The latest CCRL Rating Lists and Statistics are available for viewing from the following links:
http://computerchess.org.uk/ccrl/4040/ (40/40)
http://www.computerchess.org.uk/ccrl/404/ (40/4)
http://www.computerchess.org.uk/ccrl/404FRC/ (FRC 40/4)

Please note that the three lists are often updated separately to each other. The FRC list is only updated when a new engine or engine version is being/has been tested.

The links to the various rating lists can be found just beneath the default Best Versions list (as in this screenshot). Specific 32-bit rating lists are denoted as such to the right of the default list in each category. The default lists contain the 64-bit engines.

Image

Our 40 moves in 40 minutes repeating and 40 moves in 4 minutes repeating are both adjusted to the AMD64 X2 4600+ (2.4GHz).
This time control is roughly equivalent to 40 moves in 20 minutes repeating or 40 moves in 2 minutes repeating on an Intel i7.

Be aware that in the early stages of testing, an engine's rating can often fluctuate a lot.
It is strongly advised to look at the many other rating lists available in order to get a more accurate overall picture of an engine's rating relative to others.

The LOS (likelihood of superiority) stats to the right hand side of each rating list tell you the likelihood in percentage terms of each engine being superior to the engine directly below them.

All games are available for download by engine or ECO code. The total games database in its entirety is always available.
The current ELO ratings are saved in all game databases for those engines that have 200 games or more.

Clicking on an engine name will give details as to opponents played plus homepage links where applicable.

Custom lists of engines can be selected for comparison.

An openings report page lists the number of games played by ECO codes with draw percentage and White win percentage. Clicking on a column heading will sort the list by that column.
User avatar
Graham Banks
 
Posts: 18191
Joined: Sun Dec 18, 2005 5:47 pm
Location: Auckland, NZ

Re: The latest CCRL update (9th January 2009)

Postby ChessKnight » Fri Jan 23, 2009 6:53 am

Graham Banks

The latest CCRL update


I am highly appreciative and thankful to your efforts for regular updates.

thanks.
ChessKnight
 
Posts: 22
Joined: Fri Jan 02, 2009 12:43 pm

Re: The latest CCRL update (9th January 2009)

Postby Graham Banks » Fri Jan 23, 2009 7:00 am

ChessKnight wrote:Graham Banks

The latest CCRL update


I am highly appreciative and thankful to your efforts for regular updates.

thanks.


Thanks. It's a team effort to do all that we do. Producing an update report every two weeks is just one little way I can help the team. :)

Regards, Graham.
User avatar
Graham Banks
 
Posts: 18191
Joined: Sun Dec 18, 2005 5:47 pm
Location: Auckland, NZ

Re: The latest CCRL update (17th April 2009)

Postby Mindbreaker » Wed Apr 22, 2009 10:26 pm

It would be nice to see a rating for Deep Fritz 11 in the 40/4 category. I love all the stats, results matrices, links to engine homepages, and version comparisons. Great job. I have enjoyed this site for years.
Mindbreaker
 
Posts: 14
Joined: Tue Dec 05, 2006 10:07 pm
Location: San Deigo

Re: The latest CCRL update (17th April 2009)

Postby Graham Banks » Thu Apr 23, 2009 9:39 pm

Mindbreaker wrote:It would be nice to see a rating for Deep Fritz 11 in the 40/4 category. I love all the stats, results matrices, links to engine homepages, and version comparisons. Great job. I have enjoyed this site for years.


Thanks for your kind comments. :D
I'm sure that the 40/4 testers will read your request. Hopefully one of them might have purchased Deep Fritz 11, but I can't guarantee it. We'll see what we can do.
User avatar
Graham Banks
 
Posts: 18191
Joined: Sun Dec 18, 2005 5:47 pm
Location: Auckland, NZ

Re: The latest CCRL update (15th May 2009)

Postby Davemt01 » Mon May 18, 2009 7:59 am

Please excuse me for asking a very basic question. Perhaps it has already been answered somewhere on these forums, but if so, I have been unable to find the answer.

The question is regarding the CPU mentioned in the listings. The abbreviations used for the listing include "single-CPU", "2CPU" and "4CPU", and it's separately mentioned that the CPU in question is the Athlon 64 X2 4600+.

However, the Athlon 64 X2 4600+ is a dual core CPU. Am I correct in assuming, for example, that the top listing, which is:

Rybka 3 64-bit 4CPU

actually means:

Chess engine: Rybka, version 3, 64 bit
OS: some 64 bit OS, probably Windows Vista?
CPU Type: Athlon 64 X2 4600+
CPU Number: 4 separate CPUs, probably on some server-type motherboard? With a total of 8 cores?

By the same token, am I correct in assuming that "2CPU" means two separate Athlon 64 X2 4600+ CPUs (4 cores total)?

I would really appreciate it if I could see a description of the actual hardware, including CPUs, motherboard, amount and type of memory in particular, as well as the name and version of the operating system. Is this information available somewhere?

Thanks.

Dave M.
Davemt01
 
Posts: 3
Joined: Mon May 18, 2009 7:44 am

Re: The latest CCRL update (15th May 2009)

Postby Kirill Kryukov » Mon May 18, 2009 8:13 am

Davemt01 wrote:Please excuse me for asking a very basic question. Perhaps it has already been answered somewhere on these forums, but if so, I have been unable to find the answer.

The question is regarding the CPU mentioned in the listings. The abbreviations used for the listing include "single-CPU", "2CPU" and "4CPU", and it's separately mentioned that the CPU in question is the Athlon 64 X2 4600+.

However, the Athlon 64 X2 4600+ is a dual core CPU. Am I correct in assuming, for example, that the top listing, which is:

Rybka 3 64-bit 4CPU

actually means:

Chess engine: Rybka, version 3, 64 bit
OS: some 64 bit OS, probably Windows Vista?
CPU Type: Athlon 64 X2 4600+
CPU Number: 4 separate CPUs, probably on some server-type motherboard? With a total of 8 cores?

By the same token, am I correct in assuming that "2CPU" means two separate Athlon 64 X2 4600+ CPUs (4 cores total)?



Hi Dave,

"single-CPU", "2CPU" and "4CPU" in fact mean "single-core", "2-core" and "4-core". They are called CPUs because this list started before multi-core CPUs became common. Perhaps it is time to change the wording.

When Athlon 64 X2 4600+ is referred as a reference for 1-CPU testing, it should be taken as "one of the cores of Athlon 64 X2 4600+.

Davemt01 wrote:I would really appreciate it if I could see a description of the actual hardware, including CPUs, motherboard, amount and type of memory in particular, as well as the name and version of the operating system. Is this information available somewhere?

Thanks.

Dave M.

We use a broad variety of CPUs, motherboards, and other parts. It is simply not practical to try unifying the hardware. We are not that rich to buy identical machines just for chess, so we use our ordinary machines, whatever we have, for testing.

So, instead we use calibrated time control. We benchmark all of our machines, and adjust the time control so that it becomes close to what it would be on the reference machine (Athlon 64 X2 4600+).

Best,
Kirill
User avatar
Kirill Kryukov
Site Admin
 
Posts: 7380
Joined: Sun Dec 18, 2005 9:58 am
Location: Mishima, Japan

Re: The latest CCRL update (15th May 2009)

Postby Davemt01 » Mon May 18, 2009 8:57 am

Hi Kirill

Thanks very much for the reply. Could you let me know what you use to benchmark the machines, and if this benchmarking software is available to me for testing?

Thanks.

Dave M.
Davemt01
 
Posts: 3
Joined: Mon May 18, 2009 7:44 am

Re: The latest CCRL update (15th May 2009)

Postby Kirill Kryukov » Mon May 18, 2009 9:36 am

Hi Dave,

You're welcome! For historical reasons, we use Crafty 19.17 BH (Brian Hoffman's compile) as a benchmark. You can get a copy here.
User avatar
Kirill Kryukov
Site Admin
 
Posts: 7380
Joined: Sun Dec 18, 2005 9:58 am
Location: Mishima, Japan

Re: The latest CCRL update (15th May 2009)

Postby Davemt01 » Mon May 18, 2009 10:29 am

I downloaded the benchmark and was able to successfully run it on 3 different machines.

I am interested in chess engines (in a purely amateur/hobbyist way; I have no deep understanding of them), and would like at some point to help in the testing you're using to rate and compare engines. I can see that I still have some reading to do before I fully understand your methodology. I will continue to follow these forums.

Thanks again for the link to Crafty, and for answering my questions.

Dave M.
Davemt01
 
Posts: 3
Joined: Mon May 18, 2009 7:44 am

Re: The latest CCRL update (15th May 2009)

Postby Kirill Kryukov » Mon May 18, 2009 12:49 pm

Hi again, Dave. In this thread you can find discussion about CCRL testing conditions, including details on benchmarking and time control calibration. Good luck!
User avatar
Kirill Kryukov
Site Admin
 
Posts: 7380
Joined: Sun Dec 18, 2005 9:58 am
Location: Mishima, Japan

Re: The latest CCRL update (12th June 2009)

Postby guillef » Sun Jun 14, 2009 1:53 pm

For the CCRL Team,

I really enjoy using all the information you publish. CCRL is currently the primary source of information to know the progress of each new version of Chronos.

Thank you very much for your efforts in keeping the site updated. :wink:

Guillef.
guillef
 
Posts: 5
Joined: Tue Dec 09, 2008 10:23 am

Re: The latest CCRL update (12th June 2009)

Postby Graham Banks » Thu Jun 18, 2009 6:11 am

guillef wrote:For the CCRL Team,

I really enjoy using all the information you publish. CCRL is currently the primary source of information to know the progress of each new version of Chronos.

Thank you very much for your efforts in keeping the site updated. :wink:

Guillef.


Hi Guillef,

Chronos 1.9.6 will pass the 200 game mark in the next update. :)
Thanks for your kind words.

Cheers,
Graham.
User avatar
Graham Banks
 
Posts: 18191
Joined: Sun Dec 18, 2005 5:47 pm
Location: Auckland, NZ

Re: The latest CCRL update (23rd April 2010)

Postby Mindbreaker » Sat Apr 24, 2010 7:44 pm

I am getting a very different result between Rybka 3 Human vs StockFish 1.7.1 than vs Rybka 3. Rybka 3 Human appears to be a more difficult opponent for StockFish 1.7.1 It would be nice if someone official would run a match between these two.
Mindbreaker
 
Posts: 14
Joined: Tue Dec 05, 2006 10:07 pm
Location: San Deigo

Re: The latest CCRL update (23rd April 2010)

Postby Shaun » Mon Apr 26, 2010 8:06 pm

Mindbreaker wrote:I am getting a very different result between Rybka 3 Human vs StockFish 1.7.1 than vs Rybka 3. Rybka 3 Human appears to be a more difficult opponent for StockFish 1.7.1 It would be nice if someone official would run a match between these two.


I am running a head to head R3 v SF1.71 I will also run R3Human and R3Dynamic - Stockfish and R3 a close so lots of games are required for a conclusion one way or the other

Shaun
User avatar
Shaun
 
Posts: 6739
Joined: Sat May 13, 2006 3:24 pm
Location: Brighton. UK

Re: The latest CCRL update (31st July 2010)

Postby Philippe » Wed Aug 04, 2010 3:15 pm

Hi all,

It is just for a confirmation please... My processor is an Intel Core i7 940 (2.93 Ghz), I have 12 Gb RAM, I have Windows 7 64 bits. I think it is a quad core... So could you really confirm that it is better for me to buy the Deep Rybka 4?

Multi processor, multi core but... monolithic processor, so it sounds not so easy for me to be sure that the Deep Rybka 4 will be better for me than the "not Deep" :-)

Thank you very much for you answer!
Philippe
 
Posts: 1
Joined: Wed Aug 04, 2010 3:05 pm

Re: The latest CCRL update (7th August 2010)

Postby Quitch » Thu Aug 12, 2010 11:23 pm

I'm confused by the pure list in relation to this update. Why, if Deep Fritz 11 has a higher elo than Deep Fritz 10.1, is the latter used on the pure listing?
Quitch
 
Posts: 1
Joined: Thu Aug 12, 2010 11:21 pm

Re: The latest CCRL update (31st July 2010)

Postby Kirill Kryukov » Fri Aug 13, 2010 1:46 am

Philippe wrote:Hi all,

It is just for a confirmation please... My processor is an Intel Core i7 940 (2.93 Ghz), I have 12 Gb RAM, I have Windows 7 64 bits. I think it is a quad core... So could you really confirm that it is better for me to buy the Deep Rybka 4?

Multi processor, multi core but... monolithic processor, so it sounds not so easy for me to be sure that the Deep Rybka 4 will be better for me than the "not Deep" :-)

Thank you very much for you answer!

Hi Philippe,

Deep Rybka would certainly be better than single-core Rybka on a Core i7 940, but it's more expensive too. Whether it is better for you is something only you can tell, especially with so many strong freeware engines around. :-)

Best,
Kirill
User avatar
Kirill Kryukov
Site Admin
 
Posts: 7380
Joined: Sun Dec 18, 2005 9:58 am
Location: Mishima, Japan

Re: The latest CCRL update (7th August 2010)

Postby Kirill Kryukov » Fri Aug 13, 2010 1:48 am

Quitch wrote:I'm confused by the pure list in relation to this update. Why, if Deep Fritz 11 has a higher elo than Deep Fritz 10.1, is the latter used on the pure listing?

Hi Quitch,

I guess that you are talking about "Deep Fritz 11 4CPU" vs "Deep Fritz 10.1 4CPU". Most probably "Deep Fritz 11 4CPU" did not play enough games with the other participants of the pure list. At least 200 games are required.

Best,
Kirill
User avatar
Kirill Kryukov
Site Admin
 
Posts: 7380
Joined: Sun Dec 18, 2005 9:58 am
Location: Mishima, Japan

Re: The latest CCRL update (26th February 2011)

Postby Classen » Thu Mar 03, 2011 6:43 pm

This might be the wrong place to ask, but how can I get the testers to include Ghost 2.0.1 in the 40/4 testing? It has shown some nice results in 40/40, so I'm curious to see how it performs under shorter time limits.
Classen
 
Posts: 8
Joined: Mon Jun 14, 2010 11:04 am
Location: Germany

Re: The latest CCRL update (26th February 2011)

Postby Adam Hair » Thu Mar 03, 2011 7:02 pm

Classen wrote:This might be the wrong place to ask, but how can I get the testers to include Ghost 2.0.1 in the 40/4 testing? It has shown some nice results in 40/40, so I'm curious to see how it performs under shorter time limits.


It can happen fairly soon. I am always eager to include an engine when an author makes a request.
User avatar
Adam Hair
 
Posts: 1571
Joined: Sun May 30, 2010 3:28 am
Location: Fuquay-Varina, North Carolina, USA

Re: The latest CCRL update (26th February 2011)

Postby Classen » Sat Mar 05, 2011 2:21 pm

It can happen fairly soon. I am always eager to include an engine when an author makes a request.[/quote]

Ok, just to make sure: you can understand that as an official request. :)

I didn't see any postings related to 40/4 test matches yet, I'll be looking forward to that...
Classen
 
Posts: 8
Joined: Mon Jun 14, 2010 11:04 am
Location: Germany

Re: The latest CCRL update (26th February 2011)

Postby Adam Hair » Sat Mar 05, 2011 5:27 pm

Classen wrote:It can happen fairly soon. I am always eager to include an engine when an author makes a request.


Ok, just to make sure: you can understand that as an official request. :) [/quote]

:)

Classen wrote:I didn't see any postings related to 40/4 test matches yet, I'll be looking forward to that...


I will be starting the gauntlet this week.
User avatar
Adam Hair
 
Posts: 1571
Joined: Sun May 30, 2010 3:28 am
Location: Fuquay-Varina, North Carolina, USA

Re: The latest CCRL update (7th January 2012)

Postby Mindbreaker » Tue Jan 10, 2012 3:41 am

Maybe I am fussing a bit, but I like the live list but the links on the top direct instead to the normal list. It would make more sense that if one was already on a live list say the 40/4 list that when the 40/40 or Fischer Random list button is pushed that the live version shows up. Currently, I have to go back to the CCRL discussion board to find the live version. Alternatively, you could have both sets of buttons on each type of page.

I consider your list the best out there by quite a lot. I really appreciate your efforts and quality of presentation.

I also have another suggestion that is not quite as simple to implement. I think it would be outstanding if there was a graphical view of how rating changes for an engine relative to the others with more time / more threads. They could be combined as different ways to quantify the same thing: "Computation." There are losses in computing efficiency associated with more cores but the data points can be adjusted to appear be in appropriate locations. One only has to take the average kn/s from several games by an engine and average it and compare that to the average kn/s in games with different time controls or # of cores. On one axis would be Computation, on the other would be Relative Elo. It would mean that all the lists should have something like the same Elo average for the top 15 competitors that are on both lists. That really is the only change that would be needed to the way the current lists are made.

The beauty of such a graph is that it can project ratings for times that are inconvenient to test. Each engine fully tested would have a data point for 1-core/40/4, 2-core/40/4, 4-core/40/4, 6-core/40/4, 1-core/40/40, 2-core/40/40, 4-core/40/40, and 6-core/40/40. The points would lie either correctly spaced based on computation or on a log scale and the appropriate locations. "Computation" is average (kn/s)*(seconds/40 moves).

To make calculations easier, you initially can take averages of "Computation" among the leading engines at the various time controls and core activity levels, and just use those placements on the graph to put any future engine that is also of the same time control and thread level. That way it should be easy to generate the graph. I would also mean that you would not have to require testers to provide the average kn/s for each engine each game.

It would also be convenient if when one rolls over a particular line the name of the corresponding engine is shown.

Another nice aspect of the graph is that it can suggest where there might be statistical anomalies as a typical pattern should emerge for engines.
Mindbreaker
 
Posts: 14
Joined: Tue Dec 05, 2006 10:07 pm
Location: San Deigo

Re: The latest CCRL update (7th January 2012)

Postby Kirill Kryukov » Thu Jan 12, 2012 9:42 am

Thanks for suggestions. I don't have time at the moment to implement any of this, but will look into this in the future.
User avatar
Kirill Kryukov
Site Admin
 
Posts: 7380
Joined: Sun Dec 18, 2005 9:58 am
Location: Mishima, Japan

Next

Return to CCRL Public

Who is online

Users browsing this forum: No registered users and 1 guest

cron