Page 1 of 3

The latest CCRL update (16th December 2017)

PostPosted: Sat Jan 26, 2008 10:42 pm
by Graham Banks
The purpose of our rating lists is provide engine authors and enthusiasts with a general comparison of engine strength. We also provide other data that could be of interest.
For various reasons, there may be some engines that do not appear on our lists, therefore it is useful to look at other rating lists as well as ours.
The inclusion or exclusion of engines in our lists should not be taken as our group making a statement about their legality or status.
We test chess engines for our own enjoyment and receive no payment for doing so.


The latest CCRL Rating Lists and Statistics are available for viewing from the following links:
http://computerchess.org.uk/ccrl/4040/ (40/40)
http://www.computerchess.org.uk/ccrl/404/ (40/4)
http://www.computerchess.org.uk/ccrl/404FRC/ (FRC 40/4)

Please note that the three lists are often updated separately to each other. The FRC list is only updated when a new engine or engine version is being/has been tested.

The links to the various rating lists can be found just beneath the default Best Versions list (as in this screenshot). Specific 32-bit rating lists are denoted as such to the right of the default list in each category. The default lists contain the 64-bit engines.

Image

Our 40 moves in 40 minutes repeating and 40 moves in 4 minutes repeating are both adjusted to the AMD64 X2 4600+ (2.4GHz).
This time control is roughly equivalent to 40 moves in 20 minutes repeating or 40 moves in 2 minutes repeating on an Intel i7.

Be aware that in the early stages of testing, an engine's rating can often fluctuate a lot.
It is strongly advised to look at the many other rating lists available in order to get a more accurate overall picture of an engine's rating relative to others.

The LOS (likelihood of superiority) stats to the right hand side of each rating list tell you the likelihood in percentage terms of each engine being superior to the engine directly below them.

All games are available for download by engine or ECO code. The total games database in its entirety is always available.
The current ELO ratings are saved in all game databases for those engines that have 200 games or more.

Clicking on an engine name will give details as to opponents played plus homepage links where applicable.

Custom lists of engines can be selected for comparison.

An openings report page lists the number of games played by ECO codes with draw percentage and White win percentage. Clicking on a column heading will sort the list by that column.

Re: The latest CCRL update (9th January 2009)

PostPosted: Fri Jan 23, 2009 6:53 am
by ChessKnight
Graham Banks

The latest CCRL update


I am highly appreciative and thankful to your efforts for regular updates.

thanks.

Re: The latest CCRL update (9th January 2009)

PostPosted: Fri Jan 23, 2009 7:00 am
by Graham Banks
ChessKnight wrote:Graham Banks

The latest CCRL update


I am highly appreciative and thankful to your efforts for regular updates.

thanks.


Thanks. It's a team effort to do all that we do. Producing an update report every two weeks is just one little way I can help the team. :)

Regards, Graham.

Re: The latest CCRL update (17th April 2009)

PostPosted: Wed Apr 22, 2009 10:26 pm
by Mindbreaker
It would be nice to see a rating for Deep Fritz 11 in the 40/4 category. I love all the stats, results matrices, links to engine homepages, and version comparisons. Great job. I have enjoyed this site for years.

Re: The latest CCRL update (17th April 2009)

PostPosted: Thu Apr 23, 2009 9:39 pm
by Graham Banks
Mindbreaker wrote:It would be nice to see a rating for Deep Fritz 11 in the 40/4 category. I love all the stats, results matrices, links to engine homepages, and version comparisons. Great job. I have enjoyed this site for years.


Thanks for your kind comments. :D
I'm sure that the 40/4 testers will read your request. Hopefully one of them might have purchased Deep Fritz 11, but I can't guarantee it. We'll see what we can do.

Re: The latest CCRL update (15th May 2009)

PostPosted: Mon May 18, 2009 7:59 am
by Davemt01
Please excuse me for asking a very basic question. Perhaps it has already been answered somewhere on these forums, but if so, I have been unable to find the answer.

The question is regarding the CPU mentioned in the listings. The abbreviations used for the listing include "single-CPU", "2CPU" and "4CPU", and it's separately mentioned that the CPU in question is the Athlon 64 X2 4600+.

However, the Athlon 64 X2 4600+ is a dual core CPU. Am I correct in assuming, for example, that the top listing, which is:

Rybka 3 64-bit 4CPU

actually means:

Chess engine: Rybka, version 3, 64 bit
OS: some 64 bit OS, probably Windows Vista?
CPU Type: Athlon 64 X2 4600+
CPU Number: 4 separate CPUs, probably on some server-type motherboard? With a total of 8 cores?

By the same token, am I correct in assuming that "2CPU" means two separate Athlon 64 X2 4600+ CPUs (4 cores total)?

I would really appreciate it if I could see a description of the actual hardware, including CPUs, motherboard, amount and type of memory in particular, as well as the name and version of the operating system. Is this information available somewhere?

Thanks.

Dave M.

Re: The latest CCRL update (15th May 2009)

PostPosted: Mon May 18, 2009 8:13 am
by Kirill Kryukov
Davemt01 wrote:Please excuse me for asking a very basic question. Perhaps it has already been answered somewhere on these forums, but if so, I have been unable to find the answer.

The question is regarding the CPU mentioned in the listings. The abbreviations used for the listing include "single-CPU", "2CPU" and "4CPU", and it's separately mentioned that the CPU in question is the Athlon 64 X2 4600+.

However, the Athlon 64 X2 4600+ is a dual core CPU. Am I correct in assuming, for example, that the top listing, which is:

Rybka 3 64-bit 4CPU

actually means:

Chess engine: Rybka, version 3, 64 bit
OS: some 64 bit OS, probably Windows Vista?
CPU Type: Athlon 64 X2 4600+
CPU Number: 4 separate CPUs, probably on some server-type motherboard? With a total of 8 cores?

By the same token, am I correct in assuming that "2CPU" means two separate Athlon 64 X2 4600+ CPUs (4 cores total)?



Hi Dave,

"single-CPU", "2CPU" and "4CPU" in fact mean "single-core", "2-core" and "4-core". They are called CPUs because this list started before multi-core CPUs became common. Perhaps it is time to change the wording.

When Athlon 64 X2 4600+ is referred as a reference for 1-CPU testing, it should be taken as "one of the cores of Athlon 64 X2 4600+.

Davemt01 wrote:I would really appreciate it if I could see a description of the actual hardware, including CPUs, motherboard, amount and type of memory in particular, as well as the name and version of the operating system. Is this information available somewhere?

Thanks.

Dave M.

We use a broad variety of CPUs, motherboards, and other parts. It is simply not practical to try unifying the hardware. We are not that rich to buy identical machines just for chess, so we use our ordinary machines, whatever we have, for testing.

So, instead we use calibrated time control. We benchmark all of our machines, and adjust the time control so that it becomes close to what it would be on the reference machine (Athlon 64 X2 4600+).

Best,
Kirill

Re: The latest CCRL update (15th May 2009)

PostPosted: Mon May 18, 2009 8:57 am
by Davemt01
Hi Kirill

Thanks very much for the reply. Could you let me know what you use to benchmark the machines, and if this benchmarking software is available to me for testing?

Thanks.

Dave M.

Re: The latest CCRL update (15th May 2009)

PostPosted: Mon May 18, 2009 9:36 am
by Kirill Kryukov
Hi Dave,

You're welcome! For historical reasons, we use Crafty 19.17 BH (Brian Hoffman's compile) as a benchmark. You can get a copy here.

Re: The latest CCRL update (15th May 2009)

PostPosted: Mon May 18, 2009 10:29 am
by Davemt01
I downloaded the benchmark and was able to successfully run it on 3 different machines.

I am interested in chess engines (in a purely amateur/hobbyist way; I have no deep understanding of them), and would like at some point to help in the testing you're using to rate and compare engines. I can see that I still have some reading to do before I fully understand your methodology. I will continue to follow these forums.

Thanks again for the link to Crafty, and for answering my questions.

Dave M.

Re: The latest CCRL update (15th May 2009)

PostPosted: Mon May 18, 2009 12:49 pm
by Kirill Kryukov
Hi again, Dave. In this thread you can find discussion about CCRL testing conditions, including details on benchmarking and time control calibration. Good luck!

Re: The latest CCRL update (12th June 2009)

PostPosted: Sun Jun 14, 2009 1:53 pm
by guillef
For the CCRL Team,

I really enjoy using all the information you publish. CCRL is currently the primary source of information to know the progress of each new version of Chronos.

Thank you very much for your efforts in keeping the site updated. :wink:

Guillef.

Re: The latest CCRL update (12th June 2009)

PostPosted: Thu Jun 18, 2009 6:11 am
by Graham Banks
guillef wrote:For the CCRL Team,

I really enjoy using all the information you publish. CCRL is currently the primary source of information to know the progress of each new version of Chronos.

Thank you very much for your efforts in keeping the site updated. :wink:

Guillef.


Hi Guillef,

Chronos 1.9.6 will pass the 200 game mark in the next update. :)
Thanks for your kind words.

Cheers,
Graham.

Re: The latest CCRL update (23rd April 2010)

PostPosted: Sat Apr 24, 2010 7:44 pm
by Mindbreaker
I am getting a very different result between Rybka 3 Human vs StockFish 1.7.1 than vs Rybka 3. Rybka 3 Human appears to be a more difficult opponent for StockFish 1.7.1 It would be nice if someone official would run a match between these two.

Re: The latest CCRL update (23rd April 2010)

PostPosted: Mon Apr 26, 2010 8:06 pm
by Shaun
Mindbreaker wrote:I am getting a very different result between Rybka 3 Human vs StockFish 1.7.1 than vs Rybka 3. Rybka 3 Human appears to be a more difficult opponent for StockFish 1.7.1 It would be nice if someone official would run a match between these two.


I am running a head to head R3 v SF1.71 I will also run R3Human and R3Dynamic - Stockfish and R3 a close so lots of games are required for a conclusion one way or the other

Shaun

Re: The latest CCRL update (31st July 2010)

PostPosted: Wed Aug 04, 2010 3:15 pm
by Philippe
Hi all,

It is just for a confirmation please... My processor is an Intel Core i7 940 (2.93 Ghz), I have 12 Gb RAM, I have Windows 7 64 bits. I think it is a quad core... So could you really confirm that it is better for me to buy the Deep Rybka 4?

Multi processor, multi core but... monolithic processor, so it sounds not so easy for me to be sure that the Deep Rybka 4 will be better for me than the "not Deep" :-)

Thank you very much for you answer!

Re: The latest CCRL update (7th August 2010)

PostPosted: Thu Aug 12, 2010 11:23 pm
by Quitch
I'm confused by the pure list in relation to this update. Why, if Deep Fritz 11 has a higher elo than Deep Fritz 10.1, is the latter used on the pure listing?

Re: The latest CCRL update (31st July 2010)

PostPosted: Fri Aug 13, 2010 1:46 am
by Kirill Kryukov
Philippe wrote:Hi all,

It is just for a confirmation please... My processor is an Intel Core i7 940 (2.93 Ghz), I have 12 Gb RAM, I have Windows 7 64 bits. I think it is a quad core... So could you really confirm that it is better for me to buy the Deep Rybka 4?

Multi processor, multi core but... monolithic processor, so it sounds not so easy for me to be sure that the Deep Rybka 4 will be better for me than the "not Deep" :-)

Thank you very much for you answer!

Hi Philippe,

Deep Rybka would certainly be better than single-core Rybka on a Core i7 940, but it's more expensive too. Whether it is better for you is something only you can tell, especially with so many strong freeware engines around. :-)

Best,
Kirill

Re: The latest CCRL update (7th August 2010)

PostPosted: Fri Aug 13, 2010 1:48 am
by Kirill Kryukov
Quitch wrote:I'm confused by the pure list in relation to this update. Why, if Deep Fritz 11 has a higher elo than Deep Fritz 10.1, is the latter used on the pure listing?

Hi Quitch,

I guess that you are talking about "Deep Fritz 11 4CPU" vs "Deep Fritz 10.1 4CPU". Most probably "Deep Fritz 11 4CPU" did not play enough games with the other participants of the pure list. At least 200 games are required.

Best,
Kirill

Re: The latest CCRL update (26th February 2011)

PostPosted: Thu Mar 03, 2011 6:43 pm
by Classen
This might be the wrong place to ask, but how can I get the testers to include Ghost 2.0.1 in the 40/4 testing? It has shown some nice results in 40/40, so I'm curious to see how it performs under shorter time limits.

Re: The latest CCRL update (26th February 2011)

PostPosted: Thu Mar 03, 2011 7:02 pm
by Adam Hair
Classen wrote:This might be the wrong place to ask, but how can I get the testers to include Ghost 2.0.1 in the 40/4 testing? It has shown some nice results in 40/40, so I'm curious to see how it performs under shorter time limits.


It can happen fairly soon. I am always eager to include an engine when an author makes a request.

Re: The latest CCRL update (26th February 2011)

PostPosted: Sat Mar 05, 2011 2:21 pm
by Classen
It can happen fairly soon. I am always eager to include an engine when an author makes a request.[/quote]

Ok, just to make sure: you can understand that as an official request. :)

I didn't see any postings related to 40/4 test matches yet, I'll be looking forward to that...

Re: The latest CCRL update (26th February 2011)

PostPosted: Sat Mar 05, 2011 5:27 pm
by Adam Hair
Classen wrote:It can happen fairly soon. I am always eager to include an engine when an author makes a request.


Ok, just to make sure: you can understand that as an official request. :) [/quote]

:)

Classen wrote:I didn't see any postings related to 40/4 test matches yet, I'll be looking forward to that...


I will be starting the gauntlet this week.

Re: The latest CCRL update (7th January 2012)

PostPosted: Tue Jan 10, 2012 3:41 am
by Mindbreaker
Maybe I am fussing a bit, but I like the live list but the links on the top direct instead to the normal list. It would make more sense that if one was already on a live list say the 40/4 list that when the 40/40 or Fischer Random list button is pushed that the live version shows up. Currently, I have to go back to the CCRL discussion board to find the live version. Alternatively, you could have both sets of buttons on each type of page.

I consider your list the best out there by quite a lot. I really appreciate your efforts and quality of presentation.

I also have another suggestion that is not quite as simple to implement. I think it would be outstanding if there was a graphical view of how rating changes for an engine relative to the others with more time / more threads. They could be combined as different ways to quantify the same thing: "Computation." There are losses in computing efficiency associated with more cores but the data points can be adjusted to appear be in appropriate locations. One only has to take the average kn/s from several games by an engine and average it and compare that to the average kn/s in games with different time controls or # of cores. On one axis would be Computation, on the other would be Relative Elo. It would mean that all the lists should have something like the same Elo average for the top 15 competitors that are on both lists. That really is the only change that would be needed to the way the current lists are made.

The beauty of such a graph is that it can project ratings for times that are inconvenient to test. Each engine fully tested would have a data point for 1-core/40/4, 2-core/40/4, 4-core/40/4, 6-core/40/4, 1-core/40/40, 2-core/40/40, 4-core/40/40, and 6-core/40/40. The points would lie either correctly spaced based on computation or on a log scale and the appropriate locations. "Computation" is average (kn/s)*(seconds/40 moves).

To make calculations easier, you initially can take averages of "Computation" among the leading engines at the various time controls and core activity levels, and just use those placements on the graph to put any future engine that is also of the same time control and thread level. That way it should be easy to generate the graph. I would also mean that you would not have to require testers to provide the average kn/s for each engine each game.

It would also be convenient if when one rolls over a particular line the name of the corresponding engine is shown.

Another nice aspect of the graph is that it can suggest where there might be statistical anomalies as a typical pattern should emerge for engines.

Re: The latest CCRL update (7th January 2012)

PostPosted: Thu Jan 12, 2012 9:42 am
by Kirill Kryukov
Thanks for suggestions. I don't have time at the moment to implement any of this, but will look into this in the future.