CCRL 40/15 Testing Conditions (previously 40/40)

[ThatsMe]

Thanks for the specific link. I can see Stockfish 14 in this selection. But I am unsure what it means. Does it mean that Stockfish 14 is not better than its predecessor Stockfish 13 and therefore is hidden in the main page http://ccrl.chessdom.com/ccrl/4040/ ? Thanks

[Graham Banks]

Thanks for the specific link. I can see Stockfish 14 in this selection. But I am unsure what it means. Does it mean that Stockfish 14 is not better than its predecessor Stockfish 13 and therefore is hidden in the main page http://ccrl.chessdom.com/ccrl/4040/ ? Thanks

That is the situation at present, but could well change.

4CPU testing is in progress also:

https://www.computerchess.org.uk/ccrl/4 ... ons_only=1

[ThatsMe]

Thanks for the clarification. What I take away from this is that the CCRL differences are in any case not so huge that you can clearly see them.

Funny that the Stockfish community's own measurements showed an increase of about 41 ELO points between Stockfish 13 and Stockfish 14.
For example, their list https://nextchessmove.com/dev-builds says:
v14 (+415.16 ELO increase over Stockfish 7).
v13 (+373.78 ELO increase compared to Stockfish 7)
This means for me (of course I can be totally wrong) that between Stockfish 13 and Stockfish 14 there must have been an ELO increase of about 41 points.

I wonder why are all these ELO measurements so contradictory? And who's right..?

[Graham Banks]

Never rely on just one rating list.

Check the other well-known ones and you'll get a pretty good overall picture.

[ThatsMe]

That's what I thought. Maybe the Stockfish team overestimated themselves in their own assessments. Thanks.

[Ray]

The Stockfish results are all over the place in the various lists. A couple of them do in fact agree with the fishtest results. There are so many variables - time control, threads and book choice for example. In our case I believe different hardware can also impact the results especially today with NNUE engines performing better on AVX2 machines etc. I don't think that the hardware made a significant difference a few years ago, with adapted time control of course, but now the situation has probably changed.

In terms of the list lagging behind, that is simply down to resources of course.

[Madeleine Birchfield]

That's what I thought. Maybe the Stockfish team overestimated themselves in their own assessments. Thanks.

CCRL is the only ratings list that.I know of that doesn't have any visible elo difference between Stockfish 13 and Stockfish 14. Other ratings lists like FGRL and CEGT largely agree with Fishtest estimates.

[Graham Banks]

That's what I thought. Maybe the Stockfish team overestimated themselves in their own assessments. Thanks.

CCRL is the only ratings list that.I know of that doesn't have any visible elo difference between Stockfish 13 and Stockfish 14. Other ratings lists like FGRL and CEGT largely agree with Fishtest estimates.

That's why you should look at many rating lists in order to get the overall picture.

[xmchess]

Hie
1) I would like to know which version of program are used for ccrl 40/15 rating list :
64 bit win ?
avx 2 ?
bmi 2 ?
sse 42 ?
2) The xmchess programm has won aginst stockfish_14_1 ( 20 seconds for each moves) for the first time yesterday (he has lost too against this great program !!) . I was very happy.
this new proramm will be avalaible soon
xmchess

[Graham Banks]

Hie
1) I would like to know which version of program are used for ccrl 40/15 rating list :
64 bit win ?
avx 2 ?
bmi 2 ?
sse 42 ?
2) The xmchess programm has won aginst stockfish_14_1 ( 20 seconds for each moves) for the first time yesterday (he has lost too against this great program !!) . I was very happy.
this new proramm will be avalaible soon
xmchess

I would imagine that we're all using either bmi2 or avx2, but an sse4.2 compile is probably a good idea too.

[xmchess]

hie thanxs
the experimental version 0.1 of xmchess will be avalaible soon.
this version is about 3300-3400 elo in 40/15
Is it possible to be in the ccrl list 40/15 ?
is it free ?
how to get ranked
thanks a lot

[Gabor Szots]

the experimental version 0.1 of xmchess will be avalaible soon.
this version is about 3300-3400 elo in 40/15

Will it be open source?

[xmchess]

Hie
No it will not be open source
sorry ...
I am looking for investors to value the new ia algorithms used in xmchess.

[bastiball]

Hie
No it will not be open source
sorry ...
I am looking for investors to value the new ia algorithms used in xmchess.

How long u work for your engine?

[xmchess]

i im working for xmchess since 2 years !!!
the final version will have a new neural network with 300 000 parameters (a non nnue neural network). It will use two new lerning algorithms and a genetic algorithm.

[xmchess]

Hie
the experimental version 0.1 of xmchess will be avalaible soon.
this version is about 3300-3400 elo in 40/15
Is it possible to be in the ccrl list 40/15 ?
how to get ranked ?
Thx a lot

[Gabor Szots]

how to get ranked ?

There is no rule for that. Just release it, announce it and hope that someone might pick it up.

I recommend you reveal your real name and country but that is not obligatory if you want to remain anonym. However, some raw background information about your engine (origins, history, etc.) will be required. And most of all: exes running under various Windows platforms.

PS. Please open a new thread for your engine. This one is not for announcing and discussing engines.

[xmchess]

thxs a lot
xavier merrheim researcher in ai University of Lyon, France i will open a new topic

[Paul Galbraith]

In Windows you just double-click the Stockfish 10 exe and type "bench", Adapted time control for 40/15 would be/:

T minutes for 40 moves where T = Your time / 2054 x 15 (for 40/15)

I think this is the only place I've been able to find this information, maybe it's documented privately I'm guessing from some of the other discussion here, can anyone confirm this? From this post I'm assuming that 2054 is the time produced on real hardware for the "standard" machine running the Stockfish 10 bench ... can anyone confirm which binary was used to obtain this value (bmi2, x64, etc.)?

[Ray]

In Windows you just double-click the Stockfish 10 exe and type "bench", Adapted time control for 40/15 would be/:

T minutes for 40 moves where T = Your time / 2054 x 15 (for 40/15)

From this post I'm assuming that 2054 is the time produced on real hardware for the "standard" machine running the Stockfish 10 bench ... can anyone confirm which binary was used to obtain this value (bmi2, x64, etc.)?

Yes, Graham has an i7-4770k so it is an actual result on an actual machine. Another tester has similar Intel hardware and got the same results.

[Paul Galbraith]

From this post I'm assuming that 2054 is the time produced on real hardware for the "standard" machine running the Stockfish 10 bench ... can anyone confirm which binary was used to obtain this value (bmi2, x64, etc.)?

Yes, Graham has an i7-4770k so it is an actual result on an actual machine. Another tester has similar Intel hardware and got the same results.

Thanks, Ray. Do you happen to know which binary was used by Graham?

[Graham Banks]

From this post I'm assuming that 2054 is the time produced on real hardware for the "standard" machine running the Stockfish 10 bench ... can anyone confirm which binary was used to obtain this value (bmi2, x64, etc.)?

Yes, Graham has an i7-4770k so it is an actual result on an actual machine. Another tester has similar Intel hardware and got the same results.

Thanks, Ray. Do you happen to know which binary was used by Graham?

The machine supports both bm12 and avx2, so would have been the bm12 one, as the modern one is just popcount as far as I'm aware.

[Paul Galbraith]

The machine supports both bm12 and avx2, so would have been the bm12 one, as the modern one is just popcount as far as I'm aware.

Thanks, Graham!

[jkominek]

I have a question in regards to relating the benchmark calibration to running conditions. The answer is perhaps obvious, but better to ask since I have not seen it addressed in this forum thread.

i) How many simultaneous games do CCRL testers typically run?
ii) If it is more than one, is benchmark calibration performed when the machine running a CPU load in line with what is expected during the tournament?

In Kirill's post he suggests rebooting to get a clean slate before benchmarking. But that was back in the years of single-CPU computers.

To be specific to my situation I have a 48-CPU test computer, and use cutechess-cli. I set "-concurrency 48" when running engines single-threaded, or "-concurrency 12" with engines configured to 4 threads. The benchmark result under heavy load is about double the time when run while the machine is idle. That makes a big difference to time control settings if I want a good match CCRL testing conditions.

[Gabor Szots]

I have a question in regards to relating the benchmark calibration to running conditions. The answer is perhaps obvious, but better to ask since I have not seen it addressed in this forum thread.

i) How many simultaneous games do CCRL testers typically run?
ii) If it is more than one, is benchmark calibration performed when the machine running a CPU load in line with what is expected during the tournament?

In Kirill's post he suggests rebooting to get a clean slate before benchmarking. But that was back in the years of single-CPU computers.

To be specific to my situation I have a 48-CPU test computer, and use cutechess-cli. I set "-concurrency 48" when running engines single-threaded, or "-concurrency 12" with engines configured to 4 threads. The benchmark result under heavy load is about double the time when run while the machine is idle. That makes a big difference to time control settings if I want a good match CCRL testing conditions.

To begin with, I would not let the engines use all threads simultaneously. I'd reserve a thread or two for system things. Just to be on the safe side. I may be wrong though.

The machine I use has only 4 cores. They are all run always at the same speed. I have set this in BIOS and check the benchmarks every now and then. Load has no impact. I never run more than 3 tournaments at a time.