Greetings and new Eubos chess engine

[c_j_bolt]

It is hard for me to be sure without seeing profiling data. However, it could be that the Java Virtual Machine is working particularly hard on garbage collection, for some reason. That can peak at between 8%-10% in my tests. It will depend on the size of the heap, as to how hard it is working.

I'm currently working on Multithreading Eubos (will be v2.0.0), where CPU utilisation does rise accordingly, but v1.1.6 is a single threaded search, with 2 other threads for UCI messaging and periodic UCI info reporting. It is also possible onr of those threads is blocking on a resource for some reason.

[c_j_bolt]

Can I ask for a clarification of the CCRL rules surrounding multithreaded engines and hash size?

I have added an option to Eubos called "NumberOfWorkerThreads", where the minimum setting is 1, the maximum is the number of cores available on the PC and the default is (num cores - 2).

When testing the engine, am I correct in thinking it is allowed 256Mb of hash for each thread?

If so, should the engine itself multiply the thread number by the configured hash size internally (e.g. user sets 256 hash, engine multiplies by number of threads - say 4, so engine uses 1024), or should it be left to the CCRL user to specify the hash size to use (e.g. 1024 for a 4 thread test).

I am assuming that the engine should treat the settings completely distinct from one another, not make assumptions and let the user configure the engine as they see fit.

I just want to be sure about this so I can do representative testing of the CCRL conditions during development.

[Gabor Szots]

When testing the engine, am I correct in thinking it is allowed 256Mb of hash for each thread?

If so, should the engine itself multiply the thread number by the configured hash size internally (e.g. user sets 256 hash, engine multiplies by number of threads - say 4, so engine uses 1024), or should it be left to the CCRL user to specify the hash size to use (e.g. 1024 for a 4 thread test).

I am assuming that the engine should treat the settings completely distinct from one another, not make assumptions and let the user configure the engine as they see fit.

256 MB is the max. hash per core used in blitz testing. In 40/15 testing the hash size may be bigger, I don't know.

Settings shall be independent from each other. The user should be allowed the freedom of varying them as he wants.

[c_j_bolt]

Great thanks, I will configure 1024 hash for my 4 core testing, from Arena, in that case.

[c_j_bolt]

Thanks for the testing.

I see that Eubos only managed a 15% win rateagainst Pwned 1.3, that is comfortably the worst run I have ever seen. I will try and look through the PGN to find out what went wrong, on the surface it really looks like an outlier as that engine is (theoretically) weaker! Though I have never tested against it myself. It also fared worse against Tarrasch Toy Engine than in my testing.

Can I ask what system you ran it on Gabor? I want to understand if I have issues with a certain JVM version/hardware configuration.

[Gabor Szots]

Thanks for the testing.

I see that Eubos only managed a 15% win rateagainst Pwned 1.3, that is comfortably the worst run I have ever seen. I will try and look through the PGN to find out what went wrong, on the surface it really looks like an outlier as that engine is (theoretically) weaker! Though I have never tested against it myself. It also fared worse against Tarrasch Toy Engine than in my testing.

Can I ask what system you ran it on Gabor? I want to understand if I have issues with a certain JVM version/hardware configuration.

The result against Pwned is indeed weird.

Against Tarrasch the result is not that surprising, they are close to one another on the list so that is a small anomaly.

This is my system:

CPU: i5-4690K at 3.5 GHz
OS: Windows 10 Home Edition 64-bit
RAM: 8 GB - I run 3 tournaments at a time and I also use the machine for other purposes while they are running.
Java version: 14.0.1 (64-bit)

Games were played under Arena 3.5.1. I used the jar file directly not the batch. That way the -Xshare:off part was not included (I don't know what it does).

[c_j_bolt]

I skimmed over a few of the PGN games for Eubos and Pwned (it is great that they are made available). It does look like Eubos was just outplayed. I'll try and do some testing locally against this engine. Maybe I have a new nemesis to test against, it can be a new sparring partner. Thanks for the details.

[Gabor Szots]

I'm testing 2.0 and Process explorer shows that it uses about 35 % CPU (out of 4 cores).

[c_j_bolt]

Hi Gabor,

There is a defect in v2.0 which can cause one of the worker threads to get a null pointer exception and crash. It is possible that this could have affected the CPU load. However that would only happen when the hash table needed to be trimmed to make way for more transpositions, so depending on the configuration (I mean hash table size), that might not explain it. I actually find in 2/+1 time control that the hash table may only need to be trimmed once in each game, so this defect might be occuring infrequently in your runs.

How may threads are you configuring? In my testing 2 threads is optimum and adding more after that has negligible benefit in strength. This is because my initial work sharing algorithm is rather crude and inefficient. I will improve this as my next development activity. Bascially threads get blocked on the access to the hash table, or they are searching positions which should be pruned by the alpha beta algorithm. It is like a traffic jam!

On my PC I find that if I run with 2 threads, it takes about 60% of the available processing resource - I have 4 physical cores in my PC (Java garbage collection and other threads will account for some of the additional 10%, I think).

[Gabor Szots]

How may threads are you configuring?

Hi Chris,

One. I only do 1-CPU testing.

[c_j_bolt]

Having spent the best part of several weeks testing different worker thread and hash table configurations (against a single opponent engine!) I can see the appeal of just testing with a single thread! The permutations start to get cumbersome very quickly. Even with one thread v2.0 should be stronger than v1.1.6, so it is worth a test. I'm not sure about strength with 1 thread, but with 2, I estimate it to be about 1650-1700 elo.

Another thing to mention, I recently found out that the Java runtime compiles native instructions dynamically at run-time (they call it JIT). Some of the unexpected CPU overhead we are seeing will be the JVM optimiser compiling the Java bytecode into native code. This is a really neat feature, but annoyingly for Chess Engine testing it means that this native code is compiled once again, every time the Jar file is executed in Arena - i.e. it is recompiled and optimised (probably the same way) in every single game of a tournament!

[Gabor Szots]

I can see the appeal of just testing with a single thread!

And I am yet to see the appeal of multi-thread testing. But I am aware that I represent a unique standpoint here.

The thing about java is interesting even if I don't understand it very much.

[c_j_bolt]

Hi Gabor,

I have recently released Eubos v2.2 on my github page. There are some change notes there about what is different, most importantly I no longer cap the extended search at each ply of the iterative deepening to an additional depth of 8 ply. this prevented Eubos seeing some imprtant tactical variations to completion.

I have tested Eubos v2.2 against TSCP (300 games 58% win rate), Robin (300games, 55% win rate) and Beaches 2.32 (200games, 41%win rate). I estimate its ELO to be in the region of 1770 to 1780.

When you get some time, do you think you could include it in your test queue, against comparable engines. Thanks in advance!

Chris

[Gabor Szots]

When you get some time, do you think you could include it in your test queue, against comparable engines. Thanks in advance!

Sure. Possibly already tomorrow. I always try to be up-to-date.

Cheers,
Gabor

PS. Could you please spell Elo as such? It is a name, same as e.g. Watt.

[c_j_bolt]

Hi Gabor, thanks very much for the testing

Good to see the solid increase I had seen replicated in the CCRL listing. In a couple of weeks/months I might look at trying to introduce some pruning of the search tree, but for now I think I'll take a break!

Thanks again, Chris

[Gabor Szots]

Hi Gabor, thanks very much for the testing

Good to see the solid increase I had seen replicated in the CCRL listing. In a couple of weeks/months I might look at trying to introduce some pruning of the search tree, but for now I think I'll take a break!

Thanks again, Chris

Hi Chris,

As you can see opponent selection was not the best. Further testing against stronger engines is coming, which may decrease Eubos's current rating. But not by very much, I guess.

[c_j_bolt]

Hello again,

I would have posted this over at TalkChess, but I seem to have an issue with my credentials over there.

I have recently released a couple of versions of Eubos. I see v2.4 was picked up by one of the CCRL testers, thanks for that. Was it you Gabor?

I just wanted to let you know that I had to do a bug fix for v2.5, which contained a serious bug that I missed. Please don't test or use that build. It is ok for single-threaded, but will crash if configured to use more than one thread.

I missed this because I now tend to test exclusively with single-threaded configuration, and an optimisation in v2.5 broke multi-threaded operation. This wouldn't be too bad, if it wasn't for the fact that Eubos defaults to multi-threaded if not configured explicitly to use 1 thread (on multicore systems). I.e. it would crash out-of-the-box for any new user

In v2.6, which I just released, it shall always run single-threaded. I will fix it properly in the next release.

Cheers,
Chris

[Gabor Szots]

I have recently released a couple of versions of Eubos. I see v2.4 was picked up by one of the CCRL testers, thanks for that. Was it you Gabor?

Hi Chris,

Yes, it was me.

What strength increase do you expect from version 2.6? I might pick it up as well although my resources are scarce.

Oh, and I see now that the jar has taken up some fat, being 7 times as large as the 2.4 one. Why is that?

[c_j_bolt]

I estimate Eubos is now at about 1890-1895 Elo, so that is an increase of about 50-60 points (on v2.4).

I have greatly improved the speed of the move generation, but I still have a long way to go. I need to tune up the evaluation function some more and, beyond fixing the major bug, that shall be my next objective. I want to get to around 2000 before the next hiatus. I'm not sure how realistic that is, but it is good to have a goal.

[c_j_bolt]

Hi Gabor,

If you can insert Eubos 2.10 (just released) into your test queue I would be very grateful.

I have been working on pruning and reductions in the last few releases and, as a result, I believe Eubos is now in the 2000+ range. Making the step from pure brute force to selective path searching has been a real revelation for me; I can see that there is a great deal to be gleaned from tweaking algorithms like late move reduction. It is very satisfying to find that Eubos isn't typically being out searched (in terms of depth) with respect to other engines in its class. I worked for ages optimising, when what I really needed was algorithms to curb the problem of search explosion.

I'm going to focus on staged move generation next, so I don't expect any big change in strength for a little while.

Thanks in advance,
Chris

[Gabor Szots]

Hi Chris,

I'll do my best but there are many waiting out there. But one of my fellow testers might pick it before me.

[c_j_bolt]

Thanks, there's no rush and I'm very happy to wait my turn

[bastiball]

Hello Chris,

I'll put it on my queue

[bastiball]

http://www.computerchess.org.uk/ccrl/40 ... _10_64-bit

[Gabor Szots]

http://www.computerchess.org.uk/ccrl/40 ... _10_64-bit

Unfortunately very few opponents. I am currently running a gauntlet against 13 opponents (32 games against each) and that may change the rating although probably not by much.