New engine: Jangine

[Jan-Frederik Konopka]

Hi everybody! I would be grateful if my engine "Jangine" could be rated and added to the CCRL list. I just released v2022.12.29 I actually unsuccessfully tried signing up and contacting you a few weeks ago, so I'm glad it finally worked out.

Comment: The engine is a traditional alphabeta searcher. I think the engine's strongest suit is speed, as it is written in C++, and because I have tried optimizing move generation and search (e.g. only generating captures and quiet moves when absolutely needed, not sorting quiet moves, board eval updated at each node instead of calculated fresh at each leaf, etc.). So my prediction is that Jangine is strongest in hyperbullet relative to other engines, and getting weaker at slower time controls.

Open work: The most obvious addition would be to add more pruning/reductions, which I think could improve rating significantly. bitboards could add a bit as well, if I can wrap my head around them. I added principal variation search with modest (~10%) gain, so aspiration windows are a logical next step. The halfhearted UCI/Xboard support also needs to be replaced with proper support. General code/repo cleanup is also in order as this was my first ever C++ project.

Name: Jangine
Version: v2022.12.29
Protocol: UCI/Xboard
Author: Jan-Frederik Konopka
Country: Germany
Website: https://github.com/xjcl/jangine
Download: https://github.com/xjcl/jangine/releases
Estimated Elo: 2150 (at 10-second time control, actual elo could be lower)

Thanks for reading, and don't hesitate to suggest improvements!

[Gabor Szots]

Hello Jan-Frederik,

I have just noticed your post. Glad to see a new engine around, I am going to look at it and see if I can test it.

Gabor

EDIT: I suggest including the version in the engine id. If all versions display simply Jangine, a confusion may arise.

And under Arena no PV is displayed, only the move that is played eventually.

[Gabor Szots]

I have started a tournament with Jangine. First results show it will be around 2000 Elos, probably a bit lower.

[Gabor Szots]

Tournament over. 552 games have been played out of which 49 was lost on time and there were 9 illegal moves. Results at next update.

[Jan-Frederik Konopka]

Yikes at those results! Thank you for your diligent testing.

rules infraction (1.6%): I'm surprised by this as well, I found only one illegal move in my dataset but I made a mistake analyzing it. Presumably illegal moves happen due to hash collisions, so if we search a billion positions during a game we have about a 5% chance of collision (as (10**9)**2 * 2**-64 = 0.054). Maybe this can be fixed by checking a move for legality when fetching it from the transposition table, but I don't have a function for that.

A better approach might be to do some sort of aging, by also storing the ply at which the transposition was encountered so we don't use a transposition from the early game later when it is very unlikely to occur again, and more likely to be a collision.

time forfeit (8.9%): I had a much lower rate of time forfeits in my dataset (3.6%), although I was playing weaker opponents. When we finish searching depth $n$, we make a simple binary decision of whether to search depth $n+1$ by checking whether $5 * T(n)$ (5 times the time needed to search up to depth $n$) would fit in our time budget, but I have seen 2 failure modes: (1) If there are many queens on the board, the branching factor can be very high so we take longer than expected; (2) if all moves at depth $n$ get fetched from the transposition table, this takes no time at all, so we vastly underestimate $T(n+1)$ and get stuck doing an extremely deep search. This bites us in the endgame a lot, as there might be a perpetual check, which indirectly forces us to search increasing depths (for example in this position: 8/4q1KP/8/8/3k4/8/8/8 w - - 21 77).

One approach I have seen is to simply stop the search after a set time limit. I used to think this was wasteful because we don't "complete" a certain depth, but since we do use the time to fill the transposition table with valuable information, it might be okay.

uci: I will add the engine version when I get around to making a build script. I did not know about UCI when starting this project, so if you check the engine output, you can see I am using a custom format to print the current best move. After seeing my engine battle in Arena, I can see the benefit of displaying the top move plus eval and comparing that to the opponent engine. I will add this.

[Gabor Szots]

Yikes at those results! Thank you for your diligent testing.

rules infraction (1.6%): I'm surprised by this as well, I found only one illegal move in my dataset but I made a mistake analyzing it. Presumably illegal moves happen due to hash collisions, so if we search a billion positions during a game we have about a 5% chance of collision (as (10**9)**2 * 2**-64 = 0.054). Maybe this can be fixed by checking a move for legality when fetching it from the transposition table, but I don't have a function for that.

A better approach might be to do some sort of aging, by also storing the ply at which the transposition was encountered so we don't use a transposition from the early game later when it is very unlikely to occur again, and more likely to be a collision.

time forfeit (8.9%): I had a much lower rate of time forfeits in my dataset (3.6%), although I was playing weaker opponents. When we finish searching depth $n$, we make a simple binary decision of whether to search depth $n+1$ by checking whether $5 * T(n)$ (5 times the time needed to search up to depth $n$) would fit in our time budget, but I have seen 2 failure modes: (1) If there are many queens on the board, the branching factor can be very high so we take longer than expected; (2) if all moves at depth $n$ get fetched from the transposition table, this takes no time at all, so we vastly underestimate $T(n+1)$ and get stuck doing an extremely deep search. This bites us in the endgame a lot, as there might be a perpetual check, which indirectly forces us to search increasing depths (for example in this position: 8/4q1KP/8/8/3k4/8/8/8 w - - 21 77).

One approach I have seen is to simply stop the search after a set time limit. I used to think this was wasteful because we don't "complete" a certain depth, but since we do use the time to fill the transposition table with valuable information, it might be okay.

uci: I will add the engine version when I get around to making a build script. I did not know about UCI when starting this project, so if you check the engine output, you can see I am using a custom format to print the current best move. After seeing my engine battle in Arena, I can see the benefit of displaying the top move plus eval and comparing that to the opponent engine. I will add this.

1. What is '$'?
2. I ran your engine under Arena in winboard mode.
3. Arena is notorious of time forfeits if an engine is sensitive to that. You may consider introducing a Move Overhead parameter.

[Yinuo Huang]

Hi, Avalanche author here. Regarding the time forfeit thing - unfortunately sometimes you have to “waste” a depth, if the engine has less than a certain amount of time (the Move Overhead parameter). Say 50ms; when time goes below 50ms we must immediately stop even if we are in the middle of the depth or we might lose on time or have little time to search next move. In Avalanche I basically divide the time by # of moves to go, and force that amount of time for each search. Some engines implement a soft limit and a hard limit (if we are in the middle of a depth after soft limit we should finish that depth, but if we are in the middle of a depth and we hit the hard limit we stop immediately). You can try and test different algorithms out
And regarding move collision - you should not directly use TT’s best move under any circumstances (as far as I know). You should always generate legal moves and sort the hashmove first.

[Jan-Frederik Konopka]

@Gabor: I haven't had any problems with latency on Arena, only on Lichess. I use '$' as a reference to math expressions in LaTeX.

@Yinuo: Regarding move collisions, I just implemented pseudo-legal move checking when probing the hash table, meaning we only execute the hash move if it appears consistent with our board state. This can still lead to illegal moves (king in check), but makes them much less likely. I try to save time in my engine by not generating any moves if we have a hash move, if we don't have a hash move I only generate captures, and only if absolutely necessary I will generate quiet moves. But your engine is much stronger, so what do I know

Regarding time use, I tried always stopping the search after using 1/20, 1/30, 1/40 of the remaining time, but in each case this led to a big drop in the engine's strength, so it looks like "finishing" depths is valuable. I guess I will try a hybrid approach where I generally try to finish depths and abort only if a depth is taking significantly longer than expected.

I've also noticed that my engine was able to win 80% of games over a set of 495 test games in December, and now the exact same binary is stuck around 72-75%, it's kind of crazy-making. Maybe that explains the poor results you see, but other than some sort of mix-up I have no explanation for this.

[Yinuo Huang]

Your approach might work Avalanche uses a legal-move generator so I never faced king in check problems. I also couldn't bother generating moves lazily, I just generate all legal moves at once and completely sort it lol. Partially generating moves is probably a much better technique. Good luck on development!
By the way, just a small tip, sometimes it's much easier to gain elo through better evaluation function than better search improvements

[Jan-Frederik Konopka]

Hi everyone, I'm excited to see that Jangine was finally put on the CCRL list! Thank you again for your time and effort testing my engine.

I am a bit puzzled as to why its rating (1915) falls so far below my expectations.

In my own gauntlet against a group of engines with an average rating of 1920 ("G1920"), Jangine scored 74.5%, which implies a rating of 2106:

Engine Score Ja 01: Jangine_2022-12-29_fe9c599e_linux64 82,0/110 .......... 02: Gully-maybe-2.16pl1 7,0/10 0111011=1= 03: Sissa-64-2.0 5,0/10 0=011=0=1= 04: Matmoi-7.15.0-cct 3,0/10 000==1==00 05: Rustic-alpha-3.0.0-linux-64-bit-popcnt 2,5/10 0001=00001 06: Heracles 0.6.16 2,0/10 0000000101 06: Sayuri-2018.05.23-linux-64bit 2,0/10 0000=10=00 06: Fatalii-v0.3.1-x86_64-unknown-linux-musl 2,0/10 0001000001 09: Deepov-0.4 1,5/10 00001000=0 10: ALChess_184_x64 1,0/10 0100000000 10: Tinman-v0.4.0 1,0/10 000=00000= 10: Weasel-1.0.2-Beta 1,0/10 0000=0000= 110 games played / Tournament is finished Name of the tournament: 2022-12-29T18:00 Jangine Gauntlet v2022.12.29 2+1 Site/ Country: ssd, Germany Level: Blitz 2/1 Hardware: AMD Ryzen 5 3600 6-Core Processor 3739.004 MHz with 32.177 MB Memory Operating system: Linux 4.15.0-45-generic x86_64

That makes it hard for me to believe Jangine is 1915 rated. Even if I (say) downloaded the wrong version of an engine, that wouldn't skew the results so much. Are you passing any special parameters to the engines? (I just used the default settings.) Do most engines just have crappy Linux builds? (Or is my Windows build crappy?) I would be grateful to hear your insight.

(I also had some difficulty finding engines to play against, most homepages were dead links, and most of the remaining ones didn't have Linux builds.)

In another gauntlet against a group of engines with an average rating of 2315 ("G2315"), Jangine scored 18.5%, implying a rating of 2057.

---

Either way, I am announcing the release of version 2023-01-15 today, and would be happy if it could be added to your testing program. I am now checking pseudo-legality of transposition table moves, which should get rid of illegal moves, and which allows me to store more positions in the transposition table safely. This also had a positive impact on losing on time, although I haven't added the stop-the-world fix to completely prevent time losses (yet).

I am guiding a rating gain of +110 (G2315: 18.5% -> 30%, 2057 -> 2168).

[Gabor Szots]

Hi Jan-Frederik,

A possible explanation might be that I used Jangine in WinBoard mode. WB and UCI support might be different.

To check, I have played a 40-game match between their UCI and WB versions under Cute Chess. TC was 30s+0.2s. Here are the results:

Score of Jangine 2022-12-29 64-bit UCI vs Jangine 2022-12-29 64-bit WinBoard: 31 - 9 - 0 [0.775]
... Jangine 2022-12-29 64-bit UCI playing White: 15 - 5 - 0 [0.750] 20
... Jangine 2022-12-29 64-bit UCI playing Black: 16 - 4 - 0 [0.800] 20
... White vs Black: 19 - 21 - 0 [0.475] 40
Elo difference: 214.8 +/- 142.8, LOS: 100.0 %, DrawRatio: 0.0 %
40 of 40 games finished.

I'm convinced. Are you?

[Jan-Frederik Konopka]

Hi Gabor,

thanks for your analysis. I was unable to verify your results in my own testing (again on Arena+Linux):

Engine Score Ja Ja S-B 1: Jangine_2022-12-29_fe9c599e_linux64 53,5/100 .................................................................................................... =1===1100=0=000011111111=10101=1=10==10100==011=001=====01===0111=1=01000==1011==10=0101010=01=10101 2487,7 2: Jangine_2022-12-29_fe9c599e_linux64 WINBOARD 46,5/100 =0===0011=1=111100000000=01010=0=01==01011==100=110=====10===1000=0=10111==0100==01=1010101=10=01010 .................................................................................................... 2487,7 100 games played / Tournament is finished Name of the tournament: 2023-01-16T19:00 Jangine UCI VS WINBOARD VS WINBOARD_OLD Site/ Country: ssd, Germany Level: Blitz 0:30/0,2 Hardware: AMD Ryzen 5 3600 6-Core Processor 3739.004 MHz with 32.177 MB Memory Operating system: Linux 4.15.0-45-generic x86_64 PGN-File: /home/jan/Tournaments/2023-01-16T19_00 Jangine UCI VS WINBOARD VS WINBOARD_OLD.pgn

I don't really "trust" self-play tournaments too much anyway because the moves are almost the same, so wild swings can happen.

I don't understand how UCI vs Winboard (Xboard) could make a difference. I do everything exactly the same in both modes, the only difference is I multiply the time given by the time command by 10 as it is supposed to be centiseconds (versus wtime/btime in UCI which is milliseconds). The engine seem to play on normal speed in Arena and Lichess though, so I can't be off by a factor of 10.

Are you absolutely sure you didn't pick the wrong binary? Or set any additional parameters? Or set the time given to it to below 100%?

[Jan-Frederik Konopka]

I went hunting through the published PGN files to check for a case of mistaken identity, and the moves all matched with my local engine output, so it likely is some version of Jangine.

I noticed the openings coming from an opening book, and thought maybe the startpos code was different for Winboard. My theory is that Jangine responds wrongly or not at all, leading either to a time penalty, or time spent thinking on opening moves that don't happen in the game. I tried to test this, but openings from a PGN file worked fine in both modes, and openings from an EPD file didn't work in UCI, but worked fine in Winboard.

Could you kindly test this by taking a look at one of the games and seeing if the Winboard version either receives penalty time at the start, or has other unusual time use?

The only other explanation I can think of is it being an older version of Jangine.

[Gabor Szots]

I use this version: https://github.com/xjcl/jangine/release ... dows64.exe

If it were an older version it would have a different name in the list. Or I don't understand what you mean.

[Gabor Szots]

Could you kindly test this by taking a look at one of the games and seeing if the Winboard version either receives penalty time at the start, or has other unusual time use?

I don't know how to do that. Maybe I don't understand what you are asking from me. Why should it receive a penalty? Also, why should that make a 200 Elo difference?

[Jan-Frederik Konopka]

Sorry if I was unclear. Basically I was wondering if there was anything unusual when you watch a "Jangine UCI" vs "Jangine Winboard" match. The engines should move at around the same speed. If the Winboard version receives a time penalty or starts up slower, it will have less time to think during the game, meaning less search depth, which would explain the lower-quality moves.

[Gabor Szots]

I'm going to do another test run in the next couple of days. I will submit an Arena debug for you to see if there is something wrong. But in fact you can do it yourself, I think Arena can be run through Wine under Linux.

[Jan-Frederik Konopka]

Thank you a lot for offering to do testing. I actually just repeated the experiment on Windows, and I was able to reproduce the Winboard version performing worse, and was then able to locate the issue using the arena.debug file.

In my code I print out the state of the board as ASCII art (UTF-8 art to be specific). What's wrong with this log?:

2023-01-18 21:50:10,385<--2:INPUT: f7e6 2023-01-18 21:50:10,400<--2: 8 . . ♜ . ♝ . ♚ . 2023-01-18 21:50:10,415<--2: 7 . ♟ ♟ . . ♛ ♟ . 2023-01-18 21:50:10,430<--2: 6 ♟ . . . . . . ♟ 2023-01-18 21:50:10,445<--2: 5 . . ♖ . ♘ ♟ . . 2023-01-18 21:50:10,447<--2: 4 . . . ♙ . . . . 2023-01-18 21:50:10,460<--2: 3 ♕ . . . . . . . 2023-01-18 21:50:10,475<--2: 2 ♙ ♙ . . . ♙ ♙ ♙ 2023-01-18 21:50:10,490<--2: 1 . . . . . . ♔ . 2023-01-18 21:50:10,490<--2: a b c d e f g h 2023-01-18 21:50:10,505<--2: 8 . . ♜ . ♝ . ♚ . 2023-01-18 21:50:10,520<--2: 7 . ♟ ♟ . . . ♟ . 2023-01-18 21:50:10,535<--2: 6 ♟ . . . ♛ . . ♟ 2023-01-18 21:50:10,537<--2: 5 . . ♖ . ♘ ♟ . . 2023-01-18 21:50:10,550<--2: 4 . . . ♙ . . . . 2023-01-18 21:50:10,565<--2: 3 ♕ . . . . . . . 2023-01-18 21:50:10,580<--2: 2 ♙ ♙ . . . ♙ ♙ ♙ 2023-01-18 21:50:10,595<--2: 1 . . . . . . ♔ . 2023-01-18 21:50:10,610<--2: a b c d e f g h 2023-01-18 21:50:10,625<--2:Setting piece-square tables to middlegame

My Winboard engine is losing ~240 milliseconds seconds per move just printing out the board twice.

For comparison, it takes 1-3 milliseconds on Linux.

2023-01-16 23:22:47,808<--1:INPUT: f4e3 2023-01-16 23:22:47,808<--1: 8 . . . . . . . . 2023-01-16 23:22:47,808<--1: 7 . . . . . . . . 2023-01-16 23:22:47,808<--1: 6 . . ♔ ♟ . . . . 2023-01-16 23:22:47,808<--1: 5 . . . ♙ . . . . 2023-01-16 23:22:47,808<--1: 4 . . ♙ . . ♚ . . 2023-01-16 23:22:47,808<--1: 3 . . . ♗ . ♟ . . 2023-01-16 23:22:47,808<--1: 2 . . . . . . . . 2023-01-16 23:22:47,808<--1: 1 . . . . . . . . 2023-01-16 23:22:47,808<--1: a b c d e f g h 2023-01-16 23:22:47,808<--1: 8 . . . . . . . . 2023-01-16 23:22:47,809<--1: 7 . . . . . . . . 2023-01-16 23:22:47,809<--1: 6 . . ♔ ♟ . . . . 2023-01-16 23:22:47,809<--1: 5 . . . ♙ . . . . 2023-01-16 23:22:47,809<--1: 4 . . ♙ . . . . . 2023-01-16 23:22:47,809<--1: 3 . . . ♗ ♚ ♟ . . 2023-01-16 23:22:47,809<--1: 2 . . . . . . . . 2023-01-16 23:22:47,809<--1: 1 . . . . . . . . 2023-01-16 23:22:47,809<--1: a b c d e f g h 2023-01-16 23:22:47,809<--1:Setting piece-square tables to endgame

This is extremely frustrating. Even a basic function such as subprocess communication is hopelessly broken on Windows. It doesn't really help that the Windows version of Arena hasn't been updated since 2015. For clarification, I am losing a lot of time on just printing the output in both UCI (~300 ms per move) and Winboard (~550 ms per move) modes. It is just that I print more output in Winboard mode, leading to the difference between modes.

I guess this leaves me with no other option except for removing all output except for the final move. Even just communicating the top move at each depth for depth 10 would lose me around ~150 milliseconds per move, or about 15 seconds for a 100-move match. Does anyone know a way to print output in C/C++ without losing a lot of time? I am also thinking about other engines, some of which print the current best move many times per depth, potentially losing 100s of rating points. Either way, I will try to create a fix this weekend.

Thank you again for your time.

[Gabor Szots]

I don't know how others do it but there must be a way. Do you have to print the board at all?

Anyway, good luck with finding a solution.

[Jan-Frederik Konopka]

I will just not print the boards at all. I'm hoping there is some sort of magic with turning off output stream flushing that could work. But since I observed 2 of the pre-installed Arena engines having the same problem, I am not too optimistic.

I think this is a significant problem with the testing method. Jangine was losing 190 elo simply by printing more than other engines, and now that I know about this issue, I will be printing less than other engines, which will very likely improve elo further, turning my undeserved disadvantage into an undeserved advantage, which does not really feel fair either way. This should be a contest of engine cleverness, and not a contest of whether the developer knows about obscure Arena bugs.

Switching to Linux is likely too big of a hassle, but maybe switching to a different Windows chess GUI could be considered. I would be interested to hear the opinion of other developers on this issue.

[Gabor Szots]

I will just not print the boards at all. I'm hoping there is some sort of magic with turning off output stream flushing that could work. But since I observed 2 of the pre-installed Arena engines having the same problem, I am not too optimistic.

I think this is a significant problem with the testing method. Jangine was losing 190 elo simply by printing more than other engines, and now that I know about this issue, I will be printing less than other engines, which will very likely improve elo further, turning my undeserved disadvantage into an undeserved advantage, which does not really feel fair either way. This should be a contest of engine cleverness, and not a contest of whether the developer knows about obscure Arena bugs.

Switching to Linux is likely too big of a hassle, but maybe switching to a different Windows chess GUI could be considered. I would be interested to hear the opinion of other developers on this issue.

You have opened up my eyes to the problem, only I don't know how to solve it. Of the GUI's I use only Arena can handle WB engines. That will put all of them in disadvantage. In the future, using Arena, I may run all engines as UCI but that will not do any good to WB engines.
BTW, I don't understand why UCI engines are not affected.

No Windows chess tester will switch to Linux, that's for sure. I tried it once but I did not like it. Linux is for programmers, not everyday testers.

If you want to consult other authors you may sign up to the CCC forums where there is a section on technical problems.

[Gabor Szots]

Just for the sake of experiment, I did the same test with KnightX, also a UCI/WB hybrid.

Score of KnightX_3.5-x64 UCI vs KnightX_3.5-x64 WB: 14 - 8 - 18 [0.575]
... KnightX_3.5-x64 UCI playing White: 8 - 5 - 7 [0.575] 20
... KnightX_3.5-x64 UCI playing Black: 6 - 3 - 11 [0.575] 20
... White vs Black: 11 - 11 - 18 [0.500] 40
Elo difference: 52.5 +/- 81.4, LOS: 90.0 %, DrawRatio: 45.0 %
40 of 40 games finished.

[Gabor Szots]

And here is Cheese:

Score of Cheese_3.1.1-x64 UCI vs Cheese_3.1.1-x64 WB: 10 - 6 - 24 [0.550]
... Cheese_3.1.1-x64 UCI playing White: 7 - 1 - 12 [0.650] 20
... Cheese_3.1.1-x64 UCI playing Black: 3 - 5 - 12 [0.450] 20
... White vs Black: 12 - 4 - 24 [0.600] 40
Elo difference: 34.9 +/- 68.7, LOS: 84.1 %, DrawRatio: 60.0 %
40 of 40 games finished.

These matches were all played at 30+0.2.

It seems these two do (or not do) something different to what you do and are less sensitive to the UCI-WB difference.

[Jan-Frederik Konopka]

Thanks for your understanding. The problem here is not about UCI vs Winboard, the problem is that engines lose a lot of time if they attempt to write output. Since every engine chooses to write a different amount of output, they are punished differently. Usually you want an engine to write a lot of output so you can see which move it is currently considering, but here it is bad to write a lot because it loses an unreasonable amount of clock time for no good reason.

My engine wrote more output in Winboard mode compared to UCI, mostly just due to my own sloppiness. It is not a problem with UCI vs Winboard generally.

If you want to do a test, you would need an engine with a "verbosity" setting. "Verbosity" means how much output it will try to write. The low-verbosity setting should outperform the high-verbosity setting.

[Gabor Szots]

As promised, here is a bit of an Arena debug. I can see much less wasted time here than in your example debug but it might not matter.