new engine: Thrawn

[fefty]

Hi,

I've coded an open source uci compliant chess engine and would like it to be tested against other engines to determine its rating: https://github.com/feftywacky/Thrawn

engine name: Thrawn
name: Feiyu Lin

Thrawn v2.0
evaluation type: nnue
estimated rating: 2500+?
executable: https://github.com/feftywacky/Thrawn/releases/tag/v2.0

... and if there's time and availability
Thrawn v1.1
evaluation type: hand crafted eval
estimated rating: 1800?
executable: https://github.com/feftywacky/Thrawn/releases/tag/v1.1

Thank you for your time and consideration! Please reach out if there are any inquiries

[Graham Banks]

Does it use its own NNUE?

[fefty]

no it does not.

if you refer to the README in the github repo, at the bottom I referenced the specific nnue dataset I've found online: https://tests.stockfishchess.org/nns

[Gabor Szots]

To assess its strength I ran a small test match under Cute Chess. It made an illegal move in all but the first game. It seems it often does not know which side it is playing, but even more weird things happen (see game 9). Here is the PGN.

[Event "Thrawn_1.1"]
[Site "?"]
[Date "2024.01.01"]
[Round "1"]
[White "Thrawn 1.1 64-bit"]
[Black "Athena 1.0.1 64-bit"]
[Result "1-0"]
[ECO "A46"]
[GameDuration "00:05:09"]
[GameEndTime "2024-01-01T09:42:57.537 Közép-európai téli idő "]
[GameStartTime "2024-01-01T09:37:48.493 Közép-európai téli idő "]
[Opening "Queen's pawn game"]
[PlyCount "123"]
[TimeControl "120+1"]

1. d4 {book} Nf6 {book} 2. Nf3 {book} e6 {book} 3. g3 {book} b5 {book}
4. Bg2 {book} Bb7 {book} 5. O-O {book} c5 {book} 6. dxc5 {book} Bxc5 {book}
7. Bf4 {+0.02/10 5.0s} O-O {6.0s} 8. Nc3 {0.00/10 4.9s} b4 {5.8s}
9. Nb5 {+0.32/10 4.7s} Nc6 {5.6s} 10. Bd6 {0.00/11 4.6s} Qb6 {5.3s}
11. Bxc5 {0.00/11 4.5s} Qxc5 {5.1s} 12. Nd6 {0.00/12 4.4s} Ba6 {4.9s}
13. Re1 {+0.09/10 4.3s} Nd5 {4.7s} 14. Ne4 {+0.40/11 4.1s} Qe7 {4.5s}
15. a3 {+0.24/11 4.0s} Bb7 {4.3s} 16. axb4 {+0.48/10 3.9s} Qxb4 {4.2s}
17. Qd3 {+0.53/9 3.8s} d6 {4.0s} 18. c3 {+1.44/10 3.7s} Qb6 {3.9s}
19. Nxd6 {+1.62/10 3.7s} Rfd8 {3.7s} 20. Nxb7 {+1.53/10 3.6s} Qxb7 {3.6s}
21. Ng5 {+1.67/9 3.5s} h6 {3.5s} 22. Qh7+ {+2.25/11 3.4s} Kf8 {3.3s}
23. Ne4 {+2.28/10 3.3s} Nde7 {3.2s} 24. Qh8+ {+2.61/10 3.2s} Ng8 {3.1s}
25. b4 {+2.50/9 3.2s} Rac8 {3.0s} 26. Red1 {+2.41/9 3.1s} Qc7 {2.9s}
27. Nc5 {+2.38/9 3.0s} a5 {2.8s} 28. Rac1 {+2.27/10 3.0s} axb4 {2.7s}
29. Bxc6 {+1.76/11 2.9s} Qxc6 {2.6s} 30. cxb4 {+1.83/11 2.8s} e5 {2.6s}
31. e4 {0.00/10 2.8s} Qb5 {2.5s} 32. Rb1 {+1.31/10 2.7s} Rxd1+ {2.4s}
33. Rxd1 {+1.77/10 2.7s} Qe2 {2.3s} 34. Ra1 {+1.97/11 2.6s} Rd8 {2.3s}
35. Kg2 {+1.94/10 2.5s} Qd2 {2.2s} 36. Ra7 {+1.83/11 2.5s} Qd6 {2.1s}
37. h4 {+1.82/11 2.4s} Qc6 {2.1s} 38. h5 {+1.76/10 2.4s} Qc8 {2.0s}
39. Kf3 {+1.66/11 2.3s} Qh3 {2.0s} 40. Ke3 {+1.29/11 2.3s} Qg4 {1.9s}
41. Rb7 {+1.60/11 2.3s} Qg5+ {1.9s} 42. Kf3 {+1.65/11 2.2s} Qxh5+ {1.8s}
43. Kg2 {+2.06/11 2.2s} Qg4 {1.8s} 44. Rc7 {+2.06/10 2.1s} Rd1 {1.8s}
45. b5 {+3.09/10 2.1s} Rd6 {1.7s} 46. b6 {+6.89/11 2.1s} Rd3 {1.7s}
47. Nxd3 {+11.70/13 2.0s} Qxe4+ {1.6s} 48. Kh2 {+8.54/10 2.0s} Qxd3 {1.6s}
49. b7 {+12.44/14 2.0s} Qb5 {1.6s} 50. Rc8+ {+14.04/14 1.9s} Ke7 {1.6s}
51. b8=Q {+14.19/13 1.9s} Qxb8 {1.5s} 52. Rxb8 {+16.11/13 1.9s} Nf6 {1.5s}
53. Qf8+ {+16.25/12 1.8s} Ke6 {1.5s} 54. Qxg7 {+16.30/12 1.8s} Nd5 {1.5s}
55. Re8+ {+16.86/11 1.8s} Kd6 {1.4s} 56. Qxe5+ {+17.59/12 1.8s} Kd7 {1.4s}
57. Rf8 {+17.94/12 1.7s} Kc6 {1.4s} 58. Rc8+ {+M9/14 1.7s} Kb6 {1.4s}
59. Qxd5 {+M7/13 1.7s} f5 {1.4s} 60. Rc6+ {+M5/15 1.7s} Kb7 {1.3s}
61. Qb5+ {+M3/14 1.6s} Ka7 {1.3s} 62. Ra6# {+M1/16 1.6s, White mates} 1-0

[Event "Thrawn_1.1"]
[Site "?"]
[Date "2024.01.01"]
[Round "2"]
[White "Athena 1.0.1 64-bit"]
[Black "Thrawn 1.1 64-bit"]
[Result "1-0"]
[ECO "A45"]
[GameDuration "00:00:10"]
[GameEndTime "2024-01-01T09:43:09.441 Közép-európai téli idő "]
[GameStartTime "2024-01-01T09:42:58.552 Közép-európai téli idő "]
[Opening "Queen's pawn game"]
[PlyCount "13"]
[Termination "illegal move"]
[TimeControl "120+1"]

1. d4 {book} Nf6 {book} 2. Nc3 {book} d5 {book} 3. Bg5 {book} Nbd7 {book}
4. e3 {book} h6 {book} 5. Bh4 {book} e6 {book} 6. a3 {book} Be7 {book}
7. Bd3 {6.0s, Black makes an illegal move: h4g3} 1-0

[Event "Thrawn_1.1"]
[Site "?"]
[Date "2024.01.01"]
[Round "3"]
[White "Thrawn 1.1 64-bit"]
[Black "Athena 1.0.1 64-bit"]
[Result "0-1"]
[ECO "B10"]
[GameDuration "00:01:05"]
[GameEndTime "2024-01-01T09:44:15.735 Közép-európai téli idő "]
[GameStartTime "2024-01-01T09:43:10.449 Közép-európai téli idő "]
[Opening "Caro-Kann defense"]
[PlyCount "24"]
[Termination "illegal move"]
[TimeControl "120+1"]

1. e4 {book} c6 {book} 2. Nf3 {book} d5 {book} 3. Nc3 {book} Bg4 {book}
4. Be2 {book} e6 {book} 5. h3 {book} Bh5 {book} 6. d4 {book} Nd7 {book}
7. Bf4 {+0.58/9 5.0s} Bb4 {6.0s} 8. e5 {+0.72/9 4.9s} Bxc3+ {5.8s}
9. bxc3 {+0.66/9 4.7s} Bxf3 {5.6s} 10. Bxf3 {+0.83/11 4.6s} Ne7 {5.3s}
11. O-O {+0.75/10 4.5s} O-O {5.1s} 12. Bg5 {+0.79/10 4.4s}
h6 {4.9s, White makes an illegal move: h6g5} 0-1

[Event "Thrawn_1.1"]
[Site "?"]
[Date "2024.01.01"]
[Round "4"]
[White "Athena 1.0.1 64-bit"]
[Black "Thrawn 1.1 64-bit"]
[Result "1-0"]
[ECO "B40"]
[GameDuration "00:01:00"]
[GameEndTime "2024-01-01T09:45:17.362 Közép-európai téli idő "]
[GameStartTime "2024-01-01T09:44:16.745 Közép-európai téli idő "]
[Opening "Sicilian defense"]
[PlyCount "23"]
[Termination "illegal move"]
[TimeControl "120+1"]

1. e4 {book} c5 {book} 2. Nf3 {book} e6 {book} 3. d4 {book} cxd4 {book}
4. Nxd4 {book} Qb6 {book} 5. Nb3 {book} Qc7 {book} 6. g3 {book} Nf6 {book}
7. Nc3 {6.0s} Qxc3+ {+0.92/10 5.0s} 8. bxc3 {5.8s} Nxe4 {-6.00/11 4.8s}
9. Qd3 {5.6s} d5 {-5.85/11 4.7s} 10. Bg2 {5.3s} Nd7 {-5.84/10 4.6s}
11. Bxe4 {5.1s} dxe4 {-5.97/12 4.5s}
12. Qxe4 {4.9s, Black makes an illegal move: c1f4} 1-0

[Event "Thrawn_1.1"]
[Site "?"]
[Date "2024.01.01"]
[Round "5"]
[White "Thrawn 1.1 64-bit"]
[Black "Athena 1.0.1 64-bit"]
[Result "0-1"]
[ECO "A41"]
[GameDuration "00:00:05"]
[GameEndTime "2024-01-01T09:45:23.643 Közép-európai téli idő "]
[GameStartTime "2024-01-01T09:45:18.368 Közép-európai téli idő "]
[Opening "Queen's Pawn"]
[PlyCount "12"]
[Termination "illegal move"]
[TimeControl "120+1"]

1. d4 {book} d6 {book} 2. g3 {book} g6 {book} 3. Bg2 {book} Bg7 {book}
4. e4 {book} Nc6 {book} 5. Ne2 {book} e5 {book} 6. dxe5 {book}
dxe5 {book, White makes an illegal move: d8d1} 0-1

[Event "Thrawn_1.1"]
[Site "?"]
[Date "2024.01.01"]
[Round "6"]
[White "Athena 1.0.1 64-bit"]
[Black "Thrawn 1.1 64-bit"]
[Result "1-0"]
[ECO "E70"]
[GameDuration "00:00:21"]
[GameEndTime "2024-01-01T09:45:46.191 Közép-európai téli idő "]
[GameStartTime "2024-01-01T09:45:24.656 Közép-európai téli idő "]
[Opening "King's Indian"]
[PlyCount "15"]
[Termination "illegal move"]
[TimeControl "120+1"]
[Variation "4.e4"]

1. d4 {book} Nf6 {book} 2. c4 {book} g6 {book} 3. Nc3 {book} Bg7 {book}
4. e4 {book} d6 {book} 5. Bd3 {book} Nfd7 {book} 6. Be3 {book} O-O {book}
7. Nf3 {6.0s} Nf6 {-0.85/9 5.0s}
8. O-O {5.8s, Black makes an illegal move: d3e4} 1-0

[Event "Thrawn_1.1"]
[Site "?"]
[Date "2024.01.01"]
[Round "7"]
[White "Thrawn 1.1 64-bit"]
[Black "Athena 1.0.1 64-bit"]
[Result "0-1"]
[ECO "A40"]
[GameDuration "00:00:26"]
[GameEndTime "2024-01-01T09:46:13.904 Közép-európai téli idő "]
[GameStartTime "2024-01-01T09:45:47.202 Közép-európai téli idő "]
[Opening "Queen's pawn"]
[PlyCount "16"]
[Termination "illegal move"]
[TimeControl "120+1"]

1. d4 {book} e6 {book} 2. Nf3 {book} f5 {book} 3. h3 {book} Nf6 {book}
4. g4 {book} d5 {book} 5. Rg1 {book} Ne4 {book} 6. c3 {book} Nc6 {book}
7. Bf4 {0.00/11 5.0s} fxg4 {6.0s} 8. hxg4 {+0.16/11 4.8s}
Bd6 {5.8s, White makes an illegal move: d6f4} 0-1

[Event "Thrawn_1.1"]
[Site "?"]
[Date "2024.01.01"]
[Round "8"]
[White "Athena 1.0.1 64-bit"]
[Black "Thrawn 1.1 64-bit"]
[Result "1-0"]
[ECO "B22"]
[GameDuration "00:00:41"]
[GameEndTime "2024-01-01T09:46:56.671 Közép-európai téli idő "]
[GameStartTime "2024-01-01T09:46:14.909 Közép-európai téli idő "]
[Opening "Sicilian"]
[PlyCount "19"]
[Termination "illegal move"]
[TimeControl "120+1"]
[Variation "Alapin's variation (2.c3)"]

1. e4 {book} c5 {book} 2. c3 {book} d5 {book} 3. exd5 {book} Qxd5 {book}
4. d4 {book} g6 {book} 5. Nf3 {book} Bg7 {book} 6. Be3 {book} Nh6 {book}
7. Nbd2 {6.0s} Nf5 {-0.51/10 5.0s} 8. Bc4 {5.8s} Qc6 {-0.35/10 4.9s}
9. O-O {5.6s} Nxe3 {0.00/11 4.7s}
10. fxe3 {5.3s, Black makes an illegal move: c4f7} 1-0

[Event "Thrawn_1.1"]
[Site "?"]
[Date "2024.01.01"]
[Round "9"]
[White "Thrawn 1.1 64-bit"]
[Black "Athena 1.0.1 64-bit"]
[Result "0-1"]
[ECO "B10"]
[GameDuration "00:00:56"]
[GameEndTime "2024-01-01T09:47:53.954 Közép-európai téli idő "]
[GameStartTime "2024-01-01T09:46:57.675 Közép-európai téli idő "]
[Opening "Caro-Kann"]
[PlyCount "22"]
[Termination "illegal move"]
[TimeControl "120+1"]
[Variation "Anti-anti-Caro-Kann defense"]

1. e4 {book} c6 {book} 2. c4 {book} d5 {book} 3. exd5 {book} cxd5 {book}
4. cxd5 {book} Nf6 {book} 5. Bb5+ {book} Nbd7 {book} 6. Nc3 {book} g6 {book}
7. d4 {+0.76/10 5.0s} a6 {6.0s} 8. Bd3 {+0.75/10 4.9s} Nb6 {5.8s}
9. Bf4 {+0.56/10 4.7s} Nbxd5 {5.6s} 10. Nxd5 {+0.40/11 4.6s} Nxd5 {5.3s}
11. Be5 {0.00/12 4.5s} f6 {5.1s, White makes an illegal move: e4g6} 0-1

[Event "Thrawn_1.1"]
[Site "?"]
[Date "2024.01.01"]
[Round "10"]
[White "Athena 1.0.1 64-bit"]
[Black "Thrawn 1.1 64-bit"]
[Result "1-0"]
[ECO "A02"]
[GameDuration "00:00:10"]
[GameEndTime "2024-01-01T09:48:05.858 Közép-európai téli idő "]
[GameStartTime "2024-01-01T09:47:54.964 Közép-európai téli idő "]
[Opening "Bird's Opening"]
[PlyCount "13"]
[Termination "illegal move"]
[TimeControl "120+1"]

1. f4 {book} c5 {book} 2. g3 {book} Nc6 {book} 3. Bg2 {book} d5 {book}
4. Nf3 {book} g6 {book} 5. e3 {book} Bg7 {book} 6. d4 {book} cxd4 {book}
7. Nxd4 {6.0s, Black makes an illegal move: d4c6} 1-0

[fefty]

Hi,

Thank you for the feedback! And I appreciate your patience!

I've looked into the code and found out it to be a bug with the uci_loop and how I was parsing inputs.

I've tested Thrawn_v2.0 and Thrawn_v1.1 by also using CuteChess Tournament mode: 4 games, 15sec + 0.1sec time control

I have implemented a fix for this issue for Thrawn_v2.0 and Thrawn_v1.1; The updated executable and source code has been updated in the releases:
https://github.com/feftywacky/Thrawn/releases/tag/v2.0
https://github.com/feftywacky/Thrawn/releases/tag/v1.1

============================================================

Thrawn_v2.0 vs Lynx.Cli (rated 2497 in 2+1 blitz by ccrl):
Score of Thrawn_v2.0 vs Lynx.Cli: 4 - 0 - 0 [1.000]
... Thrawn_v2.0 playing White: 2 - 0 - 0 [1.000] 2
... Thrawn_v2.0 playing Black: 2 - 0 - 0 [1.000] 2
... White vs Black: 2 - 2 - 0 [0.500] 4
Elo difference: inf +/- nan, LOS: 97.7 %, DrawRatio: 0.0 %
SPRT: llr 0 (0.0%), lbound -inf, ubound inf
4 of 4 games finished.

Player: Thrawn_v2.0
"Win: Black mates": 2
"Win: White mates": 2
Player: Lynx.Cli
"Loss: Black mates": 2
"Loss: White mates": 2

============================================================

Thrawn_v1.0 vs sayuri (rated 1856 in 2+1 blitz by ccrl)
Elo difference: 88.7 +/- 172.5, LOS: 84.1 %, DrawRatio: 75.0 %
SPRT: llr 0 (0.0%), lbound -inf, ubound inf
4 of 4 games finished.

Player: Thrawn_v1.1
"Draw by 3-fold repetition": 2
"Draw by insufficient mating material": 1
"Win: White mates": 1
Player: sayuri
"Draw by 3-fold repetition": 2
"Draw by insufficient mating material": 1
"Loss: White mates": 1

============================================================

Note that for Thrawn_v2.0, the nnue file must be placed in the same directory as the executable or it will play random moves.

Thank you for your time and consideration!
Feel free to reach out again if any other issues arises

[fefty]

I've found another bug where my engine would promote to a knight instead of a queen when clearly promoting to a queen is a better move.

I've implemented a fix and updated the exe and source code for the release: https://github.com/feftywacky/Thrawn/releases/tag/v2.0

I should've made another release (v2.1) addressing the bug fix above and the previous one, but unfortunately, at that time I somehow decide to update the release directly (may cause a bit of confusion).

Sorry if this caused any inconvenience and thank you for your consideration.

[Gabor Szots]

It's OK as I am going to test v1.1 which does not have that bug (if I understood you correctly).

[fefty]

I didn't update v1.1 with the second bug fix. However, I don't think v1.1 had this issue since it used a handcraft evaluation instead of nnue, so it should be good.

I speculate the nnue version had that bug because during the endgame the nnue gave knight promotions a extremely high score (and knight promotion was searched first), which exceeded the score threshold greatly, and so the rest of the other promotions were pruned out during search.

[fefty]

Hi, I hope you're doing well,

I'm just wondering if there's an update for the evaluation of my engine.
I'm especially curious for the rating of v2.0, and wondering if this version could be listed?

Understanding the ongoing evaluations for numerous engines, I appreciate your team's efforts and will patiently await the results.

Thank you!

[Gabor Szots]

Hi, I hope you're doing well,

I'm just wondering if there's an update for the evaluation of my engine.
I'm especially curious for the rating of v2.0, and wondering if this version could be listed?

Understanding the ongoing evaluations for numerous engines, I appreciate your team's efforts and will patiently await the results.

Thank you!

Testing of version 1.1 of your engine has not finished yet. As many other engines are waiting to be tested it is currently unlikely that we'll have the resources to test your engine again in such a short time.
BTW, it looks you have turned your attention to NNUE evaluation. Using a Stockfish net is a good starting point towards using a self-developed net, although I am sure you haven't yet exploited all your possibilities with HCE evaluation.

Good luck!

[fefty]

I feel like I currently lack the knowledge to developed a well scaled HCE evaluation. As a result, I am focusing on NNUE. I've found a few NN training libraries online and I'm attempting to train my own NNUE.

In the future, I planning to release a new version incorporating my own NNUE and even perhaps, a hybrid model between HCE and NNUE depending on position and game state.

Just out of curiosity, would you have an estimate of when v2.0 will be tested for an approximate rating/elo? This would give me solid insights during the production of a self-developed net, and how it compares with other engines (and their evaluation methods) within that elo margin.

Regardless, I appreciate your advice and feedback! Feel free to let me know if there are any inquiries or issues about the testing of my engine.
Thank you!

[Gabor Szots]

Just out of curiosity, would you have an estimate of when v2.0 will be tested for an approximate rating/elo?

I can only speak for myself, not for other testers. I can't see myself testing v2.0 in the near future, in fact ever.

Let me quote what I once wrote to another author:

I myself also allow NNUE, I disable only the SF ones. I personally don't find it interesting to test such engines. On our list you can find Coiled (no NNUE) with which I did the same. And there were engines which I did not test at all for the same reason. The only exception I made was with BBC 1.4 a couple of years ago when NNUE was new to me and I was simply curious how much strength can be gained by using an SF net instead of traditional evaluation.
I could as well say I would like to encourage authors to develop their own unique nets (or stick with HCE). Of course using an SF net could be a step in that direction but I would rather skip that step in my testing.