New attempt at 8-man and beyond

Endgame analysis using tablebases, EGTB generation, exchange, sharing, discussions, etc..
Post Reply
koistinen
Posts: 92
Joined: Fri May 02, 2008 7:59 pm
Sign-up code: 0
Location: Stockholm

New attempt at 8-man and beyond

Post by koistinen »

I got some good advice here which I didn't take, but now I am trying to do better.
To do so I am starting with writing a design document: https://github.com/Koistinen/Endgame/bl ... r/algo.txt
The goal is to avoid optimizing much except what is needed to fit in ram.
In the beginning extra disk i/o is allowed in order to keep down the complexity of the algorithm.
Once we have working code, improving performance should be relatively easy as testing is possible.
I have gone through a few iterations of debugging the design document, but there might still be something wrong with it, if you can find faults, please tell.

Code: Select all

Goals of the basic algorithm
It should be easy to understand and turn into code.
It should not make adding optimizations difficult,
i.e. no premature optimizations.

50 move rule applies.

Motivation
It is the quickest and is simple to extend to infinity.
Modifying to distance to mate or distance to conversion is not difficult.


Black is the losing side.

Motivation
It is the traditional side to win in chess problems.
Only computing wins for one side means fewer bits are needed at a time.
It is easy to flip colours.
Once both sides are computed, they can be combined into a single table for the
storage benefits that might be there.


Indexing index of the square in octal
  A  B  C  D  E  F  G  H
8 72 02 12 22 23 13 03 73
7 06 62 32 42 43 33 63 07
6 16 36 66 52 53 67 47 17
5 26 46 56 76 77 57 47 27
4 24 44 54 74 75 55 45 25
3 14 34 64 50 51 65 45 15
2 04 60 30 40 41 31 61 05
1 70 00 10 20 21 11 01 71

Motivation
Indexing the squares this way keeps the first digit constant when mirroring.
a1 and d4 sharing the same first digit allows a minor optimization and does not
make the system harder to understand.


Without pawns the black king is indexed in decimal with:
  A B C D
4       9
3     7 8
2   4 5 6
1 0 1 2 3

Motivation
It is simple and adds 25% to the size of the table.
Optimizing for the mirroring of the diagonal can be done later.

White pawns are indexed in octal with:
  A  B  C  D  E  F  G  H
8
7 57 56 55 54 50 51 52 53
6 47 46 45 44 40 41 42 43
5 37 36 35 34 30 31 32 33
4 27 26 25 24 20 21 22 23
3 17 16 15 14 10 11 12 13
2 07 06 05 04 00 01 02 03
1

Black pawns are indexed in octal with:
  A  B  C  D  E  F  G  H
8
7 07 06 05 04 00 01 02 03
6 17 16 15 14 10 11 12 13
5 27 26 25 24 20 21 22 23
4 37 36 35 34 30 31 32 33
3 47 46 45 44 40 41 42 43
2 57 56 55 54 50 51 52 53
1

First pawn is indexed with bit 2 set

Index is constructed by joining the bits in most significant bits first:
Pawns,
first 3 bits of black and white pieces interleaved,
remaining bits.

The main bit tables used are
bn: black loses with g50=n but not with less
wn: white wins with g50=n
b0: black check mate
w0: all zeroes

The bn tables are sparse and zeroes can be stored in 3.14 bits per position
using bytes to encode length to next set bit.
Ones use at most 1 extra byte per position.

For each pawn position, starting at 57 and decrementing, with each pawn of the
  same position having a lesser value than the previous
For g50=1 to 100

  Calculate b(g50)
    if g50=1, no legal move without capture or pawn move and all and at least
      one capture or pawn move lose
    if g50>1, no legal move(including capture or pawn move) avoids loss and at
      least one move was turned into a loss by the latest increment of g50

  Calculate w(g50)
    any capture or pawn move, or any move into b(g50-1) wins, or w(g50-1)

  Leave lower bits for later, this to allow a certain minimum block size,
    say 25 lower bits means minimum block size is 4 MiB
  For each value of white upper bits
    For each black piece px,
      For each value of upper bits, excluding those of px,
        For each position
          Store if any move avoids loss and if it might be a new loss (2 bits)
    For each position
      b(g50)=is new loss (using stored values)

  For each value of black upper bits
    For each white piece px,
      For each value of upper bits, excluding those of px,
        For each position
          Store if any win into b(g50-1)
    For each position
      w(g50)=w(g50-1) or new win (using stored values)
jkominek
Posts: 150
Joined: Mon Dec 04, 2006 9:02 am
Sign-up code: 0
Location: Pittsburgh, PA

Re: New attempt at 8-man and beyond

Post by jkominek »

The part of your algorithm that stands out are the square map tables and indexing function. But I must be missing something because I'm puzzled. What problem is this supposed to solve? That is to say, how does it bring 8-man and beyond tablebase construction in the the realm of current computer capabilities?

Is the idea that interleaving bits from white and black pieces in the index improves data access locality, reducing memory requirements? Or is the indexing geared towards applying 8-way symmetry through simple bitwise operators, thereby improving calculating speed? (Or both.)

I am not a good low-level programmer. The approach you propose seems in line with HGM's aptitudes and know-how. Perhaps you can persuade him to implement your ideas.
koistinen
Posts: 92
Joined: Fri May 02, 2008 7:59 pm
Sign-up code: 0
Location: Stockholm

Re: New attempt at 8-man and beyond

Post by koistinen »

jkominek wrote: Sat Jun 19, 2021 6:50 am Is the idea that interleaving bits from white and black pieces in the index improves data access locality, reducing memory requirements? Or is the indexing geared towards applying 8-way symmetry through simple bitwise operators, thereby improving calculating speed? (Or both.)
Yes, the idea is to improve data access locality.
The main purpose of the indexing scheme is to separate out rotations and mirroring as they will only cause the lower 3 bits of each pieces 6 bits to possibly change, allowing the top bits to remain constant for a partial calculation.
Pawn positions remain constant except once per g50 step.
When computing the result of the black moves, the upper white bits are guaranteed to be constant.
When computing the result of the white moves, the upper black bits are guaranteed to be constant.
By computing the moves of each piece separately, the needed ram is small, i think about 24*block size.
Disk space is still large, about 2 bytes per position. (Likely plenty room for improvement.)

I intend to have a go at making at least a naïve implementation in C.
koistinen
Posts: 92
Joined: Fri May 02, 2008 7:59 pm
Sign-up code: 0
Location: Stockholm

Re: New attempt at 8-man and beyond

Post by koistinen »

After 18 months of failing I have decided I am not as good at programming as I thought.

Next step is writing a plan for solving the problem for someone who has trouble juggling all the details required for the solution I have in mind.
First part of it will be a very naïve way of doing the computation: Leaving aside taking advantage of mirroring, permutations, tricky encodings, bitmaps et.c.
Just using simple indexed arrays and files storing distance to 50 move rule for white to win.
jkominek
Posts: 150
Joined: Mon Dec 04, 2006 9:02 am
Sign-up code: 0
Location: Pittsburgh, PA

Re: New attempt at 8-man and beyond

Post by jkominek »

I'm not as good at programming as I once thought, either!

However, I believe Ronald de Man is as good a programmer as he thinks he is (which is very high). Based on his comments over the years he only works on projects if and when he wants to, and is not likely to take on someone else's plan. So, you'll be fishing for someone else, most likely. Saying that, Ronald once categorically stated in this forum that he would never ever work on 7-man tablebases. That changed.

I wish you success in writing up your epic spec.

P.S.
Next step is writing a plan for solving the problem for someone who has trouble juggling all the details required for the solution I have in mind.
I hope you meant "no trouble" or "little trouble". An ominous indication of the complexity of your scheme.
koistinen
Posts: 92
Joined: Fri May 02, 2008 7:59 pm
Sign-up code: 0
Location: Stockholm

Re: New attempt at 8-man and beyond

Post by koistinen »

jkominek wrote: Thu Jan 05, 2023 9:26 pm I hope you meant "no trouble" or "little trouble". An ominous indication of the complexity of your scheme.
No, I mean to make the problem easier, so someone like myself, who has trouble juggling all the details, is able to do it.

One part of it is applying the good advice I got on this forum and start with a simpler program which solves the problem inefficiently.

I will of course keep the way I want the problem to be solved in the end in mind so it will be easy to get there in small steps.
koistinen
Posts: 92
Joined: Fri May 02, 2008 7:59 pm
Sign-up code: 0
Location: Stockholm

Re: New attempt at 8-man and beyond

Post by koistinen »

Here is my plan to make the problem easier so someone with my ability might solve it:
  • Initial limitation: Only compute white wins as it simplifies the code a little. Also don't worry about pawns.
  • Make a naïve implementation including code working for array.
  • Save result. Write query code for files. Initialize w.r.t. captures. Now code should work for any pawnless endgame.
  • Change indexing scheme so 3 bits of 6 will be invariant under rotations.
  • Store as bitmaps while computing.
  • Use bitmaps for computing when data is in ram.
  • Use mirroring to reduce table size.
  • Use permutations of identical pieces to reduce table size.
  • Split computation into parts so they may be done separately.
  • Add pawns.
  • Perhaps work on data compression.
  • Make it work on a network, where each node store parts of the table.
  • Use tricks, including extra tables and sequences of communication to reduce amount of network traffic.
  • Find better compression techniques.
  • Encourage interested people to get a large enough hard disk to have a functional node.
The order of the listed items is not strict, just something I think is reasonable.
Post Reply