Share Your Tablebases

[Vegan]

I note when I view my shares, there are hardly any blue files, most are gray, meaning there are not many shares of the tablebases out there.

If you are short on disk, sharing a subset can still help. For example the full 5 piece set is a bit over 7 GB making it easy to share.

The 42 subset is bigger at around 95 GB but is also sharable with a larger hard disk. The 33 subset is slightly larger and could be shared with or as an alternative to the 42 subset.

The 42p subset is very large and needs a large 750 GB disk to hold them all, and that is not including the 42 or 33 subsets. If you have a secondary disk, then sharing the 42p is feasible, as is the 33p subset which is equally as large.

To share the full 6 pieces needs 1.2 terabytes of storage, and a pair of 1 TB disks would be useful (RAID 0) to share them. Larger storage (3 or 4 hard disks) is useful if you have have large numbers of other games installed.

Some use disks as stand-alone groups and this works as well. Using 3 or 4 disks from a SATA controller is easy to set up. eMule can use all of the disks for sharing easily.

A 320 GB disk could share the 5 piece set with 42 and 33 if used for tablebases only. A 500 GB disk is needed for the 33p and a 750 GB disk is needed for the 42p subsets meaning 3 hard disks can hold all the tablebases from 6 pieces, using low cost disks.

Share those tablebases so P2P can work.

[Daz]

Hi

I have 85% of the six man, about 150Gig to go (taking into account emule has about 100 Gig of incomplete)

I am sharing all I have.

setup = 2 x 750Gb drives. 33P and 42 on one and 33 and 42P on the other.

Regards

Daz

[ath]

I note when I view my shares, there are hardly any blue files, most are gray, meaning there are not many shares of the tablebases out there.

Are you sure? I ask, because the documentation seems to be zero on how to interpret 'Complete Sources'. I see that several files I
know are shared by at least three hosts are listed as '1' under Complete Sources, which leads me to believe that this is something
else than 'how many sources for this file are there out there'. (What does 'Complete Sources' = 0 mean, for instance -- most
of those are in my 'incoming' directory. And why do I have 13 complete sources listed for kppkpp.nbw.emd, when a straight search
finds only three?)

[Kirill Kryukov]

Are you sure? I ask, because the documentation seems to be zero on how to interpret 'Complete Sources'. I see that several files I
know are shared by at least three hosts are listed as '1' under Complete Sources, which leads me to believe that this is something
else than 'how many sources for this file are there out there'. (What does 'Complete Sources' = 0 mean, for instance -- most
of those are in my 'incoming' directory.

Estimating the number of complete sources does not work very well in eMule. It's works more or less for the files you are currently downloading, but for other files you share it's usually not reliable at all. I share about 2000 files, and most of them show 1 complete source, although I know it's not correct in all cases.

And why do I have 13 complete sources listed for kppkpp.nbw.emd, when a straight search
finds only three?)

13 sources were found using KAD and source exchange, but only 3 of them are connected to the servers in your server list. So when you search using Servers you only get those 3. Just my guess.

[Kirill Kryukov]

..... (skipped)

Share those tablebases so P2P can work.

You forgot the magic word "please".

Most of people reading this forum know all you are saying, and share what they can. One good way to improve the situation is to post on other forums, inform and invite more people to join. If more people learn about EGTB and join the network, we all win.

[Mark]

Question about downloaded files: All of the files I've downloaded have "[content.emule-project.net]" embedded in the file name. Why is that? If I take that out of the file name (which you have to do to use them) will they still be shared?

Thanks!

[Kirill Kryukov]

Question about downloaded files: All of the files I've downloaded have "[content.emule-project.net]" embedded in the file name. Why is that?

Probably someone is sharing them with that name?

If I take that out of the file name (which you have to do to use them) will they still be shared?

Yes. I think it would be a good idea to remove that part from file names.

[Vegan]

Right now I am finishing off the 42 group and as expected it is slowing down to just a few feeds from the top people I have received the most from. Others are conspicuously absent from the client list.

I have saved the file list from all the people i could get them from as a marker of availability. So I can estimate when I will finish. I wish.

[lsvll1]

Hey:

What does 'Complete Sources' = 0 mean, for instance -- most
of those are in my 'incoming' directory.

I thought perhaps you were referring to the "Transfers" page on eMule when you said your "Incoming" directory. Perhaps not. After all, there is no column of "COMPLETE Sources" on that page, although there is a column of "Sources." The reason I bring it up is because that "Sources" column on the "Transfers" page confused me (as it might have others) after I began downloading EGTB files in particular (i.e., not other files), since it said "0" sources for almost all of the EGTB files I was downloading even when doing global server searches on those files indicated multiple sources.

Since then I believe I've figured out the meaning of those 0's, and if you haven't, try this:

On eMule, go to Options -> Extended -> Show more controls (advanced mode controls). If you click this box, the "Sources" column under "Transfers" will now be in the form of "x + y", where x + y is the number of sources eMule is presently "aware" of relating to each of your files. When "Show more controls" is not enabled, all you see under "Sources" is "x" -- which is often 0 and is obviously NOT the number of sources (and thus very misleading).

What is "x" and what is "y"? If you double click a few of the files you are downloading, you will see individual representations of the sources/clients eMule has lined you up with. There are smiley and frowny faces assigned to each one. "x" is simply the number of Green Smiley faces plus Yellow Smiley faces, and "y" is the number of Grey Frownies.

Green Smiley means that that particular client is transferring down to you. Yellow Smiley means that you are on that client's queue. Grey Frowny means one of the following: No needed parts, asked for another file, or unable to connect due to a low ID or full queue. (And a red means you're currently connecting to that client.)

Interestingly, when you are downloading files within such a small "community" as ours (i.e., EGTB downloaders), there seems to be many more "y" (Grey Frowny) clients than when you're downloading a non-EGTB file. This is almost always because I am asking you for file A while I'm already downloading or in queue for file B (C, D, E...) from you, and vice-versa....

It's kind of like the inbreeding that goes on in little West Virginian towns -- Not enough chromosomes to go around.


Patrick


P.S.: Vegan, you're trying to make $ off this?! I hope you can pull it off. And $169 does seem pretty reasonable for a 750 MB hard drive full of tablebases.

[ath]

What does 'Complete Sources' = 0 mean, for instance -- most
of those are in my 'incoming' directory.

I thought perhaps you were referring to the "Transfers" page on eMule when you said your "Incoming" directory. Perhaps not.

No, I was referring to the 'Shared Files' tab (where, indeed, most shared parts are grey) -- but perhaps I misinterpreted the original post?

[lsvll1]

No, you didn't misinterpret the original post at all, which is why I hedged on that statement, and why I even thought to myself that it was more likely you were still referring to the "Shared Files." I think you are absolutely correct that we can't assume what the blue and grey files (and 1's) mean under "Shared Files" since they make no sense if we interpret them the same way as files we're downloading on "Transfers."

The truth is that I used what you wrote primarily to segue into what I wrote about on "Transfers." Which I think is relevant to the post because (IMHO) I believe it shows that the info on "Sources" presented on the "Transfers" page is much more informative and reliable than on "Shared Files." (with the caveat that I'm no expert and I may well not be entirely correct, and I do know that that's not ALL there is to know on that subject)

I don't know if everyone already knew what I wrote about, but I thought that at least people new to eMule might find it enlightening since the "Show More Controls" option is buried in a place in "Options" that's not easy to get to, and because the "Extended" option even says "WARNING: Do not change these settings unless you know what you're doing. Otherwise you can easily make things worse for yourself. eMule will run fine without adjusting any of these settings." I didn't thoroughly look there until I was irked enough about the "Sources" column to look for more info (again, the problem only came up when I was downloading EGTB files in particular).


Regards,
Patrick


P.S.: Regarding my other P.S. to Vegan, I wanted to clarify that I think it's great what you are trying to do (if you look at what ChessBase and Convekta charge for databases, it's obviously a much better deal, and I'm assuming it must be non-profit.)

[Vegan]

My offer of tablebases on DVD or HD is more courtesy than profit driven. While I will make a few dollars, the bulk of profit goes to keeping tablebases available on-line for download.

I lost a good server after a power supply shorted out, costing me a system with a quad-core processor, 4 hard disks and 8 GB of RAM. Even the video card was toast. Power supply was 650W and the system needed 475W so I am not sure why it failed.

I hope that offering tablebases on DVD to raise some cash to replace the burned out equipment. As I said, the DVD offer is a courtesy. I encourage everyone who is into computer chess to use tablebases.

[lsvll1]

Vegan:

Regarding the way you organize and apparently download your TBs, FWIW, I do take a different approach to collecting sets. I wonder how the rest of us approach collecting? Personally, I want the smallest # of tablebase sets which still hit a very large percentage of 6-man endings that occur in actual games.

A post on this forum by Peter Kasinki (which is linked in Kirill’s website on EGTB, viewtopic.php?t=74) has an analysis of 6-piece ending frequencies that occurred in 23,000 comp-comp matches. Peter lists a “Top 365” of the TB sets ordered from most common to least common.

There are some pretty helpful conclusions Peter makes: That of the 365 possible sets (which includes the 70 5 v 1’s), you only need 83 TBs to cover 90% of comp-comp endings. And only 42 TB sets are needed to cover 90% of human-human matches.

A couple of calculations of my own: 1) In those 23,000 matches, 118 sets of the 365 showed up 0% of the time [though this would obviously change (minutely, I'd guess) with a larger source of matches]; 2) The Top 25 of the 365 accounts for a whopping 65% of all 6-piece endings.

So what I’ve done is whittle down the list of 365 sets according to their prevalence. My total will eventually amount to “only” about 650 GB (I have about 430 GB already downloaded). I have a 1 TB HD. My plan was to first download Peter’s Top 25 (a handful at a time) so that I could have a nice set of 6-piece EGTBs that is pretty effective for its size. In all, I am downloading 184 of the 365 sets (or of 295 sets if not counting the 5 v 1’s). My downloads include 125 of his Top 150, but just 59 of the rest. And a key thing is that this includes all of the promotion possibilities for sets with pawns, as well as a number I thought were otherwise important for one reason or another (e.g., long checkmates).

Regarding your website, I think a lot of chess enthusiasts would like a collection of TBs that isn’t overwhelming and yet hits a great % of actual 6-piece endings, while being very economical. I see that Convekta currently sells TBs totaling 100 GB (12 DVDs for $79). ChessBase’s product is so minimal as to be nearly irrelevant. Both of those companies hit a number of important sets but a lot of uncommon ones too; and no sets to cover promotion possibilities (!) (When I e-mailed ChessBase asking if Fritz would need the "sister" TBs to counter the promotion problem, they wrote back one line: "The sister sets have not been developed." And that was about 2 months ago!) Meanwhile, Peter’s Top 25 totals just 132.29 GB. A top 25 with the promotion sets could probably fit on a relatively small HD. Just a thought.

Patrick

[Vegan]

That's a tough one, I am organizing them the same way they were when I first downloaded some from Bob's FTP before his RAID croaked. Once the block is done, they go into C:\Nalimov

The way chess programs play is different from human games so the analysis of a bunch of games is misleading. I use the full set of pieces and have asked Eugene to implement 43 and 52 and even 44 tablebases for the next version of TBGEN so that it will resist obsolescence well into the future.

Once I have a block, I have to move them to DVD as I currently lack the disk space for sharing them P2P for people to download. I had a power supply burn out killing a high-end machine.

[lsvll1]

The analysis of the list I mentioned is taken from 23,000 computer vs. computer matches, although there's also a human vs. human analysis of over 3,000,000 matches that's on the post above the comp-comp matches, so you probably saw that list first. I'm not saying 23,000 games is enough for perfection, but I've also scoured it thoroughly and think it's the best way to go IF you're like me and don't have the space. Also, aside from the top 58, I was forced to remove a number of sets that I would rather not have simply because they were much larger than average (specifically, a set plus all of its promotion possibilities).

Your set-up is fine the way it is. Great even. I would be a completist too if my wife wouldn't divorce me if she knew how much I was spending on multiple hard drives for "some chess thing."

[Vegan]

Over on Bob's FTP he has a large PGN file of games. I built an opening book from it, but I have never looked at it beyond that. I wonder if it is too big for some of these programs to generate an analysis of positions and generate a list of tablebases most commonly used.

[gambit3]

such a table has been done, when first 6 man tables were coming out. kirill and others here would remember the debate that ensued about the actual effectiveness of it, given that in order for it to actually represent percentages, you would ave to have more than 100%. further, what about games that have tb accesses entailed by depth but never actually make it to 6 man positions. the usefulness of the tables on commonality of ending tables was very quickly and very thoroughly smashed logically, since engines don't record which tables they accecss every move.

short end of point summary for those without mathematic inclination: it is not possible within a reasonable error margin to tell which tables were used by the engines and how frequently in the lead-up to the first 6-man position in the game. any position that involves a pawn MUST, by definition, also have had the engine access ALL of the dependancies of that particular pawned table, so for kppkpp endings EVERY 3v3 6man, 3v2 5man, and ALL 3 and 4 man tables have been accessed, whether or not the game ends at 6 man.

this is actually covered elsewhere on the board, but i'm not going to go through years-old archives to find it myself to re-discard something that has already been discarded long ago (the commonality list).

[lsvll1]

You do know that it is from Kirill’s own website (linked earlier) that this particular “commonality” list arose, RIGHT? CITE: “You may like to download tablebases by ‘importance’ order, which is based on statistics of occurrance of each ending in real games. Several such lists exist: by Dieter Bürßner, Nelson Hernandez, and Peter Kasinski.” (BTW, I saw nothing negative about these lists in prior posts, so you’ll have to give me a specific search. “commonality” only points back to you.)

While frequency of occurrence analysis is an incomplete way of looking at tablebases, I think you’re throwing out the baby with the bathwater (which is a nicer way of saying that you are incorrect at the most fundamental level). Not only do you dismiss frequency of occurrence as a useful criterion for downloading, but your logic goes so far as to imply that no particular tablebase is any more useful than any other tablebase.

Tablebase anarchy! So why is it that we don’t download 5 v 1 bases? Why is a kqqqkn match-up not as daunting as a krppkr match-up? If you take a moment to scan the top 365 list, you find in general that the more useful tablebases tend to occur in the first half of the list, and that the less useful ones occur further down (useful in terms of competition – i.e., Are you solely into the mathematics of tablebases, or do you also know competition?). A nexus seems to appear between frequency of occurrence and usefulness (and you can think of good reasons for why this is the case). E.g., near the end of the list we have most of the heavily un-balanced matches (kqqqkn, kqrrkb, krrnkb, etc.) and the heavily drawish matches (knnknn, kbbkbb, etc..). That is, stuff we think our engine can figure out without the help of the tablebases.

So, “short end of point summary for those without mathematical inclination:” IF #1 and #2 ARE MORE COMMON ENDINGS, THEN I’D LIKE MY CHESS ENGINE TO HAVE MORE INFORMATION ABOUT THEM THAN IT DOES ABOUT #364 and #365. OR: You can take knnknn into battle; I'll take krppkr.

[jshriver]

Sadly I've let my olympuschess.com site go a bit stale. I tried deeply to get amule and such working but I dont know if it's the linux client or what I was barely getting a gig a month if that. 99% of what I have is from people literally sending my DVD's and one time a hard drive. (which I'm very grateful for)

I really hoped the google share option would have found ground because they are the perfect place to host such a data set.. and really for google 1.5 TB is nothing.

Now I have a windows partition I need to try the windows client and see if I get better luck. *crosses fingers*

-Josh

[Mark]

any position that involves a pawn MUST, by definition, also have had the engine access ALL of the dependancies of that particular pawned table, so for kppkpp endings EVERY 3v3 6man, 3v2 5man, and ALL 3 and 4 man tables have been accessed, whether or not the game ends at 6 man.

I have to disagree with this. When a tablebase is encountered in the search, say kppkpp, it becomes a terminal node, since the game outcome from that position is then known. For that position, the computer will not search any further and no other tablebases will be accessed.

It would be interesting if a programmer inserted code to keep track of the tablebases accessed. This would greatly slow down the search, but you could do a lot of interesting experiments with it. For example, you could do a search from number of positions with say 8 or 9 pieces and then keep track of the tablebases accessed.

Anyway, although not perfect, I think the lists of the most important tablebases are certainly "good enough."

Regards,

Mark

[Vegan]

Sadly I've let my olympuschess.com site go a bit stale. I tried deeply to get amule and such working but I dont know if it's the linux client or what I was barely getting a gig a month if that. 99% of what I have is from people literally sending my DVD's and one time a hard drive. (which I'm very grateful for)

I really hoped the google share option would have found ground because they are the perfect place to host such a data set.. and really for google 1.5 TB is nothing.

Now I have a windows partition I need to try the windows client and see if I get better luck. *crosses fingers*

-Josh

Check your files, some are corrupt.

[lsvll1]

If you take a moment to scan the top 365 list, you find in general that the more useful tablebases tend to occur in the first half of the list...

Regarding a tablebase like kppkpp with multiple pawns, and thus multiple promotion tablebases, it is fairly instructive that of the approximately 71 tablebases with 2 or more pawns, only 2 are in the second half of the list, and those are 5 v 1. If that’s not evidence of the connection between commonality to usefulness then I don’t know what is. And that means that all of the promotion possibilities for the pawn-stacked kppkpp tablebase, less one pawn remaining, are in the first half of the list. Q’s, R’s, B’s, N’s, and any combination thereof. Further, in a case like knnknn or kbbkbb, the chances of 4 underpromotions is astronomically small.

[Vegan]

Generally KPPKPP will promote to KQPKPP and possibly KQPKQP for rare cases. Under promotion here is not suitable. Other positions with various piece combinations are where the remaining tablebases are used. It all depends on whats left coming into the final part of the game. It could be anything.

Do not be misled by under promotion consideration. There are other positions where these tablebases are useful.

[guyhaw]

KPPKPP does not promote directly to KQPKQP, only to KxPKPP.
There have been studies of the value of underpromotion, and these could be completed for 3- to 6-man chess, now all the Nalimov DTM EGTs are in.
Peter Karrer did a remarkable study of underpromotions in KQQKQP and KQPKQP some years ago in connection with the Kasparov-World Game.
See ICCA/ICGA Journal back issues around the year 2000.
Not surprisingly, the % was small (but larger than one might expect) and decreased as the pawns receded from the last rank.
g

[Vegan]

Given KPPKPP, and pawns on the 7th rank respectively, its possible for KPPKPP to convert to KQPKQP if the position does not check on queening. I use full sets of tablebases due to the fact no stats will cover every case.

I organize them based on pieces and dependency. 4 pieces need 3 pieces for captures. 5 pieces need 4 pieces for captures. 32p needs 32 in case of promotion. 33p needs 33 in case of promotion.

I download/generate tablebases with regard to the dependencies so the next layer will function properly even if incomplete.

Given KRPKRP, the position needs KQRKRP and KQRKQR for promotions among others. Then the 5 piece tables stand as deeper minors. With pawns, the layers are based on each pawn present. For n pawns, the minor is n-1 pawns.

Because of the interrelated dependencies, that is how I organize my tablebases.

Once I get all the 6 pieces together, I can then move on to 7 pieces and grind away for the next 10 years.

35 countries on the web page, not bad.