Google Hosting

[jshriver]

I came across this article today, and thought about egtb's.
http://blog.wired.com/wiredscience/2008 ... provi.html

Google is willing to host "open source scientific data" for free, by sending out a multi-TB drive array for researchs to copy data to and send back to google.

This would be a nice place I feel for storing the entire 3-4-5-6 (7?) men egtb's. Guess the big question is if egtb's count as "scientific data". In my view it would be.

What do you all think?

[ZeroOne]

Cool! I'd think EGTBs definitely count. 1,2 TBs would also be practically nothing for Google as they're already planning to host the 120 TBs of Hubble Space Telescope data. Yet 1,2 TB is enough so that one can say that it couldn't possible be hosted elsewhere. Maybe Google would also be inspired to build their own endgame tablebase querying interface... Not that there wasn't enough of them already, but who knows what Google might come up with. Maybe a special chess game search engine? Actually, why wouldn't they host millions of complete chess games too...

[Codeman]

These guys don't lack the financial resources anyway. Good Idea!

[Kirill Kryukov]

In case anyone makes any progress contacting Google, I would be happy to provide complete Nalimov 3-4-5-6-men EGTB collection (excluding the 5-1).

[jshriver]

I sent an email, but no response. Kind of a shot in the dark, but I would really like to see this become a reality. I also mentioned this project.

-Josh

[Ray]

Hi Joshua,

It's a long shot, but if we don't try then we'll never know. Nothing ventured nothing gained as the saying goes.

[jkominek]

An attractive idea indeed! As it is, I have a contact within google that already was making plans to have the full complement of tablebases hosted by Google. He has most of everything already, and I've offered to fill out the few he misses, plus the forthcoming 5-1 set.

This is nothing official by any mean, and it's really still in the contemplation stage. If something comes of it I'll ask for permission to keep this audience apprised of developments. (Permission might not be permitted until one day, boom, there it is.) The larger issue is not likely hosting itself, but what the search interface will be.

But Josh, perhaps your tablebase probing from the outside is helping draw interest? Can't hurt.

john

[Kirill Kryukov]

I think the main problem we have to solve is hosting the EGTB files. Making sure that the files are safe and available. Any web-interface or remote probing are less important, even though they would be nice to have as well. It's the raw hosting we are looking for first of all, because of the huge size of the dataset (1.2 TB right now, possibly more in future).

Right now the hosting is done via p2p network (eD2K/KAD), and I am very happy that I can say we have the files safe now: There are multiple sources for every file, and there should be some complete collections offline as well. However the availability is still suffering. All sharing member has bandwidth limitations, so it takes very long time (months probably) to download the complete 1.2 TB.

Reasons why creating another web-interface is less important: 1. Web-interfaces to complete 6-men EGTB already exist. 2. Web-interfaces can usually probe only single position, they don't do any search, and this limits their usefulness. (making a web-interface with search would be a breakthrough, but I'm not sure it's practical because it would be very popular). 3. Hard disks are getting larger, these days it's much easier to afford having a complete 6-men EGTB than it was a few years ago when we were starting the sharing project.

[jkominek]

Yes, the matter of simple hosting is more important to us. But it is not necessarily more important to them.

john

[jshriver]

jkominek: wonderful news. Hope your source makes a dent. I still haven't received any feedback about my email.

I urge everyone on this list to email google about this project. Perhaps with enough info they'll be more interested. The best email source I've found is research-blog@google.com. If anyone else knows a better means please post it here.

Also anyone else with the entire 6men set willing to copy the data? Many praises to Kirill for willing to do it, but also nice to have a 2nd source in case we are granted.

[jshriver]

Wanted to add, that 3-4-5 isn't a high priority. Between Dr. Hyatt and myself and this project I feel that void is well taken care of. I'm uploading the 3-4-5 dataset to my site right now and it's paid for at least another 4 years (let alone Dr. Hyatt's main source). I just truly would like another solid source for the entire 6men set. On a side note I'm excited for 7men to see more light. BTW it will take some time to finish uploading, but my 3-4-5 share will be up within 48 hours.

http://www.olympuschess.com/tb/ specifically http://www.olympuschess.com/tb/3-4-5 and http://www.olympuschess.com/tb/tbs

-Josh

[ZeroOne]

Wanted to add, that 3-4-5 isn't a high priority. Between Dr. Hyatt and myself and this project I feel that void is well taken care of.

I agree on that. I've personally removed 3-4-5 tablebases from my eMule shared directory, just to distribute the 6-men tablebases more efficiently.

[Kirill Kryukov]

Wanted to add, that 3-4-5 isn't a high priority. Between Dr. Hyatt and myself and this project I feel that void is well taken care of. I'm uploading the 3-4-5 dataset to my site right now and it's paid for at least another 4 years (let alone Dr. Hyatt's main source). I just truly would like another solid source for the entire 6men set. On a side note I'm excited for 7men to see more light. BTW it will take some time to finish uploading, but my 3-4-5 share will be up within 48 hours.

http://www.olympuschess.com/tb/ specifically http://www.olympuschess.com/tb/3-4-5 and http://www.olympuschess.com/tb/tbs

-Josh

Hi Josh! I added the link to the project page. I hope that you don't mind. Thanks!

[guyhaw]

I believe that JT, who has worked with me and Marc Bourzutschky on DTC, DTZ and DTZ50 EGTs, works for Google. He had the idea himself, independent (?) of Google's 'Research Data' initiative, and I've asked him how things might stand. Given that the Nalimov DTM EGTs account for the most eMule traffic, it makes sense. It does not make sense _not_ to include the 3/4/5-man EGTs as they are small in comparison with a single P-ful 6-man DTM EGT.
I have also written to a colleague in the USA who has all the EGTs. They would need to be MD5sum-checked after transfer to Google's disc: checking that the transferred data is complete and correct will consume some time.
JT may also be interested in transferring the available DTC, DTZ and DTZ50 EGTs to Google. Marc B probably has lots of 6-man DTC EGTs I don't know about, but in a non-Nalimov format.
g

[Kirill Kryukov]

Hi Guy! I keep hearing from you about DTC, DTZ and DTZ50 tablebases, and also about 7-men tablebases. It seems you and others invested lot of time and energy into this. So I wonder, why don't you release the generator source code? Or may be you did and I overlooked it? As long as you don't publish the generator code (together with the probing code), all these talks are useless. Reporting 517 moves long mates has 0 relevance to the people. What humanity needs desperately is a free (released under OSI approved license) generator code and probing code. (Ideally also fast, portable and feature-rich code).

[Kirill Kryukov]

I believe that JT, who has worked with me and Marc Bourzutschky on DTC, DTZ and DTZ50 EGTs, works for Google. He had the idea himself, independent (?) of Google's 'Research Data' initiative, and I've asked him how things might stand. Given that the Nalimov DTM EGTs account for the most eMule traffic, it makes sense. It does not make sense _not_ to include the 3/4/5-man EGTs as they are small in comparison with a single P-ful 6-man DTM EGT.
I have also written to a colleague in the USA who has all the EGTs. They would need to be MD5sum-checked after transfer to Google's disc: checking that the transferred data is complete and correct will consume some time.
JT may also be interested in transferring the available DTC, DTZ and DTZ50 EGTs to Google. Marc B probably has lots of 6-man DTC EGTs I don't know about, but in a non-Nalimov format.
g

So did he reply to you?

[guyhaw]

Kirill,

The code that generated the DTC, DTZ and DTZ50 EGTs is Marc Bourzutschky's generalisation of Eugene Nalimov's 'DTM' code. IT accommodates differently-sized rectangular boards, Fairy Chess pieces, and the metrics DTC(k), DTM(k) and DTZ(k).
Marc has not said anything about releasing the source, and I have only binary copies compiled for specific metrics like DTC(infinity), DTM(inf), DTZ(inf) and DTZ50. The computations were done by John Tamplin, and I don't have copies of the EGTs (3- to 5-man and P-less 6-man).
Marc is more focussed on Yakov Konoval's code now which is in Pentium Assembler (whatever that looks like) and I don't know if he has any plans re code related to Eugene's code.
I agree that it would be excellent to have an 'Open Source' code, or a library of basic procedures that do the Board Representation, the indexing and other fundamental things. I'd welcome that myself: none exists - yet. Of course, the downside of everyone generating EGTs is that we might finish up with a few hack, incorrect products which would then have to be painfully weeded out. There is more to generating these EGTs than meets the eye: only those who have tried can know what Ken Thompson, Christoph Wirth, Eugene Nalimov, Peter Karrer, Yakov Konoval and Marc Bourzutschky have achieved. I tip my hat to them all.

Nelson H, who is in the USA and was instrumental in receiving the 'last 16' Nalimov DTM EGTs from Eugene, is happy to work with Google as and when. Haven't heard from JT yet but I'm sure I will shortly.
g

[Kirill Kryukov]

Thanks for clarification, Guy.

[guyhaw]

KK: Yes, JT has passed on the 'EGT hosting' idea to colleagues, and we will see if Google are interested in our discrete-mathematical 'game'-domain, 1400 years of history, and eMule-evidenced international interest. g