Backup tablebases

[Dhanish]

I am downloading the 6 men tablebase files set by set.

When any set is complete, I would like to copy them to CDs and free disk space.

Copying each file to a separate CD looks like waste of CD.

What is the best method for backing up each set to several CDs?

1) Convert all files in the set into a 7z or zip file split into pieces of 700MB and copy each to a CD? But if the first or last CD gets corrupted, the entire set becomes useless.

2) Use a file splitter to split the files into batches of 700MB and copy each to CD? The good thing about this method is that the files are independently retrievable (if one CD is corrupted, I have to redownload only those particular files), but a lot of work is involved.

3) Use specialised backup software? (I have not tried this, any recommendations?).

What method do you follow?

[Shaun]

Given the data size - I would suggest an external HD?

By default I never have only one copy of any important files (I normally get in a mess with too many backups ).

So either 2 physical disks (in different machines or an external disk), or Disk + CD/DVD or 2 x CD/DVD.

P.S. I agree a system where if one disk is unreadable the set is useless if not acceptable.

Shaun

[redpawn]

Hi all:

For backup i use something may be a bit strange, for every tablebase I generate or download I put one copy on a DVD (4.7GB) as it is (emd files) and the other copy will be the nbw or nbb original file before datacomping but after compressing it with 7zip (a great program by the way) & again burned on DVD.

7zip applied on nbb & nbw filez can save really a lot of space, for example i have all 3+4+5 men nalimov on one 4.7GB DVD, and that's not all i remember even putting a set or two of the 6 men on the same DVD.

I'm planning two 1TB external HDD for the purpose of using those files.

as you have noticed I have two sets of backup DVD's [emd's & 7zipped]

BlueRay double sided dual layer disks seems to be my great next step to backup those files [waiting for thier prices to become more reasonable]

I don't know if I have added anything usefull here but that's what I'm doing & planning to do

Best Regards,

[Dhanish]

Given the data size - I would suggest an external HD?

Shaun

Thankyou Shawn, for the suggestion. I started with an 80GB external hard disk, but it is about to be full now. Having a hard disk only as a backup seems to be a waste. I use tablebases only for analysis, and hence don't need all of them together.

I don't know if I have added anything usefull here but that's what I'm doing & planning to do

Best Regards,

Hi Redpawn,

Glad to know what you are doing. For me, at my present speed of download, another 80-GB hard disk appears to be sufficient for one more year. I may have to change if Kirrill's http server turns out to be faster . Please, Kirrill, upload your 6 men tablebases in order of frequency of usage!

I decided not to backup to DVDs, as I found that even a small scratch is sufficient to make the data unusable, while CDs seem to be more robust as backup media.

Any ideas on how to copy to several CDs?

Regards,
Dhanish

[jkominek]

The best way to back up a hard drive full of data is with another hard drive. It's really that simple.

In my case I have a 2 TB external drive dedicated to storing chess data. It has firewire and usb connectors, but not eSATA, which is emerging as the preferred connection technology (faster speeds, lower error rate). External drives don't seem to be designed for running 24/7, so get one with an auto-sleep mode or just turn it off when not in use. Once a month I run md5sum on the entire collection to catch any data corruption, and to give each sector a little "exercise".

I was once big on backing up data to CD ROMs and have hundreds in my closet attesting to this. But the time attending to burning is high, the data verification process lengthy, and the coaster-creation ratio frustrating. As mentioned above in another posting, DVDs are vulnerable to wear; you're more likely to lose data with discs than with disks. Plus, the cost savings is marginal. Dual-layer DVD-R discs run about $0.15 a GB. A 750 GB SATA drive can be had for $0.16 a GB. (I sampled these figures from pricewatch.com). Copying to an internal drive is more hassle with opening the case and all, but is the cheapest and least-error prone route, with the fastest data transfer.

Holographic discs would be the ideal archival medium, if only the vendors could have been 6 years earlier on the technology curve. It looks unlikely they'll catch up quickly enough to make significant inroads into the consumer market.

In the mid-90s I used Magneto-Optical discs. That technology didn't survive. The write speeds were too slow and the capacity didn't ramp up quickly, and the media was expensive.

Tape backup is a dead-end technology. I used to use tapes -- and hated them.

So again, copy your drives to other drives. That's what Google, Yahoo, and Amazon do with their huge storage arrays. The alternatives are not attractive anymore.

john

[Dhanish]

The best way to back up a hard drive full of data is with another hard drive. It's really that simple.
....
I was once big on backing up data to CD ROMs and have hundreds in my closet attesting to this. But the time attending to burning is high, the data verification process lengthy, and the coaster-creation ratio frustrating.
....
john

Thank you John, for the advice. Based on your experience, I think I will have to rethink my whole backup strategy.

Regards,
Dhanish

[guyhaw]

One backup strategy might be to backup less in GByte terms, at the cost of having to invest some processing time to recreate the full files.

For example, I believe that if the Bitbase of an endgame was saved, it would help create the (say) DTM EGT more quickly than that EGT could be created from nothing. The reason is that all the draws would be known, and the number of 'dead ends', specially on backing up to stm-losses would be much reduced.

Checking out the contribution that Bitbases make in creating (or, as above, recreating) the 'Depth EGT' might be a neat project for someone.

g

[jkominek]

For example, I believe that if the Bitbase of an endgame was saved, it would help create the (say) DTM EGT more quickly than that EGT could be created from nothing. The reason is that all the draws would be known, and the number of 'dead ends', specially on backing up to stm-losses would be much reduced.g

That's an interesting idea. It might cut the overall time in half, to take a wild guess. Not sure if that's worth the programming effort, since on first thought it looks a tad awkward to implement, but one will never know until it is tried.

jk

[Kirill Kryukov]

I tried to un-datacomp some Nalimov files and then 7zip them. The result is several times smaller than original compressed Nalimov files (though I tried only several so may be I was lucky). So this could be one way of making backup. Though it would take lot of time to repack everything. My guess is that a successful repack should fit on a single 500 GB drive.

[Vegan]

I know from using TBGEN that uncompressed, the entire 6 piece TB set is 4.7 TB so you need a storage server like the model I suggested on my site which is common in the rack based system world.

I tried every compression tool for backups and all of them are better than DATACOMP at compressing tablebases.

RAR was best using the max setting, ZIP was close using max setting. With 4 piece tablebases, is one thing, but as the tablebases get larger the issues of complexity in the blocks means its tricky to compress them optimally.

DATACOMP works only because its dictionary does not slide. This is to allow the opening of the compressed file without uncompressing it.

I am starting to see some ad revenue trickle in on the web site so funding from Google may yet afford one of those storage servers. Then experiments would be simple.

[Shaun]

RAR was best using the max setting, ZIP was close using max setting. With 4 piece tablebases, is one thing, but as the tablebases get larger the issues of complexity in the blocks means its tricky to compress them optimally.

Try 7-Zip I find this compresses better than RAR and WinZip for everything I have tried.

There are many options to tweak compression, however the only one I change is

Compression method: either PPMd or LZMA compression level always Ultra.

Shaun

[Vegan]

I have looked at 7-zip before, its ok but not widely used yet. I am happy enough with DATACOMP as the tablebases are usable compressed. As for backup, I have Roxio Easy Media Creator 9 (older version) and it can make a multi-disk backup of whatever you want to DVD or Blu-Ray (not at $30 a disk). DVD is cheap, and if there are any problems I can regenerate/download what was damaged.

I have a web page on file integrity with some comments on it. Might try some compression tools on tablebases once I get more hard disks.

[guyhaw]

I think Vegan is missing one piece of information re the compression capability of DATACOMP.

Eugene Nalimov asked Rob Hyatt to investigate what would be the best block size for compression, given the use the EGTs were to be put. The goal was not max compression but max performance at runtime.

Rob concluded that 8Kb was the optimum size. Obviously, if larger block sizes were used, the compression would be greater and more impressive (but the runtime performance during search would be worse).

g

[Vegan]

I think I asked Bob about the 8K back a couple of years ago and he said it was to keep RAM usage at bay. Larger blocks mean more RAM to open, and with 2804 files open, it adds up. Ever look at the memory used by TBGEN as it grinds out tablebases?

I have tested DATACOMP with various settings, and bigger blocks do help a bit, bit not as much as you would hope. DATACOMP is more or less built for TBs as its unique compared to other LZ77 derivatives.

With those new fat Seagate disks announced recently, the 9 yards of 6 piece tablebases will now fit on a single disk. Makes it a lot more convenient than the swarm of disks everyone uses now.

jk's comments about tape, new tape is bigger and faster, but its near the end of the road for the technology at the high-end. Low cost magnetic disks are now widely used everywhere as storage, and copies are made where additional reliability is needed. My own forays into tape are long ago, when I abandoned tape I was using 8mm before I jumped on the RAID bandwagon. I have also used DC6000 class tape (before the Exabyte 8mm). RAID was cheaper and faster. The DVD recorders put low-end tape in the grave, mind you early DVD burners were garbage.