How do you store/secure your research data?

[Richard Hedrick]

So I recently upgraded my home network storage to accommodate additional space. It was up for about a week when a power outage caused the device to lose the RAID configuration and with it all my data. RAID is supposed to provide fault tolerance if one drive fails but not if the configuration itself is lost. Luckily I am paranoid enough to have had another backup copy and could restore my old RAID setup on my PC. As I was going through this process I could not help but wonder what I would do if this single remaining copy of my 2 TB worth of data were lost before I had a chance to get things back to normal. It can be very time consuming to move around 2 TB of data so I had two full days to ponder this question and decided that I am probably not being paranoid enough.

Things are now back to normal but I feel I need to take a different or enhanced approach to securing my data. The data not only represents about 12 years of my own piddling around but also 12 years of collected archival material. If I was to lose all this data it is unlike I could find the motivation or enthusiasm to begin again. I am currently back to my old setup which is two 2 TB drives setup in a RAID 1 configuration with all data synchronized to a backup 2 TB drive nightly. I am near capacity so I have to upgrade.

In this digital age I am sure I am not the only one dealing with these issues of hard drive space or pondering the question of whether I am being paranoid enough with my precious data.

I would like to hear others input or take on this subject of how careful or protective you are of your data. Most importantly, since I rolled back my upgrade and still need a long term data storage solution, I would really like to hear the storage/backup setup others have in place.

Thanks for any feedback,
Richard

[Jeff Leach]

For the bulk of my research material, I have two 1 TB USB drives, which I try and keep at seperate physical locations. Besides that the material I am working on is usually kept on USB Memory Sticks. The actual material I produce is really pretty small compared to the research material and these I try to keep an up to date copy on my Hotmail SkyDrive.

Another method is to trade / share your research material with others, which can easily work as another back up.

One problem with the above set up is my computer won't recongize more than one Passport drive at a time, so I can't directly copy from one to the other.

Thanks for your post, I reminded me that I need to be more carefully with my back ups

[David W]

You guys are scaring me. I recently finished moving all my work from hand written (I started in about '88) to computer. But I ditched all the paper.

I am sloppy by comparison, only transferring to memory stick every couple of months or so. I don't have photos, only Word & Excel docs.
But on a much smaller scale than you, as it fits onto 1 stick.

[Jeff Leach]

Do yourself a favor, buy another Memory Stick as big as you can easily afford and use it as a back up.

[Richard Hedrick]

Guys, thanks for the feedback. I suspect that USB drives are the typical backup medium for most. I also use about 7 GB on SkyDrive to store my active research material. Since cloud storage like SkyDrive and Drop Box are free no reason not to benefit from their use since they are basically a real-time backup. They also allow you to access the data easily from multiple devices.

I think it’s good to be a little scared about losing your data. I suspect if one lost all their data they would be willing to pay a great deal more to get it back than what it would cost to take some preemptive backup steps. I don’t believe one should be wondering if their computer or hard drive will crash but when. I think this gets you in the right frame of mind when considering your backup strategy.

Any other input on the topic is welcome.

Maybe in a future topic we can discuss how members are organizing all these files and documents. They sure can be hard to locate sometimes.

Richard

[Maxschnauzer]

While not a serious researcher, I have an extensive photo collection and have had bad luck in the past with external hard drives and flash drives failing on occasion over the past several years. I now back up my files not only on a backup HD but in the cloud automatically using Mcafee Online Backup. Peace of mind.

[Richard Hedrick]

Max you make a good point. The reliability of the device you are backing up to is important. The mean time between failures for hard drives is actually quite good but I have seen too many crash over the years not to consider it as a real possibility.

Something else to consider for those of us who order DVDs from NARA or simply use DVDs as a backup/storage medium. DVDs are actually not that reliable for long term storage and have a much shorter shelf life than one would imagine. In looking over my orders it seems that NARA switched to DVDs in early 2008 (maybe late 2007). NARAs own position on the matter, see link below, is that experiential life expectancy is 2 to 5 years. This of course is based on a variety of conditions but I suspect most of us don’t have a “clean room” at home to store our DVDs. This means that I should expect that some of my earlier DVDs from NARA may already be corrupt.

Richard

http://www.archives.gov/records-mgmt/in ... a-faq.html

[John P. Moore]

Besides running RAID on my desktop PC I keep a spare HD of many TB in my bank safe deposit box. I rotate that HD twice a year.

I had a HD failure about 15 years ago. Fortunately I was able to recover all data with a disk recovery program.

I have been burning vacation video to DVDs for more than 15 years. I watch these videos once a year and have never noticed any data corruption. Same goes for BluRay discs that I have burned as long as 5 years ago.

john

[Tom Peters]

I round-robbin two external 2TB disks using a Newertech Voyager (very handy) to backup everything. I really ought to do what John does and put a third one in a safe deposit box.

Always backup.

Mad Dog

[Habu]

Backups? You mean my reel-to-reel tapes of interviews with vets from the Spanish-American war, WWI, the Polar Bear Expedition, the Banana Wars, etc won't last forever?!?!?!? What about the cassette tapes? And 8-track tapes--surely they have are truly archival? VCR tapes? What about floppy discs? ZIP drives? CDs? Hard drives and USBs? And now DVDs?

I gave up. Seriously. I can spend my life transferring from "old" media to "new" media, putting up with degradation due to recording of recordings, and fearing The Day of the Inevitable Crash (or just as bad, the day when I can no longer find a way to play or transfer older files). Or, after a realistic examination of my needs, I can take a multi-pronged approach.

By the end of 2016, hopefully, all of my older recordings (well over 10,000 hours) will be computerized. Hard copies will be done in the form of DVDs, with off-site storage for backup. Not ideal, particularly with the ongoing problems with software changes and the risk of orphan software. For now, I'm going with mp3 because I think it likely that for the next ten years I should be able to find software to change from mp3 to newer formats. Future interviews will be transferred to computer media immediately.

At the same time, I'm working on transcribing the recordings. That is an ongoing project, and I don't expect it to be done in the near future. I figure I'll save files in whatever word processing software I'm using at the time I do the transcriptions, with backup text files. (I still remember the headaches when I was forced to transition from WordPerfect to Word, and prefer to avoid a repetition.) Computerized files are just incredibly handy, particularly when you need to do searches for terms/locations/etc. Again, the goal is a ten-year life span before I am forced to do updates.

But ten years before obsolescence is not something I consider acceptable. Those transcriptions will also be printed out on paper. I know, I know, just incredibly old-school. It is bulky, it is dusty, you can't search it with a couple of key-strokes. But it works, and has worked for long enough to have an established track record. It won't be made obsolete in ten years.

Paper. The original backup media. Might be worth considering as part of your backup plan.

[Reegor]

This is a sophisticated discussion, and everyone seems to have thought about this much better than the average computer use.
I did academic research on related issues ten years ago, and it has affected my backup strategy since then. One way I think about this is based on scenarios: what can go wrong?
Think about the following scenarios:

• House fire or theft - that's pretty much the worst case
  Accidental deletion of an important file - how long will it take to figure out it is missing?
  File corruption.
  Failure of backup device - I once had a backup system fail when I needed it. Fortunately I had a second one.
  Misplacing the backup device - you forgot which USB drive contains what.
  Loss of passwords
  Theft of notebook computer.
  Your own death or disability. you won't care, but your wife may not be able to find things.
  Total failure of computer (that's a problem with RAID).
  Long term, obsolescence = changes in file formats, storage formats, and interface formats

Personally, I have 2 main backup methods:
An Apple "Time Machine" system which backs up my Mac every hour. But it does not provide a good archive of work I did years ago that is no longer on my machine.
Portable hard drives, one of which goes in my house fireproof safe, and another in my safe deposit box occasionally.
Also, I occasionally send things via email to my daughter, where they will live on her Google mailbox. And USB drives are now large enough to back up databases although not an entire machine, and they are convenient, so I will add one.

File formats: I agree that paper is really the only universal format. It will last at least 50 years. Electronically, JPG and .txt seem to be pretty safe. Most of my archive is PDF, and I have no sense of how long that will last.

Interfaces and physical formats: I have bunches of devices and hard drives that use SCSI, ATA, SATA, firewire, etc. All are basically unreadable now. I think USB will still be readable in 15 years, and has a decent chance to last another 25. I still have material on 3.5 inch floppies and zip drive disks - basically unreadable today.

CD and DVD - yes, cheap recordable CDs can fail in as little as 5 years. Good brand names last much better.
Hard drives have 2 ways to fail. First, they will wear out, if they remain running. Although MTBFs are listed as around 500,000 hours (50 years), realistic numbers are that they will start to fail after 3 years of continuous use. More expensive server-grade drives will do much better than cheap consumer drives.
Hard drives can also fail if they are turned off and left in a vault. I'm not sure anyone really understands all the processes, but things like evaporation of lubricant are relevant. Also just "freezing." The best thing to do is apparently to take them out and run them for an hour, a few times a year. But who is going to bother?

Cloud storage: not reliable in the long run. Some day, the company will disappear. Many small cloud backup services have already died through bankruptcy, government action, hacking, etc. At a personal level, you can fail to pay, fail to stay "active" in the company's database, or lose your password.

In short, a mixed strategy is needed. At a minimum, as everyone has implied, a frequent backup method (daily) and an off-site long-term method. At least in my case, the backups also have to be convenient, or I will rarely bother.