Data storage

[Andrew1]

For those of you who want a longer storage for your pics.

http://www.technologyreview.com/view...disk-unveiled/

[RustBeltRaw]

The 10-year number from that article, though basically a footnote, is an important take-away for anyone storing their backups on magnetic media. Regularly upgrading/replacing one's backup storage system should be considered a necessity in the long term.

[NorthernFocus]

...Regularly upgrading/replacing one's backup storage system should be considered a necessity in the long term.

Anyone who has had a hard drive fail on a computer should have this etched into their psychie.

[RustBeltRaw]

Anyone who has had a hard drive fail on a computer should have this etched into their psychie.

Guess why I advocate replacing drives?! Fortunately, I just lost my OS/programs drive. Data was fine, and I wasn't even a photog then, but it made me a RAID 1 disciple. Just replaced my main backup array since files on one drive started to get corrupted. Those were Seagate 7200.5 750GB drive which lived in open air their entire lives, and narrowly avoided disaster when my apartment flooded. Easy to access and swap out, but not very well protected. The new array is a mirrored pair of 1TB Western Digital Reds built into my laptop.

[Manfred M]

Anyone who has had a hard drive fail on a computer should have this etched into their psychie.

Anyone who hasn't had a hard drive fail is living on borrowed time. They all do eventually; I had a drive die within 4 months of installation and I have several drives that are a good 6 or 7 years old going strong. RAID or RAID-like (I use a Data Robotics Drobo-FS) is really the only way to go.

Even that is not fool-proof.

[jcuknz]

Pardon me but what is 'RAID' or 'RAID-like'
You worry me as I am using the HD out of my old crashed computer as my back-up in an external box with new machine. It was the latest addition and hardly used when the rest went away.

EDIT .. ignore me I read Wiki

[Manfred M]

Pardon me but what is 'RAID' or 'RAID-like'
You worry me as I am using the HD out of my old crashed computer as my back-up in an external box with new machine. It was the latest addition and hardly used when the rest went away.

EDIT .. ignore me I read Wiki

I have had two relatively new hard drives in a 4-drive box backup "die" within a 4 month period. Fortunately, the RAID (actually RAID-like) technology meant replacing the hard drive (on warranty) and letting it restore the backup data. Even that is not 100% as had two disks failed before I did the restore, I would have lost data; based on the configuration I use.

I was a bit concerned after the first drive died, and bought an additional backup, so when the second one went, I was able to replace it right away. Lesson learned; two drives from the same production batch was the problem, so I now try to buy different drives to reduce this rist. The nice thing about the Data Robotics system is that the drives do not have to be identical.

[RustBeltRaw]

I was a bit concerned after the first drive died, and bought an additional backup, so when the second one went, I was able to replace it right away. Lesson learned; two drives from the same production batch was the problem, so I now try to buy different drives to reduce this risk.

Interesting point. I wonder if the brand-new Western Digital Reds in my laptop were a bigger risk than I thought. They came out 4 days before I got them, so they're definitely from similar batches.

[Adrian]

This is a black art.

To start off with, buy reliable drives for Raids. We have a hell of a lot of business servers and our experience is that some manufacturers have a far better reputation and better warranty than others on their drives.
I use a 6 disk Raid array at home configured as automatic (hourly) backup storage across 4 drives, one permanent archive drive and one spare in case of drive failure. Plus other on site archive backup (including some prints of really stand out imagery).
This Raid array is further backed up overnight to a cloud server that is linked to our office system (my home system is also used for business, not just photography)
Both systems "report" to an off site IT office that checks for disk failure and software errors at least every day.
Temporary cloud storage / backup via dropbox is used when travelling

Clearly this is overkill for most people but anyone who handles a lot of data needs to accept that it can all be lost (maybe years of work) if you do not have multiple backups and the ability to restore those backups onto a new device. (So don't just rely on Time Machine if you use Mac, as this is tied to the originating computer to quite a large extent!).

Cloud backup is expensive for high frequency / high volume. It is very cheap though for just your prime work as a last ditch safety net.

[ajohnw]

I think one of the problem with many raid boxes is inadequate cooling leading to early drive failure. I also think that the recent increases in storage densities on a single platter within the drive don't help. Basically they aren't as reliable as they were and some form of redundant storage is required. I part trust usb drives - fill them up with backups and then never touch them again unless needed. I am a bit queasy about using rather high capacity ones though.

Another thing I have done for a long time is buy a new card once the one in use if full. Not a bad idea as the data retention time of flash is very very long - or was. It was of a type called SLC which stored a single bit state in a single cell. These days they seem to be working on or already have made it 2 or more. One thing for sure on this media though is that writing new data over old wears it out and also increases the time it takes.

I recently bought an iPad. iCloud isn't of much use to me as I run Linux but there are others about. HP for instance seem to charge 9 cents per gb /month. An outfit such as JustCloud.com seem to charge £2.95 a month without any limit on the amount used. When I look at the cost of building up a decent raid box these prices look very attractive. I've already bought the server box and 4 500gb enterprise discs, raid 5 and one spare though. Maybe I should put the lot on ebay.

All in all the cloud is an interesting area - one day many will be running their software from it and possibly no local discs at all.

John
-

[Manfred M]

Interesting point. I wonder if the brand-new Western Digital Reds in my laptop were a bigger risk than I thought. They came out 4 days before I got them, so they're definitely from similar batches.

On the other hand, if they come off a very good production run, having two identical drives would be a benefit, rather than a liability. To bad that is an unknown quantity until the drive starts to fail.

[Colin Southern]

Just a few early morning thoughts while I'm still waking up ...

1. Great that folks can create media that's good for 1,000,000 years - unfortunately though, even though the media may be good to go in 1,000,000 years, I doubt that there's be any device that can interface with it in 40 years.

2. Normally having at least 3 copies of data, with at least one in a different location is considered the minimum for backups.

3. Backups and archiving are different.

4. Personally, I like using cloud backup, although due to size issues, it's not a perfect fit yet - however - one can adopt ones style to make it fit a little better eg; if I take 50 shots of the same scene (eg sunset) and only process 1 into a commercial piece, then for the cloud backup I'll only backup the DNG & PSD of "the chosen one" - if my local backup of the other 49 fail then I'm not too concerned. With some client work that I do I know it's unlikely that they'll ever need to access it again - so in that situation I may only upload a full-resolution JPEG to the cloud, and skip several GB of originals.

5. Because standards change over time (eg hard drives went from MFM -> IDE -> SATA -> SATA2 -> SATA3, and we've gone from CDs to DVDs to Blueray now) it's important to migrate data from one standard to another whilst the old and new technologies overlap.

[drjuice]

I'm with Colin on the millions of years. But, one thing I try to do is to keep at least one functioning device for each type of archival media I have. Also, because of my rather strict winnowing of images, I tend to have a relatively small number of "active, useful" images. Maybe they total 250 or 300 images. And, I've recently started regularly archiving them on 100-year media. If anybody wants them after that, they can check with the Library of Congress. ;~)

virginia

[LocalHero1953]

In my IT career I have not just had drive failures but also viruses. Fortunately none of the latter have done damage or been difficult to remove, but I still worry about what they could do. So I avoid connecting my back up drive to my PC except when backing up. Not foolproof, I know, but every little helps.

[Colin Southern]

In my IT career I have not just had drive failures but also viruses. Fortunately none of the latter have done damage or been difficult to remove, but I still worry about what they could do. So I avoid connecting my back up drive to my PC except when backing up. Not foolproof, I know, but every little helps.

I like to think that this helps from another angle too; hard drives wear out (or the chances of a failure increases with run time) -- so it doesn't make any sense to me to keep a backup attached to my PC and running whenever the PC is, thus increasing the chances of having a time-on related failure when it's needed most (and being more susceptible to being knocked, spiked, virus etc).

[xpatUSA]

About twenty years ago, the Engineering manager proudly presented to us a 72MB behemoth complete with cooling fans an a UPS. But I digress . . .

I was involved with gaining ISO 9001 certification and wrote the Company software procedures (gas turbine controls). Back then the deal was three copies, one in-house and two elsewhere. The Cloud was nascent at the time and the phone line was as good as it got. I recall there was rotation involved.

We got the certification but the procedure was never really implemented

[RustBeltRaw]

On the other hand, if they come off a very good production run, having two identical drives would be a benefit, rather than a liability. To bad that is an unknown quantity until the drive starts to fail.

As an engineer, I am amused by the fact that the best I can do - despite a fancy degree, lots of research, and decent knowledge regarding the involved technologies - is cross my fingers. Progress!

5. Because standards change over time (eg hard drives went from MFM -> IDE -> SATA -> SATA2 -> SATA3, and we've gone from CDs to DVDs to Blueray now) it's important to migrate data from one standard to another whilst the old and new technologies overlap.

Another interesting point, and another reason to replace your drives regularly. I've gone from IDE to SATA2 to SATA3 in only six years, so I'm definitely refreshing drives more quickly than necessary. But at least obsolescence is a non-issue.

We should also consider how much data we're willing to lose. If I lost both of my RAID 1 arrays, my entire body of work would be gone. But that requires 4 independent hard drive failures. Personally, I'd be bothered by losing more than three or four months of work, or a significant chunk of my portfolio. There's a good cost/benefit case to be made for giving your portfolio images better protection (including the negatives/original files) than the whole archive.

[Colin Southern]

If I lost both of my RAID 1 arrays, my entire body of work would be gone. But that requires 4 independent hard drive failures.

Just be sure to step back and consider a bigger picture, eg the chances of 4 drives failing at the same time from different causes are incredible small - but that may not be true for them all to fail at once from the same cause (eg fire, theft, flood, power surge).

Just some food for though!