What should I use to backup all my video?
March 24th, 2008 | Published in Backup, Post | 10 Comments
It’s hard to sleep when I know that all of my video, from all my different shows, could forever disappear with just a small fire. I have 4 hard drives in my Mac Pro, and they’re nearly full of data, most of it video. Three terabytes of data. I also have external hard drives which I’ve been using to backup, but I’ve now arrived at that point where I no longer have room on my external drives, or on my internal hard drives.
I am almost out of space. I clearly need to go bigger. But that’s just part of my problem.
At the same time, I want to back up the data off site as well.
For the local backup solution, I’ve been looking at the Drobo with four individual terabyte drives in it. It’s an expensive solution though. All together, the package would run around $1,500 with all the drives in it. Still, most of the reviews rave about what a great backup tool the Drobo is. I can’t help salivate over the thought of the Drobo and Time Machine working in combination.
But what about the off site solution? At first I tried Mozy, an online service that would upload my data onto the internet, to store it “in the cloud” as it were. I started the backup, and after 3 days of uploading, I hadn’t even completed 1% of the data uploaded. At this rate, it would take more than a year to backup my one computer. That wasn’t feasible, so I canceled the service. Then I registered for JungleDisk which uses Amazon’s S3 service. At 15 cents per gigabyte, how could I go wrong? I started uploading, and at my upload rate, it was much faster than Mozy, but it was still going to take about 5 months. And, once you add up all the gigabytes, it was going to be an expensive solution as well. And that leaves me…
Well, I don’t know. I’m thinking that at this point, I’m better off simply copying my backups onto hard drives and storying them at a friend’s house. Maybe once per month we could swap. I’ll store their data, and they can store mine. I guess that means I’ll also have to secure my data with some sort of encryption.
What do you do for backup? Do you have any recommendations for me?
March 24th, 2008 at 8:47 pm (#)
I’m friends with former Mozy execs and use it for my typical “file” storage . However, like you, I’ve learned that video backup is NOT a good application for web storage. I’ve settled for local storage. After losing a 1TB disk with 750MB of storage on it, I am moving to a local disk “backup” of “backups.”
March 25th, 2008 at 12:07 am (#)
Patience is a virtue. It will take a long time, but separate site is the only way to have real disaster protection.
I think some of the services will let you mail them a drive to see the data (ElephantDrive offered this originally, though I could find info on the site when I just searched).
What is your connection speed? Is the chokepoint at the service provider’s end or at your last mile connection?
March 25th, 2008 at 4:11 am (#)
Hey Izzy,
I’ve been in the same place, and the biggest conclusion that I came to was for me to put any finished projects back to MiniDV tape, burn a DVD with project files and additional assets, such as sound tracks and graphics, and then have the ability to automate the recreation of a project from the original tapes in the event of HD failure.
In your case, I’m not sure if this would work, since you shoot with the EXCAM, and I’m not familiar with how you can back any of that up to tape. Maybe consider a DLT type drive?
Personally, I like the DROBO solution, but it sounds like you’d be nearly at capacity already.
BTW, I’m a huge fan of your work on Izzy Video. Thanks for all you do!
-Brian
March 25th, 2008 at 9:31 am (#)
I feel your pain. On-line “back up to the cloud” storage solutions fail for consumer use. As you calculated, it will take 5 months to upload your data. This is not a solution for people with lots of media. They are being stupid promoting this stuff to consumers. Businesses are different.
For financial records and other critical files including some photographs we have copies on a Drobo, other computers and also USB thumbsticks one of which is the in the safety deposit box at the bank.
Its too painful to backup all the music and movies. If a fire or flood strikes they are vulnerable.
September 7th, 2008 at 10:22 am (#)
Make a bootable backup of your hard drive (I use Super Duper) onto an external hard drive, so if your machine breaks you can recover from that. Then do it again on another drive. Take them both somewhere else and store them -one there and one there.
Plan to rotate/update them once a month. So don’t store them too far away.
That leaves your data - back it up to two separate drives and take those off site as well.
Set up time machine (if you have Leopard) and run that.
Now imagine you are living in a pestilence-prone area of Germany during the 30-years war, with roving armies likely to appear over the hill and burn everything. Now switch off that image and relax. It is the 21st Century and you have backed-up as well as you can.
September 8th, 2008 at 3:42 am (#)
I just came across this:
http://www.techcrunch.com/2008/09/07/parascale-promises-data-center-heaven-private-cloud-storage-at-about-1-a-gig/
December 2nd, 2008 at 5:19 pm (#)
Like you I my digital storage needs are growing by leaps and bounds daily. At my office we have a good long term solution but it’s not all too cheep. But I do recommend it. Often parts can be purchased second hand to help lower the cost. In our setup we use an xserve raid with 14TB of storage. Of course you can use less drives and build up to a high capacity as needed. This runs the daily activities for file serving and backup through raid 5. For off-site backups we use a tape backup system from Tandberg data. We have three sets of tapes that rotate out on a weekly basis. Tapes are still better for long term data storage with high capacity. Hard drives are too volatile and seem to self destruct when you need them most.
In your case A drobo may be a good replacement for your daily backups offering large capacity with room to grow in the future. As it is now, you will see little benefit in a drobo if you already have 4 drives that are maxed out. Until 4TB drives become available your stuck with only 1TB not a lot of growing room. And probably not money well spent for a long term solution. Also, unless this has been fixed in new units, there is a 2TB(?) cap on USB. So 4TB would show as two drives on your system. Not exactly ideal for organizing your projects. :(
One thing to conciser is if the digital copies are worth keeping of your videos? The tapes are less volatile than a hard drive if placed into a proper storage location. You can always re import that video back into your editing system. Keeping a project file and and other assets is far less demanding on your storage needs than all the digital video. By eliminating the old video files you find yourself in a better position and not having to fork over serious cash for a long term solution.
A few TB of storage is not really a long term solution for someone like yourself. You May need to look at larger “enterprise class” solutions like Xraid and tapes vs consumer hard drives.
Sorry this is not very well written and too long. But I hope it helps.
December 2nd, 2008 at 6:38 pm (#)
Mark,
That was a very thorough comment, and I love your ideas.
One thing I forgot to mention in my original post is that I’m shooting with a tapeless workflow. This means I have no original tapes to archive because everything is just a digital file.
Since I wrote this post, I’ve managed to get a decent temporary backup system I’m happy with.
I purchased a little port that allows me to plug in SATA drives as if they were game cartridges. I make exact duplicates of each of the four internal drives weekly, using four separate external drives. This at least ensures all my files are in at least two places.
By the way, I use SuperDuper for duplicating the drives.
What’s still missing is the off-site backup. That’s coming. I’m a few drive purchases away from making that a reality. It’s not automated, but it’s getting the job done at least.
And I can certainly sleep better!
You’re right about going to the big RAIDS in the future. I’m definitely headed that way.
December 4th, 2008 at 9:50 am (#)
I am all about raw footage and sound effects, and find it very difficult to part with old audio recordings and video footage.. I may just want to use it in another video.. so i dont do much throwing away..
one thing that just came to my mind, a Safety Deposit Box.
you know those are temerature controlled an humidity controlled rooms which will never see your hard drives even nudged..
wrap them in atistatic foam and plastic, so they are in an “as new” condition, and lock them up.
another idea, maybe try using your own safe, those often withstand fires and natural disasters
the bottom of your friends gun safe or camera chamber would be safe..
i feel the most important thing is o have a motherboard with rear power and sata jacks in a PCI slot.,
and put your computer on a big spacious desk with the back easily accessible so u have a nicework space.
simplly take out your new hard drive, carefuly plug it into the back, power on your computer, transfer thedata, , turn off your computer, carefully unplug it, and warp it right back up and put it in the styrofoam ends and back into the box.
label the box that it came in with a magic marker
and put it somewhere it will never be bumped
if you do anything less than that, you may need double backups…
December 4th, 2008 at 7:40 pm (#)
Also, please let me clarify a small point, thank you
—————————-
Drive Set 1: “Back-up” Drives:
A common misconception is that anything taken off of your computer is known as a “back-up”, and 99% if the time, its actually the original copies, simply removed from the computer to free-up storage space.
all of your real “Back-up” drives should be kept off site. as you will never use them or alter them, once they have been originally written, until you have had a catastrophic loss of your normal data stored in your regularly accessible external drives. They will only be used one time. and The back-up drives should be left unopened and stored in a safe, humidity controlled environment, before and after use.
Another misconception is the “Terabyte” being good for Back-ups. A “Terabyte” drive is not an important consideration when doing back-ups because your drives will all be in a storage container, off-site, at least nowhere near you computer (unless you or your friend live in a very small apartment with limited storage space)
Any drive will do, 250, 320, and a 500, whatever is cheapest per Gigabyte.
if all your video/audio files are well-named, you can print out a text-only version of the contents of the drive in DOS, and keep it stored in a notebook, with the drives (but not in the drive’s box), with a corresponding number from paper to box.
———————————
Drive Set 2: Regular “Storage-Only” Drives (often incorrectly called “Back-ups”):
All of your regularly accessible drives contain all the files you don’t use regularly, but may need in the future. These drives should always be kept “read-only” and never altered. And only written at one time, in one go. .. Don’t change any information on these drives, this ensures your pristine back-up drives are always accurate and up-to-date.
This is your original unused footage, stored in outside your computer. It is the original copy, not a back-up, and these drives can be thrown into an external drive case on a whim and accessed frequently, but should still be handled with care.
———————————
Drive Set 3: “Scratch Drives”
These are the drives stored in your computer. This should NOT be the same HDD your Windows is on, and should be a Drive Specifically for Video/Audio only.
This is your Biggest Drive, it should be as big, or bigger than all your other Drives in the 2 sets above. A Terabyte works well here. This is called your “Scratch Drive” and is the only drive that will be re-written and edited and have the actual information tampered with.
As your “Scratch Drive” fills up to the capacity of one of your “Storage-Only” HDD’s, transfer the information to the external Storage Drive, and set that drive as a “Read-Only” Drive. You can then Make a “Back-Up” of that Drive to be kept in safe storage.
Its also a good idea to make a “Scratch Back-Up” of that “Scratch Drive”, especially if your information isn’t filling up Drives fast enough (Which is why smaller 250G drives are much preferred for both Storage and Back-up drives, as they fill up faster, so you can quickly get the information of your computer’s
“Scratch Terabyte/”, or the Largest Drive you have)
I have found that most people are doing things in reverse order, smaller Internal Drives and large external Terabyte Drives which they are rewriting every day in order to keep them up to date, which wears out the drive, and defeats the purpose of creating safe storage.
——————————–
Both Drives should be kept in Safe Locations, with your “Storage-Only” Drives in one side of the shelf, and your “Back-Up” Drives on the other side of the shelf, with separate notebooks for each set.