State of Emergency

posted Sep 4th, 2008 View Comments

Pedal * Blog

Or should I say: “state of #@%$ing emergency”. I’ve always tried very hard to not let day to day problems build up and cause high anxiety in my life – I know that taking my first bicycle trip in ‘01 has helped with that greatly. Whenever I get stressed or run into road blocks, something in my head has always snapped me back to the road, and it “turns the volume down” on what ever I’m dealing with at that moment. Having said that… I think both Amanda and I are about to have heart attacks.

I have 3 TBs of storage for Pedal, the first TB is a drive called ‘Desmond’, it housed 70% of the film’s raw footage, the second TB drive, Hugo, was used to copy all the content from ‘Desmond’, both of these two drives where Western Digital MyBooks. The third TB of space, ‘Locke’, a Lacie Big drive, contained the last 30% of footage and also my Time Machine back-ups for my MacBook Pro.

DSC00560
In the last week, I’ve lost ‘Desmond’, and ‘Locke’ – which means I’m working off my back-up, ‘Hugo’, as a primary drive. And ‘Locke’, well if it doesn’t ever mount again, I’m fucked on that last 35 hours of footage, I’ll be recapturing it and re-logging… assuming I don’t end up in a stress coma.

All this drive failure has left us in a position of being 1 drive away from losing 100% of the last 6 months of work. I plan on calling Western Digital and demanding my money back – but who knows how far that will get me. Amanda and I are, in a state of panic, buying what we can off Amazon. I just purchased a 500 GB Lacie USB drive, which is to be used as my Time Machine drive, I have lots of important Pedal related files that need to be backed up on a daily basis. Our goal is, in the next few days, to purchase six separate 500 GB drives, two of which will be 800 firewire, and the other four will be USB. I’ll work off the firewire drives, and be constantly backed up on the USB drives. If something happens, we’ll replace the firewire drive and continue working.

If you happen to be interested in helping us out, there’s always our “we heart donations” page, which is greatly, greatly, greatly appreciated – there’s also Pedal’s Amazon wishlist, we believe we have the USB drives covered, that’s about $400 we’ve scrapped together to begin to mitigate this problem. If you’re like us, and are broke, then please just keep your fingers crossed for us. We are trying to remain semi-calm’ish.

View Comments
  • Mr. Puzzled in S.B.
    So the nagging question at hand is "What happened with Mozy?" It didn't update the archive fast enough? I thought that was your backup and you were covered.
  • Hey Wane - I know we talked about this in email already, but for other people who might wondering the same thing:

    Mozy is still always backing up my footage, 24/7, but it's doing so over the internet, so it's still a long ways away from being 100% synced. But someday it will be a good last resort fail safe for our footage.
  • I hope you get everything figured out and check your "chip in" account.
  • Hey Brian :) I hope so too - thanks so much for the donation, I used it tonight towards getting new drives.
  • Mr. Learning to be awesome
    I think the problem is these drives aren't designed for massive amounts of video throughput. They're probably designed for low duty cycle applications such as back-up, and can't handle the heat generated by constant use.

    A 'pro' system would certainly have much better cooling and bigger power supplies on the drives. I'm actually surprised how quiet the MacPro tower is, considering it has four internal SATA drive slots. The external RAID box I got roars like a small windtunnel compared to the MacPro.

    As someone that's never lost a hard drive, (I'm vigorously knocking on wood... the rest of the day! Okay, I do have ONE firewire drive that might have a problem, just haven't troubleshot it), let's see, I was working up to make a point here... As someone that's never relied on external drives as mainline storage, I would like to point out that drives are generally reliable when they're inside computers or enclosures where adequate cooling is present.

    Let's look for a RAID storage rack for you. Something up to the task, and fill it with drives as you need them. This will be the cheapest and most reliable storage in the long run. My RAID box has 5 SATA slots, and I picked it up on ebay for $150. Then I spent another $120 on a stack of ten 160 GB drives. Yes, you can get drives that cheap, as data centers upgrade to the larger units they unload their inventory (both used and new) for next to nothing.

    Since you don't have an external SATA port, you might also consider using 1000Mbit Ethernet (aka Gig-E). I would avoid USB-2, they advertise a 'peak data rate' of 480 Mbps, but their latency bites into the data transfer and brings down the average considerably. And FC doesn't like USB as I recall.

    Call if you need a hand. We'll figure this out.
  • It sucks cause they should be - they are built with 800 firewire ports :/ And as for the heat, that's what really confuses me, I had a fan set up blowing cool air on them 24/7, they were always cooled-off when I touched them to check.

    Yes, I hope that doesn't change for you regarding the drives! Thanks again for all your help with the RAID, I'll be writing a post soon thanking you and everyone else who has helped out so much in the last day!

    The USB drives we bought are just for backing up - yea, FCP would freak out if I tried to ever edit off them :P
  • Mr. Learning to be awesome
    I'm thinking there must be some common denominator here to explain this rash of failures. So what do we know?

    1. You're editing HD video. But it's still in some compressed format, so we're not talking about extraordinarily high data rates. But you're doing it in marathon sessions fueled by cases of RedBull. And I think once you get some momentum you work pretty fast.

    2. You've had some interconnect problems. Mostly bad firewire cables. Life on the road is hard, especially on laptop computers.

    3. You've had some warm weather lately, but nothing like the peak of summer. You have A/C and seem to be using it.

    4. You have them sitting on the floor. On or near carpet.

    6. You said they were high density TB drives.

    7. You found out Western Digital doesn't ventilate their product very well.

    8. You raise small furry woodland creatures that like to nest in your hard drives. Oh wait... maybe that's just me. I've never seen them, but at least that would explain those dust bunnies they leave behind.

    ***
    If you're certain your drives are running reasonably cool, then maybe there's something else contributing to their mortality.

    I'm thinking out loud here, but you've had your share of cable problems, and perhaps you have a situation where the drive gets bad signals and has to do lots of head seeks and retries. All day long.

    I notice a lot of dust collects around vents when things are near the floor. More so than I would expect. Even in pristine lab type computer rooms it's amazing how much dust will find its way into fans and filters. So keep them somewhere isolated from dust and where they get better air circulation.

    And there's the density issue. TB drives are cutting edge right now, and maybe they have some reliability issues. I remember how floppy drives went through the same thing when they first came out. Staying with lower density drives might be safer.

    Moving to ethernet will eliminate any problems you might have on your Firewire port. Maybe the motherboard connector is just plain worn out, or got zapped by static one day. We'll just have to see on that one.

    What we need is a diagnostic program. Something that exercises your drive and FW port and tells you how well it is working. Preferably something that gives you real-time error counts so you can wiggle connectors and see if they're solid or not. Anyone know of such an ap?? If so, chime in!
  • All my drives for Pedal are sitting on the second shelf of a bookcase with a fan in front of them - I think one change I'll make is to turn the fan and have them blow air off the drives, instead of on. I'm hoping that will be suck all the dust out of that space.

    I'm hoping when we take 'Locke' to get the data pulled off that maybe they can tell us what the problem is.
  • Mr. Learning to be awesome
    Make that 'can's of RedBull. Just to be accurate. :)
  • "Can's" is more accurate - I'd be dead by now if it were cases :P
  • Mr. Puzzled in S.B.
    I was exaggerating for effect, but realized 'cases' was a bit extreme. Writers do that, you know : )

    And while we're on the subject of getting wired, I'm not sure you have a hard drive problem as much as a Firewire interface problem. Check the drives on a different computer with a different cable. And I'll look for that diagnostic test.
  • Question: how to the pros handle massive data storage? Going HD is going to mean more and more storage... My brother is an animator at Dreamworks and I'm sure they have some huge storage array. But can't indie filmmakers come up with a workable micro version solution? Do tell when you find out.
  • Hey Mark - when I worked at NBC they had an Apple RAID set up in the editing room that all the computers accessed *shrug* Your brother is an animator at Dreamworks!!?!? That's awesome :)

    But yes, anywho, I'm working on a solution *fingers crossed* I'm hoping that mirroring everything over USB drives is a good solution... it's definitely cheaper... ahhh!
  • Goodness me, what a horrible situation. I hope the drives come back online: how can they just fail?! And two of them?! Western Digital need to man the f*ck up on this.

    Bets of luck with your contingency plan, I don't want this to break you guys.

    Take it easy ok?
  • Hey Paul :) I really, really hope so too - I'm kinda scared to plug it back in tonight... if it doesn't come online I might cry a little. Yea, I'm looking forward to the phone call to them... bastards.

    I'm taking it as easy as I can :P Thanks for the support.
blog comments powered by Disqus