Tech Meltdown 2011

Tech Meltdown 2011

110815 harddrive

Long time readers might recall a very bad day in 2005 where we had a hard drive die, successfully rescue it and in general, pull a lot of hair out. This was a hard drive that held photos of Chuck and Leta from their entry into my life. Back then, there was 40 gigabytes of critical life data that was in danger of being lost. Hard drives have ballooned to many times that size, so recovering lost data takes even longer.

Last week, I decided to plunge in and upgrade my operating system to Mac OS 10.7, aka, Lion, aka anal probe. I didn’t realize this at the time, but this upgrade occurred almost 6 years TO THE DAY after the aforementioned tech meltdown. I should have known better than to tempt any of the surviving fates. The upgrade appeared to go okay. I spend a few hours upgrading software I forgot to upgrade prior to the install of 10.7; drivers, utilities and apps that I rely on to work productively. I am lucky that most of the drivers and software I need is represented to be 10.7 “ready”. I begin the following day to “do work” on the “computer” that I “upgraded”. I’m noticing some slow downs, but I attribute that to the upgrade as opposed to any other of a million things that might cause a computer to seem slower. New system things like re-indexing the hard drive can slow things down, so no big whoop.

I decided to further tempt fate by choosing to do a project in the recently upgraded Final Cut Pro X, an app that worked great for me when I did the work from home video for dooce.com. Seemed faster than the old Final Cut Pro. I’ll talk about Final Cut Pro X at a later time, so save your questions. As I began to pull in source video files from my scratch drive, Final Cut became suddenly extremely crashy. Not good. Not good at all. I do a lot of searching and a lot of reading of horror stories. This is mine.It’s at this point that I opened up the nerd-dilettante and friendly Console.app. I notice that in between crashes, the kernel of the OS is spitting out the following message:

8/13/11 4:42:05 PM kernel disk2s2: I/O error.

110815 console app icon

This is not a good message. Ever. Note the best answer in this thread on Apple’s support discussion site. I spent half a day trying to get Final Cut Pro X to simply keep event files and not jack the project I was trying to start. It was hopeless. I figured that something in the upgrade hosed the drive and I needed to roll back my system. I had tried Safe Boot, Single User Boot and a number of other tricks to “fix” the bad drive. I tried the 10.7 Recovery Tool. I decided that getting work done was more important than beta testing 10.7. Since I had completed a project in Final Cut Pro X, I was less concerned about that being the culprit.

A lot of things have changed since 2005. One of them is that Apple introduced a great app called Time Machine. Time Machine automatically backs up your hard drive. A lot. I use Time Machine as a first course of action for situations exactly like this. Problem: upgrading to 10.7 and then rolling back to 10.6.8 filled the Time Machine drive nearly full. I was able to roll back successfully and start working again. For a short while. Until Saturday, when I realized that I was facing a near catastrophic hard drive failure. Even copying the critical files I needed (source video) was spitting out I/O errors. I stopped rying to copy files and ran to a local Mac store (Apple stores don’t sell internal drives) to buy a 2 Terabyte hard drive.

I formatted that drive and found, on the Time Machine drive, a backup of the scratch drive. An incomplete backup that Time Machine CAN’T SEE, but I can via the Finder. I’ve spent two days reconstructing the data from the Time Machine backup. Before I assign the malfunctioning drive to the scrap heap, I wanted to ask: has anybody used anything besides Disk Warrior to rescue data on a Mac?

Finally, I usually recommend that backing up files onto a second drive, while great, isn’t enough. For any mission-critical (yes, photos of family and kids is mission-critical), it’s better to back them up on two separate drives and then have a third drive offsite. We usually follow this. Except on the scratch drive. Since we’ve been doing more videos for dooce.com, the scratch drive was the only drive large enough to hold all the source video and captured (and processed) video. And I had backed up some of the files to our ultra-redundant Drobos. Except for the ones that the previous version of Final Cut Pro hides inside a folder called “Capture Scratch”. There sat over 300 gigabytes of processed and rendered video files. Fortunately, the freaky, now visible Time Machine backup has all those files. I’m very lucky. I figure I’ve lost a fraction of what I could have. I’ve lost several days of time, which frankly, is the worst part about the 2011 Tech meltdown.

If you use a computer: Back up your data. The data is worth more than the computer, worth more than your time. After I get it all recovered, I’m going to do a major backup and spend another day getting the new scratch drive sorted. And then I’ll buy another, even more massive drive to do weekly backups of my whole system; the boot drive, the Time Machine drive and the scratch drive. I’ll be moving most of these files over to the Drobo. Normally, I copy all my photos and any other files I work on to the Drobo. That Drobo wasn’t fast enough to do video editing, which is why I bought the scratch drive in the first place. I need to automate copying files from the scratch to the Drobo, as the Drobo is backed up by another Drobo. Precisely for times like this when drives fail. This was one of the to do list items that kept getting pushed to the bottom because I wasn’t editing video or hitting the scratch drive that hard.

Final note: This is the first Western Digital drive I’ve had that failed. I know, many of you have horror stories about any and all drive manufacturers. I have them as well. However, I’ve had an extremely good run with the Western Digital Caviar Black internal drives.