My Digital Shoebox, Pt. 1: Photos

Self Portrait (unedited), 300D shot #26, 7 January 2005

Not to start this post with the typical "I haven't updated this in a long time...," but I haven't updated this in a long time. I basically treated this like a Tumblr (before finally creating a Tumblr [and one I would actually stick with]), which I knew wasn't ideal for the blogging format, but I didn't care. Really, I most often used this to test out technical stuff, so there's a shitton of private posts I'll have to clean out later.

But I've been thinking that I want to return back to updating this to do more as what Stu Maschwitz likes to do with his ProLost, which is basically to think aloud and share ideas with his followers. This blog, of course, has no followers, but I still need to maintain my Googlejuice somehow.

I purchased my Mac Pro tower around 18 months ago, and with it I wanted to clean up and organize my digital archive of everything- every photo I've taken and every video project I've worked on. I installed a 6TB RAID 0 into the tower (3 of the 4 HDD bays with 2TB drives), which seemed plentiful at the time, but is now nearly at capacity in just this last year working on Looking Glass Children's Videos (a good 3.5TB, or 73% of the RAID's use). Early on, I imported all of the files on the several external hard drives I had amassed for the better part of the last decade (an easy feat, considering most of the drives were 160GB or less), so at least I was confident I wouldn't lose any data due to the drives hardware expiring. But now comes the difficult and long process of making sense of everything.

Again, I'm just going to use this to put up my thoughts on what I've done so far, but I'll start this off with what I've done to my photo collection, as that area's mostly finished with the first step of updating and organizing, and I have yet to figure out where to go from here (in terms of how I want to re-consume the photos in the future).

My photos, 7 January 2012, c/o Disk Inventory X

Unfortunately, I didn't log a lot of info before starting. This photo was taken in January 2012, so I was already halfway through the process, and I didn't write down the raw numbers of photos before starting, nor can I now compare the size on disc before and after.

To start things off, I got my first digital camera, a Canon EOS 300D Digital Rebel, in January 2005. My friend had one, and I didn't know much about what to look for in a Digital SLR, so I figured that'd be a good starting point (when I did a study abroad in Italy in the summer of 2004, I knew that would be the last major photo event for me that would be done on film). For the first year, I only took maximum-resolution JPEG's, because I didn't understand the pipeline yet for taking raw (how I'd be able to enjoy and share them). I took about 2,200 pictures this way, sometimes taking a shot or two in raw, but it wasn't until I was in a college and had access to powerful Mac towers that I'd be able to really play with the Adobe Camera Raw Editor and know what I was doing (or at least know how to convert them to JPEG's to share).

Deuce (unedited), 450D shot #1, 14 January 2009

From then on (February 2006) I took photos exclusively in raw. My archiving system back then was to shoot and dump all photos into a folder on an external drive until the folder reached 4.3GB, which I'd then burn to DVD and backup and create a new folder. This was a wise choice, because it easily organized the photos over time, and was a pretty simple ladder system for backup. When I graduated from college and got a full-time job, I had the opportunity to go to Washington DC for the 2008 Presidential inauguration. I figured that would be a perfect time to upgrade the camera body (for which I had only the two kit lenses), and picked the Canon 450D Digital Rebel XSi. I debated for some time to pick the Canon 40D for its larger sensor, but the price difference over the XSi let me easily afford a 50mm 1.4f lens.

I continued this same pattern of shooting and archiving, again, shooting exclusively in raw, until I purchased an intervalometer which allowed me to do timelapse videos (and because I'd be shooting hundreds of photos at a time, I'd shoot in hi-res JPEG's).

Then the video DSLR's started coming out. I borrowed a friend's Canon 550D Rebel T2i for a trip to Austin, and though I loved being able to shoot photos and videos easily, I thought I should wait out until a really great DSLR from Canon gets released (the 550D was still using the exact same sensor as my 450D). I took nearly 17,000 photos with that camera, before finally getting the Canon 600D Rebel T3i in March 2011 (while again on vacation in Austin).

Zach (edited), 600D shot #1, 19 March 2011

And this is the camera I still use today. So far, I've taken 5,600 photos in the year that I've had it, but that number really doesn't capture the amount that I've used the camera. I started producing Looking Glass Children's videos in May, and launched the product in September, and my 600D has been the go-to camera for 90% of those shoots (on a few of the videos, we used a Canon 60D, 7D, 5Dmii, and Panasonic G2). The 600D is one of the cheapest video DSLR's you can buy, and I think I've gotten more than my money's worth out of it. I can't recommend it enough.

So now we get back to storing. By this point in time, I've had a lot of experience shooting digitally, know how to build a good digital pipeline, and now have some 33,000 photos to take care of.

• Convert all photos to the same format

For many, many years, I kept all the photos in the same raw file format they were taken in, wanting to keep raw photos raw. Any time I played around with a photo, it was exported to JPEG, but the raw file remained. And this is where it can get really tricky, because when you start thinking how things will work 10, 20, 50 years from now, it's debatable if I'd be able to open those raw files at will. I believe because JPEG is such a standard format that JPEG files will still be openable 50 years from now, but that probably won't be the case for the 300D raw .CR2 files.

Several years back, the This Week In Media podcast covered a lot of these issues, and talked about the flaws in JPEG's, so I considered converting everything to HDPhoto spec they described (originally Windows Media Photos, now JPEG XR), but this isn't a raw format, just a better JPEG. When I pointed my photographer friend Ian Adams Alexander about this episode, he said he had just finished converting all of his photos to Adobe DNG, which started me thinking that should be the format of choice. Later listening to an episode of This Week In Photography, where Adobe Lightroom Product Manager Tom Hogarty goes into detail about the format, I figured that'd be the safest bet for long-term file use. Not only are there very few other standard raw file formats to compete with, I only use Adobe Photoshop to play with photos, and every other photo editor (if I decide to ever use it) utilizes DNG's.

This process took months (in particular to be sure all files converted and I wasn't accidentally tossing anything out). I used Adobe DNG Converter on default settings.

My photos, 27 March 2012

• Create a standard naming convention

This became a noticeable issue quickly when you take more than 10,000 photos and Canon's naming system only gives you 4 digits. For a while, I was manually adding two extra digits to the list of files, but that got confusing really quickly and wasn't going to work. It became a big issue when I started using the T3i for video, because even though the camera shoots MOV files, the naming continues for every shutter release (meaning the card will read:

IMG_2614

IMG_2615

MVI_2616

MVI_2617

IMG_2618

With the amount of video footage I shot this last year, I would easily go through 10,000 shutter releases within 3-4 months. So how should I name files?

yyyymmdd_camerabody_filename

Now, any time I take a photo and dump the card's contents into my computer, I update the info above in Adobe DNG Converter and the program names the files as it converts them to DNG's. The camera body part may be excessive, but I figured this would be a safeguard in case I use multiple types of cameras, continue using old camera bodies after I upgrade, or just generally want to easily focus down on a certain group of photos (all of which have happened).

• Backup

Bubbles (edited), 450D, 29 August 2010

I'm falling behind on this one. Right now, every digital photo I've taken is on my Mac Pro RAID 0 drive. RAID 0 means 3 physical hard drives read as 1 giant drive on the computer. Nothing is duplicated, nothing is redundant. If the computer is damaged by fire, all of those files are lost. If one of those discs on one of those hard drives fails, the whole system fails. Why did I do it this way? Because hard drives are still pretty robust these days (compared to 10 years ago), so the likelihood of one platter failing in the drive after a year, instead of immediately when I first got the drive, is low. The raid is more likely to run into trouble if tower itself is in physical trouble, so having duplicate drives in the same physical location doesn't feel as practical to me.

Fortunately, all of the photos represent only 370GB, so I can easily back them up to an external 1TB drive. Another benefit to DNG is that they're on average 20% smaller in file size, which adds up for 33K photos. I couldn't find a general photo database program that I liked (more on that later), so I'd have to backup the files manually. I found arRsync just a few months ago, and liked how clean and simple it backs up folders to an external drive, and only adds what needs to be updated (most applications completely rewrite everything, which to me risks losing files). So everything is at least duplicated to an external drive, but it's only one drive, and it sits right next to the computer. Ideally I'd store this drive in a different room, or even a different building (eventually I'll store this at my folks' home in a different city).

I should backup to DVD discs, and burn new discs every 5 years. This worked for me very well for the last 5 years, but it's now more questionable if we'll have disc readers 5 years from now.

Mentor (edited), 300D, 27 July 2008

I also should back them up to the cloud, but 370GB stored in the cloud is a TREMENDOUS amount of money, so that'll probably become a consideration when price goes down, which will probably be around 5 years. I've considered printing out every picture to hi-res JPEG's and automatically backing them up to Flickr privately, so at least I have something in the cloud for only $40/year, but each cloud-based service is less likely to be around for 5 years, so there's not a whole lot of practicality (it sounds like Flickr is being adopted into OS Mountain Lion, so they may stick around longer than I would have thought).

• Enjoy the library

This last point is elusive because I don't really know what I want. I just know I don't want any photo archive program that's out there now.

iPhoto does not handle large libraries of photos well, duplicates and erases photos automatically (which is easy to accidentally lose photos and have a mess of duplicates in the same folder), creates multiple-size copies (wasting space on the drive and complicates things), and I don't like the interface.

Picasa has a terrible interface because it shows you every single media file on your computer- software logos and slices, videos, application files, etc. It's date organization is messy, and the way it backs up to its cloud service is expensive.

I'm using Adobe Bridge now. It definitely has kinks when it comes to going through the library, but I feel like this will probably be the most useful, because I only intend to use Photoshop (and one day Lightroom) for photos. I played with Lightroom a little, and I know it's good for large libraries, so I'll have to get into it to see if it'll make sense to me.

I've never used Apple Aperture, but a lot of professionals aren't crazy about it, and I use Photoshop and After Effects more than ever, so I want to stick with that pipeline as much as possible. Another surprising benefit is the AppleTV accepts DNG files for its screensaver, so whenever our TV is idle, all of my photos play on the big screen.

Dragon Kite (edited), 600D, 9 October 2011

There are 1001 photo sites online now, only some of them wanting to address the problems I've had, but nothing's really impressed me much in the last few years that would make me rethink using Bridge. In the last 3 years, I dropped Flickr for 500Px and Facebook, but I'm also finding I'm not as compelled as I used to be to share all of my photos with the world. The truth is, the more I shoot and play with photography, the more I realize there's a lot more I need to learn and practice.

In future posts, I'll go into how I developed video pipelines for my own projects, and pipelines for working with teams of editors (often remotely), as well as my thoughts back on photography and scanning all of my film-based work (which I'm still researching).