Dealing with the Pain of Photo Storage

Over the holidays, I finally decided to start a thorough cleanup of my Lightroom catalog and images. Several years of shooting resulted in about 400 thousand images and videos accumulated in native formats, and I got tired of trying to backup my data and sync it between different storage solutions. I took a look at some of the photos in my Lightroom catalog and realized that from personal and professional work I’ve done over the years, most of the images I’m keeping will likely never be touched again. So what’s the point of having them in the first place? It was a time of massive purification…

All photographers start their digital camera journey with a lot of camera clicks, and I was no exception. The first few years were filled with:

• “Experimental” images of almost anything around the house
• Images of the same subject and scene using different camera settings
• Poorly composed images without clear subject(s) or context
• Identical/duplicate family photos
• Photographs of subjects in and out of focus (after acquiring the first prime lens)
• Several similar images of birds and mammals (after acquiring the first telephoto lens)
• Several close-up images of flowers and insects (after getting the first macro lens)

It was also fun to learn more about when I got into HDR photography. Does anyone remember an early version of Photomatix? I remember sometimes I would feed Photomatix JPEG images with different brightnesses from the same RAW image to produce that popular “HDR look”. Yes, single-image HDR photos were a real thing, even if many of us wouldn’t admit it 🙂

people “talk abouthdr hole“, but the “panorama hole” turned out to be very bad for me, because I ended up making panoramas of anything that looked beautiful at the time. Below was my sequence, in this order:

• Horizontal panorama (wow, I can do that?)
• Unstitchable panorama (nodal point and parallax search)
• Horizontal panorama in vertical orientation (more resolution baby!)
• Vertical panorama in horizontal orientation (long print anyone?)
• Ridiculously detailed panoramas (don’t know why, but will do anyway)
• Multi-row panorama (first panoramic top)
• HDR Panorama (Photomatics + PTGUI Craze)
• HDR multi-row panorama (all of the above, time for a computer upgrade)

I had them all – hundreds and hundreds of awesome panoramas:

The worst culprit was a panoramic view where I somehow managed to capture over 500 multi-row HDR images with the telephoto lens. The software at the time and my computer couldn’t even process such a large pano, let alone so many images, so I had to extract smaller sized JPEG photos from the original RAW to stitch it together. In the end, it was a terrible image anyway, but for some reason I kept it all – maybe at the time I thought it might be useful someday? I still don’t know.

Does anyone remember the “pro” tip from the early DSLR days, where we used to shoot multiple photos to reduce the “mirror slap” effect to get sharper images? Found lots of them. How about post-processed images of the same photos in color as well as black and white? check. Images of bricks and walls for lens sharpness? Another check. Images of the same subject zoomed in and zoomed out for no reason? Yes. Fully open and stopped all the way down? a few words. Vertical and horizontal shots of the same subject? Absolutely. “Creamy” Bokeh Highlights? Oh yes! In touch with left, center and right? you betcha. With and without filter? Definitely. On-camera and off-camera flash? Yes to that too.

I could go on and on, and the cycle seems to repeat with more camera gear purchases, new software, and newly-acquired tricks.

All of the above, combined with a lot of terrible images, meant that the cleanup process for my early work would result in a reduction of over 95% in my image storage and Lightroom catalog. This is a great start!

However, the problem turned out to be much more difficult than I thought – trying to view hundreds of thousands of images in Lightroom and delete them is no easy feat, especially when the images are not starred or tagged properly. Lightroom was never good at drawing fast images, so I thought it would be much faster to use this fastrawviewer To delete the images, then go back to Lightroom and re-sync My Photo Catalog to remove all missing files. But how would you do this for 400K images? Sadly, one year at a time, and one folder at a time. While I was glad I kept my Images are properly organized in the Lightroom catalogThe sheer volume of images meant potentially several weeks of cleanup work.

In the end, I still decided to go through the pain and follow the process, no matter how long it took:

1. Backup all images to a large external drive before clean-up (about 16 TB of total data).
2. Consolidate all annual Lightroom catalogs into a single mega-catalog, but keep the images organized by year in the file system. This makes it easier to see the bigger picture.
3. Save all Lightroom metadata to sidecar files for FastRowViewer (FRV) to be able to access starred and marked images.
4. Set FRV to dump rejected images to a dedicated folder, as FRV doesn’t actually delete any data.
5. Go through each year with FRV, one folder at a time, as quickly as possible. Start by rejecting all easily identified bad images (out of focus, poor exposure, duplicates, etc.), unwanted HDR and panoramic images, and other photos that will never be used. Discard any suspicious images.
6. If the “Rejected” folder remains in the main “Photos” folder, delete it.
7. Resync the Lightroom catalog to remove all missing images.
8. Repeat #5-#7, but this time view the images at a slower speed to retain only the strongest groups.
9. Customize the final Lightroom catalog, and rename/reorganize images if necessary.

My biggest savings are expected to be from the commercial work (headshots, weddings, corporate photography, etc.) that Lola and I did for about 10 years from 2009 to 2019. Most of those images will be permanently removed from the working image and Lightroom catalogs. Although I do not expect any of our former clients to request images in the future, I do want to keep the strongest images as part of the portfolio.

If you want to avoid image clutter and resulting troubles in the future, it is always a good idea to properly pull down the images after each trip and photo shoot, then delete the rest. I wish I had done this from the beginning.

more to come…