About mid July of this year, I realized that my current storage solution had reached a breaking point.  With about 44TB of digital storage for both stills and video spread out over 3 Western Digital MyBook Duo drives and coming to grips with the fact that I was adding about 3.5-4TB of data every year, I decided I needed a fresh approach to the problem. This 44TB was backed up and archived on offsite drives as well so the total number of drives was rapidly increasing.   My wife had a replicate copy of those 44TB sitting on her computer, but there was no backup of her drives (not great!).

There are about 363,000 digital stills and an unknown number of video files on external drives.  The still digital files were all linked to two different Lightroom CC catalogs and the video files to two separate Final Cut Pro X libraries.  My Lightroom catalog over time has diverged from my wife’s catalog, but the FCPX libraries had not diverged.  This divergence creates some difficulties to think about.

Problem to be solved: Reduce the every expanding external devices with a limited amount of storage (16TB) available for individual external drives.

Current workflow: I use to shoot professionally, but now my time is spent mostly on photography of my family.  My workflow has been very simple.  I load new pictures onto a external WD passport drive (4TB) into a working Lightroom Catalog, once done editing I import the working LR catalog to a master LR catalog on one of my 16 TB My Book Duo.  The 16 TB drives were backed up and archived (an additional 32 TB of storage space).  I also move the working library on to a 16TB My Book Duo that was attached to my wife’s computer so she would have access as well.

After looking at the options and talking to a number of different people, I decided to build a network storage device (NAS).  Presumably, such a device would allow me to substantially increase my storage capabilities, allow my wife and I to share the same set of digital negatives (so I would not need to replicate them) and have a single backup and archive copy of the NAS.

We have three computers: a 2011 iMac, a 2014 iMac and mid 2012 MacBook Pro (Retina display).  These computers were linked to the external world via an Airport Extreme and Xfinity cable modem, but not really linked together.  So for the NAS, I wanted to set up a local area network so all the computers could attach to the NAS.  

HARDWARE: This is a list of the pieces of equipment I purchased to accomplish this goal:

1. A Synology 2415+: This is a 12 bay NAS and should satisfy my needs for a long time.  But if not it can be expanded to another 12 bays using a DX1215.  I also added a 4GB Synology DDR3 RAM Module (RAM1600 DDR3-4GB).  

2. I populated the drive with 8 10TB Red Pro NAS-7200 RPM Class 6 Gb/s 3.5 inch drives.

3. For my switch I purchased a D-Link 8-Port EasySmart Gigabit Ethernet PoE Switch (DGS-1100-08P)

4. Cat 6a cable

5. An APC Back-UPS 1500VA Battery Backup (SMC1500) because I wanted to protect the system electrically.

The following items are directly connected to the D-Link: iMac, Synology 2415+ NAS, Airport Extreme.  The cable modem is connected through the Air Airport Extreme.  My wife’s computer and my MacBook Pro are connected wireless through the Airport Extreme.

SET UP: The physical configuration of these pieces was relatively straight forward and the manuals and/or online support was pretty clear.  Initially, I formated the NAS initially as AFP, but learned later I should have formated it as an SMB drive.  The reason was I wanted to open FCPX libraries residing on the NAS and that required a SMB formatted drive.  This late learning cost me a fair amount of time as I need to rebuild the NAS.

Initial populating the NAS: This task required moving large amounts of data from my existing external drives to the NAS.  Initially I tried to use Carbon Copy Cloner (CCC) but my previous work created two obstacles.  First, my legacy was a migration from Aperture to Lightroom a couple of years ago and I had number of old Aperture Library just in case I might need them.  It turns out those Aperture Libraries have a huge number of very small files and this has a hugely negative impact on trying to copy files over to the NAS.  These old Aperture Libraries were unlikely to be important to me in the future.  Second issue was I had a number of files with special characters in them: mainly a ‘?’. While that was ok on an AFP formated system, the special character was not allowed on an SMB formated system.

Eventually I was able to copy the complete content of my external drives over to the NAS by connecting my external drive to the USB ports of the Synology and using File Station to move them over.  This allowed for transfer rates up to about 80-100 MB/s. But when I tried to create a backup of the NAS to a new drive (I wanted to save my original drives for a while) using CCC I received a number of errors.  These errors only occurred in old iPhoto libraries or .aplibrary files and were only a couple of 100 files out of 10’s of thousands.  All of the errors were Permission denial errors or special character errors.   

After some discussion with Mike B of Bombich software and a fair amount of work on his part, he suggested to delete the offending files on the NAS and then copy a fresh version from the original source on the NAS.  This worked!  I am speculating that some file attributes were not copied with File Station, thus resulting in the denied permission, but I don’t really know.

LESSON LEARNED: make sure there are NO special characters in any of the titles of your files and avoid data structures that have LOTS of small files, like old Aperture Libraries.  Another thing I learned was I could signficantly reduce the amount of files needed to be copied by simply cleaning of my original hard drive of data that I really no longer needed, but had retained ‘just in case’.

I now have three shared folders on my single volume NAS.  One for my 2001-14 pictures, one for my 2015-xx pictures and one for all of my video content.  I have been successful in backing up these shared folders without problem using CCC at roughly 30 MB/s.  My Lightroom catalog resides on an external SSD drive on my computer, and a preliminary test suggest that my FCPX library will actually open and run from the NAS just fine (more on this later).

At this point, I discovered Smart Previews in Lightroom.  These allow me to edit images without being attached to the original digital file.  I did a small test and discovered that Smart Previews were roughly 1 MB in size and it really allowed me to edit the picture and allows me to post Facebook pictures and email pictures.  My combined picture library (2001-14 and 15-xx) was 363,000 pictures so a 1 TB SSD drive could easily handle the complete Lightroom catalog, previews and smart previews!

At this point, I had to go back and create Smart Previews for all my pictures.  Not because I needed them, but it is not uncommon for me to be traveling and someone wants to post a picture on FB or email it to a friend.  NOW I have all of the pictures available!

LESSON LEARNED: If you want Smart Previews for your files, build those before starting your migration your originals to the NAS.  Building Smart Previews with the originals on the NAS is not particularly fast.

Connecting my LR catalog and my wife’s catalog to the NAS:  With my two separate LR catalogs, 2001-14 and 2015—xx residing on my SSD external drive, I made sure my original data files were disconnected.  When I started up each catalog on the SSD, the folders showed a question mark, but I was able to use the find the folder function in LR and very quickly, my catalogues were now connected to the NAS folders.  Once I had the individual catalogues connected to the NAS, I combined them into a single catalog and I had all 363,000 pictures in one catalog!

I repeated the process with my wife’s catalogs and since she had never changed the location of the original digital files, using an identical process, I was able to connect her catalogs to the NAS and combine them into a single catalog.  I am connected via an ethernet cable, but she is only connected via WiFi.  Her catalog is stored on an internal drive on a newer computer, but the speed of opening this large catalog is quite speedy and faster than my opening speed.  I suspect that my bottleneck is going through USB 3.0 to my SSD external drive.  CAVEAT: If either of us make changes to the processing of the images, those changes are NOT transferred to the other person’s catalog, but we can use the identical set of original digital files.

For pictures there was one last step: transferring new pictures into my catalog (originals sitting on the NAS) and then also transfer them to her catalog.  For my catalog it was relatively simple.  I opened my Master catalog (contained links to all 363,000 pictures) and imported the working catalog into the Master catalog while moving a copy of the original digital files onto the NAS into their proper location.

Moving the working catalog into my wife’s Master catalog required some additional steps.  

1. Opened my wife’s Master catalog and imported the working catalog (remember this working catalog resides on a Western Digital Passport with the original digital files) BUT, and this is important, I did NOT move the original files (selected that during the import process), but left them in place 

2. After the import process was completed, I closed my wife’s Master catalog and ejected the WD passport drive that contained the working catalog and primary digtial files.  I then re-opened my wife’s Master catalog and as expected the recently imported catalog information had question marks, indicating that the link between the Master catalog and the original files had been broken.  More importantly, the Master catalog showed the location to be the passport drive.

I then used the ‘Update Folder Location’ that is in the pop up menu with a right click on the folder.  Using the dialog box, I navigated to the folder location on the NAS.  When I did this the folder appeared as it should in my wife’s catalog with the parent folders properly displayed.  I repeated this process for each remaining ‘lost folder’ and used the ‘Merge’ option.

At this point I had accomplished the following critical steps:

1. Moved my original files to the NAS

2. Created Smart Previews for all of my pictures (optional)

3. Moved my Lightroom catalogs to an external SSD drive (Mobile catalog)

4. Linked the Mobile Catalogs to the NAS original files

5. Consolidated the two large Mobile catalogs into a single MasterCatalog

6. Backedup the original digital files to a fresh backup drive using CCC (Backup and Archive strategy for the NAS now working)

7. Moved new pictures and edits into both my Master catalog and my wife’s Master catalog

FCPX AND VIDEO CONTENT

All of my video content was managed in a FCPX library.  To move the original and rendered files over to the NAS was a relatively straightforward task.

1. I created a new library (MasterVideoLibrary) on my external drive that contained all of the managed video content, but instead of have the MasterVideoLibrary manage the content, I only wanted it to reference the original media so the MasterVideoLibary would be relatively small.  I also wanted the orignal media on the NAS so all of our computers should access it, when needed.  To accomplish this task, when I created the MasterVideoLibrary, I changed the location for storage by selecting the new MastervideoLibary, File—>Library Properties and a dialog shows up on the right side that looks like this:

By selecting Modify Settings under Storage Locations, I was able to set the storage location for media and Cache on the NAS in a folder called MasterVideoCollection.  Once this was done, I simply dragged and dropped events and projects from my existing library (thus making a copy, but not moving them) into the MasterVideoLibrary.  In this process, FCPX rearranges the media into a strict date created folder system.   In the end, my MasterVideoLibrary was less than a GB of data pointing to the NAS for the 21250 original media files.  This MasterVideoLibrary resides with my Master Lightroom Catalog on the SSD.

WORK FLOW CONSIDERATIONS:

For small editing projects, the speeds seems adequate to do some editing in FCPX with the original media residing on the NAS.  However, for larger projects, I will create a new Working Library on a fast external drive and copy the files over to the Working Library by setting the location of the original and cache media to the external drive and dragging the events/projects from the MasterVideoLibary to the Working Library.  Once my editing is done, I will copy the updated project and events back to the MasterVideoLibrary and then delete the content of the Working Library.  I have discovered that if I do not directly touch the original media (that is only work through the FCPX window), when I copy back it does not add duplicate original media, but simply relinks to existing original media stored on the NAS.  This copy back to the MasterVideoLibrary tends to happen very quickly and determined by how much new original media was added to the Working Library.

CONCLUSIONS:

1. Make sure your original external drives have been cleaned up before moving the data over to the NAS.

2. I would set the NAS as an SMB format, but that may be a personal preference.

3. Make sure there are no special characters in your file names.

4. If you use Lightroom and want to be able to edit while not connected to the original media files, set up Smart Previews before you start moving data over to the NAS.

5. The Synology 2415+ and other necessary pieces of hardware were relatively simple to set up.

6. Carbon Copy Cloner is an excellent backup tool.  Learn how to use backing up individual folders in addition to whole drives. File Station can be used as well.  Much to my surprise my Journaled HFS+ drive was visible when connected directly to the NAS.  I had originally planned on directly connecting my back up drives to the NAS but did not like the backup solutions offered.

7. Have a step by step plan on the moving and organization before you start.

8. My final structure looks like this:

     a. My MasterLightroomCatalog (384GB with all pictures with Smart Previews)  and MasterVideoLibary (960MB) reside on a 1 TB external USB 3/USB C SSD drive from G-Technologies.  There are 363,000 stills and 21,250 video files connected to the Master catalogs/libaries. 

     b. The original media is stored on a Synology 2415+ NAS as a RAID 6 equivalent redundancy.

     c. The catalogs/libraries are connect to the Synology NAS via either hardware ethernet connect (my computer) or wirelessly for my wife’s computer or my laptop.

     d. Speed of access for the Lightroom Catalog for my purposes is more than adequate and workable for FCPX access.  Large FCPX projects will need to be checked out  to a fast external drive Working Library for significant work projects.