NAS Notes

FOREWORD

Each year, I’ve been upgrading externals. From 100GB to 250GB, to 500GB, 1TB, 2TB, 3TB, 4TB....  Each time, I would upgrade in pairs to assure proper backups.  When it came to for the upgrade to the 8TB externals, a friend of mine suggested I make the switch to NAS.   

In the early days, NAS units were a "black art" to me.   Units were built around proprietary operating systems and reliability was dodgy.  Over the years, these gadgets have matured nicely around Linux cores with easy interfacing to Windows and Mac.   Taking my friends advice, I researched for about a year and when the horizon appeared on my 8TB externals, decided to squeeze the trigger.  As usual, each project begets all sorts of side projects.   This one has been no exception and at this time, I'm three months into the implementation.    I've cobbled up a few notes along the way of what happened to me.   Hope they help you...

HOW LARGE?   After looking at my yearly needs and mindful of my past upgrades, 16 TB of storage seemed to be the amount to suit my needs for awhile.   Following the natural inclination to size up and catching a source for the 10TB at at the best price point, a 20TB in a RAID 1-0 configuration was selected.   The trade-offs are faster recoveries with smaller multiple disks - With the larger disk formats, recovery time increases into the realm of inconvenience.  So, the 1 - 0 “supposedly” restores quicker given the spare is on hand, at least that was the plan going in.

The question form on the Synology website consistently recommended the DS418.  I took this question for test many many times, researching units over and below the recommended DS418.

I thought long and hard on their DS918+ with many cool bells and whistles.   In the end, I realized it would take me lots of time I didn't have to learn how to take full advantage of these features - Plus, I was on course to fill my existing externals within a couple months.   The money saved (by choosing the more affordable DS418 ($350 vs $550 plus extra for added memory) made it easier to select the 10TB drives that became an attractive buy.   It was possible to order a fully populated and tested DS418 but that would take away lots of the fun.  The better units were indeed better but provided capabilities for services I had no idea about so the DS418 it was...   Your Mileage May Vary....




ORDERING AND ASSEMBLY - Straight forward...   Newegg it was.    There was a price special on the 10TB WD Reds that had these units in the best price sweet spot.   The following week, that was no long the case.

Add caption

The instructions don’t come with words and no tools were needed - Everything snapped together. 

Time to configure.   The word is that with larger drives, the recovery time naturally takes long.  The present restoration time for my external 8TB USB 3.0 drives is about a day already so I initially opted for a RAID 10 of 4x10TB (redundant 20TB drives).  The 1 - 0 “supposedly” restores quicker given the spare is on hand, at least that was the plan going in.  However....

During startup, the Synology decided it new better and automatically concluded that 27 TB of SHR was ideal (1 drive tolerant with graceful upgrades and Synology's proprietary BTFRS for continuous soft healing).   Being tied to a proprietary setup was not what I originally wanted or planned. Plus, the SHR and BTFRS in my mind would add overhead and forever lock me into Synology.

I read how to to restart from scratch to get my RAID 10 but it was a long day and a clear head would definitely be needed.   After sleeping on it, I gave in and decided just to leave it be I had an adequate number of backups to rebuild from scratch if I indeed decided to.

"You can't always get what you want....   But if you try sometimes, you might find, that you get what you need..."    Jagger & Richards

Uninterrupted Power Source  Almost immediately, it occurred to me that I needed an UPS - We've spotty power and data errors are simply not acceptable.    I was able to get the APC 650 kVA from Office Depot for $63.   This unit has a two USB ports, one assigned to be connected to and to notify the DS418 to perform an orderly shut down when power outages occur.   The NAS has setup menus to guide the shutdown process and  was easy to test by simply unplugging the UPS and watching the NAS shut itself down. 

Update: It's taken until now (15 daze) to migrate my content to the NAS. By avoiding dupes and just focusing on the "meaningful" stuff, I was able to reduce the content size from 6.8TB down to 5.7TB.  This included not only the photos but also music and other personal information.  Apparently, I had been carrying excess storage over the years simply due to changing workflows and had up to triplicate copies floating around.  Imagine that.   

Throughout this initial consolidation, it looked like I would be able to put  4 external drives (24TB) to cold storage.  It was time for a planned family vacation and a job change - A good time to back off the heavy moving and refine work flows, backup schemes include remote site backups and remote site access.... 

ARCHITECTURE, HARDWARE - Earlier in the year, I had just upgraded the home wifi after 10 years.   The performance was remarkable.   It also became clear that I simply can't remember everything like I used to.   Throughout the project, it was helpful to keep updated a sketch (Google Slides) of the hardware, port assignments, and node assignments.   Not only does this help keep things straight, it also helps identifying bottlenecks.  Be sure to keep your usernames and passwords neatly organized on a separate encrypted file.

Along the way, keeping in mind the mantra of  3-2-1 (three copies, two different media, one remotely stored) was a foundation for many of the decisions made along the way.

ARCHITECTURE FOLLOWUP - A hardware centric drawing wasn't enough - It was also necessary to map out my workflow.   I'm a photo journalist and write for a monthly publication soo my workflow has become somewhat established.   Depending on what you do, you may find doodling out a workflow map handy.   Dunno.....    Just thought I'd mention it.  

Years ago, I had Lightroom catalogs by the month as that coincided with my monthly magazine workflow.   I eventually found that was impossible to peruse over the years and underwent a consolidation to yearly catalogs along with catalogs for special projects.   In early August, while setting up remote site synchronization, I began an attempt to merge "all" off my Lightroom catalogs into a single catalog - I still maintained backups of all the prior catalog scheme.   I made the mistake starting this task upstairs on the MacBook Pro on the SSD.  Within a couple years into the project (1970-2012) I quickly realized that the SSD had not enough space for Lightroom to work plus it was away from the NAS relying on wifi connection and connected to the disks via USB 2.0 (duh).   It took TWO DAYS to do a single year of 27,000 image importing.   I took the job downstairs (using my new Resilio folder synching) to the iMac which was hard wired to the router/NAS, imported catalogs off the downstairs SSD and accumulated the single catalog on the USB 3.0 HDD and back on track from the years 2014-current all within a Saturday.   Takeaway - Depending on how much heavy lifting you're doing, where you work within the Architecture is "very" important.   Takeaway - Use the available activity monitors to help judge performance.

BACKUPS (not Synchronization) - Before, the only automated backups running is Time Machine on the two macs.   Each month, my workflow would have me manually backing up content amongst the external drives.   This monthly backup would tie up machines for about a full 24 hours.

At the present time, each Mac has automatic local Time Machine backup.   Each Mac also maintains a copy of work space catalog files locally on external HDD & SSD and on the NAS making for three copies on two different medias.  

The large RAW library also maintains three copies on two different medias - 1 on the NAS and 2 on external USB drives.

I'll get to the remote site backups down the road.

DATA LOCATION - During this transition, it became obvious that data transfer rates depended on where the work was performed - The need for a data/work location map became obvious.   I expect this is an area that will vary amongst people as they work with video, photos, music, documents, etc...

By veering from the DS918+, I had enough cash to bring in Patriot Burst 480GB SSDs at each of the MACs to handle the current projects.  These are this years new SSDs from Patriot and at first blush, I thought I got ripped off as they have a reduced memory cache size of 32MB compared to 500 MB of cache in prior models.   Turns out that Patriot instead opted for a much improved memory controller which closely aligns in performance - Twice the usable space for the same price plus in my application, was RIDICULOUS FAST!   Setting these two SSDs into the workflow required some tweaking on where data would reside but was well worth the exercise.  The price was good enough for the Patriot Blast 480 gb SSD, I got a third.   This extra unit has really been a help debugging and setting up work flow.   Go figure... 

Also at each Mac are 4TB external HDDs which provide adequate speeds for photo and audio.   

FAMILY SHARING - This was one of the reasons why I went with Synology, to share family photos.   Presently, I'm paying about $190/year for a Zenfolio business store front.   The store front gets very little action and the site is essentially a fancy viewing gallery and place to hand off work to my magazine editors.   Google Photos is another alternative for family sharing - already, I use Google Photos to back up pictures from my phone.   Google has a habit of changing things every couple of years.   HP had the same habit and while providing good solutions, would leave users high and dry.   I've time - To be continued...

FILE MANAGEMENT - While the NAS was good for robust storage, this robustness made for slower file management.  Transfer speeds of a few small files is nothing, the hardware kicks but.   It it is the catalogs where a large number of large sized files are involved where the breakdown occures.  In addition to moving at a snails pace, this would hold up file managing in the faster workspaces.  This pointed out the need to develop and adhere to a thought out work flow.  Tasks like RAW image culling is best done on the faster SSDs.



FILE INDEXING - File Indexing is used by  the Synology App, "Drive".   It seems that fine indexing is a must if you'll be using the mobile app "FILES".   File Indexing takes place as a background task and takes place over a period of days/weeks/months for large initial loads.  Not knowing this made for annoying times.  

For reference, the RAW library was closely near 400,000 files while the audio library was around a mere 28,000  files.   It's August the 4th and the damn thing is still indexing...   The good thing is that you can switch the indexing feature to pause if things gets bogged down which in the beginning, is often.

This is perhaps where the advanced DS918+ with additional CPU and Memory would possibly shine.  Said differently, it became better obvious to use a NAS as a NAS and to perform the fancy tasks at the CPU level making it clear the benefits of a planning the architecture layouts and workflows.  Decide if you need to have your Files Indexed for the native Synology Apps.   

FILE INDEXING FOLLOWUP:   After uploading my data in early July, the bulk of the indexing appears to have calmed - It is now mid August.   About 400,000 raw images, catalogs (and previews), and music.    I believe it was the previews that really bogged things down.   In hind site, I would likely not have made so much content subject to being indexed.




FILE INDEXING, NOT FOR ME - I yanked the cord on Drive and File indexing. The number of photos I'm working with (nearly 1/2 million) simply overwhelm the indexing effort.  For what I'm using the NAS for, indexing doesn't seem important - For those needing to store a bunch of doc files, indexing makes sense.

MUSIC - Before I got into photography, a did lots of audio production (recording, mixing, mastering and distribution) and accumulated an honest collection of music.   After giving Synology's Audio Station a try, I yanked it and went the iTunes route.   The entire music collection with iTunes Libraries resides on the NAS.   Audio Station turned out to be buggy and its phone apps had lots of peculiarities.   I may decided to rehost all my audio later.   I've backups residing on the SSDs (portable) and also on drives targeted for cold storage.

Anyway, I did purchase a lifetime license to Plex to try.   After a week, it seemed that it was a tool that needed to be fed rather than a tool that would work upon command.   Presently sticking with iTunes.



MUSIC, FOLLOWUP - In my early years, I was heavy into audio recording and captures lots of live music performances.  This means that the services (like Plex) would often get the metadata for the music flat out wrong.   I'll be sticking with iTunes as I've invested lots of time cataloging my music collection.




SPARES - The present SHR/BTFRS system is one fault tolerant. Pick your poison.  I purchased one extra 10TB drive for a spare.   All four slots of the DS418 are currently filled.   In hind site, setting things up for a hot spare may be a future possibility.   Don't forget to register all of your hardware - I've read that breakdowns "do" happen and those with proper registration reports prompt no-hassle RMA service.



PORTALS  - Up until recently, I've been a heavy user of the Google Chrome and it's suite of office products.   Overtime, I noticed it was consuming a good share of my system resources.  For general use, I've started using Mozilla's Firefox more but for "infrastructure" use, selected Safari for router and NAS web based portals.  Over time, I started sensing that Mozilla was simply too slow.   On my PC platforms, I'm using Internet Explorer more and more.  



PHOTOS - With my decades of photography safely backed up on external drives, I've the entire collection of Lightroom RAWS and Catalogs residing on the NAS.   LR Catalogs need to run on external drives local to the computer because of SQL rules.   Catalogs may point to either the local drive or to the NAS.   The 480 GB SSD are able to hold the Catalogs and the most currently RAW photo collection.    Presently, synchronization is maintained manually and targeted for automation soon.

REMOTE ACCESS - In the 3-2-1 setup, there is to be 1 remote storage location.  Synology's app, FILES appears to allow me to access any file on the NAS to my phone.  Ahead will be setting up  remote storage site(s).   One thing I'll have to remember is that I'm currently limited to around 12-15MB/Sec uploading speed and have an upload limit before higher data rates apply.

SURVEILLANCE - I thought long and hard about using my NAS to store security surveillance video.   This is a field I know little about abut would like to explore in the future.

SYNCHRONIZING -  At the beginning, this was one thing high on my priority list and early on, this was annoying.   I did lots of research reading the forums on this to prepare.   As it turned out, most of the apps that looked favorable had long since been overcome with newer apps. 

Another trouble area was that I located located the NAS beneath the home router which resides beneath the cable model/router.  This presents addressing issues which may be solvable by setting the proper bridging instructions in the cable modem router.     

I "believe" the free Synology synch apps (Drive) has my data on the NAS being sent (encrypted) to the Synology Server where it is passed through, returned to my router and wi-fi'd to the laptop upstairs.   Given that the data can only flow as the slowest link, update speeds using this path peaks at and annoying 10-15 MB/S - Not good when synchronizing large files.

Researching suggests that using Resilio Synch, a peer to peer tool, is promising.   Once the link is set up, transfers proceed in a peer-to-peer manner so if you've a fast home network with Gigabit Ethernet or a fast wi-fi,  transfer rates "should" hit around 120'ish MB/Second.   Early tests are hitting 20MB/Second so I've a connection issue I'll need to work - Good thing I've an Architecture map to help me run this down.   This is seemingly the preferred path with a copy of Resilio Synch ranging from free to $100 for a life time license.   

A third path would be to use the Remote Management tools that are available through Apple.   You would think that this would be a bug free approach but my history is that Apple makes money selling software and proceeding on this path, I feel, will have me dealing with update issues in the future (iPhoto, Logic, Aperture, etc. comes to mind).   If you go this route, good luck and let me know how it goes.

The plan is to set up small sharable folders on the various computers.    These will be called Transport Pads that will contain files I "must" have synchronized (i.e. Lightroom Catalogs and current photo projects).



SYNCHRONIZING STATUS I - I've learned how to enable synchronized files using both DRIVE and Resilio Synch.   I'm a ways to go before I master either and presently relying on manually sneaker disk between the machines to keep things current.   Here's the rub - I'm working on a 10 year project and would like to peruse the 10 years worth of material.   Synching 10 years worth of catalogs, I'm learning, is ridiculous.   Others have written only synching their most current projects which presently is not in my workflow.

The Catalogs are presently over 82,000 files and 211 GB is size plus what are in the preview folders.  This appears ridiculous to synch.   I will have to revamp the workflow to a smaller current project synch.  To be continued...

SYNCHRONIZING - I think I'm done.   I learned that Resilio Synch requires a type of memory map of the spaces to be synched so that it know when changes takes place.  Makes sense....   That it also keeps this memory map in the computer's RAM makes sense too.   Until the time you realize that the size of the Catalog requires a total mapping space of nearly 3.5gb and your laptop you've assigned to tend to the NAS synching is going on the fritz.  Luckily, I had a decent i5 mining rig that was not longer affordable to have mine crypto currencies and so I bumped up the mining rig to 16 gigs of ram and assigned it to cover the NAS synching and with that, I think I've got this part nailed....

DATA LOCATION - During this transition, it became obvious that data transfer rates depended on where the work was performed - The need for a data/work location map became obvious.   I expect this is an area that will vary amongst people as they work with video, photos, music, documents, etc...

By veering from the DS918+, I had enough cash to bring in Patriot Burst 480GB SSDs at each of the MACs to handle the current projects.  These are this years new SSDs from Patriot and at first blush, I thought I got ripped off as they have a reduced memory cache size of 32MB compared to 500 MB of cache in prior models.   Turns out that Patriot instead opted for a much improved memory controller which closely aligns in performance - Twice the usable space for the same price plus in my application, was RIDICULOUS FAST!   Setting these two SSDs into the workflow required some tweaking on where data would reside but was well worth the exercise.  The price was good enough for the Patriot Blast 480 gb SSD, I got a third.   This extra unit has really been a help debugging and setting up work flow.   Go figure... 

Also at each Mac are 4TB external HDDs which provide adequate speeds for photo and audio.   

FAMILY SHARING - This was one of the reasons why I went with Synology, to share family photos.   Presently, I'm paying about $190/year for a Zenfolio business store front.   The store front gets very little action and the site is essentially a fancy viewing gallery and place to hand off work to my magazine editors.   Google Photos is another alternative for family sharing - already, I use Google Photos to back up pictures from my phone.   Google has a habit of changing things every couple of years.   HP had the same habit and while providing good solutions, would leave users high and dry.   I've time - To be continued...

FILE MANAGEMENT - While the NAS was good for robust storage, this robustness made for slower file management.  Transfer speeds of a few small files is nothing, the hardware kicks but.   It it is the catalogs where a large number of large sized files are involved where the breakdown occures.  In addition to moving at a snails pace, this would hold up file managing in the faster workspaces.  This pointed out the need to develop and adhere to a thought out work flow.  Tasks like RAW image culling is best done on the faster SSDs.



FILE INDEXING - File Indexing is used by  the Synology App, "Drive".   It seems that fine indexing is a must if you'll be using the mobile app "FILES".   File Indexing takes place as a background task and takes place over a period of days/weeks/months for large initial loads.  Not knowing this made for annoying times.  

For reference, the RAW library was closely near 400,000 files while the audio library was around a mere 28,000  files.   It's August the 4th and the damn thing is still indexing...   The good thing is that you can switch the indexing feature to pause if things gets bogged down which in the beginning, is often.

This is perhaps where the advanced DS918+ with additional CPU and Memory would possibly shine.  Said differently, it became better obvious to use a NAS as a NAS and to perform the fancy tasks at the CPU level making it clear the benefits of a planning the architecture layouts and workflows.  Decide if you need to have your Files Indexed for the native Synology Apps.   

FILE INDEXING FOLLOWUP:   After uploading my data in early July, the bulk of the indexing appears to have calmed - It is now mid August.   About 400,000 raw images, catalogs (and previews), and music.    I believe it was the previews that really bogged things down.   In hind site, I would likely not have made so much content subject to being indexed.



FILE INDEXING, NOT FOR ME - I yanked the cord on Drive and File indexing. The number of photos I'm working with (nearly 1/2 million) simply overwhelm the indexing effort.  For what I'm using the NAS for, indexing doesn't seem important - For those needing to store a bunch of doc files, indexing makes sense.

MUSIC - Before I got into photography, a did lots of audio production (recording, mixing, mastering and distribution) and accumulated an honest collection of music.   After giving Synology's Audio Station a try, I yanked it and went the iTunes route.   The entire music collection with iTunes Libraries resides on the NAS.   Audio Station turned out to be buggy and its phone apps had lots of peculiarities.   I may decided to rehost all my audio later.   I've backups residing on the SSDs (portable) and also on drives targeted for cold storage.

Anyway, I did purchase a lifetime license to Plex to try.   After a week, it seemed that it was a tool that needed to be fed rather than a tool that would work upon command.   Presently sticking with iTunes.



MUSIC, FOLLOWUP - In my early years, I was heavy into audio recording and captures lots of live music performances.  This means that the services (like Plex) would often get the metadata for the music flat out wrong.   I'll be sticking with iTunes as I've invested lots of time cataloging my music collection.



SPARES - The present SHR/BTFRS system is one fault tolerant. Pick your poison.  I purchased one extra 10TB drive for a spare.   All four slots of the DS418 are currently filled.   In hind site, setting things up for a hot spare may be a future possibility.   Don't forget to register all of your hardware - I've read that breakdowns "do" happen and those with proper registration reports prompt no-hassle RMA service.



PORTALS  - Up until recently, I've been a heavy user of the Google Chrome and it's suite of office products.   Overtime, I noticed it was consuming a good share of my system resources.  For general use, I've started using Mozilla's Firefox more but for "infrastructure" use, selected Safari for router and NAS web based portals.  Over time, I started sensing that Mozilla was simply too slow.   On my PC platforms, I'm using Internet Explorer more and more.  



PHOTOS - With my decades of photography safely backed up on external drives, I've the entire collection of Lightroom RAWS and Catalogs residing on the NAS.   LR Catalogs need to run on external drives local to the computer because of SQL rules.   Catalogs may point to either the local drive or to the NAS.   The 480 GB SSD are able to hold the Catalogs and the most currently RAW photo collection.    Presently, synchronization is maintained manually and targeted for automation soon.

REMOTE ACCESS - In the 3-2-1 setup, there is to be 1 remote storage location.  Synology's app, FILES appears to allow me to access any file on the NAS to my phone.  Ahead will be setting up  remote storage site(s).   One thing I'll have to remember is that I'm currently limited to around 12-15MB/Sec uploading speed and have an upload limit before higher data rates apply.

SURVEILLANCE - I thought long and hard about using my NAS to store security surveillance video.   This is a field I know little about abut would like to explore in the future.

SYNCHRONIZING -  At the beginning, this was one thing high on my priority list and early on, this was annoying.   I did lots of research reading the forums on this to prepare.   As it turned out, most of the apps that looked favorable had long since been overcome with newer apps. 

Another trouble area was that I located located the NAS beneath the home router which resides beneath the cable model/router.  This presents addressing issues which may be solvable by setting the proper bridging instructions in the cable modem router.     

I "believe" the free Synology synch apps (Drive) has my data on the NAS being sent (encrypted) to the Synology Server where it is passed through, returned to my router and wi-fi'd to the laptop upstairs.   Given that the data can only flow as the slowest link, update speeds using this path peaks at and annoying 10-15 MB/S - Not good when synchronizing large files.

Researching suggests that using Resilio Synch, a peer to peer tool, is promising.   Once the link is set up, transfers proceed in a peer-to-peer manner so if you've a fast home network with Gigabit Ethernet or a fast wi-fi,  transfer rates "should" hit around 120'ish MB/Second.   Early tests are hitting 20MB/Second so I've a connection issue I'll need to work - Good thing I've an Architecture map to help me run this down.   This is seemingly the preferred path with a copy of Resilio Synch ranging from free to $100 for a life time license.   

A third path would be to use the Remote Management tools that are available through Apple.   You would think that this would be a bug free approach but my history is that Apple makes money selling software and proceeding on this path, I feel, will have me dealing with update issues in the future (iPhoto, Logic, Aperture, etc. comes to mind).   If you go this route, good luck and let me know how it goes.

The plan is to set up small sharable folders on the various computers.    These will be called Transport Pads that will contain files I "must" have synchronized (i.e. Lightroom Catalogs and current photo projects).

SYNCHRONIZING STATUS I - I've learned how to enable synchronized files using both DRIVE and Resilio Synch.   I'm a ways to go before I master either and presently relying on manually sneaker disk between the machines to keep things current.   Here's the rub - I'm working on a 10 year project and would like to peruse the 10 years worth of material.   Synching 10 years worth of catalogs, I'm learning, is ridiculous.   Others have written only synching their most current projects which presently is not in my workflow.

The Catalogs are presently over 82,000 files and 211 GB is size plus what are in the preview folders.  This appears ridiculous to synch.   I will have to revamp the workflow to a smaller current project synch.

To be continued...

SYNCHRONIZING STATUS II  - Resilio is cool.   I've synchronized folders on the SSDs connected to the iMac and to the MacBook Pro.    The synchronized folder presently carries my collection of annual catalogs along with some data I like having readily available.  I'm thinking of other possible synchronization schemes that I may try later.   One thing I picked up from the pros was that synchronization is not a backup and setting up a remote backup is an important item on my list of goals.

SYNCHRONIZING STATUS III - What I found out about Resilio is that a computer can only be associated with one synched folder at a time - The situation where the Laptop would access its local folder while the iMac access its local folder *and* the folder on the NAS does not work.   I was able to solve this by recalling my Lenovo laptop into service and assigned it to the folders on the NAS.   This will only be temporary as the Lenovo is old and down to 4gb of working memory from its original 8gb.   Resilio took "forever" indexing the Lightroom Catalogs which makes sense as there were nearly 1/2 million files to index.   Update,   I reassigned a Windows 10 based mining rig with an memory upgrade to 16 gb total to replace the failing laptop.    I was able to get the extra 8 gigs of ram at 1/2 the price from when I bought the first 8gb of ram as prices had fallen nicely.

VIDEOS -  Surprise me....   This was an area I was least interested in yet enjoying spending lots of time with.   Go Figure....    At first, I had a few videos on hand, mostly historical clips of aerospace project.  This lead to to VCR conversion and DVD conversions.   Presently, I'm using Roxio Video Converter for the VCR tapes but may explore other tools.    For DVD conversions, I tried VLC, Cisdem Video Converter,  and MacX DVD Ripper Pro.