Article: A Digital Audio Drawback - Music Storage Delicacy, Compatibility, and Longevity

10 minutes ago, The Computer Audiophile said:

 

I rsync this web server from one continent to another daily :~)

 

I am grateful to know that you're diligently saving all my nonsense I type here Chris.  ;-)

7 minutes ago, AudioDoctor said:

 

I am grateful to know that you're diligently saving all my nonsense I type here Chris.  ;-)

HA!

3 hours ago, Sal1950 said:

In my desire for a silent music-video server PC I currently run a Linux OS and have 12tb of internal storage spread across 6 SSD fat formated drives. I have no real use for a networked system so have avoided those complications. All files are stored as either flac or mkv with only few exceptions.

Backup is done manually to an outboard 14tb spinner drive using a rsync terminal command about every 3 months or so. I'm still procrastinating creating some sort of off-site backup in case of catastrophe. My uploading data speed is just too damn slow to mess with, looks like I'll end up with second large spinner drive kept at some friends house or whatever.

Going back to my years passed, I would have never imagined, even just a few short years ago the need for this much storage. But the advent of Atmos coded bluray music discs along with 5.1, etc has spun things out of control and I'm avoiding storage of large video files as much as possible.  😛

Good to shake folks up now and then to remind everyone of what "can" happen.

Backblaze will send you a USB drive you can copy your files to, you send that drive back to them with your files and they upload it to their cloud for you. After that you would only need to upload the new files.

9 hours ago, The Computer Audiophile said:

This is exactly what scares me most. When I discovered that the QNAP HBS utility couldn't backup files with quotation marks in the name, I thought, what else can't it backup? I have no clue.

Doesn't it tell you which files are filtered from the backup?

On 8/21/2024 at 12:08 PM, Bob2803 said:

Tidal and Qobuz are good enough for me. But, "to each his own".

What happens if/when the streaming services "disappear" because they can't stay financially viable?

6 minutes ago, Axiom05 said:

What happens if/when the streaming services "disappear" because they can't stay financially viable?

The whole streaming service library will be lost … the same will apply with any subscription based service like Roon, Audirvana …

19 hours ago, Kal Rubinson said:

Doesn't it tell you which files are filtered from the backup?

To answer my own question......................

HBS keeps a log of the filtered files for each transfer.  I just completed creation of a new backup of about 37Tb of music (and associated info), ~150,00 files.  None were skipped.  Many ".@__thumb/" files were filtered but none of the music content files.  FWIW.

17 minutes ago, Kal Rubinson said:

To answer my own question......................

HBS keeps a log of the filtered files for each transfer.  I just completed creation of a new backup of about 37Tb of music (and associated info), ~150,00 files.  Of them, only 13 were filtered and none were music content files.  FWIW.

I've been experimenting with this over the years and found that sometimes the logs are great, while others times they aren't I've had thousands of files skipped before. Some of them were nonsense files like .DS_Store, while others were real music. It gets tricky to weed through all the skipped files, looking for legit stuff. 

2 minutes ago, The Computer Audiophile said:

I've been experimenting with this over the years and found that sometimes the logs are great, while others times they aren't I've had thousands of files skipped before. Some of them were nonsense files like .DS_Store, while others were real music. It gets tricky to weed through all the skipped files, looking for legit stuff. 

I've re-edited my post but over the span of this backup, no files were skipped.

2 minutes ago, Kal Rubinson said:

I've re-edited my post but over the span of this backup, no files were skipped.

Wow. Very nice.

On 8/22/2024 at 1:47 PM, AudioDoctor said:

 

 

Rsync might be able to identify which files are missing, or the Forklift app on a mac, if you have a known source to compare the backups to.

I use Chronosync on the Mac, which is a front end to rsync I'm fairly certain. Just like rsync it does bit for bit comparisons, so I know what's stored at Backblaze isn't some compressed version that's only accessible via proprietary software, but the very same AIFF files I play from local storage.

There are several front ends for rsync on Linux. Grsync seems to be the most up to date, though the interface is rudimentary. Rclone also has an rclone-gui version.

4 hours ago, Jud said:

 

I use Chronosync on the Mac, which is a front end to rsync I'm fairly certain. Just like rsync it does bit for bit comparisons, so I know what's stored at Backblaze isn't some compressed version that's only accessible via proprietary software, but the very same AIFF files I play from local storage.

 

There are several front ends for rsync on Linux. Grsync seems to be the most up to date, though the interface is rudimentary. Rclone also has an rclone-gui version.

I think Forklift also uses rsync, and can upload to Backblaze directly. It's a great program that I wish had an equivalent on Linux.

add me to the camp that uses local hard drives – both solid-state and spinning for about 6TB of music. 
 

I also have all 2channel stuff on 2 separate Aurender servers. 
 

I don’t buy a ton of new music unless there is an amazing version that is not streamable (typically High Def Tape Transfers) I tend to stream, newer albums I might be interested in, and then if I really like them, I will purchase. I simply keep my new purchases in a separate folder and add then add the new files to the drives at the time of purchase/download of new albums. (but that’s obviously much easier if you’re not purchasing a lot of new music)

I have 2 back up drives in a fireproof safe, and one back up on the desktop.  I had used a Synology for a while, but was getting all sorts of errors and ultimately just abandoned it and do it by hand, the old-fashioned Way. 
 

so redundant local hard drives, 2 Aurenders, and nothing in the cloud for me (although attempted by reading everyone’s comments on back blaze and iDrive)

You seem to favor NAS drives for local backup. Is there any reason for that compared to other types of storage? I have a 4 bay QNAP but I don’t use it anymore.

26 minutes ago, daverich4 said:

You seem to favor NAS drives for local backup. Is there any reason for that compared to other types of storage? I have a 4 bay QNAP but I don’t use it anymore.

I love them for the size of storage I need, accessibility from any device, redundant disk arrays, built-in apps, etc…

7 hours ago, blaven said:

add me to the camp that uses local hard drives – both solid-state and spinning for about 6TB of music. 
 

I also have all 2channel stuff on 2 separate Aurender servers. 
 

I don’t buy a ton of new music unless there is an amazing version that is not streamable (typically High Def Tape Transfers) I tend to stream, newer albums I might be interested in, and then if I really like them, I will purchase. I simply keep my new purchases in a separate folder and add then add the new files to the drives at the time of purchase/download of new albums. (but that’s obviously much easier if you’re not purchasing a lot of new music)

I have 2 back up drives in a fireproof safe, and one back up on the desktop.  I had used a Synology for a while, but was getting all sorts of errors and ultimately just abandoned it and do it by hand, the old-fashioned Way. 
 

so redundant local hard drives, 2 Aurenders, and nothing in the cloud for me (although attempted by reading everyone’s comments on back blaze and iDrive)

Every single one of those disks, solid state and spinners will absolutely fail. Before they fail completely, they will be silently corrupting your data and that bad data will be backed up. Once the good data is gone, it is unrecoverable.

42 minutes ago, The Computer Audiophile said:

I love them for the size of storage I need, accessibility from any device, redundant disk arrays, built-in apps, etc…

Thanks, I think my needs are more basic than your’s. I have two Time Machine backups & one Carbon Copy Cloner. One of the Time Machines is connected all the time and the other one as well as the CCC backup are in a safe in the garage. I use those every couple of weeks. It sounds like I should look into an online backup as well. Useful article, thanks again.

6 hours ago, AudioDoctor said:

 

Every single one of those disks, solid state and spinners will absolutely fail. Before they fail completely, they will be silently corrupting your data and that bad data will be backed up. Once the good data is gone, it is unrecoverable.

Well, I certainly understand that risk, and genuinely defer to your expertise, what I guess I don’t understand is how other backup solutions are any different?

The bottom line seems to be redundancy of data to protect against drive failure. Whether you are backing up in the cloud, to a NAS or any other solution, ultimately, isn’t it all just being stored on drives that can fail?  
 

Providing that you have another copy of your data on a drive that hasn’t failed, you would seem to be as protected as possible (I am asking not telling)
 

Isn’t the point to be redundant, whether it is local drives, cloud based storage, or network storage?

I guess that is what I’m not understanding.  In the end, aren’t they all ultimately drive based storage and thus all subject to failure?

8 minutes ago, blaven said:

Well, I certainly understand that risk, and genuinely defer to your expertise, what I guess I don’t understand is how other backup solutions are any different?

 

The bottom line seems to be redundancy of data to protect against drive failure. Whether you are backing up in the cloud, to a NAS or any other solution, ultimately, isn’t it all just being stored on drives that can fail?  
 

Providing that you have another copy of your data on a drive that hasn’t failed, you would seem to be as protected as possible (I am asking not telling)
 

Isn’t the point to be redundant, whether it is local drives, cloud based storage, or network storage?

 

I guess that is what I’m not understanding.  In the end, aren’t they all ultimately drive based storage and thus all subject to failure?

 

 

By backing up to the cloud you're likely backing up to a ZFS file system. I use TrueNAS here in my home which also uses ZFS. ZFS is the ONLY file system that can detect, fix, and alert you to, bad data and errors allowing you to replace the bad disk, it rebuilds the array with no data loss. At the Moment I have ~40TBs of ZFS storage space available across 15 4TB disks, any three of which can fail simultaneously before I lose data. IMO, the only way to be absolutely sure is to have at least two backups on ZFS, at least one of which is offsite.

Also, SSDs can lose data when left powered off for long enough lengths of time. 6 months or a year, age of the drive also effects this.

5 hours ago, daverich4 said:

Thanks, I think my needs are more basic than your’s. I have two Time Machine backups & one Carbon Copy Cloner. One of the Time Machines is connected all the time and the other one as well as the CCC backup are in a safe in the garage. I use those every couple of weeks. It sounds like I should look into an online backup as well. Useful article, thanks again.

I lost a lot of photos to a Time Machine/iCloud backup scheme. Time Machine and iCloud will happily backup bad data and erase teh good data to make more space for the new, bad data.

At a minimum, Backblaze has unlimited personal storage for, I think $10 USD per month, and a Mac App that handles the backups for you. You can use versioning on Backblaze (maybe, I use B2 so it may be different on the personal account) and if you wanted to buy the Forklift App, know beyond any doubt your data is good on Backblaze.

12 minutes ago, AudioDoctor said:

 

By backing up to the cloud you're likely backing up to a ZFS file system. I use TrueNAS here in my home which also uses ZFS. ZFS is the ONLY file system that can detect, fix, and alert you to, bad data and errors allowing you to replace the bad disk, it rebuilds the array with no data loss. At the Moment I have ~40TBs of ZFS storage space available across 15 4TB disks, any three of which can fail simultaneously before I lose data. IMO, the only way to be absolutely sure is to have at least two backups on ZFS, at least one of which is offsite.

 

Also, SSDs can lose data when left powered off for long enough lengths of time. 6 months or a year, age of the drive also effects this.

Or if you're like me and decide you don't want to ever lose data again. You run Fedora with ZFS on Root across a 2 nvme RAIDZ Mirror, and have your home folder on a 3 SSD RAIDZ1. One of my NVMEs can completely fail and my computer will boot, letting me know a drive has failed. A new NVME to replace the bad one, a few terminal commands, and I am back up and running again with fully protected data. It is the same with my 3 SSD RAIDZ1 set up. Any one of the disks can fail and I will not lose data.

When ZFS reads a file, whatever that file may be, lets say its a music file. It reads the checksum for that file and checks it against the parity. If they differ ZFS automatically fixes the file, and then loads it for you. All that happens without you even noticing, except that ZFS logs the error and it will tell you, hey, I encountered this error on this disk at this time. I fixed it but I wanted to let you know. Ok, it's not worded exactly like that but all that information is available.

Here is a ZFS Scrub, a check for errors, on my RAIDZ1, aka, my home folder on my desktop.

jason@fedora:~$ zpool status
  pool: Tank
 state: ONLINE
  scan: scrub repaired 0B in 00:05:45 with 0 errors on Sat Aug 24 23:10:08 2024
config:

	NAME                        STATE     READ WRITE CKSUM
	Tank                        ONLINE       0     0     0
	  raidz1-0                  ONLINE       0     0     0
	    wwn-0x500a0751e6d7c60a  ONLINE       0     0     0
	    wwn-0x500a0751e6d5f0ed  ONLINE       0     0     0
	    wwn-0x500a0751e6d9dd98  ONLINE       0     0     0

errors: No known data errors