Too many external HDD's! How do you keep track?

[papadage]

As a side comment, thanks to this thread I ordered the additional high-capacity drives to do a full backup of my NAS, and the last backups were done last night. All three are labeled and in their padded carriers, and a log in Excel that I update on my desktop and a copy on each drive.

Now to take them to my mom's house for safekeeping and to set up a schedule to bring them back quarterly for a refresh.

Thanks for the push to get this done!

 

[Lord Evermore]

I might actually have to pay for it myself, sadge

A hammer is pretty cheap and reasonably secure.

 

[evan_s]

A hammer is pretty cheap and reasonably secure.

If the drive still works a single pass of anything, 0's 1's random, should make sure an undelete utility can't recover anything. If the drive doesn't work well enough for that just open up the drive so you can see the platters and/or remove the PCB. Either is likely to make the data unreadable for anyone but a professional data recovery service or nation state.

 

[Lord Evermore]

If the drive still works a single pass of anything, 0's 1's random, should make sure an undelete utility can't recover anything. If the drive doesn't work well enough for that just open up the drive so you can see the platters and/or remove the PCB. Either is likely to make the data unreadable for anyone but a professional data recovery service or nation state.

For personal data a wipe is 99.999999999% fine. For a business, destruction is a good idea and may be mandatory, and might even require certification. For a long time my company just destroyed them with a hammer after we got a pile of them, and clients were fine with that, but eventually we got with a PC recycler that certified destruction for us. A hammer to break or bend the platters (some are just glass), even a single blow, is faster than taking a drive apart or spending time hooking the drive up and doing a wipe/format.

 

[singebob]

A hammer is pretty cheap and reasonably secure.

I cannot be bothered to take the cover off each HDD and take a chisel etc to the platters is the simple answer

I would rather I spent that time posting

 

[Lord Evermore]

I cannot be bothered to take the cover off each HDD and take a chisel etc to the platters is the simple answer

I would rather I spent that time posting

Why would you take the cover off? Just hit it hard enough to bend the cover which will bend the platters. (Often cracking the cast metal casing.) Literally 3 seconds of labor. Just wear protective gear, at least googles/glasses and a towel over your face and a glove on the hammer hand. I had a couple of shards fly up once that left a scar on my thumb and cut my index finger deep enough to affect the tendon. But, that's just the kind of thing that makes computing exciting.

 

[singebob]

But, that's just the kind of thing that makes computing exciting.

not for me

 

[Lord Evermore]

not for me

If blood hasn't been shed, the job wasn't done right.

 

[tiredoldtech]

If blood hasn't been shed, the job wasn't done right.

You mean like my former boss in the 2000's who would use a hammer on drives made live with an AT power supply to destroy the data... to discover that IBM had moved to glass-metal substrate platters around that time with an explosive shower of glass bits everywhere when he hit the drives with a hammer (and no eye or other protection)? After that, he resorted back to drilling drives and using big magnets.

 

[Lord Evermore]

You mean like my former boss in the 2000's who would use a hammer on drives made live with an AT power supply to destroy the data

Was he trying to make sure the heads would crash and scrape across the platters? Extra paranoid to think somebody might recover a beat up platter and still read the data on it. (Of course, back then, the bits could be seen with the naked eye.)

 

[doraemon]

Typical 3.5" form factor drives are going to be aluminum media, so just drill a hole through the cover and through all of them. Fast and effective.

 

[Lord Evermore]

Typical 3.5" form factor drives are going to be aluminum media, so just drill a hole through the cover and through all of them. Fast and effective.

Technically those could still be mounted and read, accessing at least the undrilled parts and allowing reconstruction, if someone REALLY wanted to. Deforming them is more likely to damage the magnetic layer and warp it and make it nearly impossible to recover enough data to make any sense. The point of multiple writes with DoD-level erasure is to ensure as many magnetic particles as possible get flipped at least once, because a single write doesn't ever change every particle of rust to the new orientation; a normal write will only flip the vast majority of them, which is enough for a read to say "this bit is supposed to be a 1" or "this bit is supposed to be a 0" because it's over a threshold of strength in that direction. If nothing at all is done to a section of the platter, it could be mounted in a custom device with a read head applied to it to detect the bits very slowly. Even multiple overwrites isn't fully secure and scanning electron microscopes have been able to recover data.

And not everybody has a drill, but most people could improvise a hammer even if they don't have that, and they're cheap. Bashing a drive with a hammer is also more fun. Of course, a single blow might not damage metal platters to the point they're unrecoverable, either, so a few blows might be needed.

You could pour some naval jelly into the drilled holes and shake it around and let it eat through the iron oxide on the platters.

 

[continuum]

/me waits for @Lord Evermore to realize who he's mansplaining to...

 

[doraemon]

Guess it really comes down to how paranoid you want to be about someone recovering the data. I've never had to think very much about how to efficiently damage a drive beyond repair/recovery, but it does make for a fun thought experiment. With an air drive, you could peel away certain seals (or poke holes in them) and soak the drives in water. It'd rust inside and you wouldn't have to do anything.

 

[Lord Evermore]

/me waits for @Lord Evermore to realize who he's mansplaining to...

Some dude on the Internet? They may know all those details, but not everybody reading will.

 

[Lord Evermore]

Guess it really comes down to how paranoid you want to be about someone recovering the data. I've never had to think very much about how to efficiently damage a drive beyond repair/recovery, but it does make for a fun thought experiment. With an air drive, you could peel away certain seals (or poke holes in them) and soak the drives in water. It'd rust inside and you wouldn't have to do anything.

I thought of that, but the magnetic media is already rust. Would water actually change anything? And isn't it just one very small hole, just enough for air pressure to move in and out? Might take a while to fill.

 

[Lord Evermore]

Guess it really comes down to how paranoid you want to be about someone recovering the data.

It's kind of like quicksand. When I was younger, "they" made it seem like the bad guys would be much more interested in recovering my data from old hard drives than it turned out anybody actually is. Now that paranoia is legally required in many situations, and a requirement for insurance coverage in most cases outside of personal use.

 

[singebob]

Guess it really comes down to how paranoid you want to be about someone recovering the data.

It also depends on e.g. whether you've always worked in a heavily regulated sector.

If blood hasn't been shed, the job wasn't done right.

You sir, have some weird ideas about IT

 

[doraemon]

I thought of that, but the magnetic media is already rust. Would water actually change anything? And isn't it just one very small hole, just enough for air pressure to move in and out? Might take a while to fill.

If you know where to puncture the hole(s), it wouldn't be that hard to fill with water. I can say that from firsthand experience. If you get enough in there, it'll corrode quickly enough.

Media isn't the only thing inside the drive that can corrode and cause problems. It's why there's usually desiccant inside, to help control humidity within a particular range.

I still think drilling a big hole would be faster.

 

[Mechjaz]

I asked an unrelated question the other day and got some really good answers, so I was going to try again.

I have a LOT of HDD's. Some are external, some are internals in a enclosure, and some are just bare internals that I slot into a USB drivebay. Backups, backup of backups, TV shows, 80's cartoons, books, taxes... other stuff. My issue is that I have no idea what I have or where it is. I was looking for an empty HDD the other day, and ran across an old mini external HDD that had a show on it I had been looking for for years.

So I'd like to know if there is some way to organize all this stuff. The closest idea I've been able to come up with was sticky notes on the HDD and something like exporting the DIR to a text file. That seems stupid. I can't be the only one with this problem. Someone has to have already solved it.

So... is there something out there like this? Something that can keep track of everything on these HDDs, which files I have backed up more than twice, how much free space is left of the drive, etc... I'd say I want to get organized, but I already know that is beyond my capabilities.

I'm considering boxing up all my old disc videogames and using the bookshelf to physically keep my HDD's. I will probably also have to get a label maker at some point. It's just that as I move things around, consolidate, delete backups of backups of backups, contents change, so labeling a HDD "Old shows" doesn't help when I need to move that to another HDD because it isn't big enough, or I wind up with Old Shows #1, Old Shows #2, Old Shows #3, and then I'm back to having to look through them for the one I want again.

So if anyone has an idea or could point me in a direction... other than Marie Kondo because deleting things sparks terror for me, not joy.

Pending a NAS to finally get my life on track...

Four is personal stuff - documents etc.

Eight is shows, because somehow my shows are smaller than my movies.

Fourteen is a backup of Four, movies, extras, and general scratch drive - things I've been too lazy or haven't gotten around to playing nicely with Plex.

Twenty is the main drive that is the daily drive(r) for all the content that is backed up on Eight (shows) and Fourteen (movies).

I just got Twenty, so now things are fully backed up again across Eight and Fourteen (my TNG rips from the spring were living on the edge for a while).

They're all WD easystores, so all the AC adapters work with any of the drives, which is handy. One day I'll shuck them, stuff them in a NAS, and make a storage pool, and the three-drive monte will finally be put to rest as I only have to pull out a (presumably new, more capacious) external HDD to back up the NAS.

 

[Fruity Pebbles]

A hammer is pretty cheap and reasonably secure.

For glass platter drives I slam them on concrete or give them a tap with a sledgehammer. For aluminum platter drives I drill a 1/2" hole or two through the drive.

 

[tiredoldtech]

...Or, if you're in a certain category of person without going into detail: homemade Thermite on a few drives at a time sat in the center of an in-ground fire pit so the prying neighbors don't see what exactly you're doing other than "fire and smoke".

 

[Lord Evermore]

...Or, if you're in a certain category of person without going into detail: homemade Thermite on a few drives at a time sat in the center of an in-ground fire pit so the prying neighbors don't see what exactly you're doing other than "fire and smoke".

I have wanted to do that for so long, since I saw a video decades ago on how easy it is to make and watched an entire PC be destroyed with it.

 

[tiredoldtech]

I have wanted to do that for so long, since I saw a video decades ago on how easy it is to make and watched an entire PC be destroyed with it.

Are we referring to a certain notorious episode that aired only once before it was pulled on G4/TechTV prior to 2000? Grin.

 

[Mental Gear Reduction]

I hoard redundant drives purely for the reason that every time the office had a clearout of disks for secure disposal I kept forgetting to sneak my disks into the pile. And I think I've finally missed the boat altogether since we're out of all our datacenters and have no spinny rust hardware left - everything that isn't still on tape in archive is somewhere in the cloud. There isn't even a NAS at the office anymore.

I might actually have to pay for it myself, sadge

Also, for the OP - anything wrong with a labelmaker?

A screwdriver and a hammer are pretty cheap.

edit: oops, well-covered ground, already. Sorry!

 

[Ananke]

Probably around thirty or so. Other than some Western Digital drives from a long time ago, I really haven't had many failures. However, EVERY WD drive I ever bought had failed. They have a good RMA process, but I got tired of using it so often.

Now a problem I have, though I mentioned it earlier, is sometimes I'll take a smaller drive and copy it to a larger drive to consolidate... then forget I've copied the data already or keep the old drive for a back-up of the data and suddenly I now have two drives I have to keep track of. That is why I wanted a way to identify if I have duplicate files (like identical name, size, and created date or something).

I have been wanting to take it easy for a little bit, so I haven't moved forward on this side project after I got the labeler.

I'm late to the party, but two points stand out to me:

First, this quote right here has me professionally breaking out in a cold sweat. Giant waving red flag: far bigger issue than not being able to track which drive is which. To borrow a phrase - it sounds like you're not controlling your data, but in effect, it's controlling you, either due to fear of losing something, or forgetfulness. And that is magnifying a problem which is already more complicated than it need be.

Secondly, you mention bare hard drives. Sata connectors are not designed for regular cycling in the same way that USB is. You'll probably be fine - I suspect you're still more likely to destroy a drive by dropping it on the floor than by wearing out the connector - but this isn't the expected use case for sata connectors, so if any of your drives are at the marginal end of the quality curve, it's a risk.

As for suggestions:

* The easiest problem to attack is the number and spread of drive sizes. The fewer drives you're juggling, the easier it will be for you to track. You mention multiple 64GB flash drives - these are tiny compared even to hard drives sold in, say, 2008 (when 750GB was the cutting edge of enterprise storage disks). You must have space on larger drives to consolidate your data together, and these tiny drives are the lowest hanging fruit.

* If you're paranoid about deleting data, there exist copy/cut programs that will verify that a copied file has been copied fully and equally, and delete the source file if and only if that check completes (e.g. Teracopy). Avoid, at all costs, multiplying your problems by creating new (duplicate) data that you then have to catalogue and carry around with you

* Don't get trapped by analysis paralysis over relatively trivial decisions. Your naming scheme is not carved in granite (unless you were planning to engrave it on the hard drive shell, in which case, uh, don't do that) - picking something non-optimal but functional now is better than being forever put off by the thought that you might find the perfect solution tomorrow. 1, 2, 3 is a functional start, and you can change it later.

* As far as possible, keep your directory structures consistent across disks. Suppose you have a collection of all sorts of audio/visual/image media spread across N disks - it's a lot easier to find what you're looking for if all N disks have a structure like media/film/genre/film.mp4 or media/music/artist/album/track.mp3 than if disk 1 has films/film.mp4, disk 5 has oldwindows/users/him/documents/films/film.mp4 disk 17 has backup/disk2/video/film.mp4.

* Flatten your data structures as much as possible. The more sub- and sub-sub-categories you include, the more likely that you'll end up with files in unexpected or inappropriate places, or that you'll just throw your hands in the air and give up. Especially try to avoid just duplicating old WIndows or Linux operating system structures in their entirety. With consistent, flat, general-> specific, directory structures, you can much more easily identify at a glance what a drive does or does not contain. And you can more easily see if you have similar or identical files in a given category.

* Aim to have your drives no more than 80% full and absolutely no more than 90% full: that both gives headroom for disk management, and provides spare space to include new files in existing categories without requiring starting new disks too often.

* Even if you don't want a NAS, seriously consider buying a single large hard drive to consolidate many older, smaller disks onto. Consider as in "think about it for at least 20s before saying either 'no' or 'yes'". Assuming you have a bunch of older 1, 2, 4TB disks, for a moderate outlay you could replace, say, 6-8 of them with a single 16-20TB disk. It would absolutely cost you money, but that might be worth it to also save yourself frustration and delays in having to sort through your collection to find what you're looking for.


(Managing data produced by a bunch of people who care a great deal about it existing an arbitrary number of years later, but who are not convinced that it's worth putting much effort into organising it here and now, and who may or may not resemble a herd of cats, is a non-trivial part of my job. I won't pretend to know the answers, but I recognise a lot of the problems)

 

[HO]

So I'd like to know if there is some way to organize all this stuff.

I have pared down my drive count considerably, but back when that was not so I used a drive cataloging software called Cathy (scroll down). Not splashy (far from it), but simple, tiny (~150 Kb) and reasonably quick. And way better than post-it notes.

 

[N00balicious]

All I'm saying is don't be hugely surprised if you go to read one of these drives in a few years and some or all or it is gibberish, or the drive won't respond, or whatever. It's not (quite) the same as keeping books on a shelf. That said, if they don't get too hot or suffer mechanical shock, you'll probably do OK.

This is anecdotal, but recently, a friend of mine went for the external HDD, that he'd stored all his digital treasures on. It contained his: music, years of correspondence, financial records, pics, sex tapes, etc.. We wasn't storing it under his bed, amongst the: dust bunnies, used condoms, "dead soldiers", and detritus that accumulates in places like that. It was stored under ideal conditions in a: cool, dark, place-- a fireproof, strongbox.

However, the drive was more than 5-years old. As the OP mention, "out of sight, out of mind" with external HDDs. He hadn't accessed it in several months.

About two weeks ago, he went to retrieve some golden oldies from the daze of his youth, to titillate his new Asian girlfriend. (I think he was looking for an AAD version of Iggy Pop's "China Girl"?) He found the drive was all rotted-out. It was unreadable.

Nobody ever expects Bit-rot to happen to them. He was bereft. He though "HDDs were forever".

I found it interesting that a drive can "go bad" in a period of months? But who knows, he may have unknowingly been losing sectors for years?

I suggested he send it out to a recovery shop, to see what they could salvage.

Still, I think the suggested "NAS Option", with regular device health checking is the best advice. Better yet, put it all up in some Enterprise Cloud and let them do the care and feeding of the storage?

 

[Lord Evermore]

I found it interesting that a drive can "go bad" in a period of months? But who knows, he may have unknowingly been losing sectors for years?

Most likely, it was degrading for much longer. He may have accessed it several months ago, but performance may not have been what it used to be due to the drive needing to re-read sectors, but he might not have noticed. And "unreadable" may only mean that the master file table or partition table was no good, not necessarily that all the data was gone (unless he also tried using recovery software to scan the entire drive), or may have had some other mechanical issue which the OS can't see other than "the drive won't read data".

Running something like Recuva would be a good first attempt to get the data. Recovery services can be expensive, but some of that data sounds like it could be important enough to justify it. Just bear in mind that while the company as a whole may have policies about anything being done with your data, it's still possible that an individual tech may view some of the stuff while spot-checking and see embarrassing stuff (but of course, they don't know a customer from Adam). That would be a good task for AI, which could be allowed to check every single file for quality without a real person seeing your data, and the AI could then forget what it saw. (Of course some illegal items have to be reported if seen by a person, and it might be required to have AI identify that as well.)

 

[Mental Gear Reduction]

Nobody ever expects Bit-rot to happen to them. He was bereft. He though "HDDs were forever".

I like to always have a live spinning version of the data I care about, along with any offline backups. I particularly like ZFS on this front, because of its parity checking and "scrub" capability. I scrub my live disks twice a month.

Sadly, cloud backup is very expensive. It rapidly becomes much cheaper to buy a pair of drives and run them in a ZFS mirror.

 

[N00balicious]

Sadly, cloud backup is very expensive. It rapidly becomes much cheaper to buy a pair of drives and run them in a ZFS mirror.

That is until the house burns down or the river rises.

The folks in Pasadena, CA and Ashville, NC by now understand the 3-2-1 Backup Maxim. That is (I'm paraphraseing) "You need three (3) copies of your data. Two copies, each on different types of storage media, and one copy not with the other two.

Putting that off-site copy up into a cloud, any cloud, in write once, read (hopefully) never storage could save the pics from your first TJ bachelor party, back when you had hair and a 28" waistline from fire and flood?

 

[evan_s]

That is until the house burns down or the river rises.

The folks in Pasadena, CA and Ashville, NC by now understand the 3-2-1 Backup Maxim. That is (I'm paraphraseing) "You need three (3) copies of your data. Two copies, each on different types of storage media, and one copy not with the other two.

Putting that off-site copy up into a cloud, any cloud, in write once, read (hopefully) never storage could save the pics from your first TJ bachelor party, back when you had hair and a 28" waistline from fire and flood?

Yeah. I'm not sure how relevant the 2 different storage types is anymore, especially for home users. I get why for businesses you might want different two different storage media, so that a tape machine dying doesn't take out access to all copies of your backups etc.

For home users I think the realistic option is a local back up copy to something like a NAS or external HD and remote copy, probably on a Cloud service or maybe on a friend/families NAS. That is on top of what ever location actually stores those in the first place. Personally for me that's my local QNAP nas and backing up to my OneDrive space from my family Office 365 sub for my important files (mainly pictures etc backed up automatically to the NAS from our phones). Luckily the important stuff I actually want to back up is easily small enough to fit into the 1tb that provides.

An External hard drive in a fire safe is better than nothing as that third copy if you are adamant about not using cloud storage but certainly not as apparently the only copy of important data. Any type of drive sitting on a shelf or in a fireproof box is a bit of a Schrodinger's drive. You know it's there but you don't know if it's still functional until you try accessing it so it is and isn't a valid backup at the same time.

The other part of that is actually testing the backup periodically. Just because the process is running and not failing doesn't mean it is working correctly. You don't want to discover it hasn't actually been working when you go to try to restore something.

 

[Mental Gear Reduction]

That is until the house burns down or the river rises.

The folks in Pasadena, CA and Ashville, NC by now understand the 3-2-1 Backup Maxim. That is (I'm paraphraseing) "You need three (3) copies of your data. Two copies, each on different types of storage media, and one copy not with the other two.

Putting that off-site copy up into a cloud, any cloud, in write once, read (hopefully) never storage could save the pics from your first TJ bachelor party, back when you had hair and a 28" waistline from fire and flood?

Too expensive for me. What I've done instead is to have an external USB enclosure with a mirrored backup; that's the only thing I need to grab if I have to evacuate. Won't help against instant disasters, or a fire where I can't reach that room, but it covers many types of emergency.

It would be even nicer to take the whole NAS, but that's way too heavy to just grab and go.

The other part of that is actually testing the backup periodically.

Yep, that's important. I've run through it a couple times, and it worked fine. I've since been depending on ZFS to tell me if there are underlying problems.

I probably should test it more often, but it's an encrypted backup, so bringing it online to test it is a real pain in the butt. I have to do a bunch of monkeying around with mountpoints and key inheritance on sub-filesystems. Normally it's just an offline backup; plugged in, but without mounted filesystems, and then it takes a zfs receive every day.

edit: I also have a single internal drive with a second layer of backup for the extra-important stuff. That's only a single drive, but it's an old 4TB Hitachi that seems indestructible.

second edit: no parity on that drive, though. That's just good old ext4.

third edit: another reason I haven't tested the restore much is because ZFS has a semi-backup built right in. I store three months of checkpoints, so when I need to recover something, I can usually just drop into an old version of the filesystem and copy it out. It's a really nice extra layer of defense against fat fingers. But then the mirrored pair is an exact replica, every day, so there's no additional backup layer. If I don't realize I need something within three months, it's gone.

 

[Chito]

Local NAS (RAIDZ2), critical data off of which is backed up to cloud, and also occasionally to another local HDD which stays physically disconnected. That HDD goes with me sometimes when I'm gone for a longer period of time. The intent was to leave it at a distant physical location entirely but I've not done that in practice.

And tested! I restore data occasionally from crash plan just to make sure it's there....