Can you damage or harm a hard drive by formatting or partitioning it?

[Campbell Ritchie]

Bought some cheap external HDDs from somebody on eBay, and these are obviously home‑made things with a drive stuck inside a caddy and a label over the join. Said label says not to format or partition the drive, but leave it as it is with exFAT. It says you can damage the drive by formatting or partitioning it. That seems a bit far‑fetched to me, so I searched and found a few hits:- 1 2 3, all of which suggest you can't damage a disc by partitioning or formatting it. Is it all right to let the heavies and hitmen gParted at it? Since I hardly ever use Windows®, it would be a lot easier to have the disc formatted with ext4 or similar.

Maybe when they said damage the disc, they meant delete all its data?

[Knute Snortum]

Personally, if I bought a cheap HDD from someone I don't know, and it said not to format the drive, I would be worried that it contains some kind of malware.

[Campbell Ritchie]

Hahahahaha! All the more reason to format it then!

[Norm Radder]

I have a vague memory of a HDD I bought over 10 years ago that had proprietary features to allow it to be used. No useful details on it.

Have you tried mounting it to take a peek at its contents?

[Martin Vashko]

In the distant past, physically formatting a drive meant writing marks that would identify sectors on the magnetic medium. So it was, for example, possible to format a 3.5" floppy, which was normally 1.44 MB, to 1.72 MB or so, simply by squeezing sectors on a track closer together. Performing this operation on a HDD incorrectly probably might cause permanent damage to it.

I think modern HDDs typically don't even allow such an operation to be performed without specialized software. When performing a standard (that is, not quick) format, the OS just rewrites the contents of all sectors on the disc to zero or something similar.

What partitioning does is that it simply writes new information into the master boot record. Should he MBR be damaged and fail, the drive is toast (modern HDDs might remap a bad MBR to somewhere else, but that's just a guess). Similarly, a quick format just writes new empty control structures on the disc - a few sectors at most. Can't see how this could hurt a healthy drive.

In any case, if your drive shouldn't survive partitioning and formatting, I'd say would be dangerous to trust it with any data at all. I'd actually suggest running some kind of health check on the drive, if you plan to store on it any data you care about. In Windows, that would be chkdsk /R, on Linux I think (actually, Google says) there is the badblocks command.

[Campbell Ritchie]

Norm Radder wrote:. . . Have you tried mounting it . . .?

Yes, but no success yet.
And good to see you again, Martin

[Tim Holloway]

Knute Snortum wrote:Personally, if I bought a cheap HDD from someone I don't know, and it said not to format the drive, I would be worried that it contains some kind of malware.

So would I.

However, as Martin has pointed out, things were different in the past. Take floppies, in particular. There were actually 2 types of floppies. One was hard-sectored, one was soft-sectored. A hard-sectored drive had multiple index holes on the disk, marking the start of each sector. Hard-sectored drives weren't common, however.

More often, floppies were soft-sectored. As Martin mentioned, instead of physical sector marks, the drive used the single index hole as a baseline and all the other sectors on the track were indicated by metadata markers visible to the hardware.

What few people knew was that this was actually the IBM CKD disk formatting technique. CKD stands for Count, Key, Data and refers to the IBM architecture where a disk sector's metadata contained a count (the sector length), a key (so that you could rapidly search files based on a key value), and data, which was the primary payload. It was, in fact, the exact same architecture used on their mainframe DASD devices. Direct Access Storage Device, which is IBM-ese for disk, the way IPL is IBM-ese for boot. Well, technically, IBM did make some direct-access CKD devices using other technologies like drums and datacells, but for most people, DASD meant disk.

You didn't format a DASD device. Instead, when you brought it into service, you initialized it. That process consisted of writing a set of 80-byte data-only ANSI-format label records to track zero. Most importantly, the VOL1 label, which, in EBCDIC, contained the 6-character volume ID. The ANSI labels were all text, incidentally, no binary data.

You also had the Volume Table of Contents, or VTOC. This was the "directory" of the disk and mapped the 44-character filenames to specific cylinder/track number extents on the disk. A file might spread over multiple extents. The directory was flat. No subdirectories.

Allocating an extent didn't format the extent itself, just marked its location. When you wrote a file to the disk, you'd write data or write key and data on the fly. Parts of the extent beyond the last physical record were undefined garbage. CKD not only supported hardware search by key, but also variable length records. And that's all I need to say, if not more. If you want further details, most of that documentation is online.

So CKD didn't work with fixed-length sectors. Unless it wanted to. And the sector length and existence was solely defined by the programmer. CKD did have its advantages, though. The gap between one sector and the next was "wasted" disk capacity back when a 5MB (!) disk was a respectable size. So breaking a disk track into lots of small sectors was undesirable usually.

When PCs came along, they ignored the key-related parts of the disk controller logic and drives because the original PCs were pretty limited. They also lifted their filesystem pretty much straight from Digital Equipment Corporation systems, which is where we got the A:, B:, C: stuff and the 8.3 filenames. Using fixed-length sectors was simply easier. However, you could choose your sector length when you formatted the disk, as long as you had your BIOS set up to work with that length. On CP/M systems, you actually built your own customized BIOS - it wasn't in ROM like on the IBM PCs.

SCSI disks introduced the concept of hard-sectored drives. With SCSI, you didn't address records in terms of cylinder, track, record, you used a simple block offset. This practice got carried into the IDE architecture and IBM-compatible BIOS configuration got a lot easier as a result. So did the location logic in PC operating systems. It used to be that you'd set up the sectors as part of installing a new disk (hard formatting). These days, hard formatting is done at the factory and you only do the volume initialization (soft formatting). Soft formatting comes in 2 flavours: short, which just initializes basic volume info and full, which physically erases (and thereby tests) each physical sector. Full formatting can also be done on a per-partition/filesystem basis for only that partition.


OK so much for the history lesson. What this all means in practical terms is that it's possible that funny things were done with Campbell's disk to make it look bigger than it is. Although that's just one of many possibilities. Some that I can thing of are:

A) Possible malware hidden on it.
B) Making the disk look bigger
C) It actually is a bigger disk, but there's bad zones, so it was formatted to look smaller, avoiding the bad zones. Or the physical access mechanism cannot reliably move to some parts of the physical disk.
D) ???

I wouldn't be surprised to see that it's option C. It's common practice to sell not-up-to-spec devices as smaller/slower units. Even the manufacturers themselves do that. Although when a no-name supplier does it, there's a high probability that they didn't fully test the downscaled unit before fobbing it off.

On a Linux computer, the best bet would be to run the badblks utility using the writeback option. This will totally destroy the data on the disk, but will exhaustively test and record whether they read and write correctly, including under stress conditions. A few bad blocks can be mapped by filesystem. If there are large bad zones, you can isolate them into unused partitions and use the rest of the disk. I have a few of those.

Of course, any drive with massive bad spot infection is more likely to fail catastrophically, so I'd keep it for non-critical applications.

[Campbell Ritchie]

Thank you for all the detail Tim, and sorry for not replying earlier; am a bit under the weather. I think it is most likely they have got a job lot of discs cheap because they have potential bad sectors, so I can't send it back or anything. I think it will be

One of those things you put down to experi‑ence.

It opened with Windows 7, and Kaspersky didn't say anything about malware. Can't open it on Linux [edit]didn't therefore run badblks[/edit], so maybe I shall give it to somebody who uses Windows® and tell them to screen it for malware and not use it for anything critical.


I have definite evidence the disc is OK, from its ID number:-










0123456789ABCDEF

[Tim Holloway]

Sorry you're not at your best. The badblks utility actually doesn't like to work with accessible (mounted) disks. In fact, a favorite way of analysing disks is to boot Linux off a thumb drive (meaning even a machine whose native OS is Windows can do it), then run forensics.

Badblks can scan partitions (/dev/sdb1) or the entire drive (/dev/sdb), and in your case, a full drive scan is indicated.

One quirk I've seen on my own system lately is that if the OS auto-mounts the disk and I manually unmount it, the OS device (/dev/sdb) completely vanishes. Apparently something in the hot-swap device config does this. So I have to be careful about how I moun/unmount.

Since these are questionable drives, it might also be worthwhile to scan the OS logs when plugging them in to see if anything unusual is reported by the low-level services.

[paul nisset]

As Martin suggested ,the key to a hard disk is the Master Boot Record. That's what you need to mount it and get the bios to recognize it.

You should have no issues formatting it . I believe EX FAT formatting means it can be seen by non Microsoft operating systems as well as Windows.
I've done this with old hard drives from computers they are chucking out at work and mounted them in a caddy with a cable connector and used them as an external drive.

I use one as a back up for a Mac that has a small HDD. With the amount of crap (log files, bad/old registry keys etc) that gets stored on hard drives ,you are asking for  headache trying to "clean" it. Formatting it and putting on a new OS is a lot quicker . In your case , it would also provide some piece of mind in terms of knowing  what is for sure not on the disk  .

Also ,you will know whether issues the disk(s)are software or hardware. you have nothing to lose by formatting then throwing Linux on it and seeing how it performs.
The exception this is if you want Windows and want to keep using the licence that is already on the disk. Then formatting  means you would have to have a valid licence handy before reinstalling windows.


Sometimes ,there are bad sectors on the disks .It just means that part of the disk is marked as dead and can't be written to. When you do defrag on a disk in Windows ,there is a graphic that actually shows the bad sectors.

Good luck.

[Caleb Cruze]

Formatting does not damage a disk drive. I would like to point out that there are different kinds of formatting. The most common way to format a hard disk is to do a file system format. There is also a low-level format and a secure format. These are the 3 most common methods of formatting a disk and none of them “damage” a drive, but it is a normal use of a drive so it will contribute to normal wear and tear.

Doing a file system format is usually something that happens when you are installing an operating system or adding a new hard disk to an existing OS. This lays out the file system structure by which all files saved will be accounted for. Quick formats of this typewrite out the file allocation tables, and/or journals depending on the file system, but typically only does a few hundred megabytes at best. This is trivial for a modern drive and only takes a minute or three to finish. As files are written to the drive, new entries are added to the table. The sectors not touched after a quick format must be dynamically formatted when a file is stored in that location. This will add overhead to the write, but again on a modern drive, this is trivial. A full format will take longer, but every sector is aligned and enumerated at the time of formatting. New write will not have dynamic formatting overhead, which means it’s slightly faster. Again, this is usually a very small write benefit of about 1–2% on our fast modern drives and even less on solid-state drives.

A low-level format is done with a special tool or controller that will write sequentially block by block some pattern of data across the disk drive. This is usually zero’s if it’s not a secure format. These types of formats take a long time, usually multiple hours. A secure format is similar to a low-level format, however, the data written is typically pseudo-random and may take multiple passes to format the entire disk and doubles or triples the time it takes to format.

Let me know if this was helpful.

Regards,
Caleb

[Campbell Ritchie]

CC: thank you. I had forgotten asking this question in the first place. And welcome to the Ranch

[Tim Holloway]

Caleb Cruze wrote:
A low-level format is done with a special tool or controller that will write sequentially block by block some pattern of data across the disk drive. This is usually zero’s if it’s not a secure format. These types of formats take a long time, usually multiple hours. A secure format is similar to a low-level format, however, the data written is typically pseudo-random and may take multiple passes to format the entire disk and doubles or triples the time it takes to format.

This isn't precisely accurate. Disk tracks are subdivided into physical records (sectors/blocks). The detection of sector boundaries can be done in one of two methods: hard sectoring and soft sectoring. Way back in the early days when floppy disks still flopped, both schemes were used.  You'd have a hole in the disk envelope near the hub that a light would shine through. A corresponding hole(s) in the spinning media would gate light pulses through to a photodetector. Hard-sectored disks would have multiple holes in the media. Soft-sectored disks had only one hole. Over time. soft-sectoring became the norm and hard-sectoring disappeared.

Soft-sectored disks and their control circuitry were shaped by the IBM I/O addressing model, as I've mentioned before, which was known as MBBCCHHR (module/bin/cylinder/head/record). And incidentally, the module/bin parts were rarely used. I never dealt with them myself.

A soft sector didn't have a senseable hole for each sector, so it relied instead on meta-data seen only by the controller and not passed as data. IBM supported a scheme known as CKD (count, key, data) which defined the length of the sector and the length of an optional key data value. Sectors could thus be scanned by the controller hardware looking for the key and could be variable in length. This was the norm on IBM's mainframe Direct Access Storage Devices (DASD) and its virtue was that it offloaded a lot of I/O intelligence onto the storage controllers. IBM mainframes were not faster than other computers - in fact the last IBM machine I worked with had 600MHz processors in it. But mainframes were optimised for I/O and for reliability.

While floppy controller chips could use the key-searching capabilities of such a scheme, I never saw any system that did so. And in fact, I can't recall any floppy that used anything other than fixed-sized records (sectors). So, in short, formatting a floppy meant writing tracks with one or more sequences of CKD, data, and CRC (Cyclic Redundancy Check), with a timing gap between them. Incidentally, you didn't format a disk with variable-length sectors, just wrote a label record at the front. Formatting could not be done in advance, since the format varied on each track written. You formatted a track as part of writing data to it. With the so-called Fixed-Byte Architecture (FBA), you'd pre-allocate the track sector space with standard CKD, blank data, and CRC for the entire disk when you brought it into service. That way, later I/O only needed to update the data and CRC parts of the record and you could easily calculate how long to wait between the time the sensor hole was detected and the sector of interest came under the heads.

As floppies became common, most were fixed-length sectors, although some attempted to maximise capacity by doing various tricks to take advantage of the larger surface on the outer tracks. Macintosh and Amiga drives in particular slowed down the rotation rate and/or packed more or fewer sectors on a track based on its cylinder number (and hence radius).

This is where the BIOS was critical, since not all necessary information was accessible directly from the disk and disk drive. In CP/M, you had to custom-compile a BIOS with the geometry information. When hard disks came available, where PC-DOS was supposed to work with IBM drives, the information was simply hard-coded into the BIOS and it became an immutable file and as third parties started providing drives of differing capacities and sector sizes, a whole table full of information ended up in the BIOS until, thankfully, disk drives got smart enough to volunteer the information themselves.

In the mean time, SCSI was built, predicated on the idea that the controller was smart enough to find sectors without any CPU help, so the SCSI architecture used linear sector addressing and the actual cylinder/head values were not visible to the OS or to apps in normal operation. And, of course, SATA built itself off the concepts of SCSI.

So these days at the OS/driver/application level, low-level disk formatting isn't even visible, much less user-controllable. The original CKD/data/CRC scheme (or some variant) is probably still in use internally to the disk drives, but disks are reliable enough now that they are formatted at the factory and will likely never need formatting again. And if they did, it would likely require specialised knowledge, software tools, and perhaps even hardware.