TideLog Archive for the “Data Recovery” Category

When a hard disk is manufactured, there are areas on the platter that have bad sectors. Considering that on a 2 TB hard disk there are 4 billion sectors, then a few bad sectors is only a tiny proportion of the total number of sectors on the drive. During the test phases of a hard disk, the platters are scanned at the factory and the bad sectors are mapped out – these are generally called ‘Primary Defects’. The primary defects are stored in tables in the firmware zone, or in some cases the ROM of a hard disk. When you buy a brand new hard disk, you will most likely be completely unaware of these bad sectors and the numbers because they are ‘mapped out’ using ‘translator‘ algorithms.

Modern hard disks use Logical Block Addressing or LBA, this describes the sector numbering system on the hard disk, and goes in sequence
0,1,2,3,4,5,…..n-1,n (where n is the last sector on the drive.

Spare sector pools

All modern hard disk drives have a spare sector pool. This is used when bad sectors develop during the normal life of the hard disk and any newly found bad sectors are ‘replaced’ with good ones from the spare sector pool. This process is invisible to the user and they will probably never know that anything has changed.

How Bad Sector Mapping Works:

There are at least two methods of bad sector re-mapping (or translation) these are P-List and G-List.

  • P-list are defects found during manufacture and are also know as Primary Defects
  • G-List are defects that develop in normal use of the drive and are known as Grown Defects

There are other defect lists found in modern drives but the principles are similar. For example, you may find a T-List or a Track defect list, or an S-List or System area defect list.

P-List Remapping
Lets get into how these defect lists actually work, so let’s say we have a small hard disk with 100 sectors and a 10 sector spares pool.

When bad sectors are found at the factory, shift-points are entered into the P-List, if we take the following LBA sequence 0,1,2,3,4,5,6,7,8,9,10 …99, 100 Lets say that Sectors 3, 6 and 9 are found to be bad. When the first bad sector is found, the first part of the re-mapping process will look like this

0,1,2,B,4,5,6,7,8,9,10 ..

What happens here is the bad sector at position 3 is recorded in the P-List. The new map now looks like this;

0,1,2,P,3,4,5,6,7,8,9,10 ..  You can see now that 3 is where 4 was.

The next bad sector at LBA 6 is now found

0,1,2,P,3,4,5,B,7 and is again mapped out giving 0,1,2,P,3,4,5,P,6,7

When the whole sequence is complete, our final map looks like this.


Because these sectors are mapped out, the user will never be aware that they exist. If you want to look at sector 6, the drive will translate that to physical sector 8. It takes the 6 and adds the shift points to it, +1 for the bad sector at LBA3 and +1 for the bad sector at LBA 6. When the testing gets to the end of the drive, in order that it is of the correct size of 100 sectors, it allocates the sectors from the spare sector pool completely concealing the fact that there are bad sectors on the media. To all intents and purposes the drive looks just like the original as 1,2,3,4,5,6,7,8,9,10. However, our spare pool has reduced in size and there are now 7 sectors remaining in the spares pool.

After using the drive for a while some bad sectors develop the drive takes care of these using a grown defect list.

G-List Remapping
The grown defect list or G-List is a table containing the location of bad sector defects found during normal operation of the hard disk drive. When a bad sector occurs during normal use of the drive, something a similar process to P-List generation occurs – resulting with the bad sectors being mapped out. The process for G-List mapping out is slightly different. Lets say our hard disk develops a bad sector at the current LBA 6. What happens in this case is first the bad sector is mapped out. Giving; 0,1,2,3,4,5,G,7,8,9,10 .. A sector from the spare pool is allocated in the bad sectors place. We used 3 of these sectors in factory testing, so the next available bad sector is 104 this now becomes mapped to LBA 6 so our sequence would look like this; 0,1,2,3,4,5,104,7,8,9,10

Again, this process is completely invisible to the user and will still look like the original sequence of 0,1,2,3,4,5,6,7,8,9,10

You might ask, ‘why don’t the new defects get added to the P-List?‘ the answer is that if you add a grown defect to the P-List it has the effect of shifting the data up the drive for each sector from the point where the new bad sector is found. If you look again at the methodology behind the P-List it will help you understand this.

Where a G-List entry can help to revive hard disk, if there was data stored in the original sector attempts then usually it is lost. This may appear to the user as a file that not longer opens, or a a program that doesn’t run anymore or some other errant behaviour. This will not become apparent until the next time the file is attempted to be opened. It may also be that it is such a long time since it was opened that a backup plan means there are no backups of the working version. So bear this in mind when developing you backup plan.

Defect Mapping in a live system
When a hard disk is powered up, the p-list and g-list are usually loaded into RAM on the controller card. As requests for data come through, the location where the data is required from is passed to the translator, which makes the calculations necessary so as to determine which sectors to actually read in order to get to the actual data requested. In our example above, if we wanted the data from LBA 6 the translator would first run through the p-list and add 2 sectors to the count for the two bad sectors found at the factory, it then checks this value in the G-list and finds it has been re-allocated to sector 104. It then reads sector 104 and presents you with the data.

All the magic that goes unnoticed by normal people 🙂

Comments No Comments »

Bad sectors are little clusters of data on your hard disk that cannot be read. More than that, though, they have the potential to cause real damage to your hard drive (catastrophic failure) if they build up over time, stressing your hard drive’s arm, which contains the read/write head, there are two for each platter, one for each side. Bad sectors are fairly common with normal computer use and the imperfections of the world we live in. Like chip fabrication and LCD panel manufacturing, HDD manufacture is a very critical, precise process, and like a TFT with bad pixels from the factory, you do get bad sectors with a HDD due to imperfections when it’s made. The manufacturers make legal allowances for a certain limit to these imperfections before warranty claims can be made, like the legal limit of 5 dead pixels on a TFT. However, there are several simple steps you can take to prevent HDD bad sectors and to repair any that you do have. Having bad sectors will slow down computer performance as well, as your drive takes time attempting to read them. Here is a step-by-step guide. The most common questions I get as a computer engineer are “What is a sector?”, and “How are HDD bad sectors created?”

A sector is simply a unit of information stored on your hard disk. Rather than being a mass of fluid information, your hard disk stores things neatly into “sectors”, a bit like us humans putting things into boxes, and the box only holds so much, and all boxes are the same size. The standard sector size is 512 bytes.

There are various problems that can cause HDD bad sectors:

  • Improper shutdown of Windows, especially power loss while the HDD is writing data;
  • Defects of the hard disk, including general surface wear, pollution of the air inside the unit due to a dirty or clogged air filter, or the head touching the surface of the disk;
  • Other poor quality or aging hardware, including dodgy data cables, an overheated hard drive, and even a power supply problem, if your drive’s power is erratic;
  • Malware.

Hard and soft bad sectors

There are two types of bad sectors – hard and soft.

Hard bad sectors are the ones that are physically damaged (that can happen because of a head crash if your drive is dropped while running and writing data), or in a fixed magnetic state. If your computer is bumped while the hard disk is writing data, is exposed to extreme heat, or simply has a faulty mechanical part that is allowing the head to contact the disk surface, a “hard bad sector” might be created. Hard bad sectors cannot be repaired, but they can be prevented. The heads of a hard drive float on the air cushion generated by the platters spinning, they fly less than the width of a human hair away from the platters, even a small speck of dust is like a mountain, so knocks are definitely to be avoided.

Soft bad sectors occur when an error correction code (ECC) found in the sector does not match the content of the sector. Whenever a file is written to a sector, the drive calculates a “checksum”, which is used to verify the data, if it doesn’t match upon read, the drive knows the sector is weak. A soft bad sector is sometimes explained as the “hard drive formatting wearing out”, in other words the magnetic field is weakening, like an old video cassette – they are logical errors, not physical damage ones. These are repairable by overwriting everything on the disk with zeros. Like tapes and CD’s, the magnetic surface on a hard disk is not infinite, it is affected by other magnetic fields around it, so data recovery guys like me recommend regularly imaging a drive directly to another, frequently, to keep the data fresh and readable.

Preventing bad sectors

You can help prevent bad sectors (always better than trying to repair them, as they say prevention is better than cure!) by paying attention to both the hardware and the software on your computer.

Preventing bad sectors caused by hardware:

  • Make sure your computer is kept cool and dust free;
  • Make sure you buy good quality hardware from respected brands. Cheap RAM and power supplies are my biggest culprits from experience;
  • Always move your computer carefully, and make sure it is TURNED OFF, not in Sleep mode, it can wake up while being moved, especially a laptop;
  • Keep your data cables as short as possible;
  • Always shut down your computer correctly – use an uninterrupted power supply if your house is prone to blackouts.

Preventing bad sectors using software

  • Use a quality disk defragmenter program with automated scheduling to help prevent head crashes (head crashes can create hard bad sectors). Disk defragmentation reduces hard drive wear and tear, thus prolonging its lifetime and preventing bad sectors;
  • Run a quality anti-virus and anti-malware software and keep the programs updated.

Monitoring bad sectors

If you use a tool like HD Sentinel, or CrystalDiskInfo, and you notice bad sectors on your drive, keep an eye on it. A few sectors bad is not normally a problem, as I mentioned at the start of the article, up to 5 bad pixels on a new TFT is allowed before it becomes a warranty claim, hard drives are allowed a few bad sectors due to the imperfections of their manufacturing process. They are manufactured with what are known as “reserved sectors”, a spare area of the disk only accessible by the controller board. If a sector is weak, the controller will attempt to move the data to the reserved area, if this is successful it then attempts a quick read/write test on the old sector (takes less than a few milliseconds), if it fails it marks it as bad in the sector map, also stored in the drive reserved area, along with drive firmware, so that it doesn’t attempt to use it again.

If the number of bad sectors starts increasing, or you start to experience other symptoms, such as the drive dropping out completely as if you unplugged it, or any clicking, and data taking longer to read or copy, this could indicate a fault with the read/write heads, or the control circuitry. Stop using it immediately and back up any important data to another drive. If the failing drive is under warranty, print a log off from HD Sentinel and take it along with you to return the drive, as evidence.

S.M.A.R.T Values to look for

When looking at S.M.A.R.T (Smart Monitoring And Reporting Tool) analysis, the two main areas to look out for are:

Reallocated Sector Count

This shows how many of the drive’s Reserved sectors have been used. If too many of these are used it generally indicates a problem with the disk surface.

Current Pending Sector

This shows how many bad sectors are currently pending a rewrite. A hard drive will always try to rewrite the sector, if it fails, the sector is reallocated into the reserved, the drive adds the sector on to the Reallocated Sector Count, and the original sector is then marked as unusable. If the rewrite is successful, the Pending Sector count will drop.


Comments No Comments »

HDD Regenerator is the backbone of our data recovery services at Kitamura, coupled with Runtime Software’s GetDataBack series, they are paid for software but pay for themselves if you’re a data recovery specialist.

One big warning I need to make people aware of about using HDD Regenerator 1.71 and lower with an Advanced Format drive that has 4k sectors is that they ARE NOT COMPATIBLE. Versions 1.71 and below DO NOT support 4k sectors on Advanced Format drives! You need to use HDD Regenerator 2011 and NEWER.

If you run these older versions either in DOS or Windows on an advanced drive formatted with 4k sectors, once it sees a bad sector it will regenerate it as a 512k sector, and every single sector after this will be seen as bad, making the drive look totally damaged. IMMEDIATELY STOP THE PROCESS, and purchase HDD Regenerator 2011, and re-run it either as a CD or USB. The newer version will correct the sectors to the proper size.

Don’t torrent it either. Dimitriy Primochenko and his team have done such a great job with this program since it started over 10 years ago, and any IT recovery professional worth his salt, like me, will reward a great team with purchases of their software. The price pays for itself, I use it in Kitamura and personally at home, so reward greatness with kindness and tip him an extra £10 on top of the purchase price.

Comments No Comments »