Hardware RAID Cards and Disk Array Monitoring

Understand the RAID card and disk array on your bare-metal server, and learn to catch dropped disks and degraded arrays with storcli / mdadm before it's too late.

On a bare-metal (dedicated) server, multiple disks are usually grouped into a RAID array that trades capacity for reliability and performance. The real danger is a degraded array that nobody notices: lose one more disk and the entire dataset can go with it. Knowing how to read array status, spot a dropped disk, and replace a bad one correctly is a core skill for anyone running physical hardware.

What a Hardware RAID Card Is

A hardware RAID card (such as MegaRAID / LSI, now part of Broadcom) is a dedicated controller that owns all the physical disks (PDs) and presents one or more virtual disks (VDs) to the operating system. What you see with lsblk is typically just the VD, not the health of each underlying drive. That's exactly why a vendor tool is required to see the truth.

Common RAID levels:

  • RAID 1 — two-disk mirror, survives one failure.
  • RAID 5 — distributed parity, survives one failure.
  • RAID 6 — dual parity, survives two simultaneous failures.
  • RAID 10 — mirror + stripe, balancing performance and redundancy.

Inspecting the Array with storcli / MegaCli

storcli (formerly MegaCli) is the command-line tool for LSI-family RAID cards. Start with the overall topology and status:

# Summary of all controllers, virtual disks, and physical disks
storcli /c0 show
# Virtual disk state only (Optl=optimal, Dgrd=degraded, Pdgd=partially degraded)
storcli /c0/vall show
# Each physical disk (State: Onln=online, Offln=offline, Rbld=rebuilding)
storcli /c0/eall/sall show

Focus on two things: whether each VD State is Optl (Optimal), and whether every PD State is Onln (Online). Anything like Dgrd, Offln, Failed, or UBad (Unconfigured Bad) means a disk has dropped or the array is already degraded.

Hot Spares

A hot spare is a spare drive kept idle in the chassis. When a member disk fails, the RAID card automatically pulls in the hot spare and begins rebuilding right away, with no human in the loop. This shrinks the window during which the array runs without redundancy. Configuring a hot spare on any critical array is strongly recommended.

Replacing a Bad Disk and Rebuilding Safely

Once a disk shows Failed, keep these points in mind before you swap it:

  • Locate the physical drive first. Blink its locate LED so you don't pull the wrong disk:
   storcli /c0/e252/s3 start locate
  • Confirm it's degraded, not a multi-disk failure. If losses already exceed the redundancy limit (e.g., two disks in a RAID 5), stop and recover data first rather than acting blindly.
  • Hot-swap in a replacement of the same type and no smaller than the original. In most cases the card rebuilds automatically; if it doesn't, drive it manually:
   storcli /c0/e252/s3 insert dpd disk
   storcli /c0/e252/s3 start rebuild
   storcli /c0/e252/s3 show rebuild   # check rebuild progress as a percentage
  • A rebuild is the most fragile moment, with degraded performance and no fallback if another disk fails. Avoid heavy load until State returns to Optl.

Software RAID: mdadm

If the server uses Linux software RAID (no hardware card), switch to mdadm:

cat /proc/mdstat              # quick view; [UU] all up, [U_] a disk dropped
mdadm --detail /dev/md0       # detailed state: clean / degraded / recovering
mdadm /dev/md0 --add /dev/sdc1   # add the replacement and trigger a rebuild

Alerting So You Catch Degradation in Time

A degraded array usually fails silently, and manual spot-checks aren't reliable. Recommended practices:

  • Hardware RAID: poll VD/PD status on a schedule via storcli, or vendor SNMP / ipmitool, and alert whenever anything is not Optl / Onln.
  • Software RAID: set up mdadm --monitor for email notifications, or scrape nodeexporter's nodemd metrics into your monitoring stack.
  • Treat degraded, rebuilding, and disk dropped as high-priority alerts, pushed immediately to a channel you actually watch.

Summary

RAID gives you redundancy, not a backup, and certainly not a guarantee against data loss. What actually saves you is the ability to read array status with storcli / mdadm, keep a hot spare ready, locate before you rebuild, and alert on degradation so you can act before the second disk fails. The fault-tolerance window that redundancy buys you only matters if something is watching it.