What Happens When an Embedded System Crashes?

Embedded systems run silently inside your car, your microwave, your medical devices. Unlike a PC, there's no user to click "restart" no IT helpdesk. When one crashes, the consequences range from a frozen screen to something far worse.

So what actually happens inside the chip when it all goes wrong?

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

What Counts as a Crash?

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

A crash isn't always a dramatic explosion. In embedded systems, it's any state where the device stops doing its job correctly. The four most common types:

🔁 Infinite Loop / Hang : The CPU is running but stuck, usually waiting for a sensor or peripheral that stopped responding.

💥 Hard Fault: The CPU tried to access invalid memory or execute an illegal instruction. The hardware catches it and panics.

📦 Stack Overflow: The call stack grew too large and started corrupting nearby memory silently, before anything obviously breaks.

🧨 Watchdog Reset: A hardware timer noticed the software hadn't checked in, assumed it was hung, and forced a reboot.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

The Moment of a Hard Fault

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

On ARM Cortex-M chips (the most popular embedded processors), a Hard Fault triggers a precise sequence in microseconds:

1. The illegal operation is detected and aborted instantly.

2. The CPU automatically saves 8 key registers to the stack including exactly where the code was and what it was doing.

3. Execution jumps to your HardFault_Handler function.

4. Fault status registers tell you why it happened bad memory access, bus error, or misaligned read.

That saved context is gold. A good fault handler reads it, writes it to flash memory, and then resets the device cleanly. A bad fault handler is just an infinite loop and you'll never know what went wrong.

━━━━━━━━━━━━━━━━━━━━━━━━━━━

The Watchdog: The System's Heartbeat Check

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Every production embedded system should have a hardware watchdog timer. It counts down on its own, independent of the CPU. Your software must "kick" it (reset the count) regularly to say "I'm alive." Miss the deadline — automatic reset.

It's a dead man's switch built into silicon.

Common mistake: kicking the watchdog inside an interrupt handler. If your main loop hangs but interrupts still run, the watchdog gets kicked anyway and hides the problem. Always kick it from your main execution path.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Stack Overflow: The Quiet Killer

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Stack overflows are uniquely nasty because they don't crash immediately. The stack silently overwrites adjacent memory global variables, other task stacks, critical data. The actual crash happens later, somewhere completely unrelated, making it extremely hard to debug.

On a typical microcontroller with 64–256KB of total RAM, a single 4KB local array in a function can overflow the stack entirely. The fix? Use static or globally allocated buffers for anything large, and never recurse without strict depth limits.

The Memory Protection Unit (MPU), available on most modern ARM chips, can catch overflow the instant it happens turning silent corruption into an informative fault.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

HEADING: How Systems Recover

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Recovery is where good and bad firmware really separates:

✅ Basic: Watchdog fires, system reboots cleanly from a known state.

✅ Better: Fault handler logs the crash data to flash before rebooting. Next boot, you have a full record of what happened, even in the field with no debugger attached.

✅ Best: After 3+ consecutive watchdog resets, the system enters Safe Mode: a minimal, protected firmware image that can receive an OTA update and recover itself. Used in automotive, medical, and industrial devices.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

When It Goes Wrong in the Real World

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

These aren't hypotheticals:

🚀 Ariane 5 (1996): A velocity value overflowed a 16-bit integer. The unhandled exception shut down both the primary and backup navigation systems. The rocket destroyed itself 37 seconds into its maiden flight. Cost: $370 million.

☢️ Therac-25 (1985–1987): A race condition in a radiation therapy machine caused it to deliver lethal overdoses to patients. There were no hardware safety interlocks — software was the only protection, and it failed silently.

🚗 Toyota (2009–2011): Stack overflow issues and over 11,000 global variables in the engine control unit contributed to unintended acceleration events. Result: deaths, recalls, and a $1.2 billion settlement.

The pattern is always the same: no fault logging, no meaningful recovery, and assumptions that were never tested against reality.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

The Bottom Line

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Every embedded system will eventually face a condition it wasn't designed for. The question isn't if it crashes it's what happens when it does.

Enable the watchdog before you write main(). Write the fault handler before you write the application. Design the fail-safe state before you design normal operation.

The craft of embedded engineering isn't just making it work. It's making it fail well.