Q: How to implement watchdog and self-healing mechanisms for always-on IoT devices?
Answer
Multi-level watchdog strategy: (1) Hardware watchdog: STM32 independent watchdog (IWDG) – resets MCU if software hangs. Configure timeout < worst-case loop time. (2) Software watchdog: external process monitor (systemd watchdog for Linux) that kills and restarts unresponsive services. (3) Network watchdog: monitor MQTT broker connection - reconnect with exponential backoff (initial 1s max 60s). (4) Application watchdog: send heartbeat to cloud every 60s - if no heartbeat for 5 minutes trigger remote diagnostics. (5) Boot watchdog: U-Boot or SPL watchdog resets if kernel does not load within timeout. (6) Peripheral watchdog: monitor critical sensors - if data stops updating for >5 min alert and optionally reset sensor via power cycle GPIO. Implement state machine for boot -> running -> degraded -> offline transitions. Log all restarts to persistent storage for diagnostics.
Filed under: FAQ