Watchdog – independent monitoring of VM status in the cloud: from /dev/watchdog to the reset test

WebDisk Blog · category: Cloud computing · reading time: ~9 minutes

In brief:- The cloud panel may show "Running" even though the system inside the VM has been unresponsive for the last fifteen minutes – the hypervisor's HA mechanisms see the machine from the outside, not its interior.- A watchdog is a timer that ticks down outside the guest system: if the system stops resetting it regularly, the hypervisor performs a hard reset of the VM – even when the kernel is completely frozen.- The response is a reset, not a repair – which is why in WebDisk Cloud the /dev/watchdog device is available in the machines, but we do not enable the mechanism on your behalf; in this article we show how to do it deliberately. >Not working with a terminal? Skip the command blocks – the description of the mechanism, the side-by-side comparison and the section on the limits of the solution read just fine without them.

It is 3:00 a.m. Infrastructure monitoring is all green, the cloud panel shows the status "Running" next to your machine – and for twenty minutes customers have not been able to open the application. The operating system inside the VM has frozen for good: the kernel has stopped scheduling processes, the console does not react, there is no sign of life over SSH. From the infrastructure's perspective nothing has happened, because the virtual machine process on the physical server is running flawlessly. The failure is invisible in exactly the place where the automated checks are looking.

There is a mechanism for this scenario that is older than the cloud: the watchdog – literally a "guard dog", a timer that has to be reset regularly, because otherwise it bites. In this article we explain how a watchdog running inside the machine differs from high availability (HA) at the hypervisor level, how the /dev/watchdog device works, how to enable and test a watchdog in Ubuntu step by step – and, honestly, when it is better not to enable it. The text will be useful to anyone who maintains virtual machines in the cloud – in WebDisk Cloud and beyond.

Why does the cloud panel show "Running" even though the VM is not working?

Classic high-availability mechanisms in the cloud operate at the infrastructure level. The hypervisor (the software that runs virtual machines on a physical server) and the management layer watch two things: whether the host is alive and whether the VM process is alive. When a physical server fails, the machines come up on another one; when a VM process unexpectedly disappears, it is started again. This is an important safety net, but it looks at the machine exclusively from the outside.

Meanwhile, a large share of real failures happens inside the guest system: a kernel lockup, a frozen init/systemd process, exhausted memory after which the system thrashes between killing processes and swapping, a driver bug. The machine "works" – sometimes it even answers a ping – but it does not deliver the service and you cannot log in to it. For the hypervisor's HA mechanisms such a state is indistinguishable from a healthy one: the VM process exists, so from their point of view there is nothing to rescue.

It is precisely this gap that a watchdog running in the machine itself closes.

How do a watchdog and /dev/watchdog work?

The principle of operation is deliberately primitive – and that is where its strength lies. A watchdog is a countdown timer placed outside the control of the system it supervises. The system has to reset it cyclically; if it stops, the timer runs out and triggers a predetermined response. In physical servers that role is played by a chip on the motherboard; in a virtual machine – by a virtual watchdog device that the hypervisor exposes to the guest. In Linux it is visible as the device file /dev/watchdog. Once it is opened, the timer is armed: from that moment on somebody has to write to it regularly.

In practice this is done by the watchdog daemon: it opens the device and "reports in" every few to a dozen or so seconds. As long as the system is working, the reports keep coming. When the kernel freezes, the daemon does not get processor time, the timer reaches zero, and the hypervisor carries out the configured action – a hard reset of the machine. The crucial point is that the execution takes place outside the guest system: a frozen kernel cannot stop it, because the timer is not ticking in that kernel but one floor below.

The daemon can do more than just report "I am alive". In the configuration file you can add health tests: maximum system load, the presence of a process with a given PID file, the freshness of a file (whether a log is still growing), whether a specified host answers a ping, and finally your own test scripts. If one of the tests fails persistently, the daemon deliberately stops resetting the timer and brings about a reset – even though the kernel is formally working. Thanks to this it also reacts to "soft" hangs, not only to the clinical death of the system.

How does a watchdog differ from hypervisor HA?

• What it observes – HA at the hypervisor level: the physical host and the VM process · Watchdog inside the VM: the interior of the guest system
• What it detects – HA at the hypervisor level: a host failure, the disappearance of the VM process · Watchdog inside the VM: a kernel freeze, an unresponsive system, failed health tests
• What it does not detect – HA at the hypervisor level: a hang inside a running VM process · Watchdog inside the VM: a host failure (that is not its role)
• Response – HA at the hypervisor level: restarting the VM, e.g. on another host · Watchdog inside the VM: a hard reset of the VM
• Who enables and configures it – HA at the hypervisor level: the cloud provider · Watchdog inside the VM: the user, inside their own machine

These two mechanisms do not compete with each other – they look at the same machine from two opposite sides, and only together do they complete the picture of a failure: from the crash of a physical server to a frozen kernel. What matters in a shared environment: the watchdog is configured entirely inside the VM, without changes on the hypervisor side – so a cloud user can raise the resilience of their own system independently, without violating the separation of responsibilities between them and the provider.

How do you enable a watchdog in Ubuntu step by step?

The example uses Ubuntu Server (e.g. 22.04 or 24.04); in other Linux distributions at most the package names differ.

1. Check whether the machine sees the device:

ls -l /dev/watchdog*
# crw------- 1 root root 10, 130 ... /dev/watchdog

If the file is not there, the hypervisor has not exposed a virtual watchdog device to the machine (what to do in such a situation – see the FAQ; in WebDisk Cloud the device is available).

2. Install the daemon:

sudo apt update
sudo apt install watchdog

3. Configure the /etc/watchdog.conf file:

sudo nano /etc/watchdog.conf

A minimal, sensible starting configuration:

# device and reporting rhythm
watchdog-device = /dev/watchdog
interval = 10 # the daemon reports in every 10 s; this has to be
                           # clearly less than the device timeout

# optional health tests – uncomment deliberately:
#max-load-1 = 24 # reset when 1-min. load > 24
#pidfile = /run/mysqld/mysqld.pid # watch a specific process
#file = /var/log/syslog # the given file has to change…
#change = 1800 # …at least every 1800 s

4. Enable the service and observe:

sudo systemctl enable --now watchdog
systemctl status watchdog
journalctl -u watchdog -f # reports and test results live

From this moment on the machine is under supervision. Two housekeeping notes: a planned stop of the service (systemctl stop watchdog) closes the device "cleanly" and disarms the timer, so in the default configuration it does not end in a reset; add health tests one at a time and watch the logs after each change – we come back to why that matters in the honest reckoning below.

Test it before you trust it

Warning. The test below hard-resets the machine – exactly as the watchdog would do during a real failure. Perform it only on a test machine or in an agreed maintenance window; all unsaved data will be lost.

A watchdog that has never been verified is just an entry in a configuration file. The most credible test simulates precisely the scenario the mechanism is supposed to protect against – an immediate kernel freeze:

# forces a kernel crash – the system immediately stops responding
echo c | sudo tee /proc/sysrq-trigger

The machine goes dead: the SSH session drops, the console does not react. The daemon stops reporting to the device, the timer reaches zero and the hypervisor resets the VM. Once the system is back, check how the event unfolded:

uptime # fresh uptime = the reset worked
journalctl -b -1 -e # last entries from the previous boot

If, instead of resetting once the timeout expires, the machine hangs indefinitely – go back to the previous section: most often it turns out that the daemon was not running or was not opening the right device.

When should you not enable a watchdog and what are its limitations?

Let us call things what they are: a watchdog does not diagnose or repair anything. Its only response is a hard reset – the equivalent of pulling the plug out of the socket:

• A reset interrupts everything, including operations in progress. Journaling file systems (ext4, XFS) and databases with crash-recovery mechanisms are designed to survive such a restart, but data that has not been written to disk is lost. A watchdog is in no way a substitute for backups – we wrote about them in the article Cloud backup – the foundation of modern IT security.
• A reset masks the cause. A machine that restarts "by itself" every few days looks healthy in the availability statistics, even though it is sick. A watchdog should go hand in hand with monitoring and log analysis – a reset is a plaster, not a cure.
• You lose the evidence. A frozen machine is also a freeze-frame of the failure; a reset erases it. When you are trying to diagnose a hang that is hard to reproduce, a watchdog can be the enemy of the investigation.
• Health tests can produce false positives. A rule such as max-load-1 on a machine that is computing something heavy exactly as planned can reset the system in the middle of correctly performed work. That is why thresholds are matched to the nature of the workload, and new tests are enabled one at a time, with a period of observation.

When should you not enable a watchdog? When the machine runs long batch jobs without checkpoints; when you are analysing a recurring hang and need a "live" machine for diagnosis; when the service already sits behind a load balancer that pulls unhealthy instances out of rotation on its own. A watchdog pays off most on standalone machines that nobody duplicates: a single application server, a VPN gateway, an administration panel, a mail server.

Watchdog in WebDisk Cloud

Our cloud runs on the Apache CloudStack platform, and we take care of the health of the physical servers and the virtualization layer – failures on the infrastructure side are our responsibility. What remains, however, is exactly the gap we started with: a machine that has frozen internally may look perfectly functional from the outside – and no mechanism on the provider's side should then guess on your behalf whether a hard reset right now is safe.

That is why the division of roles is simple:

• Machines in WebDisk Cloud start with the /dev/watchdog device available – you do not have to request anything or change anything in your service configuration.
• The mechanism is not enabled by default in our system templates. Since the response is a reset, the decision to arm it – and the choice of health tests for your application – belongs to you.
• Enabling it happens entirely inside the machine, exactly as in the instructions above – without interfering with the hypervisor and without the involvement of our team.

If you are moving to us from a VMware environment, where a similar role is played by VM monitoring based on a signal from the guest tools, the concept will feel familiar – and we write about the move itself in the article on migration from VMware to WebDisk Cloud.

Frequently asked questions

How does a watchdog inside the VM differ from hypervisor HA? Hypervisor HA looks at the machine from the outside: it watches the physical host and the VM process, so it does not see a hang inside a running system. A watchdog works the other way round – it supervises the interior of the guest system and, when the reports stop, brings about a hard reset of the machine carried out by the hypervisor. They are complementary mechanisms: only together do they cover failures from the crash of a physical server to a frozen kernel.

Does a watchdog replace monitoring – charts, alerts, notifications? No, it complements it. Monitoring tells you what is happening and why, and wakes a human up; a watchdog merely brings the machine back to life when nobody is looking. A mature setup has both: the watchdog shortens the outage, monitoring lets you remove the cause.

The application has frozen, but the system is working. Will a watchdog detect that? Not on its own – without health tests it reacts only to an unresponsive system. The first line of defence for a single service is restarting the process itself (e.g. Restart=on-failure in systemd). A test with a PID file or your own script can be the second line, but remember: its effect is a reset of the entire machine – it is the bluntest tool available and it should be treated as such.

Can a hard reset damage data? Modern journaling file systems and databases with crash recovery generally take it without harm – at the cost of losing data that did not make it to disk. The risk is real for applications that write without transactional safeguards. The conclusion is always the same: a reset does not excuse you from keeping tested backups.

My machine has no /dev/watchdog. What now? Outside WebDisk Cloud – ask your provider whether it exposes a virtual watchdog device. An alternative is softdog, a kernel module that emulates a watchdog purely in software (sudo modprobe softdog). Honestly: softdog runs in the same kernel that it is supposed to supervise, so in the event of a full kernel lockup it may fail together with it – a device exposed by the hypervisor gives a stronger guarantee.

How do you test whether a watchdog really works? On a test machine or in an agreed maintenance window, force a kernel crash with the command echo c | sudo tee /proc/sysrq-trigger – the system immediately stops responding. If the watchdog is working, the hypervisor will hard-reset the VM once the timeout expires; after the system comes back you will confirm it with a fresh uptime and entries from the previous boot in journalctl -b -1. A machine that hangs indefinitely most often means that the daemon was not running or was not opening the right device.

Does this work only in Linux? The examples in this article are for Linux – it is the best-documented and simplest route. Other systems require a driver for the virtual watchdog device and an equivalent of the daemon; before deployment, check in the documentation whether your system supports such a device.

Summary

A watchdog is one of those "simple, therefore effective" mechanisms: a timer that ticks down outside the guest system turns the worst kind of failure – the silent one, invisible from the outside – into an ordinary, short restart. Three things are worth remembering: hypervisor HA and a watchdog look at the machine from two different sides and only together complete the picture; a watchdog's response is a hard reset, so it is armed deliberately and after a test; in WebDisk Cloud the /dev/watchdog device is waiting in your machine – arming it is your decision and fifteen minutes of work. And if you are only just looking around for an environment for your machines, see what a public cloud at a reasonable price really means.

Not sure whether a watchdog suits your workload? Write to us – the WebDisk support team will help you assess whether this solution fits your scenario.