I had a process in a "uninterruptible sleep" state. Trying to kill it is, unsurprisingly, unhelpful. All the literature on the subject will say that it cannot be killed, and they're right. It's called "uninterruptible" for a reason. An uninterruptable process is in a system call that cannot be interrupted by a signal (such as a SIGKILL, SIGTERM etc).

These typically arise when the kernel needs to do something that could take "a while", and that certainly was the case in this particular situation (a user filled a disk partition and NFS lost its head, but that's for another post). Uniterruptible sleeps are actually very useful; they're needed when a process reads/writes to disk for example.


root@storage2:~# ps afux | grep unlink
root 24587 0.0 0.0 8036 852 pts/15 S+ 14:57 0:00 \_ grep unlink
root 18328 0.0 0.0 11092 1356 ? D Jan25 0:00 /bin/bash ./unlinked
root@storage2:~# kill 18328
root@storage2:~# ps afux | grep unlink
root 24589 0.0 0.0 8036 848 pts/15 S+ 14:58 0:00 \_ grep unlink
root 18328 0.0 0.0 11092 1356 ? D Jan25 0:00 /bin/bash ./unlinked


As expected. "This is an uninterrible sleep and dammit, I'm going to sleep", says the process.

But I didn't expect that it would survive a shutdown.


root@storage1:~# ssh stg2 shutdown -r now
^[[A^[[A^[[A
root@storage1:~# ssh stg2
Last login: Wed Jan 27 14:51:43 2016 from edward-m.hpc.unimelb.edu.au
root@storage2:~# ps afux | grep unlink
root 24615 0.0 0.0 8036 848 pts/15 S+ 15:03 0:00 \_ grep unlink
root 18328 0.0 0.0 11092 1356 ? D Jan25 0:00 /bin/bash ./unlinked


"That's bananas!" was my first reaction. Doesn't a shutdown clear memory and reset all process IDs?

Indeed, it most certainly does.


root@storage2:/var/lib/nfs# ps afux | grep shutdown
root 24653 0.0 0.0 8036 856 pts/21 S+ 15:15 0:00 | \_ grep shutdown
root 23083 0.0 0.0 5964 732 ? D 11:00 0:00 shutdown -r 0 w


It looked like it has shutdown, but it hadn't!

The shutdown process starts with all logged-in users receiving notification that the system going down, and login operations are blocked. It also notifies processes that the system will be going down with a SIGTERM, allowing programs to save edited files etc.

Being a moron in a hurry, I initially exited, waited, and logged in again, pleased that the system had come up again.

Of course it had never gone down in the first place. The system was such a state that the shutdown command itself had gone into an uninterruptible sleep as well. So yes, a process can survive a shutdown command because the command can fail to complete!

A lesson learned: to shutdown an system that has uniterruptible sleep processes, be aware that a number of other commands could also fall into such a state. In such cases, a non-operating system shutdown process (IPMI) should be used instead. But try reboot -f - that initiates a shutdown without contacting the init system.