Rebooting a server that wont reboot

Posted by HostsVault | Posted in How-To's | Posted on 05-09-2008-05-2008

0

Have you ever faced this kind of problem where you couldn’t even reboot your sever because it had a HDD error and you cant access the reboot or shutdown commands and they showed this kind of errors :

[root@hostsvault ~]# reboot
bash: /sbin/reboot: Input/output error
[root@hostsvault ~]# shutdown -r now
bash: /sbin/shutdown: Input/output error

Obviously, there is a problem with your drive. These commands are failing because the kernel is unable to access the /sbin/reboot and /sbin/shutdown binaries from the disk so that it can execute them.

A fsck on the next boot might be able to correct whatever is wrong with the disk, but first you need to get the system to reboot. If your machine is located at a managed hosting provider then you could submit a reboot ticket, but you’ll have to wait for someone to take responsibility.

Wouldn’t it be nice if there was a way to ask the kernel to reboot without needing to access the failing drive? Well, there is a way, and it is remarkably simple.

The “magic SysRq key” provides a way to send commands directly to the kernel through the /proc filesystem. It is enabled via a kernel compile time option, CONFIG_MAGIC_SYSRQ, which seems to be standard on most distributions. First you must activate the magic SysRq option:

echo 1 > /proc/sys/kernel/sysrq

When you are ready to reboot the machine simply run the following:

echo b > /proc/sysrq-trigger

 

This does not attempt to unmount or sync filesystems, so it should only be used when absolutely necessary, but if your drive is already failing then that may not be a concern.
In addition to rebooting the system the sysrq trick can be used to dump memory information to the console, sync all filesystems, remount all filesystems in read-only mode, send SIGTERM or SIGKILL to all processes except init, or power off the machine entirely, among other things.

Also, instead of echoing into /proc/sys/kernel/sysrq each time you can activate the magic SysRq key at system boot time using sysctl, where supported:

echo "kernel.sysrq = 1" >> /etc/sysctl.conf

 

If you would like to learn more about magic SysRq you can read the sysrq.txt file in the kernel documentation.

VN:F [1.9.3_1094]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.3_1094]
Rating: -1 (from 1 vote)
Share and Enjoy:
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Google Bookmarks
  • Furl
  • Slashdot
  • StumbleUpon
  • Technorati

Write a comment