failsafe booting..

[Andrew]

hi,

I was wondering if someone could give me some advice regarding grub and
or bootloaders. 

What i'm trying to achieve is failsafe bootloading of a kernel when a
kernel upgrade to a remote server has occurred. I have to be sure that
the machine will reboot successfully. Because I will not have "on site"
access to any of the machines I wish to have this facility on.

One approach would be to manipulate which kernel is booted at boot time,
using the default kernel flag.

At boot time the bootloader would change the default kernel to the
failsafe one and boot the new kernel. Thus if the new kernel hangs, the
machine will after a certain amount of time reboot ( with the help of a
watchdog card designed for this purppose).  After the watchdog card
reboots the machine, the failsafe kernel will be used to boot
successfully and error logs etc can be examined. If the new upgraded
kernel boots successfully then a script can be run to change the default
kernal back to the new kernel so that on subsequent reboots the upgraded
kernel is used.

So basically I wanted to know if something like this has been done
before in grub or other bootloaders, if not is it possible/feasible ? Or
is it a completely stupid way of doing things ? Are there other ways to
achieve what I want ?

Any feedback would be much appreciated ....

thanks 
Andrew