Full /home hoses a box

[Michael Hipp]

Michael Hipp wrote:
> One of my servers is hosed, I'm offsite and can't get to it until 
> tomorrow. Nmap shows ssh and samba to be active as normal but all 
> attempts to connect to the box are rejected.
> 
> Based on when it apparently went south I'm guessing that a runaway job 
> has filled a disk (it was getting low on free GB anyway). But the 
> overflow would have been only on /home which is its own disk.
> 
> I thought a full /home would be survivable. No?
> 
> What can I do to prevent this in the future? I'm loathe to use quotas 
> because some of the accounts need large amounts of space at various times.
> 
> Any possible creative ideas to get the box to respond?


Got here a bit earlier than expected ... and my guess about a full disk 
was WRONG. Here's what was on the console:

   Filesystem "hda2": Corruption of in-memory data detected. Shutting
   down filesystem: hda2
   Please unmount the filesystem, and rectify the problem.

Oh joy, hda2 is the root filesystem.

The box would give me a login prompt but no password prompt. I hit reset 
and it seemed to boot clean. All logs stopped abruptly at 06:25:27 this 
morning. Interestingly enough, cron.daily is set to run at 06:25. But it 
evidently crash immediately because I was using it remotely as late as 
09:00 this morning.

I note that Ubuntu released a kernel update last night. It installed 
flawlessly on several other machines. I wonder if that could be what 
hosed it here - a failed kernel install.

Flash: it has something to do with the kernel upgrade. 'Cause it just 
did it again. More later.

Michael