We've got a hard disk failure (bad blocks - reported the array controller
bios) on a scsi hard-disk on an INTEL platform (running Fedora Core 2 Linux
operating system). What is interesting is that this hard-disk failure
occurred after a "I don't know what it is... let's reboot it and see after
that" situation. Situation describe by many "segmentation fault" when using
typical application like vi or service or even grub-install. Grub did not
start again after that (we tried to reinstall it with an Install CD 1 from
Fedora and grub-install did said "segmentation fault" again)
We did recover the data on that scsi hard-drive by mounting it on another
machine.
So far so good (sort of)
After reading this part, there are two likely explanations, IMO:
- Bad RAM
- Bad sectors in swap
The best way to check for the first issue, is to use something like
memtest86. I have found a number of bad sticks with it.
After a week or so, another Linux server, began to show the same errors
while giving shell commands and also sshd listened on port 22 we cannot do a
ssh on it. We did not make the connection to the previous case (as we
thought was a possible hardware failure), reboot it and grub did not start.
We boot again with an install CD from redhat 7.3 (as we had redhat 7.3
installed on that hard-disk, and thought if any files are missing...), the
hard-disk was recognized by controller (again scsi hard-disk), fdisk view
the partitions, and cannot this time mount them. (As I write this the "much
more important data that hardware" hard-disk is at a computer service, for
data recovery.
Again, on a third Linux server (redhat 7.3) we got some messages at the
primary console (kernel BUG commit.c #some number, lots of stack text and
hexa symbols...) and again can't do ssh on it (it responds to ping and
traceroute, telnet ip_address port 22 works...). We are kind of worried
regarding the reboot of this machine...
Could that be a worm, exploit or something, or looks like a human
intervention situation?!
After these two, it isn't looking good to me. Are these machines
running all the same hardware? If so, maybe you just got a bad batch of
something.
If not, then that kind of highly-unstable kernel behavior would lead me
to start searching for signs of kernel exploits. There's been quite
a few local kernel holes in the last 6 months, after all.
Of course, that's just pure speculation.
tim
