Hello.
Most of the symptoms you describe and the "sudden" falling of more
systems does point to a rootkit that was installed on the first
compromised machine (FC2). That machine might have been later used to
gain access to the other servers in your network.
The segfaults when running usual commands (mostly grep, netstat, ps and
so on) while some other software runs just fine makes the rootkit
explaination quite certain. Also the failure to restart the server
usually is a consequence of that. One way to make that sure is to get
the hdd from the possibly compromised machine, put it on an offline
system which has rkhunter (or other rootkit-detection software)
installed and check it. After the signs you described, it quite very
probably you'll find a rootkit.
RH's before RHEL are ok (from the stability point of view) as long as
you keep the exposed services uptodate (recompilation from source).
Don't use the old software they come with, cause you might just open a
door to your system.
Kernel error messages may also be a sign of intrusion (local root
exploit maybe, that breaks something).
About firewall and passwords, that should have been the first step,
before making any server accessible from the Internet.
Also, be careful because once any server of yours got compromised, this
means the attackers may already have the passwords for most of the users
on that system.
About the last 7.3 you spoke about, it's posible (if the attackers
haven't already got the machine) to see some intrusion trace in the
system logs (or ssh and other services).
Good luck.
Cristian Stanca wrote:
Hello,
We've got a hard disk failure (bad blocks - reported the array controller
bios) on a scsi hard-disk on an INTEL platform (running Fedora Core 2 Linux
operating system). What is interesting is that this hard-disk failure
occurred after a "I don't know what it is... let's reboot it and see after
that" situation. Situation describe by many "segmentation fault" when using
typical application like vi or service or even grub-install. Grub did not
start again after that (we tried to reinstall it with an Install CD 1 from
Fedora and grub-install did said "segmentation fault" again)
We did recover the data on that scsi hard-drive by mounting it on another
machine.
So far so good (sort of)
After a week or so, another Linux server, began to show the same errors
while giving shell commands and also sshd listened on port 22 we cannot do a
ssh on it. We did not make the connection to the previous case (as we
thought was a possible hardware failure), reboot it and grub did not start.
We boot again with an install CD from redhat 7.3 (as we had redhat 7.3
installed on that hard-disk, and thought if any files are missing...), the
hard-disk was recognized by controller (again scsi hard-disk), fdisk view
the partitions, and cannot this time mount them. (As I write this the "much
more important data that hardware" hard-disk is at a computer service, for
data recovery.
Again, on a third Linux server (redhat 7.3) we got some messages at the
primary console (kernel BUG commit.c #some number, lots of stack text and
hexa symbols...) and again can't do ssh on it (it responds to ping and
traceroute, telnet ip_address port 22 works...). We are kind of worried
regarding the reboot of this machine...
Could that be a worm, exploit or something, or looks like a human
intervention situation?!
In the mean time, we are working at a firewall and password policies.
--
Valentin AVRAM
IT Security Engineer
GeCAD NET
Phone: +40-21-321.78.03
E-mail: vavram@gecadnet.ro
Web: www.gecadnet.ro
