SSH server and/or client binaries appear to have been modified outside of normal RPM

I received this mail from my Control Panel:

[SIZE=3]Hello, this is an automated e-mail generated by your control panel. I’ve detected a potential problem on this server. You may want to look into it. Details of the problem are below.[/SIZE]
[SIZE=3]SSH server and/or client binaries appear to have been modified outside of normal RPM updates. This is possible evidence of a compromised server. InterWorx has attempted to reinstall the openssh, openssh-server, and openssh-clients RPMs (if this server’s version of yum supports the “reinstall” option). [/SIZE]
Since modifying modifying these files required root access, a more thorough investigation of the server is recommended, along with resetting the root password immediately.

I have resetted the root password immediatley.

In Nodeworx, System Health shows this:


2011-09-14 21:15:33

Cleared Date
2011-09-14 21:20:19

I am not sure what has happened. Is the server hacked?
Is the problem solved or am I facing troubles?

This is actually because of a bug in iworx 4.8.2 that’s causing a false positive with the sshd in centos5.7. iworx 4.8.3 is out now, which fixes this problem.

Thanks for your fast answer Robert!!!

I am on iworx 4.8.2. and centos 5.7.

I have tried yum update, but it doesn’t update to 4.8.3, maybe I have to wait a little while?

I received the same email a few days ago as well and the first thought I had was that it was probably just an update. I am so glad to hear that confirmed.

Yeah, this got me worried for a second, then saw in NodeWorx there was an update and figured it was a bug.

The same day iWorx alerted me to the changed ssh binaries my VPS node was failing because of a hardware problem. It was just a coincidence but until I knew better I was certain there had been a successful break-in. It’s a crappy feeling and the episode nudged me out of a state of complacency. This is one iWorx bug I can appreciate, sort of.

Just FWIW, here’s an overview of how this problem detector actually works -

It runs “rpm -V openssh openssh-server openssh-clients”. That’s a capital V (verify) not lowercase (verbose).

Rpm then checks all the files it installed (rpm keeps track of all files added/created by an rpm install) and looks for changes. It checks modified dates, md5s of the files themselves, and a few other things. If there’s a change to a file in /usr/bin or /usr/sbin, then it’s unexpected, and the problem detection is triggered.

Our problem detectors also have the ability to attempt a fix. In this case, we do a force reinstall of the sshd programs - server, clients, and the base library. This can fail for a number of reasons - one, it only works on CentOS/Redhat 5+, or two, hackers that know what they’re doing break yum installs while they’re at it. This is often done messing with things using chattr.

This detector had a bug that triggered false positives. If yum was currently running an update that was actively upgrading openssh, etc, then the rpm -V was comparing previously calculated md5s from the old version with the md5s of the files on disk, which didn’t work.

Because the detector runs rather often by default, it runs several times during the time it takes to do a full distro upgrade. The release of 5.7 triggered these false positives. We’ve now added code to detect if yum is currently running, and just hold off the checks until it’s done.

So, sorry about crying wolf!


It’s just one of those things and you guys were right on top of it. Any bug or mishap that raises security awareness, like it did for me, is a positive outcome.