cPanel : SSHD Rootkit

We have been seeing this SSHD rootkit from time to time with much improved versions 🙂

One could see number of SSH processes on the server with nothing under process details. There is a huge discussion thread at WHT www.webhostingtalk.com/showthread.php?t=1235797

The current one which I have seen today was with keyutils-libs legit version name. However the “Signature” was missing.

root@linuxbabu [/var/log]# ls -la /lib64 | grep libkeyutils
lrwxrwxrwx 1 root root 18 Jun 22 2012 libkeyutils.so.1 -> libkeyutils.so.1.3*
-rwxr-xr-x 1 root root 35320 Jun 22 2012 libkeyutils.so.1.3*

root@linuxbabu [/var/log]# rpm -qi keyutils-libs
Name : keyutils-libs Relocations: (not relocatable)
Version : 1.4 Vendor: CentOS
Release : 4.el6 Build Date: Fri 22 Jun 2012 01:20:38 AM CDT
Install Date: Tue 14 Jan 2014 04:56:45 AM CST Build Host: c6b10.bsys.dev.centos.org
Group : System Environment/Base Source RPM: keyutils-1.4-4.el6.src.rpm
Size : 59336 License: GPLv2+ and LGPLv2+
Signature : (none)
URL : http://people.redhat.com/~dhowells/keyutils/
Summary : Key utilities library
Description :
This package provides a wrapper library for the key management facility system
calls.

I matched the size from a clean server and found different in the file size of libkeyutils.so.1.3.

So I reinstalled keyutils-libs replacing the file using yum.

yum reinstall keyutils-libs -y

After reinstall I could see the file size changes and also the Signature shown was correct :

root@linuxbabu [~]# ls -la /lib64/libkeyutils*
lrwxrwxrwx 1 root root 18 Feb 25 01:03 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.3*
-rwxr-xr-x 1 root root 10192 Jun 22 2012 /lib64/libkeyutils.so.1.3*
root@linuxbabu [~]# rpm -qi keyutils-libs
Name : keyutils-libs Relocations: (not relocatable)
Version : 1.4 Vendor: CentOS
Release : 4.el6 Build Date: Fri 22 Jun 2012 01:20:38 AM CDT
Install Date: Tue 25 Feb 2014 01:03:47 AM CST Build Host: c6b10.bsys.dev.centos.org
Group : System Environment/Base Source RPM: keyutils-1.4-4.el6.src.rpm
Size : 36624 License: GPLv2+ and LGPLv2+
Signature : RSA/SHA1, Sun 24 Jun 2012 05:18:51 PM CDT, Key ID 0946fca2c105b9de
Packager : CentOS BuildSystem
URL : http://people.redhat.com/~dhowells/keyutils/
Summary : Key utilities library
Description :
This package provides a wrapper library for the key management facility system
calls.
root@linuxbabu [~]#

Hope this helps to disable the rootkit and avoid further damage to the server.

Another MySQL daemon already running with the same unix socket.

After upgrading MySQL binaries mysqld will not start at all any more and shows below error :

CT-1977-bash-4.1# service mysqld start
Another MySQL daemon already running with the same unix socket.
Starting mysqld: [FAILED]
CT-1977-bash-4.1#

MySQL service does not shut down gracefully during the OS reboot, leaving the old /var/lib/mysql/mysql.sock such that mysqld will not start up. Some people were able to reproduce this error on a CentOS 6.5 KVM guest virtual system by rebooting the host CentOS 6.5 system. CentOS is supposed to gracefully shut down the guest systems, but this seems to be failing in the case of mysqld.

Confirmed Red Hat Linux 6.5 bug – https://bugzilla.redhat.com/show_bug.cgi?id=1037650

Issue discussion on MySQL bug tracker – http://bugs.mysql.com/bug.php?id=71086

Simple steps to reproduce this issue:

service mysqld start
killall -9 mysqld_safe mysqld
service mysqld start

A quick way to restart MySQL is as below :
Remove socket file and restart mysql:
CT-1977-bash-4.1# ls -la /var/lib/mysql/mysql.sock
srwxrwxrwx 1 mysql mysql 0 Jan 8 20:13 /var/lib/mysql/mysql.sock
CT-1977-bash-4.1# rm /var/lib/mysql/mysql.sock
CT-1977-bash-4.1# /etc/init.d/mysqld restart
Stopping mysqld: [ OK ]
Starting mysqld: [ OK ]
CT-1977-bash-4.1#

A work around for the issue is to modify ‘/etc/init.d/mysqld’ script:

Make a backup copy of the startup script.
cp -p /etc/init.d/mysqld /etc/init.d/mysqld.orig

Edit the file /etc/init.d/mysqld to look as below :
=========================
# if fuser "$socketfile" &>/dev/null ; then
# echo "Socket file $socketfile exists. Is another MySQL daemon already running with the same unix socket?"
# action $"Starting $prog: " /bin/false
# return 1

# We check if there is already a process using the socket file,
# since otherwise this init script could report false positive
# result and mysqld_safe would remove the socket file, which
# actually uses a different daemon.
if fuser "$socketfile" &>/dev/null ; then
echo "Socket file $socketfile exists. Is another MySQL daemon already running with the same unix socket?"
action $"Starting $prog: " /bin/false
return 1
fi
=========================

Now restart MySQL :
CT-1977-bash-4.1# service mysqld start
Starting mysqld: [ OK ]
CT-1977-bash-4.1#

Virtuozzo : RTNETLINK answers: Operation not supported

Here is a new bug which I faced on virtuozzo 4 and 4.6 for Centos 6 VPS.

VPS fails to add IP during start or networking fails when network is restarted :

-bash-4.1# /etc/init.d/network restart
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: RTNETLINK answers: Operation not supported
Failed to bring up lo.
[FAILED]
Bringing up interface venet0: RTNETLINK answers: Operation not supported
Failed to bring up venet0.
[FAILED]
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
-bash-4.1#

ifconfig shows blank as networking fails to start :

-bash-4.1# ifconfig
-bash-4.1# rpm -q iproute
iproute-2.6.32-31.el6.x86_64
-bash-4.1# cat /etc/redhat-release
CentOS release 6.5 (Final)
-bash-4.1#

Here is the fix, download the rpm’s as per your VPS arch.

for 64-bit systems: http://mirror.centos.org/centos/6.4/os/x86_64/Packages/iproute-2.6.32-23.el6.x86_64.rpm
for 32-bit systems: http://mirror.centos.org/centos/6.4/os/i386/Packages/iproute-2.6.32-23.el6.i686.rpm

Networking is not available in VPS so download it on the Hardware Node :

cd /root/dino
wget http://mirror.centos.org/centos/6.4/os/x86_64/Packages/iproute-2.6.32-23.el6.x86_64.rpm

Copy it to the affected VPS :
cp iproute-2.6.32-23.el6.x86_64.rpm /vz/private/VEID/fs/root/root/
replace VEID with the affected VPS ID.

vzctl enter VEID

-bash-4.1# cd /root/
-bash-4.1# rpm -q iproute
iproute-2.6.32-31.el6.x86_64
-bash-4.1# rpm -e iproute --nodeps
-bash-4.1# rpm -Uvh iproute-2.6.32-23.el6.x86_64.rpm
Preparing... ########################################### [100%]
1:iproute ########################################### [100%]
-bash-4.1# /etc/init.d/network restart
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface venet0: Determining if ip address 127.0.0.1 is already in use for device venet0...
SIOCADDRT: Network is unreachable
SIOCADDRT: Network is unreachable
[ OK ]
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
-bash-4.1#

Now you can see the IP’s responding. Please make sure you check the RPM and OS versions:

Parallels KB : http://kb.parallels.com/en/118992

UPDATE :

to avoid updates due to yum append iproute* to exclude line in file /etc/yum.conf

cPanel : /root/ Inode issue!

Today I had an issue on / with shortage of disk inodes on a cPanel dedicated server.

In computing, an inode is a data structure on a traditional Unix-style file system such as UFS. An inode stores basic information about a regular file, directory, or other file system object.

In layman’s term – Every time a file is created or uploaded on a server, an inode is created. Simply say, inode is the count of the number of files on your server may it be a VPS or dedicated server.

The number of inodes used can be checked using below command (example is for /root) :

for i in /root ; do echo $i; find $i -type f | wc -l ;done

For my current issue it was the comet directory which was using up the space / inodes. The comet directory is usually filled on a high mail traffic server.

/root/.cpanel/comet

You can safely clean up the comet directory by running the following command that will remove all files in this directory that have not been accessed in more than three days.

/usr/local/cpanel/bin/purge_dead_comet_files

Example :

root@linuxbabu [~]# df -i /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda6 960992 960992 0 100% /
root@linuxbabu [~]#

root@linuxbabu [~]# /usr/local/cpanel/bin/purge_dead_comet_files
******Cleaning up comet for root...Done
root@linuxbabu [~]#

root@linuxbabu [~]# df -i /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda6 960992 31278 929714 4% /
root@linuxbabu [~]#

This should free your Inodes / Disk space.
🙂

wp-cron.php – High CPU usage

What is wp-cron.php ?

This file is a PHP script which runs all the automated tasks that let WordPress do all it’s wonderful tricks. Some examples include:

 

  • Posting content when it is scheduled to be posted at specific times
  • Check all pending comments for spam (if you have plugins like Akismet running)
  • Send emails (i.e. if you have the option enabled where you get emailed whenever a comment is posted, this script handles the email)

Basically wp-cron.php is the automatic part of WordPress.

WP-Cron.php is a very common cause of high CPU loads. Disabling WP-Cron can drastically reduce CPU-load and prevent the chances your account is suspended due to exceeding your resources.

You can disable WP-Cron by editing your wp-config.php and adding the following line;

define('DISABLE_WP_CRON', 'true');

Create a cron job and run wp-cron.php every hour or two using the following command:

wget -O /dev/null http://www.example.com/wp-cron.php?doing_wp_cron

OR (if wget is disabled) :

cd /home/cpanel_user/public_html; php -q wp-cron.php

🙂

cPanel :: cPHulk error – Error while connecting to MySQL

WHM shows below error for cPHulk Brute Force Protection :

=========
cPHulk Brute Force Protection
Mysql is currently disabled. To enable mysql go to: Service Manager
Once there check Enable and Monitor for mysql. Then at the bottom of the page click save.

=========

Running below command should fix the issue :

/usr/local/cpanel/bin/hulkdsetup

Sample Output :
===========
root@1 [/]# /usr/local/cpanel/bin/hulkdsetup
hulkdsetup: synchronizing database schema

## mysqldiff 0.43
##
## Run on Sat Apr 6 04:05:04 2013
## Options: debug=0, host=localhost
##
## — db: cphulkd (host=localhost)
## +++ file: /usr/local/cpanel/etc/cphulkd_db.sql

CREATE TABLE auths (
SERVER char(128) NOT NULL,
USER char(128) NOT NULL,
PASS char(128) NOT NULL,
PRIMARY KEY (SERVER,USER)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE blacklist (
IP char(128) NOT NULL,
ISPREFIX int(1) DEFAULT ‘0’,
UNIQUE KEY IP (IP),
KEY ISPREFIX_index (ISPREFIX),
KEY IP_index (IP)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE brutes (
IP char(255) NOT NULL DEFAULT ”,
NOTES text,
BRUTETIME datetime DEFAULT NULL,
EXPTIME datetime DEFAULT NULL,
PRIMARY KEY (IP),
KEY EXPTIME_index (EXPTIME)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE good_logins (
USER char(128) NOT NULL,
IP char(255) DEFAULT NULL,
LOGINSERVICE char(64) DEFAULT NULL,
LOGINTIME datetime DEFAULT NULL,
KEY LOGINTIME_LOGINSERVICE_USER_index (LOGINTIME,LOGINSERVICE,USER)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE logins (
USER char(128) NOT NULL,
IP char(255) DEFAULT NULL,
SERVICE char(64) DEFAULT NULL,
STATUS int(1) DEFAULT NULL,
LOGINTIME datetime DEFAULT NULL,
KEY LOGINTIME_SERVICE_STATUS_USER_index (LOGINTIME,SERVICE,STATUS,USER),
KEY LOGINTIME_IP_index (LOGINTIME,IP)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE report (
type char(16) NOT NULL,
service char(16) NOT NULL,
login_service char(16) DEFAULT NULL,
ip char(200) DEFAULT NULL,
user char(100) DEFAULT NULL,
failcount int(11) DEFAULT NULL,
logintime datetime DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE whitelist (
IP char(128) NOT NULL,
ISPREFIX int(1) DEFAULT ‘0’,
UNIQUE KEY IP (IP),
KEY ISPREFIX_index (ISPREFIX),
KEY IP_index (IP)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
root@1 [/]#

===========

Virtuozzo 4.7 Install Error :: lspci

Virtuozzo 4.7 install show Hardware check errors as below :

=========
[general] error getting information about file /sbin/lspci (No s|
=========

The error is due to missing package pciutils which is not included in CentOS 6 minimal install.

Install pciutils using yum on CentOS server :

yum install pciutils

Once done you can start the install without issues.

Uninstall CloudLinux :: cPanel server

CloudLinux is a good tool however we had some issues with our test production server and had to remove it. Here are the steps to remove CloudLinux :

CloudLinux Kb has below steps :
https://helpdesk.cloudlinux.com/index.php?/Knowledgebase/Article/View/33/0/how-do-i-convert-back-to-centos

yum remove liblve-devel
cd /usr/local/src/
wget http://repo.cloudlinux.com/cloudlinux/sources/cln/centos2cl
sh centos2cl -c

After executing script from CloudLinux, you need to remove the CL kernels as provided in the output of the script.

eg : (This may be different so watch the output of “sh centos2cl -c”)
rpm -e --nodeps kernel-2.6.32-379.5.1.lve1.1.9.6.1.el6.x86_64 kernel-headers-2.6.32-379.5.1.lve1.1.9.6.1.el6.x86_64 kernel-2.6.32-379.9.1.lve1.1.9.7.1.el6.x86_64 kernel-firmware-2.6.32-379.9.1.lve1.1.9.7.2.el6.noarch kernel-2.6.32-379.9.1.lve1.1.9.7.2.el6.x86_64

Once done install the CentOs kernel :

yum install kernel kernel-headers
yum update

Make sure grub entries are fine and reboot.

Once the server is back execute below command to cleanup CL :

yum remove lve liblve lve-utils lve-stats

Finally run easyapache to rebuild :

/scripts/easyapache --build

You should be fine / free from CloudLinux 🙂

CloudLinux Uninstall : Easyapache error : configure: error: C preprocessor “/lib/cpp” fails sanity check

I was testing CloudLinux on a production server however it turned out bad and had to remove it. After removing the kernel and reboot, last step was to run easyapache which gave errors as below :

==========
checking how to run the C preprocessor... /lib/cpp
configure: error: in `/home/cpeasyapache/src/httpd-2.2.23/srclib/apr':
configure: error: C preprocessor "/lib/cpp" fails sanity check
See `config.log' for more details
configure failed for srclib/apr
==========

Further checking i found that the kernel-headers package was missing which was installed using command :

=============
yum install kernel-headers-$(uname -r)
=============

After installing kernel headers easyapache executed fine and the server was free from CloudLinux. 🙂

Directadmin :: httpd error Invalid command ‘php_admin_flag’

On Directadmin server httpd fails to start with below error :

# service httpd restart
Stopping httpd: [FAILED]
Starting httpd: Syntax error on line 47 of /usr/local/directadmin/data/users/linuxbabu/httpd.conf:
Invalid command 'php_admin_flag', perhaps misspelled or defined by a module not included in the server configuration

Solution :

Execute below commands to rebuild php and http config files.

./build all d
./build rewrite_confs