Basics

Securing the TMP Partition and Tracking Hacks

by

Are your temp partitions putting out behind your back? Anyone who’s ever administered a Linux server would know the risk of leaving the /tmp directory unsecured, moreso on a webserver that is shared among multiple websites.

The tmp directory is world-writeable and used by a majority of services on a machine — including the storage of PHP and MySQL session files. One issue that I’ve seen on older servers is that one customer’s poorly-coded website would get exploited and end up downloading a file into the /tmp directory, then have that hack file executed on the server as a nobody-owned process. Hack processes are the easiest to find, as they always have a little something extra.

This should go without saying, but hack processes can run from almost anywhere, not just /tmp. I’m mainly bringing up the tmp directory because it’s the most targeted location for hack files if it isn’t secured properly.

Identifying Hack Processes

To start out, log into your server as root as issue the following command to see all the nobody-owned processes running on the machine. This is assuming that Apache on your webserver runs as ‘nobody’:

root@localhost[~] ps -efw |grep nobody |more

A majority of the output will reflect legitimate Apache processes:

nobody 8748 25841 0 21:35 ? 00:00:00 /usr/local/apache/bin/httpd -DSSL
nobody 8785 25841 0 21:35 ? 00:00:01 /usr/local/apache/bin/httpd -DSSL
nobody 8988 25841 0 21:36 ? 00:00:00 /usr/local/apache/bin/httpd -DSSL

You’ll see that they all have the same parent process ID of ‘25841′ and all of the commands look about the same. However, you may see something that looks like this:

nobody 15707 26407 0 11:18 ? 00:00:00 [sh <defunct>]
nobody 15717 1 0 11:18 ? 00:00:04 /usr/bin/perl
nobody 13016 1 0 14:14 ? 00:00:00 /usr/bin/perl

nobody 8988 1 0 21:36 ? 00:00:00 /usr/local/apache/bin/httpd -DSSL -d3

All of these are hack processes. A few things to look for:

– Perl and shell (sh) processes on most systems run as the user that is executing them, not ‘nobody’. If this is true on your system, a nobody-owned perl process is probably a hack

– The parent ID of the process is different than all the others of the same service — these can be easily faked

– For Apache, an extra little switch on the end of the command (../httpd -DSSL -d3) – ‘d3′ is usually the name of the hack file that is being executed through Apache (not always).

Now I know you probably have the urge to kill -9 it, but don’t. Killing a hack process before determining what it is or where it came from is only going to let it happen again.

Tracking the process:

Once you know which processes should not be running, you need to know where they came from. The *easiest* way to track a process is through the lsof command. If you take the process ID (not the parent) and run an lsof it will tell you everything you want to know:

lsof -p 8748

From here I can see that this process is running from a folder in a user’s account. All I need to do now is kill it off, then remove the hack files from that user’s directory and let them know to update their shit.

TMP Directory Hacks

I’ve seen my fair share, but many hack processes will be spawned from the temp partitions of the server (like /tmp, /tmpDIR, /dev/shm), as these are usually set to 777 to allow programs to store their temporary files. When you first look in your TMP directory you’re going to see a lot of stuff — this is normal, but you’ll want to look for nobody-owned files that stand out. Just a hint, some hacks will try to trick you with their names. I have seen many that are named “…” or “httpd” to make them look legitimate, but I know that httpd (the service that runs Apache) should not be in the tmp directory…and that’s what makes it stand out.

Securing the tmp Partition

I’m not going to go too far into this here, because there’s already a very good website with this information available.

If you are on a cPanel server, you should run /scripts/securetmp in addition to following the instructions above.

Installing IonCube loader with Zend Optimizer – cPanel

by

This is a common request we get for Ioncube to be installed. It’s generally not an issue, but when you factor in other optimization plugins like Zend and eAccelerator, a common misconception is that the three don’t get along. It’s very easy to install Ioncube into a PHP installation that already has Zend and eAccelerator.

This tutorial is specific to cPanel, assuming that you are using php 5.2.x with Zend 3.x.x.

If you need help installing eAccelerator, you can see this tutorial. For help with installing Zend, you can go here. The versions in both these tutorials are outdated, so you’ll probably want to apply the instructions to the newest versions available.
Go to http://www.ioncube.com/loader_download.php and pick your download. This example assumes that you are using php 5.2.

cd /usr/src
wget http://downloads2.ioncube.com/loader_downloads/ioncube_loaders_lin_x86.tar.gz
tar -xvzf ioncube_loaders_lin_x86.tar.gz
cd ioncube

Copy the loader config to the user’s public_html or another location where you can access it from a browser.

cp ioncube-loader-helper.php /home/username/public_html

Now in your browser go to the loader file that you just copied. This file will tell you exactly which extension you need to use. Choose the ‘php.ini Installation Instructions’ link, and you should see something like this after the php config output:

zend_extension = /<path>/ioncube_loader_lin_5.2.so

Move the ioncube directory to a more permanent location:

mv /usr/src/ioncube /usr/local
chown -Rf root:root /usr/local/ioncube

Edit the php.ini and add look for this section (may not be exact):

[Zend]
zend_extension_manager.optimizer=/usr/local/Zend/lib/Optimizer-3.0.1
zend_extension_manager.optimizer_ts=/usr/local/Zend/lib/Optimizer_TS-3.0.1
zend_optimizer.version=3.0.1
zend_extension=/usr/local/Zend/lib/ZendExtensionManager.so
zend_extension_ts=/usr/local/Zend/lib/ZendExtensionManager_TS.so

Above this section, add this line:

zend_extension=/usr/local/ioncube/ioncube_loader_lin_5.2.so

Of course, make sure that the .so file is the one that the loader helper told you to use! After that is added, STOP and then START Apache to make sure that it’s loading. You should now see IonCube in your phpinfo file.

If you’re using eAccelerator, you shouldn’t need to change the location of the plugin loader in your php.ini.

Note that if Apache doesn’t start, it’s probably because of the order in which you have Zend and ioncube loading.  The lines for Ioncube should be above those for Zend optimizer.

Lastly, you should test your IonCube installation to make sure that it can decode its own files. In the original ‘ioncube’ directory that you moved, there’s a test ‘ioncube-encoded-file.php’ file that you can load through a browser to make sure that it works.

eAccelerator on PHP 5

by

Here is what I did to get eAccelerator 0.9.5 running on PHP 5 on my V-Dedicated server. I did the exact steps when I had PHP 4 running the only difference was that with PHP 4 I used the 0.9.4 install of eAccelerator instead of 0.9.5. Everything was done via SSH access.

1. cd /usr/local/src/
2. wget http://bart.eaccelerator.net/source/0.9.5/eaccelerator-0.9.5.tar.bz2
3. bunzip eaccelerator-0.9.5.tar.bz2
3. tar -xvf eaccelerator-0.9.5.tar
4. cd eaccelerator-0.9.5
5. Run the following command inside that directory /usr/local/bin/phpize. You should get output like the following

Configuring for:
PHP Api Version: 20041225
Zend Module Api No: 20050922
Zend Extension Api No: 220051025

6. Run the next commands to get it all configured to work with your system and compiled/installed

./configure –enable-eaccelerator=shared –with-php-config=/usr/local/bin/php-config
make
make install

7. After running the make install the system will tell you know where it installed the module to. Take note of that directory, it should be something much like the following

Installing shared extensions: /usr/local/lib/php/extensions/no-debug-non-zts -20050922/

you will need this information for the php.ini else the module will not load.

8. Now you need to create the eaccelerator cache directory and give it the correct permissions.

mkdir /tmp/eaccelerator
chmod 0777 /tmp/eaccelerator

9. This is reallt the last main step, Next you need to add the following into your php.ini so that the module loads.

;;;;;;;;;;;;;;;;
; eAccelerator ;
;;;;;;;;;;;;;;;;
extension=”/usr/local/lib/php/extensions/no-debug-non-zts-20050922/eaccelerator.so”
eaccelerator.shm_size=”16″
eaccelerator.cache_dir=”/tmp/eaccelerator”
eaccelerator.enable=”1″
eaccelerator.optimizer=”1″
eaccelerator.check_mtime=”1″
eaccelerator.debug=”0″
eaccelerator.filter=””
eaccelerator.shm_max=”0″
eaccelerator.shm_ttl=”0″
eaccelerator.shm_prune_period=”0″
eaccelerator.shm_only=”0″
eaccelerator.compress=”1″
eaccelerator.compress_level=”9″

Of course make sure to change out extension=”/usr/local/lib/php/extensions/no-debug-non-zts-20050922/eaccelerator.so” with the exact location of your module as told to you after the make install command.

10. Restart Apapche and it should be all set. You can load up your phpinfo page and you should see that the eAccelerator is loaded on the system

Preventing DDOS Attacks with Mod_Evasive

by

Denial of Service attacks are among the oldest yet most common form of attacking a server. Most system administrators have had to deal with DOS attacks taking down a server, router, or other networking device and know how difficult they can be to prevent.

Mod_evasive is an Apache module that limits the number of Apache connections to the server at once, and blocks an offending IP for a specified amount of time. This tutorial will show you how to install mod_evasive on your system.

1. Install:

wget http://www.zdziarski.com/projects/mod_evasive/mod_evasive_1.10.1.tar.gz
tar -xvzf mod_evasive_1.10.1.tar.gz
cd mod_evasive_1.10.1
/usr/local/apache/bin/apxs -cia mod_evasive.c

2. Configure:

Once the module is compiled, add these lines to httpd.conf and stop Apache completely before starting it up again:

<IfModule mod_evasive.c>
DOSHashTableSize 3097
DOSPageCount 6
DOSSiteCount 50
DOSPageInterval 2
DOSSiteInterval 2
DOSBlockingPeriod 600
</IfModule>

Below is an explanation of these settings:
DOSHashTableSize – Size of the hash table. The greater this setting, the more memory is required – faster

DOSPageCount – Max number of requests for the same page within the ‘DOSPageInterval’ interval

DOSSiteCount – Max number of requests for a given site, uses the ‘DOSSiteInterval’ interval.

DOSPageInterval – Interval for the ‘DOSPageCount’ threshold in second intervals.

DOSSiteInterval– Interval for the ‘DOSSiteCount’ threshold in second intervals.

DOSBlockingPeriod – Blocking period in seconds if any of the thresholds are met. The user will recieve a 403 (Forbidden) when blocked, and the timer will be reset each time the site gets hit when the user is still blocked.

Changing the SSH Port

by

Sometimes you don’t want SSH running on standard port 22. Here are the steps to change it:

It’s VERY important that you leave port 22 open while you are testing the new port, otherwise you may lock yourself out of the server!
SSH into the server normally…

cp /etc/ssh/sshd_config /etc/ssh/sshd_config.root

In the /etc/ssh/sshd_config.root file that you just made, specify a new port number with the Port line:

Port 9670
#Protocol 2,1
#ListenAddress 0.0.0.0
#ListenAddress ::

Also, make sure that PermitRootLogin is set to yes or commented out.
Now open /etc/init.d/sshd and locate this section:

# Some functions to make the below more readable
KEYGEN=/usr/bin/ssh-keygen

Directly below that, add this line:

OPTIONS=”-f /etc/ssh/sshd_config.root”

Now locate this line:

initlog -c “$SSHD $OPTIONS” && success || failure

Directly ABOVE that add this line:

initlog -c “$SSHD” && success || failure

Now you need to restart sshd (service sshd restart). Before shutting down port 22, make sure that whatever port you created for ssh is either opened in the firewall or has the appropriate IPs added. Make sure to test this in a separate ssh window to make sure you can log in!

Once you’ve verified that you can log in through the new port, you have a couple options for port 22.

  • Close it altogether or just open it for specific IPs as usual
  • Disable root logins through port 22 (preferred)

To disable root logins, open the original /etc/ssh/sshd_config file and uncomment out PermitRootLogin and set to No. This will essentially allow someone to log in as the user, but not root. You didn’t really change the SSH port, you’re just running a copy of it on another port and making port 22 useless.

If you want to disable port 22 altogether, you can just directly edit the /etc/ssh/sshd_config file and restart SSHD, but you only have one chance to get it right.

Once the port is changed you would ssh into the server with the command ssh -p 9670 root@server with 9670 being the port you chose for ssh.

Closing Open Nameservers – DNS recursion

by

Open nameservers allow anyone in the world to perform queries on them, which can often lead to DOS attacks and slower performance. Some system administrators prefer to have their nameservers restricted to only trust parties. To do this kind of setup, you will need to configure your named configuration:
On command line:

pico /etc/named.conf

Look for this line at the top:

include “/etc/rndc.key”;

Now add this right below it:

acl “trusted” {
205.134.252.71;66.117.3.128;127.0.0.1;
};

The IPs should be those of the nameservers…you can add other trusted IPs as well.

Now in the options section right below that, add these lines:

allow-recursion { trusted; };
allow-notify { trusted; };
allow-transfer { trusted; };

So your options section will look like this:

options {
directory “/var/named”;
dump-file “/var/named/data/cache_dump.db”;
statistics-file “/var/named/data/named_stats.txt”;
/*
* If there is a firewall between you and nameservers you want
* to talk to, you might need to uncomment the query-source
* directive below. Previous versions of BIND always asked
* questions using port 53, but BIND 8.1 uses an unprivileged
* port by default.
*/
// query-source address * port 53;
allow-recursion { trusted; };
allow-notify { trusted; };
allow-transfer { trusted; };
};

Restart named

/etc/init.d/named restart

Now you can check a service like DNSreport to make sure the changes took.

Useful MySQL Commands

by

These command assume you are running a cPanel MySQL installation and logged into the server as a root user where you can do a mysql -u root without having to specify a password.
Import a database:

mysql -u root db_name < file.sql

Dump a Database (with special characters):

mysqldump -Q –add-drop-table db_name > file.sql

Dump a Database, 4.0 compatible:

mysqldump –compatible=mysql40 –add-drop-table –quote-name db_name > file.sql

Dump Multiple Databases:

mysqldump -Q –add-drop-table –databases db_name1 db_name2 > file.sql

Restore Multiple Databases (must be done as root):

mysql -u root < file.sql

Log into MySQL prompt as the user (or root):

mysql -u username -p

Show Databases: (will only show databases the user has access to. Root has all.)

show databases;

Drop a whole database:

drop database user_databasename;

Create a database: *only a root mysql user can use this command

create database user_newdb;

Select a database to work on:

use user_testdb;

Drop a specific table:

drop table table_name;

Show all MySQL Processes:

show full processlist;

MySQL Root Login Failure

by

When trying to log into the MySQL shell as root you get an access denied error. To fix:

Add this line to /etc/my.cnf and restart MySQL:

skip-grant-tables

Follow these steps to reset the MySQL root password:

mysql -u root

mysql> FLUSH PRIVILEGES;
mysql> GRANT ALL PRIVILEGES ON *.* TO root@localhost IDENTIFIED BY ‘password’ WITH GRANT OPTION;
mysql> FLUSH PRIVILEGES;
mysql> exit;

service mysql restart

Remove the line you added to my.cnf and restart MySQL again.

Now go into WHM and reset the MySQL root password again. This is is necessary in order to build an association with cPanel and phpmyadmin, as well as the root user on the server to MySQL (meaning, when logged in as root to the server, you don’t have to specify a mysql root password to log in).

Missing MySQL Socket (mysql.sock) Error

by

If you get some variation of the error below:

error: ‘Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’

You need to recreate the MySQL socket symlink in /tmp:

ln -s /var/lib/mysql/mysql.sock /tmp/

If you have magically disappearing mysql.sock file, it could point to a memory issue or disk space issue on the /tmp partition.

Missing libmysqlclient.so.14 Error

by

This is a common error when dealing with command line software that uses MySQL:

error while loading shared libraries: libmysqlclient.so.14: cannot open shared object file: No such file or directory

Usually this simple command will correct the problem:

cp /usr/lib/mysql/libmysqlclient.so.14 /usr/lib