I've recently been dealing with a very typical security problem. I rebuilt gsarchives.net some 6 years ago, and I occasionally deal with its maintenance issues. On January 1, 2, and 3, its front page was defaced by teh hax0rs. I looked into it and found some exploit code in with the code for a popular php forum (Simple Machines) that, regrettably, we hadn't updated in years. I wiped that out an installed a new version of the same system. Lesson one: When using third party software packages, apply security patches.

A week or two later, I found the entire public_html directory empty, with the exception of one of the admin's personal directories. This contained another forum, again containing malicious php placed by hackers. We let any administrators keep whatever files they wanted on our web server, and that was a mistake. Lesson two: Anyone who can put files on your server can introduce their own security risks. Make sure they know what they're doing, or better, don't allow server-side scripts to run in users' personal directories.

Because php applications are incredibly easy to deploy, it's also incredibly easy not to heed the rather ubiquitous security warnings. If directories with executable code are also directories to which your scripts have write access, you're in a very dangerous situation.

Fortunately, after the first hack, I had made a full backup. I downloaded the tar, and I'm putting sections of content back on the web server after I sure there's no more malicious in code them. I've learned some fun unix stuff in the process.

First, I had a tough time downloading the backup. The 2.5-gigabyte download kept stalling out, so I had to break it up into smaller files.

split -b 250000000 backup.tar.gz

This cuts backup.tar.gz into ten 250-meg pieces, which are easy to scp down and concatenate back together on my machine.

cat xaa xab xac xad xae \
    xaf xag xah xai xaj > backup.tgz

Next, I've been doing a ton of grepping to make sure the content is clean. The content (due to my own dumb design years ago) has a lot of index.php files in it. So first I'm checking to make sure there are no php files that aren't called index.php:

find . | grep .php | grep -v /index.php$

Now to do a quick check of the code inside them, I do a search on php files, call cat on each of them, and pipe the output into less so I can look through the contents of all of these files at once.

find . | grep .php$ | xargs cat | less

All in all, this was a good learning experience.