WHM/cPanel cPHulk Brute Force Protection Lock Out Via SSH 

One of my company’s servers is hosted with Liquid Web. Yesterday one of my co-workers tries to log intoWHM and sees the following message:

Brute Force Protection

This account is currently locked out because a brute force attempt was detected. Please wait 10 minutes and try again. Attempting to login again will only increase this delay. If you frequently experience this problem, we recommend having your username changed to something less generic.

He tries to log in a bit later and receives the message again. He contacts Liquid Web’s Heroic Support, and the support person “helpfully” recommend a server reboot to fix the problem. My co-worker asked for another solution, and support said that it was the only way.

My co-worker tells me about the situation, and I tell him that the support guy is an idiot. It takes all of about ten seconds to quickly find a solution to the problem on google. I then tell my co-worker that I’ll fix the problem, so he “thanks” the support guy and closes the chat.

I logged into the box via SSH and had the problem fixed in a couple of minutes.

Information Gathering

Since access to the box wasn’t urgent, I did what I recommend that everyone does in situations like this: gather information first. I wanted to know what caused the lock out first. In other words, who or what was brute forcing the box and caused this issue.

It was quite possible that my box was still under attack. If I simply turned off brute force protection to bypass the block, I could have opened up my box to being compromised.

cPHulk stores all of its information in a database called cphulkd. There are two tables of interest:logins and brutes. The logins table stores login authentication failures. The brutes table stores excessive authentication failures indicative of a brute force attack.

Here is what I saw on the server:

[chris@office ~]$ ssh server
Last login: Wed Oct 14 11:02:14 2009 from host
[user@server ~]$ mysql -u user -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is id
Server version: version

Type 'help;' or '\h' for help. Type '\c' to clear the current input
statement.

mysql> connect cphulkd
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Connection id:    id
Current database: cphulkd

mysql> select IP, BRUTETIME from brutes order by BRUTETIME;
Empty set (0.00 sec)

mysql> select IP, LOGINTIME FROM logins order by LOGINTIME;
+---------------------------------+---------------------+
| IP                              | LOGINTIME           |
+---------------------------------+---------------------+
| 220.199.6.48                    | 2009-10-14 11:23:10 |
| 220.199.6.48                    | 2009-10-14 11:23:10 |
| 220.199.6.48                    | 2009-10-14 11:23:10 |
| 118.212.186.59                  | 2009-10-14 11:23:40 |
| 118.212.186.59                  | 2009-10-14 11:23:40 |
| 118.212.186.59                  | 2009-10-14 11:23:40 |
| djdeatheater.liquidweb.com      | 2009-10-14 11:24:03 |
| 221.7.58.37                     | 2009-10-14 11:24:07 |
| 221.7.58.37                     | 2009-10-14 11:24:07 |
| 221.7.58.37                     | 2009-10-14 11:24:07 |
| djdeatheater.liquidweb.com      | 2009-10-14 11:24:09 |
| djdeatheater.liquidweb.com      | 2009-10-14 11:24:15 |
| mail.ingener.com                | 2009-10-14 11:24:53 |
| mail.ingener.com                | 2009-10-14 11:24:57 |
| 123.147.144.45                  | 2009-10-14 11:25:16 |
| 123.147.144.45                  | 2009-10-14 11:25:16 |
| 123.147.144.45                  | 2009-10-14 11:25:16 |
| 119.62.128.42                   | 2009-10-14 11:25:41 |
| 119.62.128.42                   | 2009-10-14 11:25:41 |
| 119.62.128.42                   | 2009-10-14 11:25:41 |
| pomme.sai.msu.ru                | 2009-10-14 11:26:13 |
| pomme.sai.msu.ru                | 2009-10-14 11:26:13 |
| pomme.sai.msu.ru                | 2009-10-14 11:26:13 |
| 84-74-21-119.dclient.hispeed.ch | 2009-10-14 11:26:48 |
| 84-74-21-119.dclient.hispeed.ch | 2009-10-14 11:26:48 |
| 84-74-21-119.dclient.hispeed.ch | 2009-10-14 11:26:48 |
| 114.143.242.51                  | 2009-10-14 11:27:23 |
| 114.143.242.51                  | 2009-10-14 11:27:23 |
| 114.143.242.51                  | 2009-10-14 11:27:23 |
| 222.179.116.53                  | 2009-10-14 11:27:47 |
| 222.179.116.53                  | 2009-10-14 11:27:47 |
| 222.179.116.53                  | 2009-10-14 11:27:47 |
+---------------------------------+---------------------+
32 rows in set (0.00 sec)

As you can see, a distributed brute force login attempt was launched starting at 11:23am. Fortunately, cPHulk quickly recognized the attack and had completely shut down login access for the user account.

Thus, when we tried to log in, it of course didn’t work.

Regaining Access

Since I now knew the problem, and now felt good about how cPHulk protected our box, I decided to open up access again so we could go about our work. There are two different ways that I could have done this.

The first way would be to disable cPHulk to regain access, log into WHM, clear out the the block by using the “Flush DB” option in the cPHulk settings page, and then re-enable cPHulk. A number of people recommended this method, but I didn’t like it. I certainly don’t want to disable a security measure that successfully protected the box just to be able to regain access. What would happen if a huge wave of brute force authentication attempts hit the box in the time between disabling and re-enabling cPHulk? The answer is that the box wouldn’t protest and would tell the attacking program whether each attempt was successful or not.

If you need to use this method, the two commands you will want to use are:/usr/local/cpanel/bin/cphulk_pam_ctl --disable and/usr/local/cpanel/bin/cphulk_pam_ctl --enable. These two commands will disable and enable cPHulk, respectively.

I decided to use another method. This method didn’t require disabling cPHulk, and thus, didn’t require reducing protection to regain access. Essentially, I cleared the tables manually, so that I could log in once again. Since this opens up the login process again, a brute force could still proceed but would be quickly shut down again.

While still connected to the database through the MySQL monitor, I ran a couple more queries.

Note: Before I ran these queries. I stored the details of these attacks in a file on the server in case I needed to refer to the information later. Never delete attack data without keeping a record of it somewhere. You never know when you may need it.

mysql> delete from brutes;
Query OK, 0 rows affected (0.00 sec)

mysql> delete from logins;
Query OK, 32 rows affected (0.00 sec)

Now, we can log back into the box.

Finishing Up

I’m sure that many people encountering this situation would just pat themselves on the back and go about their day, but I wasn’t satisfied.

I wanted to ensure that the office wouldn’t be blocked in the event that another brute force attack happens, so I added our office to the whitelist. The whitelist supports both IP numbers and hostnames. So, if you have a dynamic IP, you can use a DDNS (Dynamic DNS) service to get your own hostname to use. Both DynDNSand No-IP offer reliable free solutions that provide you with a hostname to use. In addition, EveryDNS is a free DNS service provider that also offers the ability to create subdomains of your personal domain that can link to dynamic IPs. Many current routers support either DynDNS, No-IP, or both so that the router can handle updating the hostname automatically. Some router firmwares, such as Tomato support a much larger variety of DDNS services.

Another nice feature about using the DDNS services and whitelisting a hostname is that you can update that DNS pointer remotely. This means that if you get locked out of WHM when you are away from the location that is whitelisted, you can update the IP on the DDNS system, wait a few minutes for the IP to update, and then log into WHM via the whitelist.

One final thing I did was to change the account’s password to an even more complicated and much longer one. Good luck cracking that in 32 tries.

Final Thoughts

I have to say that I was very pleased with the performance of cPHulk, a piece of software that I didn’t configure and which did its job admirably with no intervention by us. I wasn’t as pleased with the performance of the Liquid Web support person that “helped” us. Maybe I’ve just been pampered with the amazing support quality that I’ve experience at Host Gator, but I don’t think that “reboot the server” is an appropriate response to “the security software locked us out”. In fact, I’m not even sure if a reboot would have done anything other than just piss us off.

I sent a tweet about this yesterday and was pleased to get a response from Nick Campbell, a manager at Liquid Web, who wanted to know who recommended the reboot solution so that he could “thwack them if needed”. Thanks Nick.

The terminal output styling is new. I worked on it last night, and I’m quite pleased with the results. I think it makes the output much easier to scan and understand. I’d love to know what you think about it.