“My web server is acting up strange, could you take a look?”
Recently, I’ve been tasked with several web server investigation engagements.
Having detected and removed many webshells and other malicious files, I’ve noticed that several simple techniques consistently prove themselves to be of value across many different environments and servers.
Today, I‘m starting this series of articles, sharing with you some of the techniques and tips that I often find valuable.
This is an introductory level article to the topic of webshell detection, I will include several references and recommendations to more advanced topics and techniques at the end of this article.
Locate the Web Server Logs
Quite often I encounter the following paths that store the web server logs:
Typical filenames for web server logs may be any of the following:
While the web server error logs usually include any of the following names:
I like to use the locate command to make sure that I haven’t missed any new paths and filenames to keep updating my list.
Analyze the Web Server Logs
As the amount of data in web server logs may be overwhelming sometimes, especially in high traffic servers, my preferred tactic is analyzing them using the bottom-up approach.
Let’s start with only one log file and check only one date in each iteration, focusing only on one HTTP method that is quite often utilized by Cybercriminals to achieve their goals using your server:
grep “02/May” /var/www/vhosts/*/logs/access_ssl_log.processed | grep POST
A similar technique may be used to investigate additional HTTP methods, or simply all the methods excluding the most common method — GET:
grep “02/May” /var/www/vhosts/*/logs/access_ssl_log.processed | grep -v GET
You will receive a long list in the following format:
<IP Address> — — <Date and Time> “<HTTP Method> /<Requested Resource> <HTTP/Protocol Version>” <Status Code> <Response Size> “<Referer>” “User-Agent”
I will focus on the following data points:
Let’s review the unique requests focusing only on the combination of IP Address, Requested Resource, Status Code, and User-Agent:
grep “02/May” /var/www/vhosts/*/logs/access_ssl_log.processed | grep POST | cut -d ‘ ‘ -f 1,7,9,12 | sort -u
For popular CMS platforms, such as WordPress, we can filter out some of the common file names and plugin generated requests to look for the outliers:
grep “02/May” /var/www/vhosts/*/logs/access_ssl_log.processed | grep POST | cut -d ‘ ‘ -f 1,7,9,12 | grep -E -v “wp-cron.php|admin-ajax.php|xmlrpc.php|wp-login.php|admin-post.php|wordfence_syncAttackData” | sort -u
And another useful combination of “locate” + “xargs” + “grep” to locate all the logs and execute our search pattern on each one of the discovered log files:
locate “/access.” | xargs grep “02/May” | grep -v GET | cut -d ‘ ‘ -f 1,7,9,12 | grep -E -v “wp-cron.php|admin-ajax.php|xmlrpc.php|wp-login.php|admin-post.php|wordfence_syncAttackData” | sort -u
locate “/access_” | xargs grep “02/May” | grep -v GET | cut -d ‘ ‘ -f 1,7,9,12 | grep -E -v “wp-cron.php|admin-ajax.php|xmlrpc.php|wp-login.php|admin-post.php|wordfence_syncAttackData” | sort -u
At this point, I usually start to see very suspicious-looking requests, exposing the paths of the webshells that Cybercriminals planted on the server.
For persistency purposes, oftentimes, once a Cybercriminal found and exploited a vulnerability against an unpatched component of a given website (e.g., outdated WP version, theme, or plugin) he will plant many webshells in different directories that he may access using the same set of permissions you have granted to your WP installation — this is why following the principle of least privilege is so important when installing software.
Once you detect and remove some of the malicious files, you will notice how the cybercriminal tries to reach his removed webshells multiple times, before revealing the path of additional files he has planted on your server.
For example, in one of my recent engagements, the technique described above helped me to find out that a single IP address (“126.96.36.199”) executed file enumeration attack using 570 unique file names across dozens of domains on the same hosting server, many of which I identified as common webshell names based on my previous engagements.
The following pattern includes another filter to find only the requests that the server returned successfully (Status Code = 200).
It can assist you to find out which webshells are active at this point and require you to remove them from the server:
locate “/access.” | xargs grep “02/May” | grep -v GET | cut -d ‘ ‘ -f 1,7,9,12 | grep -E -v “wp-cron.php|admin-ajax.php|xmlrpc.php|wp-login.php|admin-post.php|wordfence_syncAttackData” | grep “ 200 “ | sort -u
locate “/access_” | xargs grep “02/May” | grep -v GET | cut -d ‘ ‘ -f 1,7,9,12 | grep -E -v “wp-cron.php|admin-ajax.php|xmlrpc.php|wp-login.php|admin-post.php|wordfence_syncAttackData” | grep “ 200 “ | sort -u
Overall, I don’t believe in blocking “bad” IP addresses, at least not as long as they’re not involved in a denial of service attack against your server.
When you discover any information about your attackers, like a list of IP addresses and unique User-Agents they’re using, you should leverage this information against them, instead of simply blocking them using your WAF and losing your advantage.
Know about them, but don’t let them know that you know about them, this is the best way to protect your assets in the digital world, IMO.
References and Recommendations
I invite you to clone my public repository of confirmed indicators of compromise (IOCs).
You should also check out the following, more advanced resources on the topic of webshell detection:
Update: November 29, 2020
Recently, I’ve launched a website security scanner that allows you to run a basic external self-assessment for your website for free.
My goal for this project is assisting you to improve the security and reputation of your website by detecting and fixing common issues like expired domain registration, malware infections, SSL (TLS) certificate misconfigurations, implementation of email security protocols to improve email deliverability, discover your exposed credentials from data breaches, and review your subdomains just to make sure that all of them are still in use, patched and properly secured and monitored.
Later on, I will also add the options to discover all the open ports and vulnerabilities across your servers.
I invite you to check it out using the following link: