In this blog post I’d like to outline an idea on how to perform an automated compromise assessment on Citrix Netscaler / Citrix ADC appliances.

 

I you haven’t heard about that vulnerability yet, you should read my tweets over the past weekend or try to get a full picture with the help of this Reddit

This blog post is about a subsequent forensic analysis, not detection, not protection and contains no marketing blah-blah.

The described solution should work with the free / open scanners as well. They just don’t have that huge signature set as it is included in THOR.  

 

The reason for this is that although you’ve implemented the workaround on Monday 13th of January, you can’t trust your gateway anymore.

The vulnerability is known since December 17th and the public exploit code available since Friday the 10th of January. Nation state actors have most likely used Christmas holiday season to analyze Netscaler systems in their lab and produce an exploit, just like many offensive researchers did. Since Friday the 10th, even script kiddies can exploit this vulnerbility.

Strictly speaking, you can’t trust these gateways anymore. 

But how can we regain trust? 

A. Wait for a patched version and reinstall the gateways from the scratch

B. Perform a manual forensic analysis 

C. Perform an automated forensic analysis 

Automated Analysis with THOR

We don’t have and plan a THOR (SPARK Core or LOKI) version for FreeBSD 8.4, but we could try to mount the remote filesystem of the Netscaler gateway with SSHFS. 

DigitalOcean has a good tutorial on how to enable it on your workstation.  

mkdir ~/netscaler-mount
sudo sshfs -o allow_other,defer_permissions nsroot@citrix-gateway.nextron:/netscaler ~/netscaler-mount/

We can then use one of our scanners on that mounted volume using the `–fsonly` parameter (which activates forensic lab mode and scans directories regardless of their type). 

./thor64 --fsonly -p ~/netscaler-mount

With this method, we can detect dropped malicious templates. 

You can use the public YARA rule, that I’Ve published in LOKI’s and SPARK Core’s signature base. (note that a SPARK Core version that includes that rule hasn’t been released yet) 

Important note: this only works for templates that have been dropped since the last reboot. Users reported that dropped malicious templates get removed during a reboot. As the workaround described by Citrix requires a reboot, scanning with this single YARA rule alone won’t be a reliable method.

I recommend scanning with the full signatures set in all scanners, as they also include rules for web shells and other suspicious content in files. I’ll also extend the YARA rules on github and add further indicators. 

Additional Indicators

We can use the following YARA rule to check for suspicious strings in log files. But we don’t want to use that rule on any Linux/FreeBSD system. Therefore I won’t integrate that YARA rule in the public signature base. You can just copy it from this blog post and place it in the ‘custom-signatures/yara’ sub folder (‘/signatures’ for LOKI). 

This rule for the forensic artefacts is based on remarks made in this article by TrustedSec and shared as gist. 

First, we mount the ‘/var/log’ directory from our remote Netscaler system. 

mkdir ~/netscaler-logs
sudo sshfs -o allow_other,defer_permissions nsroot@citrix-gateway.nextron:/var/log ~/netscaler-logs/
We then start a scan on the mounted log directory.

As you can see, the scanner noticed a an issued ‘whoami’ command in the ‘bash.log’ and ‘notice.log’ files, which is a good starting point for further investigations. 

Final Thoughts

I think we’ll develop more rules for post-exploitation payloads over the coming days. Stay tuned and follow my twitter account for updates.