Automated Citrix Netscaler Forensic Analysis with THOR

by Jan 14, 2020

Update 14.02.2023
The information in this blog post is outdated. For more information on how to scan appliances remotely using SSH see this newer blog post.

In this blog post I’d like to outline an idea on how to perform an automated compromise assessment on Citrix Netscaler / Citrix ADC appliances.

I you haven’t heard about that vulnerability yet, you should read my tweets over the past weekend or try to get a full picture with the help of this Reddit

This blog post is about a subsequent forensic analysis, not detection, not protection and contains no marketing blah-blah.

The described solution should work with the free / open scanners as well. They just don’t have that huge signature set as it is included in THOR.  

 

The reason for this is that although you’ve implemented the workaround on Monday 13th of January, you can’t trust your gateway anymore.

The vulnerability is known since December 17th and the public exploit code available since Friday the 10th of January. Nation state actors have most likely used Christmas holiday season to analyze Netscaler systems in their lab and produce an exploit, just like many offensive researchers did. Since Friday the 10th, even script kiddies can exploit this vulnerbility.

Strictly speaking, you can’t trust these gateways anymore. 

But how can we regain trust? 

A. Wait for a patched version and reinstall the gateways from scratch

B. Perform a manual forensic analysis 

C. Perform an automated forensic analysis 

Automated Analysis with THOR

We don’t have or plan to release a THOR version for FreeBSD 8.4, but we could try to mount the remote filesystem of the Citrix Netscaler / ADC gateway with SSHFS. 

DigitalOcean has a good tutorial on how to enable it on your workstation.  

mkdir ~/netscaler-mount
sudo sshfs -o allow_other,defer_permissions nsroot@citrix-gateway.nextron:/netscaler ~/netscaler-mount/

We can then use the full version of THOR to scan that drive in lab scan mode.

./thor64 --lab -p ~/netscaler-mount

The free version doesn’t have that lab scanning feature, but we can use a set of commands that achieves something very similar. (this blog post explains the differences in more detail)

./thor-linux-64 -a Filescan --intense --norescontrol --cross-platform --alldrives -p ~/netscaler-mountan

With this method we can detect dropped malicious templates. 

You can use the public YARA rule, which I’ve published in LOKI’s and THOR Lite’s signature base.

Important note: this only works for templates that have been dropped since the last reboot. Users reported that dropped malicious templates get removed during a reboot. As the workaround described by Citrix requires a reboot, scanning with this single YARA rule alone won’t be a reliable method.

I recommend scanning with the full signatures set in all scanners, as they also include rules for web shells and other suspicious content in files. I’ll also extend the YARA rules on github and add further indicators. 

Additional Indicators

We can use the following YARA rule to check for suspicious strings in log files. But we don’t want to use that rule on any Linux/FreeBSD system. Therefore I won’t integrate that YARA rule in the public signature base. You can just copy it from this blog post and place it in the ‘custom-signatures/yara’ sub folder (‘/signatures’ for LOKI). 

This rule for the forensic artefacts is based on remarks made in this article by TrustedSec and shared as gist. 

First, we mount the ‘/var/log’ directory from our remote Netscaler system. 

mkdir ~/netscaler-logs
sudo sshfs -o allow_other,defer_permissions nsroot@citrix-gateway.nextron:/var/log ~/netscaler-logs/
We then start a scan on the mounted log directory.

As you can see, the scanner noticed a an issued ‘whoami’ command in the ‘bash.log’ and ‘notice.log’ files, which is a good starting point for further investigations. 

Final Thoughts

I think we’ll develop more rules for post-exploitation payloads over the coming days. Stay tuned and follow my twitter account for updates.

About the author:

Florian Roth

Florian Roth serves as the Head of Research and Development at Nextron Systems. With a background in IT security since 2000, he has delved deep into nation-state cyber attacks since 2012. Florian has developed the THOR Scanner and actively engages with the community via his Twitter handle @cyb3rops. He has contributed to open-source projects, including 'Sigma', a generic SIEM rule format, and 'LOKI', an open-source scanner. Additionally, he has shared valuable resources like a mapping of APT groups and operations and an Antivirus Event Analysis Cheat Sheet.

Newsletter

New blog posts (~1 email/month)

GDPR Cookie Consent with Real Cookie Banner