In this blog post I’d like to outline an idea on how to perform an automated compromise assessment on Citrix Netscaler / Citrix ADC appliances.
I you haven’t heard about that vulnerability yet, you should read my tweets over the past weekend or try to get a full picture with the help of this Reddit.
This blog post is about a subsequent forensic analysis, not detection, not protection and contains no marketing blah-blah.
The described solution should work with the free / open scanners as well. They just don’t have that huge signature set as it is included in THOR.
The reason for this is that although you’ve implemented the workaround on Monday 13th of January, you can’t trust your gateway anymore.
The vulnerability is known since December 17th and the public exploit code available since Friday the 10th of January. Nation state actors have most likely used Christmas holiday season to analyze Netscaler systems in their lab and produce an exploit, just like many offensive researchers did. Since Friday the 10th, even script kiddies can exploit this vulnerbility.
Strictly speaking, you can’t trust these gateways anymore.
But how can we regain trust?
A. Wait for a patched version and reinstall the gateways from the scratch
B. Perform a manual forensic analysis
C. Perform an automated forensic analysis
Automated Analysis with THOR
We don’t have and plan a THOR (SPARK Core or LOKI) version for FreeBSD 8.4, but we could try to mount the remote filesystem of the Netscaler gateway with SSHFS.
DigitalOcean has a good tutorial on how to enable it on your workstation.
sudo sshfs -o allow_other,defer_permissions firstname.lastname@example.org:/netscaler ~/netscaler-mount/
We can then use one of our scanners on that mounted volume using the `–fsonly` parameter (which activates forensic lab mode and scans directories regardless of their type).
With this method, we can detect dropped malicious templates.
Important note: this only works for templates that have been dropped since the last reboot. Users reported that dropped malicious templates get removed during a reboot. As the workaround described by Citrix requires a reboot, scanning with this single YARA rule alone won’t be a reliable method.
I recommend scanning with the full signatures set in all scanners, as they also include rules for web shells and other suspicious content in files. I’ll also extend the YARA rules on github and add further indicators.
We can use the following YARA rule to check for suspicious strings in log files. But we don’t want to use that rule on any Linux/FreeBSD system. Therefore I won’t integrate that YARA rule in the public signature base. You can just copy it from this blog post and place it in the ‘custom-signatures/yara’ sub folder (‘/signatures’ for LOKI).
First, we mount the ‘/var/log’ directory from our remote Netscaler system.
sudo sshfs -o allow_other,defer_permissions email@example.com:/var/log ~/netscaler-logs/
As you can see, the scanner noticed a an issued ‘whoami’ command in the ‘bash.log’ and ‘notice.log’ files, which is a good starting point for further investigations.