Automated Citrix Netscaler Forensic Analysis with THOR

Jan 14, 2020 | Newsletter, THOR

In this blog post I’d like to outline an idea on how to perform an automated compromise assessment on Citrix Netscaler / Citrix ADC appliances.

 

I you haven’t heard about that vulnerability yet, you should read my tweets over the past weekend or try to get a full picture with the help of this Reddit. 

This blog post is about a subsequent forensic analysis, not detection, not protection and contains no marketing blah-blah.

The described solution should work with the free / open scanners as well. They just don’t have that huge signature set as it is included in THOR.  

 

The reason for this is that although you’ve implemented the workaround on Monday 13th of January, you can’t trust your gateway anymore.

The vulnerability is known since December 17th and the public exploit code available since Friday the 10th of January. Nation state actors have most likely used Christmas holiday season to analyze Netscaler systems in their lab and produce an exploit, just like many offensive researchers did. Since Friday the 10th, even script kiddies can exploit this vulnerbility.

Strictly speaking, you can’t trust these gateways anymore. 

But how can we regain trust? 

A. Wait for a patched version and reinstall the gateways from the scratch

B. Perform a manual forensic analysis 

C. Perform an automated forensic analysis 

Automated Analysis with THOR

We don’t have and plan a THOR (SPARK Core or LOKI) version for FreeBSD 8.4, but we could try to mount the remote filesystem of the Netscaler gateway with SSHFS. 

DigitalOcean has a good tutorial on how to enable it on your workstation.  

mkdir ~/netscaler-mount
sudo sshfs -o allow_other,defer_permissions nsroot@citrix-gateway.nextron:/netscaler ~/netscaler-mount/

We can then use one of our scanners on that mounted volume using the `–fsonly` parameter (which activates forensic lab mode and scans directories regardless of their type). 

./thor64 --fsonly -p ~/netscaler-mount

With this method, we can detect dropped malicious templates. 

You can use the public YARA rule, that I’Ve published in LOKI’s and SPARK Core’s signature base. (note that a SPARK Core version that includes that rule hasn’t been released yet) 

Important note: this only works for templates that have been dropped since the last reboot. Users reported that dropped malicious templates get removed during a reboot. As the workaround described by Citrix requires a reboot, scanning with this single YARA rule alone won’t be a reliable method.

I recommend scanning with the full signatures set in all scanners, as they also include rules for web shells and other suspicious content in files. I’ll also extend the YARA rules on github and add further indicators. 

Additional Indicators

We can use the following YARA rule to check for suspicious strings in log files. But we don’t want to use that rule on any Linux/FreeBSD system. Therefore I won’t integrate that YARA rule in the public signature base. You can just copy it from this blog post and place it in the ‘custom-signatures/yara’ sub folder (‘/signatures’ for LOKI). 

This rule for the forensic artefacts is based on remarks made in this article by TrustedSec and shared as gist. 

First, we mount the ‘/var/log’ directory from our remote Netscaler system. 

mkdir ~/netscaler-logs
sudo sshfs -o allow_other,defer_permissions nsroot@citrix-gateway.nextron:/var/log ~/netscaler-logs/
We then start a scan on the mounted log directory.

As you can see, the scanner noticed a an issued ‘whoami’ command in the ‘bash.log’ and ‘notice.log’ files, which is a good starting point for further investigations. 

Final Thoughts

I think we’ll develop more rules for post-exploitation payloads over the coming days. Stay tuned and follow my twitter account for updates.

THOR Integration into Microsoft Defender ATP

Jan 7, 2020 | ASGARD Analysis Cockpit, ASGARD Management Center, Newsletter, Partner, THOR

Why Integrate THOR into Microsoft Defender ATP

While Microsoft Defender ATP fully plays off its strength in detecting live attacks, suspicious process starts and network connections, THOR shines as a live forensic scanner that scans the local filesystem, registry, logs and other elements for traces of hacking activity.

While Microsoft Defender ATP features a forensic package collection that retrieves elements from a remote system, THOR scans these elements on the remote system, applying more than 10,000 hand-written YARA rules and thousands of filename, C2, hash, mutex and named pipe IOCs to them. This live forensic scan reduces the work of your forensic analysts to a minimum and generates results as fast as possible for you to react in a timely manner. 

THOR extends Microsoft Defender ATP’s real-time monitoring by intense local scans to allow a full on-demand compromise assessment.

Deployment Options

Due to the fact that both Microsoft Defender ATP and THOR are very flexible and open products, the integration is no one-lane road with a single possible solution. Depending on the network size, segmentation and available 3rd party solutions like a SIEM the integration allows and requires different setups.

This blog post starts with an example use case and then outlines many of these setup options.

Live Response Scripts

The Microsoft Defender Security Center allows us to upload PowerShell scripts into a so called “live response library”, which is available on the endpoint during “live response” sessions.

These scripts allow us to facilitate the download and execution of THOR on the endpoint.

There are two ways to implement different scan modes and parameters. THOR has numerous command line options, which can be passed either as parameters of the PowerShell scripts or predefined in YAML config files.

Example: Turla Malware

We’ll use a simple demo script that contains a path to a file share providing the THOR package. 

It uses a config file named “rootkit-check.yml”, which is located in the program folder on the file share. It activates 3 rootkit related modules, sets the path for all output files as rebase-dir and deactivates some features. 

We upload that script into a live response session to investigate suspicious behaviour of a workstation that showed several alerts regarding a malware and the use of a “living-off-the-land” binary to run malicious code. 

The details reveal that the use of certutil.exe triggered the alert.

We can see other commands like tasklist, net and netstat, which are often used in reconnaissance scripts, executed in the context of a user named “admin”. 

We start a “Live Response Session” for further live forensic investigations with the help of THOR. 

Since this is our first investigation with that specific script, we have to upload it to the live response library. 

We can then verify the upload using the “library” command and run the script from the command line. 

It takes about a minute to complete the Rootkit check.

THOR recognized a malicious mutex used by Turla malware and gives further information on the related process and process binary, which can be used for additional verification of the threat. 

The HTML report and text log file have been saved back to the file share.

Other Setup Options

Scanner Provisioning

In this chapter we describe different methods to provide a THOR package to an end system during live response investigations.

Option A: File Share

The complete THOR package including binaries and signatures can be provided on a network share. This network share should be read-only to avoid that attackers notice the activity and manipulate the program or signatures on the file share.

Advantages:

  • Quick setup
  • Only a file server is needed

Disadvantages:

  • Requires SMB/CIFS connection from end system to file share
  • Scanner / signature updates must be scripted (thor-util.exe)
  • Manual license generation (in Nextron’s customer portal) or expensive IR license (not host-based)

Option B: ASGARD Management Center

The central management platform ASGARD Management Center is hardened Debian-based soft appliance that serves as software repository and licensing server in our use case.

The PowerShell scripts in the script library can retrieve THOR packages via HTTPS from the ASGARD Management Center.

Advantages:

  • HTTPS download of THOR packages
  • Integrated licensing
  • Automatic scanner and signature updates

Disadvantages:

  • Additional server system (VM; maintenance)

Option C: THOR via Script Library as SFX

The complete THOR program folder can be packaged into a self-extracting & executing archive (SFX), which could then be uploaded into the “live response library”. It could then be executed right from the script library (run) or uploaded to the end system (put).

Advantages:

  • No servers needed
  • Microsoft Defender ATP native solution

Disadvantages:

  • Scanner / signature updates and SFX creation must be scripted on an analyst system (thor-util.exe)
  • Manual license generation (in Nextron’s customer portal) or expensive IR license (not host-based)

Output Options

The results of the scans can be stored and transmitted to different locations.

Option A: Log and Report on File Share

THOR writes a log file in real-time during the scan and renders an HTML report at the end of the scan. Users can set an output directory other than the working directory for all output files with the “–rebase-dir” parameter.

This output folder can be a file share, e.g. “\\server\share”.

Analysts can check the log file during the scan, which takes between minutes and hours to complete.

Advantages:

  • Only a file server required

Disadvantages:

  • Requires access to file share from the end system (SMB/CIFS)
  • File share must be writable (possible manipulation by the attackers)

Option B: SYSLOG, JSON or CEF to SIEM

THOR can send the logs via SYSLOG (UDP, TCP, TCP+SSL, CEF) or in JSON (UDP, TCP, TCP+SSL) to a remote SIEM or log management system.

Advantages:

  • Integrates into existing solution and processes

Disadvantages:

  • Requires SIEM system and some base-lining
  • Requires connection to port 514 from end system to SIEM system

Option C: SYSLOG, JSON or CEF to ASGARD Analysis Cockpit

ASGARD Analysis Cockpit is the optimized log analysis platform (soft appliance) to process, baseline and forward THOR logs.

It most relevant features in this use case are:

  • Base-lining and central false positive filtering
  • Event forwarding of filtered events

ASGARD Analysis Cockpit already has several options to create alerts for incoming logs.

Similar to the current “Webhook” output, Analysis Cockpit could add a feature to connect with Microsoft Defender Security Center and create Alerts as described in the official API documentation.

Advantages:

  • Optimal THOR log base-lining and forwarding of relevant events only

Disadvantages:

  • Additional server system (VM; maintenance)
  • Requires connection to port 514 from end system to Analysis Cockpit

Option D: Local Eventlog

THOR can be instructed to log to the local Windows Eventlog with the “—eventlog” command line parameter. Customers that already forward their Windows Application Eventlog to a central SIEM could then use the existing integration and analyze the THOR events in their SIEM.

Advantages:

  • Integrates into existing security monitoring
  • No additional open port needed

Disadvantages:

  • Requires SIEM system and some base-lining

Option E: Live Response – “getfile”

Local log files that were written to the working directory can be retrieved with the “getfile” command.

Advantages:

  • Integrates into analyst workflow
  • No additional open port needed

Disadvantages:

  • Files could be left on the end system
    (causing false positives in other products; in plain sight for attackers)

Future Integrations

This chapter contains an outlook on expected future integrations based on upcoming features and APIs. 

Sample Collection

The Microsoft Defender ATP API allows to fetch a certain file from a remote system. Similar to the alerting mechanisms via Webhooks in ASGARD Analysis Cockpit, users will be able to fetch any suspicious or malicious file reported by THOR with a given minimum threat score using the Microsoft Defender ATP API. 

THOR Cloud

The upcoming cloud based version of our licensing and download server, which is currently integrated into our customer portal, will be able to serve THOR packages that contain an integrated license for the host which is supposed to be scanned. 

This way, you will we be able to run a PowerShell script from the live response library that downloads an up-to-date THOR package with a valid license file right from the new online service and don’t need a local ASGARD server that provides the THOR packages and licenses.

Not All IOC Scanning Is The Same

Dec 9, 2019 | SPARK, THOR

People often tell us that EDR product X already does IOC scanning and that they don’t have to check for these indicators a second time using our scanners. Especially when it comes to network wide sweeps for traces of activity due to an ongoing incident I recommend scanning a second time with one of our scanners or a tool of similar quality.

This blog post explains why.

People usually spend a fair amount of time on selecting threat intel feeds and interesting indicators for their scans. However when it comes to the actual application of these indicators they seem to be satisfied with the simplest form of checks.

Especially when we look at C2 or Filename IOCs I can easily explain the difference between the „compulsory“ and „freestyle“ methods of IOC scanning.

A plain „compulsory“ filename IOC check would walk the disk or query a database looking for a certain filename, right?

 

However if you think about it for a second and ask yourself „where else could we check for that filename?“ you’ll realize that the following elements could also contain the malicious filename:

  • Eventlog entries (e.g. process starts, service installs with image path, access failures …)
  • Log files (local Antivirus log file, access to file in web root > web server access log, backup errors, PowerShell history …)
  • Registry (recently opened files, shell bags, service image path, other caches …)
  • MFT (deleted entries)
  • Archive content (packed in ZIP file)
  • WMI (scripts – e.g. see this PoC by Matt Graeber)
  • Crash dumps
  • Windows Error Report (WER – file names and content)
  • Free disk space (filename as content of batch files or other scripts, scheduled tasks …)

Actually we often see that during lateral movement attackers access systems, run their tools remotely, copy the output, delete the output files and leave no file system traces behind. Our scanners use the locations mentioned above and others to detect them although all the files have already been removed from disk. That’s the „freestyle“ method.

The same counts for the C2 IOCs. The „compulsory“ plain method would check the system’s network connections.

The „freestyle“ method also includes checking for these C2 IOCs in the following locations:

  • Process memory (C2 strings loaded and decrypted in process memory)
  • Log files (web server access logs, Windows firewall log file, AV module log file …)
  • Hosts file
  • Files (in backdoor config files on disk)
  • Registry (hard coded C2 server in registry key)

It is sad to see great indicators from expensive feeds used into tools that do „IOC scanning“ the „compulsory“ way missing so many interesting spots.

If all you have is a hammer, everything looks like a nail.

So – the next time when someone tells you that their tool checks for IOCs on the endpoint, your question should be „How and where do you check for these IOCs?“.

Changes in Upcoming THOR Version 10.3

Dec 4, 2019 | Newsletter, THOR

Refactored Handle Detection

We have completely refactored THOR’s malicious Handle detection.

We now allow the use of regular expressions and combined all types in a single signature file named “malicious-handles.dat”. 

Users can provide custom indicators by placing a file with the keyword ‘handles’ in name into the folder ./custom-signatures/iocs

 

Process Handle Match Enrichment

Mutex, Named Pipe or Event matches will now trigger message enrichment in which the alert message is “enriched” with plenty of helpful information on the underlying process, including image file path, parent process and image file hashes. 

This helps analysts to evaluate and classify the event. 

More Eventlog Sources

We’ve integrated more Windows logs in the default Eventlog scanning to detect keyword and filename IOCs in even more data sources. 

This could increase scan duration significantly in cases in which you have defined custom unusually large maximum log sizes. 

Custom Eventlog and Registry Exclusions

Users can now exclude certain Registry paths or Eventlogs from scanning. 

We’ve added two new and empty exclude files in the ./config sub folder for your convenience.  

 

Support for new Sigma Modifiers

THOR now supports the newest value modifiers used in the most recent version of the Sigma standard.

  • endswith
  • startswith

Case ID (CID) get renamed

Previous versions of THOR offered a parameter -i / –case-id to set a unique identifier that appears as “CID: identifier” in all log lines of this specific scan. 

With version 10.3 this CID gets renamed to SCANID and will be generated by default.

Every log line will contain a SCANID that makes it easier to associate a single  line with the complete scan in reporting platforms like SIEM systems or ASGARD Analysis Cockpit.

You can still overwrite that value manually for legacy reasons or in order to group multiple scan runs into a single logical report.

Other Changes

  • Feature: Log to local syslog with `–local-syslog` (Linux and macOS only)
  • Improvement: Filename IOC “slash to backslash” transformation now works platform independent 
  • Improvement: SHIMCache Analysis in Registry Hives (not just live Registry)
  • Improvement: Support for new license type “silent” (allows silent scans used in deployment tests)
  • Improvement: YARA rule date in output messages
  • Change: -j hostname will also adjust hostname in output file names
  • Change: Don’t skip certain Registry paths with `–fullregistry` or in `–intense` mode
  • Change: New value “max_file_size_intense” to be able to set a dedicated value for “intense” or “lab” scans
  • Bugfix: missing start and end date in HTML report
  • Bugfix: Message enrichment module processed duplicate values (e.g. Eventlog contained the filepath C:\w1.exe three times, the message enrichment included the additional attributes like EXISTS, MD5, SHA1, FIRSTBYTES three times, which is unnecessary) 

THOR v10.2 Changes

Oct 22, 2019 | Newsletter, THOR

New Module “Events”

This module checks registered Events in the system environment as they are used by advanced malware and rootkits. 

We have checked for malicious Events before, in the Rootkit module, but these checks were hardcoded. We’ve spun out that section and can now provide regular updates in a separate signature file. 

The “Events” module extends our set of rootkit related modules that already include the “Mutex” and “Named Pipe” modules.

THOR DB with Timing Statistics

THOR v10.2 features an unencrypted table in THOR DB that shows timing information for the scanned elements. This could help you identifying elements that lengthen scans significantly and determine a time range in which certain elements have been scanned. 

Reduced Output

A new switch “–reduced” allows to limit events to “Warning” and “Alert” types only. 

Other Changes

  • Upgrade to YARA 3.11
  • Improved module messages (better description)
  • Bugfix: Golden ticket detection module reported far too many Kerberos tickets with too long lifetime, message: “Kerberos ticket with very long life time detected – likely a Golden Ticket”. The issue has been fixed. Please make sure that you haven’t filtered / base-lined that event type. 
  • Added ExecFlag to SHIMCache output
  • Apply YARA rules on WMI Event Filters
  • Passing new external YARA variables ‘timezone’ and ‘language’ to registry rule set
  • More robust custom YARA signature initialisation (syntax check and tests before compilation)

New Feature in THOR v10.1 – Remote Scanning

Aug 29, 2019 | LOKI, Newsletter, THOR

THOR v10.1 features a mode of operation that is especially helpful in incident response or compromise assessment scenarios – remote scanning. 

Imagine that you’re in a firefighting scenario – a breach has been confirmed and management wants to have quick results on the extent of the compromise. 

The new remote scanning feature called “THOR Remote” allows you to perform triage scans on hundreds of remote systems from a single admin workstation. You can think of it as an integrated PsExec. 

No scripting, no agents, no hustle.  

Benefits

  • No agent
  • No scripting
  • Painless scans of many remote systems

Requirements

  • Available on Windows only
  • Accessible remote ports (135/tcp, 445/tcp)
  • Account with local admin rights

All you need is the new version 10.1 of THOR and a command line of an admin user with the required privileges and open Windows ports (135/tcp, 445/tcp) on the remote systems.

THOR will then switch into a new mode of operation and present a command line interface showing scan information and a scrollable pane for each log file. (see screenshot)

THOR writes the log files to a local folder on the admin workstation or sends them via SYSLOG to your SIEM system.  

You can also define a number of concurrent scans (workers) and delay the scan starts to distribute the load evenly among the target systems. This is beneficial when you scan numerous virtual machines running on a few host systems. 

A complete triage scan of your internal domain can’t be more comfortable. 

THOR 10 Fusion Released

Jul 2, 2019 | Newsletter, THOR

THOR 10 Fusion has arrived. 

It replaces our successful scanners THOR 8 and SPARK and combines the best of both worlds. It is a completely new code base that features all modules of our 4 year old compromise assessment flagship THOR 8 and the speed and extra features of our triage scanner SPARK.

You can find an overview of the major changes in this article.

Download

All customers with an active contract (rental license) and license pack users can download THOR 10 from the “downloads” section in the customer portal.

You can find the new manual as PDF in that section and the ‘./docs’ folder of the downloaded program package. 

 

Updates

Please note that signatures updates will be much more frequent due to the decoupling of program and signature files. Make sure to use thor-util version 1.8 or higher. 

We plan to release new signature packs every 1-3 days and new program binaries about once a month. 

The old scanners will receive updates until mid-2019. However, these updates will be less frequent. 

 

ASGARD

After upgrading to ASGARD version 1.10 you’ll immediately see the new scanner in all menus. 

THOR 10 will be the new default for newly scheduled scan jobs. Old scan jobs will not be touched.

Updates of program binaries and signatures can now be managed separately from the “Updates” section. 

 

Changes to Consider

All the old command line options stayed the same as in THOR 8. However, we’d like to bring some addition features and changes to your attention. 

  • The THOR 10 program package now also contains a 64-bit executable (thor-x64.exe), which should produce much better process memory detection results. (ASGARD automatically selects the right binary)
  • Custom settings are now configured via ./conf/thor.yml and not ./conf/thor.cfg.
  • The active modules per scan mode and the log contents have been reworked. You can’t make a comparison with previous THOR 8 scan data. The log format (default) stayed the same, so that old field extractions should still work. 
  • The log contents are more detailed and more consistent (e.g. timestamp format).
  • THOR has more output options (SYSLOG formats and JSON log file output, see manual).
  • Scan durations will change. The scanner is faster but has more active features like “archive YARA scanning” (better detection for Office document macro droppers).
  • Sigma scanning is available, but has to be activated with “–sigma”. It uses all rules from the public rule repository.

See the already mentioned article for more changes. 

 

Get THOR

Check our license packs for many DFIR and SOC scenarios or request a trial of our new scanner.

Questions

If you have any questions, please contact via the support link in the customer portal. 

THOR 10 Fusion – Major Changes

Jun 11, 2019 | Newsletter, Nextron, THOR

In anticipation of our new scanner THOR 10 Fusion, we would like to show you some of the exciting new features and upcoming changes. 

Modes and Feature Cleanup

We’ve reviewed and reworked all scan modes in order to clarify the overview of active modules and features for the user. 

In the past, it wasn’t always clear which module and feature has been auto-deactivated and auto-activated during the scan runs. 

We’ve dropped the “–fast” mode, which was rarely used intentionally but auto-activated on Workstations.

Most of the modules have been completely rewritten. 

Due to higher scan speeds we didn’t have to make many compromises. The “default” scan should take roughly as long as with THOR 8 but is much more intensive. 

Modules like the “Rootkit” module have been split up in two different sections, one with important and less dangerous checks and one with less relevant checks that could lead to an Antivirus intervention (e.g. Double Pulsar check).

This refactoring allows us to activate the module in “Soft” scan mode and set e.g. “Double Pulsar” as extra feature for that module, which is activated in “Default”, “Quick” and “Intense” scan mode. 

Separate Program and Signature Updates

Former versions of THOR have been shipped and upgraded as a complete package.

The new thor-util allows you to upgrade program files and signatures separately.

We’ll try to publish new signature packs as fast as new YARA signatures get published in VALHALLA.  

Time Stamp Harmonization

The timestamps in all the different modules have been harmonized to ANSIC standard.

This was an important step to allow the creation of meaningful timelines of the discovered events. 

Configuration Files Become Scan Templates

THOR 10 uses so-called scan templates in YAML format, instead of the old config file format.

The parameters in these scan templates reflect 1:1 the command line parameters. With these new scan templates it is easy to define a set of parameters for your scan and ship them as the default scan template. 

You can even mix the configurations from multiple scan templates, e.g. define a default template and separate templates with different syslog targets for each branch office.  

 

JSON and Key/Value Output

You can choose from multiple options to influence the output format of the log files and SYSLOG messages sent to  remote servers. 

We handle log messages internally as objects and can easily render JSON or Key/Value pair outputs. 

This greatly simplifies the SIEM integration of all output streams. 

 

Difference Scan

The difference scan makes use of the THOR DB and checks only elements on disk that have been created or changed since the last scan start.

This is a new ultra fast scan mode, albeit susceptible to timestomping attacks. 

Sigma Scanning

THOR 10 inherits the Sigma scanning feature from SPARK and can now apply Sigma rules to local Eventlog entries (Windows) or log files (Windows, Linux and macOS). 

Find more information on the Sigma scanning feature in this older blog post. 

 

Better Process Memory Matches

Process memory matches now show the matching strings or code sequences found in the memory of scanned processes. 

Tagged Matches

Since our YARA rules are tagged during the integration into VALHALLA, all of them have tags including the MITRE ATT&CK tags, that help your analysts putting matches into context. 

ASGARD Integration

THOR 10 integrates seamlessly with ASGARD and shows up as third scanner next to THOR 8 and SPARK. 

The “Updates” section will show separate update settings for the scanner’s program components and signatures. 

The ASGARD menu to create new THOR 10 hunts contains all command line options dynamically extracted from the current executable.

This way it adapts to all future features and command line options that will be integrated into THOR 10 over time. 

These are only some of the changes coming with THOR 10 Fusion.

We are in schedule and excited to release it in July.

Upcoming : THOR 10 “Fusion”

May 27, 2019 | Newsletter, Nextron, SPARK, THOR

We are proud to announce the upcoming release of THOR 10 code named “Fusion”.

It will replace our scanners THOR 8 and SPARK before the end of this year. Both of the current scanners will still receive updates until the end of this year. 

THOR 10 “Fusion” combines the advantages of our current scanners, the intensive analysis capabilities of THOR with the unmatched flexibility and speed of SPARK.

It features all modules of THOR 8, including Registry, SHIM Cache, Eventlog, Mutex, WMI, Service and Autoruns analysis.   

It runs on all major operating systems – Windows, Linux and macOS.

With THOR 10 “Fusion” you will not have to decide between an intense or fast scan anymore. THOR 10 provides the best of both worlds. 

We plan to release THOR 10 in July this year. Follow us on twitter or subscribe to the newsletter for updates.

Customers will get a separate notification with changes and instructions for an upgrade.