There’s a Thunderstorm Coming

We are proud to announce a groundbreaking new scan mode named “Thunderstorm” that we’ve integrated into preview builds of the upcoming THOR version 10.6.

This mode of operation turns THOR into a RESTful web service that is able to process thousands of samples per minute sent from any device within the network.

Think of it as your ultra-fast on-premise scan service, wich is bundled with more than 13,000 hand-crafted YARA rules focusing on persistent threats and forensic artefacts.

Collect files and submit them for analysis from any operating system and any hardware platform. The possibilities are limitless.

With this blog post, we’d like to highlight some of these new possibilities.

Thunder rolls, lightning strikes & the hammer flies across the sky.
God of the weather,
chariot of the storm,
master of rain & torrents.
Son of the strength
of Mother Earth,
I ask you to grant me that strength for myself.

Norse Poem

What is THOR Thunderstorm?

A RESTful web service that receives samples and returns a scan result. It is feature-rich and very fast.

Use Cases

Use Case 1 – Remote File Collection

During forensic investigations, automated file collection (ESI) from one or multiple remote systems can be combined with THOR Thunderstorm to improve the forensic anylsis.

Alerts and warnings produced by THOR Thunderstorm highlight interesting elements in file data, registry hives, eventlog files and more.

Use Case 2 – ICS Networks

ICS networks are mission critical, requiring immediate and high-availability. The installation of an endpoint agent or running a portable scanner is often out of question.

With THOR Thunderstorm, you just have to collect and submit the files.

Use Case 3 – Out of Reach Devices

Since file collection is a lot easier than endpoint scanning, all you need is way to export the remote system’s files or directly send them to THOR Thunderstorm.

Imagine that you can collect and submit files from network devices, telephone systems or embedded devices.

Use Case 4 – Out of Reach Operating Systems

File collection scripts for many old or usually unsupported operating systems allow you to upload samples for analysis.

Select files based on size, age or type and schedule frequent upload tasks to analyze only new or modified files. 

Use Case 5 – S3 Bucket Scanning

We’ve been working with our partner Adolus to showcase a tuned version of AirBnb’s BinaryAlert in which the standard YARA analyzer has been replaced by THOR Thunderstorm.

By using it in a container that scales with the demand, you can process millions of files in a few minutes.

Flexibility

Most operating system provide tools to walk the file system and submit files via HTTP. The following examples are intentionally short and compact to inspire you with their simplicity. Think of all devices that you could analyze this way. No agent, no portable scanner, just simple file submission via HTTP.

Windows 10 Batch

This example shows a simple batch file that walks recursively over a given folder an submits all files. You could extend it to the whole disk and reduce the submission to certain file extensions (e.g. exe, bat, ps1, js).

Linux Web Server

This examples shows how easy it is to get all files in a web server root checked by THOR Thunderstorm just by using bash, find and curl.

 

Thunderstorm Components

The following slide lists the different components that can be used with THOR Thunderstorm. We provide a server installer script, collectors, a Python API client and update scripts. 

In addition to the Thunderstorm server we provide a set of simple sample collection tools called Thunderstorm Collectors, a Python-based API library with command line client and a set of helper scripts

Thunderstorm Collectors

The Thunderstorm Collector repository contains a Go based collector, precompiled for many different operating systems and architectures as well as collectors scripts (Batch, Bash, PowerShell).

We have pre-build collectors for Windows, Linux, macOS, AIX, Solaris on x86, x64, Arm, PowerPC, MIPS, RISC-V, Plan9, S390x (IBM Z) architectures.

These collectors allow you select files based on age, size and type for submission to a Thunderstorm server.

It is easy to set up a task like: 

“Select all files that have been created or modified within the last 24 hours and submit them to Thunderstorm for analysis. Run this task daily.”

Low CPU and RAM Usage

A collection task requires 0.75-2% of the CPU and 20MB memory. 

Any OS, Any Arch

Our collectors run on any operating system and processor architecture

High Speed

It allows ultra fast collection runs. (Our tests: Win 10, collect last 3 days, any type, full disk = 3 minutes run)

Thunderstorm API Client

We provide a Python module and Python based API client that supports multi-threaded submission to the THOR Thunderstorm service.

Modes of Operation

Service Mode

The service can be started in two scan modes:

  • Pure YARA
  • Full-Featured

Pure YARA

In the pure YARA mode (–pure-yara) THOR Thunderstorm only applies the 13,000 internal and all custom YARA rules to the submitted samples. It’s leightweight and super fast.

Full-Featured

The full-featured mode is the default. In this mode Thunderstorm also parses and analyses Windows Eventlogs (EVTX), registry hives, memory dumps, Windows error reports (WER) and more. It’s not just a YARA scan, but a full forensic processing.

More Features

Completely On Premise

THOR Thunderstorm can be installed on any internal system and runs as a service within your network

Sample Storage

Store suspicious or all transmitted samples with a reference to the source system to facilitate the deeper analysis

Forensic Modules

THOR Thunderstorm supports the analysis of different file types that get collected for forensic analysis purposes (e.g. EVTX files, Registry Hives)

Custom Signatures and IOCs

Add you own YARA signatures, Sigma rules, hash and filename IOCs and apply them to incoming samples

SIEM Integration

THOR Thunderstorm offers many ways to output information (Text, JSON, Syslog), which makes it easy to integrate the findings into your favorite SIEM system

Web GUI and API Documentation

The API documentation is embedded into the web service itself. You can even send requests right from the browser to test it live.

The Web GUI contains important information about the service like the signature set version, uptime, number of processed and queued samples and much more. 

It contains some graphs that help you to assess the actual server load and processing speed. 

It also contains links to the API documentation, the Python API library and the Thunderstorm Collectors for your convenience. 

 

On The Roadmap

The following tasks are on our roadmap for THOR Thunderstorm

  • Collector service that uses file system notifications to submit new files in real-time
  • Cortex Analyzer
  • ICAP Support (allows interfacing with Web Proxies)
  • File format support: PCAP, MFT
  • Recursive extraction of nested archives
  • Docker setup guide

Getting Started

Please use the “GET STARTED” button in the upper right corner or this link to request more information.

The release slide deck contains more detailed information on some of the mentioned aspects.

 

THOR v10.6 TechPreview

We are proud do announce the version 10.6 of THOR, which is the first one that gets released as a TechPreview. We’ve discussed the split-up into THOR and THOR TechPreview in a previous post.  

The following post describes the most important new feature of the THOR v10.6 TechPreview version.

THOR Thunderstorm

THOR 10.6 is the first version that support a new mode of operation – a RESTful web API service named THOR Thunderstorm. THOR Thunderstorm is able to receive thousands of samples per minute via web requests, scans them and returns a scan result. 

We’ve outlined many use cases and features of THOR Thunderstorm in a separate blog post

THOR Thunderstorm requires a separate license named “service license” to run. 

 

Multi-Threaded Scanning

Especially the customers with a lab license should be happy to hear that we’ve implemented multi-threaded scanning. 

From now on, THOR can use multiple threads to process elements (files, registry keys, events in eventlog etc.). 

This can boost the scan speed on mounted images significantly. Our tests on a 16 Core system showed a scan speed improvement of 1400%. 

Reworked Quick Scan

Quick scan (–quick) is used when fast scan results are crucial. It usually takes less than 25 minutes to complete. This is achieved by skipping elements in the scan. Quick in versions previous to 10.6 do the following: they skip the Eventlog scan and scan only a set of 40+ highly relevant folders on disk. 

The new quick scan doesn’t skip whole modules or directories anymore. For all previously skipped elements the new quick scan evaluates if they have been modified or created within the last 72 hours and scans only these elements. 

This way the new quick scan is much more intense but should  be only slightly slower. 

Other Changes

  • We’ve changed the ambigious “–fsonly” flag to “–lab” to indicate the best settings for scanning in a forensic lab (the old flag is still usable but hidden in the usage description)
  • Virtual drive name mapping (used in lab scans to map the actual mount point to the original one)
  • Minor changes to some log lines (extended field values) 

Getting Started

Customers can download the THOR TechPreview version 10.6 in the Downloads section of the customer portal or use thor-util in it’s newest version to download that version with the flag “–techpreview”. ASGARD version 2.5.3 also supports scan runs with THOR TechPreview. 

Introduction THOR TechPreview

Since its early days, THOR has always been focused on stability and detection rate. With the early module and feature set, we never had to make a compromise. 

However, during the last 1-2 years, we had to make some decisions on the integration of new features and their default state in favor of stability. These decisions include e.g. the process dump feature, the PE-Sieve integration and Sigma scanning. 

Detection and stability have become two competing goals. We do not want to make these hard decisions anymore and leave them to you. You decide, based on your use case, if you want to use the version with newest features and detection capabilities or the one with a maximum of stability. 

With THOR version 10.6 we introduce a version named THOR TechPreview, which includes the newest features, refactored modules and new modes of operation. 

THOR TechPreview is a special THOR version that contains the newest modules and great detection features, which have not yet been tested on thousands of systems.

Florian Roth

Head of Research

The first release of THOR TechPreview will be version 10.6.
The standard version of THOR remains version 10.5 until the refactored and new features of the TechPreview have been proved to be stable. The expected release cycles of new version of THOR Tech Preview will be once a month, while new minor versions of THOR will be released only twice a year. Both versions receive bugfix updates and use the same signature set. 

ASGARD and THOR TechPreview

The current ASGARD Management Centers continue to use the standard THOR versions. The next minor release ASGARD 2.6, which is planned for October 2020, includes the option to use the TechPreview variant.  

Recommended Use Cases

The TechPreview version is recommended for all use cases in which detection capabilities have higher priority than stability. We would e.g. always recommend the TechPreview for image scans in a forensics labs.

Internal Testing

THOR TechPreview is not an untested version. It still goes through our internal testing on almost a hundred different test systems in 4 different test configurations.

Getting Started

Customers can download the new THOR TechPreview version from the download section in the customer portal once it gets released. Thor-Util version 1.11+ also supports the TechPreview download. We’ve planned the release for September 8.

Use THOR in CrowdStrike Falcon Real Time Response

One of our customers has successfully deployed THOR using CrowdStrike’s Falcon Real Time Response.

Falcon’s Real Time Response provides a remote shell that is very similar to Microsoft Defenders ATP’s Live Response, which we’ve already combined with THOR Cloud (see this page). 

The most important feature that allows us to integrate THOR is the ability to upload binaries to a remote system and execute them.

The Real Time Response shell offers a set of commands to interact with the remote system.

We used “put” and “run” to upload and run THOR and “get” to download the scan results.

Since the “run” command doesn’t accept any command line flags, it comes in handy that THOR accepts all his command line flags with config files in YAML format.

You can simply put all command line flags that you’d like to use into `thor.yml`, which is the default config file that gets initialized automatically during startup.

You can find more information on the YAML config files in chapter 10 of THOR’s manual. 

 

We recommend editing the “thor.yml” before uploading the THOR package to the remote system using the “put” command. 

It is advisable to set a specific output path for the log file or use the “rebase-dir” flag to create all output files in a certain folder. Many users reduce THOR’s CPU load to a lower value to influence the remote system as little as possible.

...
rebase-dir: C:\Temp
cpulimit: 70

You can then expand the archive using “Expand-Archive thor.zip” (PowerShell) or upload a “7z.exe” on very old systems with old PowerShell versions that don’t support the “Expand-Archive” command. 

You have to configure a “Sensor Visibility Exclusion” for THOR for the complete host group. Exclusions are relative to the file system’s root, so the pattern should look like: 

\temp\thor\**\*.exe

After a successful scan run, you can download the results using the “get” command and remove the exclusion.

It is possible to integrate THOR into many endpoint solutions. If you’d like to check the integration into your solution, please contact us to get a trial version of THOR. We offer special discounts for customers that help us showcase interesting integrations. 

Sigma Scanning with THOR

Our compromise assessment scanner THOR is able to apply Sigma rules during the local Eventlog analysis. This can help any customer that has no central SIEM system or performs a live forensic analysis on a system group that does not report to central monitoring. 

By running THOR on these systems with activated Sigma feature, THOR becomes a kind of a distributed and portable SIEM.

Since the Sigma scan module isn’t active by default, we thought it a good idea to explain how to activate an use it in the best possible way. 

Open Source Rule Set

By default THOR uses the open-source Sigma rule set with more than 500+ rules provided by the Sigma project on their Github page

Since our head of research is also one of the project maintainers, it was reasonable to combine the detection capabilities of Sigma with THOR’s scanning functionality on the endpoint. 

We comply with Sigma’s DRL (Detection Rule License) by including the rule authors in the event data produced by these rules.  

Custom Sigma Rules

You can easily add you own Sigma rules by placing them in the “./custom-signatures/sigma” sub folder.

THOR’s Sigma Config

The THOR default configuration for Sigma can be found in the Sigma repository. 

This configuration shows you, which Windows Eventlogs and Linux/Unix log files get analyzed by the Sigma module in THOR.

 

Sigma Scanning

To activate the Sigma module simply use the “–sigma” flag (or “sigma: True” in a YML config file).

You can start a THOR scan that analyzes the local Eventlog and activates the Sigma feature with:

thor64.exe -a Eventlog --sigma

To run a Sigma scan on a single Eventlog e.g. Sysmon’s log, use the “-n” flag.

thor64.exe -a Eventlog --sigma -n "Microsoft-Windows-Sysmon/Operational"

To include the Sigma feature in a standard THOR scan and check only the last 3 days of the Windows Eventlogs to reduce the scan duration, use:

thor64.exe --sigma -lookback 3

Sigma Matches

Once a Sigma rule matches on a log entry, you’ll see it listed in one of the REASON’s that lead to the classification of an event. 

The following example shows the detection of a China Chopper (Caidao) ASP web shell. That web shell has been detected by multiple Sigma rules. 

  1. Webshell Detection With Command Line Keywords 
  2. Shells Spawned by Web Servers
  3. Whoami Execution

Getting Started

Since this feature isn’t available in THOR Lite, please contact us via the “Get Started” button in the upper right corner and get a free trial voucher. Most customers that use THOR with Sigma choose one of our THOR license packs, especially the SOC Toolkit Pack, which was geared to the needs of today’s SOC teams. 

Upcoming Changes in THOR v10.5

PE Sieve Integration

With the integration of @hasharezade‘s PE Sieve project THOR is able to detect and report a variety of process implants like replaced or injected portable executables (process hollowing), injected shellcodes, hooks and in-memory patches.

Naturally, since @hasharezade’s project is an open source project, this feature will also be available in THOR Lite, the free version of THOR. 

Process Dumps

THOR v10.5 creates a process dump of any process that is considered suspicious or malicious. 

This process dump can then be analyzed with standard tools later to examine the findings. Use the flag “–dump-procs” to activate this feature.

To prevent excessive disk space usage, new dumps overwrite old dumps of the same process. Also, THOR stores the dumps in a compressed form and will not generate dumps if less than 5 GB disk space is available. 

Global Module Lookback

The current “–lookback” option allows you to restrict the Eventlog and log file scan to a given amount of days. E.g. by using “–lookback 3” you instruct THOR to check only the log entries that have been created in the last 3 days.

We’ve extended this feature to include all applicable modules, including “FileScan”, “Registry”, “Services”, “Registry Hives” and “EVTX Scan”. By setting the flags “–global-lookback –lookback 2” you instruct THOR to scan only elements that have been created or modified during the last 2 days. This reduces the scan duration significantly.

On our test systems, we were able to reduce the scan duration of a full filesystem scan and a lookback of three days to less than 4 minutes.

LNK File Parser

The link file parser module processes .lnk files, extracts relevant data and gathers more information on the linked contents. It also applies the anomaly detection methods to its contents to allow the detection of unknown threats. 

 

More Changes

  • Default output files include a timestamp and not just the date
  • Outputs include non-ASCII characters in a hex encoded form (use –ascii to revert to ASCII only output)
  • THOR DBs “–resume” feature is deactivated by default and has to be manually activated using “–resume” due to significant performance implications caused by updating resume states in THOR DB 
  • New –portal* flags allow the licenses generation at runtime using our Netxron portal API
  • New –yara-max-strings-per-rule flag limits the output of matching strings
  • New –nofserrors flag suppresses all error messages regarding access permissions
  • New –scanid-prefix allows users to set a custom prefix to allow the identification of group of scans
  • New –print-signatures flag lists names and meta data of all included YARA and Sigma rules

THOR 8 and SPARK End-of-Support

With this blog post we would like to inform you that our End-of-Life (EOL) products THOR 8 and SPARK will reach their End-if-Service-Life (EoSL) on 31th of October 2020.

From this day onwards, product and signature updates will not be available anymore.

Please consider upgrading to THOR 10, which is available for all relevant platforms and architectures.

THOR Lite – Free YARA and IOC Scanner

THOR Lite – Free YARA and IOC Scanner

We are proud to announce the release of THOR Lite. It is a trimmed-down version of THOR v10 with a reduced feature set and the open source signature base used in LOKI and the now obsolete scanner SPARK Core.

It uses the completely rewritten code base of THOR v10 “Fusion” and is therefore faster, more thorough and stable than SPARK.

 

As you can see in the table below, we’ve come a long way since 2012. We’ve phased out the old THOR version based on Python and SPARK in 2019. Today, we’re replacing the community version of SPARK named SPARK Core with a community version of THOR v10, named THOR Lite. 

There are two main differences between THOR Lite and THOR: 

  1. Reduced feature set
  2. Open source signature base

Apart from that, you’ll get a fully maintained and tested scanner pre-compiled for the Windows, Linux and macOS platform. A limited support is available via the issues section on the github page for auxiliary scripts.

Upgrading from SPARK Core

There is no direct upgrade path from SPARK Core, since SPARK Core and THOR Lite are completely different products and have different upgrade paths.

New users have to subscribe to the newsletter to get download links and a free license. You can subscribe and download THOR Lite using the link on the product page

SPARK Core users that already have a valid license can use the following download links to download THOR Lite:

THOR Lite for Windows
THOR Lite for Linux
THOR Lite for macOS

Important: These download packages do not include a license. You need to subscribe on the product page to receive a valid license OR use your existing SPARK Core license with THOR Lite. 

Issues

Please report problems in the issues section of THOR Lite’s helper scripts github page

Automated Citrix Netscaler Forensic Analysis with THOR

In this blog post I’d like to outline an idea on how to perform an automated compromise assessment on Citrix Netscaler / Citrix ADC appliances.

 

I you haven’t heard about that vulnerability yet, you should read my tweets over the past weekend or try to get a full picture with the help of this Reddit

This blog post is about a subsequent forensic analysis, not detection, not protection and contains no marketing blah-blah.

The described solution should work with the free / open scanners as well. They just don’t have that huge signature set as it is included in THOR.  

 

The reason for this is that although you’ve implemented the workaround on Monday 13th of January, you can’t trust your gateway anymore.

The vulnerability is known since December 17th and the public exploit code available since Friday the 10th of January. Nation state actors have most likely used Christmas holiday season to analyze Netscaler systems in their lab and produce an exploit, just like many offensive researchers did. Since Friday the 10th, even script kiddies can exploit this vulnerbility.

Strictly speaking, you can’t trust these gateways anymore. 

But how can we regain trust? 

A. Wait for a patched version and reinstall the gateways from the scratch

B. Perform a manual forensic analysis 

C. Perform an automated forensic analysis 

Automated Analysis with THOR

We don’t have and plan a THOR (SPARK Core or LOKI) version for FreeBSD 8.4, but we could try to mount the remote filesystem of the Netscaler gateway with SSHFS. 

DigitalOcean has a good tutorial on how to enable it on your workstation.  

mkdir ~/netscaler-mount
sudo sshfs -o allow_other,defer_permissions nsroot@citrix-gateway.nextron:/netscaler ~/netscaler-mount/

We can then use one of our scanners on that mounted volume using the `–fsonly` parameter (which activates forensic lab mode and scans directories regardless of their type). 

./thor64 --fsonly -p ~/netscaler-mount

With this method, we can detect dropped malicious templates. 

You can use the public YARA rule, that I’Ve published in LOKI’s and SPARK Core’s signature base. (note that a SPARK Core version that includes that rule hasn’t been released yet) 

Important note: this only works for templates that have been dropped since the last reboot. Users reported that dropped malicious templates get removed during a reboot. As the workaround described by Citrix requires a reboot, scanning with this single YARA rule alone won’t be a reliable method.

I recommend scanning with the full signatures set in all scanners, as they also include rules for web shells and other suspicious content in files. I’ll also extend the YARA rules on github and add further indicators. 

Additional Indicators

We can use the following YARA rule to check for suspicious strings in log files. But we don’t want to use that rule on any Linux/FreeBSD system. Therefore I won’t integrate that YARA rule in the public signature base. You can just copy it from this blog post and place it in the ‘custom-signatures/yara’ sub folder (‘/signatures’ for LOKI). 

This rule for the forensic artefacts is based on remarks made in this article by TrustedSec and shared as gist. 

First, we mount the ‘/var/log’ directory from our remote Netscaler system. 

mkdir ~/netscaler-logs
sudo sshfs -o allow_other,defer_permissions nsroot@citrix-gateway.nextron:/var/log ~/netscaler-logs/
We then start a scan on the mounted log directory.

As you can see, the scanner noticed a an issued ‘whoami’ command in the ‘bash.log’ and ‘notice.log’ files, which is a good starting point for further investigations. 

Final Thoughts

I think we’ll develop more rules for post-exploitation payloads over the coming days. Stay tuned and follow my twitter account for updates.