Spotlight: Threat Hunting YARA Rule Example

With this post, we would like to demonstration the YARA rule creation process for the so-called “threat hunting” rule category that we use in VALHALLA.

We noticed that many interested parties thought that “threat hunting” YARA rules are just rules with lower scores indicating a lower certainty. But in fact, they’re our most successful rules. The reason behind this is that they focus on anomalies as they appear in obfuscated samples and we’re not just talking about different forms of encoding.

Looking at the current table named “Successful YARA Rules in Set” on the VALHALLA start page, you’ll see many rule names that start with “SUSP_” for “suspicious”. 

These rules don’t match on a specific threat / malware but detect

  • certain methods (evasion, exploitation, side-loading, LOLBASs, LOLBINs)
  • casing anomalies (like cMd.ExE)
  • many forms of suspicious encodings
  • reversed strings
  • suspicious parameter combinations (e.g. certutil -decode)
  • suspicious packer / PE information combinations (like AutoIt executables from Microsoft)
  • and much more

So, these rules cannot be used for classification but they’re certainly priceless to detect new unknown threats.

Genesis of a New Threat Hunting YARA Rule

Processing different samples from various threat groups we often notice patterns in malicious code that looks as if it could be used for a generic “threat hunting” rule. 

The MuddyWater sample (8f0c6a09d1fca3d0002d3047733b50fe5153a33436d576c5020f0a21761242f1) contains the following base64 encoded block. 

While looking at this code block you can see repeating patterns even before decoding it just by scrolling over it. 

A good analysts asks himself “could this pattern serve as a signature?”.

To answer the question he decodes the base64 encoded chunk and gets a script with the following content:

He’ll notice a block of hex encoded values in a list. It seems that the obfuscation of the lower level (hex) can be detected in the upper layer (base64). So, by using a combination of these two forms of obfuscation, the attackers provide us a pretty specific pattern to detect a malicious – or rather – a highly suspicious code.

Next we try to figure out the exact usable patterns and put them to the test with different offsets. We use simple regular expressions in CyberChef to highlight matches. 

For our YARA rules, we don’t want to use regular expressions but byte patterns with place holders. Even for this task we can make use of CyberChef. 

The output can be used in a YARA rule that looks like this:

rule SUSP_Base64_Encoded_Hex_Encoded_Code {
   meta:
      author = "Florian Roth"
      description = "Detects hex encoded code that has been base64 encoded"
      date = "2019-04-29"
      score = 65
      reference = "Internal Research"
   strings:
      $x1 = { 78 34 4e ?? ?? 63 65 44 ?? ?? 58 48 67 }
      $x2 = { 63 45 44 ?? ?? 58 48 67 ?? ?? ?? 78 34 4e }
   condition:
      1 of them
}

To us it is not surprising that a test with the rule returned a lot of samples with low or no AV detection at all. We tested the hash list of the samples retrieved from a Virustotal Retrohunt with Munin and got the following results: 

As you can see, it’s not possible to verify the results based on the AV detection ratio. However, it’s a good sign that other threat hunting rules or even rules for known webshells from our ruleset match on these samples as well. We typically evaluate the false positive rate of this type of rules with the help of the file names (e.g. c99.php, virus.txt, *_codexgigas, Virusshare_*) and some spot checks.

You’ll also note that the rule matches many different content types – emails (.eml), executables, web shells, scripts. That’s one of the reasons why we love these rules so much.

The second screenshot contains some reassuring matches of the customized older version of the LaZagne credential dumper used by MuddyWater and apparently also encoded in the described form. (b8e97c96aa18916c15eea5c78d5a20b966aa45f332a5ea4d9ac2c87ebe5adff6)

You can find a full munin result file of the retrohunt matches here.

The YARA rule will be pushed to the signature-base that we provide for the community and will also be available in a streamlined form in the VALHALLA demo feed very soon. 

I hope you liked it.

For more information like this, please subscribe to the newsletter or follow us on twitter: @thor_scanner 

Remarks on Products and Services

We constantly improve the quality of our products and services, add features and create new bundles. Follow ups with our customers showed that not all of these changes reach their attention. They are often surprised and excited to hear about these features, free tools or license bundles. This is a list of the changes that often go unnoticed.

1. Scanner licenses allow you to run THOR and SPARK

Customers who have bought scanner licenses to scan Servers and Workstations, be it an Enterprise or Host-based license, can use both our scanners THOR and SPARK. If you have bought an Enterprise license for THOR in the past, you are also allowed to download and use this license with SPARK on Linux or macOS endpoints. Download SPARK from the “Downloads” section in the customer portal.

2. SPARK applies Sigma rules on endpoints

Customers are often surprised to hear that. We have customers that are not allowed to collect logs on endpoints due to legal restrictions but they are able to start executables like our scanner SPARK on endpoints, which is able to apply Sigma rules on local Eventlogs. This way, they can apply detection rules on systems that they do not actively monitor. The blog post – SPARK uses Sigma Rules in Eventlog Scan has more information on that feature.

3. Some contracts include a free ASGARD Management Center and Analysis Cockpit

Enterprise customers with a valid support contract for our scanners are eligible for a free ASGARD Management Center, which is able to control and schedule scans on up to 10.000 end points and an Analysis Cockpit, that allows you to ingest and analyze the logs of up to 50.000 end points in a comfortable manner.

Customers with more than 10.000 licensed endpoints are eligible for additional ASGARD Management Centers and a MASTER ASGARD, which is the central management for multiple ASGARD systems. 

See the Video Tutorials page to learn how these systems can help you with you daily management and analysis tasks. If you are interested in these systems and your account status, please contact your account manager.

4. YARA signature overview in Customer Portal

The customer portal contains a CSV with information on all 9973 YARA rules in our signature set (as of 16.02.2019). This way you can verify if a certain threat group or campaign is covered by our rules or not. You can find that CSV in the “Software Information” section together with binary hashes and an update server status on all our products.

5. Kibana can be installed together with ASGARD Analysis Cockpit

We do not support this coexistence but prepared everything to make it easier for you to install Kibana next to our own interface to analyze the collected log data. The analysis cockpit manual has a chapter that explains how to install Kibana on an Analysis Cockpit. The Analysis Cockpit wraps Kibana and serves access as reverse proxy providing a common authentication. You can manage the service from within the “Settings” section of the Analysis Cockpit.

Feedback

If you have any feedback, questions on these features, please let us know.

MASTER ASGARD – One ASGARD to Rule Them All

We are glad to announce our new product MASTER ASGARD, a central control for a set of ASGARD systems.

MASTER ASGARD is designed to control multiple instances of ASGARD, which itself supports up to 10,000 endpoint agents. Using MASTER ASGARD you are able to control more than 100,000 end points from a single central location. 

This control includes:

  • Run distributed THOR and SPARK scans
  • Schedule distributed THOR and SPARK scans
  • Manage and distribute IOCs
  • Collect files and memory from Windows and Linux end systems

 

 

Here are some screenshots:

Management of multiple ASGARDs

Evidence Collection

Distributed Scans

MASTER ASGARD will be available for BETA program customers at the end of February and to the full customer base in May 2019.  

Antivirus Event Analysis Cheat Sheet v1.7

We’ve just released an updated version of our Antivirus Event Analysis cheat sheet. You can download version 1.7 here.

The major changes are:

  • Updated AV signature lists
  • Split AV signature cells into two columns to save space
  • Fixed and added some directory names
  • Extended file extension list
  • Google filename search
  • More Virustotal checks

ASGARD v1.7.2 with File and Memory Collection

Our brand new ASGARD 1.7 comes with a shiny new feature: Evidence Collection

The evidence collection feature allows you to collect files or main memory from connected end systems.

The memory and file collection tasks provide a throttling option to reduce the upload speed of the dump files in order to save bandwidth and avoid higher response times of servers or workstations. 

The file collection feature allows you to get a single file, the contents of a folder with or without its sub directories. You can set size limits for each file and the whole archive.

The “Evidence Collection” tab lists all active and completed tasks. 

A log shows you the details of all the collection tasks.

ASGARD version 1.7.2 has been released today and can be upgraded via the “Updates” section. 

Please note that the memory collection on Linux endpoints is integrated but not fully supported. 

 

50 Shades of YARA

A long time ago I’ve noticed that there is no single best YARA rule for a given sample, but different best solutions depending on the user’s requirements and use case. I noticed that I often create 2 to 3 YARA rules for a single sample that I process, while each of them serves a different purpose.

In this blog post, I’d like to describe the three most common rule types.

In the following example I’ll use the malware sample with hash 7415ac9d4dac5cb5051bc0e0abff69fbca4967c7 (VirusBay, Hybrid-Analysis)

While looking at the strings extracted by yarGen, you’ll notice that it contains a lot of interesting strings. In my past tutorials (1, 2, 3) I’ve always distinguished between “Highly Specific” and “Suspicious” strings (see Part 3 of the blog post series). Today I’d like to show you a more purpose oriented approach. 

The following screenshots shows what types of strings I see while looking at these strings:

The strings that are marked with yellow look very specific. I’d use them as “Highly Specific” strings ($x*) of which only a single one is required to trigger the rule: 1 of ($x*)

The strings marked green will be used in combination with other green strings. A reasonable set of these strings is required to trigger the rule: $u1 and 1 of ($f*)

The strings marked with red color could serve in a rule that tracks the C2 addresses used by this sample and the strings marked blue could be used for a generic detection of malicious samples that can be completely unrelated.

The different rule categories are:

  • Regular Rules: Detect a certain malware or malware family 
  • Threat Intel Tracking Rules: Detect specific indicators that relate to a certain actor
  • Method Detection Rules: Detect methods or anomalies 

The following table describes these three different types of rules and gives some string examples. 

Regular Rules

In the case of the “Regular Rules” I distinguish between two different flavors: 

  • Threat Detection Rules
  • Threat Hunting Rules

The difference between these flavors is based on a different level of strictness in the conditions and not on the different selection of strings. While a “threat detection” rule may require “6 of them”, a “threat hunting” rule may be satisfied with “3 of them”, accepting some false positives. 

The reason why someone distinguishes between “threat detection” and “threat hunting” rules is that the response to matches can be very different. Antivirus solutions that respond to matches with “delete” or “disinfect” reactions do not accept false positives and avoid false positives by any means.

In “threat hunting” use cases which include direct destructive reactions to signatures matches are rare. Typically analysts investigate such an event, classify and react to it manually. In “threat hunting” scenarios analysts try to avoid “false negatives” by all means. 

(Source: Chris Gerritz @gerritzc

Threat Intel Tracking

In threat intel, we can use YARA rules to track the activity of certain actors in cases in which there are certain characteristics or keywords that persist over longer periods and campaigns. 

A very convenient form of tracking without having access to the telemetry data of OS and AV vendors is offered in the form of YARA match notification services as provided by VirusTotal or ReversingLabs

Method Detection Rules

During the past year I focussed on the last rule type “Method Detection” whenever I had the opportunity as it allows me to provide very generic rules that produce amazing results with a minimum of false positives.

However, those rule matches lack a reference like a malware name or an adversary group that used the detected method in their samples. Here is an example with one of the few public YARA rules published in the “signature-base” repository:

Sample: fc18bc1c2891b18bfe644e93c60a2822ad367a697bebc8c527bc9f14dad61db5 

The comment tab shows a match with generic rule “SUSP_LNK_SuspiciousCommands” . No reference is given. The Antivirus detection ratio is low. 

You can find more matches with this rule on Virustotal using the search function – URL: https://www.virustotal.com/#/search/lnk 

Conclusion

These are the reasons why the analysis of a single sample often results in 2-3 different YARA rules.

Using this method the coverage is exceptionally good as the set of rules covers specific samples of the same family and the different malware families the use the same methods.

YARA Rule Sets and Rule Feed

As previously announced our YARA rule packs and feeds will be available in March/April 2019. We’ve put a lot of effort into a internal system named “Mjolnir” that parses, normalizes, filters, tags and automatically modifies our rule base, which contains more than 9000 YARA rules. 

This system will now fill a database of tagged YARA rules – the basis of our new YARA services. 

The services will be divided into two categories:

  • YARA Rule Set
  • YARA Rule Feed

YARA Rule Set

The YARA rule set consist of more than 7000 YARA rules of different categories that are used in our scanners.

Some of our rules use extensions (external variables) that are only usable in our scanner products. These rules, experimental, third party and other classified rules will not be part of the purchasable rule set. 

YARA Rule Feed 

The YARA rule feed is a subscription on our rules. The feed always contains the rules of the last 90 days, which is between 250-400 YARA rules. 

Rule Samples

The quality of the rules in the rule set are comparable to the rules in our public “signature-base” repository. 

Some good examples for the different rule categories are:

Quality and Focus

The rules are tested against a data set of more than 350 TB of goodware. The goodware file repository consists of Windows OS files, several full Linux distributions and a big collection of commercial and free software. 

However, false positives are always possible. We do not recommend any destructive action on a signature match, like delete or blocking.

The main focus of our rules are:

  • Threat Hunting
  • Classification
  • Anomaly Detection
  • Compromise Assessment 

Subscribe to our Early Access Mailing List

THOR 8.53 Feature: Diff Mode

With the upcoming version 8.53 of THOR, we’re testing a new feature called “Difference” or “Diff” mode (–diff).

The idea behind “Diff” mode is that a scan could be much faster, if it would only consider elements that have been created or changed since the last scan on that system. We can apply this principle to various modules and increase scan speed massively.

Diff mode is currently supported in the long running modules

  • Filesystem – files with MAC timestamps older than the last scan (start) will be skipped
  • Registry – registry keys with last modification dates older than the last scan (start) will be skipped
  • Eventlog – runs until it reaches eventlog entries with timestamps older than the last scan (start)

Diff mode requires the use of THOR DB, which is the default but could have been disabled with “–nothordb”. This is necessary to determine information from the last scan, e.g. “when did it start” but also “which modules were used in the last scan”.

The main advantage is an incredible fast scan. Our tests showed that scans in “Diff” mode complete within 5 and 15 minutes. In “Diff” mode, the longest running module is “ProcessCheck” with run times between 3 and 6 minutes.

The main disadvantage of “Diff” mode is the inability to detect Timestomping attacks, in which attackers or malware changes the timestamps of files and other elements.

ASGARD Analysis Cockpit 2.2 Feature Overview

Later this month the new version 2.2 of ASGARD Analysis Cockpit will be released. These are the most important new features.

The Optimize Button

The new “Optimize” button allows you to add all unassigned log lines to existing cases with matching filters. It is possible that you miss some events when creating a new case, either by the wrong selection or due to the fact that new log lines can arrive at any time via SYSLOG or log file import in the background.

Now it is possible to add all unassigned log lines to previously created cases by using the “Optimize” button.   

It will not remove previously assigned log lines from existing cases. It just helps you to clear up the base lining section by removing events that are related to existing cases but haven’t been assigned to these cases yet.

You can later review all automatic assignments in the “Automatic Event Assignment” protocol.

Notification Settings

The new “notification” settings allow you to create notification rules for the following type of events:

  1. Log lines that are automatically assigned to an existing case
  2. Status changes of cases

The current supported actions are:

  1. Syslog Forwarding
  2. Email Notification

This allows you to define flexible rules for many different events. You may e.g. create a rule that sends an email notification whenever a new “Incident” case is opened. 

You could also forward all incoming log lines that are automatically assigned to a case of type “Incident” to your remote SIEM system. (each syslog message will be extended by two new fields: case_type and case_id)

An email for a opened “Incident” case will then look like this:

The attachments of these emails contain the included log lines (text) and a JSON with all case information in machine readable form.

File Importer

The File Importer status view has been improved so that it shows the number of total files in queue and the number of processed files.

Improved Reporting

The new improved reporting allows you to generate reports not only for a given period of time (e.g. last month) but custom queries on the ElasticSearch database. E.g. you can generate report for the scans on your SuSE linux systems only. 

The reports contain more panels and information on the data set. 

The data from all reports can be downloaded as JSON file. 

Upgrade to 2.2

The upgrade will be visible in the “Updates” section of your Analysis Cockpit once it is released. See the change.log notes for a full list of changes. 

 

ASGARD Management Center Feature: Scanner Package Download Links

ASGARD features a new section since the last upgrade that you may have missed. It’s called “Downloads” and contains a section in which you can configure a download link for scanner packages.

In previous versions, the scanners have been accessible right from the login screen without any authentication, just like the GRR agents, which are still accessible in that way.

We’ve removed these unauthenticated scanner downloads and created that new “Downloads” section, which can be used by authenticated users in different ways.

While selecting different options in the form, the download link changes.

After you have selected the correct scanner, operating system and target hostname (not FQDN), you can copy the download link and use it to retrieve a full scanner package with included license file for that host. These download links can be send to administrators or team members that don’t have access to ASGARD management center. Remember that the recipients of that link still need to be able to reach ASGARD’s web server port 443/tcp. 

If you don’t set a hostname in the “Target Hostname” field, the scanner package will not contain a license file. If you have an unlimited “Enterprise” license, you’ll have to provide it separately.  

Use Case 1 – Provide Download Links

You can generate download links for the different scanner packages without included license for yourself or the administration team. A valid license (e.g. “Enterprise” or “Incident Response”) has to be provided and placed in the program folder. You can also use “thor-util” to retrieve licenses for specific hostnames from an ASGARD server (see the “THOR_Util_Manual.pdf” in each scanners “./docs” folder for details)

Use Case 2 – Administrator Asked to Run a Scan

You can copy the final download link and send it to an administrator, which can use this link on one of the servers to retrieve a full scanner package with license and run a scan. 

Use Case 3 – Use the URL in Script

You can use the URL in Bash or PowerShell scripts to automate scan runs on systems without installed ASGARD agent. Replace the hostname value with the value of the current host on which the script runs to get a URL for scanner download package with a host-specific license.