Using THOR Lite to scan for indicators of Lazarus activity related to the 3CX compromise

Using THOR Lite to scan for indicators of Lazarus activity related to the 3CX compromise

Mar 31, 2023 | Alert, APT, Nextron, THOR, THOR Lite, Tutorial, YARA

On March 29, 2023 CrowdStrike detected malicious activity, originating from a legitimate, signed binary called 3CXDesktopApp. The binary is part of a softphone system developed by 3CX.
The observed malicious activity consisted of beaconing to infrastructure controlled by the actors, leading to the deployment of second-stage payloads and in a few cases direct on-keyboard activity from the attackers.

You can find more information on the threat in the following articles by CrowdStrike, Volexity and Huntress Labs:

CrowdStrike Report

The first report on the activity linking it to LABYRINTH CHOLLIMA aka Lazarus group.

Volexity Article

This article by Volexity lists a lot of indicators and reports on the final stage in form of the stealer ICONIC

Huntress Labs Article

Huntress Labs report on the activity including process patterns, rules and IOCs

After the compromise became first known, we began our own investigation and in the following few hours released a number of detection rules to our public repositories.

While having the detection in place is a great start, often times it’s not an easy task to assess the situation and make sure that no system in the network is affected by the threat.

One way to leverage these rules and quickly scan your own environment for free, is using THOR Lite scanner.

Enter THOR Lite

THOR Lite is the reduced version of our compromise assessment scanner THOR. It uses YARA rules and Indicators of Compromise (IOC) like hash values and file names to detect malicious activity. 

In this technical blog article, we’ll explore how to levreage THOR Lite to scan end systems for signs of malicious activity related to the 3CX compromise.

We’ll also discuss the various types of indicators that THOR Lite can detect, walk through the process of setting up and configuring the tool, and provide tips for interpreting the scan results.

By the end of this article, you should have a solid understanding of how to use THOR Lite to run a compromise assessments within your network.

Download THOR Lite

Visit the product page, subscribe to the newsletter to get the program package and the license file.

(note: we offer a special license file to 3CX customers that enables an additional module from the full version to extend the detection coverage even more)

You can download this special license here: (expires 30.04.2023) 

Download 3cx.lic

Email content:

Getting Started

After you’ve downloaded the program package as a ZIP archive, extract it and place the license file (.lic) in the program folder.

Double click on the “thor64-lite.exe” to run it without any flags or open a Windows command line as an administrator and navigate to the folder where you’ve extracted the program package.

You should then see the scan window that closes automatically when the scan is complete. Usually scans take between 1-4 hours, but there are some ways to speed up the scan.

 

Flags to Consider

--nosoft --nolowprio

If you’re scanning virtual machines or systems that are under a constant high load by other processes, it could be helpful to use the “–nosoft” and “–nolowprio” flags to let THOR run with the same process priority as any other regular process.

--lookback 150 --global-lookback

If you’re interested in scanning recently created files and log entries. These flags instruct THOR to only scan elements created or changed within the last 150 days (why 150?). It would ignore any file or eventlog entry older than that and thus scan a much smaller set of elements.

--cpulimit 30

To minimize the impact for the end user working on a system while it is getting scanned, you can reduce the CPU usage of the scanner to e.g. 30% to avoid them taking notice of the scan by reducing the overall load and fan noise.

Recommended CommandLine Flags For The 3CX Use Case

If a normal scan takes too long, we recommend the following command line flags in order to reduce the scan duration by restricting the scan to the changes of the last 150 days:

thor64-lite.exe --nolowprio --lookback 150 --global-lookback

In order to reduce the CPU usage and make it as imperceptible as possible to the end user working on the scanned systems use the following command:

thor64-lite.exe --lookback 150 --global-lookback --cpulimit 35

Update the Signatures

We’re constantly working on enhancing and updating the signatures related to the 3CX compromise. Updates are to be expected over the weekend and next week. To make sure THOR always works with the newest set of signatures use the following command:

thor-lite-util.exe upgrade

Interpreting the Scan Results

During the scan you’ll see several messages in green and blue colours. Warning and alert messages use a yellow or red color. But don’t worry when you notice a message of that color. Remember that THOR is a scanner that highlights malicious and suspicious elements for review by an administrator or forensic analyst. Not everything shown as a “warning” message has to be a real threat.

After the scan finishes, users can find an HTML report in the program folder that lists all findings. 

We recommend searching the HTML report for the “3CX” keyword and only review matches with the specific IOCs and YARA rules related to this activity.

THOR Lite is able to detect various forensic artefacts:

  • The installer files
  • The malicious binaries
  • The loaded malware in-memory
  • Process connections to known C2 addresses
  • Traces of activity in local log files

We’re also offering a special license (3cx.lic) to 3CX and their customers that will activate a special feature called “Sigma Scanning” in THOR Lite instances. This allows them to apply the Sigma rules mentioned below (and 1600+ more) on the event logs of a scanned end system.

A match with one of these Sigma rules would look like this: 

You can download this special license here: (expires 30.04.2023) 

Download 3cx.lic

Continuous Compromise Assessment: Enhancing Detection Capabilities to Mitigate High-Profile Cyber Attacks

One more time, we are all taken aback by yet another sudden high-profile compromise. Just like the Sunburst or HAFNIUM Attack, the 3CX compromise arose out of nowhere, putting companies of all kinds across the globe at risk. We may later discover that some organizations were exploited for months before the 3CX compromise was ultimately made public.

But does it truly have to come as such a surprise to everyone? Looking back at the Hafnium attack, Nextron discovered that many organizations had been breached by various attack groups, all of whom appeared to have used the proxy shell/proxy logon weakness. All attackers who expanded their breach brought their own toolset for persistence and post-exploitation. Nothing new so far.

However, what if we could automatically detect an attacker’s toolkit after it has been deployed? In this case, we could efficiently detect these breaches long before day zero simply by identifying secondary tools that appear magically on a system. Let’s assume we scan our systems weekly, searching for all kinds of Indicators of Compromise, known attacker tools, or traces of their methods. Then, even without knowing that the 3CX compromise exists, we would most likely be able to detect attacks that make use of it within a week. This would give us a heads up before bad things even begin to happen, shocking everyone.

This is precisely Nextron’s “Continuous Compromise Assessment” approach. With our orchestration platform ASGARD, we can conduct recurrent and automated compromise assessments using our full-featured Scanner THOR. Our first and initial scan represents what we call the baseline. We would analyze all events from the first scan and, starting with the next week, focus on any deviations from this baseline. In such a scenario, we would detect breaches based on secondary toolsets from one week to another. While we still would not detect the 0-day itself, the secondary toolset would show up very prominently as a deviation from the baseline.

There is not much effort required to gain a considerable amount of additional detection capabilities.

Signatures 

The following listings show all the signatures we’ve made public and used in THOR Lite to detect malicious activity

YARA (public)

SIGMA (public)

Potential Compromised 3CXDesktopApp Beaconing Activity – Proxy
UUID: 3c4b3bbf-36b4-470c-b6cf-f07e8b1c7e26

Potential Compromised 3CXDesktopApp ICO C2 File Download
UUID: 76bc1601-9546-4b75-9419-06e0e8d10651

Potential Compromised 3CXDesktopApp Beaconing Activity – DNS
UUID: bd03a0dc-5d93-49eb-b2e8-2dfd268600f8

Potential Compromised 3CXDesktopApp Beaconing Activity – Netcon
UUID: 51eecf75-d069-43c7-9ea2-63f75499edd4

Potential Suspicious Child Process Of 3CXDesktopApp
UUID: 63f3605b-979f-48c2-b7cc-7f90523fed88

Malicious DLL Load By Compromised 3CXDesktopApp
UUID: d0b65ad3-e945-435e-a7a9-438e62dd48e9

Potential Compromised 3CXDesktopApp Execution
UUID: 93bbde78-dc86-4e73-9ffc-ff8a384ca89c

Potential Compromised 3CXDesktopApp Update Activity
UUID: e7581747-1e44-4d4b-85a6-0db0b4a00f2a

IOCs

C2 IOCs
c2-iocs.txt @ signature-base

Filename IOCs
filename-iocs.txt @ signature-base

Hash IOCs
hash-iocs.txt @ signature-base

Full THOR Version

Keep in mind that THOR Lite is only a demo version of our full scanner with more than 27 detection modules and more than 20,000 YARA rues compared to the 5 modules and 2,500 rules used in THOR Lite.

You can find a full feature comparison here and a blog post that explains the differences in more detail here. 

 

Not All IOC Scanning Is the Same

Jun 12, 2016 | APT, Security Monitoring

In the recent months I had several talks with friends and coworkers about IOC scanning and how to integrate IOCs from threat intel feeds into our scanners or other products that our customers already use. People often tell me that EDR or client management product X already does IOC scanning and that we don’t need to check for these indicators a second time. Especially when it comes to network wide sweeps for traces of activity due to an ongoing incident I recommend scanning a second time with one of our scanners or a tool of similar quality.
This blog post explains why.
People usually spend a fair amount of time on selecting threat intel feeds and interesting indicators for their scans. However when it comes to the actual application of these indicators they seem to be satisfied with the simplest form of checks.
Especially when we look at C2 or Filename IOCs I can easily explain the difference between the “compulsory” and “freestyle” methods of IOC scanning.
IOC scanning automation
A plain “compulsory” filename IOC check would walk the disk or query a database looking for a certain filename, right?
However if you think about it for a second and ask yourself “where else could we check for that filename?” you’ll realize that the following elements could also contain the malicious filename:

  • Eventlog entries (e.g. process starts, service installs with image path, access failures …)
  • Log files (local Antivirus log file, access to file in web root > web server access log, backup errors, PowerShell history …)
  • Registry (recently opened files, shell bags, service image path, other caches …)
  • MFT (deleted entry)
  • Archive content (packed in ZIP file)
  • WMI (scripts – e.g. see this PoC by Matt Graeber)
  • Crash dumps
  • Windows Error Report (WER – file names and content)
  • Free disk space (filename as content of batch files or other scripts, scheduled tasks …)

Actually we often see that during lateral movement attackers access systems, run their tools remotely, copy the output, delete the output files and leave no file system traces behind. We use the locations that I mentioned above and others to detect them using their tools although all the files have been removed from disk. That’s the “freestyle” method.
The same counts for the C2 IOCs. The “compulsory” plain method would check the system’s network connections. The “freestyle” method also includes checking for these C2 IOCs in the following locations:

  • Process memory (C2 strings loaded and decrypted in process memory)
  • Log files (web server access logs, Windows firewall log file, AV module log file …)
  • Hosts file
  • Files (in backdoor config files on disk)
  • Registry (hard coded C2 server in registry key)

I am sure that digital forensics experts would come up with other fruitful locations. It is just sad to see those great indicators feed into tools that do “IOC scanning” only to get another check mark in a product comparison table – aka the “compulsory” way.

If all you have is a hammer, everything looks like a nail.

So – the next time when someone tells you that their tool checks for IOCs on the endpoint, your question should be “How and where do you check for these IOCs?”.

How to Fall Victim to Advanced Persistent Threats

May 4, 2016 | APT, Security Monitoring, Tutorial

During the last four years, I was engaged on incident response teams for several large advanced persistent threat (APT) cases involving different German corporations. In this time, we have developed methods and tools to detect compromised systems, while also planning and performing remediation. During the course of these investigations, I noticed that certain circumstances supported the chance that a corporation would fall victim to advanced persistent threats. In this article I want to focus on favorable preconditions that nearly ensure a successful APT attack.
The recommendations in this article were discussed and extended with the help of red team leaders and fellow CERT team members. This is an article inspired by a book named “Anleitung zum Herzinfarkt” by Bernhard Ludwig. This book provides serious guidance on how to increase the risk of dying from heart attack in someone’s early years. I found that this principle of a ‘humorous reversed guide’ could be useful to describe the typical pitfalls and mistakes that we regard as crucial for the development of advanced persistent threats with the aim to help organizations, which have not been hit by this type of attacks so far.

Don’t Create Network Segments

At first, strictly avoid placing systems in different network segments. Instead – keep it simple. Let standard Windows workstations, admin workstations, server systems, print servers, industrial control systems, backup servers, network management systems, monitoring servers, terminal servers, mobile management servers, development systems, IP cameras, house automation, and SIP telephones be in a single huge network in order to avoid firewalling issues.
That principle applies also to subsidiaries and affiliates. After the acquisition of other companies, don’t waste time asking them for compliance with your security policies. Save time and connect their network directly with your backbone! Make sure to allow any type of connection and create domain trusts as fast as possible to enable cross-domain resource access to your data center in Berlin from your user workstations in Brazil, as well as the call center in Bangladesh.
If you are required to create network zones for some reason, interconnect these zones with mutual Windows domain trust relationships. (see Titanic as an example)
why-titanic-sank
In order to maintain an environment of mutual trust and respect, never monitor the gateways between segments for port scanning, network sweeps, or other types of suspicious network activity. Also, please do not commit the new business unit to provide technical support in cases of suspicious activity and incident response.

Don’t Limit Privileged Account Usage

You trust your administrator, don’t you? So, let administrators use their privileged accounts to surf the web and read email while they are connected via SSH and RDP to their domain controllers and Internet facing servers in DMZ networks. Do not use hardened and intensely monitored jump servers from special admin workstations to manage your most important server networks.
Don’t waste precious resources creating administrative roles. If someone needs to install a text editor on a server, add him to the Domain Administrators group and make sure that no one ever figures out why you did this.
Do not use Microsoft’s solution called LAPS to ensure that all computers have different, complex, local administrator passwords or Privileged Admin Workstations (PAW) to provide a dedicated operating system for sensitive tasks. This would give attackers an unnecessarily hard time finding highly privileged credentials and fewer opportunities to use them.

Don’t Collect and Analyze Useful Log Data in a Central Location

Attackers tend to make themselves familiar with the environment and therefore cause strange peaks in log volumes that could be easily detected by an over-attentive employee – so, don’t log security events. If you do log events for any reason, make sure that all of your systems keep their logs in a local log file and do not transmit them to a central SIEM system.
IMG_6213
If you have to use a SIEM system, use Active Directory for authentication, so that attackers can find all the fine log data in a central place. If you have any problems on a server, such as with disk space due to the fact that you configured your golden image for Windows servers to use only 20GB of space on drive ‘C:’, don’t hesitate to reduce the log file size to a minimum. That may cause the log to overwrite itself every 30 minutes, but hey, each line that is overwritten can help prevent successful detection.
On Linux systems configure your log file rotation to keep 7 days, not more. Your IT staff typically needs a couple of days investigate the system owner and have some meaningful discussion with your data protection officer.
If you have to comply with certain policies that require the collection of log data, you still can do a lot to make sure that breaches remain undetected for months. Here is our Top 10:

  1. Only log and report high amounts of failed logins, because attackers tend to use valid credentials after taking the first hop
  2. Disregard all antivirus events that have the status “deleted” or “moved to quarantine”. They are gone, so they won’t trouble you.
  3. Do not collect the logs of client workstations to avoid detecting zero-day attacks. Detecting them would cause pressure to take action.
  4. Do not search for anomalies in your log files because hacker activities generate some very special or completely new log types. If you look for them you could find them – so again, don’t do that.
  5. Do only collect the log files at system level and disregard the logs of applications running higher up.
  6. Do not spend time understanding your organization’s structure, logging completeness, and logging behavior of your assets. It’s usually pretty complex and the less you know the better.
  7. Use the default logging configuration on your systems. Maybe you miss the most relevant event types, but if they really were relevant, wouldn’t they be in the default anyway?
  8. Monitor and report attacks on your Internet-facing firewall to tie up valuable resources with useless pie charts.
  9. Do not collect the logs of SysInternals Sysmon, AppLocker, Windows Defender, or Microsoft EMET. The protection provided from their use, especially when combined, is far too effective.
  10. Let the data protection officers and workers’ council decide on what you’re allowed to log and analyze.

Use your Active Directory for All Types of Authentication

Centralization is good. It saves resources and simplifies user administration. Use Active Directory authentication for everything: the logins to your proxy servers, network devices, online certificate authorities, virtual machine consoles, administrative jump hosts, security monitoring servers, VPN servers, SIEM system, and last but not least – backup servers.
This ensures that attackers take over the complete infrastructure after compromising a single outdated member server that has domain admin logon sessions. Remediation becomes much more exciting when attackers have access to proxies, DNS servers, and mail gateways all at once!

Don’t Regard the Client Workstations as the First Line of Defense

Since corporate workstations operate deep in the internal network, which are well protected by several firewalls and proxy servers, consider them to be far outside of the danger zone. Don’t audit them as frequently as you do with DMZ servers. There are so many more clients than servers that it is certainly not efficient to scan them all. Also, random samples never give a valid result, so let’s just drop the whole thing.
Don’t patch workstation software like Microsoft Office, PDF viewers, JAVA and Flash plugins, media players, and archivers as soon as a patch is published. Don’t use exploit protection, like Microsoft EMET, in order to increase the risk of zero-day attacks and finally,  don’t deactivate content in document viewers, such as JavaScript in Acrobat Reader or VBA in Microsoft Office.
Simplify administration of user workstations by granting standard users local administrative rights.
Also allow workstations to access the Internet directly on any service port. Don’t use a proxy server and do not monitor dropped connections on typical Trojan back connect ports. Developers and administrators are highly skilled professionals that tend to download and install suspicious software from the shadiest websites. Remember: You trust them and if they think that they need that (suspicious) software than of course they do. To increase the probability of such events, block Sourceforge, Github, and the whole category “Software Downloads” on your Internet proxy server.

Don’t Mind Antivirus Alerts

As mentioned before, do not consider antivirus alerts that have the status “deleted”, “cleaned”, or “moved to quarantine”. A deeper analysis of these events could reveal a Trojan that had control of the system for weeks before the right signature returned a match on components of a hack tool set that attackers moved from server to server. Therefore, do only check for errors and unresolved operational issues.
Make sure that no one pays special attention to antivirus events that report “Hack Tools”, “Password Dumpers”, or “Scanners”. Tell everyone that this would cause too many false positives because system administrators need these tools to find their assets and regain access to them.
FullSizeRender
Also avoid rating antivirus events according to an evaluation method. That’s so bureaucratic.
To allow attackers the greatest possible leeway, create vast exclusion lists and don’t use special Antivirus functions like PUA scanning or application controls that block password dumpers from accessing the memory.

Handle Web Servers Like Other Server Types

Regard web servers like any other server type. Frequently patching the web server service is perfectly sufficient. Do not audit the applications running on that web server; if you have to due to corporate policies, do it once, print the report, and archive it.
Place the web servers behind reverse proxy servers and tell everyone that this will protect them. If you repeat that constantly everyone will believe it one day. Do not protect the web servers with costly Web Application Firewalls and don’t collect the logs of such a system for central attack detection and analysis.
Allow developers to access the management interfaces, like JMX or Tomcat Manager, from remote locations. Don’t log access to these applications and don’t ban source IPs on security violations. Tell everyone that each developer may access the servers anytime, from anywhere, to reduce operational risks.
If you have to run Apache or Tomcat on a server, choose Windows as the operating system. Do not use limited user accounts to run the web server services in order to maximize the impact of a successful attack. Last but not least: Avoid annoying security features like SELinux.

A Few Last Words

I know that it is hard to guarantee a successful APT attack and that all of these recommendations require a certain amount of stubbornness and resistance to advice; however, even if this advice does not guarantee falling victim to advanced persistent threats, chances increase exponentially the more of my advice you apply. Good luck – I’ll keep my fingers crossed.
If you have further ideas that you want to share, please comment on this article or contact me on Twitter @cyb3rops.

Credits

Many thanks to Stephan Kaiser for the idea, Julia Stolz and Jeff for major reviews and Matthias Kaiser (@matthias_kaiser), Daniel Sauder (@DanielX4v3r), Thomas Patzke (@blubbfiction), Claas Rettinghausen, Robert Haist (@SleuthKid), Alexander Döhne for their valuable feedback.

Synergetic Effects of Network and Host Based APT Detection

Oct 27, 2015 | APT, THOR

People often ask me if they still need our host based scanner THOR now that they have bought a network appliance that already checks all content that goes into and leaves their network. I normally answer that it is not a question of one solution versus another, but a combination of solutions to achieve the best possible result.
It is not difficult to understand that both solutions apply different detection techniques as they analyze different elements and provide different perspectives. It is difficult for an host based solution to detect Zero Day exploits, C2 back connects and malicious content in a network connection. But, in the same way it is impossible or difficult for a network based solution to detect system anomalies, malware-less backdoors, web shells and Eventlog or Registry based traces of hacking activities.
I collected and composed different aspects of advanced persistent threat protection in the following info graphic. The color (grey and aquamarin) indicates the coverage by the different solutions. The graphic is not based on research and may vary in specific cases. It is meant to roughly visualize the different perspectives and high coverage you achieve by combining both solutions.
Endpoint Attacker Detection

Endpoint APT Detection and Network APT Detection


I should add that we currently provide THOR only for a limited group of customers, mainly European corporations, government institutions and certain CSIRTs within the European Union. THOR’s little brother LOKI provides a very reduced feature set but may be enough and FENRIR is a dependency-less IOC scanner for Unix based target systems written in bash. For a Windows Powershell solution check out Kansa by Dave Hull. It also allows a distributed scan run using LOKI.

Detect System File Manipulations with SysInternals Sysmon

Mar 21, 2015 | APT, Security Monitoring, Splunk, Tutorial

SysInternals Sysmon is a powerful tool especially when it comes to anomaly detection. I recently developed a method to detect system file manipulations, which I would like to share with you.
We know how to track processes with the standard Windows audit policy option “Audit process tracking”, but Sysmon messages contain much more information to evaluate. By using Sysmon on many systems within the network and collecting all the logs in a central location you’ll get a database full of interesting attributes and Metadata which can be statistically analyzed in order to identify anomalies.
Carlos Perez wrote a really good article on Sysmon, which you should check out if you’re new to Sysmon and its capabilities.

Anomaly Detection

In recent years “anomaly detection” has often been used as marketing buzzword and as a result lost some of its shine. I am still a strong believer and often phrase sentences like “anomaly detection is the only method to detect yet unknown threats”. In security monitoring we call it anomaly detection, Antivirus vendors call it heuristics and SPAM appliances evaluate it in a “X-Spam-Score”.

Anomaly detection requires the ability to describe what is normal and exclude it from the evaluation.

With the data collected from the different Sysmon sources, this is an easy task to do. Sysmon provides the executable hash as MD5, SHA1 or SHA256 in the log entries that enables an analyst to identify the few different versions of a certain system executable. A hash of a system program like “cmd.exe” executed on the different systems on your domain should always be the same on all systems running the same version of Windows. But let me give you some examples.
A sane system environment analysis for the “cmd.exe” would look like this:

Hash - Image - Count
3C77C39347A6FA560A74587B0498FE84 - C:\WINDOWS\system32\cmd.exe - 56
AD7B9C14083B52BC532FBA5948342B98 - C:\Windows\System32\cmd.exe - 34

The following analysis includes an anomaly, which is worth to be investigated:

Hash - Image - Count
3C77C39347A6FA560A74587B0498FE84 - C:\WINDOWS\system32\cmd.exe - 56
AD7B9C14083B52BC532FBA5948342B98 - C:\Windows\System32\cmd.exe - 34
D8B7B276710127D233ABCDB7313AAC36 - C:\WINDOWS\system32\cmd.exe - 1

Let’s take a look at two analysis examples in which I use this method to identify different anomalies.

Anomaly 1: “StickyKeys” backdoor and the like

I use my favorite log analysis system for the analysis, which is Splunk. Getting the Sysmon data into splunk is easy as there is already a Sysmon Add-on available in the App Store. Just use the deployment manager to push the Add-on to the Splunk Forwarders and install Sysmon. (see my other blog post on Sysmon for more appropriate configuration options)
Then you can do things like that:

source="WinEventLog:Microsoft-Windows-Sysmon/Operational" NOT Image=*Sysmon.exe | dedup host,Image | stats distinct_count(Image) AS different_names,values(Image),values(host) by Hash | sort -different_names

It gives you an overview of files with the same hash but different names. It is pretty easy to spot the manipulation.

StickyKey Backdoor Detection with Splunk

StickyKey Backdoor Detection with Splunk and Sysmon

We detected a so called “StickyKeys” backdoor, which is a system’s own “cmd.exe” copied over the “sethc.exe”, which is located in the same folder and provides the Sticky Keys functionality right in the login screen. Replacing it with a system command line establishes a shell running as LOCAL_SYSTEM that pops up when you RDP to a server and press 5 times shift consecutively. (see this blog post for more information on this backdoor)

Anomaly 2: The Black Sheep

If you create the statistics by “Image” instead of “Hash” you’ll get an overview of the different versions of system files in use and are able to identify system file versions that are unique.
Look at the following example to get an impression what can be done with this method.

source="WinEventLog:Microsoft-Windows-Sysmon/Operational" NOT Image=*Sysmon.exe | dedup host,Image | rex field=Image "(?[^\\\]+)$" | eval Executable=lower(Executable) | stats count by Executable,Hash | sort +count

I am sorry but I can’t give you a nice screenshot on what would it look like in a big environment. These are the results from 3 different demo systems only (Win2003, Win7 and Win8), but in order to see what it would look like in a environment with hundreds or thousands of systems, see the listing below.

Sysmon Detection Splunk

Sysmon Anomaly Detection with Splunk

The result would look like this:

Hash - Image - Count
AD7B9C14083B52BC532FBA5948342B98 - cmd.exe - 1480
3C77C39347A6FA560A74587B0498FE84 - cmd.exe - 256
D8B7B276710127D233ABCDB7313AAC36 - cmd.exe - 2

Consider the image files with a low count as anomalies and try to figure out, why the hash of the system executable is different from the variants on the other systems.
I would google the hash of the black sheep, which is “D8B7B276710127D233ABCDB7313AAC36” and see if I can get more details. An empty google result is NOT a good sign as some may be inclined to believe. If the google results are ambiguous you should try to figure out if these systems are somehow special – e.g. certain readout system on embedded OS versions, systems that do not receive patches. If the findings are still suspicious you should drop the samples in a sandbox and see how they behave.
Hope you liked it. Please give me feedback if you actually tested this method in your environment so that I can improve the search statements or handle false positive conditions.

APT Detection is About Metadata

Mar 10, 2015 | APT, THOR, Tool, Tutorial, YARA

People often ask me, why we changed the name of our scanner from “IOC” to “APT” scanner and if we did that only for marketing reasons. But don’t worry, this blog post is just as little a sales pitch as it is an attempt to create a new product class.
I’ll show you why APT detection is difficult – for the big players and spirited newcomers like us.

Metadata is the Key

Only recently I recognized and named the methods that we apply since we introduced the scoring system in our scanner product. Instead of looking at a file only by its content we collect numerous attributes and evaluate a score based on certain rules that indicate conspicuous features or anomalies.
What I recognized was that Metadata is the key to successful APT detection. Let me give you some examples.

The “Sticky Keys” Backdoor

During our investigations we found that the attackers used a simple backdoor that allowed them to avoid AV detection and use tools that were already available on the target systems. What they did was to copy a valid “cmd.exe” over the “sethc.exe” in the System32 folder in order to establish a backdoor that waits for the user pressing five times shift consecutively on a RDP logon screen and pops up a Windows command line running as LOCAL_SYSTEM. Another method sets the Windows command line as debugger for the stickykeys binary.

wmic /user:username /password:secret /node:system1 process call create
"C:\Windows\system32\reg.exe add \"HKLM\SOFTWARE\Microsoft\Windows
NT\CurrentVersion\Image File Execution Options\sethc.exe\"
/v \"Debugger\" /t REG_SZ /d \"cmd.exe\" /f"

With the necessary rights it is easy to install and difficult to detect.

StickyKeys Backdoor APT

APT Detection StickyKeys Backdoor

As I already said, an Antivirus engine won’t detect this backdoor as the content of the file is a valid Windows executable with an intact signature. Windows 7+ users won’t stumble over it as they have Network Level Authentication (NLA) enabled by default, which prompts the user for username and password before fully establishing a Terminal Services connection. Attackers modify their local “default.rdp” file and add “enablecredSSPsupport:i:0:1” in order to disable this behaviour.
APT detection therefore means the following:

  • Check system files like the stickykey binary for modifications – not by comparing MD5 hashes from whitelist databases like the NSRL, but by comparing the expected content for a certain file name with the actual content of the file. I described this method in a blog article and Chad Tilbury from Crowdstrike described how to apply this method using their CrowdResponse tool.
  • Identify “default.rdp” files on server systems that have NLA disabled. (Administrators shouldn’t do that)
  • Check if the Windows command line (cmd.exe) is registered as a debugger for any program

PsExec’s Evil Clone

I often use the example of the well-known Sysinternals tool “PsExec”, which is likewise used by administrators and APT groups. It doesn’t make much sense adding it to the indicators of compromise (IOCs) of your triage sweep although it may have played a substantial role in lateral attacker movement.
The human eye is able to distinguish between a PsExec that has been used for administration and a PsExec that has been used by the attackers. The essential difference, which enables us to distinguish between both versions does not lie in the content of the file but the Metadata.
Look at the following table and tell me which of both files is valid and which has been placed on the system by the attackers. Remember, the file content is the same – a MD5 hash of both files is equal.

File 1 File 2
MD5 aeee996fd3484f28e5cd85fe26b6bdcd aeee996fd3484f28e5cd85fe26b6bdcd
Filename PsExec.exe p.exe
Path C:\SysInternals C:\TEMP
Owner Administrators LOCAL_SYSTEM
Modified Time Stamp 2013-02-10 09:22:04 1970-01-01 00:00:00

It is not that difficult, isn’t it?
APT detection therefore means the following:

  • Imitating the human point of view by pulling together all Metadata connected with an element, be it a file, a process or eventlog message and evaluate the legitimacy of the element based on all available Metadata attributes

The Simplest Webshell

We often encounter so-called webshells that were placed in web server directories to establish a simple backdoor. Webshells can be very specific and therefore easy to detect. The C99 webshell is a good example for a PHP webshell, JspSpy is a well-known JSP webshell. Both are easily detected, even by Antivirus engines (see: C99 on VT, JSPSpy on VT).
However, APT groups tend to use two different types of webshells:

  • Tiny webshells
  • Code snippets with certain functions copied from legitimate software

There are a lot of well-known tiny webshells. The following one is my favorite. Add a space or change the request parameter “abc” to something else and the detection ratio is alarmingly low (Example). It allows an attacker to evaluate (execute) an arbitrary command on the web server. There are numerous blog posts and other articles describing what can be done with a webshell like this. However, the protection level provided by AV engines, firewalls and NIDS is almost zero.

<%eval request("abc")%>

Another method we discovered was the use code snippets copied from blog entries or tutorial pages that allowed them to use only certain functions like “file upload” or “directory listing”.
They often use a weakness in web applications to upload and run their own scripts or even whole application containers (.war). By placing a known webshell like the JspSpy webshell into that web server folder, they would run the risk of being detected. What they really need is a distribution point for their toolset or a simple tool to execute code on the server (like a tiny webshell). We’ve seen simple upload scripts that provide nothing more than a upload function, which they use to store their toolsets for lateral movement. A google search for “upload jsp” revealed various scripts they used in their attacks. It’s obvious that AV engines won’t detect this type of threat. How could they? The attackers abuse benign pieces of code to establish malicious backdoors.
APT detection therefore means the following:

  • Use the Metadata like file size, creation timestamp, file extension in combination with generic content detection rules – e.g. check for the string “eval” in a file smaller 40 byte and a script extension like “.jsp”.
  • Check the content of upload directories for the expected file types (and don’t use the extension to determine the file type)
  • Check web server processes for executables running in the web server directories – e.g. curl.exe in D:\Inetpub\wwwroot
  • Generate and send frequent reports on modified files within the web server directories

The Heavy Burden of Definite Detection

One could be tempted to believe that I wrote this article in order to degrade Antivirus engines, but this isn’t the case. Antivirus solutions are still play a key role and carry the heavy burden of definite detection. Their scan result has to be “thumbs up” or “thumbs down” as there is no middle ground.
Years ago they introduced signatures to detect “Potentially Unwanted Applications” (PUA). Users or administrators decide on what to do if one of these “dual use” tools has been found on a system. Handling thousands of the events generated by the Antivirus agents is a difficult task, even with a central console or SIEM integrated log files. It is easy to understand why PUA events do not play an important role in view of dozens of Trojan detections per day.

APT detection is the art of suspicion. A missing “stickykeys” string in the “sethc.exe” indicates a manipulation, a replaced system file. It is not a definite detection but the certainty that something is wrong.

Conclusion

Considering the given examples an attentive reader may be inclined to believe that Antivirus and simple IOC scanning (Triage) is not enough to defend against Advanced Persistent Threats. After the experiences of the last 3 years I have to confirm that assumption.
Who would recognize and report the execution of a “sethc.exe” on a server system, the “PUA/PsExec” message generate by the Antivirus or another JSP file on the web server?
I even doubt that so-called “APT solutions” are able to detect

  • a “.war”-file upload to a Tomcat server by the use of “tomcat/tomcat” as credentials,
  • encrypted file uploads,
  • lateral movement using PsExec, Powershell or WMIC and
  • “StickyKey” backdoor access via RDP.

An extensive security monitoring in form of a SIEM system allows you to detect a needle in the haystack but only if you are able to distinguish between straw and needles.

The question is: How can I define such soft indicators to detect the described anomalies? The OpenIOC framework already contains options to combine certain characteristics like filename and filesize, but rather than using it as a tool to describe anomalies it is often used to tighten the detection to the level of a hash value. I prefer hash values over “Name:PsExec.exe” combined with “Filesize:381816” because it doesn’t make you believe that you’re looking at a clever rule.
I therefore recommend the following:

  1. Assume compromise and start from there
    Ask yourself: How would I detect a breach? What if attackers already took control of the Windows Domain and worked with domain admin accounts? What if they worked with tools that my Antivirus is unable to detect?
  2. Use all Metadata you can get to determine the legitimacy of an element
    This does no only apply to files or processes in APT and IOC scanning, but also to the discipline of security monitoring.E.g. Select interesting Antivirus events based on various characteristics and not only the status that indicates if the malware has be removed or not. Consider the location of the malware (Temporary Internet files or System32 folder), the user account (Restricted user, Administrator or LOCAL_SYSTEM), the malware type (JS/Redirector or PUA/PasswordDump), system type (server or client workstation), detection time (02:00am with noone working in the night shift), detected form (in a RAR archive or extracted). Develop similar schemes for other log types. The most interesting ones are Antivirus, Proxy and Windows logs.
    Useful Links:
    SysForensics published an article about process anomaly detection, which was adapted to other OS versions and included in our THOR APT Scanner and my LOKI IOC Scanner.
    Use SysInternals Sysmon to enrich you Windows log data.
  3. Don’t create detection rules that are too tight but concentrate on filtering the false positives
    If you regard filtering false positives as a pain in the neck you’re probably using the wrong SIEM system. (Warning – product placement: I prefer Splunk over all others, especially with the Enterprise Security App installed)
  4. Use IOCs from published APT reports to enrich your detection rules
    We use the APT reports to create new rules in our customer’s SIEM systems and as input for our APT scanner THOR.Useful Links:
    APT Notes is a IOC repository with hundreds of reports from the last years. You can download the github repository from here.
How to Scan for System File Manipulations with Yara (Part 2/2)

How to Scan for System File Manipulations with Yara (Part 2/2)

Aug 28, 2014 | APT, Command Line, Tool, Tutorial, YARA

As a follow up on my first article about inverse matching yara rules I would like to add a tutorial on how to scan for system file manipulations using Yara and Powershell. The idea of inverse matching is that we do not scan for something malicious that we already know but for anomalies within the system files. Chad Tilbury from Crowdstrike related to this method in his article describing a way to scan for this type of anomaly using their incident collection tool CrowdResponse. In my first article I described how we utilize this method in our incident response tool and promised a free solution based on available system tools.
The yara rules used to apply this method require the name of the observed file. Yara allows the file name to be passed via an external variable like in the following listing.

yara32.exe -d filename=iexplore.exe inverse-matching.yar iexplore.exe

But we have to define and pass this “filename” variable for every file we analyse while walking the directory tree.
So – what do we do?
First – we need a powershell script that walks a directory tree and feeds each file with an “.exe” extension together with the rule set and the file name as external variable to a yara32.exe. You could copy the script and paste it directly to the command line but I would recommend the following:
Prepare a folder with the following content:

  1. The powershell script as listed below – name it “inverse-scan.ps1”
  2. The ruleset listed below as “inverse-matching.yar”
  3. A version of Yara for Windows
  4. A batch script that invokes the powershell script with some parameters named “runit.bat”

The final result looks like this:

Yara Scan on Anomalies

Inverse Yara Matching Script Set

You can copy that folder to the target system, take it with you on a USB drive or provide a network share with its contents.
inverse-scan.ps1

Get-ChildItem -Recurse -filter *.exe C:\Windows 2> $null |
ForEach-Object { Write-Host -foregroundcolor "green" "Scanning"$_.FullName $_.Name; ./yara32.exe -d filename=$_.Name inverse-matching.yar $_.FullName 2> $null }

runit.bat

@ECHO OFF
powershell -ExecutionPolicy ByPass -File ./inverse-scan.ps1

inverse-matching.yar

rule iexplore_ANOMALY {
	meta:
		author = "Florian Roth"
		description = "Abnormal iexplore.exe - typical strings not found in file"
		date = "23/04/2014"
		score = 55
	strings:
		$upd_magic = { 44 43 }
		$win2003_win7_u1 = "IEXPLORE.EXE" wide nocase
		$win2003_win7_u2 = "Internet Explorer" wide fullword
		$win2003_win7_u3 = "translation" wide fullword nocase
		$win2003_win7_u4 = "varfileinfo" wide fullword nocase
	condition:
		not ( $upd_magic at 0 ) and not 1 of ($win*) and filename matches /iexplore\.exe/is
}
rule svchost_ANOMALY {
	meta:
		author = "Florian Roth"
		description = "Abnormal svchost.exe - typical strings not found in file"
		date = "23/04/2014"
		score = 55
	strings:
		$upd_magic = { 44 43 }
		$win2003_win7_u1 = "svchost.exe" wide nocase
		$win2003_win7_u3 = "coinitializesecurityparam" wide fullword nocase
		$win2003_win7_u4 = "servicedllunloadonstop" wide fullword nocase
		$win2000 = "Generic Host Process for Win32 Services" wide fullword
		$win2012 = "Host Process for Windows Services" wide fullword
	condition:
		filename matches /svchost\.exe/is and not 1 of ($win*) and not ( $upd_magic at 0 )
}
rule explorer_ANOMALY {
	meta:
		author = "Florian Roth"
		description = "Abnormal explorer.exe - typical strings not found in file"
		date = "27/05/2014"
		score = 55
	strings:
		$upd_magic = { 44 43 }
		$s1 = "EXPLORER.EXE" wide fullword
		$s2 = "Windows Explorer" wide fullword
	condition:
		filename matches /explorer\.exe/is and not 1 of ($s*) and not ( $upd_magic at 0 )
}
rule sethc_ANOMALY {
	meta:
		description = "Sethc.exe has been replaced - Indicates Remote Access Hack RDP"
		author = "F. Roth"
		reference = "http://www.emc.com/collateral/white-papers/h12756-wp-shell-crew.pdf"
		date = "2014/01/23"
		score = 70
	strings:
		$upd_magic = { 44 43 }
		$s1 = "stickykeys" fullword nocase
		$s2 = "stickykeys" wide nocase
		$s3 = "Control_RunDLL access.cpl" wide fullword
		$s4 = "SETHC.EXE" wide fullword
	condition:
		filename matches /sethc\.exe/ and not 1 of ($s*) and not ( $upd_magic at 0 )
}
rule Utilman_ANOMALY {
	meta:
		author = "Florian Roth"
		description = "Abnormal utilman.exe - typical strings not found in file"
		date = "01/06/2014"
		score = 55
	strings:
		$upd_magic = { 44 43 }
		$win7 = "utilman.exe" wide fullword
		$win2000 = "Start with Utility Manager" fullword wide
		$win2012 = "utilman2.exe" fullword wide
	condition:
		filename matches /utilman\.exe/is and not 1 of ($win*) and not ( $upd_magic at 0 )
}
rule osk_ANOMALY {
	meta:
		author = "Florian Roth"
		description = "Abnormal osk.exe (On Screen Keyboard) - typical strings not found in file"
		date = "01/06/2014"
		score = 55
	strings:
		$upd_magic = { 44 43 }
		$s1 = "Accessibility On-Screen Keyboard" wide fullword
		$s2 = "\\oskmenu" wide fullword
		$s3 = "&About On-Screen Keyboard..." wide fullword
		$s4 = "Software\\Microsoft\\Osk" wide
	condition:
		filename matches /osk\.exe/is and not 1 of ($s*) and not ( $upd_magic at 0 )
}
rule magnify_ANOMALY {
	meta:
		author = "Florian Roth"
		description = "Abnormal magnify.exe (Magnifier) - typical strings not found in file"
		date = "01/06/2014"
		score = 55
	strings:
		$upd_magic = { 44 43 }
		$win7 = "Microsoft Screen Magnifier" wide fullword
		$win2000 = "Microsoft Magnifier" wide fullword
		$winxp = "Software\\Microsoft\\Magnify" wide
	condition:
		filename matches /magnify\.exe/is and not 1 of ($win*) and not ( $upd_magic at 0 )
}
rule narrator_ANOMALY {
	meta:
		author = "Florian Roth"
		description = "Abnormal narrator.exe - typical strings not found in file"
		date = "01/06/2014"
		score = 55
	strings:
		$upd_magic = { 44 43 }
		$win7 = "Microsoft-Windows-Narrator" wide fullword
		$win2000 = "&About Narrator..." wide fullword
		$win2012 = "Screen Reader" wide fullword
		$winxp = "Software\\Microsoft\\Narrator"
		$winxp_en = "SOFTWARE\\Microsoft\\Speech\\Voices" wide
	condition:
		filename matches /narrator\.exe/is and not 1 of ($win*) and not ( $upd_magic at 0 )
}
rule notepad_ANOMALY {
	meta:
		author = "Florian Roth"
		description = "Abnormal notepad.exe - typical strings not found in file"
		date = "01/06/2014"
		score = 55
	strings:
		$upd_magic = { 44 43 }
		$win7 = "HELP_ENTRY_ID_NOTEPAD_HELP" wide fullword
		$win2000 = "Do you want to create a new file?" wide fullword
		$win2003 = "Do you want to save the changes?" wide
		$winxp = "Software\\Microsoft\\Notepad" wide
	condition:
		filename matches /notepad\.exe/is and not 1 of ($win*) and not ( $upd_magic at 0 )
}

Although the string descriptors list only some of the windows versions we’ve tested it against the following versions:
Windows 2000
Windows 2003 Server
Windows 7 (x64)
Windows 2008 R2
Windows 2012
What you get as result is a small anomaly scanner made completely with Windows tools and Yara. An administrator would just have to click the Batch file and run the script with admin rights. The following screenshot shows a scan on the Windows folder with a prepared malicious “iexplore.exe” in the subfolder “C:\Windows\AA_Testing”.

Yara Anomaly Scanner

Yara Inverse Matching Anomaly Scanner in Action

You could remove the section “Write-Host -foregroundcolor “green” “Scanning”$_.FullName $_.Name;” to show only the alerts or modify the script that it writes a log file.
We use all of these rules in our APT Scanner THOR and added further rules matching 3rd party tools attackers tend to replace or rename.

GDPR Cookie Consent with Real Cookie Banner