Antivirus Event Analysis Cheat Sheet v1.8

The “Web Proxy Event Analysis Cheat Sheet” can help SOCs and security analysts classify proxy events (blocks, alerts) and is based on my ideas and many ideas from experts that helped me collect detection ideas for this document.
You can download version 1.0 here.
We also recommend checking Sigma’s “proxy” section for detection rules that can be used to detect threats in web proxy or similar logs as long as they contain web connection information (EDR, HIDS etc.).
At first, strictly avoid placing systems in different network segments. Instead – keep it simple. Let standard Windows workstations, admin workstations, server systems, print servers, industrial control systems, backup servers, network management systems, monitoring servers, terminal servers, mobile management servers, development systems, IP cameras, house automation, and SIP telephones be in a single huge network in order to avoid firewalling issues.
That principle applies also to subsidiaries and affiliates. After the acquisition of other companies, don’t waste time asking them for compliance with your security policies. Save time and connect their network directly with your backbone! Make sure to allow any type of connection and create domain trusts as fast as possible to enable cross-domain resource access to your data center in Berlin from your user workstations in Brazil, as well as the call center in Bangladesh.
If you are required to create network zones for some reason, interconnect these zones with mutual Windows domain trust relationships. (see Titanic as an example)
In order to maintain an environment of mutual trust and respect, never monitor the gateways between segments for port scanning, network sweeps, or other types of suspicious network activity. Also, please do not commit the new business unit to provide technical support in cases of suspicious activity and incident response.
You trust your administrator, don’t you? So, let administrators use their privileged accounts to surf the web and read email while they are connected via SSH and RDP to their domain controllers and Internet facing servers in DMZ networks. Do not use hardened and intensely monitored jump servers from special admin workstations to manage your most important server networks.
Don’t waste precious resources creating administrative roles. If someone needs to install a text editor on a server, add him to the Domain Administrators group and make sure that no one ever figures out why you did this.
Do not use Microsoft’s solution called LAPS to ensure that all computers have different, complex, local administrator passwords or Privileged Admin Workstations (PAW) to provide a dedicated operating system for sensitive tasks. This would give attackers an unnecessarily hard time finding highly privileged credentials and fewer opportunities to use them.
Attackers tend to make themselves familiar with the environment and therefore cause strange peaks in log volumes that could be easily detected by an over-attentive employee – so, don’t log security events. If you do log events for any reason, make sure that all of your systems keep their logs in a local log file and do not transmit them to a central SIEM system.
If you have to use a SIEM system, use Active Directory for authentication, so that attackers can find all the fine log data in a central place. If you have any problems on a server, such as with disk space due to the fact that you configured your golden image for Windows servers to use only 20GB of space on drive ‘C:’, don’t hesitate to reduce the log file size to a minimum. That may cause the log to overwrite itself every 30 minutes, but hey, each line that is overwritten can help prevent successful detection.
On Linux systems configure your log file rotation to keep 7 days, not more. Your IT staff typically needs a couple of days investigate the system owner and have some meaningful discussion with your data protection officer.
If you have to comply with certain policies that require the collection of log data, you still can do a lot to make sure that breaches remain undetected for months. Here is our Top 10:
Centralization is good. It saves resources and simplifies user administration. Use Active Directory authentication for everything: the logins to your proxy servers, network devices, online certificate authorities, virtual machine consoles, administrative jump hosts, security monitoring servers, VPN servers, SIEM system, and last but not least – backup servers.
This ensures that attackers take over the complete infrastructure after compromising a single outdated member server that has domain admin logon sessions. Remediation becomes much more exciting when attackers have access to proxies, DNS servers, and mail gateways all at once!
Since corporate workstations operate deep in the internal network, which are well protected by several firewalls and proxy servers, consider them to be far outside of the danger zone. Don’t audit them as frequently as you do with DMZ servers. There are so many more clients than servers that it is certainly not efficient to scan them all. Also, random samples never give a valid result, so let’s just drop the whole thing.
Don’t patch workstation software like Microsoft Office, PDF viewers, JAVA and Flash plugins, media players, and archivers as soon as a patch is published. Don’t use exploit protection, like Microsoft EMET, in order to increase the risk of zero-day attacks and finally, don’t deactivate content in document viewers, such as JavaScript in Acrobat Reader or VBA in Microsoft Office.
Simplify administration of user workstations by granting standard users local administrative rights.
Also allow workstations to access the Internet directly on any service port. Don’t use a proxy server and do not monitor dropped connections on typical Trojan back connect ports. Developers and administrators are highly skilled professionals that tend to download and install suspicious software from the shadiest websites. Remember: You trust them and if they think that they need that (suspicious) software than of course they do. To increase the probability of such events, block Sourceforge, Github, and the whole category “Software Downloads” on your Internet proxy server.
As mentioned before, do not consider antivirus alerts that have the status “deleted”, “cleaned”, or “moved to quarantine”. A deeper analysis of these events could reveal a Trojan that had control of the system for weeks before the right signature returned a match on components of a hack tool set that attackers moved from server to server. Therefore, do only check for errors and unresolved operational issues.
Make sure that no one pays special attention to antivirus events that report “Hack Tools”, “Password Dumpers”, or “Scanners”. Tell everyone that this would cause too many false positives because system administrators need these tools to find their assets and regain access to them.
Also avoid rating antivirus events according to an evaluation method. That’s so bureaucratic.
To allow attackers the greatest possible leeway, create vast exclusion lists and don’t use special Antivirus functions like PUA scanning or application controls that block password dumpers from accessing the memory.
Regard web servers like any other server type. Frequently patching the web server service is perfectly sufficient. Do not audit the applications running on that web server; if you have to due to corporate policies, do it once, print the report, and archive it.
Place the web servers behind reverse proxy servers and tell everyone that this will protect them. If you repeat that constantly everyone will believe it one day. Do not protect the web servers with costly Web Application Firewalls and don’t collect the logs of such a system for central attack detection and analysis.
Allow developers to access the management interfaces, like JMX or Tomcat Manager, from remote locations. Don’t log access to these applications and don’t ban source IPs on security violations. Tell everyone that each developer may access the servers anytime, from anywhere, to reduce operational risks.
If you have to run Apache or Tomcat on a server, choose Windows as the operating system. Do not use limited user accounts to run the web server services in order to maximize the impact of a successful attack. Last but not least: Avoid annoying security features like SELinux.
I know that it is hard to guarantee a successful APT attack and that all of these recommendations require a certain amount of stubbornness and resistance to advice; however, even if this advice does not guarantee falling victim to advanced persistent threats, chances increase exponentially the more of my advice you apply. Good luck – I’ll keep my fingers crossed.
If you have further ideas that you want to share, please comment on this article or contact me on Twitter @cyb3rops.
Many thanks to Stephan Kaiser for the idea, Julia Stolz and Jeff for major reviews and Matthias Kaiser (@matthias_kaiser), Daniel Sauder (@DanielX4v3r), Thomas Patzke (@blubbfiction), Claas Rettinghausen, Robert Haist (@SleuthKid), Alexander Döhne for their valuable feedback.
The most important feature of the upcoming yarGen YARA Rule Generator release is the Binarly API integration.
Binarly is a “binary search engine” that can search arbitrary byte patterns through the contents of tens of millions of samples, instantly. It allows you to quickly get answers to questions like:
Binary Search Engine – Binar.ly
Binarly Service Lookup in yarGen 0.16
But let’s talk about the improved rule generation process.
As described in my previous articles, I try to divide the list of strings generated by yarGen into two different groups:
In previous examples I always tended to combine these strings with magic header and file size. yarGen 0.15 and older versions generated those rules by default. The problem with these rules is that they do not detect the malware or tools to process memory.
Therefore I changed my rule generation process and adjusted yarGen to follow that example. As I said before, yarGen is not designed to generate perfect rules. Its main purpose is to generate raw rules that require the least effort to complete and could also work without further modification.
The following image shows how new rules are composed. They contain two main conditions, one for the file detection and one for the in-memory detection. I tried to copy the manual rule generation process as far as possible.
YARA rule composition (manual composition and yarGen v0.16)
Here is an example of a rule produced by yarGen v0.16 (sample Unit 78020 – WininetMM.exe). I shows a ‘raw’ rule without further editing and the ‘scores’ included as comments:
You may ask “Why do the ‘DATA_BEGINS:’ and ‘.?AVCWinnetSocket@@’ do have such high scores”? Well, that’s the reason why analysts needs the support of big data:
I have to add that Binarly offers two query modes (fast/exact) of which yarGen uses the ‘fast’ mode. An analyst that doubts the produced results would use ‘exact’ query mode to verify the results manually. Please ask Daniel about the details.
The new version of yarAnalyzer allows to generate an inventory of your YARA rule sets. This features comes in very handy in cases in which you have to handle a big set of rules. The ‘–inventory’ option generates a CSV file that can be prettied up in MS Excel or Openoffice Calc.
yarAnalyzer Inventory
System Files and Sizes
I ran this rule set over my goodware database and got only a few false positives. Feel free to use these rules wherever you like but please share new rules or statistical analyses on other system files.
False Positives
In the past I was glad to see very specific strings in samples and sometimes used these strings as the only indicator for detection. E.g. whenever I’ve found a certain typo in the PE header fields like “Micorsoft Corportation” I cheered and thought that this would make a great signature. But – and I have to admit that now – this only makes a nice signature. Great signatures require not only to match on a certain sample in the most condensed way but aims to match on similar samples created by the same author or group.
Look at the following rule:
What I do when I review the 20 strings that are generated by yarGen is that I try to categorize the extracted strings in 3 different groups:
Then I create a condition that defines:
Here is another example that does only have very specific strings (x) and common strings (s):
If you can’t create a rule that is sufficiently specific, I recommend the following methods to restrict the rule:
One of the most underestimated features of Yara is the possibility to define a range in which strings occur in order to match. I used this technique to create a rule that detect metasploit meterpreter payloads quite reliably even if it’s encoded/cloaked. How that?
If you see malware code that is hidden in an overlay at the end of a valid executable (e.g. “ab.exe”) and you see only strings that are typical function exports or mimics a well-known executable ask the following questions:
Malware Strings
Webshell Code PHP
I collected many ideas by Wesley Shields and Victor M. Alvarez and composed a gist called “Yara
Performance Guidelines”. This guide shows you how to write Yara rules that use less CPU cycles by avoiding CPU intensive checks or using new condition checking shortcuts introduced in Yara version 3.4.
Yara Performance Guidelines
People sometimes ask why I don’t use the PE module. The reason is simple: I avoid using modules that are rather new and would like to see it thoroughly tested prior using it in my scanners running in productive environments. It is a great module and a lot of effort went into it. I would always recommend using the PE module in lab environments or sandboxes. In scanners that walk huge directory trees a minor memory leak in one of the modules could lead to severe memory shortages. I’ll give it another year to prove its stability and then start using it in my rules.
yarGen has an opcode feature since the last minor version. It is active by default but only useful in cases in which not enough strings could be extracted.
I currently use the following parameters to create my rules:
The problem with the opcode feature is that it requires about 2,5 GB more main memory during rule creation. I’ll change it to an optional parameter in the next version.
yarAnalyzer is a rather new tool that focuses on rule coverage. After creating a bigger rule set or a generic rule that should match on several samples you’d like to check the coverage of your rules in order to detect overlapping rules (which is often OK).
yarAnalyzer helps you to get an overview on:
yarAnalayzer Screenshot
To review the strings in a sample I use a simple shell one-liner that a good friend sent me once.
“strings” version for Linux
“gstrings” version for OS X (sudo port install binutils)
It produces an output as shown in the above screenshot with green text and the description “Malware Strings” showing the offset, ascii (A) or wide (W) and the string at this offset.
For a colorization of the string check my new tool “prisma” that colorizes random type standard output.
Prisma STDOUT colorization
Follow me on Twitter: @Cyb3rOps
Example – Threat Intel Feed OTX Receiver (LOKI)
Threat Intel Hash CSV for Splunk Lookup
Threat Intel CSV Files in Splunk Search App Lookup Folder
Threat Intel CSV File Lookup Definition in Splunk
Windows Sysmon Log Data in Splunk
Windows Sysmon Log Data with Hash Values of Executables
After the lookup I search for all entries that have a “threat_description” field set and display them in a easy-to-read table view. Only entries that had a “Hash” matching on a “hash” from the CSV will have this new field set. In the example below I had a match on an unwanted application called “Pantsoff” that I used in my Lab environment for this POC.
Threat Intel Lookup in Splunk
In recent years “anomaly detection” has often been used as marketing buzzword and as a result lost some of its shine. I am still a strong believer and often phrase sentences like “anomaly detection is the only method to detect yet unknown threats”. In security monitoring we call it anomaly detection, Antivirus vendors call it heuristics and SPAM appliances evaluate it in a “X-Spam-Score”.
Anomaly detection requires the ability to describe what is normal and exclude it from the evaluation.
With the data collected from the different Sysmon sources, this is an easy task to do. Sysmon provides the executable hash as MD5, SHA1 or SHA256 in the log entries that enables an analyst to identify the few different versions of a certain system executable. A hash of a system program like “cmd.exe” executed on the different systems on your domain should always be the same on all systems running the same version of Windows. But let me give you some examples.
A sane system environment analysis for the “cmd.exe” would look like this:
The following analysis includes an anomaly, which is worth to be investigated:
Let’s take a look at two analysis examples in which I use this method to identify different anomalies.
I use my favorite log analysis system for the analysis, which is Splunk. Getting the Sysmon data into splunk is easy as there is already a Sysmon Add-on available in the App Store. Just use the deployment manager to push the Add-on to the Splunk Forwarders and install Sysmon. (see my other blog post on Sysmon for more appropriate configuration options)
Then you can do things like that:
It gives you an overview of files with the same hash but different names. It is pretty easy to spot the manipulation.
StickyKey Backdoor Detection with Splunk and Sysmon
If you create the statistics by “Image” instead of “Hash” you’ll get an overview of the different versions of system files in use and are able to identify system file versions that are unique.
Look at the following example to get an impression what can be done with this method.
I am sorry but I can’t give you a nice screenshot on what would it look like in a big environment. These are the results from 3 different demo systems only (Win2003, Win7 and Win8), but in order to see what it would look like in a environment with hundreds or thousands of systems, see the listing below.
Sysmon Anomaly Detection with Splunk
Consider the image files with a low count as anomalies and try to figure out, why the hash of the system executable is different from the variants on the other systems.
I would google the hash of the black sheep, which is “D8B7B276710127D233ABCDB7313AAC36” and see if I can get more details. An empty google result is NOT a good sign as some may be inclined to believe. If the google results are ambiguous you should try to figure out if these systems are somehow special – e.g. certain readout system on embedded OS versions, systems that do not receive patches. If the findings are still suspicious you should drop the samples in a sandbox and see how they behave.
Hope you liked it. Please give me feedback if you actually tested this method in your environment so that I can improve the search statements or handle false positive conditions.
Only recently I recognized and named the methods that we apply since we introduced the scoring system in our scanner product. Instead of looking at a file only by its content we collect numerous attributes and evaluate a score based on certain rules that indicate conspicuous features or anomalies.
What I recognized was that Metadata is the key to successful APT detection. Let me give you some examples.
During our investigations we found that the attackers used a simple backdoor that allowed them to avoid AV detection and use tools that were already available on the target systems. What they did was to copy a valid “cmd.exe” over the “sethc.exe” in the System32 folder in order to establish a backdoor that waits for the user pressing five times shift consecutively on a RDP logon screen and pops up a Windows command line running as LOCAL_SYSTEM. Another method sets the Windows command line as debugger for the stickykeys binary.
With the necessary rights it is easy to install and difficult to detect.
APT Detection StickyKeys Backdoor
I often use the example of the well-known Sysinternals tool “PsExec”, which is likewise used by administrators and APT groups. It doesn’t make much sense adding it to the indicators of compromise (IOCs) of your triage sweep although it may have played a substantial role in lateral attacker movement.
The human eye is able to distinguish between a PsExec that has been used for administration and a PsExec that has been used by the attackers. The essential difference, which enables us to distinguish between both versions does not lie in the content of the file but the Metadata.
Look at the following table and tell me which of both files is valid and which has been placed on the system by the attackers. Remember, the file content is the same – a MD5 hash of both files is equal.
File 1 | File 2 | |
MD5 | aeee996fd3484f28e5cd85fe26b6bdcd | aeee996fd3484f28e5cd85fe26b6bdcd |
Filename | PsExec.exe | p.exe |
Path | C:\SysInternals | C:\TEMP |
Owner | Administrators | LOCAL_SYSTEM |
Modified Time Stamp | 2013-02-10 09:22:04 | 1970-01-01 00:00:00 |
It is not that difficult, isn’t it?
APT detection therefore means the following:
We often encounter so-called webshells that were placed in web server directories to establish a simple backdoor. Webshells can be very specific and therefore easy to detect. The C99 webshell is a good example for a PHP webshell, JspSpy is a well-known JSP webshell. Both are easily detected, even by Antivirus engines (see: C99 on VT, JSPSpy on VT).
However, APT groups tend to use two different types of webshells:
There are a lot of well-known tiny webshells. The following one is my favorite. Add a space or change the request parameter “abc” to something else and the detection ratio is alarmingly low (Example). It allows an attacker to evaluate (execute) an arbitrary command on the web server. There are numerous blog posts and other articles describing what can be done with a webshell like this. However, the protection level provided by AV engines, firewalls and NIDS is almost zero.
Another method we discovered was the use code snippets copied from blog entries or tutorial pages that allowed them to use only certain functions like “file upload” or “directory listing”.
They often use a weakness in web applications to upload and run their own scripts or even whole application containers (.war). By placing a known webshell like the JspSpy webshell into that web server folder, they would run the risk of being detected. What they really need is a distribution point for their toolset or a simple tool to execute code on the server (like a tiny webshell). We’ve seen simple upload scripts that provide nothing more than a upload function, which they use to store their toolsets for lateral movement. A google search for “upload jsp” revealed various scripts they used in their attacks. It’s obvious that AV engines won’t detect this type of threat. How could they? The attackers abuse benign pieces of code to establish malicious backdoors.
APT detection therefore means the following:
One could be tempted to believe that I wrote this article in order to degrade Antivirus engines, but this isn’t the case. Antivirus solutions are still play a key role and carry the heavy burden of definite detection. Their scan result has to be “thumbs up” or “thumbs down” as there is no middle ground.
Years ago they introduced signatures to detect “Potentially Unwanted Applications” (PUA). Users or administrators decide on what to do if one of these “dual use” tools has been found on a system. Handling thousands of the events generated by the Antivirus agents is a difficult task, even with a central console or SIEM integrated log files. It is easy to understand why PUA events do not play an important role in view of dozens of Trojan detections per day.
APT detection is the art of suspicion. A missing “stickykeys” string in the “sethc.exe” indicates a manipulation, a replaced system file. It is not a definite detection but the certainty that something is wrong.
Considering the given examples an attentive reader may be inclined to believe that Antivirus and simple IOC scanning (Triage) is not enough to defend against Advanced Persistent Threats. After the experiences of the last 3 years I have to confirm that assumption.
Who would recognize and report the execution of a “sethc.exe” on a server system, the “PUA/PsExec” message generate by the Antivirus or another JSP file on the web server?
I even doubt that so-called “APT solutions” are able to detect
An extensive security monitoring in form of a SIEM system allows you to detect a needle in the haystack but only if you are able to distinguish between straw and needles.
The question is: How can I define such soft indicators to detect the described anomalies? The OpenIOC framework already contains options to combine certain characteristics like filename and filesize, but rather than using it as a tool to describe anomalies it is often used to tighten the detection to the level of a hash value. I prefer hash values over “Name:PsExec.exe” combined with “Filesize:381816” because it doesn’t make you believe that you’re looking at a clever rule.
I therefore recommend the following:
I therefore decided to write an article on how to build optimal Yara rules, which can be used to scan single samples uploaded to a sandbox and whole file systems with a minimal chance of false positives.
These rules are based on contained strings and easy to comprehend. You do not need to understand the reverse engineering of executables and I decided to avoid the new Yara modules like “pe” which I still consider as “testing” features that may lead to memory leaks or other errors when used in practice.
First I believed that automatically generated rules can never be as good as manually created ones. During my work for out IOC scanners THOR and LOKI I had to create hundreds of Yara rules manually and it became clear that there is an obvious disadvantage. What I used to do was to extract UNICODE and ASCII strings from my samples by the following commands:
I prefer the UNICODE strings as they are often overlooked and less frequently changed within a certain malware/tool family. Make sure that you use UNICODE strings with the “wide” keyword and ASCII strings with the “ascii” keyword in your rules and use “fullword” if there is a word boundary before and after the string. The problem with this method is that you cannot decide if the string that is returned by the commands is unique for this malware or often used in goodware samples as well.
Look at the extracted strings in the following example:
Could you be sure that the string “NT LM 0.12” is a unique one, which is not used by legitimate software?
To accomplish this task for me I developed “yarGen“, a Yara rule generator that ships with a huge string database of common and benign software. I used the Windows system folder files of Windows 2003, Windows 7 and Windows 2008 R2 server, typical software like Microsoft Office, 7zip, Firefox, Chrome, Cygwin and various Antivirus solution program folders to generate the database. yarGen allows you to generate your own database or add folders with more goodware to the existing database.
yarGen extracts all ASCII and UNICODE strings from a sample and removes all strings that do also appear in the goodware string database. Then it evaluates and scores every string by using fuzzy regular expressions and the “Gibberish Detector” that allows yarGen to detect and prefer real language over character chains without meaning. The top 20 of the strings will be integrated in the resulting rule.
Let’s look at two examples from my work. A sample of the Enfal Trojan and a SMB Worm sample.
yarGen generates the following rule for the Enfal Trojan sample:
The resulting string set contains many useful strings but also random ASCII characters ($s9, $s11, $s12) that do match on the given sample but are less likely to produce the same result on other samples of the family.
yarGen generates the following rule for the SMB Worm sample:
The resulting rules are good enough to use them as they are, but they are far from an optimal solution. However it is good that so many strings have been found, which do not appear in the analyzed goodware samples.
If you don’t want to use or download yarGen, you could also use the online tool Yara Rule Generator provided by Joe Security, which was inspired by/based on yarGen.
It is not necessary to use a generator if your eye is trained and experienced. In this case just read the next section and select the strings to match the requirements of the (what I call) sufficiently generic Yara rules.
As I said in the introduction rules that generate false positives are pretty annoying. However the real tragedy is that most of the rules are far too specific to match on more than one sample and are therefore almost as useful as a file hash.
What I tend to do with the rules is to check all the strings and put them into at least 2 different categories:
Check out the modified rules in order to understand this splitting. Ignore the definition named $mz, I’ll explain it later and look at the string definitions below.
The definitions starting with $s contain the very specific strings, which I regard as so special that they would not appear in legitimate software. Note the typos in both strings: “Micorsoft Corportation” instead of “Microsoft Corporation” and “Monnitor” instead of “Monitor”.
The strings starting with $x seem to be special (I tend to google the strings) but I cannot say if they also appear in legitimate software. The definitions starting with $z seem to be ordinary but have not been part of the goodware string database so they have to be special in some way.
Now check the condition statement and notice that I combine the rules with a magic header of an executable defined by $mz and a file size to exclude typical false positives like Antivirus signature files, browser cache or dictionary files. Set an ample file size value to avoid false negatives. (e.g. samples between 100K and 200K => set file size < 300K)
You can see that I decided that a single occurrence of one of the very specific strings would trigger that rule. ( 1 of $s* )
Than I combine a bunch of less unique strings with most or all of the ordinary looking strings. ( 2 of $x* and all of $z* )
Let’s look at second example. (see below)
$s1 is a very special string with string formatting placeholders “%s” in combination with an Admin$ share. $s2 seems to be the typical “svchost.exe” but contains the number “0” instead of an “O”, which is very uncommon and a clear indicator for something malicious.
All the definitions starting with $a are special but I cannot say for sure if they won’t appear in legitimate software. The strings defined by $x seem ordinary but were produced by yarGen, which means that they did not appear in the goodware string database.
This special example contains a list of typical passwords which is defined by $z1..z8.
You see that I combined the string definitions in a similar way as before. This method in combination with the magic header and the file size should be a good starting point for the final stage – testing.
Testing the rules is very important. It seems that most authors decide that the rules are good enough if they match on the given samples.
You should definitely do the following checks:
To carry out the tests download the Yara scanner and run it from the command line. The goodware directory should include system files from various Windows versions, typical software and possible false positive sources (e.g. typical CMS software if you wrote Yara rules that match on malicious web shells)
Yara Rule Testing on Samples and Goodware
Make sure to check Part 2 of “How to Write Simple and Sound YARA Rules”.