How we combined THOR Thunderstorm’s rule-based detection with LLM triage to reduce 114,000 daily artifacts to about 4,600 initial model reviews and roughly 100 analyst reviews, while reserving RuneAI for analyst-initiated deep analysis.

The problem: you can’t afford to ask an LLM about everything

The open-source software (OSS) ecosystem never sleeps. npm alone publishes well over a hundred thousand package versions per week. Add PyPI, crates.io, NuGet, browser extensions and VS Code extensions, and you are looking at more than 100,000 newly published artifacts every day (millions per month), each one a potential vector for supply-chain attacks, typosquatting, or malicious code injection.

Feeding every new artifact to an LLM and asking “is this malicious?” doesn’t work at this scale. The token cost would be prohibitive, and most of it would be wasted: the overwhelming majority of artifacts are benign, and spending model cycles on a straightforward utility library is a poor use of both compute and budget.
So the system is built as a funnel. Cheap, deterministic, rule-based scanning runs first across the whole feed and gates the expensive semantic analysis, which only ever sees the artifacts the rules flag. 

The numbers below show how much volume reaches each stage and what the initial LLM triage costs in practice.

The funnel in numbers

Across ten package registries and extension marketplaces, our pipeline scans an average of 114,000 newly published artifacts per day. Around 4% of them – approximately 4,618 – produce at least one THOR finding and enter the initial LLM triage stage.

The triage LLM resolves roughly 4,500 of those artifacts without human review. About 100 per day are classified as suspicious or remain inconclusive and are therefore escalated to an analyst. That means around 2.2% of the Stage 3 queue, or approximately 0.09% of the original feed, reaches human review.

The initial triage pass currently averages 20.9 million prompt tokens and $2.88 per day, roughly $0.025 per thousand artifacts scanned.

The pipeline: three automated stages, followed by selective analyst review

Our scanning architecture separates high-volume automation from human review. Indexing, THOR Thunderstorm scanning and LLM triage run automatically. Only suspicious or inconclusive findings – around 100 artifacts per day, or 0.09% of the original feed – reach an analyst. The analyst can invoke RuneAI on demand when additional files require deeper analysis.

Stage 1: Indexers

Indexers continuously monitor ten sources for newly published artifacts and collect metadata: name, version, publish time, download counts. No downloads yet, just lightweight polling.
Registries covered:

• npm
• PyPI
• crates.io
• NuGet
• OpenVSX and the Visual Studio Marketplace
• Chrome Web Store, Microsoft Edge Add-ons and Firefox Add-ons
• ClawHub (the public registry for OpenClaw skills and plugins)

Stage 2: THOR Thunderstorm scanning

Scanner workers download and extract each artifact, then submit its files to THOR Thunderstorm. THOR evaluates each submitted file against the detection logic applicable to its type, including YARA rules, IOCs and file-analysis modules. Sigma rules and additional parsers apply where relevant to the artifact format.

The rule set combines specific detections with generic signals for suspicious characteristics such as obfuscation, encoded payloads and unusual program behavior. Many of these signals deliberately carry low scores rather than making a binary malware decision. THOR combines the signals into a cumulative score and reports the result as a Notice, Warning or Alert.

For each relevant string match, we configure THOR to include 2,048 bytes of surrounding context. No LLM is involved at this stage.

Stage 3: Triage LLM

Only artifacts with THOR findings reach this stage. On average, approximately 4,618 artifacts per day enter LLM triage. The model receives the matched strings and their surrounding context, not the full artifact, and returns a structured assessment based on the findings, the artifact’s purpose and its metadata.

Most artifacts can be classified at this stage without human involvement. Around 100 artifacts per day are classified as suspicious or remain inconclusive and are escalated to an analyst. This represents approximately 2.2% of the artifacts entering Stage 3 and less than 0.1% of the original feed.

Stage 4: Analyst review

An analyst reviews the roughly 100 daily escalations, validates the available evidence and determines whether the artifact can be classified or requires additional analysis.

Optional: RuneAI deep review

When the available evidence is insufficient, the analyst can invoke RuneAI, our agentic analysis worker. RuneAI retrieves and inspects selected files from the artifact and returns additional evidence to the analyst.

RuneAI is not invoked automatically for every Stage 4 case. The deeper analysis is analyst-initiated and reserved for cases that cannot be resolved from the THOR match context alone.

Where the token savings come from

The efficiency is not one big win. It is several compounding filters, each reducing the volume that reaches the next, more expensive step.

The core insight: THOR doesn’t replace the LLM; it shrinks the haystack the LLM has to search. Generic rules cannot determine intent, but they are very good at flagging anomalies worth a second look. That is exactly the right job for a fast, deterministic system.

What the initial triage pass actually costs

The initial triage step, where the triage LLM receives THOR match context and surrounding strings rather than full artifacts, has run in production since early May 2026. The figures below are daily averages across the past month of live traffic.

The $2.88 figure covers the initial LLM triage stage only. It excludes any RuneAI analysis initiated by an analyst after escalation.

The initial LLM triage costs under $3 per day while the pipeline scans approximately 114,000 artifacts across ten sources. Of the roughly 4,618 artifacts entering LLM triage, about 100 require analyst review. In other words, the LLM resolves approximately 98% of the THOR-matched artifacts without human involvement, and less than 0.1% of the original feed reaches an analyst.

The triage stage is designed to return a short, structured classification. It includes a verdict, a confidence value and a recommendation on whether deeper analysis is needed. Suspicious or low-confidence results are flagged for an analyst.

Why rule-based prefiltering is a better operational fit at this scale

Speed and cost are not the only reasons to put rules first.

THOR’s rule set has been refined over years of malware research. Many signatures encode patterns that recur across unrelated campaigns: obfuscation techniques, persistence mechanisms, command-and-control patterns. These patterns can surface previously unseen malware even when no sample-specific rule exists.

An LLM handed a raw artifact with no prior filtering faces a different problem: it has to decide what to pay attention to across thousands of lines, and it can miss subtle anomalies buried in boilerplate. THOR evaluates every submitted file deterministically against the applicable detection logic, rather than relying on a model to decide which parts look interesting.

Where the LLM earns its place is judgment about context and intent. A base64-encoded string in a test fixture is not the same threat as a base64 blob that is decoded and executed at install time, but a YARA rule flags both. Given the match context and the artifact’s stated purpose, the model can separate many of those cases at far lower cost than sending every match directly to human review.

The surrounding context is often rich enough to settle the question on its own. A classic example: a base64 blob triggers a rule, but the surrounding bytes show it sitting inside a //# sourceMappingURL=data:application/json;base64,... comment. That is a standard source map emitted by a bundler. The model can mark it as a likely false positive and move on, with no escalation.

The combination gives us a practical balance of deterministic coverage, contextual analysis and cost.

Worked example: a real detection from the pipeline

In November 2025 the pipeline caught a malicious VS Code extension in the Visual Studio Marketplace. The extension appeared to impersonate the popular Material Icon Theme by Philipp Kief: a similar name and the same visual presentation, but a different publisher.

The listing looked plausible at a glance. The decisive signal was in the files. The signal was in the bytes. Version 5.29.0 was clean; the publisher then pushed 5.29.1, and that release bundled two Rust implants, one Windows PE and one macOS Mach-O, under extension/dist/extension/desktop/. The Mach-O carried a user-path string matching the GlassWorm samples reported around the same time.

How it moved through the pipeline:

1. Indexer picked up the new extension version shortly after it was published.

2. Scanner unpacked it and submitted the files to THOR Thunderstorm. The implants did not match a signature written for this specific sample.
   They tripped generic rules: SUSP_Implant_Indicators_Jul24_1 and SUSP_HKTL_Gen_Pattern_Feb25_2, signatures built to recognize implant and hacktool tradecraft regardless of campaign. That is the case for rule-based prefiltering in one detection: the malware was novel, but the tradecraft was not.

3. Triage and review. THOR’s hits crossed the review threshold and were confirmed by an analyst, who identified the impersonation, the bundled cross-platform implants, and the GlassWorm overlap. The extension was reported to Microsoft. In today’s pipeline the triage LLM sits at exactly this point, taking THOR’s match context first and classifying or prioritizing each hit before it reaches a human, with RuneAI available on demand for the binary deep-dive an analyst would otherwise do entirely by hand.

The point this illustrates: a metadata-only check would have seen a popular, normal-looking icon theme. The deterministic byte-level scan is what surfaced the implants, and the rules that caught them were generic, not sample-specific.

This was not an isolated catch. To date, the pipeline has surfaced and reported more than 100 malicious artifacts across npm, the Visual Studio Marketplace, Open VSX and PyPI. We post individual findings as we confirm and report them, on our Nextron Research account at x.com/nextronresearch.

Limitations and validation

The pre-filter is a deliberate design choice, but it creates a clear limitation: a malicious artifact that matches no configured THOR rule does not reach the LLM. Deterministic prefiltering therefore trades semantic completeness for throughput.

To date, the pipeline has surfaced and reported more than 100 malicious packages across the monitored ecosystems. This demonstrates that the rule pre-filter produces valuable results, but it does not quantify how many malicious artifacts may pass through without a finding.

The broader architecture in practice

Running across ten sources at once requires the indexers to be fast and resilient. Registry APIs rate-limit, and artifacts can be yanked between discovery and download. The scanners run as a distributed fleet, pulling artifacts in parallel and submitting their files concurrently to THOR Thunderstorm through its asynchronous API. THOR Thunderstorm is built for exactly this kind of high-throughput, API-driven scanning, which makes it a natural workhorse at the center of the pipeline.

The triage tier is where the automated intelligence concentrates. The triage LLM manages the assessment, judges when the match context is sufficient, and flags low-confidence findings for an analyst rather than resolving them itself. When deeper analysis is needed after an escalation, the analyst can invoke RuneAI on demand. RuneAI retrieves and examines selected files autonomously rather than running on every artifact.

Here’s an example analysis by Rune AI:

The triage tier also keeps state across an artifact’s findings: when several files in the same artifact produce matches, the model sees them together rather than in isolation, which provides artifact-level context and avoids redundant calls.

The practical result is continuous coverage across more than 100,000 daily artifact publications, with rule-based scanning keeping pace with the registries and LLM cost staying proportional to the findings, not to the total volume.

What this means for OSS supply-chain security

Supply-chain attacks have become a recurring way to compromise developer environments and downstream infrastructure. A malicious package on npm or PyPI can spread quickly into downstream projects before anyone notices, so speed of detection matters.

A pipeline that only looks after a human tip-off, or that runs weekly batch scans, can miss that window. This architecture operates continuously, at the pace of the registries: an artifact enters the indexer within minutes of publication, and if THOR flags anything, triage follows automatically.

Cost efficiency is what makes continuous full-feed rule-based scanning practical. Deep LLM analysis of every artifact would either incur much higher costs or force the semantic layer to sample. Putting rules first lets us scan the complete feed deterministically while reserving model attention for artifacts that produce findings.