Detecting Monero miners with bpftrace

Phillip Kuznetsov

February 17, 2022 • 7 minutes read

Lead SWE @ New Relic, Founding Engineer @ Pixie Labs

Cryptomining is expensive if you have to pay for the equipment and energy. But if you “borrow” those resources, cryptomining switches from marginal returns to entirely profit. This asymmetry is why cybercrime groups increasingly focus on cryptojacking – stealing compute time for the purpose of cryptomining – as part of malware deployments.

Despite a common misconception, most cryptocurrencies are not actually anonymous. If these cryptojackers were to mine Bitcoin or Ethereum, their transaction details would be open to the public, making it possible for law enforcement to track them down. Because of this, many cybercriminals opt to mine Monero: a privacy focused cryptocurrency that makes transactions confidential and untraceable.

In this article we’ll discuss the following:

• Existing methods for detecting cryptojackers
• How to leverage bpftrace to detect Monero miners

Detection scripts and test environment can be found in this repo.

• What happens during cryptomining?
• What signals can we detect?
• Monero mining signals
• Building our bpftrace script
  • What is bpftrace?
  • Test environment
  • Where should we trace?
  • What data do we need?

What happens during cryptomining and why is it important? This blog post by Anthony Albertorio provides more detail, but here's what's relevant:

Miners race to create the next block for the blockchain. The network rewards them with cryptocurrency when they submit a valid block. Each block contains the hash of the previous block (hence the “chain”), the list of transactions, and a Proof of Work (PoW) ¹. A miner wins when it successfully finds a valid Proof of Work for that list of transactions. The Bitcoin Proof of Work is a string that causes the entire block to hash to a bit-string with a “target” number of leading 0s.

Verifying the proof is computationally easy: you hash the block and verify that the bitstring matches the expected target. Finding the proof is difficult: the only way to discover it is by guessing. When a miner finds a proof, they broadcast the solution to the network of other miners, who quickly verify the solution. Once the solution is accepted, each miner updates their local copy of the blockchain and starts work on the next block.

Now that we know how cryptomining works, we can evaluate ways to detect cryptojackers. Note that no matter what we propose below, the landscape will shift and new techniques will be necessary. Attackers adapt to defenses and detections as they confront them in the field.

Many cryptojackers opt to use open-source mining software without modification. Scanning binaries running on the operating system for common mining software names and signatures of mining software is a simple yet effective barrier.

🟢 Pros: simple to implement, large surface area.

🔴 Cons: easy to bypass with obfuscation of code. Can also be hidden from tools like ps or top using libprocesshider.

Many cryptominers choose to contribute to a mining pool, which will require some outgoing network connection to a central location. You can make a blocklist of the top 100 cryptomining pools and block a large portion of miners.

🟢 Pros: simple to implement, large surface area

🔴 Cons: easy to bypass with proxies or by searching for allowed pools

Most miners opt for SSL which means reading the body of messages is impossible, but there are still signatures that exist for the network patterns. Michele Russo et al. collect network data on these traces and trained an ML classifier to discriminate between normal network patterns and cryptominer network patterns.

Because the miners must receive block updates from the rest of the network as well as updates from mining pools, they must rely on the network.

🟢 Pros: robust to proxies, miners are guaranteed to leave a trace due to dependence on the network.

🔴 Cons: large upfront investment to collect data and train models. Operational investment to update models with new data after discovery of new attacks. Risk of steganographic obfuscation or adversarial examples.

Similarly, you can collect data from hardware counters and train a model that discriminates between mining and not-mining use of CPU, GPU, etc., as discussed in Gangwal et al. and Tahir et al.

🟢 Pros: robust to binary obfuscation

🔴 Cons: large upfront investment to collect data and train models. Operational investment to update models with new data after discovery of new attacks. Risk of steganographic obfuscation or adversarial examples.

We mentioned earlier that cryptojackers opt to mine Monero because of the privacy guarantees. It turns out that Monero’s Proof of Work algorithm, RandomX, actually leaves behind a detectable trace.

RandomX adds a layer on top of the Bitcoin PoW. Instead of guessing the “proof string” directly, you need to find a “proof program” in the RandomX instruction set that outputs the “proof string” when run in the RandomX VM. Because every correct length bitstring is a valid program, Monero miners randomly generate "proof programs" and evaluate each in the RandomX VM.

These RandomX programs are easy to spot. They leverage a large set of CPU features, some of which are rarely used by other programs. The instruction set attempts to hit many features available on commodity CPUs. This design decision curtails the effectiveness of GPUS and ASICs, forcing miners to use CPUs.

One RandomX instruction in particular leaves behind a strong signal in the CPU. CFROUND changes the rounding mode for floating point operations. Other programs rarely set this mode. When they do, they rarely toggle this value as much as RandomX does. The main RandomX contributor, tevador, created randomx-sniffer which looks for programs that change the rounding-mode often on Windows machines. Nothing exists for Linux yet - but we can build this with bpftrace.

We want to detect traces of RandomX (the CPU-intensive mining function for Monero) running on a cluster. Specifically, we want to find the forensic trace of RandomX changing the floating-point rounding mode. We can do this with bpftrace.

bpftrace makes it easy to collect data about running Linux processes. bpftrace is a simple interface on top of eBPF, a Linux kernel technology that allows you to add operating system capabilities safely at runtime. We want to leverage bpftrace to collect data from running programs.

We specifically want a script that grabs information about the floating-point unit (FPU) configuration. The FPU configuration contains the rounding mode setting that Monero miners change often.

The scripts and environment setup instructions are available here.

The cluster: I deployed a Kubernetes cluster on a few machines running Linux Kernel 5.13, using x86 processors ².

The target: I deployed xmrig, a popular open source Monero miner to my cluster.

The bpftrace environment: You can use the bpftrace CLI directly on nodes. I chose to use Pixie instead because I wanted to deploy bpftrace to all the nodes on my cluster and leverage Pixie's data engine.

We need to find a probe accessible from bpftrace that stores information about the FPU. Where do we look? One option is to search for matching probes in the bpftrace CLI: bpftrace -l *fpu*. Unfortunately this leaves a large set of options (62 on my cluster) that can take a while to evaluate.

I had better luck searching for existing eBPF programs that attach to fpu probes. Intel's cri-resource-manager attaches to the x86_fpu_regs_deactivated tracepoint to track context switches of x86 CPUs. They accessed an fpu struct, grabbing a field called avx512_timestamp. That fpu struct felt like a good place to investigate for register values.

But before diving into the struct definition, let’s make sure the tracepoint actually collects data on our system. Here’s our first bpftrace script:

tracepoint:x86_fpu:x86_fpu_regs_deactivated
{
    printf("time_:%llu pid:%d comm:%s",
            nsecs, pid, comm);
}

It runs whenever the x86_fpu_regs_deactivated event triggers. For each event, the script outputs the time of the event, the pid of the triggering process and the name of the triggering process (aka comm).

We see a bunch of events, some of which come from the miner, xmrig.

Aggregating, we see that xmrig shows up near the top of the list, but it doesn’t show up alone and is definitely not the heaviest user of the FPU. We need to inspect the actual data in the x86_fpu_regs_deactivated probe to help us discriminate xmrig.

We're trying to detect a signature of the CFROUND instruction from RandomX. CFROUND changes the floating-point rounding mode for future floating-point operations. Most programs do not change this value so it gives us a strong signal of something fishy.

What happens when RandomX executes CFROUND? If you inspect the RandomX x86 implementation of CFROUND, you'll find the last instruction calls LDMXCSR.

This sets the MXCSR register in the FPU; the register is responsible for control and status values for Streaming SIMD Extensions. CFROUND sets the two rounding bits (with mask 0x6000) of MXCSR to the argument value. (See the x86 assembly for CFROUND).

Let’s try to find the MXCSR register in the fpu struct exposed by x86_fpu_regs_deactivated. What fields are available inside the fpu struct? First, I searched for the tracepoint definition, which led me to the x86/include/fpu namespace. Then I searched for the struct definition inside the namespace, which surfaced: asm/fpu/types.h. We’ll include this header in our bpftrace script. Finally, I searched in that file for mxcsr and found it's available via fpu->state.xsave.i387.mxcsr

Let's access this register value in bpftrace ³.

#include <asm/fpu/internal.h>
#include <asm/fpu/types.h>
tracepoint:x86_fpu:x86_fpu_regs_deactivated
{
    $f = (struct fpu *)args->fpu;
    $mxcsr = $f->state.xsave.i387.mxcsr;
    printf("time_:%llu pid:%d comm:%s mxcsr:%d",
            nsecs, pid, comm, $mxcsr);
}

Source

Running this, xmrig stands out from the rest based on the register value alone. We’re on the right track.

Now for the sinker: filtering by the rounding-bits from $mxcsr. We can find the rounding bits of the register by masking 0x6000 and shifting the bits towards the least significant bits. Then we filter out all non-zero values.

#include <asm/fpu/internal.h>
#include <asm/fpu/types.h>
tracepoint:x86_fpu:x86_fpu_regs_deactivated
{
    $f = (struct fpu *)args->fpu;
    $mxcsr = $f->state.xsave.i387.mxcsr;
    $fpcr = ($mxcsr & 0x6000) >> 13;
    if ($fpcr != 0) { 
        printf("time_:%llu pid:%d comm:%s fpcr:%d",
            nsecs, pid, comm, $fpcr);
    }
}

Source

And now we only see xmrig! Our detector successfully isolates the Monero miner running in the cluster.

We can then connect the process to the hosting Kubernetes pod using Pixie.

And there you have it - easily detect Monero mining on your cluster! If only it could be so simple. Like any other security tool, this mining detector is only one turn of the security cat and mouse game. ⁴

The best anti-cryptojacking solution will be a combination of defenses and detections using different modalities. That might mean combining this approach with the approaches listed earlier in the article or with other detectors written in bpftrace.

All the above code is available on Github. If you come up with some new Monero detectors or clever ways to use bpftrace, let us know on our Slack or tag us on Twitter @pixie_run.

Footnotes

---

Related posts

Come meet us at Kubecon EU 2022

Pixie contributors will be attending this year’s KubeCon + CloudNativeCon Europe…

Zain Asgar, Michelle Nguyen
May 13, 2022 • 3 minutes read

Pixie Team BlogsAnnouncementsKubernetes

Open Observability with the Pixie Plugin System

At Pixie, we believe that the future of observability lies in open source…

Michelle Nguyen
May 12, 2022 • 2 minutes read

Pixie Team BlogsAnnouncementsKubernetes

A brief stroll through the CNCF eBPF landscape

eBPF has been steadily gaining traction in the past few years. The foundation…

Hannah Troisi
Apr 19, 2022 • 4 minutes read

Pixie Team BlogseBPF