devirt.re
Create account
Menu

Devirtualization methods

Devirtualization is the recovery of program logic from code that a software protector has translated into bytecode for its own virtual machine. Rolf Rolles published an approach to unpacking virtualization obfuscators in 2009, and this hub groups the methods published since then into five families: what each one does, what its output can prove, and where it stops. Every Devirt result reports each protected function in one of four proof states, and the proof section explains what each state means.

4 research tracks · Updated

Research policyDated · Sources linked · Limits stated

Published research and permission-cleared samples only. No customer binaries.

What is devirtualization in reverse engineering?

Devirtualization in reverse engineering is the translation of virtualized code back into a form an analyst can read, such as native instructions, an intermediate representation like LLVM IR, or decompiled pseudocode. Protectors such as VMProtect, Themida and Code Virtualizer replace selected functions with bytecode and an interpreter, so a disassembler shows the interpreter instead of the function. Devirtualization reverses that step for the functions in scope.

Devirtualization does not recover original source code. The names, comments, types and file structure that compilation discarded were gone before any protector touched the program, so a devirtualized function is recovered logic, not the developer's text. The term is also unrelated to the compiler optimization called devirtualization, which turns C++ and Java virtual calls into direct calls.

Devirtualization is also a different job from unpacking. A packer compresses or encrypts code and restores it in memory when the program starts, while a virtualizing protector never restores the original instructions, which is why an unpacked file can still contain unreadable virtualized functions.

How does virtualization obfuscation work?

Virtualization obfuscation, also called code virtualization or VM-based obfuscation, is a protection technique that translates selected native functions into bytecode for a custom instruction set and embeds an interpreter that executes that bytecode at run time. Five parts make up a virtualized function:

  1. VM entry. Saves the native registers and switches execution into the virtual machine.
  2. Dispatcher. Fetches the virtual instruction at the virtual program counter, decodes it and selects the handler that implements it.
  3. Handlers. Small routines that each carry out one virtual operation, such as an addition or a memory load.
  4. Virtual context. The virtual registers or stack that the handlers read and write.
  5. VM exit. Restores the native registers and returns control to ordinary code.
Diagram: a virtualized function's VM entry, dispatcher, handlers, virtual context and VM exit, with the dispatcher selecting a handler for each instruction.
Virtualization obfuscation replaces a function with bytecode and an interpreter. The VM entry saves the native registers, and the dispatcher fetches and decodes each virtual instruction, then selects the handler that implements it. The handlers read and write the virtual context, and the VM exit restores the registers and returns control to native code.

Virtualizing protectors generate the virtual instruction set when they protect a program, so the same handler layout rarely carries over from one protected build to the next. Oreans states that Code Virtualizer's virtual opcodes and virtual machine are unique for every protected application.

Many protectors add further layers, such as duplicated handlers, opaque predicates and virtual machines nested inside other virtual machines, all covered in a 2021 systematization of the field. Another is threaded dispatch, in which each handler decodes the next instruction and jumps to its handler without returning to a central loop, which Kim, Lee and Cho describe in a 2026 study of commercial protectors.

Protector vendors warn that virtualized code runs significantly more slowly than native code, so developers usually virtualize only selected functions, such as license checks or proprietary logic, and leave the rest of the program native. VMProtect research and Themida research show how two commercial protectors apply these ideas.

What are the main devirtualization methods?

Devirtualization methods, or techniques, fall into five families: handler analysis and bytecode translation, trace simplification, symbolic execution, program synthesis, and lifting to an intermediate representation followed by compiler optimization. Published tools often combine several families. A 2021 systematization by Kochberger, Schrittwieser, Schweighofer, Kieseberg and Weippl found that automatic deobfuscation of virtualization-protected code is possible in specific scenarios, that existing methods have clear limitations, and that multiple challenges remain.

Diagram: five devirtualization method families, from handler analysis and trace simplification to symbolic execution, program synthesis and LLVM lifting.
Devirtualization methods fall into five families, and each one works from a different starting point: the interpreter's handlers, an execution trace, symbolic inputs, sampled inputs and outputs, or code lifted to an intermediate representation such as LLVM IR.

Handler analysis and bytecode translation

Handler analysis and bytecode translation work from the outside in: the analyst first reverse engineers the interpreter, then uses what each handler does to translate the bytecode program. Rolf Rolles described this route at the 2009 USENIX Workshop on Offensive Technologies and applied it to VMProtect: generate a disassembler for the virtual instruction set, translate the bytecode into an intermediate language, then optimize the result and compile it back into x86 machine code. Bytecode translation covers the whole bytecode program at once, but its output is only as correct as the handler semantics behind it, and each new virtual machine design needs the handler analysis again.

Trace simplification of virtualized code

Trace simplification works from the inside out: it runs the protected code, records an execution trace, and keeps only the instructions that influence the result. Coogan, Lu and Debray (ACM CCS 2011) kept the instructions that affect a program's observable behavior, such as its system calls, without first reverse engineering the interpreter. Yadegari, Johannesmeyer, Whitely and Debray (IEEE Symposium on Security and Privacy 2015) combined taint analysis with semantics-preserving transformations, rebuilt a control-flow graph from the simplified trace, and applied one generic approach to emulation-based obfuscation and to return-oriented programming. A simplified trace is exact for the inputs that produced it, but trace simplification cannot recover code on paths the trace never took.

Symbolic execution on virtualized code

Symbolic execution replaces concrete input values with symbols and expresses what each path computes as formulas over those inputs, which a solver can simplify or compare. Salwan, Bardin and Potet (DIMVA 2018) combined taint analysis, symbolic execution and path exploration to recover hash functions protected with the Tigress virtualizer, then translated the result into LLVM IR and recompiled it without the virtual machine. Symbolic execution covers every input on an explored path at once, but the number of paths grows with every branch, a problem called path explosion, and heavily obfuscated arithmetic can make the formulas too large to solve.

Program synthesis of handler semantics

Program synthesis treats obfuscated code as a black box: it samples inputs and outputs, then searches for a simple expression that behaves the same way. Syntia, presented by Blazytko, Contag, Aschermann and Holz at USENIX Security 2017, split execution traces into windows and used program synthesis guided by Monte Carlo Tree Search to learn what each window computes; it synthesized the semantics of arithmetic instruction handlers in VMProtect and Themida with a success rate above 94 percent. Program synthesis ignores how a handler is written, which makes it resistant to obfuscated handler code, but it is limited by how complex the underlying semantics are, and a synthesized expression is a candidate that matches the samples, not a proof.

Lifting virtualized code to an intermediate representation

Lifting translates machine code into an intermediate representation such as LLVM IR, where standard compiler optimizations, including constant folding and dead code elimination, can strip the interpreter away once the bytecode it executes is known, leaving the computation it performed. Lifting libraries such as remill translate machine code into LLVM IR, and a 2024 article on Thalium's research blog combined a taint-guided execution trace with LLVM optimization to recover virtualized functions. Lifting is only as complete as the code it receives: optimization preserves meaning, but it cannot add paths that were never lifted.

What is the difference between static and dynamic devirtualization?

In reverse engineering, static devirtualization analyzes a protected binary without running it, and dynamic devirtualization works from what the protected code does when it runs. Static analysis can examine decoded bytecode beyond the paths visited in a single execution, subject to analysis limits. It depends on recognizing the interpreter, which gets harder when handlers are encrypted, duplicated or threaded. Dynamic analysis sees exactly what executed, including bytecode that was decrypted at run time, but only for the inputs used.

Static and dynamic devirtualization compared
ApproachWhat it coversWhere it breaks
StaticPaths in decoded bytecode, analyzed without running the program and subject to analysis limitsEncrypted, duplicated or threaded handlers, and interpreters it cannot recognize
DynamicExactly what ran, including bytecode decrypted at run time, but only for the inputs usedPaths no input reached, and code that detects analysis and changes its behavior

Static devirtualization leans on handler analysis, bytecode translation, static symbolic execution and lifting. Dynamic devirtualization leans on tracing, taint analysis, trace simplification and the samples program synthesis needs. Many published approaches combine the two, and the 2021 systematization treats symbolic execution on an execution trace as the typical combination. VMHunt (Xu, Ming, Fu and Wu, ACM CCS 2018) finds the boundaries of virtualized code in an execution trace and simplifies the code inside them with symbolic execution, and Salwan, Bardin and Potet explore further paths from a first trace to cover more of each function.

Binary analysis research compares static and dynamic analysis for ordinary binaries. The difference for virtualized code is what the interpreter hides from each of them.

What each devirtualization method can prove

Each devirtualization method family produces a different kind of evidence, and that evidence decides what a recovered function can be trusted for.

What each devirtualization method can prove
Method familyWhat the output can showWhat it cannot show on its own
Handler analysis and bytecode translationThe whole bytecode program, translated through the handler semantics the analysis establishedThat every handler was understood correctly
Trace simplificationExactly what the code did for the traced inputsBehavior on paths the trace did not take
Symbolic executionWhat each explored path computes, for every input on that pathPaths that were not explored, or formulas too large to solve
Program synthesisA simple expression that matches the sampled behaviorEquivalence, until the expression is checked
Lifting and compiler optimizationThe computation with the interpreter removed, as precise as the lifted inputPaths that were never lifted

No single method family proves everything, which is why a devirtualized result has to state, function by function, what was shown and under which assumptions.

Why does devirtualization fail on some functions?

Devirtualization fails on a function when the protection hides information that recovery needs, or when the function's behavior depends on conditions no analysis observed. Six causes recur in published work:

  • Single-path traces. A trace records one execution, so branches it never took stay unrecovered.
  • Path explosion. Symbolic execution has to follow every branch, and the number of paths multiplies with each one.
  • Per-build diversity. Generated instruction sets and duplicated handlers defeat pattern matching and tools tuned to one virtual machine design.
  • Mixed boolean-arithmetic. MBA expressions rewrite simple arithmetic as combinations of arithmetic and bitwise operations that resist simplification.
  • Threaded dispatch. Without a central dispatcher loop, methods that begin by locating that loop have nothing to locate.
  • Runtime-dependent behavior. Self-modifying code, dynamic stack allocation and input from the environment make parts of a function depend on state that recovery never saw.

The Thalium write-up lists single-path traces among its own limits, and Kim, Lee and Cho note that threaded dispatch in current commercial protectors neutralizes analyses built around the central loop.

Obfuscators are designed against devirtualization methods as well. The Loki obfuscator, presented by Schloegel, Blazytko, Contag, Aschermann, Basler, Holz and Abbasi at USENIX Security 2022, was built to resist automated attacks such as symbolic execution, taint analysis and program synthesis. The Loki VM devirtualization page explains why every Loki-derived binary, including Obfuscator X and Griffin, requires its own free assessment before any engagement. These are not automated profiles. A Devirt result reports functions that fail for these reasons as runtime-dependent, unresolved or bounded, and keeps them in the count.

Devirtualization methods research tracks

Each track lists the articles in preparation. Articles publish when the evidence exists, so the list is direction, not a schedule.

Methods in depth

Each method family on its own, with its assumptions and its evidence stated.

In preparation

  • Bytecode devirtualization explained
  • Binary lifting for virtualized code
  • Lifting protected code to LLVM: what it does and does not prove
  • Symbolic execution in devirtualization
  • Recovering control flow from virtualized code

Limits and failure modes

Where recovery stops, shown on reproducible samples built for the purpose.

In preparation

  • Self-modifying code and devirtualization
  • Loki VM obfuscation: how it works and what gets recovered
  • CodeDefender and native obfuscation
  • Building safe test fixtures with Tigress

How we publish methods research.

Every methods article is dated and versioned, links the published work it relies on and states the assumptions behind each claim. Samples are permission-cleared or built for the purpose, and customer files are never published. Every result separates what was proven from what stayed runtime-dependent, unresolved or bounded. Failures get published too.

The same rules apply across Devirt Research.