Beyond the Packer Myth: Architectural EDR Evasion and Durable Telemetry

The persistence of the “just run it through a new packer” mindset among entry-level practitioners misrepresents the modern state of endpoint detection and response. For years, security tools relied heavily on static signatures, entry-point scanning, and basic string matching. When EDRs shifted toward dynamic runtime monitoring, they frequently implemented this visibility through userland API hooking—patching the import tables or function preludes of sensitive system DLLs like ntdll.dll inside the target process space. This architectural choice created a temporary illusion of control, but it ultimately built detection logic on top of memory that the executing context inherently owned.

If an attacker controls the process context, relying on that same context to report its own malicious behavior is a fundamental design flaw. Understanding why userland evasion techniques work requires examining the execution boundary, and understanding how to defend against them requires moving visibility entirely outside the attacker’s sphere of control.

The Userland Illusion: Why Ring 3 Hooks Were Always Borrowed Time

When Windows executes an operation—whether allocating virtual memory, creating a thread, or writing to a process—the application invoking the action eventually calls an API within kernel32.dll or kernelbase.dll. These higher-level APIs wrap native system services exported by ntdll.dll. The native functions setup the system call number in a register (such as EAX on x64) and execute a syscall instruction, transitioning execution from Ring 3 (user mode) to Ring 0 (kernel mode).

Early EDR architectures inserted 5-byte relative jumps (JMP) at the beginning of exported functions in ntdll.dll within every newly spawned process. When an application attempted to call NtMapViewOfSection or NtAllocateVirtualMemory, execution was immediately redirected to the EDR’s injected monitoring dynamic link library. The security DLL inspected the arguments, evaluated heuristics, and either allowed the execution to continue or terminated the process.

This mechanism contained a fatal assumption: that the memory space of ntdll.dll within a userland process was immutable or protected from the process itself. Because the process owns its virtual memory space, the process possesses the permission to modify its own memory. An attacker executing within that process context can alter permissions on ntdll.dll using standard memory protection routines, read a clean copy of ntdll.dll from disk or a freshly spawned suspended process, and overwrite the modified hooks with the original bytes. Once the hooks are removed—a technique known as unhooking—the process regains direct, unmonitored access to the operating system’s native API interface.

The Mechanics of Bypass: Direct Syscalls and Call Stack Spoofing

As unhooking became well-understood, offensive research evolved away from modifying memory in place toward bypassing the userland DLL altogether. Because ntdll.dll merely acts as a trampoline that populates registers and invokes the syscall instruction, an operator does not strictly need to call ntdll.dll at all.

Direct syscall implementation involves embedding assembly instructions directly within the payload executable to load the appropriate System Service Descriptor Table (SSDT) index and execute syscall directly from the payload’s memory space. By skipping ntdll.dll completely, execution never passes through userland jump instructions, rendering hook-based monitoring entirely blind to the invocation.

When defenders reacted by inspecting the source location of system calls or monitoring for anomalous executable memory pages issuing syscall instructions, the offensive methodology refined further into indirect syscalls. In an indirect syscall pattern, the payload manually sets up the requisite registers and stack parameters, but instead of executing the syscall instruction from its own memory region, it jumps to a syscall; ret instruction gadget located within the legitimate, signed memory region of ntdll.dll. To an observer looking solely at the instruction pointer during the kernel transition, the call appears to originate from legitimate system code.

This dynamic created an escalating operational complexity. To distinguish legitimate indirect syscalls from malicious ones, security teams had to look beyond the execution point and analyze the complete call stack. However, userland stack frames can also be manipulated or unwound artificially using return-oriented programming techniques or custom frame-spoofing logic, proving once again that telemetry gathered or parsed entirely within Ring 3 is subject to manipulation by an adversary operating at the same privilege tier.

Why Signatures and Heuristics Keep Failing

The persistent failure of traditional detection paradigms stems from confusing artifacts with behavior. A signature detects a specific representation of code at rest or a specific sequence of bytes in memory. A heuristic attempts to identify known-bad patterns derived from previously observed attacks. Both models operate on the assumption that the attacker will use recognizable tools or standard compilation output.

When an adversary implements direct or indirect syscalls, custom memory allocators, or modified call stacks, they are not necessarily changing the core intent of their actions—they are stripping away the predictable artifacts that defenders rely on for easy identification. A payload allocating executable memory and writing shellcode into a remote process performs the exact same kernel-level state transitions regardless of whether it was compiled with a commercial C2 framework, written in raw assembly, or packed inside a custom loader.

Attempting to catch these actions by writing signatures for each iteration of an obfuscation tool creates a perpetual reactive cycle. Every time an offensive research group publishes a new method for dynamically resolving syscall numbers (such as reading SSDT indices sequentially or calculating them via relative addresses), defenders release detection rules targeting those specific memory structures or code patterns. Within days, the pattern changes, the signature breaks, and the underlying capability remains intact.

Durable Telemetry: Shifting the Gate to Kernel Callbacks and ETW-TI

Durable defense requires shifting the point of observation to a boundary the userland process cannot alter or bypass: the kernel. Regardless of how cleverly an adversary handles execution within Ring 3—whether using direct syscalls, indirect syscalls, or unhooked DLLs—the execution context must ultimately pass through the kernel to interact with hardware, memory, disk, or network resources.

Windows provides several built-in mechanisms that allow security software operating at Ring 0 to collect immutable telemetry:

1. Kernel Executive Callbacks: Registers such as ObRegisterCallbacks, PsSetCreateProcessNotifyRoutineEx, and PsSetCreateThreadNotifyRoutineEx allow driver-level security software to be notified synchronously whenever handle operations, process creation, or thread creation occur. Because these callbacks execute within kernel mode before or immediately after the state change occurs, a userland process cannot unhook them or modify their logic.

2. Event Tracing for Windows Threat Intelligence (ETW-TI): ETW-TI is a specialized, kernel-level tracing provider logged directly from the kernel executive routines (such as NtMapViewOfSection or NtWriteVirtualMemory). When a process attempts to allocate, protect, or write memory into another process, the kernel itself generates an ETW-TI log event. This event contains the target process, source process, allocation size, and permissions requested. Because this log is generated inside the kernel routine, bypassing userland hooks in ntdll.dll has zero impact on the generation of the ETW-TI event.

3. Hypervisor-Protected Code Integrity (HVCI) and Virtualization-Based Security (VBS): By leveraging hardware virtualization, the OS isolates sensitive security operations and kernel memory from even Ring 0 driver compromises, ensuring that the integrity of the underlying telemetry collection mechanisms remains verified.

When telemetry is collected at the kernel layer, the “how” of the userland call becomes secondary to the “what” of the OS request. If a process writes executable bytes into a remote process address space, a kernel driver receives notification of the handle access and the memory modification regardless of whether the userland call was executed via standard APIs, raw assembly syscalls, or an unhooked library.

The New Playing Field: Detecting the Anomalies of Evasion Itself

While kernel-level telemetry strips away the invisibility granted by userland evasion, it introduces a new paradigm for detection engineering: detecting the structural anomalies generated by the evasion techniques themselves.

Evasion techniques leave distinct operational signatures when viewed against normal system baselines:

• Call Stack Inconsistencies: When thread execution is intercepted via kernel callbacks or thread sampling, the userland call stack is unwound. Legitimate system API calls follow predictable unwinding paths through kernelbase.dll down to ntdll.dll. An indirect syscall that manually jumps into ntdll.dll without a valid preceding call frame creates a disconnected or truncated stack trace.
• Unbacked Executable Memory: Modern Windows applications load executable code from signed binaries on disk, resulting in memory pages backed by image files. Payloads running direct syscalls or reflectively loaded code often reside within private, unbacked memory allocations (MEM_PRIVATE with PAGE_EXECUTE_READWRITE or PAGE_EXECUTE_READ). Observing a thread issuing kernel requests from unbacked memory is inherently anomalous in enterprise environments.
• Kernel Object Handle Auditing: Evasion frameworks must obtain process or thread handles with high-privilege access masks (such as PROCESS_VM_WRITE or PROCESS_VM_OPERATION) to perform injection. Kernel callbacks can evaluate whether a requested handle mask matches the expected profile of the requesting application, flagging suspicious requests regardless of the userland source.

The objective of modern detection engineering is not to prevent an adversary from executing a syscall or modifying their own memory. The objective is to build detection logic on top of kernel-enforced, unalterable telemetry primitives where the act of attempting to evade visibility inherently generates an operational anomaly that stands out against legitimate system activity.