LLDBagility: practical macOS kernel debugging

Virtual machine introspection via the Fast Debugging Protocol

The Fast Debugging Protocol [4] is a light, fast and efficient debugging API for stealthy introspection of virtual machines, written in C and currently only released as a patch to VirtualBox's source code [FDP]. This API makes it possible to:

• read and write VMs registers and memory;

• set and unset hardware/software breakpoints and watchpoints;

• pause, resume and single-step the execution of the virtual machines;

• save and restore the VMs state.

FDP's stealthiness comes from the manipulation of the Extended Page Tables of the virtual machines (operation that can be very difficult to detect from the guests), while speed is the result of using shared memory between FDP and the VMs (reaching in some cases one million of memory reads per second).

In addition to the low-level C interface, the API also provides Python bindings (PyFDP) that can be useful both for quick proof of concepts and for bigger projects like LLDBagility; as an example, here is a script that breaks on every system call:

#!/usr/bin/env python2
from PyFDP.FDP import FDP

# Open the debuggee VM by its name
fdp = FDP("18E226")

# Pause the VM to install a breakpoint
fdp.Pause()
# Install a hardware breakpoint (very fast and stealth)
UnixSyscall64 = 0xffffff80115bae84
fdp.dr0 = UnixSyscall64
fdp.dr7 = 0x403
# Resume the execution after the breakpoint installation
fdp.Resume()

while True:
    # Wait for a breakpoint hit
    fdp.WaitForStateChanged()
    print "%x" % (fdp.rax)
    # Jump over the breakpoint
    fdp.SingleStep()
    # Resume the debuggee
    fdp.Resume()

More details about FDP can be found in the original article on Winbagility [4] (in French) [WinbagilitySSTIC2016], LLDBagility's older counterpart for Windows and WinDbg.

Attaching LLDB to virtual machines

As presented at length in the previous post, during regular two-machine debugging, LLDB interacts with the macOS kernel of the debuggee by sending commands to its internal KDP stub, which, being itself part of the kernel, is then able to inspect and alter the state of the machine as requested and communicate back the results. The key idea behind LLDBagility is to replicate the functionalities provided by such agent using the analogous capabilities of (virtual) machine introspection and modification offered by FDP at the hypervisor level, so that the kernel's debugging stub can be replaced by an equivalent alternative external to the debugged machine. In addition, by maintaining compatibility with the KDP protocol, this new stub can take advantage of LLDB's existing support for the macOS kernel without modifying the debugger in any aspect, since there's no difference in communications with respect to the KDP server in the kernel. In this regard, Ian Beer used an analogous solution for his iOS kernel debugger [5].

This approach brings in two notable advantages: the elimination of the necessity of enabling KDP in the kernel for debugging, and the possibility of augmenting LLDBagility's stub with additional features: all the limitations of the classic debugging method described in the first blog post can now be overcome. First of all, since setting up XNU for debugging is not required anymore, it becomes unnecessary to modify the NVRAM and possibly disabling SIP; likewise, there is also no need to install DEBUG or DEVELOPMENT builds. Secondly, all system-wide side effects related to KDP code disappear, implying for example a more difficult debugger detection by rootkits. Thirdly, debugging can now start before the initialisation of the KDP stub during the kernel boot process. Finally, having complete control over the machine thanks to FDP makes it easy to implement hardware breakpoints, watchpoints and a mechanism for pausing the kernel at debugger's command.

Overview of the tool

As mentioned in the introduction, LLDBagility's front end is a set of new LLDB commands, implemented in lldbagility.py (from [LLDBagility100]):

• fdp-attach or simply fa, to connect the debugger to a running macOS VirtualBox virtual machine;

• fdp-hbreakpoint or fh, to set and unset read/write/execute hardware breakpoints;

• fdp-interrupt or fi, to pause the execution of the VM and return the control to the debugger (equivalent to the known sudo dtrace -w -n "BEGIN { breakpoint(); }");

• fdp-save or fs, to save the current state of the VM;

• fdp-restore or fr, to restore the VM to the last saved state.

On the other hand, the back end is composed of:

• kdpserver.py, the new KDP server (replacement of XNU's) to which LLDB is connected, whose job is to translate the KDP requests received by the debugger into commands for the VM and send back responses and exception notifications (e.g. breakpoint hits);

• stubvm.py, for abstracting common VM operations and actualising them with FDP.

The connection between LLDB and the virtual machine to debug is established on fdp-attach. When the user executes this command, LLDBagility:

• instantiates a PyFDP client for interacting with the VM (lldbagility.py#L73);

• starts its own KDP server on a secondary thread (lldbagility.py#L93), waiting for requests (kdpserver.py#L182);

• executes the LLDB command kdp-remote to connect the debugger to the KDP server that has just started (lldbagility.py#L99, procedure explained in detail in the first blog post), kicking off the debugging process.

As a last interesting note, the translation of KDP requests into corresponding FDP commands is not sufficient to have lldbmacros working correctly. In fact, such macros rely on the ability to use some fields of the kdp struct (osfmk/kdp/kdp_internal.h#L41/#L55, tools/lldbmacros/core/operating_system.py#L777 both from [XNU49032212]), populated by the kernel only when its KDP stub is enabled (e.g. osfmk/kdp/kdp_udp.c#L1349). LLDBagility obviates the problem by hooking memory reads from the debugger to the kdp struct and returning a patched memory chunk in which necessary fields (e.g. kdp_thread) are filled with proper values (stubvm.py#L242).

Reusing debug information between kernels

In this last section we explore some ideas about debugging kernels with lldbmacros coming from a Kernel Debug Kit released for a different kernel build. The methods proposed here have their obvious limitations, but they also have been proven to work sufficiently well and are worth a shot; see KDKutils/ for code and examples.

As mentioned in the first post, the absence of KDKs for most kernel builds makes debugging more difficult, since in such cases types information and lldbmacros are not available. Luckily, the debug information required by these macros to work correctly comprises a relatively small number of global variables and structs, like version (config/version.c#L40) and kdp, whose definitions do not change much (or at all) from one kernel version to the next. In this light, it seems reasonable to try to reuse debug information from available KDKs so that most features of existing lldbmacros can be used with kernels lacking their own debug kit. As an example, let us assume we have debug information (i.e. the DWARF file generated during compilation) and lldbmacros for kernel build A, but not for build B. With the goal of loading and using A's lldbmacros for debugging B in LLDB, it is possible to provide the debugger with a "fake" DWARF file for kernel B, created by patching A's in such a way that: its Mach-O UUID matches B's instead of A's, and the load addresses of all symbols used by the macros match the virtual load addresses of the same symbols in B's memory at run time. These two small changes should be enough to have most macros working.

The simple approach just discussed works well when the symbols used by lldbmacros have very similar definitions in the code of the two kernels, which is naturally the case when they are successive releases. When this does not happen (for example when a field is added or removed from the thread struct), then the workaround above is no longer sufficient, since it becomes necessary to also modify the DWARF file to alter types definitions; but this is not easy. For such cases, it is possible and better to create the fake DWARF from scratch, by first parsing a source DWARF file to extract types information about the structures used by lldbmacros, and then generating C source files containing the types definitions. These sources can easily be edited by hand to modify types as desired and then compiled so that a new DWARF file is created. Before loading it in LLDB, the DWARF file must be patched as above to change its UUID and load address to match the kernel being debugged.

Wrapping up

With this post, we presented LLDBagility, a convenient alternative to classic macOS kernel debugging based on virtual machine introspection. While not perfect (yet), our solution sets aside or greatly alleviates all current limitations of the standard approach, and at the same time offers powerful additional features that make debugging much more practical. LLDBagility has been actively used at Quarkslab in the past months and it is being released for others to experiment with; feedback via GitHub is welcome and encouraged. Happy debugging!

References and footnotes