https://www.moritz.systems/blog/improving-gdb-register-model-compatibility-in-lldb/
Moritz Systems
* Articles
* BSD Tips & Tricks
* Services
* About us
* Careers
* Contact
ARTICLES
Improving GDB register model compatibility in LLDB
By Michal Gorny
September 21, 2021 - 31 minutes read - 6412 words
BSD contract debugger FreeBSD GDB LLDB LLVM
Moritz Systems have been contracted by the FreeBSD Foundation to
continue our work on modernizing the LLDB debugger's support for
FreeBSD.
The primary goal of our contract is to bring kernel debugging into
LLDB. The complete Project Schedule is divided into six milestones,
each taking approximately one month:
1. Improve LLDB compatibility with the GDB protocol: fix LLDB
implementation errors, implement missing packets, except
registers.
2. Improve LLDB compatibility with the GDB protocol: support
gdb-style flexible register API.
3. Support for debugging via serial port.
4. libkvm-portable and support for debugging kernel core files in
LLDB, on amd64 + arm64 platform. Support for other platforms as
time permits.
5. Support for debugging the running kernel on amd64 + arm64
platform. Support for other platforms as time permits.
6. Extra month for kgdb work, processing patches on LLDB reviews or
miscellaneous tasks - as time permits. Examples of misc tasks:
access to extended system and process information, starting
processes via shell, $_siginfo support.
This month, we have continued our effort towards improving the LLDB
client compatibility with the GDB Remote Protocol and thus other
implementations, keeping the focus on the compatibility with the
reference implementation in GNU gdbserver and the builtin debug
server present in QEMU.
The GDB Remote Protocol is a mechanism to divide a debugger into two
processes: server and client. The server due to some specific
circumstances can be located in an environment without resources or
conditions to run a full debugging session (like headless embedded
AArch64 or RISCV boards) or lacks resources in reimplementing a
full-featured debugging purposes, exporting hooks to be accessed by a
preexisting debugger client.
As the final deliverable of this and the past month we increased the
compatibility between LLDB and GDB, giving an option of a
permissively licensed software stack to be used instead of the GPLv3
GDB code.
We selected the QEMU emulator as a reference implementation of a
debugger server in an emulator. Future versions of other software
like bhyve can extend its capabilities to GDB protocol and get the
native LLDB client support.
In the previous month, we worked on generic compatibility problems.
This month we've focused on the problems related to register
interfaces. The register interfaces are fundamental for proper
investigation of the state of a process inside a debugger, and this
month we achieved better compatibility for cross-OS, cross-CPU and
cross-debugger scenarios. We have formulated our goal as ensuring
that LLDB connected to GNU gdbserver and QEMU should have the same
level of functionality as LLDB connected to lldb-server.
The most important problem in the difference between LLDB and GDB
that we faced was the difference in abstractions. The LLDB
abstraction assumed most of the logic stayed on the server side and
was exposed to the client by the server. On the other hand, in the
GDB abstraction the server logic was minimal and additional functions
were implemented on the client side. To achieve maximum
compatibility, we had to replicate the GDB solution while preserving
compatibility with the original lldb-server.
GDB and LLDB register abstractions
Both GDB and LLDB are using register layout abstractions internally.
Notably, this enables the UI to expose the register sets of various
targets without carrying the specific target knowledge itself. In a
debugger operating in a client-server layout, this means that the
client does not need to be explicitly aware of target's register
layout. This makes it possible to use new targets without having to
update the client.
The server exposes its register layout to the client via the target
definition file target.xml. This file has originally been devised by
gdbserver but it has been adapted and extended by lldb-server as
well. The modern versions of LLDB support both formats.
To compare the two formats, it is best to look at a few example
registers. In the GDB format, their definitions look as follows:
In the LLDB format, the definitions include more attributes:
The LLDB target descriptions include more information than GDB's.
This is because in LLDB practically all register abstraction is
handled on the server, and exposed to the client via target.xml. In
GDB, part of the abstraction is handled on the client side.
GDB-LLDB compatibility mapping between target.xml formats
The common information found in both formats are:
* register name
* the size of register in bits
* its number (as used by the server)
The register type is spelled in both formats differently. In the GDB
format, specific types are defined, while in LLDB format the type is
described through encoding (indicating how the raw data is encoded)
and format (indicating the preferred visual representation). When
connected to gdbserver, LLDB maps individual GDB types into their
respective LLDB encodings and formats.
lldb-server explicitly indicates register offsets into the raw data.
When working with the GDB format, offsets are calculated based on
previous register sizes.
Grouping is also handled differently. In the GDB format, register
groups are indicated via separate tags. In the LLDB
format, a single tag is used and each register specifies a
group explicitly.
Finally, the LLDB format provides for carrying additional
information:
* the register's alternate name (alias), generally used to allow
querying registers via more generic names (e.g. rip via pc)
* the register's number in the exception handling frame and DWARF
debug information notations
* the register's generic role (if applicable)
This information is missing when working with gdbserver, and
therefore it needs to be filled in by the gdb-remote client.
Improving generic register support
View of registers in a debugger
Aside from the fundamental architecture-specific view of registers,
LLDB has a generic view that maps a subset of registers through their
functions, as defined by the platform's ABI. At the time of writing,
LLDB defines the following generic registers:
* PC (Program Counter) register that stores the address of the code
currently being executed (e.g. RIP on amd64)
* SP (Stack Pointer) register that stores the address to the
current position on the stack (e.g. RSP in amd64/SysV ABI)
* FP (Frame Pointer) register that stores the address to the top of
the current stack frame (e.g. RBP in amd64/SysV ABI)
* RA (Return Address) register that stores the address to return to
when the current function finishes (not used on amd64, e.g. LR on
arm64)
* Flags Register that is a bitmap indicating processor state (e.g.
RFLAGS/EFLAGS on amd64)
* and up to 8 Argument Registers that are used to pass arguments to
functions
The generic register codes are associated with specific registers in
the lldb-server plugins for individual systems, and are transmitted
as part of its target.xml. Along with the register codes, the
respective generic names are transmitted as alternative names of the
registers.
When LLDB is connected to gdbserver, this data is missing. Therefore,
LLDB client needs to fill it in for the generic register
functionality to work.
When we started working on this, the fallback was already partially
implemented, and even implemented twice. Firstly, for each register
received from the server, the gdb-remote plugin called the
AugmentRegisterInfo() method of the currently used ABI plugin that
assigned generic roles to the registers based on the current ABI.
Secondly, the finalization code called after receiving all registers
verified whether generic roles were assigned, and implemented a
fallback to roles defined per architecture.
However, the logic only assigned generic register codes and not
alternative names that could be used by the user to reference the
registers. In other words:
register read pc
worked with lldb-server but not gdbserver. The problem could be
resolved by modifying the fallbacks to set alternative names as well.
Nonetheless, we have decided to take an alternative approach and
instead enhance the register lookup methods to support finding the
registers via their generic names.
This method has the explicit advantage that it reduces the
duplication of data (i.e. it is no longer necessary to repeat generic
register names as alternative names to registers) and at the same
time it frees up the altname field for other purposes (LLDB supports
only one altname per register).
Partial/overlapping registers
It is not uncommon for newer architectures to reuse and extend older
registers, as well as provide accessors to smaller parts of larger
registers. The registers specific to amd64 are:
* the lower halves of amd64 RAX..RDX, RSI, RDI, RSP and RBP
registers can be accessed using i386-compatible EAX..EDX, ESI,
EDI, ESP and EBP designators
* the lower halves of amd64 and i386 EAX..EDX, ESI, EDI, ESP and
EBP registers can be accessed using 16-bit x86-compatible AX..DX,
SI, DI, SP and BP designators
* the two halves of amd64 and i386 AX..DX registers can be accessed
via AH..DH and AL..DL designators respectively
* the lower halves of amd64 and i386 SI, DI, SP and BP registers
can be accessed using SIL, DIL, SPL and BPL designators
* the least significant 32, 16 and 8 bits of amd64 R8..R15
registers can be accessed via R8D..R15D, R8W..R15W and R8L..R15L
designators respectively
* the MMX MM0..MM7 registers overlap with the least significant 64
bits of ST0..ST7 registers
* the AVX/AVX-512 YMM/ZMM registers overlap with the least
significant bits of SSE XMM registers
The AArch64 platform has similar layout:
* the lower halves of 64-bit x0..x31 registers can be accessed via
w0..w31 designators
* the least significant 64 bits of vector v0..v31 registers overlap
with double-precision floating-point d0..d31 registers
* the least significant 32 bits of vector v0..v31 registers overlap
with single-precision floating-point s0..s31 registers
LLDB implements these 'pseudo-registers' on server-side, and exposes
them in the target description. On the other hand, GDB exposes only
real registers in the target description and requires the client to
handle the partial registers explicitly. Therefore, to enable support
for them and bring LLDB in line with GDB, we had to implement the
client logic for this.
Regiser groups with subregisters
We have decided to hook into the finalization mechanism that is
invoked after receiving the remote register definitions. We have
implemented a pre-finalization hook and post-finalization hook.
The pre-finalization hook verifies whether the partial registers were
transmitted as a part of the remote register definitions. If they
were not, it adds the missing registers as smaller-sized registers
referencing the original registers.
Afterwards, the logic already existing in the finalization mechanism
updates the partial register definitions with appropriate offsets
referencing the full registers.
Finally, the post-finalization hook adjusts the offsets of AH..DH
registers as the existing logic did not support offsetting partial
registers from their parents.
Composite YMM registers
Along with the introduction of AVX instruction set, the 128-bit XMM
registers were replaced with 256-bit YMM registers. The code
referencing XMM registers was now using the lower half of the YMM
registers. Both LLDB and GDB support reading and writing both XMM and
YMM registers, though they use different approaches to handle the
overlap internally.
View of composite YMM registers
The lldb-server approach is to expose both complete XMM and YMM
registers. This implies that the overlapping bits are transmitted
twice. If the client requests writing inconsistent values into XMM
and lower bits of YMM, the behavior is undefined.
The gdbserver approach resembles the one used in the processor's
XSAVE instruction. The lower halves of YMM registers are transmitted
as XMM registers (for backwards compatibility), while the higher
halves are transmitted as ymmih pseudo-registers. The client combines
both into complete YMM registers.
To implement this, we have decided to extend the existing logic for
creating pseudo-registers. Before our changes, the logic supported
specifying only a single 'parent' register and extracting a part of
its value. We have modified it to support multiple 'parent'
registers, and combine their values to construct the larger register.
Comparison: LLDB against amd64 gdbserver
LLDB prior to our changes
Prior to our changes, LLDB featured baseline support for registers on
amd64. It was capable of displaying low-level register information
from server. However, the ST registers were reported as unavailable.
Partial registers such as EAX or MM0 were missing entirely, as were
combined YMM registers.
(skip terminal snippet)
mgorny@freebsd:~ $ lldb-devel --version
lldb version 13.0.0
mgorny@freebsd:~ $ lldb-devel
(lldb) gdb-remote 192.168.1.14:1234
Process 3396 stopped
* thread #1, stop reason = signal SIGTRAP
frame #0: 0x00007ffff7fcd050
-> 0x7ffff7fcd050: movq %rsp, %rdi
0x7ffff7fcd053: callq 0x7ffff7fcde00
0x7ffff7fcd058: movq %rax, %r12
0x7ffff7fcd05b: movl 0x2ec57(%rip), %eax
(lldb) cont
Process 3396 resuming
(lldb) process interrupt
Process 3396 stopped
* thread #1, stop reason = signal SIGINT
frame #0: 0x00007ffff7e7dd76
-> 0x7ffff7e7dd76: negl %eax
0x7ffff7e7dd78: retq
0x7ffff7e7dd79: nopl (%rax)
0x7ffff7e7dd80: movl $0x16, %eax
(lldb) # some of the registers are unavailable
(lldb) # some helpful aliases are missing
(lldb) register read --all
general:
rax = 0xfffffffffffffdfc
rbx = 0xffffffffffffff88
rcx = 0x00007ffff7e7dd76
rdx = 0x00007fffffffcbf0
rsi = 0x0000000000000000
rdi = 0x0000000000000000
rbp = 0x0000000000000000
rsp = 0x00007fffffffcbd8
r8 = 0x0000000000000000
r9 = 0x00007fffffffcb07
r10 = 0x00007fffffffcbf0
r11 = 0x0000000000000246
r12 = 0x00005555555550a0
r13 = 0x0000000000000000
r14 = 0x0000000000000000
r15 = 0x0000000000000000
rip = 0x00007ffff7e7dd76
eflags = 0x00000246
cs = 0x00000033
ss = 0x0000002b
ds = 0x00000000
es = 0x00000000
fs = 0x00000000
gs = 0x00000000
xmm0 = 0x25252525252525252525252525252525
xmm1 = 0x00000000000000000000ff0000ff0000
xmm2 = 0x00000000000000000000ff0000000000
xmm3 = 0x00000000ff000000ff000000000000ff
xmm4 = 0x2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f
xmm5 = 0x00000000000000000000000000000000
xmm6 = 0x00000000000000000000000000000000
xmm7 = 0x00000000000000000000000000000000
xmm8 = 0x2069205d74756f64735b000a6425203d
xmm9 = 0x00000000000000000000000000000000
xmm10 = 0x00000000000000000000000000000000
xmm11 = 0x00000000000000000000000000000000
xmm12 = 0x00000000000000000000000000000000
xmm13 = 0x00000000000000000000000000000000
xmm14 = 0x00000000000000000000000000000000
xmm15 = 0x00000000000000000000000000000000
orig_rax = 0x00000000000000e6
fs_base = 0x00007ffff7f84580
gs_base = 0x0000000000000000
ymm0h = 0x00000000000000000000000000000000
ymm1h = 0x00000000000000000000000000000000
ymm2h = 0x00000000000000000000000000000000
ymm3h = 0x00000000000000000000000000000000
ymm4h = 0x00000000000000000000000000000000
ymm5h = 0x00000000000000000000000000000000
ymm6h = 0x00000000000000000000000000000000
ymm7h = 0x00000000000000000000000000000000
ymm8h = 0x00000000000000000000000000000000
ymm9h = 0x00000000000000000000000000000000
ymm10h = 0x00000000000000000000000000000000
ymm11h = 0x00000000000000000000000000000000
ymm12h = 0x00000000000000000000000000000000
ymm13h = 0x00000000000000000000000000000000
ymm14h = 0x00000000000000000000000000000000
ymm15h = 0x00000000000000000000000000000000
8 registers were unavailable.
float:
fctrl = 0x0000037f
fstat = 0x00000000
ftag = 0x0000ffff
fiseg = 0x00000000
fioff = 0x00000000
foseg = 0x00000000
fooff = 0x00000000
fop = 0x00000000
vector:
mxcsr = 0x00001f80
LLDB after our changes
With our changes, LLDB displays a complete set of server and virtual
registers. Vector registers are consistently displayed as vectors.
Partial registers are created from their parent registers (e.g. EAX
from RAX, MM0 from ST0). The complete YMM registers are recombined
from the server data.
(skip terminal snippet)
mgorny@freebsd:~/llvm-project/build $ ./bin/lldb --version
lldb version 14.0.0
mgorny@freebsd:~/llvm-project/build $ ./bin/lldb
(lldb) gdb-remote 192.168.1.14:1234
Process 4186 stopped
* thread #1, stop reason = signal SIGTRAP
frame #0: 0x00007ffff7fcd050
-> 0x7ffff7fcd050: movq %rsp, %rdi
0x7ffff7fcd053: callq 0x7ffff7fcde00
0x7ffff7fcd058: movq %rax, %r12
0x7ffff7fcd05b: movl 0x2ec57(%rip), %eax
(lldb) cont
Process 4186 resuming
(lldb) process interrupt
Process 4186 stopped
* thread #1, stop reason = signal SIGINT
frame #0: 0x00007ffff7e7dd76
-> 0x7ffff7e7dd76: negl %eax
0x7ffff7e7dd78: retq
0x7ffff7e7dd79: nopl (%rax)
0x7ffff7e7dd80: movl $0x16, %eax
(lldb) # more registers are displayed
(lldb) # data types are also more consistent
(lldb) register read --all
general:
rax = 0xfffffffffffffdfc
rbx = 0xffffffffffffff88
rcx = 0x00007ffff7e7dd76
rdx = 0x00007fffffffcbf0
rsi = 0x0000000000000000
rdi = 0x0000000000000000
rbp = 0x0000000000000000
rsp = 0x00007fffffffcbd8
r8 = 0x0000000000000000
r9 = 0x00007fffffffcb07
r10 = 0x00007fffffffcbf0
r11 = 0x0000000000000246
r12 = 0x00005555555550a0
r13 = 0x0000000000000000
r14 = 0x0000000000000000
r15 = 0x0000000000000000
rip = 0x00007ffff7e7dd76
eflags = 0x00000246
cs = 0x00000033
ss = 0x0000002b
ds = 0x00000000
es = 0x00000000
fs = 0x00000000
gs = 0x00000000
st0 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
st1 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
st2 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
st3 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
st4 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
st5 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
st6 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
st7 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
xmm0 = {0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25}
xmm1 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0xff 0x00 0x00 0xff 0x00 0x00}
xmm2 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0xff 0x00 0x00}
xmm3 = {0x00 0x00 0x00 0xff 0x00 0x00 0x00 0x00 0xff 0x00 0x00 0x00 0x00 0x00 0x00 0xff}
xmm4 = {0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f}
xmm5 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
xmm6 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
xmm7 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
xmm8 = {0x64 0x6f 0x75 0x74 0x5d 0x20 0x69 0x20 0x3d 0x20 0x25 0x64 0x0a 0x00 0x5b 0x73}
xmm9 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
xmm10 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
xmm11 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
xmm12 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
xmm13 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
xmm14 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
xmm15 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
orig_rax = 0x00000000000000e6
fs_base = 0x00007ffff7f84580
gs_base = 0x0000000000000000
ymm0h = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm1h = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm2h = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm3h = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm4h = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm5h = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm6h = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm7h = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm8h = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm9h = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm10h = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm11h = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm12h = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm13h = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm14h = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm15h = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
float:
fctrl = 0x0000037f
fstat = 0x00000000
ftag = 0x0000ffff
fiseg = 0x00000000
fioff = 0x00000000
foseg = 0x00000000
fooff = 0x00000000
fop = 0x00000000
vector:
mxcsr = 0x00001f80
partial registers:
eax = 0xfffffdfc
ax = 0xfdfc
ah = 0xfd
al = 0xfc
ebx = 0xffffff88
bx = 0xff88
bh = 0xff
bl = 0x88
ecx = 0xf7e7dd76
cx = 0xdd76
ch = 0xdd
cl = 0x76
edx = 0xffffcbf0
dx = 0xcbf0
dh = 0xcb
dl = 0xf0
edi = 0x00000000
di = 0x0000
dil = 0x00
esi = 0x00000000
si = 0x0000
sil = 0x00
ebp = 0x00000000
bp = 0x0000
bpl = 0x00
esp = 0xffffcbd8
sp = 0xcbd8
spl = 0xd8
r8d = 0x00000000
r8w = 0x0000
r8l = 0x00
r9d = 0xffffcb07
r9w = 0xcb07
r9l = 0x07
r10d = 0xffffcbf0
r10w = 0xcbf0
r10l = 0xf0
r11d = 0x00000246
r11w = 0x0246
r11l = 0x46
r12d = 0x555550a0
r12w = 0x50a0
r12l = 0xa0
r13d = 0x00000000
r13w = 0x0000
r13l = 0x00
r14d = 0x00000000
r14w = 0x0000
r14l = 0x00
r15d = 0x00000000
r15w = 0x0000
r15l = 0x00
mm0 = 0x0000000000000000
mm1 = 0x0000000000000000
mm2 = 0x0000000000000000
mm3 = 0x0000000000000000
mm4 = 0x0000000000000000
mm5 = 0x0000000000000000
mm6 = 0x0000000000000000
mm7 = 0x0000000000000000
ymm0 = {0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm1 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0xff 0x00 0x00 0xff 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm2 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0xff 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm3 = {0x00 0x00 0x00 0xff 0x00 0x00 0x00 0x00 0xff 0x00 0x00 0x00 0x00 0x00 0x00 0xff 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm4 = {0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x2f 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm5 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm6 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm7 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm8 = {0x64 0x6f 0x75 0x74 0x5d 0x20 0x69 0x20 0x3d 0x20 0x25 0x64 0x0a 0x00 0x5b 0x73 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm9 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm10 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm11 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm12 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm13 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm14 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
ymm15 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
Improving arch recognition logic in LLDB
In order to apply architecture-specific customization on the client
side, LLDB needs to be able to recognize the architecture of the
debugged process. Normally, this is done using one of two packets:
qProcessInfo to get the information on the debugged process, or
qHostInfo to get information on the host running the debugger. The
former packet is preferred as the latter can yield incorrect
information if the host can run more than one type of executables,
i.e. when debugging a 32-bit program on amd64 host.
However, these packets are LLDB-specific and are not implemented by
gdbserver. When interacting with gdbserver, LLDB can obtain the
architecture from target.xml instead. Prior to our changes, it did
this only for two special cases -- i386:x86-64 used by various GDB
remote protocol implementations on amd64, and arm used by SEGGER
J-Link jtag boards.
We have enhanced this logic to permit fallback to the target.xml
supplied architecture for any value of the architecture, with special
casing for architecture names that do not match between gdbserver and
LLDB. As a result, we have been able to successfully connect both to
a gdbserver running 32-bit i386 executables, as well to an AArch64
server.
Comparison: LLDB against aarch64 gdbserver
LLDB prior to our changes
Prior to our changes, LLDB did not recognize aarch64 as a valid
architecture. It supported minimal program flow control but was
unable to read registers and display stack frames.
mgorny@freebsd:~ $ lldb-devel --version
lldb version 13.0.0
mgorny@freebsd:~ $ lldb-devel
(lldb) gdb-remote 192.168.1.18:1234
Process 1355 stopped
* thread #1, stop reason = signal SIGTRAP
frame #0: 0xffffffffffffffff
(lldb) cont
Process 1355 resuming
(lldb) process interrupt
Process 1355 stopped
* thread #1, stop reason = signal SIGINT
frame #0: 0xffffffffffffffff
(lldb) register read --all
error: invalid frame
LLDB after our changes
With our changes, LLDB recognizes the architecture correctly. It
displays the complete set of server and virtual registers.
(skip terminal snippet)
mgorny@freebsd:~/llvm-project/build $ ./bin/lldb --version
lldb version 14.0.0
mgorny@freebsd:~/llvm-project/build $ ./bin/lldb
(lldb) gdb-remote 192.168.1.18:1234
Process 2361 stopped
* thread #1, stop reason = signal SIGTRAP
frame #0: 0x0000fffff7fbed80
-> 0xfffff7fbed80: bti c
0xfffff7fbed84: mov x0, sp
0xfffff7fbed88: bl 0xfffff7fbf894
0xfffff7fbed8c: mov x21, x0
(lldb) cont
Process 2361 resuming
(lldb) process interrupt
Process 2361 stopped
* thread #1, stop reason = signal SIGINT
frame #0: 0x0000fffff7ed9a58
-> 0xfffff7ed9a58: svc #0
0xfffff7ed9a5c: mov x19, x0
0xfffff7ed9a60: neg w0, w19
0xfffff7ed9a64: ldp x19, x20, [sp, #0x10]
(lldb) register read --all
general:
x0 = 0x0000000000000000
x1 = 0x0000000000000000
x2 = 0x0000fffffffff228
x3 = 0x0000fffffffff228
x4 = 0x0000000000000000
x5 = 0x00000000fffffffa
x6 = 0x000000000000000a
x7 = 0x00000000ffffffff
x8 = 0x0000000000000073
x9 = 0x000000000000000a
x10 = 0x000000000000000a
x11 = 0x0000000000000020
x12 = 0x0000000000000000
x13 = 0x0000000000000000
x14 = 0x0000000000000000
x15 = 0x0000fffff7f63d58
x16 = 0x0000000000420010
x17 = 0x0000fffff7edeee0
x18 = 0x0000000000000001
x19 = 0x0000000000000000
x20 = 0x0000fffff7ff13c0
x21 = 0x0000000000000000
x22 = 0x0000000000000000
x23 = 0x0000000000000000
x24 = 0x0000000000000000
x25 = 0x0000000000000000
x26 = 0x0000000000000000
x27 = 0x0000000000000000
x28 = 0x0000000000000000
x29 = 0x0000fffffffff1a0
x30 = 0x0000fffff7edf070
sp = 0x0000fffffffff1a0
pc = 0x0000fffff7ed9a58
cpsr = 0x80001000
v0 = {0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25 0x25}
v1 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x5b 0x73 0x74 0x64 0x65 0x72 0x72 0x5d}
v2 = {0x00 0x30 0x00 0x00 0x00 0x00 0x00 0x00 0x03 0x00 0x00 0x00 0x00 0x00 0xfc 0x00}
v3 = {0x0f 0xf0 0x0f 0xf0 0x0f 0xf0 0x0f 0xf0 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v4 = {0x0f 0xf0 0x0f 0xf0 0x0f 0xf0 0x0f 0xf0 0x0f 0xf0 0x0f 0xf0 0x0f 0xf0 0x0f 0xf0}
v5 = {0xff 0xff 0xff 0xff 0x00 0x00 0x00 0x00 0xff 0xff 0xff 0xff 0x00 0x00 0x00 0x00}
v6 = {0x00 0x00 0xc0 0x00 0x00 0x00 0x00 0xc0 0x00 0x00 0xc0 0x00 0x00 0x00 0x00 0xc0}
v7 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v8 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v9 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v10 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v11 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v12 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v13 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v14 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v15 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v16 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v17 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v18 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v19 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v20 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v21 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v22 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v23 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v24 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v25 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v26 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v27 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v28 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v29 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v30 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
v31 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
fpsr = 0x00000000
fpcr = 0x00000000
partial registers:
w0 = 0x00000000
w1 = 0x00000000
w2 = 0xfffff228
w3 = 0xfffff228
w4 = 0x00000000
w5 = 0xfffffffa
w6 = 0x0000000a
w7 = 0xffffffff
w8 = 0x00000073
w9 = 0x0000000a
w10 = 0x0000000a
w11 = 0x00000020
w12 = 0x00000000
w13 = 0x00000000
w14 = 0x00000000
w15 = 0xf7f63d58
w16 = 0x00420010
w17 = 0xf7edeee0
w18 = 0x00000001
w19 = 0x00000000
w20 = 0xf7ff13c0
w21 = 0x00000000
w22 = 0x00000000
w23 = 0x00000000
w24 = 0x00000000
w25 = 0x00000000
w26 = 0x00000000
w27 = 0x00000000
w28 = 0x00000000
w29 = 0xfffff1a0
w30 = 0xf7edf070
w31 = 0xfffff1a0
s0 = 1.43241e-16
s1 = 0
s2 = 1.72192e-41
s3 = -1.78186e+29
s4 = -1.78186e+29
s5 = -nan
s6 = 1.76324e-38
s7 = 0
s8 = 0
s9 = 0
s10 = 0
s11 = 0
s12 = 0
s13 = 0
s14 = 0
s15 = 0
s16 = 0
s17 = 0
s18 = 0
s19 = 0
s20 = 0
s21 = 0
s22 = 0
s23 = 0
s24 = 0
s25 = 0
s26 = 0
s27 = 0
s28 = 0
s29 = 0
s30 = 0
s31 = 0
d0 = 9.53282412436824e-130
d1 = 0
d2 = 6.07107865609724e-320
d3 = -6.19799051634446e+231
d4 = -6.19799051634446e+231
d5 = 2.12199579047121e-314
d6 = -2.00000000558794
d7 = 0
d8 = 0
d9 = 0
d10 = 0
d11 = 0
d12 = 0
d13 = 0
d14 = 0
d15 = 0
d16 = 0
d17 = 0
d18 = 0
d19 = 0
d20 = 0
d21 = 0
d22 = 0
d23 = 0
d24 = 0
d25 = 0
d26 = 0
d27 = 0
d28 = 0
d29 = 0
d30 = 0
d31 = 0
Testing other servers using the GDB remote protocol
Our posts so far have focused on two implementations of the GDB
remote protocol: the reference implementation in GDB, and the one in
LLDB. However, there are other implementations of interest and at
this point we've decided to perform some testing of two more
scenarios, both connected from FreeBSD/amd64 running LLDB: QEMU with
two FreeBSD guests: AMD64 and ARM64. For completeness we ensured that
Linux setup also works.
QEMU is an open source machine emulator and virtualizer. It provides
a GDB remote protocol server that can be used to attach directly to
the raw virtual hardware. Unlike a monitor run inside the VM, the
debugger is not aware of processes but instead observes the raw CPU
activity. On multiprocessor systems, the different processors, cores
and threads are reported as GDB remote protocol threads.
To enable this function, qemu needs to be started with the -gdb or
the shorthand -s option.
QEMU FreeBSD/AMD64
For this scenario, we have used a FreeBSD 13.0 amd64 'bootonly' CD.
QEMU was started using the following command-line:
qemu-system-x86_64 -m 4G -display curses -smp 12 \
-cdrom FreeBSD-13.0-RELEASE-amd64-bootonly.iso -s
This started QEMU with a GDB remote protocol-compatible server on
port 1234.
QEMU FreeBSD/ARM64
For this scenario, we have used the FreeBSD 13.0 arm64
'mini-memstick' image. We have converted the image to QCOW2 format
using:
qemu-img convert -O qcow2 \
FreeBSD-13.0-RELEASE-arm64-aarch64-mini-memstick.img \
FreeBSD-13.0-RELEASE-arm64-aarch64-mini-memstick.qcow
Then we have started QEMU using:
qemu-system-aarch64 -display curses -M virt -cpu cortex-a57 -m 4G \
-hda FreeBSD-13.0-RELEASE-arm64-aarch64-mini-memstick.qcow \
-smp 8 -bios edk2-aarch64-code.rd -serial stdio -s
This started QEMU with a GDB remote protocol-compatible server on
port 1234.
Valgrind
We have performed additional tests with Valgrind. Valgrind is a set
of program instrumentation tools, probably most widely known for its
memory error detector. When a program is being instrumented via a
valgrind tool, the debugger attaches to the tool rather than the
actual program. In order to permit debugging the program being
instrumented, valgrind provides a GDB-compatible wrapper called vgdb.
To use this functionality, valgrind needs to be used with --vgdb=yes
option. Afterwards, the vgdb tool serves as a proxy between valgrind
and GDB remote protocol-compatible client.
The GDB Protocol compatibility gap with Valgrind was decreased in the
past two months, but unfortunately there is still room for
improvement, as once we attach to a Valgrind session, our set of
features is restricted and there is limited support for Valgrind
shadow registers.
Changes merged upstream
* [lldb] Support "eflags" register name in generic reg fallback
* [lldb] Support querying registers via generic names without
alt_names
* [lldb] Remove redundant register alt_names
* [lldb] [gdb-remote] Try using for remote arch unconditionally
* [lldb] [ABI/AArch64] Recognize special regs by their xN names too
* [lldb] [DynamicRegisterInfo] Pass name/alt_name via RegisterInfo
* [lldb] [Process/gdb-remote] Alias sp to x31 on AArch64 for
gdbserver
* [lldb] [gdb-remote] Remove unused arg from
GDBRemoteRegisterContext::ReadRegisterBytes()
* [lldb] [gdb-remote] Recognize aarch64v type from gdbserver
* [lldb] [gdb-remote] Always send PID when detaching w/
multiprocess
* [lldb] [test] Add unittest for DynamicRegisterInfo::Finalize()
* [lldb] [DynamicRegisterInfo] Unset value_regs/invalidate_regs
before Finalize()
Changes pending review
* [lldb] [gdb-remote] Use local regnos for value_regs/
invalidate_regs
* [lldb] [DynamicRegisterInfo] Add a convenience method to add
suppl. registers
* [lldb] [ABI/AArch64] Add pseudo-regs if missing
* [lldb] [gdb-remote] Fix displaying i387_ext & vec regs with
gdbserver
* [lldb] [gdb-remote] Add x86_64 pseudo-registers when using
gdbserver
* [lldb] [gdb-remote] Remove unused arg from
GDBRemoteRegisterContext::ReadRegisterBytes()
* [lldb] [gdb-remote] Support combining xmm* and ymm*h regs into
ymm*
* [lldb] [gdb-remote] Extend x86 pseudo-regs to i386
Summary
Our primary focus during M2 work was improving register-level
compatibility between LLDB and a variety of servers implementing the
GDB remote protocol. We wanted to ensure that the user experience
when working with these servers, would be no worse than when working
against lldb-server, effectively working towards making LLDB a
drop-in replacement for GDB in a growing number of scenarios.
On the most common amd64 architecture, our work has primarily
resulted in improving the convenience of register access. Prior to
our work, LLDB was only capable of exposing the low-level register
data supplied by the server. We have enhanced it to include
high-level registers such as supplementary EAX, AX or MM0 registers,
as well as registers that are transmitted in parts by GDB such as
YMM0.
However, much greater difference has been made on other
architectures. Prior to our work, LLDB was unable to recognize them
unless interacting with lldb-server. While a minimal flow control was
still available, the debugger couldn't read stack frames, registers,
variables. Our changes have made it possible to use other
architectures including but not limited to i386 and aarch64.
We have tested our changes not only against genuine gdbserver but
also against QEMU and Valgrind. We have tested cross-architecture and
cross-platform (particularly between FreeBSD and Linux) use of LLDB
client and various servers.
Finally, we have made major changes to the internal architecture of
LLDB. We have made the code simpler and more robust. Some of the
duplicate logic has been removed already, and other major
deduplication of code that is currently copied between all platform
plugins is planned after the next release. We have prepared new API
that others can use to follow up on our work in the future.
Future plans
The next step of our work is to implement the support for debugging
via the serial port. This is especially important as it provides an
interface to kernel debugging with minimal dependency on kernel
drivers, or on embedded hardware that does not run a general purpose
operating system capable of running the debugger locally.
GDB supports a number of remote connection options including serial
port connections. However, at the time of writing LLDB supports only
TCP/IP-based connections to the remote server. It needs to be
enhanced to support serial port connections as well.
This involves adding support for configuring various serial port
properties such as baud rate, number of data bits, use of parity
bits, etc.
Please enable JavaScript to view the comments powered by Disqus.
comments powered by Disqus
Related pages
* Blog: Improving GDB protocol compatibility in LLDB
* Blog: FreeBSD KGDB support in LLDB
* Blog: LLDB core dump support improvements
* Blog: LLDB support for fork(2) and vfork(2)
* Blog: LLDB: FreeBSD Legacy Process Plugin Removed
* Blog: FreeBSD Remote Process Plugin on Non-x86 Architectures
* Blog: LLDB FreeBSD CPU target support and userland debugging
improvements
* Blog: FreeBSD Remote Process Plugin: Final Milestone Achieved
* Blog: FreeBSD Remote Process Plugin is now the default in LLDB
* Blog: Introduction of a new FreeBSD Remote Process Plugin in LLDB
* Blog: LLDB Debugger Improvements for FreeBSD
* Services: BSD Software
* Blog: Mastering UNIX pipes, Part 1
* Blog: Before the BSD Kernel starts: Part One on AMD64
* Blog: How Debuggers Work: Getting and Setting x86 Registers, Part
2: XSAVE
(c) 2020-2021 Moritz Systems Technology Company Sp. z o.o. All rights
reserved.