https://and0uille.net/misc/debug90.html

Reverse Engineering DOS Software as if It Were 1990

Jonas Santoso,Last updated at Sun Nov 19 2023*7 min read
retrocrackingreverse engineering
Back

Quick Start 

The video above shows a speed run of what's being done in the How to
Build and Install section. TL;DR: This is about running SoftICE, a
popular DOS and Windows debugger, in an emulated MS-DOS environment.
And about overcoming a limitation of Bochs (opens in a new tab), the
IA-32 (x86) PC emulator.

All of the following assumes a Linux host system. I don't know if and
how it would work in macOS, let alone Windows.

Emulating SoftICE 

NuMega started making SoftICE, which became the most widely used tool
for debugging DOS, and later Windows software, in 1987. The last
version that was released exclusively for DOS is SoftICE 2.80 from
1997. For someone who wishes to use live debugging sessions in an
effort to reverse engineer old software today, there is a number of
options:

  * DOSBox-X (opens in a new tab), a fork of DOSBox, features a
    debugger that bears some resemblance to SoftICE.
  * IDA (opens in a new tab), whilst primarily a disassembler, has a
    debugger mode too, and there's dsbxida (opens in a new tab) which
    interfaces IDA with DOSBox-X in order to facilitate debugging DOS
    software.
  * Custom tooling may be built using the Unicorn (opens in a new
    tab) CPU emulator framework. See UniDOS (opens in a new tab) for
    a (yet limited) Unicorn-based DOS emulator.

However, when the intention is not solely to attain a reverse
engineering objective, but also to (re-)live the experience of using
period tooling, there's nothing like SoftICE for DOS. Where period
hardware is not available, we'll be looking at running it in an
emulator, and that's where things get a little bit hairy.

SoftICE doesn't work in DOSBox-X. It does work in Bochs which, unlike
DOSBox, is based on full emulation of an x86 CPU. Setting up Bochs is
a little more involved though, and that aside, there's another
problem.

Say we're done installing MS-DOS and SoftICE in a Bochs environment.
Pressing Ctrl+d brings up SoftICE, and all is good and well until we
try doing that while running something that's using a VGA graphics
mode--a game? Execution is paused, and SoftICE is actually in the
foreground, but we don't see it, because text mode is not being
restored properly. The Bochs window just shrinks to the height of the
UI.

The solution, in actual hardware terms, would be to connect a second
monitor to a secondary graphics adapter providing output from an
MC6845 (opens in a new tab) display controller, i.e. an MDA/Hercules
card. By issuing ALTSCR ON in SoftICE, the debugger's display could
then be switched to the secondary monitor.

Unfortunately Bochs doesn't emulate either MDA or a secondary
display. Emulating MDA text mode isn't hard though, and ultimately
that's what I ended up doing. I found a discussion about adding MDA
emulation to Bochs in an old thread (opens in a new tab) on
SourceForge. There wasn't a complete implementation anywhere, so I
created a Bochs fork with the necessary changes:

https://github.com/santosoj/bochs-dual-monitor (opens in a new tab)

Implementation details can be found in the repo readme. The most
important thing to note is probably that the MDA text screen isn't
displayed in Bochs itself. It's written to a named pipe as UTF-8 code
points, along with ANSI escape sequences for rendering the MC6845's
character attributes. The named pipe would then be polled by an
external program which dumps the text screen on a terminal. (A sample
program doing that was added to the repo (opens in a new tab).)

This screenshot shows Bochs running a game while SoftICE is displayed
in a terminal window:

Bochs running a game while SoftICE is displayed in a terminal window

How to Build and Install 

Build Bochs 

Clone my Bochs fork (opens in a new tab) and compile it with CPU
level 3 in order to target 386. (More on compiling Bochs here (opens
in a new tab).) E.g.

$ ./configure --with-sdl2 --enable-cpu-level=3
$ make

Use bximage (part of the Bochs build) to create a harddisk image. The
defaults (hd / flat / 512) and a size of 30mb will be fine. I'm
calling it hdd.img.

We need a .bochsrc file, which we initialize by starting Bochs and
picking 4. Save options to.... Then we edit it and make these changes
(YMMV):

romimage: file="/usr/local/share/bochs/BIOS-bochs-legacy", address=0x00000000, options=fastboot
floppya: 1_44=dos622_d1.img, status=inserted
ata0-master: type=disk, mode=flat, path=hdd.img
#port_e9_hack_all_rings: enabled=0

I had to comment port_e9_hack_all_rings because otherwise Bochs would
throw an error. dos622_d1.img is the first of the MS-DOS 6.22 install
disks. You can find these in the Downloads section below.

Install DOS 

One little annoyance observed in other versions of Bochs--it doesn't
seem to be occurring in the version the fork is based on--is that
Bochs goes straight into the debugger after picking 6. Begin
simulation, requiring you to enter c <newline> to continue execution.
This can be mitigated by creating a file (named bochs_init here)
containing c and a newline, and passing this file as an argument to
Bochs's -rc command line option. Also, the startup menu can be
skipped by passing -q, so the following should make for a quick Bochs
startup:

$ bochs -q -rc bochs_init

Another annoyance is that Bochs will segfault on starting when a
harddisk image lock file is left over from an unclean exit. So when
there's a dangling hdd.img.lock, delete it before starting Bochs.

Start Bochs and proceed with the MS-DOS installation. When prompted
to insert another disk, click CONFIG in the Bochs UI, and change the
image file for Floppy disk 0. Once installation is finished, change
the boot option in .bochsrc from floppy to disk. When you start Bochs
again, it should boot straight into DOS.

Install a CD-ROM Driver 

Next, use bximage again to create a floppy image (options: fd /
1.44M). I'm naming it floppy.img. In Bochs, insert this disk (CONFIG
/ 1. Floppy disk 0) and format it by entering format a: in DOS. We
may as well change the floppya option in .bochsrc so this disk image
is used by default. Once formatted, we mount this floppy image on our
host system and copy oakcdrom.sys, the Oak CD-ROM driver (also found
in the Downloads section):

$ sudo mount floppy.img /mnt/tmp
$ sudo cp oakcdrom.sys /mnt/tmp
$ sudo umount /mnt/tmp

Install the driver in DOS.

C:\> copy a:\oakcdrom.sys c:\dos
C:\> edit config.sys

Add this line to CONFIG.SYS:

DEVICEHIGH=C:\DOS\OAKCDROM.SYS /D:MSCD001

And this line to AUTOEXEC.BAT:

LH C:\DOS\MSCDEX.EXE /D:MSCD001

Install SoftICE 

Now that we can use CD images, let's create one on our host system.
Assuming the SoftICE 2.80 files (those found in S-ICE280.ZIP inside
sice280_and_extras.zip from the Downloads section) are in a directory
called sice, this command creates an ISO image from it:

$ mkisofs -o cdrom.iso -V CDROM sice

Change the ata1-master option in .bochsrc to insert the CD:

ata1-master: type=cdrom, path=cdrom.iso, status=inserted

After booting into DOS again, we can proceed to install SoftICE.

C:\> mkdir sice
C:\> xcopy /e/s d: c:\sice

Add C:\SICE to the PATH environment variable in AUTOEXEC.BAT. Note
that, unlike Bash, DOS uses the semicolon as path separator.

SET PATH=%PATH%;C:\SICE

In CONFIG.SYS, load SoftICE before any other driver:

DEVICE=C:\SICE\S_ICE.EXE

That's it. After rebooting, you can bring up SoftICE by pressing
Ctrl+d.

Bochs running SoftICE

Displaying SoftICE on the MDA Adapter 

One step is left in order to be able to debug games and such. The
forked Bochs's MDA emulation is enabled when Bochs finds an
environment variable named MDA_PIPE containing the path to a named
pipe. Create the pipe on the host system:

$ mkfifo /tmp/mda_pipe

Note that, once MDA is enabled, execution is blocked unless there's a
program running that consumes the pipe. So let's start that, the
program being the mda_viewer (opens in a new tab) example from the
repo.

$ cd ~/bochs-dual-monitor/mda_viewer/
$ MDA_PIPE=/tmp/mda_pipe python .

Start Bochs, also specifying MDA_PIPE:

$ MDA_PIPE=/tmp/mda_pipe ~/bochs-dual-monitor/bochs/bochs -q

You can verify that MDA is enabled by looking for this line in
Bochs's console output:

[MDA] Using named pipe "/tmp/mda_pipe".

Run a game, or something that uses a VGA graphics mode. Press Ctrl+d.
The graphics display will disappear. But confident that SoftICE is
now in the foreground (and with the Bochs window still focused),
enter the following SoftICE command.

altscr on

The graphics display comes back, and the terminal window where
mda_viewer is running now shows SoftICE. It may take a little getting
used to not to try typing into the terminal window. It's still the
Bochs window that needs to be focused when entering SoftICE commands.
But with this setup, we can reverse engineer DOS software as if it
were 1990.

Downloads 

De facto abandonware, as none of this is more recent than from 1997.

MS-DOS 6.22 (opens in a new tab)
SoftICE 2.80 and extras (opens in a new tab)
OAKCDROM.SYS (opens in a new tab)

[ ::.. AND0UiLLE 2023 ..:: ]
A The Hardliner serie.

CC BY-NC 4.0 2024 (c) Jonas Santoso.