Title

Whitespace Normalization on BSD Unix: Collapsing Runs and Resetting Indentation Using ed(1) and Standard Utilities

Abstract

We fix whitespace and indentation problems manually using standard Unix utilities, learning each tool as we go. Two tasks are covered: collapsing runs of whitespace into a single space, and resetting drifted indentation in text and HTML files back to a clean baseline. The tools are ed(1), sed(1), tr(1), awk(1), and expand(1) — all present on any Unix or BSD base system with no packages required. Each command is shown exactly as typed so the reader can follow along at a terminal.

I. Introduction

Whitespace normalization is a fundamental text-processing operation in Unix system administration, source code maintenance, and document preparation. Two forms are routinely required: collapsing, in which any run of one or more whitespace characters is replaced by a single space; and indentation reset, in which the leading whitespace on each line is stripped and optionally replaced with a canonical indent.

The BSD base system provides four tools suited to these operations without any package installation: ed(1), the original Unix line editor; sed(1), the stream editor; tr(1), the character translator; and awk(1), the pattern-scanning language. Each operates differently on whitespace, and the correct choice depends on whether line boundaries must be crossed, whether the edit must be in-place, and whether POSIX portability is required [1].

HTML source files are a common target for both operations. Whitespace collapses when files are processed by automated tools or pasted between editors. Indentation drifts when lines are added interactively in ed(1) without awareness of the current nesting level. The standard POSIX character class [[:space:]] covers the full set of whitespace characters: horizontal tab (HT), vertical tab (VT), form feed (FF), carriage return (CR), and space (SP) [2]. This experiment quantifies each tool’s effectiveness for both normalization tasks.

II. Materials and Methods

A. Equipment

BSD Unix System — FreeBSD 14.0-RELEASE

Base system only; no ports or packages installed beyond the default distribution. Shell: /bin/sh (POSIX sh, ash-derived). All commands invoked from a login shell over SSH or at the console.

ed(1) — BSD ed, POSIX line editor

The standard line-oriented text editor. Accepts commands from stdin or a here-document. Operates on an in-memory buffer; the w command writes back to the original file without a temporary file. POSIX-specified; present on all BSD and Linux systems.

sed(1) — BSD stream editor

Reads input line by line, applies substitution expressions, writes to stdout. The -i '' flag (BSD sed) enables in-place editing by creating a temporary file and renaming it.

tr(1) — character translator

Maps characters at the byte-stream level; uniquely able to process newline characters without line-at-a-time restrictions. The -s flag squeezes repeated characters in the output class to a single instance, making it the natural tool for collapsing runs.

awk(1) — pattern scanning and processing language

Field-aware; the built-in field-splitting mechanism collapses whitespace by default. Reassigning $1=$1 on any record forces awk to rebuild the line using the output field separator (OFS, default single space), collapsing all inter-field whitespace.

expand(1) / unexpand(1)

Convert between tab characters and space sequences. expand -t 4 replaces each tab with four spaces; unexpand -t 4 reverses the operation. Required as a pre-processing step when input files use mixed tab/space indentation.

cat(1) with -A flag (GNU) / -e -t flags (BSD)

Reveals invisible whitespace characters: BSD cat -e marks line endings with $; cat -t renders tab characters as ^I. Used to verify input file whitespace composition before normalization.

Test Files

Three plain-text and HTML source files of 20–80 lines each, constructed to contain representative mixtures of tabs, multiple spaces, carriage returns, and inconsistent indentation depth.

B. Procedure — Whitespace Collapsing

Each command below was applied to a fresh copy of the test file. Output was captured to a second file and diffed against the expected result (every run of whitespace replaced by exactly one space, count not preserved).

1. Inspect the file for whitespace composition

Before editing, identify what kinds of whitespace are present. On BSD, cat -e marks line endings and cat -t shows tabs:

  cat -e file.txt    # $ marks every line ending; \r\n appears as ^M$
  cat -t file.txt    # ^I marks tab characters

2. Collapse using tr(1) — recommended for cross-line whitespace

tr -s squeezes; the POSIX class [:space:] covers all whitespace including newlines:

  tr -s '[:space:]' ' ' < file.txt > collapsed.txt

Note: this flattens the entire file to one line if newlines are present. To collapse only horizontal whitespace while preserving line structure, restrict the class:

  tr -s '[:blank:]' ' ' < file.txt > collapsed.txt

[:blank:] covers space and horizontal tab only; newlines are passed through unchanged.

3. Collapse using ed(1) — in-place, no temp file

The ed(1) substitution operates line-at-a-time, so newlines are not visible to it. This collapses horizontal whitespace runs within each line and writes back to the original file:

  ed -s file.txt <<'EOF'
  ,s/[[:space:]][[:space:]]*/  /g
  w
  q
  EOF

The pattern [[:space:]][[:space:]]* is POSIX BRE for “one or more whitespace characters” (equivalent to [[:space:]]+ in ERE). Each matching run is replaced by a single space.

4. Collapse using sed(1) — stdout

  sed 's/[[:space:]][[:space:]]*/  /g' file.txt

For BSD in-place editing, add -i '':

  sed -i '' 's/[[:space:]][[:space:]]*/  /g' file.txt

5. Collapse using awk(1) — field-level

Forcing awk to rebuild each record collapses all inter-field whitespace. Leading and trailing whitespace is also stripped by default field-splitting:

  awk '{$1=$1; print}' file.txt

C. Procedure — Indentation Reset in ed(1)

When editing HTML interactively in ed(1), indentation drifts because ed has no awareness of document structure. The following procedure restores a clean indent baseline. All commands are typed at the * prompt inside an active ed session.

1. Check current indentation state

Print a range of lines with line numbers to see the current indent:

  1,20n          ← print lines 1-20 with line numbers
  .n             ← print the current line with its number
  .p             ← print the current line without its number

2. Strip all leading whitespace from every line

The anchor ^ matches the start of each line; the class [[:space:]]* matches zero or more whitespace characters; replacing with nothing removes it:

  ,s/^[[:space:]]*//

After this command all lines are flush left. This is the reset. Write if satisfied:

  w

3. Apply a uniform indent to all lines

Prepend two spaces to every line:

  ,s/^/  /

Prepend one tab to every line:

  ,s/^//       ← the replacement contains a literal tab character

4. Indent or de-indent a specific range

Add four spaces of indent to lines 10 through 25:

  10,25s/^/    /

Remove exactly four spaces of leading indent from lines 10 through 25:

  10,25s/^    //

Remove any amount of leading whitespace from lines 10 through 25:

  10,25s/^[[:space:]]*//

5. Resolve mixed tabs and spaces before resetting

If the file contains mixed tab/space indentation, normalize to spaces first. From inside an ed session, the ! command runs a shell command and reloads the result:

  w                           ← save current buffer first
  ,d                          ← delete all lines
  r !expand -t 4 file.txt     ← read back with tabs expanded to 4 spaces

Then proceed with ,s/^[[:space:]]*// to reset.

6. Undo

ed(1) supports a single level of undo with the u command. It reverses the most recent command that modified the buffer. There is no multi-level undo; if multiple commands have been applied, quit without writing (q) and re-open the last saved version:

  u              ← undo the most recent buffer change
  q              ← quit without writing (ed prompts if unsaved changes exist)
  Q              ← quit unconditionally, discarding all unsaved changes

NOTE: ed(1) will print a ? and refuse to quit with q if the buffer has unsaved changes. Type q a second time to force quit, or use Q. Neither q nor Q writes the file; only w does.

III. Results

Table I summarizes the behavior of each tool across both normalization tasks. Input files contained spaces, horizontal tabs, and in some tests carriage returns (\r).

Table II shows the key ed(1) indentation commands as a quick reference. All are typed at the * prompt inside an active ed session.

Before/after example. Input line with mixed indentation:

  BEFORE:  "  <p>Hello    world.</p>"
           (tab + 2 spaces before <p>; 4 spaces between words)

  AFTER strip+collapse:
    step 1  expand -t 4   →  "      <p>Hello    world.</p>"
    step 2  ,s/^[[:space:]]*/   →  "<p>Hello    world.</p>"
    step 3  ,s/[[:space:]][[:space:]]*/  /g  →  "<p>Hello world.</p>"

IV. Discussion

The choice of tool depends on a single primary question: must the operation cross line boundaries? If the answer is yes — for example, collapsing a paragraph that was broken across lines with a hard newline — only tr with [:space:] handles it directly in a single expression. All other tools are line-at-a-time and require a join step first.

A. Why ed for In-Place Editing

sed -i '' on BSD creates a temporary file, unlinks the original, and renames the temporary to the original name. This breaks hard links, resets the inode, and is not POSIX-specified. ed writes directly to the original inode via w: the file’s inode number, hard link count, and permissions are unchanged. For files under version control or with known inode references, ed is the safer choice [3].

B. The awk Field-Collapse Side Effect

awk '{$1=$1; print}' strips leading and trailing whitespace as a side effect of field splitting, then rebuilds the record using OFS (a single space by default). This is convenient but potentially destructive: it collapses whitespace inside quoted strings and HTML attribute values. Use it only on plain prose, not on structured markup.

C. Indentation in HTML Edited with ed

When composing or editing HTML in ed(1), the editor provides no structural awareness — there is no auto-indent, no bracket matching, no tree view. Indentation is the editor’s only visual cue to nesting depth. It drifts for two common reasons:

• The a (append) and i (insert) commands enter text verbatim; the user must type leading spaces manually on each line.
• Pasting pre-indented content from another source introduces a different indent convention (tabs vs. spaces, 2 vs. 4 spaces).

The two-command reset sequence is the practical solution: strip all leading whitespace first (,s/^[[:space:]]*/), confirm the result with 1,20n, then re-apply a uniform indent to the desired range. Because ed supports one level of undo, the strip step can be reversed immediately with u if the result is wrong.

WARNING. The substitution ,s/^[[:space:]]*/ is destructive across the entire file. Always write a backup first: w backup.html before running any global substitution. The u command only undoes the single most recent change; it cannot recover from a sequence of edits.

D. Handling Mixed Tab and Space Indentation

BSD cat -t reveals tab characters as ^I. If both tabs and spaces appear as indent, normalize to spaces before resetting. The recommended sequence from the shell:

  expand -t 4 file.html | sed 's/^[[:space:]]*//' > clean.html

Or entirely within an ed session (write first, then reload via the shell escape):

  w
  ,d
  r !expand -t 4 file.html

After reloading, all leading whitespace is spaces and the reset commands apply uniformly.

V. Conclusion

On a BSD Unix base system, tr -s '[:blank:]' ' ' is the most concise expression for collapsing horizontal whitespace runs while preserving line structure. tr -s '[:space:]' ' ' extends this across newlines at the cost of flattening the file to a single line. For in-place normalization that preserves inode identity, the ed(1) global substitution ,s/[[:space:]][[:space:]]*/ /g followed by w is the correct POSIX-portable approach.

For indentation reset when editing HTML in ed(1), the sequence to type is: ,s/^[[:space:]]*/ to strip all leading whitespace on every line, then n,ms/^/    / to re-apply a four-space indent to the desired range. Use 1,20n at any point to inspect line numbers and current indentation state. If mixed tabs and spaces are present, run expand -t 4 before any indent operation.

References

1. The Open Group. The Single UNIX Specification, Version 4 (POSIX.1-2017). IEEE Std 1003.1-2017. https://pubs.opengroup.org/onlinepubs/9699919799/
2. The Open Group. “Regular Expressions.” POSIX.1-2017, Volume 1: Base Definitions, Chapter 9. Section 9.3.5: Character Classes and POSIX Character Classes.
3. B. W. Kernighan and R. Pike. The Unix Programming Environment. Englewood Cliffs, NJ: Prentice-Hall, 1984, pp. 73–104.
4. D. Dougherty and A. Robbins. sed & awk, 2nd ed. Sebastopol, CA: O’Reilly Media, 1997.
5. FreeBSD Project. FreeBSD 14.0 Base System Manual Pages: ed(1), sed(1), tr(1), awk(1), expand(1). https://www.freebsd.org/cgi/man.cgi