[HN Gopher] Awk Technical Notes (2023)
___________________________________________________________________
Awk Technical Notes (2023)
Author : signa11
Score : 82 points
Date : 2025-11-03 10:40 UTC (11 days ago)
(HTM) web link (maximullaris.com)
(TXT) w3m dump (maximullaris.com)
| dietrichepp wrote:
| Awk is still one of my favorite tools because its power is
| underestimated by nearly everyone I see using it.
| ls -l | awk '{print $3}'
|
| That's typical usage of Awk, where you use it in place of cut
| because you can't be bothered to remember the right flags for
| cut.
|
| But... Awk, by itself, can often replace entire pipelines. Reduce
| your pipeline to a single Awk invocation! The only drawback is
| that very few people know Awk well enough to do this, and this
| means that if you write non-trivial Awk code, nobody on your team
| will be able to read it.
|
| Every once in a while, I write some tool in Awk or figure out how
| to rewrite some pipeline as Awk. It's an enrichment activity for
| me, like those toys they put in animal habitats at the zoo.
| RGBCube wrote:
| Stop using awk, use a real programming language+shell instead,
| with structured data instead of bytestream wrangling:
| > ls -l | get user +----+------+ | 0 | cube |
| | 1 | cube | | 2 | cube | | 3 | cube | |
| 4 | cube | | 5 | cube | | 6 | cube | | 7 |
| cube | | 8 | cube | | 9 | cube | | 10 |
| cube | | 11 | cube | | 12 | cube | | 13 |
| cube | | 14 | cube | | 15 | cube |
| +----+------+
|
| You don't need to memorize bad tools' quirks. You can just use
| good tools.
|
| https://nushell.sh - try Nushell now! It's like PowerShell, if
| it was good.
| electricEmu wrote:
| PowerShell is open source and available on Linux today for
| those who enjoy an OO terminal.
|
| MIT licensed.
|
| https://learn.microsoft.com/en-
| us/powershell/scripting/insta...
| anthk wrote:
| Once you get TSV and CSV related tools, nushell and psh are
| like toys.
| meken wrote:
| This sounds interesting. Could you give an example where you
| rewrote a pipeline in awk?
| ketanmaheshwari wrote:
| Not the op but here is an example: TOKEN=$(kubectl describe
| secret -n kube-system $(kubectl get secrets -n kube-system |
| grep default | cut -f1 -d ' ') | grep -E '^token' | cut -f2
| -d':' | tr -d '\t' | tr -d " ")
|
| This pipeline may be significantly reduced by replacing cut's
| with awk, accommodating grep within awk and using awk's gsub
| in place of tr.
| rbonvall wrote:
| Example of replacing grep+cut with a single awk invokation:
| $ echo token:abc:def | grep -E ^token | cut -d: -f2
| abc $ echo token:abc:def | awk -F:
| '/^token/ { print $2 }' abc
|
| Conditions don't have to be regular expressions. For
| example: $ echo $CSV foo:24
| bar:15 baz:49 $ echo $CSV | awk
| -F: '$2 > 20 { print $1 }' foo baz
| dietrichepp wrote:
| Somebody wanted to set breakpoints in their C code by marking
| them with a comment (note "d" for "debugger"):
| //d
|
| You can get a list of them with a single Awk line.
| awk -F'//d[[:space:]]*' 'NF > 1 {print FILENAME ":" FNR " "
| $2}' source/*.c
|
| You can even create a GDB script, pretty easily.
|
| (IMO, easier still to configure your editor to support
| breakpoints, but I'm not the one who chose to do it this
| way.)
| kazinator wrote:
| Why are you using the locale-specific [:space:] on source
| code? In your C source code, are you using spaces other
| than ASCII 0x20?
|
| Would you have //d<0xA0>rest of comment?
|
| Or some fancy Unicode space made using several UTF-8 bytes?
| wtallis wrote:
| Tab characters can also be found in source code.
| kazinator wrote:
| Since you control the \\\d format, why would you
| allow/support anything but a space as a separator? That's
| just to distinguish it from a comment like "\\\delete
| empty nodes" that is not the \\\d debug notation.
|
| If tabs are supported, [ \t]
|
| is still shorter than [[:space:]]
|
| and if we include all the "isspace" characters from ASCII
| (vertical tab, form feed, embedded carriage return)
| except for the line feed that would never occur due to
| separating lines, we just break even on pure character
| count: [_\t\v\f\r]
|
| TVFR all fall under the left hand, backspace under the
| right, and nothing requires Shift.
|
| The resulting character class does exactly the same thing
| under any locale.
| tetris11 wrote:
| one of the best word-wrapping implementations I've seen
| (handles color codes and emojis just fine!) is written in pure
| mawk
|
| very fast, highly underrated language
|
| I'm not sure how good it would be for pipelines, if a step
| should fail, or if a step should need to resume, etc.
| PopAlongKid wrote:
| >To Perl connoisseurs, this feature may be known as
| Autovivification. In general, AWK is quite unequivocally a
| prototype of Perl. You can even say that Perl is a kind of AWK
| overgrowth on steroids...
|
| Before I learned Perl, I used to write non-trivial awk
| programs. Associative arrays, and other features are indeed
| very powerful. I'm no longer fluent, but I think I could still
| read a sophisticated awk script.
|
| Even sed can be used for some fancy processing (i.e scripts),
| if one knows regex well.
| abhgh wrote:
| Love awk. In the early days of my career, I used to write ETL
| pipelines and awk helped me condense a lot of stuff into a
| small number of LOC. I particularly prided myself in writing
| terse one-liners (some probably undecipherable, ha!); but did
| occasionally write scripts. Now I mostly reach for Python.
| nerdponx wrote:
| > this means that if you write non-trivial Awk code, nobody on
| your team will be able to read it.
|
| Sort of! A lot of AWK is easy to read even if you don't
| remember how to write it. There are a few quirks like how gsub
| modifies its target in-place (and how its default target is
| $0), and of course understanding the overall pattern-action
| layout. But I think most _reasonable_ (not too clever, not too
| complicated) AWK scripts would also be readable to a typical
| programmer even if they don 't know AWK specifically.
| packetlost wrote:
| AWK, rc, and mk are the 3 big tools in my shell toolkit. It's
| great
| nmz wrote:
| Why mk instead of any of the other builders?
| benjaminogles wrote:
| I feel the same about using Awk, it is just fun to use. I like
| that variables have defined initial values so they don't need
| to be declared. And the most common bits of control flow needed
| to process an input file are implicit. Some fun things I've
| written with awk
|
| Plain text accounting program in awk
| https://github.com/benjaminogles/ledger.bash
|
| Literate programming/static site generator in awk
| https://github.com/benjaminogles/lit
|
| Although the latter just uses awk as a weird shell and
| maintains a couple child processes for converting md to html
| and executing code blocks with output piped into the document
| 1vuio0pswjnm7 wrote:
| stat -c %U *
| kevg123 wrote:
| Nice link to the canonical book on Awk within the first linked
| page in the article:
| https://ia903404.us.archive.org/0/items/pdfy-MgN0H1joIoDVoIC...
| layer8 wrote:
| There's a second edition: https://www.awk.dev/
| jjice wrote:
| Everyone should read "The AWK Programming Language". It's so
| short, with both the first and second editions floating around
| 200 pages in an A5 (could be off on that page size) form
| factor.
|
| Aside from AWK being a handy language to know, understanding
| the ideas behind it from a language design and use case
| perspective can help open your eyes to new constructs and
| ideas.
| cholantesh wrote:
| This was a great read, and the previous post in the series. I see
| a lot of very convincing arguments here
| (https://maximullaris.com/awk.html#why) but for me one of the
| biggest points in favour of python (and I say this is someone
| who, for learning, will always just reach for C++ because of my
| muscle memory) is its eminent readability. If I'm writing a
| script, quite a lot of the time, it's meant not just for myself
| but for my peers to use with some degree of regularity. I feel
| pretty confident that there would be much more operational
| overhead and a lot of time spent explaining internals with awk
| than with python.
| gist wrote:
| I am not a programmer, but I have used awk since the 1980's. And
| normally I would read this type of info or really many things
| about typical unix tools. I've done a small amount of helpful
| things with awk (again dating to the 1980's). (Wrote an
| estimating system using awk and flat txt files as an example).
|
| However given what I've been able to acomplish with Claude Code,
| I no longer find it necessary to know any details, tips, or
| tricks, or to really learn anything more (at least for the types
| of projects I am involved in for my own benefit).
| anthk wrote:
| FreeCell written in AWK:
|
| https://git.luxferre.top/nnfc/
|
| AWK goodies (git clone --recursive) :
|
| https://git.luxferre.top/awk-gold-collection
| knlb wrote:
| I used to be scared of Awk, and then I read through the appendix
| / chapters of "More Programming Pearls"
| (https://www.amazon.com/More-Programming-Pearls-Confessions-C...)
| and it became a much easier to reason about language.
|
| The structure can be a bit confusing if you've only seen one
| liners because it has a lot of defaults that kick in when not
| specified.
|
| The pleasant surprise from learning to use awk was that bpftrace
| suddenly became much more understandable and easier to write as
| well, because it's partially inspired by awk.
| svat wrote:
| I learned the basics of AWK in a few minutes from here:
| https://learnxinyminutes.com/awk/ -- and I agree with you, it
| was worth it!
___________________________________________________________________
(page generated 2025-11-14 23:00 UTC)