[HN Gopher] Show HN: A pure bash web server. No netcat, socat, etc.
___________________________________________________________________
Show HN: A pure bash web server. No netcat, socat, etc.
Author : dzove855
Score : 172 points
Date : 2022-01-04 14:20 UTC (8 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| [deleted]
| spicybright wrote:
| Definitely scary, but I love things like this. Takes a lot of
| creativity!
|
| There was a post a bit ago that had a one liner for a basic web
| server using net cat: while : ; do cat page.html
| | nc -l 80; done
| dzove855 wrote:
| Yeah sure. I have see it, but basically this one use bash
| builtins instead of external commands.
| hericium wrote:
| or even while :; do nc -l 80 < page.html;
| done
| skrtskrt wrote:
| what is the difference in these two approaches?
|
| I always use the pipe, it is intuitive to me to think left-
| to-right, output of this becomes input of next. But I always
| see stackoverflow answers using the other approach.
| arbitrage wrote:
| The difference is in the first code example, you're
| launching both cat and nc.
|
| In the second block, you're just launching one program that
| reads input, not two. It's slightly more efficient.
|
| I also tend to build large *NIX pipelines using `cat $foo |
| [...]`, but it's some times regarded as bad form. See:
| "useless use of cat" for more examples.
| aidenn0 wrote:
| If you want to do left-to-right, you can do that:
| while :; do <page.html nc -l 80 ; done
| tyingq wrote:
| But, bash can't do listen sockets.
|
| _" loadable accept builtin"_
|
| Ahh.
|
| Gawk can be a web server also, typically just "as-shipped",
| without any new extensions. Doesn't scale well though :)
|
| https://news.ycombinator.com/item?id=22085459
| drvdevd wrote:
| This was very interesting to me - I was expecting using linux
| socket virtual FS paths to create the socket.
| tyingq wrote:
| Pretty printed: https://gist.github.com/tyingq/4e568425e2e68e
| 6390f3105e58878...
|
| The /inet/tcp/8080/0/0 is a gawk-ism, versus a Linux thing:
|
| https://www.gnu.org/software/gawk/manual/html_node/TCP_002fI.
| ..
|
| Unfortunately, gawk doesn't let you control the listen/accept
| part of it, so it doesn't scale well. Not sure why they did
| it that way. A separate accept() call would have made it
| actually usable in a non-toy way.
| BeefWellington wrote:
| I appreciate the ingenuity here given the other recent posts, but
| this still requires executing an external application to perform
| the accept() part.
|
| Definitely closest to the mark though, kudos!
| dzove855 wrote:
| Basically yes, but it's a loadable builtin.
| mro_name wrote:
| I like those unorthodox, against-cargo-cult approaches. Thx!
| sillysaurusx wrote:
| This script taught me a few things! Since it's kind of impossible
| to google, what does this construct mean? while
| :; do
|
| It's clearly a while true loop, but why does an ascii crying face
| produce true?
| mturmon wrote:
| The other common idiom in scripts that uses the : is
| : > $LOGFILE
|
| which creates a zero-length log file, and removes the contents
| if it already existed.
| bewuethr wrote:
| Bash also has the equivalent "true" as a builtin, and Coreutils
| provides one as well, allowing to write this instead:
| while true; do
| veltas wrote:
| true is also a POSIX utility (not builtin), so it can be used
| in POSIX-y sh scripts as well.
|
| https://pubs.opengroup.org/onlinepubs/9699919799/utilities/t.
| ..
| framecowbird wrote:
| https://stackoverflow.com/a/3224910
| StanAngeloff wrote:
| See https://stackoverflow.com/questions/3224878/what-is-the-
| purp... $ type : : is a shell builtin
| veltas wrote:
| : is a sort of 'null' function in bash and bourne shells. It
| also always returns 0 so it's an alternative to 'true'.
|
| https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
| vultour wrote:
| What's impossible to google about that? "bash colon" gives tons
| of results
| tyingq wrote:
| Including the somewhat unexpected "Bottoms Up Bash - Colon
| Cancer Prevention Project".
| Trufa wrote:
| You get decent results if you put it on quotes.
| maxbond wrote:
| My humble submission to this genre is this pipeline I wrote once,
| when I needed an HTTP proxy, but I couldn't use install a real
| one do to work limitations, and I couldn't use firewall rules
| because of weird limitations of the Mac OS at the time:
| echo "" > /tmp/buffer; tail -f /tmp/buffer | netcat httpbin.org
| 80 | netcat -lp 8080 > /tmp/buffer
|
| I call it a circular pipeline.
| dundarious wrote:
| I know this is a toy meant for short-term use, but you can use
| a fifo/named pipe so you don't have a growing file used for
| data transfer. mkfifo -m0600 /tmp/buffer &&
| netcat httpbin.org 80 < /tmp/buffer | netcat -lp 8080 >
| /tmp/buffer
|
| Might want the file if you want a record of course.
| maxbond wrote:
| Thanks! TIL! That's a much cleaner way of doing it.
|
| Regarding keeping the file, if memory serves the "real"
| pipeline had some `tee`s in it. I recreated this much more
| recently to make a GitHub repo of funny code snippets
| (https://github.com/MaxBondABE/one-thread-crashing/).
|
| You reminded me of a different toy I made many years ago. I
| created FIFOs for files that you might try to read when
| exploiting an arbitrary file read in a web app, say
| /etc/www/apache.conf (if that's the right path - it's been a
| while since I've configured a web server!) and I'd have a
| program that opened them. This would block until someone
| opened the other end of the FIFO, at which point I'd raise an
| alarm (which was just a print statement).
| thyrox wrote:
| That's awesome but would you mind explaining this a little bit
| more to those not very unix savvy? I think I get the first
| three parts:
|
| 1) Create a temp buffer file
|
| 2) Read the file constantly to STDOUT?
|
| 3) What does netcat httpbin.org 80 | netcat -lp 8080 do?
|
| What would be a good use of this circular pipeline? Can you
| give me some example of what you use this for? Thanks
| maxbond wrote:
| Of course!
|
| A.)
|
| Some context:
|
| - Our goal here is to match up to two network connections, so
| that anything we receive on one is forwarded to the other.
|
| - netcat is a "networking swiss army knife", it's a tool for
| making network connections
|
| - HTTP and bash pipelines both happen to be "line oriented",
| meaning they operate one line at a time. Exploiting this
| happy coincedence is what makes this work
|
| - httpbin.org is a metanym for any website you need.
|
| So: echo "" > /tmp/buffer
|
| Initialize our buffer to an empty state
| tail -f /tmp/buffer
|
| Read each line in the buffer, one at a time, and feed them to
| the next command. Do this forever (-f).
| netcat httpbin.org 80
|
| Connect to the website httpbin.org. Transmit each line from
| our buffer to the website. netcat -lp 8080
| > /tmp/buffer
|
| Accept a connection. Transmit any data we receive from
| httpbin.org to this client; write any data we receive from
| the client to the buffer, so that it can be transmitted to
| httpbin.org.
|
| B.) I don't think there is one! I've never needed to do this
| again in the, gosh, seven intervening years. Normally bash
| pipelines operate like a bucket brigade, moving data
| unidirectionally through various processing steps, and
| usually dumping it into a file at the end.
|
| They aren't normally for creating long-lived programs that
| manage bidirectional flows of data. Which is why writing a
| web server in bash is a fun challenge.
| dundarious wrote:
| I'm not OP, but as they said, it's an HTTP proxy. Run it,
| then point your browser at http://localhost:8080. The message
| is proxied over to http://httpbin.org (on port 80, the normal
| HTTP port). If you're in an overly restricted environment and
| have a legit need to proxy to a very limited set of hosts,
| you can ad-hoc make a proxy mapping from local port to a
| single specific remote host. Just change the 8080,
| destination host, and buffer file.
|
| And I mean this as encouragement, but you should read the
| synopsis from the GNU netcat manual. It's short and explains
| it well enough. netcat httpbin.org 80 is using connect mode,
| the -l one is using listen mode, and -p is --local-port (the
| port to listen on in listen mode).
| dheera wrote:
| If Bash is Turing-complete could we just make a Python to Bash
| transpiler?
| pmarreck wrote:
| The other day I wrote this bash code after getting frustrated
| with the password generators available to me, just so I had
| something to use right in the shell:
|
| https://gist.github.com/pmarreck/f4dbb02396c4532762a7a64511c...
|
| It actually took an entire afternoon and went through a couple
| iterations, and I hadn't even written a test for it yet (and it
| just so happens that I started an MVP shell-script testing
| library for just such an occasion when I ALSO got frustrated with
| the bash-accessible testing libraries available to me, also in
| Bash: https://github.com/pmarreck/tinytestlib)
|
| But then I immediately felt bad. I could have written all of this
| in less than an hour in a language like Elixir, with test
| coverage (and the testing/assertion lib is built-in, so it would
| have been free).
|
| It is impressive and code-golfish that someone can "write a
| webserver in pure Bash" but Bash is a crap language to script in
| at this point, and adding more Bash code to the world just seems
| wrong now. Bourne shells were invented in the 70's (!), long
| before many language advancements and understandings, and they
| thus have MANY warts (many of which we're familiar with).
|
| The next time I feel the urge to code anything more than a one-
| liner in Bash, I'm writing it in elixir script
| https://thinkingelixir.com/2019-04-running-an-elixir-file-as...
| and just sucking up the extra run time (or compiling it into an
| executable with something like
| https://github.com/spawnfest/bakeware/ and sucking up the extra
| megabytes), and then MAYBE writing a wrapper function for my
| .profile
|
| Incidentally, if you DO want a shell that "tastes like" Bourne
| but is functional and thus sucks a lot less, check out
| https://github.com/wryun/es-shell, or my fork of it
| https://github.com/pmarreck/es-shell/
| foxhop wrote:
| Here is a fun one I made:
|
| https://github.com/russellballestrini/bash-kira/blob/master/...
|
| Kira (bash-kira): a small bash script for killing programs
| which run too long.
| dzove855 wrote:
| Bash is not mean for kind of projects like this. I even develop
| on other languages. But i like bash. A good bash script, is
| like reading a novel, because it's challenging to have a clean
| script.
| pmarreck wrote:
| I get it. The challenging part is fun. You can learn all the
| corner cases, the clever bash-isms, and then show off. But
| the code you end up with is essentially unmaintainable,
| especially since you don't have any test coverage, so...
| dzove855 wrote:
| Well that's true. I took me two hours to get this done.
| Since accept was not clear at the beginning. But it will be
| cleaner in the future, i'm still not convinced how it works
| currently.
| carlhjerpe wrote:
| You could be a doctor! "I don't really know why or how
| this works, but I'll prescribe enough for you to last a
| month".
| ramses0 wrote:
| This is my favorite "reference" bash script:
| https://git.einval.com/cgi-
| bin/gitweb.cgi?p=abcde.git;a=blob...
|
| ...it feels like "if it's possible to do in bash, this does
| it".
| istjohn wrote:
| 5500 lines of bash. Wow.
| wgjordan wrote:
| > and it just so happens that I started an MVP shell-script
| testing library for just such an occasion when I ALSO got
| frustrated with the bash-accessible testing libraries available
| to me
|
| I found Bats straightforward and simple to use, never felt the
| need to reinvent the wheel there.
| pmarreck wrote:
| BATS is
|
| 1) bloated
|
| 2) doesn't use a very nice syntax (hint: bashisms are not a
| very nice syntax). Compare BATS assertions with this for
| example (the test of my test library, which uses the
| library's own functions):
| https://github.com/pmarreck/tinytestlib/blob/yolo/test Call
| me crazy but I think `assert_equal "4" "$result"` is a lot
| more readable than `[ "$result" -eq 4 ]`
|
| 3) doesn't let you assert on all of (return code, stdout,
| stderr) from running a single command. I created a `capture`
| function to do that and bring those all into named variables
| that you can then assert on.
|
| But overall I wanted something tiny that you could easily
| include alongside a bash script instead of establishing as a
| new dependency you'd have to install separately. Honestly, I
| feel that functions like assert_equal, assert_not_equal,
| assert_success, assert_failure, assert_match and
| assert_no_match (which all raise on failure) should be built
| into every programming language's standard library.
|
| I know that you can sort of get by just using `set -e` and
| just do a comparison operation, but...
| codeivore wrote:
| one day this code will end up on a small/home office router
| pdkl95 wrote:
| Regarding the creation and removal of a tempfile (line 72 & 85):
| local tmpFile="${TMPDIR:-/tmp/bash-web-server.$$}" #
| ... rm "$tmpFile"
|
| To avoid collisions when the PID is reused, and to clean up0 the
| tempfile on errors, I recommend using mktemp and trap:
| local tmpfile="$(mktemp --tmpdir="${TMPDIR:-/tmp}" bash-web-
| server.XXXXXX)" trap "rm -f \"${tmpfile}\"" RETURN EXIT
| # ... # Do nothing at the end of the function; trap
| will # remove the file at RETURN automatically.
|
| Otherwise, I like the implementation! It's nice to see good bash
| techniques like parsing with "IFS='&' read -ra data" and
| rewriting variables with %%/etc.
| dzove855 wrote:
| I always use mktemp and trap to remove tmp files.
|
| But in this script i would like to avoid external commands. I
| will even remove the use of tmp files kn thw future.
| goombacloud wrote:
| To spare you the trap you can open the file as FD in your
| bash process (e.g., exec {my_fd}>"$TMPFILE") and then
| directly delete it before doing anything else, and use the
| /proc handle to access it ("/proc/$$/fd/${my_fd}")
| ghthor wrote:
| That's super clever, thanks for pointing this out!
| stouset wrote:
| That's an awesome technique, thanks for it! It _is_ Linux-
| specific though.
| Klasiaster wrote:
| It's also useful for signaling between processes to have
| them continue doing something as long as this proc path
| exists (the fd can also just be backed with >/dev/null
| instead of a real file that needs to be removed)
| nottorp wrote:
| I wonder if it's faster than the same thing in javascript...
| dzove855 wrote:
| Well benchmarking it os currently not possible. I will try it
| in a few days and provide some benchmarks.
| tyingq wrote:
| Most of the node.js solutions I'm aware of would support
| http/1.1 and async so I can't imagine this faring well in a
| performance comparison.
| benbristow wrote:
| Right, time to make this into a Docker image and ship it to
| production
| encryptluks2 wrote:
| I used to see value in scripts like this, but often these Bash
| scripts do quite the opposite. It may be readable for those that
| only know Bash, but you'll probably be happier if you learn Go or
| C. There are also a good amount of lightweight C web servers like
| darkhttpd.
| drran wrote:
| It's written in C:
| http://git.savannah.gnu.org/cgit/bash.git/tree/examples/load...
| ape4 wrote:
| Sort of related, if you do "man read" you the the huge "bash" man
| page. It would be nice if man could be improved to take you to
| the "read" command within that page. Or a new command "manbash
| read"
| dzove855 wrote:
| Well just type: help BUILTIN
|
| Example : help read
| garaetjjte wrote:
| This prints abridged usage help, though. You can use `info
| bash read`, but confusingly it might not always contain the
| same content as manpages.
| umvi wrote:
| Ha, this is more like a puzzle than anything - trying to
| implement a protocol in a highly restrictive environment.
|
| For anyone looking for an easy way to spin up a command line web
| server, python has one built in, so if you are running linux
| usually you can do:
|
| `python3 -m http.server`
|
| From a terminal and it will spin up a web server that serves the
| current directory. I use this all the time for quick tinkering
| vs. installing/configuring nginx/node.js/caddy or whatever the
| cool kids are using these days.
| asiachick wrote:
| and here's lots of alternatives since that solution has issues
|
| https://stackoverflow.com/questions/12905426/what-is-a-faste...
|
| Also, the python solution is only relevant if python is already
| installed. Something that's not true on Windows and is or will
| soon be true on MacOS.
| TingPing wrote:
| Why does it matter if it's included, it's trivial to install
| and extremely broadly useful?
| teh_klev wrote:
| > Something that's not true on Windows.
|
| Getting python up an running on Windows is fairly trivial
| though. You don't even need local admin permissions if your
| machine is locked down by your IT dept.
| vimsee wrote:
| Will Apple ditch Python in the future? Typing on an M1 Air
| and I`m pretty sure Python came per-installed.
| moreati wrote:
| macOS 12 dropped PHP. It still has /usr/bin/python (CPython
| 2.7), but - any app the uses it triggers a message box
| about the the developer needing to update their app -
| before any REPL session it prints
|
| """ WARNING: Python 2.7 is not recommended. This version is
| included in macOS for compatibility with legacy software.
| Future versions of macOS will not include Python 2.7.
| Instead, it is recommended that you transition to using
| 'python3' from within Terminal. """
|
| There is no version of Python 3 included with the OS. The
| XCode app/developer tools include Python 3.8. I expect
| macOS 13 won't include any Python interpreter out the box
| 0xbadcafebee wrote:
| If you have access to a _busybox_ executable, it comes with a
| quite capable little httpd with /cgi-bin/ for CGI apps. If you
| have a wifi router, the web interface might be running off this
| :) (https://git.busybox.net/busybox/tree/networking/httpd.c)
| pcranaway wrote:
| I use darkhttpd from time to time, it supports streaming:
| https://github.com/emikulic/darkhttpd
| masklinn wrote:
| The biggest issue with `http.server` is it's a blocking single-
| threaded server, so it can serve a single client at a time.
|
| This makes it really _really_ not great when trying to serve
| pages (with various attending static files), or to ad-hoc share
| large resources with other people on the network.
| javajosh wrote:
| A single-threaded server is fine if you're only mildly IO
| bound, which is the case when you're serving files-as-
| resources. My understanding anyway is that http.server is
| explicitly not for production use!
| masklinn wrote:
| > A single-threaded server is fine if you're only mildly IO
| bound, which is the case when you're serving files-as-
| resources.
|
| It's fine if you're also evented, but that's not the case
| here, http.server literally can only interact with one
| connection at a time, the next connection(s) will be queued
| up waiting for http.server to finish its thing and accept
| them.
|
| > My understanding anyway is that http.server is explicitly
| not for production use!
|
| Sure, but "I'm at a lan party and have the installer for
| $game so we don't explode the 'net connection" is not the
| sort of "production use" that's intended by this statement.
|
| Neither is "I want to open this HTML file I wrote which has
| a bunch of static assets booting up the webapp".
| londons_explore wrote:
| It also means a mis-configured client that doesn't close
| the connection can hang the server.
|
| Looking at you HttpClient-on-esp8266...
| jcims wrote:
| We had an engineer add it to his unit tests and started
| blaming the network team when everything started slowing
| down. Took me two days to convince him to try another
| server.
| teh_klev wrote:
| Parent does point out they use this _for quick tinkering_.
| masklinn wrote:
| Yes, and I point out issues I've hit when using this _for
| quick tinkering_.
| pxeger1 wrote:
| printf '%s\n' "$(<$tmpFile)"
|
| Why not just cat $tmpFile printf '\n'
| dzove855 wrote:
| to avoid external calls
___________________________________________________________________
(page generated 2022-01-04 23:01 UTC)