[HN Gopher] Beej's Guide to Network Programming
___________________________________________________________________
Beej's Guide to Network Programming
Author : mutant_glofish
Score : 461 points
Date : 2023-05-21 23:28 UTC (23 hours ago)
(HTM) web link (beej.us)
(TXT) w3m dump (beej.us)
| saboot wrote:
| I'll ask here in case I can get help.
|
| I followed Beej's linked guide to transfer data over udp sometime
| ago, written in C on a Windows machine. I then compiled the code
| on Mac os x but could never get it to work properly. Tried all
| the Google suggestions at the time, posting this comment in case
| there's an obvious "oh did you set X?" tip from someone that has
| done a similar task.
| a2800276 wrote:
| Maybe the reason you're not getting help is is that you're not
| asking a question. You're not even describing a problem. You're
| only stating that you have a problem.
| daydream wrote:
| What was the specific issue you ran into?
| Dowwie wrote:
| I've been working on the Protohackers [1] exercises in Elixir,
| putting my own spin on the problems by creating fault tolerant,
| scalable, high performance solutions-- and have been touching a
| lot of Elixir that I haven't had the opportunity to otherwise.
| The "Unusual Database" problem, particularly, has become a very
| deep rabbit hole assignment as it involves a UDP-based KV
| database. I stopped just short of clustering.
|
| Writing servers in Elixir is straightforward because of the
| foundation exposed by Erlang's gen_tcp and gen_udp libraries.
| When you want to take your TCP server up a notch, try TCP
| acceptor pooling. There are two viable TCP acceptor pool
| libraries: Thousand Island, written in pure Elixir, and Ranch,
| written in Erlang.
|
| Because of these well- abstracted foundation libraries and
| acceptor pool libraries, it's possible to write robust servers
| without venturing much into low level network programming. Beej
| explains the lowest level details well. I've done some of the
| equivalent work in Rust but found the slow progress of my work
| unsatisfying and moved back to Elixir quickly.
|
| [1] protohackers.com
| macintux wrote:
| I was disappointed last night when I looked at the Protohackers
| website and it seems to be on hiatus. Such a great idea, but I
| imagine it's not trivial to manage.
| Dowwie wrote:
| I'm grateful that the author managed to take it even this
| far!
| ftxbro wrote:
| > "The way I see it, it's the educators who hold all the power in
| the world. No, stop laughing and hear me out. A good instructor
| can, over the course of their career, create hundreds of millions
| or even billions of dollars of GDP. Let's say a teacher teaches
| 40 students a year for 30 years. And Each of those students goes
| on to work for 40 years. And, due to the quality of instruction,
| they earn $10,000 more per year than they would have otherwise.
| That's $480 million in extra revenue. From one good instructor."
|
| OK but providing value isn't the same as having power. It's
| almost the opposite thing.
| oreally wrote:
| Yea he's using economic output as a measure.. then equating it
| to having power, which is laughable.
|
| Anyway, in an age of information abundance and videos,
| educators are overvalued if they just give instruction alone.
| It's the whole package, the checks and balances on the student,
| that possibly provides some sort of value. Then again, there
| are irregularities, like the programmer who's never been to
| college who contributes as much or even more.
| whimsicalism wrote:
| What are you quoting?
| ftxbro wrote:
| it's beej saying why he write guides
| jrumbut wrote:
| > OK but providing value isn't the same as having power.
|
| Providing value is absolutely a way of having power. You change
| the world, that's power.
|
| Having power isn't the same as yielding benefit (for yourself).
| TeMPOraL wrote:
| When people talk about having or not having power, they
| rarely mean magnitude alone. The more important component is
| _control_. Can you wield your power to achieve some specific
| goal of yours? Can you use this power to overall improve your
| life, and lives of those you care about? Can you use this
| power to protect yourself from someone else using _their_
| power to achieve something at your expense?
|
| Beej's teaching power only answers "Yes" to the first
| question here - and that's only if your goals have something
| to do with teaching or improving the world overall,
| somewhere, for some people.
| jrumbut wrote:
| I'm not sure I understand your third question, but I think
| beej's teaching power fulfills the second question in a
| very radical way which is that it is an act of caring for a
| large audience of strangers and does improve their lives.
|
| Power that isn't attached to the desire to reap a direct
| benefit is a strange thing to witness, but it's power
| nonetheless!
| alberth wrote:
| Beyond this being a fantastic document, I really appreciate it
| being offered as a single HTML page:
|
| https://beej.us/guide/bgnet/html/
| schaefer wrote:
| And a physical print book.
| asdfkljvalkj wrote:
| The funny thing is at the time downloading that took like 5
| minutes back in the day.
| p4bl0 wrote:
| I love this guide, I gave this link as a reference to my students
| when I was teaching networks a few years ago. But the last time I
| looked at it the guide didn't have IPv6 taken into account! Very
| nice to know that it is now the case! Thanks for posting :).
| davidw wrote:
| Pre-pandemic, beej used to come to programmer/hacker events here
| in Bend. Haven't seen him in a while though.
| caust1c wrote:
| Beej's guides are fantastic. He went to Chico State and I was
| happy to learn that his books were used for teaching network
| programming there when I attended.
|
| Genuinely inspirational hacker. Feel's great to have had that
| experience at a state school.
| jackphilson wrote:
| If you're trying to learn networking, better to start off with
| the foundational knowledge rather than jump straight into the
| code. Right now I'm going through the networking section on
| https://teachyourselfcs.com/ which is pretty good so far.
| ascar wrote:
| Contrary to popular belief, formal education wasn't useless
| here. All the foundational knowledge was covered in depth in my
| mandatory CS undergrad networking class. So this is a neat
| quick guide to all the stuff CS graduates already have learned
| once about but might've forgotten the details or don't know the
| hands on code.
| samtho wrote:
| I think there is (or was, depending on the institution) a
| disconnect between the curriculum and what skills students
| needed. For context, I was a tutor, instructional assistant,
| adjunct professor, and finally a curriculum advisor for the
| Los Rios Community College system and the University of
| California Davis Extension.
|
| First, CS programs that went 100% into Java did not age well
| and most of these courses were taught in a way that led
| students on to believe this stuff was static and won't really
| improve. The concept of new languages beyond what was
| available in circa 2006 was unimaginable.
|
| Another quirk that a Java-centric education brought on was
| that low-level courses were often taught in Java and you
| ended up writing all your data structures coursework in Java
| taught by someone with a C background. But you couldn't use C
| or anything sensible for this because universities promised
| students would know object oriented programming. This created
| a lot of disconnect and meaningless ritual around a paradigm
| that did not fit the language.
|
| CS programs that succeeded understood that a foundational
| knowledge coupled with language flexibility. They had to
| understand that the specific languages did not matter so
| much, rather it should fit the paradigm and principles of the
| context in which it was written. Once students are able to
| think in terms of computer procedures, instructors can
| leverage this knowledge to introduce abstractions and a grand
| unification principle.
| d1n037 wrote:
| Funny I was just going over this again yesterday. Also didn't
| know he had one called Beej's Guide to Network Concepts[0], which
| has some really good explanations in it.
|
| [0] https://beej.us/guide/bgnet0/html/split/
| [deleted]
| keyle wrote:
| There is also a C guide [1] which has a pretty interesting email
| policy... I'm generally available to help out
| with email questions so feel free to write in, but I can't
| guarantee a response. I lead a pretty busy life and
| there are times when I just can't answer a question you have.
| When that's the case, I usually just delete the message. It's
| nothing personal; I just won't ever have the time to
| give the detailed answer you require. [...]
|
| [1] https://beej.us/guide/bgc/html/split/
| mu53 wrote:
| I love this level of honesty. It is true, but people never say
| it
| ryandrake wrote:
| I had a work colleague with a similarly refreshing Out-Of-
| Office E-mail autoresponder that included something like:
| "I'm on vacation. When I get back I'm not going to have time
| to read all my missed E-mails. So your mail is going to be
| ignored and deleted with the other 2,000 unread mails. If
| it's important, E-mail me again after I return on [date]."
|
| That's how you do it!
| keyle wrote:
| There is a massive caveat to this strategy "Email me again
| after I return".
|
| Tssk tssk. Big mistake. Always do the "If it's urgent,
| contact <someone else>" approach ;)
| ly3xqhl8g9 wrote:
| Here's a product (Gmail/Outlook add-on or so): have your own
| LLM trained over your public texts (articles, books, blog
| entries, READMEs, Hacker News comments, and so on), mark the
| emails you can't respond to with the "GPT" label which puts
| the mail in a queue and if you don't remove the label in 7
| days or so, it sends a response with a polite intro "I didn't
| have the time to get to your email, but here's what I might
| think as a response".
| InitialLastName wrote:
| Yep, absolutely no career-terminating downside opportunity
| in this one.
| ly3xqhl8g9 wrote:
| Good thing that with the coming devastating job purges
| due to statistical learning such "careers" based around
| bullshit jobs [1] where some words in a generated email
| can be "career-terminating" will be forever gone.
|
| [1] https://en.wikipedia.org/wiki/Bullshit_Jobs
| InitialLastName wrote:
| I think I'd argue that the people whose emails can be
| sufficiently answered with an LLM are the ones whose jobs
| are first on the chopping block, so the product has a
| self-destructing user base.
| ly3xqhl8g9 wrote:
| Not necessarily. When I was writing the initial comment I
| wasn't even thinking about job-related mails, but at
| someone like Sean Carroll who does a regular Ask me
| Anything [1] where lots of questions are skipped for lack
| of time, or in general academics who receive plenty of
| emails about articles and various field-related
| questions.
|
| [1] at the beginning of each month, May 2023
| https://www.youtube.com/watch?v=h_GD4E27Bt8
| lannisterstark wrote:
| Why not like ...just leave the message alone lol?
|
| We don't have 64 Kilobytes of limited memory on our email
| servers anymore.
| pluijzer wrote:
| When you know you definitly do not have time for an answer
| deleting it might unload your mental burden. Keeping hundreds
| of unamswered messages around might feel like an unresolved
| issue.
| remram wrote:
| There's no functional difference between "not looking at it
| ever again" and "deleting it".
|
| You should understand when they say "delete" that they mean
| "remove from their queue", whether it still exists in an
| "archive" folder, backups, trash, ... is irrelevant.
| Jorengarenar wrote:
| Liking to have clean and empty inbox isn't related to limited
| memory
| bachmeier wrote:
| I can speak from experience that posting that type of material
| will generate some interesting email inquiries. I used to get
| email messages with a 4000-line program, multiple datasets, and
| "My program doesn't work. Can you tell me what the error is?"
| The awkwardness of deleting the message is less than the
| awkwardness of refusing what someone else views as a 30-second
| request.
| nickelpro wrote:
| This guide taught me how to program. I owe my entire career to
| beej
| kqr wrote:
| Open question: in what fields are you working where you _really_
| need to work with actual sockets? Most of the network programming
| I do fall into one of two categories:
|
| 1. A library exists to abstract over the bytes-into-sockets
| layer, meaning I don't need to deal with it; or
|
| 2. A library probably exists to abstract over the bytes-into-
| sockets layer, but I've chosen to ignore it either because I'm
| truly ignorant of it, or for intellectual stimulation.
|
| (Note that in category 2 I count things like "not using
| protobuf/thrift/etc. for IPC.)
|
| When are these things not true? Genuinely curious! I'm guessing
| interfacing with existing obscure third parties, or using obscure
| language environments, but what are examples of that? And what
| other usages are there?
| samtho wrote:
| I often work in a resource constrained environment where my
| program size is measured in two-digit kilobytes, RAM is
| measured in the single-digit kilobytes, and it can't exceed
| some current measured in mA/h. When I am simply sending a UDP
| packet, I didn't need nor can I afford the footprint that these
| libraries have.
| DoingIsLearning wrote:
| Had plenty of use cases where even zmq was overkill.
|
| E.g. interfacing a research machine with an OEM machine or a
| machine to a standalone embedded target, sometimes it makes
| sense to just serialize, send over a socket, ethernet crossover
| cable and be done.
| kqr wrote:
| Your description makes it sound sort of like temporary code
| to perform a one-time (or few-times) job, and not something
| that's running and maintained for years. Is that a fair
| characterisation or am I misunderstanding?
| kragen wrote:
| at knownow i wrote an event-driven nonblocking http server in
| python with asyncore which could efficiently do comet over
| hundreds of concurrent http connections; existing http
| libraries were all blocking at the time
|
| i also wrote a similar event-driven nonblocking http client in
| c for benchmarking and load-testing both that server and a
| different compatible one being written by another team in c++.
| this was nonblocking because it was important for the client to
| be faster than the server. at the time there was no
| libevent/libev/libuv so i wrote my own
|
| at satellogic i maintained the upper layers of our cubesat
| space protocol stack, which provided a socket-like interface to
| our coap stack, which i also maintained part of
|
| i also had to tweak kernel socket buffering settings for
| rabbitmq there
|
| for wercam i'm experimenting with different approaches to ipc
| for efficiency; the data being transferred are pixel buffers so
| i don't think protobuf/thrift/etc. is going to help
|
| for the mail server i share with some friends i wrote a milter
| in python to smtp-reject outside email to certain recipients
| unless it matched a whitelist; milter doesn't speak pb either
| and i couldn't find a python library
|
| i wrote a multicast file transfer program in python
| (bccpo/bccpi) for copying files across the lan without having
| to fiddle around with ip addresses; the data being transferred
| is just (multicast) the ip and port and (unicast) the filename,
| file size, and file contents, so again i'm not sure pb would
| simplify anything. the hard part of getting it working was just
| multicast
|
| i wrote a chat client for icb in python with asyncore, and
| again i couldn't find a suitable library
|
| even when i'm using a library, it's often pretty useful to be
| able to read strace output or use select or poll with the file
| descriptor it's trying to hide from me
|
| i don't think any of these qualify as either your category 1 or
| 2
|
| i've also done a huge number of things that fall into category
| 2, of course, like writing an irc client in bash, writing a web
| server in assembly, and writing a chat server in three
| 80-column lines of c, writing a mud, and so on
| GoblinSlayer wrote:
| At least on Unix and Windows file read and write methods work
| for sockets too, so streaming bytes isn't different.
|
| How would you implement, say, netcat? Launch a jango rest
| service with kubernetes, which will run in a wm if it's a wrong
| OS, create a certificate with letsencrypt and access it with a
| react electron app?
| kqr wrote:
| Why would I implement netcat when netcat is already
| implemented? That's sort of why I'm asking - at least in the
| circles I move, all the low hanging fruit are already picked,
| and picking the high fruit seems rare, so I'm interested in
| which people do and what those fruits are!
| fredrb wrote:
| I don't work with it either. But I find that learning the
| abstraction below where you're working at can be quite
| beneficial to understanding the constraints of your layer,
| debug, and solve problems.
|
| E.g.: Learn the basics Transport Layer protocols (TCP/UDP) if
| you work with HTTP
| synack wrote:
| It's useful to understand the socket API so that you can do
| things like read the output from strace, notice that accept()
| returns EAGAIN too often, and know what that indicates.
| JustSomeNobody wrote:
| I implement protocols to communicate with various hardware
| (RFID, LiDAR, etc). Most of these communicate over sockets.
| Some still use serial. I have a library I wrote of socket
| routines that I use, so it's not like I re-implement that every
| time, though. And for this, I'd really rather use my library
| than a 3rd party in case I have to dig in and see what's going
| on.
| dan-robertson wrote:
| 1. Writing the libraries you describe
|
| 2. Speaking unusual protocols for which (good) libraries don't
| exist.
|
| 3. I think interacting with the sockets api can also be
| relevant for tweaking various options to get better performance
| (though often there are Linux defaults that can be tweaked
| instead)
| kqr wrote:
| I'm fully aware of the hypotheticals - what I'm curious about
| is which your personal experiences are with e.g. those three
| points you list.
| dan-robertson wrote:
| I've done/do all three of those things. Knowing about the
| networking stack (protocols, the various things inside
| Linux, etc) has been pretty relevant.
| forrestthewoods wrote:
| There is a whole universe of programming that exists outside of
| webdev. Firmware, video games, high performance systems, etc.
| JustSomeNobody wrote:
| This. I think HN leans very heavy on webdev and somehow
| forgets that the entire rest of the planet runs on software
| also.
| zabzonk wrote:
| it is what a great short guide should be. but i am sure dang will
| be posting a huge number of previous links.
| peter_l_downs wrote:
| Posted a million times before, and should be posted a million
| times again in the future. This is one of the only reference
| books I've ever used, and it got me through my first real
| programming job. Wonderful text. Thank you, Beej!
| eep_social wrote:
| This is probably the best one:
| https://news.ycombinator.com/item?id=26100075
| gabagool wrote:
| Could be wrong, but isn't this the same Guide?
| eep_social wrote:
| whoops, best thread is what I was thinking.
| bdcravens wrote:
| I think that's the point. They were talking about how many
| times it's been posted, so that's probably the best of
| those submissions.
| snarkypixel wrote:
| This guide was my bible when I was a kid. I learned not just
| network programming, but also C with it.
| CamperBob2 wrote:
| That's how Andrej Karpathy's video lectures are shaping up for
| me... I'm learning a lot about not just neural networks, but
| Python and Pytorch.
|
| It's more or less magical when you find just the right
| conjugate match between tutorial material and your own learning
| style. I can see how Beej's Guide might do the same for someone
| new to both socket programming and C.
| b33j0r wrote:
| My nick has been Beej or b33j ever since starcraft or
| counterstrike 1.5.
|
| I will always bring up "but you know who the real beej is,
| right?" if the other person is a nerd.
| girvo wrote:
| > counterstrike 1.5
|
| I miss WON. Man my friends and I hated Steam when it came
| out...
| tomcam wrote:
| Now maybe I have to change all my usernames to animats0r for
| the same reason
| b33j0r wrote:
| Funny enough I only got this joke on my second read. One of
| my uncles had a company with a name similar to your joke :D
| ADD probably.
|
| So, to me it was beej. But in reality, to the public [you
| can't nickname yourself].
|
| So it's always been b33j0r. My friend who had the first
| dedicated server I encountered named me that because my name
| is Brian Jor---
|
| Finding me has been trivial since I first AWP'd in a CAL-O
| match. This was before scoutznknives were more than merit
| badges.
| [deleted]
| dang wrote:
| Related (note: reposts are fine after a year or so -
| https://news.ycombinator.com/newsfaq.html):
|
| _Beej 's Guide to Network Programming (1994-2020)_ -
| https://news.ycombinator.com/item?id=26100075 - Feb 2021 (165
| comments)
|
| _Beej 's Guide to Network Programming_ -
| https://news.ycombinator.com/item?id=13983212 - March 2017 (44
| comments)
|
| _Beej 's Guide to Network Programming_ -
| https://news.ycombinator.com/item?id=13670971 - Feb 2017 (6
| comments)
|
| _Beej 's Guide to Network Programming_ -
| https://news.ycombinator.com/item?id=12402313 - Aug 2016 (4
| comments)
|
| _Beej 's Guide to Network Programming (2012)_ -
| https://news.ycombinator.com/item?id=9445692 - April 2015 (31
| comments)
|
| _Beej 's Guide to Network Programming_ -
| https://news.ycombinator.com/item?id=7584974 - April 2014 (1
| comment)
|
| _Beej 's Guide to Network Programming_ -
| https://news.ycombinator.com/item?id=5241220 - Feb 2013 (47
| comments)
|
| _Beej 's Guide to Network Programming_ -
| https://news.ycombinator.com/item?id=584557 - April 2009 (22
| comments)
|
| _Guide to Network Programming_ -
| https://news.ycombinator.com/item?id=337371 - Oct 2008 (7
| comments)
| ghoshbishakh wrote:
| This is a fantastic guide and this was used in our programming
| lab at IIT Kharagpur. The lab was headed by an equally awesome
| instructor - Abhijit Das [1]
|
| [1] https://cse.iitkgp.ac.in/~abhij/
| thrwawy74 wrote:
| I'm not saying Rust is the answer to all things, I'm saying I
| want a Rust translation for continuity of the wisdom and
| concision I loved in this guide as a 14 year old.
| deaddodo wrote:
| The only problem is that it wouldn't be quite as useful. Zig,
| Rust, D, etc offer libraries and methodologies (async/await,
| for instance) that deprecate traditional raw socket
| programming. Instead, you would use something like tokio in
| Rust to do the same thing.
|
| You certainly _can_ program raw sockets, the techniques are
| almost orthogonal to the beej guide. It 's the same API and
| Structs (using FFI) and you would change some glue logic to
| make it more rust idiomatic. It would just be less useful in
| those languages, generally.
| verandaguy wrote:
| This gets reposted on HN every few months or so,
|
| And I couldn't be happier, Beej's guide is a seminal read for
| anyone looking to get into net programming. Exposing as many
| newbies to it as possible helps strengthen the field.
| code_witch_sam wrote:
| learned sockets with this like 20 years ago programming IRC
| fserve bots in C
| TallGuyShort wrote:
| There is a handful of people whose tabs I would instantly pick up
| were I to meet them somewhere because of the immense value I have
| gotten from their free work online, writing stuff, answering
| forum posts, etc.
|
| Beej possibly tops that list.
| vajrabum wrote:
| All Beej's guides are fantastic, but if you're interested in
| Network programming you could do way worse than follow this up
| with his guide to network concepts. https://beej.us/guide/bgnet0/
|
| If you search Algolia for Beej you'll see his material has been
| on hacker news numerous times.
| sph wrote:
| Anyone knows of a well-rounded guide that explains the
| structure of the Internet at large? I know everything listed in
| the one you linked, but I have a nebulous idea of how anything
| works after my packet exits my gateway.
|
| So, BGP, autonomous systems, peering agreements and other site-
| to-site routing protocols that keep the entire Internet
| infrastructure ticking along.
| tonightstoast wrote:
| https://learn.cantrill.io/p/tech-fundamentals
|
| I found Cantrill's course pretty solid. It is free and you
| can just skip to DNS & DNSSEC.
| sph wrote:
| Oh, that's exactly what I was looking for. Thanks!
| capableweb wrote:
| I don't know of any one resource that gives a overview of
| everything, I find that browsing Wikipedia for each topic
| gives a good overview of each topic though, with the overview
| being the "Internet Architecture" category:
| https://en.wikipedia.org/wiki/Category:Internet_architecture
|
| Some bookmarks I have regarding BGP and peering that dives
| into it a bit:
|
| - https://www.fortinet.com/resources/cyberglossary/bgp-
| border-...
|
| - https://aws.amazon.com/blogs/architecture/internet-
| routing-a...
|
| - https://www.cisco.com/c/en/us/td/docs/ios-
| xml/ios/iproute_bg...
| tombert wrote:
| I agree with everything you said. Additionally, I think that
| the ZeroMQ guide is also a great resource to learn a lot of the
| common networking patterns, even if you don't plan on using
| ZeroMQ directly.
|
| https://zguide.zeromq.org/
| kayge wrote:
| > If you search Algolia for Beej...
|
| Or, as I just recently learned, you can click on the (beej.us)
| domain name next to the title at the top of this page and it
| will take you to a list of all the HN submissions for the site:
|
| https://news.ycombinator.com/from?site=beej.us
| jamestimmins wrote:
| Is this better as a preface to Network Programming or a suffix?
| vram22 wrote:
| Yes, it seems reading about the concepts should be done
| first.
___________________________________________________________________
(page generated 2023-05-22 23:01 UTC)