[HN Gopher] I wrote a children's book / illustrated guide to Apa...
       ___________________________________________________________________
        
       I wrote a children's book / illustrated guide to Apache Kafka
        
       Author : mitchum_
       Score  : 1666 points
       Date   : 2021-06-17 16:19 UTC (1 days ago)
        
 (HTM) web link (gentlydownthe.stream)
 (TXT) w3m dump (gentlydownthe.stream)
        
       | faster_harder wrote:
       | This is awesome! I will prefer all my technical documentation in
       | children's book format from now on.
        
         | searchableguy wrote:
         | Agreed. Author has done magnificent work on the illustrations
         | and animations.
         | 
         | This prompted me to make an awesome repo to collect children's
         | books on technical topics (which I have seen a few on HN).
         | 
         | https://github.com/searchableguy/awesome-illustrated-guides
         | 
         | I couldn't find a similar list. If there's one, let me know.
        
           | wbl wrote:
           | SELinux coloring book!
        
           | rossdavidh wrote:
           | Well, you should really include the story of Ping:
           | https://www.amazon.com/review/R2VDKZ4X1F992Q :)
        
           | vincentmarle wrote:
           | Google Chrome - https://www.google.com/googlebooks/chrome/
           | 
           | Google Machine Learning -
           | https://cloud.google.com/products/ai/ml-comic-1
           | 
           | Google Federated Learning - https://federated.withgoogle.com/
        
           | redsummer wrote:
           | Some of these may be of use: https://nostarch.com/manga-
           | guide-13-book-set
           | 
           | Also, why's guide to ruby, though it's probably out-of-date: 
           | https://en.m.wikipedia.org/wiki/Why%27s_(poignant)_Guide_to_.
           | ..
        
         | monksy wrote:
         | Why?
        
           | jabroni_salad wrote:
           | I like the novelty of it. I have a copy of The Manga Guide to
           | Databases at my desk that I occasionally 'lend out' to people
           | that mess up my databases. It won't turn anyone into a db
           | hero but it's a decent primer at least.
        
             | sdoering wrote:
             | What the heck. I didn't know this was a thing. Thanks for
             | the pointing it out.
        
           | michaelcampbell wrote:
           | Different strokes, I guess; I can't count the number of
           | people that openly gush about _why's Ruby book with the cats
           | or whatever, but to me it just read like the ravings of a
           | highly functional something-opath. But reasonable people can
           | disagree, and I'm sure I'll get downvoted for disagreeing
           | with the hivemind.
        
             | ericwood wrote:
             | I never found it helpful from a purely technical
             | perspective, but I found it extremely eye-opening as an
             | unorthodox approach to programming that really captured my
             | imagination. It was definitely something that encouraged me
             | to dig deeper into Ruby and do more explorative "creative"
             | coding.
             | 
             | There's no shortage of dry technical documentation, so
             | seeing something akin to outsider art in that space was
             | really refreshing.
             | 
             | Personally I would love to see more technical books come
             | with a soundtrack!
        
             | bee_rider wrote:
             | I don't really get these weird ways of explaining different
             | technologies, just give me a straightforward text
             | description. But straightforward text isn't going anywhere
             | and it doesn't hurt me if people want to read mangas about
             | foxes or whatever.
        
               | monksy wrote:
               | Yea that's where I was going with asking why. It feels
               | like it's a trendy thing is to convert a technical thing
               | into a children's book. (Which is odd.. children don't
               | need that book.. so it's for an adult that wants to
               | consume children's literature)
               | 
               | Note: I'm not against creative attempts to explain
               | technical concepts. But the form to me seems odd, and
               | that it feels like we're producing very short tutorials
               | in a childrens format. That's even weirder.
        
               | haswell wrote:
               | To me, the attraction is less about the specific art used
               | to communicate the concept, and more about the careful
               | attention to a fully-formed analogy that explains the
               | tech in completely different terms.
               | 
               | These kinds of explanations tend to focus on the most
               | critical/important concepts, and help validate (or
               | dispel) assumptions I've made about the tech.
               | 
               | This focus on analogy also lets the author tell the story
               | faster, because I already understand:
               | 
               | - What an otter is
               | 
               | - That rivers flow
               | 
               | - The water flowing down a river that forks will be
               | spread across those forks
               | 
               | - etc...
               | 
               | Depending on the strength of the analogy, it's possible
               | to get the reader on the same page much faster than an
               | intro/tutorial that must first explain foundational
               | concepts just to get to the basics of the technology
               | itself.
        
               | bee_rider wrote:
               | I've never really been a fan of analogies, because I feel
               | like in the end I just need to first understand what the
               | thing is underneath, so I can understand how the author
               | has built their analogy... so the task of understanding
               | why they used this analogy is equivalent to understanding
               | the thing itself.
               | 
               | But I can't deny they are very popular!
        
               | caymanjim wrote:
               | Do people find these analogies helpful? The concepts
               | aren't that difficult, the audience for them is already
               | technical, and adding a cutesy abstraction about it makes
               | it harder to understand.
               | 
               | The art and animation in this is great, but I feel like
               | the author's talents are wasted on a document with no
               | audience. Make a kids' book instead!
        
             | filterfish wrote:
             | > ravings of a highly functional something-opath
             | 
             | I'm going to try and fit this into as many sentences as I
             | can get away with!
        
             | throw1234651234 wrote:
             | My initial impression was "more of this garbage" - but it
             | was really well done in the sense that it distilled the
             | core functionality of the system in an easy-to-understand
             | form.
             | 
             | The guy above you mentioned manga-guides to stuff, which
             | utterly fail at their job, which again, is to distill key-
             | information in an entertaining, easy-to-read, general (but
             | fully accurate) manner.
        
           | teekert wrote:
           | Because to me it seems like a very pure way of separating a
           | concept from implementation. In language and metaphors that
           | are easy to understand and fun to read. I really like it.
           | Now, in a very short time I can decide if Kafka can solve my
           | problem or whether I should move on. I for one store
           | information presented in this way much better, it's feasible
           | that in 5 years I'm presented with a problem and the Otters
           | pop into my mind.
           | 
           | Different people, different preferences I guess.
        
           | caffeine wrote:
           | _why, that's why.
        
             | firebones wrote:
             | caffeine gets it...
        
       | thih9 wrote:
       | Why did the otters use streams as opposed to e.g. having a post
       | office? Is their throughput really that high? Or would a more
       | regular message passing solution be good enough for them?
       | 
       | Perhaps they usually work with streams and couldn't resist
       | applying a familiar solution to the problem?...
        
       | jsmo wrote:
       | Kudos to you for writing this! Well done!
        
       | djohnston wrote:
       | Really well done, bravo!
        
       | Vaslo wrote:
       | For kids only? I personally had little understanding of Kafka
       | before and now I get the gist of what it does. Maybe books should
       | look at this style of explanation instead of just putting a bunch
       | of crazy equations on day 1 that put 90% of people off.
        
         | RussianCow wrote:
         | If you like explanations that are more playful, Why's
         | (Poignant) Guide to Ruby comes to mind: http://poignant.guide/
         | 
         | Not necessarily reminiscent of a children's book, but still a
         | more entertaining way to learn a programming language than most
         | guides.
        
         | jiggawatts wrote:
         | No crazy equations? But how else can I prove my intellectual
         | superiority over lesser men but by demonstrating trivial
         | concepts with greek letters, in keeping with the way Plato
         | originally taught computer science?
        
       | alex_g wrote:
       | This is really cool! Clearly a lot of thought went into making
       | it.
       | 
       | That said, I feel like the otters could have just made a bulletin
       | board to solve their problem
        
       | danellis wrote:
       | Who is this actually for? Programmers need clarity and precision,
       | not "fun" analogies stretched past their breaking point. (I
       | mentally facepalmed when it got to the glass floats. Why even
       | bother with the river at that point?)
       | 
       | And it can't be aimed at children, given that amount of technical
       | jargon (we suddenly go from otters and rivers and bees to
       | headers, keys, values and timestamps). And, well, children don't
       | need a book about Kafka.
        
       | d_t_w wrote:
       | For kids only? I've been working with Kafka for a decade and
       | found this to be an absolute delight.
        
       | andrewnicolalde wrote:
       | Very cute! I bought one of the Round Robin Publishing T-shirts.
       | Truly beautiful drawings :)
        
       | cupcake-unicorn wrote:
       | Hey, this is great - what tools did you use to make this?
        
       | phyrex wrote:
       | That was glorious! Do kubernetes next!
        
       | mtnGoat wrote:
       | pretty clever!
       | 
       | on page 12, i think you mean "streams" rather than "seams". no?
       | also, the page title is "gently down the steam", is an R missing?
        
       | temp8964 wrote:
       | How is this different from REST API? Serious question. Why get
       | downvoted?
        
         | krapht wrote:
         | You're getting downvoted because your question makes no sense.
         | They are not related at all, which means you must not have read
         | the link.
        
           | temp8964 wrote:
           | I read the whole thing. That's why I have the question. So
           | they used to have messages send to everyone (websocks?), now
           | they have the messages persistent in different channels for
           | people to consume. Isn't this just different entry points of
           | a Rest API?
        
             | krapht wrote:
             | Why are you confusing something that could be used to
             | implement the back-end of a REST API (Kafka) with the idea
             | of a REST API?
             | 
             | Your question makes as much sense as: why is <new database
             | X> any different from SQL?
        
               | temp8964 wrote:
               | I got my understanding from the illustrated book...
               | 
               | OK. So I didn't get what Kafka is.
        
         | simonh wrote:
         | Because there's no meaningful way to answer that here. It's
         | like asking what's the difference between trawler fishing and
         | open cast mining.
        
           | temp8964 wrote:
           | I don't understand your metaphor. But thanks.
        
             | simonh wrote:
             | Well, fishing and mining are both resource extraction but
             | apart from that have nothing in common. Kafka and REST both
             | have APIs but apart from that have nothing in common.
        
         | C0d3r wrote:
         | I'm sorry you're getting downvoted, but I think this question
         | is legitimate because the book is peddling Kafka as if it's the
         | only way to do _event sourcing_. Event sourcing is what you
         | should compare with REST APIs, Kafka is one way of doing it,
         | but you can do the same with any database, as long as you have
         | a way to write things in and read things out and organize them,
         | you can achieve event sourcing.
         | 
         | With REST APIs (first few pages of the book), services talk
         | directly with each other, with event sourcing (the rest of the
         | book) services talk with an event store (Kafka in the book) as
         | the intermediary.
        
       | inshadows wrote:
       | Your drawing skills are good. Is this scanned pencil drawing with
       | post-processing? Forget Kafka! How do I learn to draw like this?
       | :-D
        
         | book-sandworm wrote:
         | I think it is using procreate, it also has frame by frame
         | animation options. Awesome Ipad app
        
       | vhodges wrote:
       | Does anyone remember the one about Nitix? Does anyone have a
       | copy?
        
       | crispyambulance wrote:
       | It's really beautiful.
       | 
       | Surprising too. I would have expected a far darker approach to
       | this subject!
       | 
       | Perhaps something like 60's horror comics genre, or better, like
       | those Jack Chick religious tracts where people end up in burning
       | in Hell forever as an immediate consequence of a sin against god.
        
         | jimbokun wrote:
         | Or something written in the style of Franz Kafka.
        
           | hutzlibu wrote:
           | "Childrens book" ... was the goal.
           | 
           | I don't think many children are into Franz Kafka. Kafka is
           | more for cynical grownups. (And teenagers about to become
           | it). It was very Kafkaesque, having to read Kafka in school
           | ..
        
             | opaque wrote:
             | https://en.wikipedia.org/wiki/A_Little_Fable is probably a
             | close as Kafka got to a children's book.
             | 
             | "Alas", said the mouse, "the whole world is growing smaller
             | every day. At the beginning it was so big that I was
             | afraid, I kept running and running, and I was glad when I
             | saw walls far away to the right and left, but these long
             | walls have narrowed so quickly that I am in the last
             | chamber already, and there in the corner stands the trap
             | that I am running into."
             | 
             | "You only need to change your direction," said the cat, and
             | ate it up.
        
         | zwkrt wrote:
         | It's rare a piece of tech has a more fitting name!
         | 
         | "Is your orgs politics so complicated that direct team-to-team
         | communication has broken down? Is your business process subject
         | to unannounced violent change? Bogged down by consistent DB
         | schemas and API versioning? Tired of retries on failed
         | messages? Introducing Kafka by Apache: an easy to use, highly
         | configurable, persistently stored ephemeral centralized
         | federated event based messaging data analytics Single Source of
         | Truth as a Service that can scale with any enterprise-sized
         | dysfunction!"
        
           | [deleted]
        
         | asimjalis wrote:
         | You might be channeling a chapter on dead-letter-queues in
         | Kafka.
        
       | dunningkruger27 wrote:
       | Your kids are very lucky!
        
       | jcolella wrote:
       | This is amazing. Great way to illustrate. What software did you
       | use?
        
       | RcouF1uZ4gsC wrote:
       | This reminds me of "The Story about Ping". The Amazon review of
       | is classic.
       | 
       | https://www.amazon.com/review/R2VDKZ4X1F992Q
        
         | allenu wrote:
         | The OP link is blocked for me (work VPN) but seeing the title
         | immediately made me think of "Mommy, Why is There a Server in
         | the House?" https://www.amazon.com/Mommy-Why-There-Server-
         | House/dp/16053...
        
         | cbsmith wrote:
         | I was thinking more the Little LISPer/Smalltalker.
        
         | mttpgn wrote:
         | > Using deft allegory, the authors have provided an insightful
         | and intuitive explanation of one of Unix's most venerable
         | networking utilities. Even more stunning is that they were
         | clearly working with a very early beta of the program, as their
         | book first appeared in 1933, years (decades!) before the
         | operating system and network infrastructure were finalized.
         | 
         | The fact that since January 2000 nearly 16k people "found this
         | helpful" opened my eyes to how Amazon book reviews really do
         | have the potential to be "classic".
        
         | kmtrowbr wrote:
         | From the review: "The Story About Ping has earned a place on my
         | bookshelf, right between Stevens' Advanced Programming in the
         | Unix Environment, and my dog-eared copy of Dante's seminal work
         | on MS Windows, Inferno. Who can read that passage on the
         | Windows API ("Obscure, profound it was, and nebulous, So that
         | by fixing on its depths my sight -- Nothing whatever I
         | discerned therein."), without shaking their head with deep
         | understanding. But I digress."
        
       | bg24 wrote:
       | Should come with a disclaimer - It is addictive. Once you click
       | the link, be prepared to spend 10minutes...
        
       | thruhiker wrote:
       | Delightful!
        
       | yeswecatan wrote:
       | Can someone please explain why to use Kafka over Redis streams?
       | Is it basically use Kafka if you need your data long term and/or
       | if it's way too big for an in-memory store? Redis seems much,
       | much simpler to setup.
       | 
       | Even if you do need to keep your events long term, why not use
       | something like Eventstore?
        
         | saynay wrote:
         | Kafka is more focused on distributed use-case, I think. It is
         | built around being able to write to fault-tolerant queues, and
         | having multiple consumers in a group reading from it; you can
         | have it configured so that losing a queue node will not lose
         | availability of any of the data in the queue, and losing a
         | consumer will not drop any messages. All of that while being
         | able to handle very high throughput, even with persistence. The
         | tradeoff is in the baseline complexity of Kafka is much higher.
         | 
         | As far as I know, Redis Streams does not offer all of that on
         | its own. You could certainly build a lot of that for Redis
         | yourself (and someone probably has already), but then you start
         | getting back that complexity.
         | 
         | So if you have high requirements for performance, durability
         | and availability, are don't mind the added complexity then
         | Kafka is worth a look.
        
           | yeswecatan wrote:
           | From my limited research, Redis streams offers pretty good
           | durability if you run with AOF and sync frequently.
           | Performance should be great since it's all in memory. As for
           | availability, there is sentintel and cluster mode which I
           | know nothing about.
           | 
           | edit: the differences in consumer groups confuse me as well.
           | With Redis streams, if you have a single stream with a single
           | consumer group with multiple consumers, each consumer will
           | get a new message at different times so processing of each
           | message may happen out of order. That seems... fine to me.
           | Apparently that is not the case with Kafka which makes me
           | ask, why have multiple consumers in a single group if one
           | will be blocked on the other?
        
             | manigandham wrote:
             | Kafka consumer groups do not block. Kafka uses partitions
             | to scale out a single stream/topic and a consumer group
             | allows consumers to exclusively read a partition. Multiple
             | consumers can read from multiple partitions simultaneously,
             | as long as there are enough partitions available.
             | 
             | Redis doesn't have partitions, and instead has a single
             | host managing a stream, while taking on all the consumer
             | tracking functionality. It's fast, but not as scalable, and
             | eventually if you keep growing then scale _is_ how you get
             | speed.
        
               | yeswecatan wrote:
               | If a topic is split across multiple partitions and we
               | have multiple consumers, won't messages be read out of
               | order?
        
               | cmorgan31 wrote:
               | Order guarantees is not part of the promise with Kafka.
               | If your consumer drops or times out the broker will pause
               | all active consumers for the topic, revoke assignments
               | and issue their new assigned blocks based on the new
               | consumer group acknowledgement count.
        
         | xgb84j wrote:
         | 4 years ago at work we compared Kafka to alternatives and ended
         | up using it. In my opinion the only reason for using Kafka is
         | that you need a very very highly performant message queue. It
         | had bad usability at the time and some very annoying
         | undocumented behaviour (e.g. there was no official guide or
         | feature to permanently remove a node).
         | 
         | It feels like whenever Kafka developers had to make any
         | tradeoffs they chose performance over everything else. I would
         | only ever use it again if I the sheer data volume makes it
         | impossible to use an alternative.
        
           | thibauts wrote:
           | With a friend we've built this [1]. Usability has been our
           | main focus, though we ended up having very good perfs
           | (millions of mps out of the box, usually 2x to 10x kafka). We
           | provide REST and WebSocket APIs, + a simple to implement
           | binary protocol for the most demanding workloads. It's in Go
           | so very light and rock solid in stress tests. We provide
           | trivial to implement and understandable usage patterns for
           | exactly once processing. We're currently working on HA and
           | cloud integration. And we're looking for community feedback
           | BTW ! :)
           | 
           | [1] https://github.com/dataptive/styx
        
       | mumblingbee wrote:
       | loved it ... the merchandise should do good ... sadly
       | international shipping is expensive.
        
       | devnull255 wrote:
       | This is so awesome. It almost made me cry. I think it's a
       | brilliant way to introduce a complex technology because we have
       | all grown up to absorb lessons through stories, especially
       | stories about cute animals. And you know what? we remember those
       | stories and the lessons we glean from this. I'm also glad I can
       | swap the image I have of Franz Kafka's Gregor Samsa bug with cute
       | otters.
        
       | dyml wrote:
       | This was really great. I would like to see more content like
       | this.
        
       | vages wrote:
       | I've seen a lot of these children's books about technical topics.
       | Is there any record of an actual child reading any of them? And,
       | as children do with good books: reading them more than once?
       | (Excluding children in the author's family.)
       | 
       | I have read about 10 of these "children's books about
       | programming" and while I enjoy them myself, I find that they lack
       | most of the things that grab children's attention, such as
       | repetition and visual-only sub-plots.
       | 
       | This is not to criticize the use of an illustrated fable as a
       | storytelling device for adults: They're great! It's just sad that
       | we have to frame an illustrated fable as "for children" in order
       | for it to be accepted. I think it says something about how
       | content dictates and narrows the expected presentation format,
       | sometimes to the detriment of clear communication.
        
         | Siira wrote:
         | Can you list the books?
        
           | sireat wrote:
           | Probably the most famous: https://en.wikipedia.org/wiki/Why%2
           | 7s_(poignant)_Guide_to_Ru...
           | 
           | Not my cup of tea (loved the current Kafka one)
           | 
           | I prefer when books for learning use a consistent art style.
        
             | sciurus wrote:
             | I've (rightly) never seen Why's Guide marketed as being
             | "for children" though.
        
             | yitchelle wrote:
             | While not directly related to tech, this story is great
             | analogy about ping.
             | 
             | https://en.wikipedia.org/wiki/The_Story_About_Ping
        
           | vages wrote:
           | Most of the books have been low-key attempts from programmer
           | parents, probably to get their own children into programming.
           | They were posted on different forums, and I have a hard time
           | finding my way back to them.
           | 
           | The example that I think illustrates my point best is:
           | http://arthur-johnston.com/hacker_writes_a_childrens_book/
           | (which was posted here in 2017:
           | https://news.ycombinator.com/item?id=15879519) The book works
           | well as entertainment for grown-up programmers: "G is for
           | Garbage Collector/when something's no longer needed/it frees
           | up the memory/so your program is speeded." You can look
           | inside the book on Amazon for more examples. I judge this
           | rhyme as too advanced for any child below 8 that I've read
           | aloud to. Most theories about cognitive development agree
           | with Piaget (1896-1980) in that children have a hard time
           | grasping difficult abstract concepts before the age of 12-13,
           | so there is at least some "scientific backing" to my hunch: h
           | ttps://en.wikipedia.org/wiki/Cognitive_development#Concrete..
           | . (This is only a hunch though; the children I've read aloud
           | to are all picked from the same group - children of family
           | and friends.)
           | 
           | My observation is that children seem to need _lots_ of
           | concrete verbal and visual imagery in order to stay
           | interested. You can also get away with more abstract themes
           | if the actual text is lyrically well-crafted, like Dr. Seuss'
           | books (or Andre Bjerke's children's rhymes in Norwegian, my
           | mother tongue). The most successful attempts at teaching
           | programming to children that I've seen, seem to give the
           | children a lot of practical tasks they can work on (e.g. the
           | Hello Ruby book series). You also have some programming
           | elements in Minecraft that children could pick up, because
           | the concepts are implemented as concrete objects.
           | 
           | All of this makes me suspect that the main audience for these
           | "children's books" teaching programming is adults that
           | already know a fair bit about the subject. And that's
           | completely all right by me, because it gives the books
           | traction and brings the book's readers entertainment.
           | 
           | PS: You also have the Javascript/HTML/CSS for babies series,
           | which are only jocularly presented as children's books:
           | https://imgur.com/eOYc8fC
        
         | ufoolme wrote:
         | My 5yo loves reading the kubernetes book (paperback copy of
         | https://www.cncf.io/the-childrens-illustrated-guide-to-
         | kuber...), I found him reading it by himself the other day I
         | guess it has sort of a plot.
         | 
         | I think he likes it mostly because I tell him that's what I do
         | at work.
        
           | newtwilly wrote:
           | Wow this one has quite the vocabulary! Right in the first
           | pages we have "hosting provider", "environment", "OS", "Web
           | Server", etc.
        
           | vages wrote:
           | It's got very pretty pictures, so that's very understandable!
           | Have you been able to find out if he understands any of the
           | abstract concepts that are explained, or is he mostly
           | interested in the character interactions?
        
       | gabriel9 wrote:
       | This was nice and refreshing. Thank you.
        
       | MartijnBraam wrote:
       | Reminds me of the Microsoft Server illustrated childrens book :D
       | 
       | https://imgur.com/gallery/kvATA
        
         | jedberg wrote:
         | Holy cow. I had no idea this existed, but it's amazing in a
         | terrible way.
        
         | chairmanwow1 wrote:
         | Oh my goodness. This had me cracking up:
         | 
         | >> "When a mommy and a daddy love each other very much, the
         | daddy wants to give the mommy a special gift.
         | 
         | >> So he buys a "stay-at-home" server."
        
       | DubDouble wrote:
       | I think ...Hm... That it's awesome. The notion that a book can
       | possess child-like qualities... which convey complex ideas is
       | appealing. The audience can be anyone.
        
       | chrismorgan wrote:
       | https://www.gentlydownthe.stream/#/10/1: I was wondering why the
       | "//gi" and "/round-robin-books/g" were purple, while the rest of
       | the line was white. hljs decided the whole block was awk, and
       | those parts regular expressions. Might be nice to tell hljs it's
       | a different language.
       | 
       | https://www.gentlydownthe.stream/#/11: "In the rivers gleam"
       | doesn't seem quite right; I can't decide whether that should be
       | "In the river's gleam" (the river doing the gleaming) or whether
       | it was intended to be "In the rivers gleam" (the events doing the
       | gleaming, in multiple rivers) in which case I suspect it should
       | be a singular river.
        
       | pashariger wrote:
       | Cutest thing I've seen today. Every tech team should hire a
       | children's book illustrator to explain their products/tech, IMO.
        
       | racl101 wrote:
       | Wow! That was cool.
       | 
       | Now I want a Docker children's book please.
        
         | CGamesPlay wrote:
         | I mean, Kubernetes is a superset of Docker, right?
         | https://www.cncf.io/the-childrens-illustrated-guide-to-kuber...
        
           | alexjplant wrote:
           | Not really (unfortunately). Kubernetes can leverage a number
           | [1] of container runtimes - OpenShift uses CRI-O, for
           | instance. There's also the problem that Docker can refer to a
           | container runtime, orchestration engine (Swarm, which is
           | analogous to k8s), CLI tools (`docker` and `docker-compose`),
           | or HTTP API (registry). At my last few gigs the maintenance
           | of separate configs for docker[-]compose and k8s has been a
           | consistent pain point.
           | 
           | The container ecosystem is pretty complicated :-).
           | 
           | [1] https://kubernetes.io/docs/setup/production-
           | environment/cont...
        
         | chrischen wrote:
         | And then do Monads.
        
           | flaie wrote:
           | https://adit.io/posts/2013-04-17-functors,_applicatives,_and.
           | ..
        
       | strzibny wrote:
       | Beautiful. Great work, really enjoyed the animations!
        
         | augustk wrote:
         | I found the animations very distracting when trying to read the
         | text. To me the animations didn't add anything.
        
       | el_seano wrote:
       | Absolutely adorable! Really excellent work :D
        
       | kolla wrote:
       | I work with Splunk and this book would be good for some of my
       | users..
        
       | sciencewolf wrote:
       | This is outstanding!
        
       | jwilber wrote:
       | Extremely well done! What did you use for the drawings?
        
       | bdamm wrote:
       | Reminds me of "SCADA and Me (A book for Children and
       | Management)"[1].
       | 
       | [1]: https://www.amazon.com/SCADA-Me-Book-Children-
       | Management/dp/...
        
       | maurits wrote:
       | Super cool, thank you. Might I request Kubernetes?
        
         | momothereal wrote:
         | https://azure.microsoft.com/en-ca/resources/videos/the-illus...
         | :)
        
         | samspenc wrote:
         | This has been shared in other comments here but just in case :)
         | https://www.cncf.io/the-childrens-illustrated-guide-to-kuber...
        
       | octos4murai wrote:
       | Taking ELI5 to another level!
        
         | abledon wrote:
         | level 2 ELI5 , Illustrate like im 5 (ILI5)
        
       | yongjik wrote:
       | I can't shake the feeling that the otters could have used a
       | bulletin board - on the left side they could have written "what
       | kind?", in the middle "when", and on the right more detailed
       | description as necessary. Then any otter in a hurry could skim
       | through the left column, find topics they liked, and just read
       | that part from the right.
       | 
       | Sorry, it's not really a fair criticism, and I did like the art
       | style - I just don't like Kafka much because I had to deal with
       | it in places where it didn't make much sense.
        
         | ndrfrhlch wrote:
         | your bulletin board could be an RSS feed. most often you do not
         | need kafka (or a service bus) for distributed systems.
         | 
         | RSS feeds are great as they are easy to implement and debug.
        
         | LotToLearn wrote:
         | Where would you say it makes more sense, design wise? I feel
         | like its hard to find perfect use cases for Kafta.
        
           | swagasaurus-rex wrote:
           | When you have many servers who all need to see the same
           | chronological data stream (including messages they might miss
           | during network downtime), and see new events in real time.
           | 
           | If you set "log.retention.hours = -1" and
           | "log.retention.bytes = -1" in kafka config, kafka stores
           | messages forever.
           | 
           | In a game for example, user-inputs and other events can be
           | produced in Kafka, then reconstruct the entire game-state by
           | reading and processing the kafka-stream from start to finish.
           | It has an advantage over most DBs because it's real-time.
           | 
           | You can use also chronological data streams to represent data
           | structures more complicated than a simple array. For example,
           | a tree can be represented while preserving chronology. This
           | is far from the ideal use case however.
        
             | spockz wrote:
             | > In a game for example, user-inputs and other events can
             | be produced in Kafka, then reconstruct the entire game-
             | state by reading and processing the kafka-stream from start
             | to finish.
             | 
             | This is called event sourcing.
        
           | lmm wrote:
           | I'd recommend Kafka for any case where you need to act on
           | data that changes over time - almost any software system. In
           | almost every system you want to be able to reliably record
           | events, and you want to be able to partially replay your
           | state in a deterministic way (e.g. after you fix a bug); both
           | those things should be table stakes for a serious datastore,
           | but AFAIK Kafka is practically the only one that offers them.
           | 
           | Kafka itself kind of only solves half the problem, because it
           | doesn't offer indexes as a builtin, so you have to build your
           | indexed state yourself, or maybe even use a "traditional"
           | datastore as a de facto cache. But since you've moved all the
           | hard distributed part of the problem into kafka, that part is
           | not so bad.
        
           | yongjik wrote:
           | Sorry, I'm not qualified to answer the question. You will
           | have to ask someone who has seen Kafka being used in a place
           | where it made sense...
        
             | andreskytt wrote:
             | I have not used Kafka, but what the book describes works
             | with most queue systems. And those are great for localizing
             | dynamic complexity. One part of your system behaving
             | weirdly (spiking, dying, fluctuating etc. ) can't affect
             | other parts because they sit at the other end of a queue
             | happily consuming the messages at their own pace. This
             | means tour complex system becomes less complex in terms of
             | behavior and is less likely to end up in some weird
             | metstable state
        
               | nvarsj wrote:
               | This is sort of funny to me, since ActiveMQ had this
               | exact problem. A single slow client could break the
               | entire system because the broker would slow down
               | producers to prevent exhausting the queue space. Which
               | was exactly some "weird" state because it wasn't obvious
               | what happened until you spent a lot of time debugging.
        
             | rising-sky wrote:
             | this made me chuckle
        
           | mdtusz wrote:
           | Wherever you have a firehose of data that needs to be
           | processed.
           | 
           | I've heard of it being used as a sort of message queue for
           | application level events before, but that sounds like a
           | nightmare of trying to reinvent the actor model with 1000x
           | the complexity.
        
             | halfmatthalfcat wrote:
             | Bah god, that's Akka's music!
        
               | oneplane wrote:
               | Akka, and a lot of actor model services break
               | microservice availability, durability and general
               | reliability because nuking a random node messes with Akka
               | and now whatever actor happened to be on that node is now
               | stalled until it's transferred.
               | 
               | Just like SOA and ESB, the concept isn't the problem, is
               | the technical constraints of the design at the time.
               | Decoupling and messaging isn't bad, but having a legacy
               | message queue on physical hardware doesn't really hold
               | up. Any derived architecture faces the same problem.
               | 
               | Then again, Kafka isn't an actor model implementation,
               | and Akka isn't a partitioned redundant stream processing
               | system, they don't have all that much overlap ;-)
        
               | halfmatthalfcat wrote:
               | If you shard your Akka actors, the messages are buffered
               | and passed to the actor when it's initialized on the new
               | node. You get even more stability if you persist your
               | actors backed by a DB or some other persistent store.
               | 
               | Not saying Akka can replace Kafka but many of the issues
               | around availability, durability and reliability have been
               | attempted to be solved in Akka.
        
               | oneplane wrote:
               | Yeah, that's true. I think the main issue in most cases
               | where I see problems are the ones that only use in-memory
               | stores.
        
           | Joeri wrote:
           | For me a perfect use case is sensor data processing. I've
           | been involved with two independent sensor data platforms that
           | used kafka as the backbone. Sensor data is persisted
           | unprocessed to raw kafka topics, and then gradually
           | deduplicated, refined, enriched and aggregated across
           | multiple stages until finally a normalized, functionally
           | sharded and aggregated view of the data is stored in
           | databases for dashboarding.
           | 
           | It is easy to scale horizontally to massive volumes of data,
           | and any issues in the processing pipeline can be fixed
           | without losing any raw sensor data (restarting the consumers
           | from the last known valid point).
        
           | guhcampos wrote:
           | Kafka is a streaming log service. People who use it just for
           | passing messages are hugely overengineering something that
           | would fit just fine for something like RabbitMQ, Redis or
           | other tools.
           | 
           | An RSS feed must be powered by something underneath, and any
           | of those tools can do the job. In most situations it would be
           | extremely impractical to use a relational database for this
           | kind of thing. If you are getting thousands of messages in
           | per second, which is not uncommon, no transactional database
           | will give you enough write performance and won't be able to
           | handle too many queries per second for clients polling for
           | updates, like an RSS feed would require. Note that caching
           | queries is almost useless here because the latest content is
           | updated every few milisseconds.
           | 
           | Kafka, as pretty much any other queueing datastore, is
           | optimized for append-only writes with no deletes or updates.
           | Reading from the end of the queue is extremely fast and
           | sequential reads down the stream are quite fast too. Random
           | reads such as the ones commonly handled by SQL databases are
           | either not available or are less efficient than with SQL.
           | 
           | That said, Kafka can be used to pass any kind of message
           | between applications: from simple text messages and small
           | JSON data to video frames, protobuf messages and other types
           | of chunks of serialized data.
           | 
           | It is also a very durable data store for immutable, time-
           | ordered data, and is widely used in the financial world to
           | store transaction logs.
        
       | mechero wrote:
       | This is awesome, excellent work! Like some others here, I feel
       | like the otters over-engineered the entire thing using fancy
       | glass balls and partitioning a river where a categorized
       | bulleting board would have been good enough.
       | 
       | But then I realized that that's what makes it a realistic story
       | ;)
       | 
       | Jokes aside, the best explanation I've ever seen. It should be a
       | must-do for people who want a quick and understandable
       | introduction to the concepts.
        
       | stakkur wrote:
       | This...is brilliant. I could also see this as a print book.
        
       | muhammadusman wrote:
       | Finally, I can learn what kafka is :D
        
       | teekert wrote:
       | I like it, I only heard of Kafka before, now I know it's mqtt on
       | steroids.
        
       | Hackbraten wrote:
       | This is amazingly cute. And the analogies feel like they've been
       | very carefully chosen. Bravo!
        
       | smusamashah wrote:
       | Wow this is such a beautiful read. Has anything like this been
       | done before for some other topic?
       | 
       | P.S. This sentence is hard to grasp for me "This Unawareness
       | helps Decouple systems that produce events from otters that read
       | events." https://www.gentlydownthe.stream/#/20
        
         | jpdb wrote:
         | The selinux coloring book is another great example:
         | https://people.redhat.com/duffy/selinux/selinux-coloring-boo...
        
           | tbrock wrote:
           | The penguin in this one is nightmare fuel
        
         | MasterScrat wrote:
         | The Reinforcement Learning algo A2C was also illustrated in a
         | similar way:
         | 
         | https://medium.com/hackernoon/intuitive-rl-intro-to-advantag...
        
         | jedberg wrote:
         | I don't have a link but the NSA has a coloring book about
         | cryptography that's in the public domain, since it was created
         | by the government with tax dollars and has no copyright notice.
         | I actually have a copy I've been meaning to scan, I'm just
         | afraid of ending up on a list if I do. :)
        
         | sideshowb wrote:
         | I tried it with a bunch of 101 comsc topics in one story -
         | boundary conditions, recursion, high vs low level languages,
         | undefined behaviour, memory protection
         | https://www.royalroad.com/fiction/41545/manifest-logic/chapt...
         | 
         | More generally there's this list of similar fiction
         | https://fiftysevendegreesofrad.github.io/hard-comp-fi-fictio...
        
         | clintonc wrote:
         | https://www.cncf.io/the-childrens-illustrated-guide-to-kuber...
         | is a famous example.
        
         | jimbokun wrote:
         | Does Why's Poignant Guide to Ruby count?
         | 
         | https://poignant.guide/
        
         | onlyrealcuzzo wrote:
         | I'm glad people are enjoying this. I wrote a story two years
         | ago about Queues / Kinesis - and am in the process of getting
         | it illustrated. I doubt it'll make any money, but it's good to
         | know that people enjoy this type of thing!
        
         | SamBam wrote:
         | > "This Unawareness helps Decouple systems that produce events
         | from otters that read events."
         | 
         | It simply means that the producer doesn't need to maintain a
         | list of listeners. It just throws the event into the stream,
         | and assumes that anyone who wants to read it will be able to do
         | so.
         | 
         | RSS vs an email list, I guess.
        
         | jbrnh wrote:
         | Not quite the same, but "Life on the Infinite Farm"
         | https://www.math.brown.edu/reschwar/farm.pdf
        
         | ramoz wrote:
         | Kubernetes https://www.cncf.io/the-childrens-illustrated-guide-
         | to-kuber...
        
       | alexkreidler wrote:
       | I've seen many comments here along the lines of "but when should
       | Kafka be used?"
       | 
       | If you want a deep dive into event streaming systems (logs) that
       | also gives some examples of big data systems they interact with,
       | I highly recommend "The Log: What every software engineer should
       | know about real-time data's unifying abstraction," from LinkedIn,
       | the creators of Kafka.
       | 
       | Their engineering blog also has many other interesting articles
       | on their data systems.
       | 
       | [0] https://engineering.linkedin.com/distributed-systems/log-
       | wha...
        
       | c141charlie wrote:
       | Outstanding! Purchased the eBook. Will read to my son who wants
       | to be a coder when he grows up.
        
         | c141charlie wrote:
         | Also more approachable than the little schemer :-)
        
           | touisteur wrote:
           | Or the little prover. Ugh.
        
       | artellectual wrote:
       | This is so well done. I've made it a mandate for all my engineers
       | to read it.
        
         | penguinlinux wrote:
         | the book was really good. However, just because you liked it
         | why do you have to mandate your employees to read it?
         | 
         | Why not just make a recommendation.
        
           | acmecorps wrote:
           | Not OP, but I think it was a bit of tongue in cheek comment
           | (I might be wrong tho) ;)
        
       | 41209 wrote:
       | Can you do this for every technology stack, maybe one for multi-
       | threading in C sharp
        
         | parafactual wrote:
         | monad tutorial via children's book
        
       | williesleg wrote:
       | Nobody gives a shit.
        
       | mawise wrote:
       | With the initial focus on families sharing news, this made me
       | think about Facebook (or better yet, RSS!)
        
       | nebulous1 wrote:
       | I love that this is life before Kafka:
       | https://www.gentlydownthe.stream/#/4
       | 
       | And this is life after Kafka:
       | https://www.gentlydownthe.stream/#/24
        
         | StavrosK wrote:
         | While I love the commentary and found it funny, it would more
         | accurately be "life before and after a huge volume of
         | messages". Kafka isn't the problem in this specific case.
        
         | sushisource wrote:
         | That page 24 image was exactly where I felt the otter metaphor
         | was perhaps being stretched to its breaking point.
        
         | kccqzy wrote:
         | What a Kafkaesque situation. Really gives new meaning to the
         | word "Kafkaesque."
        
       | chairmanwow1 wrote:
       | Honestly did not expect this to be so fun to go through. Doesn't
       | pull any punches when digging into the details either.
        
       | soheil wrote:
       | I hope this doesn't encourage these children to grow up and start
       | using Kafka willy-nilly in places that it's not meant to be used.
       | 
       | It's often hard to find a legitimate use case for
       | Kafka/RabbitMq/etc. where a much simpler solution won't do.
        
       | hank_z wrote:
       | I love this. Great work!
        
       | mbfg wrote:
       | Fantastic. I'm betting a large group of people would much more
       | likely watch/read this than a typical technical paper, and still
       | gain 60% of the knowledge.
       | 
       | I have to say, i was looking for a slide on how the otters
       | cleverly know what messages they've already read, and that was
       | unfortunate, but kudos!
        
       | dante_dev wrote:
       | Love it! But the cdn serving the images is improperly configured.
       | The browser hits the cdn each time you go back and forth,
       | downloading the images (multiple megabytes each) every time.
       | 
       | Likely the `expires` header is the problem (it's a date in the
       | past)
        
       | markphip wrote:
       | This is great, I was pleasantly surprised to see the feedback
       | here has all been positive as well.
        
       | posharma wrote:
       | Amazing! Really nice.
        
       | winkelwagen wrote:
       | WOW, this is so nice. The drawings are really cute and the
       | animations give it its own style. Bit of feedback the color palet
       | all over the place. Could use a bit of hierarchy to help the
       | focus of the viewer and reuse same colors to make it more
       | coherent.
        
       | jmcguckin wrote:
       | cute. ELI5, but with beavers.
        
       | vladstudio wrote:
       | I've been thinking about comics explaining different IT concepts,
       | but so far only produced one :-)
       | https://vlad.studio/wallpaper/?how_internet_works
        
         | exikyut wrote:
         | Psst, files.vlad.studio's cert expired yesterday. Really nice
         | site design and artwork!
        
           | vladstudio wrote:
           | ooh, perfect timing :-(
        
         | duck wrote:
         | Cool to see you on HN! I use to rotate through all your
         | wallpapers back in the day (what seems like forever ago) - glad
         | to see you're still at it!
        
           | vladstudio wrote:
           | Thank you! I'm way less productive than I used to be, but I
           | like to think my best drawings are yet to come :-)
        
       | peterburkimsher wrote:
       | I started writing a bedtime story called "Ed and the Awk". If
       | someone could illustrate it, that would be amazing!
       | 
       | My homestay family sister Anna, 9 years old, told me that I
       | spelled "sed" wrong :D
       | 
       | ed likes to write books. His home is near the C.
       | 
       | ed has short black hair on his head, with a curl.
       | 
       | ed has a cat with a long tail. His cat likes to read books. He
       | likes it when you tcl his tummy.
       | 
       | Last time, ed looked into the clear sky. It's a nice view. On top
       | of the arch, he saw a bird.
       | 
       | "Whatis your name?" asked ed.
       | 
       | "Awk" said the bird.
       | 
       | "Can you talk to a man?" asked ed.
       | 
       | "Yes" said awk. "I read books. But only sum words."
       | 
       | "Can you find all the words that start with a B?" asked ed.
       | 
       | "books" "black" "bird" "but" "bin" "big" "bits" "bash" sed awk.
       | 
       | "That's right! You're very good at reading. Which place do you
       | live in?" asked ed.
       | 
       | "Now I live in a bin." said awk.
       | 
       | Awk is hungry.
       | 
       | To goto the field, he crawls through a pipe.
       | 
       | He gets some food from the field, and puts it on the table to eat
       | it with a fork.
       | 
       | Yummy! He takes a byte.
       | 
       | Oh no! Now he's too fat to go back home.
       | 
       | "Don't worry", says ed. "I can sort that out."
       | 
       | "My cat will cut food from the field and keep on sending it
       | through the pipe for you."
       | 
       | "Now I need to go to the toilet." says awk.
       | 
       | They go to the toilet. Wait while IP, says ed.
       | 
       | Awk sits down and unzips his trousers.
       | 
       | awk does one poo, two, three, four, five, six, seven, eight!
       | 
       | The wc has 8 bits of poo. There's a big login the heap. Stinky!
       | 
       | Flush! The poo goes away through another pipe.
       | 
       | The awk must wash his hands. But the sink is grotty and it leaks.
       | 
       | ed has strong arms. He can flex his muscles. "I'll make it better
       | with tar!" he says.
       | 
       | With a bash and a clang, the sink is flowing again.
       | 
       | "Eat less next time." says ed. "The only animal who eats more
       | than you is the GNU."
       | 
       | "I was hungry!" says awk. "What did you expect?"
       | 
       | It's time to leave. There's no reason to stay here idle. "Welcome
       | back to our group any time. cu later!"
       | 
       | Questions:
       | 
       | 1. Who does ed meet in the story? (a) Gnu (b) Awk (c) Python
       | 
       | 2. Can you remember a word starting with B? (any of "books"
       | "black" "bird" "but" "bin" "big" "bit" "bash")
       | 
       | 3. What takes food to the awk, and flushes the poo away? (a) Pipe
       | (b) Bin (c) Field
       | 
       | 4. What do you do after going to the toilet? (a) Watch YouTube
       | (b) Brush your teeth (c) Wash your hands
        
       | sandreas wrote:
       | Awesome art - thanks for sharing. Who is the artist? You?
        
         | j4ah4n wrote:
         | Written and illustrated by Mitch Seymour (on the cover)
        
       | bombcar wrote:
       | I can't find the feedback link but I feel the animations could be
       | a bit slower - hard to read the small words when flashing so fast
       | on iPhone.
       | 
       | Otherwise quite well done!
        
       | dave_sid wrote:
       | This is great. Can you write me one for elastic search now :-D
        
       | ghufran_syed wrote:
       | This is fabulous!
       | 
       | Is there a typo at https://www.gentlydownthe.stream/#/22 ?
       | 
       | Current version: "First, they dropped large stones into the
       | river, splitting each topic into a _smaller_ number of streams,
       | or Partitions. "
       | 
       | I know nothing about Kafka, but I think maybe this should be:
       | "First, they dropped large stones into the river, splitting each
       | topic into a number of _smaller_ streams, or Partitions. "
       | 
       | (my emphasis for both versions) "
        
       | janci wrote:
       | Nice. I expected Kafka to be a bird, not a river.
        
       | neovive wrote:
       | Beautiful work! I love the illustrations and animations. Did you
       | use Photoshop or a another illustration tool?
        
       | throwaway_fjmr wrote:
       | This is effin' beautiful.
        
       | driton wrote:
       | Great job and excellent art style. It provides enough material to
       | get any curious mind interested in reading further about Kafka.
        
       | marvindanig wrote:
       | Awesome! Love the illustrations and the subtle animations
       | underneath.
       | 
       | Quick feedback: My nephew prefers page turns (curling) on their
       | iPad. I assume that he prefers it over a powerpoint style slides
       | that we currently have on your book. Well, at least he spent some
       | time flipping through the animated pages!
        
       | dj_gitmo wrote:
       | This we very helpful for someone unfamiliar with Kafka. You could
       | probably get a job doing this.
        
       | juliend2 wrote:
       | This is great because (like _why's {poignant} guide to ruby did
       | for me) it breaks the psychological barrier that I have to even
       | get an intro to those technologies -- as an adult developer --
       | because it has new concepts that are foreign to me.
        
       | testplzignore wrote:
       | While the otters are busy planning elaborate ways to tell each
       | other about neat rocks, the much smarter humans are revving up
       | their chainsaws to cut down the forest.
        
         | parafactual wrote:
         | I'm not sure I'm following the analogy anymore.
        
       | rundmc wrote:
       | Can you do Haskell next please?
        
       | trilinearnz wrote:
       | This is a wonderful contribution to the software development
       | community, in the vein of Why's Poignant Guide and Land of Lisp.
       | I fully support such combinations of artistry, whimsy and
       | technology - true to the core hacker spirit.
        
       | karanbirsingh7 wrote:
       | Great read and well put explanations. Thanks for this!
        
       | data_spy wrote:
       | I will show this to my daughter later today!
        
       | [deleted]
        
       | cdnsteve wrote:
       | An exceptional form of creative education on a difficult
       | technical topic with stunning visuals that is intuitive and fun.
       | Brilliant!
        
       | 101008 wrote:
       | This is awesome. Congratulations for creating this!
        
       | primaprashant wrote:
       | PSA: If you are anxious like me, there are total of 32 pages.
        
       | Epskampie wrote:
       | I love this kind of high-level introduction of what a thing is.
       | Sometimes very hard to find for technologies. Also otters are
       | cute.
        
       | ramoz wrote:
       | I'd pay for the magical place sequel.
        
       | rdevsrex wrote:
       | It's so cute and I really enjoyed the high level overview of
       | Kafka.
        
       | fierro wrote:
       | How will the otters handle cross region communication?
        
       | pixelmonkey wrote:
       | I've been trying to put my finger on why Kafka so well captured
       | the imagination of many distributed systems engineers. My best
       | answer is, "low-cost publish and multi-consumer data-sharded
       | subscribe is the key to resilient horizontal scaling and
       | parallelism."
       | 
       | Kafka has its flaws, but it really served us well. We have Python
       | Data Engineers who focus on distributed system design[1], and
       | Kafka is one of the team's least finicky open source components,
       | but it is used _everywhere_ , and it basically enables the entire
       | rest of the real-time data processing stack.
       | 
       | [1]: https://www.parse.ly/careers/python_data_engineer
        
         | jeffbee wrote:
         | "Captures the imagination of distributed systems engineers" is
         | the biggest red flag I can think of.
        
       | geodel wrote:
       | Could be just me. I see Apache kafka more nightmarish than
       | bureaucracy in 'The Castle' or Police/judiciary in 'The Trial'
       | from author Franz Kafka. Mr Franz btw stole his last name from
       | this very famous Apache Kafka project.
        
         | dkarl wrote:
         | Individual Kafka consumers and producers tend to have simple
         | behavior, which is good, and initially plugging them together
         | yields simple, predictable systems, but people tend to keep
         | going until they experience pain, at which point they have a
         | system that is right at the limit of what they can understand.
         | Then further feature work pushes them over the limit into
         | darkness.
         | 
         | I think Kafka needs the equivalent of OpenAPI and Redoc, a
         | simple spec and document generator, but for groups of consumers
         | and producers rather than single applications. This would
         | increase the tractability of complex systems, but it would also
         | let you see the system getting more complex over time, even
         | when you haven't reached the pain point yet.
        
           | cbsmith wrote:
           | KSQL seems to address this gap.
        
           | morelisp wrote:
           | https://www.asyncapi.com/ is in its infancy but is attempting
           | what you describe.
           | 
           | (I view OpenAPI as a near-failure, but, good luck to everyone
           | trying.)
        
             | dkarl wrote:
             | Thanks for the tip; that's a really cool project. It looks
             | like the documentation generator for it works on a single
             | file at a time, and a single file defines a single service.
             | I was thinking of something that works on the level of a
             | system of interacting services. I want a tool that reads
             | the API specifications for a group of services and
             | documents the interactions between them. I'd like to look
             | at service X and see that it emits message Y on queue Z,
             | and then see which services read message Y on queue Z, jump
             | to their documentation, etc. I think the AsyncAPI format is
             | perfect to build on, though. I'll start there if I decide
             | to take a hack at it.
        
               | ridiculous_lol wrote:
               | I am not sure if I completely understand your use-case
               | but it seems that the config can be distributed across
               | multiple files.
               | 
               | https://www.asyncapi.com/docs/specifications/v2.0.0#refer
               | enc...
        
               | dkarl wrote:
               | I'm taking a closer look today, and it looks to me like
               | one AsyncAPI document, which can be defined using
               | multiple files, defines one application. If you look at
               | the "fixed fields" section under the root object, the
               | second field is "id"[0]:                   id Identifier
               | Identifier of the application the AsyncAPI document is
               | defining.
               | 
               | This format is fine, but the tool I'm looking for is
               | something that will read the definitions of multiple
               | AsyncAPI documents (multiple applications) and show how
               | their inputs and outputs connect, so I can answer a
               | question like, "When application X publishes message Y on
               | channel Z, which applications consume that message?"
               | 
               | AsyncAPI gets me 90% there by defining the service spec
               | and providing code to parse it. It's possible somebody
               | has already written the rest; I'll have to see.
               | 
               | [0] https://www.asyncapi.com/docs/specifications/v2.0.0#f
               | ixed-fi...
        
         | gilbetron wrote:
         | What do you find nightmarish about it?
        
           | manishsharan wrote:
           | Kafka is very configurable .. and teams struggle to optimize
           | the production deployment configuration and tend to second
           | guess their decisions. Getting a managed deployment from
           | Confluent etc. mitigates these issues as you can count on
           | proven SMEs to deliver a well optimized solution. However, if
           | you were to DIY , you might not be so blessed. Disclosure: I
           | never did a DIY Kafka prod deploy -- my employer was already
           | a Confluent customer.
        
         | rjzzleep wrote:
         | It's quite beautiful, but put so simply it does look
         | nightmarish.
         | 
         | "Then, the otters would decide which part of the river to put
         | the message in"
         | 
         | I'd imagine a child looking at this going like "why are they
         | doing this to the river". Why are they are throwing things into
         | the river. :D
        
           | lostcolony wrote:
           | No, put so simply it's cute. In production it's nightmarish.
        
         | monkeyfacebag wrote:
         | Maybe it's just you, maybe not. It would be beneficial for you
         | to provide some substance to your claim. Then we would have
         | something to discuss.
        
           | geodel wrote:
           | Unfortunately we do not have something to discuss. All these
           | consumer/producer/broker failing randomly, system behaving
           | erratically is me doing it wrong. Those white papers from
           | Confluent and testimonial from customers are proof that Kafka
           | works fine.
        
         | Hamuko wrote:
         | Is this comment intentionally Kafkaesque?
        
       | asimjalis wrote:
       | I love this.
        
       | operatorius wrote:
       | Really enjoyed reading it. the analogies are very clear, easy to
       | understand and fun to read. the illustrations are amazing as
       | well. wish there was more of it :)
        
       | unknown_error wrote:
       | This is one of the best things I've ever seen on the web (or off
       | it!). It's now one of my favorite books <3
       | 
       | Thank you for sharing this and really brightening my day.
        
       | jimmyjazz14 wrote:
       | I can't help but feel the otters kinda over-engineered this whole
       | thing.
        
         | jeffbee wrote:
         | I think the otters should question some of the things that are
         | meant to be axiomatic. Is tight coupling really a scalability
         | problem? Otters should be prepared to defend these statements.
         | Nixie's song might be the vacuous equivalent of "Mongo is web
         | scale".
        
           | jasonhansel wrote:
           | Also: did adding the stream really decouple the otters, or
           | did it just make the coupling less visible? After all, the
           | consuming otters may still depend a great deal on the precise
           | behaviors of the producing otters. It's just that now the
           | producing otters don't _know_ in what ways other otters
           | depend on them, making it harder for them to make changes
           | without harming their otter-dependents.
        
             | saynay wrote:
             | The fact the producing otters don't know the ways in which
             | the consuming otters handle the message is precisely the
             | decoupling its talking about, no?
        
             | roccomathijn wrote:
             | Nobody used the word decoupling. They just went from being
             | tightly coupled to being loosely coupled.
        
             | [deleted]
        
         | codeulike wrote:
         | You need some other animals - Beavers maybe? - who are in the
         | same forest and who just have a simple system of scratching
         | messages onto the side of a single large tree somewhere that
         | everyone goes to look at regularly. Then at the end see how
         | many picnics the beavers missed vs how many otters lost their
         | fucking minds from the complexity of the system and then see
         | who comes out ahead.
        
       | zdmc wrote:
       | Love the concept, and nicely executed! But what is Kafka-specific
       | here? It seems like this illustrated guide describes _any_ highly
       | scalable, persistent pub /sub system
        
       | redsummer wrote:
       | would be good in pdf or cbr/cbz format for offline reading
        
       | metrue wrote:
       | Soooo lovely, this is the most romantic things I've ever seen.
        
       | void_mint wrote:
       | This is great. Is there a way I can pay for it?
        
       | zebnyc wrote:
       | This is great! I am going to read this to my 3 year old and hope
       | he likes it,
        
       ___________________________________________________________________
       (page generated 2021-06-18 23:02 UTC)