[HN Gopher] The Basics of Apache Kafka
       ___________________________________________________________________
        
       The Basics of Apache Kafka
        
       Author : the2ndfloorguy
       Score  : 54 points
       Date   : 2021-09-07 08:05 UTC (14 hours ago)
        
 (HTM) web link (www.pankajtanwar.in)
 (TXT) w3m dump (www.pankajtanwar.in)
        
       | Railsify wrote:
       | Kafka 2.8.0 removes the mandatory zookeeper dependency.
        
       | rad_gruchalski wrote:
       | This should be labelled as Show HN. One comment - towards the end
       | you say
       | 
       | > You might think that Kafka would be using queue data structure
       | internally. It's not true. Kafka uses a "log" data structure. It
       | is a persistent data structure which allows only appends, no
       | editing, no deletion. In detail, we will cover this some other
       | day.
       | 
       | Almost immediately followed by:
       | 
       | > Ok, I have a confession . I lied. A Kafka topic is not just a
       | single queue. It's a combination of queues which helps kafka
       | scale. Every queue is called partition.
        
         | Tomte wrote:
         | > This should be labelled as Show HN
         | 
         | No, it's a simple blog post.
        
           | rad_gruchalski wrote:
           | Yes, posted by the author of the blog post.
        
             | thunderbong wrote:
             | From the FAQ [0] -
             | 
             | > Show HN is for sharing your personal work and has special
             | rules.
             | 
             | The first line of the rules [1] -
             | 
             | > Show HN is for something you've made that other people
             | can play with. HN users can try it out, give you feedback,
             | and ask questions in the thread.
             | 
             | [0]: https://news.ycombinator.com/newsfaq.html
             | 
             | [1]: https://news.ycombinator.com/showhn.html
        
               | rad_gruchalski wrote:
               | You're correct. I have learned something new:
               | 
               | > Off topic: blog posts, sign-up pages, newsletters,
               | lists, and other reading material. Those can't be tried
               | out, so can't be Show HNs. Make a regular submission
               | instead.
        
       | moedersmooiste wrote:
       | We use Kafka a lot within the company I work for. I think it's
       | great. The only thing I miss is fast lookups based on some key
       | and/or the ability for subscribers to only receive messages for
       | certain keys.
        
         | rad_gruchalski wrote:
         | That's most likely the trade off between the queue and a log.
         | The property of the log is having to look at every message.
         | 
         | The prefixes can be somewhat done with a custom partitioner,
         | the lookup requires another technology. The only way to know
         | the final value under the key is to look at the complete log.
        
         | dmead wrote:
         | oddly enough, zeromq has the prefix/key style subscription.
         | 
         | you might look at ksql for filtering purposes or just tried to
         | work such things into your topics/partitions?
        
       | brudgers wrote:
       | At the computer science level, there's _What Every Software
       | Engineer Should Know About Real-Time Data Unifying_ from 2013:
       | 
       | https://engineering.linkedin.com/distributed-systems/log-wha...
       | 
       | Reading it was an aha. At its core, the log is the simplest thing
       | that might work in a lot of cases.
        
       ___________________________________________________________________
       (page generated 2021-09-07 23:02 UTC)