[HN Gopher] Apache Beam for Search: Getting Started by Hacking Time
       ___________________________________________________________________
        
       Apache Beam for Search: Getting Started by Hacking Time
        
       Author : clandry94
       Score  : 50 points
       Date   : 2021-01-08 17:13 UTC (5 hours ago)
        
 (HTM) web link (shopify.engineering)
 (TXT) w3m dump (shopify.engineering)
        
       | kordlessagain wrote:
       | The link for the Apache Beam project has the hostname and domain
       | transposed. The correct URL is: https://beam.apache.org/
        
         | softwaredoug wrote:
         | Author here, thanks. I'll get it fixed.
        
           | kordlessagain wrote:
           | You got it fixed in no time at all! :)
        
       | waynesonfire wrote:
       | does it support data lineage?
        
       | runT1ME wrote:
       | It would be helpful if this thorough of an example could live on
       | the apache beam website, I think it would avoid a lot of
       | confusion. I certainly found the way it handles windowing with
       | triggers to be quite diffrent than say, Spark.
        
         | softwaredoug wrote:
         | Yeah I agree. I basically wrote this blog out of my challenges
         | learning this content. It involved a lot of code spellunking
         | and trial and error to figure out precisely what these concepts
         | meant. I do find Beam powerful, but also to be a bit esoteric
         | at times and difficult to follow how watermarks, windows, and
         | triggers all work. And we encounter sometimes unexpected
         | behavior that frequently causes us to revise our understanding
         | of these concepts.
         | 
         | There's a fair amount of Stackoverflow highly voted answers out
         | there like "I dunno, try this trigger, see if it works" without
         | much understanding of how everything work underneath. Probably
         | cause it's tricky to grok
        
           | runT1ME wrote:
           | I'm not even sure the core Beam engineers understand it all!
           | Look at how Kafka offset acks are handled now:
           | 
           | https://github.com/apache/beam/blob/master/sdks/java/io/kafk.
           | ..
        
       ___________________________________________________________________
       (page generated 2021-01-08 23:01 UTC)