[HN Gopher] Alert Evaluations: Incremental Merges in ClickHouse
       ___________________________________________________________________
        
       Alert Evaluations: Incremental Merges in ClickHouse
        
       Author : vadman97
       Score  : 31 points
       Date   : 2024-10-02 12:55 UTC (1 days ago)
        
 (HTM) web link (www.highlight.io)
 (TXT) w3m dump (www.highlight.io)
        
       | hodgesrm wrote:
       | It sounds as if you used your own algorithm for pre-aggregation
       | into the schema you showed. Did you consider using a materialized
       | view to populate it?
        
         | nitinreddy88 wrote:
         | Precisely, whats stopping them from using simple Incremental
         | Materialised view?
        
           | vadman97 wrote:
           | If there was only one alert criteria, that'd be simple. Our
           | alerts can be configured for any data filters (eg. only
           | matching logs with column `level='error'`); we would have to
           | create a unique MV for each alerts' filter condition.
        
         | vadman97 wrote:
         | We insert from our alerts worker because we want the
         | aggregation to happen per alert (with the aggregated data
         | filtered by the particular alert definition). As each alert is
         | evaluated, we run the following [1] INSERT INTO ... SELECT ...
         | statement based on the alert definition. We can't aggregate
         | with an MV since we'd need to create an MV per unique alert
         | that a customer may set up.
         | 
         | [1]:
         | https://github.com/highlight/highlight/blob/c526daea31fdf764...
        
       | iampims wrote:
       | At a certain scale, exact computations (p50 for instance) become
       | impractical. I've had great luck switching to approximate
       | calculations with guaranteed error bounds.
       | 
       | An approachable paper on the topic is "Effective Computation of
       | Biased Quantiles over Data Streams"
       | http://dimacs.rutgers.edu/%7Egraham/pubs/papers/bquant-icde....
        
         | hipadev23 wrote:
         | The quantile() method in clickhouse is also approximate
         | although it uses a more simplistic reservoir sampling model
         | than GK, but quantileGK() is also available. quantileExact()
         | exists but indeed becomes impractical as you point out.
        
       ___________________________________________________________________
       (page generated 2024-10-03 23:01 UTC)