[HN Gopher] My toy project turned into a viral challenge
       ___________________________________________________________________
        
       My toy project turned into a viral challenge
        
       Author : jinay
       Score  : 48 points
       Date   : 2021-11-30 16:30 UTC (2 days ago)
        
 (HTM) web link (blog.jinay.dev)
 (TXT) w3m dump (blog.jinay.dev)
        
       | tyingq wrote:
       | You were sending the entire database (all times from all users)
       | to each individual end user to compute histograms client-side
       | after they finished? Ah, yeah, that could get expensive on
       | Firebase.
        
         | jinay wrote:
         | I guess it wasn't top of mind at the small scale I planned to
         | operate at, but definitely a facepalm when you put it that way.
        
       | notwhereyouare wrote:
       | is there a way to see the results without going through the
       | challenge?
        
         | jinay wrote:
         | I filter out any data points <5 seconds (as seen in the graph),
         | so completing your attempt in under 5 seconds should do it.
         | 
         | Ideally, people would do the challenge first and then see where
         | they lie on the graph before seeing the data itself.
        
         | dwighttk wrote:
         | Nice bell curve around 60s but a little higher on left than
         | right (more people underestimate, but proportionally) and
         | there's a spike at 0-8 seconds (from people who just wanted to
         | see results or decided to quit quickly.)
        
         | tyingq wrote:
         | https://imgur.com/a/jVYbnSn
        
       | blopker wrote:
       | Regarding the histogram issue, I worked on a project that had a
       | few hundred histograms based on data from over 3 billion data
       | points. It turns out that after a few thousand data points many
       | histograms will stop changing significantly.
       | 
       | So, unless you really need to show exactly how many data points
       | each bucket contains, it's much easier to run the analysis once
       | offline, then serve just the histogram percentage data. From that
       | you can make an SVG and overlay additional user-specific data on
       | top. The point is that this histogram data is small and easy to
       | cache.
       | 
       | You can then rerun the histogram analysis later if you'd like.
       | However, for this project I never saw anything change with more
       | data. It was overkill even to run it as a cron job.
        
         | jinay wrote:
         | Right now this project is on the scale of ~100k points, but I'm
         | starting to see a drop in percentage change as you mentioned.
         | In the beginning, though, the trends weren't as clear so I
         | wanted to keep it updating.
        
       | boberoni wrote:
       | > I pushed a quick fix to the issue by freezing the data being
       | sent to the client, thereby halting the rapid growth in data
       | consumption.
       | 
       | What do you mean by "freezing the data"?
       | 
       | Regarding the excessive download problem, my first instinct is to
       | periodically (for example, every hour) compute summary statistics
       | for the bar chart and store that in Firebase. This, of course,
       | would require an additional script/service to perform these
       | periodic jobs.
       | 
       | I'm not sure if that's what you ended up doing and I'm curious
       | what your solution is.
        
         | malfist wrote:
         | Probably (and I'm guessing, not the author here) took a
         | snapshot of the data and hardcoded that to be sent to the
         | client instead of live data.
        
         | bocytron wrote:
         | > What do you mean by "freezing the data"?
         | 
         | https://github.com/JinayJain/just-a-minute/blob/master/app.j...
        
         | jinay wrote:
         | As others have mentioned, my solution in the moment was to
         | download the JSON file from Firebase, compute histogram
         | statistics manually, and hardcode the histogram into the JS
         | itself.
         | 
         | Obviously not a scalable solution, and I think I would have
         | done something very similar to the periodic updates like you
         | mentioned (if I had more experience with cloud functions etc.)
        
         | noahtallen wrote:
         | > This, of course, would require an additional script/service
         | to perform these periodic jobs.
         | 
         | Also worth noting that Firebase has built-in "cloud functions"
         | which have access to the database API. It would be pretty easy
         | to run one on a schedule.
        
       | post_break wrote:
       | Just some feed back. I started the challenge, but the constant
       | text changing like "I'm bored" etc was enough to throw off my
       | internal clock to guess when to stop it.
        
       ___________________________________________________________________
       (page generated 2021-12-02 23:02 UTC)