[HN Gopher] Open sourcing Dicer: Databricks's auto-sharder
       ___________________________________________________________________
        
       Open sourcing Dicer: Databricks's auto-sharder
        
       Author : vivek-jain
       Score  : 38 points
       Date   : 2026-01-13 19:56 UTC (3 hours ago)
        
 (HTM) web link (www.databricks.com)
 (TXT) w3m dump (www.databricks.com)
        
       | ayf wrote:
       | Does anyone else have something similar?
       | 
       | What are some use cases that you found are useful?
        
         | vivek-jain wrote:
         | Sharded in-memory caching turns out to be rather useful at
         | scale :)
         | 
         | Some of the key examples highlighted on our blog are Unity
         | Catalog, which is essentially the metadata layer for
         | Databricks, our Query Orchestration Engine, and our distributed
         | remote cache. See the blog post for more!
        
         | louis-paul wrote:
         | Sounds related to Google Slicer:
         | https://research.google/pubs/slicer-auto-sharding-for-datace...
        
       | khaki54 wrote:
       | Seems weird to call it sharding since it's not sharding indexed
       | datasets or anything like that. Is this just a tool to mitigate
       | Databricks' internal service-scaling challenges?
        
       ___________________________________________________________________
       (page generated 2026-01-13 23:00 UTC)