[HN Gopher] The Next Big Challenge for Data Is Organizational
       ___________________________________________________________________
        
       The Next Big Challenge for Data Is Organizational
        
       Author : kiyanwang
       Score  : 16 points
       Date   : 2021-10-30 10:30 UTC (1 days ago)
        
 (HTM) web link (locallyoptimistic.com)
 (TXT) w3m dump (locallyoptimistic.com)
        
       | rightbyte wrote:
       | I wonder how much money is wasted trying to engineer ourself out
       | of the mess left from when the secretaries were fired. There is
       | no way to organize data without ongoing work in catalogizing it
       | and purge irrelevant items.
        
         | civilized wrote:
         | You've got it. My company just got a fancy data catalog and
         | while it is helpful, what it's really revealed is the godawful
         | mess of 1000x duplicated, completely undocumented tables and
         | schemas we have, thanks to decades of neglect and shortcuts.
         | 
         | I've been trying to populate the catalog with a bit more
         | explanation of what things are for + data dictionaries, but it
         | feels like boiling the ocean.
        
       | iblaine wrote:
       | Organizing data is a problem everyone wants to solve until they
       | figure out the cost. It's much easier for organizations to throw
       | data over the fence and let data engineers fend for themselves
       | than it is to get buy-in from the top to organize data closer to
       | the source.
       | 
       | There seems to be a lot of hype and hope to organize domains of
       | data much like you would products of a business. I just don't see
       | it happening today, largely because people in
       | engineering/product, upsteam of DE teams, are unwilling to put in
       | the effort.
        
       | diob wrote:
       | Check out data mesh. It's pretty much supposed to be where the
       | next shift is.
        
         | throwawaysea wrote:
         | Can you share a good resource on that topic? I found the
         | following from a quick search but it all seems a bit too
         | abstract to understand for a casual reader. Is there something
         | more practical and approachable?
         | https://martinfowler.com/articles/data-mesh-principles.html
        
       | politelemon wrote:
       | > Error establishing a database connection
       | 
       | That's an older challenge.
        
       ___________________________________________________________________
       (page generated 2021-10-31 23:01 UTC)