[HN Gopher] Is BI dead? - On dismantling data's ship of Theseus
___________________________________________________________________
Is BI dead? - On dismantling data's ship of Theseus
Author : sebg
Score : 109 points
Date : 2021-09-18 23:30 UTC (2 days ago)
(HTM) web link (benn.substack.com)
(TXT) w3m dump (benn.substack.com)
| MangoCoffee wrote:
| >BI should include all consumption
|
| we use Microsoft PowerBI. it work great within the Microsoft
| ecosystem. if you try to pull data from NetSuite with PowerBI. it
| went to crap.
| riskneutral wrote:
| Why? What is special about NetSuite?
| MangoCoffee wrote:
| its the ERP software that we use. people needs to do report
| with data from NetSuite.
| 58x14 wrote:
| I thoroughly appreciate when an author uses an extensive variety
| of sources that are particularly relevant to the topic.
|
| I wonder how long this took to write.
| 1-6 wrote:
| I've gotten away with Plotly. Easy enough and looks great.
| drfuchs wrote:
| If you make it to the 8th paragraph, you may learn that "BI"
| stands for "business intelligence".
| da_chicken wrote:
| And what it actually means is "visualizations and dashboards".
| Why make decisions based on analyzing the data from a report
| when you can get a vague impression from a pretty chart and go
| with that instead?
|
| IMX, the core problem is that all BI software is built around
| the idea that everything is a financial value that can always
| be arbitrarily aggregated meaningfully, and that looking pretty
| is more important than presenting the data in a way that your
| organization actually finds logical.
|
| It's a way to sell very expensive software to decision-makers,
| while also producing software complex enough that it can't be
| configured well enough to properly evaluate until well after
| you've already bought it. Only then do you find that the
| feature set is very broad, but very shallow. It's only 18
| months later that you discover how limited the report writer
| is, or that you have to do it this one way for everything even
| if you really could use it formatted slightly differently.
|
| It's like using a pivot table in Excel and trying to control
| the order of the columns, or to make one table aggregate two
| values distinctly, etc. You end up with 10 seconds to pull your
| data and create the table, and 4 hours trying to get it to
| display in the manner you want before giving up.
| fifilura wrote:
| Where I used to work I feel like it was actually used. And it
| gave some surprises to deals that were until then seen as
| profitable.
|
| I worked with consumer apps and it was used for making
| business decisions related to return of investment and
| marketing.
|
| But it took some time to get there and before that I think
| most decisions were based on questionable assumptions from
| half-baked results.
|
| What was the final building bricks were to be able to
| calculate the lifetime value of recently acquired users, but
| also to collect all the types of revenue and cost and through
| various tricks (based on user base numbers mostly) break it
| down into countries and marketing channels.
| otabdeveloper4 wrote:
| Where I've worked BI wasn't ever used by decision makers.
| It's used to make dashboards that are part of a sales pitch
| to clients, because clients love pretty pictures and
| "science".
| x86_64Ubuntu wrote:
| That largely depends on the culture of the company. I've
| been at places that used BI to make decisions, and other
| places that just used it to nitpick over events from years
| ago.
|
| BI, like every other tool doesn't exist in a vacuum. It has
| to be present in an environment that has peopeople that
| know how to help themselves use the self-service attributes
| (i.e not Boomers), it also has to occur in a culture where
| KPIs and metrics are understood, and used in day-to-day
| discussions of the business, there should also be a
| training component to deploying your BI solution too.
|
| Without these aspects being present, your BI solution will
| devolve into either a sales prop with pretty colors, or it
| will become a web version of Excel.
| nemacol wrote:
| If you can pepper in the phrase "single pane of glass" you
| will be sure to hook em with the sale pitch.
| 58x14 wrote:
| I laughed, I exclaimed... and then I had a moment of
| silence.
| horsawlarway wrote:
| This matches my experience entirely. BI was not for making
| good decisions, it was for making better sales pitches.
| BeefWellington wrote:
| > IMX, the core problem is that all BI software is built
| around the idea that everything is a financial value that can
| always be arbitrarily aggregated meaningfully, and that
| looking pretty is more important than presenting the data in
| a way that your organization actually finds logical.
|
| I used to work directly modeling data for various sized
| companies on behalf of two different BI sellers.
|
| As I see it there are perpetually two problems:
|
| 1. BI is marketed at and sold to anyone needing any kind of
| data visualization capabilities, and most packages I've
| worked with can do this but that's not really where they
| shine.
|
| 2. They really shine when you have huge amounts of data
| stored in different systems and you are trying to build an
| environment up where you can coalesce that data into one
| place and then visualize it, and you need to routinely report
| on these sorts of things.
|
| I think most BI solutions fall into the same space as JIRA
| does -- highly-customizable solutions that are sold as "Turn-
| key" without any warning ahead of time that it'll require
| someone (or several people) on your team become ridiculous
| levels of expert in areas that don't necessarily help the
| business.
|
| It's a specialized skill most places don't really need when
| what they're looking for is a simple reporting package that
| can connect to just any database.
|
| Similarly, I saw a lot of people with BI insisting they had
| "Big Data" and thus "needed" things like Hadoop in order to
| process their stuff, when in reality the MySQL, SQL Server,
| or Oracle DB they were already invested in would do the same
| job faster 99.9999% of the time.
| meepmorp wrote:
| If, like me, you've sustained enough corporate brain damage,
| you assumed that's what it meant from reading the headline.
| kgwgk wrote:
| It was either that or Boehringer Ingelheim.
| majormajor wrote:
| To be fair - if you aren't already familiar with that, you
| probably aren't the target market for the article at all.
|
| (Though even in the target audience, I'm not taking much away
| from this. Tools should be better. Yep.)
| Joeboy wrote:
| My first though was bisexuality, then Basic Income, then
| glanced at the article and found it was something to do with
| Salesforce, then found your comment. Thank you so much.
| [deleted]
| pram wrote:
| I've spent the last 10 years of my life building "Big Data"
| platforms for companies (Kafka, Druid, Hadoop, Databricks,
| Presto, Teradata, Informatica, MicroStrategy, etc etc) and I
| still have no idea what people need most of this crap for.
| There's so much duplicative functionality, and a basically never
| ending list of OLAP adjacent products emerging every year. I mean
| it's good for me personally, it's pretty much a gravy train, but
| I always question if a lot of it is a solution searching for a
| problem lol
| clusterhacks wrote:
| Similar experience for me. Years ago I made a decision to jump
| into data engineering and reporting instead of pushing more
| into enterprise app development (meaning, Java/C# stuff).
|
| I share the question about a solution searching for a problem.
| I will say that I continue to see efforts being made that seem
| focused on adding a bullet point to some executive leader's
| yearly accomplishments rather than specifically providing value
| back to an org . . .
| the_af wrote:
| Interesting. To me, you seem to be conflating some pieces of
| data infra plumbing with very sensible and concrete goals
| (Kafka, Hadoop) with others that are adjacent tools which may
| or may not make sense. Regardless of any tech details, the idea
| behind Kafka makes a lot of sense:
| https://engineering.linkedin.com/distributed-systems/log-wha...
|
| Or did you mean, "I have no idea what business needs all this
| data for", a much broader question?
| geodel wrote:
| Well ideas behind all the other stuff will also make lot of
| sense if one align with perspective of snake oil salesman.
| And I do think Kafka/hadoop etc fit in snake oil category.
|
| Linkedin saying Kafka is good for them is not saying much.
| pram wrote:
| It can become the opposite of sensible, very easily. I've had
| devs demand to use Kafka for storing 100mb+ blobs (which of
| course required a custom broker config + cluster)
| geodel wrote:
| Indeed. The idea that somehow Kafka is beyond reproach
| seems amusing to me.
| chris_wot wrote:
| This seems like a promotion for Salesforce.
| oxfordmale wrote:
| Salesforce is anything but "No Code", it was just a clever
| marketing ploy. Just look at job postings for Salesforce
| Developers.
|
| I don't think BI analysts have to get worried quite yet.
| jl6 wrote:
| The ugly truth of BI tools is that they are of most benefit to
| the organizations which are least capable of using them.
|
| The goal of properly exploiting BI comes with many prerequisites
| which sound superficially reasonable but turn out to be decade-
| long side-quests. Things like having all your data in one data
| model. Things like understanding where your data comes from, and
| exactly what it means.
|
| These prerequisites are easy for small orgs, but small orgs
| benefit least from BI and typically get better bang-for-buck from
| Excel.
|
| Large orgs find themselves mired in the political meta-problems
| of meeting those prerequisites, like joining up the fiefdoms that
| own data sources with the cabals that run data governance and the
| accountants who want a return on the investment of simplifying a
| sprawling legacy estate.
|
| BI tools are generally incredibly poor at dealing with the bag-
| of-spanners heterogeneous data landscapes that exist in these
| organizations, and their analytics nirvana remains largely
| unattainable.
|
| The trend that the article describes - towards simpler,
| composable BI components, each with more modest goals - is
| progress. It helps move focus away from the relatively-easy
| problem of visualization, to the rest of the data stack.
| Foobar8568 wrote:
| Or that most people are totally clueless about their own
| companies, processes and data.
| d--b wrote:
| I am building Jig (based on observable): https://www.jigdev.com
|
| Because I believe that BI should be done by people who have a
| mixed set of skills between stats and programming.
|
| A programmer alone can't make the stats, and a mathematician
| can't build a reporting tool.
|
| Jig is a shortcut to build reports quickly, but I am somehow
| convinced by javascript is just not good enough to do that job.
|
| I think I might have to try and design a language that has
| indexed containers as first class objects...
| b9a2cab5 wrote:
| In my experience BI has no business being done by technical
| people. You need business people who are just technical enough
| to use the system for what they need. You need domain knowledge
| and deep product understanding to know what data to be looking
| for, not a stats and CS whiz. In 99% of cases, SQL backed by a
| fast distributed analytics database and Tableau+Excel is enough
| for that.
|
| Even as a software engineer doing basic analytics charts for
| the service I own I don't want to be writing JavaScript.
| Ideally I wouldn't be writing SQL either.
| monkeydust wrote:
| As someone who runs teams deploying BI to internal stakeholders
| (product, sales) I am pretty fed up with them.
|
| Firstly Tableau, QlikView or PowerBI are all pretty much doing
| the same thing whatever flavour you prefer.
|
| We find that maybe 10% of users will actually use them to garner
| new actionable insights and 90% will moan there's too much data
| for them to process and no one appreciates the immense amount of
| data munging that goes on behind the scenes to make those pretty
| graphs.
|
| Where do we go? Personally I see a lot of the insights being
| automated and turned into actions on the server side without
| anyone touching a BI tool. This can be achieved with rules based
| approach and perhaps some correlation, trend analysis over time
| series data. Where you do need to go beyond that then perhaps
| AR/VR will provide a novel and more valuable approach.
| TOMDM wrote:
| I've been doing a fair amount of BI work myself recently.
|
| I reckon of the 90% that don't make any data driven decisions
| with the dashboards they're provided, there are three causes I
| run into frequently.
|
| 1. They can't describe what info they need
|
| 2. That info isn't available.
|
| 3. They don't make decisions based on data at all.
|
| I feel I'm getting better at addressing 1 and 2 the more I do
| it, but I figure there's a fair amount from group 3 who demand
| the reports nonetheless...
|
| As an aside, you mention Tableau, QlikView or PowerBI, have you
| tried Apache Superset at all?
| monkeydust wrote:
| Yes on 3.
|
| One way to solve is to automate the action from insight part
| and inject into users workflow. I am looking at this now with
| support of senior sales stakeholders.
|
| Simple example:
|
| Client X activity has been trending down vs it's peer group,
| Call Client X to understand what's going on.
| gav wrote:
| This is the direction that I'd like to see more people
| moving in: pushing insights to users rather than expecting
| users to come to dashboards and get those insights.
|
| Nobody is saying "I'd love another tool that I am expected
| to check every day", both embedded analytics and insights
| are how to drive adoption and unlock value from data.
| TOMDM wrote:
| Are there any good examples of this already out there?
|
| It feels like a good idea, but I imagine it would be very
| fragile in black swan events.
|
| Maybe a mix?
|
| Provide the dashboards, but allow users to set up
| alerts/workflows, they understand the problem well, and
| if they know what they're going to be going to the dash
| to find, let them skip that step. In case something new
| happens, they can set up a new alert/workflow/whatever
| for the next time it happens.
| tomnipotent wrote:
| > pushing insights to users
|
| This is part of the expectation of business analysts,
| that they will surface and communicate data not
| immediately being asked for by executives or other
| stakeholders. This usually means running a meeting and
| presenting findings, and then pairing with another
| stakeholder to flesh out further opportunity.
|
| If there's no one to pick up the baton, there's very
| little that can be done about that.
| TOMDM wrote:
| I like the idea.
|
| Only works if you've got buy-in from management though.
| gumby wrote:
| > there's a fair amount from group 3 who demand the reports
| nonetheless...
|
| It's a very important function: provide data that can be
| cherry-picked to support a decision already made.
|
| If the decision is too complex for that, hire a big
| consulting firm to write a report suggesting and justifying
| the action desired.
|
| A decison staple surely going back millennia.
| RandomLensman wrote:
| I'd add a fourth: having no idea or model how the past
| relates to future/what are predictors of success or failure
| in the data
| FranzFerdiNaN wrote:
| I work as a data engineer in a BI team, with our product being
| part of a large software suite in healthcare. We got a small
| number power users who really use the dashboards, who get real
| result from them and who actively help our development. We also
| got a larger group of people who use it once in a while.
|
| But most people seem to log in once or twice and then never
| again. And if you ask why it's often because they couldn't find
| the answer to their incredibly specific one-time only question,
| so they decided it's all useless and they continue to work on
| their own personal homegrown BI-suite in excel.
|
| Also, we use Qlik and I absolutely hate it. Luckily I don't
| have to work with it too often but whenever I do I feel like
| I'm always fighting it. Using Power BI and Tableau felt much
| more like working together to solve issues.
| perl4ever wrote:
| >they continue to work on their own personal homegrown BI-
| suite in excel.
|
| Why aren't you providing your dashboards via Excel, then?
| What exactly does the fancy BI software do that Excel won't?
| I have used Qlikview but not Tableau or Informatica, and I'm
| a bit fuzzy on what people mean when they say "Power BI". Is
| using Power Query what you would call Power BI? Or does that
| mean using the data model stuff? Power Query is the part of
| Excel that I particularly like because data kind of stays
| where you put it.
| _delirium wrote:
| > And if you ask why it's often because they couldn't find
| the answer to their incredibly specific one-time only
| question, so they decided it's all useless and they continue
| to work on their own personal homegrown BI-suite in excel.
|
| Hah this is probably me. The university I work for uses a BI
| tool (MicroStrategy) to track students/majors/etc. But I
| usually find it easier to use the BI tool primarily as a way
| to get a CSV or XLS export of enough source data to answer my
| question, and then do the actual analysis in Python+pandas. I
| can _probably_ formulate my query in their weird proprietary
| query builder, but it seems unnecessarily complicated and I
| already know how to do data analysis in Python.
| perl4ever wrote:
| >their weird proprietary query builder
|
| If you're talking about Power Query, the best way to use it
| is to write code directly once you observe what the
| interface generates. It's a rather nice kind-of-functional
| language that really comes into its own when you start
| using/creating higher order functions.
|
| Power Query gave me an intense craving for a form of SQL
| with a similar facility for functions.
|
| For example, I have a lot of experience writing Oracle SQL,
| and Power Query didn't offer "grouping sets". But I
| realized it could be implemented using functions. It would
| be so great if SQL supported that.
| ndnichols wrote:
| I couldn't agree more! We built Lexio to address exactly that
| issue. (Our tagline is literally "Stop building dashboards that
| no one uses".) It does data storytelling (discovering the
| insights and conveying them in natural language) on the server
| and gives users a newsfeed experience for consuming them. I'm
| happy to talk shop if folks are interested in how it works.
| https://narrativescience.com/lexio
| marcinzm wrote:
| I run a data team. We do two things:
|
| * Provide a clean minimal view of the data that other teams can
| pivot table on top of or download in the BI tool. We do
| onboarding, support and all that for this data.
|
| * Provide dashboards and reports for more complicated ongoing
| questions that require data that's not cleaned yet.
|
| We don't use Looker but based on the sales demo it seems
| optimized for this type of workflow. The core analytics team
| maintains the data and complex workflows while other teams have
| business analysts for day to day questions.
| Guest42 wrote:
| Having used looker, it felt rather clunky and proprietary
|
| I was generally able to create better results via power bi,
| ssrs, or shiny
| marcinzm wrote:
| As I understood it the goal of Looker isn't to make the
| lives of people who can use something like Shiny better but
| rather the lives of people who use pivot tables 99% of the
| time. So rather than having technical analysts making
| dashboards you have business analysts making dashboards (or
| at least digging into generic dashboards).
| haddr wrote:
| This is the direction where most BI work is heading for: you
| need a team of data engineers that can make sure that the
| data for your BI dashboards is delivered and in a good
| quality. Everything else that doesn't need it is probably
| simple enough to be run as some PowerBI dashboard on top of
| production DB. I think today most effort in those heavier BI
| cases is spent on data transformations and making sure that
| data flows/batch jobs run uninterrupted.
| 0xbadcafebee wrote:
| Data is clay. We are still struggling to make bricks. We are
| still very far from building a house out of it, much less a large
| building. And we're going to eventually find out that we can't
| make skyscrapers out of it at all.
|
| Just like bricklayers haven't even remotely been replaced with
| robots, I doubt the data-mungers will be, either. We are probably
| on the cusp of a whole new generation of "data people" who will
| have careers that span a generation, and do nothing but sift
| through data.
| jpmonteiro wrote:
| I entered in a discussion with Benn on Twitter on Friday around
| this topic. Here is my take on it in case you are interested in a
| different perspective:
| https://jpmonteiro.substack.com/p/a-friday-fight-and-the-int...
|
| TL;DR: I believe the trend will be to have an ecosystem-like
| approach to BI with tools complementing each other and
| communicating with each other through open standards.
| mdoms wrote:
| Not sure why the author spent half the article writing about
| Salesforce which turned out to be entirely irrelevant.
| isoprophlex wrote:
| Oh, I OTOH really enjoyed the historical perspective they
| sketched. I appreciated the work put in to give some context.
| slt2021 wrote:
| it is because Salesforce conference is happening right now in
| San Francisco
| intrasight wrote:
| I remember learing decades ago that "In UNIX, everything is a
| file" I approach BI now as "In BI, everything is in your
| database". With everything in the database (SQL Server in my
| case) I can deliver BI to my users with quality tools like
| Tableau and Power BI - or just with generic web reports.
| jon-wood wrote:
| It really is amazing how far you can get by just dumping
| everything into a single database and letting people do joins.
| We're doing the same thing with Stitch replicating data from
| production databases and a few SaaS products into a big
| Postgres instance and it's so low on maintenance hassle. The
| most advanced our "BI pipeline" gets is a few stored procedures
| run from a cron job to aggregate data in a particularly large
| table.
| PhilipA wrote:
| I do believe that we have seen solutions trying to be a
| horizontal solution to problems, and I think the evolution will
| turn into vertical solutions. I have also invested a lot of time,
| as we are a couple of people who has been building on a vertical
| BI tool for the SaaS space for about a year.
|
| The main goals are:
|
| - Easy accessible for novice users. We want to make the Google
| for BI to help empower non-technical people (no more salespeople
| asking for reports from devs)
|
| - More advanced editors for the more technical people
|
| - Advanced alerts + integrations to 3rd parties
|
| - Later on proactive reports
|
| Hit me up on @philipanderse if you want to test it out.
| molsongolden wrote:
| Interested in learning more but it looks like your DMs aren't
| open.
| PhilipA wrote:
| Just follow me and I can follow you back.
___________________________________________________________________
(page generated 2021-09-21 23:00 UTC)