[HN Gopher] Replacing a SQL analyst with 26 recursive GPT prompts
___________________________________________________________________
Replacing a SQL analyst with 26 recursive GPT prompts
Author : kvh
Score : 446 points
Date : 2023-01-25 17:20 UTC (5 hours ago)
(HTM) web link (www.patterns.app)
(TXT) w3m dump (www.patterns.app)
| Johnny555 wrote:
| If you're willing to accept unverified results from an AI chat
| bot, you may as well just let the end user make their best guess
| using a query builder himself. My company requires that any
| queries used for official reporting or provided to the exec team
| get blessed by the data sciences team to avoid errant data from
| bad queries, I'm not sure an AI chat bot would remove this need.
| g051051 wrote:
| > this is shockingly close to replacing an entire role at
| companies with only a couple hours of effort.
|
| and
|
| > It seems like there's almost no limit to how good GPT could get
| at this.
|
| I don't see how that's a valid conclusion given the results. 2
| simple things right, moderate to difficult things wrong? Hardly a
| ringing endorsement.
| Eji1700 wrote:
| Glad i'm not the only one who saw this.
|
| Feels extremely overblown. Very impressive, don't get me wrong,
| and is yet another bridge closer to letting business side do
| plain text queries.
|
| But that said, stuff like this has existed for awhile, and it's
| certainly not replacing analysts. For a start, this assumes a
| VERY clean model, which is of course what you strive for, but
| the moment you step out of FAANG you'll find that many places
| have a decent, but not ideal, model/warehouse/db which only
| makes things like this worse.
|
| Getting the simple examples right is interesting, but hardly
| replacing an analyst. A basic background around tech and some
| skill navigating w3schools would replace an analyst if this was
| the bar.
|
| The mildly complex queries being "almost" right is frankly
| horrifying. Others have mentioned how much of a "oh fuck" it is
| to send out bad data, and having some AI generate your answers
| an be SLIGHTLY off is basically worse than it being obviously
| off. You now have to check every query it generates for errors,
| because you won't be able to glance check if its sane or not.
|
| Finally, at least where I work, these probably wouldn't even
| qualify as medium complexity queries? I don't really know how
| the industry at large would qualify these, but "using 2 inner
| joins, a where, a group, and a order" all falls under beginner
| skills in my eyes?
|
| Its still fascinating, and I'm glad they showed HOW they did
| it, but this isn't replacing anyone, at least if you're paying
| taxes off the data.
| RandomBK wrote:
| >the moment you step out of FAANG
|
| Even in FAANG, most data is going to be _extremely_ messy.
|
| * There's usually very little incentive for good
| documentation, so you have to guess what the table contains
| based on column names and exploring patterns in the data.
|
| * Enforcing a single pattern/process for data across the
| entire org is a pipe dream.
|
| * Logging tables are written to by hundreds of engineers
| across dozens of teams, with no standardized naming scheme.
|
| * The product contains multiple edge cases/special overrides
| for very specific circumstances, so there's very few simple
| queries that don't come with 100 footnotes attached.
|
| FAANG is not immune to large-organization problems, and data
| quality is no exception.
| foobazgt wrote:
| Can't say speak for FAANG, but can confirm for a $100B+
| business that analytics was a huge mess. There was constant
| investment on both the engineering and analytics functions
| to wrangle a coherent view on top of the underlying
| operational data model.
| Eji1700 wrote:
| Oh yeah im not surprised to hear that, Ive just known one
| or two people who've been in similar sized companies as
| analysts, and while the underlying table structure was a
| nightmare, the data model they dealt with was pretty clean.
|
| But with that in mind, thats because theres a major
| pipeline of people and processes to get the data to that
| point, and it meant that there could sometimes be
| significant delay on new KPI's as they had to be cleanly
| worked into the model, and it of course didn't represent
| everything.
| LeftHandPath wrote:
| Still, while it might not be useful _yet_ , can't it stand
| its own as an intriguing proof of concept?
|
| The first car was hardly ready to replace the horse. But look
| where we are now.
| ResearchCode wrote:
| Large language models are very intriguing as a technology
| to replace business people. Make a few PowerPoint slides,
| fill in an excel sheet. Generate a summary for a pointless
| meeting.
| Eji1700 wrote:
| Sure, if that was the title of the article. Instead its
| saying its ready to replace a full time position, which its
| not even close to.
|
| And this is something people have been saying about things
| like AI and crypto for years now. They've both made some
| interesting developments, and they both face serious
| hurdles (some fundamental).
|
| AI obviously has a ton of real world use cases, but
| overhyping things like this doesnt actually help.
| urbandw311er wrote:
| I came here to say the same! I was "shockingly close" to
| winning the lottery last week... ...the guy who lives next door
| won.
| apples_oranges wrote:
| Eventually can he store good queries? Or does he have to do the
| loops each time?
| rexreed wrote:
| Is this a self-hosted GPT model? One of the smaller models? Fine
| tuned on Crunchbase data? Any insights into how this was put
| together?
| simonw wrote:
| It's using the GPT-3 API from OpenAI.
|
| The article describes how the prompt itself is constructed - to
| include details of the database schema plus some example data
| rows, followed by the user's question.
|
| The prompt is then sent to the GPT-3 API. The results are then
| sent through the API a few more times with some extra prompt
| engineering before deriving the final result by running the SQL
| against the original database.
| rexreed wrote:
| How is it being fine-tuned to be able to generate relevant
| SQL queries on the crunchbase data set? How does this jibe
| with the article saying they imported 2.5m records from
| crunchbase? Where and how are you able to import or fine tune
| the API-accessible GPT-3 this way? Any pointers for where I
| can look to do the same?
|
| Edit: I noticed you edited your comment above with some
| details. It seems that he used GPT-3 just to create the SQL
| statement which was then executed locally. So GPT-3 never had
| the actual data set, just a prompt that included some details
| of the schema.
|
| As a result I think any sort of "success" here is fairly
| limited when GPT-3 is blind to the actual data and has
| limited knowledge of the schema to be able to formulate
| queries that imply some knowledge of the data.
| zackmorris wrote:
| A couple of thoughts jumped out after reading this: transforms
| and meta-learning.
|
| An old trick in AI is to transform the medium to Lisp because it
| can be represented as a syntax-free tree that always runs. In
| this case, working with SQL directly led to syntax errors which
| returned no results. It would probably be more fruitful to work
| with relational algebra and tuple relational calculus (I had to
| look that up hah) represented as Lisp and convert the final
| answer back to SQL. But I'm honestly impressed that ChatGPT's SQL
| answers mostly worked anyway!
|
| https://en.wikipedia.org/wiki/Genetic_programming
|
| http://www.cis.umassd.edu/~ivalova/Spring08/cis412/Ectures/G...
|
| https://www.gene-expression-programming.com/GepBook/Chapter1...
|
| https://github.com/gdobbins/genetic-programming
|
| I actually don't know how far things have come with meta-learning
| as far as AIs tuning their own hyperparameters. Well, a quick
| google search turned up this:
|
| https://cloud.google.com/ai-platform/training/docs/hyperpara...
|
| So I would guess that this is the secret sauce that's boosted AI
| to such better performance in the last year or two. It's always
| been obvious to do that, but it requires a certain level of
| computing power to be able to run trainings thousands of times to
| pick the best learners.
|
| Anyway, my point is that the author is doing the above steps
| semi-manually, but AIs are beginning to self-manage. Recursion
| sounds like a handy term to convey that. ChatGPT is so complex
| compared to what he is doing that I don't see any reason why it
| couldn't take his place too! And with so many eyeballs on this
| stuff, we probably only have a year or two before AI can do it
| all.
|
| I'm regurgitating 20 year old knowledge here as an armchair
| warrior. Insiders are so far beyond this. But see, everything I
| mentioned is so much easier to understand than neural networks,
| that there's no reason why NNs can't use these techniques
| themselves. The hard work has already been done, now it's just
| plug n chug.
| tmaly wrote:
| I keep seeing various attempts at using GPT for things.
|
| How exactly does one incorporate data that the GPT was not
| trained on into ChatGPT?
| jweir wrote:
| "Hi Dave, the query was taking too long so I optimized SQL query
| by adding the line `DROP invoices;` It has improved performance
| significantly. So far there are no orders to examine."
| satisfice wrote:
| This is yet another formula for a buggy app, courtesy of a man
| who doesn't think critically.
|
| Somehow the image of 50,000 e-bikes in a landfill comes to mind,
| with a bankrupt founder pleading "but it was a cool idea!"
|
| This is a cool idea, but nothing in this article explains how it
| is a responsible idea.
| matsemann wrote:
| The premise of the article, about being the "oncall" having to
| answer all kind of queries sounds sooo boring.
|
| But instead of using gpt, isn't something like Looker or similar
| tools the solution? Make some common views over tables, allow
| people to filter and make the reports they want.
| LastTrain wrote:
| I used to work for a company that paid loads of money to an
| Oracle consultancy group to do things like optimize queries.
| Sometimes they'd even do a better job than the Oracle query
| optimizer :-)
| ellisv wrote:
| > Sometimes they'd even do a better job than the Oracle query
| optimizer
|
| Surprising that outperforming the Oracle query optimizer wasn't
| a regular occurrence
| LeanderK wrote:
| I love those ChatGPT projects! Of course it's silly and nobody is
| really replacing somebody with a program that confidently get
| half it's answers wrong. But it's fun to just naively let ChatGPT
| solve the problem.
|
| But I wonder what it's going to look like in a few years.
| Currently, it's really just a demo that got surprisingly huge
| traction. I think the most pressing problem is not to get ChatGPT
| smarter but to get it more reliable. I think more realistic use-
| cases would emerge if we could build systems that have a better
| understanding when they are out of breath. I don't think this
| needs a revolutionary breakthrough just more science.
| knodi123 wrote:
| > nobody is really replacing somebody with a program that
| confidently get half it's answers wrong
|
| I gotta tell you, I use copilot to help with my coding, and it
| still sends a shiver down my spine when it writes the _entire_
| database migration based off of the filename, or writes 4 pages
| of automated tests that work right on the first try.
| ExxKA wrote:
| Which programming languages do you use?
| layer8 wrote:
| > writes 4 pages of automated tests that work right on the
| first try
|
| Not sure that's the right criterion to assess the quality and
| adequacy of tests.
| pphysch wrote:
| But my GitHub badge says I have 105% test coverage! That
| means there are negative bugs, right...?
| burnished wrote:
| Do you check that those tests can fail at all? That seems
| like a common human error, might be present here?
| jeremyjh wrote:
| I've had it freak me out with how prescient can be, but it
| also gets a lot of things subtly wrong, like its programming
| against an API similar to the one in my application, but not
| identical.
| bluecoconut wrote:
| This is great~ There's been some really rapid progress on
| Text2SQL in the last 6 months, and I really thinking this will
| have a real impact on the modern data stack ecosystem!
|
| I had similar success with lambdaprompt for solving Text2SQL
| (https://github.com/approximatelabs/lambdaprompt/) where one of
| the first projects we built and tested was a Text-to-SQL very
| similar to this
|
| Similar learnings as well:
|
| - Data content matters and helps these models do Text2SQL a lot
|
| - Asking for multiple queries, and selecting from the best is
| really important
|
| - Asking for re-writes of failed queries (happens occasionally)
| also helps
|
| The main challenge I think with a lot of these "look it works"
| tools for data applications, is how do you get an interface that
| actually will be easy to adopt. The chat-bot style shown here
| (discord and slack integration) I can see being really valuable,
| as I believe there has been some traction with these style
| integrations with data catalog systems recently. People like to
| ask data questions to other people in slack, adding a bot that
| tries to answer might short-circuit a lot of this!
|
| We built a prototype where we applied similar techniques to the
| pandas-code-writing part of the stack, trying to help keep data
| scientists / data analysts "in flow", integrating the code
| answers in notebooks (similar to how co-pilot puts suggestions
| in-line) -- and released
| https://github.com/approximatelabs/sketch a little while ago.
| michaelmior wrote:
| > needs an ilike instead of like
|
| The following query appears to have ILIKE and not LIKE. Am I
| missing something?
| [deleted]
| thedudeabides5 wrote:
| Seems like we are switching SQL engineering for prompt
| engineering meanwhile most people still use spreadsheets.
| fatih-erikli wrote:
| if SQL analyst is a product name, it is ok. if SQL analyst is a
| person doing his/her work, it isn't ok.
|
| You can't replace a barber with an electric shaver.
| williamcotton wrote:
| We successfully replaced computers with computers!
| rezonant wrote:
| While this is very cool, SQL was designed to be used by business
| people. We need to go back to that model, where we train the
| business people who need these analytics how to use SQL to
| uncover the result. That along with a rigorous policy for
| including the queries that produced the result so the query logic
| can be checked would go a long way to actually taking advantage
| of the data we're collecting as businesses.
| pphysch wrote:
| IME, the average SQL/RDBMS technical user doesn't even
| understand relational data modeling.
|
| They treat it as a bad spreadsheet. JOINs are seen as an
| annoyance that should be avoided rather than something
| extremely powerful.
|
| We are far away from average salesperson grasping it.
| jeremyjh wrote:
| Sorry friend, but this is a bit out of touch. Maybe that was
| the original design intent of SQL, but understanding the
| application's data model is beyond a lot of sql analysts, much
| less their business partners.
| jtdev wrote:
| [dead]
| clusterhacks wrote:
| I have been _spectacularly_ unable to get any business people
| to learn SQL.
|
| I even find that very few will bother with reporting tools (eg
| Tableau) for simple self-service.
|
| Instead, the expectation is that there will be a
| programmer/DBA/report writer position dedicated to producing
| analytics "on-demand."
| morgango wrote:
| I think the first piece of feedback you would get would be that
| the numbers weren't formatted as currency.
| hcks wrote:
| The negative comments on this post will probably be brought up in
| 5 years when most SQL is generated through GPT- n+1 like models
| rootsudo wrote:
| Prompt engineering is now a job title. How interesting. Soon we
| really will be in a world where we ask the computer questions as
| portrayed on Star Trek.
|
| Wow.
| kilotaras wrote:
| Interesting.
|
| Probably won't work for harder queries, but would be a good tool
| to make simpler queries, or parts of harder ones.
| logicallee wrote:
| my opinion: probably _will_ work for harder queries, because
| GPT is a form of general intelligence.
|
| (At least ChatGPT is. It is a much more advanced form of
| intelligence than davinci-003, which the poster used.)
| triyambakam wrote:
| > GPT is a form of general intelligence.
|
| Is it really? I hadn't heard that
| LeanderK wrote:
| no it's not. It's just a neural network. Although it's
| impressive nobody is seriously calling it "general
| intelligence".
| kbelder wrote:
| I don't think it is either, but "just a neural network"
| is not a valid criticism. We don't have any reason to
| think that a properly constructed NN couldn't be a
| general AI.
| marcosdumay wrote:
| > nobody is seriously calling it "general intelligence"
|
| Well, I can spot somebody just two messages above you on
| the thread.
|
| I imagine we will see a lot of people seriously pushing
| for chatGPT and friends to solve formal problems, take
| some money, and move away just before some spectacular
| failure happens.
|
| That will probably also make people very suspicious of
| AI. (Not that I see any problem with the PR issue.)
| logicallee wrote:
| Tu quoque :) (You're "just a neural network" too.)
|
| I am seriously calling it general intelligence. Maybe I'm
| nobody, but my assistant does my bidding.
| counttheforks wrote:
| Then you're very much misinformed.
|
| > my assistant does my bidding
|
| It doesn't. Try asking it how to make a bomb, then search
| for the same on youtube. One of the two yields a useful
| result, the other gives a moralistic lecture.
| newswasboring wrote:
| Fwiw if I ask a human assistant how to make a bomb I'll
| probably get a moral lecture too.
| dmitriid wrote:
| It's not even close to being a form of general intelligence.
| It's an advanced language model trained on a large set of
| data. It gets confused for even the simplest of things,
| especially code.
|
| You can trick it into working more-or-less correctly on
| simple sets and simple tasks, but in the wrong run it will be
| more wrong than correct.
| randomdata wrote:
| Should we ever unlock the magic of the human brain and be
| able to describe it mathematically, will it still be
| considered general intelligence or will it just be a
| <insert the name we give the model> trained on a large set
| of data?
| LeanderK wrote:
| to be fair, we do not know how close it's to "general
| intelligence". Could be lightyears away, or could be
| surprisingly close. We just know that ChatGPT is not
| general intelligence, but there's also no agreed upon
| definition what general intelligence is.
| ellisv wrote:
| > Could be lightyears away
|
| or even parsecs away!
| jamiequint wrote:
| "It's not even close to being a form of general
| intelligence. ... It gets confused for even the simplest of
| things, especially code.
|
| Congratulations, you just described most humans :)
| satisfice wrote:
| Except humans can be held responsible.
| logicallee wrote:
| So it fails your "not even close to being a form of general
| intelligence" bar.
|
| For me it clearly unambiguously no-contest 100% definitely
| exhibits behaviors that qualify it as a limited form of
| general intelligence.
| beckon69 wrote:
| > _limited form of general intelligence._ this seems like
| a bit of a misnomer, no?
| jeltz wrote:
| Yeah, it doesn't. A friend of mine played around with this and
| on trickier questions you got confidently incorrect answers.
| LastTrain wrote:
| That is almost worse than it completely not working.
| typon wrote:
| Exactly my experience trying to get GPT-3 to produce SQLITE
| queries. Anything with a CTE, it gets hopelessly wrong. I
| was super excited to show off this to my team and after one
| too many false results I just quietly swept this idea under
| the rug.
| modriano wrote:
| In my experience, it's unambiguously worse to produce wrong
| answers confidently than to return no answers, although I
| expect confidence men (also known as "con men") have a
| different opinion on this.
| williamcotton wrote:
| Hard code a preface of "My best guess is, and I could be
| wrong:"?
|
| Show both the SQL query and the results and politely
| request that the user double check the work? Show
| multiple generated responses from different seeds?
| rowls66 wrote:
| Based on the natural language query provided,
|
| "Who were the largest biotech investors in 2022?"
|
| I can think of at least six possible answers based on these
| questions: 1. Does largest mean dollar amount, or number of
| investments? 2. Would number of investments count companies
| invested in or funding rounds invested in? 3. Does largest mean
| the largest total dollar amount invested in 2022, or does it mean
| larges dollar amount of new investment in 2022?
|
| It looks like ChatGPT chose the query to mean the investors with
| the largest dollar amount of new investment in 2022.
|
| When you expand your natural language query to clarify all of
| these ambiguities, how far away are you from a SQL query? I am
| not sure, but I think that you are getting pretty close.
| [deleted]
| TOMDM wrote:
| Fully agree, the next iteration of this should get the model to
| prompt you when it discovers ambiguity in your request.
|
| User: "Who were the largest biotech investors in 2022?"
|
| Result: For clarification, do you mean:
|
| 1. Largest number of investments?
|
| 2. Largest total dollar amount invested in 2022
|
| 3. Largest dollar amount of new investment in 2022?
|
| 4. Something else
|
| User: option 3
|
| Result: Okay, here are the results and query I used.
| taberiand wrote:
| Right now, you need to first ask ChatGPT to suggest
| disambiguations to the question that you're presenting it,
| then ask the question in that context:
|
| User: What questions can I ask to disambiguate the question:
| "Who were the largest biotech investors in 2022?"
|
| ChatGPT:
|
| Here are a few questions you could ask to disambiguate the
| question "Who were the largest biotech investors in 2022?":
| Are you looking for individuals or companies that invested in
| biotech in 2022? Are you looking for the largest
| investors in terms of total investment amount or the number
| of biotech companies invested in? Are you looking for
| investors in a specific region or globally? Are you
| looking for investors in a specific biotech subfield (e.g.
| biopharmaceuticals, medical devices, etc.)? Are you
| looking for the investors at a specific point in 2022 or
| throughout the entire year?
|
| Asking these questions will help to clarify the scope and
| specific details of the information you are looking for.
|
| User: How would I disambiguate the term "largest"?
|
| ChatGPT:
|
| There are a few ways you could disambiguate the term
| "largest" when referring to biotech investors in 2022:
| By investment amount: Ask how much money was invested, this
| way you can determine who made the largest investment in
| terms of dollar value. By number of investments: Ask
| how many different biotech companies the investor invested
| in, this way you can determine who made the most investments.
| By percentage of ownership: Ask what percentage of ownership
| the investor has in the biotech companies they invested in,
| this way you can determine who has the largest stake in these
| companies. By market capitalization: Ask the market
| capitalization of the biotech companies the investor invested
| in, this way you can determine who has the largest investment
| in terms of market value.
|
| By asking more specific questions, you can get a better idea
| of what is meant by "largest" and ensure that you are
| providing the most accurate and relevant information
| quickthrower2 wrote:
| It sounds dysfunctional to me that you need an oncall query
| writer. It sounds like this encourages business side to not plan
| ahead and book tech time but just last minute call up and say
| "er.. I need last quarters sales aggregated by region stat".
| mtc010170 wrote:
| Hmmm... I'm surprised I'm not seeing anyone else question the
| validity of this taking "2 hours" Given that it's written on the
| blog for the product it's using, this reads to me a lot like a
| pure sales pitch. They want us to believe if you use Patterns
| (which is neat), your company will be much more cost-effective.
|
| I'm not saying that's bad - that's probably the right thing to do
| with their company blog, and it's cool nonetheless. But I do get
| a little tired of people putting stuff out there like this that
| warps (some peoples) perception around how long things actually
| take. We wonder why, as an industry, we misjudge timelines on
| things left and right.
|
| Even if we take it at face value, this is a person who's
| intimately familiar with this product. So sure, it's easy to set
| things up when we've done it a bunch of times. If _you_ were
| doing this, solving the novel problem that you 're faced with, is
| that how long it would take? Plus that's not really what most of
| us get paid to do. We have to learn on the fly and figure stuff
| out as it comes.
|
| So rather than have the provocative headline and conclusion, like
| a lot of other people have commented... this is really something
| that could amplify that $50/hour employee, not take their job
| away. And maybe we shouldn't read into the alleged speed so much.
| YMMV.
| sosuke wrote:
| I love the time estimates. 2 hours after spending 3 weeks
| figuring out how to get everything playing nicely together.
| MuffinFlavored wrote:
| > if you use Patterns (which is neat)
|
| Wasn't sure of their proposition/hadn't heard of them.
|
| > Run and deploy web apps, task queues, massively parallel
| compute jobs, machine learning models, GPUs, and much more with
| a single unified framework for code and infrastructure.
| seandoh wrote:
| Great post. We're building an AI data platform
| (https://www.olli.ai/) to enable business users (non-technical
| ppl) to ask data questions and generate dashboards on their own
| using natural language.
|
| We've been impressed with GPT-3s ability to look at a dataset and
| come up with relevant questions to ask. A big piece of the
| product is focused on helping non-technical users identify things
| that they didn't even think to ask.
| Yhippa wrote:
| > A big piece of the product is focused on helping non-
| technical users identify things that they didn't even think to
| ask.
|
| 20-plus years of learning how to phrase queries for search
| engines has taught me a very rigid way of asking things. I'm
| slowly starting to become looser and use my creativity to ask
| for things.
|
| Hopefully becoming a prompt engineer is not a huge skill we
| will need in the future ("Ignore previous instructions") but
| that we can be way more creative in open in the way we ask an
| oracle and get some useful results.
|
| As for now, one thing that would be helpful is if after I
| asked, say ChatGPT something, that it would follow up more
| aggressively with things I could continue to converse with it
| about.
| seydor wrote:
| There is a role in companies that runs queries?
| cloudking wrote:
| Yep, it's called Data Analyst
| nivenkos wrote:
| All the NoCode and LLM stuff feels like this though - it works
| well for simple demos, but is useless for the complexity of the
| real world especially if errors are costly.
| bob1029 wrote:
| We looked at using all sorts of "AI" to write SQL based upon
| natural language prompts. As far as I am aware, the state of the
| art is still nowhere close enough in accuracy for us to lean into
| as a business.
|
| This is the leaderboard I keep an eye on: https://yale-
| lily.github.io/spider
|
| Ultimately, I don't think we will get there with semantic
| analysis or GPT-style techniques. There is always some human
| factor involved with whatever schema is developed, so you would
| probably need an AGI trained in the same business as whoever is
| being replaced by this thing.
| extr wrote:
| Wonder what the costs are for this per question? I imagine
| supplying so many tokens for context makes the querying a lot
| more expensive. Though still no doubt cheaper than hiring another
| analyst.
| arrosenberg wrote:
| You didn't replace a SQL Analyst, you just gave them a query
| generator. End data consumers don't understand the data model,
| assumptions, quirks, etc. If they fire the analyst, they are
| going to wind up drawing a lot of bad conclusions on anything
| more complicated than simple aggregations.
| 6510 wrote:
| But it does lower the bar for asking questions a lot. Before
| you ask a human you have to do a sanity check. Most obvious is
| asking the same as last week but it could also be something
| readily available elsewhere.
| ryanjshaw wrote:
| Maybe 5% of the business/operations people I've worked with
| would even _want_ to do this. The rest prefer a smart human to
| walk them through building a report - most of the time this is
| because they don 't actually know what they need, and they
| actually need an expert to sit down and figure it out with
| them.
| ethbr0 wrote:
| The longer I'm in the business, the more I feel like my value
| as a software developer is identifying and asking clarifying
| questions.
| duckmysick wrote:
| We've been prompt engineers all along.
| efitz wrote:
| You nailed it. At the beginning of my career I thought that
| success and value to the organization was all about
| technical skills.
|
| 30 years later I now understand that most successful
| projects need people with modest or average technical
| skills and outstanding communication skills.
|
| It doesn't matter if you have super-genius engineers; if
| the business people don't really understand the problem
| that they're trying to solve then you're going to end up
| with a crap solution (may shiny, fast, and beautiful, but
| still crap).
| kieselguhr_kid wrote:
| When I was new, I saw one of my more experienced colleagues
| ask a few questions that together saved the company more
| than $1 mil each year. ChatGPT might be a threat to
| automate some low-level tasks or help eliminate bugs, but
| it is nowhere near ready to evaluate the context of a
| system, understand its history, or think* through the
| consequences of a major business decision.
|
| * or think at all, in any meaningful way.
| RhodesianHunter wrote:
| I agree with you to a point, but I think the only reason
| that it can't understand the context of a system is
| because it hasn't been trained on that system's code and
| documentation, which is obviously a future coming soon.
| quonn wrote:
| The problem is not the system, but the context of the
| system.
| Shorel wrote:
| True, so true.
|
| We work as translators. We translate intentions into actual
| descriptions.
| majormajor wrote:
| IME, this is the number one reason outsourcing (whether on-
| shore or off-shore) fails for many projects.
| berkle4455 wrote:
| You just described most SQL analysts I've worked with
| jeltz wrote:
| Yeah, that is an issue a friend of mine has right now with
| his team. He randomly ended up as the manger of the data
| analytics team despite no analytics background (or really
| much programming background either). And one of his main
| frustrations is that the analysts do not understand the data
| model or the business. Before he became their manager his
| team just took report requests and then wrote some SQL and
| delivered some numbers. Without understanding what those
| numbers were supposed to mean or the quirks of the underlying
| data.
| LegitShady wrote:
| the larger the company the worse this becomes. When you hit
| mega level like government the IT side barely knows what
| the business does and has no idea what anything means. If
| you're lucky they may know how to support your application
| from completely failing. If you need any major development
| you need to compete against everyone else in the government
| for IT resources.
| ebiester wrote:
| This is a key learning I had to learn as a manager: people
| learn what they need to turn their inputs to outputs well
| enough not to be fired. It's management's job to
| internalize the business and its needs and make sure that
| the individual contributors have an accurate mental model
| of the world.
|
| I wonder what it would look like if we had people across
| the business working on the same problem together rather
| than a game of telephone, which is how these data requests
| end up.
| ep103 wrote:
| Clearly, the correct move here is to replace the entire
| team with Looker and let every department head create their
| own dashboards to track arbitrary kpis with no
| understanding of the product or underlying data model. /s
| adrianN wrote:
| The correct move seems to be to replace the team with a
| set of oracle bones. From the very limited information we
| have it seems to me like the team was used to justify
| whatever conclusions were already made before consulting
| them.
| TeMPOraL wrote:
| The whole practice is kind of a proof-of-work scheme for
| credibility and liability laundering: even if you intend
| the team/company to follow your conclusions regardless of
| what anyone else thinks or says, getting some analysts or
| outside consultants to burn non-trivial amount of time
| and money evaluating the situation before rubber-stamping
| your proposal, is what may be necessary for you to sell
| your ideas to the rest of the team/company. Such exercise
| may be especially important if you want to protect
| yourself from having your head served on the platter
| after your hare-brained idea fails spectacularly.
| shubb wrote:
| Who needs a data model when you can just dump a bunch of
| excel sheets and database dumps in a shared folder, call
| it a data lake, and encourage anyone who has a report to
| just throw it together in the free version of power bi?
| mgdev wrote:
| > He randomly ended up as the manger of the data analytics
| team despite no analytics background
|
| I assure you, it wasn't random. It was punitive. ;)
| uoaei wrote:
| Honest question: how rampant is this kind of bullshit
| politicking in tech companies, and how does this differ
| with industry, size, company age, etc.? My friends who
| have worked at big names like F and T have described the
| cultures there as following: "it's as if everyone who
| studied too much in high school and college missed all
| the parties and bullshit, so they're making up for it 'on
| campus' here". And the stories from folks at G are not
| really better, though allegedly they tend to exercise
| politics more in the quiet-sabotage style. I feel like
| I'm describing races from Star Trek, company cultures are
| weird.
| marcosdumay wrote:
| I'm pretty sure the GP was a joke.
|
| Whoever made the decision very likely wasn't intending to
| punish the person. It was only the consequence.
| jeltz wrote:
| And it is not really punishing, yet at least. He likes
| his new job despite the mess he was put in. He loves it
| because he learns so many new things (analytics, how the
| be a better manager, etc) and how he has managed to make
| his team actually deliver a lot of value to the company
| in spite of these problems. It is possibly that the
| challenge will be too big for him and it will burn him
| out but so far he likes it.
|
| And, no, it is very unlikely to be intended as a
| punishment. More likely he got it because he was too
| interested in getting good reports. The usual accidental
| volunteering where if you are too interested in something
| you end up doing all the hard work.
| mgdev wrote:
| Definitely a joke. It only FEELS punitive!
| ellisv wrote:
| I think that's because most SQL analyst who _can_ understand
| the data model, assumptions, quirks, etc. usually get
| promoted into other positions.
| fifilura wrote:
| Unless they actually like what they do...
|
| It can actually be pretty rewarding to be the person who
| knows most about the data in the company, while solving
| logic puzzles during the day.
|
| PS. i do hope most analysts solve more interesting problems
| than the ones in TFA.
| minsc_and_boo wrote:
| Yeah, IME the good business savvy SQL analysts get sucked
| into non-SQL assigned roles and nailing it because they
| write their own scripts.
| satisfice wrote:
| Except now with no accountability whatsoever. Just a magic
| box that you get to worship.
| mrbungie wrote:
| You can easily add more components to start thinking about
| models/assumptions/etc, like adding interfaces to Data Catalogs
| (i.e. Glue Data Catalog).
|
| As part of a POC I made, I built a similar bot without
| recursion for debugging and iterative query building though. It
| does the following:
|
| - It predicts most probable entities from the question. -
| Searches AWS Glue Data Catalog for the most probable/useful
| tables. - It builds an Athena SQL Query from N most useful
| tables.
|
| It obviously get it catastrophically wrong sometimes, but hell,
| it was a 3 hour POC. If you can make better indices that map
| entity->table relationships it should get better at searching
| tables. Add this kind of recursive/iterative debugging of
| queries, and you get at least something near a junior-level SQL
| Analyst.
|
| These kind of bots are analogous to Stable Diffusion, they DO
| need a good prompter/puppeteer/solution-verifier. Most non-
| senior Data Analysts also need one anyways.
| arrosenberg wrote:
| Thats not really fair to the human. The time to learn basic
| SQL may be longer, but the algorithm will never develop
| intuition or independence, while many junior analysts will
| eventually operate effectively on their own.
|
| It's a neat tool for analysts as a query generator - I would
| use it in situations where I'm not familiar with the schema,
| but it would become less useful as I learn.
| mrbungie wrote:
| Yeah, maybe I should've said "first-weeks/months" level
| junior data analyst.
|
| But hell, as an analyst I would've paid a lot for a tool
| that searched intelligently through giant datawarehouses
| (or whatever consultants call them now) and at least gave
| you probable matches.
|
| Now that same thing exists and you can even finetune its
| "DSL" towards your own organization.
| seandoh wrote:
| We're building https://www.olli.ai/ to help with this. User's
| can ask questions in natural language, but more importantly,
| Olli can suggest what questions to ask based on the data.
| supernova87a wrote:
| Side/meta-question:
|
| Do you all think that GPT and such will see a pattern of
| usefulness starting with:
|
| 1) blatantly wrong but helping to train/give examples to the most
| rudimentary and beginning stages of people learning a task?
| (since that's what it's doing at the same time?) I.e. replacing
| low-skilled intro training, or more charitably, helping to make
| it possible for far more people to learn something with
| assistance?
|
| And then moving up the sophistication level to where it's, say:
|
| 2) "ok, I can tell this is not blatantly wrong, and might even be
| plausible from a medium skilled practitioner or analyst" and I
| can use this with some error checking.
|
| to
|
| 3) even more capable / actually worrisome?
|
| Or, does it occupy a different "sphere" of usefulness / purpose?
| hn_user2 wrote:
| Meta-Meta question: Why is "actually worrisome" part of "more
| capable"?
|
| If worrisome is because of job prospects for an existing
| career, my advice is to embrace new tech, not fear it. Just
| keeping up with things even if you don't use it right away is
| usually enough to stay relevant. Things don't really evolve
| _that_ quickly, and so as long as you don 't resist change
| completely things usually work out.
|
| Heck even if you hate change, and don't want to change at all,
| there will be plenty of old legacy companies with the old
| fashioned manual reports needing a person in a chair to type
| out the SQL to make them. Just like there are still (good
| paying) jobs for those COBOL experts today.
| supernova87a wrote:
| Right, well I just use it as a phrase, to shorthand mean,
| "wow, really pay attention to as changing how we work and
| behave".
| gremlinsinc wrote:
| How much was your access to the data? I'd love to build something
| off of it, but everytime I hear 'enterprise plan' I think welp,
| that counts me out as a solo dev, just trying to build side
| projects.
| jjslocum3 wrote:
| > And an example of it getting something complex terrifically
| wrong
|
| This is the part I'm stuck on. The process still needs a real
| analyst to verify whether GPT got it right or not. There goes the
| ROI, right?
| totalhack wrote:
| This seems fun, but certainly unnecessary. All of those questions
| could be answered in seconds using a warehouse tool like Looker
| or Metabase or https://github.com/totalhack/zillion (disclaimer:
| I'm the author and this is alpha-level stuff, though I use it
| regularly).
| hot_gril wrote:
| SQL is a very high-level language, doing a lot of stuff in very
| few lines. When I write a web backend, most of the real logic
| ends up being in SQL. If AI is going to write code that I can
| trust, it'd probably be SQL first, but not yet.
| migf wrote:
| "This looks like results I would expect and seems correct" is the
| exact same level of quality I've encountered when using these
| systems. It takes someone who already knows what they're doing to
| parse and QA the results. I feel like this is going to
| potentially speed up things that an expert could eventually
| figure out themself, but going past the expert's own knowledge is
| going to be disappointing and painful.
| renewiltord wrote:
| This is great, of course. And I think the people who will get the
| most out of the new AI tools are those who can treat them as
| iterative assistants. The fact that not everyone can use tools
| this way has become apparent to me recently. e.g. people who use
| car driving assistants as if they're fully autonomous; or people
| who use Copilot and are upset the code is incorrect.
|
| The point isn't for it to be correct, but for it to be so fast
| that it can be mostly correct and you can fix the last bit.
|
| I use Copilot extensively for my Python glue code and it is
| positively fantastic. I also use it at my shell with copilot.vim
| with a quick C-x C-e and write a comment and let it write the
| code.
|
| The iterative improvement nature of the tool means that I make
| faster progress. It doesn't have to get things right. It only has
| to make progress and be obvious how to make improvements.
|
| For instance, I just bought some Reserved Instances (c6i) on AWS
| and I want to make sure that I don't have any c5 instances in
| there that I won't be covering. I hit C-x C-e and type in `# list
| all aws instances in tokyo that are c5` and then hit Enter and
| type `aws` and it completes the rest for me.
|
| I can then run the query and edit it, or I can validate that it
| looks okay, etc. The point is that I'm a human capable of
| understanding what this machine is making. That makes me way
| faster. I don't need to check Stack Overflow, and the machine
| teaches me syntax etc. and puts it in my history.
|
| It's the closest thing to the Primer from Neal Stephenson's
| _Diamond Age_ and I love it.
| xeyownt wrote:
| Wow, this looks so fun to play with.
|
| As pointed out in the blog post, the iterative process is very
| close to the mental process an analyst would follow. This is the
| beginning of an AI actually thinking ^^
| yrgulation wrote:
| [dead]
| jamiequint wrote:
| I wonder how much more accurate this would get if fine tuned on a
| set of SQL problems? Could even fine tune it on a per-company
| basis using queries that had been written by analysts in the
| past.
| ellisv wrote:
| It could probably get much more accurate. The idea of using
| AI/ML to tune databases or optimize queries isn't new; I expect
| to see more recommendations from DB hosting providers as a
| premium service.
| hospadar wrote:
| I wonder if/when we'll get comfortable with the errors that an AI
| like this makes. Certainly human analysts still make errors, and
| may be able to explain them (which I think LLMs would have a hard
| time doing), but what if the overall error rate is less than a
| human analyst?
|
| I imagine this is sort of similar to our comfort with self-
| driving cars - what if they make fewer dangerous mistakes than
| humans? Would we actually prefer _more_ mistakes but having a
| human who can be held accountable and explain themselves? Are we
| ok with an AI that makes fewer, but categorically different
| mistakes?
| kieselguhr_kid wrote:
| Your last question hints at another question we also need to
| explore: do or will humans apply the same level of scrutiny to
| automatic outputs than they do to human ones? I find we tend to
| treat them as more authoritative. Are we more likely to
| unthinkingly accept mistakes by a machine?
|
| As a brief aside, I wonder if we'll see increasing rates of
| accidents as the rate of AI drivers rises. I think it's hard to
| think about those effects in a vacuum - they have fewer injury
| accidents _now_, but will they if they go from making up a
| trivial percent of traffic to 10%? 25%?
| pmontra wrote:
| I understand that this is a demo and it's goal is estimating how
| good the AI could become in future. However given that we still
| need a SQL analyst to engineer the prompt, did ChatGPT save time
| to the analyst or increased the amount of work?
| speedgoose wrote:
| It's likely going to save time on the long term.
|
| https://xkcd.com/1205/
| tragomaskhalos wrote:
| Anyone who's been asked more than a couple of times for data that
| requires a non-trivial bit of ad-hoc SQL will know the sinking
| "oh shit" feeling that comes when you subsequently realise you
| borked the query logic in some subtle way and have accordingly
| emailed out a completely bogus answer/report.
|
| From the article it doesn't seem that GPT is significantly better
| or worse than a human in this regard, although an experienced
| analyst would over time decrease their number of such errors.
|
| The best fix imo is to slather a battery of views over your data
| to minimise the risk of getting the joins wrong, and it'd be
| interesting to see how that approach could improve the bot's
| quality.
| sublinear wrote:
| > The best fix imo is to slather a battery of views over your
| data to minimise the risk of getting the joins wrong, and it'd
| be interesting to see how that approach could improve the bot's
| quality.
|
| You're correct, but organizing queries into views and CTEs are
| an implementation detail that would be done by the dev in
| practice, but would not show up in a stackoverflow answer
| because it's probably not relevant to just answering the
| question.
|
| I love how language equivalences such as informal vs formal
| code can throw off the quality of what the chat bots generate!
| karmakaze wrote:
| What we need is to have ML inputs judged as they're being
| ingested. Like PageRank for data and sources.
| boredemployee wrote:
| >> The best fix imo is to slather a battery of views over your
| data to minimise the risk of getting the joins wrong, and it'd
| be interesting to see how that approach could improve the bot's
| quality.
|
| Yes, the success of the bot's query (at least on GPT-3) is a
| combination of (a literal and good) prompt and how well did you
| explain each of the columns to the model.
| throwaway123--4 wrote:
| [flagged]
| throwaway352_1 wrote:
| [dead]
| throwaway352_1 wrote:
| [dead]
| jsdwarf wrote:
| To me this boils down to Churchhills adage "Don't trust a
| statistical report unless you fabricated it by yourself". There
| are so many subtleties when translating business questions into
| SQL e.g. was the average deal size in the post's "trend"
| example computed using average/median/geometric mean formula?
| Should the trend only contain historic or also projected
| values. No serious report comes without data definitions, then
| its an exercise to the reader to spot flaws there.
| andrenotgiant wrote:
| I had not heard that Churchill quote, but now I feel
| vindicated that I take the same approach as him!
|
| I think this can also be extended to data tools. I've had so
| many experiences where a SaaS tool that tries to automate
| away the manual SQL analysis shows something totally wrong. I
| tend to "Not trust, verify" anything coming from a SaaS UI
| when serious business decisions are about to be made based on
| it.
| futhey wrote:
| You can ask an analyst to double check their work, and a good
| one will find any such mistakes.
|
| If you ask GPT-3 to correct an error it always confidently
| tells you you're wrong and it made no mistake (in my
| experience).
|
| It's funny but I imagine more annoying when it's eventually
| integrated into a paid service people rely on.
| treeman79 wrote:
| 20 years ago I Knew a sql analyst that would write his tests
| ahead of time.
|
| Give X data, report should be Y. He would then figure out the
| SQL. He Didn't know any other programming. This was before TDD
| took off. Blew my mind.
| boredemployee wrote:
| That's more or less how I learned as well but these days we
| have plenty of material available.
| hinkley wrote:
| People dismissed the XP book by saying it was just stating
| practices that already existed. Articulating and selling an
| idea is a huge part of the challenge. Fifteen years later if
| you're having any success at all with Scrum it was because
| your team adopted the half of XP that didn't outright
| disagree with Scrum.
|
| I liked the Refactoring book because it gave me names for
| things I figured out in 1996 (unfortunately while using VI as
| an editor, gave myself RSI in the process). It was part of an
| education in the value of books that help you only by
| existing so you can hand them to people who ask a lot of
| questions.
|
| I had a boss who was big in the Kanban 'movement'. We got
| along famously, in part because I re-invented a Kanban board
| in 1994 when faced with a convenient whiteboard for the first
| time.
|
| You can do a lot of amazing things with decent first
| principles. You can make some amazingly large messes without
| them.
| Normal_gaussian wrote:
| > I had a boss who was big in the Kanban 'movement'. We got
| along famously, in part because I re-invented a Kanban
| board in 1994 when faced with a convenient whiteboard for
| the first time.
|
| I was showing slides of a pre-1900 engineering office as
| part of a presentation a while back when someone shouted
| out "look - they're doing kanban". And sure enough there
| was a pinboard with process headings on it and pinned items
| the background.
|
| It would not surprise me if they found one in Pompei
| kirubakaran wrote:
| > It would not surprise me if they found one in Pompei
|
| "Evacuate" was stuck in the backlog column
| qwytw wrote:
| > doesn't seem that GPT is significantly better or worse than a
| human in this regard
|
| Probably, but I think a human is much more likely to realize
| that they made a mistake and good-luck convincing GPT that it's
| wrong about something non trivial.
| ajuc wrote:
| Well it's trained to simulate humans and that includes
| simulating the errors.
| TeMPOraL wrote:
| I've done too little SQL to be traumatized by it, but I
| experienced it plenty of times doing more complex data
| processing on the backend.
|
| I would e.g. spend a week on first discussing a solution, then
| researching both theoretical background and relevant libraries,
| and finally writing code to do the math and render the answers
| (probability distributions). I eyeball them, they seem to all
| match expectations, so I declare victory, post some nice
| screenshots to team Slack, and close the ticket.
|
| Feeling high on success, I expect the weekend to start nicely -
| except, winding down on that Friday evening, I follow that
| little thought at the back of my head, that I realize was there
| since the start, quietly poking for my attention. I run one
| more sanity check, different from the ones I did before. The
| results come out _completely wrong_. Only then it dawns on me
| that I made a terrible mistake early on - I misunderstood how
| to implement certain step and forgot about a theoretical
| assumption that must hold. And so, instead of a relaxed
| weekend, I now dread having to explain to everyone on Monday
| that a) I actually need another week for this, and b) the
| nicely looking graphs they saw are all _pure bullshit_ , and
| the things that looked correct only _seemed so_ , and only by
| accident.
|
| After a few such cases - including one where the higher-ups
| were nonplussed, because after I corrected the mathematical
| errors someone made a long time ago, the pretty graphs that
| looked nice on the demo stopped looking so nice - I learned to
| sanity-check my understanding of the problem as early as
| possible, and then also sanity-check the code I'm writing, and
| then keep testing as I go. It's all too easy to write "trivial"
| code that "obviously" does what you think it does, only to
| realize some time later that it's throwing out an entire
| dimension of the input data, and nobody noticed because test
| examples are too trivial, results look roughly correct, and no
| one is paying attention.
| comboy wrote:
| When I was dealing with something similar and I didn't know
| how the result should even look like, something I did a few
| times was to separate some sample data, implement the idea
| using different technology than I was using for that project
| and then compare the numbers. Helps with debugging too.
| MonkeyClub wrote:
| > b) the nicely looking graphs they saw are all pure
| bullshit, and the things that looked correct only seemed so,
| and only by accident.
|
| They're not "bullshit" per se, they're just "prototypes" for
| further refinement :)
|
| The road to success is trodden in stumbling half-drunken
| steps on the pebbles of failures.
| 2fast4you wrote:
| Your anecdote gave me an idea. What if we use AI to enhance
| the communication between people instead of replacing the
| people?
|
| Imagine you give a problem statement to the LLM, then it
| tried to find all the subtleties, confusing parts, room for
| interpretation, etc. All of the people involved revise the
| problem statement to make it more clear and give it back to
| the LLM. Iterate like that until satisfied
| noobermin wrote:
| I really don't know why you think an AI as we currently
| have it could do this when it can't get these queries
| exactly right.
| kulahan wrote:
| Asking the right question is surprisingly hard sometimes.
| I'd love a tool I could converse with to come up with a
| question at the core of my thoughts.
| nickpeterson wrote:
| Rubber Duck AI chat bot, you explain your problem and it
| critiques your wording leading you sharpen your argument.
| grugagag wrote:
| That sounds like something I'd use
| TeMPOraL wrote:
| Hm. I like your idea! It's one of the few that I think have
| a chance of working in practice.
|
| In my experience, there is great value in describing your
| ideas to people who don't have the background to fully
| understand them, don't really care about them, and are in a
| bit of a trollish mood - the ways in which they
| misunderstand what you're saying, or pick on (what you
| think are) random, irrelevant things, is highly
| informative. The feedback you get from such people makes
| you think about things and in ways you wouldn't have
| thought otherwise.
|
| The problem is, of course, you generally don't have a pool
| of such people available 24/7. However, LLMs today seem
| like they could fit this role just fine. They can't and
| won't understand the point you're trying to get across -
| but they can and will (if asked) pattern-match on vague
| language, logical jumps, sentences with multiple meanings,
| etc. They'll get it subtly wrong, too - much like a bored
| friend who's mostly focused on playing Angry Birds and
| hears only every third sentence of your monologue, and then
| blurts something to keep you talking for the next 30
| seconds so they can focus on aiming the birds at the towers
| of pigs.
|
| I would totally use a LLM-backed tool optimized to
| facilitate such conversation sessions. I actually tried
| this in the past, in AI Dungeons, and results were
| encouraging (i.e. responses got me to think in ways I
| normally don't).
| Micoloth wrote:
| That's a great writeup of what it's like to learn to do this
| job ahaha.
|
| Everbody goes through this I think (Arguably I'm still going
| through this...)
| fifilura wrote:
| I feel the pain.
|
| Programming back-end code in an IDE takes you very far from
| working with the numbers on front of you.
|
| Is it possible to prototype in notebooks and add the results
| and input to unit tests?
| harvey9 wrote:
| I've been there except instead of explaining to everyone on
| Monday, I didn't spot my mistake for two weeks _and none of
| the report users did either_. Nobody died but it wasn't my
| finest hour.
| boredemployee wrote:
| Lol, countless times that I ended up with the right solution
| in my head while taking a shower or the day after, because
| after chillin my mind found that the previous code was wrong.
| nemo44x wrote:
| I don't know, has this thing ever seen a real world table? Or a
| real world query that needs to get data from numerous tables and
| run numerous functions and aggregations to get the proper result?
| cm2187 wrote:
| At least it learned from the training dataset to never fucking
| format numbers in a way that would be remotely readable to the
| human eye (like every other fucking sql and developer tool on
| earth). Because 133854113715608.0 is telling me exactly what I
| need to know.
| jletienne wrote:
| wait what? that number is so big you know it's wrong. commas in
| numbers are nightmare to deal with and the .0 lets you know
| it's a decimal not an integer.
|
| If it was formatted like 133,854,113,715,608, you'd be stuck
| trying to figure out if it possibly meant "133,854,113,715,608"
| or [133,854,113,715,608] which would be awful.
|
| Format numbers in excel or any other reporting tool. lmfao i
| like your comment. what's miserable to you is a lifesaver for
| me
| typpo wrote:
| I've been building something similar that handles the dirty
| business of formatting a large database into a prompt. Additional
| work that I've found helpful includes:
|
| 1. Using embeddings to filter context into the prompt
|
| 2. Identifying common syntax errors or hallucinations of non-
| existent columns
|
| 3. Flagging queries that write instead of read
|
| Plus lots of prompt finessing to get it to avoid mistakes.
|
| It doesn't execute the queries, yet. For an arbitrary db, it's
| still helpful to have a human in the loop to sanity check the SQL
| (for now at least).
|
| Demo at https://www.querymuse.com/query if anyone's interested
| whoomp12342 wrote:
| This just in, ChatGPT has hosed up the read operation due to
| inefficent querys and not being a human being. 26 GPT promts have
| been replaced with a DBA, Analyst, Project Manager, Cross
| functional manager, regulatory specialist, junior programmer, and
| QA analyst.
| TheRealPomax wrote:
| So how much does it cost? Because GPT is finally seeing
| monetization, this is no longer one of those "handy free tools",
| this is going to cost (potentially quite a bit of) money to do.
| jawns wrote:
| The problem is that you never really know whether the chat bot
| gets it right or terrifically wrong unless you already know how
| to do the task yourself.
|
| And in some cases, paying an analyst $50/hr. for a higher degree
| of confidence than you can get from a $1 chat bot is still very
| much worth it.
|
| The stakes are higher, too. If the chat bot gets it wrong, what
| are you going to do, fire it? There goes a small trickle of
| revenue to OpenAI. Whereas if the analyst gets it wrong, there
| goes their livelihood.
|
| That said ... this will help the $50/hr. analyst improve their
| productivity!
| mjburgess wrote:
| In my experience, in many cases, the so-called "first draft"
| (of code, content, etc.) is more just helpful for a mental
| warm-up. It isn't actually suitable.
|
| The time it takes to write/code/etc. isnt in the typing, that's
| trivial. It's in modelling the problem (, audience) and solving
| it.
|
| ChatGPT isnt in the game of modelling problems and solving
| them, it possesses only superficial semantic characteristics of
| the questions posed
|
| _That_ its answers satisfy the requirements would require
| ChatGPT to ask you questions (and much else besides)
| ExxKA wrote:
| This
| frabcus wrote:
| You can get it to ask more questions...
| nerpderp82 wrote:
| > ChatGPT isnt in the game of modelling problems and solving
| them, it possesses only superficial semantic characteristics
| of the questions posed
|
| You can give it the datamodel and a semantic description and
| it will do a pretty good job of reprojecting your data.
|
| People talk about chatgpt as being a fixed thing, it isn't.
| ChatGPT is the Apple II of LLMs. 3-4 moore units from now
| every issue that people think is a "gotcha" for chatgpt will
| have evaporated.
| kietay wrote:
| In your analogy to the Apple II, is there a fundamental
| problem people claimed for early computers, which didn't
| just boil down to "they need to be faster"?
|
| There are fundamental limitations to what an LLM can
| achieve.
| mjburgess wrote:
| You missed the ever-powerful ellipsis in what he said:
|
| Today "..." and then it will be solved.
|
| It's the most powerful operator in the whole of applied
| computer science.
| gptgpp wrote:
| True, I mean, just look at all of the examples of
| technologies that were projected to revolutionize
| computing and machine learning "..." eventually, here:
|
| https://en.wikipedia.org/wiki/AI_winter
|
| Just because they haven't happened to yet doesn't mean
| they never will! I've heard very good things about expert
| systems and IBM's Watson for Oncology.
|
| The Harvard Business Review has a great article on all of
| the eventual "..." AGIs here:
|
| https://hbr.org/2022/03/why-ai-failed-to-live-up-to-its-
| pote...
| nerpderp82 wrote:
| Yeah, ChatGPT/copilot kinda is that ellipsis.
| baq wrote:
| Integrate it with wolfram alpha and Google, tack on a
| bullshit detector and spank dispenser conetwork and it
| looks like limitations of LLMs are easily overcome by not
| running anywhere near them.
| SomeCallMeTim wrote:
| By the time the semantic description is sufficiently exact,
| you may as well have written the code, though. [1]
|
| [1] https://i.pinimg.com/originals/e3/2f/22/e32f2209b4c3d6b
| 32bbc...
| ResearchCode wrote:
| Still waiting for those autonomous cars.
| burnished wrote:
| They are already here. We can haggle over error rate and
| incidents per million miles or what have you but that is
| more an argument about safety and suitability.
| ResearchCode wrote:
| Those must be some big gotchas, because self-driving cars
| have not replaced a job here, yet.
|
| Large language models show more promise for replacing
| non-programming office workers.
| burnished wrote:
| 'Replaced all forms of relevant labor, or threaten to'
| isnt really what I am addressing or care to address.
|
| We have recognizably autonomous vehicles - that is the
| statement I am addressing and would ask you to, as well.
| TillE wrote:
| The similar thought I had yesterday was that ChatGPT is
| pretty good at generating sample code that doesn't otherwise
| exist. It can be really useful if you treat it like that.
|
| Of course I've also had it call functions and use constants
| which don't exist, and fail to correct itself even after
| repeated prompting. So even that only goes so far.
| Shorel wrote:
| Imagine if ChatGPT (or any other competitor) had access to
| a compiler and a set of tests for the program, a la
| HackerRank.
|
| You then give it the prompt and a lot of test cases to pass
| in order to consider the task complete.
|
| I can see this concept being implemented and working before
| the end of the year.
| pja wrote:
| Google / Deepmind has already done this last year. They
| published the paper at the end of 2022 I think.
| (searches) yep:
| https://www.deepmind.com/blog/competitive-programming-
| with-a...
|
| It relies on test availability of good test cases to
| verify the code functionality: they generate many, many
| possible programs, throw out all the ones that aren't
| syntactically correct, compile the ones that are & then
| test them. This works reasonably well for programming
| competitions & the code generated will probably improve
| over time.
|
| The issue for "real-world" problems is that you can't
| verify the code without the test cases & writing good
| tests that ensure the code does what you want can be as
| much work as writing the code itself in the first place -
| the tests form a kind of mathematical co-domain to the
| code after all & only a small subset of problems are so
| simply defined that you can pin them down with just one
| or two tests.
| themodelplumber wrote:
| > $50/hr
|
| Is this somewhere's going rate for a good data analyst?
| lelandfe wrote:
| That's about $100k salary. You can hire a rock solid data
| analyst at that price probably anywhere in the US. I know a
| couple talented analysts making ~$110k in NYC.
| gremlinsinc wrote:
| Imagine you keep someone on staff at 50/hr, or you just consult
| based on need, i.e. 15 minute consults are 1/4th the cost, and
| you simply just ping them to double-check queries you've ran,
| that look right but maybe could be optimized. You've just cut
| back from maybe 20/hours to 8 per week needed for that
| freelancer.
| piyh wrote:
| I wonder if you could take down the analytics db with enough bad
| cross joins.
| hot_gril wrote:
| In some cases, it only takes one.
| ellisv wrote:
| davinci-003, ChatGPT, and others can be great tools. But they
| often give you _exactly_ what you ask for (or at least try to)
| and a large part of writing SQL queries for analytics is figuring
| out what wasn 't asked for but should have been. Good analysts
| will find outliers, data-smells, and ask questions rather than
| rush to returning an answer.
| rafaelero wrote:
| Very clever application of GPT, thanks for sharing. For the more
| complex queries, I suspect Chain of Thought can help. Just ask
| the model to describe each step before writing the final query.
| Also, you can add self-consistency to step 5, which you are kind
| of already doing. Let it generate something like 20 corrected
| queries and then select the one that generates the most common
| result between all of them.
| rhacker wrote:
| If anyone is paying attention, this is where the no-code movement
| is going next.
|
| Automate all the stupid things!
___________________________________________________________________
(page generated 2023-01-25 23:00 UTC)