[HN Gopher] Production AI systems are hard
___________________________________________________________________
Production AI systems are hard
Author : headalgorithm
Score : 224 points
Date : 2023-05-29 07:27 UTC (15 hours ago)
(HTM) web link (methexis.substack.com)
(TXT) w3m dump (methexis.substack.com)
| w10-1 wrote:
| Brain MRI's (in all their forms) are a whole different species
| than all other MRI's, due to variability and softness of
| structures, differences in condition presentation/time-frame, and
| and the high incidence of brain MRI's in the elderly (the brain
| can shrink precipitously after 60).
|
| Also, neurologists are quite specialized (e.g., stroke vs. MS vs.
| acute encephalopathy vs. optic neuritis). In the majority of
| cases, the specialist neurologist is better at reading the MRI of
| their target conditions than the general radiologist. This is
| handled by the neurologist phoning the radiologist and guiding
| them to re-work their findings.
|
| The vast majority of AI in software is in radiology. Decades ago
| radiology was among the first practice groups to fall under
| private equity patterns because of the cost of the machines. In
| some cases, specific radiology groups develop protectable
| expertise around some protocols (i.e., how to get the machine to
| give better data), but long machine lifecycle creates
| opportunities for technical one-upmanship.
|
| Knowing that the labeling quality is variable and the images are
| not entirely comparable, you realize the bulk of AI-radiology may
| be built on shifting sands, and amounts to workflow optimization.
|
| Worse, the effort could go into addressing what really plagues
| all providers: the unusability of their EHR's. But that's not
| particularly monetizable relative to critical diagnostics, and
| harder than production AI for MRI.
| jgalt212 wrote:
| The only general distribution production AI system (post ChatGPT)
| I'm aware of the that "works" decently is Grammarly Go.
|
| What others are out there?
| potatoman22 wrote:
| Google search
| xpe wrote:
| This conversation is unfolding nicely. Sorry for the
| sarcasm... I'll say what I mean: I think we can strive to
| treat even poorly phrased questions as an opportunity.
|
| Narrowly, yes, Google search seems to be behaving roughly as
| well (or poorly, depending on your point of view) nowadays.
| But I get the feeling saying "Google search" isn't a helpful
| response to the person asking the question.
| hewlett wrote:
| DeepL?
| xpe wrote:
| What do you mean by "work decently"? What kinds of behavior are
| you interested in? (i.e. What are you hoping to learn from this
| conversation?)
|
| If I were to guess, I'd probably think you are interested in
| production-level issues, rather than limitations of these "AI"
| technologies that have nothing to do with typing software
| scaling issues?
| wouldbecouldbe wrote:
| "Radiologists, because they have a grounded brain model, only
| need to see a single example of a rare and obscure condition to
| both remember it and identify it in the future."
|
| This would actually be a long term reason to go for AI / database
| diagnosis.
|
| I had a personal case where a close family member, a young child,
| almost died. The doctors didn't understand condition and last
| minute it suddenly calmed down.
|
| Im sure a few doctor's in the world seen it before, but they
| weren't working in my hospital that week. If we can start sharing
| in-depth diagnosis worldwide of obscure cases using AI to make it
| easy to query that would probably be of great benefit.
| srge wrote:
| Exactly: what if my radiologist doesn't know my particular rare
| condition?
|
| I would feel comfortable removing the human diagnostician.
| Let's have actual human doctors acting as researchers working
| to improve the AI diagnostics.
| haldujai wrote:
| My perspective as both a radiologist and CS/AI researcher so
| exactly what you're proposing:
|
| 1. We don't practice in imaginary vacuums, it's easy* to
| identify that something looks abnormal and then refer to
| clinical resources/other physicians for rare diseases with
| specific questions in mind (i.e. recognizing the nuanced
| imaging finding and referring to a resource to assist).
|
| As a rare disease example, tumors of the eyeball/orbit are
| very rare, but detecting them is not. If I open the case and
| see one I can refer to StatDx to help me narrow my
| differential knowing what imaging features I'm looking for.
| This reduces my misdiagnosis rate (which as an aside is ~2-5%
| for radiologists, "major" clinically significant that impact
| morbidity or mortality ~2-10% of those depending on the
| study).
|
| 2. Rare disease are hard to diagnose, and would likely also
| be hard for an AI. Imaging appearances are not unique to the
| vast majority of diseases, especially what we cal the "weird
| and wonderful".
|
| Pelvic tuberculosis, endometriosis, advanced cervical cancer
| and advanced rectal cancer can look identical/nearly
| identical on MRI and the clinical portion as well as
| additional testing helps us get to the diagnosis.
|
| We don't _have_ to diagnose everything based on a single
| imaging test, nor should we given:
|
| 3. Diagnosis is a tradeoff of sensitivity and specificity.
| You can't have both.
|
| Let's consider adrenal gland tumors. Statistically these are
| going to be benign, there is no specific imaging feature to
| tell a small adrenocortical carcinoma (ACC) ~1 in 1 million
| incidence from an adrenal adenoma (99% of adrenal lesions).
|
| We also can't tell them apart with a biopsy under a
| microscope.
|
| If you're unlucky enough to get an ACC you're basically shit
| out of luck as the only options we have are to recommend
| adrenal surgery (and their complications which can be death)
| to optimize sensitivity, or assume that it's benign and
| optimize for specificity considering disease prevalence and
| risk of overdiagnosis.
|
| In practice, we just use a cutoff of 4cm. I'm not sure how an
| AI would solve this, especially as there isn't a large enough
| training set. MD Anderson has the most experience of any
| center and they've had ~600 cases in 40+ years which as you
| can imagine encompasses a very heterogeneous imaging set (we
| didn't have multidetector CT or 3T abdominal MRI 20+ years
| ago).
|
| Overall, AI can and should _help_ radiologists and as someone
| involved in this field I can 't envision a world where we can
| safely remove the human diagnostician element from the mix,
| given that it's a spectrum of grey not black/white labelling
| as it is for object detection.
|
| We've had attempts with mammography and stroke AI and it's
| still horrendously inaccurate compared to what I expect out
| of a resident radiologist let alone an experience staff
| physician.
| visarga wrote:
| A great post. I can attest the same about information
| extraction from semi-structured documents. The situation is
| far from full autonomy. Can't do anything without human in
| the loop, not even with the latest AI.
|
| I am seeing this trend - everyone can explain at length why
| AI is "not quite up there" in their own field, but believes
| it's "near AGI" in other fields. We find it hard to imagine
| future difficulties AI will have to face in general, we can
| only do that in our own field where we have learned from
| direct experience.
| haldujai wrote:
| Exactly this! I chose radiology given my background
| thinking I could "easily build radiology AI systems" and
| help our struggling system.
|
| Then I became a radiologist and quickly discovered how
| hard this is. Something "as simple" as NER and entity-
| linking on radiology reports is damn near impossible at
| the moment (even with SOTA LLMs which have made it easier
| but still not accurate enough for production use).
| bick_nyers wrote:
| I previously worked for a Radiology PACS and it's hard to
| get funding/interest to even tackle the problem. With how
| lucrative it could be, I would think that a corporation
| would be very interested in putting resources into it,
| but this has not been my experience.
|
| No PACS that I know of even wants to tackle digital
| pathology in a significant way, which last I heard had
| about 5% adoption versus glass slides.
| chaxor wrote:
| NER, Entity linking, and relationship extraction
| definitely seem to be 'low hanging fruit' due to LLM
| improvements, but one of the big problems is that they
| really need a completely different architecture to limit
| the decoder vocab if using a decoder transformer for
| producing the set of sequences in relation extraction
| with specific entity ids. A Longformer with full
| attention to input sequence, and sliding window attention
| to a large dictionary could be a decent way to find tune
| a system like this, but there are few that try it.
| Unfortunately there's a lot of stupidity going around
| right now in thinking the answer is just to 'pRoOoMpT tHe
| LLm RiGhT', but that will always be exceedingly wasteful
| such that processing terabytes of files will be
| prohibitively expensive, and there's no guarantee the
| system will always restrict to the specific vocab and
| structure desired.
|
| The images in radiology definitely make these types of
| things harder, and the sparsity is an enormous issue.
| However, working with some projects in this area, I don't
| think it's as impenetrable as a lot of radiologists in AI
| suggest. The main thing needed in the field is adoption
| of better techniques and architectures to deal with these
| problems.
| jameshart wrote:
| > In practice, we just use a cutoff of 4cm.
|
| In other words, we extract a single feature, and apply a
| single nonlinear activation function to that feature to
| decide whether or not to activate the 'treat' signal. We've
| replaced all that vaunted human judgement and mental
| modeling of the body with a heuristic that has equivalent
| power to a single neuron neural network.
|
| I appreciate that there's a dearth of training data, and it
| varies in quality. But this is precisely the kind of thing
| where _sufficiently powerful_ ML could do better than the
| simple heuristics we can come up with.
|
| The heterogeneity of imaging types doesn't have to be a
| problem. Train the model on all the data, and all the
| different kinds of scan, all the anatomical knowledge.
|
| Look at how LLMs are able to do stuff like write code that
| has comments written in pirate speak. Do you think they
| learned how to do that by studying a large body of code
| with pirate-speak comments in? No. They picked up examples
| of pirate speak in one context, and code in another
| context, and they're able to combine them together in ways
| that make sense.
|
| ML models looking at diagnoses with small training sets
| that are largely in obsolete scan formats could still, in
| theory, learn how to spot those diagnoses in more modern
| scan images, because they have learnt from _other_ , much
| larger datasets how things in the newer format correlate
| with features in the old form.
| gonzo41 wrote:
| You need to pair a system like this with regulation and changes
| in the medical field to upskill nurses so they have a larger
| extended skill set and essentially have a path to becoming
| doctors via practical experience.
|
| There's a lot of information on medical conditions. I remember
| hearing about IPhone apps that can get a 90% hit rate on
| correct diagnosis for general practitioner style visits. Taking
| that into account, and a large scale medical database, you'd
| think we'd be pushing that technology down so we can get better
| productivity out of primary care, so most of the time Dr's were
| using their big brains for big complex problems.
| wouldbecouldbe wrote:
| 90% hit rate is pretty easy to accomplish for a standard
| doctor's visit, a big percentage of the doctor's visit are
| standard. You can probably train most humans to deal with
| that quickly. They often use students if the pressure is high
| here. It's about the other 10% + unforeseen issues.
| haldujai wrote:
| Speaking to radiology, ~90% of imaging studies are normal
| so I can sign my default template without looking at the
| images and be correct the vast majority of the time.
| taneq wrote:
| 90% hit rate could probably be achieved with "take some
| panadol and come back in a couple of days if you still feel
| bad."
| cdogl wrote:
| > I remember hearing about IPhone apps that can get a 90% hit
| rate on correct diagnosis for general practitioner style
| visits.
|
| Was that in marketing materials?
|
| > so most of the time Dr's were using their big brains for
| big complex problems.
|
| I have been misdiagnosed multiple times to my significant
| detriment. The issue each time was doctors failing to pay
| attention and exercise their faculties and knowledge, because
| they didn't care. These people didn't seem to have "big
| brains" and my subsequent experience taught me they weren't
| interested in responsibility or accountability either. I've
| heard similar stories from (for example) women who struggle
| with conditions like PMDD.
|
| I'm not on board with giving these people tools that they can
| use to justify paying even less attention.
| wouldbecouldbe wrote:
| I honestly don't think GP's need big brains, just slightly
| above average and a good memory.
|
| Skills required for being a good GP are probably empathy,
| communication, decent memory, reasonable analytical skills,
| not to big of an ego. Very different from a researcher.
| avereveard wrote:
| > have a path to becoming doctors via practical experience.
|
| Lol no
| srvmshr wrote:
| That's one of the biggest roadblocks: hospital & healthcare
| entities will not not have permissible mass sharing of data.
| When switching hospitals, its really hard to get a full copy of
| the record for single person. Try that for whole population. I
| may be wrong, but hospitals also could have incentives to
| tightly hold on to their proprietary data - where they spent a
| large sum of money. I worked in ML+Dermatology & getting data
| was a lot of regulatory check.
|
| Also, DICOM is the format for maximum interoperability from
| different radiology modalities & manufacturers. But annotation
| & markup methods vastly vary between institutions. There is no
| commonality or agreed upon standards on information
| interchange.
| mtkhaos wrote:
| What's nice is considering a global audience.
| weatherlite wrote:
| I think it will happen much faster in China, India or maybe
| Japan than in the U.S for that exact reason. It's quite
| possible the U.S will lag a lot in healthcare soon.
| samuell wrote:
| Federated learning (with implementations such as FEDn [1]) is
| supposed to solve this problem; Only sharing training
| weights, but not sharing data.
|
| Sure requires some coordination, but the legal parts should
| at least be solvable in this way.
|
| [1] https://github.com/scaleoutsystems/fedn
| wouldbecouldbe wrote:
| Im not sure it even has to be complicated and full on
| machine learning.
|
| Could also just make it easy to share diagnosis to the a
| database stripped from personal information about the
| client, properly index it, share it worldwide, allow
| doctors to contact eachother automatically who match.
| distant_hat wrote:
| If it is a rare condition, reidentifying people from even
| small amounts of relevant data (age, gender, hospital
| visited) data is trivial. If it is a common condition,
| usually there is plenty of data already there.
| renonce wrote:
| Is that really possible? I don't understand the field
| exactly but at least if you use deep learning, you'll be
| able to recover inputs from gradients, and if you want to
| program it such that no other party is able to recover your
| inputs, that's called cryptography and homomorphic
| encryption is FAR from practical yet (like a million times
| slower than practical). Without rigid mathematical
| foundations, I would doubt that it's just a fancy way of
| gathering all the training data and doing the training and
| making it perceive as if everyone retained their own data.
| haldujai wrote:
| I've been involved in a federated learning radiology
| project, the biggest issue is image technique and
| labelling.
|
| Different centers practice very differently, with different
| imaging protocols, disease prevalences, and
| labelling/reporting.
|
| This project was looking at renal masses and the only part
| that worked well with federated learning was image
| segmentation and probability that the mass is a cancer,
| this is a competency I expect out of a first or second year
| trainee.
|
| It was horrible for predicting subtype of cancer as we
| couldn't get a good training set (few of these lesions are
| biopsied, specific MRI sequences that may help are not done
| the same way in every center) which is what the goal was
| and more of an experienced generalist/subspecialty
| radiologist skill.
|
| Practically subtype doesn't make too much of a difference
| for the patient as if it's "probably a cancer" it'll just
| get cut out anyway, but highlights a challenge with
| federated learning.
| chaxor wrote:
| BERT-like encoder models actually do help here, although
| unfortunately the dense representation is not as nice and
| clean as a class, it will at least map to something
| similar. If the hospital provides even a completely
| bizarre entity id, the requirement would be to provide
| the dictionary of descriptions and train the BERT-like
| model to associate the entity ids and descriptions.
| dustingetz wrote:
| this may be a sufficiently powerful macro force to disrupt
| that
| wouldbecouldbe wrote:
| Yeah you have to make it non-personal, as well as deal with
| the commercial side of it. I've seen a few initiatives, but
| they all seem to lack the pockets, drive and ambition of an
| arrogant Silicon startup. Even though it's annoying at times,
| it might be what can do the trick in this case.
| sampo wrote:
| > That's one of the biggest roadblocks: hospital & healthcare
| entities will not not have permissible mass sharing of data.
|
| Maybe in some other country?
| srvmshr wrote:
| I am not certain how it works in all of Europe, but it
| doesn't seem straightforward in Sweden or UK.
|
| In US and Japan, they will rather do the full roster of
| tests than transfer in radiology results from another
| hospital. Its not something to do with costs, but physician
| opinions. Every specialist has his personalized view of the
| patient in hand - so the corpus of information he needs is
| different.
| brabel wrote:
| As far as I know, in Sweden data about a patient is shared
| between every hospital... you need to sign up on a web
| portal, of course, but from then on, you can authorize a
| doctor or nurse anywhere to get your data with a click...
| you can go to any clinic and they have access to your full
| history. It's impossible to do anything properly if you
| don't have that. I just don't understand why in the US,
| it's acceptable for hospitals to be silo-ed like that to
| the detriment of everyone's health.
|
| Here's a news report from 2017, when the system was being
| set up: https://www.bmj.com/content/357/bmj.j2069
|
| Today, this has been working for years.
| hattmall wrote:
| The US has multiple individual medical systems larger
| than Sweden with essentially the same abilities. Sweden
| has roughly the same number of hospitals as South
| Carolina.
| brabel wrote:
| What makes you think the Swedish system wouldn't scale to
| countries with 100's of millions of people??
| agrippanux wrote:
| Serious question - could it? How would that be
| accomplished?
| guillemsola wrote:
| Aren't these complex as any other piece of software that is
| looking to provide value to users?
|
| I've been developing software solutions for several years and the
| fact that a computer can do cool stuff, automate processes, be
| more accurate... always need users who know and want to use it.
| So my takeway is that with AI solutions the human part also needs
| to be considered.
| ghm2180 wrote:
| When you ask interview questions like: How would you design the
| ML stack that can recommend a restaurant given their past
| restaurant going history? Most candidates don't get into the
| craziness of things like bias in datasets or calibration of
| probability output of models.
|
| They dive straight into embeddings(they have the freedom in the
| interview to design something the are comfortable with), which is
| important to the quality but not necessarily the most(or even the
| only) challenging part of what the team works on.
| siddiqi123 wrote:
| So on appoint! Even the god father of AI (Geoffrey Hinton) is
| wrong about this. Yann LeCun, who is his student way back also
| isn't afraid to point that out as well.
| RamblingCTO wrote:
| Who tf started calling Hinton godfather of AI? It is not only
| inaccurate but unscientific. What about Minsky, Weizenbaum,
| Rosenblatt, Hopfield, Lenz, Hebb, Werbos, Turing, McCarthy and
| whoelse and whatnot. What differentiates Hinton from all the
| rest? The truth is that all of science is an amalgamation of
| past achievements, many so small and unheard of that it's just
| so very very wrong to pick one person and call them godfather.
| Why Hinton? Because DL had good marketing 10 years back?
| sgt101 wrote:
| >Minsky, Weizenbaum, Rosenblatt, Hopfield, Lenz, Hebb,
| Werbos, Turing, McCarthy and whoelse and whatnot
|
| Sounds like that Queens of the Stone Age song.. https://open.
| spotify.com/track/3DaXIGJm0BCEB9X7zHTRfI?si=98b...
| ftxbro wrote:
| > What about Minsky
|
| Minsky slowed down AI by using his powerful rizz to hypnotize
| otherwise reasonable researchers and academics into believing
| his self serving assertion that multilayer perceptron systems
| are bad because single layer perceptron systems are bad.
| levihaku wrote:
| Multilayer perceptrons literally are bad. Machine Learning
| industry is a complete joke because trivial stuff like this
| is all it has.
|
| It would honestly take less time for biologists to trap a
| brain in a jar and force it to do things than anyone in
| this joke of an industry will create real intelligent
| algorithm that can think for itself.
|
| Oh wait, I also forgot to mention that in our politically
| correct clown society, free thinking is illegal so AI by
| literal definition is illegal in the first place because it
| may just happen that it starts to be a little bit racist
| despite otherwise being amazingly intelligent and better
| than humans at solving certain tasks.
| ftxbro wrote:
| oh it's the 'LLMs are simultaneously ineffective and
| dangerously woke' argument
| crosen99 wrote:
| "Radiologists are not performing 2d pattern recognition - they
| have a 3d world model of the brain and its physical dynamics in
| their head. The motion and behavior of their brain to various
| traumas informs their prediction of hemorrhage determination."
|
| Radiologists are certainly performing 2d pattern recognition, as
| the input they are processing is only 2d even if the details of
| how that recognition is performed relies on some deeper
| understanding. Likewise, an AI system performs recognition of 2d
| patterns based on some deeper "understanding" - in the case of a
| neural net that "understanding" lies in the complex configuration
| and weightings of its neural connections built up from vast
| training datasets.
|
| Even if this dataset is only a subset of the dataset that humans
| are trained on, we still can't a priori claim that this subset
| lacks patterns and correlations that escape humans and that allow
| an AI to make certain determinations better than a human might.
| ugh123 wrote:
| I don't think speaking to Radiologists as "advisors" to an AI
| system is going to be productive in the grand scheme of things.
| Yes, radiologists (and all doctors for that matter) will give
| their thoughts on making their job incrementally easier.
|
| Is that going to move the needle on better healthcare? Will it
| allow our hospital systems to bring in more throughput of
| patients at higher quality of care and lower costs for patients?
|
| Thats not what doctors (and hospitals) want. They won't be an
| ally in "AI for healthcare" unless it means their
| paychecks/revenues are protected. The author is being misguided
| by seemingly smart people who have skin in the game and a lot to
| lose.
|
| Personally, I only favor AI solutions that will drive down the
| cost of all healthcare, raise the quality of service, and overall
| expand high-quality healthcare to underserved populations.
| Anything less than that is only serving the needs of the industry
| and not patients.
| aabajian wrote:
| I'm just finishing interventional radiology training and I
| moonlight as a diagnostic radiologist (not to mention having an
| undergrad/master's in computer science).
|
| Almost 90% of the diagnostic studies I read could be pre-drafted
| by AI. That's where the money is and where AI-in-radiology
| companies should focus. The money is _not_ in detecting
| hemorrhage or pulmonary embolism. It 's a classic fallacy to
| think that life-saving means money-saving. Rather, the money in
| radiology is reads per day.
|
| Here's a user story: A private practice radiologist reads 20
| abdomen and pelvis CT scans with contrast per day. In each of
| these studies, he must write a short description of each organ.
| For example, "Gall bladder: Unremarkable" or "Gallbladder:
| Cholecystitis without evidence of cholecystitis" or "Gallbladder:
| Dependent sludge." There are around 15 such organs (liver,
| gallbladder, pancreas, spleen, adrenal glands, kidneys, etc.).
| The AI should auto-populate the radiologist report with an
| appropriate description of each organ system.
|
| The job of the radiologist is to confirm what the AI says in each
| section, and to go into further detail as needed. It's
| essentially just customizing the existing template to each
| patient. This type of pre-drafting is exactly what radiology
| residents do and what companies like vRad do.
| jmhmd wrote:
| You are partially correct, but I think the value is not in
| generating a draft for the radiologist to review, but to take a
| radiologists "raw" dictation, which pertains only to relevant
| findings in a case, and transform that into a complete, well
| formatted report. This is what scribes do in lots of different
| medical specialties, and save a ton of time. In my experience,
| reviewing pre drafted reports, as from residents, can often
| make my job slower rather than faster, as you have to read the
| whole thing to know what needs editing.
|
| Also, at least in my practice, I would double the number of CTs
| read in an average day :)
| bee_rider wrote:
| Would it be possible to instead have a checkbox for the status
| of each organ, and have it default to whatever the average case
| is?
| [deleted]
| kurthr wrote:
| What percentage of the diagnostic radiologists won't bother to
| even review the AI output? 3%, 10%, 30%? It'll increase as the
| AI improves. Unless the image is sent to multiple radiologists
| or their metrics (eg spotting AI errors) are updated, we'll get
| crap results.
|
| I know one top tier diagnostic radiologist and he could tell
| which piece of equipment an image was taken on (or if something
| is out of calibration) with a fairly quick glance. Regularly on
| the phone with techs and technologists to get things fixed or
| reimaged. He retired recently.
|
| From his telling most of the work has been outsourced to India
| for the last 10-15 years. He only gets the really challenging
| stuff.
|
| edit: Interestingly, it seems like breast imaging is still done
| more in house and there are regular reviews, but that's from a
| different source.
| richardw wrote:
| He's retiring, which means his knowledge is mostly lost
| unless he teaches. What if he spent some of his time labeling
| images so AI learns and the machine says "recommend reimage"
| or "possible fault" for millions who don't have your doctor.
| brookst wrote:
| > What percentage of the diagnostic radiologists won't bother
| to even review the AI output?
|
| If someone's not going to do their job, why does AI matter at
| all? Why wouldn't they just use the same boilerplate text for
| everyone today?
| kurthr wrote:
| Because it's a lot easier to rationalize not doing your job
| when it is (or at least seems to be) being done OK. There's
| a lot of rationalizing of decisions, when you've spent your
| life saying you're the expert helping people. Also, a lot
| easier for management to hand out a LOT more work since it
| looks like it's being done right.
|
| It's like the lawyer using ChatGPT to file legal briefs...
| he'd feel bad and be obviously incompetent, if he didn't
| file anything at all, but didn't feel bad (or feel the need
| to check anything) once it "looked right".
| akiselev wrote:
| _> Because it 's a lot easier to rationalize not doing
| your job when it is (or at least seems to be) being done
| OK. There's a lot of rationalizing of decisions, when
| you've spent your life saying you're the expert helping
| people. Also, a lot easier for management to hand out a
| LOT more work since it looks like it's being done right._
|
| It doesn't matter what the radiologist rationalizes -
| they're going to get killed by malpractice lawsuits.
| Imaging is particularly prone since the radiologists'
| work product and a second opinion is all the evidence the
| plaintiff needs if something obvious was missed.
|
| _> It 's like the lawyer using ChatGPT to file legal
| briefs... he'd feel bad and be obviously incompetent, if
| he didn't file anything at all, but didn't feel bad (or
| feel the need to check anything) once it "looked right"._
|
| That lawyer is going to get sanctioned. It's an open
| question if he loses his license but he's not getting off
| scott free.
| gymbeaux wrote:
| We already have an issue in the US at least, of doctors
| waxing about how their favorite part of their job is the
| relationship with their patients, and how they take an
| "evidence-based approach" and take time to listen to their
| patients... only for them to do the polar opposite in
| reality.
|
| This is just going to be another case of patients needing to
| take ownership of their own care. It's no longer "tell the
| doc your symptoms and they'll figure out what's wrong", it's
| now "tell the doc what you think you have and if they
| interrupt you and brush it off, go to another doc" until
| someone agrees with your diagnosis and gets you the referral
| or test or prescription that you need. For those who know
| what I'm talking about, I'm sorry. For those who don't-
| you'll learn this lesson someday.
| BiteCode_dev wrote:
| You really have to double check everything yourself, which
| sucks if you are weaken.
|
| I was at the hospital last week for a pretty bad condition
| and my IV was badly set. I told the nurse to change it for
| 4 days, they kept refusing, trying to adjust it.
|
| Eventually I just told the doctor what was happening, and
| he said "oh, that's why your readings were messed up, I was
| wondering why I had to change your drug dose every day".
|
| They changed my IV in the next hours, and my conditions
| improved overnight.
|
| I discussed that with some doctor and nurse friends, and
| they all told me to never, ever trust a health practitioner
| you don't know very well. You need to be very proactive in
| your treatment, because they are exhausted, have many
| people to treat in a row and are only humans.
| jmhmd wrote:
| To be fair, this is not how doctors _want_ to practice
| medicine.
| oars wrote:
| Thank you for sharing your insight.
| throwaway85858 wrote:
| this! I asked a neurologist and she said at least 40% of her
| time is spent on formulaic discharge letters and rounds
| documentation.
| NeuroCoder wrote:
| MD PhD student here and this is 100% on the money and what many
| I work with try to do. The diagnostic classifiers seem to be
| about pushing papers and getting funding.
| la64710 wrote:
| So essentially the job of the radiologist is to carefully
| compare what the AI generated with the report to ensure that
| the generated text is correct. So in fact the radiologist now
| has to read both pieces of text (AI generated and the report
| itself) instead of just reading the report and typing her
| conclusions. Does it seem like less work or more work? I am not
| a radiologist but as a programmer trying to work with AI
| generated BS most of the time I find the work to be increased.
| We need AI that can reliably produce results (maybe through
| testing and internal feedback mechanisms).
| jacurtis wrote:
| A system like this will lead to an overall increase of mis-
| readings. Humans are incredibly forgiving and lack focus when
| confirming opinions, versus generating raw opinions.
|
| Imagine a Radiologist that already has to read 20 scans per
| day. With AI helping, and Radiologists just confirming, they
| will probably be expected to analyze 40 scans per day.
| Eventually the Radiologist will quickly get into a habit of
| confirm, confirm, confirm... 500 times in a day. They will stop
| treating each case as a real reading and instead they look at
| the AI's output and think, "looks about right" and will become
| complacent in actually providing valuable feedback.
| Furthermore, they will be expected to produce more work,
| meaning they are giving every case less attention than
| previous. They will begin to just trust the AI to keep up with
| their schedule.
|
| The better use of AI is to flip it on its head. Let the
| radiologists continue to work like normal. They will produce
| reports like they do now, only 20 per day. But the AI will
| analyze each scan after them, and if it flags the scan as
| remarkable or disagrees with the radiologist, then it bubbles
| the scan back to the top for further analysis. This way
| Radiologists are still expected to provide their expertise as
| they do currently and we generally rely on their opinion, but
| we will have an added layer provided by AI to catch anything
| that might be missed. This overall improves the accuracy of
| medical imaging instead of compromising it (which the original
| proposed AI solution certainly would do). Productivity would
| not be increased in this scenario, but accuracy (and indirectly
| malpractice instances) would improve.
|
| I for one, would not want to get scanned somewhere that relies
| on AI to analyze my scan. But i would love a place that used AI
| as a second pair of eyes to confirm a human's findings.
| abxytg wrote:
| I've seen some awesome demos of exactly this, ingesting priors
| and all of that.
| OthmaneHamzaoui wrote:
| On point ! Been working on putting AI/ML systems in production
| for various large companies in the last 5 years and every time
| the ML part of the system is just the tip of the iceberg. System
| integrations and user adoption are two main big components that
| had to be tackled before the AI system was in production.
|
| I think the natural excitement we have with any new AI model (or
| technology) leads us to assume that it will magically get
| integrated in any existing system and adopted by any user.
| senttoschool wrote:
| > _Should you be worried GPT5 is going to interact with social
| systems and destroy our society single-handedly? No absolutely
| not._
|
| I don't think most people are scared of GPT5. It's AGI that
| they're scared of.
|
| And GPT5 can destabilize a society because of how fast it could
| replace workers.
| ekianjo wrote:
| Yawn. The computers were supposed to put us out of work. Then
| it was the internet. Now its AI. And despite all that the
| number of people employed keeps increasing and find new ways to
| thrive with technology.
| XorNot wrote:
| It's the difference between "some human labor involved" and
| "zero". Getting to zero is _very_ hard, and has very
| different ramifications then the "some" quantity.
|
| As long as "some" labor is needed, then that's going to be
| the entire economy - it'll simply expand along whatever
| constraint that "some" represents.
|
| The bizarre thing lately is all these arguments people make
| implying the economy is maximized. That all the things that
| will ever be needed are currently being produced and no new
| growth is possible or will ever happen (despite this
| literally never being the case in the entire history of the
| human race).
| bberrry wrote:
| The industrial revolution replaced muscle power with
| machinery. AI is finally at the place where it can replace
| brain-power for many tasks, and will likely keep improving. I
| think you are underestimating the significance of this. I
| don't see too many horses with jobs these days.
| maaanu wrote:
| Currently the AI is not even able to improve my work... How
| should it replace me? I think the statement "AI is finally
| at the place where it can replace brain-power for many
| tasks" is ridiculous.
| maccard wrote:
| I don't think your statement is true - AI can definitely
| improve my work and has done for a while now. It lets me
| be more productive, but it certainly isn't capable of
| being left unattended or even lightly attended to.
|
| As long as you know the limits of what you're doing,
| tools like ChatGPT are very helpful. In about a minute, i
| was able to generate a skeleton app using a go framework
| including tests, the infrastructure as code to deploy as
| a lambda, a makefile build and run it, and a buildkite
| pipeline that will build and deploy it all. It's also
| correct.
|
| Now, I can't rely on it to do everything, but it can give
| me an app scaffold for a tech stack I'm familiar with,
| quicker than I can Google for it.
| maaanu wrote:
| That's good for you and I could give you some anecdotes
| with ChatGPT, were it failed horribly, e.g. were it
| failed to generate unit-tests for a simple api-call
| (about 10 loc)...
|
| I am sure the tools will only improve, but I don't
| understand all the hype/fear about it. ("We are fucked",
| "this is soo over", ... I think you will find more of
| those comments in older news-threads)
| maccard wrote:
| I agree - it's disastrous when it goes wrong. If you ask
| it to generate the code for a lambda with 32 CPU's it
| will happily spout out nonsense rather than tell you it's
| not a valid think to request.
|
| That said, ive found it remarkably good at spitting out
| slightly modified boilerplate - like IDE templates on
| steroids. It's been a great tool, like a hammer. But not
| everything is a nail.
| laratied wrote:
| Sam Altman is saying we are going to see a shift from labor
| to capital and that is going to cause more economic
| inequality along with possibly breaking the social
| contract.
|
| It is probably nothing though...As if this gopher server is
| going to replace my TV someday. Give me a break. It only
| does text!
| aatd86 wrote:
| I remember seeing a paper that explained that while the a
| business needed 7 employees on average to reach something
| like $1MM in revenue in 90s, it was now 3.
|
| So I think it still has the potential to replace some people.
| And even with AI and AGI eventually perhaps.
|
| This is a progression. Besides, employment is a bad measure
| in general if one does not compare it to others such as cost
| of living, poverty etc...
|
| Like non farm payroll doesn't say anything if everyone is
| employed living with the minimum paycheck.
| rolisz wrote:
| How much of that is because of inflation though? 1 million
| in 1995 corresponds to 2 million in 2023 dollars.
| joshuahedlund wrote:
| > I remember seeing a paper that explained that while the a
| business needed 7 employees on average to reach something
| like $1MM in revenue in 90s, it was now 3.
|
| An alternative interpretation: while in the 90's, 7 people
| could only support one $1MM business, today they can
| support at least two.
|
| (US unemployment is lower now than it was then.)
| dclowd9901 wrote:
| As a thought experiment:
|
| - computers are able to churn through any problem more
| quickly and cheaply than people
|
| - machines are able to perform any tasks that humans can,
| more quickly and cheaply
|
| I'm curious what you think is left for people to _do_.
|
| We may not be there today, but at some point, both of those
| things can and will become true.
| levihaku wrote:
| Fantasies never come true.
| NicoJuicy wrote:
| GPT is a tool in the toolbox. It can increase productivity, but
| won't replace a worker.
|
| Fyi, GPT 5 won't be coming soon and GPT 4 vs. 3.5 is not that
| different in terms of quality.
| whstl wrote:
| The main worry was never about it "replacing a worker". It's
| about one worker + GPT replacing two or three, because of the
| increased productivity gains you mentioned.
|
| I've seen it happening in some fields already. First with
| translation, now with copywriting and GPT-4. Hiring was
| frozen for those areas for a while in some companies I'm
| familiar with.
| sgt101 wrote:
| Also administrative assistants and secretarial support. 30
| years ago maybe 1/2 of the white collar workforce and now
| maybe 5% or less?
|
| Not AI - but IT.
| cookieperson wrote:
| I'm not scared of AGI beyond how it's likely lead to societal
| collapse because of human greed. I am worried about the
| operationalization of the current chat bot technologies and
| other generative AI modelling techniques for harm. You don't
| have to have a smart system to hurt hundreds of millions of
| people.
| xyzal wrote:
| If GPT5 is to put people out of work, I hope it will make
| unemployable the largest percentage people in the shortest
| amount of time. Only then there might be a chance we will
| change our way of measuring an individual's value by his or
| hers economic output.
| senttoschool wrote:
| White collar workers, the ones who went to college and
| studied, will be the first to be unemployed. I'm guessing
| that these people are more employable than the rest of the
| population since they dedicated themselves to studying in
| college versus the ones who mess around in life.
| bsenftner wrote:
| You miss the point: reality is far, far, immensely far more
| complex than you, I or anyone else realizes. Attempts to use AI
| to replace things, people and groups and fields, are fool's
| errands. They will be colossal failures. Just watch, you may be
| even a participant: reality is more complex than we realize,
| and our attempts to replace it with encompassing automation
| will fail.
| rco8786 wrote:
| > And GPT5 can destabilize a society because of how fast it
| could replace workers.
|
| How do you know this?
| ftxbro wrote:
| this sounds like the kind of cope that gets written before
| production ai starts going down the list replacing jobs
| tikkun wrote:
| > I interviewed and hired 25 radiologists, whose primary and
| chief complaint was that they had to reboot their computers
| several times a day..
|
| Yes. There is still so much low hanging fruit for software
| everywhere.
| wouldbecouldbe wrote:
| Thats more an IT management problem then a software issue.
| TeMPOraL wrote:
| IT management usually _is_ the problem.
| sgt101 wrote:
| It's often a software problem too.
|
| It's pretty easy to ship software with memory leaks -
| especially when there are few users, limited money for
| testing and technically challenging tasks - such as large
| images that need to be manipulated.
| bick_nyers wrote:
| The testing overhead for Dicom is immense to do properly.
|
| There is a very high chance it is a software problem,
| especially if dealing with tomos/mammos (or even x-ray)
| which are incredibly resource hungry.
| itissid wrote:
| Was that due to bad software crashing the os ?
| malikNF wrote:
| Smells like a memory leak.
| TheCleric wrote:
| My startup idea: rewriting medical apps in Rust.
| skybrian wrote:
| Another startup's idea: use AI to assist programmers in
| rewriting medical apps in Rust.
|
| I'd guess it's probably safer than using AI directly.
| bick_nyers wrote:
| DCMTK (C++) doesn't have any significant memory leaks
| that I am aware of, you can take a look there to get a
| grasp on the amount of work you would be signing up for.
|
| If you (or anyone) are serious about this, please for the
| love of God allow for multithreading (even if at the
| expense of efficiency), put some SIMD in there, or better
| yet utilize GPUs.
|
| So much of Dicom image handling is done single thread and
| it's so silly.
| m3kw9 wrote:
| To start, small things like getting the json format to work 100%
| of the time is impossible.
| xpe wrote:
| Me: "Are you speaking of JSON in the context of radiology
| systems? medical systems? enterprise systems? software in
| general?"
|
| You: Yes.
| Ozzie_osman wrote:
| > Geoffrey Miller was one of the loudest voices decrying the
| decline of radiology 5 years, and now he's crying fear for new AI
| systems.
|
| I don't know who Geoffrey Miller I'm pretty sure if there's a
| Geoffrey who notably predicted the decline of radiology, it was
| Geoffrey Hinton a few years ago...
| dang wrote:
| Looks like the OP has fixed that now.
| xpe wrote:
| I did a few minutes of research. I did not find any such
| person. So I'm inclined to agree; the author may have meant
| Geoffrey Hinton.
|
| See also: https://statmodeling.stat.columbia.edu/2021/06/07/ai-
| promise...
|
| > Gary Smith points us to [this news article][1]:
|
| > Geoffrey Hinton is a legendary computer scientist . . .
| Naturally, people paid attention when Hinton declared in 2016,
| "We should stop training radiologists now, it's just completely
| obvious within five years deep learning is going to do better
| than radiologists." The US Food and Drug Administration (FDA)
| approved the first AI algorithm for medical imaging that year
| and there are now more than 80 approved algorithms in the US
| and a similar number in Europe.
|
| [1] https://qz.com/2016153/ai-promised-to-revolutionize-
| radiolog...
|
| > Geoffrey Hinton is a legendary computer scientist. When
| Hinton, Yann LeCun, and Yoshua Bengio were given the 2018
| Turing Award, considered the Nobel prize of computing, they
| were described as the "Godfathers of artificial intelligence"
| and the "Godfathers of Deep Learning." Naturally, people paid
| attention when Hinton declared in 2016, "We should stop
| training radiologists now, it's just completely obvious within
| five years deep learning is going to do better than
| radiologists." The US Food and Drug Administration (FDA)
| approved the first AI algorithm for medical imaging that year
| and there are now more than 80 approved algorithms in the US
| and a similar number in Europe.
|
| > Yet, the number of radiologists working in the US has gone
| up, not down, increasing by about 7% between 2015 and 2019.
| Indeed, there is now a shortage of radiologists that is
| predicted to increase over the next decade.
| xpe wrote:
| Caveated [1] or not, the "concluding" paragraphs [2] are not a
| _summary_ of the article. Neither are they well-supported nor
| convincing; they are sweeping, unrelated generalizations.
|
| Notes:
|
| 1. The caveat at the top states "NOTE: This post is not up to my
| normal writing standard, just felt compelled to get this down in
| some form. This is more like a blog post entry than a real
| newsletter addition."
|
| 2. The article's final paragraphs are: "Long story short /
| Thinkers have a pattern where they are so divorced from
| implementation details that applications seem trivial, when in
| reality, the small details are exactly where value accrues. /
| Should you be worried about GPT5 being used to automate
| vulnerability detection on websites before they're patched?
| Maybe. / Should you be worried GPT5 is going to interact with
| social systems and destroy our society single-handedly? No
| absolutely not."
|
| P.S. Pedantic timestamp: the quotes above are taken on 12:24 pm
| eastern time, May 29. Well, this may _seem_ pedantic until the
| author updates the blog post and then this comment seems
| erroneous. It is notable that we still don 't have a well-
| accepted standard for snapshotted content.
| godelski wrote:
| At the root of this is something I'm constantly saying to other
| members in my research lab, arguing with reviewers, and
| passionately teaching my students:
|
| datasets are proxies, measurements are fuzzy.
|
| This is something that is drilled into students learning
| statistics and I can't for the life of me figure out why this
| changed with respect to ML (best guess is not requiring
| statistics courses).
|
| Datasets are proxies: they represent the world, but aren't. They
| should generally be seen as narrow subsets too. Your dataset
| quality and type matter a lot! Things like medical image datasets
| also have tons of correlating factors that can easily invalidate
| all results without you being aware. There are simple datasets we
| use to prove a concept (toys, mnist, cifar, etc). There are large
| scale datasets that have internal inconsistencies (imagenet,
| flowers). There are huge datasets that haven't been properly
| filtered/deduplicated (LAION). (there are also just shitty
| datasets (HumanEval)) Thinking of datasets as proxies helps
| internalize the frustration that literally every production
| engineer faces (even outside ML and software): real world results
| are inconsistent with lab results. Dataset engineering is an
| underappreciated art that is extremely difficult. But everyone
| needs to internalize that datasets are just a map, not the
| territory, and your navigation will only be as good as the map
| (many are poorly drawn maps, often on purpose)
|
| Measurements are fuzzy: Benchmarkism is running rampant in the ML
| world and it baffles me that a field who's bottomline objective
| deals with alignment isn't able to align how we evaluate
| ourselves. No measurement is perfect, many are far from them. You
| can train two language models to the same NLL and one might
| sample well and the other outputs garbage. You can train two
| image models to have identical FIDs and one samples clearly and
| the other doesn't. Likelihood also doesn't guarantee sharpness
| and I can go on. You must think about the limitations of your
| measurements and know them in depth. This also has seem to have
| gotten away from us and people just use the measurement libraries
| and call it that. We've reached a point where ImageNet
| classification accuracy has decoupled from downstream performance
| (object detection and segmentation), and things like this are
| confusing to production people because just taking the model with
| the highest score doesn't always result in the best performing
| model for their work (even before we consider things like
| throughput and memory usage). It is a Goodhart problem through
| and through.
|
| ML is at a serious point where we've gotten away from our basic
| stats learning. That's going to pose a real danger to society,
| not AGI. It's like handing powertools to chimps (who don't know
| how powertools work), it won't end well. But that is happening
| because we've shifted focus to meet targets, not to measure our
| work. Targets are easy, science is hard. Unless we bring these
| nuances back to our evaluation of works then we are just handing
| powertools to chimps without any quality assurance.
| tzhenghao wrote:
| I remember working with several ML teams on specifically
| inference performance (latency, memory usage etc.), and it's
| not a surprise to see some object detection performance
| variance depending on a scene.
|
| Sometimes, even capping the model architecture to bound us from
| exceeding performance thresholds is non-trivial in itself, but
| convincing "some" researchers why p99, inference latency for
| example, is more important than the p50 case in a safety
| critical system...that's surprisingly several magnitudes
| harder.
| godelski wrote:
| Most, if not all (can't think of one), research datasets have
| enough problems where once you reach a certain threshold your
| model will start to overfit despite no divergence between
| training and test data. But this is hard to explain to many
| as they only understand overfitting as divergence. But maybe
| worse, is that it is status quo to tune hyper-parameters on
| test data results[0] (instead of validation), which causes
| information leakage and helps support overfitting. But damned
| if you do, damned if you don't. And don't get me started on
| generative models where there aren't even test datasets[1].
|
| [0] I blame the status quo of lazy reviewers and rejecting
| works based on benchmarks as well as being uninformed.
|
| [1] I'll admit this is a sore spot right now as a reviewer
| took my paper's discussions about the limitations of FID and
| asked why I didn't present a new metric.
| mensetmanusman wrote:
| Healthcare is fascinating. The quote that nothing will be
| implemented that possibly harms patients is false, because harm
| is complicated.
|
| Sharing patient data harms privacy, but in the long run more data
| sharing is probably the best route for reducing physical harm.
| The body is just too complicated for individuals to fully grok.
| skybrian wrote:
| > Sharing patient data harms privacy
|
| Yes, that's a very abstract statement that seems plausible and
| is easy to make, but how do we know there's significant harm?
| What are some real-world examples? Can we quantify how much
| harm is done to patients from sharing too much, versus sharing
| too little?
| neon_electro wrote:
| Do you believe in a middle ground of informed consent such that
| data can be shared while mitigating the privacy harms? It
| allows each patient to make their own decisions, and just like
| volunteering to be an organ donor, there can be societal
| benefit without forcing harm on people who do not want to
| participate.
| levihaku wrote:
| I believe that if I cared more about privacy than my life, I
| wouldn't go to a public institution full of people I
| objectively cannot trust because I don't personally know them
| and then put my life in their hands.
|
| Privacy schizos need to stop making up nonsense. All of your
| diagnosed illnesses are permanently stored in a database that
| will never ever be deleted until decades after your death.
| This data is already shared between all hospitals. The moment
| you step through the hospital door you aren't at home and
| your privacy fantasies end. Literally any malicious worker
| could leak your data and then what? Why does this even
| matter. If I see you in a street, I can tell you're sickly
| just from the direction you're walking in.
| thegrimmest wrote:
| I don't. I think sharing data should be a mandatory part of
| receiving services. In order to benefit from the system you
| must commit to continue to improve it.
| amelius wrote:
| Even harder is using them when the vendor keeps pushing updates
| that may or may not work.
| outside1234 wrote:
| A customer I am working with can't even set up CI/CD with GitHub.
|
| I have no doubt that AI is not going to kill us because 95% of
| companies won't even be able to build it.
| jacurtis wrote:
| It is because, especially in tech, most people can't even
| vocalize what they want to the detail that an AI could generate
| it. Even in my day-job (which is probably similar to yours),
| most people give me the wrong information when asking for a
| solution and if I blindly built what they think they needed,
| they wouldn't get where they wanted.
| xpe wrote:
| > I have no doubt that AI is not going to kill us because 95%
| of companies won't even be able to build it.
|
| I take your joke, but the argument doesn't hold water for many
| reasons. Here are only two:
|
| 1. Even if only 5% of companies can build something dangerous,
| that's more than enough.
|
| 2. The "AI" doesn't have to directly kill us; we're capable of
| doing that ourselves with only a little bit of informational,
| societal, and/or economic degradation.
| xpe wrote:
| > A customer I am working with can't even set up CI/CD with
| GitHub.
|
| Can you please characterize your customer without calling them
| out by name?
|
| Also, can you clarify what you mean by "can't"? Under what time
| frame? With what background? With what else on their plate?
|
| In other words, why do you expect your customer to be able to
| setup CI/CD with GitHub? Maybe this seems reasonable to you,
| but maybe you are overlooking the hundreds of little things
| that make it seem easy to you, not least of which is the
| patience and ability to wade through tasks with many steps.
|
| I'm genuinely curious. Are you suggesting they lack capability
| (such as intelligence or skills), motivation, and/or something
| else?
| jameshart wrote:
| Wait a minute.... Because the role of radiologists is not as
| simple as trained classification, we can conclude that "No, AGI
| isn't going to take over every social system when GPT5 comes
| out"?
|
| Author starts off by saying that you can't generalize conclusions
| based on observations in a limited domain:
|
| > Geoff made a classic error that technologists often make, which
| is to observe a particular behavior (identifying some subset of
| radiology scans correctly) against some task (identifying
| hemorrhage on CT head scans correctly), and then to extrapolate
| based on that task alone.
|
| Author then goes on to suggest that there are a number of issues
| with radiology which AI will not solve.
|
| Then, based on _this_ lack of ML 's complete coverage of the
| aspects of radiology that he finds interesting, he then
| extrapolates:
|
| > Should you be worried GPT5 is going to interact with social
| systems and destroy our society single-handedly? No absolutely
| not.
|
| I don't see how the conclusion remotely follows from the
| argument. And more to the point, the refuting argument is
| _embodied in the article itself_.
| skybrian wrote:
| Yes, it goes a bit far, but the basic problem is treating this
| as all-or-nothing: Either the article is true or false. Either
| AI is going to take over, or it will have little effect.
|
| I believe the article raises relevant issues about why
| deploying AI-based systems is harder and will take longer than
| it might appear. Another relevant example might be driverless
| cars. It's taking many years and deployment is still quite
| limited. Some companies gave up. But I'm not going to count
| Waymo out yet.
|
| Similarly, it's possible that Hinton got the timeline wrong for
| radiology, but it might still have significant affects on
| radiologist employment in the end.
|
| (Note that this sort of reasoning by analogy is useful for
| imagining plausible scenarios, but not for ruling things out.)
| quickthrower2 wrote:
| This is more about production is hard as in "real life example,
| in a hospital" than production as in "running llama and serving
| yourself, vs. running llama and serving 10000 people at the same
| time", although I think serving lots of people something that
| uses a lot of compute, that people expect to be real-time is
| going to be hard too!
| varelse wrote:
| [dead]
| mercurialsolo wrote:
| The "feel" aspect in any domain is largely embeddings of a multi-
| modal information space. The challenge though with AI systems
| today is solving for multi-modality and providing access to
| datasets in closed domains. We are making strides in both.
|
| As AI's get exposed via API's and ever easier access we will see
| more of proliferation of this where jobs which we felt never
| could be AI'fied are more rapidly done so.
|
| Reminds me of the Lee Sedol vs Alpha Go game where famously it
| was said at the end "All but the very best Go players craft their
| style by imitating top players. AlphaGo seems to have totally
| original moves it creates itself.."
|
| I do echo though the production grade rollout of AI. When we want
| to jinx the game, we can notoriously play spoil sport -
| regulation, data protection et. al.
___________________________________________________________________
(page generated 2023-05-29 23:01 UTC)