[HN Gopher] To safely deploy generative AI in health care, model...
___________________________________________________________________
To safely deploy generative AI in health care, models must be open
source
Author : thecal
Score : 58 points
Date : 2023-11-30 19:38 UTC (3 hours ago)
(HTM) web link (www.nature.com)
(TXT) w3m dump (www.nature.com)
| verdverm wrote:
| It would seem to me, that the data is the more important part of
| the equation, and the health care providers, being the holders of
| this data, and also needing to find new revenue streams, want to
| profit from this.
|
| With federated learning and homomorphic encryption, can we
| satisfy both parties?
| TaylorAlexander wrote:
| Recently there has been a trend in calling models with weights
| and code available "open source" even if the training data is not
| available. For safe deployment in health care and other safety
| critical fields, transparency on the training data and process
| are vital too, which means developing clear terminology for
| models full transparency! Even this article title suffers from
| this ambiguity.
| HPsquared wrote:
| Yeah it's a pretty obvious misuse of the term. Training data is
| (part of) the "source"; weights are clearly "binaries".
| Training is "compiling".
| troupo wrote:
| That's why EU's upcoming AI regulation requires foundational
| models to have full documentation , including detailed
| descriptions of training data etc.
| Spivak wrote:
| I can't fathom why they didn't just require the models to
| make available the training data itself. Sure you might need
| to fork some cash so they can ship you hard drives but surely
| being audited by someone anyone is better than none.
| nradov wrote:
| Training data may be licensed from third parties which
| don't allow redistribution.
| huytersd wrote:
| If you give up your training data, you don't have a product
| anymore.
| pardoned_turkey wrote:
| How does open source improve safety if we simply don't have the
| analytical tools to intuitively reason about LLMs?
|
| You can't use this to prove that the model will always behave
| correctly (or desirably). At best, you can build test-suites to
| empirically check that it kinda-sorta appears to be doing the
| right thing most of the time. Which you can just as easily do
| with a black-box model.
|
| It's not that I'm against openness. I just don't see how you can
| posit that it gets us close enough to safety.
| Espressosaurus wrote:
| Step one is transparency--let's get the black boxes under our
| control open.
|
| It is not _sufficient_ but it is _necessary_.
| pardoned_turkey wrote:
| Right, but the article doesn't make that point. It is full of
| magical thinking that openness is the one hurdle we need to
| clear.
|
| I wouldn't feel any more comfortable getting diagnosed by an
| open-source LLM than I would be by a proprietary one made by
| OpenAI.
| notahacker wrote:
| Full openness in healthcare also comes with a cost: no training
| or fine-tuning on patient records or real world interactions
| which really really ought to not be fully open.
| dontreact wrote:
| To deploy generative AI in healthcare someone has to pay for the
| salaries of a lot of people to do the work. That means there
| needs to be a business model.
|
| I am not sure who will take an AI through regulatory procedures
| if it is open source and there is no way to make money from it.
|
| Open source is a useful tool for research yes. More of it would
| be nice.
|
| But I don't understand how or why anyone is going to go through
| all the hurdles of deploying technology if all of it is open
| source.
|
| Maybe an open source enthusiast can explain to me how that is
| supposed to work?
| lrvick wrote:
| Redhat, Element, Prusa, Adafruit, Sidero Labs, Arduino...
| plenty of companies that open source everything or almost
| everything and have have staying power.
|
| Many consumers, myself included, will -only- pay for technology
| if it is open source. In fact if something is proprietary I
| feel I am being cheated anyway and I might as well pirate it
| until I find something open to support.
|
| Many of us are willing to pay for time and labor and to support
| development for our personal projects and businesses so long as
| we have the power to change that relationship later and keep
| the tech if third party company later goes evil or goes under.
|
| If I do not have the source code, I do not own it. If I cannot
| own it, then why pay for it.
|
| Also everything becomes open source eventually. Companies can
| choose to accelerate this and earn community goodwill that
| might make them money selling open source turn-key services, or
| be replaced by that same community eventually doing it all
| themselves.
|
| No one pays for a license fee for the Linux kernel, but they
| pay their choice of cloud provider to host it. Choice. That is
| what I will pay for.
|
| https://staltz.com/time-till-open-source-alternative.html
|
| https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
| nradov wrote:
| Only certain classes of healthcare products require regulatory
| approval. For example, you could likely build and distribute an
| open source AI tool for summarizing patient charts, and the FDA
| probably wouldn't object (this is not legal advice).
| robwwilliams wrote:
| In progress---this is a natural for a joint effort by NIH, VA,
| NSF, DOE, DoD, companies, many universities across the globe.
|
| Not just biomedical research but all of science. The effort is
| being managed out of Argonne National Laboratory by Rick
| Stevens.
|
| https://www.anl.gov/article/new-international-consortium-for...
| KaiserPro wrote:
| I mean opensource is nice, but that's not actually going to make
| healthcare safer.
|
| Whatever flavour of AI needs to be deterministic, which llama, et
| al are not. even if you turn the temperature right down.
|
| As others have pointed out, its the training set that actually
| makes a model behave, hence why models are freely given away by
| large companies.
| Davidzheng wrote:
| Why must it be deterministic? Humans are not but we're still
| can have trust in humans.
| simonw wrote:
| If the training data for this is completely published, we'll
| have a huge head-start in terms of building a deterministic LLM
| once someone figures out how to do that.
| ribosometronome wrote:
| It seems like all of their criticisms can be easily applied to
| essentially any technology used or medical company relied on.
|
| For example: >In the rush to deploy off-the-shelf proprietary
| LLMs, however, health-care institutions and other organizations
| risk ceding the control of medicine to opaque corporate
| interests. Medical care could rapidly become dependent on LLMs
| that are difficult to evaluate, and that can be modified or even
| taken offline without notice should the service be deemed no
| longer profitable
|
| Even:
|
| >LLMs often generate ... convincing outputs that are false
|
| is already a problem the medical community has to address with
| existing tests.
|
| Or:
|
| >Another problem specific to proprietary LLMs is that companies'
| dependency on profits creates an inherent conflict of interest
| that could inject instability into the provision of medical care.
|
| Seemingly applies to essentially the entirety of medical supplies
| and medications.
| glitchc wrote:
| Just like an MRI machine is open source? I'm not sure if the
| authors have thought any of this through.
| kordlessagain wrote:
| As mentioned in another comment, the problem is that Open Source
| does not necessarily apply to all aspects of models. Open code
| allows everyone access to the "source" of an application. It does
| not mean the information that the code stores, when used, is also
| open to viewing.
|
| In models, the training data (dataset) is frequently "closed",
| where it is not open to viewing. That's just the default behavior
| of publishing models. You don't need the dataset to use the
| model. The weights or tensors may be "open" in that we can see
| them, but they are fairly "not worth viewing" if we don't know
| the nature of the relationships between the tensors.
|
| If we were able to figure out relationships between the tensors,
| and the dataset was not made open, then there might be a debate
| on whether or not certain use of that extracted or "transfer"
| knowledge is allowed.
|
| For a "model" to be fully "open", it must publish the data it was
| trained on, the code it used to train itself, and its tensors or
| weights must not be encrypted or disallow establishing
| relationships in the weights.
___________________________________________________________________
(page generated 2023-11-30 23:01 UTC)