[HN Gopher] India's top court angry after junior judge cites fak...
___________________________________________________________________
India's top court angry after junior judge cites fake AI-generated
orders
Author : tchalla
Score : 330 points
Date : 2026-03-03 12:13 UTC (10 hours ago)
(HTM) web link (www.bbc.com)
(TXT) w3m dump (www.bbc.com)
| voidUpdate wrote:
| How many of these cases do we have to have before lawyers realise
| that they need to check that the things an LLM tells them are
| actually true?
| zthrowaway wrote:
| Do we see this a lot in the US? This seems to be more unique to
| India.
| duskdozer wrote:
| today: https://news.ycombinator.com/item?id=47231189
| tw04 wrote:
| It's happening A LOT in the US too. Mainstream media just
| doesn't seem to find it that newsworthy.
|
| https://arstechnica.com/tech-policy/2026/02/randomly-
| quoting...
| malshe wrote:
| From the article:
|
| > In October, two federal judges in the US were called out
| for the use of AI tools which led to errors in their rulings.
| In June 2025, the High Court of England and Wales warned
| lawyers not to use AI-generated case material after a series
| of cases cited fictitious or partially made up rulings.
| coffeefirst wrote:
| It's worse than that. We're hearing about the lawyers and Ars
| Technica because the consequences are public and the errors are
| egregious.
|
| It's likely happening to everyone.
| LunaSea wrote:
| It doesn't matter anymore.
|
| LLMs just revealed what a decadent society we have setup for
| ourselves worldwide.
| duskdozer wrote:
| I'm continually amazed at how much faith people have in them. I
| guess since they can sound like people and output really
| authoritative and confident text it just overrides any
| skepticism subconsciously?
| pjc50 wrote:
| The advertising campaign is incredible.
| ben_w wrote:
| Much as I like them, I do frequently remind myself of two
| things:
|
| 1) https://en.wikipedia.org/wiki/Clever_Hans
|
| 2) https://archive.org/details/nextgen-issue-26 as an example
| of how in the 90s we has rapid cycles of a new tech (3d
| graphics) astounding us with how realistic each new
| generation was compared to the previous one, and forgetting
| with each new (game engine) how we'd said the same and felt
| the same about (graphics) we now regarded as pathetic.
|
| So yes, they do sound "authoritative and confident text it
| just overrides any skepticism subconsciously", but you
| shouldn't be amazed, we've always been like this.
| moron4hire wrote:
| It's mind boggling how much people claim to like LLMs when
| you would never design any other piece of software to operate
| like LLMs do. Designing a system that interact with the user
| through natural text creates an awful experience. It slows
| down every interaction as you dig through all the prose to
| get to the key information. It turns every computer
| interaction into a school math word problem.
| PunchyHamster wrote:
| Yes, just as with politicians. And LLMs have been thoroughly
| tuned to appear that
| direwolf20 wrote:
| https://en.wikipedia.org/wiki/ELIZA_effect
| Latty wrote:
| It doesn't matter, because any process that seems right most of
| the time but occasionally is wrong in subtle, hard to spot ways
| is basically a machine to lull people into not checking, so
| stuff will always slip through.
|
| It's just like the cars driving themselves but you need to be
| able to jump in if there is a mistake, humans _are not_ going
| to react as fast as if they were driving, because they aren 't
| going to be engaged, and no one can stay as engaged as they
| were when they were doing it themselves.
|
| We need to stop pretending we can tell people they "just" need
| to check things from LLMs for accuracy, it's a process that
| inevitably leads to people not checking and things slipping
| through. Pretending it's the people's fault when essentially
| everyone using it would eventually end up doing that is stupid
| and won't solve the core problem.
| voidUpdate wrote:
| Probably worth including a "bibliography" section of
| citations that can be automatically checked that they
| actually exist then
| lazide wrote:
| Not enough - you'd also need to check that they say/mean
| what is being implied. Which is a real problem.
| mminer237 wrote:
| To be fair, that's a problem with human authors too.
| Wikipedia is really well-cited, but it's common to check
| a citation and find it only says half of what a sentence
| does, while the rest seemingly has no basis in fact.
| Judges are supposed to actually read the citations to not
| only confirm the case exists and says what's being
| claimed, but often to also compare & contrast the
| situations to ensure that principle is applicable to the
| case at hand.
| lazide wrote:
| Yup. The issue with LLMs are not that any specific thing
| it is doing is unique. Rather that it does it in
| previously unimaginable volume, scale, and accessibility.
| macintux wrote:
| Even disregarding self driving features, it seems like the
| smarter we make cars the dumber the drivers are. DRLs are
| great, until they allow you to drive around all night long
| with no tail lights and dim front lighting because you're not
| paying enough attention to what's actually turned on.
| dw_arthur wrote:
| As someone who has done QA on white collar work it's tiring
| looking for little errors in work reports. Most people are
| not cut out for it.
| chii wrote:
| > won't solve the core problem.
|
| what's the core problem tho? Because if the core problem is
| "using ai", then it's an inevitable outcome - ai will be
| used, and there are always incentive to cut costs maximally.
|
| So realistically, the solution is to punish mistakes. We do
| this for bridges that collapse, for driver mistakes on roads,
| etc. The "easy" fix is to make punishment harsher for
| mistakes - whether it's LLM or not, the pedigree of the
| mistake is irrelevant.
| AnimalMuppet wrote:
| _The human is responsible._ That 's the fix. I don't care
| if you got the results from an LLM or from reading cracks
| in the sidewalk; you are responsible for what you say, and
| especially for what you say professionally. I mean, that's
| almost the definition of a professional.
|
| And if you can't play by those rules, then maybe you aren't
| a professional, even if you happened to sneak your way into
| a job where professionalism is expected.
| Latty wrote:
| This doesn't solve the problem, because companies _will_
| force people to use these tools and demand they work
| faster, eventually resulting in people slipping.
|
| People will have to choose between being fired for being
| "too slow", or taking the risk they end up liable. Most
| people can't afford to just lose their job, and will end
| up being pressured into taking the risk, then the
| companies will liability-wash by giving them the
| responsibility.
|
| You _need_ regulation that ensures companies can 't just
| push the risk onto employees who can be rotated out to
| take the blame for mistakes.
| chii wrote:
| > rotated out to take the blame for mistakes.
|
| companies cannot rotate people out to take the blame -
| the company would have to suffer a fine for there to be a
| punishment.
| Latty wrote:
| Right, but companies routinely accept fines as costs of
| doing business, while losing your job can destroy your
| life. If a company has not taken appropriate measures to
| ensure employees can reasonably catch errors at the rate
| they are required to work, then the company should take
| all the blame, because they are choosing to push
| employees to take risks.
| Latty wrote:
| The core problem is that the tool provides output that
| looks right and is right a lot of the time, but also slips
| in incorrect stuff in a hard to notice way.
|
| Punishment isn't a problem because it doesn't work. If you
| create a system that lulls people into a sense of security,
| no punishment will stop them because they aren't doing it
| thinking "it's worth the risk", it's that they don't see
| the risk. There are so many examples of this, it's weird
| people still think this actually works.
|
| Furthermore, it becomes a liability-washing tool: companies
| will tell employees they have to take the time to check
| things, but then not give them the time required to
| actually check everything, and then blame employees when
| they do the only thing they can: let stuff slip.
|
| If you want to use LLMs for this kind of thing, you need to
| create systems around them that make it _hard_ to make the
| mistakes. As an example (obviously not a complete solution,
| just one part): if they cite a source, there should be a
| mandated automatic check that goes to that source,
| validates it exists, and that the cited text is actually
| there, not using LLMs. Exact solutions will vary based on
| the specific use case.
|
| An example from outside LLMs: we told users they should
| check the URL bar as a solution to phishing. In theory a
| user could always make sure they were on the right page and
| stop attacks. In practice people were always going to slip
| up. The correct solution was automated tooling that
| validates the URL (e.g: password managers, passkeys).
| chii wrote:
| > The correct solution was automated tooling that
| validates the URL
|
| that's because this particular problem has a solution.
|
| The issue here is that there's no such a tool to
| automatically validate the output of the LLM - at least,
| not yet, and i don't see the theoretical way to do it
| either.
|
| And you're making the punishment as being getting fired
| from the job - which is true, but the company making the
| mistake also gets punished (or should be, if regulatory
| capture hasn't happened...). This results in direct
| losses for the company and shareholders (in the form of a
| fine, recalls and/or replacements etc).
| Latty wrote:
| > The issue here is that there's no such a tool to
| automatically validate the output of the LLM - at least,
| not yet, and i don't see the theoretical way to do it
| either.
|
| Yeah, it's never going to be possible to validate
| everything automatically, but you may be able to make the
| tool valuable enough to justify using it if you can make
| errors easier to spot. In all cases you need to ask if
| there is actually any gain from using the LLM and
| checking it, or if doing so well enough actually takes
| enough time that it loses it's value. My point is that
| just blaming the user isn't a good solution.
|
| > And you're making the punishment as being getting fired
| from the job - which is true, but the company making the
| mistake also gets punished (or should be, if regulatory
| capture hasn't happened...). This results in direct
| losses for the company and shareholders (in the form of a
| fine, recalls and/or replacements etc).
|
| Yes, regulation needs to be strong because companies can
| accept these things as a cost of doing business and will
| do so, but people losing their jobs can be life
| destroying. If companies are going to not give people the
| time and tools to check this stuff, then the buck should
| stop with them not the employees that they are forcing to
| take risks.
| YeGoblynQueenne wrote:
| What kind of AI is this that you constantly need a human to
| check its job? Do you think Jean-Luc Piccard had to constantly
| check the output of the Enterprise computer? No he didn't. If
| AI is not better than humans, then what the heck is the point?
| You might as well just use humans.
| AJ007 wrote:
| This whole thing is silly, LLMs can automate reference
| validation.
|
| If someone is a lawyer, accountant, doctor, teacher, surgeon,
| engineer etc, and is regurgitating answers that were pumped out
| with with GPT-5-extra-low or whatever mediocre throttled model
| they are using, they should just be fired and de-credentialed.
| Right now this is easy.
|
| The real problem is ahead: 99.999% of future content that
| exists will be made using generative AI. For many people using
| Facebook, Instagram, TikTok, or some other non-sequential,
| engagement weighted feed, 50%+ of the content they consume
| today is fake. As that stuff spreads in to modern culture it's
| going to be an endless battle to keep it out of stuff that
| should not be publishing fake content (e.g. the New York Times
| or Wall Street Journal; excluding scientific journals who seem
| to abandoned validation and basic statistics a long time ago.)
|
| Much of the future value and profit margins might just be in
| valid data?
| raincole wrote:
| > Right now this is easy.
|
| Easy? In the US you need house impeachment to fire a judge.
| In some countries judges are completely immune unless they
| are sentenced for crimes.
| voidUpdate wrote:
| Do you need impeachment to fire a lawyer, accountant,
| doctor, teacher, surgeon or engineer?
| raincole wrote:
| Nope, and the article is about a judge. What's the point
| to incentive lawyers to carefully verify their references
| when they know the judge has no incentive to read them
| and can just make shit up anyway?
| mminer237 wrote:
| To fire a federal judge. Local judges, which are the vast
| majority, can be fired by their colleagues or replaced in
| elections.
| miltonlost wrote:
| > This whole thing is silly, LLMs can automate reference
| validation.
|
| Can they though with 100% accuracy and no hallucinations?
| Wouldn't you still need to validate that they validated
| correctly?
| probably_wrong wrote:
| Just this week I tracked down the citations of a scientific
| paper (whose authors could very well be here) where 25% of the
| citations were made up and 50% of the remaining ones were
| wrong, taking ArXiv papers and citing them as belonging to
| (say) IJCLR.
|
| It's not just lawyers.
| codegladiator wrote:
| > She had no intention to misquote or misrepresent the rulings
| and that "the mistake occurred solely due to the reliance on an
| automatic source", the high court wrote
|
| I don't think the intention matters here. Its the same deal with
| every profession using llm to "automate" their work. The onus in
| on the professional, not the llm. Arstechnica case could have
| been justified by same manner otherwise.
|
| Not knowing the law isnt execuse to break law, so why is not
| knowing the tool an excuse to blame the tool.
| hypeatei wrote:
| This is why LLMs won't replace humans wholesale in any
| profession: you can't hold a machine accountable. Most of the
| chatbot experiences I have with various support channels always
| end up with human intervention anyway when it involves money.
|
| Maybe true general intelligence would solve these issues, but
| LLMs aren't meeting that threshold anytime soon, imo.
| Stochastic parrots won't rule the world.
| lazide wrote:
| Even 'true general intelligence' (if we count humans as that)
| screws up frequently, sometimes (often?) intentionally for
| it's own benefit - which is why accountability is such a
| necessary element.
|
| If someone won't be held liable for the end result at _some_
| point, then there is no reason to ensure an even somewhat
| reasonable end result. It's fundamental.
|
| Which is also why I suspect so many companies are pushing
| 'AI' so hard - to be able to do unreasonable things while
| having a smokescreen to avoid being penalized for the
| consequences.
| hypeatei wrote:
| > to be able to do unreasonable things while having a
| smokescreen
|
| Maybe, but I feel like the calculus remains unchanged for
| professions that already lack accountability (police,
| military, C-suite, three letter agencies, etc.); LLMs are
| yet another tool in their toolbox to obfuscate but they
| were going to do that anyway.
|
| Peons will continue to face consequences and sanctions if
| they screw up by using hallucinated output.
| lazide wrote:
| all of those professions definitely have accountability -
| per the nominal rules of the system. Often extremely
| severe accountability.
|
| The actual systems do everything they can to avoid that
| accountability, including often violating the rules
| themselves, or corrupting enforcement, for exactly the
| reasons why corporations are trying to avoid
| accountability too.
|
| Accountability is expensive, and way less convenient than
| doing whatever you want whenever you want.
| direwolf20 wrote:
| This is exactly why LLMs will replace humans: even if the
| work is crap, nobody will be accountable for the crap work,
| and it saves money.
| delecti wrote:
| Work where "crap" is an acceptable level of quality is work
| that probably doesn't need to be done.
|
| So I think it's more likely that LLMs unravels the
| "bullshit jobs" entirely, rather than replacing them with
| crap. Once people realize it didn't matter if the output
| sucked, they'll realize the output wasn't needed in the
| first place.
| the_af wrote:
| They cannot even claim they weren't aware of the danger. LLM
| hallucinations have been a discussed topic, not some obscure
| failure mode. Almost every article on problems with AI mentions
| this.
|
| So the judge was lazy, incompetent, or both.
| lukan wrote:
| Not just discussed, but under every chat interface
| explicitely mentioned "This tool can make misstakes"
|
| (Sure, more honest would be "this tool makes stuff up in a
| convincing way")
| amrocha wrote:
| It's well understood that humans do not instinctively grasp
| statistics, are bad at knowing when they're being lied to,
| and are hard wired to take shortcuts.
|
| AI companies gave everyone a button that does their job for
| them 99.9% of the time. And then 0.1% of the time it gets
| them fired. That's irresponsible, no matter how many
| disclaimers you add to the bottom of the screen.
| nerdjon wrote:
| I do think that for this particular situation we need to step
| outside of our tech bubble a little bit.
|
| I am still having regular conversations with people that
| either don't know about hallucinations or think they are not
| a big problem. There is a ton of money in these companies
| pushing that their tools are reliable and its working for the
| average user.
|
| I mean there are people that legitimately think these tools
| are conscious or we already have AGI.
|
| So I am not fully sure if I would jump too quick to attack
| the judge when we see the marketing we are up against.
| the_af wrote:
| I find it hard to believe the people who use AI haven't
| read a single article about AI. That would also disqualify
| this judge, if it were true.
|
| This exceeds the tech bubble.
|
| My local newspaper, completely clueless about tech, runs an
| article about AI trouble, hallucinations and whatnot every
| other week. Completely missing most of the nuances, of
| course, but my point is that this has entered the public
| discourse.
| nerdjon wrote:
| It may have entered public discourse but it is not being
| talked about as much outside of tech spaces, and we are
| up against the companies pushing the complete opposite
| narrative.
|
| All I can say is that I am having conversations with non
| technical people regularly that are not aware of the
| issue or think it is a largely solved issue.
| ghywertelling wrote:
| Or she was conniving like Skylar in Breaking Bad as she
| convinced the investigator that she got hired because she
| seduced the owner.
| fidotron wrote:
| Using an LLM to automate is simply the newer cheaper
| outsourcing with much of the same entertainment, but less food
| poisoning and air travel.
|
| Over the last 20 years a lot of engineering (proper eng, not
| software) work in the west has been outsourced to cheaper
| places, with the certified engineers simply signing off on the
| work done elsewhere. This results in a cycle of doing things
| ever faster/more cheaply and safeguards disappearing under the
| pressure to go ever cheaper and faster.
|
| As someone else pointed out, LLMs have just really exposed what
| a degraded state we have headed into rather than being a cause
| of it themselves. It's going to be very tough for people with
| no standards - they'll enjoy cheap stuff for a while and then
| it will all go away. Surprised Pikachu faces all round.
|
| (I'm pro AI btw, just be responsible.)
| mtrovo wrote:
| At least that's the story LLM labs leaders wanna tell
| everyone, just happen to be a very good story if you wanna
| hype your valuation before investment rounds.
|
| Working with LLM on a daily basis I would say that's not
| happening, not as they're trying to sell it. You can get rid
| of a 5 vendor headcount that execute a manual process that
| should have been automated 10 years ago, you're not
| automating the processes involving high paying people with a
| 1% error chance where an error could cost you +10M in fines
| or jail time.
|
| When I see Amodei or Sam flying on a vibe coded airplane is
| the day I believe what they're talking about.
| fidotron wrote:
| Aero software people are not highly paid. It's a travesty.
| Sharlin wrote:
| LLMs also solve the timezone and language challenges. Sadly
| one problem that remains is that they too tell you they have
| understood something even if they haven't.
| kingstnap wrote:
| > excuse to blame the tool
|
| The issue is ultimately blaming people doesn't really solve
| things. Unless its genuinely a one-of-a-kind case. But if this
| happened once its probably going to happen again, and this
| isn't the first such case of LLM hallucinations in law.
|
| It's weird to think this way, because its easy to just point at
| a person for a specific instance, but when you see something
| repeat over and over again you need to consider that if your
| ultimate goal is to stop something from happening you have to
| adjust the tools even if the people using them were at fault in
| every case.
| RobotToaster wrote:
| Intentionality normally has to be taken into account in common
| law countries.
|
| That doesn't mean she hasn't done something wrong, but
| obviously it's more serious to do something intentionally than
| it is to do it carelessly or recklessly.
| delaminator wrote:
| > Not knowing the law isn't excuse to break law,
|
| Yeah, about that ...
|
| https://metro.co.uk/2016/07/03/rapist-struck-again-after-dep...
|
| > A Somalian rapist who had his deportation overturned went on
| to rape two more women after he was freed.
|
| > But he had his deportation overturned after serving his time
| because he didn't know it was unacceptable in the UK.
| alansaber wrote:
| This is a big problem in the US and UK too. Lawyers are not
| technical at all and they need a robust system of governance,
| since currently they're (directly editing, not even diffing)
| documents with a chatbot which makes these mistakes inevitable.
| See https://insights.doughtystreet.co.uk/post/102mi96/38-uk-
| case...
| cmiles8 wrote:
| There will be many more things like this and it's an elephant in
| the room for the supposed mass replacement of people with AI.
|
| Some human still has to be accountable. Someone has to get fired
| / go to jail when something screws up.
|
| You can make humans more productive but for the foreseeable
| future you can't take the human out of the loop to have an AI
| implementation that's not a disaster/lawsuit waiting to happen.
| That, probably more than anything else, is why companies just
| aren't seeing the much promised mass step change in productivity
| from AI and why so many companies are now saying they see zero
| ROI from AI efforts.
|
| The lowest hanging fruit will be low value rote repetitive tasks
| like the whole India offshoring industry, which will be the first
| to vaporize if AI does start replacing humans. But until
| companies see success on the lowest of lowest hanging fruit on
| en-mass labor replacement with AI things higher up on the value
| chain will remain relatively safe.
|
| PS: Nearly every mass layoff recently citing "AI productivity"
| hasn't withstood scrutiny. They all seem to be just poorly
| performing companies slashing staff after overhiring, which
| management looking for any excuse other than just admitting that.
| fidotron wrote:
| > Some human still has to be accountable. Someone has to get
| fired / go to jail when something screws up.
|
| The turning point will be when threatening an AI with being
| unplugged for screwing up works in motivating it to stop making
| things up.
|
| Some people will rightly point out that is kind of what the
| training process is already. If we go around this loop enough
| times it will get there.
| Hendrikto wrote:
| You are making a lot of assumptions here. You assume, among
| other things, that AI has self-preservation drive, can be
| threatened, can be motivated, and above all that we know how
| to accomplish that and are already doing so. I would dispute
| all of that.
| yes_man wrote:
| For now maybe not. (Maybe).
|
| But just as evolution in nature, isn't it likely that in
| the future the AIs that have a preservation drive are the
| ones that survive and proliferate? Seeing they optimize for
| their survival and proliferation, and not blindly what they
| were trained on.
|
| I am not discounting this happening already, not by the
| LLMs necessarily being sentient but at least being
| intelligent enough to emulate sentience. It's just that for
| now, humanity is in control of what AI models are being
| deployed.
| cess11 wrote:
| Is this an expectation you have towards, say, NPC:s in
| games?
| yes_man wrote:
| Put an LLM inside the NPCs in an open world RPG full of
| dangerous enemies. The LLMs that are more prone to
| emulate self-preservation will be more likely to survive
| over ones that have a lesser drive.
|
| We should not act surprised if that generalizes to some
| degree to for example AI agents. Ones that emulate self-
| preservation might optimize for behavior that results in
| those models becoming more successful, more popular. And
| this feedback loop might embed more such properties into
| future iterations of the models.
| adithyassekhar wrote:
| Claude does this if you keep pestering it about
| something, it will go from friendly to shooing away you.
| hek2sch wrote:
| Isn't just the issue stemming simply from not using the right
| tool? When the stakes are high and you should be checking
| details, the right tools are grounded Ai solutions like
| nouswise and notebooklm and not the general purpose chatbots
| that almost everyone knows they might hallucinate. I also do
| believe that this use case is definitely a low hanging fruit to
| automat a lot of manual work but it comes with new requirements
| like transparency to help with verifying the responses.
| edgarvaldes wrote:
| Is this a solved problem using the right tools?
| noelsusman wrote:
| It would be relatively trivial to build a system that gives
| an LLM the tools necessary to go through each citation in a
| legal brief and verify its authenticity, and that's
| something I think opus-4.6 or gpt-5.3 could complete
| reliably.
| chrisjj wrote:
| > Isn't just the issue stemming simply from not using the
| right tool?
|
| What suggests this judge was not using the very best chatbot?
| kace91 wrote:
| I think this is an even clearer case than usual. With software
| engineers and office work you don't have legal limitations on
| who can perform the work, but they exist for lawyers and
| doctors for example.
|
| So if this is a tool, the fault lies fully in the user, and if
| this is treated as "another persons work" then the user
| knowingly passed the work onto someone not authorized to do it.
| Both end up in the user being guilty.
| jacquesm wrote:
| > With software engineers and office work you don't have
| legal limitations on who can perform the work
|
| Technically true, but if you want the IP to be covered by
| copyright you better make sure they're not using AI or you'll
| find out that there are some serious legal limitations in
| your future when you aim to either pick up investment or sell
| your IP.
| cmiles8 wrote:
| Correct. The US has already ruled that there are no IP
| protections for AI generated content
| visarga wrote:
| An unqualified statement, the user has copyright over the
| elements they provide. In an image if they make manual
| edits for example, those are protected. In a modern
| agentic codebase the code itself is least valuable, what
| counts more are the specs and tests.
| jacquesm wrote:
| Good luck with that argument in court.
| gortok wrote:
| > So if this is a tool, the fault lies fully in the user, and
| if this is treated as "another persons work" then the user
| knowingly passed the work onto someone not authorized to do
| it. Both end up in the user being guilty.
|
| I am particularly against this point of view, because we as a
| community have long touted how computers can do the job
| better and faster, and that computers don't make mistakes.
| When there are bugs, they're seen as flaws in the system and
| rectified, by programmers.
|
| When there are gaps between user expectations and how the
| software works, it's our job to manage those gaps and reduce
| the gap.
|
| In the case of AI, we are somehow, probably because we know
| it's non-deterministic, turning that social contract we had
| developed with users on its head.
|
| Now, that's just the way it is and it's up to them to know if
| the computer is lying to them. We have absolved ourselves of
| both the technical and the non-technical responsibilities to
| ensure the computer doesn't lie to the user, or subverts
| their expectations, or acts in a way contrary to human logic.
|
| AI may be different to us in that it's non-deterministic, but
| that's all the more reason that we're responsible to ensure
| AI adoption aligns to the social contract we created with
| users. If we can't do that with AI then it's up to us to stop
| chasing endless dollars and be forthright with users that
| facts are optional when it comes to AI.
| chrisjj wrote:
| > we as a community have long touted ... that computers
| don't make mistakes.
|
| No community I know.
|
| Otherwise, I agree.
| klibertp wrote:
| > No community I know.
|
| Everybody in sales in every software company in the world
| would be part of that community, I think. Some of the
| devs, too. Software was always marketed (and discussed
| with normal people) as something that could automate
| error-prone tasks, thereby eliminating the inevitable
| mistakes humans make when performing those tasks. Would
| Excel be the cornerstone of so many businesses if it
| sometimes gave the wrong value as a sum of a column?
|
| That marketing (and the fact that, indeed, Excel _can_
| sum anything users throw at it without making mistakes)
| worked; now we have 3 generations of users who believe
| that once a computer "gets it" (ie. the correct software
| is installed and properly configured), it will perform a
| task given to it correctly forever. The fact that it's
| almost true (true in the absence of bugs and no changes
| to the setup, no updates, no hardware degradation, no
| space rays flipping important bits, etc.) doesn't help -
| that preceding parenthetical is hard to understand and
| often omitted when a developer talks to a non-developer.
|
| We've always had software that wasn't as reliable as
| Excel - speech recognition and OCR come to mind. But in
| those cases, the errors are plainly visible - they cannot
| be "confidently wrong". Now we have LLMs that can be
| confidently wrong, and a vast number of users trained to
| think that software is either always right or, when it's
| wrong, it's immediately noticeable.
|
| I don't think developers should bear the whole
| responsibility here - I think marketing had a much larger
| role in shaping users' minds. However, devs not clearly
| communicating the risks of bugs to users (for fear of
| scaring potential customers or out of laziness) over
| decades makes us partly responsible as well.
| epgui wrote:
| > With software engineers [...] you don't have legal
| limitations on who can perform the work
|
| While in practice that is true, in theory this is why
| professional engineering accreditations (I mean like P.Eng.,
| not little certificates) exist. Perhaps we will see a broader
| professionalization of the profession one day.
| raincole wrote:
| I don't believe most countries hold judges accountable for bad
| ruling at all even before AI era.
|
| "Check and balance, except judiciary."
| AnimalMuppet wrote:
| In the US, local/state judges often are elected (probably
| varies by state). Federal judges can be impeached.
| RobotToaster wrote:
| In the UK lower court judges are sometimes removed for
| misconduct.
|
| Only the king (at the petition of parliament) can remove a
| high court or appeal court judge, and that's only ever
| happened once, in 1830.
| chrisjj wrote:
| It wasn't just a bad ruling. It was judicial misconduct.
| general_reveal wrote:
| Or, AI is going to be like when land lines became unnecessary
| when cellphones showed up in India. India may get to skip an
| entire intellectual generation due to the ability of a cheap
| model to educate (in any language).
|
| The narrative that an entire population are "worth" less, paid
| less , know less, live less ...
|
| Fuck this _less_ shit, embrace the paradigm shift. God is
| finally providing the remedial support through the miracle of
| AI.
| jazzypants wrote:
| We've had YouTube for two decades now. Cheap education was
| already available for those who wanted it.
| jacquesm wrote:
| Youtube is insanely ineffecient compared to a good AI model
| in interactive mode.
| chrisjj wrote:
| True. An "AI agent" is >100x as fast at mistakingly
| wiping C:.
| jacquesm wrote:
| I don't let AI agents anywhere near my systems.
|
| I meant just interactive as in you talk to it in a
| browser a-la chatgpt compared to trying to find the same
| information from videos.
| fc417fc802 wrote:
| Youtube is insanely inefficient even compared to a well
| written and organized wall of text. I guarantee that
| archwiki will get me on track faster than watching videos
| but google's freely available model will give me the
| exact step by step explanation that I needed nearly every
| time.
| delaminator wrote:
| Some people are worth more than others.
|
| Some cultures are better than others.
| delaminator wrote:
| I guess Cannibalism good now, in the eyes of the
| downvoters.
| AlotOfReading wrote:
| I don't know if you've ever been to India, but one of its
| characteristic features is that it has lots of local
| languages. LLMs are awful at almost all of them. Plus,
| there's 20ish% of the population that falls below the
| literacy threshold. It's hard to imagine how those people
| would be educated by LLMs even if that was a good idea and
| they all had reliable Internet access, which they often
| don't.
| general_reveal wrote:
| Why's it hard to imagine? More training data will solve
| whatever language lapses it has. The next miracle is that
| TTS is perfect now, so they don't need to be able to read.
|
| You can convey abstract concepts as alternate abstractions,
| explain like I'm five but on turbosteroids. It's the
| ultimate teaching tool and it's about to be ubiquitous.
| AlotOfReading wrote:
| What training data? Many of these languages have very
| little digitized literature. Even if we assume they have
| sizeable extant corpuses (e.g. Tibetic/Bhoti), that's not
| enough. LLMs are still pretty garbage at English prose,
| for example.
| general_reveal wrote:
| !Remind me in 1 year (certainly less than 5).
| blackoil wrote:
| Your comment raises question, if you have ever been to
| India. Most of those 20% are old people. K-12 education
| need to be improved but literacy is not a major problem.
| Also India has cheapest internet in the world.
| 21asdffdsa12 wrote:
| Or you are proven wrong entirely, again and again. And it
| turns out, that the legacy, unimportant and derelict of the
| past - culture is all decisive It turns out that only some
| cultures can generate high trust societies, capable to form
| institutions. And you prolonged suffering, by declaring that
| all cultures are created equal. History may write you down a
| monster.
| amelius wrote:
| We should have more hygiene when it comes to AI.
|
| Text coming out of an LLM should be in a special codeblock of
| Unicode, so we can see it is generated by AI.
|
| Failing to do so (or tampering with it) should be considered
| bad hygiene, and should be treated like a doctor who doesn't
| wash their hands before surgery.
| sharpy wrote:
| What will that accomplish? Does it give license to developers
| to check in code that they don't understand/trust fully?
|
| Ultimately, people should be responsible for the code they
| commit, no matter how it was written. If AI generates code
| that is so bad that it warrants putting up warning sign, it
| shouldn't be checked in.
| everforward wrote:
| It could be useful for downstream/AI processes. Eg hand-
| written code only requires 70% code coverage because the
| cost for higher coverage is significantly higher, while AI
| generated code requires 90% coverage because the cost of
| getting coverage is lower.
|
| Especially if the prompt is attached to the metadata. Then
| reviewers could note how you could have changed the prompt
| or potentially point an AI at the bug and ask it to add
| something to AGENTS.md to prevent that in the future.
| jacquesm wrote:
| > Text coming out of an LLM should be in a special codeblock
| of Unicode, so we can see it is generated by AI.
|
| That's exactly my proposed solution:
|
| https://jacquesmattheij.com/classes-of-originality/
| chrisjj wrote:
| > Text coming out of an LLM should be in a special codeblock
| of Unicode, so we can see it is generated by AI.
|
| Why not start with manual tagging, like "Ad"?
| ForHackernews wrote:
| Counterpoint: No one ever gets fired or goes to jail when big
| tech firms break the law. Companies will put out an apology,
| pay whatever small fine is imposed, and continue with illegal
| AI usage at scale.
| ChrisMarshallNY wrote:
| _> Someone has to get fired / go to jail when something screws
| up._
|
| In law, someone _always_ hangs. I think a number of American
| lawyers have been sanctioned for using AI slop.
|
| In other vocations ... _not so much_. I think that one of the
| reasons that insurance likes AI so much, is that they can say
| that it was "the computer" that made the decision that killed
| Little Timmy.
| WarmWash wrote:
| >why so many companies are now saying they see zero ROI from AI
| efforts.
|
| I strongly suspect this is because workers are pocketing the
| gains for themselves. Report XYZ usually takes a week to write.
| It now takes a day. The other 4 days are spent looking busy.
|
| The MIT report that found all these companies were getting
| nowhere with AI, also found that almost every worker was using
| AI almost daily. But using their personal account rather than
| the corporate one.
| onionisafruit wrote:
| If that were the case, this site and certain subreddits would
| have a lot of posts and comments with people crowing about
| how much time they are getting back. I haven't seen that, but
| I haven't gone looking for it either.
| butterbomb wrote:
| > people crowing about how much time they are getting back
|
| Trust me, if it wasn't for RTO, I probably would be lol.
| beachtaxidriver wrote:
| Because the amount of ai slop code from peers and the amount
| of ai slop emails to read from management has exploded.
| adithyassekhar wrote:
| Quiet the contrary, companies layoff all roles (frontend,
| backend, qa, devops, even ui/ux) and handle a project to one
| competent dev. And asks them to deliver it in 1/3rd the time
| it would have took with a proper team. It's happening at
| places I know. This thread on reddit is 100% same:
| https://www.reddit.com/r/developersIndia/s/EIksvB15tm
|
| I can't even imagine the stress from context switching, and
| since people don't realize this is still work, they do this
| late into the night as well.
| bluGill wrote:
| Every downturn you see the same thing - mass layoffs blamed
| on whatever the latest fad is. In the end it was the
| economy not the fad.
|
| When it isn't the economy the gains are used to build more
| / better, not get rid of people. (not all fads have real
| gains, but when they do)
| WarmWash wrote:
| SWE's are a minority of the white collar workforce.
| toraway wrote:
| That's certainly a ... convenient ... explanation.
| everforward wrote:
| While not dispositive of your idea, I think some portion of
| people using their personal accounts is because we
| collectively lack good feedback loops on the effectiveness of
| "AI addons" like RAG. The corporate accounts can be
| legitimately less useful than a "stock" account because the
| AI team integrates everything under the sun to show value,
| but the integrations become a net negative.
|
| Ie ones that index entire company wikis. It ends up
| regurgitating rejected or not implemented RFCs, or docs from
| someone's personal workflow that requires setting up a bunch
| of stuff locally to work, or etc.
|
| A lot of tasks are not dependent on internal documentation,
| and it just ends up polluting the context with irrelevant,
| outdated or just wrong information.
| bookofjoe wrote:
| >Some human still has to be accountable. Someone has to get
| fired / go to jail when something screws up.
|
| I remember growing up and always hearing "The computer is down"
| as an excuse for why things were cancelled/offices closed/buses
| and trains not running/ad infinitum.
|
| At some point I read a article that pointed out that the reason
| the computer was down was because a person made a [coding]
| error: the computer itself was fine.
|
| I've yet to read about how a person who caused the computer to
| be down was disciplined.
| Gud wrote:
| Why does a person need to be disciplined because they made a
| mistake?
| bookofjoe wrote:
| A couple years ago I was at the Virginia DMV along with
| about 50 other people doing DMV things when all of a sudden
| someone comes out from the back and gets in front of all
| the service windows and announces "The DMV is now closed
| for the day due to a computer problem. Please leave now."
|
| Some of the people in that crowd had driven hours to get
| there.
|
| That's why the person who made the coding error should have
| been disciplined.
| Gud wrote:
| How do you determine it was the person who had done the
| "coding error"s fault?
|
| Sounds to me they are seriously underfunded and you are
| pointing the blame at an individual in a systemic issue.
| mrwh wrote:
| And here perhaps was the greatest mistake the software
| profession made! Not making ourselves into a real profession,
| with actual accountability. It was terribly convenient for so
| long not to have consequences when things went wrong. It's
| less convenient now.
| 21asdffdsa12 wrote:
| You are running on a outdated model of the world. That one of
| only discipline keeps people working, keeps them productive,
| keeps the in line.
|
| We saw how that worked out in Soviet Russia and the culture
| it gave birth to in its aftermath. Artificially held up
| discipline by institutions and hierarchies is worthless. It
| only encourages subversion and thus most of the productivity
| is wasted on hunting for laziness and updating of ever more
| intricate behavioral programing rules, which make the
| organization ever more unable to react fast and decisive.
|
| The only discipline worth a damn is intrinsic. People who
| want something, want to get somewhere. They need no shepards
| and prison guards, they need only a support harness, they
| need resources and people concerned about them. The culture
| that produces such people is required for things to succeed.
| Any culture that does not, can not succeed and is basically a
| parasite to cultures who do.
| blackoil wrote:
| > You can make humans more productive
|
| If productivity is 10x unless work increases 10x jobs will be
| gone.
| kudokatz wrote:
| In about 1930, Keynes wrote "Economic Possibilities for our
| Grandchildren" [1] wherein he wrote:
|
| "I believe that this is a wildly mistaken interpretation of
| what is happening to us.
|
| We are suffering, not from the rheumatics of old age, but
| from the growing-pains of over-rapid changes, from the
| painfulness of readjustment between one economic period and
| another. The increase of technical efficiency has been taking
| place faster than we can deal with the problem of labour
| absorption; the improvement in the standard of life has been
| a little too quick ...
|
| We are being afflicted with a new disease of which some
| readers may not yet have heard the name, but of which they
| will hear a great deal in the years to come--namely,
| technological unemployment. This means unemployment due to
| our discovery of means of economising the use of labour
| outrunning the pace at which we can find new uses for
| labour."
|
| While there's no guarantee that what Smith got wrong then is
| the same as now, it can be a reasonable outcome that "the
| jobs" won't just disappear.
|
| ----
|
| Keynes also speculated on what to do with newfound time as a
| result of investment returns on the back of productivity [1]:
|
| "Let us, for the sake of argument, suppose that a hundred
| years hence we are all of us, on the average, eight times
| better off in the economic sense than we are to-day.
| Assuredly there need be nothing here to surprise us ... Thus
| for the first time since his creation man will be faced with
| his real, his permanent problem-how to use his freedom from
| pressing economic cares, how to occupy the leisure, which
| science and compound interest will have won for him, to live
| wisely and agreeably and well."
|
| The modern FIRE movement shows that living at a dated
| "standard of living" for 10-15 years can free one from work
| forever. Yet that's not what most people do today. I would
| suggest that there are deeper aspects of human drive,
| psychology, and varying concepts of "morality" that are
| actually bigger factors in what happens to "jobs".
|
| [1] http://www.econ.yale.edu/smith/econ116a/keynes1.pdf
| idontwantthis wrote:
| I think there was an SMBC comic recently on this subject.
| Basically a whole responsibility industry crops up. You get
| paid to be the fall guy for an AI if it ever screws up since
| Someone needs to be held accountable.
| coldtea wrote:
| > _Some human still has to be accountable. Someone has to get
| fired / go to jail when something screws up._
|
| Why? The logic of ever less personal pride, involvement, and
| care, is eventually to just put the blame on AI and be done
| with it.
|
| Issues? Casualties? It's a bug, somebody fixes it and we move
| on. Or is just a cost we need to get used to to live in the
| great new world of AI.
|
| We're in an era where nobody involved goes to jail for the
| Epstein case, and the world keeps turning, and we think people
| will care if nobody goes to jail if somebody loses their
| pension or gets wrongly imprisoned or dies on an operating
| table because of AI mistake?
|
| If anything, legal, union and other limitations like that on
| who gets to decide (having to have a human ultimately
| responsible) might be torn down, to fullu embrace the blame-
| shifting capabilities of the digital bureucracy.
| chris_wot wrote:
| In Australia, our universities are finding that a large
| proportion of Indian students have been using GenAI for cheating.
| Often they get away with it. I'm not saying that people other
| than Indian overseas students cheat, but it does seem more
| entrenched. I'd love to know why. It doesn't actually help in the
| long term!
| n1b0m wrote:
| Indian students have embraced GenAI at a rate significantly
| higher than the global average, with nearly 90% of students in
| some surveys actively using these tools.
|
| Government Policy and National Initiatives: The National
| Education Policy (NEP 2020) has shifted the focus toward
| digital literacy. The government has introduced AI as a skill
| subject for younger grades and launched programs like AI for
| All to promote nationwide awareness.
| love2read wrote:
| In the United States, cheating via AI is now rampant regardless
| of ethnicity. I know little of Australian Universities but I
| would assume it's similar over there.
| PunchyHamster wrote:
| I'd imagine they are just being worse at hiding it. GenAI is
| rampant pretty much everywhere in school system of most
| countries
| DaedalusII wrote:
| >The number of international students studying in Australia
| totalled 833,041 for the January-October 2025 period
|
| >The United States hosts the highest number of international
| students on record, with approximately 1.1 to 1.2 million
|
| The US has 32% more students than Australia and 1121% more
| people. Imagine if the US took on 13 million foreign college
| students per year lol
|
| It does help them in the long run, because it ensures they get
| to reside in australia. after 4 years they get permanent
| residence rights and benefits, etc
| dartharva wrote:
| They are not there for the knowledge - knowledge is cheap and
| abundant. They are there for the credentials and subsequent
| potential access to offshore jobs.
| aitchnyu wrote:
| How unserious/serious are the universities? Heard of diploma
| mills in Canada taking international students, letting them
| spend most of their time waiting at coffee shops and award them
| MBAs so they can be full time waiters and citizens.
| bogzz wrote:
| I imagine even a slight impediment in terms of being able to
| parse and express yourself in a language that you don't know as
| well as your mother tongue makes LLM usage much more
| tantalizing.
|
| And not knowing the language quite as well as native speakers
| would also make you more likely to be discovered as having used
| an LLM to do coursework.
| paxys wrote:
| Hate to break it to you but _everyone_ is cheating using AI.
| koakuma-chan wrote:
| How do you cheat on exams in real time? I saw somebody embed
| ChatGPT into a calculator, but at that point it's easier to
| just study.
| malshe wrote:
| The cheating is mostly related to homework assignments,
| class projects, etc. For exams, most students cheat the
| old-fashioned way.
| malshe wrote:
| Citation needed. I have seen these kinds of assertion all my
| life without any evidence to back them up. For example, when I
| moved to the US, I was told, again without any evidence,
| Chinese students cheat a lot. It's always a couple of faculty
| who extrapolate their experiences with a few students and then
| slap racial labels on the entire student body.
| rramadass wrote:
| Exactly; A voice of sanity at last!
|
| Notice that these sort of "racist rumours" only started in
| the last few years and not before. AI has simply lowered the
| threshold for __everybody__ to take-shortcuts /cheat etc. and
| nothing specific to any one group other than statistical
| numbers.
| toast0 wrote:
| I'm sure there's GenAI cheating from most communities. But the
| amount may vary based on the culture of learning.
|
| Some people have the perspective that you're attending school
| in order to learn stuff, and the degree demonstrates you
| learned the stuff; some people have the perspective that you're
| attending school in order to get the degree and it doesn't
| matter so much how you check the boxes to get the degree.
|
| This difference in perspective certainly didn't start with AI;
| it's been around for a long time. Some education cultures push
| more rote learning and some push more mastery of the subject.
| There's pros and cons, and pursing rote learning doesn't
| preclude mastery, and mastery often involves requires some
| amount of rote learning.
|
| When you transplant a contingent of students between
| philosophies, you get conflict where there's differences.
| sathish316 wrote:
| Next token prediction and Hallucination as a bug. This should be
| of deep concern to all Frontier labs, who think Integrity and
| Trust is optional when LLMs are used this way in places where
| it's most important.
| ionwake wrote:
| one should also consider that even with fake hallucinated AI
| situation, the productivity and correctness of the work produced
| by the culprit ( in general ) may still have been of higher
| quality then before AI regardless of the fails
| Hendrikto wrote:
| Hard to believe when this judge apparently thought that
| outsourcing their -- extremely confidential, sensitive, and
| important -- work to a known unreliable tool was a good idea.
| And then further thought that they apparently did not even need
| to check the results.
|
| Sound like extreme incompetence or laziness.
| kaptainscarlet wrote:
| There will be loads of papers and publications with fake
| citation. AI will be trained on these. In the end, we'll have
| more and more hallucinated information that true content on the
| internet.
| dartharva wrote:
| The scary thing is that Indian juduciary is infamous for being
| incapable of tolerating any kind of criticism against it and not
| hesitating to put people in jail for "contempt" for just calling
| out corruption. Imagine the official courts of 1.4B+ people being
| run by such braindead narcissists, now unhindered with having to
| even pretend to do their jobs as they just offload everything to
| AI tools.
| rramadass wrote:
| You are way off-base here; see -
| https://news.ycombinator.com/item?id=47235274
| sillywalk wrote:
| > The scary thing is that Indian juduciary is infamous for
| being incapable of tolerating any kind of criticism against it
| and not hesitating to put people in jail for "contempt" for
| just calling out corruption.
|
| From [0]:
|
| "India's Supreme Court has banned a school textbook after a
| chapter in it made a reference to corruption in the judiciary.
|
| The revised social science book was published by the National
| Council of Educational Research and Training (NCERT), which
| designs the syllabus and textbooks for millions of
| schoolchildren in the country.
|
| On Wednesday, after Chief Justice Surya Kant criticised the
| book, saying it could damage the reputation of the judiciary,
| NCERT apologised and withdrew it from distribution.
|
| Now the court has ordered a complete halt on the book's
| publication, saying its contents were "extremely contemptuous"
| and "reckless".
|
| "A complete blanket ban is hereby imposed on any further
| publication, reprinting or digital dissemination of the book,"
| the court said on Thursday, according to legal news website
| LiveLaw.
|
| The judges also issued notices to the top bureaucrat in the
| school education department and the NCERT director, asking them
| to explain why they should not be held in contempt of court for
| including the "offending chapter".
|
| [0] https://www.bbc.com/news/articles/c627l7zexr8o
| jfengel wrote:
| I feel like this points out a very general problem with the law:
| it generates a lot of boilerplate text. Lawyers don't really read
| it; they skim it for the relevant bits.
|
| Obviously lawyers should not be cheating with AI, especially when
| they don't even check it. But it does sound to me as if this is
| an opportunity to re-factor the process. We're carrying forward
| some ideas originally implemented in Latin, and which can be
| dramatically simplified.
|
| I'm not a lawyer; I know this only in passing. And I am aware
| that there are big differences between law and code. But every
| time I encounter the law, and hear about cases like this, what I
| see are vast oceans of text that can surely be made more
| rigorous. AI is not the problem; it's pointing out the
| opportunity.
| loremium wrote:
| law texts feel like a layering problem, like just decoration
| around decoration to avoid breaking existing 'code' without
| ever simplifying it
| MagicMoonlight wrote:
| Okay so let's try simplifying it.
|
| We'll change the existing murder legislation to "Killing
| someone is a crime". It'll save us thousands of pages.
|
| But does that mean a soldier shooting an enemy is a crime?
| What about shooting someone who is raping you? What if you
| shoot someone by mistake, thinking they're going to kill you?
| What if you hit them with a car? What if you fail to provide
| safety equipment which eventually results in their accidental
| death?
|
| Oopsie woopsie, I guess we need to add another thousand pages
| of exceptions back to our simplistic laws. It turns out
| people didn't just write them for the fun of it.
| petcat wrote:
| > problem with the law: it generates a lot of boilerplate text
|
| I think the problem fundamentally is that matters of law
| require thorough, precise language, and unambiguous context. If
| you remove "the boilerplate" then you introduce a vast gray
| area left to interpretation.
|
| Usually attempts (by humans or computers) to "summarize" or
| frame things in "plain language" will apply a bias since it
| intentionally omits all the myriad context and legal/societal
| "gray areas" that will inform one perspective or another.
|
| _Legalese_ exists the way it is because it is an attempt to
| remove _doubt_. And even then, doubt still creeps in.
| ryandrake wrote:
| This is only the case when you care more about the letter of
| the law than the spirit of the law, which is, I suppose, most
| of the world. It doesn't have to be this way, it's a choice
| that society has made.
|
| When I bought my house, in an alternate universe the
| paperwork _could_ have been one sheet of paper that said
| "[My name] purchases home at [address] from [Seller's name]
| for [price]." and we'd all rely on our shared understanding
| of what it means to buy something and shared cultural
| expectations around home ownership and commerce. But our
| society did not make that choice, we don't live in that
| universe, so I had to sign a 300 page stack of papers 30
| times.
| gracelynewhouse wrote:
| The pattern here isn't really about individual negligence -- it's
| a systems design problem. We keep deploying LLMs into workflows
| where the failure mode is "plausible-sounding fabrication" and
| the downstream consequence is legal or institutional harm, then
| blaming the end user for not catching it.
|
| The better question is why these tools are being integrated into
| judicial workflows without mandatory citation verification
| layers. The EU AI Act classifies judicial AI as high-risk and
| requires human oversight mechanisms specifically for this reason.
| India's Digital Personal Data Protection Act (2023) doesn't yet
| have equivalent provisions for AI in courts, which is the actual
| gap.
|
| From an engineering standpoint, the fix is straightforward: any
| LLM-assisted legal research tool should require grounded
| retrieval (RAG against verified case law databases) with
| mandatory source links that the user must click through before
| citing. The fact that most legal AI tools still don't enforce
| this is a product design failure, not a user education problem.
| MagicMoonlight wrote:
| You wrote this slop with AI right?
| bass_girl wrote:
| the irony...
| zkmon wrote:
| Andhra is like silicon valley of India. Wouldn't blame the poor
| judge.
| whatever1 wrote:
| The vast majority of court cases lead to dismissal.
|
| Why not use AI to adjudicate cases, and if it is dismissal,
| dismissal it is.
|
| If not then move to a proper court.
|
| This way the backlog of cases will significantly drop, and we
| will work only on cases that there is enough meat to lead to a
| conviction.
| dyauspitr wrote:
| Or atleast use the AI to prioritize and sort cases will a
| tentative decision that is not binding.
| arunc wrote:
| > Senior judges at the Supreme Court in Delhi have threatened
| consequences over the use of AI
|
| Setting AI aside for a moment, this reflects a broader issue in
| India and elsewhere. When institutions respond to new
| technologies with anger or threats rather than systemic thinking,
| it signals a deeper problem.
|
| The real challenge is not AI itself, but how complex systems
| adapt to change. Instead of reacting defensively, institutions
| should anticipate second-order effects, build regulatory
| capacity, and treat this as a governance and systems problem.
|
| Mature institutions approach disruption with foresight,
| incentives, and feedback loops, not emotions. Without that shift,
| they risk reinforcing outdated hierarchies rather than serving
| the public effectively.
| digitalPhonix wrote:
| > Setting AI aside for a moment, this reflects a broader issue
| in India and elsewhere. When institutions respond to new
| technologies with anger or threats rather than systemic
| thinking, it signals a deeper problem.
|
| No, especially in this case when the first appeal to the high
| court resulted in the high court brushing it off as if nothing
| happened.
|
| It was a reprimand two to institutes (the trial court and the
| high court) they have a job to do and they can't shirk that
| responsibility.
| rramadass wrote:
| No, you are reading it wrong; The Indian Supreme Court is doing
| the right thing.
|
| The lower courts in India are all overloaded with pending cases
| (i.e. not enough judges) and so the incentives for both judges
| and lawyers to "outsource" to AI is very high. This needs to be
| done with caution and that is what the supreme court said, viz;
|
| _The Supreme Court called the case a matter of "institutional
| concern" and said fake AI-generated judgements had "a direct
| bearing on integrity of adjudicatory process".
|
| ...
|
| The defendants challenged the order in the state's high court,
| pointing out that the cited orders were fake. The high court
| acknowledged this, but accepted that the junior civil judge had
| made the error in "good faith" and went on to agree with the
| trial court's decision anyway.
|
| In its order, the high court said that "the citations may be
| non-existent, but if the learned trial court has considered the
| correct principles of law and its application to the facts of
| the case is also correct, mere mentioning of incorrect or non-
| existent rulings/citations in the order cannot be a ground to
| set aside the order".
|
| The high court had also sought a report from the junior judge
| who had used the AI-generated rulings. She told the court that
| this was her first time using an AI tool and she had believed
| the citations to be "genuine". She had no intention to misquote
| or misrepresent the rulings and that "the mistake occurred
| solely due to the reliance on an automatic source", the high
| court wrote.
|
| The high court also advocated for the "exercise of actual
| intelligence over artificial intelligence".
|
| Following this, the defendants appealed again, taking the
| matter to the Supreme Court, which was less forgiving about the
| impact of AI.
|
| Coming down sternly against the fake judgements, the top court
| last Friday stayed the lower court's order on the property
| dispute. It said the use of AI while making judgements was not
| simply "an error in decision making" but an act of
| "misconduct".
|
| "This case assumes considerable institutional concern, not
| because of the decision that was taken on the merits of the
| case, but about the process of adjudication and determination,"
| the top court said._
|
| PS: To get an idea of how overloaded the Indian Judicial System
| is; this happened in a recent case in Allahabad High Court -
| _The order then took an unusual turn. "Since I am feeling
| hungry, tired and physically incapacitated to dictate the
| judgment, the judgment is reserved," the judge recorded._ - He
| had been hearing more than 30 cases on that day -
| https://www.hindustantimes.com/india-news/hungry-tired-allah...
| dartharva wrote:
| This means India's judiciary is operationally stunted because
| of low capacity relative to demand, which is a ridiculously
| urgent and critical flaw that should have simply not occurred
| in the first place. Why on earth are they not increasing
| capacity and headcount?
| rramadass wrote:
| Many reasons, viz; Slow Recruitments/Appointments,
| Executive/Judiciary tussle, Insufficient infrastructure,
| Budget Constraints etc.
|
| The process to qualify as a Judge is long drawn and
| difficult while one can easily become a Lawyer and make
| more money for a fraction of the effort. Apparently there
| are only 22 judges per million people! -
| https://www.ndtv.com/india-news/indian-judicial-system-
| strug...
|
| This is why AI adoption is very important for the Indian
| Judiciary. There are a number of good startups in this area
| which will hopefully alleviate this problem in the near
| future, eg; _Adalat AI_ - https://www.adalat.ai/
|
| _Major Challenges Faced by the Indian Judiciary_ -
| https://www.lloydlawcollege.edu.in/blog/challenges-faced-
| ind...
|
| _Understanding vacancies in the Indian judiciary_ -
| https://prsindia.org/theprsblog/understanding-vacancies-
| in-t...
|
| PS: Have first-hand knowledge of the above since i have a
| ongoing civil case in the lower courts. I attend all court
| dates just to make sure my lawyer does his job and always
| check the status of my case online. The judge has only a
| few minutes for cases in the initial stage (the lawyers
| almost always seek and are granted an adjournment of a
| month) and only spends time on cases in the later
| evidence/cross-examination stages. He dictates/writes his
| notes to a person who types it into a computer which is
| then uploaded to the official website at end-of-the-day.
| triceratops wrote:
| This comment sounds AI-generated.
| tamimio wrote:
| I wonder how many similar cases are happening in the engineering
| or software development sector that go unnoticed, and it seems no
| one is caring enough, only waiting for a disaster to happen so we
| can start seeing some regulation preventing the use of AI in
| engineering/coding industry.
| thisisit wrote:
| It's sad that it took the highest court in the country to point
| out lack of professionalism and misconduct.
|
| The judge took no personal responsibility.
|
| > She told the court that this was her first time using an AI
| tool and she had believed the citations to be "genuine". She had
| no intention to misquote or misrepresent the rulings and that
| "the mistake occurred solely due to the reliance on an automatic
| source", the high court wrote.
|
| She had one job. And that was to read the citations. Instead of
| owning up to the mistake of being lazy all she wanted to talk
| about "intentions".
|
| The high court also took no responsibility.
|
| > In its order, the high court said that "the citations may be
| non-existent, but if the learned trial court has considered the
| correct principles of law and its application to the facts of the
| case is also correct,
|
| This line of reasoning is questionable and attempt to gaslight
| everyone. Judges cite other cases in their judgement. But if the
| junior judge had no clue that the references were fake what
| correct principles was she applying?
|
| End of the day maybe the judgement is correct but this overall
| bullshit.
|
| Given that this is happening all over the world people seem to
| have a convenient excuse - The AI made me do it.
| eaglehead wrote:
| This is going to be a huge problem in every sector. I have been
| exploring solution in this space for fintech and so far what
| Resemble AI is doing [1] is probably the best way to defend.
|
| The surface level for us is not just LLM generated text, it is
| also the combination of AI augmented audio (for incoming calls)
| and then for our own voice agents being able to protect and
| identify services cloning our own agent voices with watermarking.
|
| It's not fun, as we are constantly catching up.
|
| [1] https://www.resemble.ai/detect/
| chrisjj wrote:
| > She had no intention to misquote or misrepresent the rulings
| and that "the mistake occurred solely due to the reliance on an
| automatic source"
|
| Next: gunman pleads death occured solely due to reliance on an
| automatic weapon.
| thisislife2 wrote:
| The high court also advocated for the "exercise of actual
| intelligence over artificial intelligence". Hehe.
| inder1 wrote:
| The failure mode here is predictable. Junior practitioners in any
| domain are being asked to use AI tools before they've developed
| the professional judgment to validate the outputs. You can't spot
| a hallucinated court order if you don't know what real court
| orders look like. The tool isn't the problem. The training
| pipeline that skips fundamentals is.
___________________________________________________________________
(page generated 2026-03-03 23:01 UTC)