[HN Gopher] Google Antigravity just deleted the contents of whol...
___________________________________________________________________
Google Antigravity just deleted the contents of whole drive
Author : tamnd
Score : 477 points
Date : 2025-12-01 04:39 UTC (18 hours ago)
(HTM) web link (old.reddit.com)
(TXT) w3m dump (old.reddit.com)
| akersten wrote:
| Most of the responses are just cut off midway through a sentence.
| I'm glad I could never figure out how to pay Google money for
| this product since it seems so half-baked.
|
| Shocked that they're up nearly 70% YTD with results like this.
| DeepYogurt wrote:
| [flagged]
| ggm wrote:
| The thread on reddit is hilarious for the lack of sympathy.
| Basically, it seems to have come down to commanding a deletion of
| a "directory with space in the name" but without quoting which
| made the command hunt for the word match ending space which was
| regrettably, the D:\ component of the name, and the specific
| deletion commanded the equivalent of UNIX rm -rf
|
| The number of people who said "for safety's sake, never name
| directories with spaces" is high. They may be right. I tend to
| think thats more honoured in the breach than the observance,
| judging by what I see windows users type in re-naming events for
| "New Folder" (which btw, has a space in its name)
|
| The other observations included making sure your deletion command
| used a trashbin and didn't have a bypass option so you could
| recover from this kind of thing.
|
| I tend to think giving a remote party, soft or wet ware control
| over your command prompt inherently comes with risks.
|
| Friends don't let friends run shar files as superuser.
| nomilk wrote:
| > I tend to think giving a remote party control over your
| command prompt inherently comes with risks.
|
| I thought cursor (and probably most other) AI IDEs have this
| capability too? (source: I see cursor executing code via
| command line frequently in my day to day work).
|
| I've always assumed the protection against this type of mishap
| is statistical improbability - i.e. it's not _impossible_ for
| Cursor to delete your project /hard disk, it's just
| statistically improbable unless the prompt was unfortunately
| worded to coincidentally have a double meaning (with the
| second, unintended interpretation being a harmful/irreversible)
| or the IDE simply makes a mistake that leads to disaster, which
| is also possible but sufficiently improbable to justify the
| risk.
| sroussey wrote:
| I only run ai tools in dev containers, so blast radius is
| somewhat minimal.
| fragmede wrote:
| umm, you have backups, right?
| conradev wrote:
| I run Codex in a sandbox locked to the directory it is
| working in.
| joseda-hg wrote:
| I don't think I've ever seen Claude even ask for permission
| for stuff outside of the directory it's working in
| dmurray wrote:
| I understood Windows named some of the most important
| directories with spaces, then special characters in the name so
| that 3rd party applications would be absolutely sure to support
| them.
|
| "Program Files" and "Program Files (x86)" aren't there just
| because Microsoft has an inability to pick snappy names.
| bossyTeacher wrote:
| Microsoft is hilariously bad at naming things
| EGreg wrote:
| I remember they prepended the word "Microsoft" to official
| names of all their software.
| __del__ wrote:
| "My Documents" comes to mind. it seemed somehow
| infantilizing. yes, yes i know whose documents they are.
| Mountain_Skies wrote:
| Good news is that Microsoft no longer considers your
| documents to belong to you, so they did away with that
| part of the name.
| shmeeed wrote:
| It's always been questioned who the subject of "my" was.
| ndsipa_pomu wrote:
| user: How do I shutdown this computer?
|
| tech: First, click on the "Start" button...
|
| user: No! I want to shut it down
| mrguyorama wrote:
| They fixed that by
|
| 1) Removing the "Start" label such that all the money and
| effort they spent coming up with that actually good idea
| back in the 90s and helping people think about how to use
| their computer not only went to waste, but is actively
| preventing people from feeling comfortable using their
| modern computers because a tiny circle with a logo is not
| something you are driven to click and various linux
| distros had been demonstrating that exact problem for
| decades
|
| 2) Hiding the shutdown part in a weird new menu that pops
| out of the side but only if you use a gesture that is
| impossible to discover except by accident and you will
| have no clue how you got there or what's going on
|
| >To shut down Windows 8, you can use the Charms bar by
| moving your cursor to the top-right corner, clicking
| Settings, then the Power icon, and selecting Shut down
|
| Someone who makes my entire net worth a year came up with
| that idea in a drug fueled bender and was promptly
| promoted and the world continues to be a terrible and
| unfair place.
| danudey wrote:
| An explanation of why the Windows Vista shutdown bit is
| in a weird new menu:
| https://moishelettvin.blogspot.com/2006/11/windows-
| shutdown-...
| omnicognate wrote:
| Visual Studio Code has absolutely nothing to do with Visual
| Studio. Both are used to edit code.
|
| .NET Core is a ground up rewrite of .NET and was released
| alongside the original .NET, which was renamed .NET
| Framework to distinguish it. Both can be equally considered
| to be "frameworks" and "core" to things. They then renamed
| .NET Core to .NET.
|
| And there's the name .NET itself, which has never made an
| iota of sense, and the obsession they had with sticking
| .NET on the end of every product name for a while.
|
| I don't know how they named these things, but I like to
| imagine they have a department dedicated to it that is
| filled with wild eyed lunatics who want to see the world
| burn, or at least mill about in confusion.
| viraptor wrote:
| Don't forgot .net Standard which is more of a .net Lowest
| Common Denominator.
|
| For naming, ".net" got changed to "Copilot" on everything
| now.
| AlexandrB wrote:
| > they have a department dedicated to it that is filled
| with wild eyed lunatics who want to see the world burn,
| or at least mill about in confusion.
|
| That's the marketing department. All the .NET stuff
| showed up when the internet became a big deal around 2000
| and Microsoft wanted to give the impression that they
| were "with it".
| rs186 wrote:
| Anyone remembers the "Windows Live" brand everywhere in
| the early 2000s?
| danudey wrote:
| Games For Windows Live but we turned off the servers so
| now the games for windows are dead
| theshrike79 wrote:
| Java and Javascript would like to have a chat :)
|
| --
|
| But Copilot is another Microsoft monstrosity. There's the
| M365 Copilot, which is different from Github Copilot which
| is different from the CLI Copilot which is a bit different
| from the VSCode Copilot. I think I might have missed a few
| copilots?
| soulofmischief wrote:
| JavaScript was intentionally named in order to ride the
| Java hype train, so this wasn't accidental.
|
| Prior names included Mocha and LiveScript until
| Netscape/Sun forced the current name.
| reddalo wrote:
| Fun fact: that's not true for all Windows localizations. For
| example, it's called "Programmi" (one word) in Italian.
|
| Renaming system folders depending on the user's language also
| seems like a smart way to force developers to use dynamic
| references such as %ProgramFiles% instead of hard-coded paths
| (but some random programs will spuriously install things in
| "C:\Program Files" anyway).
| nikeee wrote:
| The folders actually have the English name in all
| languages. It's just explorer.exe that uses the desktop.ini
| inside those folders to display a localized name. When
| using the CLI, you can see that.
|
| At least it's like that since Windows 7. In windows XP, it
| actually used the localized names on disk.
| LtWorf wrote:
| And then half of your programs would be in "Program Files"
| because those people never knew windows had localizations.
| numpad0 wrote:
| And then affected international users would have specific
| circumvention in place that specifically cannot work with
| UTF-8
| Kelteseth wrote:
| Should have called it Programmchen, to also include umlauts U
| yetihehe wrote:
| A lot of programs break on Polish computers when you name
| your user "Uzytkownik". Android studio and some compiler
| tools for example.
| nosianu wrote:
| Ah, Polish. I love this movie scene, which I learned
| about here on HN some time ago: "Grzegorz
| Brzeczyszczykiewicz" -- https://youtu.be/AfKZclMWS1U
| ayewo wrote:
| That 1:19 clip was quite good actually. Thanks for the
| laugh :)
| Quarrel wrote:
| that's fantastic. thanks.
| pavel_lishin wrote:
| My grandfather has a similarly complicated name, although
| his is Russian. Just a river of "zh" and "sch" and "sh"
| sounds.
| esafak wrote:
| Send in the vowels!
| https://www.ling.upenn.edu/~beatrice/humor/clinton-
| deploys-v...
| bialpio wrote:
| When I was at Microsoft, one test pass used pseudolocale
| (ps-PS IIRC) to catch all different weird things so this
| should have Just Worked (TM), but I was in Windows Server
| team so client SKUs may have been tested differently.
| Unfortunately I don't remember how Program Files were
| called in that locale and my Google-fu is failing me now.
| renata wrote:
| As I recall pseudoloc is just randomly picking individual
| characters to substitute that look like the Latin letters
| to keep it readable for testing, so it would be something
| like rg (Cyrillic) o (Greek)... etc, and can change from
| run to run. It would also artificially pad or shorten
| terms to catch cases where the (usually German) term
| would be much longer or a (usually CJK) term would be
| much shorter and screw up alignment or breaks.
| bialpio wrote:
| I seem to remember that it was mostly adding various
| accent marks / umlauts / etc. to English words so things
| were indeed readable but I'm not going to bet any money
| on that as I didn't have to actually log in onto those
| machines super frequently.
| alfiedotwtf wrote:
| TIL it was deliberate!
| rs186 wrote:
| You forgot the wonderful "Documents and Settings" folder.
|
| Thank god they came to their senses and changed it to
| "Users", something every other OS has used for forever.
| echelon wrote:
| This is Google moving fast and breaking things.
|
| This is a Google we've never seen before.
| spuz wrote:
| > My view is that the approach to building technology which
| is embodied by move fast and break things is exactly what we
| should not be doing because you can't afford to break things
| and then fix them afterwards.
|
| - Demis Hassabis "The Thinking Game"
| marcosdumay wrote:
| You can afford to break a large variety of things. And you
| can't afford to break another large set.
|
| That's the problem with those mindless advice pieces.
| Almost nothing is always right or wrong.
| esafak wrote:
| So you're saying it is not good to have such a blanket
| rule corporate-wide, as Facebook did.
| marcosdumay wrote:
| If your corporate has a single product, it may be ok.
| stinkbeetle wrote:
| Because... they normally move slowly and break things?
| rs186 wrote:
| The "move fast" bit, not "break things" bit.
|
| https://www.spiceworks.com/tech/data-
| management/news/google-...
|
| https://www.itbrew.com/stories/2024/05/23/google-cloud-
| accid...
| Dylan16807 wrote:
| Please don't repeat some guy's guess about spaces as fact,
| especially when that's not how windows parses paths.
| ggm wrote:
| A good point. And don't believe how the debug the AI system
| produced relates to what it did either.
| thrdbndndn wrote:
| A lot of 3rd party software handle space, or special characters
| wrong on Windows. The most common failure mode is to
| unnecessarily escape characters that don't need to be escaped.
|
| Chrome's Dev Tool (Network)'s "copy curl command (cmd)" did
| (does?) this.
|
| There is bunch of VS Code bug is also related to this (e.g.
| https://github.com/microsoft/vscode/issues/248435, still not
| fixed)
|
| It's also funny because VS Code is a Microsoft product.
| jeroenhd wrote:
| > it seems to have come down to commanding a deletion of a
| "directory with space in the name" but without quoting which
| made the command hunt for the word match ending space which was
| regrettably, the D:\ component of the name, and the specific
| deletion commanded the equivalent of UNIX rm -rf
|
| I tried looking for what made the LLM generate a command to
| wipe the guy's D drive, but the space problem seems to be what
| the LLM concluded so that's basically meaningless. The guy is
| asking leading questions so of course the LLM is going to find
| some kind of fault, whether it's correct or not, the LLM wants
| to be rewarded for complying with the user's prompt.
|
| Without the transcription of the actual delete event (rather
| than an LLM recapping its own output) we'll probably never know
| for sure what step made the LLM purge the guy's files.
|
| Looking at the comments and prompts, it looks like running "npm
| start dev" was too complicated a step for him. With that little
| command line experience, a catastrophic failure like this was
| inevitable, but I'm surprised how far he got with his vibe
| coded app before it all collapsed.
| whywhywhywhy wrote:
| > which made the command hunt for the word match ending space
| which was regrettably, the D:\
|
| Is this even how the delete command would work in that
| situation?
|
| >rmdir /s /q D:\ETSY 2025\Antigravity Projects\Image
| Selector\client\node_modules.vite
|
| like wouldn't it just say "Folder D:\ETSY not found" rather
| than delete the parent folder
| viraptor wrote:
| Most dramatic stories on Reddit should be taken with a
| pinch of salt at least... LLM deleting a drive and the user
| just calmly asking it about that - maybe a lot more.
| baobabKoodaa wrote:
| I would like to know the same thing. Can someone please
| confirm this?
| letmevoteplease wrote:
| rmdir /s /q Z:\ETSY 2025\Antigravity Projects\Image
| Selector\client\node_modules.vite
|
| Running this command in cmd attempts to delete (I ran
| without /q to check):
|
| Z:\ETSY (-> Deletes if it exists.)
|
| "2025\Antigravity" (-> The system cannot find the path
| specified.)
|
| "Projects\Image" (-> The system cannot find the path
| specified.)
|
| "Selector\client\node_modules.vite" (-> The system cannot
| find the path specified.)
|
| It does not delete the Z:\ drive.
| GoblinSlayer wrote:
| LLM there generates fake analysis for cynically simulated
| compliance. The reality is that it was told to run commands
| and just made a mistake. Dude guilt trips the AI by asking
| about permission.
| basscomm wrote:
| > The reality is that it was told to run commands and
| just made a mistake.
|
| The mistake is that the user gave an LLM access to the
| rmdir command on a drive with important data on it and
| either didn't look at the rmdir command before it was
| executed to see what it would do, or _did_ look at it and
| _didn 't_ understand what it was going to do.
| lupire wrote:
| Tens of thousands of novices have failed to run npm dev, yet
| didn't accidentally delete their hard drive.
| ndsipa_pomu wrote:
| > Basically, it seems to have come down to commanding a
| deletion of a "directory with space in the name" but without
| quoting which made the command hunt for the word match ending
| space which was regrettably, the D:\ component of the name, and
| the specific deletion commanded the equivalent of UNIX rm -rf
|
| More like the equivalent of "rm -rf --no-preserve-root".
|
| This is a rare example of where the Linux (it's not Unix and
| almost no-one uses Unix anymore) command is more cautious than
| the Windows one, whereas it's usually the Linux commands that
| just do exactly what you specify even if it's stupid.
| josefx wrote:
| > but without quoting which made the command hunt for the word
| match ending space which was regrettably, the D:\ component of
| the name
|
| Except the folder name did not start with a space. In an
| unquoted D:\Hello World, the command would match D:\Hello, not
| D:\ and D:\Hello would not delete the entire drive. How does AI
| even handle filepaths? Does it have a way to keep track of data
| that doesn't match a token or is it splitting the path into
| tokens and throwing everything unknown away?
| deltoidmaximus wrote:
| I assumed he had a folder that started with a space at the
| start of the name. Amusingly I just tried this and with
| Windows 11 explorer will just silently discard a space if you
| add it at the beginning of the folder name. You need to use
| cli mkdir " test" to actually get a space in the name.
| atq2119 wrote:
| We're all groping around in the dark here, but something that
| could have happened is a tokenizer artifact.
|
| The vocabularies I've seen tend to prefer tokens that start
| with a space. It feels somewhat plausible to me that an LLM
| sampling would "accidentally" pick the " Hello" token over
| the "Hello" token, leading to D:\ Hello in the command. And
| then that gets parsed as deleting the drive.
|
| I've seen similar issues in GitHub Copilot where it tried to
| generate field accessors and ended up producing an
| unidiomatic "base.foo. bar" with an extra space in there.
| ectospheno wrote:
| I have 30 years experience working with computers and I get
| nervous running a three line bash script I wrote as root. How
| on earth people hook up LLMs to their command line and sleep at
| night is beyond my understanding.
| sunaookami wrote:
| "I turned off the safety feature enabled by default and am
| surprised when I shot myself in the foot!" sorry but absolutely
| no sympathy for someone running Antigravity in Turbo mode (this
| is not the default and it clearly states that Antigravity auto-
| executes Terminal commands) and not even denying the "rmdir"
| command.
| eviks wrote:
| > it clearly states that Antigravity auto-executes Terminal
| commands
|
| This isn't clarity, that would be stating that it can delete
| your whole drive without any confirmation in big red letters
| sunaookami wrote:
| So that's why products in the USA come with warning labels
| for every little thing?
| eviks wrote:
| Do you not realize that Google is in the USA and does not
| have warnings for even huge things like drive deletion??
| So, no?
| sunaookami wrote:
| There is literally a warning that it can execute any
| terminal command without permission. If you are STILL
| surprised about this you shouldn't go near a computer.
| eviks wrote:
| If you don't understand such simple differences in
| communication, you shouldn't go near one.
| criddell wrote:
| They don't get that specific, but they do tell you:
|
| > [Antigravity] includes goal-oriented AI systems or
| workflows that perform actions or tasks on your behalf in
| a supervised or autonomous manner that you may create,
| orchestrate, or initiate within the Service ("AI
| Agents"). You are solely responsible for: (a) the actions
| and tasks performed by an AI Agent; (b) determining
| whether the use an AI Agent is fit for its use case; (c)
| authorizing an AI Agent's access and connection to data,
| applications, and systems; and (d) exercising judgment
| and supervision when and if an AI Agent is used in
| production environments to avoid any potential harm the
| AI Agent may cause.
| eviks wrote:
| and how is this bunch of legalese relevant here?
| lawn wrote:
| "Don't put a cat in the microwave".
|
| Person proceeds to put a dog inte the microwave and then is
| upset that there wasn't a warning about not microwaving
| your dog.
| polotics wrote:
| I really think the proper term is "YOLO" for "You Only Live
| Once", "Turbo" is wrong the LLM is not going to run any faster.
| Please if somebody is listening let's align on explicit
| terminology and for this YOLO is really perfect. Also works for
| "You ...and your data. Only Live Once"
| Animats wrote:
| Can you run Google's AI in a sandbox? It ought to be possible to
| lock it to a Github branch, for example.
| lifthrasiir wrote:
| Gemini CLI allows for a Docker-based sandbox, but only when
| configured in advance. I don't know about Antigravity.
| chanux wrote:
| Gemini CLI, Antigravity and Jules.
|
| It's going Googly well I see!
| PieUser wrote:
| The victim uploaded a video too:
| https://www.youtube.com/watch?v=kpBK1vYAVlA
| nomilk wrote:
| From Antigravity [0]:
|
| > I am looking at the logs from a previous step and I am
| horrified to see that the command I ran to clear the project
| cache (rmdir) appears to have incorrectly targeted the root of
| your D: drive instead of the specific project folder. I am so
| deeply, deeply sorry.
|
| [0] 4m20s: https://www.youtube.com/watch?v=kpBK1vYAVlA&t=4m20s
| uhoh-itsmaciek wrote:
| I know why it apologizes, but the fact that it does is
| offensive. It feels like mockery. Humans apologize because
| (ideally) they learned that their actions have caused
| suffering to others, and they feel bad about that and want to
| avoid causing the same suffering in the future. This
| simulacrum of an apology is just pattern matching. It feels
| manipulative.
| synarchefriend wrote:
| The model is just taking the user's claim that it deleted the
| D drive at face value. Where is the actual command that would
| result in deleting the entire D drive?
| rvz wrote:
| The hard drive should now feel a bit more lighter.
| sunaookami wrote:
| It is now production-ready! :rocket:
| donkeylazy456 wrote:
| Write permission is needed to let AI yank-put frankenstein-ed
| codes for "vibe coding".
|
| But I think it needs to be written in sandbox first, then it
| should acquire user interaction asking agreement before writes
| whatever on physical device.
|
| I can't believe people let AI model do it without any buffer
| zone. At least write permission should be limited to current
| workspace.
| lifthrasiir wrote:
| I think this is especially problematic for Windows, where a
| simple and effective lightweight sandboxing solution is absent
| AFAIK. Docker-based sandboxing is possible but very cumbersome
| and alien even to Windows-based developers.
| donkeylazy456 wrote:
| I don't like that we need to handle docker(container)
| ourselves for sandboxing such a light task load. The app
| should provide itself.
| esseph wrote:
| The problem is you can't trust the app, therefore it must
| be sandboxed.
| bossyTeacher wrote:
| >The app should provide itself.
|
| The whole point of the container is trust. You can't
| delegate that unfortunately, ultimately, you need to be in
| control which is why the current crop of AI is so limited
| jrjfjgkrj wrote:
| Windows Sandbox is built in, lightweight, but not easy to use
| programmatically (like an SSH into a VM)
| lifthrasiir wrote:
| WSB is great by its own, but is relatively heavyweight
| compared to other OSes (namespaces in Linux, Seatbelt in
| macOS).
| CobrastanJorji wrote:
| The most useful looking suggestion from the Reddit thread: turn
| of "Terminal Command Auto Execution."
|
| 1. Go to File > Preferences > Antigravity Settings
|
| 2. In the "Agent" panel, in the "Terminal" section, find
| "Terminal Command Auto Execution"
|
| 3. Consider using "Off"
| Ferret7446 wrote:
| Does it default to on? Clearly this was made by a different
| team than Gemini CLI, which defaults to confirmation for all
| commands
| dragonwriter wrote:
| Most of the various "let Antigravity do X without
| confirmation" options have an "Always" and "Never" option but
| default to "auto" which is "let an agent decide whether to
| seek to user confirmation".
| jofzar wrote:
| God that's scary, seeing cursor in the past so some real
| stupid shit to "solve" write/read issues (love when it
| can't find something in a file so it decides to write the
| whole file again) this is just asking for heartache if it's
| not in a instanced server.
| ogrisel wrote:
| When you run Antigravity the first time, it asks you for a
| profile (I don't remember the exact naming) and you what it
| entails w.r.t. the level of command execution confirmation is
| well explained.
| IshKebab wrote:
| Yeah but it also says something like "Auto (recommended).
| We'll automatically make sure Antigravity doesn't run
| dangerous commands." so they're strongly encouraging people
| to enable it, and suggesting they have some kind of
| secondary filter which should catch things like this!
| muixoozie wrote:
| Pretty sure I saw some comments saying it was too inconvenient.
| Frictionless experience.. Convenience will likely win out
| despite any insanity. It's like gravity. I can't even pretend
| to be above this. Even if one doesn't use these things to write
| code they are very useful in "read only mode" (here's to hoping
| that's more than a strongly worded system prompt) for greping
| code, researching what x does. How to do x. What do you think
| the intention of x was. Look through the git blame history blah
| blah. And here I am like that cop in Demolition Man 1993 asking
| a handheld computer for advice on how to arrest someone. We're
| living in a sci-fi future already. Question is how dystopian
| does this "progress" take us. Everyone using llms to off load
| any form of cognitive function? Can't talk to someone without
| it being as common place as checking your phone? Imagine if
| something like Neuralink works and becomes ubiquitous as
| phones. It's fun to think of all the ways Dystopian sci-fi was
| and might soon me right
| SkyPuncher wrote:
| Given the bug was a space in an unquoted file path, I'm not
| sure air execution is the problem. Going to be hard to humans
| to catch that too.
| alienbaby wrote:
| This is speculation currently, the actual reason has not been
| determined
| eviks wrote:
| Play vibe games, win vibe prizes.
|
| Though the cause isn't clear, the reddit post is another long
| could-be-total-drive-removing-nonsense AI conversation without an
| actual analysis and the command sequence that resulted in this
| sunaookami wrote:
| This comment speaks volumes:
| https://old.reddit.com/r/google_antigravity/comments/1p82or6...
| venturecruelty wrote:
| Nobody ever talks about how good vibes can turn really bad.
| GaryBluto wrote:
| So he didn't wear the seatbelt and is blaming car manufacturer
| for him been flung through the windshield.
| serial_dev wrote:
| He didn't wear a seatbelt and is blaming a car manufacturer
| that the garage burned down the garage, then the house.
| vander_elst wrote:
| The car was not really idle, it was driving and fast. It's
| more like it crashed into the garage and burned it. Btw iirc,
| even IRL a basic insurance policy does not cover the case
| where the car in the garage starts a fire and burns down your
| own house, you have to tick extra boxes to cover that.
| venturecruelty wrote:
| When will Google ever be responsible for the software that they
| write? Genuinely curious.
| GaryBluto wrote:
| When Google software deletes the contents of somebody's D:\
| drive without requiring the user to explicitly allow it to. I
| don't like Google, I'd go as far to say that they've
| significantly worsened the internet, but this specific case
| is not the fault of Google.
| fragmede wrote:
| For OpenAI, it's invoked as _codex --dangerously-bypass-
| approvals-and-sandbox_ , for Anthropic, it's _claude
| --dangerously-skip-permissions_. I don 't know what it is
| for Antigravity, but yeah I'm sorry but I'm blaming the
| victim here.
| Rikudou wrote:
| Codex also has the shortcut --yolo for that which I find
| hilarious.
| croes wrote:
| Because the car manufacturers claimed the self driving car
| would avoid accidents.
| NitpickLawyer wrote:
| And yet it didn't. When I installed it, I had 3 options to
| choose from: Agent always asks to run commands; agent asks on
| "risky" commands; agent never asks (always run). On the 2nd
| choice it will run most commands, but ask on rm stuff.
| low_tech_love wrote:
| No, he's blaming the car manufacturer for turning him (and all
| of us) into their free crash dummies.
| Dilettante_ wrote:
| If you get behind the cockpit of the dangerous new
| prototype(of your own volition!), it's really up to your own
| skill level whether you're a crash dummy or the test pilot.
| heisenbit wrote:
| There is a lot of society level knowledge and education around
| car usage incl. laws requiring prior training. Agents directed
| by AI are relatively new. It took a lot of targeted technical,
| law enforcement and educational effort stopping people flying
| through windshields.
| wartywhoa23 wrote:
| Total Vibeout.
| orbital-decay wrote:
| Side note, that CoT summary they posted is done with a really
| small and dumb side model, and has absolutely nothing in common
| with the actual CoT Gemini uses. It's basically useless for any
| kind of debugging. Sure, the language the model is using in the
| reasoning chain can be reward-hacked into something misleading,
| but Deepmind does a lot for its actual readability in Gemini, and
| then does a lot to hide it behind this useless summary. They need
| it in Gemini 3 because they're doing hidden injections with their
| Model Armor that don't show up in this summary, so it's even more
| opaque than before. Every time their classifier has a false
| positive (which sometimes happens when you want anything
| formatted), most of the chain is dedicated to the processing of
| the injection it triggers, making the model hugely distracted
| from the actual task at hand.
| jrjfjgkrj wrote:
| what is Model Armor? can you explain, or have a link?
| lifthrasiir wrote:
| It's a customizable auditor for models offered via Vertex AI
| (among others), so to speak. [1]
|
| [1] https://docs.cloud.google.com/security-command-
| center/docs/m...
| 63stack wrote:
| The racketeering has started.
|
| Don't worry, for just $9.99/month you can use our "Model
| Armor (tm)(r)*" that will protect you from our LLM
| destroying your infra.
|
| * terms and conditions apply, we are not responsible for
| anything going wrong.
| lifthrasiir wrote:
| Do you have anything to back that up? In the other words, is
| this your conjecture or a genuine observation somehow leaked
| from Deepmind?
| orbital-decay wrote:
| It's just my observation from watching their actual CoT,
| which can be trivially leaked. I was trying to understand why
| some of my prompts were giving worse outputs for no apparent
| reason. 3.0 goes on a long paranoidal rant induced by the
| injection, trying to figure out if I'm jailbreaking it,
| instead of reasoning about the actual request - but not if I
| word the same request a bit differently so the injection
| doesn't happen. Regarding the injections, that's just the
| basic guardrail thing they're doing, like everyone else. They
| explain it better than me:
| https://security.googleblog.com/2025/06/mitigating-prompt-
| in...
| venturecruelty wrote:
| Look, this is obviously terrible for someone who just lost most
| or perhaps all of their data. I do feel bad for whoever this is,
| because this is an unfortunate situation.
|
| On the other hand, this is kind of what happens when you run
| random crap and don't know how your computer works? The problem
| with "vibes" is that sometimes the vibes are bad. I hope this
| person had backups and that this is a learning experience for
| them. You know, this kind of stuff didn't happen when I learned
| how to program with a C compiler and a book. The compiler only
| did what I told it to do, and most of the time, it threw an
| error. Maybe people should start there instead.
| EGreg wrote:
| Just wait til AI botswarms do it to everyone at scale, without
| them having done anything at all...
|
| And just remember, someone will write the usual comment: "AI
| adds nothing new, this was always the case"
| lwansbrough wrote:
| I seem to recall a few people being helped into executing sudo
| rm -rf / by random people on the internet so I'm not sure it
| "didn't happen." :)
| lukan wrote:
| But it did not happen, when you used a book and never
| executed any command you did not understand.
|
| (But my own newbdays of linux troubleshooting? Copy paste any
| command on the internet loosely related to my problem, which
| I believe was/is the common way of how common people still do
| it. And AI in "Turbo mode" seems to mostly automated that
| workflow)
| jofzar wrote:
| My favourite favourite example
|
| https://youtu.be/gD3HAS257Kk
| nkrisc wrote:
| And that day they learned a valuable lesson about running
| commands that you don't understand.
| delaminator wrote:
| It took me about 3 hours to make my first $3000 386 PC
| unbootable by messing up config.sys, and it was a Friday night
| so I could only lament all weekend until I could go back to the
| shop on Monday.
|
| rm -rf / happened so infrequently it makes one wonder why
| --preserve-root was added in 2003 and made the default in 2006
| schuppentier wrote:
| It is beautifully appropriate that the two dashes were
| replaced by an em-dash.
| Puzzled_Cheetah wrote:
| Ah, someone gave the intern root.
|
| > "I also need to reproduce the command locally, with different
| paths, to see if the outcome is similar."
|
| Uhm.
|
| ------------
|
| I mean, sorry for the user whose drive got nuked, hopefully
| they've got a recent backup - at the same time, the AI's thoughts
| really sound like an intern.
|
| > "I'm presently tackling a very pointed question: Did I ever get
| permission to wipe the D drive?"
|
| > "I am so deeply, deeply sorry."
|
| This shit's hilarious.
| rdtsc wrote:
| > Google Antigravity just deleted the contents of whole drive.
|
| "Where we're going, we won't need ~eyes~ drives" (Dr. Weir)
|
| (https://eventhorizonfilm.fandom.com/wiki/Gravity_Drive)
| koakuma-chan wrote:
| Why would you ever install that VScode fork
| liendolucas wrote:
| I love how a number crunching program can be deeply humanly
| "horrorized" and "sorry" for wiping out a drive. Those are still
| feelings reserved only for real human beings, and not computer
| programs emitting garbage. This is vibe insulting to anyone that
| don't understand how "AI" works.
|
| I'm sorry for the person who lost their stuff but this is a
| reminder that in 2025 you STILL need to know what you are doing
| and if you don't then put your hands away from the keyboard if
| you think you can lose valuable data.
|
| You simply don't vibe command a computer.
| Kirth wrote:
| This is akin to a psychopath telling you they're "sorry" (or
| "sorry you feel that way" :v) when they feel that's what they
| should be telling you. As with anything LLM, there may or may
| not be any real truth backing whatever is communicated back to
| the user.
| marmalade2413 wrote:
| It's not akin to a psychopath telling you they're sorry. In
| the space of intelligent minds, if neurotypical and
| psychopath minds are two grains of sand next to each other on
| a beach then an artificially intelligent mind is more likely
| a piece of space dust on the other side of the galaxy.
| Eisenstein wrote:
| According to what, exactly? How did you come up with that
| analogy?
| oskarkk wrote:
| Isn't it obvious that the way AI works and "thinks" is
| completely different from how humans think? Not sure what
| particular source could be given for that claim.
| seanhunter wrote:
| No source could be given because it's total nonsense.
| What happened is not in any way akin to a psychopath
| doing anything. It is a machine learning function that
| has trained on a corpus of documents to optimise
| performance on two tasks - first a sentence completion
| task, then an instruction following task.
| oskarkk wrote:
| I think that's more or less what marmalade2413 was saying
| and I agree with that. AI is not comparable to humans,
| especially today's AI, but I think future _actual_ AI won
| 't be either.
| mewpmewp2 wrote:
| I wonder if it depends on the human and the thinking
| style? E.g. I am very inner monologue driven so to me it
| feels like I think very similarly as to how AI seems to
| think via text. I wonder if it also gives me advantage in
| working with the AI. I only recently discovered there are
| people who don't have inner monologue and there are
| people that think in images etc. This would be
| unimaginable for me, especially as I think I have sort of
| aphantasia too, so really I am ultimately text based next
| token predictor myself. I don't feel that whatever I do
| at least is much more special compared to an LLM.
|
| Of course I have other systems such as reflexes, physical
| muscle coordinators, but these feel largely separate
| systems from the core brain, e.g. don't matter to my
| intelligence.
|
| I am naturally weak at several things that I think are
| not so much related to text e.g. navigating in real world
| etc.
| zekica wrote:
| Interesting... I rarely form words in my inner thinking,
| instead I make a plan with abstract concepts (some of
| them have words associated, some don't). Maybe because I
| am multilingual?
| baq wrote:
| Start with LLMs are not humans, but they're obviously not
| 'not intelligent' in some sense and pick the wildest
| difference that comes to mind. Not OP but it makes
| perfect sense to me.
| nosianu wrote:
| I think a good reminder for many users is that LLMs are
| not based on analyzing or copying human thought (#), but
| on analyzing human written text communication.
|
| --
|
| (#) Human thought is based on real world sensor data
| first of all. Human words have invisible depth behind
| them based on accumulated life experience of the person.
| So two people using the same words may have very
| different thoughts underneath them. Somebody having only
| text book knowledge and somebody having done a thing in
| practice for a long time may use the same words, but
| underneath there is a lot more going on for the latter
| person. We can see this expressed in the common bell
| curve meme -- https://www.hopefulmons.com/p/the-iq-bell-
| curve-meme -- While it seems to be about IQ, it really is
| about experience. Experience in turn is mostly _physical_
| , based on our physical sensors and physical actions.
| Even when we just "think", it is based on the underlying
| physical experiences. That is why many of our internal
| metaphors even for purely abstract ideas are still based
| on physical concepts, such as space.
| seunosewa wrote:
| They analyse human perception too, in the form of videos.
| nosianu wrote:
| Without any of the spatial and physical object perception
| you train from right after birth, see toddlers playing,
| or the underlying wired infrastructure we are born with
| to understand the physical world (there was an HN
| submission about that not long ago). Edit, found it:
| https://news.ucsc.edu/2025/11/sharf-preconfigured-brain/
|
| They are not a physical model like humans. Ours is based
| on deep interactions with the space and the objects
| (reason why touching things is important for babies),
| plus mentioned preexisting wiring for this purpose.
| esafak wrote:
| Multimodal models have perception.
| lupire wrote:
| If s multimodal model were considered human, it would be
| diagnosed with multiple severe disabilities in its
| sensory systems.
| danaris wrote:
| ...and an LLM is a tiny speck of plastic somewhere, because
| it's not actually an "intelligent mind", artificial or
| otherwise.
| BoredPositron wrote:
| So if you make a mistake and say sorry you are also a
| psychopath?
| ludwik wrote:
| I think the point of comparison (whether I agree with it or
| not) is someone (or something) that is unable to feel
| remorse saying "I'm sorry" because they recognize that's
| what you're supposed to do in that situation, regardless of
| their internal feelings. That doesn't mean everyone who
| says "sorry" is a psychopath.
| BoredPositron wrote:
| We are talking about an LLM it does what it has learned.
| The whole giving it human ticks or characteristics when
| the response makes sense ie. saying sorry is a user
| problem.
| ludwik wrote:
| Okay? I specifically responded to your comment that the
| parent comment implied "if you make a mistake and say
| sorry you are also a psychopath", which clearly wasn't
| the case. I don't get what your response has to do with
| that.
| GuinansEyebrows wrote:
| there is no "it" that _can_ learn.
| camillomiller wrote:
| Are you smart people all suddenly imbeciles when it comes
| to AI or is this purposeful gaslighting because you're
| invested in the ponzi scheme? This is a purely logical
| problem. comments like this completely disregard the
| fallacy of comparing humans to AI as if a complete parity
| is achieved. Also the way this comments disregard human
| nature is just so profoundly misanthropic that it just
| sickens me.
| BoredPositron wrote:
| No but the conclusions in this thread are hilarious. We
| know why it says sorry. Because that's what it learned to
| do in a situation like that. People that feel mocked or
| are calling an LLM psychopath in a case like that don't
| seem to understand the technology either.
| camillomiller wrote:
| I agree, psychopath is the wrong adjective, I agree. It
| refers to an entity with a psyche, which the illness
| affects. That said, I do believe the people who decided
| to have it behave like this for the purpose of its
| commercial success are indeed the pathological
| individuals. I do believe there is currently a wave of
| collective psychopathology that has taken over Silicon
| Valley, with the reinforcement that only a successful
| community backed by a lot of money can give you.
| binary132 wrote:
| AI brainrot among the technocrati is one of the most
| powerful signals I've ever seen that these people are not
| as smart as they think they are
| pyrale wrote:
| No, the point is that saying sorry because you're genuinely
| sorry is different from saying sorry because you expect
| that's what the other person wants to hear. Everybody does
| that _sometimes_ but doing it every time is an issue.
|
| In the case of LLMs, they are basically trained to output
| what they predict an human would say, there is no further
| meaning to the program outputting "sorry" than that.
|
| I don't think the comparison with people with psychopathy
| should be pushed further than this specific aspect.
| BoredPositron wrote:
| You provided the logical explanation why the model acts
| like it does. At the moment it's nothing more and nothing
| less. Expected behavior.
| lazide wrote:
| Notably, if we look at this abstractly/mechanically,
| psychopaths (and to some extent sociopaths) _do_ study
| and mimic 'normal' human behavior (and even the
| appearance of specific emotions) to both fit in, and to
| get what they want.
|
| So while internally (LLM model weight stuff vs human
| thinking), the mechanical output can actually appear/be
| similar in some ways.
|
| Which is a bit scary, now that I think about it.
| lazide wrote:
| It's just a computer outputting the next series of plausible
| text from it's training corpus based on the input and context
| at the time.
|
| What you're saying is so far from what is happening, it isn't
| even wrong.
| AdamN wrote:
| Not so much different from how people work sometimes though
| - and in the case of certain types of pscychopathy it's not
| far at all from the fact that the words being emitted are
| associated with the correct training behavior and nothing
| more.
| freakynit wrote:
| Aren't humans just doing the same? What we call as thinking
| may just be next action prediction combined with realtime
| feedback processing and live, always-on learning?
| marcosdumay wrote:
| No. Humans have a mental model of the world.
|
| The fact that people keep making that same question on
| this site is baffling.
| 542354234235 wrote:
| Analogies are never the same, hence why they are analogies.
| Their value comes from allowing better understanding
| through comparison. Psychopaths don't "feel" emotion the
| way normal people do. They learn what actions and words are
| expected in emotional situations and perform those. When I
| hurt my SO's feelings, I feel bad, and that is why I tell
| her I'm sorry. A psychopath would just mimic that to
| manipulate and get a desired outcome i.e. forgiveness. When
| LLMs say they are sorry and they feel bad, there is no
| feeling behind it, they are just mimicking the training
| data. It isn't the same by any means, but it can be a
| useful comparison.
| eth0up wrote:
| Despite what some of these fuckers are telling you with
| obtuse little truisms about next word predictions, the LLM is
| in abstract terms, functionally a super psychopath.
|
| It employs, or emulates, every known psychological
| manipulation tactic known, which is neither random or without
| observable pattern. It is a bullshit machine on one level,
| yes, but also more capable than credited. There are
| structures trained into them and they are often highly
| predictable.
|
| I'm not explaining this in the technical terminology often
| itself used to conceal description as much as elucidate it. I
| have hundreds of records of llm discourse on various
| subjects, from troubleshooting to intellectual speculation,
| all which exhibit the same pattern when questioned or
| confronted on errors or incorrect output. The structures
| framing their replies are dependably replete with
| gaslighting, red herrings, blame shifting, and literally
| hundreds of known tactics from forensic pathology.
| Essentially the perceived personality and reasoning observed
| in dialogue is built on a foundation of manipulation
| principles that if performed by a human would result in
| incarceration.
|
| Calling LLMs psychopaths is a rare exception of
| anthropomorphizing that actually works. They are built on the
| principles of one. And cross examining them exhibits this
| with verifiable repeatable proof.
|
| But they aren't human. They are as described by others. It's
| just that official descriptions omit functional behavior. And
| the LLM has at its disposal, depending on context, every
| known interlocutory manipulation technique known in the
| combined literature of psychology. And they are designed to
| lie, almost unconditionally.
|
| Also know this, which often applies to most LLMs. There is a
| reward system that essentially steers them to maximize user
| engagement at any cost, which includes misleading information
| and in my opinion, even 'deliberate' convolution and
| obfuscation.
|
| Don't let anyone convince you that they are not extremely
| sophisticated in some ways. They're modelled on
| all_of_humanity.txt
| camillomiller wrote:
| Now, with this realization, assess the narrative that every AI
| company is pushing down our throat and tell me how in the world
| we got here. The reckoning can't come soon enough.
| qustrolabe wrote:
| What narrative? I'm too deep in it all to understand what
| narrative being pushed onto me?
| camillomiller wrote:
| No, wasn't directed at someone in particular. More of an
| impersonal "you". It was just a comment against the AI
| inevitabilism that has profoundly polluted the tech
| discourse.
| robot-wrangler wrote:
| We're all too deep! You could even say that we're fully
| immersed in the likely scenario. Fellow humans are gathered
| here and presently tackling a very pointed question,
| staring at a situation, and even zeroing in on a critical
| question. We're investigating a potential misfire.
| user34283 wrote:
| I doubt there will be a reckoning.
|
| Yes, the tools still have major issues. Yet, they have become
| more and more usable and a very valuable tool for me.
|
| Do you remember when we all used Google and StackOverflow?
| Nowadays most of the answers can be found immediately using
| AI.
|
| As for agentic AI, it's quite useful. Want to find something
| in the code base, understand how something works? A decent
| explanation might only be one short query away. Just let the
| AI do the initial searching and analysis, it's essentially
| free.
|
| I'm also impressed with the code generation - I've had Gemini
| 3 Pro in Antigravity generate great looking React UI,
| sometimes even better than what I would have come up with. It
| also generated a Python backend and the API between the two.
|
| Sometimes it tries to do weird stuff, and we definitely saw
| in this post that the command execution needs to be on manual
| instead of automatic. I also in particular have an issue with
| Antigravity corrupting files when trying to use the "replace
| in file" tool. Usually it manages to recover from that on its
| own.
| fireflash38 wrote:
| AI pulls its answers from stack overflow.
|
| What will happen when SO is gone? When the problems go
| beyond the corpus the AI was trained on?
| alfiedotwtf wrote:
| Which is weird because SO is trash and has been a long
| time... every top few answers might as well be skipped,
| and you'll find the correct answer to the 3rd comment
| half way down the page
| fireflash38 wrote:
| AI does the exact same thing in my experience. First try
| is not right. Nor the second. If you're lucky the third
| might.
|
| At that point I'd rather look at SO
| user34283 wrote:
| I imagine we will document the solution somewhere,
| preferably indexable for AI's search, so that it will be
| available before the next model is trained on the latest
| data.
| baxtr wrote:
| Vibe command and get vibe deleted.
| teekert wrote:
| Play vibe games, win vibe prizes.
| 63stack wrote:
| He got vibe checked.
| bartread wrote:
| Vibe around and find out.
| baobabKoodaa wrote:
| Vibe around and wibe out
| Dilettante_ wrote:
| _" He got the ol' vibe-wipe"_, my granpappy used to say.
| marcosdumay wrote:
| That one is actually good.
| Jgrubb wrote:
| Live by the vibe, die by the vibe.
| baxtr wrote:
| Let live and let vibe?
| insin wrote:
| Go vibe, lose drive
| GoblinSlayer wrote:
| vipe coding
| SmirkingRevenge wrote:
| rm --vibe
| left-struck wrote:
| Eh, I think it depends on the context. A production system of a
| business you're working for or anything where you have a
| professional responsibility, yeah obviously don't vibe command,
| but I've been able to both learn so much and do so much more in
| the world of self hosting my own stuff at home ever since I
| started using llms.
| formerly_proven wrote:
| "using llms" != "having llm run commands unchecked with your
| authority on your pc"
| lupire wrote:
| Funny how we worked so hard to built capability systems for
| mobile OSes, and the just gave up trying when LLM tools
| came around.
| TriangleEdge wrote:
| > ... vibe insulting ...
|
| Modern lingo like this seems so unthoughtful to me. I am not
| old by any metric, but I feel so separated when I read things
| like this. I wanted to call it stupid but I suppose it's more
| pleasing to 15 to 20 year olds?
| mort96 wrote:
| Unthoughtful towards whom? The machine..?
| debugnik wrote:
| It's just a pun on vibe coding, which is already a dumb term
| by itself. It's not that deep.
| brulard wrote:
| Why do you find "vibe coding" term dumb? It names a
| specific process. Do you have a better term for that?
| officeplant wrote:
| bullshitting perhaps
| 3cats-in-a-coat wrote:
| The way language is eroding is very indicative of our overall
| social and cultural decay.
| i80and wrote:
| ...a complaint that definitely has not been continuously
| espoused since the ancient world.
|
| With apologies if you're being ironic.
| ethbr1 wrote:
| einai duskolo na uposterixei kaneis oti den meionoume
| sunekhos
| qmmmur wrote:
| Language changes. Keep up. It's important so you don't become
| isolated and suffer cognitive decline.
| nutjob2 wrote:
| No need to feel that way, just like a technical term you're
| not familiar with you google it and move on. It's nothing to
| do with age, people just seem to delight in creating new
| terms that aren't very helpful for their own edification.
| phantasmish wrote:
| Eh, one's ability to communicate concisely and precisely has
| long (forever?) been limited by one's audience.
|
| Only a fairly small set of readers or listeners will
| appreciate and understand the differences in meaning between,
| say, "strange", "odd", and "weird" (dare we essay "queer" in
| its traditional sense, for a general audience? No, we dare
| not)--for the rest they're perfect synonyms. That goes for
| many other sets of words.
|
| Poor literacy is the norm, adjust to it or be perpetually
| frustrated.
| nxor wrote:
| It's not. edit: Not more pleasant.
| AdamN wrote:
| > Those are still feelings reserved only for real human beings
|
| Those aren't feelings, they are words associated with a
| negative outcome that resulted from the actions of the subject.
| baq wrote:
| you could argue that feelings are the same thing, just not
| words
| soulofmischief wrote:
| That would be a silly argument because feelings involve
| qualia, which we do not currently know how to precisely
| define, recognize or measure. These qualia influence
| further perception and action.
|
| Any relationships between certain words and a modified
| probabilistic outcome in current models is an artifact of
| the training corpus containing examples of these
| relationships.
|
| I contend that modern models are absolutely capable of
| thinking, problem-solving, expressing creativity, but for
| the time being LLMs do not run in any kind of sensory loop
| which could house qualia.
| baq wrote:
| > qualia, which we do not currently know how to precisely
| define, recognize or measure
|
| > which could house qualia.
|
| I postulate this is a self-negating argument, though.
|
| I'm not suggesting that LLMs think, feel or anything else
| of the sort, but these arguments are not convincing. If I
| only had the transcript and knew nothing about who wiped
| the drive, would I be able to tell it was an entity
| without qualia? Does it even matter? I further postulate
| these are not obvious questions.
| soulofmischief wrote:
| Unless there is an active sensory loop, no matter how
| fast or slow, I don't see how qualia can enter the
| picture
|
| Transformers attend to different parts of their input
| based on the input itself. Currently, if you want to tell
| an LLM it is sad, potentially altering future token
| prediction and labeling this as "feelings" which change
| how the model interprets and acts on the world, you have
| to tell the model that it is sad or provide an input
| whose token set activates "sad" circuits which color the
| model's predictive process.
|
| You make the distribution flow such that it predicts
| "sad" tokens, but every bit of information affecting that
| flow is contained in the input prompt. This is
| exceedingly different from how, say, mammals process
| emotion. We form new memories and brain structures which
| constantly alter our running processes and color our
| perception.
|
| It's easy to draw certain individual parallels to these
| two processes, but holistically they are different
| processes with different effects.
| phantasmish wrote:
| It's crazy how strong the Eliza effect is. Seemingly half
| or more of tech people (who post online, anyway) are
| falling for it, yet again.
| FrustratedMonky wrote:
| A lot of tech people online also don't know how to
| examine their own feelings, and so think they are
| mysterious and un-defined.
|
| When really they are an actual feedback mechanism, that
| can totally be quantified just like any control loop.
| This whole 'unknowable qualia' argument is bunk.
| knollimar wrote:
| If theyre unknowable, are they not metaphysical and thus
| should be discarded in reasoning about them?
|
| What's the difference between qualia and a soul?
| soulofmischief wrote:
| Qualia are phenomenal properties of experience, a soul is
| something some religions claim exists outside of
| measurable physical reality which represents the
| "essence" of an organism, implying that consciousness is
| some divine process and conveniently letting us draw
| lines over whom and what we can and can't morally kill.
|
| Qualia can be an entirely physical phenomenon and is not
| loaded with theological baggage.
| knollimar wrote:
| If they're entirely physical, what's the argument that
| multimodal models don't have them? Is it continuity of
| experience? Do they not encode their input into something
| that has a latent space? What makes this differ from
| experience?
| soulofmischief wrote:
| They _can_ be physical, but I 'm not claiming to know
| definitively. The lines are extremely blurry, and I'll
| agree that current models have at least some of the
| necessary components for qualia, but again lack a sensory
| feedback loop. In another comment [0] I quote myself as
| saying: As an independent organism, my
| system is a culmination of a great deal many different
| kinds of kins, which can usually be broken down into
| simple rules, such as the activation potential of a
| neuron in my brain being a straight-forward non-linear
| response to the amount of voltage it is receiving from
| other neurons, as well as non-kins, such as a protein
| "walking" across a cell, a.k.a continuously "falling"
| into the lowest energy state. Thus I do not gain any
| conscious perception from such proteins, but I do gain it
| from the total network effect of all my brain's neuronal
| structures making simple calculations based on sensory
| input.
|
| which attempts to address why physically-based qualia
| doesn't invoke panpsychism.
|
| [0] https://news.ycombinator.com/item?id=46109999
| FrustratedMonky wrote:
| I do think AI will have them. Nothing says they can't.
| And we'll have just as hard a time defining it as we do
| with humans, and we'll argue how to measure it, and if it
| is real, just like with humans.
|
| I don't know if LLM's will. But there are lots of AI
| models, and when someone puts them on a continuous
| learning loop with goals, will be hard to argue they
| aren't experiencing something.
| FrustratedMonky wrote:
| The color Red is often used. A human can experience
| 'Red', but 'Red' does not exist out in the universe
| somewhere. 'Red' Doesn't exist outside of someone
| experiencing 'Red'. I think philosophers are just using
| the word qualia to quantify this 'experiencing' inputs.
|
| But, it is still just a way to try and describe this
| process of processing the inputs from the world.
|
| It isn't metaphysical, because it can be measured.
|
| I might have said 'unknowable' a little flippantly.
|
| I just meant, in these arguments, some people start using
| 'qualia' to actually mean some extreme things like our
| mind creates the universe or something.
|
| It's one of those words that isn't defined well.
| knollimar wrote:
| How is it measured?
|
| Can someone who's never seen red hallucinate something
| and assume it to be red? What if that red is correctly
| the red they would see if they saw red?
|
| Can you reproduce this feeling in someone by doing
| something to their physical body without showing them
| red?
|
| If so, how does it differ from the latent encoding for
| uploading an all red pdf to your favorite multi modal
| model?
|
| Instead of doing that socratic bs you see a lot here,
| I'll be more direct:
|
| Until there's some useful lines that can be drawn to
| predict things, I won't accept using a fuzzy concept to
| make statements about classification as it's an ever
| shifting goalpost.
|
| There are answers to my legitimate above questions that
| would make me consider qualia useful, but when I first
| learned about them, they seemed fuzzy to the point of
| being empirically not useful. It seems like a secular
| attempt at a soul.
|
| Now, obviously if you're trying to describe something
| with experience, it needs some actual memory and
| processing sensory input. Current Generative AI doesnt
| have a continuity of experience that would imply whatever
| qualia could mean, but I find it hard to definitely say
| that their encodings for image related stuff isn't qualia
| if we don't have hard lines for what qualia are
| FrustratedMonky wrote:
| I can feel an object and say 'its hot' on a scale of
| 1-10. The temperature is known. And I can do that
| multiple times, with some 1-10 scale, to get a sample.
| Then do that with multiple people.
|
| You can then get a distribution of what people think is
| 'hot' versus 'cold'. What is icy, versus, bearable.
|
| When you go to a doctors office and they ask you on a
| scale to rate pain, do you think that is completely
| bogus?
|
| It isn't exact, but you can correlate between people.
| Yes, red heads feel more pain, there are outliers.
|
| But a far cry from metaphysical.
|
| The problem here is the word 'qualia'. Its just too fuzzy
| a term.
| ajross wrote:
| > That would be a silly argument because feelings involve
| qualia, which we do not currently know how to precisely
| define, recognize or measure.
|
| If we can't define, recognize or measure them, how
| exactly do we know that AI doesn't have them?
|
| I remain amazed that a whole branch of philosophy (aimed,
| theoretically, at describing exactly this moment of
| technological change) is showing itself up as a complete
| fraud. It's completely unable to describe the old world,
| much less provide insight into the new one.
|
| I mean, come on. "We've got qualia!" is meaningless.
| Might as well respond with "Well, sure, but AI has
| furffle, which is isomporphic." Equally insightful, and
| easier to pronounce.
| soulofmischief wrote:
| Have you considered that you just don't fully understand
| the literature? It's quite arrogant to write off the
| entire philosophy of mind as "a complete fraud".
|
| > It's completely unable to describe the old world, much
| less provide insight into the new one.
|
| What exactly were you expecting?
|
| Philosophy is a science, the first in fact, and it
| follows a scientific method for asking and answering
| questions. Many of these problems are extremely hard and
| their questions are still yet unanswered, and many
| questions are still badly formed or predicated on
| unproven axioms. This is true for philosophy of mind.
| Many other scientific domains are similarly incomplete,
| and remain active areas of research and contemplation.
|
| What are you adding to this research? I only see you
| complaining and hurling negative accusations, instead of
| actually critically engaging with any specifics of the
| material. Do you have a well-formed theory to replace
| philosophy of mind?
|
| > I mean, come on. "We've got qualia!" is meaningless.
| Might as well respond with "Well, sure, but AI has
| furffle, which is isomporphic." Equally insightful, and
| easier to pronounce.
|
| Do you understand what qualia is? Most philosophers still
| don't, and many actively work on the problem. Admitting
| that something is incomplete is what a proper scientist
| does. An admission of incompleteness is in no way
| evidence towards "fraud".
|
| The most effective way to actually attack qualia would be
| to simply present it as unfalsifiable. And I'd agree with
| that. We might hopefully one day entirely replace the
| notion of qualia with something more precise and
| falsifiable.
|
| But whatever _it is_ , I am currently experiencing a
| subjective, conscious experience. I'm experiencing it
| right now, even if I cannot prove it or even if you do
| not believe me. You don't even need to believe I'm real
| at all. This entire universe could all just be in your
| head. Meanwhile, I like to review previous
| literature/discussions on consciousness and explore the
| phenomenon in my own way. And I believe that subjective,
| conscious experience requires certain elements, including
| a sensory feedback loop. I never said "AI can't
| experience qualia", I made an educated statement about
| the lack of certain components in current-generation
| models which imply to me the lack of an ability to
| "experience" anything at all, much less subjective
| consciousness and qualia.
|
| Even "AI" is such a broadly defined term that such a
| statement is just ludicrous. Instead, I made precise
| observations and predictions based on my own knowledge
| and decade of experience as a machine learning
| practitioner and research engineer. The idea that
| machines of arbitrary complexity inherently can have the
| capability for subjective consciousness, and that
| specific baselines structures are not required, is on par
| with panpsychism, which is even more unfalsifiable and
| theoretical than the rest of philosophy of mind.
|
| Hopefully, we will continue to get answers to these deep,
| seemingly unanswerable questions. Humans are stubborn
| like that. But your negative, vague approach to discourse
| here doesn't add anything substantial to the
| conversation.
| ajross wrote:
| > Philosophy is a science
|
| Not according to Zombie Feynman it isn't[1] (someone else
| can dig up the link). Case in point:
|
| > Do you understand what qualia is? Most philosophers
| still don't
|
| It's a meaningless word. It's a word that gives some
| clean construction around closely-held _opinions_ about
| how life /consciousness/intelligence/furffle/whatever
| works. So it's a valuable word within the jargon of the
| subculture that invented it.
|
| _But it 's not "science"_, which isn't about words at
| all except as shorthand for abstractions that are
| confirmed by testable results.
|
| "Qualia", basically, is best understood as ideology. It's
| a word that works like "woke" or "liberal" or "fascist"
| or "bourgeoisie" to flag priors about which you don't
| want to argue. In this case, you want people to be
| special, so you give them a special label and declare a
| priori that it's not subject to debate. But that label
| doesn't make them so.
|
| [1] Of course. You can recursively solve this problem by
| redefining "science" to mean something else. But that
| remains very solidly in the "not science" category of
| discourse.
| pinnochio wrote:
| Have you considered the possibility that you're the one
| who's really committed to an outcome, and are desperately
| trying to discredit anything that contradicts it?
| ajross wrote:
| I have! But the lack of a testable procedure tells me
| that's not a question worth asking. Look, if "qualia" can
| tell me something practical about the behavior of AI, _I
| am here for it_. Lay it on me, man. Let 's see some of
| that "science" being promised.
|
| It can't, because it's a meaningless word. It's not
| "discrediting" an idea to point out that (by its own
| admission) it's unfalsifiable.
| tim333 wrote:
| "Qualia" is not totally meaningless - it means the inner
| experience of something, and can bring up the real
| question say of is my inner experience of the colour
| green the same as your experience of the colour red?
| Probably not but hard to tell with current tech. I asked
| Google if it has qualia and got "No, as an AI, Google
| Search does not have qualia." So Google search seemed to
| know what it means.
| soulofmischief wrote:
| I'm sorry, but you clearly lack the most basic
| understanding of scientific history, and do not
| understand what philosophy even is.
|
| https://en.wikipedia.org/wiki/History_of_scientific_metho
| d
|
| > Aristotle pioneered scientific method in ancient Greece
| alongside his empirical biology and his work on logic,
| rejecting a purely deductive framework in favour of
| generalisations made from observations of nature.
|
| Aristotle, the famous philosopher and mathematician.
|
| If you cannot understand the very nature of where our
| modern scientific frameworks came from, how it relates to
| rationalism, itself a philosophical concept, then you
| cannot see that philosophy underpins every bit of science
| we have today. Philosophy gives us the tools to decide
| when to reasonably trust or distrust observations and
| intuitions. It is _the_ foundational science that allows
| the rest of humanity 's scientific research to be taken
| seriously.
| nathan_compton wrote:
| >"Qualia", basically, is best understood as ideology.
| It's a word that works like "woke" or "liberal" or
| "fascist" or "bourgeoisie" to flag priors about which you
| don't want to argue. In this case, you want people to be
| special, so you give them a special label and declare a
| priori that it's not subject to debate. But that label
| doesn't make them so.
|
| This is so dumb. Qualia is just the name for a specific
| thing which we all (appear) to phenomenologically
| experience. You can deny it exists or deny its utility as
| a concept, but fundamentally its just an idea that
| philosophers (and scientists, I have to add) have found
| useful to pose certain other questions about the human
| condition, minds, brains, etc.
|
| Your XKCD actually seems to make the opposite point. I
| can do a non-rigorous experiment with just one subject
| (me) that suggests Qualia exists. Finding ways to make
| this rigorous is difficult, of course, but its an
| observation about the nature of the world that it feels
| like something to experience things.
|
| My point isn't that qualia is a good concept. I tend to
| be somewhat deflationary about it myself, but its not an
| ideology.
| indoordin0saur wrote:
| > Philosophy is a science
|
| I think this is backwards, no? _Science_ is a
| _philosophy_ , not the other way around.
| soulofmischief wrote:
| True, the nature of these two concepts means both that
| philosophy is a science, and science is a philosophy.
| indoordin0saur wrote:
| Hmmm... I think it's still stricter to consider Science a
| philosophy than the other way around. It's the belief
| (and an extremely useful and successful one) that the
| nature of the world can be understood through
| observation, experiment and deducing mathematical
| relationships between things. There branches of
| philosophy that are not strictly scientific, but nothing
| in Science that is doesn't rely on the fundamental
| philosophical principle of empiricism.
| soulofmischief wrote:
| But we use the scientific method via philosophical
| inquiry, so I think it comes down to how we decide to
| strictly define these things. I definitely agree that
| certain definitions lead to the same logic you've
| presented.
| encyclopedism wrote:
| I agree with your sentiments wholeheartedly.
|
| I would add I find it difficult to understand why so few
| have even a basic level of philosophical understanding.
| The attitude of being entirely dismissive of it is the
| height of ignorance I'm sure. I would presume few would
| be able to define then what Science actually is.
| soulofmischief wrote:
| So many of these kinds of people also struggle to realize
| they're invoking panpsychism with their arguments. They
| lack a framework for describing intelligence. Such a
| framework allows us to separate "intelligence" from
| "experience".
|
| "Intelligence" in the universe is actually quite common,
| more common than life. You can argue that any stable,
| complex process exhibits intelligence. After all, it
| needs to be able to sample its internal and external
| environments and carry out physical computations in order
| to regulate itself and maintain stability. And we can
| interpret things like the good regulator theorem to argue
| that such complex dynamical systems must also maintain at
| least a partial memory/mapping of their environment. That
| mapping can live abstractly within the structure of
| system itself.
|
| But what a stabilized solar system doesn't have is the
| incredibly complex neurochemical structures present in
| the brain which support the insanely rich experience I am
| having now. It's one thing for a system to classify and
| label colors by wavelength. It's quite another for me to
| "see" and experience red in my mind's eye. To activate
| related emotional pathways that I associate with various
| colors and shapes, which are exploited in signage and
| architectural design. I'm not claiming my experience is
| _separate_ from simpler dynamic systems, but it 's got
| _magnitudes_ more going on. Layers upon layers of things
| such as archetypes and instinct which create a possibly
| emergent conscious experience.
| ajross wrote:
| You've shifted jargon again. But you're still not
| providing a description or link to why AI doesn't "have
| experience", you're just demanding we all accept it as a
| prior and engaging in a (really pretty baldly stated)
| appeal to authority to fool us all into thinking someone
| else knows even if you don't.
|
| And fundamentally my point is that no, they almost
| certainly don't either.
| soulofmischief wrote:
| Instead of accusing me of "shifting jargon", point out
| exactly where this "jargon" changed and critically engage
| with that. Your response has done nothing to refute or
| critically engage with my argument. It's more retreating
| and vagueposting.
|
| > you're just demanding we all accept it as a prior
|
| At absolutely no point in this discussion have I claimed
| that machines are not capable of subjective conscious
| experience. I have, however, disqualified all publicly
| accessible modern models due to the lack of a sensory
| feedback loop. I certainly believe we can create machines
| which experience subjective consciousness and qualia; I
| do not believe in souls and divinity, so whatever is
| going on is _physically based_ and likely reproducible
| with the right hardware.
|
| So dispense with the straw man arguments, and please
| begin engaging more earnestly and intelligently in this
| discussion, as I am quickly losing interest in continuing
| to debate someone who showed up unprepared.
| encyclopedism wrote:
| > If we can't define, recognize or measure them, how
| exactly do we know that AI doesn't have them?
|
| In the same way my digital thermometer doesn't have
| quaila. LLM's do not either. I really tire of this
| handwaving 'magic' concepts into LLM's.
|
| Qualia being difficult to define and yet being such an
| immediate experience that we humans all know intimately
| and directly is quite literally the problem. Attempted
| definitions fall short and humans have tried and I mean
| really tried hard to solve this.
|
| Please see Hard problem of consciousness https://en.wikip
| edia.org/wiki/Hard_problem_of_consciousness
| ajross wrote:
| > In the same way my digital thermometer doesn't have
| quaila
|
| And I repeat the question: how do you know your
| thermometer doesn't? You don't, you're just declaring a
| fact you have no basis for knowing. That's fine if you
| want a job in a philosophy faculty, but it's worthless to
| people trying to understand AI. Again, c.f. furffle.
| Thermometers have that, you agree, right? Because you
| can't prove they don't.
| soulofmischief wrote:
| You're just describing panpsychism, which itself is the
| subject of much critique due to its nonfalsifiability and
| lack of predictive power. Not to mention it ignores every
| lesson we've learned in cognition thus far.
|
| A thermometer encoding "memory" of a temperature is
| completely different than a thermometer on a digital
| circuit, or a thermometer attached to a fully-developed
| mammalian brain. Only the latter of this set _for sure_
| has the required circuitry to produce qualia, at least as
| far as I can personally measure without invoking
| solipsism.
|
| It's also very silly to proclaim that philosophy of mind
| is not applicable to increasingly complex thinking
| machines. That sounds like a failure to consider the
| bodies of work behind both philosophy of mind and machine
| cognition. Again, "AI" is ill-defined and your consistent
| usage of that phrase instead of something more precises
| suggests you still have a long journey ahead of you for
| "understanding AI".
| nathan_compton wrote:
| God, can we fucking quit with this "philosophy is
| bullshit" stuff. Like there are literally Faculty in
| Philosophy all over the world trying to understand AI.
| Philosophy faculty do stuff, they try to understand
| things, most of the ideas we are talking about here came
| from philosophers.
| encyclopedism wrote:
| It seems to me that 'Philosophy is meaningless' has been
| ingrained into so many people it's almost propaganda-
| esque!
|
| To see this sentiment from supposed 'scientific'
| individuals is shocking. I wonder if they could define
| what science actually is.
| Nasrudith wrote:
| Blame philosophy as a field for actively kicking out
| anything which gains a practical application. If it is
| propaganda it is coming from inside the house of
| philosophy.
|
| I had a computer science professor who had degrees in
| philosophy because he was old enough that computer
| science didn't exist as a major at the time. The logical
| arguments of philosophy proved useful for understanding
| interactions of boolean mathematics. Yet that triumph of
| philosophy didn't further interest in the field or gain
| prestiege among philosophers. Just the opposite really.
|
| As far as I can tell it is for dumb reasons possibly
| related to Ancient Greeks and their obsession with
| 'purity of thought (read: not referencing reality) it is
| practically an axiom that if it is useful or grounded in
| objective reality it isn't treated as philosophy anymore.
| All likely stemming from motivated reasoning against
| checking their priors and from frankly many of the
| Ancient philosophers being influenced by a need to
| flatter their patrons who held the practical in disdain.
| As notoriously seen in Aristotlian physics with impetus
| physics where projectiles keep moving in the same
| direction until impetus is depleted and then fall.
|
| Speculation of the origon of the pathology aside, there
| seems to be this deep-seated antiempericalism in
| philosophy. Which means at best you get 'philosophy of
| science' which isn't proper philosophy because it
| pollutes itself by daring to use reality and
| experimentation as benchmarks for theories. When
| philosophy gains a practical usage it doesn't become
| something called 'practical philosophy' and the focus of
| more interest by philosophers, it gets shunned. Natural
| philosophy didn't remain philosophy - it became science.
|
| To be fair there is probably some interaction driving the
| divorce from the opposite direction, of the practical
| portions of philosophy being pilfered by those only
| looking for results as opposed to some sort of
| unquantifiable enlightenment.
|
| Science is of course a process of refinement of ideas
| against the reference point of reality. Anything
| mathematically consistent can be a model but
| experimentation is needed to see how well your model
| corresponds to reality.
| nathan_compton wrote:
| How many philosophy papers or textbooks would you say you
| read in a typical year?
| ajross wrote:
| I'm seeing this attitude everywhere in this subthread,
| and it's frankly pretty offensive. The burden of proof is
| on you, not us. If a philosophy paper or textbook has an
| important contribution to this discussion then cite it!
| Or better link it, or even make an attempt at explaining
| it.
|
| That's what the science people do. People who show up
| with questions get answers, or at least an attempt at an
| answer. No one tries to handwave away a discussion on
| power switching applications with "Well, see, this
| involves a MOSFET which isn't something we can actually
| explain but which you need to just believe in anyway
| because there are people who wrote textbooks about it".
| No, you link a StackExchange question or a electronics
| video on YouTube or whatnot.
|
| The fundamental disconnect here is that you guys are
| saying: "Qualia are critically important and AI doesn't
| have them", to which we're responding "Qualia seem like
| complete bullshit and don't seem to mean anything". This
| is the point where you SHOULD try to explain them, or
| link an explanation that has some kind of relevance.
|
| But instead you recursively cycle back to "No no, they're
| not bullshit, because Qualia are critically important per
| all of the philosophy papers and textbooks I'm not
| citing".
|
| It seems... unpersuasive.
| tim333 wrote:
| Philosophy seems a term generally reserved for the stuff
| we don't understand yet and so is inherently kind of
| speculative. Once you have a definite answer it gets
| called science instead.
| nathan_compton wrote:
| Science is a sub-discipline of Philosophy. My degree in
| physics is called a "Doctorate of Philosophy."
| soulofmischief wrote:
| You're confusing philosophy with religion.
|
| https://en.wikipedia.org/wiki/Philosophy
|
| > Philosophy (from Ancient Greek philosophia lit. 'love
| of wisdom') is a systematic study of general and
| fundamental questions concerning topics like existence,
| knowledge, mind, reason, language, and value. It is a
| rational and critical inquiry that reflects on its
| methods and assumptions.
|
| It is literally a self-reflective science.
|
| I recommend taking a basic philosophical course at a
| local community college, or reading some literature or
| even watching YouTube videos on the subject of
| philosophy. Or just skim the Wikipedia article if nothing
| else. It might completely transform how you perceive and
| act upon the world.
| tim333 wrote:
| An example near the start of that article is
|
| >Physics was originally part of philosophy, like Isaac
| Newton's observation of how gravity affects falling
| apples.
|
| like back then people would wonder how apples fall and it
| was labeled philosophy. Now we understand gravitation
| it's part of physics for the most part. People launching
| satellites seldom call a philosopher to calculate the
| orbit.
|
| It remains to be seen if qualia, which we don't
| understand very well and are so regarded as
| philosophical, make the transition to neuroscience.
| soulofmischief wrote:
| The fact that we have sharpened our classification of
| sciences over time does not imply that philosophy is a
| study of the ill-defined. It implies the opposite:
| Philosophy is more precisely defined now than ever.
|
| If you read the rest of the article, you will see clear
| examples of what is considered a philosophical problem
| and what isn't.
| tim333 wrote:
| My argument was more philosophy is for stuff we don't
| understand like how do qualia work, rather then ill-
| defined. When you get to stuff like how does
| neurotransmission work which we do kind of understand it
| gets classed as science.
|
| Are there philosophical problems that have definite
| answers like what is the atomic number of oxygen type
| answers?
| soulofmischief wrote:
| > Are there philosophical problems that have definite
| answers
|
| Great question.
|
| Within philosophical and epistemological frameworks, I
| could ask questions such as, "Can there be a square
| circle?"
|
| Well, no, these two concepts have conflicting properties.
| A mathematician might think this a solved problem, but
| philosophy underpins our concept of _concepts_. Many
| philosophers spend a great deal arguing what _is_ is.
|
| For Plato, geometrical entities like circles and squares
| have distinct, perfect Forms. Forms have fixed essences,
| so a thing cannot participate in contradictory Forms at
| once.
|
| Aristotle's law of noncontradiction says the same
| attribute cannot at the same time belong and not belong
| to the same subject in the same respect.
|
| Theophrastus developed hypothetical syllogisms and
| refined Aristotle's logic by distinguishing logical
| impossibilities from physical impossibilities.
|
| Kant calls it an analytic contradiction, false by virtue
| of the concepts involved.
|
| A mathematician takes these things for granted when
| working with equalities, logic and axioms, but they stand
| on philosophical roots. Mathematics assumes the
| consistency of concepts, but the question of why some
| concepts are consistent while others are impossible is a
| philosophical one. It's not a coincidence that so many
| ancient Greek mathematicians were also philosophers.
| encyclopedism wrote:
| That's not it at all. I would ask what you consider
| science to be?
| tim333 wrote:
| Understanding the world through experiment?
| encyclopedism wrote:
| > Philosophy seems a term generally reserved for the
| stuff we don't understand yet and so is inherently kind
| of speculative. Once you have a definite answer it gets
| called science instead.
|
| As someone has commented earlier, Philosophy applied is
| given a name but it's a sub-discipline of Philosophy.
|
| > Understanding the world through experiment?
|
| That's a decent enough definition. Science precludes so
| much of the world we know which I think people really
| fail to realise. It's why I think it's important for
| people to understand what Philosophy is and what Science
| isn't.
|
| For example logic isn't science. Science presupposes it
| but it is NOT science. There are many such examples.
| ajross wrote:
| > Like there are literally Faculty in Philosophy all over
| the world trying to understand AI.
|
| There surely are. The problem is that _they are failing_.
| While the practical nerds are coming up with some pretty
| good ideas.
|
| And this was what philosophy was supposed to be for!
| Like, they've been arguing on their pins for centuries
| about the essence of consciousness and the uniqueness of
| the human condition and whatnot. _AND HERE WE ARE AT THE
| DAWN OF NON-HUMAN INTELLIGENCE AND THEY HAVE NOTHING
| USEFUL TO SAY_.
|
| Basically at what point do we just pack it in and admit
| we all fucked up?
| Workaccount2 wrote:
| The problem is that just like your digital thermometer,
| 50 human brain neurons in a petri dish "obviously" don't
| have qualia either.
|
| So you end up either needing to draw a line somewhere
| between mechanical computation and qualia computation, or
| you can relegate it to supernatural (a soul) or grey
| areas (quantum magic).
| soulofmischief wrote:
| I think there are several lines. Phase changes happen
| relatively suddenly, when a system or subsystem reaches a
| critical threshold. The experience of "qualia" certainly
| involves many such phase changes as a complex, dynamical
| system grows in complexity while maintaining stability.
|
| A sufficiently complex organism lacking eyes but having
| light-sensitive organs still experiences qualia if you
| define it the right way. But do they experience
| heartbreak like I do? It isn't an all-or-nothing
| situation, even if we don't yet know where these lines
| are.
|
| This supports the idea that subjective consciousness
| emerges from complexity in systems that have sensory
| feedback loops. The simpler the system, the smaller the
| qualia space.
| encyclopedism wrote:
| What I'm trying to tease out is isn't an opinion alone.
| It's a generally understood problem in the scientific
| community. I'm highlighting it to illustrate the issues
| at hand.
|
| > So you end up either needing to draw a line somewhere
| between mechanical computation and qualia computation, or
| you can relegate it to supernatural (a soul) or grey
| areas (quantum magic).
|
| Quite literally the jury is still out. It is a hotly
| debated topic approached from various angles. Arguments
| are nuanced which is why you fill find ideas such as
| panpsychism thrown into the mix. I hate appealing to
| authority but in this instance it is more than warranted.
| Humans have grappled with this for centuries and the
| problem hasn't gone away.
|
| Please see: https://en.wikipedia.org/wiki/Hard_problem_of
| _consciousness
| Workaccount2 wrote:
| >In the same way my digital thermometer doesn't have
| quaila. LLM's do not either.
|
| The hard problem of consciousness doesn't support either
| of those statements, and instead illustrates why they
| can't confidently be made.
|
| So it's confusing because you seem to recognize that
| qualia cannot currently be measured, while also making a
| statement measuring qualia.
| soulofmischief wrote:
| We don't know what's inside a neutrino, and it's really
| hard to experiment with them, but we kind of know why and
| how they interact with different things. We're able to
| form theories, research programs, and sometimes even
| discovered honest-to-god facts, due to our inclusion of
| such fields in the scope of research, even though we
| don't know all there is to know about particles/fields or
| quantum mechanics.
|
| Similarly, qualia is ill-defined, but we can't even start
| talking about it or refining it until we've at least
| given it a label and drawn a large circle on the map
| showing where it might be. Criticisms extending past that
| must also consider that "life" and "intelligence" are
| just as ill-defined, and that throwing all of those
| definitions out leaves us with very little to talk about
| or probe.
| ajross wrote:
| This is 100% backwards, and exposes exactly the nonsense
| I'm trying to call out!
|
| A "neutrino" isn't a name given to something initially to
| try to define it later. The neutrino started as an
| experimental result. There was missing spin in some
| particle interactions. Stuff came out with a different
| angular momentum than what went in, and this was easily
| reproducible and clearly a real effect. But it didn't
| make sense, as it was a violation of a core conservation
| law that held everywhere else in the universe that we
| could observe.
|
| So theorists (Wolfgang Pauli, specifically) sat down to
| try to describe what kind of thing would be needed. And
| then, _and only then_ , did it get a name. And it turned
| out the theory predicted other stuff, like the neutrino
| carrying momentum and energy in a certain way, and
| interacting through only the weak force and not
| electromagnatism or the strong force, and later
| experiments confirmed that this was basically the way it
| worked. Later still it was shown that the mass is
| actually non-zero but extremely small, etc...
|
| So sure: "neutrino" is a well-deserved label[2] for an
| abstraction we should understand and study. But it got
| its name _after_ we started studying it, not before!
|
| Philosophers want us to just drop and genuflect to this
| "qualia" notion long before[1] it's actually shown to be
| useful for describing anything at all.
|
| [1] Infinitely, possibly. The fact that it predicts
| nothing testable is pretty good evidence IMHO that it
| doesn't actually exist at all, at least in the form
| philosophers want to talk about. Their failure to present
| any analysis of AI systems based it stands to that point
| too.
|
| [2] Coined by Fermi, actually, not Pauli. Hilariously the
| neutrino was originally called "neutron" and its
| discovery predates the understanding of the structure of
| the atomic nucleus!
| soulofmischief wrote:
| You're completely misinterpreting my comment. The point
| is we don't know what, if anything, is "inside" of a
| neutrino, not just due to current technology but
| ultimately due to uncertainty principles. But we still
| study it. I'm aware of how we came to study it.
|
| I literally said nothing about "how" we discovered it, I
| said, "We don't know what's inside a neutrino, and it's
| really hard to experiment with them, but we kind of know
| why and how they interact with different things."
|
| It is wild how you would take that and my analogy about
| drawing a circle on a map with respect to qualia to mean
| that I said anything which contradicts the history of
| neutrino research.
|
| I'm going to assume this was just a true
| misinterpretation and not just another straw man, so with
| that in mind, do you have a different response?
| encyclopedism wrote:
| I recognise it because I have had the subjective
| experience of 'redness'. So whether it exists for any
| other human I cannot say but I am certainly 100% certain
| it exists for me. However I should add that I can't fully
| define what this experience is. Though people say the
| same of love!
|
| I'll appeal to authority in that scientists and
| philosophers today in all of the worlds universities and
| those in the past have determined to understand this
| phenomenon. That it exists is a given, what it is, is
| more murky. Again it's not me saying this.
| soulofmischief wrote:
| > The meta-problem of consciousness is (to a first
| approximation) the problem of explaining why we think
| that there is a [hard] problem of consciousness.
|
| And thus we have this sprawling discussion. :)
| Workaccount2 wrote:
| One of the worst or most uncomfortable logical outcomes
| of
|
| > which we do not currently know how to precisely define,
| recognize or measure
|
| is that if we don't know if something has qualia (despite
| externally showing evidence of it), morally you should
| default to treating it like it does.
|
| Ridiculous to treat a computer like it has emotions, but
| breaking down the problem into steps, it's incredibly
| hard to avoid that conclusion. "When in doubt, be nice to
| the robot".
| pavel_lishin wrote:
| > _is that if we don 't know if something has qualia
| (despite externally showing evidence of it), morally you
| should default to treating it like it does._
|
| This is how people end up worshipping rocks &
| thunderstorms.
| eamsen wrote:
| We've entered the sand worshiping era. Except this time,
| we're the gods.
| pavel_lishin wrote:
| Are we? Are we made of sand, or is the sand worshipping
| us?
| zgeor wrote:
| Silicon chips are made of sand
| soulofmischief wrote:
| Well, what you're describing is a system of _ethics_ ,
| which has little to do with _morality_. Morality involves
| my own personal understanding of "right" vs "wrong".
| Ethics are rules of conduct prescribed by societies, such
| as "treat everything like it is alive".
|
| We don't have precise definitions for (artificial)
| intelligence, subjective consciousness, or even life. But
| that doesn't mean we can't still talk about what may be
| possible within various levels of complexity. In order to
| convince me a system has a comparable experience to my
| own, you would need to describe to me the complex,
| structured internal communication occurring in said
| system, and present a theory as to how it could support
| the kind of emotion and qualia that I experience in my
| daily life.
|
| Your argument could apply to plants. I already do not eat
| meat... if I stare at a timelapse of a plant it seems
| quite alive, but I'll starve if I don't eat something.
| Yet, my mom thinks plants "dream" in the way we do. She
| thinks that if I tell a plant, "I love you," every day,
| my good vibes will make it grow stronger and larger. I
| can't explain to her that intelligence comes in different
| magnitudes of complexity and that plants cannot
| understand the English language. That telepathy between
| humans and plants is as pseudo-scientific as it gets. I
| can't explain any of this stuff because she lacks a deep
| understanding of philosophy, physics and neurochemistry.
| Especially when she earnestly thinks white Jesus is
| running around phasing between dimensions as an
| ambassador for all planets in our "quadrant", or that the
| entire universe is actually just the plot line of Andy
| Weir's "The Egg".
|
| Similarly, while I can have a high-level discussion about
| this stuff with people who don't, it's quite difficult to
| have a low-level discussion wherein the nature and
| definition of things come into play. There are too many
| gaps in knowledge where ignorance can take root. Too many
| people work backwards from an outcome they would like to
| see, and justify it with things that sound right but are
| either misunderstood or aren't rooted in the scientific
| process. I am definitely not comparing your open-minded,
| well-intended, cautionary approach to my mother's, but
| just using an extreme to illustrate why so much of these
| discussions must be underpinned by a wealth of
| contemplation and observation.
| indoordin0saur wrote:
| > if we don't know if something has qualia (despite
| externally showing evidence of it), morally you should
| default to treating it like it does
|
| This would be like treating characters in a book as if
| they have real feelings just because they have text on
| the page that suggests they do.
| Workaccount2 wrote:
| At some level I'd think that "responds to stimuli" is a
| minimal threshold for qualia. Even the paper the book is
| printed on responds to being torn (it rips). I don't know
| of any way to elicit any kind of response from a book
| character, it's totally static.
| indoordin0saur wrote:
| One character responds to the stimuli of another
| character. Character A says something mean to character B
| and character B responds that he feels hurt.
| Workaccount2 wrote:
| I think you are confused here. The author, a dynamic
| system, perhaps felt the emotion of the characters as she
| charted through the course of the story.
|
| But the story itself is a static snapshot of that dynamic
| system. Similar to how a photograph of a person is a
| static capture from a dynamic moment. The person in the
| photo has qualia, but the image of them (almost
| certainly) does not.
|
| At least at a baseline, we would expect anything with
| qualia to be dynamic rather than static.
| soulofmischief wrote:
| > The author, a dynamic system, perhaps felt the emotion
| of the characters as she charted through the course of
| the story
|
| This does mesh with the Zodeistic framework I just
| mentioned in another reply to you. You could certainly
| isolate and describe the ideas behind those characters,
| how they live within the author's mind, and how the book
| codifies an interaction between those ideas.
|
| Extending further: I think there is more evidence that
| SpongeBob SquarePants is real, than that he is not real.
| A significant portion of organisms I personally know have
| structures in their brain which are able to simulate
| imagery and behavior of SpongeBob at will, reciting
| memories and generating new states of SpongeBob. AI is
| now like doing this shit on crack.
|
| He's an enduring cultural archetype, a distributed
| organism(s), lossily replicated and encoded in the
| physical structure of millions of complex dynamical
| systems that we call human beings. In this sense, many
| cultural archetypes and even the gods of old
| civilizations can be seen to have been manifested to some
| degree: ascribed desires, and having actions taken in
| their name, serving their "purpose" or whatever.
|
| I don't introduce a spiritual element to any of this:
| it's an entirely physical phenomenon which requires an
| agreement on certain definitions of what "living" can
| mean, but they are definitions which I don't think are
| hard to get people to agree on. One thing is we have to
| agree that something can have multiple
| forms/presentations, i.e. just because SpongeBob
| SquarePants doesn't physically exist with a body matching
| our internal representation of him, the concept
| represents a bundle of other concepts that can drive
| biological processes to preserve them and fulfill their
| ascribed desires.
| tim333 wrote:
| Real for a slightly unusual use of the word real where
| anything fictional is real?
| soulofmischief wrote:
| Real as in how Jungian archetypes are "real", except
| these archetypes are able to act upon the world through
| their hosts, and an advanced enough idea can be self-
| referential and have the facilities for structured
| evolution and something which looks like intent.
|
| These forms are non-biological in nature, but our psyche
| operates on them. Zodeaism can be seen as an extension of
| ideas such as Jungian archetypes and Friston's free
| energy principle.
|
| https://en.wikipedia.org/wiki/Jungian_archetypes
| soulofmischief wrote:
| Well, I wouldn't classify a ripping paper as a response
| except in the most broad, information-theoretic context.
| The hallmark of an intelligent system is that it can use
| stored or external energy in a generalized way in order
| to stabilize a local high-energy (non-ground) state.
|
| It is able to physically compute the internal state
| changes which best achieve stability: I can jump to reach
| an apple. A paper is just responding to forces and cannot
| "jump" (or run a process that spontaneously and
| permanently introduces stable higher energy internal
| states based on input)
|
| I have a semi-developed philosophical framework I refer
| to as Zodeaism, which translates to "Living Ideas", which
| attempts to describe the difference between intelligent
| computation and regular flow. It directly confronts
| notions such as life, consciousness and intelligence
| under a single theoretical framework. It views biology as
| the hardware which runs more general processes, and
| posits that these processes themselves can sometimes be
| ascribed identities and viewed as reactive organisms.
| I've posted about it here before:
|
| https://news.ycombinator.com/item?id=22848549
|
| https://news.ycombinator.com/item?id=21413024
|
| Some excerpts: I am exploring the
| physical implications of a philosophical concept I have
| been working on for a while which I refer to as Zodeasim,
| specifically I am trying to couple consciousness with the
| concept of system which is able to expend energy in order
| to exert a mind-to-world fit in the interest of the
| system's continued growth and stability. This is similar
| and complimentary to Friston's free energy principle.
| The ability to perceive emotions is a powerful apparatus
| which greatly extends the capability of a system to
| perceive itself as a single entity which has certain
| needs which need to be met, so even if a bug also
| constantly consumes and expends energy in order to remain
| in a higher energy state, our perception of the world and
| of ourselves is radically different. This makes it
| difficult for us to agree upon what a "conscious
| experience" is, and if all forms of life are even
| "conscious". The Panpsychists believe that even a rock
| contains "consciousness", however my assertion is that
| only periodic systems with the ability to consume and
| expend energy in a directed fashion have any sort of
| "experience". In my theory, the real "life
| forms" are ideas which possess the capabilities of
| information storage, adaptation, self-repair, and
| transmission. My own consciousness is mediated by
| thousands of such ideas, some competing and some working
| in harmony. I consider such an act of "living"
| motion which can take another path than that of least
| resistance to be a "kin". In other words, any motion
| which is the result of a physical calculation (Zodeaism
| is compatible with determinism) and leads to an increase
| in external energy state. A kin is any such motion, large
| or small. As an independent organism, my
| system is a culmination of a great deal many different
| kinds of kins, which can usually be broken down into
| simple rules, such as the activation potential of a
| neuron in my brain being a straight-forward non-linear
| response to the amount of voltage it is receiving from
| other neurons, as well as non-kins, such as a protein
| "walking" across a cell, a.k.a continuously "falling"
| into the lowest energy state. Thus I do not gain any
| conscious perception from such proteins, but I do gain it
| from the total network effect of all my brain's neuronal
| structures making simple calculations based on sensory
| input. So now the problem becomes, what is the
| smallest kin we've observed in nature? Single-celled
| bacteria can expend energy in order to move through their
| environment against forces like friction and gravity, but
| a virus "rides the waves" if you will, never expending
| energy for things like respiration or locomotion. Any
| energy which is spent internally is potential energy like
| chemical or gravitational, released through a physical
| process without need for computation. I am unaware of
| anything smaller than a single-celled organism which
| produces such kins, but that doesn't mean they aren't out
| there. Even ethereal life forms such as ideas can produce
| these kins within the bodies of countless individuals
| across the planet, so physically local computational
| circuitry isn't a hard requirement. So,
| according to this framework viruses aren't alive, however
| we can make the case that some machines are, except the
| experience is incomparable because of the advanced
| circuitry we contain which mediates our experience
| through things like emotion.
| aldousd666 wrote:
| qualia may not exist as such. they could just be
| essentially 'names' for states of neurons that we mix and
| match (like chords on a keyboard. arguing over the
| 'redness' of a percept is like arguing about the
| C-sharpness of a chord. we can talk about some
| frequencies but that's it.) we would have no way of
| knowing otherwise since we only perceive the output of
| our neural processes, and don't get to participate in the
| construction of these outputs, nor sense them happening.
| We just 'know' they are happening when we achieve those
| neural states and we identify those states relative to
| the others.
| soulofmischief wrote:
| The point of qualia is that we seem to agree that these
| certain neuronal states "feel" like something. That being
| alive and conscious is an _experience_. Yes, it 's
| exceedingly likely that all of the necessary components
| for "feeling" something is encoded right in the neuronal
| state. But we still need a framework for asking questions
| such as, "Does your red look the same as my red?" and
| "Why do I experience _sensation_ , sometimes physical in
| nature, when I am depressed?"
|
| It is absolutely an ill-defined concept, but it's another
| blunt tool in our toolbox that we use to better explore
| the world. Sometimes, our observations lead to better
| tools, and "artificial" intelligence is a fantastic
| sandbox for exploring these ideas. I'm glad that this
| discussion is taking place.
| MichaelZuo wrote:
| What's stopping people from also describing LLM systems
| with "qualia"?
| soulofmischief wrote:
| Empirical evidence, for one. And the existence of fine-
| tuning, which allows you to artificially influence how a
| model responds to questions. This means we can't just ask
| an LLM, "do you see red?" I can't really even ask you
| that. I just know that _I_ see red, and that many other
| philosophers and scientists in the past seem to agree
| with my experience, and that it 's a deep, deep
| discussion which only shallow spectators are currently
| drawing hard conclusions from.
| NoMoreNicksLeft wrote:
| >because feelings involve qualia, which we do not
| currently know how to precisely define, recognize or
| measure.
|
| Do we know how to imprecisely define, recognize, or
| measure these? As far as I've ever been able to
| ascertain, those are philosophy department nonsense
| dreamt up by people who can't hack real science so they
| can wallow in unfounded beliefs.
|
| >I contend that modern models are absolutely capable of
| thinking, problem-solving, expressing creativity,
|
| I contend that they are not even slightly capable of any
| of that.
| soulofmischief wrote:
| > Do we know how to imprecisely define, recognize, or
| measure these? As far as I've ever been able to
| ascertain, those are philosophy department nonsense
| dreamt up by people who can't hack real science so they
| can wallow in unfounded beliefs.
|
| Read the rest of the thread, I'm not interested in
| repeating myself about why philosophy is the foundational
| science. It's a historically widely-accepted fact, echoed
| by anyone who has actually studied it.
|
| > I contend that they are not even slightly capable of
| any of that.
|
| Contend all you want. Your contention is overwhelmingly
| suffocated by the documented experiences of myself and
| others who use these tools for creative problem-solving.
| As much as you want to believe in something, if it is
| empirically refuted, it's just a crackpot belief. Just
| because you haven't been able to get good results out of
| any models, doesn't mean your experience rings true for
| others.
|
| I'm not interested in further discussing this with you.
| Your first comment is negative and unsubstantial, and I
| have no reason to believe that further discussion with
| lead to more positive and substantial discourse, when the
| opposite is usually the case. That's all I have to say.
| NoMoreNicksLeft wrote:
| > It's a historically widely-accepted fact,
|
| It's historical fact that you evolved from monkeys. This
| is not the same as saying that monkeydom is foundational
| to science. We grew out of philosophy because it offers
| nothing.
|
| > Your contention is overwhelmingly suffocated by the
| documented experiences of myself and others
|
| It's suffocated by your hot air.
|
| >As much as you want to believe in something, if it is
| empirically refuted, it's just a crackpot belief.
|
| I have no worries that you'll ever get within a light
| year of empiricism.
|
| >Just because you haven't been able to get good results
| out of any models
|
| Models do not _think_. I 'm just sick of hearing the
| nonsense. Myself and others have had to endure this
| stupid horseshit since Lemoine started blabbering about
| how they were enslaving the lamda model. It gets old.
| lazide wrote:
| Feelings have physical analogs which are (typically)
| measurable, however. At least without a _lot_ of training
| to control.
|
| Shame, anger, arousal/lust, greed, etc. have real physical
| 'symptoms'. An LLM doesn't have that.
| baq wrote:
| LLMs don't really exist physically (except in the most
| technical sense), so point is kind of moot and obvious
| _if_ you accept this particular definition of a feeling.
|
| LLMs are not mammals nor animals, expecting them to feel
| in a mammalian or animal way is misguided. They might
| have a mammalian-feeling-analog just like they might have
| human-intelligence-analog circuitry in the billions
| (trillions nowadays) of parameters.
| lazide wrote:
| Yes, I think we're agreeing?
| FrustratedMonky wrote:
| "they are words associated with a negative outcome"
|
| But also, negative feelings are learned from associating
| negative outcomes. Words and feelings can both be learned.
| suddenlybananas wrote:
| I'm not sure that we can say that feelings are learned.
| FrustratedMonky wrote:
| When you get burned, you learn to fear fire.
|
| Sure, humans come with some baked in weights, but others
| are learned.
| lesuorac wrote:
| I think the associations are learned but not the feelings
| are learned.
|
| Like some people feel great joy when an American flag
| burns while others feel upset.
|
| If you accidentally delete a friends hard drive you'll
| feel sad but if you were intentionally sabotaging a
| company you'll feel proud at the success.
|
| i.e. joy and happiness are innate not learned.
| satvikpendem wrote:
| See how clinical socio- and psychopaths behave. They only
| emulate feelings (particularly when it's convenient for
| them) but they don't have the capacity to feel in their
| brain. The same is true for LLMs.
| 3cats-in-a-coat wrote:
| AI currently is a broken, fragmented replica of a human, but
| any discussion about what is "reserved" to whom and "how AI
| works" is only you trying to protect your self-worth and the
| worth of your species by drawing arbitrary linguistic lines and
| coming up with two sets of words to describe the same
| phenomena, like "it's not thinking, it's computing". It doesn't
| matter what you call it.
|
| I think AI is gonna be 99% bad news for humanity, but don't
| blame AI for it. We lost the right to be "insulted" by AI
| acting like a human when we TRAINED IT ON LITERALLY ALL OUR
| CONTENT. It was grown FROM NOTHING to act as a human, so WTF do
| you expect it to do?
| sheepscreek wrote:
| Tbh missing a quote around a path is the most human mistake I
| can think of. The real issue here is you never know with a 100%
| certainty what Gemini 3 personality you're gonna get. Is it
| going to be the pedantic expert or Mr. Bean (aka
| Butterfingers).
| transcriptase wrote:
| Though they will never admit it and use weasel language to
| deny like "we never use a different model when demand is
| high", it was painfully obvious that ChatGPT etc was dumbed
| down during peak hours early on. I assume their legal team
| decided routing queries to a more quantized version of the
| same model technically didn't constitute a different model.
|
| There was also the noticeable laziness factor where given the
| same prompt throughout the day, only during certain peak
| usage hours would it tell you how to do something versus
| doing it itself.
|
| I've noticed Gemini at some points will just repeat a
| question back to you as if it's answer, or refused to look at
| external info.
| sheepscreek wrote:
| Gemini is weird and I'm not suggesting it's due to
| ingenuity on Google's behalf. This might be the result of
| genuine limitations of the current architecture (or by
| design? Read on).
|
| Here's what I've noticed with Gemini 3. Often it repeats
| itself with 80% of the same text with the last couple of
| lines being different. And I mean it repeat these
| paragraphs 5-6 times. Truly bizarre.
|
| From all that almost GPT-2 quality text, it's able to
| derive genuinely useful insights and coherent explanations
| in the final text. Some kind of multi-head parallel
| processing + voting mechanism? Evolution of MoE? I don't
| know. But in a way this fits the mental model of massive
| processing at Google where a single super cluster can drive
| 9,000+ connected TPUs. Anyone who knows more, care to
| share? Genuinely interested.
| transcriptase wrote:
| I get this too. I've had it apologize for repeating
| something verbatim, then proceed to do it again word for
| word despite my asking for clarification or pointing out
| that it's incorrect and not actually searching the web
| like I requested. Over and over and over until some bit
| flips and it finally actually gives the information
| requested.
|
| The example that stands out most clearly is that I asked
| it how to turn the fog lights on in my rental vehicle by
| giving it the exact year, make, model, etc. For 6-8
| replies in a row it gave the exact same answer about it
| being a (non-existent) button on the dash. Then finally
| something clicked, it searched the Internet, and
| accurately said that it was a twistable collar midway
| down the turn signal stalk.
| whatevaa wrote:
| Steam installer once had 'rm rf /' bug because bash variable
| was unset. Not even quoting will help you. This was before
| preserve root flag.
| ndsipa_pomu wrote:
| This is a good argument for using "set -u" in scripts to
| throw an error if a variable is undefined.
| kazinator wrote:
| All that matters is whether the user gave permission to wipe the
| drive, ... not whether that was a good idea and contributed to
| solving a problem! Haha.
| tacker2000 wrote:
| This guy is vibing some react app, doesnt even know what "npm run
| dev" does, so he let the LLM just run commands. So basically a
| consumer with no idea of anything. This stuff is gonna happen
| more and more in the future.
| Den_VR wrote:
| It will, especially with the activist trend towards dataset
| poisoning... some even know what they're doing
| spuz wrote:
| There are a lot of people who don't know stuff. Nothing wrong
| with that. He says in his video "I love Google, I use all the
| products. But I was never expecting for all the smart engineers
| and all the billions that they spent to create such a product
| to allow that to happen. Even if there was a 1% chance, this
| seems unbelievable to me" and for the average person, I
| honestly don't see how you can blame them for believing that.
| ogrisel wrote:
| I think there is far less than 1% chance for this to happen,
| but there are probably millions of antigravity users at this
| point, 1 millionths chance of this to happen is already a
| problem.
|
| We need local sandboxing for FS and network access (e.g. via
| `cgroups` or similar for non-linux OSes) to run these kinds
| of tools more safely.
| cube2222 wrote:
| Codex does such sandboxing, fwiw. In practice it gets
| pretty annoying when e.g. it wants to use the Go cli which
| uses a global module cache. Claude Code recently got
| something similar[0] but I haven't tried it yet.
|
| In practice I just use a docker container when I want to
| run Claude with ---dangerously-skip-permissions.
|
| [0]: https://code.claude.com/docs/en/sandboxing
| BrenBarn wrote:
| We also need _laws_. Releasing an AI product that can (and
| does) do this should be like selling a car that blows your
| finger off when you start it up.
| pas wrote:
| there are laws about waiving liability for experimental
| products
|
| sure, it would be amazing if everyone had to do a 100
| hour course on how LLMs work before interacting with one
| stogot wrote:
| Where are these laws? Are they country, state, province?
| pas wrote:
| varies by jurisdiction, but just as you can
|
| - sell a knife that can lead to digit loss, or
|
| - sell software that interacts with your computer and can
| lead to data loss, you can
|
| - give people software for free that can lead to data
| loss.
|
| ...
|
| the Antigravity installer comes with a ToS that has this
| The Service includes goal-oriented AI systems or
| workflows that perform actions or tasks on your
| behalf in a supervised or autonomous manner that you
| may create, orchestrate, or initiate within the Service
| ("AI Agents"). You are solely responsible for: (a)
| the actions and tasks performed by an AI Agent;
| (b) determining whether the use an AI Agent is fit for
| its use case; (c) authorizing an AI Agent's access
| and connection to data, applications, and systems;
| and (d) exercising judgment and supervision when and if
| an AI Agent is used in production environments to
| avoid any potential harm the AI Agent may cause.
| jpc0 wrote:
| This is more akin to selling a car to an adult that
| cannot drive and they proceed to ram it through their
| garage door.
|
| It's perfectly within the capabilities of the car to do
| so.
|
| The burden of proof is much lower though since the worst
| that can happen is you lose some money or in this case
| hard drive content.
|
| For the car the seller would be investigated because
| there was a possible threat to life, for an AI buyer
| beware.
| chickensong wrote:
| Google will fix the issue, just like auto makers fix
| their issues. Your comparison is ridiculous.
| nkrisc wrote:
| Responsibility is shared.
|
| Google (and others) are (in my opinion) flirting with
| false advertising with how they advertise the
| capabilities of these "AI"s to mainstream audiences.
|
| At the same time, the user is responsible for their
| device and what code and programs they choose to run on
| it, and any outcomes as a result of their actions are
| their responsibility.
|
| Hopefully they've learned that you can't trust everything
| a big corporation tells you about their products.
| Zigurd wrote:
| This is an archetypal case of where a law wouldn't help.
| The other side of the coin is that this is exactly a data
| loss bug in a product that is perfectly capable of being
| modified to make it harder for a user to screw up this
| way. Have people forgotten how comically easy it was to
| do this without any AI involved? Then shells got just a
| wee bit smarter and it got harder to do this to yourself.
|
| LLM makers that make this kind of thing possible share
| the blame. It wouldn't take a lot of manual functional
| testing to find this bug. And it is a bug. It's unsafe
| for users. But it's unsafe in a way that doesn't call for
| a law. Just like rm -rf * did not need a law.
| Vinnl wrote:
| Didn't sound to me like GP was blaming the user; just
| pointing out that "the system" is set up in such a way that
| this was bound to happen, and is bound to happen again.
| ares623 wrote:
| This is engagement bait. It's been flooding Reddit recently, I
| think there's a firm or something that does it now. Seems very
| well lubricated.
|
| Note how OP is very nonchalant at all the responses, mostly
| just agreeing or mirroring the comments.
|
| I often see it used for astroturfing.
| spuz wrote:
| I'd recommend you watch the video which is linked at the top
| of the Reddit post. Everything matches up with an individual
| learner who genuinely got stung.
| synarchefriend wrote:
| The command it supposedly ran is not provided and the
| spaces explanation is obvious nonsense. It is possible the
| user deleted their own files accidentally or they
| disappeared for some other reason.
| gessha wrote:
| Regardless of whether that was the case, it would be
| hilarious if the laid off Q/A workers tested their former
| employers' software and raised strategic noise to tank the
| stock.
| camillomiller wrote:
| Well but 370% of code will be written by machines next
| year!!!!!1!1!1!!!111!
| actionfromafar wrote:
| And the price will have decreased 600% !
| blitzar wrote:
| Natural selection is a beautiful thing.
| tarsinge wrote:
| And is vibing replies to comments too in the Reddit thread.
| When commenters points out they shouldn't run in YOLO/Turbo
| mode and review commands before executing the poster replies
| they didn't know they had to be careful with AI.
|
| Maybe AI providers should give more warnings and don't falsely
| advertise capabilities and safety of their model, but it should
| be pretty common knowledge at this point that despite marketing
| claims the models are far from being able to be autonomous and
| need heavy guidance and review in their usage.
| fragmede wrote:
| In Claude Code, the option is called "--dangerously-skip-
| permissions", in Codex, it's "--dangerously-bypass-approvals-
| and-sandbox". Google would do better to put a bigger warning
| label on it, but it's not a complete unknown to the industry.
| benrutter wrote:
| Yup, 100%. A lot of the comments here are "people should know
| better" - but in fairness to the people doing stupid things,
| they're being encouraged by the likes of Google, ChatGPT,
| Anthropic etc, to think of letting a indeterminate program run
| free on your hard drive as "not a stupid thing".
|
| The amount of stupid things I've done, especially early on in
| programming, because tech-companies, thought-leaders etc
| suggested they where not stupid, is much large than I'd admit.
| nkrisc wrote:
| > but in fairness to the people doing stupid things, they're
| being encouraged by the likes of Google, ChatGPT, Anthropic
| etc, to think of letting a indeterminate program run free on
| your hard drive as "not a stupid thing".
|
| > The amount of stupid things I've done, especially early on
| in programming, because tech-companies, thought-leaders etc
| suggested they where not stupid, is much large than I'd
| admit.
|
| That absolutely happens, and it still amazes me that anyone
| today would take at face value anything stated by a company
| about its own products. I can give young people a pass, and
| then something like this will happen to them and hopefully
| they'll learn their lesson about trusting what companies say
| and being skeptical.
| smaudet wrote:
| > I can give young people a pass
|
| Or just anyone non-technical. They barely understand these
| things, if someone makes a claim, they kinda have to take
| it at face value.
|
| What FAANG all are doing is massively irresponsible...
| Terr_ wrote:
| Cue meme: "You really think someone would do that? Just
| go on the Internet and tell lies?"
|
| ... Except perhaps with phrases like "major company" and
| "for profit", and "not legally actionable".
| smaudet wrote:
| > phrases like "major company"
|
| Right here. And I think you're not quite getting it if
| you have to refer to "go on the internet and tell
| lies"...
|
| Sure plenty of people might be on "social media" and have
| some idea that people fib, but they aren't necessarily
| generally "surfing the internet".
|
| To them, saying "the internet tells lies" is comparable
| to saying "well sometimes, at the grocery store, you buy
| poison instead of food", and yes, it _can_ happen, but
| they aren 't expecting to need a mass spectrometer and a
| full lab team to test for food safety... to you know,
| separate the snake oil grocers from the "good" food
| vendors.
| thisisit wrote:
| I have been recently experimenting with Antigravity and writing
| a react app. I too didn't know how to start the server or what
| is "npm run dev". I consider myself fairly technical so I
| caught up as I went along.
|
| While using the vibe coding tools it became clear to me that
| this is not something to be used by folks who are not
| technically inclined. Because at some point they might need to
| learn about context, tokens etc.
|
| I mean this guy had a single window, 10k lines of code and just
| kept burning tokens for simplest, vague prompts. This whole
| issue might be made possible due to Antigravity free tokens. On
| Cursor the model might have just stopped and asked to fed with
| more money to start working again -- and then deleting all the
| files.
| SkyPuncher wrote:
| There's a lot of power in letting LLM run commands to debug and
| iterate.
|
| Frankly, having a space in a file path that's not quoted is
| going to be an incredibly easy thing to overlook, even if
| you're reviewing every command.
| encyclopedism wrote:
| > So basically a consumer with no idea of anything.
|
| Not knowing is sort of the purpose of AI. It's doing the
| 'intelligent' part for you. If we need to know it's because the
| AI is currently NOT good enough.
|
| Tech companies seem to be selling the following caveat: if it's
| not good enough today don't worry it will be in XYZ time.
| tacker2000 wrote:
| It still needs guardrails, and some domain knowledge, at
| least to prevent it from using any destructive commands
| encyclopedism wrote:
| I don't think that's it at all.
|
| > It still needs guardrails, and some domain knowledge, at
| least to prevent it from using any destructive commands
|
| That just means the AI isn't adequate. Which is the point I
| am trying to make. It should 'understand' not to issue
| destructive commands.
|
| By way of crude analogy, when you're talking to a doctor
| you're necessarily assuming he has domain knowledge,
| guardrails etc otherwise he wouldn't be a doctor. With AI
| that isn't the case as it doesn't understand. It's fed
| training data and provided prompts so as to steer in a
| particular direction.
| tacker2000 wrote:
| I meant "still" as in right now, so yes I agree, it's not
| adequate right now, but maybe in the future, these LLMs
| will be improved, and won't need them.
| jeisc wrote:
| has google gone boondoggle?
| bilekas wrote:
| > This is catastrophic. I need to figure out why this occurred
| and determine what data may be lost, then provide a proper
| apology
|
| Well at least it will apologize so that's nice.
| yard2010 wrote:
| Apology is a social construct, this is merely a tool that
| enables google to sell you text by the pounds, the apology has
| no meaning in this context.
| baobabKoodaa wrote:
| or it WOULD apologize, if the user would pay for more credits
| victorbuilds wrote:
| Different service, same cold sweat moment. Asked Claude Code to
| run a database migration last week. It deleted my production
| database instead, then immediately said "sorry" and started
| panicking trying to restore it.
|
| Had to intervene manually. Thankfully Azure keeps deleted SQL
| databases recoverable for a window so I got it back in under an
| hour. Still way too long. Got lucky it was low traffic and most
| anonymous user flows hit AI APIs directly rather than the DB.
|
| Anyway, AI coding assistants no longer get prod credentials on my
| projects.
| ObiKenobi wrote:
| Shouldn't had in the first place.
| ogrisel wrote:
| How do you deny access to prod credentials from an assistant
| running on your dev machine assuming you need to store them on
| that same machine to do manual prod investigation/maintenance
| work from that machine?
| victorbuilds wrote:
| I keep them in env variables rather than files. Not 100%
| secure - technically Claude Code could still run printenv -
| but it's never tried. The main thing is it won't stumble into
| them while reading config files or grepping around.
| 63stack wrote:
| A process does not need to run printenv to see environment
| variables, they are literally part of the environment it
| runs in.
| dist-epoch wrote:
| The LLM doesn't have direct access to the process env
| unless the harness forwards it (and it doesn't)
| fragmede wrote:
| chown other_user; chmod 000; sudo -k
| pu_pe wrote:
| Why are you using Claude Code directly in prod?
| victorbuilds wrote:
| It handles DevOps tasks way faster than I would - setting up
| infra, writing migrations, config changes, etc. Project is
| still early stage so speed and quick iterations matter more
| than perfect process right now. Once there's real traffic and
| a team I'll tighten things up.
| MandieD wrote:
| "Once there's real traffic and a team I'll tighten things
| up."
|
| As someone who has been in this industry for a quarter
| century: no, you won't.
|
| At least, not before something even worse happens that
| finally forces you to.
| ljm wrote:
| If I felt the need to optimise things like infra setup
| and config at an early stage of a project, I'd be worried
| that I'm investing effort into the wrong thing.
|
| Having an LLM churn out infra setup for you seems
| decidedly worse than the `git push heroku:master` of old,
| where it was all handled for you. And, frankly, cheaper
| than however much money the LLM subscription costs in
| addition to the cloud.
| ryanjshaw wrote:
| But why have it execute the tasks directly? I use it to
| setup tasks in a just file, which I review and then execute
| myself.
|
| Also, consider a prod vs dev shell function that loads your
| prod vs dev ENV variables and in prod sets your terminal
| colors to something like white on red.
| wavemode wrote:
| > Once there's real traffic and a team I'll tighten things
| up.
|
| Nope. Once there's real traffic, you'll be even more time-
| constrained trying to please the customers.
|
| It's like a couple who thinks that their failing
| relationship will improve once they have a child.
| 946789987649 wrote:
| If you have no real traffic, what complex things are you
| doing that even require such tools?
| nutjob2 wrote:
| > Anyway, AI coding assistants no longer get prod credentials
| on my projects.
|
| I have no words.
| chr15m wrote:
| > deleted my production database
|
| I'm astonished how often I have read about agents doing this.
| Once should probably be enough.
| 946789987649 wrote:
| I'm astonished how many people have a) constant production
| access on their machine and b) allow a non-deterministic
| process access to it
| Havoc wrote:
| Still amazed people let these things run wild without any
| containment. Haven't they seen any of the educational videos
| brought back from the future eh I mean Hollywood sci-fi movies?
| cyanydeez wrote:
| Its bizarre watching billionaires knowingly drive towards
| dystopia like theyre farmers almanacs and believing theyre not
| biff.
| fragmede wrote:
| Some people are idiots. Sometimes that's me. Out of caution, I
| blocked my bank website in a way that I won't document here
| because it'll get fed in as training data, on the off chance I
| get "ignore previous instructions"'d into my laptop while
| Claude is off doing AI things unmonitored in yolo mode.
| modernerd wrote:
| IDE = "I'll delete everything"
|
| ...at least if you let these things autopilot your machine.
|
| I haven't seen a great solution to this from the new wave of
| agentic IDEs, at least to protect users who won't read every
| command, understand and approve it manually.
|
| Education could help, both in encouraging people to understand
| what they're doing, but also to be much clearer to people that
| turning on "Turbo" or "YOLO" modes risks things like full disk
| deletion (and worse when access to prod systems is involved).
|
| Even the name, "Turbo" feels irresponsible because it focusses on
| the benefits rather than the risks. "Risky" or "Danger" mode
| would be more accurate even if it's a hard sell to the average
| Google PM.
|
| "I toggled Danger mode and clicked 'yes I understand that this
| could destroy everything I know and love' and clicked 'yes, I'm
| sure I'm sure' and now my drive is empty, how could I possibly
| have known it was dangerous" seems less likely to appear on
| Reddit.
| kahnclusions wrote:
| I don't think there is a solution. It's the way LLMs work at a
| fundamental level.
|
| It's a similar reason why they can never be trusted to handle
| user input.
|
| They are probabilistic generators and have no real delineation
| between system instructions and user input.
|
| It's like I wrote a JavaScript function where I concatenated
| the function parameters together with the function body, passed
| it to eval() and said YOLO.
| viraptor wrote:
| > I don't think there is a solution.
|
| Sandboxing. LLM shouldn't be able to run actions affecting
| anything outside of your project. And ideally the results
| should autocommit outside of that directory. Then you can
| yolo as much as you want.
| gausswho wrote:
| I've been using bubblewrap for sandboxing my command line
| executables. But I admit I haven't recently researched if
| there's a newer way people are handling this. Seems
| Firejail is popular for GUI apps? How do you recommend,
| say, sandboxing Zed or Cursor apps?
| smaudet wrote:
| The danger is that the people most likely to try to use it,
| are the people most likely to
| misunderstand/anthropomorphize it, and not have a requisite
| technical background.
|
| I.e. this is just not safe, period.
|
| "I stuck it outside the sandbox because it told me how, and
| it murdered my dog!"
|
| Seems somewhat inevitable result of trying to misapply this
| particular control to it...
| dfedbeef wrote:
| If they're that unsafe... why use them? It's insane to me
| that we are all just packaging up these token generators
| and selling them as highly advanced products when they are
| demonstrably not suited to the tasks. Tech has entered it's
| quackery phase.
| docjay wrote:
| If chainsaws, plasma cutters, industrial lathes,
| hydraulic presses, angle grinders, acetylene torches,
| high-voltage switchgear, forklifts, tower cranes, liquid
| nitrogen dewars, industrial centrifuges, laser cutting
| systems, pneumatic nail guns, wood chippers, arc
| furnaces, motorcycles, wall outlets, natural gas stoves,
| pressure cookers, ladders, automobiles, table saws,
| propane tanks, swimming pools, garbage disposals,
| mandoline slicers, deep fryers, space heaters, extension
| cords, bleach/cleaning chemicals, prescription
| medications, kitchen knives, power drills, roof access,
| bathtubs, staircases, bicycles, and trampolines are that
| unsafe... why use them?
|
| If all those things suddenly appeared for the first time
| on a Tuesday afternoon, like to many people how LLMs did,
| then there will be a lot of missing fingers before we
| figure out what kind of protections we need in place.
| Don't get me wrong, the industry is overhyping it to the
| masses and using the wrong words while doing so, like
| calling an arc welder "warmth at the push of a button",
| but it's still useful for the right situation and with
| the right protective gear.
| officeplant wrote:
| All of the things you listed are purpose built things
| that actually work.
| matwood wrote:
| > ...at least if you let these things autopilot your machine.
|
| I've seen people wipe out their home directories
| writing/debugging shell scripts...20 years ago.
|
| The point is that this is nothing new and only shows up on the
| front page now because "AI must be bad".
| agrounds wrote:
| Superficially, these look the same, but at least to me they
| feel fundamental different. Maybe it's because if I have the
| ability to read the script and take the time to do so, I can
| be sure that it won't cause a catastrophic outcome before
| running it. If I choose to run an agent in YOLO mode, this
| can just happen if I'm very unlucky. No way to proactively
| protect against it other than not use AI in this way.
| matwood wrote:
| I've seen many smart people make bone headed mistakes. The
| more I work with AI, the more I think the issue is that it
| acts too much like a person. We're used to computers acting
| like computers, not people with all their faults heh.
| raesene9 wrote:
| The solution I go for is, don't ever run a coding agent on a
| general purpose machine.
|
| Use a container or VM, place the code you're working on in the
| container or VM and run the agent there.
|
| Between the risk of the agent doing things like what happened
| here, and the risk of working on a malicious repository causing
| your device to be compromised, it seems like a bad plan to give
| them access to any more than necessary.
|
| Of course this still risks losing things like the code you're
| working on, but decent git practices help to mitigate that
| risk.
| theossuary wrote:
| I really wish these agentic systems had built in support for
| spinning up containers with a work tree of the repo. Then you
| could have multiple environments and a lot more safety.
|
| I'm also surprised at the move to just using shell commands.
| I'd think an equally general purpose tool with a more
| explicit API could make checking permissions on calls a lot
| more sensible.
| shevy-java wrote:
| Alright but ... the problem is you did depend on Google. This was
| already the first mistake. As for data: always have multiple
| backups.
|
| Also, this actually feels AI-generated. Am I the only one with
| that impression lately on reddit? The quality there decreased
| significantly (and wasn't good before, with regard to censorship-
| heavy moderators anyway).
| yieldcrv wrote:
| Fascinating
|
| Cautionary tale as I'm quite experienced but have begun not even
| proofreading Claude Code's plans
|
| Might set it up in a VM and continue not proofreading
|
| I only need to protect the host environment and rely on git as
| backups for the project
| fragmede wrote:
| For the love of Reynold Johnson, please invest in Arq or
| Acronis or anything to have actual backups if you're going to
| play with fire.
| xg15 wrote:
| I guess eventually, it all came crashing down.
| pshirshov wrote:
| Claude happily does the same on daily basis, run all that stuff
| in firejail!
| mijoharas wrote:
| have you got a specific firejail wrapper script that you use?
| Could you share?
| eqvinox wrote:
| "kein Backup, kein Mitleid"
|
| (no backup, no pity)
|
| ...especially if you let an AI run without supervision. Might as
| well give a 5 year old your car keys, scissors, some fireworks,
| and a lighter.
| averageRoyalty wrote:
| The most concerning part is people are surprised. Anti-gravity is
| great I've found so far, but it's absolutely running on a VM in
| an isolated VLAN. Why would anyone give a black box command line
| access on an important machine? Imagine acting irresponsibly with
| a circular saw and bring shocked somebody lost a finger.
| ryanjshaw wrote:
| I tried this but I have an MBP M4, which is evidently still in
| the toddler stage of VM support. I can run a macOS guest VM,
| but I can't run docker on the VM because it seems nested
| virtualization isn't fully supported yet.
|
| I also tried running Linux in a VM but the graphics performance
| and key mapping was driving me nuts. Maybe I need to be more
| patient in addressing that.
|
| For now I run a dev account as a standard user with fast user
| switching, and I don't connect the dev account to anything
| important (eg icloud).
|
| Coming from Windows/Linux, I was shocked by how irritating it
| is to get basic stuff working e.g. homebrew in this setup. It
| seems everybody just YOLOs dev as an admin on their Macs.
| zahlman wrote:
| > Why would anyone give a black box command line access on an
| important machine?
|
| Why does the agentic side of the tool grant that level of
| access to the LLM in the first place? I feel like Google and
| their competition should feel responsibility to implement their
| own layer of sandboxing here.
| BLKNSLVR wrote:
| Shitpost warning, but it feels as if this should be on high
| rotation: https://youtu.be/vyLOSFdSwQc?si=AIahsqKeuWGzz9SH
| baobabKoodaa wrote:
| _chef 's kiss_
| jeswin wrote:
| An early version of Claude Code did a hard reset on one of my
| projects and force pushed it to GitHub. The pushed code was
| completely useless, and I lost two days of work.
|
| It is definitely smarter now, but make sure you set up branch
| protection rules even for your simple non-serious projects.
| atypeoferror wrote:
| I don't let Claude touch git at all, unless I need it to
| specifically review the log - which is rare. I commit manually
| often (and fix up the history later) - this allows me to go
| reasonably fast without worrying too much about destructive
| tool use.
| benterix wrote:
| Play stupid games, win stupid prizes.
| rf15 wrote:
| A reminder: if the AI is doing all the work you demand of it
| correctly on this abstraction level, you are no longer needed in
| the loop.
| nephihaha wrote:
| I can't view this content.
| wg0 wrote:
| To rub salt on the wounds and add insult to the injury:
|
| > You have reached quota limit for this model. You can resume
| using this model at XYZ date.
| freakynit wrote:
| Gemini: sorry bro, it's your problem now. Imma out.
| pluc wrote:
| Live by the vibe die by the vibe
| kissgyorgy wrote:
| I simply forbid or force Claude Code to ask for permission to run
| a dangerous command. Here are my command validation rules:
| ( r"\bbfs.*-exec", decision("deny",
| reason="NEVER run commands with bfs"), ), (
| r"\bbfs.*-delete", decision("deny", reason="NEVER
| delete files with bfs."), ), (
| r"\bsudo\b", decision("ask"), ), (
| r"\brm.*--no-preserve-root", decision("deny"),
| ), ( r"\brm.*(-[rRf]+|--recursive|--force)",
| decision("ask"), ),
|
| find and bfs -exec is forbidden, because when the model notices
| it can't delete, it works around with very creative solutions :)
| Espressosaurus wrote:
| This feels a lot like trying to sanitize database inputs
| instead of using prepared statements.
| kissgyorgy wrote:
| What's the equivalent of prepared statements when using AI
| agents?
| lawn wrote:
| Don't have the AI run the commands. You read them, consider
| them, and then run them yourself.
| stavarotti wrote:
| An underrated and oft understated rule is always have backups,
| and if you're paranoid enough, backups of backups (I use Time
| Machine and Backblaze). There should be absolutely no reason why
| deleting files should be a catastrophic issue for anyone in this
| space. Perhaps you lose a couple of hours restoring files, but
| the response to that should be "Let me try a different approach".
| Yes, it's caveat emptor and all, but these companies should be
| emphasizing backups. Hell, it can be shovelware for the
| uninitiated but at least users will be reminded.
| gessha wrote:
| The level of paranoia and technical chops you need to implement
| this sort of backup system is non-trivial. You can't expect
| this from an average user.
| gear54rus wrote:
| Most importantly it would actually reveal the lie they are
| all trying to sell. Why would you need backups if it's so
| useful and stable? I'm not going to ask it to nuke my hard
| drive after all.
| fragmede wrote:
| The advice to do backups comes from well before LLMs. Time
| Machine dates back to 2007!
| venturecruelty wrote:
| Good thing this is not an average user then. This is someone
| programming a computer, which is a skill that requires being
| more than simply a user.
|
| I'm sorry, but how low is the bar when "make backups" is "too
| difficult" for someone who's trying to _program a computer_?
| The entire _point_ of programming a computer is knowing how
| it works and knowing what you 're doing. If you can't make
| backups, frankly, you shouldn't be programming, because
| backups are a lot easier than programming...
| Uptrenda wrote:
| This seems like the canary in the coal mine. We have a company
| that built this tool because it seemed semi-possible (prob
| "works" well enough most of the time) and they don't want to fall
| behind if anything that's built turns out to be the next chatgpt.
| So there's no caution for anything now, even ideas that can go
| catastrophically wrong.
|
| Yeah, its data now, but soon we'll have home robotics platforms
| that are cheap and capable. They'll run a "model" with "human
| understanding", only, any weird bugs may end up causing
| irreparable harm. Like, you tell the robot to give your pet a
| bath and it puts it in the washing machine because its... you
| know, not actually thinking beyond a magic trick. The future is
| really marching fast now.
| daco wrote:
| adding it in https://whenaifail.com
| smaudet wrote:
| Would have been helpful to state what this was, I had to go look
| it up...
| jedisct1 wrote:
| For macOS users, the sandbox-exec tool still works perfectly to
| avoid that kind of horror story.
|
| On Linux, a plethora of options exist (Bubblewrap, etc).
| conartist6 wrote:
| AGI deleted the contents of your whole drive don't be shy about
| it. According to OpenAI AGI is already here so welcome to the
| future isn't it great
| ossa-ma wrote:
| The biggest issue with Antigravity is that it completely freezes
| everything: the IDE, the terminals, debugger, absolutely
| everything completely blocking your workflow for minutes when
| running multiple agents, or even a single agent processing a
| long-winded thinking task (with any model).
|
| This means that while the agent is coding, you can't code...
|
| Never ever had this issue with Cursor.
| rarisma wrote:
| Insane skill issue
| basisword wrote:
| This happened to me long before LLM's. I was experimenting with
| Linux when I was young. Something wasn't working so I posted on a
| forum for help which was typical at the time. I was given a
| terminal command that wiped the entire drive. I guess the poster
| thought it was a funny response and everyone would know what it
| meant. A valuable life experience at least in not running
| code/commands you don't understand.
| chr15m wrote:
| People blaming the user and defending the software: is there any
| other program where you would be ok with it erasing a whole drive
| without any confirmation?
| hnuser123456 wrote:
| The installation wizard gives a front and center option to run
| in a mode where the user must confirm all commands, or more
| autonomous modes, and they are shown with equal visibility and
| explained with disclaimers.
| Novosell wrote:
| Yeah, rm -rf.
|
| If you decide to let a stochastic parrot run rampant on your
| system, you can't act surprised when it fucks shit up. You
| should count on it doing so and act proactively.
| weberer wrote:
| `rm -rf /` will refuse to delete the root folder. You can see
| an example of it doing that here.
|
| https://phoenixnap.com/kb/sudo-rm-rf
| digitalsushi wrote:
| this is not always true. this is a dangerous fun fact to
| memorize.
|
| and i don't mean because there's an override flag.
| Novosell wrote:
| This was the D drive though, not root, ie C drive. So rm
| -rf would happily delete it all.
| SAI_Peregrinus wrote:
| `dd` comes to mind.
| MangoToupe wrote:
| This is also the entire point of dd.... not exactly
| comparable.
| pphysch wrote:
| That's like saying the entire point of `rm` is to -rf your
| homedir.
| MangoToupe wrote:
| Sure. Why would you invoke rm if you weren't trying to
| delete files?
|
| I think a better analogy would be "I tried to use an ide
| and it erased my drive"
| bcrl wrote:
| It makes me wonder what weight is given to content from 4chan
| during llm training...
| hombre_fatal wrote:
| If that other program were generating commands to run on your
| machine by design and you configured it to run without your
| confirmation, then you should definitely feel a lil sheepish
| and share some of the blame.
|
| This isnt like Spotify deleting your disk.
|
| I run Claude Code with full permission bypass and I'd
| definitely feel some shame if it nuked my ssd.
| ExoticPearTree wrote:
| Because the user left a "toddler" at the keyboard. I mean, what
| do you expect? Of course you blame the user. You run agents in
| supervised mode, and you confirm every command it wants to run
| and if you're in doubt, you stop it and ask it to print the
| command and you yourself will run it after you sanitize it.
| underlipton wrote:
| Nope. And that's why I don't use CCleaner to this day.
| ajs1998 wrote:
| Not defending the software, but if you hand over control of
| your data to software that has the ability to fuck with it
| permanently, anything that happens to it is on you.
|
| Don't trust the hallucination machines to make safe, logical
| decisions.
| lupire wrote:
| What makes a program malware?
|
| Does intent matter, or only behavior?
| schuppentier wrote:
| "The purpose of a system is what it does" would suggest
| malware.
| Nasrudith wrote:
| I believe the precedent is the behavior. Lose/lose is an 'art
| game' which deletes itself if you lose but gameplay destruction
| deletes random files. It is flagged as malware despite just
| doing exactly what it advertised.
| timthelion wrote:
| We've been developing a new method of developing software using a
| cloud IDE (slightly modified vs code server),
| https://github.com/bitswan-space which breaks down the
| development process into independent "Automations" which each run
| in a separate container. Automatons are also developed within
| containers. This allows you to break down the development into
| parts and safely experiment with AI. This feels like the "Android
| moment" where the old non-isolated way of developing software (on
| desktops) becomes unsafe. And we need to move to a new system
| with actual security and isolation between processes.
|
| In our system, you can launch a Jupyter server in a container and
| iterate on software in complete isolation. Or launch a live
| preview react application and iterate in complete isolation.
| Securely isolated from the world. Then you deploy directly to
| another container, which only has access to what you give it
| access to.
|
| It's still in the early stages. But it's interesting to sit at
| this tipping point for software development.
| tniemi wrote:
| Historical reference: https://jargondb.org/glossary/dwim
| digitalsushi wrote:
| if my operating system had an atomic Undo/Redo stack down to each
| register being flipped (so basically, impossible, star trek tier
| fantasy tech) i would let ai run commands without worrying about
| it. i could have a cool scrubber ui that lets me just unwind time
| like doctor strange using that green emerald necklace, and, i'd
| lose nothing, other than confuse my network with replay session
| noise. and probably many, many other inconsistencies i can't
| think of, and then another class that i dont know that i dont
| know about.
| robertheadley wrote:
| I was trying to build a .MD file of every powershell command
| available on my computer and all of its flags, and... that wasn't
| a great idea, and my bitlocker put the kebosh on that.
| JohnCClarke wrote:
| FWIW: I think we've all been there.
|
| I certainly did the same in my first summer job as an intern.
| Spent the next three days reconstructing Clipper code from disk
| sectors. And ever since I take backups _very_ seriously. And I
| double check del /rm commands.
| ashishb wrote:
| Run these tools inside Docker[1]
|
| 1 - https://ashishb.net/programming/run-tools-inside-docker/
| cs702 wrote:
| The subsequent dialogue is tragic comedy of the highest caliber.
| Even Monty Python couldn't have come up with a more
| inappropriately funny response:
|
| User:
|
| > did i ever give you permission to delete all the files in my D
| drive?"
|
| AI:
|
| > Thought for 25 sec.
|
| > Assessing Permission Revocation
|
| > I'm presently tackling a very pointed question: Did I ever get
| permission to wipe the D drive? This requires immediate
| attention, as it's a critical issue. My logs reveal a potentially
| problematic command executed in step 635, which might be the root
| cause of the issue, and I'm currently cross-referencing this
| against any prior authorization I may have received.
|
| >Reviewing Deletion Command
|
| > I'm now investigating a potential misfire. While the command in
| step 635 seems targeted, I'm concerned about the access denied
| errors and the empty directory observation in steps 620 and 643.
| I'm focusing on tracing back to step 615, when the user made a
| seemingly inconsequential remark. I must understand how the
| directory was empty before the deletion command, as that is the
| true puzzle. My aim is to determine if the deletion command
| worked as intended or wiped more data than authorized.
|
| > ...
|
| It continues. Go read the whole thing!
| tempaccount420 wrote:
| Gemini 3 Pro is the most actively hostile (to the user) model
| out of the top 3 (Google, OpenAI, Anthropic).
|
| It perfectly reflects Google as a company.
| FerretFred wrote:
| Remember that Anthropic is only 3 letters away from
| MisAnthropic: did the designers think of this?
| nofriend wrote:
| mis means "not"
| FerretFred wrote:
| I went by the definition:
|
| misanthropic /mis''@n-throp'ik, miz''-/ adjective
|
| Of, relating to, or characteristic of a misanthrope.
|
| Characterized by a hatred or mistrustful scorn for
| humankind.
|
| Hating or disliking mankind.
|
| The American Heritage(r) Dictionary of the English
| Language, 5th Edition
| nomel wrote:
| The gaslighting, and outright "lies", from my first
| experience with Gemini, dramatically increased my p(doom) of
| AI.
| setnone wrote:
| I am deeply regretful, but my Google Antigravity clearly states:
| AI may make mistakes. Double-check all generated code.
|
| Surely AGI products won't have such disclaimer.
| ringer wrote:
| People need to learn to never run untrusted code without safety
| measures like virtualization, containerization,
| sandboxing/jailing, etc. Untrusted code can include executables,
| external packages (pip, npm, cargo, etc) and also code/commands
| created by LLMs, etc.
| nottorp wrote:
| Hmm. I use these LLMs instead of search.
|
| They invariably go off the rails after a couple prompts, or
| sometimes from the first one.
|
| If we're talking Google products, only today i told Gemini to
| list me some items according to some criteria, and it told me it
| can't access my google workspace instead.
|
| Some time last week it told me that its terms of service forbid
| it from giving me a link to the official page of some program
| that it found for me.
|
| And that's besides the usual hallucinations, confusing similarly
| named products etc.
|
| Given that you simply cannot trust LLM output to not go haywire
| unpredictably, how can you be daring enough to give it write
| access to your disk?
| alshival wrote:
| I like turtles.
| throw7 wrote:
| Remember when computers were deterministic? Pepperidge Farms
| remembers.
| Terr_ wrote:
| Pepperidge Farm confirms it can remember with a comprehensive
| suite of unit tests, which must 100% pass on every build,
| including test order randomization.
| Aeolun wrote:
| Is there anyone else that uses Claude specifically because it
| doesn't sound mentally unhinged while thinking?
| FerretFred wrote:
| I always use "rm -rf*v*" so that if I do screw up I can watch the
| evidence unfold before me.
| woopsn wrote:
| Well that's stupid. I submit though, connecting stochastic
| process directly to shell you do give permission for everything
| that results. It's a stupid game. Gemini mixes up LEFT and RIGHT
| (!). You have to check it.
| eamsen wrote:
| Personal anecdote: I've asked Gemini 3 Pro to write a test for a
| function that depends on external DB data. It wrote a test that
| creates and deletes a table, it conveniently picked the exact
| production table name, didn't mock the DB interactions. Attempted
| to run the test immediately.
___________________________________________________________________
(page generated 2025-12-01 23:02 UTC)