Posts by arclight@oldbytes.space
 (DIR) Post #B5CxpmA7vQ5AjseMbo by arclight@oldbytes.space
       0 likes, 0 repeats
       
       @azonenberg @whitequark You should never be allowed near a nuclear plant. :)Aside: I've seen the results of arcing within the potting compound of a cable penetration from a power plant. Safety analysis is weird.
       
 (DIR) Post #B5SWJbk1Xuccv2EhDU by arclight@oldbytes.space
       0 likes, 0 repeats
       
       @azonenberg @dianea This is the initial scene of half a dozen USCSB videos...
       
 (DIR) Post #B5kZwP7YCRRvmKi6Hg by arclight@oldbytes.space
       0 likes, 0 repeats
       
       RE: https://c.im/@cdarwin/116479704797697865Nothing "went rogue". AI didn't delete the firm's database and backups. A human operator built admin automation and ran it in production without adequate testing or backups.I'm sorry, but no: a human gave admin privileges to unverified tools and run them in a production environment.Own your work. You as sysadmin, developer, etc. are paid to perform a job with skill and diligence. Ultimately you are responsible for your professional work. If there was someone upstream responsible for V&V of the tool, ensuring users are trained, cautions and limitations of the tool are communicated, and confirming the tool is fit for its intended use, they bear a share of that responsibility.If you're the manager that forced worker to use an unreliable tool on production systems without putting it through proper V&V, without effective user training, use case development, or risk assessment, you bear a share of the responsibility.Repeating this for those in the back: AI does not launder away your job responsibilities.RT: https://c.im/users/cdarwin/statuses/116479704797697865
       
 (DIR) Post #B5lb8Ksc5TYpDFw8dk by arclight@oldbytes.space
       0 likes, 0 repeats
       
       Once upon a time - around 2007 or so, just before I left sysadminery to do risk assessment on legacy radioctive waste cleanup - I was upgrading a Blackboard LMS. Hardware load balancer in front of two unreliable web/app servers, Oracle db with RMAN backups, NFS file store backed by iSCSI SAN storage for user files. The web front-ends were intentionally provisioned with low disk because they didn't need it - content was on the NFS server or in the database and all the front-ends needed disk for was swap and log files from Apache and Tomcat. Those were religiously scraped because why would you expect the vendor to rotate their giant useless log files or record to a remote log host when they could just let crap accumulate everywhere until their system fell over?But I digress. I had automated log cleanup and database backup (with tested restores) and main IT managed the SAN backups. It was that brief Windows of low usage between semesters when we could run the vendor-supplied binaries to upgrade this expensive and cursed assemblage of Java, Perl, Oracle, and human misery. Read the documentation multiple times to understand the order of upgrade operations, clean and quiesce the system, take a few final backup snapshots and pull the trigger.The upgrade worked as intended, dutifully _moving_ files from the NFS mount of the large iSCSI drive to the web front-ends, filling the disk, then shitting the bed and falling over leaving the system in an unknown and unrecoverable state. As one does when you are Blackboard, the usurious vandal of LMS vendors.Surveying the flaming wreckage, I called Bb support to as for guidance. The support peon was impressed with my calm tone. I responded that being outwardly furious and losing my shit at them was unlikely to recover my system or improve any outcome I cared about. I did ask in my support ticket if this behavior from the update was documented and if there was any way I had missed a critical "do this to avoid incinerating prod" step in the upgrade process.A few hours later I got a response back from upper tier support that no, I had read and done everything correctly according to their documentation and that this whole fiasco could have been avoided by the use of an _intentionally undocumented_ option to the updater.Their words: _"intentionally undocumented"_.Why? Somebody might get confused by the explanation so it was omitted in the interest of ... clarity?I spent several harrowing hours waiting for the iSCSI restore to complete. I did my best to verify no user content was lost but to this day I don't know if we lost data.Deleting one symlink in the filesystem would have prevented this problem, a symlink that was required in a previous version of the code for the system to work properly (one actually described in the vendor documentation).We were a small university and did not have a full replica dev system to test the updater on. Why would we? We explicitly did not do development, we ran a vendor-supplied code in production. Dev systems were for developers which we were not. This wasn't a matter of a spinning up some virtuals in the cloud - we bought and managed real hardware and there was no way in hell we could justify doubling our hardware investment to have a test environment just to verify vendor supplied code worked as advertised. There wasn't a possibility of auditing the updater to detect that it would copy and delete the entirety of user-uploaded content, not without decompiling a big blob of Java. I exercised what diligence I could given the garbage state of the vendor's code and still got royally fucked over.I owned that. I informed my management chain of the situation and kept them updated with new developments and a revised ETA until the system was stabilized, recovered, and updated. That is how I practiced server ops for a decade (1998-2008). You did your dligence, said a prayer as you pushed the button, and you owned the outcome.No idea what current practice is. That was almost 20 years ago before devops, virtuals, clouds, and containers replaced real machines and dedicated sysadmins. I would like to believe that outlook and practice carried forward since then but I don't know - I left for greener, safer, more relaxing and fulfilling pastures helping package, transport, and store Cold War era uranium-metal-bearing radioactive sludge, moving it out of crumbling fuel pools at Hanford to interim storage elsewhere at Hanford. Then later projects for Dounreay, Sellafield, US commercial plants, Swedish interim waste storage at CLAB underneath Oskarshamn, fire PRA for plants in the US, Sweden, and Spain, and a whole lot of safety analysis code development and software QA. Somewhere in there I live-tweeted Fukushima melting and exploding. That's possibly the most direct act of nuclear safety I've performed with the goal of keeping people informed well enough to contextualize what was happening so they didn't panic and hurt themselves.
       
 (DIR) Post #B5lb8LhJ306tkTecee by arclight@oldbytes.space
       0 likes, 0 repeats
       
       So if you wonder how I kept my shit together when Blackboard torched itself, it came from the perspective of nuclear safety analysis. Nobody was going to die, nobody was going to be hurt, the entire web service indistry wasn't going to be wiped out, and the worst thing that could happen is a few people would be inconvenienced for a few hours and there might be minor data loss of instructional material. Important in the local context but meaningless when compared with human safety and environmental protection.Skill and diligence are underrated. They may not save you when all hell breaks loose but more often than not they turn a disaster into a near miss. We ignore the deskilling and low/no-responsibility aspects of AI at our collective peril.
       
 (DIR) Post #B5lb8LzjwSr6feHLO4 by arclight@oldbytes.space
       0 likes, 0 repeats
       
       Note that this isn't a "back in my day..." post as much as a description of how easy it is to get screwed over by _deterministic_ tools in an environment you mostly control. Modern systems have so many layers abstraction between an admin and the actual hardware, there are so many hidden points of failure and latent vulnerabilities that a) it's understandable that someone with tenuous employment (i.e. contractor held to a Jira ticket clearance rate or time window) would turn to a chatbot especially if chatbot use was now a KPI and a mandate, and b) if we couldn't avoid Blackboard-level disasters in an age of deterministic tools and more mentally tractable architectures, what makes anyone believe a chatbot can do any better today? We've painted ourselves into a corner chasing velocity and scale and layer upon layer of abstraction and the same people pushing all this are now pushing the magic expensive pirated text extrusion engine as a solution.Do you honestly need all that to build a stupid web page?"If I want to meet my KPIs and keep my pittance of health insurance I do."I can't fault that logic but seriously wonder why more tech workers haven't invested more of their energy into unions and guillotines. That supposes there's any discretionary energy and support networks left and that anything can survive the decades of libertarian propaganda. I suppose the silver lining in all this is with chatbots, you don't need to organize and recruit saboteurs to fight back; the whole system has evolved to tear itself apart on its own.
       
 (DIR) Post #B5lb8MMmYnHrp73kIq by arclight@oldbytes.space
       0 likes, 1 repeats
       
       We really need to organize to take back control of our technology and our jobs and our profession. We are failing ourselves, the people who use and are affected by our systems, and the generation that is entering the field. Until we build more solidarity than a bucket full of lobsters, this BS is just going to keep happening.I'm lucky that I had my original profession to go back to when I left ops in 2008. Not everyone can or wants to work on literal radioactive sludge cleanup and ideally nobody should have to make that choice given how soulcrushingly awful dev & ops have become (OTOH it's a really easy choice if you've ever managed Blackboard. Beyond all probability, the company is worse than its software. As an example, radioactive sludge has never intentionally and actively made my life worse. Sludge mostly just sits there emitting energy and hydrogen.)Nobody should have to choose between using broken black-box automation that risks their code, systems, and skills and quitting their job. But people do every damn day - how do we break this cycle of abuse and incompetence?
       
 (DIR) Post #B5lb8NZw3NP1aCDlRY by arclight@oldbytes.space
       0 likes, 0 repeats
       
       I will grudgingly accept "arson" as a viable solution. Not my first choice but I'm hard pressed to see an effective alternative.
       
 (DIR) Post #B67JOgZBhV2hPPcDia by arclight@oldbytes.space
       1 likes, 0 repeats
       
       More and more I feel that software is something that's inflicted on me rather than something I create or control that serves me.
       
 (DIR) Post #B67JOgkt00Orzh5Yx6 by arclight@oldbytes.space
       1 likes, 0 repeats
       
       I'm still going to write code to solve problems I care about but I need to rethink licensing and distribution. Making my code public may actively harm me - hosting costs, time & effort defending against AI scrapers, attribution laundering, plagiarism, all devaluing the skill I use to put food on my table. Plus the morale damage that comes from knowing that work I put care and effort into is being ground into a barely recognizable paste to fund psychofuckwit nazi oligarchs that deluded themselves into believing a box of anxious electrified sand is a god.If someone else was _just_ making money off my work it wouldn't be nearly as bad. But no, it has to be shoved into the LLM blender and extruded as "AI search results" that actively makes everyone else stupider, prevents people from finding canonical and trustworthy sources of information, actively prevents people from finding experts or recognizing expertise. That high velocity StupidityAsAService is already infecting every piece of computational technology that touches the internet. Code is churned out faster and faster, quality is plummeting in free fall, and every organization that dove headfirst into the empty pool that is AI is sweating bullets that their end of the pool will start filling up before they faceplant into the concrete.I swear, computer programmers are some of the most gullible, delusional, arrogant, and just plain stupid people on the planet. And we're stuck with them shitting up everything they touch.
       
 (DIR) Post #B67JOhTYJw84EDzEZc by arclight@oldbytes.space
       0 likes, 0 repeats
       
       Thanks to @meena for prompting this - it's not like I'm some sort of Superior Programming Being whose every line of code is elegant snd brilliant and not worthy of the sweaty masses. I'm a tediously average programmer. I detest status-driven gatekeeping. I would like to share my work so the effort can be leveraged by others to do new and useful work. I support some ideal of open source circa 1995 where useful decent quality tooling is freely shared and available.That is not the environment we live in today. Beyond the individual entitled ankle-biters that harass devs for not doing work they want for free, we now have this extreme corporate bullshit of "open source supply chain", that freely-provided _caveat utilitor_ code should be treated like code acquired under a commercial procurement agreement with formal specs, requirements, and standards for security, quality, and lifecycle management.I work in this space in my day job, I'm the one who sets and verifies those specs for our acquired codes, and I'm going to say flat out, that is that absent an explicit agreement with a supplier, all that supply chain and capital-P procurement activity is *solely* on the code user, not on the code author, full stop. Screw off with that CTO/infosec/bureaucrat thinking. Into the sea with you.For those of you in the back:_IF THAT PROCUREMENT AND ASSURANCE WORK IS IMPORTANT TO YOUR ORGANIZATION, YOUR ORGANIZATION NEEDS TO PAY FOR IT_.Further:_WITHOUT AN EXPLICIT AGREEMENT, NOBODY IS REQUIRED TO PROVIDE THAT SERVICE TO YOU AT ANY PRICE_.And finally:_WHAT PART OF 'USE AT YOUR OWN RISK' IS UNCLEAR TO YOU?_I will curse all day long about research-grade software being built and distributed by organizations who explicitly know they are working on nuclear safety applications and do not perform adequate diligence in design, implementation, and testing. This goes beyond #ResearchSoftwareEngineering - it's production software engineering practice. These are organizations who absolutely know how their code is used (we have monthly meetings with them). I hold these organizations and individuals to a higher standard because this is quite literally our jobs.By contrast, this is not the job of @bagder and expecting or demanding he do some corporation's or industry's V&V and SQA work is laughable, expecting him to do it _for free_ is arrogant, insulting, and delusional.But this is where we are now with sharing our code. Ethically, I believe anyone who puts their code in public bears some personal responsibility to ensure it's of a reasonable level of quality, functionality, and security. It's as if you were bringing homemade food to a potluck - the expectation is that it's edible and not spoiled or adulterated. You aren't expected to disclose every significant allergen, provide a list of ingredients, or a nutritional statement but you're expected to wash your hands, use clean utensils and ingredients of good quality, and ensure the food is cooked all the way through.That's gatekeeping, absolutely, and I won't apologize for setting that demand or an analogous one for "potluck" software.I am old enough to remember Matt's Script Archive and I have seen what happens when we don't do this sort of positive gatekeeping. For those unfamiliar, Matt was the Typhoid Mary of insecure internet software and his formmail.pl script was the poster child for bad and dangerous code you found for free on the internet. https://en.wikipedia.org/wiki/Matt%27s_Script_ArchiveAnd while I appreciate he was a teenager when he posted most of that code, as an adult he was repeatedly told how damaging his code was and yet he kept distributing broken versions for YEARS.
       
 (DIR) Post #B67JOiM92xnWxXWpfM by arclight@oldbytes.space
       0 likes, 0 repeats
       
       So what am I actually arguing here?This is not 1995 and the expectations surrounding publicly distributed code are very different now.Some of those expectations are reasonable. If an author knows their code is dangerously broken, it should be fixed or be pulled from distribution. You need to be better than Matt from Matt's Script Archive. You should not be publishing to npm or PyPi or CRAN or CPAN or any of the public repositories unless you accept the minimal responsibility for fixing or retiring your code._RETIRING CODE IS ALWAYS AN OPTION, EVEN IN A PUBLIC REPOSITORY_But let's take npm as a bad example. What happens when an author decides to retire their code and stops distributing it and pulling it from a public archive breaks other people's applications that rely on code from that public archive? Do we accept the breakage that results from our naïve expectations of the public archive or does the archive continue to distribute the retired code against the author's wishes without forking the code and taking responsibility for it?I was on call the night of Friday, December 31, 1999. The right answer is that organizationally, you take responsibility for your code _AND ALL ITS DEPENDENCIES_ and you factor in that the public repository is not guaranteed to be there when you need it and that any code you pulled in from the internet is your responsibility. You own those dependencies once you start relying on them. Unless you have an explicit agreement with the original author, they owe you _NOTHING_.That is apparently not the modern expectation. The modern expectation is that authors who make their code public owe free maintenance, development, and security fixes to corporations for free forever and if you ever try to extract yourself from this implicit serfdom by retiring your code, some asshole organization will keep distributing your retired code under your name and keep you yoked to that millstone.Am I arguing to have it both ways? No. It's reasonable to distribute code of average quality and maintain it on one's own schedule. It's also reasonable to retire code for any reason but specifically because you no longer have the time or interest to maintain it at an average or (personally) acceptable level of quality. Mark it as archived and unsupported so it's clear to everyone what the project's status is. In an extreme case, stop distributing the code - that's what should have happened with Matt's Script Archive.
       
 (DIR) Post #B67JOivwtqRCkaHQTQ by arclight@oldbytes.space
       0 likes, 0 repeats
       
       The other obsolete expectation is that one's public code would be used and read by other humans and that attribution was preserved. The vast corpus of code used as training data for commercial LLMs was never intended to be used in that manner. No explicit permission was granted to strip the license and authorship of all that code - it was effectively plagiarized by sampling.How much money did Clyde Stubblefield get from all the songs that sampled Funky Drummer? How much money did Gregory Coleman from The Winstons get from all the songs that sampled his drum break from their 1969 track "Amen, Brother"? https://en.wikipedia.org/wiki/Amen_breakHint: Coleman died homeless and destitute in 2006.When hip-hop was a niche genre of primarily Black artists sampling other Black artists early on, maybe "sample culture" and the same sort of attribution stripping and plagiarism-by-sampling was acceptable. Once the big money started rolling in it wasn't. People like Richard Spencer, Gregory Coleman, and Clyde Stubblefield never got their due and let's not underemphasize this was yet another instance of Black artists being screwed over by White corporations.White people _really_ don't like it when they get treated like Black people have been treated like forever. I can't be upset about AI stripping attribution and stealing code by sampling and remixing without clearly and unambiguously showing the parallels to Black musicians 4 decades ago.Let's just pause and reflect on that for a moment.
       
 (DIR) Post #B67JOjVOm2nIWWrjjE by arclight@oldbytes.space
       0 likes, 0 repeats
       
       Honestly, I'm the last person in the world who should be giving a lecture on hip-hop history. I make Weird Al look like Chuck D by comparison: https://youtu.be/N9qYF9DZPdw
       
 (DIR) Post #B6EMs3MJhK9gSu0fYW by arclight@oldbytes.space
       0 likes, 1 repeats
       
       I give up. R is an impenetrable trash fire. The system is intentionally designed to violate virtually every standard and good practice of software engineering. R is a desktop tool for desktop people and like Python should never EVER be used to create applications for distribution beyond the distance a human can walk in 30 seconds. It's a personal scripting tool that becomes prohibitively antisocial as soon as application code is shared."Tell me the name, version, and canonical source location+ of every external package needed for this application to function completely and correctly." is apparently an unanswerable question.I need the list of dependencies BEFORE I install this software because it gets installed on a machine with no internet connectivity - I need everything downloaded checked up front. Then I need to write the installation instructions that did not come with the application in such a way as someone other than me can install the software without assistance. That's just fundamental resilience and software lifecycle management. Apparently it's also completely alien thinking to anyone in research or who uses interpreted languages.This is a deeply unserious field and R is substantially worse than almost anything I've touched in the past 2 decades. If I never encountered this cursed mental tarpit again it would be too soon.The hardest part of this is finding the professional language to ask the vendor to provide a full installation manifest and clear detailed instructions for installing their software in a way that preserves an otherwise good working relationship.+ e.g. provide enough information to construct an URL which points at the package source code. Have fun getting that info out of the Byzantine nested dataframe generated by pkgdepends.
       
 (DIR) Post #B6Vu4CjZGdyJBtbd32 by arclight@oldbytes.space
       0 likes, 1 repeats
       
       Fujitsu open-sourced their compiler test suite for C/C++/Fortran https://github.com/fujitsu/compiler-test-suiteSlide deck overview: https://llvm.org/devmtg/2025-06/slides/lightning-talk/kawashima-fujitsu.pdf When was the last time you independently tested your compiler/interpreter?
       
 (DIR) Post #B6cmqB4vHq8qQgg8Js by arclight@oldbytes.space
       0 likes, 0 repeats
       
       @whitequark I don't understand how one expects to support multiple platforms without regularly testing on a variety of platforms to detect platform-specific problems as early as possible. Not just software bugs but problems with build automation, packaging, etc.
       
 (DIR) Post #B6lbtvghZxeK91KFQe by arclight@oldbytes.space
       0 likes, 1 repeats
       
       My snoozy boy #CatsOfMastodon
       
 (DIR) Post #B6nuqcgU422DmaHRVQ by arclight@oldbytes.space
       0 likes, 0 repeats
       
       So today is the end of the month. A number of our reports are due today, at least three (I'm reviewer on two). IT is patching everything right now. They uninstalled Python 3.11 from production servers without telling anyoneThe last three words of the previous sentence are the important ones. The rest are important too but are substantially less critical without those last three.So a post-processor I'm responsible for broke because it's installed in a Python 3.11 virtual environment as is good practice to avoid multiple applications fighting over dependencies under one interpreter. The application is distributed as a .whl as is good practice - it requires the least effort on end users while maintaining good configuration management.The application's runtime and development dependencies are listed in the project's pyproject.toml file as is modern practice. The appropriate requirements.txt files are created by pip-compile which also point pip to pull dependencies from our local PyPi repo, again good practice for configuration management, security, etc.Unfortunately, pip-compile does not honor ">=" as specified in pyproject.toml, it pins dependencies with "==". This great if that's what you want; this is a post-processor and we're mostly concerned that it run, not that it's installed with specific versions of dependencies.I clone the post-processor's repo, build a Python 3.13 virtual environment with the intent of refreshing the requirements.txt files to work with 3.13 compatable dependencies.pip-compile doesn't work, throwing a cryptic use_pip517 error so regenerating the requirements files won't work. So I try to install the existing requirements files. I do not expect this to work so I am not surprised when every third dependency breaks because the pinned version is incompatible or we don't have the platform- and Python-version-specific .whl in our local repo.The solution was to bump the project version and strip all the version numbers off everything in the requirements.txt files so it installs whatever version it can. I'm still left with no way to build version-aware requirements.txt files because of the should-not-be-my-problem use_pep517 problem. For the moment I'm ignoring the build system shrieking about SPDX license tags (you know what license SPDX doesn't list? "Other/Proprietary"! Do you know what all of my work code is licensed as? That's exactly right! I've sorted this BS out on other projects so this is also not a surprise to me.I think that going forward I will be writing all my utilities in Rust. Or Zig. Or D. Or Spark/Ada. Anything but Python. Good practice won't save you and the project risk is simply not worth the convenience of Python. In our environment I no longer consider it git for use. There might have been a graceful way forward had IT given us sny warning they were going to break Python on the production servers but they didn't. I can't change or fix our IT department (or the whole benighted industry), I can't fix Python but I sure as hell can replace it with something that doesn't self-destruct every fiscal quarter.A post-processor like mine does not change unless it needs to change. It may not get a change for *years* and that is reasonable for this software in this environment. Nothing says you as a developer and the projects you work on are disposable quite like the Python ecosystem does. It's a huge middle finger every time I fire up Python and find that half a dozen things are broken with no documentation or migration path. Nothing about Python compensates for this absolute lack of dependability. I just can't trust it, I just can't.
       
 (DIR) Post #B6nuqe5gpnn89355Qu by arclight@oldbytes.space
       0 likes, 0 repeats
       
       Addendum: You'll note that I did not describe any of the code changes I needed to make because of the change from 3.11 to 3.13. That's because the application source code didn't change at all - all that changed was a version bump in pyproject.toml and removal of version numbers from requirements.txt. Maybe if this inconvenience only affected me it might be tolerable but I had an engineer, his manager, and his manager affected by an outage that should not have happened. That's unacceptable project risk.