[HN Gopher] Ask HN: Best "I brought down production" story?
___________________________________________________________________
Ask HN: Best "I brought down production" story?
What is your best "and then I brought down production" story?
Author : Ozzie_osman
Score : 78 points
Date : 2021-06-26 18:17 UTC (4 hours ago)
| parkersweb wrote:
| This was 20 years ago now - it was my first day in a new job
| working for a startup.
|
| Our startup was based in the garden office of a large house and
| the production server was situated in a cupboard in the same
| room.
|
| The day I started was a cold January day and I'd had to cycle
| through flooded pathways to get to work that morning - so by the
| time I arrived my feet were soaked.
|
| Once I'd settled down to a desk I asked if I could plug a heater
| in to dry my shoes. As we were in a garden office every socket
| was an extension cable so I plugged the heater in to the one
| under my desk.
|
| A few minutes later I noticed that I couldn't access the live
| site I'd been looking through - and others were noticing the
| same.
|
| It turned out the heater I was using had popped the fuse on the
| socket. The extension I was using was plugged into the UPS used
| by the servers. So the battery had warmed my feet for a few
| minutes before shutting down and taking the servers down too.
|
| And that's how I brought production down within 3 hours of
| starting my first job in the web industry...
| maccard wrote:
| Very first job, as an intern, I was tasked with building a "free
| text search engine" for the product, using their api. Maybe my
| first week or so there, I left a script running over lunch. Turns
| out the internal IP addresses weren't subject to the rate
| limiting, and my script's queries were growing exponentially (I
| was sending the response back to the same endpoint which was
| querying with the response, and giving me back a larger response
| etc..) Within 20-30 minutes or so every production machine was
| stuck running one of my queries. And it happened on the day that
| the engineering team were taking the new intern out for a team
| lunch...
|
| At the time I was mortified, but in hindsight the fact that I was
| able to do that in the first place was really the issue, not my
| script.
| lanstin wrote:
| I had a product that had mostly stopped growing in usage. It was
| running on say fifty machines. I had put considerable effort into
| some memory optimizations, which was the scaling point for new
| hardware, so I talked Ops into bunching the active traffic load
| onto fewer machines. All of the active traffic load. Started
| hitting the memory limits (32 bits linux) and our server
| framework exited on malloc failure, so lots of exiting of long
| lived processes and loss of expensive state, delayed alerts etc.
|
| I still think it was over-provisioned, but they told Ops to stop
| listening to me unless someone else agreed. Probably ran on the
| 50 machines till it was discontinued 10 or 15 years later, but I
| left so who knows.
| jason_slack wrote:
| I worked for a company back around 1999 that used MS Access to do
| the entire companies payroll. Including the CEO. Once I was
| awoken at 1am because payroll wasn't running. In my sleepiness I
| accidentally manually committed a few database values (that I had
| never touched before) that gave the executive team 0.00
| paychecks. Not a single exec including the CEO noticed for about
| 6 weeks.
| tluyben2 wrote:
| Long ago this one (close to 2000), but we were hosting some of
| our clients on machines in our office because it was (a lot)
| cheaper for our startup to do so. We had 3 rather large (for our
| company) clients running at the time in the server room in the
| office. The servers where hooked up to 2nd hand APCs so power
| failures went unnoticed if they were short. One Friday afternoon,
| we had some drinks (tgif) and a bunch of us (I was the cto....)
| were fooling around in one of the rooms throwing tennis balls; I
| threw one straight into the firealarm. This was just a glass with
| a button behind it: if you pressed the glass it would trigger for
| the entire (20 story) building. This cut the power and switched
| on the emergency lights. There was no fire obviously, but, for
| bureaucracy rules, the firebrigade had to pull up, inspect the
| building, we had to sign docs etc and then they switched off the
| alarm and on the power. Too late for our APCs: everything was
| down and (like said: long time ago for Linux etc) we had to run
| fsck and basically spent a large portion of the evening getting
| it all back. We moved to xs4all co location after that
| incident...
| whoisjohnkid wrote:
| heard a few years back, someone accidentally deleted everything
| across all AWS accounts at our company; had to reach out to AWS
| and they helped recover everything. Took 6-12 hours to recover
| BozeWolf wrote:
| About 15 years ago, when ssh-ing into servers was quite normal.
|
| In eterm on my gentoo linux laptop with enlightenment desktop I
| typed: su - shutdown -h now
|
| Because I was tired and I wanted to go to bed. Came back after
| brushing my teeth. F### laptops and linux! Screen still on. The
| thing didn't shutdown!
|
| Strange thing was: in the terminal something said it got a
| shutdown signal.
|
| Then I realized I shutdown a remote server for a forum with 200k
| members.
|
| It was on the server of an isp employee, which happened to be
| member of that site. All for free, so no remote support and no
| kvm switches. Went to bed and took a train next day early morning
| to fix it.
| suhailpatel wrote:
| I'm in the fortunate position of having been able to tell our
| story in detail on our blog after a major outage involving
| Cassandra and bootstrap behaviour that we didn't fully
| understand. This is a story of how I bought down the bank for two
| hours.
|
| https://monzo.com/blog/2019/09/08/why-monzo-wasnt-working-on...
|
| In summary, we were scaling up our production Cassandra data
| store and we didn't migrate/backfill the data properly which led
| to data being 'missing' for an hour.
|
| In a typical Cassandra cluster when scaled up, data moves around
| the ring a single node at a time. When you want to add multiple
| nodes, this can be an extremely time and bandwidth consuming
| process. There's a flag called auto_bootstrap which controls this
| behaviour. Our understood behaviour was that it will not join the
| cluster until operators explicitly signal for it to so (and this
| is a valid scenario because as an operator, you can potentially
| backfill data from backups for example). Unfortunately it was
| completely misunderstood when we originally changed the defaults
| many months prior to the scale up.
|
| Fortunately, we were able to detect data inconsistency within
| minutes of the original scale up and we were able to fully revert
| the status of the ring to it's original state within 2 hours (it
| took that long because we did not want to lose any new writes so
| we had to carefully remove nodes in the reverse order that they
| came in and joined the ring).
|
| Through a mammoth effort across the engineering team across two
| days, we were able to reconcile the vast majority of inconsistent
| data through the use of audit events.
|
| This was a mega stressful day for everyone involved. On the plus
| side though, I've had a few emails telling me that the blog post
| has saved others from making a similar mistake.
| 42365767567 wrote:
| I was working on Nationbuilder, a horrible all-in-one master-of-
| none thing, back in 2016 or so, and ran so many concurrent tasks
| in our BE that it started affecting all of their other clients.
| Waking up an engineer in CA was fun
| breakfastduck wrote:
| That thing is a pile of absolute dog shit. New up and coming UK
| political party are using it - I volunteered to help with
| setting up / maintaining their tech stack, realized within 5
| minutes all their eggs were in that godforsaken basket and
| changed my mind.
| BrandonWatson wrote:
| Impacted iOS Kindle users ... We had an oopsie.
|
| https://www.google.com/amp/s/techcrunch.com/2013/02/27/bug-i...
|
| Customer fears were far worse than reality, but my management
| team (up to and including JeffB) we're not amused.
| h2odragon wrote:
| Drilling through a wall, routing a new network line; hit the
| power line to the server rack. There was a UPS, but it didn't
| like that kind of short apparently, and folded up into a sulk
| immediately.
|
| Best part is that I did the wiring in that building when it was
| built 5 years before that; I really should have realized it was
| there.
| imwillofficial wrote:
| Early on in my career, I worked for a secret unit of a secret
| government law enforcement group that handled surveillance. Being
| young, full of verve, and not nearly as smart as I thought I was,
| I was always trying to improve things and tinker. Knowing nothing
| about networking, I plugged a switch into itself. Due to the
| configuration it knocked the entire surveillance network offline
| and everyone was freaking out. I was cool as a cucumber, because
| it couldn't have been me. Must have been a coincidence right?
|
| Right?
|
| The sense of dread the dawned on me as the former Navy Seal
| turned Network Engineer (and later doctor) started sniffing
| around the switch I had just touched was palpable. Luckily for
| me, He kept my mistake quiet and fixed it quickly.
| neilwilson wrote:
| First time I got into the computer room back in the days when one
| unix mini-computer ran hundreds of terminals. I was asked to put
| a reel tape into the drive to load a tar archive into the Oracle
| database.
|
| I couldn't get the tape drive door open so I looked around and
| saw a key next to the door.
|
| That didn't open the door either. I was stood there scratching my
| head when the double doors burst open and half a dozen sysadmins
| came running in like a SWAT team.
|
| I was a bit surprised until I glanced down and notice all the
| lights were off.
|
| Yes the key that turned the power off had been left in the
| machine.
| gtaylor wrote:
| We pushed a CDN config that triggered a CDN provider bug. Took
| down the CDN's entire presence on one continent. Broke a whole
| bunch of recognizable sites for a bit.
| tmh88j wrote:
| Are you referring to the Fastly outage that happened a few
| weeks ago, or is this more common than I realize?
| mikecoles wrote:
| Years ago, my employer was light on funds so we cobbled together
| plugs to use as a loopback when testing and identifying network
| jacks. It worked great. Insert the plugs in cubilces then test
| the open ports in the wiring closet. Worked great many times
| until one day we plugged it in and went to lunch. When we came
| back, we were told the network had slowed to a crawl and captures
| showed floods. This was during the days of primitive DoS via
| broadcast floods. Well, this flood was self induced. The loopback
| plug was inserted into a jack that had a connection back to the
| network hub. It dutifully retransmitted everything it saw back
| onto the network. Whoops.
| shyn3 wrote:
| Exact same scenario but it was connected to the on premise data
| center. I was imaging devices and figured why not use a switch
| and provision 4 at a time. Started seeing everything go down
| figured the network team was doing things. Checked my images
| and they had stopped, unplug a cord and everything is working.
| I don't think anything of it and plug it back in and went back
| to waiting for the images to deploy but the storm started.
| After about 5 hours customers started receiving their
| electricity after the network team found my device.
| pandemic_region wrote:
| The year is 2002, OS is Solaris, trying to compile some httpd
| add-on straight on the production server (because why not) kept
| giving some weird error about /etc/ld.so not being right. So
| junior me does:
|
| $ rm /etc/ld.so*
| theon144 wrote:
| Not me, but a colleague - he wanted to look around the system as
| the `uwsgi` user, so he ran `sudo -u wsgi -s /bin/bash`.
|
| Except that he typoed, and instead ran `sudo -c wsgi -s
| /bin/bash`. What that does is instead of launching the (-s)hell
| as the uwsgi (-u)ser, it interprets the rest as a (-c)ommand.
| Now, `wsgi` is also a binary, and unfortunately, it does support
| a `-s` switch. It tries to open a socket at that address - or a
| filesystem path, as the case may be. Meaning that the command
| (under root) overwrote /bin/bash with 0 bytes.
|
| Within minutes, jobs started failing, the machine couldn't be
| SSH'd into, but funnily enough, as /bin/bash was the login shell
| for all users, not even logging in via a tty through KVM worked.
|
| Perhaps not the best story, but certainly a fun way to blow your
| foot off on a Monday morning :)
| soneil wrote:
| That's beautiful. I'm not sure I'd have had a clue what just
| happened even if it was me making the typo.
| trefgin wrote:
| Slightly off topic but I think in most cases responsibility for
| accidentally bringing down production likes with management.
|
| However I've only ever heard stories where management lays the
| blame.
| disambiguation wrote:
| this was many years ago, as a junior dev .. management was
| stressing out over how to make our app faster -- long query
| runtimes. Naively, i pitched that we should run the queries in
| advance so they would be cached. simple enough. we did some dry
| runs. it looked good to go. we pushed to prod. its sunday night
| and i'm asleep, the query runner activates .. and our app
| proceeds to DDOS our data layer. Not only taking down our prod,
| but the prod of every app subscribed to the data store.
|
| I wasn't oncall, and the oncall didnt have access to the query
| runner script -- it was on my laptop. So, oncall was desperately
| trying to fight a fire they couldn't put out while i slept like a
| baby .. that was a fun monday morning meeting.
| krylon wrote:
| During my training, ~2004, I managed to kill the TCP/IP stack on
| an IBM mainframe running z/OS by accidentally creating a fork
| bomb with a perl script meant to test the performance of the
| newly installed BIND9.
|
| Fortunately, it was on a testing system, SNA continued to work,
| and the system was due to be rebooted on the weekend anyway, so
| it was not that bad.
|
| Not myself, but a few years ago, a new coworker managed to
| accidentally delete all user accounts from our Windows domain
| while trying to "clean up" the Group Policies. Our backup
| solution, while working, was rather crappy, so we had to restore
| the entire domain controller (there only was the one), which took
| all day, even though it was not _that_ big. Fortunately, most
| users took it rather well and decided to either take the day off
| (it was a Friday) or tidy up their desks and sort through the
| papers they had lying around. A few actually thanked us for
| giving them the opportunity to "actually get some work done".
| comprev wrote:
| Learned the hard way to properly label two identical bits of
| hardware before working on them.
|
| One ISP router in production with 20k active connections... one
| "backup" router fresh from the box.
|
| My job was to backup the production firmware and flash the config
| to the spare box.
|
| The opposite happened and the customer support telephones lit up
| like a Christmas tree.
| tonyjin wrote:
| I spelled "tariff" incorrectly with "tarriff" in the config file
| that's parsed on every page load.
|
| My code reviewers didn't notice and we didn't have linting or
| warnings on that file, so I brought down production :)
| BalancedThought wrote:
| 2 days before I got married, I dropped the production database by
| accident from a GUI tool where "right-clicking" can be
| destructive if you click to fast. The application scheduled radio
| and television commercials and within 48 US states for a large
| international advertising group. The bigger problem was that the
| DBA had only been doing incremental backups and didn't have a
| full back against which to run the incremental backups. He had
| never created a full backup.
|
| Fortunately, given the nature of media buys in that time, all
| placements were printed and faxed. My team sent me to my wedding
| rehearsal dinner and spent the next two days collecting printed
| orders and re-keying them into the system.
|
| I am forever grateful to that team.
| smitty1e wrote:
| I had web server access, but they wouldn't give me a DB login.
|
| So I crafted an .asp to do my maintenance.
|
| Only I was calling CreateObject() in the _for_ loop to get a new
| AdoDb.Connection for each of the array entries of the data.
|
| That creaky IIS server crashed like the economy.
| oceanghost wrote:
| Can It be a story I was involved in but I didn't do it?
|
| I used to work for a major university as a student systems admin.
| The only thing that was "student" about it was the pay-- I had a
| whole lab of Sun and SGI servers/desktops, including an
| INCREIDBLE 1TB of storage-- we had 7xSun A1000's (an array of
| arrays) if memory serves.
|
| Our user directories were about 100GB at the time. I had sourced
| this special tape drive that could do that, but it was fidgety
| (which is not something you want in a backup drive admittedly).
| The backups worked, I'd say, 3/4ths of the time. I think the
| hardware was buggy, but the vendor could never figure it out.
| Also, before you lecture me, we were very constrained with
| finances, I couldn't just order something else.
|
| So I graduated, and as such had to find a new admin. We
| interviewed two people, one was very sharp and wore black jeans
| and a black shirt-- it was obvious he couldn't afford a suit
| which would have been the correct thing to wear. The other
| candidate had suit, and he was punching below his weight. Over my
| objections, suit guy gets hired.
|
| Friday night, my last day of employment I throw tapes into the
| machine and start a full L0 backup which would take all weekend
| to complete.
|
| Monday morning I get a panicked phone calls from my former
| colleagues. "The new guy deleted the home directories!"
|
| The suit guy literally, had in his first few hours destroyed the
| entire labs research. All of it. Anyways, I said something to the
| effect of, "Is the light on the AIT array green or amber?"
|
| "Green."
|
| "You're some lucky sons of bitches. I'll be down in an hour and
| we'll straighten it out."
| D13Fd wrote:
| This was long ago and the details are hazy.
|
| Back in 2003 or so, I was in tech support for a company that used
| desktop computers running java applets to connect to a mainframe
| via Telnet (IBM Host-on-Demand IIRC). Most of the core business
| processes were handled by mainframe apps, which the company
| largely developed. I used to hang out in the data center with the
| mainframe guys who coded in COBOL all day.
|
| On a Friday afternoon, I was working on testing deployment of an
| update to the java terminal client applet. Everything seemed to
| work fine in testing, and it was a minor update, so (idiot me) I
| went ahead and pushed it to the server.
|
| Shortly after I pushed it out, the mainframe guys' phones started
| ringing with complaints that the mainframe was down. Then my
| phone started ringing. Then all of the phones started ringing.
|
| Turns out, something I did in the update (I honestly can't
| remember the specifics now) reset every local users' mainframe
| connection information for the applet. Across the whole company.
| So as soon as they exited the applet, they couldn't get back in.
|
| That was a fun weekend.
| ramoz wrote:
| Nothing crazy, but something I always laugh at.
|
| I was so excited to meet a legit/professional dev team the first
| day of my career.
|
| I was paired with a Sr dev and sat in his cubicle so he could
| show me some backend update to a prod app with 20K internal
| users... "normally I'd run this on the dev server, but, its quick
| & easy so Ill just do it on prod"
|
| ...watched him crash the whole thing & struggle the rest of the
| day to try and bring it up. I just sat there in awe, especially
| as everyone came running over and the emails poured in, while the
| Sr Dev casually brushed it all aside. He was more interested in
| explaining how the mobile game Ingress worked.
| pschoeps wrote:
| About 5 years ago when I was just starting out I found myself
| designing a responsive course builder. My solution to a
| responsive interface at this time involved sending a very large
| stringified HTML file over websockets.
|
| This wasn't a huge problem, but the configuration on Action Cable
| (Rails wrapper around websockets), logged the entire contents of
| the message to STDOUT. At a moderate scale, this combined with a
| memory leak bug in Docker that crashed our application every time
| one of our staff members tried to perform a routine action on our
| web app. This action resulted in a single log line of > 64kb,
| which Docker was unable to handle.
|
| All of this would have been more manageable if it hadn't first
| surfaced while I was taxiing on a flight from Detroit to San
| Francisco (I was the only full time engineer). I managed to
| restart the application via our hosting providers mobile web
| interface, and frantically instructed everyone to NOT TOUCH
| ANYTHING until I landed.
| etaioinshrdlu wrote:
| I worked in online advertising and pushed infinite loops that
| froze browsers to millions of unsuspecting victims.
|
| On another occasion I had a division operation happen on integers
| instead of floats, and the code was running on some hardware that
| steered antennas for radios on airplanes. Much time was spent by
| pilots flying in circles over LA while I gathered data and found
| the "oops". It was fixed by adding a period to an int literal.
|
| On another occasion my machine learning demo API failed due to
| heavy load, but only when India's prime minister was looking at
| it.
| thatjoeoverthr wrote:
| I was 18 and had just taken over the website of a car and horse
| trailer dealership. I typed ";rm -fR ~" into an e-mail form to
| show my coworkers what would happen. On purpose. I quickly
| restored it (I was ready for this.) They were pretty amused. We
| had no concept of "on prod". "Dev" was my local PC. Damn
| teenager.
| cheese123 wrote:
| Doom on a single PC was fun. But the same game on our network
| with all those broadcasts running through our net, which hosted a
| university's PCs. That was bad. And I started it several times
| until I noticed that "I" was the cause for network slowdown.
| woutersf wrote:
| I have some of these.
|
| Does self DDOS count?
|
| We worked for Flanders radio and television (site was one of
| Flanders biggest radio stations). The site was a angularjs
| Frontend with a CMS backend.
|
| The 40x and 80x pages fetched content from the backend to show
| the relevant message (so editors can tweak it). The morning they
| started selling tickets for Tomorrowland I deploy the frontend
| breaking the js fetching a non existing 5x page, looping to doing
| this constantly. in a matter of seconds the servers were on fire
| and I was sitting sweating next to the operations people. Luckily
| they were very capable and were able to restore the peace quite
| quickly.
|
| And also (other radio station) deleting the DB in production. And
| also (on a Bank DB2) my coworker changing the AMOUNT in all rows
| of cash plans in stead of in 1 row (and OR and braquets and trust
| you know).
| neya wrote:
| 3 years ago. Took on a large traffic media site running on
| Wordpress. The usual good stuff - outdated plugins, running on a
| cheap VPS box, hosting vendor washed hands off saying they can't
| handle the tech support and traffic.
|
| I was working in a different company back then and I was
| contacted by this consultancy that specialized in Google Cloud
| (which was always my personal favorite anyway). I was offered a
| very handsome pay for just what seemed to be just 4 days worth of
| work. To me, it sounded really simple, like get in, migrate and
| get out.
|
| After I signed the contract and everything I got to know the
| client was promised a $200/mo budget and a very wrong technical
| solution proposal by an Engineer in the consultancy from what
| they were paying then which is definitely in multiples of what
| was quoted. And to make matters more interesting, this guy just
| quit after realizing his mistake. And that's how I even got this
| project.
|
| So, I went in, tried many cost effective combinations including
| various levels of caching and BAM!, the server kept going down.
| They had too much traffic for even something like Google's Paas
| to hold (it has autoscaling and all the good stuff, but it would
| die even before it could autoscale!). Their WP theme wasn't even
| the best and made tons of queries for a single page load. Their
| MySQL alone costed them in the 1000s. So, I put them on a custom
| regular compute box, slapped some partial caching on bits they
| didn't need and managed to bring the cost to slightly higher than
| what they were paying with their previous cheap hosting company.
| All this lead to a 4 hour downtime.
|
| I apologized to them profusely and built them a CMS from scratch
| that held their traffic and more and dropped their cost to 1/4th
| of what their competitors are paying. Today, this client is one
| of my best friends. They went from "Fuck this guy" to "Can we
| offer you a CTO role?" :)
|
| I make it sound like it's so easy, but it was a year long fight
| almost bundled with lots of humiliation for something I didn't do
| just to earn their trust and respect. Till date, they don't know
| the ex-consultant's screw up.
|
| In retrospect, this downtime is the best thing that happened to
| me and helped me to understand how you handle such scenarios and
| what you should do and not to do. In such situations it is
| tempting to blame other people around you, but in the long term,
| it pays off if you don't and solve it yourself.
| e67f70028a46fba wrote:
| Near miss: my first job I was working on a CRUD app for a huge
| bank. I was dumb and it was early in the enterprise era of
| software and I had built my own simple O/R tool based on codegen.
| Not a terrible tool all things considered and I was pretty
| pleased with myself.
|
| One night in bed I realized that if someone hit submit on the
| delete screen without filling in any criteria it would just
| delete the whole database.
|
| Not a fun drive in.
|
| Yes, we drove in in those days.
| throwaway6734 wrote:
| >One night in bed I realized that if someone hit submit on the
| delete screen without filling in any criteria it would just
| delete the whole database.
|
| how
| breakfastduck wrote:
| Lack of validation?
|
| I.e if no criteria, it could be sending a DELETE message with
| no where clause in SQL land.
| e67f70028a46fba wrote:
| Easy: no form input no condition on the delete
|
| aaand it's gone
| onion2k wrote:
| And this is why we use things like database users that don't
| have delete permission, and row-level security so users can
| delete things that don't belong to them.
|
| I have learned this from a very similar experience.
| nradov wrote:
| Years ago we were using NetApp Filers as storage for our database
| servers in a colo facility. During a planned maintenance window I
| installed a NetApp OS upgrade and brought everything back on
| line. At first it seemed fine but as soon as the database servers
| got some load they started dropping their connections to the
| NetApps and everything crashed.
|
| Of course I blamed NetApp and called their tech support screaming
| for help with their OS "bug". After hours of troubleshooting we
| finally figured out that the NetApp OS upgrade had included a
| network performance optimization and it was now sending out
| packets fast enough to overflow the buffer on our gigabit
| Ethernet switch. The packet loss rate was huge. Fortunately we
| had a newer switch back in the office so after swapping that out
| and repairing some corrupt databases I was able to get production
| back on line. Didn't get any sleep that night though.
| dhosek wrote:
| I wrote some software to handle charging customer's credit cards.
| It worked fine in the dev environment so a week later we deployed
| it since we hadn't been charging customers at all until that
| point. We would run bills once a day until all the unprocessed
| billing was caught up.
|
| Well, in dev, the database was refreshed with prod data every
| night at midnight, so we never saw the bug in my code. I had a
| sign error in updating the customer's balance so instead of
| lowering their balance by the payment amount, my code _increased_
| their balance. Geometric growth is an amazing thing. A few days
| later we had calls from angry customers because we had maxed out
| their credit cards. Miraculously, I was not fired. In retrospect,
| I think that it might have been because the manager would have
| then had to explain why _he_ had not made sure there was not
| adequate testing on something so central to the business.
| rootusrootus wrote:
| Many years ago (decades, in fact) as a fresh new excuse for a
| unix admin, I needed to hide the passwd binary so that users
| couldn't find it and change their local password on the terminal
| box (this was early ISP days). SunOS 4.1.4, as I recall.
|
| Anyway, I hid that binary. In /etc, where they'd never think to
| look.
|
| Gosh we do some dumb things, eh? LOL. That took a while to find a
| solution for, and no small amount of luck. The owner of the ISP
| walked back in the office a couple hours later and said "I heard
| you had some excitement?" I said, "Oh yes, it was pretty ugly for
| a bit." "Is it fixed now?" "Yup." "Carry on."
|
| For sure thought my ass was fired and I'd only been on the job a
| month or so.
| Nextgrid wrote:
| I don't get it - how would moving that binary break a running
| system? Is that binary somehow involved in something else
| beyond password changes?
| krylon wrote:
| /etc/passwd contains the user database on most Un*x systems.
| GP replaced it with the executable file, thus wiping out the
| system's users. Ouch.
| rootusrootus wrote:
| > Ouch.
|
| Ouch, indeed. We ended up getting lucky and found a
| workstation where someone had left themselves at a root
| prompt on another machine that had a shared NFS mount. This
| was before protection from this kind of attack, so we were
| able to create a setuid root script and run it on the main
| server to get root access to fix the broken passwd file.
|
| Our next step was going to be rebooting the server. We were
| pretty sure that faced with a corrupt passwd file, SunOS
| would drop to single user mode. Never tested that theory.
| Glad we didn't have to, the server in question was a hack
| job as it was. Copied over (literally, as files) from a
| previous server, it wasn't even 100% in agreement with
| itself on its own hostname, so I always kinda wondered how
| it would react to any big changes.
| zimpenfish wrote:
| Partition with the mSQL database on filled up; I moved it to
| /tmp. On a Solaris box. Which rebooted some weeks later.
___________________________________________________________________
(page generated 2021-06-26 23:02 UTC)