[HN Gopher] The day I locked everyone out of the company intranet
___________________________________________________________________
The day I locked everyone out of the company intranet
Author : dfcowell
Score : 122 points
Date : 2023-07-15 06:53 UTC (16 hours ago)
(HTM) web link (www.dancowell.com)
(TXT) w3m dump (www.dancowell.com)
| justsomehnguy wrote:
| pfft rmdir . /s /q
|
| only to notice what I am in the wrong folder, way up in the
| hierarchy.
|
| And repeat the same error years later when the batch file failed
| to cd to the destination folder. Added if %CD%
| == %DESTDIR%
|
| to avoid that problem.
| asmor wrote:
| I did worse.
|
| We had a redis with sessions. Early on, someone decided every
| write to redis should also cause a write to S3 as backup. My
| first task was to get rid of this 4-digits a month extra cost in
| PUT requests. I decided to instead write all changed session
| objects into a set keyed by half-hour timestamps and then write
| only those sessions every 30 minutes. Unfortunately initially I
| used a KEYS to find the set corresponding to my half-hour stamp,
| not having read up exactly on what it does. It's not exactly
| advisable to do on a redis with a million or so objects. A later
| version of the archiver wrote the last emptied set to a stable
| key instead and then checked the set keys between then and now
| instead...
| rrrrrrrrrrrryan wrote:
| A few months into my first job out of college, I brought down the
| main production server in the middle of the workday. It took us
| about an hour to recover. Afterward, I was very embarrassed and
| apologetic, but my boss just shrugged and said:
|
| "You're not a real technology worker until you've brought the
| company down. Welcome."
|
| Might not be the best words to live by, but it was exactly what I
| needed to hear at that time early in my career.
| IronWolve wrote:
| My first sysadmin job at a call center, the call center reps use
| the same directory for all the users. And, I'm working tickets to
| delete old users accounts...
|
| The old grey haired sysadmin backs up the directory so he can
| instantly restore it. Seems this happens all the time.
|
| Whew.
| croo wrote:
| My chooosen database explorer is Dbeaver. Horrible name but great
| app.
|
| You can set colours for local/test/prod servers and a red colored
| tab will scream at you to be cautious. And with red color every
| edit will pop up an "are you sure?" question. And autocommit is
| off.
|
| I sorta stopped making unrecoverable mistakes.
| acomjean wrote:
| DBeaver is great. I came from a different tool, and had to get
| used to persisting the changes. I'm totally use to it.
|
| I run a local copy of the database I connect to remotely and I
| have to be careful about which database/table I'm connected
| too.
| reaperducer wrote:
| _My chooosen database explorer is Dbeaver_
|
| I just checked it out, and it looks promising.
|
| Con: There are too many licensing options. I'm not even sure
| which one I would need.
|
| Pro: They'll send out an invoice, like a real company. I work
| for a billion-dollar healthcare company, and if your company
| only takes PayPal, or Venmo, or something else that makes it
| look like two guys in a basement, it'll never get through our
| purchase approval process. Big companies do business with other
| big companies. Startups have to learn to interface with
| businesses on a business level if they ever want to grow out of
| being a startup.
| fbdab103 wrote:
| Start with the Community Edition and see where it takes you.
| jareklupinski wrote:
| i'm using it on Mac, and it seems to want to update every time
| I turn it on... i get that having timely updates is great, but
| the way to update the app on a mac is:
|
| click ok on the update dialog, wait until the download
| finishes, shut down the app, drag the new app over to
| Applications, click Replace, wait for the actual moving dialog
| to pop up, wait for the moving dialog to finish, minimize all
| your windows so you can see the desktop, unmount the dbeaver
| image, click dbeaver again
|
| and you can get back to work! now if only I could remember all
| the details that were in my head when i opened the app in the
| first place...
|
| dbeaver team: can you do what paint.net did please? that is,
| only bother me to update as i'm turning the app off, not on?
| fbdab103 wrote:
| My only complaint about DBeaver is how powerful it is -I feel
| like I have only ever scratched 1% of the functionality it
| offers. I need a tutorial class on all of the goodies the
| platform can do.
|
| All of the tutorials I seem to find online are basic level
| and/or mixed with introducing SQL.
| psnehanshu wrote:
| I love DBeaver too.
| philderbeast wrote:
| "All of your colleagues have done something dumb. Don't be afraid
| to tell us when you make a mistake. We all remember our first
| screw up and will be happy to help."
|
| Never have truer words been spoken.
|
| As I tell all the new juniors at work doing sysadmin type tasks,
| everyone has deleted the production database at least once.
| Mistakes will always happen, it's how you deal with them that
| defines how good you are at the end of the day.
| russfink wrote:
| It's also a measure of how well management does in response. We
| were transferring a software production repository from one
| machine to another, Lord knows why, when a junior admin I was
| supposed to supervise had arrived a half hour early and started
| the operation without me. He got the source and destination
| arguments reversed in the file transfer; we were using DD for
| this one part of it.
|
| Management reacted pretty well. They assured both of us that
| while we made the mistake, it was not our fault that data was
| lost: the problem was that backups were not being checked,
| which caused us to lose the resulting three days (120 developer
| days) of work. The manager in charge of the folks doing the
| backup got taken to task - but nobody else did.
| headline wrote:
| I've just started my career - I wonder what my first fuck-up will
| be :)
| james-skemp wrote:
| Own up to it when it happens. Can only get worse if you don't.
|
| Hopefully you work with people that follow that as well.
|
| I can still remember a more senior coworker copying a directory
| from a network share (Ctrl + C), deleting the directory, and
| then trying to paste it somewhere else.
|
| I didn't speak up (he also rarely touched the mouse so flew
| through it), so we both got a chuckle when he realized his
| mistake and pinged network folks for a backup from tape.
|
| It happens. The real world is messy. Can always learn something
| new. Can always do something you know not to do.
| jdsalaro wrote:
| Don't wonder! Plan it and execute it so you can be more chill
| !!
|
| /s
| l0ngyap wrote:
| [dead]
| mootzville wrote:
| I was taught to:
|
| 1. Write your WHERE clause first 2. Return to the beginning of
| the line to finish writing the statement 3. Check your statement
| 4. If it looks good, then -- and only then -- add your closing
| semicolon
|
| Having said that, once during my second week at a new company, I
| plugged in an ethernet cable to an APC UPS, so I could set up
| networking on it. It shut down production. Why? APC makes (for
| that model at least) proprietary ethernet cables for networking,
| and if you plug in a regular cable it does an autoshutdown...an
| engineers attempt at marketing perhaps!? I did RTFM before, and
| after out of confusion, and there was no mention of this.
| sokoloff wrote:
| I will typically do something like this when writing SQL in an
| interactive tool: SELECT * from table
| WHERE id = 12345
|
| and once that's giving me the selection I want, insert the
| update statement into the middle: SELECT * from
| table -- UPDATE table set c1 = v1, c2 = v2 WHERE id
| = 12345
|
| Then, accidentally running the entire buffer doesn't do
| anything destructive, but selecting the query from update to
| the end of the statement lets me do the update. (It's still
| imperfect, because selecting only the update line will still be
| destructive.)
|
| (Most of the RDBMS tools that I've used would happily ignore
| the lack of a closing semi-colon and that will not save you for
| a single-statement case.)
| mootzville wrote:
| True, thought I didn't include the lookup first, it is how it
| happens in practice. Also, we use the mysql client, so
| semicolon is a must, and I would avoid anything that let you
| submit the statement without it...that's your safety net.
| fukawi2 wrote:
| I was working on an old old old "ERP" system written in D3 PICK.
| It's a database, programming language and OS all in one with
| roots in tracking military helicopter parts in the 1960's. I was
| working on it in the mid-2000s.
|
| It had SQL like syntax for manipulating data, but it was
| interactive. So you would SELECT the rows from the table that you
| wanted, then those rows would be part of your state. You would
| then do UPDATE or DELETE without any kind of WHERE, because the
| state had your filter from the previous SELECT.
|
| It has a fun quirk though - if your SELECT matched no rows, the
| state would be empty. So SELECT foo WHERE 1=2 would select
| nothing.
|
| UPDATE and DELETE are perfectly valid actions even without a
| state...
|
| Working late one night, I ran a SELECT STKM WHERE something that
| matched nothing, then before I realised I realised my state had
| no rows matched, I followed up with DELETE STKM.
|
| Yep, the entire Stock Movements table four the last 20+ years of
| business were gone.
|
| The nightly backup had not run, and I didn't want to lose an
| entire day of processing to roll back to the previous night.
|
| I spent the entire night writing a program to recreate that data
| based on invoices, purchase orders, stocktake data, etc. I was
| able to recreate every record and got home about 9am. Lots of
| lessons learnt that night.
| cvoss wrote:
| > if your SELECT matched no rows, the state would be empty
|
| > UPDATE and DELETE are perfectly valid actions even without a
| state
|
| Some may call this a fun quirk :) but I'd call it a horrible
| mistake in the design of the system! It should have been
| conceptually obvious to the designer that an empty set of rows
| is a perfectly valid state and is fundamentally different from
| "no state".
| bombolo wrote:
| At university I had to use a programming language to do some
| specific calculations.
|
| It was obviously designed by someone who had not taken any
| courses in formal languages, compilers and algorithms.
|
| It had a fun quirk, the final result would change depending
| on which names you used for your variables.
|
| When the phd student guiding our lab session told me to not
| use underscores in my variable names, I thought i'd humor her
| just to show her it was a stupid suggestion. But she knew
| more than me.
| albert_e wrote:
| You recreated a production database of 20 years data by hand
| overnight?
|
| You deserve an award !!!
| fukawi2 wrote:
| Yep, the entire table. I figured since I had to write it to
| do the day at a minimum, doing the whole table would help me
| find corner cases and errors since it was a much larger range
| of data
| testemailfordg2 wrote:
| Only that day's data was added on top of backup from last
| night...
| unnouinceput wrote:
| he only recreated the ones not saved by the nightly backup,
| meaning he used the previous nightly backup and recreated
| from invoices the last ~23 hours.
| fukawi2 wrote:
| No, I did the entire table. That helped me find corner
| cases and errors in my logic having such a bigger range of
| source data.
| tester756 wrote:
| Wow,
|
| I'd still not touch previous data.
|
| I'd use it just for ensuring correctness, but I'd not
| touch it
| stevezsa8 wrote:
| Maybe I don't fully comprehend what you're saying... but
| is it even possible to do this?
|
| I mean what if at some point particular entries were
| manually tweaked and the database was updated to fix an
| error in an invoice or something. And then you recreated
| data from what you assume is 100% reliable data.
|
| I'm happy to try to understand if you don't mind
| simplifying the explanation.
| Aardwolf wrote:
| Where did you get the 20 year old data from in that case?
| Actual paper in drawers, a directory of all invoices on
| disk, or some other database?
| EricBetts wrote:
| PICK is giving be flashbacks of my tenure at ADP's dealership
| services (I think it became CDK)
| vivekv wrote:
| Whe. I first learned SQL on an oracle cli client I was told to
| type begin; then any other command. For a brief while until i
| learned about transactions, I assumed that it was part of the
| cli startup :)
| Gordonjcp wrote:
| I had a thing recently where someone was updating some entries on
| a system I look after, decided the changes hadn't applied
| properly, and clicked "Roll back" to put it back to its original
| state.
|
| Whatever had gotten into it, it rolled back to 2009. It rolled
| everything back, including user accounts.
|
| No-one who worked there in 2009 still worked there, so no-one had
| a valid password any more.
|
| Fortunately it was easy enough to copy the last-but-one backup
| over the top and lose the day before's config updates, and cure
| its Flowers for Algernon state, but it was a pretty hairy
| afternoon.
| lormayna wrote:
| I did something worst many years ago: I was working for a
| regional ISP and during a major incident, I had to reroute
| traffic through a different path. Under big pressure, I did the
| infamous Cisco mistake "switchport trunk allowed vlan 50" instead
| that "switchport trunk allowed add vlan 50" and I locked out
| myself and all the customer from our broadband customers. We had
| to call a DC technician and ask him to share a console through a
| local console server.
|
| Lesson learned: even if you are under big pressure take your time
| to plan and review the modification 15 minutes can save hours.
| matsemann wrote:
| I wish a DELETE or UPDATE only affected a single row by default
| (and perhaps even wouldn't commit if it would hit multiple rows),
| unless a keyword for MANY or something similar was added.
|
| Aka DELETE ALL where x == y or DELETE MANY where x == y or
| perhaps you need an explicit limit for it to not be 1, so DELETE
| where x == y LIMIT ALL
| sangriafria wrote:
| There are some SQL GUIs that require confirmation before
| running an update/delete query without a where clause
| stuaxo wrote:
| My manager had got me to look at backups.. but for cheap.
|
| I decided on bacula - I had the clients installed on all the
| computers in the office, and it worked for some small tests.
|
| My manager decided we would try this with a USB drive attached to
| one of the servers (somehow this didn't seem like a bad idea).
|
| In the morning, very uncaffinated he sent me to the other site -
| an unmanned basement office with the servers.
|
| Being uncaffinated I forgot the door password and set off the
| alarm.
|
| I had to go into the office and phone him with the alarm going to
| get the code to turn the alarm off.
|
| OK, that was stressful but sorted out at least.
|
| I plugged in the hard drive to the selected server and headed
| back.
|
| Once I got back it turned out all the websites on that server had
| gone down - trying to send all the backups to this poor USB
| harddrive had overwhelmed the IO on that-era Linux server and the
| poor thing just froze.
|
| Fairly soon after I was let go, and joined my friends at a much
| more fun company making mobile games.
| sacnoradhq wrote:
| It's the responsibility of the technical person to uphold
| engineering ethics, especially in the face of potentially
| inadequate recovery and security solutions.
|
| I was once let go from a big name university for refusing to
| weaken and rush changes to a payment processing network (PCI-
| DSS) when there was "no time" to review them in detail. That's
| a future FBI press conference sort of thing when it all comes
| crashing down. Not long after, all SS#s, DOBs, and deets for
| every employee was stolen from a "rogue" laptop taken by a
| consultant, likely to be sold on carder and identity theft
| forums because of an utter failure at data protection
| processes. That place was a shitshow because they didn't have
| the professional ethics or leadership backbone to do what was
| prudent and necessary.
| nonameiguess wrote:
| Early on when I'd first started making the transition from pure
| developer role working only on product to a platform role running
| the development environment, I was encountering problems with
| build scripts on CI servers leaving behind a bunch of dead
| symlinks. Tired of tracking them down manually, I wrote a nice
| script that automatically found all dead symlinks and deleted
| them.
|
| It turned out, for some arcane reason I still don't understand,
| our production instance of Artifactory was running on top of
| Docker Compose with host path volume mounts, and somehow,
| symlinks that were not valid from the perspective of the host
| actually were valid from inside the container, and doing this on
| all of our servers broke Artifactory. For some even stupider
| reason, we weren't doing full filesystem-level snapshots at any
| regular interval (which we started doing after this), so instead
| I needed to enlist the help of the classic wizard ninja guy who
| had been acting as a mostly unsupervised one-man team for the
| past six years who had hacked all of this mess together,
| documented none of it, and was the only person on the planet who
| knew how to reconstruct everything.
|
| This was probably still only the second-stupidest full on-prem
| lab outage I remember, behind the time the whole network stopped
| working and the only person who had been around long enough
| remembered they had trialed a temporary demo hardware firewall
| years earlier, management abandoned the evaluation effort, and it
| somehow remained there as the production firewall for years
| without ever being updated before finally breaking.
| cesaref wrote:
| Back in the 90s I remember a work colleague asking 'can you
| rollback a drop table?', to which I replied 'no', and all the
| blood drained from his face in seconds. It's one of those things
| you've heard happens to people, but until you see it, you can't
| quite believe it.
| jdsalaro wrote:
| > can you rollback a drop table
|
| >_< , something similar happened to me when I did an rm -Rf *
| in the old pictures directory on my system ... Well ... It
| wasn't the old pictures directory, it was the backup directory
| with all pictures from my older phone !! To say that blood
| drained from my face would be an understatement, thankfully I
| was able to recover most of it.
|
| The first thing I did after the recovery marathon was to alias
| rm so that it instead works by moving stuff to /tmp
| euroderf wrote:
| My boss was doing housecleaning in his PDP-11 (Unix v.6)
| account and did a "rm *" in the bin directory but OOPS it was
| the system bin.
| noAnswer wrote:
| inb4: /tmp partition gets full. Causing things to brake. :-)
| eCa wrote:
| In the late 90's I wanted to try this Linux thing, so I
| followed a tutorial. First step: fdisk
|
| Yes, that was my Windows partition going bye-bye.
| CoastalCoder wrote:
| So... success? ;)
| jdsalaro wrote:
| :D task failed successfully !
| [deleted]
| msravi wrote:
| Since this seems to be such a common occurrence, why haven't
| databases evolved to maintain some limited form of history that
| can be rolled back? Have no idea of the complexity, so forgive
| me if that sounds daft.
| [deleted]
| kayodelycaon wrote:
| Long story short, you're holding multiple transactions open.
| That's a lot of bookkeeping overhead if there's a lot of
| changes. A system with a few million rows and moderate load
| on a reasonably-sized server for that load will slam to a
| halt.
|
| Then come the "site is down and the world is ending" tickets,
| messages, emails, and phone calls.
| freetanga wrote:
| Always do a select with your criteria before doing a Delete or
| update.
|
| Don't ask me how I learned this.
| chrisandchris wrote:
| That, and some DB tools (line JetBrains DataGrip) block UPDATEs
| and DELETEs without a WHERE condition.
| tgv wrote:
| Then start a transaction, and only commit when the row numbers
| match.
| iJesus wrote:
| Gone are the days of hard deletes in my approach; it's
| exclusively soft deletes now!
| okeuro49 wrote:
| In the EU soft deletes can violate GDPR, so not always an
| available strategy.
| vntok wrote:
| Soft delete items immediately, purge the soft deleted rows
| automatically after 30 days or so.
| lemper wrote:
| how did you learn it?
| gumballindie wrote:
| Came here to say exactly that! I do a select count and then a
| limit against the returned count. At least it may reduce the
| blast radius.
| menacingly wrote:
| In most of my scenarios, I'd actually rather cause a
| catastrophic global change than a silent subtle corruption of
| a handful of rows
| gumballindie wrote:
| That may be fun in a trivial setup such as op's but when
| millions of customers or billions of transactions are
| affected it's a nightmare. A competent engineer runs
| queries against a local and then a uat db, verifies results
| and then on prod. But if you must do it in prod then it
| must be limited in scope.
| jojobas wrote:
| >Don't ask me how I learned this.
|
| Joke's on you, we all learned it this exact way.
| seanthemon wrote:
| This gives my heart pains I can't explain to my therapist..
| maccard wrote:
| Luckily I learned from someone else. I did figure out start
| with a transaction the hard way though
| fernandotakai wrote:
| i can still feel the pain in the pit of my stomach when i saw
| the amount of rows affected.
|
| thankfully my boss at the time was a amazing db admin and he
| helped me fixing my mistake.
| _nalply wrote:
| Once I wanted to do `rm -fr *~` to delete backup files, but the
| `~` key didn't register...
|
| Now I have learnt to instinctively stop before doing anything
| destructive and double-check and double-check again! This also
| applies to SQL `DELETE` and `UPDATE`!
|
| I know that `-r` was not neccessary but hey that was a biiiig
| mistake of mine!
| ht85 wrote:
| If you ever type really dangerous commands, it is good
| practice to prefix them with a space (or whatever your
| favorite shell convention is) to make sure they not saved in
| your history.
|
| One of my "oopsies" involved aggressive <up> and <enter>
| usage and a previous `rm -rf *` running in /home...
| CoastalCoder wrote:
| I didn't know about the space thing!
|
| Though for me, I think the greater risk would be from not
| having a record of what I'd run.
| tetha wrote:
| For entirely critical systems, I by now rather generate a
| reviewable script and run that. Something along the lines of:
| find /backups/ -mtime '-30' -printf 'rm -f %p\n' >
| very_scary_deletes.sh
|
| This gives you a static script with a bunch of rm's. You can
| read that, check it, give it to people to validate and when
| you eventually run it, it deletes exactly those files.
| _shantaram wrote:
| One time I was debugging the path resolver in a static site
| generator I was writing. I generated a site ~/foo, thinking
| it would do /home/shantaram/foo, but instead it made a dir
| '~' in the current directory. I did `rm -rf ~` without
| thinking. Command took super long, wondered what was going
| on, ctrl-c'd in horror... that was fun.
| lostmsu wrote:
| I click in Explorer. Have to confirm too. Never been
| confused.
| laurensr wrote:
| You remind me of that time I wanted to `rm -rf ./*` but the
| dot hadn't registered... I now avoid that statement.
| justinclift wrote:
| At a previous place I worked, if they were working on the cli
| (eg in psql or similar) they'd always use these two steps,
| either of which would provide adequate protection:
|
| 1. Start a transaction before even thinking of writing the
| delete/update/etc (BEGIN; ...)
|
| 2. Always write the WHERE query out first, THEN go back to the
| start of the line and fill out the DELETE/UPDATE/etc.
|
| It worked well, and it's a habit I've since tried to keep on
| doing myself as well.
| alexmolas wrote:
| The other learning I get from this story is "never hide your
| errors" or "own your errors as you own your victories".
|
| If the author had decided to say nothing the problem would have
| been bigger - an unhappy boss and probably fired.
| liampulles wrote:
| I switched companies, and moving from a MySQL cli to pgadmin is a
| godsend. Would still like a confirmation dialogue, but having to
| click a button seems less error prone than pressing enter too
| quickly.
| user6723 wrote:
| These kinds of scenarios happen when money is "cheap", and
| highlight why the current recession, and coming 2nd great
| depression aren't really a bad thing.
| CoastalCoder wrote:
| I think you'll need to flesh out your logic if you want to
| convince people that a depression is somehow a net positive.
| dfcowell wrote:
| This was 13 years ago in a small business with no significant
| investment. No "cheap money" was involved, just the realities
| of a small business with chronic NIH syndrome.
| rsynnott wrote:
| ... Okay, I've seen people blame monetary policy for a lot of
| things, but I'm really struggling to see how this one works.
| TheGigaChad wrote:
| [dead]
| drsim wrote:
| I wrap all of my production manipulations in a transaction, and
| commit only if the results are expected. Yes, it may take locks
| that block customer-facing transactions, so I have selects ready
| to go in the transaction to minimise this.
|
| 15 years and counting since wiping out a large production table
| and taking a day to restore from backup.
| m348e912 wrote:
| We had this saying: "If you fix it before anyone realized it was
| broken, you didn't break it."
| quijoteuniv wrote:
| Use chatGPT to double check too.
| throwbadubadu wrote:
| "Yes, you can delete those items via that query."
|
| "But now all has been deleted??!?! WTF HEEELP"
|
| "I apologize for my previous answer, you are correct, you now
| deleted everything!"
| archerx wrote:
| Thank you for the laugh, I can imagine someone running
| untested chatGPT code on prod and this actually happening.
| quijoteuniv wrote:
| Mmm... you can ask to explain what the query will do... yes
| you can mess thing up if you do not understand, therefore you
| try to understand. There is a lot to learn, i did not
| recommend running code in production frOm chatGPT, funny with
| downvoting
| FreshStart wrote:
| 5. Deadlines do more damage than good, but the costs they produce
| get swiped under the rug by company internal accounting.
| Nextgrid wrote:
| In the top bar of the site:
|
| > Sorry! Subscriptions were broken last week, but are now
| working. If you tried to subscribe and ran into issues, please
| try again!
|
| I wonder if a similar incident involving a "UPDATE subscriptions"
| query happened recently.
| dfcowell wrote:
| Ha, no, that was a far more mundane issue. The CMS I'm using
| requires double opt-in to subscribe, meaning you need to enter
| your email address _and_ click the confirmation link.
|
| It also apparently requires double email configuration, meaning
| it has two places where you can configure your mailer. I had
| only set it up in one place, meaning the confirmations never
| got sent.
|
| Bit of a facepalm moment.
| nytesky wrote:
| For some reason your title made me think of this classic IT Crowd
|
| https://youtu.be/Vywf48Dhyns
| Animats wrote:
| If you locked out everyone today, many would assume they had been
| laid off. Some of them would have other jobs before the login
| problem was fixed.
| makach wrote:
| A charming story which almost everyone can relate to! Only one of
| your rules will ever save you. "Don't run updates directly in the
| database console". Whole methodologies are crafted around this
| rule/principle to not do development in production environment.
___________________________________________________________________
(page generated 2023-07-15 23:00 UTC)