[HN Gopher] The absolute worst scenario happened
___________________________________________________________________
The absolute worst scenario happened
Author : logronoide
Score : 525 points
Date : 2021-03-22 11:43 UTC (11 hours ago)
(HTM) web link (www.reddit.com)
(TXT) w3m dump (www.reddit.com)
| undefined1 wrote:
| good idea in that thread:
|
| If you don't have a backup of your zone files, a good way to
| quickly pick up the main domains being used would be to turn on
| logging for your DNS server and output into a log collector. That
| way you can quickly build queries on which IPs are asking for
| which FQDNs and start rebuilding a list of your lost zones and
| can maybe do some guess-work on which IPs those FQDNs were likely
| to resolve to.
|
| I would expect irregular or extremely infrequent processes that
| use specific FQDNs will pop up from time to time as errors/failed
| processes and should prime IT teams on what to look out for.
|
| https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
| shireboy wrote:
| On a personal level, this may be a little self centered, but I
| use cases like this to put my own problems in perspective.
| Recently I had to troubleshoot lots of IT issues during a winter
| storm. It was a bad outage for the organization during high load
| important scenario. I distinctly remember thinking during the
| thick of it "this is bad, but imagine being the poor engineers in
| TX responsible for the power grid." This helped me not panic and
| focus on the problem at hand.
|
| I feel like this ability to step back and not take a problem more
| seriously than necessary can be an asset. In this case, it's
| pretty bad, but "at least people aren't in immediate danger of
| their life." Think it to yourself, but probably not best to share
| with others until after the problem is mitigated ;)
| simonw wrote:
| So many beautiful details if you search for more comments by the
| original author of the post:
|
| "The company that built this ERP solution went bankrupt 6 years
| ago, and we don't have the source code. It uses PostgreSQL and
| MySQL, and also has a built-in key-value database for which none
| of us has any credentials. We don't have the source code for this
| software (it's built in C and delivered to us as compiled
| packages..)."
| matwood wrote:
| Extremely poor planning/lawyering on the companies part. When
| larger entities deal with smaller entities, it's common to have
| a source code escrow as part of the agreement which grants the
| purchasing company rights to the source code in the event of
| this exact situation.
|
| Obviously we all know that source code alone isn't enough, but
| it is something to start from.
|
| If the ERP solution was not SaaS and installed on site, the
| purchasing companies IT should have gotten credentials as part
| of the contract. Again, normal type stuff when buying software
| from smaller companies that have somewhat higher risk of going
| out of business.
| chmod600 wrote:
| There would be a lot of value in independent organizations that
| can evaluate engineering practices. Non-technical businesses
| could just ask if you have cert XYZ before paying you.
|
| It wouldn't have to get into the weeds too much. I mean really
| basic stuff like finding documents and following them to restore
| a backup or release a new version or fix a simple bug injected in
| the system.
| ReptileMan wrote:
| No keyboard found, press f1 to continue type of situation.
| gist wrote:
| > So, everything is fucked. We have (had) a custom DNS system
| built into our custom ERP software (don't ask). It had an
| integration to our old (bind) nameservers, which was stragith up
| awful and we've been trying to replace it for years. Well, on
| Friday it all broke down. All domain information was wiped out
| and records became null. Our company's domain is down, as well as
| our customer domains (we're a medium-sized MSP). Every person who
| knew how the custom DNS system worked has left the company years
| ago.
|
| Unclear why HN wastes time on this type of 'story' with no
| attribution and scant details. (I have expertise in this area
| let's say).
|
| To start this implies that they have not added a customer or made
| a change in several years by this statement:
|
| "Every person who knew how the custom DNS system worked has left
| the company years ago."
|
| This also contradicts:
|
| "and we've been trying to replace it for years"
| xchaotic wrote:
| has anyone figured out who the company is by now? Surely there
| must be some reports of the outage by now?
| viraptor wrote:
| There's a small chance it's just a made up story for fun. If
| not, some outlets should pick it up soon.
| the_duke wrote:
| Not necessarily, there are many small companies in this
| space.
|
| But knowing Reddit, I'd give this a solid 50/50 chance of
| being fake.
| avaldeso wrote:
| > Surely there must be some reports of the outage by now?
|
| Not everyone is AWS. There's a lot of obscure software
| providers nobody cares when they're down.
| WJW wrote:
| That is an amazing story. Who knew that putting everything in a
| database without backups was risky?
| PeterisP wrote:
| They explicitly mention that they have backups, but that the
| "backups are fucked", whatever that means.
|
| Ransomware scenarios come to mind, where the attackers will
| look for and actively destroy the backups you have if they are
| remotely accessible with e.g. AD admin credentials, and only
| trigger the ransomware when the backups are gone. If you have
| offsite backups in some cloud system, that helps you against
| natural disasters but not against malicious activity. The need
| for backups that are not just offsite but also offline is
| somewhat recent and very, very many companies do not have them.
| fuzzy2 wrote:
| They backups are probably fine (as in "the data is in there")
| but there may be no plans on how to restore them. Or they may
| be incomplete.
|
| I doubt any external factors are to blame.
| brailsafe wrote:
| Didn't they only say that their backups are fucked, but not
| that they didn't try to make them? I've definitely run into
| situations where the business is only willing to pay for the
| most rudimentary conception of a backup strategy, and then it
| turned out to have been subject to corruption.
| M2Ys4U wrote:
| They said they knew they haven't been able to restore backups
| for 2 years. Which is another way of saying that they don't
| have backups at all.
| ewindal wrote:
| Backups were extant, but inaccessible, and non-functional due
| to causing a kernel panic when applied. [1]
|
| [1] https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_abs
| olu...
| pedrovhb wrote:
| Can't find it now, but I've often seen a phrase to the effect
| of "If you don't regularly test that you can restore from
| backups, then you don't actually have backups", and it seems
| to ring quite true here.
| regularfry wrote:
| It's up there with "unless you test power supply failover,
| don't rely on that generator to save you".
|
| I know of one case where power failed, the "24 hour"
| generators kicked in, power company says "We'll need 12
| hours" and the outage happened 6 hours later when they ran
| out of oil.
| arethuza wrote:
| I contracted at one place for a bit where they shut
| _everything_ down in each of their data centres once a
| year and power everything back up.
|
| When I was there this didn't go too well and they
| couldn't get one their data centres online again -
| failover to their other centres did work though. This was
| ~20 years ago in the finance sector.
| grumple wrote:
| That sounds like their testing worked out for them.
| Better than a random failure.
| myself248 wrote:
| Yup. Have the problem when all the right people are awake
| and on-site to handle it.
|
| I was in a building when someone inadvertently powered
| off the wrong equipment, which had been running for
| several years, and several of the power supplies failed
| to come back up. It was 1+1 redundant though, so we could
| quickly shuffle packs around to bring it back up without
| redundancy. Then, jogging through the building and asking
| if anyone had spares, we found a field tech in the
| lunchroom who had a pile of stuff in his van. Whole thing
| was back to 100% in less than an hour, and we let the
| beancounters sort out the field spares being used for
| office equipment.
|
| If that same failure had happened during the overnight
| maintenance window (when volatile work was supposed to be
| performed), there certainly wouldn't have been the same
| resources around.
| arethuza wrote:
| I'd go one step further - actually restore the databases
| and check that you can bring up the relevant applications
| that use those databases.
| [deleted]
| kodah wrote:
| I'm a systems engineer and a software engineer. A couple tips
| that are relevant to this story:
|
| 1. Applications do not need DNS servers, ever. Hosting platforms
| do. The way I read this is that it's some odd form of split
| horizon where one DNS server seems to replicate to larger DNS
| servers, yet the single server is what carries most of the
| critical information.
|
| 2. State should be transferred from where you gather input to
| where it becomes actionable by an eventually consistent process.
| ie: If you store your customer CNAMEs in ERP, then something like
| webhooks to a service that maintains the state in machine
| readable form should verify, accept/deny, and do something with
| that change (like update DNS).
|
| 3. Customer systems and core/critical systems should be separate.
| The fact that critical contact systems for employees were not
| separate from
|
| 4. Have a DR plan and artificially enact it at cadence, doing so
| even in a sterile environment is better than nothing. Successful
| DR's in production are best. If the first time you live test your
| DR strategy is when all is lost, then all is most likely lost and
| at best you have a lot of hours of work ahead of you.
|
| 5. The holy grail is replayable audit frameworks. This could mean
| capturing diffs but it can be as simple as logging the current
| state of a given $thing and sending it to an off-local system.
| protomyth wrote:
| Well, it looks like someone better track down the people who knew
| how it worked and offer some serious cash.
|
| The only actual backup is one that has been tested. With the cost
| of hardware, buy 2 or have an agreement with your vendor to have
| a machine ready to receive a backup.
| sethammons wrote:
| > The only actual backup is one that has [recently] been tested
| ExcavateGrandMa wrote:
| There is always a solution...
|
| but don't redo the same errors :)
| erikstarck wrote:
| This reminds me of when the Danish mega-corp Maersk was attacked
| by ransomware and had all of their computers locked and decrypted
| with no access.
|
| All computers except one, which due to a power failure had been
| offline the whole time.
|
| This computer had the critical DNS information needed to restore
| the network.
|
| Only problem: it was in Africa, in a country that required Visa
| which would take weeks to apply for and receive. So someone from
| the African office had to bring the hard drive to an airport that
| someone from the London office could travel to and pick up the
| critical hard drive, then bring it on a plane back to London.
|
| A sweaty and nervous trip, I can imagine.
|
| Whole story here: https://www.wired.com/story/notpetya-
| cyberattack-ukraine-rus...
| driton wrote:
| Sounds very interesting, but the story seems to be behind a
| paywall. I found some other articles[0][1] related to the
| incident, however none of them seem to mention the flying of a
| hard drive across continents.
|
| 0: https://www.zdnet.com/article/ransomware-the-key-lesson-
| maer...
|
| 1: https://portswigger.net/daily-swig/when-the-screens-went-
| bla...
| erikstarck wrote:
| Sorry about the paywalled article. Here's a summary of the
| same article:
| https://www.chrislouie.net/blog/2018/9/10/better-to-be-
| lucky...
| Eremotherium wrote:
| Or if you want the story as a podcast I can wholeheartedly
| recommend this episode (and the podcast in its entirety) of
| Darknet Diaries:
|
| https://darknetdiaries.com/episode/54/
| pro_zac wrote:
| Seconded! Great story and amazing podcast.
| flobosg wrote:
| There's a capture in the Wayback Machine: https://web.archive
| .org/web/20180822190926/https://www.wired...
| yabones wrote:
| Not just DNS, it was their entire Active Directory Global
| Catalog. If it weren't for that server, they would have needed
| to re-create their entire user database, all of their endpoint
| & server management policies, and it would have been almost
| impossible to safely restore their Exchange email database.
| They got very _very_ lucky.
| hef19898 wrote:
| And that is why I choose to host "my company" at the big names:
| homepage directly with Wordpress instead of a smaller local
| provider, office directly with MS and so on.
|
| Because, even there might be cheaper solutions, at least I can be
| sure to have a huge org in place to make sure stuff is up. And to
| find ways to get stuff back up and running in case something goes
| south. I value that peace of mind a lot.
| mikewarot wrote:
| As a system administrator, I never, EVER encrypted data at rest.
| Stupid policy I know, but encrypted stuff tends to be
| unrecoverable when you need it.
|
| I had an Exchange Server that had issues, and I didn't have
| enough hardware to try a backup, I kept telling management... it
| died, and I was able to recover most, but not all of it.
|
| Shortly thereafter they outsourced IT, and I assume the new crew
| was able to get the resources to actually fix it.
| Nextgrid wrote:
| This doesn't sound _that_ bad. Unless I'm misunderstanding
| something, the ERP was used as a source of truth for DNS records,
| and that mechanism broke. There's no indication of actual data
| loss so it should still be in the ERP DB, and it's a matter of
| reverse-engineering how it was all put together and then extract
| the data into a zone file (or into a managed DNS service such as
| Route 53) at least for the core domains which should at least
| bring their internal services back online and allow them to
| proceed further.
| rebuilder wrote:
| From a reply the poster made in that thread:
|
| "So, we have backups on an encrypted drive, which we can't
| access (authentication is done via our custom ERP software,
| which is down as well - in hindsight, a very bad idea). And the
| backups don't work. We can rebuild the DNS servers from
| scratch, and the records are probably still safely stored
| somewhere inone of the 3 different databases our ERP software
| uses, but we don't have an authentication key or the scripts
| used to pull the DNS domain / record information from the ERP
| system anymore. The API's are undocumented, and there's only
| one key that can access those records."
|
| Elsewhere, they say that two years ago, they found they were
| unable to restore from backups...
| raverbashing wrote:
| I'd say if it was such a house of cards, my best bet is that
| this security chain is full of holes. Maybe the ERP used a
| fixed password, which can be guessed/bruteforced.
| mysterydip wrote:
| honestly, "password" would be my first guess.
| jiggawatts wrote:
| Oh come now, there are password complexity policies in
| place in any modern software! Try: password123.
| fctorial wrote:
| Need capital letters and symbols:
|
| Password@123
| abruzzi wrote:
| it doesn't sound like modern software. At my current
| employer (to remain unnamed) when I started there were a
| bunch of MS-SQL servers with an SA password of <blank>
| (i.e. no password). It took a long time to fix that
| because we had hundreds of local installs of the client
| app that had SA/<blank> in the app config file (yes, they
| were setup to access as system administrator), so fixing
| it meant reconfiguring hundreds of local installs. (at
| the time we didn't have a mechanism for automating it.)
| We still have "enterprise" software that does no password
| rules, and would happily allow a blank password.
| yrgulation wrote:
| Oh please, even such a crappy place knows better.
|
| Its Password123. Adding capital letters makes it more
| secure.
| neogodless wrote:
| But hackers might be expecting capital letters, so
| disallowing them from passwords is more secure!
| ansible wrote:
| > _Elsewhere, they say that two years ago, they found they
| were unable to restore from backups..._
|
| So their business actually failed a couple years ago, and
| they have only noticed this month.
| ptero wrote:
| That isn't the failure of the business; backup failure is a
| screwup of the IT, which I think is not that uncommon.
|
| But the fact that after this event the management did not
| get it fixed (and requested that IT demo it, three times,
| with no failures) is an existential failure.
| lupire wrote:
| They noticed 2 years ago, and the ship sank below the water
| line this month.
| TeMPOraL wrote:
| They were already dead 2 years ago, the body just didn't
| realize it and kept moving until last Friday...
| hackeraccount wrote:
| Basically Wile E. Coyote running in the air.
| koonsolo wrote:
| The technical people noticed, left, and they had to hire
| new ones who didn't know yet. Now the new ones also know.
| arethuza wrote:
| I guess if you are in the business of selling services that
| include custom domains that isn't _quite_ as crazy as it
| initially sounds...
|
| You are, of course, assuming they have a backup and the backup
| can actually be restored and that the restore contains the
| required information.
| WJW wrote:
| It does say in the original post that:
|
| > Our backups are fucked.
|
| So I assume that they have already tried and they didn't
| work.
| arethuza wrote:
| Early in my career (>30 years) ago I nuked a companies
| salary database (a missing $ in a shell script moved
| everything to the file i)
|
| No problem they said - we have backups. They had three
| tapes.
|
| First tape failed.
|
| Second tape failed.
|
| Third tape worked.
|
| In retrospect that was a useful, if rather stressful,
| lesson.
| lupire wrote:
| > nuked a companies _salary_ database
|
| > a missing _$_
|
| Bravo!
| artursapek wrote:
| >a missing $ in a shell script moved everything to the
| file i
|
| I've always had anxiety when running a shell script, this
| is a perfect example of why
| noisy_boy wrote:
| I always wrap the actual cp/mv/rm etc in echo first
| before actually running it e.g. running echo "mv $src i"
| first would have immediately highlighted the problem
| visually.
| ansible wrote:
| It is also helpful to have at least two tape drives, and
| check the backup tapes against _both_.
|
| Early in my career, we had a storm that caused an
| electrical surge. In spite of the power protection, it
| damaged the tape drive on the main server (unknown to us
| at the time). The entire rest of the system (CPU, RAM,
| hard drives (RLL!)) were fine.
|
| Listening to the backups being run, I noticed it sounded
| a little different. Eventually I determined that the tape
| drive (old QIC, maybe 40MB capacity?) worked, but it
| would not switch tracks when reading / writing. So the
| backup ran, but it just overwrite the same track rather
| than write to subsequent tracks.
|
| A short backup / restore wouldn't have caught the
| problem, it was necessary to try to read the whole backup
| to see. This wasn't caught for a while, so all the
| backups were mostly useless.
| pc86 wrote:
| > > All domain information was wiped out and records became
| null.
|
| > > Every person who knew how the custom DNS system worked has
| left the company years ago. Our backups are fucked. Our records
| are wiped from all domain servers out there.
|
| Sure sounds like the data is lost to me.
| buro9 wrote:
| To me it sounds like those people who left can command a very
| high day rate to help with the immediate aftermath. A high
| enough rate that even if they are doing hours elsewhere some
| will be tempted to sign in after their work hours to help
| out.
|
| A company does not just stop existing or being able to
| function, deleting DNS records didn't empty the bank account
| or the ability of the accountants to be directed to pay
| someone... there are lots of options.
| goatinaboat wrote:
| _To me it sounds like those people who left can command a
| very high day rate to help with the immediate aftermath_
|
| Those people are probably employed somewhere else and would
| need very significant incentives. More than likely their
| current employers would take a dim view of them freelancing
| to help a competitor, so: enough to take a year off or even
| to retire on. Basically the idea of hiring back former
| employees as consultants is a non-starter.
| IgorPartola wrote:
| We don't know that the records are still in the database. This
| could be a matter of ON DELETE CASCADE followed by a delete of
| what looked like innocent data and then boom it's all gone.
| Then the integration updates bind which helpfully erases all
| records and then once there are no domain names to talk to
| everything is down.
|
| I would be looking at the Internet Archive and Whois too figure
| out some basic domains and what the registrars are and go from
| there but of course depends on what business they are in and
| what their domain names were doing.
| ragnese wrote:
| Yeah, the poster didn't make totally clear whether all of their
| _business_ records are totally gone, but it does sound like
| that 's the case. The ERP was very likely the source of truth
| for everything- including employee information that the poster
| says is gone (can't send SMS to remote employees).
|
| So, it _probably_ is that bad...
| [deleted]
| Nextgrid wrote:
| Nothing in his post suggests the ERP's DB is gone, merely
| that the mechanism which populates DNS records from the ERP
| has broken down, which probably has ripple effects on
| everything else as their internal DNS is broken and services
| can't talk to each other.
|
| But the data should still be there, and it's a matter of
| reverse-engineering how to go from "ERP DB" to "DNS zone
| file".
| ragnese wrote:
| Okay, fair enough. I probably skimmed the post too quickly.
| Semaphor wrote:
| > Skimmed through the MySQL and Postgres databases, no
| signs of anything domain-related. I'm betting it's all
| stored in the custom integrated [and encrypted] KV
| database..
|
| -- https://old.reddit.com/r/sysadmin/comments/ma4mwl/the_ab
| solu...
| TeMPOraL wrote:
| If the system is old (going by other replies, introduced
| in a deal 20 years ago), maybe they'll be able to brute-
| force the encryption...
| michaelt wrote:
| Encryption being 20 years old doesn't mean it's easy to
| break, given that AES is 23 years old :)
| TeMPOraL wrote:
| It doesn't, but in my experience, the ciphers and key
| sizes commonly used were often not future-proof :). This
| rests on the assumption that nobody updated the
| keys/encryption components in 20 years - which I think is
| quite possible with such internal enterprise deployment.
| Mauricebranagh wrote:
| I did something similar for a customer of ours who was a major
| UK cinema chain - they had had a falling out with some small
| company that had built an online booking system.
|
| Highlights:
|
| 1 Bookings where made at individual cinemas
|
| 2 The network was a packet radio system that blocked :-) only
| Transaction could be in flight (Leicester square had two
| channels)
|
| 3 Core of system was Perl running on three windows NT machines
| which manipulated a shonky screen based system (Pacer Cats)
|
| 4 Trying to sell tickets online for Phantom menace system
| crashed at midnight (when tickets went on sale) we made 1
| transaction at 6am
|
| % And no credentials so I had to research how to crack NT - I
| did consider setting up or fleet of a dozen or so of our suns
| as a cluster to crack NT, but not sure if the internal security
| would have liked that (aka secret squirrels and the bit that
| Bruce Shenier worked for
| Nextgrid wrote:
| I'd love to read a detailed write-up about this. Why packet
| radio - what was the real-time requirement and why couldn't
| it be fulfilled by an internet connection (I understand
| internet was primitive back then, but ISDN should've been
| able to give you enough bandwidth?).
| batch12 wrote:
| I agree. My first reaction was, "I think I could fix this".
| Worst case this may be be proceeded by a forensic recovery of
| the data. This doesn't seem impossible at all unless there is
| some other factor at play like ransomware/physical damage.
| tw04 wrote:
| I actually don't see any issue that isn't simply a matter of
| money.
|
| Everything they list that's an issue seems to revolve around the
| fact that the employees with domain knowledge for the system have
| left the company.
|
| Reach out to them and hire them on as contractors. If they left
| under bad terms because the business was a bunch of dicks, expect
| to pay 10x market rate. If this is truly "fix this or the
| business is out of business" - then it shouldn't be a tough
| decision to make.
| geoduck14 wrote:
| Let's not forget about the Mythical Man Month. Sure, specific
| people amd money can fix the problem, but we can't just assume
| that throwing people and money can fix this.
|
| I would anticipate a mix of: the people who know where the
| right bodies are buried, the right incentive (yes, that
| includes money), but also people to look over the entire
| program, a designer to make sure it is designed correctly, an
| organized person to make sure it is properly documented, and
| someone to make sure it is adhered to in the future.
| ghaff wrote:
| Even if you open a checkbook wide, there's no guarantee that
| someone remembers the specifics of some kludgy system they
| worked on years ago. And if things are really snafu-d, there's
| no guarantee that even someone who does have domain knowledge
| can recover things.
| WJW wrote:
| And, of course, there's the scenario where the people with
| domain knowledge are already dead. Money won't help in that
| case.
| Accujack wrote:
| It's slightly more complicated than that, depending on how much
| time has passed since the creators of the system left, but
| you're correct in general.
|
| Usually, since admitting that they were stupid to let all the
| knowledge of the system walk out the door requires more
| emotional and intellectual flexibility than managers who let
| this sort of thing happen possess, the biggest obstacle isn't
| even the money... it's getting the managers to admit they're
| idiots.
| hinkley wrote:
| Coworker used to have a boss who literally said where she
| could hear: "Lose a monkey, get a new monkey."
| at_a_remove wrote:
| Yes. I had an ancient system where the holders of domain
| knowledge kept either dying or retiring, one by one. I was
| frantic, but nobody listened to me. The system began to fail
| on the day I announced my departure. I would later find that
| one of the previous architects had reached out to my manager
| to offer assistance, but this was never made known to us. It
| wouldn't have even _cost_ anything.
|
| Anything but pride, I guess.
| criddell wrote:
| The person who started the thread added this comment:
|
| > The company that built this ERP solution went bankrupt 6
| years ago, and we don't have the source code. It uses
| PostgreSQL and MySQL, and also has a built-in key-value
| database for which none of us has any credentials. We don't
| have the source code for this software (it's built in C and
| delivered to us as compiled packages..).
|
| Sounds like finding their old employees wouldn't help. They
| need to find old employees of the company that wrote the
| system and hope they kept a copy of the source code.
| omgJustTest wrote:
| If business not make money, close business. Business make
| money, big boat of money.
|
| Engineers not make money.
| temp8964 wrote:
| What if they lost contact information of their former
| employees?
| drclau wrote:
| Not really related to the subject of the article, but your
| comment reminds me of employers in Romania complaining they
| can't find people to hire, while they offer small salaries and
| people just leave the country for better paying jobs abroad.
| The solution to their problem is really simple: make better
| offers.
| psychlops wrote:
| You've just described the US H1-B visa program.
| asdfman123 wrote:
| Well, the problem is this: it might not make economic sense
| for those specific companies to hire people at a higher rate.
|
| Companies aren't going to say "I guess we're inefficient and
| should just die." They're going to fight to survive and
| thrive, which is what you're seeing.
|
| Then again, some companies ARE just cheap. You never know.
| phkahler wrote:
| I was at a hotel with crappy service and asked about the
| problems with house keeping. The manager told me everyone
| in town (Traverse City MI) was trying to bring in
| foreigners to do housekeeping because they couldn't get
| local labor.
|
| They said the standard is 30 minutes per room per day. I
| figure at a very generous $50 per hour (100k / year) it
| should run $25 to have my room taken care of. Now that
| doesnt cover the shared costs - doing laundry, cleaning the
| pool, front desk, etc... but my room was something like
| $150 per night. How the fuck can they not find people to
| clean rooms?
|
| Answer: they're too cheap to pay a decent wage. But why? My
| suspicion is that all these places are actually leveraged
| far too much and paying rent/interest/dividends so much
| they can barely operate.
|
| I suspect low interest rates and other incentives to
| "stimulate" economies are actually causing these problems.
| cortesoft wrote:
| If they weren't leveraged that much, some other company
| would pay more for the property by being leveraged that
| much.
| zebnyc wrote:
| The service industry might be labor intensive but is
| hardly the only cost for the owner(s). What about
| mortgage payments, marketing, integrating with third
| parties (expedia), gas & electricity, general maintenance
| / accounting / security etc. I am sure there are hundreds
| of other costs that I have no visibility into
| thitcanh wrote:
| You make no sense. No one will pay $25 to clean a single
| room in a hotel. Just because your room is $150 it
| doesn't mean that "there's plenty to pay everyone
| 100k/year.
| ghaff wrote:
| I assume the cost of a room cleaning (including costs
| other than direct labor) is closer to $5. That's about
| what hotels will often give you in some sort of funny
| money to skip a cleaning.
| NewLogic wrote:
| The exact same thing is happening in Australia in regards to
| farmers and fruit picking labour. With international borders
| shut they are unable to exploit foreign workers and complain
| that young adults are refusing to relocate for $5 an hour
| hard labour jobs.
| kolbe wrote:
| Farming is low margin. Farmers don't get to set the price
| of their products, and they aren't handed billions in
| capital to burn through or 1000x multiples on their profit
| in the public markets. When most businesses say they can't
| hire someone and the reason is because they don't pay
| enough, the underlying cause of their low pay is usually
| because that's what the company can afford.
|
| Sure, farmers could pay $10/hr, but unless your Doritos
| were $10 a bag, they'd lose money.
| CydeWeys wrote:
| People have to eat food. Food in aggregate has one of the
| most inelastic demand curves imaginable. If the price
| rises because the production costs increase, people will
| still buy food. They might buy _different_ food, and you
| 'll see a decrease in acreage given over to labor-
| intensive crops, but that's just the free market at work.
| smnrchrds wrote:
| The thing is, if farmers in Australia increase wages and
| thus their product becomes more expensive, the product on
| supermarket shelves will not become more expensive.
| Rather, supermarkets will import food from another
| country. Food is, with few exceptions, a commodity. The
| producer does not set the price of the commodity, the
| market does.
|
| I am telling this as an engineer working in the Canada's
| oil and gas sector. Since 2014, salaries in our sector
| have decreased and firms are struggling. "Just raise the
| price of oil" is not an actionable advice. The price of
| oil is not ours to set. It's the market's
| Symbiote wrote:
| The EU sets fairly high tariffs on food imports, since
| food security means we must maintain local food
| production.
|
| Australia does not use this approach, and instead seems
| to have pushed other countries to eliminate their own
| tariffs for the TPP.
| smnrchrds wrote:
| What tariff does is that it moves the price point at
| which the retailers switch to imports a bit higher.
| Tariff does not eliminate the ceiling. There still would
| be a ceiling on price, determined by international
| commodity price + X% tariff.
| freeone3000 wrote:
| So the tariff needs to be high enough to be absolutely
| ruinous. See, for example, Canada's tariff on dairy.
| Since Canada runs a cartel on dairy producers, Canada
| does set the domestic price of milk at $4/gal. This is
| incompatible with the US subsidy model, where it prices
| out to around $1.50 a gallon -- so there's a 270% tariff
| on dairy imports. The great thing about numbers is
| there's always a bigger one.
| smnrchrds wrote:
| Yes, you can do that. It's not something the farmers
| strapped for workers can do, but the government certainly
| can. It would not be particularly popular in most
| political climates, so passing such laws in democracies
| are always difficult. But as the Canada example shows,
| not impossible.
| ksec wrote:
| It just shows most people working in Tech Industry, and
| specifically software ( or web ) development ( as oppose
| to Tech / software within Oil Industry or any other
| industry ) have absolutely zero idea on market or
| commodity trading.
| zentiggr wrote:
| So.... reduce production, and supply/demand curves should
| get you back financially on track?
|
| Of course, you might get pilloried for the artificial
| scarcity, but hey... nobody ever likes the capitalist :)
| smnrchrds wrote:
| We reduce production, US, Russia, and Saudi Arabia would
| be more than happy to fill the gap. Actually Saudi Arabia
| prefers prices that are lower than the strike point for
| non-traditional oil, so it can get rid of competition
| from likes of Canada. How should we convince Russia and
| Saudi Arabia to take a hit to their exports in order to
| help us out, when we cannot even convince our biggest
| ally, the US?
| wing-_-nuts wrote:
| I consider food security a national security issue.
| Produce is one of the few areas where I would consider a
| tarif justified. Every country should be able to feed
| it's citizens with the food grown there, or they risk
| hunger when cut off by war or natural disaster.
| smnrchrds wrote:
| And countries do that, but for a few strategic ones, not
| everything. Canada produces lots of wheat and dairy and
| maple syrup. It even has a strategic maple syrup reserve.
| But it does not produce a lot of citrus fruits or
| pistachios or dates. It may be possible for US to produce
| everything within the country--it is a gigantic country
| and has just about every climate there is. But it would
| be an ecological disaster if countries without those
| attributes do the same. Think of all the heating required
| to produce warm-climate fruits in Canada.
| wing-_-nuts wrote:
| Oh! One of my favorite ag articles talked about the
| challenges of growing oranges in russia!
|
| https://www.lowtechmagazine.com/2020/04/fruit-trenches-
| culti...
|
| You're absolutely right that perhaps not everything
| should be grown everywhere, but I strongly believe that a
| country should have a local source of all the calories
| and vitamins / minerals they need. In Canada, for
| example, maybe they encourage farmers to grow greens,
| brocolli or potatoes instead of citrus for vitamin c, but
| they absolutely need a source.
| csomar wrote:
| Apples and Oranges. If you include global supply, then
| you have to increase salaries "globally". We all (I
| assume) should know that demand/supply set the price.
|
| But imported products don't fly right into the country.
| There is customs, and denying/tariffing them is a policy
| that countries are very actively using. So at the end of
| the day, it's a political decision.
|
| Edit: On the other hand, if you are exporting stuff (oil
| for Canada), then you can't really do that for the
| external market. Might explain why the government will
| let some sectors run with illegals (to remain
| competitive); the alternative being their ultimate
| demise.
| smnrchrds wrote:
| If you ban import X from a country, the country will
| likely ban import Y from your country in retaliation.
| Sometimes the net effect would still be in your favour,
| but often times it would not be. Do you know why American
| brands dominate the US truck market? It started with a
| tariff on American chicken by some European countries and
| it quickly snowballed into much more [0][1].
|
| [0] https://en.wikipedia.org/wiki/Chicken_tax
|
| [1] https://www.npr.org/sections/money/2017/01/25/5116635
| 27/epis...
| WJW wrote:
| ...yes? The alternative to farmers is not harvesting
| their crops at all, since with the borders closed foreign
| labor is not available.
|
| If nobody wants to buy $10 doritos, that means the
| product can't exist in a market without cheap labor.
| [deleted]
| antihero wrote:
| Well perhaps that's the problem of the market. That it
| can't particularly function to enrich some people without
| gross levels of exploitation.
| kipchak wrote:
| To me it always seemed a bit similar to justifications
| used for the "peculiar institution" - the system can't
| exist without slavery, therefore slavery must exist. The
| benefits of cheaper labor was well appreciated; Carnegie
| called immigration "the golden stream which flows into
| the country every year."[1]
|
| [1]https://www.ebooksread.com/authors-eng/andrew-
| carnegie/trium...
| ihsw wrote:
| Being paid $5/hour farming in Australia vs $0.50/hour
| assembling shoes in Malaysia. Wow, such exploitation.
|
| "Exploitation" is a relative term, some people would
| rather be exploited in the countryside of a first-world
| nation rather than the countryside of their homeland.
| antihero wrote:
| Yes, they are both exploitation and they are both evil.
| ihsw wrote:
| In any case, people should have the freedom to choose.
| jacobolus wrote:
| Which is more exploitative, working for $5/hour in
| agriculture in a wealthy country, 50C//hour in an urban
| factory in a poor country, or $1/day in a rural peasant
| village somewhere or having no work at all?
|
| It's all relative.
| riskable wrote:
| That's a bit like asking, "Which is worse? A shot to the
| head or an overdose of morphine?" Either way the person's
| going to die.
| thrower123 wrote:
| Doritos aren't going to get more expensive, because like
| all the real staple crops, corn is completely mechanized,
| and you have one guy driving an air-conditioned tractor
| or combine, following GPS-plotted courses.
| relaxing wrote:
| True! Except it's not just doritos, it's everything you
| eat.
|
| And it's not just cheap labor, it's labor exploited under
| pain of deportation.
| csomar wrote:
| > Farmers don't get to set the price of their products
|
| Actually farmers could set _any_ price if they colluded
| to manipulate the market. People will buy food at _any_
| price and _everyday_.
| clipradiowallet wrote:
| This is offset by imported food. If 100% of American
| farmers colluded to fix prices, they'd have their lunch
| eaten(hah) by food importers. Obviously not everything is
| importable(eg rice), but there is more to the market than
| the American farmers.
| raffraffraff wrote:
| The reason I buy expensive eggs, milk and cheese is that
| I know that the supermarkets fuck over the farmers. I
| never buy the supermarket brand, unless it's the last one
| on the shelf. The price difference is sometimes 1.5x but
| unless you're really "broke" broke, you don't notice the
| difference. It's a few cents a week.
| kbelder wrote:
| I always buy the supermarket brand, even though I can
| easily afford more. It's not because I want to save a few
| pennies on each purchase; it's helping to drive prices
| down on all overpriced goods, helping out those really in
| need.
|
| Same reason I avoid organic whenever possible. We don't
| need boutique food.
| cammikebrown wrote:
| I agree much of the "benefits" of organic food are not
| really true or meaningful, but I do try and buy local
| when I can.
| solarkraft wrote:
| Good idea, but how do you know the extra money is
| properly passed on?
| knightofmars wrote:
| While, like most things in life, it's not a guarantee,
| doing basic research about the farms and the cooperatives
| that produce the goods will provide some insight. Almost
| anything is better than a CAFO (Concentrated animal
| feeding operation).
| skykooler wrote:
| Sure, but they'd lose less money than if they didn't hire
| anyone and couldn't sell their crops at all.
| SilasX wrote:
| Not sure that's the best example to use, since food costs
| are a trivial fraction of the retail price of a bag of
| Doritos, and harvesting labor is a trivial fraction of
| the price of food.
|
| Edit: that was just based on my general memory of non-
| perishable food economics. But I googled it and found
| this (sorry, quora) from someone who would know, a 2 oz
| bag of potato chips costs 15 cents to produce and bring
| to (retail) market. Of that, a fraction is potato prices.
|
| https://www.quora.com/How-much-does-a-bag-of-57g-of-Lays-
| act...
|
| Edit: fix Verizon math.
| scruple wrote:
| > complain that young adults are refusing to relocate for
| $5 an hour hard labour jobs.
|
| USD $5/hour is what I earned as a teenager working the
| fields by hand (picking corn, beans, and fruit for sale
| same-day at the farmers small stands throughout the area)
| in the early 1990s... That's insane. Is that USD $5 or AD?
| ballenf wrote:
| I think people will one day demand higher pay for desk
| jobs over outdoor work. Looking back on my early life,
| the hard outdoor work was incredibly valuable to me. Desk
| jobs sometimes were soul-sucking, depression inducing.
| Not always and my dev work now is desk work that doesn't
| suck.
|
| Now, outdoor work with a high injury rate is a different
| story.
| AdmiralGinge wrote:
| Sounds like exactly the same problem British farmers are
| having post-Brexit with things like fruit-picking, they're
| scratching their heads wondering why British workers aren't
| keen on having half their wages taken for "rent" to live
| six people to a crappy caravan on-site and do backbreaking
| labour day in day out. Their only choices are to improve
| their wages, conditions, both, or go bankrupt.
| wing-_-nuts wrote:
| Here in america farmers hire illegals to harvest crops,
| claiming that no american would do the work. This has
| always struck me as a _really_ terrible excuse. I 'm sorry,
| you can't simply break the law because you'd have to pay an
| american $20 / hr to do the work. Sure, it means that your
| tomatoes are more expensive, but also, now there's a hell
| of an incentive to apply automation to the problem yes? If
| you don't like the law, maybe pressure your representatives
| to reform immigration instead of just hoping everyone turns
| a blind eye to your exploitation.
| FireBeyond wrote:
| A couple of years ago, a serial offender food producer
| was raided, multiple times. 600+ undocumented workers
| were found, and all spoke of management knowing their
| status.
|
| The workers were detained, deported, and so on.
|
| After several raids, not so much as a misdemeanor was
| levied against a single person, nor a fine to the company
| in general.
| Animats wrote:
| See Postville Raid [1] That was in 2008. The company
| owners were some prominent Jews, and the political
| fallout was so severe that immigration raids pretty much
| stopped after that.
|
| [1] https://en.wikipedia.org/wiki/Postville_raid
| aaron-santos wrote:
| Kind of makes me think, what's the point when these laws
| aren't actually enforced?
| abxytg wrote:
| The point is to punish and cause pain to the immigrants.
| Regardless of the original justification of a system, the
| point of a system is what it does or will do. That is
| what this system does and will do unless changed.
| HarryHirsch wrote:
| "For my friends, everything. For my enemies, the law."
| Getulio Vargas had it all worked out.
| Animats wrote:
| About half of US hired farmworkers are illegal
| immigrants, says the U.S. Department of Agriculture.[1]
| It was only 12% in 1991, but passed 50% by 2000.
|
| [1] https://www.ers.usda.gov/topics/farm-economy/farm-
| labor
| greeneggs wrote:
| Of course it isn't that simple. If your farm's tomatoes
| are more expensive than your neighbors' tomatoes, then
| nobody will buy them. You'll quickly go bankrupt.
|
| It is also difficult because of the two tiers of labor. A
| farmer will try increasing wages by $1/hour, and still
| struggle to find enough workers. Wages would probably
| have to be doubled before new workers--Americans--would
| start to enter the industry.
| jandrese wrote:
| This is the reason tariffs exist. You levy a fee that
| makes the imported food cost as much as food picked using
| domestic labor.
| mixedCase wrote:
| Or, you know, you let unprofitable endeavors be
| substituted by profitable ones, and you just import food
| from different places where it is profitable. Just make
| sure not to wage war with them all. That's always an
| alternative to subsidies and tariffs, but it's not as
| politically useful.
| jandrese wrote:
| This is how the US gave up its ability to manufacture so
| many goods.
| lisper wrote:
| > reform immigration
|
| That's easier said than done. The problem is that in
| order to reform immigration you have to do one of two
| things:
|
| 1. Start treating people fairly, with the result being
| higher food prices, which will make some of your
| constituents very unhhappy
|
| 2. Codify the current class structure into law so that
| you can continue to exploit cheap labor legally. The
| optics on that are really bad.
|
| Exploiting people off the books is the path of least
| resistance for everyone, including the workers
| (manifestly so, or they wouldn't risk life and limb to be
| here).
| colinmhayes wrote:
| If farmers paid enough to get Americans to do seasonal
| farm labor their produce would cost twice as much as
| central/south American produce and they would immediately
| go out of business as everyone buys cheap imported fruit.
| nitrogen wrote:
| A labor-protectionist strategy has to be paired with a
| goods-protectionist strategy (tariffs or bans on
| imports).
| freen wrote:
| Sounds like they don't have a business, and should shut
| down.
| rjmunro wrote:
| You can fix this with import duties. If the country you
| are importing from doesn't have good labor laws, you
| charge them 100% extra.
| colinmhayes wrote:
| People get really pissed off when food gets more
| expensive. Politicians realize that and will therefore
| never pass legislation that would cause such drastic
| price increases on food.
| wing-_-nuts wrote:
| If they enforced the laws already on the books, they'd
| _have_ to pass new laws to fix things. I see that as a
| net positive. If you selectively enforce against illegal
| immigrants, well, that 's very much a 'rules for thee,
| not for me' situation isn't it?
| pg_bot wrote:
| Now everyone is poorer.
| reaperducer wrote:
| _Here in america farmers hire illegals to harvest crops,
| claiming that no american would do the work_
|
| This is a political talking point that has been repeated
| so often that people believe it is true.
|
| During the recession of 2008, there were stores on local
| TV news in many places showing unemployed Americans
| waiting in long lines for the opportunity do do those
| jobs that the politicians kept saying no American wanted
| to do for those wages. Meatpacking plants, especially,
| had more than enough Americans to choose from who were
| willing to work for low wages.
|
| Because these things happened in "flyover" states, they
| were only very rarely shown on television on the coasts,
| and so the meme stuck.
|
| A similar oft-repeated lie is that American farmers are
| all a bunch of Republican hillbillies who hate brown
| immigrants.
|
| Around 2010, I attended a conference in Seattle for
| farmers in the Pacific Northwest. The main topic of
| discussion was how to increase legal immigration from
| Mexico so that they could have more farmworkers to pick
| the cherries and apples and mint and whatnot. These
| people were talking about goals of making people from
| Mexico and Latin America U.S. citizens in staggering
| numbers.
|
| The more you actually go places and do things and meet
| people and talk to them, the more you realize that the
| words coming out of the political organizations on the
| coasts are mostly for their own benefit, and do not
| reflect reality.
| fredophile wrote:
| I don't think your points really contradict your first
| statement. First, people tend to drop their standards on
| the type of work they're willing to do and pay they'll
| accept when they're desperate. In 2008 a lot of people
| needed a job. Any job was better than no job. I'm not
| surprised that jobs that would normally be unpopular had
| lines of people applying at that time.
|
| Your second point about farmers trying to increase legal
| immigration in 2010 clearly indicates most Americans
| won't regularly do this work for the wages being offered.
| Why else would farmers be looking to hire immigrant
| workers just 2 years after locals were lining up to get
| these jobs?
|
| While it would be nice if there was a way to get enough
| immigrants working legally to fill those jobs that
| doesn't seem to have happened. Short of another large
| economic downturn I don't see Americans working those
| jobs either. That leaves people who are working in the US
| illegally to fill the gap.
| jbroson wrote:
| My somewhat educated guess is that these farmers in the
| PNW don't share the same politics as farmers in the rural
| south or midwest.
|
| That and Republicans have grown way more hostile to
| immigration since 2010, so it's an interesting anecdote
| but not sure how well it translates across the US, where
| polling shows farmers are overwhelmingly Republican who
| tend also tend to hate immigration legal or not.
| reaperducer wrote:
| _My somewhat educated guess is that these farmers in the
| PNW don 't share the same politics as farmers in the
| rural south or midwest_
|
| When you say it's an "educated" guess, educated how? It
| only seems to reflect the usual bigotry that comes out of
| the coasts.
|
| _polling shows farmers are overwhelmingly Republican who
| tend also tend to hate immigration legal or not_
|
| If you're going to make a statement that sweeping, you
| really should back it up with credible sources.
| enriquec wrote:
| "where polling shows farmers are overwhelmingly
| Republican who tend also tend to hate immigration legal
| or not."
|
| source?
| Verdex wrote:
| At college I had a professor who I worked for who grew up in
| Romania. Apparently the song that kids had back in the day
| was: "We pretend to work and they pretend to pay us."
| intrasight wrote:
| That is, of course, the solution everywhere that companies
| complain about not being able to hire skilled employees. Fund
| it and they will come.
| smnrchrds wrote:
| This kinda sounds like saying "why are poor countries poor?
| Can't they just pay everyone more and then they won't be
| poor?" I get what you are saying but I don't think this
| will work if the problem is country-wide. You can't just
| magically increase everyone's wages in a country.
| intrasight wrote:
| 1. The problem isn't country wide. 2. Don't have to
| increase everyone's - just those with a "shortage".
| Companies want a market economy - except when it comes to
| salaries.
| smnrchrds wrote:
| > _1. The problem isn 't country wide_
|
| Sorry, I though I was on the Romania subthread.
| k3oni wrote:
| Ha, I know this all too well(2000-2006) and it's strange to
| see/hear it's still happening now.
| vbezhenar wrote:
| https://youtu.be/-Jh0EN1De4Q
|
| sorry for pun. But that's not always a solution. Sometimes
| employer is greedy. Sometimes his business just does not
| allow to pay enough salary.
| filleduchaos wrote:
| > The solution to their problem is really simple: make better
| offers.
|
| That's certainly a _simple_ solution.
|
| The problem with simple solutions is that they are near
| guaranteed to not actually be feasible or realistic.
| alecbz wrote:
| If it's literally this or the company folds, that seems
| realistic.
| yrgulation wrote:
| Unrelated to the initial post.
|
| > The problem with simple solutions is that they are near
| guaranteed to not actually be feasible or realistic.
|
| This. There is no amount of money anyone sane can be given
| to stay in Romania. Most recently i've read they
| (hospitals) were tying covid patients to their beds.
|
| Couldn't help but comment on this as some of things i
| discovered are shocking for an eu member country (or any
| country for that matter), such as EU leading human
| trafficking, police corruption, collapsing health care, and
| the list goes on.
|
| But sure, the simple solution is to give people money, not
| the constant abuse taking place at work and outside that
| makes people want to flee that place.
| fishtacos wrote:
| That's every poor country in existence. Not everyone has
| the same mindset as you, however. Provided with basic
| incentives, many, if not most, would not want to leave
| family, friends, common culture, etc, just for a wealthy
| life - they just want basic stability.
|
| Companies paying more (offering more at least) would
| begin to slow this domino effect, which raises standards
| of living and provides more funding for the government to
| provide services and ultimately pressure for increases in
| quality of life by the voting public. Everyone wins.
|
| Ultimately not everyone can emigrate - it's an
| impossibility. The brain drain will continue to be real,
| so it's incumbent upon these companies hiring to raise
| their wages and retain talent.
| yrgulation wrote:
| Agreed - my statement is in regards to "simple
| solutions".
| fishtacos wrote:
| That was my point. The simplest solutions are sometimes
| the most effective, as well. Incentivizing brain power to
| remain the country via pay increases seems like the most
| basic approach one can take, considering the GP was
| talking about Romanian studios complaining they can't
| find/hold talent and they are competing with the rest of
| the EU.
| notahacker wrote:
| The simplest solutions are the most effective, until you
| realise it isn't at all simple to match the 200-500% pay
| increase they're emigrating for.
| [deleted]
| joezydeco wrote:
| _then it shouldn 't be a tough decision to make_
|
| You'll continually be amazed at the human tendency to double
| down instead of admitting a mistake.
| m463 wrote:
| They could also hire expensive consultants (who will just
| call up the ex-employee, and _listen_. maybe to laughter.)
| throwthescene wrote:
| Throwaway purely so that I can comment. I've had this happen
| before.
|
| Years ago I left a rather awful company - I left on great terms
| with my boss, though I hated the company. I was the last person
| who knew how to release X product. While I documented
| everything, after I left, the wiki lost its mind, backups
| apparently weren't a thing, and the director of IT moved to
| Bermuda.
|
| Fast forward 6 months.. a contractual obligation existed to
| release the software. They still hadn't successfully built (yup
| autoconf, make and friends), let alone know where to build it.
| I got the call you always dream about, 30 days prior to the
| release being due.
|
| My boss asked me to do him a favour - I reminded him that I
| quit due to a lack of promotion and raises. He asked what I
| want, and I asked for 125K, which clearly he scoffed at. Then I
| reminded him that it was a 10M deal they'd signed, because we
| had an all hands. The CEO got involved and screamed at me for
| 'torpedoing the company'. I walked out and calmly informed them
| both along the way the price was now 250K, non negotiable.
|
| I ignored every single mail (and legal letter) sent me way.
| With 10 days left, they agreed. I had the company deposit a
| cheque in escrow with lawyers of my choosing. We signed
| contracts, and I did my thing. Four days of work. It was well
| worth it.
|
| Dougie, if you're reading this I hope your cringing.
| deevolution wrote:
| Absolute chad badassery
| rolobio wrote:
| Why a check with your lawyers? Why not just deposit into your
| personal account?
| ghgdynb1 wrote:
| The idea is that you want a trusted third party because
| either of the two adversarial parties could screw the other
| over.
|
| The company could say: we're going to pay you 125 and you
| could sue us and spend 60k and two years or just settle for
| that, after he's already done the work.
|
| The guy could demand the payment directly to his account
| and then not deliver, and the firm would be in the same
| situation.
|
| By depositing the funds in escrow, they're making sure that
| the money is there and in the hands of people whose only
| incentive is to give it to the correct party upon
| fulfillment of the contract.
| dqpb wrote:
| It went into an escrow account, presumably until he
| completed the work
| bityard wrote:
| Look up what "escrow" is. When 1/4 million dollars is on
| the line, you don't take the company's word that they'll
| pay you afterward, and they don't take your word for it
| that you'll actually deliver the service.
| Aloha wrote:
| 1/8th of a million.
| brobinson wrote:
| Last time I checked, $250,000 was 1/4 of one million
| dollars.
| Aloha wrote:
| Oops, I missed the last part of the paragraph where he
| raised the price.
| [deleted]
| [deleted]
| andyv wrote:
| With the check held in escrow, it makes cheating harder.
|
| The contractor knows that the money has been paid by the
| company and won't be withheld after the job is done.
|
| The company knows that the contractor doesn't have the
| money until the job is done.
| acejam wrote:
| This is my dream. Is that bad?
| m-p-3 wrote:
| Damn, I'd pay a good part of my mortgage in 4 days with that.
| You played your cards well.
| toyg wrote:
| "A good part"? Lol, I'd pay it all and have enough spare to
| buy a midrange car...
| OneLeggedCat wrote:
| This is the most beautiful thing I've read today. Thank you.
| tofuahdude wrote:
| I love this with all my heart. You did exactly the right
| thing at every step.
| ycombinete wrote:
| Funnily enough it made me sick to my stomach. I manage a
| small company and the idea of being extorted in such a
| blase and psychopathic fashion terrifies me to even think
| about.
| DubiousPusher wrote:
| This is not extortion. The company made choices that left
| significant leverage in the hands of another business.
| That business correctly perceived the value of the
| services it could render and priced them accordingly.
|
| This is exactly the free market at work regarding labor.
| Have a problem with the free market if you like, many of
| us do but do not accuse this poster of criminal action
| equivalent to a protection racket.
| johnchristopher wrote:
| What's the rationale for doubling the first price ? Why
| is it a more fair price ?
| DubiousPusher wrote:
| The market answer to that is, the seller underpriced
| their services to begin with, regardless of the personal
| motivation for increasing the price.
|
| To clarify, the market would likely not allow this
| specific work to be sold at this price normally. But the
| buyer wasn't just buying specific work they were buying
| specific work to be completed by a specific date. The
| work in isolation does not have this value but the work
| completed by a certain day does. Companies use this
| rationale all the time charging more for expedited
| services.
|
| It's especially common in construction and manufacturing
| where costs double or even triple when expedited delivery
| is required.
| woofcat wrote:
| How is it extortion? If you need 1 resource in the whole
| world, the supply is 1. If your demand is great then you
| get to pay the price.
|
| This person left the company, and documented everything.
| What should they do, come in and save the day for $15/hr?
| 1123581321 wrote:
| It's a fair fee for such valuable work done for an
| apparently abusive, and possibly litigious, company.
| [deleted]
| exikyut wrote:
| "Lack of planning on your part does not constitute an
| emergency on my par--wait did you say $250,000?"
|
| If your company were in the hands of incompetents, but
| still kept going _in spite of the incompetents_ because
| it had gotten its foot wedged in the door or whatever,
| and then fell apart in ways you could only scorn, how
| would you view that sort of situation?
| draw_down wrote:
| God forbid engineers be paid for their work. Typical HN
| leesalminen wrote:
| I owned a small company. After selling it, and giving 2+
| more years of my life for the new owners to figure out
| how to replace me, they didn't. On my last day I told
| them that if they ever needed anything my rate is
| $5,000/day, non negotiable. It's kept them at bay for
| now. I have no ethical qualms with my decision. It's
| business.
| rangerelf wrote:
| > ...I manage a small company...
|
| And you identify with this band of miscreants? That's
| embarrassing.
| zentiggr wrote:
| Sounds like your best next step is to become an exemple
| of software/infrastructure best practices - get
| everything in the best backup/recovery solution you can,
| document every detail any time an issue comes up, and
| above all treat your people like the vaaluable assets
| they can be.
|
| You may be doing any of that already, since I don't know
| anything about you except your one comment.
|
| But the answer to your fear of exploitation is avoid
| making people hate you/your environment enough to want to
| protect themselves from it.
| sufehmi wrote:
| "extorted"
|
| This is pure slander
|
| The company can avoid this if they only care to maintain
| the documentation / wiki left behind by said person
|
| When they failed to do even just basic proper operations,
| it's their own fault.
|
| I'm a business owner and a consultant, and I too charge
| more for these kind of asshole clients, because of the
| extra stress.
|
| But nowadays I just try to avoid them, it's just too
| damaging for your own sanity. Their kind of stupidity,
| self-entitlement, and ignorance is on a completely
| different level.
| dctoedt wrote:
| > _This is pure slander_
|
| <pedantry>It's libel (written) not slander (oral);
| "defamation" is the generic noun. </pedantry>
| doodpants wrote:
| Does libel magically turn into slander if a blind person
| uses a screen reader? Asking for a friend. ;-)
| gota wrote:
| Your comment made me curious and I checked - it is not
| the physical channel that matters, it is whether the
| media is _transient_ or not.
|
| It seems that paying to skywrite (you know, small plane
| making artificial clouds) could be slander but not libel
| dctoedt wrote:
| > _whether the media is _transient_ or not_
|
| Citation? By that light, a DVD containing defamatory
| speech would be libelous, but that's generally not how
| courts see it.
|
| From the (highly-influential) U.S. Court of Appeals for
| the Second Circuit, summarizing New York law (which is
| quite typical): "Defamation, consisting of the twin torts
| of libel and slander, is the invasion of the interest in
| a reputation and good name. Generally, spoken defamatory
| words are slander; written defamatory words are libel.
| Libel is a method of defamation expressed in writing or
| print." [0]
|
| https://scholar.google.com/scholar_case?case=601727562995
| 661...
| emayljames wrote:
| What about the other way, subtitles.
| yowlingcat wrote:
| The idea of /you/ mismanaging your company in such a
| "blase and psychopathic fashion" is more what should
| terrify you. Remember, /you/ manage a small company, not
| the person you hire. If you end up in such a mismanaged
| position, the manager who let it get there is you, not
| the employee.
| burnte wrote:
| Then treat your employees better and adhere to best
| practices and you won't be subject to such "extortion".
| Once we have elft yoru company, especailly if it was
| because we were treated poorly, we are under NO
| compulsion to help you for free. If the idea of paying
| 1.25% of the value of a deal to make sure the deal
| happens seems excessive, you're greedy, and that's why
| his price went to 2.5%.
| BlargMcLarg wrote:
| Business is business. The right to fire people, deny them
| raises and form binding contracts in regards to their
| work comes at a price. GP did not act malicious, rather,
| they acted in the same self-interest most bosses, CEOs
| and companies act while trying to stand on the moral high
| ground.
|
| Coding standards, documentation requirements, meetings
| and code reviews exist for a reason, too. It's not like
| companies are defenseless.
| snerbles wrote:
| The employee documented everything, but said
| documentation was not backed up by the company.
| Furthermore, the company did not train or hire an
| adequate replacement for said employee for _six months_.
|
| This is a gross failure of leadership at the company, and
| the former employee has every right to negotiate
| compensation for the inconvenience of having to work for
| them again.
| whatever1 wrote:
| Thinking about business continuity is literally your job.
| Not your engineers' job.
| OneLeggedCat wrote:
| Hopefully you wouldn't mismanage your own company and
| treat your employees like shit. Probably you are a better
| person than that, and can avoid this ever happening to
| you.
| vagrantJin wrote:
| I see and understand your sentiment but, in my rather
| insignificant few years in the workforce - carrots,
| dipped in vanilla chocolate and honey, generally tend to
| work better than the stick.
|
| The company could have apologised, and do a lot more ego
| stroking before asking a favour which they can pay for
| and maybe throw in an all expenses paid weekend with the
| SO as a thank you. Not only will you get goodwill but
| future problems are going to be significantly cheaper to
| solve. Even if you have a bad break up - as a business
| owner you should understand the value of mending bridges.
| It's cheaper than losing sleep at night and losing your
| steak dinner to a bad stomach.
| bitcharmer wrote:
| Extortion? Wow, you have no idea what you're talking
| about. Good luck with that attitude.
| acheron wrote:
| You're terrified that you might treat your employees so
| badly that you would need to pay a lot to resolve a
| situation? Seems like that's entirely within your
| control.
| thrower123 wrote:
| In this situation, you could always just tell the former
| employer to take a flying fuck at a rolling donut, and
| not respond to them.
|
| That's probably what they deserve, if they come asking
| for a favor and then start getting abusive.
| klibertp wrote:
| I think this is just flamebait. Uses emotionally charged
| words without any other substance. Both the comment and
| responses to it (including this one) add nothing to the
| discussion.
| heterodoxxed wrote:
| Have you tried learning to code?
| Judgmentality wrote:
| > Funnily enough it made me sick to my stomach.
|
| That an overworked employee who had already left the
| company would stick up for himself? Or that a company
| would go to such efforts to try to manipulate a former
| employee?
|
| I realize my tone doesn't hide my feelings, but even if
| we disagree I'd still be curious to hear more.
| ycombinete wrote:
| My apologies, I edited my comment as you were typing I
| think. I added some more.
| marktangotango wrote:
| IMO your edit didn't help and you're not getting a lot of
| sympathy. The point wasn't that the employee is
| psychopathic as you say, but that management was
| incompetent by not ensuring a transfer of knowledge and
| validating before the employee left. You as the buinsess
| owner should plan for people to not be available and no
| one person can derail a $10M contract. This is also
| called the "bus factor".
|
| https://en.wikipedia.org/wiki/Bus_factor
| jschwartzi wrote:
| Also yelling at someone who is the only person who can
| solve your problem is not a good negotiating strategy,
| especially when it reminds them of why they left.
|
| Walking away from a toxic situation is not
| "psychopathic." It's common sense and if we all did it
| more then the psychopaths who expect us to stick around
| through it all would finally learn how to treat other
| people with respect. It's a job, not a marriage.
| HeyLaughingBoy wrote:
| > yelling at someone who is the only person who can solve
| your problem is not a good negotiating strategy
|
| Sadly, I've seen this. Not first hand, but it happened to
| a friend and I have no reason to doubt his version of the
| story.
| AnIdiotOnTheNet wrote:
| Consider that you are only hearing one side of the story
| and taking it entirely at face value.
| hharlequin wrote:
| I think enough of us have been on the same side of this
| story that we feel fairly comfortable taking it at face
| value, even if we're only applying it to said personal
| experience in a wishful fashion.
| AnIdiotOnTheNet wrote:
| Yeah, HN at it's finest I suppose. He's like you, he says
| what you want to hear, therefore you believe him. No need
| to consider the possibility that maybe there might
| actually be another point-of-view.
| cogman10 wrote:
| I mean... what could the other side of the story actually
| be?
|
| "We tried to hire back an employee to build something
| that we didn't train or try and figure out ourselves and
| a deadline forced us to pay a large lump sum to that
| employee".
|
| No matter how you slice it, that's really shitty planning
| on the businesses side. They should have spent the time
| and money learning how to maintain their shit. By not
| doing it, they paid a huge price.
| gambiting wrote:
| I imagine if it came down to it they could try to pin it
| on him and say he didn't provide documentation he said he
| did, and that's why this entire 10M project now can't be
| delivered.
|
| Luckily for him they decided it's easier to just pay the
| fee.
| cogman10 wrote:
| Even if he was a shitty employee that doesn't obligate
| him to work for a price favorable to the company. Hell,
| they could have fired him for not documenting shit.
|
| Because here's the thing, even with how weak US labor
| laws are, one thing they don't allow is for a company to
| come back and sue your for incompetence. So long as they
| didn't go out of their way to sabotage things, they are
| free and clear.
| notyourday wrote:
| That won't work in most reasonable countries. A company
| that is no longer employs someone cannot make that person
| work. In the situations similar to the ones that are
| being described, the time is not on the side of the
| company - they have a dead line, a former employees does
| not.
| sideshowb wrote:
| Op says he did provide documentation but they lost it
| (and hadn't been backing up) after he left
| gambiting wrote:
| Of course, but that wouldn't stop them from saying
| otherwise, suing and binding him in prelonged and
| expensive legal battle, would it?
| bananabreakfast wrote:
| sure it does. the company would have to prove malicious
| intent beyond a reasonable doubt and not leaving adequate
| documentation absolutely does not meet that requirement.
| ptero wrote:
| This seems absolutely unrealistic to me. Can you find a
| precedent? A single precedent of the company suing an
| engineer (individual contributor, not a manager) for
| quality of work after he left? I think if they tried they
| would have been laughed out of court; and out of all
| their future contracts, too.
| derefr wrote:
| We're talking about who's morally in the wrong here, not
| whether it's a good idea to do this.
| notyourday wrote:
| Lawsuits come with discovery. Discovery is a terrible
| thing for a plaintiff that lies in the lawsuit.
| nimih wrote:
| Even if they did "pin it on him," why should he care,
| exactly? Seems to me like he had no obligations here,
| contractual, ethical, or otherwise.
| gambiting wrote:
| Well because if he was taken to court that costs time and
| money, even if ultimately the lawsuit fails, no?
| Bud wrote:
| You seem very confused about what "extortion" means, and
| also about what "psychopathic" means. So I'd suggest
| starting there.
| arethuza wrote:
| That's not extortion, simply hard-nosed commercial
| negotiation.
| tcoff91 wrote:
| The fact that you'd refer to this as extortion makes me
| sick. The management/capital class loves the free market
| except for when it applies to labor, then suddenly it's
| extortion when supply and demand don't work in your
| favor.
| ryandrake wrote:
| Yea, this is one of those "I want to believe" stories! It's
| practically engineer-porn. Everyone who has quit a shitty
| employer whose technology was a house of cards resting on
| the point of a needle is cheering OP on. We all have that
| fantasy that the house will finally fall over, the company
| will have its comeuppance, and we're the one person in the
| world with the knowledge to fix it.
| kstrauser wrote:
| Hell. Yes. That's absolutely beautiful, and ethically
| completely correct.
|
| I have bosses at previous companies that I'd help for free
| today because they were good to me and we parted on friendly
| terms. One boss helped me move my house once, and I'm not
| about to charge him to fix some random issue.
|
| I have other bosses whom I'd require a cleared wire transfer
| of money before I'd lift a pinky to help them, because they
| were not nice people and I don't want to associate with them
| or assist them with anything. If someone in this camp
| approached me, I'd have a story just like yours afterward.
| SteveNuts wrote:
| What did their legal letters try to pin on you? Did they try
| to say you were somehow legally required to help them?
| throwthescene wrote:
| The letters stated that I had a contractual obligation to
| assist. But I noticed at the top they were stamped 'without
| prejudice'. I asked my friend what that meant, and he said
| that technically it's not a legal letter - so I ignored
| them.
|
| For a while, I made it game to light the various
| communications on fire in interesting ways, but it got
| boring.
| egorfine wrote:
| > I had a contractual obligation to assist
|
| Did they have a legal reason to think so?
| blunte wrote:
| Yes. And after the boat is righted, fire every exec from IT
| Director upward through CEO.
|
| That last part may not be necessary though, as customers will
| jump ship and the company will probably sink anyway.
| Nextgrid wrote:
| > customers will jump ship and the company will probably sink
| anyway.
|
| Depends, they might have a captive market or a customer base
| that's non technical and doesn't understand how bad this is.
|
| If their systems are like this, chances are their offering
| isn't anything groundbreaking and the competition already
| provides a better service for possibly even cheaper, so the
| customers who are willing to jump ship would've done so long
| ago and the fact they're still in business suggests they have
| a complacent customer base that's likely to stick to them
| even despite this incident.
|
| The same reason this technical debt was left unchecked for
| ages applies to customers. The task of migrating to another
| provider will most likely rot forever in a Jira board
| somewhere and they will keep paying their bill in the
| meantime.
| JKCalhoun wrote:
| Is there another industry where a few employees with "domain
| knowledge" can sink a company?
|
| Asking because, while, yes, throwing money at the problem might
| be a solution, I think the obvious _bigger_ issue is how does
| one allow themselves to get into this situation?
|
| Or rather, how does one _avoid_ getting into this situation.
| Because it seems to me that the software industry of late has
| become so mired in esoteric layers of abstraction that this
| feels less like an outlier scenario for a mismanaged company
| but more like an inevitable one for many otherwise well managed
| companies in the industry.
| hunter-2 wrote:
| >Is there another industry where a few employees with "domain
| knowledge" can sink a company?
|
| I guess any company where leadership got there without
| progress on the technical side. Law firm partners get there
| after being lawyers. Consultant partners get there after
| consulting.
|
| But if you are the CEO of an IT company without ever having
| to code, then you may never know the things you must take
| care to prevent a washout.
| foobiekr wrote:
| There are a LOT of industries with this problem. Specialist
| stuff like military shipbuilding is an example, but also
| things like precision ball bearings. Knowledge gets lost all
| the time and old timers have to be brought back in after the
| fact. Or the technology is just lost and a replacement has to
| be reinvented (famously, the production of tamper material
| for the US thermonuclear nuclear weapons).
| shagie wrote:
| The precision ball bearings thing reminded me of a news
| item from a few years ago: https://www.washingtonpost.com/n
| ews/worldviews/wp/2017/01/18...
|
| The relevant part of the article:
|
| > To anyone outside of the ballpoint pen manufacturing
| world, it might seem hard to understand what, exactly, is
| so surprising about this development. China already
| produces 38 billion ballpoint pens a year, according to
| China Daily, which is about 80 percent of all ballpoint
| pens in the world. That's a lot of pens, but there was a
| catch: China had long been unable to produce a high-quality
| version of the most important part of the pen, its tip.
|
| > The tip of a ballpoint pen is what makes it a ballpoint
| pen. At the tip, a freely rotating ball is held in a small
| socket which connects it to an ink reservoir that allows
| the pen to write or draw lines. Manufacturing a ballpoint
| pen tip that can write comfortably for a long period of
| time requires high-precision machinery and precisely thin
| steel, but for years China was unable to match those
| crafted by foreign companies.
| gregmac wrote:
| At the highest level, this is a leadership issue. There's a
| lack of someone at a sufficiently high level (eg: C-suite)
| that properly understands technology, and that's probably
| caused by the rest of the leadership team not understanding
| their limitations.
|
| At a lower level, it's allowing individuals to become
| knowledge silos (or having a bus factor [1] of 1); lack of
| documentation; not having+testing disaster recovery plans;
| neglecting systems, code bases, and platforms; letting
| technical debt accrue endlessly (or worse: not even realize
| that's what you're doing).
|
| How to avoid? Make sure there's a strong, competent
| technology leader with a vested interest in the company
| succeeding, and empower them to make the changes they need
| to.
|
| What should that person be doing? Making sure all the stuff I
| just said doesn't happen.
|
| [1] https://en.m.wikipedia.org/wiki/Bus_factor
| ufmace wrote:
| > Or rather, how does one avoid getting into this situation.
|
| IMO, the problem is technical debt. Most likely, in the
| majority of these cases, the engineers working on the system
| knew perfectly well that it was a mess, was very difficult to
| learn how to maintain, and many parts were dangerous to
| modify. Most likely they told management multiple times that
| this was a problem and what the consequences of it would be.
| They would have refused to budget resources for improving it,
| so everyone either saw the writing on the wall and quit, or
| gutted it out and learned how to manage the system
| eventually. Maybe the system limps along for a while longer,
| but it's only getting worse and the knowledge needed to
| maintain it more siloed and specialized. Eventually you lose
| a key person, something breaks that nobody left can figure
| out, or maybe the whole thing just implodes like the main
| story here.
|
| To avoid it, you must understand the pattern, hire good
| technical people, and give them the resources needed to
| improve the system in an orderly fashion.
| ISL wrote:
| > Or rather, how does one avoid getting into this situation.
|
| Documentation?
| giantrobot wrote:
| The issue with "documentation" is there are _never_ enough
| resources devoted to it at most companies. When given a
| schedule it only covers the main project goal(s) and no
| time for documentation or refactoring or any quality of
| life improvements.
|
| Significant documentation is also not something easily
| written in slack time. With any non-trivial project good
| documentation is a full time effort. It also needs to be
| updated as the project evolves lest it become out of date
| and incongruous with the behavior of the system.
|
| There's dreams of in-line documentation but that's only
| going to cover individual methods and classes and not
| necessarily whole modules or major subsystems of a project.
| It's also not going to necessarily cover in-line
| documentation from other parts of a project in different
| languages e.g. comments in config files.
|
| There's lots of technical ways to _help_ documentation but
| there 's no replacement for documentation being a high
| level project goal with resources allotted to it.
| flukus wrote:
| It doesn't help that "Technical Writer" is now an
| incredibly rare job title, it was one of the first in a
| long line of specializations that were pushed on to
| general developers. As you mentioned, they were never
| allocated the extra time but also don't have the training
| and experience that technical writers once did. It was
| also a very poor target for outsourcing.
|
| The specializations common on a team even 20 years ago
| was much closure to the "Surgical Team" model from the
| mythical man month than what we have today.
| gregmac wrote:
| Part of a way around this is broadening your definition
| of "documentation".
|
| For example, in code: good naming, simple structure, unit
| tests, and useful comments all can count. A set of easy-
| to-understand unit tests that cover real scenarios is
| much better than even the greatest documentation ever
| written.
|
| A fully scripted build and ideally also deployment (CI/CD
| pipeline) beats a checklist with tens of items that have
| to manually followed.
|
| An issue system with good descriptions of changes
| (including why and what), reproduction steps, expected
| outcomes, links to related issues, etc also can go a long
| way.
|
| All of this stuff has the side effect of making day-to-
| day work simpler, overall, while actually just being a
| part of that work.
| giantrobot wrote:
| > For example, in code: good naming, simple structure,
| unit tests, and useful comments all can count. A set of
| easy-to-understand unit tests that cover real scenarios
| is much better than even the greatest documentation ever
| written.
|
| 1. Naming things is hard. It's a bit of a joke but good
| descriptive yet usable naming is hard. It can get harder
| after refactoring or adding in features you didn't expect
| the last time you were in the module.
|
| 2. Project structures start out simple. Then you need to
| add some feature or fix some major issue with a due date
| of last week. As a project gets bigger the simple
| structure of yesterday might not serve the needs of
| today. Resources for refactoring are usually somewhere
| around the priority level of "Good Documentation".
|
| 3. Comments need maintenance like explicit documentation.
| Tight schedules lead to "# TODO add documentation here".
| Then you get the assholes that believe comments or in-
| line documentation is something you put in check in
| notes.
|
| 4. A "unit test" that covers a "real scenario" is not a
| damned unit test. It's an integration test (or however
| you label it) and it's not likely something your CI/CD
| system can run on the regular. If a real scenario is
| touching a production-load DB or something it's a big
| deal to run the test. Tests like that need resources to
| _run_ and non-trivial resources to _write_. You 're not
| just making sure a method returns a float or throws an
| error correctly.
|
| I'm not saying your suggestions are _bad_ or shouldn 't
| be used. The issue is management rarely sees any value
| with documentation, tests, or even code cleanliness.
| They're rarely incentivized to care about those things so
| they don't. They get their bonus on feature checkboxes or
| hitting a deadline. At the end of a project they'll
| schedule a "documentation" sprint or some dumbass thing
| that accomplished little useful output. It's worse when
| you've got a "move fast break things" chanting moron
| doing the scheduling.
| rantwasp wrote:
| nah. documentation only goes so far.
|
| you handle this by having and exercising a disaster
| recovery plan and having (and listening to) competent
| people that would spot this issue in a jiffy.
| foobiekr wrote:
| This. I actually think just documentation is, well not
| completely worthless but close to. Imagine telling
| someone they could learn to weld airframes by reading.
| Just no.
| twobitshifter wrote:
| For software, it's the maintenance task that's helped by
| documentation not building. They still need to know how
| to build but it's a lot easier with documentation. I know
| that when we use software packages we all rely on the
| documentation to understand how to use it and code our
| desired behaviors. It's obvious that our own code can
| benefit from documentation in the same way.
| rantwasp wrote:
| documentation is required but not sufficient. also,
| documentation is usually outdated really fast if the
| person maintaining the system leaves and/or does not care
| viraptor wrote:
| There's the possibility that those employees are happily
| retired and don't care. Or they're just not available in many
| ways.
|
| I know of a hospital which is still refusing to migrate off of
| a system which is not supported for a decade. Pay more money? 3
| people worked on it: Adam had a stroke, Bob retired with enough
| money, Charlie left the country.
| notyourday wrote:
| I bet if the hospital is to pay Bob/Charlie $1M for 6 months,
| post tax to fix it they would gladly do it. Note: this does
| not mean "We will pay you up to a million dollars, Net 90,
| after an invoice is approved and signed off by fifteen
| people". This means "We will wire a deposit of $250,000
| today. The rest will be wired every month in 24 hours after
| an invoice is issued"
|
| The reality is company execs like to talk about fixing
| problems with money but rarely are they actually willing to
| fix a problem with money. In my career I dealt with the exact
| three companies where execs decided to throw money at the
| problem bypassing all the fiefdoms and bureaucracies created
| to avoid rapidly solving problems using money in favor of
| slow moving, don't step on Jackson the IT manager's toes,
| approaches. All three solved their problems, destroying the
| local fiefdoms in a process.
| viraptor wrote:
| For $1M they can get the new version of the same system
| installed and likely get a step by step walkthrough of the
| whole thing by any number of new engineers they want. Also
| it's a public hospital - for a fraction of $1M they could
| get people who's only job is to replicate that system
| running around with pen and paper - they wouldn't have $1M
| to just drop on a support issue the same day.
|
| In context of this case - if they don't have money to fix
| their backups
| (https://news.ycombinator.com/item?id=26539988) they likely
| couldn't drop lots of cash without a serious credit line.
| tw04 wrote:
| >For $1M they can get the new version of the same system
| installed and likely get a step by step walkthrough of
| the whole thing by any number of new engineers they want.
|
| A new system isn't going to do them much good when they
| don't even know who their customers are. I doubt any
| customer is going to hang around for a year while you
| rebuild your system from scratch, ignoring the fact that
| if they're an MSP the customers may not even have their
| own copy of the data.
|
| All of that is ignoring the likely contractual
| obligations they're likely under to maintain the data on
| their systems.
|
| >Also it's a public hospital - for a fraction of $1M they
| could get people who's only job is to replicate that
| system running around with pen and paper - they wouldn't
| have $1M to just drop on a support issue the same day.
|
| Hospital? He said they're an MSP on reddit:
|
| >Yeah we're an awful MSP. I've been trying to find a new
| job for quite some time.
|
| https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_abs
| olu...
| viraptor wrote:
| > Hospital? He said they're an MSP on reddit:
|
| This thread is about a different case.
| notyourday wrote:
| That's called priorities. I'm sure as every public
| hospital it pays a lot money in salaries to those
| protecting and growing their personal fiefdoms within the
| hospital.
| TheRealDunkirk wrote:
| This modern-day feudalism with corporations is really the
| root of the problem. Managers "patch" broken IT systems
| with lots of manual labor to actively avoid pointing out
| how some other manager's system is borked. They doom the
| company to spend hundreds or even thousands of man hours a
| year to work around these processes, refusing to allow it
| to be fixed, keeping the "power" and influence of the
| broken system with some other manager. They do this hoping
| to curry favor for the sake of their career prospects.
|
| After 25 years, I'm an expert at making small, efficient
| software tools to fix local IT workflows. God knows, in the
| Fortune 250's I've worked at, there are endless
| opportunities. And, while I've written many solutions, I've
| seen this political game of thrones thwart my efforts many
| times. This is how you get the kind of 20-year legacy mess
| described in the original post: People sacrificing long-
| term success of the company for short-term success in their
| careers.
|
| I guess there are 2 kinds of people. Those that think this
| is just how the world works, and find this to be a
| perfectly-cromulent way of getting through it, and those,
| like me, who find it odious. Luckily, I now work for people
| like me. We do what we can.
| notabee wrote:
| This applies to similar problems in government as well.
| Some people care, some just kick the can to whoever is
| unfortunate enough to inherit the problem in 5-10 years.
| If you get a few of the latter people in political power,
| often the best thought out solutions are offered many
| times and denied because of internal political wrangling.
| [deleted]
| zikzak wrote:
| I have personally turned down an offer very much like this.
| I could have paid my (at the time) salary and benefits with
| about 2 days work a week with the option to bill as much
| time over that as I wanted as long as I agreed to work at
| least 2 days a week on thier top issues.
|
| The ERP system was a nightmare, though, and the company was
| difficult to work with (key, long term employees). It was a
| system I inherited, built out of garbage, that I tried to
| improve. Before I left, I found a very experienced
| contractor to take over, on-boarded him, and stayed as a
| support resource for a couple months. Six months later,
| he's retired and I'm getting panicked phone calls. I valued
| (and still do) my current job far more, even though I would
| probably have done better financially in the short term
| with the alternative.
|
| A few years ago, I heard they migrated to a new ERP at long
| last.
| plorkyeran wrote:
| There's a pretty big gap between "happily retired with enough
| money to not be worried" and having zero interest in ever
| getting more money regardless of the rate. After a decade a
| happily retired person who left amicably and didn't hate
| their job might even enjoy a short-term contract if you can
| isolate them from all the petty annoyances that come with a
| normal job. Charlie leaving the country shouldn't be a
| dealbreaker either unless you literally have tried to track
| them down and couldn't.
|
| If the long-gone ex-employees hated their employer and would
| be happy to see if go out of business then you're in quite a
| bit more trouble.
| gregmac wrote:
| I knew someone who was a specialist in doing calibrations on
| some very specific industrial measurement equipment, much of
| which was out-of-date but massively expensive to replace.
|
| He had retired, but took another 12 or 18 month contract
| flying around the country servicing these things. Apparently
| they asked, and not wanting to do it he quoted what he
| thought was an insane "go away and leave me alone" fee, but
| then they said yes. (I have no idea what that fee was or what
| multiplier he applied, though it definitely brings to mind
| patio11's standard advice [1]).
|
| [1] https://www.kalzumeus.com/2006/08/14/you-can-probably-
| stand-...
| IgorPartola wrote:
| What exactly happens if you break a three month resignation
| period? I don't advocate doing this in generally but as a
| hypothetical exercise if this person just stopped showing up for
| work, what exactly could the company do?
| technics256 wrote:
| Depends on local law. In this case the user is in Finland under
| Finnish labor laws.
| jusssi wrote:
| I just checked for curiosity. The employer can sue for
| damages they'll have to show.
|
| But there's some additional factors here. If the employer
| misses salary payments, then they're in breach of that
| employment contract, and unless they can't fix the situation
| really soon, that's grounds for immediate termination.
|
| Also in Finland, salaries earned in a normal employment are
| backed by the state for up to 3 months of missed payments
| (the state will then try to collect from the employer). So
| it's relatively safe for an employee to stick with a sinking
| company. Just get that CV polished and start warming up the
| network.
| Ekaros wrote:
| It seems to me that they can sue for 3 months salary. Or
| the amount of days employee does not work. However in this
| situation I would imagine they have more immediate concerns
| so might not get around even to that.
| danjac wrote:
| Not sure what country the person works in. In addition it could
| be a situation of a small town / tech backwater and word
| getting around, regardless of the legal question, so quitting
| out of contract could get you blacklisted.
|
| Ultimately though a person could not show up for any reason at
| all (e.g. hit by the proverbial bus) and you need to plan for
| that. Sounds like a company that just flies by the seat of its
| pants so they probably have that 3-month clause for that
| reason.
| michaelt wrote:
| In terms of personal reputation, once you're in charge of
| backups at a company that went bankrupt due to the failure of
| their backups, I'm not sure it makes much difference to your
| reputation whether you diligently work your notice period or
| not...
| hef19898 wrote:
| Depends on the contract and jurisdiction. I had one contract
| that stipulated a fixed penalty, which made any attempt of
| ignoring notice periods a calculated risk (the fine wasn't
| _that_ steep). In any other case, in Germany, I would expect a
| law suite, incurring costs, stress and time, followed by some
| kind of agreement between the two parties. As an employee, you
| don 't necessarily have the time nor the money (there's
| insurance for legal disputes like that so).
|
| In that particular case, my experience tells me the guy won't
| get _any_ credit for finding a solution, I would consider sick
| leave. Burnout, whatever. Doesn 't have any monetary
| consequences in Europe (at least no harsh ones), and Germany
| covers up to 6 weeks of salary.
|
| Any way, you can forget about getting goods references. Not
| that would get them anyway, with an employer like that.
| detaro wrote:
| At least in Germany, while it's in principle possible to sue
| the employee for damages, it is very difficult to succeed
| with that if it isn't downright sabotage. You can't force the
| employee to work, it quickly goes into negligence on your
| part if you can't handle an employee going missing, ..., so
| you'll probably not reach much more than the salary - and for
| that just terminating the contract early would be easier,
| either in agreement with the employee or one-sided after they
| stopped showing up.
| TrackerFF wrote:
| Well, for one, your work references probably won't be the best.
|
| Even if you're not responsible for the problem itself, leaving
| your employer in the ditch, when there's a major outage - well,
| it just doesn't look good (barring some scenario where your
| employer is completely unreasonable - like not paying you).
|
| It's kinda like a firefighter that decides to turn off the hose
| and go home, in the middle of a raging fire. Who in their right
| mind would want to hire him/her in the future?
| ghaff wrote:
| In general, I agree.
|
| But if they get in a situation where management is yelling at
| them and demanding they work 20 hours a day until this is
| fixed somehow and they have no idea how to. Management won't
| bring in consultants, etc. That ship has already sailed.
| dagw wrote:
| _demanding they work 20 hours a day until this is fixed_
|
| If your manager does that, you can say "No". They still
| can't fire you without paying you your 3 month wages.
| jjk166 wrote:
| > It's kinda like a firefighter that decides to turn off the
| hose and go home, in the middle of a raging fire. Who in
| their right mind would want to hire him/her in the future?
|
| The firefighter who recognizes when the building is about to
| collapse and gets out of there is a good firefighter.
| brailsafe wrote:
| Was curious about that as well. I've never heard of such an
| obsurd amount of time, and it seems doubtful it would hold up.
|
| "You need to give us 3 months heads up no matter what, but
| don't worry, we don't have to give you any if you oversleep for
| a meeting."
| nerbert wrote:
| This isn't particularly crazy in Europe. It indeed goes both
| ways.
| ratww wrote:
| In my experience it's also the same in Latin America, but
| the notice period is normally of 1-2 months rather than 3-6
| months.
| ACS_Solver wrote:
| Perfectly common in large parts of Europe. I also have a 3
| months notice period, which is pretty much the standard here
| in Sweden for tech workers or other highly-qualified
| occupations. For very senior positions, 6 months isn't
| unheard of.
|
| It goes both ways of course, my employer is also bound by the
| 3 month period in case they want to fire me.
| wonderwonder wrote:
| How does getting a new job work? You tell the new company
| that you have to give 3 months notice? That's some long
| term planning on the part of the hiring company.
| ACS_Solver wrote:
| Yes, but that's nothing unusual when the entire industry
| expects that. The last time I changed jobs, I signed a
| contract at the end of March, to start in August. That
| was the three month notice period plus most people are on
| vacation in July, so my new company would rather have me
| in August.
|
| There are contractor agencies filling the niche of
| providing employees on short notice (and usually for a
| short period), but generally having people begin 3 months
| later is just something everyone expects and factors into
| their processes.
|
| The average length of time people spend at a single job
| is also longer than in the US.
| rootusrootus wrote:
| Can't speak for Sweden, but we do pretty much exactly
| that with our India team. They have to give notice well
| in advance, and so when we hire a new team member we have
| to do so with the understanding that it will probably be
| a few months before they join the team.
| Leherenn wrote:
| Yes exactly. Sometimes they ask you if you can negotiate
| with your employer to shorten this period, which is often
| the case.
|
| Of course, they would rather it was shorter, but hiring
| is already a multi-months long process in most cases, so
| a few more months is inconvenient, but not the end of the
| world. If you need someone now, you hire contractors.
| hibbelig wrote:
| Don't forget that the notice period applies to all
| companies and all employees. So the whole culture expects
| that, and works with the long lead times.
| arethuza wrote:
| I had a 6 month notice period in one role, 3 months isn't
| uncommon for senior roles, in the UK at least.
| xchaotic wrote:
| while I agree that this is common in Europe, if the company
| ceases to exist, what good is the 3 months notice for the
| employee? If he/she has any options, I'd leave asap and not
| worry about amicable terms with a soon to be non-existent
| entity.
| arethuza wrote:
| Fair point - in that case I suspect that would be highly
| dependent on whether there are any assets remaining in
| the company and how the local legal system treats
| employees as creditors.
|
| Mind you - if the company is going down the tubes they
| are highly unlikely to enforce any job contracts.
| PeterisP wrote:
| What would be the practical consequences to the employee if
| they break that notice period in UK, i.e. stop coming to
| work (or, especially now with Covid, simply stop doing the
| remote work) sooner than these 3 months have passed?
| arethuza wrote:
| I suspect you _could_ be sued by your employer. Mind you
| I 've never heard of anyone being sued in that way -
| can't imagine it would be worth it for the employer
| unless they could demonstrate a significant financial
| loss.
| pw201 wrote:
| I investigated this once (while I was sitting around
| doing not much during a 3 month notice period, funnily
| enough). It's as you say, the breach of a contract is
| something where the employer would have to sue you for
| damages, and if you're sitting around not doing much,
| it's going to be hard to argue that losing you has caused
| them harm, much less enough to make it worth suing over.
|
| I didn't break the contract anyway out of some
| combination of feeling like it was breaking a promise and
| the worry that Cambridge is a small town.
| turbinerneiter wrote:
| It goes both ways, the employer has to give the employee 3
| months notice as well.
|
| But this is for when both of them can't find a mutual
| solution. You can always agree to terminate the contract
| earlier, if both parties agree.
|
| At least that is how it works in Germany.
| puszczyk wrote:
| Same in Poland -- after 2y on a permanent job contract the
| leave is 3 months. It goes both ways. I can be shorter if
| both parties agree.
| yetihehe wrote:
| Yes, if both parties agree, you can leave at once,
| otherwise employer can sue you for damages or lost
| profits (if damages are small, they don't bother) and you
| don't have unemployment benefits.
| matsemann wrote:
| If the company is about to go insolvent it would probably be
| happy about not having to pay for three months. Those three
| months usually go both ways.
|
| But I think it would be better to keep it above board. Present
| it as a solution, kinda. Like, "we both want the same thing
| here".
| ajb wrote:
| I guess it could be a problem in the US for COBRA (health
| insurance continuation)? apart from that, if the company is
| dead I guess there's little they can take away that you aren't
| going to lose anyway.
|
| [edited to add : not relevant, they're in Finland:
| https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
| ]
| elif wrote:
| COBRA still applies if you leave voluntarily.
|
| Source: left voluntarily
| offtop5 wrote:
| In America absolutely nothing happens. He might have some type
| of agreement where he gets a severance payment upon
| resignation, that would be forfeited. If I was him, assuming
| upper management starts yelling or whatever, I would just
| leave. Stress can kill you, and stressing over a job is
| absolutely never worth it because if you die today , they'll
| post a req for your position by the end of business.
|
| It looks like they could probably sort something out, wouldn't
| be a bad idea to call up any of the old vets who retired to see
| if they could come down to the office and contract for a week
| or so.
| mnd999 wrote:
| They could not pay you.
|
| Or
|
| If your contract is exclusive they can pay you and prevent you
| from working for anywhere else (gardening leave).
|
| You can't (at least in most countries) enforce the contract in
| such a way that you force someone to work.
| alistairSH wrote:
| Tangent Alert! _I have a 3 months resignation period, so I can 't
| even leave..._
|
| Maybe it's my (US) American experience, but being unable to quit
| a job strikes me as a terrible policy. This guy(gal?) is really
| required to submit 3 months notice or face some sort of
| repercussions (financial, I assume)?
|
| Does this 3 month period apply in reverse (employer can't fire
| employees without a full 3 month warning)?
|
| Not that the American system (no cause firing/quitting with no
| notice) is perfect, but 3 months? Yikes.
| Eremotherium wrote:
| At least in Germany this goes both ways. The period can also be
| longer depending on your seniority. This is to give the company
| time to find a proper replacement for you and allow for a
| smoother handoff. Of course if both parties agree the grace
| period can be skipped or shortened.
|
| You usually don't get stuck with duds because there's a
| probationary period of (max.) 6 months in the beginning in
| which these protections are not in place.
|
| Also (after the probationary period) the company needs a legal
| reason to fire you. If they wanna fire you for incompetence
| oder insubordination they need to have reprimanded you at least
| once officially, in written form. Only after getting a 2nd
| reprimand for the SAME THING can they act on it. Meaning if you
| told the boss to go fuck themselves once and shown up drunk to
| work once they wouldn't have legal grounds to fire you and if
| they did you'd drag their ass to labour court and get some kind
| of compensation.
| littlecranky67 wrote:
| I don't think it goes both ways. As an employee, your maximum
| notice period is always 3 months - which is also the legal
| minimum required period for employers. Now depending on your
| position, your employer will probably even have longer period
| (i.e. if covered by a union negotioted tarif). Additionally,
| you can't even be fired at will - the employer always has to
| give a reason. "Shrinking down" due to business requirements
| would be a valid reason, but that would mean that the company
| cannot fire you and hire someone else for the same position
| as long as you got the formal qualification.
| [deleted]
| acjacobson wrote:
| There are certain things that you can outright be fired for -
| such as stealing, I believe. I think that showing up drunk
| for work might also be one of them?
| Eremotherium wrote:
| AFAIK showing up drunk isn't enough but if you fuck up (I
| think that's the legal term) and say you drop the
| production DB while drunk that you can be fired
| immediately.
| clipradiowallet wrote:
| IANAL: In the US, doing something destructive [at work]
| while intoxicated can be "gross negligence", which is
| criminal if I remember correctly. Example... you show up
| to work drunk, get in a forklift, and drive through a
| wall causing damage.
| jerven wrote:
| There are more, and they are immediate. And nearly always
| lead to a lawsuit, because it is only for grave errors in
| judgement. i.e. theft, violence, endangering co-workers,
| fraud, misrepresentations to the board and such. So when
| this is triggered it is never a clean break. Lawsuits can
| come from both sides as the reason for the firing is grave
| and the argument doesn't end when a person was kicked out.
| garmaine wrote:
| Depends. Public employees in Bavaria are allowed to have up
| to 2 liters of beer for lunch. Is that drunk?
| lqet wrote:
| > I think that showing up drunk for work might also be one
| of them?
|
| Yes. Also if you just refuse to do any work.
| Mauricebranagh wrote:
| Would not these count as gross misconduct?
| dagw wrote:
| 2-3 month notice in both directions is standard in most of
| Europe.
| tadzik_ wrote:
| In Poland it scales over time: you start off with 2 weeks
| notice (for both sides), after a year or so this jumps to one
| month, and after 3 years it jumps to 3 months, or something
| like that. I've seen people quit right before their 3 year
| anniversary just so that they don't fall into the 3-months-
| notice period.
|
| I haven't worked much on regular job agreements (been
| contracting or "contracting" most of my life), but generally if
| you want to quit you go to your boss and say "hey, I wanna
| quit" and you negotiate when, regardless of the default
| requirements (which only come into play if either side doesn't
| agree). There's no business in keeping an unproductive employee
| around for 3 months.
| K2L8M11N2 wrote:
| What do you mean by "contracting" in quotes?
| megous wrote:
| When you're mostly indistinguishable from an employee,
| except for the legal relationship.
| sedeki wrote:
| "Tax minimization" or worse, I suppose.
| fogihujy wrote:
| > This guy(gal?) is really required to submit 3 months notice
| or face some sort of repercussions (financial, I assume)?
|
| In theory, the business could sue an ex-employee who just left
| for damages caused by the breach of contract, but in the real
| world it would be pretty much impossible to prove that the
| damages were caused by the employee leaving. In most cases it's
| simply considered a dick move, and you'll end up with a hole in
| your CV as you won't ever get a decent reference if you just
| get up and leave.
|
| > Does this 3 month period apply in reverse (employer can't
| fire employees without a full 3 month warning)?
|
| Usually yes. And unlike the situation above, employees could
| probably drag the employer to court for it, unless the employee
| did something illegal or completely irresponsible during
| working hours.
| 2OEH8eoCRo0 wrote:
| I'm looking to leave my current job but if voluntarily leave in
| my first year I have to surrender my signing bonus.
|
| It's certainly a crushing feeling to be chained to a job. Even
| worse are these stupid 2 weeks or unimaginable 3 months notice.
| As everyone says- you'll never get notice you're being fired.
| It'll be quick like a mob hit.
| shagie wrote:
| For many positions, the person isn't a positively
| contributing member for three to six months. A positive ROI
| on the expense of hiring them isn't realized until much
| later. Having signing and relocation bonuses be paid back if
| the person doesn't stay for a year is quite reasonable in
| that context.
|
| "We didn't pay for your relocation to X just so that you
| could change jobs after a few months."
|
| Two weeks notice is a courtesy.
|
| The much maligned PIP is often a hint that you're going to be
| let go. Rarely is there a "you're being let go today, pack
| your desk" unless there is cause (the one time I saw it was
| because someone was using their desktop support credentials
| to snoop on the CIO's computer - that one they weren't even
| given the pack your desk) or the company is in trouble (and
| in which case, things like the WARN period kick in).
| Aeolun wrote:
| > you'll never get notice you're being fired
|
| In Europe you will. Or you'll be sent home with the
| respective 3 months in salary anyway, but generally you work
| your 3 months.
|
| To be fair, I've never seen a notice period longer than a
| month.
| playing_colours wrote:
| I guess US fire-at-will system should make contracting even
| more attractive option for good engineers: you do not get any
| social protection from being fired at any time unlike say
| Germany. One solid benefit less to justify pay difference.
|
| Is it the case?
| Mauricebranagh wrote:
| US contractors don't seem to be able to charge the premium I
| would expect in the UK.
| ghaff wrote:
| Yes and no.
|
| First of all, you get benefits as a full-time employee that
| contractors don't get, like healthcare, disability insurance,
| and paid time off. (This all mostly boils down to money of
| course.)
|
| Probably more importantly, yes, a company can in general
| simply tell you your services are no longer needed one
| morning. However, in reality, absent either company
| downsizing or a documented performance issue, that is
| unlikely to happen out of the blue. Whereas not renewing a
| contract at the end of some project is commonplace.
|
| Employee status vs. independent contractors is actually
| something of a political football at the moment as
| contractors include both the stereotypical gig economy
| workers and freelance writers who want to freelance as well
| as contractors of other sorts.
| alistairSH wrote:
| Health-care and other benefits would be priced into any
| contractor's fees. IE, for a full-time job that paid $100k
| + benefits, you'd need ~$160k on contract to make up for
| the additional taxes, self-paid benefits, etc.
|
| Looking at peers who struck out on their own, you can make
| good money as a contractor, but you typically need to be
| top of your profession and able to command massive fees.
| And like any business owner, the pressure to always have
| too much work usually ruins any chance at using the money
| for longer, more extravagant vacations.
| dghlsakjg wrote:
| Not quite. If you are let go "without cause", meaning they
| fire you for no fault of your own, you are entitled to
| unemployment pay from the government. In my case it was
| around $800/week for about 4 months while I looked for a new
| position. It didn't replace my salary, but it was more than
| enough to pay my bills.
|
| Most large US companies also have a policy of not giving
| references beyond answering if the person worked there for
| the time claimed in the position claimed.
|
| There are a number of benefits to full time employment that
| good engineers appreciate. Healthcare (in the US), known
| reliable income (this can be a big factor for financing a
| house mortgage/car), various other benefits (I get massage
| credit, dental, vision/glasses, acupuncture, physiotherapy).
| The biggest for me is just avoiding the stress of feast or
| famine, managing business/taxes, etc..
| [deleted]
| polote wrote:
| > Does this 3 month period apply in reverse
|
| Yes (in France at least) it is on both sides, unless both sides
| agree to shorten it
|
| > face some sort of repercussions (financial, I assume)?
|
| I'm speaking for France. This is a tricky subject, you almost
| can't get any repercussion. But it can sucks, like the employer
| can refuse to fire you and suspend your contract, in the mean
| time, you are not paid but are not allowed to work for someone
| else.
| CaptainZapp wrote:
| > Does this 3 month period apply in reverse (employer can't
| fire employees without a full 3 month warning)?
|
| Of course.
|
| Different notice periods for employers and employees would be
| outright illegal in most of Europe.
| detaro wrote:
| > _Different notice periods for employers and employees would
| be outright illegal in most of Europe._
|
| AFAIK more common is "lower notice period for employers is
| illegal".
| CaptainZapp wrote:
| I stand corrected.
|
| See my comment below.
| driton wrote:
| I am in the Netherlands, and my current contract stipulates
| that should they want to get rid of me, the company has to
| notify me 4 months in advance, whereas should I want to
| leave, I need to notify them 2 months in advance.
|
| Do you have any sources on the legality of different notice
| periods in Europe?
| Aeolun wrote:
| I guess it's more to the extend that the notice period from
| company to employee cannot be less than employee to
| company.
| CaptainZapp wrote:
| The legal situation in Switzerland would (probably) allow
| for this. What it wouldn't allow is the reverse agreement.
| I.e. you can be fired in 2 month, but have to give 4 month
| notice.
|
| There are three types of employment laws herearound:
|
| - Statutes can be changed freely if both agree
|
| - They can be changed only to the employees advantage. For
| example: minimum vacation days are 20 days. A contract can
| stipulate 30 days, but not 10, or 15 days
|
| - Some statutes cannot be changed at all.
|
| I can obviously not speak for all of Europe (plus INAL,
| etc), but would assume that it's comparable in most of
| Europe, or even more stringent in terms of employee
| protection. Employee laws here are quite liberal compared
| to other European countries.
| jerven wrote:
| Adding to this great comment.
|
| Switzerland, default rules regarding termination (if I
| recall correctly). * first month of work, 0 second notice
| * rest of the first year of work, 1 month notice * second
| year of work, 2 months notice * years after third, 3
| months notice
|
| Impossible to terminate: pregnant women, before and also
| during the legal maternity leave. Common termination is
| the day upon returning to the office.
|
| Personnel called up for the army or civil defence. People
| on a fixed contract.
|
| Employer must give notice but may be without reasons.
| Without reasons often leads to lawsuits, because
| terminated will claim one of the illegal (discrimination
| etc) reasons applies. Still cost of a lawsuit is normally
| not prohibitive.
|
| In case of critical financial issue termination maybe
| shorter (mass-layoffs).
| Symbiote wrote:
| It's fairly common to have an X-month notice period to
| resign, and a 2X or X+1-month period to be made redundant.
| obedm wrote:
| European here. Having 3 months written in your contract (and in
| many countries by law, like France) is pretty common.
|
| You can still quit if both parties agree. I've done it before.
|
| The clause is there to avoid this type of scenario where a key
| employee just says fuck it and leave.
|
| It works both ways. So a company can say fuck it and fire you
| just because they want to.
|
| It's a fantastic system. It can be exploited, but usually works
| really well for employees and it gives you a huge safety net
| you can (usually) rely on.
| mkl wrote:
| > So a company can say fuck it and fire you
|
| "Can't", right? Otherwise it sounds like American at-will
| employment.
| AnssiH wrote:
| Right, missing "not".
|
| The law mandates that the firing notice period cannot be
| shorter than the resignation notice period.
| [deleted]
| artursapek wrote:
| > The clause is there to avoid this type of scenario where a
| key employee just says fuck it and leave.
|
| If they are at the "fuck it" point, what's going to force
| them to be useful and productive during the forced 3 month
| notice period?
| foobiekr wrote:
| The countries in Western Europe specifically are much much
| smaller pools in the tech space than anything an American
| could conceive. The reputational damage of getting to the
| "fuck it" point and becoming a pain in the ass has a lot
| more blow back when the pools of both workers and employers
| are tiny even compared to a second or third tier US geo.
| sangnoir wrote:
| > If they are at the "fuck it" point, what's going to force
| them to be useful and productive during the forced 3 month
| notice period?
|
| Nothing - beyond professionalism. It is common for
| knowledge transfer to happen in this time, and any
| outstanding vacation days are usually taken at this time.
|
| There's a _huge_ cultural difference between US and Europe
| (rest of world?) when it comes to going to work after
| serving notice. It appears the employer /employee
| relationship is more adversarial, so much so that the
| employee is not trusted/expected to do the right thing
| after being fired (or after they quit) - the whole
| "immediately getting escorted out of the building by
| security" is unheard of in Europe.
| [deleted]
| unionpivo wrote:
| It's not really a fuck it point (usualy).
|
| If you reach that you just put in the notice, take whatever
| vacations you have and then put in sick leave.
|
| But 3 months notice is used for, so the company can hire a
| replacement for you, and you can help train him, and
| transfer knowledge.
|
| A lot of people reading this site work in companies that
| have huge teams, but most people in IT I know from EU
| (where I live) are one or two person IT teams. 3 or 4
| person team is considered almost medium sized.
|
| In such situations you need yo hand of your task
| responsibly.
|
| And that the one thing that will get you burned, if you
| leave your previous employer hanging.
| mingusrude wrote:
| In Sweden (don't know where this company is located) it's usual
| with collective bargain agreement and if you've been employed
| for along time (5+ years) it's not uncommon with 3 months
| notice. It is usually accompanied by 6 months if your employer
| wants to let you go (but it's not easy if you're the employer
| to let people go). Most of the time, these periods are
| negotiated when someone resigns though.
| comeonseriously wrote:
| In the US we have the similar crappy situations. No competes
| come to mind. Healthcare tied to employment (what if you wanted
| to start your own business?) is another.
| Ekaros wrote:
| For Finland 3 months is bit special, mine is 1 month and
| previous job was 2 weeks(up to 5 years). From employer side
| legal minimums are: under 1 year 2 weeks, under 5 years 1
| month, under 8 years 2 months, under 8 years 4 months and over
| 12 years 6 months.
|
| Penalty from employee side seems to be monthly wage, so in this
| case 3 month salary. Ofc, seeing how company is probably doing
| I wouldn't be too worried of them coming after him...
| ACS_Solver wrote:
| In my EU-centric worldview, the US system of at-will
| employment, and of two-week notices as the standard courtesy,
| is crazy.
|
| Three months is the typical notice period for tech workers and
| other highly-skilled jobs. One month is the legal minimum in
| Sweden, and certainly several other countries. Of course it
| goes both ways, in fact it's even more difficult for the
| employer to get rid of an employee. An employee doesn't need a
| reason to resign, an employer needs a valid reason to fire an
| employee, in addition to observing the notice period.
|
| To complete the picture, it's necessary to say that probation
| periods exist. Employees may be initially hired on probation
| (typical for fresh graduates), which has a shorter notice
| period, and either side may unilaterally decide not to continue
| past probation. That is, an employer unhappy with a probation
| employee doesn't need additional legal grounds to say "we're
| not proceeding with full employment after the probation period
| is over".
| alistairSH wrote:
| _In my EU-centric worldview, the US system of at-will
| employment, and of two-week notices as the standard courtesy,
| is crazy._
|
| Oh, I agree. But 3 months seems crazy to me. If I have an
| open position and it isn't filled in ~4-6 weeks, it often
| gets given to some other team (on the basis that if I really
| needed that position filled, I'd do so). I can't plan hiring
| over that timeframe.
|
| And when hiring employees in India, we've had them claim a
| 2-month resignation period, but they just end up using that
| time to find some other opportunity on the previous
| employer's time. It's a real mess.
| ACS_Solver wrote:
| > Oh, I agree. But 3 months seems crazy to me. If I have an
| open position and it isn't filled in ~4-6 weeks, it often
| gets given to some other team (on the basis that if I
| really needed that position filled, I'd do so). I can't
| plan hiring over that timeframe.
|
| If you're in a culture (locality + industry) where a
| position can be filled in 4 weeks, that's one thing. If
| you're in a culture where every experienced person has 3
| months of notice, then nobody would take your inability to
| fill a position within weeks to mean that you don't really
| need it.
|
| I have some experience with part-time contracting with US
| customers. Seeing how quickly people can join or leave
| there is certainly a culture shock!
| foobiekr wrote:
| Assuming that you're talking about senior people, who are
| all employed, I don't see how this helps. Realistically
| employee X resigns, and we find a replacement; that
| replacement is working and needs to resign. Their end
| dates are still serialized.
|
| We have a bunch of employees in Europe and everyone
| involved except the very marginal employees sees this
| situation as strongly negative (and, in fact, an argument
| to not hire in Europe at all).
| Symbiote wrote:
| Are those Europeans who don't value their rights fairly
| young?
|
| I don't need my 3 months now. They might be useful if my
| employer axes a whole department. They're useful for
| people with family commitments giving them less mobility.
| 2rsf wrote:
| Since the same base rules apply for everyone this usually
| doesn't happen, plans are built accordingly and
| recruitments take into consideration those time scales.
| anyfoo wrote:
| I had a "to the next quarter" period in my last job, and
| when moving to the new job the deadline was missed, making
| the contract valid for a whole quarter.
|
| It wasn't a problem. I was on good terms with the previous
| company (I'm still getting invited to their anniversary
| parties and stuff like that), so there was no issue in
| working something out.
|
| Also, remember that ~6 weeks of vacation per year is common
| outside the US, so often it boils down to taking your
| remaining vacation. (Though I think you need to be careful
| with taxes and other regulations if the contracts overlap.)
| etripe wrote:
| > But 3 months seems crazy to me.
|
| What you might not realise is that these three months would
| be the default legal position _in case of disagreement_. If
| you want to leave earlier and the employer agrees, you can
| usually leave earlier, too (with them being off the
| financial hook).
| tkgally wrote:
| I remember reading many years ago that employees in the U.S.
| have the right to quit at any time because being penalized
| for quitting would make the employment "involuntary
| servitude," which is forbidden by the Thirteenth Amendment to
| the U.S. Constitution. I can't find any confirmation of that
| explanation on the web now, though.
| alistairSH wrote:
| Yup. Most states are "at will", which effectively means
| both parties can leave the arrangement without notice or
| repercussion.
|
| There are federal laws that limit this, typically applied
| to larger employers making entire business lines redundant
| (closing a factory, etc). State laws very, some with zero
| extra protection, some with a bit more, but AFAIK none with
| European-style rules.
|
| In professional jobs, it's traditional for an employee to
| give 2+ weeks notice. Most employees will try to stay long
| enough for a reasonable transition - depending on
| position/project status, that could be 2 weeks or a month
| or two.
|
| It's traditional for employers to give 2+ weeks severance.
| Most white-collar employers have periods that start at 2
| weeks and grow with tenure. Including benefits is hit or
| miss, and a HUGE issue because health insurance/care in the
| US is so expensive.
| staticman2 wrote:
| You can quit but they can sue you for breach of contract.
|
| I think what you are thinking about is the court almost
| certainly won't order you to go back to work after you
| quit.
| garmaine wrote:
| This is correct. If you don't have the right to quit at any
| time, you are legally a slave / indentured servant. And
| that is very, very illegal.
| ghaff wrote:
| The military is sort of a special case however. (And, of
| course, private employment contracts can make it
| financially onerous to break.)
| garmaine wrote:
| Interestingly the case can be made that the military is
| not exempt from the 13th amendment, which only makes one
| single exception for prisoners. It's just that the courts
| have ruled consistently that _OBVIOUSLY_ the military
| wasn 't meant to be included in the ban against
| involuntary servitude. Obviously.
|
| Just like they have also ruled that mandatory Jury duty
| is not in violation of the 13th amendment either. Funny
| how the system protects and perpetuates itself.
|
| But with respect to your point, I am nearly 100% sure
| that it is not the case that employers can use financial
| penalties to enforce such a contract. Anything you are
| paid as wages is pretty much untouchable after it is paid
| out, unless it was paid in mistake or you intentionally
| damaged the company or something. The best the company
| can do is withhold potential future earnings or bonuses,
| or take back conditional non-wage payments like bonuses
| within a certain window of time.
|
| This is part of why golden parachutes exist. Your
| contract has very generous severance clauses which award
| you lots of money if you follow a certain procedures for
| leaving, which involves advance notice and such. Fail to
| follow that and your paycheck just stops the day you walk
| off. So there are incentives, but they can't go after
| your already earned wages, paid or still due to you.
| ghaff wrote:
| > Your contract has very generous severance clauses
|
| That's pretty much what I meant although, as noted in
| another comment, I assume there are cases where they
| would try to claw back relocation or signing bonuses.
| Also, of course, things like RSU vesting.
|
| >they can't go after your already earned wages, paid or
| still due to you.
|
| Certainly, and depending on how paid time off accrues,
| anything earned there--which is one of the drivers with
| "unlimited" PTO because that's not owed when you leave.
| hvidgaard wrote:
| In reality, you can walk out of any job in EU as well. Unless
| you have some specific clauses in your contract, the worst
| they can do is simply not pay you anything. There may come a
| lawsuit after, but I doubt it. You can also get a doctors
| note and go on sick leave if you feel some serious stress on
| the job.
| ACS_Solver wrote:
| It's true that they probably won't sue you for breach of
| contract, though they can, but if you just walk out of a
| job, you're at the risk of ruining your professional
| reputation, arguably the most precious resource. There
| aren't many other things that would be as much of a red
| flag when hiring as an employee that just decided to walk
| out of a job.
|
| Of course if you have good reasons to stop working, there
| are good options. If it's stress or similar mental health
| issues, you can resign and spend your notice period on sick
| leave. If your employer assaults you or just stops paying,
| or otherwise grossly neglects their duties as an employer,
| you're legally entitled to quit immediately.
| benhurmarcel wrote:
| Typically a company owes you money when you leave, like the
| started part of the month, pending holidays, and the annual
| bonus/extra pay that you already earned.
|
| If you don't respect your notice period they'll remove your
| salary over the missed time from that final payment. So in
| that case they'd remove 3-month salary. Sometimes it goes
| into the negative, and some companies actually send a bill,
| but there's basically no consequence for ignoring it.
| Usually companies don't push the matter.
| Mauricebranagh wrote:
| I agree (I am from the UK) when first saw US professional
| workers complaining about > two weeks my first thought was
| wow you must have a really low status job.
| foobiekr wrote:
| Not too long ago a friend witnessed the SVP GM of an $5b a
| year division quit with two weeks notice.
|
| I'm senior enough that I give at least a month, often more,
| and this made me realize that I've been perhaps overly
| generous.
| dspillett wrote:
| UK here. Terminology may differ, I think what is being
| discussed is a _notice period_. Normally a resignation is
| either accepted or rejected pretty much on the spot and if
| accepted is officially seen as a more amicable departure.
|
| Handing in your notice to leave is something the employer
| cannot reject. Depending on circumstances you might try to
| resign first. Sometimes you may be "asked to resign"[0] and
| made a nice offer to encourage you to do so, offer will almost
| always have an NDA attached to it.
|
| [0] this can be preferable to the company as it can be quieter
| than sacking you
|
| _> This guy(gal?) is really required to submit 3 months notice
| or face some sort of repercussions (financial, I assume)?_
|
| Yes. The worst they can do is take you to court for breach of
| contract though. That can be expensive and embarrassing for
| both parties so unless they are whiter than white[1] or bitter
| enough companies often don't bother.
|
| Potentially a big risk to take though. And in an industry that
| is hardly free from nepotism, you probably don't want the
| reputation of being someone who walked that way.
|
| Three months is most common in skilled jobs, one month
| otherwise. In retail work and other low-skilled professions two
| weeks is not uncommon, though here people often do just walk
| away as the hassle of chasing them for breach of contract is
| massively not worth it.
|
| [1] i.e. there is nothing you can say on public record that
| looks bad for them going forward
|
| _> Does this 3 month period apply in reverse (employer can 't
| fire employees without a full 3 month warning)?_
|
| Yes. What usually happens if they want to have you gone without
| notice and don't want to make you an offer to resign "amicably"
| is that they put you on gardening leave. You are still employed
| and paid for the notice period but are told to stay home. They
| can't stop you looking for new work[2] as long as you don't do
| it "on their time"[3] but you can't start new work without
| resigning and them accepting that resignation until the end of
| notice period without the same risks as if you had just walked
| out without notice yourself[4].
|
| Gardening leave can also happen when you hand in your notice,
| though if you want to leave and the company wants you to leave
| then they'll just agree to waive the notice period.
|
| [2] Some contracts have clauses that say otherwise, those
| clauses are not legally enforcible. Some even have clauses
| saying you have to actively look for new work, these too are
| not legally enforceable. [3] Which is impossible for them to
| prove unless perhaps they have you check-in regularly while you
| garden. [4] Though they are likely accept the resignation as at
| that point they can stop paying you for the notice period,
| unless you are going to work for a competitor, or they
| otherwise feel like being dicks about it.
| ectopod wrote:
| At places I've worked in the UK it is asymmetrical. The
| employer has to give three months' notice but the employee
| only has to give one month's. It is intended as an employee
| benefit, and it is. I've never been asked for more than one
| month's notice from my side.
|
| I don't know where the OP is located but he should double-
| check his contract. The first time I heard of the three month
| period I was reluctant, until I realised it didn't apply to
| me.
| numbernine wrote:
| In my country, we have a 3 month notice aswell and yes it does
| also apply in reverse. You can negociate to lower the period
| but most of the time the company will decline your request. For
| high-demand jobs it is a nightmare for everyone.
| Symbiote wrote:
| I once had a colleague "go on holiday" and never come back to
| work.
|
| If you don't work your notice period, you obviously won't be
| paid for it, and any reference will state that you broke your
| contract.
|
| (Unless your employer asks that you don't come into work any
| more. This is called "gardening leave" in Britain:
| https://en.wikipedia.org/wiki/Garden_leave , and it's normal in
| some industries and circumstances.)
| Mauricebranagh wrote:
| :-) I once replaced some one (UK) who "disappeared" -
| apparently he went to NYC
| throwaway14337 wrote:
| In a previous job it slowly dawned on me that the job was
| making me quite unhappy. This was about six months in. Only six
| more months until the "vesting cliff"! I managed to power
| through and get my stock options for the year.
|
| So, while we don't have 3 months notice in the US, we certainly
| have similar ways to get stuck.
| viraptor wrote:
| These kind of posts are awesome tabletop scenarios. (and fuel for
| https://twitter.com/badthingsdaily ) Coming up with potential
| fixes is a good exercise.
|
| Things I would do that I haven't seen mentioned yet:
|
| - Grab images from the relevant servers to have current backups
| before messing with anything.
|
| - Try tcpdump-ing and decoding the custom KV protocol - other
| databases are complicated, but things like memcached/redis/...
| have trivial wire protocol. Maybe it's possible to recover some
| data by listing all entries.
|
| - Run `strings` on the KV binary or `file` on the data to make
| sure it's not just some open source project with a proprietary
| interface. With any luck it's just something like bdb that can be
| accessed externally.
|
| - Hook up a debugger to the KV and try to recover the keys (or
| maybe just strace to see if they keys are in any files)
|
| - Start tcpdump on the old DNS IPs. They mentioned they have no
| idea which domains they own anymore - as long as NS on customer
| domains wasn't trashed as well, they could collect queries coming
| in and recover a list from there.
|
| - See if the customer domains can be collected from billing data
| / some email store of notifications.
|
| (as much as I try to design for no disasters in production, the
| rush of "this is so FUBAR, what insane thing I can do to fix it"
| for me is an amazing feeling :) )
| comeonseriously wrote:
| Choice 1: Leave. Just leave.
|
| Choice 2: Put your head down and fix this. Your worth (to another
| company because you really should leave after you fix this) will
| increase dramatically.
| aidos wrote:
| I've seen a few data disasters in my time but I think this takes
| the cake. It's a good reminder of why you just need to prioritise
| the sort of work that has no immediate payoff.
| flaxton wrote:
| Can't you just pull the drive from the failed server, extract the
| DNS entries and build a new DNS server? It doesn't sound like
| everything is broken, just that it is failing because the DNS
| server isn't answering. Just install Linux on a new box and set
| up a DNS server on it. It could be done in a day I would expect.
| Or am I missing something?
| Kranar wrote:
| Its encrypted and the encryption keys are lost.
| capableweb wrote:
| It would work if they knew what the DNS entries were but seems
| they lost all DNS records and don't even know what records they
| have/had, so spinning up a new DNS server won't help as they
| don't know what it should serve.
|
| > All domain information was wiped out and records became null
| [...] Our records are wiped from all domain servers out there
| [...] We don't even know what domains we own, the listing was
| hosted in the ERP which is now busted
| drummojg wrote:
| That's part of what bothers me about this whole story--it
| says they were running BIND, which configs on text files for
| goodness sake. These critical records were tiny and could fit
| anywhere & transfer in the blink of an eye. That such a
| simple thing is buried under & dependent upon an entire
| complex and untested/maintained DR plan is mind-boggling.
| capableweb wrote:
| > That such a simple thing is buried under & dependent upon
| an entire complex and untested/maintained DR plan is mind-
| boggling
|
| I find this to pretty common in many setups where the
| engineers don't focus on simplicity and removing layers of
| abstraction. If the workforce is young and inexperienced,
| over-engineering tends to happen everywhere and you end
| with situations like this.
|
| I have seen worse in my years in the industry, that's for
| sure.
| michaelhah wrote:
| How many people had to log into their own work systems and check
| that things were running in order to be sure it wasn't their
| employer?
| loloquwowndueo wrote:
| I know it wasn't mine because we still have one guy who knows
| how everything works :) (unless he quit over the weekend oh
| crap)
| HourglassFR wrote:
| Nah, I'm not saying my employer's IT infra is rock solid but at
| the very least we have backups.
| simonebrunozzi wrote:
| Best thing to do for this poor sysadmin, before all else: talk to
| a lawyer, understand what you should do from now on to minimize
| risk of being sued by company/shareholders, etc.
| airhead969 wrote:
| 50% of companies who lose all of their data go out of business
| within 6 months. They clearly failed to test their backups with
| regularly-tested restores. And, a separate alternative system
| should've had multiple revisions of critical pieces of
| information like customers, assets, inventory, bank accounts,
| debts, and employees.
|
| A data recovery shop could attempt to forensically-reconstruct
| SMS numbers. Look for logs and other places critical deets might
| be found. Try your best to get things back and make alternatives
| processes to piece together what you can. A phone carrier or SMS
| send provider might have a log of phone numbers. The paycheck
| company may even have the address of everyone. Send them a $0.01
| paycheck with a note on them to call in.
|
| I would ride it out for a paycheck because it's not like it could
| get worse than going under. Just be sure to get paid.
| lopatin wrote:
| They didn't fail to test their backups. They tested them, they
| didn't work, and then they said "this is fine".
| airhead969 wrote:
| Oh my, I missed that, thanks. I can envision the dog in the
| burning house meme. Ouch!
| joncrane wrote:
| Only 50%?!?
| ghaff wrote:
| Reaching people seems like the least of their problems to be
| honest. If they use a payroll service, they'll have addresses--
| don't know about Europe but does just about any company do
| their own payroll in the US at this point? In any case, I
| assume most people know how to reach most of their direct
| reports and many of their co-workers. Especially at a fairly
| small company, I would think you could reconstruct contact info
| on employees fairly quickly.
|
| Of course, that may not help with the bigger problem.
| airhead969 wrote:
| Yep. It's going to be a massive problem figuring out what's
| what, who owns what, and who owes what. I don't even see how
| they'll keep enough customers to stay afloat.
| PaulHoule wrote:
| You should sue the LTO tape conspiracy that means you need to
| spend $3000 to back up the contents of a $300 hard drive.
| tclancy wrote:
| "let the boat sync."
|
| Sure, now you start thinking of backups.
| atmosx wrote:
| I loled, although non-native speaker who does similar mistakes
| all the time :-)
| sombremesa wrote:
| Loathe to be "that guy" but a sync is not a backup.
| le-mark wrote:
| It's easy to imagine there are a lot of these type of time bombs
| out there, particularly in really old legacy systems (> 20 years
| for example). I was at one place where even building the
| application for deployment led to a one week outage when some
| core people were laid off.
| protomyth wrote:
| Interestingly, some of these legacy systems run on things like
| IBM i (AS/400) which have an easy backup story and vendor
| support for recovery. I can have a new i machine receiving a
| backup and deployed fairly quickly (day or two) and I'm out in
| a rural area.
| abruzzi wrote:
| Up until 5 years ago, my employer's primary system (government,
| property tax assessment, billing, and collection) was from a
| company that had gone belly-up in the early 90's. We
| fortunately had the source, but it ran on a VERY legacy
| database called Unidata because it had originally be developed
| on the Pick operating system. Unidata was/is a Multivalue
| Database with a built in BASIC programming language. By the
| time 2014 rolled around there was one programmer in the
| organization that understood the system, and he was responsible
| for all code changes. I half-joked with management that we
| needed an insurance policy on his life/ability to work, because
| if we lost him, it would put us in a very difficult position,
| and that software accounted for 90% of our revenue (as well as
| a lot of the revenue that went to all the local towns and
| school districts for their operations.) That employee was also
| nearing retirement (he is retired now.) Fortunately we were
| able to successfully migrate off of that application, though we
| keep a minimal license pool to the old system so people can
| validate pre-migration data against post migration data.
| tsomctl wrote:
| improved link:
| https://old.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
| benlumen wrote:
| Thanks. Reddit is such a good example of how "progress" hasn't
| really been progress in web development. The new front-end
| framework site is so much slower and more buggy.
| xchaotic wrote:
| wow, it was so much better before. What the heck? Who made
| the decision to switch?
| mkl wrote:
| They want people using their app, for some reason I don't
| understand. Maybe to make ad-blocking harder by getting
| people off computers where it's easy (most mobile browsers
| don't support plugins, so it can't be about them).
| the_only_law wrote:
| Ironically, every time I click their "open in app" link
| it's drips me straight to the App Store despite me having
| the app installed.
| perlgeek wrote:
| The old design was optimized for usability, the new is
| optimized for ad revenue and/or for pushing people on the
| app.
| fsflover wrote:
| Even more improved link:
| https://teddit.net/r/sysadmin/comments/ma4mwl/the_absolute_w...
| sodality2 wrote:
| Superior link (if you can parse json): https://www.reddit.com
| /r/sysadmin/comments/ma4mwl/the_absolu...
| newsbinator wrote:
| > we don't even have a listing of our ~300 employees SMS numbers
| elif wrote:
| This surprised me. If ever there was a use case for an
| organizational structure, this is surely it.. just have every
| manager reach out to their direct reports? Following the tree
| communication should happen at something like nlogn speed
| arnaudsm wrote:
| Are the CTO and sysadmins liable in such scenarios? Can they be
| sued ?
| viraptor wrote:
| Anyone can be sued. But it wouldn't be a good idea if they kept
| the receipts:
|
| https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
|
| > We tried really hard to make management aware and actually
| succeeded in that, only for management to follow up and say we
| don't have enough money to fix it, and they understood the
| possible outcomes.
| teddyh wrote:
| They might check on robtex.com and other DNS historical
| information sites.
| TeeWEE wrote:
| Exactly what i was thinking. Its D_NS its Distributed. The
| records are out there somewhere.
| teddyh wrote:
| The 'D' in DNS stands for "Domain", not "Distributed".
| mperham wrote:
| Technical debt can lead to non-technical bankruptcy.
| [deleted]
| pjmanroe wrote:
| I worked for a newspaper group back in the 1990s. I did all their
| programming in dBase III & IV LAN. I was putting in 80+ hours a
| week. I wasn't getting paid squat, but then they wanted to put me
| on salary (at less than I was currently making) and I said no way
| I'd give my notice first. So that's what I ended up doing.
|
| They didn't believe me. But I left. After a few months of
| haggling I went back at $50 an hour on contract. I did this for a
| few weeks (about 25 hours a week), then started to not want to
| pay me. So I quit for good. Unless they made an upfront deposit.
| Which they wouldn't do.
|
| They ended up spending close to $250,000 on equipment and
| programmers and still couldn't do what I could. They were
| duplicating and tripling everything. Wasted so much money. Now
| I'm going to college for my BS in Computer Science with a
| specialty in Software Engineering. I'm currently running Windows
| 10 Pro Insiders Edition DEV and VMware with Kali Linux.
| gillesjacobs wrote:
| The managed service provider company built a custom DNS inside an
| ERP without backups or bootstrap strategy in place.
|
| House-of-cards core infrastructure is a sign of negligence and OP
| should have reconsidered working there a long time ago.
| hinkley wrote:
| Division of responsibility can make this sort of thing very
| difficult to know.
|
| If you aren't running drills then transparency may not be in
| everybody's self interest and someone somewhere is covering up
| for the fact that they only half know what they're doing.
| sorokod wrote:
| "Every person who knew how the custom DNS system worked has left
| the company years ago."
|
| Keeping the people who know how the system works is really the
| last line of defense.
|
| From, the point where such last person left without any
| compensating mechanism in place, the clock started ticking
| hinkley wrote:
| Somewhere right now there is a disgruntled former employee
| drinking a glass of Scotch and smiling.
| ChrisMarshallNY wrote:
| This is the snow on the tip of the iceberg. I'm sure there's
| plenty more where this came from.
|
| There's so many things wrong with the scenario described, that I
| can't even begin to talk about it.
|
| My experience is that folks never want to talk about backup or
| DR, as they are expensive, and the idea is that they should never
| be used.
|
| No one ever has a problem with insurance premiums, though...
| hinkley wrote:
| I keep wondering if there's an underwriter angle here where you
| offer insurance and audit the company to set the premiums based
| on how broken their IT situation is.
| blhack wrote:
| It's dark, but I kindof love situations like this (although
| luckily I haven't had something like this happen to me in digital
| space since I was a teenager). Some places I would look:
|
| 1) Are you SURE that everything is actually gone? Is there a
| logfile, a billing trail, sent emails trail from your email
| provider, something like that? If it was me I'd be looking for a
| secondary system (like alerts, logfiles, billing, etc.) that had
| some of this information stored in it as secondary function, and
| writing some scripts to parse those things and start rebuilding
| what can be rebuilt.
|
| 2) You said you lost all the emails and SMS of your team. Do you
| have an HR department? Surely they have this information. Your
| team is likely going to start trying to send emails to each other
| about this. Can you dump logs off of your mail server to rebuild
| at least the naming schema? For phone numbers: can you try an old
| school phone tree style? Start with somebody you know, and ask
| them for everybody that THEY know and so on.
|
| 3) Was there a staging environment that might have had some of
| the data you're missing on it?
|
| 4) Are you really really sure that everything was wiped out? Is
| it possible that your ERP system is broken (lost its DB
| connection or something?) and that caused a cascading failure
| that makes it LOOK like everything is gone?
|
| 5) Can you get at the BIND server somehow? Are there logfiles
| there?
|
| I'd love to know more specifics about what "wiped out" means in
| this case. It seems unlikely that it's properly "wiped out"
| unless something malicious happened, and even then I think you
| could rebuild a lot of this from sortof "secondary" data.
|
| Think like a hacker, but your system is your own target. How can
| you steal the data you're looking for from places it isn't really
| "supposed" to be?
| CobrastanJorji wrote:
| I get where you're coming from. It's a little bit like a real
| world, high stakes escape room.
|
| Come to think of it, this could be a fun online challenge. I
| picture a series of VM images, each of which describes the
| design of the system and the terrible thing that has gone
| wrong, and you are challenged to get it fixed and/or get the
| data out of it.
| hinkley wrote:
| Start connecting with everyone on LinkedIn for one...
| Graffur wrote:
| That's actually a very good idea. Even the people not on
| there might be connected through other platforms whatsapp etc
| so could be contacted
| devit wrote:
| Definitely not the "absolute worst", they merely lost all DNS
| records.
| mysterydip wrote:
| A good reminder for everyone to check your backups work, and if
| you don't have any, start!
| hinkley wrote:
| QA departments used to be good for this.
|
| Setting up a test cluster using real backups can be good. If
| you have the crosstalk problem sorted out that is.
| TheRealDunkirk wrote:
| One of the comments in the thread said, simply, "Prepare 3
| envelopes." Being a 25-year sysadmin, immersed in thinking about
| the problem, it caught me totally off-guard. I don't think I've
| ever seen a more perfect response in a Reddit thread.
| etripe wrote:
| Which 3 envelopes would those be?
| lmkg wrote:
| It's an old story.
|
| New executive starts the job. On his desk are three
| envelopes, left by his predecessor, with a post it note
| saying "When shit hits the fan, and you don't know what to,
| open one of the envelopes."
|
| Yadda yadda, exposition, things are normal and exec is
| successful for a year or two and then things go pear-shaped.
| Completely out of options, he opens the envelope, finds a
| letter which says "Blame me." He blames his predecessor for
| leaving the state of things in such a poor shape that this
| misfortunate was unavoidable, everything blows over, his
| success continues.
|
| Blah blah, narration. Another shitstorm. He opens the second
| envelope, finds a letter which says "fire all the managers."
| He promises a significant shake-up, says he has the
| leadership to turn this around, purges the old guard. Things
| go well.
|
| Hurly-burly story, and so forth. Catastrophe strikes. Having
| blamed his predecessor and fired everyone and installed his
| own managers, he has no one else to turn to for casing the
| blame. He opens the last envelope, hoping to find salvation.
|
| "Write three letters."
| unnouinceput wrote:
| My recollection of 2nd letter is "blame it on current
| economy situation"
| TheRealDunkirk wrote:
| It's an older reference, which is why it was so funny to me.
| I hadn't seen it in a long time. GIYF.
| scandox wrote:
| How people handle this level of disaster is always very
| revealing. In every case I've ever experienced someone kept their
| head and a solution was found. Usually there was reputational
| damage but surprisingly most times the actual business bounced
| back. But it was always the actions of someone in particular in
| those first hours that made the difference.
|
| Thinking you're fucked is unhelpful, because you start talking
| about resumes and expecting some deus ex machina to appear and
| resolve the situation (for good or ill).
| batch12 wrote:
| One thing the situation does show is a lack of proper care for
| their most critical assets. When the dust settles, there should
| be hard questions for several people. One being-- when was the
| last time backups were tested..
| amachefe wrote:
| The answer to the question is there
|
| >We can't access the backups. Even if we could, we noticed
| two years ago when trying a restore from the backups, that it
| doesn't work. Booting the restored server leads to a kernel
| panic we couldn't figure out. Management said we don't have
| enough money to fix any of this.
| carschno wrote:
| I have no reason to doubt that this is factual. However, I
| have serious doubt that management will acknowledge they
| made that decision.
|
| Typically, this kind of decisions are made more or less
| implicitly during long meetings about priorities, often
| without any minutes. Managers will presumably try to hold
| (senior) engineers accountable as they had not made clear
| enough how important that was.
|
| I obviously don't know who is right here.
| kwhitefoot wrote:
| > Management said we don't have enough money to fix any of
| this.
|
| It's often even worse. Many years ago I was responsible for
| backup a DEC Unix system. Asked for funds to conduct a
| disaster recover exercise to find out if our backups worked
| and was turned down even before we had tried to work out
| how much it would cost.
|
| I often wonder what proportion of backed up files and
| systems are actually in a state to be restored and have the
| necessary tools and expertise to do it in a timely manner.
|
| An even worse problem in some places, including the one I
| worked at, is a tendency for users to assume that backups
| are kept forever and for the management to totally neglect
| archiving of important documents such as specifications,
| drawings, and design calculations of products that have an
| expected lifetime of over fifty years.
|
| I had to inform several users that I could not restore the
| file that they had only just noticed was missing because it
| was last seen more than two years ago and presumably
| disappeared longer ago than our longest backup cycle which
| was one year of monthly full backups.
| benlivengood wrote:
| > I had to inform several users that I could not restore
| the file that they had only just noticed was missing
| because it was last seen more than two years ago and
| presumably disappeared longer ago than our longest backup
| cycle which was one year of monthly full backups.
|
| If you can afford 12 months of backups then you can
| almost certainly afford >=12 years of yearly backups by
| buying a new set of monthly tapes once a year and taking
| a set out of rotation and into archive.
| Symbiote wrote:
| But you are then archiving data (not just backing it up),
| and will need to work out how to delete records you are
| no longer legally able to keep.
|
| (GDPR and earlier laws, etc.)
| pedro2 wrote:
| IMO Management will find the money pretty quick if the
| business survive.
|
| Up until now, they didn't know what 'backup not working'
| meant. Now they know.
|
| Not being bitter -- there was a failure of communication
| from both parts.
| colmvp wrote:
| Thinking you're fucked is unhelpful, but it's pretty normal to
| go full Alien Hudson in a dire situation and need someone else
| to assure you that things will be okay.
| 4e530344963049 wrote:
| If you have been that person before, it can be fun and exciting
| to think up solutions and implement them under pressure.
| Definitely a real test in your ability to concentrate. But
| don't expect the credit you deserve for it, and don't even
| expect others to even bother to put in any effort towards
| fixing things.
|
| Also, make sure everyone knows you are going on vacation once
| you resolve things.
| paulz_ wrote:
| These types of situations always wind up being the most fun I
| have at work and leave me feeling most fulfilled. I wish I
| could do that all day instead of going to standups.
| 4e530344963049 wrote:
| A "firefighting" consultancy might be fun, no? And you
| would give people a lot of time off!
| pratik661 wrote:
| I always wondered this:
|
| If you are an external consultant (with no prior
| knowledge of the company's systems/processes/hierarchies)
| brought in to "fight" a particular fire, how do quickly
| get ramped up to a level of knowledge where you can
| analyze the root cause of/recommend a fix for a
| particular "fire"?
| Clubber wrote:
| I would say the analysis is 90-99% of the work, based on
| my experience. Especially the more jacked up systems
| where cheapest devs found work on the codebase and it's a
| mess. 90% only comes in when it's so jacked up it
| requires a lot of refactoring.
| paulz_ wrote:
| I've thought about that before. You get the call. "It's
| 30 minutes away. I'll be there in 10"
|
| If anyone happens to know of a career path or company
| that does that sort of work I would be interested to hear
| about it. Bonus points if the pay is half decent.
| clipradiowallet wrote:
| I don't know of a _company_ that does this sort of work,
| but I know of some technologies that experts in receive
| these types of calls. The one that comes to mind is an
| ERP-focused database system. It 's called "Progress" by
| "OpenEdge". IMHO, it's _awful_ , but this has no hindered
| adoption in the slightest. I wrote Progress/4gl (their
| query language) often enough in a prior position to have
| it on my resume. Every 2 or 3 months, I'll get an
| email/call, asking if I could be available for short-term
| contracting upwards of $200/hr for Progress emergencies.
| I have declined all of these, because I found it soul
| crushing to work with in the past. However, if you could
| enjoy that sort of thing, that's one example of a very
| lucrative field to dabble in.
| munchbunny wrote:
| Blue teaming in big company cyber security teams will get
| that for you. Not everything is a true positive but it's
| always urgent. Pay is decent too.
| meowface wrote:
| I've worked such a job at a large enterprise. It really
| does feel like firefighting (minus all the smoke
| inhalation and physical strain and death risk).
|
| However, not only is not everything a true positive,
| probably only about 0.001% of things are true positives,
| among a sea of alerts and reports and dashboards across
| myriad systems. Some coming from your SIEM, some
| generated by security appliances and products, some from
| internal employee reports.
|
| An ideal place will have people who continuously work on
| trying to reduce alert fatigue and false positive noise -
| but, in practice, at most big companies it's probably
| like working at a fire station and getting hundreds of
| dispatch calls per hour, every hour, every day, each
| about a potential fire at a different residence. And then
| you drive up and see they just used the stove for a few
| minutes or a character said the word "fire" in a TV show
| they were watching.
|
| But you have to urgently show up every time no matter
| what because, occasionally, the house actually is
| engulfed in flames and might be on the verge of igniting
| the whole town.
| htrp wrote:
| I spoke with a CISO of a large F500, his biggest gripe
| was that he has a team of 30 that can barely keep their
| heads above water, let alone respond to incidents
| insomniacity wrote:
| I know Mandiant do this (disaster response) for security
| incidents. Don't know of a generalised service, but I
| imagine the big consultancies offer it.
| easton wrote:
| This is my ideal job, I think. Charge a big rate to be
| the consultant you call when everything dies from
| ransomware or OPs problem or whatever, I calmly fix it to
| normal standards (as possible), take three weeks off
| after.
| notyourday wrote:
| > A "firefighting" consultancy might be fun, no?
|
| It is... interesting. I have been doing this over last 4
| years. The biggest problem, as I pointed out before, is
| the disconnect between what companies claim they would do
| to fix the problems vs. what they would actually do. So
| ~90% of selling is repeatedly explaining the same things
| to different people whose jobs are in the eminent danger
| of being restructured/deprioritized/eliminated if your
| services are successfully engaged.
| 4e530344963049 wrote:
| Do you mind if I email you with some questions? Thank
| you.
| notyourday wrote:
| Sure. Email is in the profile.
| beckingz wrote:
| hopefully you can skip most of those people and talk to
| the people who are in danger of being eliminated if your
| services are NOT engaged?
| notyourday wrote:
| CEO: We are doing project X. We have engaged so and so to
| do it. You are to support this.
|
| <multiple layers down, Jack, head of IT in charge of
| backups>: I'm uncomfortable sharing this information with
| the outside party. I will need to get the approvals for
| <blah blah blah blah>
|
| ...
|
| CEO: Yes, tell them to do this.
|
| ...
|
| Jack, head of IT in charge of backups: Ok, you can have
| this access.
|
| Consultant: Great. <comes back ten minutes later> It says
| I'm not authorized to perform this operation. It is a
| blocker, could this be fixed so I could continue?
|
| Jack: This was not in the scope of authorization that I
| have received. I'm uncomfortable providing this level of
| access without additional authorization.
|
| <repeat>
| beckingz wrote:
| Oh.
| digi59404 wrote:
| This happens, and honestly it happens rarely.
|
| When you're in this role your goal and job is to build
| relationships and help people. That includes Jack. The
| job is getting done with or without Jacks help. So he can
| either jump onboard or he can be removed and someone else
| can be added to give access.
|
| In the end people like Jack benefit nothing from being
| gatekeepers. Because the third time I have to ask for
| permissions to be changed - I'm asking for Jack to be
| removed from the task and someone else to be put in
| place.
| notyourday wrote:
| That's not correct.
|
| When you are in a firefighting role, your job is to fix
| the fire, not appease Jack and build relationship with
| him, the person whose practices created the situation to
| begin with. Unfortunately, even in those cases the
| highest level of stakeholder who engaged your
| firefighting service may not be willing to tell Jack's
| manager or Jack that he will do what he is told or he
| will be removed.
|
| Jack's gatekeeping is what keeps Jack employed. The
| company's ultimate bosses will need to make decision if
| they want to remove Jack and solve a problem or if they
| do not care that much that the problem is quickly solved.
| In the vast majority of cases no one in the company who
| can fire Jack wants to be seen as a bad guy.
| wiz21c wrote:
| Damn that's so true. We hired a network expert to figure
| latency somewhere between our app and the end user. Spent
| a month asking autorisations for each g _d_ mn net boxes
| which were owned by several organizations which were
| working _in the same building_ but not able to cooperate
| because they were not paid by the same institution.
|
| The guy did very creative reporting though :-)
|
| And the problem was not fixed, of course.
|
| I tried to help him as much as I could, but at some
| point, the thing is so intricate that you basically give
| up.
| Aeolun wrote:
| Come work for our company! We have an infinite amount of
| fires to put out.
|
| To be fair, the number of fires is finite, new ones are
| just getting started faster than they are put out.
| sanedigital wrote:
| I believe that implies that the fires are indeed
| infinite, just countably so.
| m463 wrote:
| It would be fun just to be able to talk to management with
| the same sort of candor the "cleaner" in pulp fiction did.
| (harvey keitel)
| papreclip wrote:
| Being an SRE is a little more of this and a little less
| scheduled work. You still have standup and plenty of other
| meetings, though
| mattlondon wrote:
| Constant "emergencies" soon wear you down.
|
| Sure it is fun, but the stakes can be high and the stress
| gets piled on.
|
| It starts with congratulations all-round. You are a hero!
| Good job! Nice save! Your boss's boss's boss's boss comes
| over and thanks you personally. Take the rest of the week
| off!
|
| Later, its people asking when will _you_ fix it? Why haven
| 't _you_ got this back working yet? Didn 'y you fix the
| same thing last month? (no) We really need _you_ to fix
| this before 5pm or the TPS reports wont go out, and the
| management will be pissed.
|
| Failures become normalised. They get reliant on people
| doing heroics. People forget that the systems are crap and
| need investment, and start to rely on _you_ being there to
| fix it, and if things don 't get fixed then it is _your_
| fault the TPS reports didn 't go out, not tech-debt/lack-
| of-investment/bad-design/whatever.
| ido wrote:
| I remember stuff like that, once the COO called the
| entire company to the common room to present me a bottle
| of champagne for my efforts (another time I was given a
| fancy bottle of whiskey that some marketing guys drank
| 3/4 of before I ever got around to it). I rather not work
| any overtime instead.
| blhack wrote:
| This is basically the reason I got out of doing IT and into
| development.
|
| "If you do things well enough, nobody will think you did
| anything at all."
| hinkley wrote:
| The first manager I ever respected was this goofball who worked
| at a fast food restaurant. I don't think the other managers
| took him seriously. He didn't really take himself seriously
| either (told the worst jokes), but when there was a rush he was
| the one you wanted running the kitchen. The other 2-3 people
| were kind of assholes under stress (they were assholes all the
| time if I'm honest). He was a machine, and it didn't hurt that
| he good at predicting orders before they were even made. We
| always made too much food in these situations but more of his
| would sell and the wait times were lower.
|
| In war room after war room there has always been at least one
| person who would be categorized as 'disorganized' by their
| coworkers. When thrown into a stressful situation they either
| don't change or become more focused, while the 'organized'
| people crumple like tissue paper, so Mr Disorganized ends up
| either fixing the problem, keeping everyone on task until it
| does get fixed, or insisted on some process/tool improvement
| that made the problem easier to identify or fix.
| danaliv wrote:
| "Are you Mr. Disorganized who shines in an emergency?" would
| be a not-terrible screening question for ADHD:
|
| https://www.additudemag.com/benefits-of-adhd-crisis/
| oehpr wrote:
| I've heard this is a kind of symptom of people who have ADHD,
| some of them have grace under pressure.
|
| I've been diagnosed with ADHD and I'm not sure I'd qualify as
| having that trait. So I don't believe this a general sweeping
| statement you can make for everyone with that, or if your
| boss even had ADHD.
|
| But maybe check this out and ask if it rings any bells:
| https://www.nhs.uk/conditions/attention-deficit-
| hyperactivit...
| danjac wrote:
| Looking through the comments, it appears concerns were raised
| for years and management refused to allow the necessary changes
| to save money.
|
| While you can and should be the cool head in the room, it's a
| route to burnout if you end up being the hero who has to keep
| saving a dysfunctional organization and management from the
| consequences of its own failures. By all means do your best fix
| the immediate problem but get the hell out as soon as you can.
| jart wrote:
| Reading comments https://www.reddit.com/r/sysadmin/comments/m
| a4mwl/the_absolu... it sounds like he ran into a kernel
| panic, maybe because cosmic radiation messed with their full
| disk encryption because they went cheap on the ecc ram, and
| then he got discouraged, he probably asked management for the
| resources to rewrite the whole system from scratch, they said
| no, and then he sat on his hands collecting paychecks for two
| years until the thing he knew would happen finally happened.
| How else can you explain an MSP not having a copy of their
| zone file? That's their core business. It's also no secret in
| the MSP world how frequently these DNS servers get hacked. I
| read about it all the time. Especially on Reddit where you
| can just sense the pervasive fear. So you'd think that'd give
| a sysadmin a moment's pause to think, OK, have I scp'd my
| zone file in case that happens to me?
| pydry wrote:
| >it's a route to burnout if you end up being the hero who has
| to keep saving a dysfunctional organization
|
| The same dysfunction that led to disaster after disaster is
| likely to lead to very little reward (if any) for actually
| being the hero, also.
| iJohnDoe wrote:
| I'm guessing the IT department reported to the CFO. Bean counters
| have zero IT knowledge but like to think they do. They also by
| default like to say no to any expenditures. Combine the two and
| you have a disaster waiting to happen.
|
| IT reporting to finance/CFO is always the worst decision.
___________________________________________________________________
(page generated 2021-03-22 23:03 UTC)