[HN Gopher] Tell HN: Canada Posts systems to be down for 6 hours...
___________________________________________________________________
Tell HN: Canada Posts systems to be down for 6 hours of "unplanned
maintenance"
Author : neom
Score : 66 points
Date : 2021-12-29 16:37 UTC (6 hours ago)
(HTM) web link (status.canadapost.ca)
(TXT) w3m dump (status.canadapost.ca)
| abrowne wrote:
| Last year, someone signed up for PO box-equivalent with my
| firstnamelastname@gmail.com address. (Not the first time this guy
| in Montreal forgot that his email has a 0 before the @gmail.com.
| Or an error from the person entering it in.) I live in the US. I
| have lived in Canada, but over 10 years ago now.
|
| The only two options to contact Canada Post were phone or ...
| Twitter -- which I don't have an account for -- no email or web
| form. And the phone CSR had to get a manager who still couldn't
| figure out how to deal with the issue since I wasn't a customer
| myself and just wanted my email removed from the account.
| 908B64B197 wrote:
| 6 hours?
|
| They know 80's mainframe could do updates without any downtime
| right? Did someone mess up the Kubernetes config and shut it all
| down at the same time?
|
| The Canadian government is an example of how not to do software.
| Their Phoenix pay system is frequently used as a cautionnary tale
| for bad design and implementation [0].
|
| [0] https://en.wikipedia.org/wiki/Phoenix_pay_system
| triceratops wrote:
| > Instead of saving $70 million a year as planned, the report
| said that the cost to taxpayers to fix Phoenix's problems could
| reach a total of $2.2 billion by 2023.
|
| Ouch.
|
| And for those who think software engineering other than medical
| devices, aviation, or nuclear power plants isn't life-or-death:
|
| > Pascale Boulay, a Quebec coroner, determined that the 2017
| death by suicide of a 52-year old woman from Val-des-Monts,
| Quebec, was preventable. The coroner assigned blame on the
| "flawed Phoenix pay system" that had "led her to emotional and
| financial ruin."
| icegreentea2 wrote:
| Phoenix was rotten from concept to post roll-out fuckery. It
| was broken before a single developer touched a line a code.
|
| Red flags independent of software development finesse or
| skill:
|
| - Centralized payroll operations to effectively the middle of
| no where (effectively ensuring that no existing payroll
| specialists would staff up the new center). Locate operations
| in the middle of no where to compensate for another
| government program and center being shut-down.
|
| - Sole source contract. To IBM no less. To customize and
| configure PeopleSoft. There's literally nothing that sends
| good vibes in that setup.
|
| - The very concept of running a big bang switch over of like
| 75% of the federal government to a new payroll system.
|
| - Launched without end to end, full system test
|
| - Launched KNOWING that critical functionality (like
| retroactive pay) was not implemented, and that critical
| requirements were failing tests.
|
| - Deciding to roll out despite their new centralized payroll
| center was already less efficient at handling payroll on
| legacy systems and that their central assumptions were
| already being challenged
|
| - Deciding to roll out despite receiving reports from client
| departments that the initial data transfer into the new
| system was already significantly incorrect
|
| - Quote: "In our opinion, they had received more than enough
| information and warning that Phoenix was not ready to be
| implemented, and therefore, they should not have proceeded as
| planned. Phoenix executives prioritized meeting schedule and
| cost over other critical elements, such as functionality and
| security, resulting in an incomprehensible failure of project
| management and oversight"
|
| You can read the audit here: https://www.oag-
| bvg.gc.ca/internet/english/parl_oag_201805_0...
|
| Software is only tangentially related in that the "magic" of
| software lets people believe that such projects are even
| possible.
| 908B64B197 wrote:
| > And for those who think software engineering other than
| medical devices, aviation, or nuclear power plants isn't
| life-or-death
|
| I don't think a single real engineer was involved in this,
| sadly. Probably offshored contractors and clueless non-
| technical government employees.
|
| It's ironic because Quebec punch way above it's weight for
| engineering.
| ayngg wrote:
| It wasn't necessarily just IBM's fault either, it was
| complete incompetence by the government who continuously
| pushed the change through in the face of ample evidence
| (and IBM's own suggestion) that it should have been
| delayed.
|
| Not only did they push it through despite issues even after
| initial limited rollout, they made a ton of terrible
| decisions like to do training in house instead of from IBM
| as cost savings, made the new Public Service Pay Centre
| elsewhere for political reasons with new staff and they
| laid off most of the people that worked on the old system
| and decommissioned it so there was no going back.
| 908B64B197 wrote:
| > It wasn't necessarily just IBM's fault either
|
| My understanding is that it wasn't IBM's fault at all.
| They delivered exactly what was aggreed upon. What the
| government asked made no sense.
|
| That's why they couldn't and didn't sue, and if you read
| about it in the media, there's no direct blame on IBM (a
| lot of it is implied; being to direct could be construed
| as defamation).
| [deleted]
| bellyfullofbac wrote:
| Also happened in the UK:
| https://www.theverge.com/2021/4/23/22399721/uk-post-
| office-s...
| MattGaiser wrote:
| Thanks IBM.
| midasuni wrote:
| To err requires a computer. To really foul things up requires
| ansible.
| ranit wrote:
| > They know 80's mainframe could do updates without any
| downtime right? Did someone mess up the Kubernetes config and
| shut it all down at the same time?
|
| > The Canadian government is an example of how not to do
| software. Their Phoenix pay system is frequently used as a
| cautionnary tale for bad design and implementation [0].
|
| Do you know what happened at CanadaPost systems today or you
| are just "speculating"? Does CanadaPost still have mainframes
| in their IT? I mean: do you know it for a fact or just trying
| to make a fun comment? Do you know of any implicit or explicit
| connection between Canadapost IT procurements and infamous
| Phoenix software every Canadian taxpayer is outraged about?
|
| Just curious to find out whether people comment based on clear
| knowledge or just common (non)sense. Sorry if I violated any HN
| rules.
|
| Disclaimer: No affiliation or interest with any of the affected
| parties.
| [deleted]
| belltaco wrote:
| Should be 'Tell HN', not 'Show HN'.
| gnicholas wrote:
| Unless it was posted by the person who took the systems down.
| /s
| perlgeek wrote:
| What are the odds this will turn out to be due to ransomware?
| admax88qqq wrote:
| That or they just realized they're using log4j
| tyingq wrote:
| Their cookies start with "JSESSIONID", so I'm guessing you're
| right.
| perlgeek wrote:
| Would be a pretty fun exploit if you printed it on the
| address field of a letter, OCR recognizes it, cannot parse it
| as a postal address, logs the failure using log4j, thus
| giving you access to their post routing system...
| inkeddeveloper wrote:
| Yes....fun...
| rbobby wrote:
| Sounds like a plan... until you realize why your label
| printer keeps restarting.
| kache_ wrote:
| Canada suffers from losing all their competent tech workers to
| jobs that actually pay them market rate :^)
| Naga wrote:
| It's actually even worse for government (or crown corps like
| Canada Post), since they pay way below Canadian market rate,
| which is already way below U.S. salaries. The government then
| isn't really in competition for the best talent who want the
| most money.
| poxrud wrote:
| Like most government dev work, Canada Post outsources to
| Accenture for their projects.
| dzhiurgis wrote:
| This. Exactly everywhere, not government only. Why pay
| market rates to actual developers when you can get bottom
| of the barrel scum for $2000 a day
| baseballMan wrote:
| All about that capex vs opex!
| markdown wrote:
| You mean the market rate in another market/country that they
| can get into. Certainly not the Canadian market rate.
|
| That's the price of free trade and open borders. Most countries
| lose their best and brightest to the US and other high-pay
| countries, and the world is poorer for it.
| 908B64B197 wrote:
| > Most countries lose their best and brightest to the US and
| other high-pay countries, and the world is poorer for it
|
| Why not simply compete?
| neom wrote:
| At least in Canada it would be consider pretty uncouth for
| a civic worker to make what they would have to make to
| compete. Personally I don't agree with this, however, if my
| farther learned a software engineer at the post office was
| making $200k his head would explode, he already wants to
| defund the Canadian Broadcasting Corporation, also a crown
| corp. My understanding is that our national defence and spy
| agency pay quite well by Canadian standards, but that's
| about it for government jobs. Our crown corps also happened
| to be big fans of using students: https://www.lcbonext.com/
| drpgq wrote:
| My sister works at CSEC and the pay isn't that great. The
| working conditions are pretty good from what I can tell.
| baseballMan wrote:
| CSIS still seems low by most standards:
| https://www.canada.ca/en/security-intelligence-
| service/corpo...
|
| You can get "level 6" salaries as a new grad at a
| canadian bank.
| [deleted]
| neom wrote:
| I just got off the phone with Canada Post because my package had
| not shown up on the Canada Post tracking system when the package
| was handed off from USPS to them yesterday. Guy on the phone said
| they've been having "issues" and he can't do any service tickets
| or status lookups because their internal systems are all offline
| for the next 6 hours due to "unplanned maintenance"
___________________________________________________________________
(page generated 2021-12-29 23:01 UTC)