[HN Gopher] US Weather Service Internet Systems Are Crumbling
___________________________________________________________________
US Weather Service Internet Systems Are Crumbling
Author : julienchastang
Score : 41 points
Date : 2021-04-02 18:34 UTC (4 hours ago)
(HTM) web link (www.washingtonpost.com)
(TXT) w3m dump (www.washingtonpost.com)
| kcmastrpc wrote:
| I've used weather.gov for almost 10 years, and I've observed is
| become more unstable and unusable over the past decade. If it
| weren't for local weather gurus who pay for the tools
| professional meteorologists utilize and broadcast through YT and
| other social media tools I wouldn't know when to shelter or take
| cover from some of the intense weather that has moved through our
| area over the past few years.
| CWuestefeld wrote:
| Why are they so attached to NWS Chat specifically? Perhaps there
| are reasons unexplained, but from this article it sounds very
| much like they've got Not Invented Here syndrome.
|
| Sure, they need bespoke computation services for their modeling
| functions and so forth. But when it comes to the dissemination of
| information, it seems like that's very much a commodity these
| days, and can be provided at such an enormous scale at such low
| prices that it should scarcely be a concern.
|
| I suspect, with no evidence whatsoever, that the NWS is holding
| this hostage for better funding overall, because it's the visible
| piece that causes pain to the public. There are other areas where
| they believe they really need better funding, but haven't been
| able to convince the budgetmakers of that. In order to force
| their hand, they're starving the public-facing stuff.
| Jtsummers wrote:
| NWSChat is apparently an XMPP-based solution and people were
| able to use off-the-shelf clients (did some digging out of
| curiosity). It was also started before things like chat-as-a-
| service offerings (like Slack) were widespread. It's also
| import to remember that they needed to coordinate with both
| government and non-government users, so a solution like MS's
| chat (what later became Skype for Business, not sure what it
| was in the 00s) would be challenging to integrate with non-
| government users even if it was trivially deployed and scaled
| across the government side. On top of that, using a commercial
| service isn't necessarily permissible as government agencies
| have to track their communications and record them with strict
| rules on how long recordings have to be maintained and how/when
| they can be destroyed. Commercial services _can_ be used, but
| they have to have the right policies in place. WFH in 2020
| revealed this issue to many as gov 't employees were using non-
| standard tools without proper recording/tracing that
| technically violated federal laws and regulations but let them
| get their job done.
|
| For better or worse, in the 00s self-hosting a chat service
| might have been the least bad option to achieve all the
| requirements they had. However, it also probably suffered (as I
| mentioned in another comment) from the fact that many gov't
| agencies don't really have proper IT staffing and don't treat
| it as something to continually invest in and maintain/sustain.
| RyanPringnitz wrote:
| Office Communicator -> Lync -> Skype for Business
| Jtsummers wrote:
| Those were the names I'd forgotten, thanks.
| CWuestefeld wrote:
| Your point may well be an important aspect of this. But to
| the extent that's true, we shouldn't be throwing money at a
| system (in the larger sense, not just NWS) that's broken.
| Unless of funding, let's find the badly-designed regulation
| and fix _that_.
|
| But it's more than just NWS Chat:
|
| _In December, because of an escalating bandwidth shortage,
| the Weather Service proposed limiting users to 60 connections
| per minute on a large number of its websites.
|
| Constituents complained about the quota and, earlier this
| month, the Weather Service announced it would instead impose
| a data limit of 120 requests per minute and only on servers
| hosting model data, beginning April 20._
|
| This is, or should be, bread and butter stuff. The rest of
| the world has figured out how to do this cheaply and with
| high availability. And their overseers even recognize this:
|
| _Neil Jacobs, former acting head of the National Oceanic
| Atmospheric Administration, which oversees the Weather
| Service, said many of the agency's Internet infrastructure
| problems are tied to the fact they run on internal hardware
| rather than through cloud service providers such as Amazon
| Web Services, Microsoft and Google Cloud.
|
| "I've demanded in writing that NWS transition these
| applications ... to our Cloud partners. It's part of an
| internal strategy I've laid out," Jacobs, a Trump
| administration appointee, told the Capital Weather Gang in an
| email before he left office._
|
| So why are they not making that architectural change?
| Jtsummers wrote:
| > So why are they not making that architectural change?
|
| Because IT is secondary to their primary mission and they
| don't have a proper dedicated IT team of sufficient size to
| enable this or a smooth transition if they tried. This is a
| repeated problem in the federal government. IT is not their
| mission, and each entity acts like a small-to-medium sized
| business, even if they're part of a larger scale
| organization. The US DOD doesn't even have one email system
| for all services. Each service has its own, and at least
| until the early 2010s many (most? all?) military bases had
| _their_ own email servers with military members (and
| civilians) having to get new email accounts at each new
| base they worked at (this isn 't the case of having a
| distributed but mutually coordinated system, it was just
| that each base literally had their own MS Exchange servers
| running and each maintained them themselves). And that's
| pretty much the single largest entity in the executive. Now
| imagine if it was a struggle for them despite a clear
| mission benefit how much more challenging this is when the
| smaller agencies are asked to do the same without being
| provided sufficient personnel and finances to do it, and no
| leadership (no real leadership) stepping up to create or
| establish a cross-service solution.
| yellowapple wrote:
| I ain't really convinced by Jacobs' take; if they're having
| these issues with on-prem hardware, then they're going to
| have these same issues with cloud services. It's a matter
| of resource allocation, effort, and the organization
| itself, not one of specific implementation details like
| where the servers live. A cloud transition sounds like a
| boondoggle and a distraction from more fundamental issues -
| and when (not if) those issues persist, now we'll be in the
| same boat but with even worse results per tax dollar.
| burnished wrote:
| Why no mention of third parties that kneecap important
| infrastructure to serve their own profit margin?
| johnnyapol wrote:
| Mods - looks like SeattleTimes is syndicating this and lacks the
| aggressive paywall. Here's an alternative link:
|
| https://www.seattletimes.com/nation-world/nation/weather-ser...
| EMM_386 wrote:
| I just tried out the new weather radar, and it was a mess.
| Extremely slow to update new tiles when scrolling around or
| zooming. Trying to "play" the radar had a 15 second delay. The
| browser console is full of errors.
|
| And then, after a couple minutes of moving around the map, I can
| no longer access the site.
|
| Is this the "data limiting" the article mentions??
|
| > Access Denied
|
| > You don't have permission to access "http://radar.weather.gov/"
| on this server.
|
| > Reference #18.95fa3b17.1617393226.1c0a631b
| julienchastang wrote:
| There was a WaPo article about this too [1]. I feel bad for the
| software developers who had to work on this website. They were
| probably understaffed and under-resourced and had to release
| something they knew was not working. Edit: Note that the old
| radar.weather.gov was great and well-liked, but they had to get
| rid of it due to its reliance on Flash.
|
| [1] https://www.washingtonpost.com/weather/2020/12/21/nws-new-
| ra...
| malwarebytess wrote:
| I wonder if it has something to do with the active sabotage in
| leadership over the last 4 years.
| 8bitsrule wrote:
| A couple months back, I was surpised to see that the radar site's
| old Windows98-refugee map was gone ... replaced by a slithering
| mess of SVC-looking intentions gone AI.
|
| But when I checked it a week ago, it was much less shaky. Good to
| see the NWS trying new ideas.
| readflaggedcomm wrote:
| Not much detail on decision making. But bandwidth and overtaxed
| servers are blamed throughout this and the other two articles
| linked near the top, and their bespoke chat and radar programs
| seem to make it worse. When will they distribute observation data
| through some swarmable system like IPFS? Yet even the rogue
| Alabama office recommended siloed Slack rather than something
| open.
| Jtsummers wrote:
| My wager is that NWS is like every other gov't agency. IT is a
| necessary evil, not a core focus. They do _enough_ to keep it
| going but until shit hits the fan it won 't get enough
| attention from their leadership to receive the funding (for
| staff or systems) necessary to actually grow or even properly
| sustain it (that is, actually replace problem components in
| servers or properly update/upgrade systems, or properly convert
| to even _minimal_ degrees of virtualization to allow mobility
| of services).
|
| The ironic/annoying part is that some agencies have figured out
| how to do it correctly, but they never share with other
| agencies. Either out of selfishness (fiefdoms, it's mine and I
| get the accolades for having it, if I shared then it would be
| ours and I'd have to share those accolades, screw you) or lack
| of communication (silos, they don't know what anyone else is
| doing and may not have any good channels of communication to
| find out). This same thing gets played out over and over again
| over the years, and it's not just with the government. It's any
| group that views what's actually their critical infrastructure
| as a cost center to minimize spending on and, ideally, cut; and
| any organization that creates overly siloed groups and forces
| communication up and down a hierarchy rather than direct
| communication between peer groups.
|
| > When will they distribute observation data through some
| swarmable system like IPFS? Yet even the rogue Alabama office
| recommended siloed Slack rather than something open.
|
| I think I'd also be comfortable making a multi-hundred dollar
| wager that they've heard of Slack and never heard of IPFS or
| similar approaches.
| tablespoon wrote:
| > My wager is that NWS is like every other gov't agency. IT
| is a necessary evil, not a core focus. They do enough to keep
| it going but until shit hits the fan it won't get enough
| attention from their leadership to receive the funding (for
| staff or systems) necessary to actually grow or even properly
| sustain it
|
| Not just every other government agency, but every other non-
| tech organization.
| Jtsummers wrote:
| Agreed, see my second paragraph above. Organizations suck
| at maintaining the infrastructure that they don't see as
| their core mission or their profit centers (and even
| then...). IT is one such thing for most businesses and
| governments. They don't appreciate how integral it is until
| it fails, then there's a lot of wailing and gnashing of
| teeth and Congressional hearings.
| edoceo wrote:
| Well, Mattermost could/would solve that chat problem nearly
| instantly and cost nearly nothing.
| Jtsummers wrote:
| Except the cost of deploying, integrating into their
| authentication system, and maintaining. In 10-15 years a self-
| hosted solution by a non-IT-centric organization will probably
| end up in the same situation as their current solution.
|
| The US government needs to get better at distributing IT
| knowledge, capabilities, and services across its numerous
| agencies instead of having them each (and their subcomponents)
| act like their own small businesses with little effort on
| cross-cutting support and capabilities.
| nattaylor wrote:
| After the radar launch I inquired about bringing back the GIF
| loops and they had something within 24 hours.
|
| I use the NWS API's in my personal weather app and they work
| fairly well.
|
| It's not all doom and gloom, but they could be better. The NDBC
| went down for about 7 days last month due to a "catastrophic
| outage," and that was very unfortunate. But, the deliberately and
| with warning took the BOX radar offline for maintenance for the
| first time in decades. A mixed bag.
| yellowapple wrote:
| In the vein of "ask not what your country can do for you, but
| what you can do for your country": how can we help? If the
| government is (evidently) unable or unwilling to operate these
| services, it seems like it is our civil duty as Americans to take
| this upon ourselves and coordinate the creation and operation of
| a system "by the people, of the people, for the people".
| 1996 wrote:
| > "ask not what your country can do for you, but what you can
| do for your country"
|
| No.
|
| You can and should ask as 1) it is at the basis of check and
| balances 2) it is not an connection that can not be severed if
| needed.
|
| If you are not willing to look at the dynamic of the
| relationship, you may help a country commit tyranny: if your
| country is not willing to help you because of your race /
| religion / anything else, maybe you should do nothing in return
| except get the fuck out and grab a bag of popcorn while
| watching the country go sour.
|
| I say that as I love China, but I understand and support my Gay
| friends who have decided to never return.
| yellowapple wrote:
| This misses the point: that the country is what we - as the
| people thereof - make of it.
|
| The American government - like, I suspect, the Chinese
| government - has a vested interest in trying to paint itself
| as "the country" and thus demand said country's people to
| serve that government unconditionally, but that couldn't be
| further from the truth; governments come and go (as they have
| here in what we now call the US, and as they have in China)
| but the the people - themselves being the country - remain.
|
| That is: countries do not commit tyranny. Their governments
| do. And when those governments commit tyranny, or otherwise
| fail to serve the people, it is the right and duty of the
| country - i.e. the people - to bring that government to heel,
| or else establish a new government obedient to the will of
| the people (or, better yet, learn to govern themselves
| instead of insisting on some government to do so). Driving
| _out_ the people of a country in order to appease the wishes
| of some government is arguably the worst-case scenario in
| terms of that country 's identity; a country which allows a
| government to dictate that identity only weakens as a result
| - as we can see quite plainly in both the US and China.
|
| So, when it comes to weather reporting or anything else, if
| the government can't or won't do it, then it's on us - the
| people - to do it, for the sake of _all_ of the rest of us.
| If any community or demographic or what have you is
| marginalized in our country, then it is on _us_ - we being
| the country - to correct that, and to extend a hand in
| friendship and mutual aid. This is how patriotism is
| _supposed_ to work: a commitment to one 's fellow human, not
| to some arbitrary state.
|
| If you do indeed love China (and I'm assuming here that
| you're a resident of it), you might want to reflect on what
| you can do to improve it for _all_ of your fellow countrymen
| - including those who currently feel excluded from it. If
| more people did the same in our respective countries, said
| countries - and given their global influence, the world -
| would be much better places than they are now.
| Mediterraneo10 wrote:
| > it is the ... duty of the country - i.e. the people
|
| No, there is no such duty. If we look at the Soviet
| dissidents, plenty of them neither wanted to support the
| Soviet regime nor challenge it, so instead they retreated
| into their own private world, and _that was a totally
| legitimate choice_.
|
| Also, you suggest that the people naturally want a free
| country ruled by the people. Yet support among Americans
| for the 18th-century Lockean ideals that informed the
| Founding Fathers (radically free speech, division of
| powers, decisionmaking by the popular will instead of some
| recognized experts, etc.) seems to be falling on both sides
| of the political spectrum. In many countries, those ideals
| were never there in the first place.
| yellowapple wrote:
| > plenty of them neither wanted to support the Soviet
| regime nor challenge it, so instead they retreated into
| their own private world, and that was a totally
| legitimate choice.
|
| So two things:
|
| 1. You seem to be conflating the country with the
| government here. The Soviet regime was not the country.
| It was one government asserting control over a country
| (and also some other countries, on that note). That
| country existed before the Soviet regime, and continues
| to exist after it.
|
| 2. "retreat[ing] into their own private world" is itself
| a means of challenging the power of a government. It
| demonstrates one way - indeed, the _simplest_ way - in
| which the people of a country can abolish a government:
| by ignoring it entirely.
|
| > Also, you suggest that the people naturally want a free
| country ruled by the people.
|
| No, I suggest that the people - and the country existing
| as an abstraction of them - are distinct from whatever
| rulership is currently in place. The power of that
| rulership is contingent on its recognition by the people.
| This is pretty inherent to whether a government is
| actually functional; if the people refuse to listen to
| the government, then what power does the government have?
|
| The form of whatever government the people do fashion for
| themselves (if they fashion one for themselves at all) is
| an entirely separate question. Doesn't matter if it's a
| direct democracy or a dictator for life; if it has power,
| it is because the people of a country enable it to have
| power.
| Mediterraneo10 wrote:
| > It demonstrates one way - indeed, the simplest way - in
| which the people of a country can abolish a government:
| by ignoring it entirely.
|
| No, not entirely. The USSR and Eastern European
| dissidents' approach was often to publicly acquiesce to
| the demands of the regime, perform the rituals expected
| of them, while in private revealing to one another their
| true opinions and tastes in art, music, literature, etc.
|
| You stated above that people have a duty to "bring the
| government to heel, or else establish a new government
| obedient to the will of the people", but clearly people
| have often not been willing to take action like that, and
| that is perfectly alright.
| yellowapple wrote:
| Fair. Still, it's those private actions that matter here;
| that perpetuation of ideas against the will of the
| ostensibly-in-power government is, too, a rejection of
| that government's power.
|
| That is:
|
| > You stated above that people have a duty to "bring the
| government to heel, or else establish a new government
| obedient to the will of the people"
|
| Chopping off the parenthetical immediately following that
| quotation changes the meaning from what I wrote entirely.
| As noted in that parenthetical, a perfectly valid way of
| fulfilling that duty is to promote and exercise self-
| governance - which is exactly what those dissidents did
| in private.
|
| And further, that promotion and exercise of self-
| governance is itself a way to bring an errant government
| to heel - namely, by introducing an alternative to it and
| prompting it to seek a more "diplomatic" outcome.
| drewda wrote:
| The private sector has been trying for years to reduce the
| National Weather Service's resources. Perhaps that's not the
| entire story of why NWS has IT constraints right now. But it's
| still pretty frustrating how Accuweather keeps trying to limit
| what NWS can distribute for free, because they would rather
| sell it to the American public.
|
| Here's an oldie but a goodie:
| https://en.wikipedia.org/wiki/National_Weather_Service_Dutie...
| throw0101a wrote:
| John Oliver covered the topic in 2019:
|
| * https://www.youtube.com/watch?v=qMGn9T37eR8
|
| * https://time.com/5699545/john-oliver-weather-last-week-
| tonig...
|
| * https://www.theguardian.com/culture/2019/oct/14/john-
| oliver-...
| kevin_thibedeau wrote:
| Accuweather: "I love it when a plan comes together"
| rckoepke wrote:
| I really miss the old weatherspark. It was extremely performant
| and also had by far the best/most useful UI visualizations of any
| weather app to date.
|
| http://radar.oreilly.com/2011/09/weatherspark-weather-data-v...
|
| https://kk.org/cooltools/weatherspark/
|
| As I understand it, they didn't have the money to migrate from
| Flash to HTML5 and the current site is a shadow of it's former
| self.
___________________________________________________________________
(page generated 2021-04-02 23:01 UTC)