[HN Gopher] Mystery Blips
___________________________________________________________________
Mystery Blips
Author : bo0tzz
Score : 164 points
Date : 2022-11-19 10:28 UTC (1 days ago)
(HTM) web link (mosquitocapital.substack.com)
(TXT) w3m dump (mosquitocapital.substack.com)
| [deleted]
| IAmGraydon wrote:
| It's funny - the very first thing I thought while reading this
| was "check the news". That's probably because I've been involved
| in markets and when you see this sort of behavior unfold in
| market price and volume, the first thing you do is check the
| news.
| MrAwesome wrote:
| Yeah! By the end of my time on the team, checking the news when
| I saw a rapid change in traffic was second nature. And later on
| in my career becoming interested in markets, even more so.
| Forex in particular - markets with trillions in daily liquidity
| don't take huge leaps unless something huge happened.
| NKosmatos wrote:
| Liked the ending of it, nice read :-) Being a data/statistics
| freak, I would really love to see statistics and usage patterns
| from big sites or games.
| xyzelement wrote:
| One of my clients back at a large financial software and data
| provider, was a southern market making firm.
|
| They kept complaining about "the whole desk being slow" and we
| spent weeks trying to figure out what was wrong with our
| software.
|
| Eventually we figured out the whole firm was nuts about golf, and
| they'd all stream golf tournament to their workstations at the
| same time and saturate their ISDN or whatever they had.
| steveBK123 wrote:
| 10+ years ago the networking team at a big European bank I
| worked for, observed that the biggest usage of network
| bandwidth on the trading floor desktop network was not the
| Bloomberg Terminal, market data, or trading application
| traffic.. but Youtube :-)
| bee_rider wrote:
| Wouldn't surprise me if YouTube, Spotify, etc were some of
| the biggest resources users in many offices.
|
| Maybe it would be useful to have a local company radio
| streaming service, haha.
| js2 wrote:
| Early in my career, in the late 90s, I was at Cox Interactive
| Media on the team responsible for the web farm which hosted all
| of Cox Enterprises news sites: newspapers, radio stations, TV
| stations. This was before the days of SREs and SROs and dev-ops.
| We were just system admins and programmers and some of us could
| do both.
|
| The web farm was about two-dozen Sun Ultra 2s connected on a FDDI
| loop (we eventually upgraded to GigE), with content on NetApps
| (4GB drives). A couple Sun E450s. Apache[1] on the Ultra 2s.
| Apache + mod_perl on the E450s. Hosted at a Global Center DC in
| Sunnyvale (same DC that early Yahoo was hosted in).
|
| Monitoring with MRTG[2].
|
| It's 1998. The Ken Starr report drops. Now, we knew it was
| coming, and we did our best to be prepared, but this is the late
| 90s. There was only so much load testing we could do and didn't
| really know how much traffic it would drive. We kept the site up,
| but it meant, as I recall, a lot of fine tuning of the mod_perl
| box and disabling interactive parts[3] of the site.
|
| I really wish I still had some of the traffic graphs.
|
| Fun times.
|
| [1] Receipt:
| https://github.com/apache/httpd/blob/1.3.x/src/CHANGES#L5754
|
| [2] Receipt:
| https://github.com/oetiker/mrtg/blob/master/src/CHANGES#L310...
|
| [3] The forums. OMG forum software was so terrible. I think we
| eventually wrote our own after nothing based on Netscape
| Application Server or that we found open source worked well at
| all.
| daveevad wrote:
| > same DC that early Yahoo was hosted in
|
| Space Park 1?
|
| Famously, Yahoo wasn't quite up to the task on 9/11 which led
| to considerable thought about planning for the unexpected 100x
| event.
| js2 wrote:
| That doesn't ring any bells. It's been years. I think it was
| on Borregas Ave and it may have been owned by company called
| ISI at the time. Equinix has a DC in the approximate area
| now, not sure if that's the same building and the address
| changed or my memory is off.
| gooseyard wrote:
| at the end 1999, my employer hosted what I guess was half NYE
| Party and half Incident Response. We were optimistic that we
| wouldn't have issues but figured it'd be wise to have a
| sufficient crew of designated drivers, so to speak.
|
| Our monitoring system included a mercator projection on some big
| screens, with colored that showed where our gear was and whose
| color indicated whether it was down, being hammered, etc.
|
| We had no equipment in UTC+14, so there was an hour of waiting
| between that zone reaching y2k and when we'd maybe see some
| action. The control room was mobbed since we also had some
| televisions running cable news there and everyone wanted to see
| it.
|
| Despite the anxiety, the gear was looking fine everywhere, until
| about 5 minutes before midnight in UTC+13, when our locations in
| New Zealand started to turn red. A hush fell over the room as the
| operators attempted to open connections to those boxes. The
| connection attempts appeared to hang, and the party got somber.
| But after the eternity of a few seconds, we were in. The machines
| were alive but being hammered; the alerts just indicated the
| excessive traffic caused by mass numbers of users refreshing NZ
| based websites as the date change approached, to see whether
| they'd fallen off the net or not. Breathing resumed.
|
| After 15 minutes or so had passed, the traffic fell off, and by
| the time y2k reached us in UTC+5 we'd been in full festive mode
| for hours. I still can't look at New Zealand on a map without
| seeing it covered in angry red circles, though.
| 2b3a51 wrote:
| Big street party here in UK on Dec 31 1999. Myself and a few
| neighbours who thought about these things relaxed a bit after
| 9pm (Midnight Moscow time) and started partying seriously.
| retrocryptid wrote:
| Lol. During "high velocity events" at Amazon, we would have a war
| room with all the engineers and a couple old hands directing
| responses.
|
| Some of our metrics came in 5 minutes delayed which wasn't a
| problem for normal days. These metrics moved slow enough that
| when you got an alarm, there was still plenty of time to take
| corrective action.
|
| But for HVEs this was an issue. During black Friday or prime day,
| sometime some metrics spiked so fast you had no time to respond
| (usually from people hitting page reload a few minutes before a
| sale kicked off.)
|
| To get an idea for what was going on, I would go in twitter and
| search for things like "amazon failure" or "amazon 502."
|
| We often got problem reports via Twitter before they showed up on
| our dashboards.
| MrAwesome wrote:
| In my time in SRO at FB, we actually had an alarm that would
| fire whenever enough #facebookisdown posts were made on
| Twitter!
| xipho wrote:
| Watch the World Cup be the death of Twitter for this type of
| thing?!
| tzs wrote:
| At a place I worked that sold a retail product online we had a
| weird rate problem that we never were able to figure out.
|
| If you graphed the visit rate to our shopping cart page it would
| follow a curve that was sort of like a sine wave (well, a sine
| wave plus a constant because you can't have negative cart
| traffic) with a 24 hour period. I'm just going to call it a sine
| wave.
|
| The amplitude varied seasonally, and was different on weekends
| and weekdays, but generally any given day was a sine wave.
|
| If you graphed order rate it also followed a sine curve too, in
| phase with the visit curve and generally with a fixed ratio
| between them. So if the cart was getting N visits per minute,
| we'd be getting fN order per minute for some number 0 < f < 1,
| and f was pretty much constant.
|
| This continued as the business grew, so the overall amplitude of
| the visit curve increased. But at some point the order rate
| seemed to hit a wall.
|
| What we would then see is that as the visit rate was heading
| toward its peak for the day when it got to a particular fraction
| of that peak, say 80% of the peak, the order rate would stop
| tracking the visit rate and would become flat. It would stay flat
| as the visits peaked and declined until the visit rate was back
| down to 80% of its peak and then orders would resume tracking
| visits.
|
| So far nothing too weird about this. What was weird is that when
| we'd have a day of especially high cart traffic such as when
| something in the news caused a big increase in demand for a few
| days for the kinds of things we sold, and we'd get 3 or 4 (or
| sometimes even 20 or 30) times the normal cart visit rate
| throughout the day--the plateau on the order rate would still
| start when the visit rate hit 80% of the visit rate peak for that
| day and go away when the visit rate fell back below 80% of peak.
|
| The order processing was handled by a different server than the
| shopping cart. They did use the same database for product and
| customer information, but we could not find any signs that the
| database was a bottleneck.
|
| This just made no sense. Suppose we'd been seeing on normal days
| that visits would peak at 1000 per minute, and orders were 10% of
| visits. Then we'd see visits grow from 0 to 800 and orders grow
| from 0 to 80. Then orders plateaued at 80 while visits continued
| on to 1000 and back to 800. Then as orders fell from 800 to 0,
| orders fell in step from 80 to 0.
|
| Now suppose the next day we had 10 times the activity due to some
| news item. So the next day visits rise from 0 to 10000. The order
| system would now not plateau until the visit rate got to 8000 and
| the order rate got to 800.
|
| And then when the increased demand was over in a few days and we
| were back to 1000 visits peak the order system would again
| plateau at 80.
|
| So whatever was causing the plateau was somehow tied to the visit
| rate, making the plateau depend on what the peak visits would be
| for that day.
|
| But how on a given day when the visits would first rise to 800
| and the orders to 80...how did it "know" if the day was going to
| be a 1000 visit peak day or a 10000 visit day so it could "know"
| if it was time to plateau?
|
| We never figured this out. Reasoning about our systems and their
| dependencies I could not figure out any way in theory that this
| could be happening nor could I find anything no matter how much
| logging and tracking I added all over the place.
|
| Everything I could think of that could even remotely cause a
| plateau would have caused a fixed plateau, not some weird plateau
| that's level gets set daily based on what the _future_ peak visit
| rate is going to be for that day.
| doubleunplussed wrote:
| If it was really close to the shape on an actual sine wave,
| then the _derivative_ at any earlier time encodes information
| about the future peak. Any way some system could be
| bottlenecked by the rate of change of the visit rate (i.e. the
| visit acceleration)?
| tonetheman wrote:
| I did SRE for years.
|
| A lot of time it would end up being DNS or routing somewhere
| outside of our control.
|
| Sometimes a disk would get close to filling up and a cron would
| clear it just in time so you would see decreasing performance
| then it would clear.
|
| Other times we would have a customer that did something that we
| just did not expect with the system and it would cause SQL
| queries to slow down in ways we could not imagine. And it was
| transient so those would be hard to find or explain.
|
| Or we would hit a limit in the load balancer (haproxy) in really
| odd ways. Too much traffic on the frontend or not enough capacity
| in the backend. And many other various ways of things not
| working. Haproxy was and still is amazing software. Really almost
| magical.
| sbierwagen wrote:
| >A lot of time it would end up being DNS
|
| Ah, good old "it's always DNS"!
| eb0la wrote:
| I was working in a Telco that owns a big backbone network in
| south America.
|
| Our director was very sensitive to traffic changes. Every week or
| two there was a big meeting at her office to explain what
| happened when some traffic went down.
|
| That day she was really mad at us. There afternoon before there
| was 20% less traffic in the backbone and nobody knew where it
| went.
|
| I was on call that week for the management systems and I was
| questioned about why the operators didn't have received any
| alarm.
|
| Turns out Spain was playing soccer against some Latin American
| team. Spain was our biggest customer. I don't remember if the
| other team was Argentina, Chile, or Brazil... but it was our 2nd
| biggest market.
|
| People just decided to watch TV instead of web browsing (that was
| before mobile phones had an affordable internet connection).
|
| Funny enough eMule traffic spiked during the match. That was the
| reason I could justify there was no alert in the system for down
| interfaces.
| userbinator wrote:
| _Funny enough eMule traffic spiked during the match._
|
| If your bandwidth is limited enough such that background
| downloading would slow down your browsing significantly, then
| it would make perfect sense to let the downloads saturate the
| connection while you watched TV.
| adw wrote:
| This happens in all infrastructure, particularly power and water;
| https://en.wikipedia.org/wiki/TV_pickup
| m3047 wrote:
| I've got a good one. I was working a "tip of the spear" job in
| threat intelligence (I suppose I was always on call). One of the
| things we monitored was DNS.
|
| It was always noisy. If it's not the miscreants, its the normal
| stupidity (DNS is so amazingly survivable of misconfigurations,
| some of them become folk wisdom; like disabling TCP/53, but I
| digress).
|
| Othodox Easter Sunday morning. Really really early in the
| morning, like I can't remember if it was light out yet (US
| Pacific coast time). The SOC calls me. Nothing is happening.
|
| Nothing is happening. There is no noise. So they've been looking
| at this, and everything is working. So they called me. At dark-
| thirty on Sunday morning.
|
| I confirm everything appears to be working correctly. I repeat
| diagnostics, do ad hoc stress tests, everything. About two hours
| later the normal crap starts coming back.
|
| We have to conclude that the miscreants, as well as the
| sysadmins, had taken the morning off.
| ricardobeat wrote:
| This footnote is almost more interesting than the post itself:
| > There's a deep beauty to the thought that untold millions
| of people using the app randomly, but oh so slightly
| habitually, aggregated together, makes such a predictable pattern
|
| That 'beauty' can be extremely scary from another angle. It's
| evidence of what makes Facebook, and other social media, such
| powerful mass manipulation tools.
| euroderf wrote:
| You just described Hari Seldon's psychohistory.
| retrocryptid wrote:
| Yes. But just about every big service has marked diurnal and
| weekly patterns.
| 13of40 wrote:
| One service I used to work on processed business-to-customer
| and business-to-business emails almost exclusively, so you
| could see a recurring weekly pattern, bumps throughout the
| day when the US east coast, US west coast, Asia, and Europe
| woke up, and spikes at the top of the hour from automation.
| So one day we got a new boss, and he called me into his
| office in a panic so I could explain why his chart of the
| traffic kept going up and down. Took about four tries, but I
| think he eventually got it.
|
| I also recall seeing someone push a global update to that
| system that was packaged wrong, and watching the graph
| gradually drop and flatline as 20,000 VM hosts across the
| planet stopped taking traffic. That had its own subtle
| beauty, in no way diminished by the fact I was just a
| bystander and couldn't get in trouble for it.
| none_to_remain wrote:
| I don't see the fright. So many things do this - electricity
| for example - you turn the lights on and off and run the
| laundry machine whenever it pleases you, but on the scale of
| millions of people the power companies predict the daily usage
| patterns very well.
| ricardobeat wrote:
| Imagine if the power company could see everything you do at
| home, and control your lights, laundry machine and other
| electronics to influence your behavior. They don't have that
| kind of power, but social networks do.
| encoderer wrote:
| Maybe it's just the trained operator in me but the whole time
| he's describing the depressed metrics and blip I'm screaming in
| my head: it's exogenous! Check the news! He finally gets there.
|
| For us (at normal company scale) I would actually go look at
| recent activity, new customers, new workloads we are running, but
| I guess at a Facebook scale it's a lot harder to do that.
| nerdponx wrote:
| One interesting aspect of the story is that the author was a
| junior at the time (almost brand new to the job, if I read it
| correctly), and it was one of the more experienced operators
| who realized it was exogenous.
|
| in addition to being a good story about the site reliability,
| it's also a great lesson in the value of collaboration,
| mentorship, and having senior people around whom you can ask
| for help!
| MrAwesome wrote:
| Author here, and that's exactly right! I think it was in my
| second or third month on the job.
|
| And I really agree with your second point too! Without that
| in-person transfer of tribal knowledge, it would have taken
| me ages to learn the ropes well enough to take such a high-
| pressure high-stakes oncall.
| GauntletWizard wrote:
| Google didn't have SRO. The closest thing was Traffic Team's
| oncall; Which was stressful, for sure, and hired some dedicated
| alcoholics, but it wasn't anything like Facebooks (I knew people
| on both teams during my time at each). Google had a constant
| stream of alerts that someone was aware of and handling. Facebook
| had a constant stream of fire that someone was beating back the
| flames of.
|
| SRO is madness. SRO is admitting that you're enough underwater
| and there's enough unexpected critical events to dedicate a team
| to burn out on it. There was a lot of spend in overcapacity and
| failover automation testing that kept it that way, and explicit
| mandates by a C-Suite and VP to keep keeping it that way. More
| importantly, though, it was baked into the architecture.
|
| Everyone's talking this week about how Twitter will go up in
| flames in the next few days; I don't think so. I think they're
| closer to Google in that respect. There's a lot of automation
| that will keep things alive and a lot of redundancy that will
| handle the immediate problems of having a lot of the team leave.
| I actually think the company can continue to run indefinitely if
| they just follow the operations manuals and replace failed drives
| and machines. But I think there will be a major outage when they
| try to launch a new feature and fail.
| MrAwesome wrote:
| Author here - a few months after the time of this story, we
| reoriented around a single purpose for the team, "Automate
| ourselves out of existence". It took a few months to remove all
| of the dependencies, tag all the alarms, train up all the team-
| specific oncalls, but we managed to do it, and I had the great
| honor of taking the very last SRO oncall on March 31, 2014 :)
| prox wrote:
| I love these kind of stories! Any more HN'ers have these?
| Kiro wrote:
| The employee shaming on here has scared away all the people
| with interesting war stories from big companies. Anyone active
| on HN nowadays probably hasn't worked on anything significant.
| gfv wrote:
| Pirate video releases generate heaps of traffic, and they won't
| be on the news. Earthquakes (mild ones, not the "cities
| collapse" kind) cause people to immediately go online and check
| on their friends. So do missile strikes, but those tend to
| generate extremely popular videos as well. During the holy
| month of Ramadan, you can see a rapid and deep traffic drop in
| Muslim countries right at their local sunset, when people have
| iftar, breaking their daily fast. A country in North Africa
| shuts down their internet access completely during their school
| exams. Popular web infrastructure sometimes reroutes your
| requests to distant data centers, leading to request latency
| exploding together with your queue lengths. In summer, the
| morning user activity peaks later than in autumn because
| schools are on their summer breaks.
|
| I bet every seasoned SRE has a few to add.
| prox wrote:
| So cool to see these things happen!
| jasonwatkinspdx wrote:
| Here's a fun one I remember seeing a little video on: the
| UK power authority has to anticipate commercial breaks in
| major broadcasts like the World Cup because everyone
| turning on their electric kettle spikes the grid.
| retrocryptid wrote:
| Another part of my career, I worked on a team that ran an
| automated content detection service (like YouTube ContentId.)
|
| The database holding signatures for known music samples was
| sharded by artist. Not as crazy as you might think. You get a
| trial sample and you send it to all the shards which chunk on it
| in parallel then you just wait for the servers hosting each shard
| to respond.
|
| But then Prince died.
|
| The queue for the server that owned the shard Prince was in
| backed up.
|
| Then our proxy that distributed trials to each shard backed up.
|
| Then our regular reverse proxy backed up.
|
| Then our anemic load balancer fell over.
|
| What we learned for about the bazillioneth time is keep a backup
| of each shard handy so you can add it into the rotation and pay
| an intern to watch Facebook looking for news about recently
| deceased musicians.
| llimos wrote:
| Once when I was working in a big bank in London during World Cup
| 2010, they told us all to stream the England games on BBC Online
| since it would be a good test of their network capacity.
|
| They knew not much work was going to get done either way.
| steveBK123 wrote:
| Having been at a finance firm trying to implement SRE without
| SRO, and without staffing either, I do find the story a bit
| entertaining.
|
| That is - we were mandated to implement all the SRE tooling on
| top of our apps, with like 1 guy owning the SRE infra, but with
| no one looking at the charts proactively.
|
| The idea was sold to dev teams that it was something like a black
| box recorder for the on-call rotation to check first when you got
| called in the middle of the night/weekend.
|
| In reality it turned into CTO reporting metrics of whether SLOs
| were being met, lol.
|
| Constantly feel like other industries read like 1/4 of a book
| about how FAANG does stuff and then adopts the laziest worst
| implementation of the part they skimmed.
| nerdponx wrote:
| > Constantly feel like other industries read like 1/4 of a book
| about how FAANG does stuff and then adopts the laziest worst
| implementation of the part they skimmed.
|
| I think this is less wrong than you think it is. In the P&C
| insurance industry, I saw a handful of initiatives that seemed
| like they were started because a senior manager read about
| something that sounded cool and high-tech in an industry
| publication, and/or heard it in a sales pitch from a Microsoft
| rep, without actually checking to see if it was feasible or
| even useful.
| jeroenhd wrote:
| This reminds me of this YouTube video:
| https://youtu.be/slDAvewWfrA. There's a monitoring room to make
| sure people are ready to switch backups and reroute in case of
| some kind of grid failure, but also in very specific scenarios.
|
| Britain being filled with Brits, what would happen is that once
| the show was over, half the nation got up from the couch to turn
| on the kettle for a cup of tea. Those electric kettles are quite
| demanding, especially if millions of them turn on at the same
| time.
|
| So every time there was a major event, dedicated people are
| monitoring the grid frequency, power plants on standby and
| foreign contracts at the ready when needed, just to increase
| capacity at the right time. You can't just schedule this stuff,
| because if a football match runs over its allotted time, you may
| suddenly add power to the grid without any load, causing all
| kinds of problems like the grid frequency increasing and making
| digital clocks run ahead. There is a temporary demand of
| gigawatts of power that rises within five minutes and lasts until
| the kettles are done.
|
| The YouTube video provides an example of 600MW of power being
| requested from France... for the end of an episode of Eastenders.
|
| I always knew the British like their tea and that there's some
| kind of planning going on for events like street lights turning
| on, but the combination of the two is a great example of complex
| behaviour that's easy to overlook.
| LeoPanthera wrote:
| > Britain being filled with Brits, what would happen is that
| once the show was over, half the nation got up from the couch
| to turn on the kettle for a cup of tea. Those electric kettles
| are quite demanding, especially if millions of them turn on at
| the same time.
|
| Particularly _British_ kettles, which pull 13 amps at 240v =
| over 3000 watts. A lot more than (standard) American outlets
| can deliver.
|
| For a while there was a theory going around that this is why
| Americans typically do not have electric kettles, because they
| would boil much slower in the USA, but the real answer is that
| Americans just don't drink much tea.
| dghughes wrote:
| It amazes me how most of the world is on 240V I'm in Canada
| and we are like the USA 120V 20A circuits.
|
| Today I made tea and I used a standalone electric kettle
| which is odd to most of us here in Canada and the US. Most
| times I would have used the microwave to boil the water. But
| I used the kettle 120V x 12.5A or 1500W and popped the
| breaker since the dishwasher was operating and on the same
| circuit.
|
| In Canada we drink a lot of tea; Red Rose, King Cole and
| especially south-eastern Canada The Maritimes and Atlantic
| Canada.
| LeoPanthera wrote:
| Most US homes actually do receive 240v, it's just done in
| an unusual biphase configuration with two "hot" phases 180
| degrees apart. If you connect a device across the phases it
| will receive the full 240 volts, which is often done for
| cookers, clothes dryers, and EV chargers.
|
| In theory you could sell a 240 volt electric kettle but
| your kitchen would have to have a 240 volt outlet, probably
| a NEMA 6-15. They're not common.
| nraynaud wrote:
| From watching a bit of YouTube, I think Canada's kitchens
| have more 220 outlets than American kitchen. They even
| seemed to have dual outlets with each outlet on a
| different phase, so all the ingredients are right next to
| each other to get an outlet un between phases.
| bbarnett wrote:
| I have never seen a 240V wall outlet, in a house,
| anywhere in Canada ever. They literally do not exists as
| a standard.
|
| Not sure why you'd think so, but no.
|
| (ovens and dryers are not wall plugs)
| [deleted]
| ericvsmith wrote:
| > Most US homes actually do receive 240v, it's just done
| in an unusual biphase configuration with two "hot" phases
| 180 degrees apart. If you connect a device across the
| phases it will receive the full 240 volts, which is often
| done for cookers, clothes dryers, and EV chargers.
|
| It's technically a single phase with two hot legs and a
| center tap for neutral, also called "split phase". From
| the neutral to either leg is 120V, from leg to leg is
| 240V.
|
| https://en.wikipedia.org/wiki/Split-phase_electric_power
|
| It would technically be possible to have a reverse-Y
| extension cord that plugged into two outlets and gave you
| a 240V circuit. The two outlets would have to be on
| different legs, of course. This is basically what a 240V
| breaker is doing. I've contemplating wiring up one of
| these for home and buying a British tea kettle, but the
| aesthetics got my plan vetoed.
|
| Something like: https://www.nooutage.com/images/adaptor-
| wye.jpg
| esaym wrote:
| > For a while there was a theory going around that this is
| why Americans typically do not have electric kettles, because
| they would boil much slower in the USA, but the real answer
| is that Americans just don't drink much tea.
|
| I would drink tea and use an electric kettle if it would boil
| faster...
| argiopetech wrote:
| If fast hot water is all that's holding you back, and hot
| water on standby is an acceptable alternative to freshly
| boiled, I can highly recommend Zojirushi's line of
| products. Mine [1] keeps water at my desired temperature in
| a vacuum insulated canister that uses barely any continuous
| power. Any time I want coffee/tea/etc., hot water is
| instantly available. Recovery time after a refill is under
| 10 minutes on a USA-standard 15A wall socket.
|
| Zojirushi has been in business since 1918 and (in my
| understanding) is a household name in Japan. Their more
| expensive (to be blunt) lines are made entirely Japan and
| replacement parts (like rubber seals, which can wear out
| over time) are available for purchase through their site. I
| fully expect mine to last 15-20 years of continuous duty.
|
| Not affiliated, just a thrilled customer.
|
| [1] https://www.zojirushi.com/app/product/cvjac
|
| Edit for additional info: the Zojirushi replaced my Sunbeam
| Hot Shot single cup boiler that had an unfortunate dry-run
| and became unusable. They were unavailable in late '21, but
| appear to be back on the market now. They boil a cup of
| water in under a minute. If the Zojirushi didn't exist (or
| I hadn't been willing to bite the bullet on the price) I
| would buy another in a heartbeat.
|
| Value is what is left when price has been forgotten. Both
| of these utensils have been high-value for me, but the
| Sunbeam has a lower initial cost of ownership.
| seesaw wrote:
| I use an electric kettle to boil water. I got it a while
| back from Costco. I find it to be faster than our earlier
| method - using the microwave to boil water.
| jeroenhd wrote:
| Many American homes receive 240V, with the right wiring and
| plugs you can use a 240V electric kettle in the USA just
| fine. You may need to import a European kettle and rewire
| it to a standard American 240V socket (or use one of those
| travel adaptors but that may be dangerous). These sockets
| are already used for high power devices like car chargers,
| AC, and other heavy electrical equipment. Your kitchen may
| even have one intended for induction cooking that you can
| use.
|
| Don't be foolish like some and have 240V wired to a
| standard plug, though! NEMA 6-15 and 6-20 exist just so you
| don't mistake your beefed up socket for a normal American
| socket and fry your devices/zap yourself or others.
| [deleted]
| gumby wrote:
| > problems like the grid frequency increasing and making
| digital clocks run ahead
|
| Ahem: the ones driven by the grid frequency are _analogue_
| clocks.
|
| This function was so important that if load changes caused the
| frequency to fluctuate the power companies will compensate as
| soon as they can to make sure the clocks can catch up. I said
| "was" because I have no idea (and does anyone?) how many such
| clocks are still in service. But it's a legacy function and,
| like POTS, such features are still maintained. Because you
| never know who'll need it.
| userbinator wrote:
| Digital clocks too. Plenty of clocks are still being
| manufactured that rely on the grid frequency, although as
| quartz crystals have become cheaper, some companies have
| started using those instead of mains-referenced clocks. Yet
| the old 70s PMOS mains-referenced clock ICs --- one of the
| first mass-produced LSI ICs for consumer electronics --- are
| apparently still in production and widespread use.
|
| https://hackaday.com/2018/03/29/ask-hackaday-is-your-
| clock-t...
|
| It's been my experience that, in the long term, mains-
| referenced clocks are extremely accurate, while local quartz
| ones seem to drift significantly. Clock also tend to last
| long enough that these are relatively common around here:
| https://en.wikipedia.org/wiki/Telechron
| naikrovek wrote:
| yes. the US grid operators also seem do go through a lot of
| effort to make sure there are exactly 5,184,000 cycles per
| 24-hour period, specifically because it helps those clocks
| keep time.
|
| I don't have any clocks that do this, so I bought one of
| these, which keeps accurate time without intervention from
| me or the power company:
| https://mitxela.com/projects/precision_clock_mk_iii
| jeroenhd wrote:
| Not just analogue clocks! In Europe we had our digital clocks
| run late for quite a while after a power crisis resulting
| from some political energy dispute in Serbia/Kosovo dropped
| the frequency for a while. This caused all manner of clocks
| to run late by five minutes or more. They even intentionally
| cranked up the frequency for a while to bring those clocks
| back in sync.
|
| Alarm clocks, microwave clocks, all manner of time indicators
| without sync mechanism built in were affected because relying
| on Europe's near perfect 50 Hz grid frequency is pretty damn
| cheap.
| b800h wrote:
| Can confirm to some extent - I started out in data analytics
| for call centres, and important episodes of Eastenders et al
| would completely bork the demand predictions by causing exactly
| the pattern of depressed traffic followed by a massive surge
| described in the blog post. Some of this was less about tea and
| more about just having four television channels.
___________________________________________________________________
(page generated 2022-11-20 23:02 UTC)