[HN Gopher] Chaos in the Cloudflare Lisbon Office
___________________________________________________________________
Chaos in the Cloudflare Lisbon Office
Author : jgrahamc
Score : 244 points
Date : 2025-03-17 14:38 UTC (8 hours ago)
(HTM) web link (blog.cloudflare.com)
(TXT) w3m dump (blog.cloudflare.com)
| umvi wrote:
| Is this just for PR/show and the real entropy generator is
| standardized across sites to be something boring in some back
| room? Or if a terrorist came in and cut power to a chaos wall
| would it really cause an extended outage?
| linsomniac wrote:
| >if a terrorist came in and cut power to a chaos wall
|
| That's an interesting way of saying "if the wall loses power".
| Your name wouldn't be Hans Gruber, would it?
| 6stringmerc wrote:
| "I'm a thief and an excellent one at that, and considering
| I've just promoted myself to kidnapper, you should show some
| respect!"
| gunian wrote:
| meh hollywood needs to upgrade its villains lol their
| narrative is just boring and trite
| jgrahamc wrote:
| Details of how we use this are here:
| https://blog.cloudflare.com/harnessing-office-chaos/
| brookst wrote:
| It's an odd world when a hypothetical terrorist is looking to
| reduce entropy.
| thinkingtoilet wrote:
| If your goal is to weaken encryption, reducing entropy could
| be a step along that path.
| brookst wrote:
| Sure, but it is amusing to imagine terrorist cells plotting
| to reduce entropy.
|
| There's a SF story to be had here: the global
| superintelligence uses bits of litter and fallen leaves and
| stuff to generate unbreakable encryption; the terrorists
| wage a global campaign to clean up litter, prune trees, get
| everything neat and orderly in order to hack the system...
| thinkingtoilet wrote:
| Agreed! Your comment did give me a chuckle.
| rzzzt wrote:
| Lava lamp factories are subverted so that the wax mixture
| flows in a straight line without any turbulence
| whatsoever. Dice land on a 3 each time they are rolled,
| anywhere in the world. Casinos disintegrate swiftly
| accompanied by a bright flash.
| deadbabe wrote:
| It's a form of eco-terrorism.
| amarcheschi wrote:
| It sounds something straight out of a cyberpunk novel, some
| sort of enthropy thief
| praptak wrote:
| Entropy does not always align with what we perceive as
| chaos/disorder!
|
| Crystals are generally considered pretty orderly, yet the
| oversaturated solution actually gains entropy when it
| crystallizes.
| pbnjay wrote:
| Given the redundancy of multiple locations, I think even
| cutting power it would be difficult to cause an extended
| outage.
| jgrahamc wrote:
| That sort of happened on "NCIS", Season 16, Episode 1:
| https://blog.cloudflare.com/statement-concerning-events-at-g...
| crazygringo wrote:
| It just injects additional randomness into randomness that is
| already perfectly fine. It's basically for PR/show, but it is
| actually used. But without it, things work fine too.
| thih9 wrote:
| If it silently crashed and started to output a static number,
| would this affect any systems negatively?
| luuio wrote:
| a random number (existing entropy) XOR with a static number
| (the crashed wall) is still a random number, me think
| crazygringo wrote:
| Surely not. If you're seeding a PRNG from multiple sources
| of entropy, you generally concatenate them. Or if you were
| limited in bytes you'd XOR them.
|
| This is why, in an app, you might seed with timestamp _and_
| process ID _and_ /dev/urandom, in case any of them happen
| to be non-unique or unsupported.
| ycombinatrix wrote:
| Probably not, unless it is their only source of entropy.
| jedberg wrote:
| I asked this last time about the one in London, and was
| told that one of the checks is that the image has changed
| since the last run. Otherwise the data isn't used.
|
| Prevents not only technical issues but attacks like someone
| blocking the camera or putting a static photo in front of
| the camera.
| thih9 wrote:
| Interesting - sounds like it would have some negative
| effect then. Thanks for sharing.
|
| Now I wonder about some periodic offsets. E.g. if the
| lights are off at night, or if the skies are overcast in
| winter, does it skew the results in some significant way.
| I seriously doubt that though.
| jedberg wrote:
| Not really. It's just one source of randomness among
| many. The entire point of having multiple sources is that
| they are redundant, you don't need them all.
| eastdakota wrote:
| Great thing about entropy is that adding more never hurts. This
| is one of many sources -- both more conventional as well as
| unconventional -- that we use. If it were to go offline, or
| somehow be corrupted, it wouldn't hurt our ability to generate
| entropy across the Cloudflare network.
|
| What I love about this, the lava lamp wall in San Francisco,
| and the double pendulums in London, is that it takes something
| very abstract and makes it tangible for our team and our
| customers.
| AdamN wrote:
| These are my favorite types of marketing - what I'll just
| call 'part of the actual stack'. Would be great if there was
| a Berlin office so I could join!
| jstasiak wrote:
| > Great thing about entropy is that adding more never hurts.
|
| I used to think the same but here's a counter-example of a
| (hypothetical) attack based on a malicious entropy source
| being able to manipulate the hash/PRNG output:
|
| https://blog.cr.yp.to/20140205-entropy.html
|
| Now, it's not necessarily the most likely attack to
| materialize, as already pointed out downthread:
| https://news.ycombinator.com/item?id=43391377.
| gunian wrote:
| man terrorists are so basic these days they should not be
| considered entropy more like extremely predictable load tests
| conducted by powers that be lmao
| tptacek wrote:
| It is 1000x for PR/show. There is no practical security benefit
| to stuff like this. There's also no meaningful risks to doing
| it. It just lets them write fun blog posts.
| dadrian wrote:
| If they actually integrate this into randomness on their TLS
| servers, the only risk is that the system for getting the
| entropy from the lamps and waves somehow screws up, fails to
| parse an HTTP request or something, and accidentally seeds
| the whole system with no entropy. Whereas doing literally
| nothing and just letting Linux boot correctly on metal would
| be perfectly secure.
| tptacek wrote:
| Right, but there's no way Cloudflare is making that kind of
| mistake. If it was a random person on HN talking about how
| they'd hooked up a bespoke hardware RNG to their TLS stack
| I'd write some tut-tutting thing about what could go wrong,
| but here the security of their system collapses down to the
| LRNG just like every else's.
| blibble wrote:
| yeah, it's not as if they've had terrible bugs like one
| that sprayed supposedly protected cleartext all over the
| internet, is it?
| swyx wrote:
| i mean its also nice office decoration and also lets them
| rank on HN. many companies spend far more on employer
| branding with far less results.
| tptacek wrote:
| Yes, we think the same thing about this. I don't have a
| grudge about it, as long as people don't try to build their
| own versions of it for security purposes.
| ivirshup wrote:
| I think there are backup systems. I sometimes walk by the
| Cloudflare SF offices on weekends and it looks like the lava
| lamps are off.
| eastdakota wrote:
| They should never all be off at the same time. We do cycle
| through each of them turning off for a period of the day.
| But, even if they were all off, there are lots of other
| sources of entropy we use, most of which are for more
| traditional if far less visually interesting.
| theandrewbailey wrote:
| Data centers are filled with loud fans that scream 24/7. I
| would think plugging in a microphone to servers and using that
| to seed entropy would be a more scalable and portable solution
| than cameras looking at lava lamps or water in the office miles
| away.
|
| There has to be some code that already does that.
| coppsilgold wrote:
| The image sensor noise will produce more bits of randomness
| than the chaotic display.
|
| There is also dedicated "TRNG" hardware which will measure
| random thermal noise. Some will even get fancy with quantum
| effects.
|
| Any source of randomness will do, you just feed it into a hash
| function and extract uniform randomness you can use in
| cryptography.
|
| For example, if you have an image sensor that takes an image
| (and does no post-processing) and you feed that image into
| SHA256 you get 256 bits which you can use for cryptography. As
| long as the image is never saved there is no practical way to
| recreate the input and in fact the input will contain more
| entropy (degrees of freedom) than the output, so no one would
| even want to try. Most of the degrees of freedom in the image
| would come from sensor noise and not the scene, so you don't
| even need to take off the cap from the camera.
|
| In practice, multiple sources are combined. The Linux kernel
| does this for /dev/[u]random though it doesn't use the camera.
| There is a potential risk with such combination: one of the
| inputs may come from a source which is able to interrogate all
| the other sources, it would then be able to adversarially
| choose its contribution to skew RNG results. This is a somewhat
| obscure and unlikely threat model.
| calmbonsai wrote:
| Yes. It's just PR. One can just reverse-bias some Zener diodes
| to get the same thing, but it is beautiful PR.
| bob1029 wrote:
| Technically speaking, we can generate and store an unlimited
| amount of information/entropy for future use.
|
| It doesn't really go bad unless you disseminate the material to
| inappropriate parties. You could store terabytes of it in S3
| buckets for an emergency.
| Mistletoe wrote:
| I wish there was more about the camera that images them. Is it
| that little security camera in the pod you can see in the video?
| Does it even have enough resolution for the minute water effects
| across the room to matter/be seen?
|
| An interesting question I would have is do you get just as much
| randomness without the fluid at all?
| Palomides wrote:
| you'd get a similar amount of randomness with a lens cap over
| the camera; it's a neat gimmick, but still a gimmick
| eastdakota wrote:
| This is correct.
| mackenly wrote:
| The goal isn't to capture all of the randomness of the water
| but instead to capture enough noise to produce a seed. All of
| the changes in the room get averaged into a pixel. The water
| just makes the room change more.
| plesiv wrote:
| Name suggestion for the wall: "Hydrobit generator"
| kmoser wrote:
| They should periodically change the name at random, and let the
| name contribute to the entropy. Of course, this won't work if
| everybody votes to name it, say, "Wavy McWaveface."
| orliesaurus wrote:
| that's kinda interesting and kinda a nice PR stunt at the same
| time
| callumprentice wrote:
| I have a real soft spot for Lisbon and Nazare - hoping to live
| there one day after I retire.
|
| Lots of similarities with the last city I lived in - San
| Francisco: Big red suspension bridge that spans the gateway to
| the ocean (same vendor), cable cars running on impossibly hilly
| streets, cosmopolitan, diverse LGBTQ+ friendly people, amazing
| food, nearby vineyards, blossoming tech scene....
| mykowebhn wrote:
| I lived in both San Francisco and Lisbon, and the only glaring
| similarities are the two bridges and the many hills. The Lisbon
| I love is much more charming, less money-oriented, less
| pretentious, and safer. The sad thing is that it's becoming
| much more like America. The more it tries to become like San
| Francisco, the sadder I become.
| gusmally wrote:
| >The sad thing is that it's becoming much more like America
|
| It cannot help that more and more Americans are moving there.
| xpe wrote:
| Who wants to bet that Steve Thompson [1], creator of Prime Target
| on Apple TV, might incorporate lava-lamp based entropy into
| season two, if we're unlucky enough to witness its creation? I've
| been wondering how much worse the show could be, and sadly I can
| envision numerous plotlines involving bizarre sources of entropy
| and their subversion.
|
| [1] https://www.imdb.com/name/nm1839162
| rafram wrote:
| Huh?
| xpe wrote:
| Prime Target gets horrible reviews for botching cryptography
| in cringe-worthy ways, not to mention horrid acting. Normally
| I would avoid such a sarcastic comment about a video series,
| but given the Apple TV platform, it gets a lot of eyes and
| affects how many people get introduced to cryptography, so it
| is good to be aware of it.
| joduplessis wrote:
| Not really related to this post specifically, but my opinion on
| Cloudflare is slowly shifting from "really like them" to "they're
| actually hostile to indie devs".
|
| Almost a month ago now my account was incorrectly charged for a
| pro plan I didn't have --- aside from the week+ long waits
| between replies from support, they refuse to refund me until the
| engineering problem is fixed. I'm still waiting for a reply to my
| follow-up mail from last week.
|
| Yea, $25 is not much - but as an indie dev this makes me HUGELY
| nervous if that amount had even 1 more zero after it.
|
| But hey, cool office guys.
| jgrahamc wrote:
| Email me details of this.
| Etheryte wrote:
| If your users need to resort to HN comments (or Twitter or
| Reddit or etc) to get a support issue fixed, something is
| definitely wrong with the pipeline. How many of your users
| out there have issues, but do not happen to see a random
| comment thread where someone from your team is participating.
| eastdakota wrote:
| Mistakes inevitably happen at scale. Sometimes they're not
| caught by traditional channels. What I try and encourage is
| our leaders to take responsibility and fix them wherever
| they see them. Which is what John did above.
| joduplessis wrote:
| Which is appreciated - emailed!
| jgrahamc wrote:
| I already passed it on to the team. Sorry you've had this
| difficulty.
| barbazoo wrote:
| If the support side of things isn't scaling the same way
| the technology is, then maybe the whole business isn't
| "scaling". You can't just make one side work and then
| throw your hands up in the air and say, well, it's just
| too many user to have proper support. I mean, that's how
| it works but it's not how it should work.
| sophacles wrote:
| What a weird take.
|
| "What I try and encourage is our leaders to take
| responsibility and fix them wherever they see them."
|
| Is kinda the opposite of "throw your hands up in the
| air".
|
| Not sure how you got there.
| barbazoo wrote:
| "Whereever they see them" implies to me that it's not a
| systematic solution that is ever able to scale unless
| they scale their "leadership" the same way.
| sophacles wrote:
| I don't think "wherever they see them" is the entire
| solution to scaling. I'd imagine that they have a whole
| system in place - an imperfect one because all systems
| are - that includes more than one part of it. That is
| "leadership fixes issues wherever they see them" is an
| component the system, not the system as a whole.
|
| Even if you saw all the "cf fucked up" tweets/posts/etc,
| you still couldn't make that assertion without knowledge
| of all the times they didn't fuck up. People are far more
| likely to make noise about issues than things just
| working. This assertion is like assuming the starbucks
| corporation is incapable of producing a coffee people
| will buy because you saw a couple tweets about a messed
| up order.
| squigz wrote:
| Good corporate non-answer.
|
| There is a big difference between mistakes due to an
| individual and mistakes due to systemic issues, which is
| what GP was referring to.
| bell-cot wrote:
| > If your users need to resort to HN....
|
| True - but if a user resorting to HN receives a very
| positive, public reply from their CxO within 2
| minutes...that's a pretty favorable sign overall.
|
| (And even if jgrahamc does nothing except fix this n=1
| problem, his brighter underlings are likely scrambling to
| minimize recurrences.)
| hinkley wrote:
| I'm still not completely over CloudBleed, if I'm honest.
| koakuma-chan wrote:
| One time I got a $70 refund from them after reporting a bug.
| kylecazar wrote:
| It's a cool concept, but I just came to compliment the incredible
| view from that Lisbon office!
| brabel wrote:
| The whole city of Lisbon is amazing. It's very underrated but
| worth a visit for sure.
| lopis wrote:
| How is Lisbon underrated? It's completely, chocked full of
| tourists and wealthy expats, and practically devoid of
| permanent residents because no one else can afford to live
| there any more.
| regularjack wrote:
| I agree with the sentiment, but it's very far from being
| devoid of permanent residents. That's maybe true for a
| couple of downtown neighborhoods, but definitely not true
| for the city as a whole.
| andrewl-hn wrote:
| Plus, many "wealthy expats" don't live in the city,
| preferring neighboring suburbs like Cascais or Sintra.
| The Lisbon proper is very dense, lacks greenery, and
| chokes in traffic, an interesting place to visit, but not
| to live in.
|
| Meanwhile, every single real estate developer / agent
| sets their prices incredibly high hoping to sell or rent
| out the property to those mythical "wealthy expats". I
| saw a stat somewhere that less than 0.2% of all real
| estate transactions in the country each year involve
| foreigners, and yet everyone blames them for high real
| estate prices.
| mykowebhn wrote:
| > lacks greenery
|
| Seriously? How about:
|
| - Avenida da Liberdade
|
| - The Botanical Garden in Principe Real
|
| - Parque Eduardo
|
| - The beautiful gardens around the Gulbenkian Museum
|
| - Jardim da Estrela
|
| - Tapada das Necessidades
|
| - Jardim do Principe Real
|
| and one of the prettiest urban neighborhood green spaces
| I've ever been to: Jardim Fialho de Almeida
|
| Edit: formatting
| epolanski wrote:
| Underrated under which metric?
|
| I would say it's rated highly by tourists and expats.
| mykowebhn wrote:
| Judging from the office view, I'd guess it's situated not far
| from the Museu do Oriente. It's a really nice up-and-coming
| area, although it's not close to any Metro stations.
|
| Edit: I just googled the location and it's right next to LX
| Factory, a rehabilitated, trendy shopping area where one of my
| favorite bookstores is located (Ler Devegar).
| rcarmo wrote:
| Yep. It's pretty nice
| aerostable_slug wrote:
| Long Live Lavarand
| acedTrex wrote:
| I have heard Cloudflare is currently in the throes of an
| offshoring to India debacle.
|
| How the mighty have fallen.
| mvdtnz wrote:
| Source?
| acedTrex wrote:
| Just murmurs through the grapevine from people internal over
| there. I trust what I'm hearing but its nothing official or
| concrete.
| andrewl-hn wrote:
| They keep hiring actively in Portugal, too, mostly because
| developer salaries there are much lower than in North, West, or
| East Europe.
| rcarmo wrote:
| Not that much, really.
| eastdakota wrote:
| Actually, it's because we believe innovation is easier the
| further you are from HQ. And Portugal is wonderful. I'm
| spending about 4 months of the year there with the team who
| hail from around the world.
| FirmwareBurner wrote:
| Would like to work for you. Any chance to seek innovation
| even further away, like in Austria? That's lovely too. :)
| eastdakota wrote:
| No we're not. We opened an India office because it's a big
| market for us. But nothing has gotten "offshored" there.
| nthingtohide wrote:
| Reminds me of this Turkish artist Refik Anadol
|
| https://refikanadol.com/works/bosphorus/
|
| Bosphorus is a data sculpture inspired by high frequency radar
| data collections of Marmara Sea provided by Turkish State
| Meteorological Service in every 30 minutes intervals. The data
| collection of 30 days long sea surface activity transformed into
| a poetic experience and visualized on a 12 meters by 3 meters
| long LED media wall. The art work exhibited at Pilevneli Gallery
| on 11th of December, 2018 - 27th of January, 2019 in Istanbul,
| Turkey.
| emot wrote:
| that's a cool experience. it's really mesmerizing and calming
| watch one or two of these machines, because of the way the
| waves move, similar to ocean waves. there's this video that
| shows it at the end: https://youtu.be/mYLvRaMmfho
| ColinDabritz wrote:
| Adding a single element from each other office would be a neat
| way to tie them together. For example, the new wave wall with a
| single lavalamp, hanging rainbow, and dual pendulum.
| jgrahamc wrote:
| There's custom wallpaper in the office which features elements
| of the other offices, but this is a fun idea. We do have a
| double pendulum and lava lamp elsewhere in the office, just not
| on the wave wall.
| thiht wrote:
| > It's exciting to see waves in Portugal now playing a role in
| keeping the Internet secure, especially given Portugal's deep
| maritime history.
|
| I hope we should not read too much in the hanging of rainbows in
| Austin, Texas
| bohdanb wrote:
| How does Cloudflare ensure that the wave machines in their Lisbon
| office generate truly unpredictable entropy for securing the
| Internet?
| tH4812378 wrote:
| Why on earth is this downvoted? I have the same question. Is
| there a formal proof that lava lamps generate random patterns?
| h1fra wrote:
| Not an expert in the field, I would assume one box is not that
| random since we can easily simulate fluid but the wall in its
| entirety is secure, right?
| pornel wrote:
| The fluid can't be "easily" simulated.
|
| It's a chaotic system (turbulent flow is chaotic). Even tiniest
| differences between the real and simulated state will add up
| and amplify over time.
|
| Fluid simulation is a notoriously hard problem. We don't have a
| solution to Navier-Stokes equations. Practical implementations
| have limited resolution in time and space, and plenty of
| simplifying assumptions.
| srinathkrishna wrote:
| I've been to Cloudflare's SF office many times for meetups and
| even for an interview and their lava lamp wall is one of the
| coolest stuff I've seen in an office.
|
| So cool to see that they've built something similar in their
| Portugal office.
| Havoc wrote:
| How much of this is actually needed for security vs just cool?
| tdeck wrote:
| You mean how much of this is theater so you won't think about
| how this company MITMs a meaningful percentage of the
| internet's encrypted traffic?
| koakuma-chan wrote:
| You can probably say the same about AWS.
| lysace wrote:
| Neat and fun PR stunt. Tech needs more of these!
|
| SGI did this almost three decades ago:
|
| https://web.archive.org/web/19971210213248/http://lavarand.s...
|
| _...harnessing the power of Lava Lite(r) lamps to generate truly
| random numbers since 1996._
|
| According to https://www.lavarand.org/news/lavadiff.html:
|
| _Seed production rate was about 8000 bits of seed per second on
| a 200 MHz SGI O2 under IRIX 6.5._
|
| The patent has since expired:
| https://patents.google.com/patent/US5732138A
|
| (And Cloudflare re-implemented it, seemingly starting just after
| the patent expired in 2016.)
| mdhb wrote:
| Instead of this PR nonsense I feel the cloudflare news we should
| be discussing today is the fact that they just admitted to
| logging the usernames and passwords of users for websites which
| use their service.
|
| https://benjojo.co.uk/u/benjojo/h/cR4dJWj3KZltPv3rqX
| nektro wrote:
| > set to the tune of an AI-generated song.
|
| way to completely ruin an otherwise lovely article
| outer_web wrote:
| Came expecting wrestlemania, left wishing for a wrestlemania
| entropy source.
| npodbielski wrote:
| OK looks cool but otherwise can be achieved much easier with few
| zener diodes and amplifiers. But I know... Does not looks that
| cool.
| nis0s wrote:
| I wonder how random it is truly given that the environment
| something resides in shapes and changes it. For instance, imagine
| if there is some sort of environmental variable (temperature,
| humidity, light, or more) which produces an indiscernible, yet
| impactful, change in the behavior of the wall over time. In that
| way, there may not be true randomness, but a complex pattern
| which can be studied and acquired over time.
| eastdakota wrote:
| If you figure out how to model this fluid dynamics accurately
| over any reasonable period of time, call me. Lots and lots of
| more valuable things you could do with that, e.g., accurately
| predicting the weather.
| bomewish wrote:
| What's the second level analysis here? We know it's not really
| necessary or helpful for the ostensible reason (far cheaper/more
| reliable ways of capturing entropy) -- so we conclude it's a
| marketing gimmick. Yet for the gimmick to work they have to
| pretend it's useful. They're not fooling themselves or anyone
| else, though.
|
| So what's really going on?
|
| Is it:
|
| - it IS somehow a good return on investment??
|
| - marketing had a budget and didn't know how else to spend it,
| and no one wanted to be the unpleasant person and say how it's
| all a silly waste of money?
|
| - they are making a tonne of money and no one really cares, so
| we'll just spend it on fun cool stuff as long as there's a
| plausibleish story to go with it?
|
| - fits with a broader global company branding concept that
| leadership seems to like, so there's just the momentum to keep it
| going (and see points above)?
|
| I can't figure it out. I agree it's cool! Just the make believe
| puzzles me a little. I've not worked at a big corp like this and
| just have to understand what's actually happening.
| babelfish wrote:
| It is a gimmick, but targeted at recruiting, not sales.
| Cloudflare is known to be "blog driven engineering"
| eastdakota wrote:
| If it were merely marketing spend for customer acquisition, I
| bet the ROI on the lava lamp wall in SF has been 100,000x. This
| isn't hard to figure out.
| bravura wrote:
| Subtext: In case a) you want to move to Europe, b) cannot
| tolerate weather that differs from California, c) like Golden
| Gate bridges, are you aware that Cloudflare has a Lisbon office?
|
| Source: I live in Portugal and notice all the discussion about it
| from my Bay Area friends who want to expatriate. To here,
| specifically.
| collingreen wrote:
| Visited Lisbon once, live in SF. Can confirm I loved it and
| want to move there.
| euroderf wrote:
| Is there much I.T. in Porto ? I've heard it's a great town.
| gusmally wrote:
| Expatriation (immigration?) to Portugal from America furthers
| Portuguese wealth inequality and housing crisis. Lisbon is
| already basically a playground for foreigners and if you do
| decide to move there please try to assimilate by learning the
| language, etc.
|
| Source: Portuguese/US citizen. Lived there for a year with a
| "good" Portuguese salary of 2000 euros a month; much different
| lifestyle than what the typical US tech worker is accustomed
| to. And not doable long term unless you plan to retire in
| Portugal and not do much travelling.
| mykowebhn wrote:
| These type of comments make me cringe. Nowhere does the OP
| describe what Portugal is like in itself. It's very often
| compared to something else, like California. Portugal is
| Portugal.
___________________________________________________________________
(page generated 2025-03-17 23:01 UTC)