[HN Gopher] Gossip Glomers: Fly.io Distributed Systems Challenges
___________________________________________________________________
Gossip Glomers: Fly.io Distributed Systems Challenges
Author : yla92
Score : 182 points
Date : 2023-02-22 16:06 UTC (6 hours ago)
(HTM) web link (fly.io)
(TXT) w3m dump (fly.io)
| yodelshady wrote:
| Dockerfile would be a really good idea.
|
| Followed instructions to letter, got : main.go:7:5: no required
| module provides package github.com/jepson-io/maelstrom/demo/go
| $GOPATH is correctly set, I can see the package there, I've spent
| half an hour looking through incorrect internet answers and
| frankly I have better things to do than debug build scripts.
| Actually, no I don't. But I still normally get paid.
| [deleted]
| benbjohnson wrote:
| I tried reproducing the error and I think I know what happened.
| There's an instruction to run "go mod tidy" in Go's "go mod
| init" command but that could be more clear in the instructions
| on our site. I'll update that. Thanks for letting me know.
| gsanderson wrote:
| ".. represented by two separate yet equally important groups
| ...". Cue the Law and Order intro in my head. I like it.
| paxys wrote:
| So this is some new elaborate hiring challenge?
|
| But then when I open the linked jobs page the company seems to
| have exactly one open position. Is it that hard to hire one
| infrastructure engineer that you need to do...all this?
| andai wrote:
| It is also marketing (by virtue of the fact we are talking
| about it!)
| packetslave wrote:
| [flagged]
| sieabahlpark wrote:
| [dead]
| tptacek wrote:
| No. We contracted Kyle to help us build a hiring challenge, and
| he and Ben overdelivered dramatically, so we turned it into a
| public thing. They're just programming challenges.
|
| If you're not interested in programming distributed systems,
| this won't be interesting to you.
| andrewfromx wrote:
| hehe I'm literally in the middle of your actual hiring
| challenge with "Fix the Glitch" 6pn networking. But I've hit
| a wall with it. Maybe I'll switch over to these :)
| specialp wrote:
| I completed that exercise along with a Nomad one w ELK
| stack. And submitted it. And never got a response ever
| besides we will look at it ASAP. (3 months ago.). It was
| billed as a "2 hour" exercise, but to even get close to
| that you would need to be completely fluent with Fly,
| Nomad, ELK, Wireguard, Linux networking stack. I thought it
| was a great take home but you would at least expect a
| response for completing it successfully.
| [deleted]
| sosodev wrote:
| That's even worse than my experience
| https://news.ycombinator.com/item?id=34523227
|
| I'm pretty skeptical of these hiring challenges at this
| point. It seems that they often require more time than
| the traditional model and provide worse outcomes for the
| candidates. For me it was particularly frustrating
| because I can typically write off the traditional model
| as irrational but I was drawn to fly.io because they make
| it seem so much better than that.
| whiplash451 wrote:
| If I were looking for a job, I would trade the usual
| stupid 1h leetcode interview for this take-home in an
| eyeblink.
|
| This one is much more time-consuming _up front_, but at
| least it gives you a sense of the job to be done _and_
| doing the exercise lets you think through whether you
| really want that job after all.
| paxys wrote:
| Well in that 1 hour leetcode session at least the
| interviewer is wasting the same amount of time as you. In
| this case you can spend all weekend working on the
| exercise and never even get a response.
| tptacek wrote:
| I don't have a better response for this than that we got
| a huge flood of candidates for that role, are doing our
| best to keep up, and have done an imperfect job. We're a
| small company. You're more than welcome to reach out
| directly to find out what happened with your application
| (it's entirely possible that what happened is that the
| ticket got buried somewhere, and that we'd have been
| excited to hire you had we not screwed something up on
| our end).
|
| As for the scope of the challenge: that's deliberate, and
| we're up front about it. If the challenge is going to
| take you many hours to complete, and doing that work
| isn't something that lights you up, then it's the wrong
| role for you. There are candidates that breeze through
| it, and there are candidates who wrestle with it for a
| couple evenings just because they're nerdsniped and enjoy
| doing that stuff. Both those kinds of candidates are
| super interesting to us! But we're very comfortable with
| the idea that the job (in this case: infra engineering)
| isn't a perfect fit for everybody.
| specialp wrote:
| I really liked the take home. It is just that if you have
| someone doing this, you are having that person commit
| hours of their time to interviewing without the same
| investment on your end. So if you do not have the means
| or time to process these submissions then it is not fair
| to have someone spend their time on it.
|
| I think your hiring process is great otherwise. And yes
| if someone isn't very familiar with every intersection of
| stacks there it will take them more time. That is fine
| too. It shows the person can learn on the spot.
| tptacek wrote:
| I agree. We came to that conclusion about platform just a
| week or so ago and have paused hiring because we can't
| meet the commitment we've made about responsiveness. We
| haven't paused infra; instead, we've hired someone to
| help keep up with it.
|
| I can't mitigate any bad experience you've had. I fucking
| _hate_ getting ghosted, and I get a little nauseous when
| I think about us having ghosted people. But ghost people
| we have! All I can say is that it 's not deliberate; it's
| just been a lot to keep up with.
|
| For everyone else: my sense is that the majority of
| people in our process have gotten timely (if not as fast
| as I'd like) responses from us, but if it's been weeks
| and you haven't heard back from me, you can hit me up
| directly. We don't ghost people "communicatively"; that's
| just not how we convey decisions.
| ayewo wrote:
| Essentially, you've paused hiring for all roles at Fly
| including these 2 roles on your jobs page
| https://fly.io/jobs/ ?
| tptacek wrote:
| We've _paused_ platform hiring; we have plenty of slots
| for it, but we don 't currently have the bandwidth to run
| hiring for it (and we're also metabolizing a bunch of the
| hires we currently made).
|
| I'm not aware of us pausing any other roles; if they're
| not listed, it's likely we don't currently have openings
| for them.
| andrewfromx wrote:
| hehe yes the Nomad one w ELK stack took me longer than 2
| hours for sure but was a lot of fun! I'll bet the fly
| people are just swamped. I loved the language in their
| email though. I've been on so many 1 hour live coding
| interviews where I can't shine like I want to shine this
| approach really made me go yes! A new way to interview.
| tptacek wrote:
| We are swamped. We kept up as long as we could, and now
| we're making some changes:
|
| 1. We have someone running infra now that also owns infra
| hiring, instead of me owning it. Infra is an ultra-
| important role here and we're still hiring for it.
|
| 2. I'm still on the hook for platform hiring, and we've
| paused it for a couple months (if you submitted a
| challenge, we're still moving forward! but we're not
| taking new applications).
|
| I know people on HN don't know these details, but knowing
| it myself makes it especially goofy to see people calling
| these challenges an elaborate scheme to recruit people
| for platform development (which is where these challenges
| apply).
| bogomipz wrote:
| >"Over the past year, we've been growing like gangbusters. That's
| great. But it also means we've been hiring, and hiring is hard."
|
| So given that hiring is hard you decided to go out an hire a
| renowned distributed systems expert Like Kyle Kingsbury to design
| distributed system to challenges to make hiring easier? And then
| you go on to say the challenges are so much fun that you couldn't
| possibly keep them to yourself. Is this the new bar in humble
| brag? This is so completely cringe-worthy. You are looking for
| people to write Terraform to deploy Nomad to AWS and you are
| testing for a candidates deep understanding of Linearizability? I
| realize this is meant as PR to show how smart are you are and how
| high your bar is but then that raises the question why did you
| need to hire Kyle to come up with these?
|
| I think this post might but not be having the effect you
| intended. In fact quite the opposite.
| tptacek wrote:
| We hire people to write distributed systems, and for those
| people, distributed systems is extremely relevant. We also hire
| people to do low-level Linux kernel stuff, and for those
| people, it's less relevant. I'm having trouble figuring out
| what it is you think you've found to dunk on here.
| bogomipz wrote:
| [flagged]
| tptacek wrote:
| Probably for the same reason that practically every other
| team that builds distributed systems engages Kyle. But I
| can't speak for them, I guess.
|
| _Later_
|
| (This comment was rewritten after I replied to it; it
| previously expressed contempt for the idea that a company
| that supposedly works on distributed systems would retain
| any of the services of Kyle Kingsbury.)
| [deleted]
| biorach wrote:
| I don't know why you're being a total dick about something
| that's actually pretty neat.
| neom wrote:
| [flagged]
| whalesalad wrote:
| This could be said about alllll the inbound marketing pieces
| that end up on HN. That being said, I appreciate what Fly is
| doing and enjoy seeing their content and problem solving.
| eatonphil wrote:
| Fly.io uses their blog to educate people. They have some of the
| highest quality writing out there and talk about unique topics
| many people can't talk about with authority (SQLite,
| distributed systems, etc.).
|
| This is amazing and I highly hope they keep doing what they do.
| I think it would be better if more companies did the same
| (sharing their knowledge).
| neom wrote:
| [flagged]
| ignoramous wrote:
| [flagged]
| yunohn wrote:
| And netlify.
| tptacek wrote:
| This whole subthread is gross. We're asked by the
| guidelines not to have discussions like this. Can we
| please not?
|
| _Please don 't post insinuations about astroturfing,
| shilling, bots, brigading, foreign agents and the like.
| It degrades discussion and is usually mistaken. If you're
| worried about abuse, email hn@ycombinator.com and we'll
| look at the data._
|
| https://news.ycombinator.com/newsguidelines.html
| yunohn wrote:
| I believe the previous poster and I were attempting to
| show that accusations of astroturfing are gross and can
| be hypocritical.
| tptacek wrote:
| You can't do that by repeating the same behavior! You fix
| this problem with the "flag" button.
| dang wrote:
| I can assure you that YC (in the form of HN admins) isn't doing
| anything to promote these posts. Like it or not, they're just
| popular with the community.
| benbjohnson wrote:
| hey John, long time! Sorry that the post came off as thought
| leadership. We thought it'd be fun to build something like
| Cryptopals but for distributed systems. Building projects like
| this helps give awareness to Fly.io but I'd like to think it
| helps the community at large as well. DigitalOcean did a great
| job of this in the past as well.
| tptacek wrote:
| 1. Not really a recruiting thing.
|
| 2. You'd get disciplined at Fly.io for non-ironically using the
| word "thought leadership".
|
| We're a company full of message board nerds working on
| something super fun. We're going to write about it. Best get
| used to it!
| sosodev wrote:
| How is this not a recruiting thing? It's promoting your
| supposedly superior hiring process.
| tptacek wrote:
| Well, for starters, we've paused hiring for a few months on
| the role it pertains to?
|
| Once again: this started as a hiring challenge, and Ben and
| Kyle overperformed and made it something more interesting.
| I guess we could have waited a few months and ran this at a
| time when we could metabolize more platform candidates,
| but, once again: _this isn 't a recruiting thing_.
| babelfish wrote:
| If you're not hiring, maybe you should remove this call
| to action implying you are at the bottom of the post:
|
| > We reserved this last challenge for evaluating our
| staff engineers at Fly.io. So if you think you'd be up to
| the challenge, we'd love to talk to you.
| tptacek wrote:
| We're happy to talk. But the first thing you'll hear from
| us is "we've paused platform hiring for a couple months".
| darkmarmot wrote:
| Looks fun and interesting. I will say your recruiting
| call at the end made some of the devs where I work swear
| off your service... I think there have been enough scams
| in this vein that it rubs certain people in a bad way.
| tptacek wrote:
| Our "recruiting call at the end" of what? I'm sorry, I
| don't understand. This post isn't advertising a new
| service; it's literally just a series of blog posts.
| johnbellone wrote:
| I am not passing judgment, but it's literally the last
| sentence of the link. Personally, I don't see any
| problems.
| dadrian wrote:
| Sorry, what? A company blog post that links to their jobs
| page means you won't use a service? Do you just not use
| any service that advertises? How do you get Internet
| access?
| darkmarmot wrote:
| I'm not that sensitive and have loved recruiting code
| games/challenges put out by other companies (like
| Scribd's AI recruiting game way back...) I was just
| noting that when I shared this link with my extended
| team, there was a pretty negative reaction because of how
| this type of thing has been used in the past by
| unscrupulous actors.
| monksy wrote:
| Is the discipline shots of malort?
| tptacek wrote:
| Not even I would come up with a disciplinary scheme
| requiring employees to ingest poison.
| tough wrote:
| Sounds cool, keep up the great work
| euroclydon wrote:
| This is rude for pretty obvious reasons.
| babelfish wrote:
| A few weeks ago, someone at Fly threatened to disable a
| customer's account over Twitter (Alex Graveley, the creator of
| Github Copilot) because he complained about a minor-version
| breaking API change. Did anything ever come of that? I like Fly
| as a company, but overall the impression I get from their
| employees (even in this thread) is just...highly defensive and
| immature.
|
| Edit: Here is the link to the thread. Alex's original tweets were
| deleted.
| https://twitter.com/alexgraveley/status/1619117645932150784
| mbStavola wrote:
| It's hard to pass judgement when the first couple of tweets are
| deleted, but this didn't read to me as Fly.io kicking anyone
| off the platform. Here is what my interpretation was of this
| interaction:
|
| > Customer: Your service is bad and I had a bad experience >
| Vendor: Okay > C: I've given you multiple chances but you're
| still not up to my standards > V: (assuming C was done with the
| product) Sorry to hear that, but here is a refund
|
| Am I missing something here?
| cldellow wrote:
| I don't see it in as clearly bad a light as babelfish does,
| but nor do I see it as positively as you do.
|
| My rubric:
|
| - CEOs shouldn't publicly call their customers dicks
|
| - if a customer thinks the CEO is threatening to kick them
| off the platform and asks for clarification about it, they
| should get a clear response (Graveley's last tweet)
|
| Both of these points are about professionalism of the vendor.
|
| Perhaps the Fly CEO realized Twitter wasn't the right place
| to have this conversation and answered Graveley's question in
| a separate channel.
| ryanjshaw wrote:
| (1) @mrkurt: ... you can go ahead and find yourself another
| vendor. (2) @alexgraveley: [I'm] Trying! [to find
| another vendor]
|
| Kurt thinks Alex is choosing to leave due to (2):
| @mrkurt: ok good luck [switching vendors by your own choice].
| I'll refund all your payments.
|
| Alex thinks Kurt was the one telling him to go due to (1):
| @alexgraveley: I'm sorry, you're kicking me off your platform
| ... ?
|
| Typical human stuff. Maybe this is a great opportunity for an
| LLM extension - have it give people the benefit of the doubt
| before you respond nastily to them, by checking your
| response.
|
| I've tried asking ChatGPT a few times to explain responses
| people have made to me on Twitter that I didn't rate highly
| but which received likes. It wasn't particularly insightful,
| but its unbiased perspective had a surprisingly calming
| effect.
|
| EDIT: I fed all but the last line of this conversation into
| ChatGPT, then asked it to evaluate 'my' proposed reply (the
| final line) with the following prompt: "What is the
| presupposition behind my response? Can you point out any
| potential mistakes I've made in my assumptions?" It correctly
| identified the ambiguity in the statements, and made a
| helpful suggestion on how to proceed.
| [deleted]
| [deleted]
| mtlynch wrote:
| > _A few weeks ago, someone at Fly threatened to disable a
| customer 's account over Twitter (Alex Graveley, the creator of
| Github Copilot) because he complained about a minor-version
| breaking API change. Did anything ever come of that?_
|
| I hadn't heard of that, but I think that's an inaccurate
| characterization based on what I'm seeing of the conversation:
|
| > _Alex Graveley: [two deleted tweets]_
|
| > _Kurt Mackey, Fly CEO: Wow, harsh. It 's fair to be upset and
| an API change. But if you're going to be a dick on twitter you
| can go ahead and find yourself another vendor._
|
| > _Alex: Trying! It 's not like this is an isolated event -
| gave you all a pass the first 4 times._
|
| > _Kurt: ok good luck. I 'll refund all your payments._
|
| > _Alex: I 'm sorry, you're kicking me off your platform
| because I was mean to the company on twitter?_
|
| Kurt was telling him that his behavior was unacceptable and
| that he didn't want to act as Alex's vendor if he didn't
| correct it. I think it's fine for companies to fire customers
| that are rude or abusive.
|
| I can understand if Kurt said, "oh yeah? Well I just nuked your
| account with zero notice!" But it seems like Kurt was giving
| Alex the opportunity to exit the platform gracefully.
|
| Also, I'd be a little more sympathetic if it was like Andy
| Jassy crushing a random company's business on AWS because they
| complained on Twitter. But this is a principal engineer at
| Microsoft presumably shitting on one of Microsft's underdog
| cloud competitors because he didn't like his experience using
| Fly on a pet project, so it's not like there's a huge power
| imbalance here.
| babelfish wrote:
| Thank you for noting he is the CEO, I didn't realize that in
| my initial comment.
|
| While I saw the two now-deleted tweets originally, I don't
| recall what they said now, but I do remember feeling like it
| wasn't anything serious enough to warrant the "find yourself
| another vendor" comment.
|
| If Kurt is willing to (gracefully, as you said) crush
| someone's side project over a rude tweet, why should Fly be
| trusted with a potential business? I don't use TinyPilot but
| I am a big fan of your blog. Would you trust running any
| servers required for TinyPilot on Fly.io after that exchange,
| running the risk that if Kurt doesn't like one of your
| tweets, he (gracefully) will ask you to move to a new
| platform?
| sodality2 wrote:
| > If Kurt is willing to (gracefully, as you said) crush
| someone's side project over a rude tweet
|
| I didn't read this at all. I read it as "we will refund
| you", not "we will kick you off". I know that fly.io
| frequently refunds payments for customers, if they feel the
| bill was unexpectedly large.
|
| > running the risk that if Kurt doesn't like one of your
| tweets, he (gracefully) will ask you to move to a new
| platform?
|
| I read this as "if you don't like us, leave us", not "I
| want you to leave us because I didn't like what you said".
| The first one is a bit more fair, because it's just a
| statement that if you don't like the way we do things,
| there are alternative providers, though some may say it's
| not the most graceful way to talk to an annoyed customer
| (picture "apologies for the breaking change, we will try to
| warn/communicate/avoid such changes in the future"). The
| second one is far more brazen.
| babelfish wrote:
| Regarding your first point - the first thing Kurt said
| was "if you're going to be a dick on twitter you can go
| ahead and find yourself another vendor". I don't believe
| there were any billing issues as a part of this exchange,
| though Kurt's offer to refund was kind.
|
| And regarding your second point, I read the above quote
| the same way Alex did, as being kicked off the platform
| ("go ahead and leave", not "you're free to leave").
| subarctic wrote:
| Just looked at the thread you linked. I wouldn't say I read
| that as a threat - more like he's saying "if you don't like it
| you can leave".
|
| Not that that sounds great, but I can't see the original tweets
| that kurt replied to because they've been deleted, so it's
| kinda hard to judge
| newaccount2021 wrote:
| [flagged]
| ignoramous wrote:
| Or stability...
| eatonphil wrote:
| Love it. Thanks for putting this together! The actual challenges
| here [0].
|
| Though I'm curious: are these different from the chapters in the
| Maelstrom documentation [1]? There seems to be a bit of overlap
| anyway.
|
| [0] https://fly.io/dist-sys/
|
| [1] https://github.com/jepsen-io/maelstrom#documentation
| benbjohnson wrote:
| Thanks, Phil! Good question. Yes, there is some overlap with
| the Maelstrom docs but most of the challenges either have
| different goals or they are new (Kafka, Totally-Available
| Transactions). These challenges also are aimed at people who
| are familiar with distributed systems and want to test their
| ability as opposed to the Maelstrom docs where they are meant
| to walk folks through the issue.
|
| We also contributed a Go implementation of a Maelstrom client
| library and we were able to fix a handful of bugs in Maelstrom
| during the process. Kyle uses Maelstrom for teaching classes as
| well so work we did on these challenges were also contributed
| back into the repo.
|
| The fun part about these challenges is there isn't one right
| way to do them. Many of the later challenges have several
| approaches and they each have their own trade-offs. Some of the
| challenge goals are performance targets so you can continue to
| optimize your implementation to get better numbers.
| Reventlov wrote:
| The json from the first echo example is not actual json, there is
| a trailing comma that shouldn't be there :p
| benbjohnson wrote:
| Ah, you solved the first challenge! Noticing my typos. :)
|
| I fixed up and validated all the JSON. The fixed version will
| be up on the web site in a couple minutes. Thanks for letting
| me know.
| LAC-Tech wrote:
| _Reading a stack of MIT PhD dissertations may be a good Friday
| night_
|
| I'm happy if I can read through a single MIT PhD dissertation in
| a stack of days!
|
| First I've heard of Maelstrom. Very cool.
___________________________________________________________________
(page generated 2023-02-22 23:00 UTC)