[HN Gopher] How Uber tests payments in production
___________________________________________________________________
How Uber tests payments in production
Author : ohduran
Score : 172 points
Date : 2024-08-07 07:16 UTC (15 hours ago)
(HTM) web link (news.alvaroduran.com)
(TXT) w3m dump (news.alvaroduran.com)
| jatins wrote:
| Extremely fluffy piece. 20% in and not one valuable piece of
| information
| rty32 wrote:
| Whenever I see an article with more than a few sentences that
| seem to be arbitrarily bolded, I know it isn't worth reading.
| Haven't had a failed case so far.
| dartos wrote:
| How would you know if you did?
| amne wrote:
| you read HN comments and confirm that you did well to skip
| the rtfa part
| colesantiago wrote:
| I'm beginning to think that Substack the new Medium, and this
| cannot and will never be solved.
|
| It would be better and respectful of the readers time to get to
| the point of the article rather than stuff the article with
| more words wasting the readers time.
|
| When I come across articles which are needlessly long, I either
| skip them or I use a summarizer and leave the page.
|
| There will always be clickbait elaborate content like this,
| (clickbait title, actual answer at the end of the article 90%
| of the time) but it just trains the reader to just scroll to
| the end of the article for the answer most of the time,
| achieving the opposite of what the article writer wants.
| red2awn wrote:
| Stopped reading after this
|
| > The reason I know this is because I've built and maintained
| systems that handle close to 100,000 payments a day.
|
| That's 1.16 payments per second.
| djtango wrote:
| 1 payment per second. 1 payment per uber ride. 10 dollars per
| ride. 864k per day. 365M per year. It's not a small system
| and could be some mixture of one market at Uber or some %age
| of rides (eg one payment provider)
|
| (it could be 2 payments per ride or drivers get batched
| payouts but w/e).
|
| There's obviously bigger payment platforms (eg Stripe or GPay
| / Apple Pay or Amazon) but not all of us work in payments
| either
| tivert wrote:
| > Extremely fluffy piece. 20% in and not one valuable piece of
| information
|
| What? Your mind wasn't _totally_ blown by the advice "Instead,
| the lesson should be this: to test your payment systems in
| sandbox for an amount of time that's reasonable. And not a
| second more."? /s
| madaxe_again wrote:
| From my experience building medium scale ecommerce systems, along
| with innumerate payment integrations of various flavours, this
| isn't unreasonable, for a few reasons.
|
| Firstly, payment service providers honestly suck at providing a
| coherent staging environment. Either it'll be out of date, or
| ahead of production, or full of garbage data that you can't clear
| that breaks their outputs, or just plain not representative of
| the production environment. You'll have stuff check out perfectly
| in staging only to be a hot mess on _their_ live environment.
|
| Secondly, if you're doing this stuff at scale, it's not as simple
| as "make an API call and get a result" - you've got your egress
| and ingress to worry about, at different levels (NAT, load
| balancing, packet routing, http(s) proxies), and there's a host
| of stuff that can go wrong for subtle reasons.
|
| We used to (for they are now just a shopify shop since my
| departure a decade ago) do exactly as is described - test in
| staging as much as it is useful, and then go live with an
| immediate test built into the deployment toolchain, with
| automatic rollback in case of failure for any reason.
|
| It worked. The only payment issues we ever had after having the
| realisation that testing on staging was damned near meaningless,
| were on the side of the payment gateway.
| michaelt wrote:
| _> test in staging as much as it is useful, and then go live
| with an immediate test built into the deployment toolchain,
| with automatic rollback in case of failure for any reason_
|
| I'm curious about the logistics of automatically testing
| payments after a deployment?
|
| Does your automated test place an order with a valid, working
| credit card? Does your test include going through 3D Secure
| too? Do you then automatically cancel the order? How do you
| make sure that whole unusual process doesn't get blocked by
| unusual activity fraud detection? Whose credit card is it? Have
| broken tests ever lead to the test order getting fulfilled?
| madaxe_again wrote:
| > Does your automated test place an order with a valid,
| working credit card?
|
| Yep. Organisation owned card for the purpose, details used by
| selenium for the test. Details stored securely, I might add,
| as PCI/DSS and ISO27k1 were important to us.
|
| > Does your test include going through 3D Secure too?
|
| Yeah. Same bank card always being used meant we could
| automate the flow.
|
| > Do you then automatically cancel the order?
|
| It went through the whole despatch process, including label
| production with couriers etc., and was then cancelled as that
| tests everything including CANCEL/VOID and the whole critical
| flow.
|
| > How do you make sure that whole unusual process doesn't get
| blocked by unusual activity fraud detection?
|
| By using the same card over and over, placing an order for a
| normal item, talking to our bank when it did occasionally get
| flagged.
|
| > Whose credit card is it?
|
| The businesses.
|
| > Have broken tests ever lead to the test order getting
| fulfilled?
|
| Yup. We had a few that appeared on our doorstep, both due to
| our error, and client error.
| altacc wrote:
| People are calling this article fluffy but I agree with you
| that if people haven't worked with a lot of payment gateways
| before then it's good advice never to fully trust the test
| environments. Many years ago I had a big launch turn to chaos
| when test worked perfectly but the payment gateway's validation
| on production turned out to be different and declined every
| payment without returning a meaningful error.
|
| A lot of work has gone into getting a customer to make a
| purchase, so it's the worst time to fail. Nothing beats testing
| on production with a real card.
| AlexDragusin wrote:
| Fluffy as they go, here is a 100-word summary of the article (via
| local instance of Llama 3.1 7B):
|
| Uber's payment system testing approach has been criticized for
| being unconventional, but it actually makes sense in retrospect.
| The company tests its payment systems in production, rather than
| in a staging environment, because this allows them to catch bugs
| that would be difficult or impossible to replicate in a simulated
| environment. This approach is based on the idea that every
| deployment should be considered an experiment, and that the goal
| of testing should not be to avoid failures at all costs, but
| rather to make rollbacks as frequent as possible when something
| does go wrong.
| lijok wrote:
| That is a blatantly incorrect summary. Might you have dismissed
| the article too early?
| simondanerd wrote:
| The summary is generated by AI.
| cglace wrote:
| Why post an AI summary if you know it's wrong?
| lifeisstillgood wrote:
| Sorry - it seems a fairly accurate summary. Quite impressed
| by Llamma 3 there
|
| I mean there is even a pull quote in the article:
|
| Not all bugs can be found until you are in production -
| therefore some testing must be done in production [and by
| implication you need to test carefully and be able to
| rollback]
| andriesm wrote:
| I see several comments calling this piece "fluffy" without much
| real insight - I have to respectfully disagree - I'm 48 and wrote
| my first code at 8, still write code for my self at 48, have
| managed teams, held all manner of roles and done some startups.
| This article is solid gold.
|
| I'm surprised people think this article doesn't have much
| important to say. I suspect their code probably crashes a lot in
| production, and will still kill many startups or otherwise end up
| destroying significant amounts of shareholder value.
|
| They think the article is banal and obvious. They will not really
| take the key insights to heart and truly live it.
|
| Crowdstrike is the perfect example of this!!!
|
| And for every crowdstrike there are tons of startups that don't
| make the news but ends up burning their early adopter users
| through inability to deal with bugs properly, delay their own
| success unnecessarily or even turns what would have been massive
| business successes into technical morasses. Imagine failing to
| capture your businesses full potential because of a bad approach
| to software defects!
| christina97 wrote:
| You don't really get at what you think the substance of this
| piece is. It'd be helpful if you pointed that out instead of
| just going on about how phenomenal it is.
| compsciphd wrote:
| I think he was being sarcastic, but can't tell exactly.
| minasmorath wrote:
| To me that's the mark of a high quality sarcastic reply.
| keybored wrote:
| Haha, that's masterful. I had no idea but reading it again
| now it feels so obvious. :D
| Tempat wrote:
| It's a parody of the writing style of the article itself, all
| excitement and noise, saying little to nothing.
| NotGMan wrote:
| TLDR: some bugs can only happen in a real production environment,
| so expect them and be ready when deploying. Thinking your deploy
| will be ok because staging env passed all tests is delusional.
| dotancohen wrote:
| Yes, exactly this. I test staging before every deployment, and
| prod after every deployment. Thirty $2 credit card payments per
| month on my personal credit card is a small price to pay for
| the piece of mind that the next $800 order won't fail.
| snowstormsun wrote:
| > First, you have to copy all production data. It's expensive,
| and a reckless breach in privacy and security, but it's doable.
|
| So, what does "doable" mean in this context? We unnecessarily
| increased the attack surface for production data and until today
| haven't suffered a data breach because of it?
|
| A staging env with actual prod data now needs be treated as a
| production environment. A system is only as secure as its weakest
| link, so an attacker will have an easier time getting into that
| "staging" environment where things are tested out, no?
| DonHopkins wrote:
| Crime is doable.
|
| https://www.youtube.com/watch?v=kYdQuuLzg2A
| AppliedQuantum wrote:
| Or, one could test in production-parallel deployment. Clone all
| requests to a parallel test system, use the same production data
| for enrichment and validation for both, the current production
| system and the new one. And automatically compare the outputs
| from both systems for those fields that have to be the same
| between the systems, and test the expected changed outputs
| automatically.
|
| Once there are no errors in the new system, you start switching
| over the systems in a controlled manner where the new system
| increasingly takes on the production role, and the old one still
| processes cloned requests for a while as a sanity check...
|
| This way you don't need an unrealistic staging environment, and
| you are not introducing any errors into production.
|
| It worked more than 20 years ago when I architected this for a
| system that had to process 50M transactions every hour.
| K0balt wrote:
| If you rely on a card-processor or a banking API this has some
| limitations.
| AppliedQuantum wrote:
| Nothing's stopping you from cloning those responses as
| well... Compare calls, clone responses.
| valicord wrote:
| Charge customers twice?
| AppliedQuantum wrote:
| I'm sure my former employer would have loved that.
|
| But no, you don't send two requests. You compare the
| calls as they are generated, but you only send one - from
| the production system. And then you clone the response
| for the tested system.
| mrbluecoat wrote:
| > For Uber, every deployment is an experiment
|
| Me: Let's do that!
|
| Boss: Ummm...
| cheschire wrote:
| > software is not like other machines. Most machines, in time,
| rot and decay. But software is just information: if it's correct,
| it stays that way. Hardware does need replacement, but the
| correct software that runs on it keeps running.
|
| Unless you have some empowered person or group in your
| organization, levels above your team, that is allowed to
| constantly move the goalposts because of "cybersecurity!!1" and
| even the most mundane internal-only systems have the be kept to
| the latest versions of everything ever just so their scanning
| software shows "green". Probably because their own OKRs are based
| on how many green circles they keep or something.
|
| They're cyberaccountants.
| zadokshi wrote:
| This article can be rewritten into one line:
|
| "Not all bugs can be found until you deploy to production. So
| deploying to production can be called 'testing in production'"
| djtango wrote:
| I thought the colour and anecdotes were useful towards
| conveying the message. Sometimes only after you've experienced
| something for yourself does the reduced pithy one liner make
| sense and resonate.
| K0balt wrote:
| The (somewhat obvious) parts about staged rollouts and
| selection criteria for initial deployments are useful. If
| CrowdStrike had rolled to a small demographic first. Billions
| of dollars could have been spared the shredder.
| boesboes wrote:
| I bet staged rollout are on some poor PO's backlog. We don't
| have the bandwidth for such niceties! :')
| AmericanChopper wrote:
| Eventually you're going to make a change that completes 100%
| of the production rollout. That change should be tested too,
| as any change is a new opportunity to break something.
| fragmede wrote:
| Everyone has a test environment, the lucky ones have a separate
| production environment.
| andrewl-hn wrote:
| Isn't it what's _everybody_ does in the industry?!
|
| Every single place that I ever worked at in a past 20 years tests
| payments using real cards and real API endpoints. Yes, refunds
| cost a few pennies and sometimes can't be automated, but most
| payment providers simply do not offer testing APIs of a
| sufficient quality.
|
| Situations when a testing endpoint has one set of bugs not found
| on production and vice versa used to be so ubiquitous in mid-2000
| to mid-2010s, that many teams make a choice agains using testing
| endpoints altogether - it's too much work to work around bugs
| unique to the environment that no real customers actually hit.
| And now the whole generation of developers grew in a world of bad
| testing APIs of PayPal, Authorize.net, BrainTree,
| BalancedPayments (remember them?), early Stripe, etc. So, now it
| became an institutional knowledge: "do not use testing endpoints
| for payments".
|
| To be exact, people often _start_ using testing endpoints for
| early stages of development when you don 't have any payment code
| at all, but before the product launch things get switched to
| production endpoints and from that point on testing endpoints
| aren't used at all. Even for local development people usually use
| corporate cards if necessary.
|
| I have a suspicion that things may be different in the US, with
| many payment providers' testing environments simulate a typical
| domestic US scenario: credit cards and not debit, no 3d-secure,
| no strict payment jurisdiction restrictions, etc.
| weinzierl wrote:
| _" Isn't it what's everybody does in the industry?!"_
|
| Everybody, some do it manually, some let their QA people use
| their private credit cards - or so I've heard.
| boesboes wrote:
| Eh, no? I've never tested payment code using real payments.
| Ever. The idea of doing it with real payments is pretty out
| there in my book even :)
|
| Then again, every payment provider/bank I've integrated with,
| had decent testing end-points and we often even support them in
| production. i.e, you can select a staging/testing env of you
| provider to test order flow or whatever.
| willcipriano wrote:
| Then you just had the customer test it with a real payment.
|
| That's pretty out there in my book.
| BonoboIO wrote:
| Amen. If you don't test it, you will test it with your
| customer and will eventually fail.
| hypeatei wrote:
| Is doing a smoke test in prod with corporate cards bad
| practice?
|
| We are rolling out subscriptions with Stripe and an internal
| business unit is will actually be using the service so they put
| it on a company card. Basically they're our first live customer
| to test all the prod systems. No refunds or anything.
| mickeyp wrote:
| No, it is not bad practice. Only developers who don't care
| about money actually coming through the door -- the same ones
| that get caught up in a local maxima trying to perfect the
| imperfect -- say that it's bad and that you should not do it.
| kbolino wrote:
| Regardless of what developers think, the payment providers
| generally forbid it. For example, Stripe says:
|
| > Don't use real card details. The Stripe Services
| Agreement prohibits testing in live mode using real payment
| method details. Use your test API keys and the card numbers
| below.
|
| https://docs.stripe.com/testing
| mjr00 wrote:
| I think you're misunderstanding here; people are talking
| about a smoke "test" using a real credit card against the
| real production payment system, using production API
| keys/authentication/etc, with real money moving around.
| No payment provider forbids that.
| anonymoushn wrote:
| That's precisely what they all forbid?
| mjr00 wrote:
| They forbid using a valid credit card to make a purchase
| on a production system?
| toast0 wrote:
| Yes. By the letter of the agreement, you are not to use
| your cards to do test purchases against your account.
|
| You ocassionally see complaints about payment processors
| when microbusinesses do this and get banned. So it is
| something that does get checked ocassionally. (There's a
| top level comment about this)
|
| I think the payment processor doesn't want you to do it
| because you may issue many transactions and then refund
| them which incurs cost, or you may be using it for
| manufactured spend which incurs issuer ire. Maybe it's a
| brown M&M thing; if you didn't read that part of the
| agreement, you didn't read anything else, and they may as
| well kick you out early and avoid hassle.
| giobox wrote:
| Generally speaking, no one is getting banned from Stripe
| for the _occasional_ transaction tested in production,
| come on. If this was the case, virtually every company I
| 've ever worked at would be banned from Stripe. It's
| reasonable to confirm your system actually works once
| deployed outside of test environments.
|
| No disagreement from me that is what the letter of the
| Stripe service agreement says, but what happens in
| reality is clearly different. I take that rule as trying
| to encourage people to use the very good test
| environments Stripe offer, or to limit scale of test
| transactions in production, rather than trying to
| shutdown a paying user (the company) for trying a
| legitimate transaction in prod with a legitimate card. I
| have no idea why you would want to risk the first ever
| transaction in prod being performed by a real customer -
| why leave it to chance that it is not setup correctly?
|
| I have also been on calls with Stripe support staff where
| we tried a card transaction in production for testing
| purposes, FWIW.
| kbolino wrote:
| But it's _not_ a purchase. You 're not exchanging real
| goods or services for that money (unless your smoke tests
| run a lot deeper than mine). Your motives may be benign,
| but from a legal/regulatory perspective, it's a
| suspicious transaction.
| JoshTriplett wrote:
| Among other things, you're not allowed to use your own
| credit card to make a purchase where the money will come
| back to you, because credit cards want to charge cash
| advance rates for that.
| kbolino wrote:
| I can't find any clarification on this by a search alone.
| When I went looking for it in the actual services
| agreement, I couldn't even find any clause about testing
| at all.
| macNchz wrote:
| I have to imagine they'd only care if you were running
| significant volumes of test transactions and refunding
| thems, like if you were using live credentials in a dev
| environment.
|
| Either way I'd be hard pressed to deploy significant
| changes to payment-related code in production without at
| the very least seeing a real $1 charge go through and
| everything work as expected. The risk of a ToS
| enforcement for this seems much lower than the risk of
| some bad logic in an if (env == 'prd') making customers
| unable to give you money.
| kbolino wrote:
| This is my read on it as well, but I'd really like an
| official clarification.
|
| Rare production smoke tests are in a gray area. They may
| be technical violations but they're allowed to happen as
| long as they stay infrequent and above-board (company
| card, small amount, no chargeback, etc.).
| squirrel wrote:
| Also very interested. I wonder if we could get a comment
| from a Stripe person or a recently ex (like @patio11 ) to
| clarify what's allowed and what's just ignored.
| Rygian wrote:
| How do those places ever pass a PCI audit? One of the first
| things the auditor asks is "please show me proof that your
| testing is never done with real credit cards"
|
| (Unless they're getting their test environments PCI certified,
| which sounds like a waste of money.)
| hermanradtke wrote:
| Don't do it in staging/test environment. As a sibling
| commented stated: smoke test in production with corporate
| cards.
| bdcravens wrote:
| Most companies are never PCI audited because they're using a
| provider who already has been (like Stripe)
| jabart wrote:
| At volume you can no longer self-certify to be PCI SAQ-D.
| There are limits based on transaction count or volume.
| bdcravens wrote:
| Makes sense. What are those volume limits like? I suspect
| the lion's share of companies like Stripe are those under
| those limits (since the largest companies are willing to
| trade simplicity for better rates)
|
| edit: according to a quick search, it looks like it's 6M
| transactions/year to require an audit vs self-assessment
| closeparen wrote:
| >Every single place that I ever worked at in a past 20 years
| tests payments using real cards and real API endpoints. Yes,
| refunds cost a few pennies and sometimes can't be automated,
| but most payment providers simply do not offer testing APIs
| of a sufficient quality.
|
| I think this means _their own_ real-money credit cards, that
| they spend a few bucks on for testing purposes. Not customer
| credit-card data.
| hnthrow289570 wrote:
| >Isn't it what's everybody does in the industry?!
|
| We built against Stripe's sandbox and never had to test in
| production, so I never had to use a real credit card. It may
| have happened when going live for the first time, but that
| would have been two/three charges with one time payments and
| recurring payments (hardly what you'd call robust testing).
| Issues observed in production can usually be reproduced in the
| sandbox. There's some other caveats between the environments
| though that I'm forgetting but I don't think we ran into those.
|
| We also had an ACH payment provider (add your bank account,
| verify it, we deduct from it, etc.) that also had a sandbox and
| had no issues there either.
| arkh wrote:
| > We built against Stripe's sandbox and never had to test in
| production, so I never had to use a real credit card.
|
| Did it, got bit in the ass when some workflow was disabled in
| production and not in their sandbox. I don't recall the exact
| thing but always fun to push into production all sure of
| yourself to have to rollback fast and ponder why you get some
| fun message in your logs. At least the error message was
| clear about what we were doing being available only on the
| sandbox.
|
| It was some years ago so I don't remember if it was in Stripe
| Connect or during the mandatory 2FA rollout.
| zhoujianfu wrote:
| Slightly off-topic, but has anybody else seen an issue with
| stripe's live checkout flow when you collect a phone number?
| People paying with Apple Pay will frequently have the first 2
| or 3 digits doubled! So like 313105551212 will come through.
| Didn't seem to happen in the test environment.
|
| (Oh also sometimes it defaults users (in the U.S.) to
| Anguilla as their country code (also +1) but then gives users
| an error their phone number is invalid.)
| allknowingfrog wrote:
| We also built our Stripe integration using only the testing
| tools they provide. The first real payment in production was
| completed by our product guy, but only as a sanity check.
| Maybe we got lucky, or maybe Stripe is just that easy to
| integrate with, but we haven't been surprised by any
| production behavior.
| ghosty141 wrote:
| Standard practice is to use testing api for development and the
| real api for verification.
| nunez wrote:
| Absolutely not. Production is still an impenetrable fortress at
| a lot of places, or at least it's perceived to be.
| zitterbewegung wrote:
| From any engineering post I have seen about Uber it seems like
| some deceptive marketing / hiring tactic when it is analyzed .
| They seem to repurpose standard practices into a blog post
| (this one might sounds like it has been derived from one).
| Tade0 wrote:
| > but most payment providers simply do not offer testing APIs
| of a sufficient quality.
|
| Moreover, sometimes they vehemently oppose testing via real
| payments and reserve the right to cancel the contract should
| this happen.
|
| To this day I have a distaste for working with payments.
| ukd1 wrote:
| My last company, Rainforest QA, developed issuing virtual
| credit-card numbers for just this purpose - kinda like the ones
| used for privacy.com. Customers use them for testing in prod.
| Simple, effective - and testing the exact same flow as
| customers.
|
| Before this, we found a lot of teams using either their own
| corp cards, or pre-paid visa type things. All, a pain to manage
| balance wise.
|
| Seemingly, the biggest problem left with doing this is
| production metrics; these transactions in prod tend to affect
| the main metrics - either your own, or payment related things.
|
| I can't find anything current, but -
| https://www.businesswire.com/news/home/20180417005414/en/Rai...
| covers it.
| rileymichael wrote:
| > Situations when a testing endpoint has one set of bugs not
| found on production and vice versa used to be so ubiquitous in
| mid-2000 to mid-2010s
|
| Honestly that's still the case, at least with Adyen. At
| $pastJob we had a pretty robust regression suite that'd
| regularly run into breaking changes in their test environment
| (against _old_ versioned api endpoints!). We seriously
| questioned whether or not anyone else used them since they were
| never aware until we submitted tickets. This also falls apart
| as soon as you use any of their "special" features that require
| an account rep to enable, they just don't work in the sandbox.
|
| Another pain point is that the test payment methods offered are
| static. You can't setup cards for specific scenarios, e.g.
| tokenize, successful payment, then it expires -- you can only
| test an expired card as a one-off.
| whstl wrote:
| I just ran a couple hours ago into such a bug with Adyen, a
| couple payment methods have completely different behaviours
| in test vs production, and we ended up having to test in prod
| anyway.
| netdevnet wrote:
| Nooope! Services like stripe let you test the payment workflow
| in their test environment which works exactly like the
| production one but without the payments going outside Stripe.
|
| Let us not normalise bad practices
| serial_dev wrote:
| I've worked on adding Google Pay / Apple Pay to the mobile app
| of a large European ecommerce company, and that's more of less
| how we went about it.
|
| Start with the sandbox / test environments, once you get
| reasonable responses end to end, release the thing behind a
| feature flag. Backend moves slowly, so you add stuff to the
| mobile app that really belongs to the backend, but f it,
| because it's the only way you meet the deadline, you will
| convince someone on the backend later (aka never) that it's
| backend responsibility. Ask (pressure) the developers into
| buying some cheap stuff from your shop with their own credit
| cards, because the company is a behemoth and approving real
| credit cards for testing would just take ages and you want to
| release yesterday. It annoys you, but you realize that 5x5
| euros is worth getting this done rather than start fighting a
| losing battle against company processes. Cancelation is
| possible, but it will take a couple of days. If there's any
| issue, you debug it across a bunch of teams and/or companies.
| Things start to work most of the time, time to release the
| stuff to x percent of your users. Check analytics and error
| logs frequently. Some production users got their payments
| through, increase rollout percentage. You discover more and
| more undocumented error codes, you improve the error messages
| to the users so that they don't retry 10 times with a card
| without sufficient funds. After a couple weeks, things start to
| stabilize, you move on to a new feature...
|
| The test environments were so complicated and had so many
| caveats that whenever I had to do something, I had to re-read
| the docs and our notes to know all the "traps" we already
| discovered. For the people who didn't work on this payment
| feature from start to finish, testing in the officially
| recommended test channels were hopeless.
| CapstanRoller wrote:
| >Ask (pressure) the developers into buying some cheap stuff
| from your shop with their own credit cards
|
| This is illegal. I've always refused such "requests" and
| asked for a company expense card.
| ceejayoz wrote:
| Illegal in what jurisdiction?
| rmbyrro wrote:
| Any jurisdiction besides China and North Korea.
| ceejayoz wrote:
| I don't think so.
|
| It would be illegal not to _reimburse_ the expense in,
| say, California; https://casetext.com/statute/california-
| codes/california-lab...
| xwolfi wrote:
| Im in China, fun fact, it is also illegal ! Now is it
| enforced, probably only a little bit more than in
| California !
| fny wrote:
| California is an example, but I'm unclear about other
| states. Honestly, I'm unclear across the board because
| there are a lot of employee made purchases that are
| conditions of employment (phone, computer), and it could
| be argued that this purchase is a similar necessity
| especially if it'll be refunded.
| insane_dreamer wrote:
| If the employee can get it refunded with an expense report,
| like any other work expense then in my experience (in
| California), it's not illegal. I've made plenty of work
| expenditures with my personal CC that I get reimbursed from
| the company with an expense report. But *pressuring*
| employees to do it is plain wrong (and may be illegal).
| singleshot_ wrote:
| I seem to remember about 4111 1111 1111 1111 times that I
| tested a payment system with a card that wasn't real, although
| I acknowledge that when I was done convincing myself that I was
| a good programmer, I would almost always be disavowed of this
| notion after using a real number.
| hot_gril wrote:
| This is also true for cryptocurrency. There's always a testnet,
| you use it whenever possible cause xact fees are high, but it
| never works the same as real.
| hoffs wrote:
| Doing penny testing yourself is different from letting a chunk
| of your user base test it
| ComputerGuru wrote:
| Stripe considers using a real card to test a zero-tolerance
| fireable offence.
|
| I, as the name on record for an organization with Stripe, made
| an actual, legitimate payment to said organization (and did not
| refund it). Stripe's automated system caught the payment and
| terminated my account automatically by the time I woke up the
| next morning; fortunately I was able to reach a real human and
| explain that in addition to working for the non-profit, I was
| also donating to it, and they begrudgingly restored the
| account.
| JoosToopit wrote:
| Pure graphomania.
|
| Look, ma, I'm a blogger! Wait, no scratch that - I'm a WRITER!
| brynb wrote:
| i've built tons of very intricate payments systems over the past
| 10 years and i honestly have no idea how "payments engineer" is
| even worthy of a distinct job title. it's a thing people do in
| the course of building products. ridiculous
| takumo wrote:
| Yes, this article is probably longer and fluffier than it needs
| to be but there are some real truths here.
|
| Payments are one of the original service orientated architecture
| systems, in production your payment is processed by at least
| three or four parties each of which will call several systems or
| sub-systems to process a payment.
|
| This method clearly works for Uber, who have a lot of payments
| going through their systems most of which are of a relatively
| small value. Dropping a payment and either asking the user to pay
| via a different option or simply writing off the revenue for a
| handful of transactions is probably workable for them.
|
| I have the opposite, the number of transactions we process is
| relatively low, but the average value of these transactions is
| high, well in excess of 1000 USD. This leads to the following
| issue:
|
| 1. Screwing up a payment and asking the user to try again can be
| a big hit to user confidence. 2. We can't write off even a single
| payment/transaction, they're too high value to write-off. 3.
| Processing fees and refunds for making test transactions in
| production are too expensive. If a test costs more than $10 (to
| test in production we must test with production transaction
| values) that's going to rack up quickly.
| lucw wrote:
| Reminder that if you test a live payment on a new Stripe
| deployment, you will get INSTANTLY banned. Don't do a live test
| with a credit card in your name !!
| mattgreenrocks wrote:
| It seems entirely natural to do this. What should you do
| instead?
| _heimdall wrote:
| A couple large corporations I worked for had two instances of
| prod, geographically isolated with one acting as a fallback in
| case the primary went down. This isn't particularly novel at all,
| but what I was always interested in was using a similar setup for
| testing production prior to flipping the release live.
|
| Effectively you'd just have prod and staging with identical
| deployment configuration. The benefit would be promoting the
| exact staging release to prod as soon as tests pass.
|
| That said, I've never tried this and I'm sure there are good
| arguments for avoiding the added complexity of regularly flipping
| production between two different environments.
| robertlagrant wrote:
| > I really like how Charity Majors put it: "staging is just a
| glorified laptop". Only production is production.
|
| Production is also just a glorified laptop.
| mannyv wrote:
| To be honest, errors in payment processing are hard to create and
| reproduce in test. Plus there are errors that apparently never
| occur anywhere except in production.
|
| So yeah, "testing" in production is normal for all payment
| systems.
| throwaway82498 wrote:
| Uber had, and probably still has, a sophisticated setup for
| directing prod traffic for specific requests to/from developer
| laptops, for isolating test tenancies in prod services, for
| simulating trips using test tenancies, for automatically
| detecting and rolling back deployments based on everything from
| the usual observability metrics to black box testing against
| prod, and last but not least, good unit test coverage.
|
| I bet their payments team runs code before it gets deployed. The
| article seems to imply that Uber engineers don't bother to test
| code before they land it, when in reality they _do_ test it,
| _and_ they also catch other stuff afterwards too.
| TYPE_FASTER wrote:
| We worked with a payment processor to implement billing for our
| services via credit card. According to the payment processor, the
| QA environment for one of the major credit cards had been broken
| for a while, so we tested in production.
|
| We were testing billing customers who were going to pay us, so
| putting a small charge on a corporate card that was going to come
| back to us wasn't a big deal, I just remember being slightly
| surprised that testing something like credit card payments was
| done against the production environment.
| lofaszvanitt wrote:
| Payment systems are the blogs of the early 2000s.
| ninju wrote:
| Testing in Production
|
| The Crowdstrike philosophy /s
| trollied wrote:
| Everybody has a testing environment. Some people are lucky enough
| enough to have a totally separate environment to run production
| in.
| kelsey98765431 wrote:
| my hot take is to test in every environment... what a concept.
| the even deeper hot take here is to reimplement mocks of your
| integrated environments AND THEN IMPLEMENT THEIR SYSTEMS! the
| process of good testing has a side effect of eventually
| eliminating technical debt, because those same set of tests that
| ensure your application is working can test if your
| reimplementation of your upstream integration is working! ta da
| you are now a growth company.
| noiv wrote:
| Just an idea, can't you just swap staging and production? So,
| actually the system you've tested goes live by switching nothing
| more than a pointer (no deployment involved).
|
| Won't raising support cost at some point suggest it's cheaper
| having two swappable live systems than the alternative?
| graeme wrote:
| What do people with smaller companies do to test with real cards?
| The terms of credit cards usually disallow using your own card to
| make a purchase from yourself.
| tqi wrote:
| > For Uber, every deployment is an experiment
|
| Blindly experimenting without a clear hypothesis is a great way
| to ship statistical noise.
| ram_rar wrote:
| Articles like these need TL;DR Testing in prod is a tale as old
| as time.
|
| It would have been more insightful to cover the underlying
| infra/tech that enables this seamlessly.
| cuttysnark wrote:
| cc: 4242424242424242
|
| cvc: 424
|
| exp: 2/4/24
|
| Fond memories of speed-running the checkout flow in Stripe
| sandbox.
___________________________________________________________________
(page generated 2024-08-07 23:01 UTC)