[HN Gopher] Deactivating an API, one step at a time
___________________________________________________________________
Deactivating an API, one step at a time
Author : bpedro
Score : 46 points
Date : 2024-07-05 09:04 UTC (4 days ago)
(HTM) web link (apichangelog.substack.com)
(TXT) w3m dump (apichangelog.substack.com)
| pocketarc wrote:
| Something that can be extremely useful as well in situations like
| this, is doing API brownouts, toward the later stages of the
| process. Disable the API for short stretches of time, on the way
| to disabling it entirely, to give consumers who might not be
| keeping up with changes a way to be alerted (they'll notice the
| downtime).
| simonw wrote:
| I really like the API brownouts trick. GitHub have been doing
| this for years, a few examples:
|
| - https://developer.github.com/changes/2018-11-05-github-
| servi...
|
| - https://github.blog/changelog/2021-08-10-brownout-notice-
| api...
| mhink wrote:
| I was thinking about this kind of thing. Another idea might be
| to introduce artificial latency and gradually increase it over
| time? Maybe dial up rate-limiting? I'm not really sure if this
| is a better or worse idea, though.
| pocketarc wrote:
| That is a good idea, but I don't think it does enough to
| serve the purpose - the point of brownouts is to trigger
| their error/alert system. If the API is just being slower
| than usual, it won't trigger anything. Even if a human was
| reviewing it manually (which is quite unlikely), they would
| only think "oh, their API's really getting slower these days,
| sad".
|
| There'd be nothing that indicates "The API is going to get
| shut down in a month and I need to move off of it ASAP!".
| Random, intermittent API failures would lead you to go check
| the API status out, and in the process you'd find out "oh,
| this API is going away".
|
| Edit: On the point of rate limiting, I think the problem with
| it is that it'll affect everyone using the API all the time,
| not just during the brownout period. It effectively shuts the
| API down for everyone still using it (if the rate limit is
| too low, and if it isn't, then it won't be noticed by low-use
| consumers).
| hooverd wrote:
| Hey, some of us look at P95 latency.
| Joel_Mckay wrote:
| Or instead of losing paying customers:
|
| 1. design API accounts to include a preferred server and
| default server (handy to explicitly load balance, or
| dynamically bounce users to specified servers.)
|
| 2. design clients to have a timed service lifecycle (expiry 2
| weeks prior to cert expiry). Then enter a semi-dormant mode
| until valid signed updates succeed.
|
| 3. add a random timed daily update check, and begin reassigning
| the users to the new API after updates install properly. Also,
| warn users the migration will happen 2 months before it is set
| to launch (do a few random A/B tests the first day).
|
| 4. Never rely on people to act, or not act for keeping
| infrastructure running. You are not going to be able to
| manually update 30000 legacy hosts with a single team. Worst
| case scenario you must auto-reconfigure the clients for a
| standalone offline mode... so the next user of the IP doesn't
| get hammered by failed connection retry attempts.
|
| Brownouts won't work for cached-edge systems designed to
| reconcile month long intermittent outages. i.e. systems that
| were designed to handle DDoS, worms, and acts of clod...
|
| Have a nice day, =3
| bogdan-lab wrote:
| Yes, I completely agree. The story sounds like fairytale:
| migration was announced to happen in 3 months and in 4 months
| it was done by removing old API.
|
| Where are all those clients, who are happy with current API
| performance and do not want to spend their money on making
| API owners life better? What happened to them? Did the
| company just decide to let those clients go?
| Joel_Mckay wrote:
| Collecting clients other people hosed is an easy business.
| Except, entrenched incompetence may still pine for the
| convenience of a quick sometimes-broken kludge (some folks
| expect everything to be glitched half the time).
|
| I definitely understand why some techs just stop caring
| about customer opinions. You'll know when you are in a
| senior role when one starts to fantasize about being a
| Plumber. =3
| simonw wrote:
| Yeah, one of the biggest problems with API deprecation is
| that you have zero control over the roadmap of your
| clients.
|
| If they can't spare the engineering time in the next six
| months to carry out the upgrade (and you aren't 100%
| mission critical to their business) they're not going to do
| that no matter how much you bug them.
|
| Depending on how old the integration is they may not even
| still employ the engineers who built the first version,
| which makes it even harder for them to roadmap the work.
| Joel_Mckay wrote:
| In general, most managers just hire external firms to
| attempt a new version:
|
| 1. if it succeeds, the manager looks smart given your
| services are no longer relevant to their operations
|
| 2. if it fails, the manager looks smart as they are not
| responsible for the external firms business operations
|
| It is a win-win situation for the client, but 100% bad
| for your business... =3
| michaelt wrote:
| _> It might be that you want to replace it with a new, more
| capable version_
|
| If you're truly replacing your API with a new, more capable
| version there's a much better option, in my experience.
|
| Roll out your new API, and replace your old API's implementation
| with a proxy that calls through to the new API.
|
| The proxy will need very little maintenance, as all it's doing is
| connecting one fixed, stable API (your old one) to another fixed,
| stable API (your new one). Lock it down to only your old
| customers, if you want.
|
| The support costs will be basically zero, and your existing
| paying customers will thank you for respecting their time with a
| dependable, low-churn API.
| n4r9 wrote:
| What if the new API is more capable precisely because the
| parameters have been completely redesigned?
| cinntaile wrote:
| Surely you can keep it backwards compatible AND give people
| access to the new good stuff?
| smallnix wrote:
| Not if the API reflects a fundamental change in flows. E.g.
| not fun to proxy a sync API into the new async one which
| splits a single operation into three.
| taeric wrote:
| The presumption would be that there is a mapping from the old
| parameters to the new? Not necessarily a simple shift/rename,
| but you aren't doing your users any favors if they have to
| speak an incompatible vocabulary to make use of the new
| system.
| tuyguntn wrote:
| This sounds a good idea in theory, but not sure about in real
| world.
|
| Consider having 4 versions of API, will you create 3 levels of
| proxies or each time re-write your old proxy?
|
| Sometimes, more capable mean additional side effects not
| existing in old contract (e.g. send an email about order), in
| that case your new API should be flexible to accommodate
| configuration of side effects, adding more maintenance cost to
| your eng. org
| tuyguntn wrote:
| also, the assumption here is "more capable = same features +
| something new (no removals)", sometimes you have to delete
| some features as part of new API (e.g. stop creating audit
| log entry for an action).
| tikhonj wrote:
| Conceptually, we should see deprecating a feature as a
| separate question from moving to a new API. So in this
| scenario, you would manage deprecating functionality in the
| same way regardless of whether you're also rolling out a
| new API at the same time.
| chipdart wrote:
| > Roll out your new API, and replace your old API's
| implementation with a proxy that calls through to the new API.
|
| I don't think you understood the problem you're commenting on.
|
| The problem is not that you have a new fancy API version you
| expect users to consume. The problem is that you need to shut
| down the old API without causing customer or business impact.
|
| It makes little to no sense to pile up technical debt and extra
| maintenance work to keep around a specific configuration in an
| API Gateway if your goal is to get people to stop using it so
| that you don't have to maintain N+1 versions of your service.
| Your vague observation of how much work a API Gateway config
| takes is meaningless because you have no idea the impact of any
| change to the API will have on the older version fed through
| adapters. I mean, even changes in performance can be
| disastrous. And even under the unlikely case a API Gateway
| applying a transformation is enough to sunset the old
| version,how is that an improvement on maintenance? You now have
| a new abstraction layer that requires testing and maintenance,
| and a multitude of scenarios you need to validate independently
| whenever a change is made. And what answer do you have to the
| question on what to do if a commit works in the new version of
| the API but introduces a regression on the old version?
|
| The truth of the matter is that there are only a couple of
| sunsetting strategies that work, which is:
|
| * In internal service calls, negotiate a sunsetting strategy
| with other internal teams. The deadline will invariably be
| pushed forward each and every single time.
|
| * In external service calls, you can announce to the world
| you're sunsetting the API and advertise it on Times Square,and
| still a hefty share of your customer base won't know and will
| still be caught by surprise. The only strategy that works is
| graceful degradation: start returning 404s and 410s
| periodically and dial it up until clients feel the need to move
| on. Dial it back when appropriate to get your point across
| without denying service, but understand that it will be
| impossible for some clients to change.
| oasisbob wrote:
| The post you're replying to is describing a straightforward
| implementation of the strangler pattern.
|
| It's a valid and useful technique. Even if you disfavor it, I
| don't see why you feel the need to judge the comprehension
| skills of the commenter.
|
| https://www.redhat.com/architect/pros-and-cons-strangler-
| arc...
| petesergeant wrote:
| > In addition to offering human-understandable communication, I
| asked the API producer to add the Deprecation HTTP header field
| to all responses
|
| Cute, but, I question the value
| bpedro wrote:
| Great question.
|
| As a consumer, you can set up an alert when any of your API
| requests has a deprecation or sunset HTTP header. You'd know
| immediately if any of the APIs you depend on is about to be
| deactivated.
| chipdart wrote:
| > Cute, but, I question the value
|
| I came here just to say that. What an half-baked idea. It might
| be trivial to mindlessly bolt on response headers, but if the
| goal is that the mechanism needs to be impactful and have
| consequences then the response header is a big red herring as
| you're actually relying on clients to implement support for
| sunsetting the endpoints. If that's the case then you already
| have meaningful mechanisms, such as passing this sort of
| metadata in responses to requests for the root resource. This
| is something that pretty much any HATEOAS spec already
| supports.
| athenot wrote:
| Other strategies that could come in handy before completely
| shutting down the API:
|
| - Rate-limit the API, with increasing aggressiveness until you're
| down to 0 requests per unit of time;
|
| - Introduce latency in serving the requests (assuming your edge
| can handle the increased volume of open connections).
|
| Both of these introduce _gradual degradation_ of the old API,
| without outright killing the business functionality that
| recalcitrant customers are nevertheless reliant upon. It helps
| spur a bit of urgency to switch to the new API, while remaining
| nice. Many (enterprise) customers will wait until the last minute
| to switch: essentially they are having to put in work without any
| tangible feature gain--at least from their perspective.
|
| Regardless of the strategy, one other point I'd add is to monitor
| actual use of the API; if important customers are still actively
| using the old API, it would be unwise to shut it off.
| croemer wrote:
| I think GitHub actions cause errors for a few minutes to alert,
| one could do that also with an API.
| Suppafly wrote:
| Man I really hate the idea of "Let's make a thing that works
| shittier so that people switch to the new thing". If you're in
| a situation where you have customer's just be honest about the
| changes coming and give them enough runway to get the changes
| made.
| abraae wrote:
| Yep. If you gradually degrade performance, and word of your
| intentions fails to make it to the right people, the victim
| may well have to devote significant effort to trying to work
| out what's going on. Boy will they be pissed when they find
| out.
| bongodongobob wrote:
| This is standard practice though. You can only communicate
| it so much. If your customers ignore it, that's on them.
| You either have a hard cutoff date or you introduce
| failures or throttling. The latter is friendlier.
| bongodongobob wrote:
| Yeah you do that and you'll still have customers that ignore
| it. That's why you have to slowly degrade it at a
| communicated point in time. You're not being sneaky about it,
| it's planned. It's that or you dedicate a ton of manpower to
| hold everyone's hand for a hard cutoff which will _always_ be
| painful.
| lazyasciiart wrote:
| We have customers using a client that was officially
| deprecated in 2020 and stopped working altogether for six
| months in 2022, along with as much announcement noise as we
| could make. A year later I was surprised to realize the
| environment change that had blocked it was gone...and people
| were still using it.
| its_ethan wrote:
| It would be really cool to see a graph of the API usage over time
| with markers showing when each "stage" was occurring. I'm
| wondering if there were significant dips shortly after each
| stage, or if it was more of a gradual decline?
|
| It would also be interesting to know if the API was still being
| used up until the final stage? Were there any ramifications/
| angry customers at the door after that?
| bpedro wrote:
| These are great suggestions.
|
| I'll write about it the next time I'm in a similar situation.
| bornfreddy wrote:
| Given the timeline of just 4 months, I would expect 90% of
| customers not migrating in time. The percent of those who
| migrated to the new API after that depends on how vital this
| service is to them, it would be interesting to see the rate of
| lost businesses. I know that if a service provider did this to
| me, I would prefer migrating to their competition, all things
| being equal.
___________________________________________________________________
(page generated 2024-07-09 23:00 UTC)