[HN Gopher] Ask HN: Handling customers that want feature preview...
___________________________________________________________________
Ask HN: Handling customers that want feature previewing?
Curious how you folks manage the release process with larger
customers that ask for advance notice (ex: 2 weeks) or previewing
of any features before release or general availability? We have a
single sandbox environment currently, but different customers are
asking for varying amounts of delay before feature release. How do
you manage bug fix deploys in these cases as well?
Author : cbono1
Score : 45 points
Date : 2021-06-10 11:35 UTC (1 days ago)
| pkrotich wrote:
| We use release branches (git) for such cases - mainly because our
| deployment is based-off release branches, thus gives us
| flexibility to move a customer to a pre-release or even have then
| stay at a particular release forever. Bug fixes can be back
| ported as needed.
|
| Tried feature flags but it introduced complexity.
|
| I wouldn't charge more as others are suggesting- bugs you'll
| address before general availability is worth the trouble.
| antonpirker wrote:
| We create a separate branch for the customer and preview this
| branch with wunderpreview and give the customer the link to the
| preview.
|
| Every once in a while we merge new stuff in the customers branch
| so she has always the latest stuff in her preview.
| leesalminen wrote:
| I also had a handful of customers like this. We ended up creating
| an opt-in "beta" channel that we treated as a first-class citizen
| along with "prod".
|
| Basically we had 2 production stacks with CI/CD for each. The
| only shared thing was the AWS ALB. Our customers each have their
| own subdomain so we used that to point to different EC2 ASGs.
|
| We would continue pushing bug fixes to "prod". Once a beta period
| ended, we merged to master and re-deployed master to both prod
| and beta channels. Rinse and repeat.
|
| It works out pretty well. Occasionally there's a really important
| bug fix that we have to cherry pick from prod to beta, but it's
| rare enough to not be a huge concern.
|
| Oh, and, charge them for it. I never got any pushback.
| cbono1 wrote:
| Thanks for sharing!
|
| Was the beta channel that you had in the pipeline for prod?
| i.e. prod would have no chance of getting ahead of beta because
| prod was always deployed based on what was last on the beta
| release?
|
| Hot or bug fixing is an interesting topic because you could end
| up with diverging histories if you're not careful. Did you try
| to maintain `master` as the single source of truth or did you
| make separate release and deploy branches for each environment?
| leesalminen wrote:
| We created rc/xxx branches in git. Developers would merge
| feature branches destined for beta to the rc. The rc gets
| deployed to the beta environment. Later, we would merge the
| rc to master and then deploy master to both prod and beta
| environments.
|
| Prod would be ahead of beta during hot fixes. Though we try
| and avoid it, it did happen occasionally.
|
| You are correct that it took some extra effort to keep it all
| clean and manageable.
| ghiculescu wrote:
| Lots of good answers below on how to make it work, how you should
| charge more, etc.
|
| Remember that saying no is also an option. You aren't obliged to
| say yes just because someone asks for something you don't sell,
| even if they are willing to pay.
|
| For most SaaS there's no logical reason why having a feature for
| 2 extra weeks would be an enormous advantage to a customer. In my
| experience what this question really is is "can we make last
| minute change requests to a feature you thought was done". I'm
| not keen to guarantee that, so it's a simple no, sorry, we can't
| do that. It's yet to be a deal breaker.
| cosmie wrote:
| > For most SaaS there's no logical reason why having a feature
| for 2 extra weeks would be an enormous advantage to a customer.
| In my experience what this question really is is "can we make
| last minute change requests to a feature you thought was done".
|
| If you frame it as early access, then people are apt to try to
| throw their wishlist at you, thinking it's still "not official
| yet" and easy to change.
|
| But based on what OP asked, I'm not sure that's the primary
| reason. It seems more like his larger clients are finding the
| changes disruptive enough that they're wanting advanced notice
| to prepare for them. So integrating a lot of the suggestions
| here can very easily address that issue, while also providing
| leverage points for client management and sales.
|
| @OP - don't think about it in terms of preview access or
| letting them into your internal sandbox environment(s). But
| rather, restructure your releases to decouple the release from
| the account migration. Using feature flags and such then allow
| end users to determine whether they immediately migrate to the
| new feature when it's available, or if they continue with the
| status quo and wait a few weeks to familiarize themselves with
| the changes before opting into them.
|
| For end users - this gives them room to both continue with
| their day-to-day needs using the process they're already
| familiar with, while giving them flexibility to familiarize
| themselves with the changes and prepare (in whatever form they
| need) before committing to them.
|
| For the tech team - there's an initial lift to adjust the app
| architecture to support this type of usage, but insulates them
| these sorts of client requests in the future since the app now
| supports these types of opt-in migrations within the live
| environment. If a user wants a sandbox account, it can be
| provisioned as a normal account in the live environment but
| with sandbox-like restrictions applied to it.
|
| For the sales/account teams - this can be used as a negotiation
| point. You can position it as normal accounts get auto-migrated
| to new features upon release, but enterprise clients can pay
| for the ability to control their own release/migration timing
| via these opt-in feature gates. And a sandbox functionality
| (which provisions a new account using the current configuration
| of their existing account, but with applied sandbox-like
| limitations) allows them to both prepare for it _and_ test it
| in a controlled environment before enabling it on their primary
| /production account. That sandbox capability is independent of
| the opt-in capability, and a second point of negotiation.
|
| And since your application now supports both of these directly
| in production and doesn't require bespoke work or custom
| environments from your tech team to support, the incremental
| cost is pretty minimum to give it to a particular client. Which
| is hugely useful for a sales team when closing larger accounts.
| They can give it away without much hassle if they have to in
| order to close a deal or provide value-add during renewal,
| while simultaneously capturing incremental revenue from clients
| that are willing to pay substantially for these sorts of
| things.
| cbono1 wrote:
| Thanks for the additional context here.
|
| I think some of the administrative pain can come from having
| a 'sandbox' or 'qa' tier beyond dev (which in this case would
| be customer facing) before production.
|
| If data separation isn't required/essential, seems as though
| a a separate environment is actually unnecessary and feature
| gating could be used. But if data separation from other
| customers is required, then new infrastructure should be
| introduced via separate CICD if I follow correctly?
| cosmie wrote:
| > I think some of the administrative pain can come from
| having a 'sandbox' or 'qa' tier beyond dev (which in this
| case would be customer facing) before production.
|
| Pretty much. It's hard to guess at the reason why without
| knowing the product, but for whatever reason customers are
| wanting access to new features/releases before using them.
| Which likely means that they need to do something on their
| end every time you release (validation, update internal
| training/process documentation, etc). And it's causing
| disruption when things change outside of their control. So
| they're wanting more control over that change timing, to
| smooth out the impact of those releases as far as their
| usage goes.
|
| If your release codebase can only support a single
| "production" version of a feature at once, you end up
| having to provision entirely new environments to hold a
| client back. And again, for every client wanting this.
| Because the timing they want isn't going to be convenient
| enough to be the same.
|
| So instead, structure things so that you can run both the
| old and new version of "the thing" simultaneously in
| production, and give the client self-service ability to
| activate the new version when they're ready. That way your
| production environment itself is able to accommodate these
| requests and client's can opt-in when they're
| ready/comfortable to.
|
| The biggest challenge to doing this tends to be
| architecting your application so you can push releases to
| the backend independently to releases in the frontend.
| Which can generally be handled by routing things through
| middleware that provides a stable response for a given
| feature version. If the interface is using the latest
| version, the middleware is basically just a stub that does
| nothing. But if the interface is using a prior version,
| then the middleware accepts the request signature of the
| prior version, translates into what's needed to execute on
| the new codebase, and returns a response that's consistent
| with the prior version. It's a few years old, but here's[1]
| an example of how we handled the data-model aspect of this
| sort of setup at a prior company
|
| So in effect you're actually upgrading all customers to the
| new release at once, but from the customer perspective
| everything still looks/feels/act the same as they're used
| to until they "upgrade/opt-in" to the new release.
|
| It's a pretty major lift to implement this architecture
| (particularly on an existing codebase), but maintaining it
| is easy to incorporate into the regular development
| process. And completely eliminates the operational and
| customer management headaches you're having now.
|
| With this architecture, it doesn't really matter if you're
| running isolated deployments or a multi-tenant
| architecture. You only ever have a single version of
| production in both cases. The "sandbox" isn't a new
| environment, but a new production account that copies over
| the configuration/setup of their existing one while
| applying sandbox-like restrictions to it. This gives them a
| temporary space to prepare for the change so they're
| prepared when they flip the switch on their primary
| account.
|
| [1] https://github.com/TomFrost/Vers
| cosmodisk wrote:
| I really like how Salesforce do it: they make an announcement
| that a feature/fix will be available from day x and depending on
| its importance, they then give 100,200,500 days or whatever to
| install it or it gets installed automatically on the deadline
| day. This gives plenty of notice to customers to prepare( updates
| can break some customisation on client's instance).
| plasma wrote:
| Have you clarified why they want this notice?
|
| In my experience it's because if you roll out a new feature (or
| non-trivial UX change), that can wreck someone's day and cause
| internal support issues into the company that uses your product.
|
| Consider your rollout of new features to be least impacting,
| don't confuse someone who's using your product to get work done
| by changing something that breaks a workflow that worked
| yesterday etc.
|
| You can put new features in new sections, or need special users
| to activate them.
|
| In some cases we would show screenshots of upcoming features to
| hear about any other feedback, but otherwise focused on new
| features not being disruptive.
| cbono1 wrote:
| Thanks for the reply - yea these are great points to consider
| and raised a few of them. Ideally rollouts benefit all
| customers, and the concern is that there is some possibility
| that something might not jive with them and they want the
| ability to do testing ahead of time.
|
| Seems like putting as many things that might risk this under
| settings/configurable as possible? This does extend development
| to have to gate everything in this way though.
| plasma wrote:
| In my experience it was just not doing huge UX
| reorganisations without prior notice.
|
| I'd dig deeper into the why with your customers, after we
| understood moving buttons around causes people's day to be
| hard we did it only when really justified etc.
|
| I wouldn't hold up releases for customers, but do dig into
| the why more with your customers about why they want this
| notice, you may find it's just an assurance something like a
| workflow isn't totally different next Monday morning.
| runwaykid wrote:
| One way is to make use of AppConfig to store customerwise feature
| roll out plan. To save cost, add 15 minute caching on it (the
| thick client comes with the local application caching feature)
| dsr_ wrote:
| Context: financial SaaS of interest to banks, brokerages and
| RIAs; no direct-to-consumer at this time.
|
| We have two special case preview environments. They're set up
| just like production, but with lower capacity and no SLA.
|
| The first one is general previews of what we're planning on
| releasing; it's a normal part of our release process and every
| release is there for 2-14 days before going to production. All
| customers have access; some use it extensively, and some
| basically ignore it.
|
| The second one is for specific in-development features; in theory
| we can spin up multiples of this, but in practice we don't want
| to diverge too wildly. Generally we showcase one feature at a
| time, and we gate the features by customers.
| theoblank wrote:
| I would seriously consider not allowing special treatment in
| cases like this. In my experience things like this can be a
| slippery slope. Decide what is most broadly reasonable for both
| your company and your customers, communicate it well, and then
| spend time and energy on your core product.
| tboyd47 wrote:
| You need at least 2 sandbox environments: one that's more
| development-oriented (little to no restrictions on merging), and
| one that's more release-oriented (as close as possible to
| production).
|
| Ideally you have 3, and the third one is not used except to demo
| new features to customers AFTER they have been released (and
| perhaps, hidden behind a feature flag).
|
| If you only have one sandbox, then it becomes a brawl between
| development, sales, and management.
|
| More than three is overdoing it and assuming a technical burden
| for no benefit. (No, you don't need the capacity to spin up
| sandboxes willy-nilly. Three is enough.)
| Arubis wrote:
| Most of the discussion here will be technical, especially around
| feature flags and friends. That's fine, but please don't miss out
| that this is a clear signal to charge these customers a lot more
| money.
| talolard wrote:
| I'll offer an illustration. We sell our SaaS at X. We have a
| customer that needed stronger SLAs and servers located in
| Europe. They pay 8X and (I think) both sides feel like they got
| a great deal.
|
| So to paraphrase Arubis, this is a business question,not a tech
| question. At what price point is the technical investment
| covered ?
| ambivalents wrote:
| > a clear signal to charge these customers a lot more money.
|
| Interesting, can you explain this a little more? Some of our
| customers chomp at the bit for particular features, and when we
| can we give them early access. We do not charge them more for
| arguably huge amounts of value add to the product.
|
| I wonder how to have these types of conversation, internally
| and with users. I'm a PM but have a direct line to management.
| I'm frustrated with how much money I believe we are leaving on
| the table, even if it doesn't directly benefit me.
| lmeyerov wrote:
| enterprise support contracts provide access to early release
| builds / feature flags / etc, or require signing up to be a
| design partner (ex: logo + joint talks)
| feifan wrote:
| If you also have a direct line to your users, you could try
| asking them how much they'd be willing to pay for either some
| new feature that's about to come out, or for the right to
| early-access. Get maybe 3-5 data points and you should have a
| pretty solid case to management (either to charge
| retroactively for existing features, to charge new users for
| existing features, or to charge all users for upcoming
| features).
| wonder_er wrote:
| rather than ask "how much they'd be willing to pay", it
| might be more compelling to simply say "It will cost $X for
| early access, limited to X slots, in exchange, {we do such-
| and-such}."
|
| If they say "OK, great", you've got immediate revenue to
| bring back to your management, and a strong case to
| continue doing more of the same.
|
| Customers routinely over-and-under-estimate future value,
| compared to what they're then willing to pay.
|
| They're often wrong in both directions, they usually won't
| pay as much as they think they will, _and_ they'll often
| pay far more than they think you will.
|
| There are many, many reasons for both kinds of errors, many
| of which can be de-risked in advance and safely
| ameliorated.
| rm_-rf_slash wrote:
| If a customer requests a changelog in advance that's one
| thing (should be as simply as fine-copy-paste-email), but
| setting up a preview sandbox is work that should be
| negotiated by a contract if it wasn't already signed or in
| the terms of service.
| zerkten wrote:
| You are right that previews can be a time sink. It
| justifies some strategic thinking because the blast radius
| can be big.
|
| Someone can forget to have a separate set of terms and
| suddenly the support team are on the hook for supporting
| previews to a greater degree than expected. If features
| aren't locked then suddenly features get jammed in because
| the preview is a good hook for customers who haven't yet
| acquired your product.
| avalys wrote:
| I don't think this is about charging for new features - it's
| almost about the opposite.
|
| I read this as "A customer that asks for advance notice of
| feature changes is likely to be a big enterprise customer
| that values stability and predictability, and the fact that
| they're asking suggests your product is important enough to
| their process to matter, so you should take this opportunity
| to charge them a ton of money for an 'enterprise support
| plan' which includes advance notice of new releases, etc."
|
| As the GP said, it's a signal that they're a certain type of
| customer.
| exolymph wrote:
| As a smaller-scale contractor I just tell people what I
| charge. If they're not willing to play ball* get to premium
| compensation, no deal. It's cliche but BATNA is everything.
|
| *I'm willing to play ball too, e.g. for a staff position I
| might take extra vacation in lieu of more cash, tradeoffs
| like that. But basically I'm not gonna take on extra hassle
| without being incentivized -- because I don't have to and
| don't want to, all there is to it
| williamdclt wrote:
| That's customer support effort: you have to have a separate
| feature release communication for these beta users, and very
| probably some support when things go wrong because they're
| using a beta product.
|
| That's also engineering effort: you need to maintain feature
| flags or a different environment, which is not cheap in the
| long run
| pjc50 wrote:
| I think the ways to approach it are:
|
| - this incurs real costs, so we should charge for this. (The
| charge doesn't have to be related to the costs! But if you're
| doing something extra there should be an extra line on the
| invoice)
|
| - this customer wants something that other customers don't
| have. There's all sorts of real-world services where you can
| pay to board the plane first or get to the front of the
| concert. These may be several times more expensive than the
| regular service. That lets you phrase it in fawning terms
| ("you can join our Super Elite Ultra Platinum Preview
| Fellows") and thereby extract much more money.
| bkuehl wrote:
| There is a lot of talk of feature flags but that only works when
| you can realistically delay your feature 2 weeks so that the
| 'special' set of clients can see it early. What we've seen that
| has worked is essentially demoing the feature in staging/test to
| stakeholders at the major/demanding client. They are happy that
| they aren't taken by surprise and then we provide training
| materials for major feature/UI changes to their users. Never have
| had an issue, client is happy and essentially zero technical
| involvement.
| bri3d wrote:
| Once you know you need to do this (and at every enterprise SaaS
| company I've seen or worked at, you eventually need to do this -
| the sibling comments about charging more $$$ are smart though):
|
| * "Release train ahead" - code takes 2 weeks (or 4 weeks) to
| reach Production from merge, but it goes out to a Preview
| environment immediately or at the next release. So, you've forked
| your code into a Prod train, a Preview train, and mainline. You
| probably need the ability to hotfix to prod and preview fix to
| the week-ahead-of-prod preview environment, as well as roll
| normal development forward. You do gain "soak time" for code in
| the Preview environment, which is a slight advantage. And your QA
| is easier, because you have three discrete, testable things
| ("does it work in Prod, does it work in Preview, does it work in
| trunk"). But, you have another runtime environment which can
| diverge, which is a big disadvantage.
|
| * Feature flagging by "release train," single execution
| environment. This is probably the most "Software Theorist
| Friendly" way to do this. You don't have to maintain/debug
| multiple environments beyond what you're already doing. Some
| customers are in the "preview ring" and get more features toggled
| on. Single runtime environment, single codeline. Fixes are just
| fixes. QA is a little harder depending on how you set up the
| possible "rings" and combinations, and the operational blast
| radius of your Preview code is now the same as your Production
| code (make a bad query that kills a backend service in Preview
| code, well, that's also Prod code. Oof.).
|
| * Free-range feature flagging. Some complex arbiter system
| manages a huge range of feature toggles on a per-tenant/per-
| customer basis. Having this range of operational knobs is very
| fun and works great for customers who are particularly risk
| averse. It also lets you sell SKU-packs of flag combinations,
| roll specific features to specific "Early Access Programs," and
| so on. But, you get a combinatorial explosion of feature flag
| permutations, which depending on how coupled your code is and how
| intelligent your QA system is, can become a disaster: "Oh no,
| FooFeature only works with BarFeature on but our QA system always
| has BarFeature on so we didn't catch the regression! And only one
| customer has BarFeature disabled for legacy reasons!"
|
| * Some combination of 1, 2, and 3 - for example a Preview
| environment that runs the same _code_ as Prod, but with the
| toggles set differently. A few feature flags which can be enabled
| for early access programs, but a general rule that code needs to
| live in a release train. This is my favorite approach. Having a
| separate Preview environment eliminates the blast radius concern
| of Preview-toggled code breaking Prod. Operational costs are
| higher, but these can be passed to the customer (making customers
| buy preview environments as a line-item is pretty popular). But
| by maintaining single codeline (rather than deploying a fork of
| last week's code to Prod), the code-theory is still easy, you
| don't have divergent release trains, and you don't have the
| combinatorial explosion that comes from a free-range feature
| flagging system.
| jitendrac wrote:
| I would suggest pre-defined staged rollout based on customer
| preference. For example, create three category Alpha, Beta,
| Stable.
|
| Alpha: Gets new features immediately and gets support within
| 24hours if something breaks.
|
| Beta: Gets new features Every 45 week, and gets priority support
| if needed.
|
| Stable: Gets new features quarterly, Mostly they are forced to
| upgrade from old features which you dont want to support, But
| they will be essential volume of your revenue.
|
| alternatively, you can give them sandboxed demo read-only(resets
| every night at 12 o'clock) account for testing.
| codegeek wrote:
| Charge them extra for a "staging" environment setup only for them
| and then deploy all latest stuff there for them to test. Then get
| their approval to apply it to their production environment.
| Charge well for this.
| cbono1 wrote:
| Would you have the staging environment share data with
| production or have its own infrastructure? or use feature
| gating? One customer may not care that it shares the same DB,
| but another might.
|
| How would this work at scale? say if there were say 10
| customers wanting approval, you'd have to have separate code
| release branches/versions for each customer? I imagine this
| could get error prone.
|
| I suppose to your point on charging additional fees for this
| you can afford any sort of workflow.
| daniellarusso wrote:
| From your description, yes, you would basically have forks of
| your codebase for each customer and deploy that to a
| sandboxed staging server.
|
| That would be used for acceptance testing.
|
| Bonus points for having up-to-date data in the staging DB.
|
| I assume this is a multi-tenant implementation.
|
| The other option is to charge the customer to disable
| unwanted features, in addition to enhanced features.
|
| If it is multi--tenant, having the ability to 'feature-gate'
| could be worth the effort, if you can charge for enabling AND
| disabling features.
| Animats wrote:
| Is it that the customers want early access to new features, or
| they don't want you breaking the product?
| quickthrower2 wrote:
| We have a powershell script that:
|
| 1. Copies the production do 2. Removes sensitive data 3. Deploys
| the chosen build of that app (from Appveyor) pointing to that db
|
| So in a nutshell CI/CD pipeline
|
| If you are having trouble perhaps there are a lot of manual steps
| to your deployment.
| zerkten wrote:
| You need to consider the blast radius and how long features may
| be in preview. Ideally, you would scope the preview to just a
| subset of users. When you go to a complete customer base you
| start to get into the situation where the customer has failed to
| communicate with their users, and have more people raising
| issues. This isn't bad necessarily, but you have to be prepared
| to handle it.
|
| If you keep something in preview for a long time then there can
| be a divergence from the main product. Make sure that data or
| work from users is migrated across when you finally release.
|
| It makes sense to think about the opportunity for support to
| participate in the preview. Let them listen in to customer calls
| and learn about the problems. Providing a dedicated support
| person for previews is tempting, but long-term it's better to
| avoid this specialization and enable the whole support team to
| participate. This shares knowledge and avoids certain people
| being granted special privileges while others toil on average
| support tickets.
|
| Microsoft publish some of their methodology at
| https://docs.microsoft.com/en-us/microsoft-365/admin/manage/....
| The key point is that they have set out the boundaries for their
| program so that customers know what to expect and it can be
| repeated for every customer.
|
| You will have to learn how to do your previews, but you should
| lay down some rules. Set it up so kudos from participants can be
| used for marketing after. You are the one to decide when the
| preview ends, but setting some minimum time in preview gives
| customers more confidence and time for you to process feedback.
| Make it clear if features are locked when you enter the preview
| and what feedback you are after.
|
| Once the preview is over, thank the participants. Call them out
| if you have a public forum. Participation can become a mini case
| study for your marketing efforts.
| Hamuko wrote:
| If a large customer wants to decide when they get new features,
| they need to pay for their own dedicated environment that is on a
| separate release cycle. Otherwise they need to adapt to our
| release schedules and make sure they get all the previewing that
| they need on a test environment.
| winkelwagen wrote:
| Difficult to answer without understanding how you are deploying
| currently. I do think feature flags can be an anti pattern. I
| would ask the customers why they want it in the first place. Is
| it a trust issue? Is it an internal process that they require?
| Ideally you would want to stay flexible, that would mean not
| tying your company to their schedule. It is a hard application to
| deploy multiple times for each customer. Perhaps a more lts
| approach for those customers might work. Just keep in mind the
| larger the releases get, the more issues it might cause.
| villasv wrote:
| We pick those customers to be beta testers. They receive pre-
| release features, but in return they must accept possible bugs
| will happen (of course, when one is identified we give them the
| choice of disabling the feature immediately).
|
| The schedule is totally in our hands, though. Once we say it's
| time to go GA, it goes. At this point there still are feature
| toggles, though, just in case.
| cweagans wrote:
| "Thank you for your inquiry. This is not a service that we offer
| at this time -- our testing and release process is fully
| automated and there is not a way for us to exclude some customers
| and not others.
|
| Can you please help us understand the purpose of the advance
| notice/release delay is? What benefit does a process like that
| provide to you or your organization?"
|
| ---
|
| I generally use some variation of that ^^. Having some customers
| on different release tracks is very painful in my experience,
| especially if it's done without properly planning for it in your
| release process. For example, let's say that right now, anything
| committed to `main` goes out to production via some automated
| process. What happens when you have a security bug that needs
| dealt with ASAP? In my org, this usually means that the security
| patch gets rolled out to everyone, and then when it's time for
| customers on a slower release schedule to "catch up", there are
| conflicts to resolve manually.
|
| It's just not worth it. Figure out why they're asking, but say
| no. If it's a problem with buggy releases, that's the _actual_
| problem to put some time into solving.
| johncessna wrote:
| The argument I always got while trying to argue against
| delaying features for large customers was that it causes a
| training headache. If you have large enough customers,
| redesigning a UI triggers a training initiative for the 100s,
| if not 1000s of customers that use your software.
| cweagans wrote:
| That's a fair concern. There are other ways to address it
| though. For instance, release the new UI as a "preview" that
| users can switch to if they want to. If they hate it or can't
| understand it, they can switch back. Even better: redesign
| the UI in smaller parts and roll it out slowly over time.
| mbrodersen wrote:
| The solution is to ask those customers to pay for the extra work
| required. If they refuse then it isn't important enough for them.
| browningstreet wrote:
| I had a similar situation in fintech. We had 3 different release
| cycles and hosting farms for fast, medium and slow accepting
| customers. Fast went to the customers who didn't require
| approvals, and who were flex when it came to errors in the
| release. Then there was a medium channel where the other
| customers were fine as long as a couple of the major customers in
| that pack approved things. The slow channel basically was for
| customers who had large validation or training requirements
| associated with releases.
| cbono1 wrote:
| Would this break down if you had 10 customers that would prefer
| the slow channel of release, but each had their own ever so
| slightly different desired release pace or training
| requirements?
| midnightmonster wrote:
| Key word "desired". Make the slow channel the longest
| reasonable time that any of them want. If they want shorter,
| well, there's medium channel or slow. You pick. If they want
| even longer, well, there's slow channel, and it's good enough
| for these 10 enterprise customers.
| browningstreet wrote:
| Yes.
|
| There are always requests and negotiations in play. But
| those three channels sufficed almost all the time. At times
| we'd combine channels if there was an urgent need and
| agreement from customers.
|
| The slowest customers were the most enterprise-y. Things
| almost couldn't go slow enough for them. When there were
| complications to the release process, they could be a
| release version or two behind and there'd be a catch-up
| release.
|
| We also had customers outside all of this, and had custom
| releases, and we charged very high fees for this.
| Ultimately, we were able to bring them back into the slow
| release because the extra work wasn't really worth it. Part
| of the requirement for these kind of customers is to have
| additional engagement efforts to make sure their internal
| teams were really aware of the details and contingencies of
| releases. Big companies move slow and need over-
| communication, and you can't trust them to do it
| themselves. It was more effective to take that
| responsibility on ourselves.
|
| I preferred to take the things that were difficult and do
| them more often. Keeping a cadence to things kept everyone
| together. The minute you break cadence -- and entertain
| special circumstances -- it's really hard to get back on
| cadence, and that cadence includes testing, notifications,
| documentation, feedback, approvals, releases.
| Jugurtha wrote:
| Figure out _why_ they 're asking that. This appears to be an "XY
| problem"[0].
|
| An example of this is a friend asking me how to weld a wire to a
| thin metallic plate. I digged further to find the actual problem
| for which his solution was to weld the two. He said it was a
| fuse's wire that melted and he wanted to weld a bigger wire to
| avoid that, to which I replied: don't. The whole point of the
| fuse is sacrifice to protect down stream equipment.
|
| The same when a relative asks how to format a disk as their
| solution to solve a problem.
|
| Generally speaking, implementing customers' solutions is a recipe
| for disaster, especially with enterprise.
|
| Ask what it is they are really trying to accomplish, and figure
| out a way to solve _that_. Listening for problems instead of
| solutions or implementations.
|
| - [0]: https://en.wikipedia.org/wiki/XY_problem
| geocrasher wrote:
| Sounds to me like the customer is trying to solve a problem and
| is coming to you with their chosen solution. My suggestion: Find
| out what problem they are trying to solve, and address _that_.
|
| Also, you might look at cPanel's notices that they send out where
| they explain upcoming feature updates well in advance of the
| rollout. They also have different release cycles, where the
| newest features are released to the EDGE cycle instead of the
| STABLE, and then the long term release cycle remains unchanged
| except or security updates.
| mooreds wrote:
| > Sounds to me like the customer is trying to solve a problem
| and is coming to you with their chosen solution. My suggestion:
| Find out what problem they are trying to solve, and address
| that.
|
| This.
|
| Problems they might be trying to solve: * want
| to be able to jump on new features as soon as they are
| released, but need to plan with the eng team * want to
| be "in the know" so not surprised by new features * want
| to be able to give feedback and help direct the roadmap
| * have been bitten by bugs in your software (or other vendors)
| and want burn in time, especially if you upgrade them without
| their choosing (as is typical in some SaaS models)
|
| I'm sure there are others, but it's worth asking why they are
| requesting this if you haven't done so already.
| zippergz wrote:
| Honestly some customers I've encountered have asked for stuff
| like this (and similarly meetings to go over product roadmap)
| almost as a power play. Either to see if they can get us to
| do it, or to feel like they have control over our
| development. If you're small enough and the customer is big
| enough, you might not have any choice, but I resist it pretty
| strongly unless it would be financially ruinous to lose the
| customer (and we have never lost a customer over refusing to
| do it).
| stronglikedan wrote:
| Reminds me of when Disney wanted us to do a considerable
| amount of unpaid work, simply because they were Disney. We
| don't do business with Disney any more.
| void_mint wrote:
| In retrospect, was it the right move?
| that_guy_iain wrote:
| Seems like you need advanced feature flags.
| cbono1 wrote:
| In your comment here, what's the distinction of advanced vs. a
| regular feature flag?
| that_guy_iain wrote:
| Regular feature flag is just on and off. When you start
| getting into the advance you can change it for user groups,
| specific users, timable so it's only enabled for a timeframe,
| only on a specific env. The possibilities are endless.
|
| I make the distinction so people don't just think they can
| put a single if statement in and be able to handle this sort
| of scenario.
| cbono1 wrote:
| Got it - that clarifies it.
| Macha wrote:
| Feature flags is usually how we handle this. We have the flags
| anyway for developer use, so making them user specific was not
| that much incremental effort though we have had a few cases where
| old workflows were not removed when they should have been.
|
| We did once have a customer with their own environment with a
| different version deployed, as they were initially a major client
| with such demands but after a few years it became clear it wasn't
| worth the maintenence effort to special case this one client so
| we made it clear in contract renewal one year that this would be
| going away.
| tombot wrote:
| +1 for flags, you can also put features into production
| earlier, safe knowing they will only be visible for the users /
| teams you've added to the list
| 1a527dd5 wrote:
| +1 on feature flags. I find them to be a pain, but fussy
| customers insist "that we need to test it in staging first".
| notJim wrote:
| +1 on feature flags. Set up your flags so you can disable
| features for certain customers, or groups/classes of customers.
| Delivery and release can then be separated. Release frequently,
| deliver features when it makes sense.
|
| Also recommend making sure customers know that this is time-
| limited, otherwise you end up supporting too many different
| configurations of your product.
|
| Also if these are customers you're in regular contact with, you
| can preview your roadmap or demo features that are coming soon.
| This helps them feel in the loop, and gives them a chance to
| provide feedback.
| petters wrote:
| Agree on this. We delivered new versions to customers once a
| week, but every new version should not have been noticed by a
| customer at all.
|
| The roll out of feature flags is a separate process.
| williamdclt wrote:
| That's about the only answer, although feature flags are hell.
| Once you have a feature flag, you now have 2 versions of your
| product. 2 feature flags, that's 4 versions to maintain, etc.
| Make sure you don't conflate "feature flag" and "setting", and
| make sure you clean up these flags when they're uneeded (and
| actively push the business to make them unneeded!).
|
| Fowler has a good write-up about different types of feature
| flags: https://martinfowler.com/articles/feature-toggles.html
| OriPekelman wrote:
| well, self horn here... but that is a basic functionality of
| https://platform.sh (CPO here). You get per branch full copies of
| production in isolated staging environments that are very cheap
| to run. So you just clone production to a specific branch apply
| whatever changes and grant access to the clone. ..you do need to
| run production with us for the magic though... and a PaaS is not
| for all use-cases (specifically when running a SaaS...).
___________________________________________________________________
(page generated 2021-06-11 23:01 UTC)