[HN Gopher] Tiered support is an anti-pattern (2013)
       ___________________________________________________________________
        
       Tiered support is an anti-pattern (2013)
        
       Author : mooreds
       Score  : 78 points
       Date   : 2024-10-02 02:35 UTC (3 days ago)
        
 (HTM) web link (peter.gillardmoss.me.uk)
 (TXT) w3m dump (peter.gillardmoss.me.uk)
        
       | patrakov wrote:
       | There is another pitfall in the layered support structure: how
       | would you hire, train, or otherwise produce L2 support engineers?
       | 
       | You cannot hire someone directly as L2, as there is a learning
       | curve for company- and product-specific issues, and a freshly
       | hired "L2" would have to learn. However, they are shielded from
       | the most frequent issues by a hoard of L1s and thus cannot learn
       | about the typical problem areas or troubleshooting approaches.
       | 
       | So they have to either graduate from L1s or accept a downgrade
       | from being a developer.
        
         | Galanwe wrote:
         | There are usually a number of people in L1 roles looking to
         | transition to more development. You can have them be L2 by
         | having 1/3 of their time spent contributing to bug fixes and
         | small features of the software so they get to know it better.
         | 
         | I dont think they would have to come from L1 from the same
         | company / project though. Experience does allow one to learn
         | faster, communicate more efficiently, etc.
         | 
         | I think it's also good practice to have devs rotates on some
         | small shifts of L1/2 so they don't get out of touch with the
         | reality of maintaining their stack.
        
         | magicalhippo wrote:
         | > You cannot hire someone directly as L2
         | 
         | We've managed to do that by recruiting from our customers.
         | During support cases and such we'll notice someone is more
         | technical, and then at some point approach them with an offer.
         | 
         | This way they already know our software well and can get fairly
         | straight into a L2 position. Of course in the start they'll get
         | some more L1 stuff and such, but it significantly reduces the
         | learning curve.
         | 
         | Customers aren't too miffed as the alternative would almost
         | always be that they'd go to a competitor.
        
         | cwbriscoe wrote:
         | Business Analysts can fit the L2 role. Where I work, they are
         | the interface between the end users and the developers. They
         | usually come from the either the business side or the support
         | center. And if they came from the support center, they usually
         | have prior experience as an end user. However, I can see how
         | that wouldn't work out for all industries.
        
         | crabbone wrote:
         | To me, it looks, like this is contingent on the kind of product
         | they need to support. For instance, in storage business, L2
         | support are often paid more than developers (and are very hard
         | to find!) They usually come from the development side of
         | things, attracted sometimes by being hired in a different
         | capacity (eg. being a consultant running their own consultancy
         | business rather than being a direct hire), some like to go on
         | business trips (this kind of work often involves being on the
         | customer's site, physically plugging and unplugging equipment
         | etc.)
         | 
         | In medical s/w development, L2 can be nurses or doctors, they
         | might be attracted to this position due to relatively low load
         | (compared to clinical medicine), flexible schedule etc.
         | 
         | I don't think there is / possible a single answer to how to
         | find these kinds of specialists.
        
       | latch wrote:
       | In my experience, the real problem with any support scheme is
       | that the decision makers don't have enough skin in the game. It's
       | easy to deprioritize tech debt and bug fixing when you aren't
       | facing 3am pages.
       | 
       | The obvious solution to this is a distributed team. There can
       | still be some holes (e.g. Jan 1 is, afaik, pretty universally
       | celebrated), but having worked at companies that had a team in
       | the US, Europe and Asia was...really nice with respect to this.
        
         | mrweasel wrote:
         | The incentives are frequently wrong as well. The support staff
         | for many companies are rewarded on volume, call-queue length
         | and "resolution time" (not actual resolution time, but how
         | quickly you can get the customer off the phone).
         | 
         | The only valuable metrics, in my mind, are "Did we actually
         | solve the issue for the customer" and "Have we effectively
         | prevented this from being an issue in the future".
        
       | the_alchemist wrote:
       | Isn't this tiered support a necessity when operating at scale?
       | How can you help thousands of customers where faqs and getting
       | starteds don't cut it? You need a filtering mechanism (L1) to
       | help users directly without overloading the product team with a
       | standard reply rtfm.
        
         | benoau wrote:
         | It's triaging the support so the people who are best-capable of
         | answering _specific_ things don 't waste all their time
         | answering the stuff a new hire can take care of after one week
         | training.
        
         | andrewaylett wrote:
         | Two layers is fine, so long as they're allowed to talk to each
         | other. Adding a third causes a lot more problems.
         | 
         | Once you get beyond having a single dev team, a layer of
         | support is necessary: you can't expect consumers to know which
         | of your dev teams they ought to be talking to.
         | 
         | My employer has an amazing support team, who are effective
         | champions for the user when interacting with development teams.
         | They're _fantastic_.
        
           | jasonjayr wrote:
           | That's an important requirement of an effective support team:
           | they have to have a seat at the project management's table.
           | They have to have the power to request features + changes
           | from developers, at the same priority level as the rest of
           | the product development team. IMHO it's important both for
           | the support team morale, as well as ensuring a positive UX.
        
         | HeatrayEnjoyer wrote:
         | If you're operating at scale you hire at scale. How do you
         | suppose Target staffs their "at scale" brick and mortar stores?
        
           | ahtihn wrote:
           | Yes, you hire L1 support staff at scale. You don't hire
           | product developers to handle the scale.
        
       | magicalhippo wrote:
       | We make B2B software, and we have roughly the setup described in
       | the article, where L3 are the devs.
       | 
       | I don't recognize what the article describes at all. There are
       | tons of questions answered by L1 which are not even issues with
       | our software. And us devs certainly feel responsible for what we
       | produce, and we still talk to our customers when needed.
       | 
       | So I think it's more a culture and possibly management problem
       | than simply layering.
        
         | jonathanlydall wrote:
         | I don't necessarily think tiered support is the wrong way to
         | go.
         | 
         | But as someone developing a product, sometimes when you're
         | closer to your users you find that many users get told to RTFM
         | on a particular feature and maybe the actual problem is that
         | the feature is unintuitive.
        
           | magicalhippo wrote:
           | But I get feedback from L1/L2 when some feature generates a
           | lot of support requests, and I've many times sat down with
           | support and even called up customers to improve such
           | features.
           | 
           | That said, I agree with the sentiment that as devs one
           | shouldn't be far removed from customers. For example I do
           | take time every now and then just looking at support working
           | if they've got a customer showing an issue or similar. Or
           | just hang around to hear what they complain about.
           | 
           | This is of course much more difficult if support is in
           | another building and not just down the hall.
        
           | HL33tibCe7 wrote:
           | There are ways to deal with that without putting devs on the
           | front line though.
        
         | mrweasel wrote:
         | Years ago I worked for a telco. As part of trying to identify
         | recurring problems and design possible solutions, we would sit
         | in on support calls for a few hours a day, for a week. While we
         | did find a few things that could be improved by adding more
         | software, or reworking solutions, that wasn't what was needed.
         | 90 - 95% of support calls where related to technical issues,
         | which required first level support to create a ticket for
         | technical support, or escalate to them immediately. First level
         | support was pretty much not needed. It would make more sense to
         | reverse the call flow and have technical staff pick up support
         | calls, then transfer to first level support if the issue turned
         | out to be non-technical, which it would be in 5% of the cases.
         | 
         | To this day that is still my issue when call ISP or telco
         | support. Why am I trying to have some 19 year old, with access
         | to nothing but my billing information, trying to solve the
         | issue of my router not connecting?
         | 
         | The type of first level support most technical companies offers
         | could be solved by better monitoring, better self-service and
         | well written documentation.
        
       | AStonesThrow wrote:
       | TIL that ".me.uk" is the domain reserved for "personal names"
       | 
       | https://en.wikipedia.org/wiki/.uk#Active
       | 
       | Which is pretty cool, considering that when I registered a domain
       | under ".us" in 1993, I used our street address as the leaf node,
       | and I've seen a few random "StonesFamily.org" or similar; it's
       | nice for UK persons to stake a claim here. How long has it been
       | active? How do they resolve disputes? Do they permit subdomains
       | for family groups?
        
         | traceroute66 wrote:
         | > How long has it been active?
         | 
         | I think pretty much forever at this point, the only addition to
         | .uk in recent memory is the ability to register .uk TLD itself
         | (before it was all second-level only).
         | 
         | I think it might have been around the 2000's when they
         | introduced me.uk.
         | 
         | > How do they resolve disputes?
         | 
         | AFAIK me.uk falls back to the general rule of "first-come
         | first-serve". The only real requirement for me.uk is that it
         | has to be registered in the name of a natural person and not an
         | agent, trustee, proxy or representative. Any transfer or
         | renewal has to continue to adhere to this rule.
         | 
         | There is no UK nationality or residency requirement for me.uk
         | either AFAIK.
         | 
         | Disputes fall back to Nominet's standard Dispute Resolution
         | Service.
         | 
         | > Do they permit subdomains for family groups?
         | 
         | Not AFAIK. Well, of course you can register the base me.uk and
         | DIY DNS subdomains.
        
       | echoangle wrote:
       | I don't even get the point, are Facebook developers supposed to
       | deal with 500k support emails per day because grandma forgot her
       | login? That's unrealistic for any product with more than maybe 1k
       | users.
        
         | andyp-kw wrote:
         | We've come to expect zero customer support from the likes of
         | Facebook and Google, but why does that need to be the norm ?
        
           | echoangle wrote:
           | I didn't say that you should have zero customer support, but
           | I don't think the level system is the problem. It's a problem
           | when the levels don't properly escalate stuff they don't
           | understand. But you just can't have every support case go
           | directly to the developers because 90% of support cases are
           | stuff developers can't help you with.
           | 
           | It's like saying every person in a hospital should directly
           | go to the operating theater. The 10% people needing urgent
           | care would like it but it's a very inefficient way to use the
           | time of the staff. That's why you have a reception (Level 1)
           | that's triaging cases and checking if you even need a doctor.
        
             | Vampiero wrote:
             | The exception being that the triage is a queue-based
             | system, and eventually it actually handles your case: you
             | show up at the hospital, describe your problem to a human
             | being, and you're assigned a priority in the queue
             | pertaining to the specialist that needs to see you. If
             | you're not dying you might have to wait a few hours, but
             | you can rest assured that you will be looked at by an
             | expert -- even if you're a hypochondriac.
             | 
             | If your case is an emergency, you can cut through all the
             | queues and go straight to the operating table.
             | 
             | Whereas most L1 support systems are either non-existent; a
             | FAQ loop that gets you nowhere; or a Dialogflow/GPT chatbot
             | with 0 reasoning skills, no technical knowledge, and
             | limited domain knowledge (really just a more convoluted
             | version of the FAQ loop).
             | 
             | If you're lucky, after some digging you might find a hidden
             | link that lets you talk with a real human being.
             | 
             | These systems are deliberately set up to massively increase
             | the effort required to get to talk to a technician, as
             | those are a limited resource and companies don't want to
             | spend money on stuff that would actually improve their
             | internal processes.
             | 
             | And if hospitals worked like this then the majority of
             | people showing up for any problem whatsoever would die
             | before receiving care. This is not how triaging works.
        
             | ethbr1 wrote:
             | > _It's a problem when the levels don't properly escalate
             | stuff they don't understand._
             | 
             | This is the real human psychology trap. By definition, L1
             | doesn't understand anything it can't handle.
             | 
             | Unfortunately, at many companies L1 is also yelled at for
             | escalating too many cases.
             | 
             | Consequently, the internalized message for L1 to not get
             | yelled at is "Juggle the ticket you don't understand
             | convincingly, until the customer abandons interaction from
             | frustration."
             | 
             | A few anti-patterns I've seen in most support shops:
             | 
             | 1) Bad KPIs / no post-hoc sampling and review. Without
             | regular, random, higher-level sampling and classification
             | of how tickets were handled, there's no signal that support
             | is technically botching a large % of incoming tickets.
             | 
             | 2) Disincentives to escalation, _even when it 's a true
             | positive escalation_ that should be. IMHO, support should
             | be incentivized to escalate true positives.
             | 
             | 3) Failure to upskill support agents and build mix-of-
             | expertise within the support function.
             | 
             | 4) To 3, failure to pay support for the value they're
             | creating. In a sane world, a few of your highest skilled
             | support people will be paid like developers, because they
             | are.
        
           | EduardoBautista wrote:
           | Both Facebook and Google provide customer support. I think
           | people are just confused about who their actual customers
           | are.
           | 
           | Hint: It's the advertisers.
        
             | echoangle wrote:
             | I bet they still have a level system for the advertiser
             | support, there's no way an actual developer reads the first
             | message. Maybe for very large customers they have a
             | dedicated support line, but not for a small business
             | advertising on google.
        
             | rolisz wrote:
             | Still a tiered support system: last year I started a side
             | project and I wanted to advertise on Google. Created the
             | AdWords account, with all the legal documentation and
             | everything, it got banned in two days. All the support I
             | could reach was bots. Luckily I had some friends at Google
             | who could escalate internally....
        
             | jeltz wrote:
             | Sounds like you have not bought ads. No, they do not
             | provide support there either. Maybe if you buy tons of ads,
             | but not for smaller advertisers.
        
           | awelxtr wrote:
           | One thing is customer support and the other is user support.
           | There is few to none of the latter.
           | 
           | Regarding the former, my company pays for google workspace
           | and an agent happily answered our questions about the recent
           | less secure apps password phasing out process.
        
         | raincole wrote:
         | I can only imagine the author works for a B2B software priced
         | at least $5000/seat/year. Or a software that practically has no
         | user. These are the only two scenarios where "let customers
         | talk to developers directly" could ever work.
        
       | smallerfish wrote:
       | Support forums for free tiers are the absolute worst. With some
       | exceptions, the general pattern seems to be power users hanging
       | out and answering questions on the company's behalf. Who are
       | these people?
       | 
       | Another support anti-pattern is an untrained offshore support
       | team whose entire mission seems to be to close the case and move
       | on. There is little in life more enraging than a support rep who
       | refuses to engage logical brain and escalate based on clear
       | evidence of a technology issue.
       | 
       | At my most successful startup, we had an extremely smart guy in
       | the support seat; he was eager to learn from the dev team how to
       | triage, and so we'd get bug reports complete with network tab
       | screenshots, api paths which had caused an error, etc. He could
       | easily have been a skilled programmer had he chosen to pursue it.
       | When we were acquired, he was merged into a larger team that had
       | much less autonomy, and his morale tanked over time.
       | 
       | The zendesk ("here are articles that might be appropriate")
       | approach sucks too, as do chatbots.
       | 
       | Perhaps fear of hallucinations is holding people back, but it
       | seems like an LLM based product could take over the "keep it
       | cheap" support vertical. 1) Have people file a report in their
       | own words; 2) synthesize an answer for them, but only if the LLM
       | identifies with high confidence their report as being a known
       | problem that has an answer; 3) cluster reports from users based
       | on salient features and escalate new clusters to the tech team;
       | 4) and if the company can afford it, route everything else to a
       | human team who can categorize, click a few buttons, and send the
       | LLM back with an answer, or escalate.
        
         | grues-dinner wrote:
         | > power users hanging out and answering questions on the
         | company's behalf. Who are these people?
         | 
         | It's genuinely strange. People will even do it for banks and
         | phone network companies.
         | 
         | As a company, all you need to do is have XP tiers for posting
         | and someone will be willing to dedicate their life to becoming
         | a _Finco Bank Guru_ and running the forum for free. Usually
         | they have a rather  "Boomer" energy, so I get the feeling it's
         | retired people who think the parish council/HOA doesn't meet
         | often enough.
        
           | thih9 wrote:
           | On one hand it's unfortunate that people don't get paid for
           | this kind of help and perhaps act in a short sighted way
           | (giving away too much control to the provider).
           | 
           | On the other, a lot of these are people willing to help
           | someone else for free and make their knowledge practically
           | public in the process. Seems aligned with a
           | hacker/wikipedia/oss ethos in many ways.
        
             | saagarjha wrote:
             | Yeah, until you realize that they get all that XP posting
             | some variation of "turn it off and on again" ten thousand
             | times
        
               | thih9 wrote:
               | As long as their comments are useful, I don't mind. Many
               | open source projects are low quality or trivial to others
               | - but still provide big help to some.
        
               | beardedwizard wrote:
               | A long time ago I supported hp scanners on windows. I
               | exclusively told people to uninstall and reinstall the
               | drivers, or windows. My success rate was 100% and I
               | outperformed all my peers who did actual troubleshooting
               | in time to resolution and number of resolutions.
               | 
               | Sometimes the right answer is just the right answer.
        
           | robocat wrote:
           | Is there some reason Boomer isn't yet seen as offensive as a
           | racist word?
           | 
           | Your comment isn't the worst, but the word is usually used as
           | a placeholder for some bigoted stereotype against someone
           | retired. Disclosure: I'm not a boomer and the word is not
           | used much in New Zealand.
        
             | bc569a80a344f9c wrote:
             | I mean, one reason is that no one has ever shouted "boomer"
             | in a hateful voice while lynching someone for looking their
             | way wrong.
             | 
             | We could also talk about how it's usually not just used as
             | an ageist slur but to describe a state of mind - if you ask
             | the kids there's plenty of old people who aren't boomers -
             | but saying it's as offensive as being racist is completely
             | inappropriate.
             | 
             | To sort of quote the comedian John Mulaney about the
             | n-word, if one of the words is one you can't say and have
             | to use "the n-word" to describe it, that's the bad word.
        
               | TeMPOraL wrote:
               | > _no one has ever shouted "boomer" in a hateful voice
               | while lynching someone for looking their way wrong._
               | 
               | That "while lynching" part is doing all the work here.
        
               | flappyeagle wrote:
               | As it should? Mexican or Chinese or Old or Gay are not
               | bad word but they have been and continue to be shouted in
               | hateful ways
        
             | HeatrayEnjoyer wrote:
             | ...really?
             | 
             | https://i.redd.it/kb6r6ajwk3pc1.jpeg
        
             | ethbr1 wrote:
             | It's definitely a lazy slur, in the same way that
             | "problematic" is a lazy complaint.
             | 
             | Why reach for a vague zeitgeist term that everyone knows,
             | but no one has a precise definition for, instead of taking
             | the time to use a specific word?
        
             | drdeca wrote:
             | The popular culture is less concerned about ageism than
             | racism.
             | 
             | (Also, like, there are clear biological differences by age,
             | which are sometimes relevant, while the concept of race is
             | a mistake. It seems understandable that people would be
             | less concerned by ageism than by racism. Also, people who
             | live long enough will experience the different age brackets
             | themselves, which puts a natural limiting pressure on how
             | badly people are likely to treat others based on their age.
             | I imagine people would be less likely to hate people of
             | "race" M if they knew that in a few years they themselves
             | would be of "race" M, while the people that currently are
             | would not be.)
        
           | bc569a80a344f9c wrote:
           | I'm sure I'm an outlier, but I did this once for a particular
           | network equipment vendor. It's a smaller vendor that aims at
           | a low price point and skimps on things like support, but
           | their software had a particular feature that was going to be
           | very important for my employer. To get good at
           | troubleshooting their gear fast - we were going to deploy the
           | equipment for scenarios where mean-time-to-resolution was
           | going to matter a lot - I camped on their forums and helped
           | everyone I came across as practice matters a lot for that.
           | After about year I'd achieved that and stopped, but I maxed
           | out their karma rankings in the process. They didn't have
           | training material or certification programs and the equipment
           | didn't break much for us in production so this was the only
           | way I could think of for getting the hours in.
        
           | generic92034 wrote:
           | > It's genuinely strange. People will even do it for banks
           | and phone network companies.
           | 
           | Helping other people can be deeply satisfying, even without
           | payment or karma points. Maybe you are already doing it
           | yourself? Otherwise I can only recommend you to try it some
           | time. ;)
        
           | autoexec wrote:
           | > someone will be willing to dedicate their life to becoming
           | a Finco Bank Guru and running the forum for free. Usually
           | they have a rather "Boomer" energy, so I get the feeling it's
           | retired people who think the parish council/HOA doesn't meet
           | often enough.
           | 
           | I started a support forum for a video game once. I wasn't a
           | boomer or retired. I just liked the game, had the ability to
           | help, and the time. I guess some people just like being
           | helpful. Certainly the company who made the game profited off
           | of my free labor but I wasn't looking for money (to their
           | credit they did offer to send me to E3 one year at a time
           | when tickets weren't being offered to the public).
        
         | yonatan8070 wrote:
         | I think it could be possible to feed the LLM with lots of data
         | about your product, potentially even give it source code with
         | instructions not to share it. Then have it act as a highly
         | knowledgeable support rep for customers, and also be able to
         | file detailed issues for the dev team when it identifies that a
         | customer finds a bug. The risk of hallucinations is still
         | there, but it could probably be mitigated by feeding the LLM's
         | output into a fact-checker model. I think the bigger risk is
         | leaking internal data through prompt injections
        
           | mooreds wrote:
           | We engaged a startup (kapa.ai; they're great) to train an LLM
           | on:
           | 
           | * our public facing docs
           | 
           | * our public repos
           | 
           | * our youtube channel
           | 
           | * our forum
           | 
           | and spit out answers. It's not right all the time, but it is
           | most of the time and, crucially, it shares not just an answer
           | but a link to the docs that led it to that answer. That makes
           | it easier to fact check the LLM.
           | 
           | We've found it very useful for scaling support and making
           | that corpus of knowledge available to folks both outside and
           | inside the company.
        
           | autoexec wrote:
           | > I think it could be possible to feed the LLM with lots of
           | data about your product, potentially even give it source code
           | with instructions not to share it.
           | 
           | I wouldn't trust the LLM to keep your secrets. There are
           | countless examples of people convincing LLMs to say things
           | they shouldn't. On the other hand, I wouldn't trust an LLM to
           | spit out your source code without hallucinating a bunch of
           | fake functions either, so maybe it balances out.
        
         | resonious wrote:
         | I also know a "support guy" who absolutely kicks ass. Probably
         | a better dev than most of us regular product engineers. Makes
         | me think support is actually way harder than the actual app
         | dev.
         | 
         | I also wonder if the incentives are a little bad with this
         | setup. Product devs' crappy code turn into tricky puzzles for
         | our support team. Legend has it, RAD Game Tools used to have
         | just 1-2 devs on each tool, and you'd do the support for your
         | tool. Feels like this would pretty strongly encourage you to
         | make your stuff good. And you'd have everything in your head
         | too. I wish I knew of more similar setups in the web world.
        
           | FiatLuxDave wrote:
           | I agree about how aligning the incentives can make a huge
           | difference. The first product I was ever a product manager
           | (and SME) for, I assisted the support team with escalations.
           | This meant that every weird new bug or strange situation
           | would hit me the same day. This really shortened the feedback
           | loop on getting stuff fixed. It also made it so that I became
           | aware of quirks or things that weren't working quite right
           | from the customer's perspective, which is gold for a PM. That
           | information I learned doing support escalations helped in the
           | design of version 2 of the product, which is still in
           | production 22 years later.
           | 
           | I strongly recommend having the PM be the highest point of
           | escalation in the support chain. It really incentives them to
           | make a quality product.
        
         | knodi123 wrote:
         | > Support forums for free tiers are the absolute worst. With
         | some exceptions, the general pattern seems to be power users
         | hanging out and answering questions on the company's behalf.
         | Who are these people?
         | 
         | In my company, we have free support forums, email support for
         | the customers who jump through some hoops (to make it slightly
         | harder than searching the forums), and phone support for select
         | clients. But the "power users" in the forum are just our
         | internal customer support staff operating a handful of sock
         | puppet accounts, and it's the same people who answer phones and
         | email.
         | 
         | The forums are a great searchable historical record. The email
         | support is versatile and scalable. And the phone support is a
         | non-scalable value-add that makes our top tier clients feel
         | very special.
        
           | mooreds wrote:
           | >The forums are a great searchable historical record.
           | 
           | Wrote a whole blog post about this:
           | https://www.mooreds.com/wordpress/archives/3451
           | 
           | Many people have a drive to solve their problems themselves,
           | and if you make content available to google, they'll find it.
           | It's a fantastic way to self-empower your users and
           | customers.
           | 
           | We even have an internal slack channel where I'll post
           | questions and answers from the other support channels (email,
           | slack) with a view to making them generic and turning them
           | into forum posts. I don't get to that task often enough, but
           | it is sure fun to do when I have time.
        
           | safety1st wrote:
           | Very early in my career I interned at a company that rotated
           | all of its developers ONTO THE SUPPORT TEAM for a few days
           | every now and then (if memory serves, for 3 days per year).
           | Yep, if you wrote the product, it meant that sooner or later,
           | you were going to be on the phone with a customer who was
           | having problems with that product. Even the interns were
           | included and I got to be a support tech for two days. The
           | support team was absolutely amazing with excellent
           | documentation on what to say and do and how to close out a
           | case.
           | 
           | We were 10x slower than the real support techs at resolving
           | cases so this didn't improve support by all the typical
           | pinhead boss metrics. Customers tended to think it was pretty
           | awesome they were talking to a dev regardless of whether we
           | solved their issue. The real benefit of the experience though
           | was FOR US, and was transformative. I credit a lot of my
           | better instincts regarding product development to having that
           | experience so early in my career. For example nothing makes
           | you more sensitive to performance issues than sitting on the
           | phone just waiting and talking to some random person for two
           | minutes every time anyone tries to load a particular screen -
           | you get back to your "real" job and the first thing you do is
           | go into the bug database and start asking where the hell is
           | that bug and when are we going to fix it.
           | 
           | I own my own business now and I have never left, and will
           | never leave, the email support alias, I will read every one
           | of those emails I can until I die (and respond to the best of
           | my ability as well).
           | 
           | Later I worked on the team for a web-based product which had
           | feedback fields all over the product. It took everything
           | anybody wrote in those fields and piped them, unfiltered,
           | into a screensaver that was installed on every computer used
           | by the development team. When I was new to the team my
           | manager told me "You just have to accept that you're going to
           | see a lot of profanity on people's screens, that's how it
           | works here." LOL! He wasn't lying. But you'd get exposed to
           | dozens of random bits of feedback about the product over the
           | course of each workweek. Tons of bugs and user pain points
           | were discovered this way. You'd be in the middle of a 1:1 and
           | some expletive-laden complaint about the feature you designed
           | that went live three weeks ago would pop up, well OK, guess
           | we're going to have to deal with that before the meeting is
           | finished! It was wild.
           | 
           | I remain of the view that relentlessly and aggressively
           | exposing your entire product team to tons of customer
           | feedback through any avenue you can is the right choice. Got
           | a popular "water cooler" or #random/#general type of Slack
           | channel? Start piping a bit of customer feedback into it. You
           | can email Slack channels! Do it today.
        
       | makeitdouble wrote:
       | The need for devs to have a sense of what's happening on the user
       | side, get support requests and work with the customer support is
       | spot on. It really makes a world of difference.
       | 
       | The crucial issue though is to also have competent CS working
       | with the dev team. It's not easy to have two separate teams share
       | info and collaborate efficiently, it requires skills, and these
       | skills need to be on both sides of the table.
       | 
       | That makes it harder to retrofit these setups in an existing org
       | when these aspect where completely ignored at hiring time for
       | instance.
        
       | crabbone wrote:
       | The article is from the time when CI / CD was relatively new and
       | many felt like it needs promotion / wanted to jump on the
       | bandwagon.
       | 
       | Today, after some practical experience, it looks like some
       | conclusions this article makes are unwarranted. For instance, the
       | idea that the development team should be directly involved in
       | addressing customer issues, if implemented, often results in
       | development plans being ruined, QA doing a lot of busy work,
       | release schedule going down the drain.
       | 
       | Ultimately, the problem is the lack of expertise necessary to
       | triage and to schedule development necessary to address the
       | problems the customer is facing. Low-tier customer support is
       | usually a low-paid position that doesn't require expertise in the
       | product being supported, or anything at all really. Is there a
       | solution that doesn't involve rising the requirements for the
       | support staff (and rising the pay)? -- I don't know. Doesn't look
       | like it. It seems like asking for having your cake and eating it
       | too.
        
       | mrweasel wrote:
       | Many of the tiered support offerings from companies also doesn't
       | make sense from a customer perspective. My favorite example is
       | Elatic Co. trying to sell us support. We want one thing, and one
       | thing only: "When this fails, in a way where we can't fix it
       | using documentation, experience or the support forums", we need
       | to be able to call someone. Everything you're offering below that
       | is useless to us. We just need the phone number in the
       | exceedingly rare case that this break down completely, everything
       | else is not something we'd be willing to pay for. But the "call
       | someone, who knows something" is the final tier, and includes all
       | the things below it and is priced accordingly.
       | 
       | Understandably being able to call a developer should be
       | expensive, but when you tier it, then customers feel ripped of,
       | because they have to buy something they don't need, to get that
       | one thing they really do need and are willing to pay for. In our
       | case we where willing to pay some fixed price per year for access
       | to phone support, and then an hourly rate if we called, but that
       | wasn't the pricing structure.
       | 
       | In fairness the support structure and pricing from Elastic has
       | improved since then, but the issue is the same, if you only need
       | one thing from the higher tiers, then the pricing doesn't make
       | much sense to you as a customer.
        
         | portaouflop wrote:
         | Would you prefer they only offer the call option without all
         | the other stuff for the same price?
         | 
         | Then you don't have the problem of paying for things you think
         | you don't need.
         | 
         | Your problem is that you want the highest value (direct on-call
         | team with actual skills is expensive AF) for the budget price -
         | as someone who helped design support tiers this is super common
         | but obviously not economically viable.
         | 
         | edit: in any case on that price level the price structure
         | should be more flexible I agree
        
           | mrweasel wrote:
           | This is not exactly how it works, but as a custom you look at
           | the price of the tier below what you want, and the price of
           | the tier which contains what you want. The price difference
           | must be what the feature you want costs.
           | 
           | So if the tier with which provides on-call is $100.000 and
           | the tier below that is $75.000, then the cost appears as
           | $25.000 to me as a customer. That not true, when you dig into
           | it, some of the cost is covered by "unused" benefits of the
           | tiers below, so that one benefit on it's own might be closer
           | to $50.000. If you truly don't need the stuff below, then
           | you'd want more flexibility in the price structure and be
           | able to spend the $50.000, rather than the $100.000. Instead
           | the provider now ends up with no sale.
           | 
           | > direct on-call team with actual skills is expensive AF
           | 
           | More than most realise. My issue is that there are occasions
           | where you don't mind paying for that service, but you're not
           | allowed to, because the tiers are designed so that it becomes
           | more expensive than it should be. I've seen a number of
           | support contracts, where they only tier that actual provides
           | much value is the one where you can call an expert, but
           | that's hidden behind "Call Us" or "Super Premium Platinum"
           | priced such that only governments and VC funded companies
           | can/will pay for it.
        
         | mooreds wrote:
         | > But the "call someone, who knows something" is the final
         | tier, and includes all the things below it and is priced
         | accordingly.
         | 
         | But it is priced accordingly not because it includes all those
         | other things, but because it is dang expensive to provide 24/7
         | on-call technical support.
         | 
         | > In our case we where willing to pay some fixed price per year
         | for access to phone support, and then an hourly rate if we
         | called, but that wasn't the pricing structure.
         | 
         | Sure, I get it. Elastic wasn't selling what you want and that
         | stinks. But there's a cost to a company of offering everything
         | ala carte. It makes support more complex (because support staff
         | have to be trained on snowflake clients), makes it harder to
         | sell online (because customers get confused by options and
         | bail) and harder to sell via a sales team (again, complexity).
         | 
         | That said, I bet there are Elasticsearch consulting companies
         | who would be thrilled to quote you for on-demand on-call
         | support (googling for "elasticsearch oncall consultant" turns
         | up a few). I'd be interested to hear how their pricing
         | compares.
        
         | hodgesrm wrote:
         | > We want one thing, and one thing only: "When this fails, in a
         | way where we can't fix it using documentation, experience or
         | the support forums", we need to be able to call someone.
         | Everything you're offering below that is useless to us.
         | 
         | I don't doubt your expertise in technology. But there are so
         | few companies that can actually do what you describe that it's
         | not a viable market for vendors.
         | 
         | The first problem is that the kind of staff you need when
         | things go really bad are (a) really hard to find and (b) are
         | looking for permanent employment. That's why support vendors
         | want term contracts--it's the only way to make the economics
         | work.
         | 
         | The second problem: While you say you just want to call your
         | support vendor when something really bad happens, that's
         | actually the most expensive and least satisfying approach for
         | most users, even very expert ones. Far better to engage before
         | there's a failure, e.g., at design time or during regular
         | health checks. Data base upgrade planning is case in point
         | where early engagement pays off in spades. That includes things
         | as simple as having an expert on-call when the upgrade happens.
         | 
         | YMMV. That's at least been my experience from 18 years of
         | supporting open source database software.
        
       | veggieroll wrote:
       | As a developer, doing one or two support cases per day has been
       | one of the best things I have done.
       | 
       | It really helps me understand what they're doing better. It short
       | circuits a lot of bureaucracy. And it's great when you can push
       | out a change for someone an hour after they wanted it for simple
       | things. So they know that you can make changes fast to address
       | their problems.
       | 
       | And it's just nice to get to know the users and be friendly. We
       | get to chatting. And it just makes everyone more chill knowing
       | that the developers care.
        
       | thom wrote:
       | Small world: my first job out of university was working for Pete
       | at the first company described here. It definitely was the kind
       | of place you could add "_old.asp" to any URL and probably hit
       | something. Ultimately I'm not sure there was much value we could
       | have salvaged in our position however we handled support or our
       | wider roadmap: this was a regional news group who fundamentally
       | rejected the idea that the internet was going to destroy their
       | classifieds business, and later couldn't cover the liabilities of
       | its enormous gold-plated pension fund. Every business must decide
       | how to optimise flows of value, but it's just as bad doing a good
       | job at the wrong ones, as it is doing a bad job at the right
       | ones.
        
       ___________________________________________________________________
       (page generated 2024-10-05 23:02 UTC)