[HN Gopher] Atlassian: We estimate the rebuilding effort to last...
___________________________________________________________________
Atlassian: We estimate the rebuilding effort to last for up to 2
more weeks
Author : tosh
Score : 333 points
Date : 2022-04-11 16:03 UTC (6 hours ago)
(HTM) web link (twitter.com)
(TXT) w3m dump (twitter.com)
| clhodapp wrote:
| I wish companies would stop with this "small number of customers"
| messaging. It always seems disingenuous and, besides, that
| matters for your _internal_ estimation of business impact but
| means absolutely nothing to the customers affected.
| tedunangst wrote:
| I think they add that because otherwise you read about the
| problem, panic, and then spend hours digging through your own
| data to make sure it's all there. Unaffected customers like
| being told they're not affected.
| dredmorbius wrote:
| https://nitter.kavin.rocks/kjartanmuller/status/151346261603...
| encryptluks2 wrote:
| I'm honestly surprised their stock hasn't plummeted yet. There is
| no way I could ever invest in a company that was this terrible.
| TameAntelope wrote:
| Not sure why an issue affecting a tiny number of clients would
| crater the stock.
|
| I've been pushing for an exit from Jira for a little while now,
| but this doesn't really add much ammo to that argument for me.
| It's like pointing at a plane crash and trying to justify the
| company no longer fly people places.
| Nextgrid wrote:
| If technical competency had any bearing on stock prices they
| should've been at 0 since long ago. Their stock price is tied
| to the amount of clueless/shitty companies that will still
| cling onto their products regardless of what happens, and I
| don't think this incident is going to change much.
| danuker wrote:
| If the market did not react instantly, it will certainly do
| so at the next financial report.
| whiteboardr wrote:
| Honest question:
|
| Since Trello is part of Atlassian aswell - what are good,
| reliable and above all lightweight alternatives for managing
| projects without the "pseudo-agile" rabbit holes of
| functionality?
| WorldMaker wrote:
| Microsoft Planner is included in most Microsoft 365 plans.
| Pretty much if you've got Teams, you've got Planner (and
| you can just add Planner as Tab in a Teams channel). At
| this point it has surprising feature parity with Trello.
| nvr219 wrote:
| Last time I checked (a few months ago), Planner still
| could not be backed up. Like, at all. If someone went in
| and deleted a whole bucket you can't recover it, not
| natively, and not with third party. So that's a big fat
| no from me.
| WorldMaker wrote:
| I seem to recall dumping Planner to JSON easily enough. I
| don't know if you can easily restore directly from its
| JSON, but the JSON was nice enough to work on for what
| little I needed to do with it.
|
| My current employer shut off Trello and forced us over to
| Jira and is threatening to disable Planner, so I'm "not
| allowed" to rely on Planner enough day-to-day so it's
| possible it is either better or worse than I remember it
| being in that department. But this Jira outage has me
| reevaluating, and they haven't turned off Planner _yet_.
| maclockard wrote:
| https://linear.app is pretty great
| indigomm wrote:
| Free until the end of the year if you get in touch with
| them - https://twitter.com/karrisaarinen/status/151357424
| 9121812482
| rglover wrote:
| I love stuff like this happening, not for the schadenfreude but
| for the "what not to do" lessons.
|
| #3,198,191 Don't automate deletion scripts w/o sufficient
| recovery options.
| polski-g wrote:
| Bigger lesson is: do not utilize cloud solutions. Your
| organization's competency floor is set to the least competent
| employee of the cloud provider.
| nvr219 wrote:
| I'd disagree with this lesson. Saying "do not utilize cloud
| solutions" period is nuts. Google and Microsoft are way
| better at email hosting and delivery than your on-prem server
| is unless you spend a ton more money on hardware and
| engineers to keep it up, which is simply not worth it for
| many companies. Dropbox is going to have better uptime and
| lower TCO than your self-hosted owncloud instance.
|
| What I will say is it's important for the customer to HAVE
| THEIR OWN BACKUPS. Don't rely on the vendor - that's the
| lesson here. If you have all your stuff in AWS back that data
| up someplace that's not AWS, etc.
| manquer wrote:
| Have your own backup works only if there are common open
| standards for export/ import. Email or storage may have
| those, project management tools don't. I can't simply
| backup from Jira and start using pivotal.
|
| Even for Email or storage or any other open system, UX
| changes and feature differences can take a lot of time to
| train properly, you don't migrate from one vendor to
| another vendor just like that.
| josephcsible wrote:
| There's something humorous when better uptime is cited as a
| reason to use cloud software instead of on-prem in a thread
| about a major cloud outage.
| nvr219 wrote:
| I mean, yes, your point is well taken, but I stand by my
| point nonetheless.
| throwaway2048 wrote:
| Good luck being able to use, or even make backups of
| proprietary SaaS offerings.
| nvr219 wrote:
| You can make backups of the data absolutely. There's a
| whole industry selling cloud-to-cloud backup solutions.
|
| Some examples: https://www.ownbackup.com/
| https://www.backupify.com/ https://www.cloudally.com/
|
| Edit: to clarify I would say if the data is important to
| you, then "the ability to back up the data" should be a
| requirement when selecting saas. See my other comment in
| this thread on ms planner.
| lapser wrote:
| For those of us not up to date, what exactly has happened? Their
| status page hasn't actually shown why they're having to rebuild.
| haunter wrote:
| >While running a maintenance script, a small number of sites
| were disabled unintentionally.
|
| https://twitter.com/Atlassian/status/1511870509973090304
|
| Most likely they wiped the data
| btgeekboy wrote:
| Supposedly they're having to basically restore everyone from
| backups because a system designed to delete old data was a bit
| more efficient than it should have been:
| https://reddit.com/r/sysadmin/comments/u14qqq/_/i4a0mk8/?con...
| xenadu02 wrote:
| Reminder: never delete data for real as your first step.
| Always mark it deleted along with a time stamp saying when.
| Then you can hide deleted itemsfrom everything. When a
| maintenance script goes haywire you can fix the problem
| quickly. Have a daily job that really deletes records marked
| deleted after 30 days.
|
| If that is too complicated to retrofit then have any mass
| cleanup script move the records to a CSV file or temporary
| table.
|
| Never ever ever be in a situation where a rogue script or bad
| SQL WHERE clause means restoring from backups.
| vbezhenar wrote:
| What's wrong with restoring from backups? This is one
| reason they exist after all. I don't think that making a
| mistake in delete statement is something you would do every
| week.
| lonelyasacloud wrote:
| Could be that they are restoring from backups - just that
| that restoring from those backups is very very slow.
| Atlassian would not be the first where resourcing and
| testing a speedy disaster recovery strategy wasn't given
| the highest engineering priority.
| btgeekboy wrote:
| This whole debacle is Exhibit A.
|
| They're lucky they have a sound backup strategy in place,
| and that the amount of data lost is appearing to be
| minimal.
| justin_oaks wrote:
| I agree that data must never be "deleted and forever gone"
| unless you've already been very sure about it a few time.
|
| But I would like to warn people about certain
| implementations of database "soft deletes" that I'm not a
| fan of. To be clear, I'm talking about the idea of having a
| "deleted" and/or a "date_deleted" column and using those
| columns in the WHERE clause to filter out rows that
| shouldn't be visible.
|
| That pattern complicates the table structure, queries, and
| indexes. It increases table and index size, thus more data
| has to be sifted through (either table data or index data)
| to ensure only non-deleted entries are returned. More data
| to go through means slower queries. It's also really easy
| for people to write SQL that accidentally leaves the
| "deleted" column out of the WHERE clause. Then old,
| irrelevant data is being returned.
|
| Accidentally deleting data that needs to be undeleted is
| usually rare so I don't think people should optimize for
| it. We should optimize for things that happen frequently.
|
| I have dealt with the rare "Oops! I deleted important
| data!" by restoring from backups and it has worked fine. I
| think it may be too strong to say you should never be in a
| position to restore data from a backup. In fact, I think
| it's important to streamline the restore process.
|
| For cases where we know ahead of time that we want to query
| deleted data I'll move deleted data to another database
| table that exists solely for maintaining history. For
| example, an ORDER table will have a DELETED_ORDER table, or
| an ORDER_HISTORY table. The HISTORY tables can also record
| data overwritten from updates.
|
| These tables take up disk space, but never affect the
| structure or size of the original table and its indexes.
| Queries to the original table don't need to be modified to
| account for soft deletes.
|
| To guarantee that things go to the delete/history tables,
| I'll usually put a trigger on the original table to move
| data over to the history tables. This way no application-
| specific code is needed.
| Trasmatta wrote:
| Agreed, soft deleting adds so much complexity to
| everything. And even has the potential for privacy
| related bugs. Like, say, accidentally forgetting to
| respect the deleted column in a query on a joining table
| that determines user permissions for some resource. Now
| people have access to something they had permissions
| revoked for. Whoops.
| bartread wrote:
| > Accidentally deleting data that needs to be undeleted
| is usually rare so I don't think people should optimize
| for it.
|
| That's very use case dependent.
|
| We've made it easy for people to undelete data they've
| accidentally deleted simply because they used to do it so
| often and the only people who could get it back were our
| tech team. We're a devops org so part of our job is of
| course to support the systems we build, but our time is
| better spent on building solutions to business problems
| than to repeatedly providing support for issues that come
| up all the time. Part of building those systems is of
| course engineering in solutions that make it hard to
| screw up, and easy to unscrew when things inevitably do
| go wrong. No mean feat given our platform dates back over
| 15 years and still includes a lot of legacy from the time
| when tech was just a couple of people.
|
| I suppose the object lesson here is that edge cases in
| one system or company can be part of core business in
| another so it's best not to make too many assumptions.
| izacus wrote:
| A "deleted" field type deletion is also how you get a
| massive fine from a GDPR agency when they find out that
| you're not actually deleting PII properly.
| noasaservice wrote:
| One smaller social media app I used to sysad over
| actually overwrote data to be purged as xX0-Deleted-oXx
| and similar (there were a few variants depending on data
| constraints). There was no "show_deleted_when" garbage.
|
| Then weekly, a task went in and then purged rows with
| those non-content placeholders to completely purge that
| user, if a user-purge was requested.
| schrismartin wrote:
| How does that work around GDPR and other "Right to be
| forgotten" legislation? Aren't we required to hard-delete
| this kind of data?
| vbezhenar wrote:
| What is hard deletion? You can restore rows from database
| files before vacuum runs. You can often restore data from
| disk sectors. Some people say SSD can remap sectors under
| your chair and you won't even know that your deleted data
| is there.
| 12baad4db82 wrote:
| As long as the data is deleted within a month there
| should not be any GDPR concerns.
|
| > The data subject shall have the right to obtain from
| the controller the erasure of personal data concerning
| him or her without undue delay
|
| > "Undue delay" is considered to be about a month
|
| https://gdpr.eu/right-to-be-forgotten/
| lazide wrote:
| You are within a certain period of time, not 'instantly'
| (depending on the exact situation you are referring to).
| The script could take that into account (using a shorter
| period of time or the like)
| danuker wrote:
| > or bad SQL WHERE clause
|
| Tip: Begin an SQL session with BEGIN TRANSACTION; at the
| end you can either COMMIT or ROLLBACK.
| Trasmatta wrote:
| Another tip: never enter queries directly in a production
| database connection with write access in the first place.
| (Ideally very few people even have that level of access.)
| Write it in your codebase, write tests for it, get it
| code reviewed, and run it in a dry run first and get a
| list of affected records before running it for real.
| davio wrote:
| I always liked LIMIT when doing dangerous stuff in prod
| bmurphy1976 wrote:
| Bonus tip: don't even type COMMIT until after you've run
| a few select queries to verify the data has in fact
| changed how you expected it to change.
| beart wrote:
| super duper bonus tip: Don't execute SQL in a production
| environment that you just typed out on the fly.
| mdoms wrote:
| It's incredibly depressing how common this is in the real
| world, but I can tell you from experience that this NEVER
| happens in Atlassian production systems.
| m0llusk wrote:
| This is also true for people cleaning house or going
| minimal. Put it in a well labeled box and tuck it away
| somewhere. If it matters then you'll fetch it, otherwise
| just chuck it when the box gets in the way.
| bpp wrote:
| Works until you get a bug in the deletion job. I've seen
| exactly this happen.
| jdbernard wrote:
| Yeah, the idea is that by expecting the deletion logic
| you can make it simpler and more rigorously tested than
| regularly changing business logic or application code.
|
| If you organizationally cannot prioritize quality then
| nothing can help you.
| hyperman1 wrote:
| You don't even need a bug. Just a wrong system clock.
|
| We had a few windows laptops where something caused them
| to time travel to 8000 years in the future. Then, they'd
| slowly spend a few hours deleting every local profile, as
| nobody had logged in to them for 8000 years. Then, they'd
| do something to their time zone database and travel back
| 8000 years.
|
| When they started the process, it was unstoppable. Trying
| to modify the system clock to something sane just caused
| them to depart to the future again, even if disconnected
| from the network. None of our users was very amused by
| this behavior, even if everything important was backed
| up.
| tetsusaiga wrote:
| that is almost cartoonishly nightmarish
| hyperman1 wrote:
| It happened specifically to 1 type of laptop and we only
| had about 30 of them. So we pulled all of them out of
| roulation. Then covid struck, so I reformatted most of
| them with Debian and we gave them away for home
| schooling. I wonder if I managed to linuxify some kid in
| the process.
| ghostly_s wrote:
| Considering the sorry state of videoconferencing on Linux
| they probably all immediately had Windows reinstalled.
| bityard wrote:
| Not sure what this means, I've been using video
| conferencing on my Linux laptop for work on a daily basis
| for the last 5 years.
| mdoms wrote:
| > This data was from a deprecated service that had been moved
| into the core datastore of our products.
|
| That is _very_ interesting. This implies they are backing
| off, at least somewhat, from their very aggressive
| microservice strategy. Perhaps they feel like they have gone
| too far in decomposing their products.
| jeffybefffy519 wrote:
| Or ransomware got them.
| seigel wrote:
| I am very surprised that there aren't more people/companies
| announcing "offsite backup capabilities" for JIRA or Confluence,
| etc.. I did just search for scripts that can do this, I'd
| probably pay a few bucks for something like that at this point.
| aaronbrethorst wrote:
| The wildest part about this service outage to me is that
| Atlassian's stock is up 10% over the past month.
| threeseed wrote:
| Might have more to do with them moving their HQ from UK to US.
|
| This would allow them access to more investors.
| Kwpolska wrote:
| Breaking news: stocks are meaningless numbers.
| [deleted]
| Thorentis wrote:
| From this thread, it sounds like Atlassian needs some better
| competition ASAP.
| mattweinberg wrote:
| More discussion here:
| https://news.ycombinator.com/item?id=30973808
| [deleted]
| [deleted]
| polski-g wrote:
| Hopefully this will make them rethink getting rid of the self-
| hosted option, as you'd be insane to use their cloud service
| after this.
| bdefore wrote:
| Engineers, do yourself a favor and add to your CV/resume: I will
| not work for companies that use Atlassian software.
| k8sToGo wrote:
| So like 90% of the companies?
| dylan604 wrote:
| Yeah, it's not really a good piece of advice. If anyone
| actually reads that bit of information, they'll just shrug
| and toss the resume in the bin and move to one of the other
| hundreds of applicants.
| threeseed wrote:
| And use what else.
|
| Their competition is equally bad and offers far less features
| for larger programs.
| floatinglotus wrote:
| Atlassian does not care about individual customers. They are
| purely driven by numbers. I listened to a presentation by one of
| their founders a long time ago where he admitted that statistics
| and number management was part of their DNA. They don't think
| about the customer's name, maybe this is right, maybe not.
| Hamuko wrote:
| My hot take is that this downtime isn't severe enough since my
| company hasn't noticed it at all, and I can't lobby for an
| alternative project manager.
| debarshri wrote:
| I guess they are using JIRA for planning their sprints.
| Rantenki wrote:
| As someone who is impacted, this is obviously immensely
| frustrating.
|
| Worse, outside of "we have rebuilt functionality for over 35% of
| the users", I haven't seen any reports from the people who have
| ostensibly been recovered.
|
| Next, their published RTO is 6 hours, so obviously they must have
| done something that completely demolished their ability to use
| their standard recovery methods:
| https://www.atlassian.com/trust/security/data-management
|
| Finally, there have been some hints that this is related to the
| decommissioning of a plugin product Atlassian recently acquired
| (Insight asset management) which is only really useful to large
| organizations. I suspect that the "0.18% impacted" number is
| relative to ALL users of Atlassian, including free/limited
| accounts, and that the percentage of large/serious organizations
| who are impacted (and who would have a use for an asset
| management product), is much higher.
| uuyi wrote:
| From experience: RTO = recovery time _objective_ = lie
| [deleted]
| mey wrote:
| Don't have a way to verify this, since you know, but I don't
| believe we had this plugin installed/enabled, but are still
| caught in the blast.
| mdeeks wrote:
| RTO is so hard to state properly even with regular testing. If
| someone blows away a critical database, sure you can meet your
| published RTO. What if we lose 300 of our databases and need to
| copy snapshots from another region. AWS limits you to 20
| concurrent snapshot copies cross region. Which of those
| databases should you do first? Do you know your entire
| dependency graph for all 1000 of your services to make the
| right call? Meanwhile tick-tock your "6 hours" is slipping
| away. And what if someone nukes our entire AWS account with all
| of our prod resources? Databases, load balancers, S3 (no such
| thing as snapshotting there), EC2 instances, etc, etc
|
| Those last two are examples are very unlikely but no company is
| going to say RTO = "probably 6 hours but it could be three
| weeks if we get ransomwared"
| manquer wrote:
| For what you suggest some combination of these things should
| have happened. - Some employee has root
| access to AWS account and uses it operationally - Given
| wildcard S3 permissions to an IAM user and allowing delete
| bucket - Not enabled object versioning - Cross
| Region replication not enabled - no large bucket
| protection - don't have basic security monitoring and
| setup of Cloudtail alerts - have not invested in full
| fledged tools for IDS and so on.
|
| If some vendor have any of these issues I don't think any
| customer would approve these software to be used, these are
| not normal or best practices .
|
| Large apps have detailed playbooks on how and what gets
| turned on in what order, and most do DR drills and time those
| runs periodically. These are well established workflows in
| any large org.
|
| Yes in a real world downtime you can't have planned for every
| scenario, maybe you miss the target by 25 % like 2 hours
| more, or maybe in a very situation you double or even triple
| it say 12-18 hours. You don't go from 6 to 600+ .
|
| The way RTO is calculated starts by looking at limits on
| cloud/ hardware / bandwidth/ machine sizes, if basic limits
| are not factored in like cross region concurrency there is no
| point in RTO being computed. Even if something like that was
| missed and you spend tens of millions of dollars on AWS then
| AWS will work with you and relax those limits .
|
| 100x missing the plan either means extremely poor planning or
| they screwed up something very very badly.
| rcurry wrote:
| Reminds me of the time a group I worked for at a National Lab
| decided to call the root folder for their project "core". I bet
| you can't guess what filename the backup scripts were configured
| to ignore...
| outsb wrote:
| This is an expensive lesson I think everyone gets to learn at
| some point. There's no such thing as a file worth excluding
| from a backup, it always fucks you, some (like me) more than
| others. Have to buy twice as much disk? Fine, at least you know
| you actually have a backup
| lmc wrote:
| Makes total sense - the core is the part of the apple everyone
| throws away.
| x86_64Ubuntu wrote:
| 2-weeks is one hell of a Recovery-Time-Objective.
| vntok wrote:
| Don't complain, you'll get a few months worth of free credits
| or something.
| Nextgrid wrote:
| I bet they screwed up royally, deleted some data and are down to
| either rebuilding it from logs, caches or other side-effects, or
| using data recovery software on the storage drives (which might
| involve third-party companies). I can't see many other reasons
| why this should take 2 weeks.
| mrits wrote:
| A few years ago I depended on Vertica (snowflae/clickhouse)
| type database.
|
| When a node went down there was no hope of ever coming back up
| unless you shut the other nodes down as well. While this was
| going down of course none of our ingress data was being
| inserted so it built a queue. When we turned things back on the
| queue would overload vertica again and we had to repeat the
| whole thing.
|
| Fortunately for us we only stored analytic type data on vertica
| where customers usually were only interested in the last few
| hours anyways. So we ended up deleting all historical data and
| just reprocessing it over months while occasionally
| prioritizing customers that complained.
| jrib wrote:
| this is a scary thought. I need to start being more aggressive
| about backing things up that are "in the cloud."
| laurent92 wrote:
| Let me bet: Rebuilding from Jira email notifications. Yes, the
| diffs in the notifications.
| BFLpL0QNek wrote:
| Not much on this on the internet.
|
| Gray*Duffy, LLP Settles Two Massive International Data Loss
| Claims Arising From Computer Server Failures
|
| Arc Touch, Inc. vs. Atlassian PTY, Ltd
|
| http://m.grayduffylaw.com/?url=https%3A%2F%2Fgrayduffylaw.co.
| ..
|
| https://www.theregister.com/2012/05/09/atlassian_cloud_stora.
| ..
| mirntyfirty wrote:
| But are they gonna be able to accomplish anything without
| Jira guiding their projects?
| daigoba66 wrote:
| I've actually done exactly that many years ago for a self-
| hosted Jira installation that didn't have any backups. You
| can bet we had backups with regular testing after that.
| uuyi wrote:
| I would have just quit if I had that landed on me.
| Aachen wrote:
| Honestly it doesn't sound too difficult and like a
| challenge to script something fun. To me. If you ever
| find yourself in that situation, email is in my profile
| and we'll work something out :)
| perlgeek wrote:
| So, let me get this straight:
|
| * It's been deleted for a week already, they estimate they might
| need two more weeks. Three in total.
|
| * They claim to have "extensive backups", and hundreds of
| engineers working on it.
|
| What? How? This simply doesn't go together. Why would restoring
| from backup take three weeks?
|
| Either their backups aren't complete, or they need new software
| written for the restore, or something else doesn't add up.
|
| I haven't administered their software yet, but what I've learned
| from the sidelines, at least Jira doesn't seem to be rocket
| science. A database, an application server (maybe a few instances
| for larger sites), a bit of config, some caches. This _really_
| shouldn 't take three weeks to restore.
| m0llusk wrote:
| If you are not regularly testing restores then you don't have
| backups.
| raffraffraff wrote:
| I'd say this is the answer.
|
| I remember a situation where we had a near miss with data
| loss (replica failed and master had a bad disk). We didn't
| want to put the production database under extra load by
| taking a live backup while it was handling all production
| traffic, so we restored a backup. But it was "bad". Tried the
| one before it, and the one before that. Apparently they were
| busted for over a month due to a config change. We restored a
| month-old backup and started applying binlogs (which
| thankfully we had been backing up). But that meant replaying
| a month of transactions into the restored database. I can't
| remember the details but I think we ended up replacing the
| bad disk, resilvering the array and live-cloning the primary
| before the binlogs got fully applied to the one we restored
| from the old backup.
| tiahura wrote:
| Went through that once in the mid- late 90's. Each restore
| and test took hours, so 3-4 attempts took 2 days - with me
| sleeping, crying and praying in a conference room.
| tpmx wrote:
| I have no specific information on this, but in general and in
| theory:
|
| Isn't this how an incompetent, insincere and desperate company
| being subjected to a ransom attack would communicate publicly?
| mdoms wrote:
| Jira cloud became much more complex when they went all in on
| AWS. The reason for the cloud/server fork 4 or 5 years ago was
| so that cloud engineers could couple to a zillion AWS services
| without having to build back any backwards compatibility. So
| data stores are very much more disparate than just a PgSQL DB
| and redis (which is how it used to be).
| Nextgrid wrote:
| My theory in my other comment is that they've deleted some data
| and are waiting on third-party data recovery specialists. That
| would explain the timescale.
| jlbooker wrote:
| Restore from off-site tape backup. The kind of service where
| you ship them ~dozen new tapes in a lockbox each week and they
| ship you the oldest dozen back. It's supposed to be the "if all
| of a data centers happen to burn to ashes simultaneously"
| option. If you say "give us all of our tapes, asap" and then
| have some pour souls swapping them out as fast as the data can
| be read... it would probably take a few weeks.
| RL_Quine wrote:
| Tapes are not that slow, 360MB/s per drive, and on a large
| scale swapping them is completely automatic.
| stonogo wrote:
| In support of your point, 360MB/s is an extremely
| conservative estimate. I'd expect that from LTO-6, which is
| around ten years old, and I would certainly hope their
| backups are on more modern gear than that.
| planb wrote:
| As the person responsible for running Jira and Confluence on
| premises at my employer I'm looking forward for the next time one
| of their sales droids contacts me to make us move to their cloud
| services (despite me stating that we are not interested multiple
| times)...
| uuyi wrote:
| I would love to listen to that call.
|
| There's nothing that makes me happier than the fearsome
| squealing noises that enterprise sales drones make when you
| drop the sales equivalent of a Paveway IV on their pitch.
|
| My favourite one was running some software supply chain
| compliance software on itself and explaining how it was
| constructed on top of a CVE riddled garbage dump.
| dylan604 wrote:
| My favorite experience with a sales rep (not Atlassian
| related at all) was when I went to the vendor's booth at NAB
| (huge industry convention for those not familiar). I saw our
| sales rep who quickly looked down to catch her breath before
| greeting me. I smiled and let her know that today was her
| lucky day if she could just introduce me to the tech team she
| promised I could meet. At one point in my conversation with
| the tech team, I noticed that a small crowd had gathered
| around my conversation. I was not intending to hold public
| court, but I was not going to miss a chance to talk directly
| in person to these team members. To a non-tech person, it
| might have been viewed as confrontational. To a tech person
| to tech person, it was just direct questions being held to
| the fire for an actual answer vs CSR/Sales rep platitudes.
| senorsmile wrote:
| They are discontinuing on prem after 2022 though. What is your
| plan if not to migrate to their SaaS offering?
| oldshatterhand wrote:
| Switching platforms
| uuyi wrote:
| Everyone should say no. That'd change their policy pretty
| quickly.
| crawl_soever wrote:
| What are disconnected customers going to do? Surely they
| won't cut off the entire US military and DoD by terminating
| that cash cow of on premises.
| bobnamob wrote:
| They're still offering on-prem for airgapped usecases
| afaik. It's just become a "contact us" pricing plan
| joshuaissac wrote:
| It is not a "contact us" pricing plan. The Data Center
| prices are publicly available on their website. It is
| more than twice as expensive as the Server offering, but
| still substantially cheaper than Cloud for the same
| number of users.
|
| And it is the exact same software as Server with some
| extras enabled like support for multiple nodes, so
| upgrading to it is as simple as pasting in a new product
| key.
| planb wrote:
| We will probably bite the sour apple (not sure if this is
| correct English but you'll get the meaning even if it's not)
| and switch to the data center edition which is still on prem
| but costs approximately twice as much.
| netsharc wrote:
| You can probably use this 1 week (3 scheduled) outage to
| ask for a discount, "your cloud offering is a bucket of
| shit, your data center edition is too expensive, my higher-
| ups told me to find something else...".
| IncRnd wrote:
| You can say, "bite the bullet", which means you accept the
| the difficulties you know will arise. It's actually a
| perfect idiom for this case.
| MandieD wrote:
| A bit from the Boom Chicago show I saw in Amsterdam around
| 2004: "The Dutch expression 'bite the sour apple' means the
| same thing as 'bite the bullet' - Americans are obsessed
| with guns, and the Netherlands is full of shitty food."
| IncRnd wrote:
| > Americans are obsessed with guns, and the Netherlands
| is full of shitty food.
|
| "Bite the Bullet" is a phrase from an Englishman, Rudyard
| Kipling, in his first novel, "The Light That Failed".
| It's believed to have come from the other English idioms,
| "to bite the cartridge" and "chew a bullet", which date
| back to 1891 and at least 1796. [1]
|
| [1] https://en.wikipedia.org/wiki/Bite_the_bullet
| gpm wrote:
| > not sure if this is correct English but you'll get the
| meaning even if it's not
|
| Not an idiom I'm familiar with, but it's grammatically
| correct and clear so I'd count it as proper.
|
| "Bite the bullet" is commonly used.
| breakingcups wrote:
| It's insane. Since they've all but killed off on-premise Jira
| and Confluence, they've been spamming me regularly trying to
| convince me to "upgrade" to the cloud. Eventually I gave in and
| replied to them asking two simple questions.
|
| One is that we've had many bad reports from partners that Jira
| Cloud is _incredibly_ slow, even when compared to the already
| underperforming Jira Server and I wonder what their performance
| guarantees are. The other one is that it 's so, so pricey.
|
| It helped, but not in the way I thought it would! There's been
| no reply, but also no more spam emails now!
| case0x wrote:
| >Jira Cloud is incredibly slow
|
| I have to say it got better after they switched to AWS but
| Jira not working/being slow is still an inside joke in the
| office
| rconti wrote:
| My employer recently switched from on-prem to cloud. The
| cloud service is insanely slow, or maybe it's my aging
| Macbook, but every single component on the page seems to have
| to load separately. (It's a newer UI versus what we had on-
| prem).
|
| Thankfully we haven't been impacted by this outage.
| andrekandre wrote:
| > or maybe it's my aging Macbook
|
| on an m1 pro: its slow as molasses
|
| sometimes i have to switch to the "old ui"[0] to get any
| use out of it (not sure what the cause is, but sometimes
| its literally unusable)
|
| [0] https://community.atlassian.com/t5/Jira-questions/Re-
| How-do-...
| judge2020 wrote:
| I've also heard that the cloud service is slow; you can
| easily check if it's your machine or the server by watching
| devtools -> network tab and seeing how many requests are
| `(waiting)`, because chances are it's Atlassian's server
| speed.
| dijit wrote:
| Can try the native Mac app until it's discontinued.
| Surprising how fast it is.
| deergomoo wrote:
| > Jira Cloud is incredibly slow
|
| I haven't used Jira Cloud in any great depth, but I did play
| around with it as part of trialling the free plan and was
| amused that you could quite easily come across warnings to
| backup your installation and consult your system
| administrator before proceeding...how exactly do I do that
| for a cloud service? Doesn't exactly inspire confidence.
|
| At my last job we used Bitbucket Cloud and that was _awful_.
| Dog slow, ridiculously low threshold for being unable to
| render diffs, and constant incidents. We used to joke that
| they could make the "Bitbucket is experiencing an incident"
| banner a permanent fixture on the page and it would be right
| more often than it was wrong.
|
| We're still using on-prem Jira at my current job, but we just
| migrated away from on-prem Bitbucket to GitHub, as Bitbucket
| was becoming infeasible and the cloud offering is a bad joke.
| znpy wrote:
| > I haven't used Jira Cloud in any great depth.
|
| I have. It's painfully slow.
|
| But slowness isn't really the problem, the problem is that
| it's unpredictable.
|
| I wait for the interface to be fully loaded, so I click on
| a text box and i start typing. Then FU--ING something takes
| the focus to some other element in the web page and now i'm
| typing random shortcuts (like reassigning tickets, changing
| status or whatever).
|
| It's painfully slow but the real problem is that it's
| unpredictable in its behaviour.
| ivanvas wrote:
| Looking at the company financials and timing, the wooden
| headquarters might have to wait.
|
| I don't envy engineers there right now, some dpts. Stay strong,
| don't burn out!
|
| https://architectureau.com/articles/worlds-tallest-hybrid-ti...
| mh- wrote:
| Wow, those renderings are gorgeous. (If you click the top
| photo, there's a gallery of 5 more you can scroll through)
| threeseed wrote:
| Atlassian Q2 revenue was up 37% y/y.
|
| Those sort of growth rates are incredible for a company this
| size.
|
| They can easily afford their timber treehouse.
| barnabee wrote:
| If I was made to use Jira I'd be pretty happy about this
| alligatorplum wrote:
| My company is in the middle of a multi year transition from
| selfhosting atlassian products to using their cloud offerings,
| and I am sure the infrastructure team/management is very thrilled
| to see this news.
| croutonwagon wrote:
| While our tenant was unaffected, i told my management of this
| issue. They just shrugged and said we could watch and eat
| popcorn. I was half way expecting them to raise eyebrows
|
| I was kinda like....."not really the point of bringing it up."
|
| Its worth noting we have had them just delete things within our
| account before. In fact one of our Senior VP's had their
| account just....disappear one day. We couldn't @ them in chats,
| tickets etc. Atlassian just shrugged and "restored" the account
| and said it was some issue with a stored proc on their backend
| or something.
|
| I have always felt uneasy about how flippant they are in their
| processes. But it seems that is not shared.
| GeorgeTirebiter wrote:
| I'm sure they will also be thrilled waiting sometimes seconds
| for a character to echo. Good Luck, I suffer with cloud
| Atlassian every day.
| yabones wrote:
| We're mere weeks aware from migrating to their could platform
| after the self-hosted rugpull. This really doesn't give me
| confidence in their ability to not break my stuff.
| uuyi wrote:
| Same situation.
|
| Absolutely no one even knew this was happening and doesn't give
| a shit now because it's a project death march.
|
| JIRA as a whole has been a fucking shit show of a product over
| the last decade even on-prem.
| digerata wrote:
| I will say that back when Bugzilla was it, JIRA rocked. It
| was amazing the new power you had and the functionality it
| provided.
|
| We self-hostd JIRA from 2008ish to 2014ish. (memory is fading
| on exact dates) By the time we decided to stop using JIRA, we
| fracking hated JIRA and would never return.
|
| Since then, GitHub Issues, Trello, Clubhouse (nee Shortcut)
| all provide less friction in day to day use. As an
| Enterprise, I do believe Shortcut is your best bet.
| timcavel wrote:
| justin_oaks wrote:
| There are two ways this can go:
|
| 1) This outage will get their organization to prioritize work
| such that it never happens again.
|
| 2) This outage is representative of a dysfunctional
| organization that can't prioritize work correctly.
|
| If you've been using Atlassian software for a while and are
| used to how they prioritize tickets then one of those options
| seems far more likely than the other.
| laurent92 wrote:
| > 1) This outage will get their organization to prioritize
| work such that it never happens again
|
| It has already happened in the past.
| justin_oaks wrote:
| Ah, you mean that outages have happened in the past, not
| that Atlassian has changed because of an outage.
| uuyi wrote:
| 2012. https://www.theregister.com/2012/05/09/atlassian_clou
| d_stora...
| lmc wrote:
| "An Atlassian spokesperson contacted by The Reg blamed
| the loss on the failure of "multiple" storage disks at
| its cloud storage provider on 28 April."
|
| 10 years ago to the month.
|
| So as long as everyone plans a vacation for this time in
| 2032 it'll be OK.
| [deleted]
| thanatos519 wrote:
| "could platform" ... perfect typo!
| abraae wrote:
| Why would you continue with that plan? You couldn't get a
| clearer warning signal.
| yabones wrote:
| To be quite honest, the problem is historical. We have over a
| decade of project plans, support tickets, change control
| logs, etc in our Jira instance. There's simply no painless
| way to export that into another product that will have
| approximately the same functionality and features. There's a
| few that come close, but all fall short of a drop-in
| replacement.
|
| The only options now are the $$$$$ "datacenter" license,
| migrating to the dangerously unstable cloud, or not doing
| anything and running unsupported EOL software.
| joshuaissac wrote:
| > the $$$$$ "datacenter" license, migrating to the
| dangerously unstable cloud, or not doing anything and
| running unsupported EOL software
|
| Data Center is pricier than Server, but isn't it still
| cheaper than Cloud? And you control your own back-ups as
| with Server, so Atlassian cannot lose your data.
| abraae wrote:
| I totally understandthat. But at the same time, a multi-
| week outage is really a sign of an org that simply does not
| have their shit together at all.
|
| But the lack of transparency is the worst. Another post
| speculated that Atlassian has lost data, doesn't even have
| backups, and is re-creating it by munging their emails and
| diffing them to re-create history. I can't really imagine
| that's tue - but what if it is, and Atlassian is concealing
| things?
| api wrote:
| It's funny that they recently killed self-hosted Jira. If you'd
| self-hosted you'd be fine.
| nolok wrote:
| To be fair, Jira screams SaaS.
|
| I don't want any of my company trapped on it, but if they were
| I'm sure as well not going to self host that spawn of hell.
| msla wrote:
| Self-hosting Jira is running a big Java application that has
| its own directory structure and talks to a Postgres database.
| It really isn't hard. Even upgrades are automated using shell
| scripts, although you do need to manually replace some
| configuration files afterwords for some damn reason.
| _Configuring_ Jira is complicated, but that 's the same
| regardless of whether you self-host.
| shrdlu2 wrote:
| Self-hosted full Atlassian stack (jira, confluence, bamboo,
| bitbucket) for 6+ years. ~200k tickets. To be honest, it just
| runs with minimal issues.
|
| Mostly downtime is just upgrades. I can remember a few times
| we've had to add (JVM) memory as our usage increased. Not
| sure what we're going to do with the discontinuation of
| server product line. We self-host to keep source code, etc.
| more than one configuration mistake (or zero-day) away from
| exposing it to the world.
| briffle wrote:
| We are in the same boat. We are looking at the cloud, but
| the migration tools are just a pain. I can migrate a
| project to the cloud from Jira, and it even gives me a
| report of workflow transitions I have to manualy
| update/change to fix, which is great.
|
| But then, there is no way to keep it in sync. I have to
| blow that project away in jira cloud, and migrate it again.
|
| So I Have to hard-cut over projects, on a system that has
| dozens and dozens of projects, and somehow have people
| figure out which ones are where. or one really, really ugly
| night to cut it all over, and hope it goes well.
|
| I'm looking for alternatives, but our team is so invested
| in some very, very customized workflows, its going to be a
| pain.
| api wrote:
| Self-hosted Jira sucked because it was shitty software, not
| because self-hosting has to suck. Mattermost and Jitsi are
| easy to self-host, to give two examples that are not complete
| shit.
| media-trivial wrote:
| We're using the cloud version and we're fine too (no outage).
| What's your point? Are you claiming that self-hosted is never
| down? Or that self-hosted is more reliable? Because I doubt
| that. Difference is just that when self-hosted goes down, it
| doesn't end up in the news.
| gregmac wrote:
| It sounds like Atlassian is doing individual restores, each
| restore takes a fair amount of time, and they don't have the
| capacity to do all 400 simultaneously (because why would
| they). So you just have to wait.
|
| If you're self-hosted, you dedicate as many people as
| possible/necessary to restoring your service, and it becomes
| their top priority.
|
| You also have a lot more insight into the detailed inner
| workings of the restore, making it easier to plan against,
| instead of just vague "we're working on it" messages for days
| a time with no clear end in sight.
| rileymat2 wrote:
| I think there is a strange psychological trait that I have,
| and others may as well, where I am much more forgiving
| breaking my own stuff than having someone else do it.
| kodah wrote:
| When you host Jira yourself, the monthly subscription pays
| for Software + Software updates. When Atlassian hosts it
| for you, you're paying for Software + Software Updates +
| Service (hosting). When you hosted yourself, a team gets
| blasted for not monitoring it or updating it correctly.
| When Atlassian fails to do it (and charges for it) then
| they get the heat.
|
| All that to say, I don't think it's a weird phenomenon,
| it's just you're realizing that you're paying _someone
| else_ for something that 's not delivered on.
| media-trivial wrote:
| People are more forgiving torwards themselves and their own
| folks, I can understand that. I'm just thinking it's
| important to make decisions based on facts. Some self-
| hosters walk around with this "my own basement is safer
| than Amazon datacenters" attitude and that's just not true
| (in most cases, I guess :D).
| akho wrote:
| In my years of selfhosting personal stuff I never had a
| service "sunset", never lost data, and never had
| significant downtime.
|
| In the external services I use, downtime of one service
| or other is to be expected at least a few times a year,
| and the "sunsets" happen occasionally.
|
| Thing is, public services are solving a much more
| difficult problem (keeping things running safely for
| millions).
| fisf wrote:
| No. The basic assumption is only, that if I pay somebody
| money for the service and to keep things running, then
| sudden data loss is unacceptable (and a multi week
| downtime even more so).
|
| Some companies cannot operate effectively without
| atlassian products, so a fuckup of that scale might just
| have legal consequences depending on whom it hits.
| vntok wrote:
| > Some companies cannot operate effectively without
| atlassian products
|
| Kind of their problem to be frank.
|
| > so a fuckup of that scale might just have legal
| consequences depending on whom it hits.
|
| Any contract those companies signed would have a cap on
| the retributions by Atlassian for trashing SLA targets
| omoikane wrote:
| One difference might be that when you self-host, you are more
| sensitive of some of the risks, whereas a hosting service
| might be balancing that risk with a need to scale, and their
| appetite for risk and tradeoff considerations might be
| different from yours. It might be that these companies know
| something that their customers don't, and thus are more
| willing to take on risks.
|
| In this case, it seems like the company took a risk and it
| did not go well. The possibility of being able to restore
| from backups might have been factored into this risk, but the
| latency of doing so might not have been.
| kuschku wrote:
| My own self-hosted services have about 20 min downtime a
| month.
|
| I've never - not at any point in the past 10 years - gone
| over 24h of downtime.
|
| JIRA will now have 3 weeks downtime.
|
| Distributed systems have complexity that grows superlinear,
| which leads to more and longer incidents.
| lousken wrote:
| if my company would be part of the outage, the restoration
| part would only take 2 days (if we're counting all the data
| they have, not just jira/conf) but not 3 weeks
|
| so selfhosting may still have certain upsides even with such
| outage
| Overtonwindow wrote:
| I worked at a company that self hosted Jira, and it was
| miserable then, I can't imagine depending on the cloud. I'll
| never approve Atlassian products after that experience.
| laurent92 wrote:
| How many tickets do you have in your non-Jira ticketing
| system? I count 100 tickets per person per year.
| throwawayboise wrote:
| > If you'd self-hosted you'd be fine.
|
| Maybe. But you're counting on your sysadmin(s), who are also
| managing dozens of other things, to keep up to speed on Jira
| and its quirks, and apply patches and new versions as they
| become available without missing any steps or screwing
| something up.
|
| On average, you're still probably better off having a company
| that knows the product also host it for you, but obviously they
| can make mistakes too, and the downside is that when they do it
| might affect all clients, not just one.
| nvr219 wrote:
| > and the downside is that when they do it might affect all
| clients, not just one.
|
| This is also potentially an upside. For example when us-
| east-1 went down recently, customers were somewhat
| understanding because it was "amazon's fault" and everyone
| was down - it was in the news, etc. If we ran our own data
| center and that went down, our customers would've just said
| "why did you morons roll your own data center instead of just
| using aws?"
| thematt wrote:
| They've only killed off one version of self-hosted Jira. Their
| datacenter edition is still alive.
| ejb999 wrote:
| I use JIRA and confluence every single day, I have to, it is
| everywhere - but imo it is such a horrific toolset in every way
| (even before this outage), I can't for the life of me figure out
| how it got so much market-share.
| alkonaut wrote:
| Because all the others are equally bad, basically. The options
| for enterprisey issue management are quite slim.
___________________________________________________________________
(page generated 2022-04-11 23:00 UTC)