[HN Gopher] Open Riak - open, modern Riak fork
___________________________________________________________________
Open Riak - open, modern Riak fork
Author : amarsahinovic
Score : 211 points
Date : 2024-11-19 20:29 UTC (1 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| amarsahinovic wrote:
| Lightning Talk: Introducing OpenRiak - Nicholas Adams:
| https://www.youtube.com/watch?v=0GLBsBeM4Kc
| carterschonwald wrote:
| Cool! I never used it but really liked the engineers I met who
| worked at basho on risk. AFAIK, they basically had an engineering
| dream team until their last ceo had them go hard in certain
| directions that didn't pan out.
| sitkack wrote:
| Thanks!
| chadd wrote:
| I used Riak for a project back in 2012, the app that became the
| Whisper App, and as a huge Erlang fanboy, I was so excited about
| it.
|
| But it was incredibly unreliable at scale, and my colleague and I
| spent a week of sleepless nights under incredible personal and
| business pressure - as the servers got busier and busier -
| ripping it out.
|
| Still love vector clocks, though, and have fond memories of the
| Basho team presenting at Erlang Factory
| amanj41 wrote:
| Vector clocks are very cool. Having read through how they were
| initially used in Riak, I was blown away that such an
| implementation could scale. I guess this is why Cassandra took
| a different approach?
| tibbar wrote:
| Vector clocks are certainly cool but fundamentally premised
| on the idea of having multiple 'live' versions of a value at
| once. Amazon's original Dynamo paper required conflict
| resolution at the application level, which is a very strange
| framework to build applications on. (Notably DynamoDB has
| moved away from this, I believe to Last Write Wins.)
| Cassandra takes the latter approach by default as well, I
| believe.
| amanj41 wrote:
| yes there's that idiosyncrasy, as well as client ideally
| needing to read the previous clock from the DB before
| writing an update for that key unless it's ok with the
| write being viewed as concurrent. Plus the extra memory
| overhead to store the clocks in the client.
| vosper wrote:
| Who would this be for in 2024?
|
| I remember evaluating Riak back in 2011 or so for an analytics
| solution, but ended up going with a more traditional OLAP
| database that was a much better option.
|
| It's hard for me to imagine where Riak would be a good option
| given how many choices we have today for various data stores.
| felixgallo wrote:
| Riak isn't remotely like OLAP. What was your use case?
| ramon156 wrote:
| Think they meant OLTP
| sitkack wrote:
| It is a fault tolerant massively scalable key value store
| capable of handling hundreds of terabytes of data.
|
| What are these options you are thinking of?
|
| The only thing that comes to my mind is Aerospike and possibly
| ScyllaDB.
| senderista wrote:
| If cloud is an option, DynamoDB?
| tptacek wrote:
| FoundationDB seems like the obvious example? But they're not
| strictly comparable in anything but scale, right? FDB is
| ACID.
| EwanToo wrote:
| It's realistically for the handful (dozens at most?) of very
| large Riak implementations where it would be enormously
| expensive to rewrite the application running on top of it.
|
| For example, the UK NHS Spine messaging system which has been
| building on Riak for 10 years
|
| https://riak.com/posts/press/nhs-launches-upgraded-it-backbo...
| nicholas-adams wrote:
| I think that in the time since 2011, things have changed more
| than a bit. As my employer provides Enterprise Grade Riak
| Support (and, of course, OpenRiak support), I'm under NDA and
| cannot really share names. However, I can share that that there
| are quite a few places that use Riak.
|
| Here are a few off the top of my head:
|
| - the biggest online betting company in the world
|
| - one of Japan's largest e-commerce sites
|
| - a large Hungarian bank
|
| - one of China's largest electronic manufacturers
|
| - arguably Asia's largest or second largest messaging platform
|
| - a significant Indian online-documentation provider
|
| - one of the largest US insurance providers
|
| - an Australian app analytics provider
|
| - a European telephone services provider
|
| - one of the world's largest travel sites
|
| - an Asian-based credit-card fraud detection service
|
| - a number of start ups in various industries
|
| - me - I do my crypto taxes using a 5 node Riak cluster running
| on Raspberry Pi's
|
| In the Basho era (up to early 2017), Riak may have only been
| targetted to larger players but now, when it comes to areas
| such as in-house data sovereignty, compliance (e.g. GDPR), the
| flexibility, speed and reliability Riak now provides plus being
| free to run, people from individuals to corporates are starting
| to wake up and see the advantages.
|
| (edited in an attempt to improve list formatting)
| brcmthrowaway wrote:
| Crypto taxes? You do automated crypto trading?
| tibbar wrote:
| I've never met an engineering team that used Riak, but it is used
| heavily as an example technology in Kleppmann's 'Designing Data
| Intensive Applications'. (I would say, informally, it's usually
| the example of the "other way" as opposed to other more well-
| known databases.) This does make me wonder what became of it, why
| it didn't take off.
| encoderer wrote:
| Inscrutable erlang stack traces definitely played a part. They
| were horrible.
| macintux wrote:
| Speaking as a former tech evangelist/engineer at Basho, there
| were a few significant challenges.
|
| Riak is horribly unfriendly as a database: no SQL, it exposes
| eventual consistency directly to the developer, it's relatively
| slow, and Erlang is a fairly unusual language.
|
| While you _can_ run Riak on a single server, you'd have to
| really want to.
|
| Its strength is the ability to scale massively, but not many
| projects need that scale, and by the time you do, you're
| probably already using some friendlier database and you'd
| rather make that one work.
| binary132 wrote:
| I wonder if some of these issues could be addressed sanely in
| an extension to the functionality
| macintux wrote:
| We were working on ways of making it easier (such as CRDTs
| to reduce the amount of work developers had to do to
| leverage eventual consistency), but these were pretty
| challenging problems to solve.
|
| One of our biggest disappointments: we had plans to add a
| way to enforce strong consistency leveraging (IIRC)
| something akin to multi-paxos, but couldn't get it to work.
| binary132 wrote:
| that sounds like a painful session
| jtuple wrote:
| TBH, we shipped fully working strong consistency in 2014.
| It just had a limited feature set, was disabled by
| default, and was never promoted/marketed since it didn't
| fit the direction the new CEO/CTO was pushing.
|
| The engineering exodus around that time sorta killed the
| project though, and we never were able to do the big
| follow-up work to make it really shine.
|
| (Disclaimer: Former Basho Principal Engineer, primary
| author of strong consistency work, lead riak_core dev
| from 2011-2015)
|
| I think another 18 months would have been enough too. But
| it just wasn't the right environment after the hostile
| take-over / leadership transition.
| macintux wrote:
| My apologies for misremembering. I'm glad you chimed in
| to correct the record.
| NickM wrote:
| There were customers happily using strong consistency in
| production, but somehow the idea that it wasn't
| "finished" kept getting repeated over and over by
| management. I was well on my way to solving the biggest
| rough edge (tombstone reaping in SC buckets) but then I
| got pulled off to work on the infamous "data platform"
| and never got to finish that work :-(
| masleeds wrote:
| I'm not sure though for how much longer it will continue
| to make sense for the project as-is to continue to roll
| riak_ensemble forward as part of future releases. As
| there are no contributors who have direct direct
| experience or knowledge of using it in production, so it
| is hard to claim it as being a supported part of the
| product in any real sense.
|
| I apologise if we do eventually cut it. Having worked
| through the code when chasing unstable tests, I developed
| an appreciation for the quality of the work.
| binary132 wrote:
| If you don't mind my asking, how did you find working
| with Erlang in such a context? I've always been curious
| about it, but these days it seems like most people are
| mostly interested in Elixir for REST APIs and I am
| content with my existing tools for that purpose. However,
| there are clearly places where OTP and other Erlang
| functionality would shine and I've always thought it
| might be fun to keep in my back pocket.
| cmrdporcupine wrote:
| Thing is, Cassandra became and remained popular, with similar
| aspects (though in JVM instead of Erlang, so).
|
| Though it had a couple years head start when there really no
| other options for people wanting that kind of kit.
| wbl wrote:
| I feel building a threat intelligence product on Cassandra
| is a bit on the nose. What's next, calling the TCB
| Palladium?
| mst wrote:
| I remember somebody whose competence I rate highly talking
| about ending up picking Cassandra over Riak (at a point
| where Riak was technically decently mature) simply because
| of availability of ops expertise - basically his take
| seemed to be that Basho support was solid but they hadn't
| succeeded at cultivating people he could *hire* to handle
| as much as possible in house before calling in the gurus.
|
| (I can't, of course, speak to the truth of this, only that
| over a couple decades of knowing the dude in question and
| working with him on and off he had sufficient Clue that I
| expect he did put in the effort before coming to that
| conclusion)
| btilly wrote:
| Back in 2011 I was working on a project that involved Riak.
| The difficulty and slowness for doing stuff corresponding to
| basic SQL operations was certainly a giant strike against it,
| and helped sink that project before it was released.
| p_l wrote:
| > corresponding to basic SQL operations
|
| Ohhh, this brings memories of developers hitting the
| wall... Between different SQL databases!
|
| Back in 2016 I was delegated at work to do ops on a project
| that had big data ambitions in Threat Intelligence space.
|
| Part of how they intended to support that was Apache
| Phoenix, an SQL database backed by HBase, running on top of
| Hadoop that also provided object storage (annoyingly
| through WebHDFS gateway).
|
| Constant problems with hung Phoenix queries and instability
| of Hadoop in entirety led me to propose moving over to
| PostgreSQL, which generally went quite well... Except
| several cases of "basic SQL operations" that turned to have
| wildly different performance compared to Phoenix and most
| importantly, to MySQL in MyISAM mode, like doing SELECT (*)
| on huge tables.
|
| Fun times, got to meet a postgres core team member thanks
| to it.
| anotherjesse wrote:
| https://howfuckedismydatabase.com/nosql/ this infamous comic
| is about riak
| rubiquity wrote:
| That could also very well be about CouchDB which
| implemented indexes/views as MapReduce functions.
| senderista wrote:
| Back in the day we had a CouchDB MapReduce view (on
| Cloudant) which took a full _month_ to rebuild (while an
| angry customer was waiting). The I /O inefficiency was
| absolutely off the charts.
| mrweasel wrote:
| I joined a company that had some investment in Basho and
| managed to sell Riak as the data store for a large client. It
| never really worked out, not enough of the SREs had be
| properly trained on Riak, the developers hated it because
| getting help and support was somewhat difficult, especially
| in an emergency.
|
| In the end more and more data was offloaded to MariaDB, until
| one day the last remaining data couldn't justify the cost of
| the Riak cluster. I think we swapped out an eight node Riak
| cluster for two largish MariaDB database (one being a hot-
| standby).
|
| For one of the other clients it was the exact same scenario,
| only we had been contracted in to help run the Riak cluster,
| which we didn't do well. Once they had migrate of it, to
| Oracle I think, the client left.
|
| To me it always felt like it was just the wrong tool for that
| particular job. Someone really wanted to be able to jump on
| the NoSQL hype and sell something. They picked Riak, because
| it honestly looked really good, and probably was, compared to
| MongoDB, CouchDB or whatever else happened to float around at
| the time. It just wasn't the right tool for the problems it
| was applied to.
| red_hare wrote:
| My old team used Riak in production for time series data in a
| real-time system.
|
| Our code was in Clojure, and we just wrapped the Java client.
| The conflict resolution was a steep learning curve, but
| overall, it was kind of nice (coming from Mongo).
|
| But man, Clojure stack traces wrapping Java stack traces
| wrapping Erlang stack traces in a Kafka consumer... I wish that
| hell on no one.
| mst wrote:
| This feels like it needs a "yo dawg, I heard you liked stack
| traces" meme.
|
| Also bourbon. Probably *lots* of bourbon.
| veyh wrote:
| We used Riak at $dayjob at around 2014-2017 (iirc). I don't
| exactly remember it fondly. It was slow and unreliable. You
| could make it freeze/crash with the wrong SOLR query. (I was
| pretty good at that...)
| masleeds wrote:
| The SOLR part has now been retired from the last few
| releases.
|
| Current development has been focused on improving the
| flexibility of secondary indexes. There was some funky stuff
| achieved by some users using overloaded 2i terms and
| distributed processing of regular expressions against those
| terms - the aim is now to make this more flexible to the
| modern developer using the language of projected attributes
| and filter expressions (ala DynamoDB). There's also some
| active work to both replicate-to and full-sync (i.e.
| reconcile with) external OpenSearch clusters.
|
| The primary goal for OpenRiak is stability under load/failure
| as a K/V store - so the ultra-flexibility of in-built SOLR
| querying has been sacrificed in the move towards that aim.
| Anything that can do harm is to be offloaded or constrained.
| 0xbadcafebee wrote:
| I worked on a team that built a massive, high-performance
| internal service based on Riak. There are many things I learned
| from that system. Here is the best takeaway I can offer:
|
| It does not matter what your technology is, or how
| theoretically superior it is. Getting it to actually work well
| "in production" is a whole separate _thing_ than simply
| designing it and writing code. When it 's a very small system,
| it will look like it's doing great. As it gets bigger, the
| seams will start to burst, and you will find out that promises
| and theory don't always match reality.
|
| In the end, while its aims are great, it takes a whoooooole lot
| of work to smooth out the bumps in such a system. You need
| experts in that technology to address bugs in a timely manner.
| You need developers versed in the system to properly build apps
| utilizing it. You need competent operators to build,
| orchestrate, operate and maintain the whole thing.
|
| All of that is made easier by using simple technology that
| everybody knows, that there's a huge support community for,
| professional services for, etc. A technology like MySQL or
| Postgres etc, has the corporate, development, support, etc to
| make it easy to work with at any scale. A little janky at
| times, limited, but dependable, predictable, controllable.
|
| A small bespoke system with a small support community and
| virtually no corporate support is, comparatively, a hell of a
| lot more difficult/costly to support and harder to make work
| reliably.
| bojo wrote:
| I'm pretty sure Stripe was a heavy user of this for a while.
| They used it due to their write-heavy system, if I recall.
|
| I fondly remember writing a Go driver for it. Was a good
| experience: https://github.com/riaken/riaken-core
| m00x wrote:
| Companies would rather use something like dynamodb than self-
| host riak. You get an army of Amazon code monkeys to help you
| if something goes wrong, and it's a click away.
| rmetzler wrote:
| > I've never met an engineering team that used Riak
|
| I was part of a recent cloud migration. Part of on-prem (though
| unfortunately not migrated by my team) were this very first
| Riak Cluster I saw in production.
|
| The engineering team used it as "kind of S3" for images, with 3
| to 5 PHP scripts providing an interface to Riak and imageMagic.
| It seemed to me like a good abstraction and I think the
| migration to S3 was mostly painless.
|
| Other than that I only had contact with Riak at university
| around 15 years ago, when we tested cluster setups of several
| NoSQL databases and tried to manually introduce faults to see
| if they could heal. Riak passed our test at that time, MongoDB
| didn't.
| isoos wrote:
| Basho team was very kind to open source contributions in
| ~2011-12: I've written an open source Riak client in Dart, and
| they had sent me t-shirts (the quality ones that are rare today).
| Nice treats for a fun project :)
| binary132 wrote:
| It's actually kinda silly how exciting this is to me
| masleeds wrote:
| Riak has been maintained through the post-basho years by
| engineers at some of its larger customers (disclaimer - including
| myself).
|
| The focus has been on trying to improve the stability of the
| database when subject to complex failure scenarios under
| stressful load, with minimal need for urgent operator
| intervention. The focus has been on keeping those existing
| operators happy rather than seeking out new users. Evolution of
| the product since basho has been slow but significant.
|
| The project now has support from Erlang Ecosystem Foundation, and
| we're looking to invest some effort over the next few months
| explaining what we've done, and to start to articulate what we
| see as the future for Riak. So if you're interested watch this
| space.
|
| It is expected to remain a niche product though. However, it may
| still find a home for those demanding specific non-functional
| requirements, with an acceptance of some functional constraints.
| jamesblonde wrote:
| Metastability is an under-rated system property for databases
| and systems software, in general.
| remram wrote:
| What does metastability mean in this context? I've only seen
| it used to mean "appears stable but not actually stable", eg
| systems that resist small perturbations but never return to
| nominal after bigger disturbance (like cold boot).
|
| Did you mean "stability"?
| 7qW24A wrote:
| Marc Brooker's blog on the topic is good:
| https://brooker.co.za/blog/2021/05/24/metastable.html
| a2800276 wrote:
| I'm not aware of any definition of metastability that would
| be a desirable property of a system. Vaguely speaking, in my
| understanding metastability in physics, chemistry and
| electronics refers to a state that has not yet fully settled
| and the final state is therefore undeterminable / random.
|
| Could you elaborate your understanding of the term? I can see
| it related to concepts like eventual consistency, just
| unclear of how it would be considered a positive
| characteristic. I'd have thought that a deterministic outcome
| would be important in a database system?
| mst wrote:
| Did you mean 'not remaining in metastable failure' ?
|
| I was under the impression that once you learned how to pet
| the cat forwards (the non-triviality of getting to that point
| being an oft cited adoption barrier, but still), Riak was
| really quite excellent at recovering from things going wrong,
| which would fit with that being what you were trying to say.
|
| Or maybe my guess has gone completely wide, in which case
| please do break out the small words and crayon drawings and
| explain what you actually meant :)
| pton_xd wrote:
| > we're looking to invest some effort over the next few months
| explaining what we've done, and to start to articulate what we
| see as the future for Riak. So if you're interested watch this
| space.
|
| Where's that going to be posted? I'm not a Riak user but I am
| interested in hearing what others are doing in regards to
| improving failure scenarios in distributed systems.
| masleeds wrote:
| When we get stuff together there will be a link from our
| discussions page -
| https://github.com/orgs/OpenRiak/discussions.
| freerobby wrote:
| I led a migration from Mongo to Riak at Shareaholic about 12
| years ago: https://www.slideshare.net/slideshow/migrating-to-
| riak-at-sh...
|
| It was successful at first, but ultimately we traded one set of
| problems for another (how novel, I know).
|
| In particular, I underestimated the pain of troubleshooting the
| database itself. Riak was a new product, we were a small team
| that had never run anything on BEAM, and ultimately we lost too
| many days debugging and trying to make sense of Erlang
| stacktraces.
|
| The Basho folks were great, and to this day I appreciate how
| quickly they fixed a number of bugs for us. But ultimately it
| wasn't enough -- we found problems faster than they could be
| patched.
| tptacek wrote:
| As someone who has used Riak in anger once in his career and who
| has a blossoming interest in FoundationDB I'd love someone to
| contrast the two systems. My knee-jerk reaction --- which I'm
| calling out as such! --- is that FDB has decreased the relevance
| of systems like Riak.
| masleeds wrote:
| I would tend to agree, perhaps a decade ago it was easier to
| define the uniqueness of Riak, and now there are alternatives
| that offer similar guarantees. So the relevance of Riak is not
| as obvious.
|
| Also as we focus on stability on OpenRiak going forward, that
| means reducing some of the capability that may have made Riak
| stand-out in the scale-out space. The preference going forward
| is to do fewer things, but do those things predictably well.
|
| There will be differences between Riak and FoundationDB, and I
| hope those differences are sufficient to make Riak interesting,
| and allow it to continue to occupy a small niche in the world
| of databases.
| nemothekid wrote:
| I spent a lot of time with NoSQL systems in the early 2010s and
| I think what "killed" them was processors and networks getting
| so much faster that Postgres "just worked" for more and more
| use cases.
|
| There are systems I've built in the past with 20+ Cassandra
| nodes and tens of thousands of ops that were originally built
| on MySQL/Postgresql but migrated to Cassandra because the
| performance/cost of the SQL systems was just to high.
|
| Now those performance requirements can be handled cheaply with
| 1 or 2 beefy PostgreSQL databases. The level of scale you need
| today to make put up with something like Cassandra is much
| higher while yesteryear it felt like every startup was falling
| over once they found pmf
| amerine wrote:
| I still use my RICON pint glass and wear my RICON jacket Basho
| gave everyone at RICON almost every week. My favorite conf swag
| ever
| remram wrote:
| What's a Riak and is there really no better link for this news?
| sriram_malhar wrote:
| https://en.wikipedia.org/wiki/Riak
| jtuple wrote:
| This really hits home and makes me happy to see on the HN front
| page.
|
| Nearly 10 years later and I still consider my time working on
| Riak at Basho the highlight of my career.
|
| After leaving, my original plan was to found "Basho 2.0" after my
| non-compete expired. But, unexpected personal/family hardships in
| 2015-2018 made big-tech money the better choice for awhile, and
| Cloud/competitors continued to chip away at the market.
|
| Often stil regret not taking that path.
|
| But, happy to see technology I'm very fond of still living on and
| providing value to the world.
| _russelldb wrote:
| I'm super happy to see YOU jtuple!!! I haven't seen or heard
| anything from you in years. I'm sorry to hear about
| personal/family hardships you suffered. I loved working for
| Basho in that era. I hope things are going well for you these
| days. No need to reply and get all mushy if you don't want to,
| I'm just really happy to see you posting and hear that you're
| out there.
| p_l wrote:
| A question possibly answered elsewhere, but did openriak include
| only Risk KV, or also other projects like CS & TS?
| masleeds wrote:
| There are forks being maintained of Riak CS and Riak TS by TI
| Tokyo - https://github.com/ti-tokyo.
|
| The focus of the OpenRiak community for the moment is on Riak
| KV only.
| p_l wrote:
| Thank you.
|
| I am tempted to try playing with the entire stack, used to
| pine for it back in 2011 or so :)
| datadeft wrote:
| For a long time Riak was my favorite key value store. It went
| into production without any significant issue in several
| companies and was running without interruption despite the face
| of hardware issues.
|
| I think it stability was due to the fact of combining great
| technologies like LevelDB and Erlang. I wish it was a bit more
| popular.
| indulona wrote:
| it's always great idea to build your company on software written
| in some obscure niche language...
| KameltoeLLM wrote:
| "mOdERn" (stupidest buzzword ever; for me: modern == stupid NIH
| rewrite)
___________________________________________________________________
(page generated 2024-11-20 23:01 UTC)