[HN Gopher] Nobody cares about our concurrency control research ...
___________________________________________________________________
Nobody cares about our concurrency control research [pdf]
Author : luu
Score : 81 points
Date : 2023-05-23 10:45 UTC (3 days ago)
(HTM) web link (www.cs.cmu.edu)
(TXT) w3m dump (www.cs.cmu.edu)
| greghn wrote:
| Talk recording:
| https://m.youtube.com/watch?t=54m&v=M2MEcvMHzkY&feature=yout...
| ripley12 wrote:
| > Assumption #1: All transactions execute as stored procedures.
|
| This kind of surprised me. Is that really a widespread
| assumption?
|
| As an application developer I'm used to starting+committing
| transactions myself, outside the context of a stored procedure;
| I've done this with SQL Server and SQLite.
| liftm wrote:
| Should have a (2017) in the title?
|
| (I was doing student assistant jobs for a database research group
| in 2015. This was old news even in 2017 (though I suppose nobody
| put it quite so obviously). I'd be curious if anything changed
| since then.)
| dekhn wrote:
| This presentation is (IMHO) worthless. There's no real data or
| information.
|
| BTW, Spanner runs on paxos.
| comex wrote:
| Content aside, I just wanted to take a moment to admire the
| visual design of the presentation. The icons on the top left of
| each slide; the two-line slide titles and the page numbers; the
| black, white, and red of the "some papers are lit up" slides; the
| circles containing rectangles containing portraits; the
| 'emphasis' slides containing just a short sentence in white on a
| black background (which is both higher contrast and inverted
| compared to the dark-gray-on-light-gray used for normal
| slides)... it's all very fancy without being too obtrusive.
| Especially for a presentation that's just an individual
| presenting at an academic conference, as opposed to some
| company's glitzy keynote trying to sell something to a mass
| audience. I'm impressed.
| mjb wrote:
| This is a good troll, and makes some good points, as Andy Pavlo
| always does.
|
| I'd diagnose the core problem differently, however. What
| application and system developers care about in modern
| architectures is application level correctness from the
| perspective of the client (customer, tester, etc). What a lot of
| the research is interested in is isolation within the context of
| a single database. That's a useful building block, but doesn't
| compose into end-to-end correctness in ways that most application
| architects and developers can reason about as soon as the
| application spans multiple stores (because SoA, or because
| microservices, or because it interacts with a SaaS, or whatever).
|
| So application builders are left with the problem of not being
| able to reason about how to build correct applications, and not
| having the tools to build those applications correctly, and so
| end up settling for some level of weird behavior under
| concurrency. Turning up the database isolation level may hurt
| performance, but does little to rein in the overall
| incorrectness, so there is little motivation to do it.
|
| If the database research community wants application developers
| and operators to care about isolation more, I suspect that we
| need to care more about atomicity, isolation, and consistency "in
| the large". Think more broadly, across whole microservice or SoA
| architectures, and provide ways to give customers ACID properties
| end-to-end, and ways to compose databases, services, and systems
| that preserve these properties.
|
| That doesn't mean that per-database or per-component isolation
| isn't important. It's practically important, and there's good
| research on the effect it has[1]. The problem is that its not
| sufficient, and so folks feel less motivated to pay the
| (perceived) performance costs.
|
| For folks who'd like to understand isolation levels better, I'd
| recommend starting with "Seeing is Believing"
| (https://www.cs.cornell.edu/lorenzo/papers/Crooks17Seeing.pdf)
| and "A critique of ANSI SQL isolation levels"
| (https://www.microsoft.com/en-us/research/wp-content/uploads/...)
| both of which do a good job explaining the differences between
| levels in a way that isn't tied to implementation.
|
| [1] One good paper there is "ACIDRain: Concurrency-Related
| Attacks on Database-Backed Web Applications"
| (http://www.bailis.org/papers/acidrain-sigmod2017.pdf).
| crabbone wrote:
| > What application and system developers care about in modern
| architectures is application level correctness
|
| Meme template: You guys care about correctness?
|
| Also, please don't lump together application and system
| developers. In terms of software quality these are worlds
| apart. It's a lot more common in system world to care about
| correctness than it is in the application world. Of course,
| there are always exceptions, but a _typical_ example of either
| camp goes about quality control very differently.
|
| In my experience, outside of a very small sector of application
| development, where correctness helps sales (eg. medicine, or
| other fields where software mistakes come with serious
| repercussions) the goal is to make the "good path" work in
| favorable conditions. It is also permissible to have "excusing
| circumstances" (a.k.a. documented defects) that render software
| inoperable.
|
| To contrast this, system developers _typically_ have to deal
| with systems that are expected to work hell or high water.
| Well, again, outside of cases when failures are forgivable,
| like, management software / UI to the system rather than
| system itself. Same reason how you'd expect library code to be
| better tested and more robust in the face of edge cases than
| any individual application using the library.
|
| ---
|
| So, as you mentioned "as soon as the application spans multiple
| stores", application developers will give up all hope because
| there are too many layers, too many components, too much opaque
| third-party code. Whereas in systems, this is typically still
| all under your control. I.e. think about products like Ceph or
| Etcd -- they are designed to span multiple stores and still be
| correct.
|
| I had to go back and forth between application development and
| systems. And, for me, the world of application development is
| very unappealing specifically because of the attitude to
| quality. From my key-hole perspective, I wish people made fewer
| applications and concentrated on making systems more robust,
| because I feel like this would make computers more reliable,
| faster and cheaper... but there's a strong financial incentive
| to make applications, even though they are less reliable,
| slower and more expensive in the long term, because initial
| development is faster and cheaper.
| pdhborges wrote:
| There is a good article [1] discussing this state of affairs
| where application developers are straight jacked because they
| end up integrating too many systems and there is no distributed
| transaction covering them all.
|
| [1] https://queue.acm.org/detail.cfm?id=3025012
| mathgladiator wrote:
| This resonates with a dry-run of my architecture (
| https://www.youtube.com/watch?v=bWYgChA_aYA ). A central thesis
| is that that "sure, you can have all these infinitely scalable
| services" but then the composition of all of them is very hard
| to reason about.
|
| The interesting thing as I play with my platform as I build a
| product right now under the thesis is that I'm focused almost
| 100% on the domain modelling problem for the business rather
| than thinking about architecture, databases, etc. I'm just
| writing code, and I have a reliable product from the get-go.
|
| Honestly, I think we have been solving the wrong problems
| collectively for many years. Granted, I'm years away from a
| platform for the masses, but it is a super power that my
| partners and I have to go storm various industries to beat the
| average.
| cmrdporcupine wrote:
| _" Maybe READ COMMITTED is good enough or maybe people don't know
| how dirty their data actually is..."_
|
| I'm going to go with the _latter_ option. And further posit that
| on the whole most people working as "full stack" devs in our
| industry don't really _get_ databases. In particular they don 't
| _get_ the relational model and its advantages at a fundamental
| level and are perfectly willing to paper over it with ORMs,
| statically structured "model" objects, and break their
| (logically connected) data up into disjoint microservices that
| end up doing inefficient server-side software joins by web
| service call and...
|
| Putting it another way, they want the DB to play second fiddle to
| frameworks, architectures, and techniques that operate with
| different approaches (usually OO & SOA) than the
| logical/relational modelling approach that an RDBMS is best for.
| And chaos and misuse is the result. That's my usual rant
| anyways...
|
| And, so yeah, it's also not a surprise that people aren't really
| "getting" transactions and concurrency. Certainly the wave of
| "NoSQL" stores (thankfully waning) showed this. People were
| willing to toss coherence in favour of scalability (often even
| when they didn't need it or hadn't proved that they did.) But
| often the "NoSQL" thing was simply a product of a mentality that
| went something like "I just want a data store" and "my _object_
| doesn 't fit in a _table_ ", not getting what a table really
| _is_... And so we got Redis and Mongo, systems that didn 't offer
| any performance advantages really, but looked on the surface like
| they'd remove complexity ... but ...
|
| And then I'd say that this general lack of enthusiasm around the
| relational model and ACID means that people haven't _demanded_
| the right things from their database tooling.
|
| Ok this rant is going to go on too long ... I just had my morning
| coffee...
| jayd16 wrote:
| You make some ok points but it's not hard to see why Redis and
| Mongo took off. During their hype cycle, they could
| cluster/shard more easily than RDBs. You could just throw
| hardware at a problem.
| cmrdporcupine wrote:
| I think it's an unfortunate history that the boom in popular
| "high scalability" type applications happened before SSDs
| became commonplace. Feeling I/O bound and having to _" go
| horizontal"_ to increase IOPS was something everybody felt
| they had to do.
|
| I remember reaching for Cassandra to solve a problem for a
| startup that needed to do what seemed like massive concurrent
| writes; the workload simply saturated what a single Postgres
| instance could be expected to do.
|
| But since then, SSD & nVME happened, and, well it's mind
| boggling.
|
| Just rough numbers.
|
| 7200 RPM HD IOPS: ~100 nVME IOPS: ~500,000
|
| Now, one doesn't always have a properly designed storage
| engine & DB that can take advantage of this huge
| differential. But what a yawning gap. Solid state storage
| changed everything.
|
| Honestly, horizontal scaling is often reached for too
| quickly. For redundancy, I get it (but that's more a
| replication story.) For performance reasons? For most regular
| OLTP workloads for most average sized website type
| applications, I am not convinced these days.
| turtleyacht wrote:
| Would this be a possible alternative?
|
| Careful, deliberate consideration of a handrolled data model,
| with plenty of prototyping up-front with Excel and the
| customer, playing out various scenarios.
| marcosdumay wrote:
| Well, it really doesn't help that from the top-5 engines on his
| slides, only Postgres doesn't have a completely horrible
| performance profile for serializable transactions. So, people
| that try to learn about it run away quite quickly.
|
| But I'm not sure if most systems need it anyway. Personally, I
| remember very few times where I had to change from the default
| transaction model.
|
| As a rule, when people need full serializability, they create
| it with some other, more flexible mechanism.
| cmrdporcupine wrote:
| Perhaps fair, but I think though this ties back to one point
| I was making:
|
| _" And then I'd say that this general lack of enthusiasm
| around the relational model and ACID means that people
| haven't demanded the right things from their database
| tooling."_
|
| In general there are plenty of solutions in database research
| and implementation. But because the broad community of
| developers doesn't put data first, and have a low expectation
| and knowledge of what a DBMS can and should do...
|
| It took us years to get to the place where Postgres is common
| place instead of e.g. MySQL; and back in the day "serious"
| folks spent tens of thousands on Oracle licenses... I guess
| that's progress, but there's so much more that can be done in
| this sphere.
| wiz21c wrote:
| > its advantages at a fundamental level
|
| I used to be a dev and I wonder what you have in mind here ???
|
| For me the two important points with a DB are:
|
| - entities connections (foreign keys), data types and a few
| other things are specified declaratively (so you don't have to
| think about enforcing them, the DB does it for you, so you
| never, ever forget). So I liked to have very strong DB models
| (no denormalization :-)) as the ultimate line of defense of my
| data quality. In my business, that was worth millions of euros
| (and politics).
|
| - transactions: Read commited was good enough in most cases
| (clients didn't access the same data) adn sometimes
| serializable was necessary (for example, toi generate ID's
| unique acorss the database and across all the clients)
| RegW wrote:
| I also agree with the later option.
|
| I've been using databases for a very long time. For the past 2
| decades there has been a drive by framework providers to
| transparently handle such things and relieve developers of any
| difficult decisions. Often there isn't an option or it is
| deeply hidden. I've arrived on jobs to find Spring applications
| where database transactions have been unwittingly disabled (or
| not enabled) and no one has realised.
|
| There is also a lack of upfront thinking about types of data
| and the required consistency. Some items really should be
| consistent and some don't need to be.
|
| In one of Martin Fowler's earlier books he describes the
| difference between knowledge and operational data. Knowledge
| data is almost like configuration - perhaps the description of
| a currency and number of decimal places it has. Operational
| data might be an invoice generated using that currency. The
| description of a currency changes rarely (unlike a currency
| rate), it could be cached a long time for use by many processes
| without issue. However, the invoice must be consistent, but
| only one process will be involved in creating it and nothing
| else is held up waiting for it, and once it is created - it
| will never change.
|
| In practice most data items lie somewhere along this spectrum.
| However, you choose your tool and then everthing is either ACID
| or NoSQL.
| RugnirViking wrote:
| im so glad I got taught database normal forms in high school
| because I sure as anything didnt learn it at university.
|
| That, in the long and short, is probably approximately where
| the problem lies.
| yyyk wrote:
| "Maybe READ COMMITTED is good enough or maybe people don't know
| how dirty their data actually is..."
|
| Look, a deadlock or slowness may technically be correct, but it
| can mean a lost sale. In practice, a worse problem than data
| inconsistency. SERIALIZABLE costs too much. I usually use
| snapshot, with a does of repeatable read where absolutely
| necessary.
| komain7 wrote:
| Don't care...
| gowld wrote:
| On slide 22, what does his fake name say in Greek? Catastropheaz
| Dedomeviv?
| apavlo wrote:
| I have no idea what it's supposed to say. I think I just put
| "Andy Pavlopolis" into Google Translate. It's meant to be a
| joke. Before I started at CMU, all of the database professors
| in the city of Pittsburgh (Carnegie Mellon + University of
| Pittsburgh) were Greek. So the joke is that the only reason
| that I got hired at CMU was because I pretended to be Greek
| with a fake Greek name.
|
| It's in the video here:
|
| https://www.youtube.com/watch?v=M2MEcvMHzkY&t=4507s
| jaspax wrote:
| Katastropheas dedomenon
|
| Means, I think, "given over to overthrows".
| greenpeas wrote:
| google translate gives a different translation: data shredder
| or data destructor
|
| https://translate.google.com/?sl=el&tl=en&text=%CE%BA%CE%B1%.
| ..
| hgsgm wrote:
| Dedomenon meant given in the Biblical past, and data now.
| ("Take is as given", in a logical/mathematical sense).
|
| For a database integrity scientist, "data destroyer" makes
| sense.
| jaspax wrote:
| Ha! Well, that's what I get for having studied Classical
| Greek in school but not Modern Greek.
___________________________________________________________________
(page generated 2023-05-26 23:02 UTC)