[HN Gopher] The LMAX Architecture (2011)
___________________________________________________________________
The LMAX Architecture (2011)
Author : mooreds
Score : 51 points
Date : 2024-08-16 15:37 UTC (7 hours ago)
(HTM) web link (martinfowler.com)
(TXT) w3m dump (martinfowler.com)
| chc4 wrote:
| See also: the discussion on the LMAX disruptor github
| documentation https://news.ycombinator.com/item?id=38313457
| senderista wrote:
| So much hype for a lock-free MPMC ring buffer. If you're familiar
| with the concurrency literature you'll see that nothing here was
| novel even at the time.
| xxs wrote:
| java had a proper memory model and the busy wait was a novelty.
| iirc, it was used mostly in a single thread processing (i.e.
| consumer) for the main business logic like matching forex
| order.
|
| Realistically aside the disruptor LMAX had an interesting way
| for fail over by replaying all the input to reach the state.
| bostik wrote:
| It is quite fitting that another HN darling, Jane Street, uses
| similar architecture in their exchange(s).
|
| Event input queue, fully deterministic matching engine, and an
| output queue. Plus importantly, a sequencer to guarantee that the
| events going into the input queue have a known order.
|
| Aspects of their architecture and design decisions have been made
| public through their podcast series, Signals and Threads.[0]
|
| 0: https://signalsandthreads.com/
| jnordwick wrote:
| This isn't even close to unique. I['ve worked at a few HFT/MM
| places that all had a similar architecture in place before LMAX
| wrote this. This, multicast rings, and other patters are copied
| a lot in the industry because they work well for the problem
| space.
| bostik wrote:
| I'm not surprised. And I think you nailed the reason too:
|
| > _copied a lot in the industry because they work well for
| the problem space_
|
| Finance as an industry could be described as ... incestuous.
| The people move around within the small circle of firms, so
| any architectural knowledge gets cross-pollinated quite
| rapidly. From what I understand, the disruptor pattern itself
| was reasonably well known in the academic circles well before
| LMAX wrote about it - but I'm not sure anyone had put
| publicly available numbers on a heavily optimised version
| before them. (FWIW, I use a simplified disruptor in one of my
| own projects because it works well and simplifies the overall
| design. )
|
| We should also be honest about the technical challenges:
| there are _very_ few as interesting problem spaces as an
| exchange or a low-latency MM.
| richliss wrote:
| So I worked at LMAX during 2009-2011 when Martin Thompson,
| Dave Farley, Mike Barker, Chris Smith and Danny Yates
| worked on the Disruptor. It was in use in 2010 inside LMAX.
|
| The heritage of Martin and Dave was video games rather than
| finance and they had the most input into it.
|
| From a friend who worked in games, the general idea behind
| the disruptor had been in use in games for a while.
| hangonhn wrote:
| Can you explain what a multicast ring is? The article itself
| just grossed over it.
|
| Thanks!
| pkaye wrote:
| Is the LMAX disruptor just a ring buffer? Anything else special
| about it?
| xxs wrote:
| It is a ring buffer indeed, with publish serialization (on
| producer side) and normally a busy wait. It sort of doesn't
| work on some VMs due to scheduling. (I mean the busy wait w/o a
| backoff)
| jnordwick wrote:
| The big win is how it allows bulk processing while taking
| serialization dependencies into account.
|
| So you algos reading market data messages and yank all that
| is there and apply them in bunk instead of trying to do each
| one individually.
|
| And your logger that is storing all the messages to a file
| and/or db knows what messages the algo has processed and can
| grab all the messages up to that point so it is always
| writing as much as it can and current.
| hangonhn wrote:
| Is this ability that is the result of each component being
| able to read the index/sequence number of all the other
| components?
| qwe2645692 wrote:
| Sounds so much like an io_uring model.
| GenericCanadian wrote:
| I wrote this in Crystal lang to learn more about it and the
| language: https://github.com/nolantait/disruptor.cr
|
| There is a Ruby version that is also very readable:
| https://github.com/ileitch/disruptor
| joezydeco wrote:
| Thanks for that. I was always curious why the finance bro down
| the block had an LMAX license plate on his car.
| BrokrnAlgorithm wrote:
| Is there any didactic implementation of the Disruptor / multicast
| ring available somewhere? I've been curious in working through
| some practical example to understand the algorithm better.
| whoevercares wrote:
| Fun fact, log4j use disrupter
___________________________________________________________________
(page generated 2024-08-16 23:00 UTC)