[HN Gopher] Fintech engineering mistakes (2022)
___________________________________________________________________
Fintech engineering mistakes (2022)
Author : 0x54MUR41
Score : 117 points
Date : 2023-06-28 14:12 UTC (8 hours ago)
(HTM) web link (startupwin.kelsus.com)
(TXT) w3m dump (startupwin.kelsus.com)
| gsatic wrote:
| And that's why I shall be sticking to cash for the next 10 years
| till all the beta testing is complete.
|
| By which time hopefully interest bearing CBDCs will show up, and
| make all these mindless intermediaries sitting between my wallet
| and someone elses wallet obsolete.
| fvdessen wrote:
| You can absolutely use doubles for money. Excel does it and so do
| many other financial tools. There are pros and cons but as long
| as you do rounding and comparisons correctly it works perfectly
| fine.
| indymike wrote:
| > There are pros and cons but as long as you do rounding and
| comparisons correctly it works perfectly fine.
|
| This is exactly the issue with using floats where an arbitrary
| precision decimal with proper rounding is really needed. Easily
| solved with a good library and if your languages supports it,
| type, but it's really easy for a dev in a hurry to not use the
| library and roll some a=b+b*c_rate code that forces some type
| conversion. The rounding rules often are tied to contracts, and
| a subtle bug that's off a few mills here (total problem created
| $2.33) and there can lead to audits (total cost of audit
| $14,800) that cost a lot.
| phoehne wrote:
| I'm tired of having to drag in another dependency and lose
| operators if I'm doing money math. I can create an experience
| in C++ that's almost rational, with operator overloading, but
| most other languages were designed well after we knew that
| doubles are not sufficient. And there's more than just
| arbitrary precision. For example,, some currencies use three
| decimal or no decimal places. Two just happens to be
| convenient for the Euro and Dollar. In addition, sometimes
| you carry prices to 3 or 4 places. But you still want
| banker's rounding. And I shouldn't be able to add Turkish
| Lira to US dollars, any more than the language allows adding
| floats and integers, without conversion. Then there's locale
| correct display for currencies (e.g. $ vs USD and before or
| after the money amount).
| fvdessen wrote:
| you can also easily hurry up subtly wrong money math with
| ints. (overflows, wrong rounding method, etc). So you should
| not in any case use the default math operators
| lightbendover wrote:
| At least with ints you can capture those issues with intent
| and handle them properly (as the better libraries that
| handle money tend to do).
| KRAKRISMOTT wrote:
| It's sufficient for quantitative finance and trading but not
| for accounting. For financial engineering and modeling it
| doesn't matter if results are a few fractions of a cent off as
| long as your profit margins/model tolerances are greater than
| the error, because your broker/bank/exchange will keep track of
| the exact values in your account. But if you are building a
| bank/broker/exchange, then tracking it precisely enough for
| GAAP is now your problem.
| recursive wrote:
| What do you mean by "correctly"? It sounds like victim blaming.
| For money, I don't think it's perfectly fine to use numbers
| where this isn't true. 0.1 + 0.2 <= 0.3 //
| sure looks true to me
| gchadwick wrote:
| The question is why? What does the double give you over a
| 64-bit integer? Sure when you divide and it leaves a fractional
| part you loose it and need to think about what happens there
| explicitly but you need to do the same for doubles to avoid
| pennies going missing and snowballing into larger errors.
| cjs_ac wrote:
| Using integer representations for currencies becomes very
| messy when dealing with more than one currency at a time.
|
| The United States dollar is famously subdivided into cents,
| but is also subdivided into 'mills' (one thousand to the
| dollar).[0]
|
| The Mauritanian ouguiya is divided into five khoums.
|
| The Madagascan ariary is divided into five iraimbilanja.
|
| The Maltese scudo is divided into twelve tari, which are
| divided into twenty grani, which are divided into six
| piccioli.
|
| Historically, such currencies were ubiquitous. For example,
| prior to 15 February 1971, the pound sterling was divided
| into twenty shillings, each of which was divided into twelve
| pence, a system that originated with Roman currency and was
| used throughout the British Empire.
|
| Exchange rates are typically quoted in terms of the largest
| unit, whereas integer representations of currency would need
| to be done in terms of the smallest unit, so extensive
| information about currency structures would need to be used
| to correctly represent exchange rates. Floating point or
| binary-coded decimal representations are consequently much
| better.
|
| [0]: https://en.wikipedia.org/wiki/Mill_(currency)
| fvdessen wrote:
| There are multiple factors:
|
| Doubles can exactly represent all 16 digit ints (IIRC) which
| is good enough for most use cases, you can catch the out of
| range cases (as you should do with ints as well)
|
| If you use long ints you must track the decimal precision
| along the value which is not always trivial if you use mixed
| currencies.
|
| Long ints are not guaranteed to correctly roundtrip through
| json serialisation / deserialisation
|
| Doubles are easier to handle in the frontend
|
| Currency math is different enough from regular math that you
| need special operator functions anyway so it's not like ints
| are easier to handle either
| jdiff wrote:
| Using ints forces you to deal with all that. Using floats
| lets you easily ignore it and sweep it under the rug.
| karmakurtisaani wrote:
| If Excel uses doubles for money, it should be a warning sign.
| It uses simple numeric data type (I guess just ints) for
| freaking dates.. I can trace at least a couple of bugs in my
| career to just that fact.
| mrbungie wrote:
| Apart from its quirks Excel is fine, if you know its
| limitations. I think the real warning sign would be an
| analyst/programmer working with Excel and expecting high
| precision results.
| recursive wrote:
| I find Excel to be fine as long as I never use it.
| Mordisquitos wrote:
| _Everything_ is fine apart from its quirks, if you know its
| limitations. The problem is when a tool 's quirks and
| limitations are neither exhaustively documented nor can
| they be inferred from having reasonable knowledge about the
| base principles of the tool, but are rather learnt by
| experience (to be read as "through bad experiences").
| esotericimpl wrote:
| [dead]
| jaclaz wrote:
| If you want a simple real life example (unconnected to the way
| numbers are stored) here it is (accounting may have quirks that
| arrive unexpected).
|
| In EU prices to customers are required to be comprehensive of
| VAT, so a price is EUR 60,00 included VAT 10%.
|
| But in an invoice/receipt you have to explicit how much is the
| net and how much is the tax, so 60 / 1.10 = 54.55 and VAT is
| 54.55 x 0.10 = 5.46 which makes a nice 60.01.
|
| You may be tempted to round down the 60 / 1.10 = 54.54 and have
| VAT 54.54 x 0.10 = 5.45 but this makes 59.99.
| phoehne wrote:
| For example 16.10 is not exactly 16.10 in IEEE floating point.
| When you do enough operations, and depending on the order of
| operations, you can wind up several cents off. That sounds
| small, but can be enough to give your auditors heart-burn.
| COBOL does BCD arithmetic, (not really but at least
| conceptually), and it's penny accurate to 31 digits (as per the
| standard - implementations may have greater accuracy). Frankly,
| it's stupid that 63 years after COBOL we're still treating
| money and currency as an afterthought in languages that are
| supposed to be business oriented. Proper currency handling
| should be part of the language.
| tiedieconderoga wrote:
| It would be neat to have a fundamental "money" primitive
| alongside int, float, string, etc.
|
| It might not be simple to implement, though. Off the top of
| my head,
|
| * Can you confidently say that $100>Y=10 in an offline
| environment?
|
| * You'd need to support different bases for splitting units
| to deal with eldritch values such as farthings.
|
| * Where is the line drawn? Is a Bitcoin a valid currency? A
| gold Krugerrand? A bundle of stock shares? A bushel of
| apples?
|
| All of that said, I would love to see a world where I could
| write code like: money balance =
| money.lumber.grade.kilograms(8);
| kdowns wrote:
| Currency conversions are a transaction not an operation,
| the conversion rate fluctuates constantly and typically
| involves fees and involve tax liabilities. For a money type
| I'd go so far as to want it to either disallow or throw an
| exception if attempting an operation on two monies in
| different currencies.
| yakshaving_jgt wrote:
| Here's the implementation we use at work. You might find
| some interesting ideas there. It's nicely documented.
|
| https://hackage.haskell.org/package/safe-
| money-0.9.1/docs/Mo...
| phoehne wrote:
| I don't think it's as big a heft. First, there are
| standards bodies that list currencies as "default" set,
| much like we have ISO standard country codes. No one really
| complains if Narnia isn't a country, that Disneyland isn't
| a country, or the Austro-Hungarian empire, for ISO Locales.
|
| At a bare minimum, it should be a reasonable fixed point
| type that correctly handles rounding and intermediate
| values. So a dollar amount like 123.45 times a rate like
| 0.3450 doesn't exceed 4 decimal places but intermediate
| values are extended so we get correct rounding. The
| destination should probably determine the number of places.
| That bare minimum wouldn't stop you from comparing yen to
| dollars, any more than a floating point representing mph
| stops you from comparing it to a value representing kph.
|
| But there are times where we need to track prices to the
| nearest tenth or hundredth of a cent. So it should be
| extensible so that 123.456 dollars * 0.3450 winds up at a
| correct round/decimal places.
|
| You also don't need always-on, real time currency
| conversion. You could have a conversion type, operator, or
| method that does safe conversion based on the value I give
| it. So if I estimate that Yen are about 130 to the dollar,
| I can just use that. If I happen to write an application
| that queries a data provider and can populate that in 'real
| time,' that's up to me.
|
| If you really wanted, you could find a way to create new
| types that represent currencies that aren't part of the
| basic implementation. That might mean you need to specify
| some things like the representation for different locales,
| or the default number of digits.
| marcosdumay wrote:
| > Can you confidently say that $100>Y=10 in an offline
| environment?
|
| Yeah, one more reason for it to be a different type.
|
| > Where is the line drawn?
|
| It's not.
|
| As a rule, measurement unities have an absurdly bad support
| from computers.
| scrollaway wrote:
| > _Can you confidently say that $100 >Y=10 in an offline
| environment?_
|
| Money is never converted. It is _exchanged_. Trying to
| solve this is like trying to solve the question "Is "100
| USD" > "1 LAPTOP".
|
| When you turn 100 USD into 90 EUR, you didn't convert it.
| You exchanged it. You bought EUR, at a price given to you
| by someone or something exchanging it. This could be a
| bank, a well-established currency office, or some dude on
| the street. There is _no real difference_ between all three
| of those: The third party gave you a price, and now has
| more USD and less EUR, whereas you have more EUR and less
| USD.
|
| There are various entities publishing standardized average
| rates which are calculated after the day closes, based on a
| variety of datapoints they have access to. Those are often
| used in eg. accounting, to establish the "real" value of
| something you bought in a currency you don't often use, but
| it's not true conversion.
|
| If you have, as a datatype, a currency becoming another,
| there is ALWAYS a "rate" attached to this. So the question
| "$100>Y=10" you asked above requires more data, it should
| be "$100>Y=10 @ 144.28". ANYTHING else is a terrible leaky
| abstraction. Don't do it. Source your rates automatically
| from a single source if you like, but make it explicit.
|
| Anyway, a "Money" object really is just this: A precise
| decimal object, with an ISO currency code. The latter
| simply being a short string among an included, limited set.
| jerf wrote:
| "Can you confidently say that $100>Y=10 in an offline
| environment?"
|
| A major problem money has is that it isn't a unit in the
| sense we usually take the term to mean. We expect, for
| instance, that translating one unit to another with a
| suitable level of precision should be translatable without
| loss back to the original unit, but that's not true for
| money, even ignoring transaction costs. If "US dollar" is a
| unit, it is a unit that technically stands alone in its own
| universe, not truly convertible to anything else, not even
| other currencies. All conversions are transient events with
| no repeatability. But that is very inconvenient to deal
| with, and with sufficient value stability of all the
| relevant values, often it's a sufficient approximation to
| just pretend it's a unit. But if you zoom in enough, the
| approximation breaks down.
|
| For that and similar reasons, while you could theoretically
| write that line of code, it would be implicitly depending
| on a huge pile of what would in most languages be global
| state. It would be a dubious line of code.
| isleyaardvark wrote:
| It looks like math but really it is describing an
| exchange of goods.
| sokoloff wrote:
| > Can you confidently say that $100>Y=10 in an offline
| environment?
|
| It's not even clear what the second currency is. US$100 is
| over 14000JPY= or 725CNY=.
|
| If you're offering me to wager whether $100 is more than
| Y=10, I'll take the wager that it is.
| NovemberWhiskey wrote:
| > _as long as you do rounding and comparisons correctly it
| works perfectly fine._
|
| At least until you need to add things, at which point you need
| Kahan's algorithm.
| gymbeaux wrote:
| This is a nit but is there a better way to say "RDBMS databases?"
| It feels like saying "ATM Machine". Could we just say "relational
| database" or "state of the art database management system?"
| nickpeterson wrote:
| A database that just got developed could be called a
| TardyBMS...
|
| I'm sorry I've drank too much coffee this morning.
| gymbeaux wrote:
| A musical database might be called a CardiB-MS
| submeta wrote:
| Have built a fintech startup here in Germany ten years ago. The
| article mentions lots of important things. Here's another one:
|
| Time synchronization. It is incredibly important in fintech
| applications for a number of reasons:
|
| 1. *Transaction Ordering:* Financial transactions often need to
| be processed in the order they were initiated. This is especially
| crucial in high-frequency trading where trades are often made in
| milliseconds or microseconds. A small difference in timing could
| potentially lead to substantial financial gains or losses.
| Therefore, accurate time synchronization ensures fairness and
| order in the execution of these transactions.
|
| 2. *Security:* Accurate timekeeping helps in maintaining
| security. For instance, time-based one-time passwords (TOTPs) are
| widely used in two-factor authentication systems. These passwords
| are valid only for a short period of time and rely on
| synchronized clocks on the server and client side.
|
| 3. *Audit Trails and Dispute Resolution:* Timestamping
| transactions can help create a precise audit trail, which is
| critical for detecting and investigating fraudulent activities.
| In case of any dispute, a detailed and accurate transaction
| history backed by synchronized time can help resolve the issue.
|
| 4. *Distributed Systems:* In distributed systems, time
| synchronization is important to ensure data consistency. Many
| financial systems are distributed over different geographical
| locations, and transactions need to be coordinated between these
| systems in an orderly fashion. This requires all servers to have
| their clocks synchronized.
|
| I am sure there are even more fields where this is relevant.
| troupo wrote:
| > Financial transactions often need to be processed in the
| order they were initiated.
|
| And then the banks send you info in batches and out of order :)
| This happened to us more than once. So the team responsible
| wouldn't settle/cancel a payment for X even if bank said so.
| They would read a few other sync batches yo make sure that
| nothing else changed for X.
|
| Because "Financial transactions need to be processed in the
| order they were initiated" is a must :)
| jorangreef wrote:
| This was also a surprise to me when we started TigerBeetle.
|
| How clock synchronization protocols can experience unaligned
| network partitions, where the ledger database cluster is able
| to continue running, but now with the risk of unsynchronized
| clocks and far-future timestamps, which can in turn then lead
| to money being locked up in 2PC payment protocols.
|
| We therefore spent considerable effort [0] on clock
| synchronization in TigerBeetle, not for the consensus protocol
| --we never risk stale reads or take a chance with clock error
| bounds--but rather simply for accurate audit trails and to keep
| inflight liquidity from being locked up if transactions take
| too long to get rolled back.
|
| [0] https://tigerbeetle.com/blog/three-clocks-are-better-than-
| on...
| yafetn wrote:
| I'd also add:
|
| -- If you're using JSON to pass around monetary quantities (eg.
| from the frontend to the backend), put them in strings as opposed
| to the native number type. You never know what the serializers
| and deserializers across languages will do to your numbers (round
| them, truncate them etc.).
|
| -- Start recording the currency of the transactions as early as
| possible. It can be a separate column in your table.
|
| -- Use TIMESTAMPTZ. Always.
| pg_1234 wrote:
| "-- If you're using JSON to pass around monetary quantities
| (eg. from the frontend to the backend), put them in strings as
| opposed to the native number type. You never know what the
| serializers and deserializers across languages will do to your
| numbers (round them, truncate them etc.)."
|
| I'd go a step further and prefix the strings with an ISO
| currency code ... to stop someone from just feeding it into
| their languages int to float converter and assuming that's ok.
| Only custom built (hopefully safe) converters will work.
| scrollaway wrote:
| Ergh, I get what you're trying to prevent, but this actively
| breaks those custom safe parsers we build, and now you have
| to do some additional active parsing. Please don't do this,
| just set contractual expectations in your API.
| hannofcart wrote:
| Politely, I think this comment may be incorrect advice.
|
| I think correctness and precision requirements of financial
| transactions outweighs any devex concerns. Lots of banking
| and trading APIs do exactly this: pass your currency fields
| as strings.
| SideburnsOfDoom wrote:
| > -- Use TIMESTAMPTZ. Always.
|
| When you're using JSON to pass around datetime data, Use
| ISO8601 date and time with offset info, always.
|
| e.g. "transactionDate": "2023-06-28T15:55:22.511Z"
| fatnoah wrote:
| > Use ISO8601 date and time with offset info, always.
|
| I think that's just sufficient for all use cases everywhere.
| I've been a software engineer for almost 25 years and, of all
| the universal truths I've encountered, implicit and
| unintended changes to datetime offset are the ones I've seen
| at every single job.
| Terr_ wrote:
| > I think [ISO8601 date and time with offset info is]
| sufficient for all use cases everywhere.
|
| TLDR: Also, _which_ timezone is used (not quite the same as
| offset) really does matter--UTC is great but you can 't use
| it _everywhere._
|
| ________
|
| One of my favorite simple examples of this "here be
| dragons" for the new developer: Any system that schedules a
| future calendar-event.
|
| Such events are typically pegged, implicitly if not
| explicitly, to a particular timezone or geographic context.
| For example: "The company's Virtual Summit will occur on
| November 2nd at 1PM Elbonian Xtremesunshine Time, hosted
| out of our central Elbonian headquarters."
|
| In that scenario, _it is impossible to know for sure_ how
| many seconds-from-now it will happen until the moment
| actually happens! "2023-11-02 13:00:00 EXT" is actually a
| contract or spec for recognizing a future condition, one
| that will shift if/when the relevant nation/province/city
| simply declares _their_ clocks shall be set differently.
|
| So if the Elbonian government alters their daylight-savings
| switchover to occur earlier on 11-01 instead of 11-06, then
| the summit just moved. Even if you scheduled everything UTC
| all along... Well, now the summit is overlapping lunchtime
| for everyone in Elbonia, so it moved from their
| perspective.
| SideburnsOfDoom wrote:
| Absolutely, e.. an offset of +2:00 in e.g.
| 2023-06-28T18:00+02:00 could mean Berlin in the summer
| (Central European Summer Time, clocks will change) or in
| Johannesburg (South African Standard Time, not summer and
| clocks don't change). Same offset, different time zone,
| different clock change rules.
|
| As you note, for some uses this _does_ make a difference
| and tracking which one you have can in these cases be
| important.
| prottog wrote:
| > Elbonian Xtremesunshine Time
|
| Ha, good one! One can only hope the Elbonian Parliament
| will have the sense to abolish this stupid Xtremesunshine
| time and observe one time zone year-round.
| Terr_ wrote:
| I took an embarrassing number of minutes trying to pick a
| plausible /E.T/ code not already in use elsewhere, then
| gave up.
| fatnoah wrote:
| > TLDR: Also, which timezone is used
|
| Good catch!
| SideburnsOfDoom wrote:
| As I put in another comment, your class library might have
| a type that is equivalent to ISO 8601 data, indeed is
| convertible to and from it, but is a binary representation
| at runtime compatible with other types in the language.
|
| So this technically isn't ISO8601, and certainly isn't
| "ISO8601 in a string", which is an _interchange format_
| between application with potentially very different
| runtimes. I don't really recommend treating ISO8601 dates
| as mere strings, unless you intend to pass them through
| without even looking at the contents.
|
| I refer to https://learn.microsoft.com/en-
| us/dotnet/api/system.datetime...
|
| Use types that support ISO8601 with an offset, always.
|
| The docs even say that, even if many are not aware:
|
| > consider DateTimeOffset as the default date and time type
| for application development.
|
| https://learn.microsoft.com/en-
| us/dotnet/standard/datetime/c...
| lightbendover wrote:
| Only if that offset info is 'Z' (IMHO).
| hirundo wrote:
| I'm generally not a fan of overloading fields, but we took the
| opportunity to add the currency code when we decided to encode
| monetary quantities in JSON as strings, e.g. "0.02USD". This
| has worked well, particularly because we use a money handling
| library that parses it unchanged.
| zeisss wrote:
| Do you always put the currency behind the number? I know some
| countries prefer the currency infront (e.g. USD), others
| behind (EUR).
|
| But since this is not a human visible field, this is probably
| irrelevant, right?
| yafetn wrote:
| It's probably fine, but this sort of thing is why
| transactionAmount and transactionCurrency are better when
| separated. Consider the case when you're doing some
| reporting from your DB based on currency; do you really
| want to have to deal with one string that contains both or
| just do a simple WHERE clause?
| hirundo wrote:
| In our case those are indeed separate in the database, so
| that we can type the value as a numeric, which is very
| necessary for many operations. Those fields only get
| combined in the serialization.
| hirundo wrote:
| I think I looked at this when I laid it out:
|
| https://ux.stackexchange.com/questions/9105/international-
| cu...
|
| So we put the ISO code at the end and ignore whitespace.
| But including a prefix sign as with `EUR1.234.567,89 EUR`
| would break it.
| hermitdev wrote:
| You also included localized formatting (the '.' and ',' )
| in your example that'll likely break on parsing without
| special handling. For interchange, you'll want to avoid
| using localized formats, e.g. stick to the "C" locale or
| something specific or agreed to between the involved
| parties.
| hannofcart wrote:
| All good points..
|
| One more from me:
|
| - Payment/order processing systems typically involve complex
| logic.
|
| - One of the best ways we managed to keep complexity in check was
| to model these as state machines (with the state itself being
| persisted to DB)
| theptip wrote:
| I think this article covers many of the main points, but could be
| worded/structured better. There are terms of art that more
| precisely refer to the concepts you need.
|
| For "using floating point data types", it's even worse; you often
| need to use strings, for example if you need to store a bank
| account number "01234567789" will have the leading zero stripped
| if you use a numeric type.
|
| "Updating transactions" would be better phrased as "use an
| append-only log / evented architecture". (Also, "use a double-
| entry ledger" is probably the most valuable advice I could have
| sent myself prior to getting into FinTech.)
|
| "Be careful with retry" should be more strictly "use idempotent
| operations" and link to the canonical Stripe article on
| idempotency keys (https://stripe.com/blog/idempotency).
|
| Another important one to think about is bitemporality. "Created
| at" vs "effective at". Not obvious at first and you'll have some
| painful migrations if you don't build it in. Fowler has a good
| overview here:
| https://martinfowler.com/eaaDev/timeNarrative.html.
|
| Edit to add - the advice that maybe using a NoSQL database is
| pretty bad IMO. I'd advise in the opposite direction - use
| SERIALIZABLE isolation in a SQL database. Read up on your Aphyr
| blog posts before trying to do anything distributed. Be paranoid
| about race conditions / serialization anomalies. If you
| eventually hit performance issues you need to think hard about
| what anomalies your access patterns might be subject to.
| (Obviously HFT won't use serializable SQL).
| gunshai wrote:
| Shit, I wish I read this prior to my interviews at Cash App.
| neon_electro wrote:
| In what way do you think it would've made a difference? Asking
| as someone who's applied to Block/Square/Cash App in the past.
| gunshai wrote:
| The difference is small, but mostly as probing questions to
| interviewers. I take the mind the more you can speak the
| language the more comfortable people become with you.
|
| Here's something small but I was doing a SQL interview one
| involved a a card_tx table with the amounts stored in
| integers values in cents. I immediately noticed this and
| chatted with the interviewer about this. My assumptions were
| wrong, and the floating point arithmetic was a simpler reason
| than what I was thinking.
|
| Anyway you can display you have thought about relevant
| business problems to where you are interview engineers tend
| to have a strong reflex to engage the person more.
| potamic wrote:
| #2 and #3 are pretty solid advice. But the rest of them are
| nothing specific to fintech. I will also offer a few additional
| points.
|
| 1. Never record an amount without its currency.
|
| 2. Reconcile all your data all the time. Never let any data go
| unaccounted for.
|
| 3. Maker-checker is a powerful concept. Embrace it to the fullest
| across your system.
|
| 4. You will be dealing with all sorts of non-standardized
| financial integrations. A lot. Think adapter pattern as early as
| possible.
|
| 5. You will be answering to multiple regulatory agencies. Create
| boundaries between them within your system and reduce the surface
| of compliance as much as possible.
| SideburnsOfDoom wrote:
| The recommendation that "developers should use integers to
| represent money" (with an implicit 2 decimal places, that will
| work for many but not all currencies) is not a great one.
|
| If your language has a dedicated type for monetary amounts, use
| that. (1)
|
| If it does not, but you can make a value object to represent,
| e.g. amount and currency code, then do that.
|
| If however, your language does not have a dedicated type for
| monetary amounts, or one cannot be trivially built or retrieved
| as a package (2), then you should ask yourself if it is really a
| suitable language for financial tasks.
|
| 1) https://learn.microsoft.com/en-us/dotnet/api/system.decimal
|
| 2) https://github.com/shopspring/decimal
| https://www.npmjs.com/package/ts-money
| adenhoed wrote:
| It's nice if your language has support for monetary amounts,
| but usually you end up using multiple languages that interact
| with the same database model, and you still end up using a
| built in numeric datatype in your RDBMS of choice.
|
| Three additional 'mistakes' to prevent when dealing with money
| representations:
|
| 1. The definition of a currency might change. For example, some
| years ago Iceland decided to change the exponent of ISK from 2
| to 0. Currencies have different versions.
|
| 2. As a FinTech you probably have integrations with many third
| parties, they don't change their exponents for a currency at
| the same time. Keep track of what third parties think the
| correct exponent is at any point in time, and convert between
| your representation and their representation. Otherwise, you'll
| have interesting incidents (e.g. transferring 100x the intended
| amount of ISK).
|
| 3. At first you think that counting minor units as an integer
| is enough, and then you need to start accounting for fractions
| of cents because sales people sold something for a fee of
| $0.0042 per transaction. If your code rounds all these fees to
| $0.00 you don't make any money.
| SideburnsOfDoom wrote:
| 1, 2 - Yeah, there's always a sanity checking and conversion
| layer around the 3rd party. Currencies do indeed sometimes
| have 2 versions in play there.
|
| 3 - Indeed, the currency type that I referenced "is
| appropriate for financial calculations that require large
| numbers of significant integral and fractional digits and no
| round-off errors"
| gytisgreitai wrote:
| Stripe uses integers [1]
|
| Adyen uses integers [2]
|
| Square uses integers [3]
|
| So not sure that the industry would agree with you
|
| [1] https://stripe.com/docs/api/prices/create#create_price-
| unit_...
|
| [2] https://docs.adyen.com/api-
| explorer/Checkout/70/post/payment...
|
| [3]
| https://developer.squareup.com/reference/square/objects/Mone...
| SideburnsOfDoom wrote:
| > Stripe uses integers [1]
|
| a) On web api, so this is by definition data interchange, not
| "in a language".
|
| b) with a currency code to cross check it.
|
| c) implicitly "in cents" which needs further documentation -
| does this mean "pence" when the currency is GBP? How does
| this work for for JPY? BHD?
|
| d) cannot represent fractions of a cent or penny.
|
| So: this int plus currency code plus docs is OK but not
| great, a lowest common denominator format for interchange,
| requires further documentation and cross-checking as "100
| USD" does not mean 100 bucks and the conversion is currency-
| specific, and cannot represent all values.
|
| I wouldn't refuse to convert _to and from_ this format at the
| edges of my application, for data interchange, but the
| conversion to something clearer and richer IMHO should remain
| there.
|
| Payment APIs are far from the only use case in fintech, for
| interest calculations you do have to care about fractions of
| a cent or penny.
|
| In fact, if your case is "I call the stripe api" ... are you
| sure you're a fintech and not an online store? Get back to me
| when you have to interop with FiServ, MasterCard or SAP.
| gytisgreitai wrote:
| yet with all those a) b) c) d) they still use that. Point
| is not "I call stripe api". Point is - those companies,
| that process wast amount of transactions, use integers.
| Probably for a reason. And its not in cents rather _minor
| units_ which you as working in fintech should ought to
| understand the difference
| SideburnsOfDoom wrote:
| > Point is - those companies, that process vast amount of
| transactions, use integers.
|
| Point missed, they use that format _on their client
| apis_, which tells you nothing about what they do in the
| code that handles it. Now it could be the same
| internally, in which case the reason is "there's crap
| code everywhere"
| molsongolden wrote:
| One more: Timezones
|
| Various platforms use: UTC, the user's timezone as set in their
| dashboard, the user's detected timezone, the timezone of the
| server that is generating the reports.
|
| Aggregating or reconciling data from different platforms can be a
| pain if the timezones aren't clearly indicated.
|
| I've had bank statements that didn't agree to CSV exports of the
| same data because the servers generating the two reports were in
| different timezones.
| SilasX wrote:
| Haha yeah for the past tax year I I looked at my transaction
| history and it was missing some that I expected to be there.
| Turns out, the exchange (Gemini, for crypto) uses UTC, but the
| IRS docs always tell you that a cutoff is relative to your own
| time zone.
|
| The transactions were in the late evening on New Year's Eve,
| central US time, but that was already the next year by UTC, so
| any readout of "transactions for 2022" would not include them.
|
| [1] e.g. https://www.irs.gov/taxtopics/tc301
| koolba wrote:
| That'd be a great subplot for a movie. The antagonist sets up
| an elaborate money laundering operation that leverages
| conflicting definitions of the taxable year and moves off
| shore profits that are "lost" between the systems. Could call
| it " _Black Ink to the Future_.
| radiator wrote:
| That is supposed to be a festive hour, not one for trading.
| SilasX wrote:
| Haha, you're not wrong. I think it was actually more like
| 8pm, shortly before I left to live it up. But either way,
| transacting near the new year shouldn't break financial
| systems.
| SideburnsOfDoom wrote:
| That's why Datetime data should be handled in a type that
| includes this data (e.g. DateTimeOffset (1) ) and exchanged as
| ISO 8601 with offset. e.g. "2023-06-28T15:55:22+01:00"
|
| 1) https://learn.microsoft.com/en-
| us/dotnet/api/system.datetime...
| theptip wrote:
| Furthermore, even for "date fields", consider using a datetime.
| Every date implicitly exists in a timezone, and if you ignore
| that ambiguity you'll get bitten later.
|
| For example, an invoice due on Friday is probably actually due
| by close of business (5pm say) in the timezone your business
| operates, and if created at 11pm it would be processed the next
| day (or even on Monday, don't get me started about business day
| calculations).
| bostik wrote:
| > _Every date implicitly exists in a timezone, and if you
| ignore that ambiguity you'll get bitten later._
|
| Solid advice, and must come from painful burns. I've been
| preaching from the same book for a few years now: a timestamp
| without timezone offset is worse than useless.
|
| Or as the DB expert in previous job so eloquently put it...
| Timestamp with zone tells you when an event actually
| happened. A timestamp without zone or offset is equal to wall
| clock time inside a windowless room, which itself is in an
| unspecified location somewhere on the planet.
| lightbendover wrote:
| We stored and transmitted exclusively UTC, which put the burden
| on the UI at display time (trivial) and backend business logic
| (complex, but simplified with robust utility functions that
| everyone understood well) in accounting for timezones when
| material.
|
| We only had a dozen issues due to it so I believe it worked
| pretty well compared to the war stories I heard from comparable
| companies in the fintech space.
| indymike wrote:
| This is a great little article, and I can't emphasize what a big
| deal #2 is.
| JohnBooty wrote:
| Some folks will laugh at this advice.
|
| Others (like me) will cry because we know of multibillion-
| dollar fintechs that still struggle with this.
|
| The one I'm thinking of didn't even _have_ cents for a long
| time. After a pretty heroic migration effort they added cents.
| And they did it properly. But within weeks folks were using
| floats all over the place for money, leading to flaky tests and
| all kinds of other errors.
| JackFr wrote:
| I have worked at banks and fintechs for the past 30 years and
| honestly have never used anything but a doubles for money
| with no issues (and a simpler code base.)
|
| I understand the sentiment and the potential issues, but it's
| really kind of domain dependent.
| Brystephor wrote:
| Is storing $1.05 really that much simpler than storing
| something as 105? Use a long to store money, pretty easy.
| gpderetta wrote:
| I might or might not have stories about fix numerical
| types underflowing.
|
| Not that fix nums are wrong, but they are far from safe.
| indymike wrote:
| Storing is one thing, computation is another :-)
| foobarian wrote:
| _JPY has entered the room_
| itsthecourier wrote:
| I'm surprised the article doesn't explain how float numbers are
| stored in memory according to IEEE that's the main reason you
| don't use them to represent money.
|
| If you are an engineer who doesn't know what a mantissa is, stop
| two minutes for a cool ride https://lashewi.medium.com/storing-
| currency-values-and-float...
| agentultra wrote:
| Don't be afraid to use formal methods. Queues, retries, event
| sourcing, payment state handling -- there are "global" properties
| we desire from these systems: certain things we want to make sure
| are always true and others that we want to make sure eventually
| happen. For the single process case, which nearly every developer
| thinks about, it can seem like a solved problem but we live in a
| concurrent world with network failures and vendors with errors in
| their own systems: it's nearly impossible for you to think really
| hard in your head and be sure that your queue retry strategy will
| maintain those properties that are important to your business.
| It's nearly impossible to do this with lightweight, informal
| testing strategies employed by busy software teams.
|
| By modelling your systems you will learn what the important
| failure modes are and you will get better at designing systems
| that are resilient and efficient.
|
| Card payment systems are fairly unreliable peer-to-peer messaging
| systems. Be prepared for a lot of complexity. Using an event-
| sourcing architecture is really useful here for that "auditing"
| requirement and for debugging transaction state when the network
| sends you messages in error, out of order, or they forget to
| retry themselves when they promised to, when merchants send bad
| data, when POS systems do weird things, etc.
| donatj wrote:
| I wish strictly INSERT-only tables were a standard SQL feature.
| It would be so very useful.
| lixtra wrote:
| What's the advantage of INSERT-only table over having a normal
| table and all normal users only have INSERT privilege on that
| table?
|
| As admin I can still recreate the whole insert table with
| modified data, etc.
| dspillett wrote:
| _> Over the last decade, fintechs using RDBMS databases to record
| transactions have built features onto their databases like a
| journal of all changes to all data and clever ways of making sure
| the data hasn't been modified by storing checksums of data as
| transactions are added. (under "Updating transactions")_
|
| It is worth noting that many of these methods don't _prevent_
| tampering, they only make it visible when you look for it at
| which point the fact you are explicitly looking for it (rather
| than having been alerted to a potential issue) might imply it is
| too late.
| bostik wrote:
| Which is also why auditors love git.
|
| The number one thing they crave is immutability. But when
| that's not an option, tamper-evident comes as a close second.
| acadavid79 wrote:
| A little off-topic, but I would have loved to see mifos /
| fineract[0] referenced in the article. Great open source banking
| core having many of the strengths listed
|
| [0] https://github.com/apache/fineract/
| say_it_as_it_is wrote:
| Why should anyone have a look at TigerBeetle, a bleeding edge
| accounting system built with a language no one understands? Since
| no one understands Zig, no one will be able to maintain
| TigerBeetle. The company is exposing itself to a great deal of
| risk with this decision. Adopting TigerBeetle is a colossal
| business mistake, not an engineering mistake.
| jorangreef wrote:
| Hey, Joran from TigerBeetle here!
|
| You raise a fair question.
|
| Coincidentally, one of the reasons we picked Zig was for how
| readable it was, and strikingly so, even for high level
| programmers who might not understand systems programming or C.
| Because Zig reads like TypeScript, and we were working in
| payment switches where the majority of programmers could read
| that. This particular switch, in fact, had this same business
| requirement, that programmers should be able to at least read
| the systems language.
|
| But generally, our experience has been that people who
| understand C will understand how to maintain Zig [0]. Zig's
| toolchain is also more accessible, and across all platforms.
| Zig's compiler is already being used by Uber for hermetic
| builds.
|
| It's also easy to learn. You can pick up Zig in a week and be
| comfortable in a month. Zig has a simple grammar. I love how,
| when we have someone join the team, we never have a discussion
| about how to learn Zig, as if it's a difficult language to
| master (like C++!). Rather, there's excitement around learning
| the language, even ahead of starting at work, and within a day
| or two they're committing.
|
| We made this decision for TigerBeetle in July 2020, and didn't
| take it lightly. We had already followed Zig's progress for 2
| years by that point, and many factors were considered
| [1][2][3]. C was the other contender, given that we had to
| handle memory allocation failure.
|
| The crux of the decision, then, was whether to invest in a
| systems language of the last 30 years, or in a systems language
| of the next 30 years. A distributed database is a big
| investment. It made sense to invest for the future. If
| anything, it would have been a colossal business mistake to
| have picked C or C++, which would have crippled our development
| velocity.
|
| Furthermore, for TigerBeetle's design goals, especially w.r.t.
| our adoption of NASA's Power of Ten Rules for Safety-Critical
| Code and thus static memory allocation, Zig made (and continues
| to make) the most sense.
|
| We also liked the efficient performance culture surrounding
| Zig, with talented game developers like Michal Ziulek and
| Stephen Gutekanst, and embedded programmers like xq, Matt
| Knight, Jens Goldberg and others moving to it. These industries
| (gaming, embedded) are often a good litmus test of where
| systems programming is at.
|
| More details (our thinking on Zig through the lens of
| safety/performance/tooling/ecosystem/hiring/marketing) here:
|
| [0] https://kristoff.it/blog/maintain-it-with-zig/
|
| [1] https://docs.tigerbeetle.com/FAQ#why-is-tigerbeetle-
| written-...
|
| [2] Building a Distributed DB in Zig:
| https://www.youtube.com/watch?v=0pHHb6ONTbw
|
| [3] TigerStyle! (How to Design Safer Systems in Less Time):
| https://www.youtube.com/watch?v=w3WYdYyjek4
| TeeWEE wrote:
| I really mis the mistake of NOT using double entry bookkeeping in
| the way you store transactions.
|
| Next to that, ensure transactions are immutable, and ensure
| reporting is idempotent: Close your periods and generate reports
| for them, when you regenerate the report it should be identical.
|
| This is more domain tips. A lot of technical tips hold for non
| fintech too.
| mhh__ wrote:
| Tip: you can express (city) time "zones" in the type system e.g.
| London time and New York time.
| gpderetta wrote:
| You can, but because the full set if timezones is only known at
| runtime, it is not terribly useful. You can statically
| distinguish a canonical static timezone (say UTC) vs a dynamic
| ine though.
|
| Same for currency.
| phoehne wrote:
| RDMS databases are not a weak point. A bad model of your
| transaction is a weak point. Depending on the application, and
| not all applications require this, you may need a consistent read
| state for the data. Eventually consistent is like saying
| periodically wrong. Let's say a trader enters a trade for that
| exceeds there allowed VAR, because they looked at their screen
| thought they had more dry powder. (Cocaine jokes aside). That
| becomes a risk management problem. (Also the risk that traders
| can just use the 'It said I was below my limit' excuse).
|
| If you want an example where consistency is not important, you
| might be able to overdraw from your bank with your ATM card. The
| bank is happy for this to be inconsistent, since they can charge
| for the overdraft.
___________________________________________________________________
(page generated 2023-06-28 23:02 UTC)