▲ 153 points
back
110 comments
Have built a fintech startup here in Germany ten years ago. The article mentions lots of important things. Here‘s another one:
Time synchronization. It is incredibly important in fintech applications for a number of reasons:
1. *Transaction Ordering:* Financial transactions often need to be processed in the order they were initiated. This is especially crucial in high-frequency trading where trades are often made in milliseconds or microseconds. A small difference in timing could potentially lead to substantial financial gains or losses. Therefore, accurate time synchronization ensures fairness and order in the execution of these transactions.
2. *Security:* Accurate timekeeping helps in maintaining security. For instance, time-based one-time passwords (TOTPs) are widely used in two-factor authentication systems. These passwords are valid only for a short period of time and rely on synchronized clocks on the server and client side.
3. *Audit Trails and Dispute Resolution:* Timestamping transactions can help create a precise audit trail, which is critical for detecting and investigating fraudulent activities. In case of any dispute, a detailed and accurate transaction history backed by synchronized time can help resolve the issue.
4. *Distributed Systems:* In distributed systems, time synchronization is important to ensure data consistency. Many financial systems are distributed over different geographical locations, and transactions need to be coordinated between these systems in an orderly fashion. This requires all servers to have their clocks synchronized.
I am sure there are even more fields where this is relevant.
This was also a surprise to me when we started TigerBeetle.
How clock synchronization protocols can experience unaligned network partitions, where the ledger database cluster is able to continue running, but now with the risk of unsynchronized clocks and far-future timestamps, which can in turn then lead to money being locked up in 2PC payment protocols.
We therefore spent considerable effort [0] on clock synchronization in TigerBeetle, not for the consensus protocol—we never risk stale reads or take a chance with clock error bounds—but rather simply for accurate audit trails and to keep inflight liquidity from being locked up if transactions take too long to get rolled back.
[0] https://tigerbeetle.com/blog/three-clocks-are-better-than-on...
Also, sometimes currencies require timestamps.
You don't represent money as (<currency>, <decimal amount>), you may want to store it as (<currency>, <decimal amount>, <timestamp>) to be able to apply the correct exchange rate post-facto, and not have to deduce from the transaction history.
It also helps with cross-checks/consistency verification.
Also, preserve the original timezone in the timestamp. It can save many headaches down the road.
> Financial transactions often need to be processed in the order they were initiated.
And then the banks send you info in batches and out of order :) This happened to us more than once. So the team responsible wouldn't settle/cancel a payment for X even if bank said so. They would read a few other sync batches yo make sure that nothing else changed for X.
Because "Financial transactions need to be processed in the order they were initiated" is a must :)
All of this is exactly why mainframes still exist and still excel at what they do.
I’d also add:
— If you’re using JSON to pass around monetary quantities (eg. from the frontend to the backend), put them in strings as opposed to the native number type. You never know what the serializers and deserializers across languages will do to your numbers (round them, truncate them etc.).
— Start recording the currency of the transactions as early as possible. It can be a separate column in your table.
— Use TIMESTAMPTZ. Always.
> — Use TIMESTAMPTZ. Always.
When you’re using JSON to pass around datetime data, Use ISO8601 date and time with offset info, always.
e.g. "transactionDate": "2023‐06‐28T15:55:22.511Z"
"— If you’re using JSON to pass around monetary quantities (eg. from the frontend to the backend), put them in strings as opposed to the native number type. You never know what the serializers and deserializers across languages will do to your numbers (round them, truncate them etc.)."
I'd go a step further and prefix the strings with an ISO currency code ... to stop someone from just feeding it into their languages int to float converter and assuming that's ok. Only custom built (hopefully safe) converters will work.
I'm generally not a fan of overloading fields, but we took the opportunity to add the currency code when we decided to encode monetary quantities in JSON as strings, e.g. "0.02USD". This has worked well, particularly because we use a money handling library that parses it unchanged.
Don't be afraid to use formal methods. Queues, retries, event sourcing, payment state handling -- there are "global" properties we desire from these systems: certain things we want to make sure are always true and others that we want to make sure eventually happen. For the single process case, which nearly every developer thinks about, it can seem like a solved problem but we live in a concurrent world with network failures and vendors with errors in their own systems: it's nearly impossible for you to think really hard in your head and be sure that your queue retry strategy will maintain those properties that are important to your business. It's nearly impossible to do this with lightweight, informal testing strategies employed by busy software teams.
By modelling your systems you will learn what the important failure modes are and you will get better at designing systems that are resilient and efficient.
Card payment systems are fairly unreliable peer-to-peer messaging systems. Be prepared for a lot of complexity. Using an event-sourcing architecture is really useful here for that "auditing" requirement and for debugging transaction state when the network sends you messages in error, out of order, or they forget to retry themselves when they promised to, when merchants send bad data, when POS systems do weird things, etc.
One more: Timezones
Various platforms use: UTC, the user’s timezone as set in their dashboard, the user’s detected timezone, the timezone of the server that is generating the reports.
Aggregating or reconciling data from different platforms can be a pain if the timezones aren’t clearly indicated.
I’ve had bank statements that didn’t agree to CSV exports of the same data because the servers generating the two reports were in different timezones.
Haha yeah for the past tax year I I looked at my transaction history and it was missing some that I expected to be there. Turns out, the exchange (Gemini, for crypto) uses UTC, but the IRS docs always tell you that a cutoff is relative to your own time zone.
The transactions were in the late evening on New Year's Eve, central US time, but that was already the next year by UTC, so any readout of "transactions for 2022" would not include them.
[1] e.g. https://www.irs.gov/taxtopics/tc301
Furthermore, even for “date fields”, consider using a datetime. Every date implicitly exists in a timezone, and if you ignore that ambiguity you’ll get bitten later.
For example, an invoice due on Friday is probably actually due by close of business (5pm say) in the timezone your business operates, and if created at 11pm it would be processed the next day (or even on Monday, don’t get me started about business day calculations).
We stored and transmitted exclusively UTC, which put the burden on the UI at display time (trivial) and backend business logic (complex, but simplified with robust utility functions that everyone understood well) in accounting for timezones when material.
We only had a dozen issues due to it so I believe it worked pretty well compared to the war stories I heard from comparable companies in the fintech space.
That's why Datetime data should be handled in a type that includes this data (e.g. DateTimeOffset (1) ) and exchanged as ISO 8601 with offset. e.g. "2023‐06‐28T15:55:22+01:00"
1) https://learn.microsoft.com/en-us/dotnet/api/system.datetime...
#2 and #3 are pretty solid advice. But the rest of them are nothing specific to fintech. I will also offer a few additional points.
1. Never record an amount without its currency.
2. Reconcile all your data all the time. Never let any data go unaccounted for.
3. Maker-checker is a powerful concept. Embrace it to the fullest across your system.
4. You will be dealing with all sorts of non-standardized financial integrations. A lot. Think adapter pattern as early as possible.
5. You will be answering to multiple regulatory agencies. Create boundaries between them within your system and reduce the surface of compliance as much as possible.
I think this article covers many of the main points, but could be worded/structured better. There are terms of art that more precisely refer to the concepts you need.
For “using floating point data types”, it’s even worse; you often need to use strings, for example if you need to store a bank account number “01234567789” will have the leading zero stripped if you use a numeric type.
“Updating transactions” would be better phrased as “use an append-only log / evented architecture”. (Also, “use a double-entry ledger” is probably the most valuable advice I could have sent myself prior to getting into FinTech.)
“Be careful with retry” should be more strictly “use idempotent operations” and link to the canonical Stripe article on idempotency keys (https://stripe.com/blog/idempotency).
Another important one to think about is bitemporality. “Created at” vs “effective at”. Not obvious at first and you’ll have some painful migrations if you don’t build it in. Fowler has a good overview here: https://martinfowler.com/eaaDev/timeNarrative.html.
Edit to add - the advice that maybe using a NoSQL database is pretty bad IMO. I’d advise in the opposite direction - use SERIALIZABLE isolation in a SQL database. Read up on your Aphyr blog posts before trying to do anything distributed. Be paranoid about race conditions / serialization anomalies. If you eventually hit performance issues you need to think hard about what anomalies your access patterns might be subject to. (Obviously HFT won’t use serializable SQL).
> if you need to store a bank account number “01234567789” will have the leading zero stripped if you use a numeric type.
This goes for any "number" that is actually an identifier. Phone numbers aren't really numbers either for example.
Rule of thumb: if it doesn't make sense to do math with it, it should probably be a numeric type unless there's a very good reason.
> Over the last decade, fintechs using RDBMS databases to record transactions have built features onto their databases like a journal of all changes to all data and clever ways of making sure the data hasn’t been modified by storing checksums of data as transactions are added. (under “Updating transactions”)
It is worth noting that many of these methods don't prevent tampering, they only make it visible when you look for it at which point the fact you are explicitly looking for it (rather than having been alerted to a potential issue) might imply it is too late.
Which is also why auditors love git.
The number one thing they crave is immutability. But when that's not an option, tamper-evident comes as a close second.
The recommendation that "developers should use integers to represent money" (with an implicit 2 decimal places, that will work for many but not all currencies) is not a great one.
If your language has a dedicated type for monetary amounts, use that. (1)
If it does not, but you can make a value object to represent, e.g. amount and currency code, then do that.
If however, your language does not have a dedicated type for monetary amounts, or one cannot be trivially built or retrieved as a package (2), then you should ask yourself if it is really a suitable language for financial tasks.
1) https://learn.microsoft.com/en-us/dotnet/api/system.decimal
2) https://github.com/shopspring/decimal https://www.npmjs.com/package/ts-money
Stripe uses integers [1]
Adyen uses integers [2]
Square uses integers [3]
So not sure that the industry would agree with you
[1] https://stripe.com/docs/api/prices/create#create_price-unit_...
[2] https://docs.adyen.com/api-explorer/Checkout/70/post/payment...
[3] https://developer.squareup.com/reference/square/objects/Mone...
It's nice if your language has support for monetary amounts, but usually you end up using multiple languages that interact with the same database model, and you still end up using a built in numeric datatype in your RDBMS of choice.
Three additional 'mistakes' to prevent when dealing with money representations:
1. The definition of a currency might change. For example, some years ago Iceland decided to change the exponent of ISK from 2 to 0. Currencies have different versions.
2. As a FinTech you probably have integrations with many third parties, they don't change their exponents for a currency at the same time. Keep track of what third parties think the correct exponent is at any point in time, and convert between your representation and their representation. Otherwise, you'll have interesting incidents (e.g. transferring 100x the intended amount of ISK).
3. At first you think that counting minor units as an integer is enough, and then you need to start accounting for fractions of cents because sales people sold something for a fee of $0.0042 per transaction. If your code rounds all these fees to $0.00 you don't make any money.
I really mis the mistake of NOT using double entry bookkeeping in the way you store transactions.
Next to that, ensure transactions are immutable, and ensure reporting is idempotent: Close your periods and generate reports for them, when you regenerate the report it should be identical.
This is more domain tips. A lot of technical tips hold for non fintech too.
RDMS databases are not a weak point. A bad model of your transaction is a weak point. Depending on the application, and not all applications require this, you may need a consistent read state for the data. Eventually consistent is like saying periodically wrong. Let's say a trader enters a trade for that exceeds there allowed VAR, because they looked at their screen thought they had more dry powder. (Cocaine jokes aside). That becomes a risk management problem. (Also the risk that traders can just use the 'It said I was below my limit' excuse).
If you want an example where consistency is not important, you might be able to overdraw from your bank with your ATM card. The bank is happy for this to be inconsistent, since they can charge for the overdraft.
The bank cannot charge for this unless you as the customer have opted in. See authorized positive settle negative. US based banks.
I'm surprised the article doesn't explain how float numbers are stored in memory according to IEEE that's the main reason you don't use them to represent money.
If you are an engineer who doesn't know what a mantissa is, stop two minutes for a cool ride https://lashewi.medium.com/storing-currency-values-and-float...
This is a great little article, and I can't emphasize what a big deal #2 is.
Some folks will laugh at this advice.
Others (like me) will cry because we know of multibillion-dollar fintechs that still struggle with this.
The one I'm thinking of didn't even have cents for a long time. After a pretty heroic migration effort they added cents. And they did it properly. But within weeks folks were using floats all over the place for money, leading to flaky tests and all kinds of other errors.
All good points..
One more from me:
- Payment/order processing systems typically involve complex logic.
- One of the best ways we managed to keep complexity in check was to model these as state machines (with the state itself being persisted to DB)
This is a nit but is there a better way to say “RDBMS databases?” It feels like saying “ATM Machine”. Could we just say “relational database” or “state of the art database management system?”
A database that just got developed could be called a TardyBMS…
I’m sorry I’ve drank too much coffee this morning.
Shit, I wish I read this prior to my interviews at Cash App.
In what way do you think it would've made a difference? Asking as someone who's applied to Block/Square/Cash App in the past.
Why should anyone have a look at TigerBeetle, a bleeding edge accounting system built with a language no one understands? Since no one understands Zig, no one will be able to maintain TigerBeetle. The company is exposing itself to a great deal of risk with this decision. Adopting TigerBeetle is a colossal business mistake, not an engineering mistake.
Hey, Joran from TigerBeetle here!
You raise a fair question.
Coincidentally, one of the reasons we picked Zig was for how readable it was, and strikingly so, even for high level programmers who might not understand systems programming or C. Because Zig reads like TypeScript, and we were working in payment switches where the majority of programmers could read that. This particular switch, in fact, had this same business requirement, that programmers should be able to at least read the systems language.
But generally, our experience has been that people who understand C will understand how to maintain Zig [0]. Zig's toolchain is also more accessible, and across all platforms. Zig's compiler is already being used by Uber for hermetic builds.
It's also easy to learn. You can pick up Zig in a week and be comfortable in a month. Zig has a simple grammar. I love how, when we have someone join the team, we never have a discussion about how to learn Zig, as if it's a difficult language to master (like C++!). Rather, there's excitement around learning the language, even ahead of starting at work, and within a day or two they're committing.
We made this decision for TigerBeetle in July 2020, and didn't take it lightly. We had already followed Zig's progress for 2 years by that point, and many factors were considered [1][2][3]. C was the other contender, given that we had to handle memory allocation failure.
The crux of the decision, then, was whether to invest in a systems language of the last 30 years, or in a systems language of the next 30 years. A distributed database is a big investment. It made sense to invest for the future. If anything, it would have been a colossal business mistake to have picked C or C++, which would have crippled our development velocity.
Furthermore, for TigerBeetle's design goals, especially w.r.t. our adoption of NASA's Power of Ten Rules for Safety-Critical Code and thus static memory allocation, Zig made (and continues to make) the most sense.
We also liked the efficient performance culture surrounding Zig, with talented game developers like Michal Ziulek and Stephen Gutekanst, and embedded programmers like xq, Matt Knight, Jens Goldberg and others moving to it. These industries (gaming, embedded) are often a good litmus test of where systems programming is at.
More details (our thinking on Zig through the lens of safety/performance/tooling/ecosystem/hiring/marketing) here:
[0] https://kristoff.it/blog/maintain-it-with-zig/
[1] https://docs.tigerbeetle.com/FAQ#why-is-tigerbeetle-written-...
[2] Building a Distributed DB in Zig: https://www.youtube.com/watch?v=0pHHb6ONTbw
[3] TigerStyle! (How to Design Safer Systems in Less Time): https://www.youtube.com/watch?v=w3WYdYyjek4
A little off-topic, but I would have loved to see mifos / fineract[0] referenced in the article. Great open source banking core having many of the strengths listed
You can absolutely use doubles for money. Excel does it and so do many other financial tools. There are pros and cons but as long as you do rounding and comparisons correctly it works perfectly fine.
For example 16.10 is not exactly 16.10 in IEEE floating point. When you do enough operations, and depending on the order of operations, you can wind up several cents off. That sounds small, but can be enough to give your auditors heart-burn. COBOL does BCD arithmetic, (not really but at least conceptually), and it's penny accurate to 31 digits (as per the standard - implementations may have greater accuracy). Frankly, it's stupid that 63 years after COBOL we're still treating money and currency as an afterthought in languages that are supposed to be business oriented. Proper currency handling should be part of the language.
> There are pros and cons but as long as you do rounding and comparisons correctly it works perfectly fine.
This is exactly the issue with using floats where an arbitrary precision decimal with proper rounding is really needed. Easily solved with a good library and if your languages supports it, type, but it's really easy for a dev in a hurry to not use the library and roll some a=b+b*c_rate code that forces some type conversion. The rounding rules often are tied to contracts, and a subtle bug that's off a few mills here (total problem created $2.33) and there can lead to audits (total cost of audit $14,800) that cost a lot.
The question is why? What does the double give you over a 64-bit integer? Sure when you divide and it leaves a fractional part you loose it and need to think about what happens there explicitly but you need to do the same for doubles to avoid pennies going missing and snowballing into larger errors.
If Excel uses doubles for money, it should be a warning sign. It uses simple numeric data type (I guess just ints) for freaking dates.. I can trace at least a couple of bugs in my career to just that fact.
If you want a simple real life example (unconnected to the way numbers are stored) here it is (accounting may have quirks that arrive unexpected).
In EU prices to customers are required to be comprehensive of VAT, so a price is € 60,00 included VAT 10%.
But in an invoice/receipt you have to explicit how much is the net and how much is the tax, so 60 / 1.10 = 54.55 and VAT is 54.55 x 0.10 = 5.46 which makes a nice 60.01.
You may be tempted to round down the 60 / 1.10 = 54.54 and have VAT 54.54 x 0.10 = 5.45 but this makes 59.99.
What do you mean by "correctly"? It sounds like victim blaming. For money, I don't think it's perfectly fine to use numbers where this isn't true.
0.1 + 0.2 <= 0.3 // sure looks true to meIt's sufficient for quantitative finance and trading but not for accounting. For financial engineering and modeling it doesn't matter if results are a few fractions of a cent off as long as your profit margins/model tolerances are greater than the error, because your broker/bank/exchange will keep track of the exact values in your account. But if you are building a bank/broker/exchange, then tracking it precisely enough for GAAP is now your problem.
>as long as you do rounding and comparisons correctly it works perfectly fine.
At least until you need to add things, at which point you need Kahan's algorithm.
Tip: you can express (city) time "zones" in the type system e.g. London time and New York time.
I wish strictly INSERT-only tables were a standard SQL feature. It would be so very useful.
And that's why I shall be sticking to cash for the next 10 years till all the beta testing is complete.
By which time hopefully interest bearing CBDCs will show up, and make all these mindless intermediaries sitting between my wallet and someone elses wallet obsolete.