The Open Banking Non-Standards
The UK's Open Banking standards have done wonders for innovation, but there's still some gotchas. This should help.
At Coconut we were one of the first companies to work with Open Banking when it was launched in 2018, won all four rounds of Nesta’s OpenUp challenge and became one of about 200 registered Account Information Services Providers (AISPs) in the UK.
So, I’ve got some thoughts on Open Banking to say the least. Tom Beckenham (Founder of Comma now VP at Weavr) recently posted about his pet peeves with Open Banking, which got me thinking about mine.
The way the UK has approached the requirements to open up their data and create secure, reliable APIs for Third Party Providers (TPPs) to integrate with is way more joined up than those in other countries.
PSD2 requires EU banking providers to provide API access to TPPs, but doesn’t provide a standard in how to do that. And the US are only just getting round to introducing Open Banking regulation which is much needed - without it I’m sure we’ll continue to see things like Fidelity recently turning off their support for anyone not using their preferred aggregation provider, Akoya.
So - the standard is great. Well done UK. Great work.
But, if you are a TPP or aspiring AISP looking to get into the Open Banking game, I’ve collected a list of gotchas that you might encounter when implementing account aggregation services. If you’re looking to use an aggregation service like Plaid, TrueLayer or Yapily, or even connect directly, then this list should help you navigate some of the lesser known nuances of working with Open Banking data.
Descriptors are non-standard
Here’s an example of the raw descriptors for Starbucks transactions in London:
Amex STARBUCKS LONDON
Barclaycard Starbucks, London
Barclays STARBUCKS COFFEE ON 01 OCT CLP
HSBC STARBUCKS LONDON
Lloyds STARBUCKSMBNA STARBUCKS LONDON GBR
Monzo Starbucks
Nationwide Contactless Payment STARBUCKS
Natwest 3041 01OCT CD STARBUCKS LONDON GB
RBS STARBUCKS LONDON GBR GBR 7779
Santander CARD PAYMENT TO STARBUCKS ON 07-11
Starling STARBUCKS LONDON GBR
Tide STARBUCKS LONDONBanks use some combination of the following:
Merchant Name
Merchant City/Location
Merchant Country
Authorisation Date
Payment Method
This problem usually gets solved at the aggregator level through some kind of transaction enrichment service. Most of the time you’ll get ‘Starbucks’ out.
But making a purchase using Santander at your local, single branch coffee shop? There’s a long tail of merchants that may not be covered by your chosen enrichment method.
In my experience this was pretty Pareto - about 20% of transactions representing 80% of merchants. At Coconut we had to build our own algorithms to cover the long tail, which duplicated the effort that had already been done by our aggregator provider for the more popular merchants.
Dealing with Payment Processors
Payment processors like Curve, SumUp and Zettle appear in transaction descriptions, but they’re not the merchants. Depending on how you approach merchant identification, you need to cater to stripping these out effectively to avoid getting this:
CRV*Trainline CD 3226 > Curve
SumUp *Mobilegear Ltd > SumUp
Zettle_*Black Sheep Co Guildford > ZettleWhen you want this:
CRV*Trainline CD 3226 > Trainline
SumUp *Mobilegear Ltd > Mobilegear Ltd
Zettle_*Black Sheep Co Guildford > Black Sheep CoThis sounds obvious and really depends on whether your merchant identification is done through regex and algorithms or a machine learning model that isn’t as good as a human at picking up on what the merchant name should be when there are a couple mentioned.
I can’t really think of use cases where you want to keep the payment processor prefix, so I think most people will want to strip them out. Here’s a pretty decent list along with some regexes to get you started, use at your own risk:
2Co ^2CO\.COM\*(.+)
Curve ^CRV\*(.+)
EventBrite ^EB\s\*(.+)
GooglePay ^GOOGLE\s+\*(.+)
Kashing ^KASHING\*(.+)
Klarna ^KLARNA\s*\s(.+)
MangoPay ^MGP\*(.+)
PayPal ^PAYPAL\s+\*(.+)
PayPoint ^PPOINT_\*(.+)
Secondary Purchases ^SP\s(.+)
Skrill ^SKR\*(.+)$
Square ^SQ \*(.+)
Sumup ^SUMUP\s+\*(.+)
WePay ^WPY\*(.+)
Zettle ^ZETTLE\_\*(.+)
ZTL ^ZTL\*(.+)You’ll want to figure out ApplePay too, but that one requires a bit more effort.
Timestamps aren’t available on everything
Some banks offer timestamps, some just give dates (ie all transactions take place at 00:00). A knock on impact of this is that you can’t match the transaction ordering in the banking app that the user sees, which can sometimes lead to customer queries:
Provider Timestamps?
Amex No - date only
Barclaycard Yes
Barclays Yes
HSBC No - date only
Lloyds No - date only
MBNA No - date only
Monzo Yes
Nationwide No - date only
Natwest No - date only
RBS No - date only
Revolut Yes
Santander No - date only
Starling Yes
Tide YesThe common theme being that neo banks provide timestamps, high street banks don’t.
This means if you want to try and check running balances in your bank account versus those in the TPP app (eg PFM or Accounting app), you can’t. At least not intra day - you have to check the day’s closing balance from accounting app to bank statement.
Running balances aren’t always available
What’s more is that running balances are available on some providers, not others. But also this is not an exact science which we’ll come to, since the running balance on a transaction can change (yes, that’s right):
Provider Running Balance?
Amex No
Barclaycard Yes
Barclays Yes
HSBC Yes
Lloyds Yes
MBNA Yes
Monzo No
Nationwide Yes
Natwest Yes
RBS Yes
Revolut Yes
Santander Yes
Starling No
Tide NoThe common theme being this time that neo banks don’t provide running balances, high street banks do. Probably because if we used the timestamps to try to confirm the transaction ordering and check the running balance is right, it would probably open a can of worms. So I guess it’s one or the other.
Also given the high street banks have had many more years to get their running balances in order. It’s a harder problem than you might think.
References aren’t always there
Want to apply logic based on references? Well you can’t because some banks don’t provide them. At Coconut we used payment references to automatically match incoming payments to invoices - it meant this feature simply was not available if you used Revolut or Nationwide.
Most banks provide the reference in the descriptor - some put a delimited (eg ‘ref:’) so you can unpick it. Some just stick it at the end, so you can’t figure out what’s descriptor and what’s reference.
Provider Provides Reference?
Barclays Yes - Part of descriptor, no delimiter
HSBC Yes - Part of descriptor, no delimiter
Lloyds Yes - Separate reference field
Monzo Yes - Reference is the descriptor
Nationwide No
Natwest Yes - Part of descriptor, no delimiter
RBS Yes - Part of descriptor, no delimiter
Revolut No
Santander Yes - Part of descriptor with delimiter
Starling Yes - Reference is the descriptor
Tide Yes - Part of descriptor with delimiterAnd if you’re Monzo or Starling, you put the reference in the descriptor.
Monzo/Starling Descriptors are References
For some reason, Monzo and Starling have mapped their ‘references’ to ‘descriptors’ in the API. I think because the ‘reference’ usually contains a description of what the transaction was for (eg birthday money) rather than who it was with (eg Mum). They have a separate ‘counterparty’ field for ‘who it was with’.
I’m sure they have their reasons for doing this. But from a TPP perspective it means these banks behave differently to all the others. Our approach was to flip these back, putting counterparty to descriptor and descriptor to reference.
Starling Spaces != Monzo Pots
While we’re on the subject. You’d expect Starling Spaces and Monzo Pots savings products to work the same, right? They don’t.
Starling Spaces are not separate accounts available through their API, instead their balances get added into the current account balance. This leads to support tickets asking why Starling app balances don’t match what the TPP is showing.
Monzo Pots are linked to their parent current account, but they are separate accounts. Although you cannot separately decide on whether they are imported, as consent is provided at the parent account level.
Most other banking apps simply treat savings accounts as separate accounts: separate balances, transactions and consents.
Transaction types are non-standard
Want to have special logic based on whether a transaction is a card purchase, direct debit or bank transfer? Well, each bank passes through this data differently.
Across all accounts all transactions we’d see a split that looks like this:
Payments In 20%
Payments Out 20%
Card Purchases/Refunds 50%
Direct Debits 8%
Bank Fees 1%
Cash Withdrawals 1%
These all have their own traits. Card Purchases and Direct Debits all have a merchant at the other side of the transaction that will look identical no matter who buys from them. For Payments In, the descriptors are set by the sending bank, so they also look identical across different users.
But for Payments Out the descriptors are set by the user - two users sending to the same account may look different. The introduction of Confirmation of Payee has started to reduce this inconsistency, but it very much depends on the bank’s UX.
There’s nothing to stop me writing weird and wonderful names in my banking app. Especially for my own accounts where I’ll often write the bank name rather than my own name, since having 4 accounts called ‘Adam Goodall’ is not helpful for me. This is widely common practice for people to write ‘Chase’ as the account name on their Chase account when sending from their HSBC account, for example.
Knowing the transaction type is therefore useful for picking up on the nuances surrounding each type. The problem is that, you guessed it, you don’t get this reliably.
Barclays doesn’t provide transaction type in the proper field, but instead appends a transaction type code to the descriptor that you can use to identify the transaction type yourself. We had to write custom code to do this.
Santander puts the transaction type in long form at the front of each descriptor (eg ‘Faster Payment to xxx’), so you have to use those to imply the transaction type, and then strip them from the transaction descriptors to make them readable, particularly on mobile devices.
In an ideal world your aggregator provider has done this for you. If they’ve made Barclays work then that’s a good sign that they understand this problem.
Duplicates happen
And it depends on the bank as to the causes. Each one a different edge case. At Coconut we wrote a deduplication service that could automatically identify duplicates and merge them with some success. But it’s an industry-wide problem.
Transactions can disappear
This happens less. When we operated a current account we would have reversing transactions take place - not refunds, but reversals of previously cleared transactions. These would sometimes come back out as a separate balance-affecting reversal. But sometimes they would simply revert the original transaction. It seemed to depend on the time being the clearing and the reversal.
Mostly when transactions ‘disappeared’, they would reappear under a different transaction ID (see below for transaction IDs changing).
Transaction amounts can change
That’s right, post clearing too. Foreign card transactions change amount from pending to clearing when the rate locks in. But post clearing, amounts can change - typically car rentals, airlines etc where merchants seemingly have superpowers to bend the rules of time and space, applying additional charges by rewriting history on a cleared transaction value.
On Coconut’s current account product, we’d see accounts go into negative balance by cleared transaction amounts changing after the fact. These were different to pre-authorisations. Post-authorisations?
The Mastercard processing rules are a heavy read but there are plenty of mentions of reversals, corrections and errors that can lead to movements in seemingly cleared transactions, which I think is the cause of the behaviour we observed (it was always card transactions).
In Open Banking, this means that you will observe transactions changing amount when you go back and hit the API a day later for the same transaction. I think generally though the window for change is finite and short.
To be honest we didn’t have much trouble with this - we just updated the amounts when we saw them change using a 14 day rolling window, which worked in our product context.
Transaction dates can change
Yep dates can change too. Either through very odd edge cases that almost never happen, or sometimes a change with a banking provider (neo banks) in how they serve the transactions.
Transaction descriptors can change
And of course descriptions can change. It’s rare, it seemed to affect foreign transactions most and probably a function of the clearing process.
Transaction IDs can change
And if all that can change, why not let the IDs change too. Our aggregator introduced a feature offering persistent IDs on transactions to help with this. But it was not supported on all banks. This certainly didn’t help deduplication efforts, since you could not rely on the IDs.
Transaction IDs aren’t unique
And if they can change, let’s make them not unique while we’re at it. HSBC comes to mind with this - the way they present FX fees is to have a separate fee transaction but with same ID. This means any deduplication logic needs to not infer two transactions as being duplicates if they have the same ID. But adding in the amount as well works in the HSBC case.
We’ve come a long way
Let’s not forget, before 2018, millions of banking customers in the UK and globally were handing over their online banking credentials to screen scraping providers like Yodlee and Salt Edge to store in decryptable format in their databases. They would send their robots to login to online banking services every day using those credentials, and download the latest transactions to serve them up through an API to TPPs.
Thank the lord those companies kept those credentials safe. It would have been pretty world ending if they hadn’t. But it was also common for those services to go down whenever the bank updated their web portals. Banks sent letters to customers telling them to stop using TPPs that used this technology, harming competition and consumers in the process.
Through Open Banking we can now manage consent, get secure feeds with decent uptime, and transaction data in a somewhat standard format. And hopefully this article helps you navigate the last mile of working with Open Banking transaction data.
(PS. If anything I’ve written here doesn’t make sense based on your own experience working with Open Banking data, this was my experience using TrueLayer as the aggregator)