Why is OAuth still hard in 2023?
We implemented OAuth for the 50 most popular APIs. TL;DR: It is still a mess.
OAuth is a standard protocol. Right? And there are client libraries for OAuth 2.0 available in basically every programming language you can imagine.
You could conclude that, armed with a client library, you should be able to implement OAuth for any API in about 10 minutes. Or at least in an hour.
If you manage, please email us: We would like to invite you for a really delicious dinner and hear how you did it.
OAuth in practice: We implemented OAuth for the 50 most popular APIs
These include Google (Gmail, Calendar, Sheets etc.), HubSpot, Shopify, Salesforce, Stripe, Jira, Slack, Microsoft (Azure, Outlook, OneDrive), LinkedIn, Facebook and many more.
If it wasn’t so annoying, it would be quite funny. Let’s dive in!
Problem 1: The OAuth standard is just too big & complex
“This API also uses OAuth 2.0, we already did that a few weeks ago. I should be done by tomorrow.”
– Famous last words from the intern
OAuth is a very big standard. The official site of OAuth 2 currently lists 17 different RFCs (documents defining a standard) that together define how OAuth 2 works. They cover everything from the OAuth framework and Bearer tokens to Threat models and private key JWTs.
“But”, I hear you say, “surely not all of these RFCs are relevant for a simple third party access token authorization with an API?”
You are right. Let’s focus only on the things that are likely to be relevant for the typical API third-party-access use case:
- OAuth standard: OAuth 2.0 is the default now, but OAuth 1.0a is still used by some (and 2.1 is around the corner). Once you know which one your API uses, move on to:
- Grant type: Do you need `authorization_code`, `client_credentials` or `device_code`? What do they do and when should you use which? If in doubt, try `authorization_code`:
- Side note, refresh tokens are also a grant type, but kind of a special one. How they work is standardized, but not how you ask for them in the first place. More on that later.
- Now that you are ready for your requests, let’s look at the many (72, to be precise) official OAuth parameters with a defined meaning and behaviour. Common examples are `prompt`, `scope`, `audience`, `resource`, `assertion` and `login_hint`. From our experience though most API providers seem to be as oblivious of this list as you probably were until now, so don't worry too much about it.
If this still feels too complicated and like a lot to learn, we tend to agree.
It seems most teams building public APIs agree as well. Instead of implementing a full OAuth 2.0 subset they just implement the parts of OAuth they think they need for their API’s use case. This leads to pretty long pages on docs outlining how OAuth works for this particular API. But we have hard time blaming them, they have only the best intentions in mind for their DX. And if they truly tried to implement the full standard you would need to read a small book!
The trouble is that everybody has a slightly different idea of which subset of OAuth is relevant for them, so you end up with lots of different (sub) implementations.
Problem 2: Everybody’s OAuth is different in subtle ways
As every API implements a different subset of OAuth, you quickly get into a situation where you are forced to read their long pages of OAuth docs in detail:
We could go on for a long time, but we think you probably got the point by now.
Problem 3: Many APIs add non-standard extensions to OAuth
Even though the OAuth standard is vast, many APIs still seem to find gaps in it for features they need. A common issue we see is that you need some data besides the `access_token` to work with the API. Wouldn’t it be neat if this additional data could be returned to you together with the `access_token` in the OAuth flow?
We think this is actually a good idea. Or at least better than forcing users to do quirky additional API requests afterwards to fetch this information (looking at you, Jira). But it does mean more non-standard behaviour that you specifically need to implement for every API.
Here is a small list of non-standard extensions we have seen:
- Quickbooks has a concept of a `realmID` which you need to pass in with every API request. The only time they tell you this `realmID` is as an additional parameter in the OAuth callback. Better store it somewhere safe!
- Braintree does the same with a `companyID`
- Salesforce uses different API base URLs for different customers, they call this the `instance_url`. Thankfully they return the `instance_url` of the user together with the access token in the token response, but you do need to parse it out from there and store it.
- Unfortunately Salesforce also does even more annoying things: Access tokens expire after a pre-set period of time, which can be customized by the user. Fine so far, but for some reason they don’t tell you in the token response when the access token you just received will expire (everybody else does this). Instead you need to query an additional token details endpoint to get the (current) expiration date of the token. Why, Salesforce, why?
- Slack has two different types of scopes: Scopes you hold as a Slack bot and scopes that allow you to take action on behalf of the user who authorized your app. Smart, but instead of just adding different scopes for each they implemented a separate `user_scopes` parameter you need to pass in the authorization call. Better know about that and good luck finding support for this in your OAuth library.
For the sake of brevity and simplicity we are skipping the many not-really-standard OAuth flows we have encountered.
Problem 4: “invalid_request” - debugging OAuth flows is hard
Debugging distributed systems is always hard. It is harder when the service you are working with uses broad, generic error messages.
OAuth2 has standardized error messages, but they are about as useful in telling you what is going on as the example in the title above (which, btw, is one of the recommended error message from the OAuth standard).
You could argue that OAuth is a standard and there are docs for every API, what is there to debug?
A lot. I cannot tell you how often the docs are wrong. Or missing a detail. Or have not been updated for the latest change. Or you missed something when you first looked at them. A good 80% of OAuth flows we implement have some problem upon first implementation and require debugging.
Some flows also break for, what seems to be, random reasons: LinkedIn OAuth for instance breaks if you pass in PKCE parameters. The error you get? “client error - invalid OAuth request”. That is… telling? It took us an hour to understand that passing in (optional, usually disregarded) PKCE parameters is what breaks the flow.
Another common mistake is sending scopes that don’t match the ones you pre-registered with the app (pre-register scopes? Yes, a lot of APIs these days demand that). This often results in a generic error message about there being an issue with scopes. Duh.
Problem 5: Cumbersome approvals to build on top of APIs
The truth is, if you build towards some other system by using their API you are probably in the weaker position. Your customers are asking for the integration because they are already using the other system. Now you need to make them happy.
To be fair, many APIs are liberal and provide easy self-service signup flows for developers to register their apps and start using OAuth. But some of the most popular APIs out there require reviews before your app becomes public and can be used by any of their users. Again, to be fair, most review processes are sane and can be completed in a few days. They are probably a net gain in terms of security and quality for end users.
But some notorious examples can take months to complete or even require you to enter into revenue share agreements:
- Google requires a “security review” if you want to access scopes with more sensitive user data, such as email contents. We have heard these can take days or weeks to pass and require a non-trivial amount of work on your side.
- Looking to integrate with Rippling? Get ready for their 30+ questions security pre-production screening. We hear access takes months (if you are approved).
- HubSpot, Notion, Atlassian, Shopify and pretty much everybody else who has an integrations marketplace/app store requires a review to get listed there. Some reviews are mild and some are asking you for demo logins, video walkthroughs, blog posts (yes!) and more. However, listing on the marketplace/store is often optional.
- Ramp, Brex, Twitter and a good number of others don’t have a self-service signup flow for developers and require you to fill in forms for manual access. Many are quick to process requests, but for some we are still waiting to hear back from after weeks…
- Xero is a particularly drastic example in how they monetize their API: If you want to remove a limit of 25 connected accounts you have to become a Xero partner and list your app in their app store. They will then take an (as of time of writing) 15% revenue cut from every lead generated from that store.
Problem 6: OAuth security is hard and a moving target
As attacks have been uncovered, and the available web technologies have evolved, the OAuth standard has changed as well. If you are looking to implement the current security best practices, the OAuth working group has rather lengthy guide for you. And if you are working with an API that is still using OAuth 1.0a today you realize that backwards compatibility is a never ending struggle.
Luckily security is getting better with every iteration, but it often comes at the cost of more work for developers. The upcoming OAuth 2.1 standard will make some current best-practices mandatory and includes mandatory PKCE (today only a handful of APIs require this) and additional restrictions for refresh tokens.
The biggest change has probably been ushered in with expiring access tokens and the rise of refresh tokens. On the surface the process seems simple: Whenever an access token expires, refresh it with the refresh token and store the new access token and refresh token.
In reality, when we implemented this we had to consider:
- Race conditions: How can we make sure no other requests run whilst we refresh the current access token?
- Some APIs also expire the refresh token if you don’t use it for X days (or the user has revoked the access). Expect some refreshes to fail.
- Some APIs issue you a new refresh token with every refresh request…
- …but some also silently assume that you will keep the old refresh token and keep on using it.
- Some APIs will tell you the access token expiration time in absolute values. Others only in relative “seconds from now”. And some, like Salesforce, don’t divulge this kind of information easily.
Last but not least: Some things we have not talked about yet
Sadly, we have only just scratched the surface of your OAuth implementation. Now that your OAuth flow runs and you get access tokens it is time to think about:
- How to securely store these access tokens and refresh tokens. They are like passwords to your user’s accounts. But hashing is not an option, you need secure, reversible encryption.
- Checking that the granted scopes match the requested scopes (some APIs allow users to change the scopes they grant in the authorize flow)
- Avoiding race conditions when refreshing tokens
- Detecting access tokens revoked by the user on the provider side
- Letting users know that access tokens have expired, so they can re-authorize your app if needed
- How to revoke access tokens you no longer need (or the user has requested you delete them under GDPR)
- Changes in available OAuth scopes, provider bugs, missing documentation etc.
A better way?
If you have read this far you might be thinking “There must be a better way!”
We think there is, which is why we are building Nango: An open-source, self-contained service that comes with pre-built OAuth flows, secure token storage and automatic token refreshes for 50+ APIs.
If you give it a try we would love to hear your feedback. And if you want to share your worst OAuth horror story with us we would love to hear about it on the Slack community.
Thanks for reading and happy authorizing!