sgo.to

Challenges consuming APIs at scale

Sam Goto

On my 2014 WS-REST keynote I noted the challenges I faced at Google consuming APIs at scale (think 10K+ APIs). These problems appeared while working on Gmail and Google Search, with an atypical imbalance in API design: few API consumers (~1-2, e.g. gmail or google search) needing to integrate with many producers (say, ~10K).

At that scale, it is easy to dismiss any type of API integration that involves manual work (yes, we've tried). So, the goal was (and still is), to offload as much as possible to API producers, ideally getting as close as possilbe to an automated and self-service platform.

Some of these challenges are still open today, some we have made progress and others I discovered in the last few years.

Each of these deserve their own space, so I'll kick off one post for each in a series.

  1. The discovery of APIs challenge
  2. The API management challenge
    • The quota management challenge
    • The key management challlenge
    • The billing management challenge
  3. The media type fragmentation challenge
  4. The versionless APIs challenge
  5. The fork on the road challenge
    • The user interface challenge
    • The vocabulary fragmentation challenge
    • The verb taxonomy fragmentation challenge