API calls occasionally returning very stale data (>1hr)

Running into an issue where an API call (via Zapier) is returning stale data after 80 minutes.

I’m aware that API calls are asynchronous (thread about this), but I’ve never found them to be out by more than about 15 seconds.

Here’s the situation

  • Doc is an extension of an outside CRM. Purpose of doc is to generate and assign serial numbers to Opportunities
  • There’s an “Opportunities” table fed from the outside CRM via Zapier. Currently 1800 of them, each with a serial number
  • There’s a “Next Available Serial Number” table with a single row, with formula [Opportunities Table].[Serial#].Sort().Last() + 1
  • Zapier reads from this table, and uses this serial number for its own records elsewhere, and also inserts a record into the Opportunities table with this number.

In theory, this should all be fine, apart from the following situation:

  1. An Opportunity comes in, gets assigned #2347 and placed in Opportunities table
  2. Another Opportunity comes in really quickly, but #2347 hasn’t been fully written to the database yet, so this new Opportunity also gets assigned #2347

This is rare and I can accept it. Has happened maybe once or twice in the last year.

But then recently (a month?), it’s happening more frequently. Worst was recently with 3 Opportunities: 11:06am, 11:32pm, 12:29pm (these times are confirmed based on Coda’s report of row creation).

I know I’ve got a non-trivial number of rows, but 80 minutes seems like quite the lag time. Any idea what could be going wrong? Or other ideas for how to architect this that would be less susceptible to this issue?

Spoke with support and it seems it was an isolated incident.

Basically the flow is:

  1. API call takes in data
  2. A snapshotting process takes place server-side to for example recalculate any formulas that need it
  3. Updated data is available to subsequent API calls

There was an issue with snapshotting latency on the day I experienced my issue, but generally support confirmed my expectation of sub 20sec latency at 95th percentile (median around 12sec).

1 Like