You trusted a codeless, near-free black-box platform to provide a mission-critical integration. I’m sorry it went sideways, but this is precisely why solution designers need to be skeptical of the many gears and machines they choose to separate processes from data.
At the heart of this problem is a failure to recognize that Coda (like Airtable) provide no event handlers. Because of this, Zapier and Integromat are forced to poll for data changes just as you would be forced had you created a custom integration with the Coda API. This is why it failed. And while it’s fair to lay [some] blame at Coda’s feet, we all must accept at least some of the responsibility. This limitation is not related to Coda’s performance or their API per-se - it’s caused because we all conspire to oversaturate the Coda infrastructure with millions of unnecessary requests every hour - perhaps every minute.
We’re high on glue - the Zapier and Integromat glue. We think it’s wonderful and yet, it is ultimately harmful without webhooks and events that drive them.
Avoiding this (for us) is not easy and adhesives the likes of Zapier only exacerbate the problem by offering vast integration options without cost or near-free and easy to setup. This serves to endear less-knowledgeable users to elevate the polling insanity because they know not what’s happening under the covers.
If the Internet depended on polling architectures, we would have shut it down in 2005. But we got smart and we realized that events are critical to interoperability success patterns. Why Coda chose to ship before events and webhooks were possible is not something I can easily rationalize.
My client solutions do not suffer from this calamity because (a) I asked deep questions about the architecture before getting clients on to it, and (b) for any processes that require some degree of event handling, I either optimize the crap out of the polling detection scheme and set clear latency expectations, or I use Slack events to mimic actual events. I also avoid polling at all costs and push manually-driven event triggers on to actual users until such time as events and webhooks may become available.
Where does that leave us?
Well, we can’t blame Zapier - they want events and webhooks as much as we do. We can certainly blame the glue-factories for referring to these integrations with Coda as “webhooks”. At best it’s a deeply harmful mischaracterization of their service.
Coda can change the API performance game with two key architectural enhancements that work the way almost every SaaS solution presently achieves optimized interoperability. I don’t think they have a choice. In the meantime, there be thin ice - best to skate carefully my friend.
[UPDATE]: BTW, a couple of key things are circling this drain.
-
Someone asked about the API accessing data via views during the Maker webinar and @shishir mentioned that he thought it was already possible but not likely in the SDKs. On this point alone it would allow us to discover changed records easily and avoid massive table-level polling to find data patterns. I will be spending a fair bit of time on this in the coming weeks because this is a significant advantage lacking webhooks and events.
-
The glue-factories do not pace or filter their API calls. In many of my integrations, I pace the change detection and processing tasks by calling a small number of API requests over many hours. This allows me to perform API tasks across many thousands of records without triggering alarms. But it does require some carefully scripted machinery to segment the activities to the record set.