User-driven pagination

At present, there doesn’t seem to be any approachable way to sync with large, paginated collections from remote APIs

When returning a continuation from an execution, the continuation is consumed immediately, ie requests the next page in a loop until there are no more continuations. This will run for as many pages as there are, which can put a server under load if there are many pages.

Is there a way to allow a gradual sync that progressively populates the sync table, based on user actions within the doc? e.g. let’s say for an API with 100 pages, sync the first page of records into the sync table… and offer the user controls to paginate through additional records (incurring a request to the remote source on paging navigation)

There are uses cases where not loading the entire set into the sync table is okay, and being able to load just the first few pages provides value, e.g. distributing the initial pages of records throughout the rest of the doc’s views; such that the user might start working with the synced records before all pages have loaded – and may also opt not to load all pages.

An example might be a Twitter API where we are syncing tweets to a table, based on the search results of a given hashtag. Let’s say the doc users are journalists who want to load tweets into a sync table and then “save” interesting ones with a checkbox. Then another Board view of saved tweets to push them through a workflow. Obviously if it’s a broad term or hashtag, we cannot load all tweets in the table, but the idea is to let the user decide how much to load into the sync table

3 Likes

Hi @mattfysh - Thanks for the feedback and providing a detailed use case. It’s not something I’ve heard discussed before, but I do see how it could be useful in that case.

To date I think sync tables have been focused on pulling in stable datasets infrequently. That model isn’t a great fit though for users that want to interactively search through some data, where you would want to optimize for speed over completeness. We had a previous discussion of search paradigms in Packs here:

I’ll pass your feedback on to the team, but unfortunately I wouldn’t expect and product changes here in the short term.

Another related feature request that came up in discussion recently (don’t remember where):

Right now the flow is always: Coda downloads the entire dataset from the server, compares against what it already has, and updates your sync table with any fresh changes.

It would be nice to be able to sometimes let the API determine what’s new instead, in order to reduce load. Coda could query the API with a timestamp of the last successful sync, and API could send any new/changed records to Coda.

This pattern could reduce what can be a pretty onerous load for large datasets being frequently refreshed by Coda. On the other hand it seems brittle (what if the data become inconsistent with each other for some reason), and you might want some way to periodically do a full-re-sync (or let user request full re-sync?) in case something falls through the cracks.

1 Like