Suggestion: allow sync tables with long running executions to recover from deadline timeout with a backpressure signal

The idea is to have the execution function of sync table be retried upon timing out, and receive an additional param to indicate that it timed out, so that it can reduce its execution time and recover.

The motivation is that in order to avoid timing out, sync table executions that take a long time should be authored to keep the execution time well below the deadline to avoid outlier cases. In some cases the execution time depends both on pack-controlled variables such as page size, but also on the nature of the user-driven data, which forces even more conservative bounding of the controlled variables. This can make the pack execution itself slower (more continuations & more network requests).

By adding a second chance when timing out, it is usually trivial to reduce the size of the workload for the execution (most packs would have the same levers in the form of page size, etc.), and allows for packs to better optimize the number of network requests made to speed up the time taken.

In practice this may not be feasible, as it would mean that for problematic packs which consistently go overtime, it would be adding an entire other 60s of compute time to each execution.

Thanks for the suggestion @loucadufault. Are you running into this scenario in one of your Packs?

Have you considered or tried to implement code that would track the length of the execution so far and use that to make decisions about whether to continue with a loop or return early? I’m not sure if that would work in all scenarios, but it could be an option in some.

Have not run into the issue. That’s one option but in my case since the processing is a multi-step pipeline reducing the input size later in the process means discarding previous work and would be tricky to get working right, and would require estimates of the timings of each step.

Setting the page size upfront (for example, halving it) would be a much simpler option, though it’s not an issue for me, just a suggestion based on a hypothetical.