Fix the timeout for long OpenAI API Requests

I’m a Notion user but I use Coda primarily for AI workflows as it’s 100x more capable than Notion in this area.

Unfortunately, with many of my submissions I’m getting an ‘OpenAI took too long to respond’ error due to the length of the responses I’m seeking. From what I’ve seen this is a standard response for API calls that take more than X seconds too complete. Given that we’re seeing LLMs with context windows of more than 16k (~12,000 words), Coda needs to make an exception for this limit for users to truly make the most of the tools that you’re providing them.

Coda has a huge opportunity steal customers from apps like Notion and ClickUp, and I see empowering users to build complex AI workflows as one way to do it.

Related issue:

1 Like

In my view, the fix is not a longer timeout period, but simply support for streaming responses.

Coda’s AI implementation is relatively closed at the moment, so there’s no way to shape the underlying parameters or use different LLMs as I mention in this article.

The only remedy (today) is to build your own implementation in a Pack or use the OpenAI Pack.

having a drop down menu to select the AI of choice and the related settings like
temperature:
maxOutputTokens:
topP:
topK:

would in certain use cases be beneficial. However seen the coda focus on simplification I am not sure this is what the team has in mind.

There are many cases where Coda handles advanced settings without compromising simplicity for common use cases.

I’m encountering this issue with the OpenAI pack i’m afraid.

I assumed this was the case. Unless you’re ready to use the source of the Pack to build a more custom approach to address your AI requirements, you don’t really have any options.

I overcome these roadblocks through many avenues and I’m fortunate to have some experience building packs and javascript applications. These are the guideposts that provide me with AI development agility to create positive AI outcomes.

  1. Chaining inference operations. Instead of a single, potentially lengthy OpenAI API call, separate the calls to perform more of them with less chances of overrunning run-time limits.
  2. Embedding. Let the faster, cheaper inferencing benefits of vectors do a lot of the heavy lifting in milliseconds instead of relying solely on chat or text completions for everything.
  3. Model choice. OpenAI was first and it is good. But there are many models that can work better and some with vastly larger prompt windows and far better performance and near-zero cost.

Sounds like a viable work around but 99% of people, including myself, won’t go to the effort to get this working. Thanks for your response though!

1 Like

I think it’s 97.5%, so let’s not sensationalize! :wink: I 100% agree that doing AI and doing it well require implementation rigor that’s difficult to bake into a no-code world.

Still, Coda AI needs a 1.1 enhancement.

Same issue. Native AI is not usable for most of my usecases. I need best possible LLM and I am willing to use my own key. Now I need to split requests to 4 steps and it’s still partial solution as sometimes even 2000 tokens result in timeout. It also costs 4 times more.
Any chance to have a fix?

1 Like

Anybody found the solution? Unfortunately sometimes even 512 token promts get timeout (
Is there any way to send request to services like make.com and then push open AI reply into the table?