How to handle large docs (Archive Docs)

Update! Don’t use this approach. For reasons I haven’t yet figured out, this doesn’t work. My syncs constantly fail, and so do my automations. I don’t know why. I’ll update this thread once I figure out what’s going on



Update! Nevermind, it works! There was a performance issue because of a formula I was using. That is now fixed.

Here’s what changed.

Before, we were detecting whether the sync table was up-to-date with the archive table by looking at all of the rows of the archive table and trying to find one that matches. That’s silly! So many comparisons.

Now, when we archive a row we cache the archived contents in a row of the sync table called Archived Contents. The column that checks if the content is archived now can just look at the previously archived contents to see if it matches the current row’s data. If it’s the same, then the row is up to date. If it’s different then it has changed and needs to be updated, so mark the row as not archived. Duh!

We went from so slow that everything was frozen to blazing fast! Thank you @Shitong_Stone_Shou on the Coda team!

The video needs to be updated to match the new formula, but I’ll have to do that later.


Thank you, Coda team! I don’t know if this is new, but just noticed it and it’s really helpful!

I don’t see this within my docs. Where do you get this JSON option?
Thanks @Connor_McCormick1

Might need to turn developer mode on?

I’m not entirely sure about this :point_up: (I would have said the same thing though :blush: ) but you need some text Coda will be able to interpret as JSON in your column :blush:.
(It seems to work like the Separate as date which is only available in the menu when right-clicking on a Date field AFAIK :blush: )

Ahh good call! Try getting some json and pasting it in the cell

1 Like

Hey those on this thread! (@Christiaan_Huizer , @Pch , @Connor_McCormick1 , @Jean_Pierre_Traets ) - Looking through some different archiving options and wanted to stay away from _merge()

The way Im approaching it now is:

  • Holding x amount of archived rows in a single row in a CSV format
  • Using ParseCSV(), formulaMap(), and addRow() to recover the rows

Its seeming to work really well but just wanted to see if ya’ll had any opinions on limitations? Don’t want to go for this method if Im going to run into obstacles down the road - (In this instance, won’t need the data for queries, charts, etc after its been archived. Just want to store the data and not lose it)

Here’s random example doc just as proof-of-concept:

Edit : The only real limits I’m noticing is cell length limits (The amount of characters you can hold in a single cell)

  • Cell with rich text: ~7,000 character limit
  • Cell with no rich text: ~43,000 character limit

You can store rich text (dates for example) as plain text and they are still recoverable. The run I did for the 43,000 character limit was:

  • 1156 rows
  • 4 columns of data per row

If you had to archive more than 43,000 characters you could easily do it in two runs. . .


that is insanely cool and clever, wow!

Compared to the archiving approach this is incredibly simple. However, it will still grow the size of the source doc and you obviously can’t use the data in the archived rows for computations (e.g. you can’t offer summary charts).

I’ll definitely be using this in the future though. Have you noticed a speedup?

hi @Scott_Collier-Weir , like @Connor_McCormick1 I like the simplicity of the approach. Like you I try to stay away from not supported functions.

I wonder why we should store so much data in a cell, what is the use case you have in mind?

The layout with the grey boxes is clever, I like it!

I noticed you applied the LineBreak(), the function to be released soon :wink:

Hey Christiaan! Yeah - I tried using both character(10) and LineBreak() - there was no difference in terms of the speed or amount of data stored, so I went with LineBreak() as, when it is released, it will be more accessible to the average user (B/c seriously, character(10) is so arbitrary).

Storing data in a cell because I am essentially storing over 1,000 rows worth of data in a single archived row thereby saving room and hopefully increasing speed of a doc by archiving data in a simple way.

I have clients with large docs and plenty of rows and this would allow the doc to scale more easily, work faster, etc

Running some tests, I think we can actually use data in the archived rows for computation:

See below:

I can pull out all dates, all names, etc the archived rows and then run different queries on them. Not yet sure If I can pull charts from them

This pulls out all dates from the archived rows:

I guess you could also combine cell data with main table data pretty easily, too.

Like if you want to answer the question, “how many sales have we made this year?” Is there an idiomatic way to simultaneously use non-archived and archived data?

Definitely easy ways to run queries on archived/non-archived data at the same time:

But you always gotta use listCombine() - Not as natural or intuitive as one would on a table but definitely possible

Yeah that’s a real nice pattern. Great find!

Love the approach! I’d suggest incorporating a .RegexReplace(",","|") step when building the CSV digest column (and a corresponding reversal on the “Recover” button), otherwise any data with commas will break your schema right?


Uhhhhh heck yeah @Steve_Simon2 - Brilliant idea

1 Like

I love your solution @Scott_Collier-Weir and found the same thing @Steve_Simon2 mentioned. I made a copy of your doc and tweaked it to try it out as a list of arrays instead of a straight list. This allows for multiple values in a column too. Pros and cons to each way of doing it, but thought I’d add the slightly different take here if you’re curious.

I’ve also seen docs where people are looking to create their own version of a changelog and I adapted the array of arrays idea to it as well.


Love it @BenLee ! Thanks for sharing and iterating on my approach. I also built a live system for a client recently like this and implemented @Steve_Simon2 's approach. Worked brilliantly!


If anyone is interested - I just walked through this archive approach in a recent youtube video!

I know these techniques are more on the advanced side, but hoping to make the more advanced workflows in Coda a bit more accessible!