JSON auto-formatting via webhook

1

Hi, I’m going a bit mad here.

I have a Python script that triggers a webhook automation with a JSON payload. The JSON has some nested data in the form of sub-arrays within lists.

When the automation dumps the JSON into a row in my Coda table, Coda does two things:

  1. It automatically formats the data, making it impossible to access and edit; the actual array data is only visible via mouseover.
  2. It strips curly brackets from internal arrays, leaving me with a list of keys instead of properly-nested arrays. This means that I can’t access any of the lower levels of data using ParseJSON().

image
image1402×901 138 KB

In the above screenshot, the first row shows what it looks like when the Python script sends its JSON object directly. The second colum uses ToText() to expose the underlying array; it contains several lists, but in the original JSON output those lists contain arrays.

The second row was delivered via the Python script to the webhook automation as a text string. In this case, Coda did not abstract away the array, but still strips the curly brackets and values from the nested array, leaving me with lists of keys.

The third row is a copy-paste of the JSON object from my IDE terminal. In this case, when not passed through the webhook automation, the cell not only preserves the full array, but automatically formats the input as code.

What is the webhook automation doing that causes it to strip nested arrays from my JSON payload? And why does Coda insist on abstracting away data by auto-formatting it? Most importantly…how do I fix both those problems?

image
image746×477 31.5 KB

This is the mouseover display of the first row. The list of sub-arrays has still been mangled.

2

Hey @Dani_Colman, welcome to the Community!

Could you share the doc please? I think I need the original payload to test this.

In my own experience, and I’m doing this a lot. Simply taking the payload and inserting it into the cell shouldn’t ruin the structure — Coda treats it as an anonymous object without a schema, and it should be possible to get its JSON with the following idiom:

thisRow.Dump._Merge() + ""

What’s happening when you don’t call _Merge() but ToText() directly is probably that Coda ‘unwraps’ the arrays because they aren’t directly corresponding to Coda’s lists. Coda’s lists are wrapped in objects {type:"arr", items: [...]} and that’s probably why you’re seeing them displayed like that, not as lists.

All the Object modification in Coda is a little bit of hidden knowledge. Most of the formulas for it are hidden and it’s not guaranteed that those tricks would work; it’s only us pro users who know these. See if the ._Merge() + "" does the job and please share the sample doc still.

1 Like
3

Same here, I’m processing close to 200,000 webhooks a month in coda docs and logging the full payloads in a cell and then running subsequent actions based on the result.

I’ve never once had a malformed payload?

I normally log the entire payload to a cell (step1Result) and then in a second column, just for visibility, I do columnWithJson.toText(). No hidden formulas needed

Another option would be to parse the incoming payload prior to actually logging it in the automation itself and therefore add the discrete values to different table columns as opposed to logging the whole payload and manipulating it afterwards.

4

Thanks for the replies, @Scott_Collier-Weir and @Paul_Danyliuk. I’m genuinely baffled. I’ve tried all your suggestions (using a smaller nested array for testing) and Coda is still stripping the inner arrays from lists of arrays.

image
image1165×174 8.91 KB

The displayed rows are, respectively:

  1. Using ._Merge()
  2. Using ._Merge().ToText()
  3. Using ._Merge()+""

image
This is a screenshot of the payload as displayed in the rule’s Activity tab. It looks as though Coda is somehow malforming the payload at the point of receipt, since the same stripping is visible. For context, here is the POST request being sent to the webhook, using Python’s requests module, with the array as the payload:

image

I’m wondering if it’s something to do with the way Python itself formats the JSON? But that seems bizarre, given that 1) I’ve used the requests module for integrations more times than I can count, and 2) the sheer number of Python developers in the world.

I am on an enterprise account that unfortunately does not allow doc sharing outside my org. If there’s a workaround, please let me know.

5 
{
    "text": "some text",
    "list": ["item 1", "item 2", "item 3"],
    "list of arrays": [
        {"item 1": "this", "item 2": "that"},
        {"param1": "who", "param2": "what"},
    ],
}

@Paul_Danyliuk, here’s the smaller test payload as code, if you want to test it yourself. I’ve had no issues copy-pasting JSON directly into Coda; it seems to be only when I send it to the doc’s automation webhook that the inner arrays are stripped.

6

Update: resolved by using json={PAYLOAD} in the POST request rather than data={PAYLOAD}. Coda still auto-formats the payload, even when using .Merge(), which I find extremely annoying and would like to resolve, but the critical issue of stripped nested arrays is solved. Thanks!

2 Likes