Feature request based on Roam Research

This made me think and come up with this:

UPD: And ultimately this:

2 Likes

Dear @Paul_Danyliuk,

First of all, all the best for 2020

You did it again, amazing to see how you managed to get this mindmap up and running. :trophy: :trophy: :trophy:

Thank you very much sharing your skills and your enormous source of know how :pray:

2 Likes

Hey guys, just wanted to add myself to this thread, sorry if this is considered over-proliferation.

And just to yet again illustrate how these links could look at the forum software Discourse will show off, here is my overly-long post mirroring what you are saying here - hopefully all this chatter will be useful to the Coda Product Team:

And @tomavatars I just started experimenting with Roam, a few thoughts on specifically what I like that could get into Coda (this is not duplicated from the other post!)

  • the use of the “/” and “#” really make creating and relating super quick when using the keyboard exclusively. Coda already has dynamite keyboard shortcuts, such as tabbing around rows and tables, selecting multiple items in a drop-down, etc. There’s some additional stuff that could result from this if you allowed, say, creation of a row and assignment to a table all with the keyboard, while in another row.

  • With Roam, I find myself really yearning for the depth of Coda - when I create a linked entity, I could just “feel” how this could look in Coda if I was creating a row with a little context, like a row for a meeting and quickly adding in a time for it. If you could add a row to a table in one operation - from another row - you could then pop open the window and complete fields that are in the destination table’s configuation

  • Really would like to put my heavy vote in for some kind of live diagram that would generate of Coda’s links, whether this be full on Mind-Mapping, or some sort of iteration of the existing Graph. I agree with @Jean_Pierre_Traets that seeing how Roam did this very simply right now is an interesting take on even a basic version we could see in Coda.

Hope that’s helpful!

2 Likes

Hey @tomavatars one more thing here:

I noticed that in Roam when you have a relationship set up, it will kind of “open” the related item within the “originating” item and let you add to it right from there. This all gets picked up in the “destination” item, what’s more the context of the content coming from the “originating” item is maintained, very cool!

Here is a use of this I could see that is in fact I have been trying to accomplish in Coda:

If you are doing a Product Planning meeting and you want to establish that you are creating a product in the first place and some ideas come up in that meeting. You can record all this at the time on the new “Product Overview” page which you actually create for the first time in the meeting. This would be a row in Coda. In Roam, you’d bullet out the ideas. In Coda, if you have a subtable, you can add in the related rows. Then, later on, the doc can be expanded, but these added ideas are not in the meeting, so they don’t have the reference. That is something my team has really thought would be useful as you can the later see the ideas from the meeting and possibly better recall why they came about when deciding how to prioritize them.

I have in fact been trying to create a way to keep a “mutual” reference between two types of rows like this (Meeting and Idea in this example), and the way I was trying to accomplish this was through a “join table” as @Jean_Pierre_Traets you are describing here.

This is in fact a good solution, as I then plan to have subtables giving the full context I’m after in each of the joined tables. But setting this up has also been one of my largest challenges in Coda, especially as in each of my tables I’m using some hierarchy within the table: I have a “Project” table that has a few levels: “Project” at the highest, and also “Phase” which is a child of “Project.” So between the three tables, and hierarchies within the tables, it has been hard for me to get this working without a lot of work and help from the Coda team!

This may involve some complexities I’m missing here, but one thing that occurred to me that would make this simpler would be to allow “instant” relating via the “hovering data” that Coda provides when you mouse over a related row, what’s shown here:

If, when that data is displayed, you could actually jump in and add to it, and the row that you are in when you are adding this data was referenced, you would basically have a duplicate functionality as what Roam is offering, without the need to have a third “Join Table” to accomplish this.

As always hope I’m making sense and this is useful!

1 Like

Yep - the unmistakable mark of a graph database.

I believe this relationship is made possible without joins in Roam because the underlying data architecture is a graph database which ironically has nothing to do with charting and little to do with relational database management systems. I suspect it is likely Neo4J under the covers, but others that ring a bell include Titan, OrientDB or SPARQL.

Graph databases - the likes of which are used to maintain the social relationships among Facebook’s 1 billion users - do not use joins. In fact, they are designed to specifically eliminate the need for joins.

Attempting to mimic the properties of a graph database in non-graph database environment is very difficult. Sure, you can implement some functionalities, but there will be some difficult challenges as you think through all of the data model requirements. And whatever you attempt to mimic will likely struggle to achieve expected performance criteria - a sure pathway to customer unhappiness.

I think it might be better for Coda to consider a series of Packs that support different data models. Ideally, this would allow users to alternatively select Neo4J, or other exotic data management platforms to serve as the underlying data storage model to Coda or perhaps a single table in Coda. This is precisely the integration partnership that ElasticSearch has forged with Neo4J to support graph data models.

1 Like

Hey @Bill_French, as expert as ever! Very cool to get some explanation of how this all works under the hood, I am learning all the time!

I am also very interested in Reciprocal Referencing across Coda, have talked about it here:

here

here

and here!

Would be curious to get your thoughts on this. I’d hoped it would be a simpler technical implementation for Coda than what sounds like a challenge to bring in some of the way Roam works when you type and generate deep references automatically. In essence, all I’d like to see is the referenced entity in Coda out of @mention have basically a notation about that reference somewhere in its canvas. Krunal from Coda once pointed out to me that you can even see all references as you search for a row. I am petitioning anywhere I can for Coda to introduce this feature, I really need it!

And I really love the way the app hosting the community, Discourse, does this! In fact I just created 4 new ones right now, haha!

PS I have some experience with Elastic as in my day job my company handles classified ads listings and we use Elastic both as a db, and as our search service.

Cheers Bill!

I think I upvoted a few of your ideas concerning reciprocal link references.

Yep, I get it, but conveying this idea effectively probably needs more than words; as you point out, it’s not easy to explain the deeper nuances of these feature requests. Have you considered a simple diagram to make this more evident?

Unfortunately - and especially the case in support forums - readers have just a few minutes to digest ideas. Complex explanations sometimes get passed over.

Hey Bill, sorry I wasn’t aware of that, and appreciate the support. My posting around here in the community needs Coda to keep it better organized, haha!

Yes I am continuing to think about how to present the concepts better, diagram a good idea! In fact, after months on my own with Coda experimenting, while looking at a range of other apps to solve my team’s needs, I am about to make the commitment to go forward “all in” for better or worse! When I do this I’m going to bring in one of my colleagues on my team - an engineer with good DB background. Hoping he can contribute to presenting our case more coherently than me!

Take care.

@tomavatars, just some more brainstorming here as I’ve been using Roam quite a bit.

First, a point about the mind mapping and diagramming. Like @Jean_Pierre_Traets I am in fact doing some outlining of my Coda Schema in Roam, which I’ve been finalizing for several months! I’ve really missed the ability to see more visually in planning my schema across dimensions. Things like I have my Software Platform (my team manages websites), built in typical layers (Front End, Mid-Tier, etc.). Then, my product components reside within these layers… And relate to partners I recruit to use my system… With whom my team interacts via communication tasks…

And All this I’m trying to track in Coda, and make sure I get the relationships right. It’s been a lot of trial and error, but Roam is helping me build this schema first, thanks to the multi-dimension relationships, without me having to build it in Coda, use it, see something wrong, change it, and rebuild.

And this is to say that a mind map or network diagram of my Coda set-up would go a long way towards solving this. So I see more and more benefit from this missing feature in my current work in Coda, almost daily:

Mind map plug aside, I also wanted to say that Roam is warming me up to the potential of “blocks” of text in Coda that could be “promoted.” By promoted, I mean converted from text into more database-type records with more depth, such as status, due date, assignee, etc.

I hadn’t been that keen on blocks, which seem to have made their presence felt in Notion, but also exist in an increasing number of apps like Slite, Fibery, and so on. But the way Roam uses blocks is very intriguing from a creative point of view:

If I am understanding correctly, every “block” you write - and basically every bullet point in a paragraph becomes a “block” - has some underlying attributes that allow it to be referenced throughout the app. So these “blocks” function like mini-data records. It occurred to me that you could create similar blocks across Coda, in some places where I happen to use a lot of expanded text, which are:

  • In Canvas
  • In Rich Text description fields
  • In Comments

Then, the idea is you could possibly “convert” them, via right click menu or ellipses, to a row in a table. All the while leaving this block where it is in its “home” in the canvas or Rich Text description or Comment. And creating a reference to the destination row. That destination row can wind up being a project, an OKR, idea to create a company goal, you name it. This would give another dimension of creativity as it would encourage write-up of ideas and similar content without hesitation across Coda, with no fear among more novice users around how to convert this into Coda’s row structure. And if these quick keyboard shortcuts are implemented that Tom talks about, there is great free-flow potential for more creative and less technical types who otherwise might look at Notion or Milanote exclusively. With this level of flow available in Coda, there would be much less intimidation between writing, and integrating this content into actionable rows in Coda.

I really don’t think it would be a stretch for Coda to implement some of this stuff. Would be a great complement to the core structure and power we have. And probably a great tool to encourage more Makers get into Coda. It is the year of the Maker after all!

Sorry to all in advance for the following lengthy comment. If I had more time this would be far shorter.

This is already possible and I’ve done some of this in varying engagements. It’s not easy, and it’s non-trivial for the reasons I mentioned in my question to @shishir during the recent webinar.

Without formulaic access or API support for text objects, both users and developers are somewhat ham-strung to advance the science of utilising text passages as “data”.

This is crucial because – as you make clear – there’s a vast underserved climate of information in text and the need for non-linear linking relationships. From entity extraction, to magical use of NLP and AI-based compositions, text is at the heart of just about everything humans do in relation to information technology.

During the webinar, @shishir was very direct - the underlying architecture supports what we (and many others have expressed in our desires concerning text accessibility and possible transformations). I look forward to the day when a simple Coda formula is able to tokenize a section or a paragraph or a sentence and allow me to exploit the content in new and innovative ways.

But there’s another aspect of all feature asks that troubles me… feature-requesting a product to death.

We’ve seen what happens when a team is pressured into adding the kitchen sink. Microsoft Word presently has 9,258 features. I know this because I was on a team that assessed the viability of making Word fully accessible under the WCAG standards, etc. It gets very ugly when [seemingly] simple writing tools are laden with plumbing fixtures.:wink: And there’s no debate - I’m part of the problem for sure!

In contrast, (and recognizing my own feature addictions) I try from time-to-time to advocate a slightly different approach - encourage teams to build features that make more features possible. Here’s a wild and crazy example -

What if … Coda could embed Python scripts much the same way Observable embeds javascript in web documents? Or the way Python can be used to test ideas in Jupyter notebooks? This is what might magically emerge -

  • Panda Dataframes would allow us to seamless interchange Coda tables and transform them in ways that negate many of the feature requests that exist in this forum.

  • We’ve all asked for data-related transformations, blending, sorting, interleaving, etc. Why not implement a pathway to every imaginable data management challenge?

  • We could build complex data models with speed and nearly without effort. Data processing would become almost trivial.

  • Data visualizations would no longer be bound to the limits of Coda’s visualization features. (the app shown below is 18 lines of Python)

  • Important (to @ABp), a mind map is a simple interpretive process that has been fully baked into many Python libraries. In just a few lines of code, relationship mapping becomes a simple approach that can be applied to arbitrary data sets; texts, lists, arrays, tables, views - with Python it all becomes easily mappable and rendered as we desire.

  • Don’t know Python? Embelish the Python plugin feature with the ability to share “recipies” and reuse snippets. Enable pack developers to openly build and share Python libraries and code generators. Now you have an attraction to those who know and those who need.

  • Big win - Coda quietly slides into the data science arena; a broader positioning is now possible.


The above 3D dynamic visualization built with 8 lines of Python code (another 10 lines to collect the data).

    dfRevenueMap['lat'] = pd.to_numeric(dfRevenueMap['lat'],errors='coerce')
    dfRevenueMap['lon'] = pd.to_numeric(dfRevenueMap['lon'],errors='coerce')

    # set up the inp selector
    fdList = sorted(dfRevenueMap['FD Name'].unique(), reverse=False)
    selectedFDs = st.multiselect(
        'Select Fire Department Name(s)',
        fdList
    )

    if (selectedFDs):
        dfFilteredRevenueMap = dfRevenueMap[dfRevenueMap['FD Name'].isin(selectedFDs)] 
    else:
        dfFilteredRevenueMap = dfRevenueMap

    # get the geo-midpoint of the data
    midPoint = (np.average(dfFilteredRevenueMap['lat']), np.average(dfFilteredRevenueMap['lon']))

    # mapboxgl chart
    st.deck_gl_chart(
        viewport = {
            'latitude': midPoint[0],
            'longitude': midPoint[1],
            'zoom': 11,
            'pitch': 50,
        },
        layers = [
            {
                'type': 'HexagonLayer',
                'data': dfFilteredRevenueMap,
                'radius': 200,
                'elevationScale': 4,
                'elevationRange': [0, 1000],
                'pickable': True,
                'extruded': True,
            },
            {
                'type': 'ScatterplotLayer',
                'data': dfFilteredRevenueMap,
            }
        ]
    )

If this is the Year of the Maker, then 2021 must be the year of Users Buying What Makers Made in 2020.

1 Like

Bill, as impressive as ever. I wouldn’t mind engaging you directly to solve my team’s software development issues, as well as our Coda set-up needs, haha!

With things like reciprocal references showing up around Coda, more manipulation of text into rows, visualizing these relationships, and the potential of the text around Coda being more readily “convert-able” to rows and data, I think you have about 90% of my desired features covered! I’m not sure if you would include those in feature bloat, but I’m hopeful most of what I am looking for is a matter of just surfacing stuff in the existing Coda architecture. I am no developer, but we use Python extensively on some of our platform (as well as Elastic Search we discussed earlier), and this all sounds like a dynamite implementation of it!

This is primitive compared to what you just wrote, but much of my desire for the reciprocal references and easy manipulation of text has to do with somewhat successful use of Confluence/Jira for a long time. As you may know, you can easily manipulate text in Confluence into Jira issues, and references are picked up all across the two apps. It was the one probably redeeming feature of an otherwise failed experience with my team and Atlassian. Also, although I just said “easy,” that is with caveats: The process in Confluence to link with Jira was simple to understand, but it took many clicks, highlighting with the mouse, and generally those two apps were S-L-O-W. So as I work in Coda, I long more and more for being able to see these wonderfully quick relations I create in Coda show up in the destination I point them to! And you can really get a sense of how well that can help with usage of a tool by seeing how Roam is manipulating this concept.

And apologies for another long post, but I thought I’d also shed some light on my own use of MindMaps /visualization. Again, hope this is helpful as I know the Coda Team likes use cases!

We wanted to create a “Feature Map” of our Software Platform. The idea was to show the relationships between components in the various tiers of the system. So, a webpage where you sign up and submit your resume (one of our sites is a job site) would have a link to the db in the Infrastructure level where that resume data does in. If there is a parsing process, the link would also incorporate the mid-tier parsing service we built. In the end, the opportunity is to wind up with a “live” diagram of our system, that will update as we adjust links in Jira, a we were still using it at the time. This eliminates the need for the likes of Lucidchart or Miro, which typically are used by a dev team building a static diagram of a Dev Platform. Then, they build completely separately in the repositories, and have a nightmare to keep the diagrams updated.

After @Paul_Danyliuk’s SVG example here…

and now reading your thoughts, makes me even more aware about how this should not be that hard to accomplish. And, I think if Coda came out with this natively, they would have a potential disruptor in the Diagramming Software space. Unlike the market leaders I just mentioned, Coda actually is a place teams will maintain the structures that they represent in external tools in Lucid and Miro.

And I am in favor of paying for doc making through Coda as soon as you guys figure it out, too!

Take care.

1 Like

Well I wouldn’t if they expose these features in a way that makes it possible to craft these features (and many more), not necessarily provide exactly those features. Rather, build for a future of possibilities.

I think we have written the start for a book. :wink:

1 Like

And I need to get back to my day job, haha.

Really hoping the Product Team shares your vision Bill. Will be eagerly looking out for your next words of wisdom, all the best!

1 Like

If Coda’s future (or anything for that matter) is dependent on my wisdom, we’re all in deep trouble.

3 Likes

I do not agree that it is not easy. I suspect this is sacrificing the 95% usecase for the 100% and stating that the 100% is unattainable. I suggest you try out the workflow as implemented in Fibery.io.

It is a simple 95% implementation, but it works because it keeps the user in the main text editing context after finishing the creation of the record - which apparently is the main issue. I don’t mind copying a text within context, I do mind having to find a table again, switch context in multiple layers and then return to the original context.

2 Likes

GJ,

I think I failed to make myself clear in the context of the comment you are referring. Let’s revisit the origin of my comment -

Asserting that (a) “this” is possible and (b) “not easy” refers specifically to doing this Coda, not Roam, and certainly not Fibery. It references @ABp’s interest in “blocks” of text that could be promoted - i.e., converted from text objects to database-like records.

As you probably know, this is not “easy” to do in Coda; most believe it is not possible, but I have done it with some clever server-side parsing and the Gmail pack.

I fully understand the capability in Fibery and if you are asserting this is easy in Fibery, that’s wonderful. My comments were about Coda specifically and in today’s current version.

As you guys know, I am finding the Fibery UX easier to navigate so far than Coda, and in fact Notion. I am getting a lot of success with the ease of creating relations. A good example of what causes me roadblocks in Coda is this feature:

@GJ_Roelofs we were both involved here. So when I use this, I was expecting simply for this to be a full bi-directional reference, that Ben is advertising can be done in “only two clicks.” Thing is, this solution is only one-directional, so not really “just two clicks” if you want the full reciprocal capabilities. With Fibery, as well as Notion in this case but with a bit more work, you set up the relation and it’s instantly visible in the receiving table. What’s more, Fibery has a notion of one-to-many vs. many-to-many. This is a little touch that allows much quicker creation of these relationships, something that requires more though to set up in Coda, and in a UI/UX arrangement that I can easily grasp. I cannot say the same about most of what I try to create in Coda.

I am really hoping we can get more of this user-friendly feature build in Coda. I don’t think it would in the least harm the underlying power, but could really make it more attainable for those “lesser” Makers such as myself. Sadly, I am getting ready to give up my efforts to build my “all-in-one” solution, for the time being, in Coda. I hope to be back though, as I’ve given up and returned several times in the last year!

Cheers guys.

I agree - this is a very powerful feature and one that is critical for many applications and even some Airtable users have asked for this. Bi-directional linking is really important because complex relationships between information objects are best able to expose deep(er) knowledge and …

… what is Coda for if not for creating amplified capacity to convey knowledge, right? :wink:

The idea of bi-directional linking dates back to the early days of the HTTP protocol and some of the earliest discussions occurred at Xerox/PARC and of course, at W3 in 1990.

Do not be fooled into thinking this concept has near-zero impact on performance. Indeed, while the feature is simple to conceptualize, the architecture required to pull it off without lots of added JOINs (in a traditional RDBMS), is non-trivial UNLESS you have the underlying power of a graph database OR a way to mimic a graph database architecture the way Fibery has [likely] achieved with this underlying technology.

Vastly More JOINs

The challenge with implementing this feature is best described using a common Facebook occurrence; the popularity of likes accumulating into a super-node. Since an unpopular target record (or document) has very few inbound and reciprocal outbound links, traditional relational databases can handle this without serious performance issues. However, when you create enough links that elevate a record (or document) to a high degree of relative popularity, a deep set of JOINs must exist. Example -

If we had a social graph of :People who :LIKE musical artists, it’s very likely that some of those artists, perhaps many, are supernodes. If we were looking at (:Person)-[:LIKES]→(:Artist {name:'Britney Spears'}) , we could reasonably assume that a :Person likely has relatively few :LIKES relationships, perhaps less than 100, but that a popular :Artist like Britney Spears may have millions of :LIKES relationships pointing her way.

Implementing this in an RDBMS would be catastrophic and this actually happened at Facebook in 2009.

Ideally, Coda would simply give us the ability to utilize a graph database for applications that require bi-directional linking. Alternatively they could throttle such a feature with a limited number of popular bidirections (I think SQL typically feels a good deal of stress after about 6 to 10 JOINs).

Yes - bi-directional linking has huge benefits, but it comes at a price.

4 Likes

@Bill_French, it’s been truly an education reading your posts on this subject.

Quick question: How much load would something like simply showing a reciprocal link in a “destination” row place on Coda? I’m talking about the exact same functionality we have here in this tool the community uses, Discourse.

So specifically, I use the @mention across Coda and reference another entity - section, or row. In that destination entity, there is a part of its details view that shows a link back that “mentioning” row. Possibly with some additional context besides a raw “link” only, or a toggle to open a bit more detail as Roam does, and Discourse also does:

This is one simple thing I recall from Jira, and some other tools like Aha!, that I think would be huge for my use of Coda. Without the need to build out more complex relations among tables, such as the ability for many-to-many table relations.

Cheers!

Sure, and Jira tickets represent a very different set of relationships than the possibilities we can create with Coda. There is no concept of a super-node in Jira - even the most popular tickets are unlikely to generate more than a few dozen inbound links even in a large team. And, can we assume Jira doesn’t use something like Neo4J to provide this functionality given Jira provides a Neo4J Plugin?

Well, that’s the point of my comment - it depends on popularity (i.e., the number of inbound links that emerge over time to a relatively popular destination). And really, this question is not something I can begin to comment on with any degree of certainty because I have no clue how Coda is designed. Coda should fear giving me any insight under the covers because my rants would be far more lengthy. :wink:

If it’s just a few bi-directional links, it’s pretty simple to track and manage these relationships even in a NoSQL database. But, imagine a corporate client where thousands of workers create their own copies of a Coda document that each link to a single source of data (or content) in another Coda document (which also links back) - a super-node has now emerged and one artefact in a Coda document must sustain thousands of outward links to separate documents. Aside from the data model (in an RDBMS) to support this possibility, any single viewing of the popular artefact must isolate your own link, and possibly (depending on permissions) many of the thousands of other outbound links because – well – that’s bi-direction linking’s objective, right?

My observation is simple; at scale, there’s probably a non-trivial cost for doing stuff like this and I’m sure there are some clever ways to implement it without supporting it at scale.

I see a lot of people in this community upset about performance. If you want more of that, implement this capability as a quick hack. Just sayin’ …

1 Like