<p>I have a language that looks something like this:</p>
<p><a href="https://i.sstatic.net/dC5iY.png" rel="noreferrer"><img src="https://i.sstatic.net/dC5iY.png" alt="enter image description here" /></a></p>
<p>Note this is a screenshot from blender, not my language but it works similarly.</p>
<p>This gets serialized currently to something like this:</p>
<pre class="lang-json prettyprint-override"><code>{
    nodes: [
        {
             &quot;id&quot;: 0,
             &quot;x&quot;: 431245.1231112445,
             &quot;y&quot;: 2122.19942312111,
             &quot;type&quot;: &quot;add&quot;,
             &quot;inputs&quot;: {
                 &quot;0&quot;: 1,
                 &quot;1&quot;: 2,
             }
        },
        {
             &quot;id&quot;: 1,
             &quot;x&quot;: 431345.7124212456,
             &quot;y&quot;: 2422.82111204065,
             &quot;type&quot;: &quot;add&quot;,
             &quot;inputs&quot;: {
                 &quot;0&quot;: null,
                 &quot;1&quot;: null,
             }
        },
        {
             &quot;id&quot;: 2,
             &quot;x&quot;: 49214.52566412456,
             &quot;y&quot;: 2112.12445122456,
             &quot;type&quot;: &quot;add&quot;,
             &quot;inputs&quot;: {
                 &quot;0&quot;: null,
                 &quot;1&quot;: null,
             }
        }
    ]
}
</code></pre>
<p>The issue comes when trying to check this into source control like git. When trying to merge branches, it just doesn't work at all:</p>
<ul>
<li>Lots of conflicts caused by just nudging the nodes around slightly. Annoying because the exact position isn't really that important yet it does need to be stored.</li>
<li>IDs shift when adding and removing nodes, leading to very tedious manual fixes</li>
<li>Sometimes git automatically merges nodes that look similar from different branches even if they are totally different. This can lead to nonsensical graphs in areas where git reports no conflicts.</li>
</ul>
<p>I do have some basic code in my editor that can parse git conflict markers and either automatically fix them or display them to the user, but it only works properly when the conflict is one individual property and not when conflict markers span multiple nodes.</p>
<p>How can I design a new version of this language so it plays nicely with git?</p>
<p>Note I'm not married to even using JSON, it can be completely different.</p>

<graphical-language>
How do I make my visual language version-control friendly?
CC BY-SA 4.0
2023-05-22T17:10:30.700Z
Xcode’s storyboard files have a similar problem (though they’re XML, not JSON). Sometimes you get a merge conflict in the “inferred metrics tie breakers” section, which is just a list of IDs of things earlier in the file and has no obvious meaning. Unfortunately, Apple doesn’t seem to care that it isn’t git-friendly.
Bbrk24
<p>Node IDs need to be stable across edits. Instead of consecutive integers, consider using strings. If you make the strings random, then it is unlikely that two programmers working in different branches will create different nodes with the same ID.</p>
<p>Also, write all of the data about each individual node on one line. This should prevent data from two different nodes from being be merged together automatically just because they happen to have some parts in common.</p>
<hr />
<p>Merge conflicts from changing the node positions are a bit trickier. Here's a suggestion which may or may not be the best option.</p>
<p>Write all of the layout data to a separate file, with a new random name each time. For example, if the source file is named <code>myprogram.ext</code>, name the layout file like <code>myprogram.sI30CXd2.layout</code>, and rename it each time the layout changes. (The random portion of the filename could be a hash of its contents.)</p>
<p>This will mean that when some node positions are changed differently on two different branches, both branches will delete the old layout file, and both branches will create a new layout files with different names, so there will be no merge conflict. Then when the project is loaded the next time, the IDE can detect that there are two different layout files, and either ask the user which one they want to keep, or just keep one with the most recent timestamp.</p>
<p>This does mean there will be a bunch of extra files cluttering up the project directory, but this may be acceptable; if not, you could write the layout files to a different directory in the project (e.g. <code>src/</code> for source files, <code>layout/</code> for layout files). Perhaps there is a similar solution which keeps the layout data inside the same file, e.g. in a list at the end, but this would require testing ─ I think merge conflicts could still happen when two different lines are inserted into the same list, but the creation of two different layout files will never conflict (as long as the hashes don't collide).</p>

CC BY-SA 4.0
2023-05-22T18:06:48.673Z
2023-05-22T18:06:48.673Z
kaya3
CC BY-SA 4.0
2023-05-22T20:51:00.070Z
Better yet, use [UUIDs](https://en.m.wikipedia.org/wiki/Universally_unique_identifier) instead of strings
Seggan
CC BY-SA 4.0
2023-05-22T20:55:03.507Z
@Seggan UUIDs would probably have to be represented as strings if you want to stick to the JSON format, but that is a decent way of generating random distinct names, sure.
kaya3
CC BY-SA 4.0
2023-05-22T20:58:34.273Z
Of course, I meant that I'd prefer UUIDs over just any random string
Seggan
<p>Git supports <a href="https://git-scm.com/docs/gitattributes#_defining_a_custom_merge_driver" rel="noreferrer">custom merge drivers, which you can apply to specific files</a>, and so it is possible to add automated tooling to help this at the merge phase. To make that job easier, some changes to the data format are useful. These changes will also make manual merging easier, so that use of the tool may not be required often.</p>
<h2>Data format</h2>
<p>First is ensuring that the <strong>node identifiers are persistent</strong>: I have used UUIDs for this before, but even six-character random alphanumeric identifiers can handle many thousands of generations before even a 1% chance of a conflict, and this may be acceptable for you. One reason to use shorter names is that it's then reasonable to <strong>prefix every line relating to that node with the identifier</strong>, to prevent the cross-node merging you saw.</p>
<p>I suggest a non-JSON format more similar to:</p>
<pre class="lang-none prettyprint-override"><code>abc543 type add
abc543 input 1 jqp3jg
abc543 input 2 q2mld1

jqp3jg type add
...
</code></pre>
<p>such that <strong>each distinct property is a separate line and unique format</strong>. This means that distinct modifications to two separate properties are not included in the same line, where they will either requiring more manual editing to merge parts of two lines or be overwritten accidentally. Every line clearly belongs to a specific node and will not be combined with another. Properties existing only on one side can be added easily.</p>
<p>In this model, stably sorting the file contents is important for mergeability. I also suggest a blank line between nodes. This format avoids most of the drawbacks mentioned and does not require specific merge tooling.</p>
<p>For the positions, if retaining the exact location is essential then accompanying modifications are unavoidable. One change that can reduce this <strong>at the language level is to snap nodes to a grid</strong>, so that movement of only a few pixels does not happen, and moving a node <em>back</em> to where it was after shifting out of the way is likely to be precisely accurate. This could happen by rounding on save only if a noticeable grid during interaction is undesirable; slight node movements when returning to the program are generally not noticeable.</p>
<h2>Tool-assisted merging</h2>
<p>A Git merge driver is an executable that handles the three-way merge of a specific file, and you can build a tool that understands your format exactly, and that even offers choices to the user if there are unresolvable conflicts. This would be a good part of the language tooling if this use case is relevant.</p>
<p>It can parse the format intelligently and not rely on line-oriented operations, and then the question becomes what mechanical steps can help reduce manual intervention when there are conflicts. For example, position data may just choose one side arbitrarily if this is not critical, while if movements are important then combining them sensibly is key.</p>
<p>An option for tool-assisted merging of numeric data in general is a conflict-free replicated data type called a PN-Counter, where instead of a single value, two values are stored representing the cumulative history of increases and decreases. The merge operation is to take the highest value of each, and the result value is the difference of the two. This means that similar movements on both sides result in similar positions in the result, but with a long history of changes the side with more substantial movement wins. This is stable across different merge orders, so that merging A into B gives the same outcome as merging B into A. It would also require tooling to merge these conveniently, but they <em>can</em> be resolved manually.</p>
<p>There is also a type of conflict-free replicated data type called a &quot;sequence CRDT&quot;, which is used for tree synchronisation and found in collaborative online editors. While CRDTs are made for distributed systems, there is nothing to stop you from writing out the packets to files and merging them together in an offline, asynchronous way later during the merge. I have considered this in the past in similar situations, but never gone as far as doing it. These changesets are likely to be much less human-readable than any of the other formats, so if that is an important property then this will not be a suitable approach. If real-time collaboration is a useful future feature, this would open the way to using the same wire and storage format.</p>
<p>A further tool-assisted option is to follow the textual format above, but in an append-only file: the last change to a property will always win. Git (and other version control) does not innately merge log-structured files like this, but both tool-assisted and manual merging are trivial. This approach does not give equal results for different merge orders, but it's not clear that this is important here.</p>

CC BY-SA 4.0
2023-05-22T23:04:23.713Z
2023-05-22T23:04:23.713Z
Michael Homer
<h1>Separate the structure from the rendering</h1>
<p>I would have the text representation of your language represent the links between the nodes only and not their location.</p>
<p>You could then have a separate document describing the layout. Think of html vs css for example.</p>
<pre><code>{
nodes: [
    {
         &quot;id&quot;: 0,
         &quot;type&quot;: &quot;add&quot;,
         &quot;inputs&quot;: {
             &quot;0&quot;: 1,
             &quot;1&quot;: 2,
         }
    },
    {
         &quot;id&quot;: 1,
         &quot;type&quot;: &quot;add&quot;,
         &quot;inputs&quot;: {
             &quot;0&quot;: null,
             &quot;1&quot;: null,
         }
    },
    {
         &quot;id&quot;: 2,
         &quot;type&quot;: &quot;add&quot;,
         &quot;inputs&quot;: {
             &quot;0&quot;: null,
             &quot;1&quot;: null,
         }
    }
]
}  
</code></pre>
<p>foobar.layout:</p>
<pre><code>{
    nodes: [
        {
             &quot;id&quot;: 0,
             &quot;x&quot;: 431245.1231112445,
             &quot;y&quot;: 2122.19942312111,
        },
        {
             &quot;id&quot;: 1,
             &quot;x&quot;: 431345.7124212456,
             &quot;y&quot;: 2422.82111204065,
        },
        {
             &quot;id&quot;: 2,
             &quot;x&quot;: 49214.52566412456,
             &quot;y&quot;: 2112.12445122456,
        }
    ]
}
    
</code></pre>
<p>Note that it is possible to mechanically transform between single and dual source file form. So you could diff just the semantics and ignore layout diffs.</p>
<p>I've stuck with your basic JSON layout here but I don't find JSON, XML or YAML very user friendly at all. Machine readable yes. But if the syntax matters enough to read in git diffs, maybe it matters enough to spend more time on a better syntax.</p>

CC BY-SA 4.0
2023-05-22T18:13:03.930Z
2023-05-22T18:13:03.930Z
Bruce Adams
CC BY-SA 4.0
2023-05-22T18:31:10.817Z
I don't think this solution would work very well. If a user removes and adds a node, the whole arrays will shuffle around. When two branches both have adds and removals, the conflicts would likely be hard to reconcile.
VLAZ
CC BY-SA 4.0
2023-05-29T21:35:32.210Z
Yes you also need to do other things such as making sure nodes are given unique ids not just sequentially numbered on creation. I just made the minimum change from OPs sample.
Bruce Adams
# How do I make my visual language version-control friendly?
Posted by: username_0 on 2023-05-22
I have a language that looks something like this:

Note this is a screenshot from blender, not my language but it works similarly.
This gets serialized currently to something like this:
{
    nodes: [
        {
             "id": 0,
             "x": 431245.1231112445,
             "y": 2122.19942312111,
             "type": "add",
             "inputs": {
                 "0": 1,
                 "1": 2,
             }
        },
        {
             "id": 1,
             "x": 431345.7124212456,
             "y": 2422.82111204065,
             "type": "add",
             "inputs": {
                 "0": null,
                 "1": null,
             }
        },
        {
             "id": 2,
             "x": 49214.52566412456,
             "y": 2112.12445122456,
             "type": "add",
             "inputs": {
                 "0": null,
                 "1": null,
             }
        }
    ]
}

The issue comes when trying to check this into source control like git. When trying to merge branches, it just doesn't work at all:

Lots of conflicts caused by just nudging the nodes around slightly. Annoying because the exact position isn't really that important yet it does need to be stored.
IDs shift when adding and removing nodes, leading to very tedious manual fixes
Sometimes git automatically merges nodes that look similar from different branches even if they are totally different. This can lead to nonsensical graphs in areas where git reports no conflicts.

I do have some basic code in my editor that can parse git conflict markers and either automatically fix them or display them to the user, but it only works properly when the conflict is one individual property and not when conflict markers span multiple nodes.
How can I design a new version of this language so it plays nicely with git?
Note I'm not married to even using JSON, it can be completely different.
**Tags:** graphical-language
> **Comments:**
> **username_1**: Xcode’s storyboard files have a similar problem (though they’re XML, not JSON). Sometimes you get a merge conflict in the “inferred metrics tie breakers” section, which is just a list of IDs of things earlier in the file and has no obvious meaning. Unfortunately, Apple doesn’t seem to care that it isn’t git-friendly.  
---
## 3 answers
### Answer by username_2 on 2023-05-22. Score: 12
Node IDs need to be stable across edits. Instead of consecutive integers, consider using strings. If you make the strings random, then it is unlikely that two programmers working in different branches will create different nodes with the same ID.

Also, write all of the data about each individual node on one line. This should prevent data from two different nodes from being be merged together automatically just because they happen to have some parts in common.

---

Merge conflicts from changing the node positions are a bit trickier. Here's a suggestion which may or may not be the best option.

Write all of the layout data to a separate file, with a new random name each time. For example, if the source file is named `myprogram.ext`, name the layout file like `myprogram.sI30CXd2.layout`, and rename it each time the layout changes. (The random portion of the filename could be a hash of its contents.)

This will mean that when some node positions are changed differently on two different branches, both branches will delete the old layout file, and both branches will create a new layout files with different names, so there will be no merge conflict. Then when the project is loaded the next time, the IDE can detect that there are two different layout files, and either ask the user which one they want to keep, or just keep one with the most recent timestamp.

This does mean there will be a bunch of extra files cluttering up the project directory, but this may be acceptable; if not, you could write the layout files to a different directory in the project (e.g. `src/` for source files, `layout/` for layout files). Perhaps there is a similar solution which keeps the layout data inside the same file, e.g. in a list at the end, but this would require testing ─ I think merge conflicts could still happen when two different lines are inserted into the same list, but the creation of two different layout files will never conflict (as long as the hashes don't collide).
> **Comments:**
> **username_3** (2023-05-22): Better yet, use [UUIDs](https://en.m.wikipedia.org/wiki/Universally_unique_identifier) instead of strings  
> **username_2** (2023-05-22): @Seggan UUIDs would probably have to be represented as strings if you want to stick to the JSON format, but that is a decent way of generating random distinct names, sure.  
> **username_3** (2023-05-22): Of course, I meant that I'd prefer UUIDs over just any random string  
---
### Answer by username_4 on 2023-05-22. Score: 9
Git supports custom merge drivers, which you can apply to specific files, and so it is possible to add automated tooling to help this at the merge phase. To make that job easier, some changes to the data format are useful. These changes will also make manual merging easier, so that use of the tool may not be required often.
Data format
First is ensuring that the node identifiers are persistent: I have used UUIDs for this before, but even six-character random alphanumeric identifiers can handle many thousands of generations before even a 1% chance of a conflict, and this may be acceptable for you. One reason to use shorter names is that it's then reasonable to prefix every line relating to that node with the identifier, to prevent the cross-node merging you saw.
I suggest a non-JSON format more similar to:
abc543 type add
abc543 input 1 jqp3jg
abc543 input 2 q2mld1

jqp3jg type add
...

such that each distinct property is a separate line and unique format. This means that distinct modifications to two separate properties are not included in the same line, where they will either requiring more manual editing to merge parts of two lines or be overwritten accidentally. Every line clearly belongs to a specific node and will not be combined with another. Properties existing only on one side can be added easily.
In this model, stably sorting the file contents is important for mergeability. I also suggest a blank line between nodes. This format avoids most of the drawbacks mentioned and does not require specific merge tooling.
For the positions, if retaining the exact location is essential then accompanying modifications are unavoidable. One change that can reduce this at the language level is to snap nodes to a grid, so that movement of only a few pixels does not happen, and moving a node back to where it was after shifting out of the way is likely to be precisely accurate. This could happen by rounding on save only if a noticeable grid during interaction is undesirable; slight node movements when returning to the program are generally not noticeable.
Tool-assisted merging
A Git merge driver is an executable that handles the three-way merge of a specific file, and you can build a tool that understands your format exactly, and that even offers choices to the user if there are unresolvable conflicts. This would be a good part of the language tooling if this use case is relevant.
It can parse the format intelligently and not rely on line-oriented operations, and then the question becomes what mechanical steps can help reduce manual intervention when there are conflicts. For example, position data may just choose one side arbitrarily if this is not critical, while if movements are important then combining them sensibly is key.
An option for tool-assisted merging of numeric data in general is a conflict-free replicated data type called a PN-Counter, where instead of a single value, two values are stored representing the cumulative history of increases and decreases. The merge operation is to take the highest value of each, and the result value is the difference of the two. This means that similar movements on both sides result in similar positions in the result, but with a long history of changes the side with more substantial movement wins. This is stable across different merge orders, so that merging A into B gives the same outcome as merging B into A. It would also require tooling to merge these conveniently, but they can be resolved manually.
There is also a type of conflict-free replicated data type called a "sequence CRDT", which is used for tree synchronisation and found in collaborative online editors. While CRDTs are made for distributed systems, there is nothing to stop you from writing out the packets to files and merging them together in an offline, asynchronous way later during the merge. I have considered this in the past in similar situations, but never gone as far as doing it. These changesets are likely to be much less human-readable than any of the other formats, so if that is an important property then this will not be a suitable approach. If real-time collaboration is a useful future feature, this would open the way to using the same wire and storage format.
A further tool-assisted option is to follow the textual format above, but in an append-only file: the last change to a property will always win. Git (and other version control) does not innately merge log-structured files like this, but both tool-assisted and manual merging are trivial. This approach does not give equal results for different merge orders, but it's not clear that this is important here.
---
### Answer by username_5 on 2023-05-22. Score: 5
Separate the structure from the rendering
I would have the text representation of your language represent the links between the nodes only and not their location.
You could then have a separate document describing the layout. Think of html vs css for example.
{
nodes: [
    {
         "id": 0,
         "type": "add",
         "inputs": {
             "0": 1,
             "1": 2,
         }
    },
    {
         "id": 1,
         "type": "add",
         "inputs": {
             "0": null,
             "1": null,
         }
    },
    {
         "id": 2,
         "type": "add",
         "inputs": {
             "0": null,
             "1": null,
         }
    }
]
}  

foobar.layout:
{
    nodes: [
        {
             "id": 0,
             "x": 431245.1231112445,
             "y": 2122.19942312111,
        },
        {
             "id": 1,
             "x": 431345.7124212456,
             "y": 2422.82111204065,
        },
        {
             "id": 2,
             "x": 49214.52566412456,
             "y": 2112.12445122456,
        }
    ]
}
    

Note that it is possible to mechanically transform between single and dual source file form. So you could diff just the semantics and ignore layout diffs.
I've stuck with your basic JSON layout here but I don't find JSON, XML or YAML very user friendly at all. Machine readable yes. But if the syntax matters enough to read in git diffs, maybe it matters enough to spend more time on a better syntax.
> **Comments:**
> **username_6** (2023-05-22): I don't think this solution would work very well. If a user removes and adds a node, the whole arrays will shuffle around. When two branches both have adds and removals, the conflicts would likely be hard to reconcile.  
> **username_5** (2023-05-29): Yes you also need to do other things such as making sure nodes are given unique ids not just sequentially numbered on creation. I just made the minimum change from OPs sample.  
---
