<p>Is the mass adoption of Unicode tokens as operators in general-purpose programming languages ​​a good idea? How acceptable is such a language to ordinary users and developers?</p>
<hr />
<h2>Background</h2>
<p>I want to add a lot of Unicode brackets and operators to my language, for use as literals for various commonly used structures and quote-free paths, because I think Unicode tokens are more visually noticeable than ASCII combination operators, with or without highlighting support.</p>
<p>I did some work to address issues I saw, hoping to improve the workflow:</p>
<ul>
<li>Designed a set of Unicode-based literal tokens</li>
<li>Adjusted the lexer to support pure QWERTY input
<ul>
<li>Unlike Julia, this is not a tab-completion, but a literal</li>
<li>Mainly supports unicode tokens by combining (<code>::()</code> / <code>.()</code>) and escaping (<code>\X</code>).</li>
</ul>
</li>
<li>Implemented a formatter for Jetbrains IDEs, which can automatically replace ASCII tokens with their Unicode counterparts during formatting</li>
</ul>
<p>According to the existing design, it looks roughly like this</p>
<div class="s-table-container"><table class="s-table">
<thead>
<tr>
<th style="text-align: left;">Collection</th>
<th style="text-align: left;">QWERTY Compatible</th>
<th style="text-align: left;">Unicode Token</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left;">Compare</td>
<td style="text-align: left;"><code>a &lt; b &gt; c</code></td>
<td style="text-align: left;"><code>(a &lt; b) &gt; c</code></td>
</tr>
<tr>
<td style="text-align: left;">Generic</td>
<td style="text-align: left;"><code>A::&lt;T&gt;</code></td>
<td style="text-align: left;"><code>A⟨T⟩</code></td>
</tr>
<tr>
<td style="text-align: left;">Block</td>
<td style="text-align: left;"><code>{ STATEMENT }</code></td>
<td style="text-align: left;"></td>
</tr>
<tr>
<td style="text-align: left;">HashMap</td>
<td style="text-align: left;"><code>#{ID: EXPR}</code></td>
<td style="text-align: left;"><code>⦃ID: EXPR⦄</code></td>
</tr>
<tr>
<td style="text-align: left;">List</td>
<td style="text-align: left;"><code>[EXPR]</code></td>
<td style="text-align: left;"></td>
</tr>
<tr>
<td style="text-align: left;">HashSet</td>
<td style="text-align: left;"><code>#[EXPR]</code></td>
<td style="text-align: left;"><code>⟦EXPR⟧</code></td>
</tr>
<tr>
<td style="text-align: left;">Range</td>
<td style="text-align: left;"><code>@[START:END:STEP]</code></td>
<td style="text-align: left;"><code>⁅START:END:STEP⁆</code></td>
</tr>
<tr>
<td style="text-align: left;">Function Invoke</td>
<td style="text-align: left;"><code>ID(ARG)</code></td>
<td style="text-align: left;"></td>
</tr>
<tr>
<td style="text-align: left;">Function Builder(like swift)</td>
<td style="text-align: left;"><code>TYPE::(AST)</code></td>
<td style="text-align: left;"><code>TYPE⦅AST⦆</code></td>
</tr>
<tr>
<td style="text-align: left;">Label</td>
<td style="text-align: left;"><code>\lID: LOOP</code></td>
<td style="text-align: left;"><code>¶ID: LOOP</code></td>
</tr>
<tr>
<td style="text-align: left;">Localized</td>
<td style="text-align: left;"><code>\Lgroup.key(SLOT)</code></td>
<td style="text-align: left;"><code>§group.key(SLOT)</code></td>
</tr>
<tr>
<td style="text-align: left;">Comment</td>
<td style="text-align: left;"><code>\\ COMMENT</code></td>
<td style="text-align: left;"><code>⍝ COMMENT</code></td>
</tr>
<tr>
<td style="text-align: left;">Nil</td>
<td style="text-align: left;"><code>\N</code></td>
<td style="text-align: left;"><code>∅</code></td>
</tr>
</tbody>
</table></div>
<p>(There are additionally also some prefix, infix and postfix operators which are not listed.)</p>
<p>Are there any other issues with implementing a language so heavily reliant on Unicode?</p>

<syntax><language-design><unicode>
What are the tradeoffs around supporting Unicode keywords and tokens?
CC BY-SA 4.0
2024-04-18T14:21:47.413Z
I'm voting to close this question because it doesn't provide any objective way to determine what a good answer is. If your question includes phrases like "Does such language make you feel bad?", it is probably not a good fit for this site.
Ginger
CC BY-SA 4.0
2024-04-18T16:28:02.103Z
Obviously opinionated, but please take readability into account. At least for me, the symbols in Range, HashSet and HashMap looked more or less the same until I gave it a deeper look. The same happened with the Unicode equivalent for Compare and Generic. It could very well lead to developers replacing only the closing character, not realizing it's not a regular character and spending hours confused.
kouta-kun
CC BY-SA 4.0
2024-04-18T18:17:17.950Z
Keep in mind that supporting Unicode is a nontrivial security issue. See [UTR #36](https://unicode.org/reports/tr36/).
RobertR
CC BY-SA 4.0
2024-04-18T19:30:01.113Z
As well as needing some objective criteria, this question needs a bit of focus. Is it about using non-ASCII characters to expand the symbology of the language - which has been explored since before Unicode, e.g. in APL? Is it about using visually similar but distinct characters, as is the case with the different types of bracket? Is it about having one form for input and a translation tool, which would explain the otherwise irrelevant example of inserted parentheses in the compare syntax? Is it about encoding (Unicode) or text entry (QWERTY) or even display (compare: programmer's ligatures)?
IMSoP
CC BY-SA 4.0
2024-04-18T21:07:49.270Z
All such usage has to be learned, but certain symbols would be far easier to learn than others. `≤` as a replacement for `<=` would be fairly uncontroversial. I have no idea what the intuition behind using `⍝` for a comment would be (though to be fair, I think syntax highlighting makes whatever syntax you choose for comment mostly irrelevant, from a readability perspective). I think the main argument against Unicode in the Python camp, however, centers around how easy it is to *type* them.
chepner
CC BY-SA 4.0
2024-04-19T18:41:03.990Z
@chepner That symbol (U+235D APL Functional Symbol Up Shoe Jot) seems to be invented specifically to mark comments for APL, but it's definitely obscure.
Jasmijn
CC BY-SA 4.0
2024-04-19T23:05:18.663Z
I think the main objection most would have to this is simply that it's inconvenient to type these things, since they don't exist on keyboards. You can use something that translates from the QWERTY syntax to Unicode, but then what's the point? Is it just because the Unicode version is "prettier"? But it means we need to know two syntaxes: One for typing, another for reading.
Barmar
CC BY-SA 4.0
2024-04-22T15:36:59.647Z
I would actually be willing to reopen the question as it is valid and there are several objective criteria against allowing unicode like performance, ability to produce symbols, complexity of proper codepoint association, lack of stability of the unicode standard, ability to render symbols, visual similarity of symbols.
feldentm
CC BY-SA 4.0
2024-04-22T20:04:07.197Z
I think this question is at its core a very good one. I've submitted some (opinionated) edits for review that hopefully make it more directed and answerable: @Aster, I hope I'm not misrepresenting your original intent.
apropos
<p>You can read about the (defunct) <a href="https://en.wikipedia.org/wiki/Fortress_(programming_language)" rel="nofollow noreferrer">Fortress programming language</a>. They implemented Unicode mathematical operators galore.</p>
<p>Check out one of the post-mortems: <a href="https://labs.oracle.com/pls/apex/f?p=94065:10:2849938467431:5316" rel="nofollow noreferrer">Fortress Features and Lessons Learned</a>.</p>
<p>Also, search for &quot;The Fortress Language Specification&quot;, and papers from Guy L. Steele Jr., for example.</p>
<p>Update (for summary request):
Of course a summary would be nice, but I'm not an expert (by far), just enthusiast. Let's try adding something, at least.</p>
<p>Some quotes from &quot;Fortress Features and Lessons Learned&quot;:</p>
<blockquote>
<p>Syntax strongly influenced by mathematics and whiteboards<br>
– Fortress libraries encourage use of ∪, ∩, ⊆, ∈ for sets<br>
– Also |a| , b x c , d x e , p ∧ q , p ∨ q ,
¬q , p ⊕ q<br>
– Multiplication expressed as a · b , a × b , or just a b</p>
</blockquote>
<p>Fortress was a project which competed with others to be &quot;the new Fortran&quot;, a programming language for High Performance Computing (it was not selected, though). So I guess being able to replicate mathematical language was intended to appeal to scientists and applications for scientific computing.</p>
<p>One point of note is that, to replicate or mathematical notation, they added support for &quot;whitespace operator&quot;, juxtaposition. They say it was possible but complicated things. I think it did not end up well, perhaps.</p>
<p>Adding lots of operators had some other effects: operator precedence, and typesetting.</p>
<blockquote>
<p>Nontransitive Operator Precedence in Fortress
• It’s okay to write a + b &gt; c + d<br>
– Why? Because + has higher precedence than &gt;.<br>
• And it’s okay to write p &lt; 0 ∨ p &gt; 9<br>
– Why? Because &gt; has higher precedence than ∨.<br>
• But in Fortress it is not okay to write a + b ∨ c + d.<br>
– There is no precedence defined between + and ∨.<br>
• We use only the most obvious and familiar rules of precedence.<br>
– This reduces opportunities for for making silly mistakes.<br>
• This matters when you have hundreds of operator symbols such as
∪ ∩ @ ÷ ⊕ ⊗ ∈ ≈. [this do not print well: 4  f }  ? b l k ]<br>
• Not a big burden—simply use parentheses to make meaning clear:<br>
(a + b) ∨ (c + d) or a + (b ∨ c) + d<br></p>
</blockquote>
<p>I have the impression that implementing typesetting required substantial effort. And gives you two versions of code: typeset vs ASCII. But perhaps typesetting was mostly for &quot;big operators&quot; (sum, lim, max) as in LaTex?</p>
<blockquote>
<p>Another Point about Syntax<br>
• There are tradeoffs between readability and writability<br>
• Mathematical syntax was designed more for certain kinds of
concision than for robustness!<br>
– In particular, simple juxtaposition is used in many ways<br>
– We tried to make this programmable<br>
-- Sometimes this worked beautifully<br>
-- Sometimes it required some weird contortions<br>
• Haskell and Scala do this more consistently<br>
– Each captures a certain PART of mathematical tradition
quite well</p>
</blockquote>
<p>So several features interact with others. And design decisions were made as a whole, so to extract guidance useful to you you will have to look into the details. Or ask the authors themselves. :-)</p>

CC BY-SA 4.0
2024-04-18T16:11:28.157Z
2024-04-22T13:12:20.013Z
2024-04-22T13:12:20.013Z
Pablo H
CC BY-SA 4.0
2024-04-18T17:05:46.283Z
Can you share a summary of the links you provided - did they decide it was a good thing or not? What points do you specifically think are relevant to the question? Not only does this save everyone reading a lot of time, it means the answer would remain relevant even if the links stop working in future (which is, sadly, extremely common).
IMSoP
# What are the tradeoffs around supporting Unicode keywords and tokens?
Posted by: username_0 on 2024-04-18
Is the mass adoption of Unicode tokens as operators in general-purpose programming languages ​​a good idea? How acceptable is such a language to ordinary users and developers?

---

## Background

I want to add a lot of Unicode brackets and operators to my language, for use as literals for various commonly used structures and quote-free paths, because I think Unicode tokens are more visually noticeable than ASCII combination operators, with or without highlighting support.

I did some work to address issues I saw, hoping to improve the workflow:

* Designed a set of Unicode-based literal tokens
* Adjusted the lexer to support pure QWERTY input
  + Unlike Julia, this is not a tab-completion, but a literal
  + Mainly supports unicode tokens by combining (`::()` / `.()`) and escaping (`\X`).
* Implemented a formatter for Jetbrains IDEs, which can automatically replace ASCII tokens with their Unicode counterparts during formatting

According to the existing design, it looks roughly like this

| Collection | QWERTY Compatible | Unicode Token |
| --- | --- | --- |
| Compare | `a < b > c` | `(a < b) > c` |
| Generic | `A::` | `A⟨T⟩` |
| Block | `{ STATEMENT }` |  |
| HashMap | `#{ID: EXPR}` | `⦃ID: EXPR⦄` |
| List | `[EXPR]` |  |
| HashSet | `#[EXPR]` | `⟦EXPR⟧` |
| Range | `@[START:END:STEP]` | `⁅START:END:STEP⁆` |
| Function Invoke | `ID(ARG)` |  |
| Function Builder(like swift) | `TYPE::(AST)` | `TYPE⦅AST⦆` |
| Label | `\lID: LOOP` | `¶ID: LOOP` |
| Localized | `\Lgroup.key(SLOT)` | `§group.key(SLOT)` |
| Comment | `\\ COMMENT` | `⍝ COMMENT` |
| Nil | `\N` | `∅` |

(There are additionally also some prefix, infix and postfix operators which are not listed.)

Are there any other issues with implementing a language so heavily reliant on Unicode?
**Tags:** syntax language-design unicode
> **Comments:**
> **username_1**: I'm voting to close this question because it doesn't provide any objective way to determine what a good answer is. If your question includes phrases like "Does such language make you feel bad?", it is probably not a good fit for this site.  
> **username_2**: Obviously opinionated, but please take readability into account. At least for me, the symbols in Range, HashSet and HashMap looked more or less the same until I gave it a deeper look. The same happened with the Unicode equivalent for Compare and Generic. It could very well lead to developers replacing only the closing character, not realizing it's not a regular character and spending hours confused.  
> **username_3**: Keep in mind that supporting Unicode is a nontrivial security issue. See [UTR #36](https://unicode.org/reports/tr36/).  
> **username_4**: As well as needing some objective criteria, this question needs a bit of focus. Is it about using non-ASCII characters to expand the symbology of the language - which has been explored since before Unicode, e.g. in APL? Is it about using visually similar but distinct characters, as is the case with the different types of bracket? Is it about having one form for input and a translation tool, which would explain the otherwise irrelevant example of inserted parentheses in the compare syntax? Is it about encoding (Unicode) or text entry (QWERTY) or even display (compare: programmer's ligatures)?  
> **username_5**: All such usage has to be learned, but certain symbols would be far easier to learn than others. `≤` as a replacement for `<=` would be fairly uncontroversial. I have no idea what the intuition behind using `⍝` for a comment would be (though to be fair, I think syntax highlighting makes whatever syntax you choose for comment mostly irrelevant, from a readability perspective). I think the main argument against Unicode in the Python camp, however, centers around how easy it is to *type* them.  
> **username_6**: @chepner That symbol (U+235D APL Functional Symbol Up Shoe Jot) seems to be invented specifically to mark comments for APL, but it's definitely obscure.  
> **username_7**: I think the main objection most would have to this is simply that it's inconvenient to type these things, since they don't exist on keyboards. You can use something that translates from the QWERTY syntax to Unicode, but then what's the point? Is it just because the Unicode version is "prettier"? But it means we need to know two syntaxes: One for typing, another for reading.  
> **username_8**: I would actually be willing to reopen the question as it is valid and there are several objective criteria against allowing unicode like performance, ability to produce symbols, complexity of proper codepoint association, lack of stability of the unicode standard, ability to render symbols, visual similarity of symbols.  
> **username_9**: I think this question is at its core a very good one. I've submitted some (opinionated) edits for review that hopefully make it more directed and answerable: @Aster, I hope I'm not misrepresenting your original intent.  
---
## 1 answers
### Answer by username_10 on 2024-04-18. Score: 3
You can read about the (defunct) [Fortress programming language](https://en.wikipedia.org/wiki/Fortress_(programming_language)). They implemented Unicode mathematical operators galore.

Check out one of the post-mortems: [Fortress Features and Lessons Learned](https://labs.oracle.com/pls/apex/f?p=94065:10:2849938467431:5316).

Also, search for "The Fortress Language Specification", and papers from Guy L. Steele Jr., for example.

Update (for summary request):
Of course a summary would be nice, but I'm not an expert (by far), just enthusiast. Let's try adding something, at least.

Some quotes from "Fortress Features and Lessons Learned":

> Syntax strongly influenced by mathematics and whiteboards  
> – Fortress libraries encourage use of ∪, ∩, ⊆, ∈ for sets  
> – Also |a| , b x c , d x e , p ∧ q , p ∨ q ,
> ¬q , p ⊕ q  
> – Multiplication expressed as a · b , a × b , or just a b

Fortress was a project which competed with others to be "the new Fortran", a programming language for High Performance Computing (it was not selected, though). So I guess being able to replicate mathematical language was intended to appeal to scientists and applications for scientific computing.

One point of note is that, to replicate or mathematical notation, they added support for "whitespace operator", juxtaposition. They say it was possible but complicated things. I think it did not end up well, perhaps.

Adding lots of operators had some other effects: operator precedence, and typesetting.

> Nontransitive Operator Precedence in Fortress
> • It’s okay to write a + b > c + d  
> – Why? Because + has higher precedence than >.  
> • And it’s okay to write p < 0 ∨ p > 9  
> – Why? Because > has higher precedence than ∨.  
> • But in Fortress it is not okay to write a + b ∨ c + d.  
> – There is no precedence defined between + and ∨.  
> • We use only the most obvious and familiar rules of precedence.  
> – This reduces opportunities for for making silly mistakes.  
> • This matters when you have hundreds of operator symbols such as
> ∪ ∩ @ ÷ ⊕ ⊗ ∈ ≈. [this do not print well: 4 f } ? b l k ]  
> • Not a big burden—simply use parentheses to make meaning clear:  
> (a + b) ∨ (c + d) or a + (b ∨ c) + d

I have the impression that implementing typesetting required substantial effort. And gives you two versions of code: typeset vs ASCII. But perhaps typesetting was mostly for "big operators" (sum, lim, max) as in LaTex?

> Another Point about Syntax  
> • There are tradeoffs between readability and writability  
> • Mathematical syntax was designed more for certain kinds of
> concision than for robustness!  
> – In particular, simple juxtaposition is used in many ways  
> – We tried to make this programmable  
> -- Sometimes this worked beautifully  
> -- Sometimes it required some weird contortions  
> • Haskell and Scala do this more consistently  
> – Each captures a certain PART of mathematical tradition
> quite well

So several features interact with others. And design decisions were made as a whole, so to extract guidance useful to you you will have to look into the details. Or ask the authors themselves. :-)
> **Comments:**
> **username_4** (2024-04-18): Can you share a summary of the links you provided - did they decide it was a good thing or not? What points do you specifically think are relevant to the question? Not only does this save everyone reading a lot of time, it means the answer would remain relevant even if the links stop working in future (which is, sadly, extremely common).  
---
