<p>So, today, I've added the support for the ternary conditional operator <code>?:</code> to <a href="https://picoblaze-simulator.sourceforge.io/" rel="nofollow noreferrer">my PicoBlaze assembler written in JavaScript</a>. Very soon, I ran into a problem: how to explain to the tokenizer that <code>:</code> is sometimes a part of a label (so that <code>label:</code> should be one single token) and sometimes a token by itself, as in when it is a part of the ternary conditional operator (like in <code>2+2&lt;5?1:0</code>).</p>
<p>For now, I dodged around that problem by telling the tokenizer that, if the <code>:</code> is followed by a new-line token, it is a part of label, <a href="https://github.com/FlatAssembler/PicoBlaze_Simulator_in_JS/blob/a862d65262ad87cb9bbdd304879eaa321deb9052/tokenizer.js#L102C1-L108C6" rel="nofollow noreferrer">like this</a>:</p>
<pre class="lang-js prettyprint-override"><code>  // Labels are single tokens.
  for (let i = 0; i &lt; tokenized.length; i++)
    if (tokenized[i].text == ':' &amp;&amp; tokenized[i + 1].text == '\n') {
      tokenized[i - 1].text += ':';
      tokenized.splice(i, 1);
      i--;
    }
</code></pre>
<p>But the problem with that solution is that labels don't necessarily have to be followed by the new-line token. For example, in the example program called &quot;Assembler Test&quot;, there is <a href="https://github.com/FlatAssembler/PicoBlaze_Simulator_in_JS/blob/a862d65262ad87cb9bbdd304879eaa321deb9052/assemblerTest.psm#L23" rel="nofollow noreferrer">this line of code</a>:</p>
<pre><code>endOfDataDirectives: hwbuild se
</code></pre>
<p>To that line of code, the parser of the new version of my assembler complains:</p>
<pre><code>Line #23: There is a colon without a matching question mark before it!
</code></pre>
<p>So, what should I do?</p>

<c><tokens>
How does the tokenizer in the C-like languages (having both labels and ternary conditional operator) know whether `:` belongs to a label or the `?:`?
CC BY-SA 4.0
2024-09-20T10:19:12.953Z
A tokenizer's job is not to know if `:` is part of a label or of a ternary. That's the parser's job. Based on the language's grammar, the parser will figure to which larger construct the `:` token belongs.
Eldritch Conundrum
CC BY-SA 4.0
2024-09-20T10:57:33.550Z
Why do you assume `label:` has to be one token?
MisterMiyagi
CC BY-SA 4.0
2024-09-20T12:29:07.127Z
As already mentioned, it is a parsing function to decide what construct that colon belongs to.  When a language supports both, the parser either looks ahead for identifier followed by colon, or, takes a second look after failing to find the matching ternary operator token.  I have found the look ahead easier in keeping a statement parser and expression parser separate, since then the expression parser doesn't have to fail back to the statement parser.  Generally speaking, the statement parser is looking at the identifier to see if it is for, while, do, etc.. so it doesn't take much lookahead.
Erik Eidt
CC BY-SA 4.0
2024-09-20T15:47:35.280Z
As others have correctly pointed out, don't do the job of the parser in the lexer. But I want to comment on a more subtle issue: is whitespace such as newlines, spaces and tabs a *token* in your language, or is it text which *separates* tokens? Having a "full fidelity lex" where the lexer keeps track of whitespace is very useful, but it often then complicates the parser if the parser then has to filter out those tokens.
Eric Lippert
CC BY-SA 4.0
2024-09-20T15:49:40.770Z
What we did in the design of the Roslyn C# and VB lexical analyzers was we made the "trivia" -- the whitespace, comments and so on -- part of each token. Every token has "leading" and "trailing" trivia, using a simple heuristic for determining whether a given space, newline or comment is logically "before" the next token or "after" the previous.   This then made it much easier to write code formatters because we had already decided which comments were logically attached to which tokens.
Eric Lippert
CC BY-SA 4.0
2024-09-20T16:09:13.187Z
@EricLippert Yes, the new-line sign is a token in the assembly language, for the simple reason that it delimits the directives.
FlatAssembler
CC BY-SA 4.0
2024-09-21T15:44:31.080Z
In the case of C, you're allowed to have arbitrary whitespace between the label and the `:`, which makes it even more obvious that they ought to be treated as separate tokens.
Nate Eldredge
<p>Generally, it does not.  In most C-like languages the <em>tokenizer</em> recognizes <code>:</code> as a single token -- often called COLON -- always.  The  <em>parser</em> will then recognize the two-token sequence IDENTIFIER COLON as a label if it is at the beginning of a statement.  Since a <code>?</code>/<code>:</code> expression cannot have a statement between the <code>?</code> and the <code>:</code> (just an expression), there is never any ambiguity.</p>

CC BY-SA 4.0
2024-09-21T05:56:22.777Z
2024-09-21T05:56:22.777Z
Chris Dodd
<p>In the most common setup, the tokenizer takes a stream of undifferentiated text, and separates it at token boundaries; a lexer is a moderately more sophisticated tokenizer that also attaches some context to each token.</p>
<p>So, your tokenizer would take <code>&quot;v1 &lt; v2 ? a : b&quot;</code> and split that into the list <code>[&quot;v1&quot;, &quot;&lt;&quot;, &quot;v2&quot;, &quot;?&quot;, &quot;a&quot;, &quot;:&quot;, &quot;b&quot;]</code> (possibly attaching a source location to each token). It's the parser that takes this list and determines the meaning of each token, and the parser is expected to maintain contextual state.</p>
<p>The parser would normally be in a state where a colon means &quot;end of label&quot;; when it sees a question mark, it changes state to one where a colon means &quot;end of first part of ternary operator&quot;. For my example, the parser would output the parsed node for a ternary operator, where the conditional is the result of parsing the tokens <code>[&quot;v1&quot;, &quot;&lt;&quot;, &quot;v2&quot;]</code>, the true branch is the result of parsing the tokens <code>[&quot;a&quot;]</code>, and the false branch is the result of parsing the tokens <code>[&quot;b&quot;]</code>.</p>
<p>For the example <code>&quot;endOfDataDirectives: hwbuild se&quot;</code>, the parser would determine that <code>[&quot;endofDataDirectives&quot;, &quot;:&quot;]</code> is the parsed node for a label with name &quot;endOfDataDirectives&quot;, and then continue on to parse <code>[&quot;hwbuild&quot;, &quot;se&quot;]</code> as if the label were absent.</p>

CC BY-SA 4.0
2024-09-20T14:03:18.623Z
2024-09-20T14:03:18.623Z
Simon Farnsworth
CC BY-SA 4.0
2024-09-20T16:03:45.237Z
One quirky problem is that the parser, in order to properly parse `?:`, needs to "move backwards", as `?:` is right-associative.
FlatAssembler
# How does the tokenizer in the C-like languages (having both labels and ternary conditional operator) know whether `:` belongs to a label or the `?:`?
Posted by: username_0 on 2024-09-20
So, today, I've added the support for the ternary conditional operator ?: to my PicoBlaze assembler written in JavaScript. Very soon, I ran into a problem: how to explain to the tokenizer that : is sometimes a part of a label (so that label: should be one single token) and sometimes a token by itself, as in when it is a part of the ternary conditional operator (like in 2+2<5?1:0).
For now, I dodged around that problem by telling the tokenizer that, if the : is followed by a new-line token, it is a part of label, like this:
  // Labels are single tokens.
  for (let i = 0; i < tokenized.length; i++)
    if (tokenized[i].text == ':' && tokenized[i + 1].text == '\n') {
      tokenized[i - 1].text += ':';
      tokenized.splice(i, 1);
      i--;
    }

But the problem with that solution is that labels don't necessarily have to be followed by the new-line token. For example, in the example program called "Assembler Test", there is this line of code:
endOfDataDirectives: hwbuild se

To that line of code, the parser of the new version of my assembler complains:
Line #23: There is a colon without a matching question mark before it!

So, what should I do?
**Tags:** c tokens
> **Comments:**
> **username_1**: A tokenizer's job is not to know if `:` is part of a label or of a ternary. That's the parser's job. Based on the language's grammar, the parser will figure to which larger construct the `:` token belongs.  
> **username_2**: Why do you assume `label:` has to be one token?  
> **username_3**: As already mentioned, it is a parsing function to decide what construct that colon belongs to.  When a language supports both, the parser either looks ahead for identifier followed by colon, or, takes a second look after failing to find the matching ternary operator token.  I have found the look ahead easier in keeping a statement parser and expression parser separate, since then the expression parser doesn't have to fail back to the statement parser.  Generally speaking, the statement parser is looking at the identifier to see if it is for, while, do, etc.. so it doesn't take much lookahead.  
> **username_4**: As others have correctly pointed out, don't do the job of the parser in the lexer. But I want to comment on a more subtle issue: is whitespace such as newlines, spaces and tabs a *token* in your language, or is it text which *separates* tokens? Having a "full fidelity lex" where the lexer keeps track of whitespace is very useful, but it often then complicates the parser if the parser then has to filter out those tokens.  
> **username_4**: What we did in the design of the Roslyn C# and VB lexical analyzers was we made the "trivia" -- the whitespace, comments and so on -- part of each token. Every token has "leading" and "trailing" trivia, using a simple heuristic for determining whether a given space, newline or comment is logically "before" the next token or "after" the previous.   This then made it much easier to write code formatters because we had already decided which comments were logically attached to which tokens.  
> **username_0**: @EricLippert Yes, the new-line sign is a token in the assembly language, for the simple reason that it delimits the directives.  
> **username_5**: In the case of C, you're allowed to have arbitrary whitespace between the label and the `:`, which makes it even more obvious that they ought to be treated as separate tokens.  
---
## 2 answers
### Answer by username_7 on 2024-09-20. Score: 8 | Accepted answer
In the most common setup, the tokenizer takes a stream of undifferentiated text, and separates it at token boundaries; a lexer is a moderately more sophisticated tokenizer that also attaches some context to each token.

So, your tokenizer would take `"v1 < v2 ? a : b"` and split that into the list `["v1", "<", "v2", "?", "a", ":", "b"]` (possibly attaching a source location to each token). It's the parser that takes this list and determines the meaning of each token, and the parser is expected to maintain contextual state.

The parser would normally be in a state where a colon means "end of label"; when it sees a question mark, it changes state to one where a colon means "end of first part of ternary operator". For my example, the parser would output the parsed node for a ternary operator, where the conditional is the result of parsing the tokens `["v1", "<", "v2"]`, the true branch is the result of parsing the tokens `["a"]`, and the false branch is the result of parsing the tokens `["b"]`.

For the example `"endOfDataDirectives: hwbuild se"`, the parser would determine that `["endofDataDirectives", ":"]` is the parsed node for a label with name "endOfDataDirectives", and then continue on to parse `["hwbuild", "se"]` as if the label were absent.
> **Comments:**
> **username_0** (2024-09-20): One quirky problem is that the parser, in order to properly parse `?:`, needs to "move backwards", as `?:` is right-associative.  
---
### Answer by username_6 on 2024-09-21. Score: 8
Generally, it does not. In most C-like languages the *tokenizer* recognizes `:` as a single token -- often called COLON -- always. The *parser* will then recognize the two-token sequence IDENTIFIER COLON as a label if it is at the beginning of a statement. Since a `?`/`:` expression cannot have a statement between the `?` and the `:` (just an expression), there is never any ambiguity.
---
