<p>What are the pros and cons of including whitespace and comments as tokens in the tokenizer output?</p>
<p>For example:</p>
<pre><code>value=  6+ 7 // A comment
</code></pre>
<p>Would result in these tokens without whitespace/comments:</p>
<pre><code>identifier  'value'
assign      '='
integer     '6'
plus        '+'
integer     '7'
</code></pre>
<p>But would result in this with whitespace and comments included:</p>
<pre><code>identifier  'value'
assign      '='
whitespace  '  '
integer     '6'
plus        '+'
whitespace  ' '
integer     '7'
whitespace  ' '
comment     '// A comment'
</code></pre>
<p>Not including comments and whitespace has the advantage of producing much more concise token output. Thus, is there any reason to use the longer version that keeps whitespace/comments as tokens (in a non-indent-based language)?</p>
<p>What are the pros and cons of including whitespace/comments as tokens?</p>

<parsing><tokens><whitespace><comments>
Pros and cons of whitespace or comments being included as tokens
CC BY-SA 4.0
2023-07-09T20:53:31.757Z
I'm afraid this is a bit opinion-based. Whether you want whitespace included or not depends on what this is for and what you and the users of your language prefer.
user
CC BY-SA 4.0
2023-07-09T20:55:59.910Z
It also depends on the internal structure and the syntax of the language
Seggan
CC BY-SA 4.0
2023-07-09T23:04:31.407Z
Surely its a pro-cons question. Granted we probably have too many of those but why single this one out?
Bruce Adams
CC BY-SA 4.0
2023-07-11T15:20:53.927Z
@BruceAdams pros/cons questions are quite useful imo, I'm glad we have them on this site
CoffeeTableEspresso
CC BY-SA 4.0
2023-07-11T16:04:23.157Z
My comment was in reference to the close votes which have subsequently disappeared. Correction the question was reopened.
Bruce Adams
CC BY-SA 4.0
2023-07-18T13:38:04.840Z
Your two options are "no trivia" and "trivia is a kind of token" but you've missed the third option: trivia is a part of the token. In the C# and VB compilers every token has a "leading trivia" and "trailing trivia" list which gives you the best of both worlds; the parser ignores the trivia, the formatter and linter pay attention to it.
Eric Lippert
<p>The problem with including whitespace and comment tokens in the output of the lexer is that this stream of tokens is typically fed to the parser directly.</p>
<p>Those tokens need to be filtered out. If you don't it the makes the grammar the parsing stage uses <em>much</em> more complex.</p>
<p>It really depends on if you have a practical use for those tokens.
This is typically not the case, however:</p>
<ul>
<li><p>I am a fan of retaining comments and attaching them the things they comment.
This can be good for document generators.</p>
</li>
<li><p>Whitespace tokens <em>might</em> be useful for inferring intention if there is a syntax error.</p>
</li>
<li><p>You could just have the parsing stage skip them or filter them out.</p>
</li>
</ul>
<p>I'm not sure I would keep the whitespace tokens even in an indent sensitive language. It would typically be better to keep track of the indent level itself instead. You could provide it as an attribute if your parser uses an attribute grammar or you could convert it into a more meaningful &quot;indent&quot; tokens equivalent to a &quot;begin&quot; and &quot;end&quot; or open and close brackets.</p>

CC BY-SA 4.0
2023-07-09T20:58:15.397Z
2023-08-05T00:33:20.217Z
2023-08-05T00:33:20.217Z
Bruce Adams
CC BY-SA 4.0
2023-07-09T21:29:48.750Z
Swift is sensitive to mid-line whitespace: https://langdev.stackexchange.com/a/100/15 Wouldn't that require whitespace be kept for the parser?
Bbrk24
CC BY-SA 4.0
2023-07-09T23:09:46.473Z
I wasn't aware of that but I think you could probably do it in the lexer just fine. For example " \* ", " \*" and "\* " coud be different tokens - infix star, prefix star, postfix star
Bruce Adams
CC BY-SA 4.0
2023-08-09T17:44:01.150Z
How much would a parser be complicated by having two forms of each token--one with preceding whitespace and one without, and having a front end on the lexer which replaces whitespace with the "with preceding whitespace" version of whatever follows it?  Such a design would allow a compiler to accept `1.2E+5` as a "partial float constant with trailing E" token, followed by "plus with no preceding white space" and "decimal literal with no preceding white space", but interpret `0x1E+3` as `hex literal`, `plus with optional whitespace`, and `decimal literal with optional white space`.
supercat
CC BY-SA 4.0
2023-08-09T20:31:20.813Z
Its possily not as bad as it sounds. For the swift example that would occur at the lexical analysis stage.  Will users like or get used to a syntax that behaves that way is a different question. Perhaps one to ask the swift community.
Bruce Adams
<p>I would expect a modern compiler to be designed as a set of small libraries that can be reused to implement interactive things like <a href="https://en.wikipedia.org/wiki/Language_Server_Protocol" rel="nofollow noreferrer">LSP</a> or parsers for code formatters. As such I would rather have the tokenizer keep all the details about the original text, so that it is possible to output the text back how it was originally, or with slight modifications (reformat code except sections that are explicitly ignored with special commands like <code>clang off</code>).</p>
<p>A difficulty comes from the fact that comments can happen almost everywhere, and that means your internal representation built from those tokens could be &quot;polluted&quot; by a lot of extra nodes. That's why you generally find a Concrete Syntax Tree for comments etc. and an Abstract Syntax Tree for the semantics (with traceability information pointing back to the CST).</p>

CC BY-SA 4.0
2023-07-10T14:54:53.843Z
2023-07-10T15:06:08.943Z
2023-07-10T15:06:08.943Z
coredump
<p>They should not be ignored, although the modern convention is to treat all consecutive whitespace and comments as a single space.  (I took you as asking about this, but some people consider filtering out whitespace a separate earlier step.)  So, most programmers would expect <code>foo bar</code> <code>foo/**/bar</code> and <code>foo /**/ bar</code> to be equivalent to each other, and not to <code>foobar</code>.  In some languages, whitespace has no function but to separate tokens and can always be filtered out of the token stream, but there are languages where that is not true.</p>
<p>Haskell, for example, inserts delimiters into the token stream to open and close blocks determined by whitespace.  Another interesting example is Fortress, in which it is a syntax error for whitespace to imply an incorrect order of operations, such as <code>a+b * c</code>.  Modern compilers typically at least warn about code like this, even when the language allows it:</p>
<pre class="lang-algol prettyprint-override"><code>if p
    doSomething();
    oopsIsThisSupposedToRunUnconditionally();
</code></pre>
<p>There’s a famous story about a bug found by Fred Webb in a code review at NASA in 1963, with the code for the Mercury space capsule.  (An urban legend, which I myself have read in textbooks and repeated in the past, mixes this up with a different bug that caused the Mariner I space probe to explode.)  Webb, posting in the Usenet newsgroup <code>comp.risks</code> in 1990, recalled seeing the line of code</p>
<pre class="lang-fortran prettyprint-override"><code>DO 10 I = 1.10
</code></pre>
<p>which was an obvious typo for the loop instruction</p>
<pre class="lang-fortran prettyprint-override"><code>DO 10 I = 1, 10
</code></pre>
<p>In that version of Fortran, there were no reserved keywords, whitespace was ignored (as programmers often did not waste lines on their <a href="https://www.computerhistory.org/revolution/punched-cards/2/2" rel="nofollow noreferrer">punch cards</a> with it) and variables could be created implicitly by assigning to them, so the compiler correctly interpreted this line as</p>
<pre class="lang-fortran prettyprint-override"><code>DO10I = 1.10
</code></pre>
<p>which implicitly declared a <code>REAL</code> variable named <code>DO10I</code> and initialized it to <code>1.1</code>.  Although this didn’t cause any catastropic failures in a space flight, it became a well-known cautionary tale, repeated in many textbooks.  As a result, mainstream languages since then, including ANSI Fortran, reserve keywords and parse whitespace.</p>
<p>For a long time, it was also fashionable for languages to force programmers to declare all variables at the beginning of their scope (either in a special section like in COBOL or the start of their local block in the Algol family, including classic C).  However, modern languages want it to be possible to write static single assignments, so the current trend is to allow variables to be declared almost anywhere, but using a keyword (like <code>auto</code> in C++17, or <code>let</code> in Rust) to prevent accidentally declaring a variable with a typo.  Today’s compilers will also warn you when a variable is declared but not subsequently used.</p>

CC BY-SA 4.0
2023-07-09T21:10:34.830Z
2023-07-10T18:17:20.013Z
2023-07-10T18:17:20.013Z
Davislor
CC BY-SA 4.0
2023-07-09T23:06:46.000Z
I strongly dispute this comment - "As a result, all mainstream languages since then reserve keywords and tokenize whitespace". Whitespace is typically consumed by the lexer of most mainstream languages.
Bruce Adams
CC BY-SA 4.0
2023-07-09T23:14:50.743Z
@BruceAdams I think you were interpreting the scope of the question differently than I was. I changed the first paragraph.
Davislor
CC BY-SA 4.0
2023-07-10T12:30:06.780Z
Nice to see Fortress mentioned. It was an interesting point in the language design space that I, as a PL nerd, would have liked to see explored further. Gone too soon.
Jörg W Mittag
CC BY-SA 4.0
2023-07-10T18:19:15.360Z
@MarcellPerger Appreciate the edit suggestion. I think it comes down to how you define “mainstream.” I compromised by taking out “all,” but “most” on top of “mainstream” feels redundant to me.
Davislor
<p>It depends on what you're doing with the tokens.</p>
<p>If you want to implement something like BASIC's <code>LIST</code> command, you'll probably want to preserve the original formatting and comments.  If you're implementing a documentation generator like Doxygen, you might not care about whitespace, but you'll need to preserve the comments in order to extract tagged documentation.</p>
<p>OTOH, if all you want to do is <em>run</em> a program in your language, then “no-op” tokens such as whitespace and comments can be safely ignored.</p>
<p>If you're writing a generic parsing library that can be used for either purpose, you might want to provide two versions of your tokenizing routine: One that includes the “no-op” tokens, and one that filters them out.</p>

CC BY-SA 4.0
2023-07-10T19:03:31.410Z
2023-07-10T19:03:31.410Z
dan04
<p>Retaining white space, or at least information about its presence or absence, until parsing, would make it possible for a language to accept <code>1.2E+3</code> but reject <code>1.2E+ 3</code>, without having to complicate the language with concepts like &quot;pp-numbers&quot; that make it necessary for programmers to add whitespace to constructs like <code>0x1E -x</code> to avoid having the preprocessor glom the <code>-x</code> into the same token as the hex constant.</p>

CC BY-SA 4.0
2023-07-14T16:03:10.420Z
2023-07-14T16:03:10.420Z
supercat
CC BY-SA 4.0
2024-01-30T20:06:23.160Z
`1.2E+3` vs `1.2E+ 3` is a lexical problem, since lexer would expect exponent until closing `float` token.
ivanjermakov
CC BY-SA 4.0
2024-01-30T20:56:32.767Z
@ivanjermakov: The only circumstances in which it would matter whether a floating-point constant which contains non-alphanumeric characters is a single token, or is subdivided into tokens by the non-alphanumeric characters, are those where a period is immediately followed by `E` or `e`.  If `1.23E+4` is treated as 5 tokens, then the grammar would need to have a rule that a simple decimal constant, followed by a period, a decimal constant with E suffix, an optional + or - sign, and another simple decimal constant, would all combine to yield a floating-point constant.  No need for the...
supercat
CC BY-SA 4.0
2024-01-30T20:58:37.257Z
...preprocessor to understand anything about floating-point constants at all.  A preprocessor that was completely oblivious to floating-point formats might expand a macro `E`, if one were defined, while processing `1.E+2`, but specifying that compilers may perform or block such expansions at their leisure would have affected fewer programs than mandating that compilers break `0x1E-x`.
supercat
<p>One reason why you might want the tokenizer to include white-space and comments is that it enables more nuanced/detailed static analysis of the code.</p>
<p>For example, <a href="https://github.com/squizlabs/PHP_CodeSniffer" rel="nofollow noreferrer">PHP CodeSniffer</a> is a tool to report on code-style issues, via static analysis of the code files.  It needs to include white-space and comments in the token stream so that individual 'Sniffs' can report on incorrect use of white-space (e.g. use of tabs instead of spaces, incorrect indentation level, etc.) or issues in comments (e.g. spelling errors, bad formatting, etc.).</p>
<p>As far as I am aware, this uses PHP's own tokenizer.  If the PHP tokenizer didn't include these symbols, then it would effectively mean the functionality has to be re-implemented in userland, with all the risk of inconsistencies that would bring.</p>

CC BY-SA 4.0
2023-07-10T18:11:49.257Z
2023-07-10T18:19:00.723Z
2023-07-10T18:19:00.723Z
HappyDog
<p>You must retain comments <em>somehow</em>, if only their source locations, in order to implement a source formatter or documentation generator. They don’t need to be considered proper tokens themselves, although they can be.</p>
<p>My preference is to produce a pair of outputs from the tokeniser: the “in-band” tokens, and a “side band” for comments and formatting information. If you have documentation comments like Rust, then they go in the token channel, because they’re a <em>syntactic</em> form: non-doc comments are purely lexical, but doc comments are a type of annotation that can only be attached to a definition. They’re just fashioned to look the same, for the sake of familiarity.</p>
<p>If you also retain whitespace, then you can do <em>exact printing</em>: your compiler can read a source file, parse it, apply a modification to it, and print it out again using the original formatting, <em>without</em> needing to reformat the whole file. That ability is helpful, too, for implementing “code actions” in the Language Server Protocol, because the API is textual, not structural—it only offers the ability to replace a range of text with a different string of text.</p>

CC BY-SA 4.0
2023-07-10T03:04:17.937Z
2023-07-10T03:04:17.937Z
Jon Purdy
CC BY-SA 4.0
2023-07-15T00:58:07.670Z
ANTLR takes this approach
Seggan
# Pros and cons of whitespace or comments being included as tokens
Posted by: username_0 on 2023-07-09
What are the pros and cons of including whitespace and comments as tokens in the tokenizer output?
For example:
value=  6+ 7 // A comment

Would result in these tokens without whitespace/comments:
identifier  'value'
assign      '='
integer     '6'
plus        '+'
integer     '7'

But would result in this with whitespace and comments included:
identifier  'value'
assign      '='
whitespace  '  '
integer     '6'
plus        '+'
whitespace  ' '
integer     '7'
whitespace  ' '
comment     '// A comment'

Not including comments and whitespace has the advantage of producing much more concise token output. Thus, is there any reason to use the longer version that keeps whitespace/comments as tokens (in a non-indent-based language)?
What are the pros and cons of including whitespace/comments as tokens?
**Tags:** parsing tokens whitespace comments
> **Comments:**
> **username_1**: I'm afraid this is a bit opinion-based. Whether you want whitespace included or not depends on what this is for and what you and the users of your language prefer.  
> **username_2**: It also depends on the internal structure and the syntax of the language  
> **username_3**: Surely its a pro-cons question. Granted we probably have too many of those but why single this one out?  
> **username_4**: @BruceAdams pros/cons questions are quite useful imo, I'm glad we have them on this site  
> **username_3**: My comment was in reference to the close votes which have subsequently disappeared. Correction the question was reopened.  
> **username_5**: Your two options are "no trivia" and "trivia is a kind of token" but you've missed the third option: trivia is a part of the token. In the C# and VB compilers every token has a "leading trivia" and "trailing trivia" list which gives you the best of both worlds; the parser ignores the trivia, the formatter and linter pay attention to it.  
---
## 7 answers
### Answer by username_14 on 2023-07-10. Score: 9 | Accepted answer
You must retain comments *somehow*, if only their source locations, in order to implement a source formatter or documentation generator. They don’t need to be considered proper tokens themselves, although they can be.

My preference is to produce a pair of outputs from the tokeniser: the “in-band” tokens, and a “side band” for comments and formatting information. If you have documentation comments like Rust, then they go in the token channel, because they’re a *syntactic* form: non-doc comments are purely lexical, but doc comments are a type of annotation that can only be attached to a definition. They’re just fashioned to look the same, for the sake of familiarity.

If you also retain whitespace, then you can do *exact printing*: your compiler can read a source file, parse it, apply a modification to it, and print it out again using the original formatting, *without* needing to reformat the whole file. That ability is helpful, too, for implementing “code actions” in the Language Server Protocol, because the API is textual, not structural—it only offers the ability to replace a range of text with a different string of text.
> **Comments:**
> **username_2** (2023-07-15): ANTLR takes this approach  
---
### Answer by username_3 on 2023-07-09. Score: 4
The problem with including whitespace and comment tokens in the output of the lexer is that this stream of tokens is typically fed to the parser directly.

Those tokens need to be filtered out. If you don't it the makes the grammar the parsing stage uses *much* more complex.

It really depends on if you have a practical use for those tokens.
This is typically not the case, however:

* I am a fan of retaining comments and attaching them the things they comment.
  This can be good for document generators.
* Whitespace tokens *might* be useful for inferring intention if there is a syntax error.
* You could just have the parsing stage skip them or filter them out.

I'm not sure I would keep the whitespace tokens even in an indent sensitive language. It would typically be better to keep track of the indent level itself instead. You could provide it as an attribute if your parser uses an attribute grammar or you could convert it into a more meaningful "indent" tokens equivalent to a "begin" and "end" or open and close brackets.
> **Comments:**
> **username_6** (2023-07-09): Swift is sensitive to mid-line whitespace: https://langdev.stackexchange.com/a/100/15 Wouldn't that require whitespace be kept for the parser?  
> **username_3** (2023-07-09): I wasn't aware of that but I think you could probably do it in the lexer just fine. For example " \* ", " \*" and "\* " coud be different tokens - infix star, prefix star, postfix star  
> **username_7** (2023-08-09): How much would a parser be complicated by having two forms of each token--one with preceding whitespace and one without, and having a front end on the lexer which replaces whitespace with the "with preceding whitespace" version of whatever follows it?  Such a design would allow a compiler to accept `1.2E+5` as a "partial float constant with trailing E" token, followed by "plus with no preceding white space" and "decimal literal with no preceding white space", but interpret `0x1E+3` as `hex literal`, `plus with optional whitespace`, and `decimal literal with optional white space`.  
> **username_3** (2023-08-09): Its possily not as bad as it sounds. For the swift example that would occur at the lexical analysis stage.  Will users like or get used to a syntax that behaves that way is a different question. Perhaps one to ask the swift community.  
---
### Answer by username_8 on 2023-07-10. Score: 4
I would expect a modern compiler to be designed as a set of small libraries that can be reused to implement interactive things like [LSP](https://en.wikipedia.org/wiki/Language_Server_Protocol) or parsers for code formatters. As such I would rather have the tokenizer keep all the details about the original text, so that it is possible to output the text back how it was originally, or with slight modifications (reformat code except sections that are explicitly ignored with special commands like `clang off`).

A difficulty comes from the fact that comments can happen almost everywhere, and that means your internal representation built from those tokens could be "polluted" by a lot of extra nodes. That's why you generally find a Concrete Syntax Tree for comments etc. and an Abstract Syntax Tree for the semantics (with traceability information pointing back to the CST).
---
### Answer by username_9 on 2023-07-09. Score: 4
They should not be ignored, although the modern convention is to treat all consecutive whitespace and comments as a single space.  (I took you as asking about this, but some people consider filtering out whitespace a separate earlier step.)  So, most programmers would expect foo bar foo/**/bar and foo /**/ bar to be equivalent to each other, and not to foobar.  In some languages, whitespace has no function but to separate tokens and can always be filtered out of the token stream, but there are languages where that is not true.
Haskell, for example, inserts delimiters into the token stream to open and close blocks determined by whitespace.  Another interesting example is Fortress, in which it is a syntax error for whitespace to imply an incorrect order of operations, such as a+b * c.  Modern compilers typically at least warn about code like this, even when the language allows it:
if p
    doSomething();
    oopsIsThisSupposedToRunUnconditionally();

There’s a famous story about a bug found by Fred Webb in a code review at NASA in 1963, with the code for the Mercury space capsule.  (An urban legend, which I myself have read in textbooks and repeated in the past, mixes this up with a different bug that caused the Mariner I space probe to explode.)  Webb, posting in the Usenet newsgroup comp.risks in 1990, recalled seeing the line of code
DO 10 I = 1.10

which was an obvious typo for the loop instruction
DO 10 I = 1, 10

In that version of Fortran, there were no reserved keywords, whitespace was ignored (as programmers often did not waste lines on their punch cards with it) and variables could be created implicitly by assigning to them, so the compiler correctly interpreted this line as
DO10I = 1.10

which implicitly declared a REAL variable named DO10I and initialized it to 1.1.  Although this didn’t cause any catastropic failures in a space flight, it became a well-known cautionary tale, repeated in many textbooks.  As a result, mainstream languages since then, including ANSI Fortran, reserve keywords and parse whitespace.
For a long time, it was also fashionable for languages to force programmers to declare all variables at the beginning of their scope (either in a special section like in COBOL or the start of their local block in the Algol family, including classic C).  However, modern languages want it to be possible to write static single assignments, so the current trend is to allow variables to be declared almost anywhere, but using a keyword (like auto in C++17, or let in Rust) to prevent accidentally declaring a variable with a typo.  Today’s compilers will also warn you when a variable is declared but not subsequently used.
> **Comments:**
> **username_3** (2023-07-09): I strongly dispute this comment - "As a result, all mainstream languages since then reserve keywords and tokenize whitespace". Whitespace is typically consumed by the lexer of most mainstream languages.  
> **username_9** (2023-07-09): @BruceAdams I think you were interpreting the scope of the question differently than I was. I changed the first paragraph.  
> **username_10** (2023-07-10): Nice to see Fortress mentioned. It was an interesting point in the language design space that I, as a PL nerd, would have liked to see explored further. Gone too soon.  
> **username_9** (2023-07-10): @MarcellPerger Appreciate the edit suggestion. I think it comes down to how you define “mainstream.” I compromised by taking out “all,” but “most” on top of “mainstream” feels redundant to me.  
---
### Answer by username_13 on 2023-07-10. Score: 3
One reason why you might want the tokenizer to include white-space and comments is that it enables more nuanced/detailed static analysis of the code.

For example, [PHP CodeSniffer](https://github.com/squizlabs/PHP_CodeSniffer) is a tool to report on code-style issues, via static analysis of the code files. It needs to include white-space and comments in the token stream so that individual 'Sniffs' can report on incorrect use of white-space (e.g. use of tabs instead of spaces, incorrect indentation level, etc.) or issues in comments (e.g. spelling errors, bad formatting, etc.).

As far as I am aware, this uses PHP's own tokenizer. If the PHP tokenizer didn't include these symbols, then it would effectively mean the functionality has to be re-implemented in userland, with all the risk of inconsistencies that would bring.
---
### Answer by username_11 on 2023-07-10. Score: 2
It depends on what you're doing with the tokens.

If you want to implement something like BASIC's `LIST` command, you'll probably want to preserve the original formatting and comments. If you're implementing a documentation generator like Doxygen, you might not care about whitespace, but you'll need to preserve the comments in order to extract tagged documentation.

OTOH, if all you want to do is *run* a program in your language, then “no-op” tokens such as whitespace and comments can be safely ignored.

If you're writing a generic parsing library that can be used for either purpose, you might want to provide two versions of your tokenizing routine: One that includes the “no-op” tokens, and one that filters them out.
---
### Answer by username_7 on 2023-07-14. Score: 1
Retaining white space, or at least information about its presence or absence, until parsing, would make it possible for a language to accept `1.2E+3` but reject `1.2E+ 3`, without having to complicate the language with concepts like "pp-numbers" that make it necessary for programmers to add whitespace to constructs like `0x1E -x` to avoid having the preprocessor glom the `-x` into the same token as the hex constant.
> **Comments:**
> **username_12** (2024-01-30): `1.2E+3` vs `1.2E+ 3` is a lexical problem, since lexer would expect exponent until closing `float` token.  
> **username_7** (2024-01-30): @ivanjermakov: The only circumstances in which it would matter whether a floating-point constant which contains non-alphanumeric characters is a single token, or is subdivided into tokens by the non-alphanumeric characters, are those where a period is immediately followed by `E` or `e`.  If `1.23E+4` is treated as 5 tokens, then the grammar would need to have a rule that a simple decimal constant, followed by a period, a decimal constant with E suffix, an optional + or - sign, and another simple decimal constant, would all combine to yield a floating-point constant.  No need for the...  
> **username_7** (2024-01-30): ...preprocessor to understand anything about floating-point constants at all.  A preprocessor that was completely oblivious to floating-point formats might expand a macro `E`, if one were defined, while processing `1.E+2`, but specifying that compilers may perform or block such expansions at their leisure would have affected fewer programs than mandating that compilers break `0x1E-x`.  
---
