<p>In a lot of older C-family languages, lines of code (statements) must be ended with an explicit character (generally a semicolon). However, newer languages support inferring where statement endings should go. What are good parser rules to use for inferring statement endings?</p>

<syntax><parsing>
What rules can be used to infer statement endings for a language where they are implicit?
CC BY-SA 4.0
2023-05-17T12:26:02.540Z
https://stackoverflow.com/a/2846298/6333444
mousetail 'he-him'
CC BY-SA 4.0
2023-05-17T13:47:59.793Z
Note that in terms of having a simple non-ambiguous grammar it is *very* helpful to have an EOS token. Automatic EOS insertion always come at the price of having some bad corner cases. The trick is to minimize them.
Nuoji
<h2>When the next token in parsing would otherwise make no sense</h2>
<p>A rule in some languages including JavaScript. For example:</p>
<pre><code>{ 1
 2 }
</code></pre>
<p>In this case <code>1 2</code> is not a valid expression so JavaScript automatically inserts a <code>;</code> in between, turning this into <code>{1; 2}</code>.</p>

CC BY-SA 4.0
2023-05-17T12:21:38.207Z
2023-05-17T12:24:14.313Z
2023-05-17T12:24:14.313Z
mousetail 'he-him'
CC BY-SA 4.0
2023-05-18T01:37:02.067Z
JS does not insert a semicolon for `1 2`, only for `1\n2`
naffetS
CC BY-SA 4.0
2023-05-18T04:38:11.097Z
Yea the specifics are either a line break needs to be in between or the offending token is a `}`
mousetail 'he-him'
<p>Python's grammar has the following:</p>
<pre><code>simple_stmt: small_stmt (';' small_stmt)* (';')? NEWLINE;
</code></pre>
<p>The <a href="https://github.com/antlr/grammars-v4/blob/master/python/python3-cpp/Python3.g4#L347-L395" rel="nofollow noreferrer">definition</a> of <code>NEWLINE</code> then has to contend with Python's indentation rules, end-of-file determination, etc. If your language doesn't require white space the same way, then your lexer logic will be a little simpler.</p>
<p><a href="https://github.com/empirical-soft/empirical-lang/blob/master/src/Empirical.g4" rel="nofollow noreferrer">A language I wrote</a> uses the following:</p>
<pre><code>eos : ';' | EOF | {endAhead()}? | {nlAhead()}?;
</code></pre>
<p>It still has to look ahead for end-of-file as well as block <code>END</code> tokens, but I didn't need to track indentation.</p>

CC BY-SA 4.0
2023-05-17T13:32:23.617Z
2023-05-17T13:32:23.617Z
chrisaycock
<h1>Indent continuation lines</h1>
<p>A logical line continues across multiple physical lines as long as the subsequent lines are indented further than the initial line, and stops at the end of the physical line otherwise. This <strong>&quot;hanging indent&quot;</strong> matches the typical indentation pattern that people use for multi-line statements, so it virtually always does what the programmer intended. However, it requires including some indentation sensitivity in the parser, which is a complication. Otherwise, the statement ends at the end of the physical line.</p>
<p>The logical line ends as soon as a line at or before the initial indentation level is reached. For example, given this code:</p>
<pre><code>while (condition) do {
    x := x ++ &quot;a very long string with {interpolated - expressions} &quot;
         ++ &quot;spread over multiple lines&quot; ++
         &quot;with varied structure&quot;
    !y or error
    z.update
}
</code></pre>
<p>the <code>x :=</code> statement spreads across three lines, and stops before the prefix <code>!</code> operator that matches the original indentation. The next two lines are single-line statements, on the same level.</p>
<p>Even in languages where line continuation works differently, programmers tend to indent the subsequent lines, so this aligns with general practice already, merely formalising it. They also, once they've learned what they're doing, don't tend to indent code at the same nesting level differently from line to line, so there are <strong>very few if any false positives</strong> in this approach. In the cases where they don't want to continue the statement, they don't need to do anything special.</p>
<p>This continuation is <strong>always explicit at the surface level</strong>: it doesn't matter what the contents of the lines are, where operators or brackets appear, or what sort of construct is in use, only the raw indentation. It is possible to pre-process the indentation away and mechanically join the logical lines together if desired, without needing to parse or understand the code. In that way the parser itself would not need to understand indentation.</p>
<hr />
<p>This was the rule we ultimately adopted <a href="https://homepages.ecs.vuw.ac.nz/%7Emwh/minigrace/js/" rel="nofollow noreferrer">in</a> <a href="https://homepages.ecs.vuw.ac.nz/%7Emwh/js-kernan/entry/" rel="nofollow noreferrer">Grace</a>, and by and large it went unnoticed — the system always just <strong>did what people intended and they never thought about it</strong>. However, that language already had enforced indentation for bracketed blocks, so (1) the parser was ready to handle it, and (2) correct indentation was well trained into people before they started trying to split lines.</p>

CC BY-SA 4.0
2023-05-18T00:49:24.177Z
2023-05-18T00:49:24.177Z
Michael Homer
<h1>Infer end-of-statement aggressively at line end</h1>
<p>If the end of the physical line <em>could</em> be the end of the statement, then it is. <strong>Only if the line ends mid-expression</strong> — within a bracketed construct, after an infix or prefix operator, or whatever else is applicable to the language — does it continue.</p>
<p>This rule ensures that lines never continue accidentally, which tends to be confusing and lead to unexpected errors. It is more or less the direct opposite of the JavaScript rule, which <em>stops</em> only if the statement could <em>not</em> continue — leading to situations like</p>
<pre><code>var a = b * 2
// Log to screen or file
(verbose ? screen : file).write(a)
</code></pre>
<p>trying to call <code>2</code> as a function and complaining that it can't.</p>
<p>On the other hand, people often like to put their infix operator at the start of the continuation line, meaning that if the language both (1) has an overlapping prefix operator (like <code>+</code> or <code>-</code>) and (2) permits unused expressions as statements, this can pass unnoticed too:</p>
<pre><code>x = y
    - z
</code></pre>
<p>In this case, the language will silently calculate and discard <code>-z</code>, rather than reporting an error. It's likely something the programmer will acclimatise to with use of the language, but there is a small trap there when the language allows.</p>
<p>A big advantage is that this rule is fully predictable based on the code the programmer is writing <em>at the time</em>, and won't ever inadvertently continue an earlier line that wasn't being edited. Each line &quot;opts in&quot; to being continued.</p>

CC BY-SA 4.0
2023-05-17T21:23:53.203Z
2023-05-17T21:36:25.013Z
2023-05-17T21:36:25.013Z
Michael Homer
CC BY-SA 4.0
2023-05-17T21:33:28.683Z
Unless you're okay with the language being newline-sensitive, wouldn't this rule mean you have to parenthesize all subtraction? `x=y-z` could feasibly parse as `x=y;-z;` even with no line break.
Bbrk24
CC BY-SA 4.0
2023-05-17T21:36:58.237Z
I tweaked the heading to mention line endings to match the body instead of leaving "whenever" open.
Michael Homer
<p>A viable option is simply for every possible expression of the language to be self contained (i.e. with an unambiguous start and end).</p>
<p>Languages that do this is (that I know of) are <a href="https://www.lua.org/" rel="nofollow noreferrer">Lua</a> and <a href="https://gleam.run/" rel="nofollow noreferrer">Gleam</a>: while at first it may appear as using newlines to end statements, every expression is either explicitly close or inferrably done by the next expression (as it's the case for infix arithmetic operators)</p>
<p>The following Gleam program</p>
<pre><code>import gleam/io
import gleam/int

pub fn main() {
  let x = 8 + 8
  let y = {
    &quot;meaningless&quot;
    24 - 8
  }
  let add1 = fn(n) { n + 1 }
  let z = add1(7)

  io.print(int.to_string(x+y+z))
}
</code></pre>
<p>can be compressed and work just as well on a single line:</p>
<pre><code>import gleam/io import gleam/int pub fn main() { let x = 8 + 8 let y = {    &quot;meaningless&quot; 24 - 8 } let add1 = fn(n) { n + 1 } let z = add1(7) io.print(int.to_string(x+y+z))}
</code></pre>
<p>This wouldn't be possible with the ML-style open expression syntax (i.e. application by juxtaposition).</p>

CC BY-SA 4.0
2023-05-17T13:55:27.650Z
2023-05-18T05:53:10.103Z
2023-05-18T05:53:10.103Z
Maldus
CC BY-SA 4.0
2023-05-18T01:36:00.963Z
Lua also does this.
naffetS
# What rules can be used to infer statement endings for a language where they are implicit?
Posted by: username_0 on 2023-05-17
In a lot of older C-family languages, lines of code (statements) must be ended with an explicit character (generally a semicolon). However, newer languages support inferring where statement endings should go. What are good parser rules to use for inferring statement endings?
**Tags:** syntax parsing
> **Comments:**
> **username_1**: https://stackoverflow.com/a/2846298/6333444  
> **username_2**: Note that in terms of having a simple non-ambiguous grammar it is *very* helpful to have an EOS token. Automatic EOS insertion always come at the price of having some bad corner cases. The trick is to minimize them.  
---
## 5 answers
### Answer by username_7 on 2023-05-17. Score: 6 | Accepted answer
A viable option is simply for every possible expression of the language to be self contained (i.e. with an unambiguous start and end).
Languages that do this is (that I know of) are Lua and Gleam: while at first it may appear as using newlines to end statements, every expression is either explicitly close or inferrably done by the next expression (as it's the case for infix arithmetic operators)
The following Gleam program
import gleam/io
import gleam/int

pub fn main() {
  let x = 8 + 8
  let y = {
    "meaningless"
    24 - 8
  }
  let add1 = fn(n) { n + 1 }
  let z = add1(7)

  io.print(int.to_string(x+y+z))
}

can be compressed and work just as well on a single line:
import gleam/io import gleam/int pub fn main() { let x = 8 + 8 let y = {    "meaningless" 24 - 8 } let add1 = fn(n) { n + 1 } let z = add1(7) io.print(int.to_string(x+y+z))}

This wouldn't be possible with the ML-style open expression syntax (i.e. application by juxtaposition).
> **Comments:**
> **username_3** (2023-05-18): Lua also does this.  
---
### Answer by username_1 on 2023-05-17. Score: 4
When the next token in parsing would otherwise make no sense
A rule in some languages including JavaScript. For example:
{ 1
 2 }

In this case 1 2 is not a valid expression so JavaScript automatically inserts a ; in between, turning this into {1; 2}.
> **Comments:**
> **username_3** (2023-05-18): JS does not insert a semicolon for `1 2`, only for `1\n2`  
> **username_1** (2023-05-18): Yea the specifics are either a line break needs to be in between or the offending token is a `}`  
---
### Answer by username_4 on 2023-05-17. Score: 4
Python's grammar has the following:
simple_stmt: small_stmt (';' small_stmt)* (';')? NEWLINE;

The definition of NEWLINE then has to contend with Python's indentation rules, end-of-file determination, etc. If your language doesn't require white space the same way, then your lexer logic will be a little simpler.
A language I wrote uses the following:
eos : ';' | EOF | {endAhead()}? | {nlAhead()}?;

It still has to look ahead for end-of-file as well as block END tokens, but I didn't need to track indentation.
---
### Answer by username_5 on 2023-05-18. Score: 2
Indent continuation lines
A logical line continues across multiple physical lines as long as the subsequent lines are indented further than the initial line, and stops at the end of the physical line otherwise. This "hanging indent" matches the typical indentation pattern that people use for multi-line statements, so it virtually always does what the programmer intended. However, it requires including some indentation sensitivity in the parser, which is a complication. Otherwise, the statement ends at the end of the physical line.
The logical line ends as soon as a line at or before the initial indentation level is reached. For example, given this code:
while (condition) do {
    x := x ++ "a very long string with {interpolated - expressions} "
         ++ "spread over multiple lines" ++
         "with varied structure"
    !y or error
    z.update
}

the x := statement spreads across three lines, and stops before the prefix ! operator that matches the original indentation. The next two lines are single-line statements, on the same level.
Even in languages where line continuation works differently, programmers tend to indent the subsequent lines, so this aligns with general practice already, merely formalising it. They also, once they've learned what they're doing, don't tend to indent code at the same nesting level differently from line to line, so there are very few if any false positives in this approach. In the cases where they don't want to continue the statement, they don't need to do anything special.
This continuation is always explicit at the surface level: it doesn't matter what the contents of the lines are, where operators or brackets appear, or what sort of construct is in use, only the raw indentation. It is possible to pre-process the indentation away and mechanically join the logical lines together if desired, without needing to parse or understand the code. In that way the parser itself would not need to understand indentation.

This was the rule we ultimately adopted in Grace, and by and large it went unnoticed — the system always just did what people intended and they never thought about it. However, that language already had enforced indentation for bracketed blocks, so (1) the parser was ready to handle it, and (2) correct indentation was well trained into people before they started trying to split lines.
---
### Answer by username_5 on 2023-05-17. Score: 2
Infer end-of-statement aggressively at line end
If the end of the physical line could be the end of the statement, then it is. Only if the line ends mid-expression — within a bracketed construct, after an infix or prefix operator, or whatever else is applicable to the language — does it continue.
This rule ensures that lines never continue accidentally, which tends to be confusing and lead to unexpected errors. It is more or less the direct opposite of the JavaScript rule, which stops only if the statement could not continue — leading to situations like
var a = b * 2
// Log to screen or file
(verbose ? screen : file).write(a)

trying to call 2 as a function and complaining that it can't.
On the other hand, people often like to put their infix operator at the start of the continuation line, meaning that if the language both (1) has an overlapping prefix operator (like + or -) and (2) permits unused expressions as statements, this can pass unnoticed too:
x = y
    - z

In this case, the language will silently calculate and discard -z, rather than reporting an error. It's likely something the programmer will acclimatise to with use of the language, but there is a small trap there when the language allows.
A big advantage is that this rule is fully predictable based on the code the programmer is writing at the time, and won't ever inadvertently continue an earlier line that wasn't being edited. Each line "opts in" to being continued.
> **Comments:**
> **username_6** (2023-05-17): Unless you're okay with the language being newline-sensitive, wouldn't this rule mean you have to parenthesize all subtraction? `x=y-z` could feasibly parse as `x=y;-z;` even with no line break.  
> **username_5** (2023-05-17): I tweaked the heading to mention line endings to match the body instead of leaving "whenever" open.  
---
