<p>In my tokenizers and parsers, once I find a syntax error (like an unexpected token) I usually throw a fatal exception that finishes parsing of the input source.</p>
<p>The downside of how I usually do it is that:</p>
<ul>
<li>I do not get a resulting AST; everything parsed before the error is discarded.</li>
<li>in a potential Language Server, I would normally not progress to type checking unless I cache the last type checking after a successful parsing of a program.</li>
</ul>
<p>I am aware that parsers like jQuery Esprima and <a href="http://www.github.com/oxyc/luaparse" rel="noreferrer">luaparse</a> (if I recall correctly) are able to tolerate these fatal syntax errors.</p>
<p>I already use a diagnostics collection per Source, but most syntax errors throw a fatal exception. How can I tolerate such syntax errors (i.e. ignore the error and move to the next parsing)?</p>

<parsing><error-handling>
How to tolerate syntax errors in a typed language?
CC BY-SA 4.0
2023-12-04T23:17:55.143Z
I think this question would be more focused if you asked only "should ...", or only "how ...". You could ask both questions separately if you want.
kaya3
CC BY-SA 4.0
2023-12-05T07:31:59.177Z
By "tolerate", do you mean "recover from" during parsing (in order to try to decide whether the rest of the code has more errors)? Or are you actually trying to design a language that accepts code that doesn't meet the strict language spec, and tries to make some sense of it?
Karl Knechtel
CC BY-SA 4.0
2023-12-05T11:01:50.950Z
@KarlKnechtel The former (there are no related error recovery questions apparently)
Hydroper
CC BY-SA 4.0
2024-01-04T07:49:04.633Z
The Eclipse Java Compiler (ECJ) also does this. See for example this [answer](https://stackoverflow.com/a/1642412) and this [wiki](https://github.com/eclipse-jdt/eclipse.jdt.core/wiki/ECJ).
Lii
<p>You can’t parse a program with syntax errors. So what you do is modify the program so there are no syntax errors, and parse that. In a simple C program</p>
<pre><code>int x = 0
int y = 0;
</code></pre>
<p>you have a blatant syntax error. A clever compiler will realise that adding a semicolon in the right place fixes it. So it displays an error, and compiles as if that ; was there. If you wrote</p>
<pre><code>Int x = 0;
</code></pre>
<p>with capital I a clever compiler will realise that with a lowercase i everything is fine and compile that. In each case an IDE might suggest to fix the problem.</p>
<p>There are cases where a syntax error or other error completely changes the program. Like an extra or missing parenthesis. The compiler can remove or insert one. If programs usually consist of functions, and the language can identify the start of a function, that’s a good point to assume that another function would have ended just before the start of a new one.</p>
<p>So in general, a good compiler will make changes that remove syntax errors, and will extract as much useful information as possible, to be able to continue compiling without undue error messages.</p>
<p>If there is a syntax error with some type, the compiler will often introduce an error type, which can be compiled without error messages. If we have a declaration that x is an int and y’s type contained a syntax error, then x + y can be compiled without an error message. The compiler assumes that errors it displays like the one for y will eventually be fixed and then x + y will be fine.</p>

CC BY-SA 4.0
2023-12-29T12:37:03.803Z
2023-12-29T12:37:03.803Z
gnasher729
CC BY-SA 4.0
2023-12-30T07:23:09.727Z
So *how* does one actually do that? Assuming that there already is a clever compiler that does all that  seems to skip the question. These examples all seem obvious for a human who knows the answer, but how would one approach writing/defining a program/parser/compiler/… to make the same judgement?
MisterMiyagi
CC BY-SA 4.0
2024-01-05T01:37:42.383Z
@MisterMiyagi Well, here is an article "Resilient LL Parsing Tutorial": https://matklad.github.io/2023/05/21/resilient-ll-parsing-tutorial.html , by the orignal author of rust-analyzer.
Joker_vD
CC BY-SA 4.0
2024-01-05T06:07:24.437Z
@Joker_vD I'm personally well aware of some strategies, and that there are many more out there. But *this answer* should contain some to address the question.
MisterMiyagi
<p>Usually, you would have anchor sets like '}' while within a block and you would only skip tokens until you reach a token in that set. While this might cause more errors, it works reasonably well in practice. Especially, if taken into account during language design.</p>
<p>In later phases like semantic analysis, you need to provide some error objects that you can perform your analyses on. Usually, you wouldn't raise any more errors there or look into error AST nodes.</p>
<p>However, be aware that you cannot always provide a meaningful error type that does not create other issues. Again, in practice common errors are mistyping or imbalanced braces or things like that <em>within</em> a method body. If your language is designed to tolerate such things. Things that are particularly bad are global type inference and other global properties that get affected when removing the content of a method body or initializer or similar.</p>
<p>Finally, if you have a look at how different IDEs work, you will notice that the value they provide is usually reduced a lot while confronted with actual parse errors and it is just somewhat reduced while confronted with semantic analysis errors.</p>
<p>Regarding the language server part, you should consider not updating your state while there are syntactic errors. At least you should try how such an implementation works out in practice before trying more sophisticated approaches.</p>

CC BY-SA 4.0
2023-12-05T16:17:07.377Z
2023-12-05T16:17:07.377Z
feldentm
# How to tolerate syntax errors in a typed language?
Posted by: username_0 on 2023-12-04
In my tokenizers and parsers, once I find a syntax error (like an unexpected token) I usually throw a fatal exception that finishes parsing of the input source.

The downside of how I usually do it is that:

* I do not get a resulting AST; everything parsed before the error is discarded.
* in a potential Language Server, I would normally not progress to type checking unless I cache the last type checking after a successful parsing of a program.

I am aware that parsers like jQuery Esprima and [luaparse](http://www.github.com/oxyc/luaparse) (if I recall correctly) are able to tolerate these fatal syntax errors.

I already use a diagnostics collection per Source, but most syntax errors throw a fatal exception. How can I tolerate such syntax errors (i.e. ignore the error and move to the next parsing)?
**Tags:** parsing error-handling
> **Comments:**
> **username_1**: I think this question would be more focused if you asked only "should ...", or only "how ...". You could ask both questions separately if you want.  
> **username_2**: By "tolerate", do you mean "recover from" during parsing (in order to try to decide whether the rest of the code has more errors)? Or are you actually trying to design a language that accepts code that doesn't meet the strict language spec, and tries to make some sense of it?  
> **username_0**: @KarlKnechtel The former (there are no related error recovery questions apparently)  
> **username_3**: The Eclipse Java Compiler (ECJ) also does this. See for example this [answer](https://stackoverflow.com/a/1642412) and this [wiki](https://github.com/eclipse-jdt/eclipse.jdt.core/wiki/ECJ).  
---
## 2 answers
### Answer by username_7 on 2023-12-05. Score: 10 | Accepted answer
Usually, you would have anchor sets like '}' while within a block and you would only skip tokens until you reach a token in that set. While this might cause more errors, it works reasonably well in practice. Especially, if taken into account during language design.

In later phases like semantic analysis, you need to provide some error objects that you can perform your analyses on. Usually, you wouldn't raise any more errors there or look into error AST nodes.

However, be aware that you cannot always provide a meaningful error type that does not create other issues. Again, in practice common errors are mistyping or imbalanced braces or things like that *within* a method body. If your language is designed to tolerate such things. Things that are particularly bad are global type inference and other global properties that get affected when removing the content of a method body or initializer or similar.

Finally, if you have a look at how different IDEs work, you will notice that the value they provide is usually reduced a lot while confronted with actual parse errors and it is just somewhat reduced while confronted with semantic analysis errors.

Regarding the language server part, you should consider not updating your state while there are syntactic errors. At least you should try how such an implementation works out in practice before trying more sophisticated approaches.
---
### Answer by username_4 on 2023-12-29. Score: 1
You can’t parse a program with syntax errors. So what you do is modify the program so there are no syntax errors, and parse that. In a simple C program
int x = 0
int y = 0;

you have a blatant syntax error. A clever compiler will realise that adding a semicolon in the right place fixes it. So it displays an error, and compiles as if that ; was there. If you wrote
Int x = 0;

with capital I a clever compiler will realise that with a lowercase i everything is fine and compile that. In each case an IDE might suggest to fix the problem.
There are cases where a syntax error or other error completely changes the program. Like an extra or missing parenthesis. The compiler can remove or insert one. If programs usually consist of functions, and the language can identify the start of a function, that’s a good point to assume that another function would have ended just before the start of a new one.
So in general, a good compiler will make changes that remove syntax errors, and will extract as much useful information as possible, to be able to continue compiling without undue error messages.
If there is a syntax error with some type, the compiler will often introduce an error type, which can be compiled without error messages. If we have a declaration that x is an int and y’s type contained a syntax error, then x + y can be compiled without an error message. The compiler assumes that errors it displays like the one for y will eventually be fixed and then x + y will be fine.
> **Comments:**
> **username_5** (2023-12-30): So *how* does one actually do that? Assuming that there already is a clever compiler that does all that  seems to skip the question. These examples all seem obvious for a human who knows the answer, but how would one approach writing/defining a program/parser/compiler/… to make the same judgement?  
> **username_6** (2024-01-05): @MisterMiyagi Well, here is an article "Resilient LL Parsing Tutorial": https://matklad.github.io/2023/05/21/resilient-ll-parsing-tutorial.html , by the orignal author of rust-analyzer.  
> **username_5** (2024-01-05): @Joker_vD I'm personally well aware of some strategies, and that there are many more out there. But *this answer* should contain some to address the question.  
---
