<p>For most modern programming languages, an LSP (<a href="https://en.wikipedia.org/wiki/Language_Server_Protocol" rel="noreferrer">Language Server Protocol</a>) server needs to be developed, which requires reusing/rewriting some of the existing code that is already used by the compiler. So, how can a compiler be designed to accommodate this need from the start?</p>
<p>Credit to gavinhoward for <a href="https://area51.stackexchange.com/proposals/127456/programming-language-design-and-implementation/127493#127493">the idea</a> on Area 51.</p>

<compilers><language-server-protocol>
How would you design a compiler upfront to be both a compiler and a LSP server?
<p>One thing you can do to save a lot of time on LSP re-parsing and re-compiling your code on frequent changes: use a parser with memoisation. Packrat, for example, or any other similar approach. This way it'll be very easy to invalidate the parsed region affected by an edit and then reparse the entire unit, whatever it is - your parser should know the boundaries of your units, be it a module, a class, a function, an entire file, etc.</p>
<p>Likewise, you can run further lowering, typing passes, etc., only on things that changed and that depend on what you've changed.</p>
<p>It can all be easily made a part of your compiler, because why would you write two different parsers anyway? A side effect of having syntax highlighting, type hints, pretty-printing and all that as a side effect of compilation is that you can add formatted literate code generation to the compiler.</p>

CC BY-SA 4.0
2023-07-02T19:18:04.597Z
2023-07-03T19:20:33.290Z
2023-07-03T19:20:33.290Z
SK-logic
<p>Gavin's answer makes a lot of great points, and I don't disagree with them. Furthermore, I'm going to risk repeating some of his closing advice, but I hope I included at least some context that makes the answer worthwhile to read.</p>
<p>If you end up deciding you want to marry your compiler to the LSP, or if you even just want to make your compiler amenable to being used in editor/IDE tooling, then I would take a look at some of the design and implementation of rust-analyzer. While it isn't actually a compiler itself, it implements a Rust frontend that is both <strong>incremental</strong> and <strong>resilient</strong>. These traits are important because when used in conjunction with an editor we always want our tooling to respond as <em>quickly</em> and <em>as completely as possible</em>, as opposed to failing to check/format/build later code for the sole reason of an earlier, unrelated piece of code not being well-formed.</p>
<p><a href="https://github.com/rust-lang/rust-analyzer/blob/master/docs/dev/architecture.md" rel="noreferrer">rust-analyzer's dev docs on syntax</a> describe some of the design that make it very resilient for parsing. One of the rust-analyzer maintainers has <a href="https://matklad.github.io/2023/05/21/resilient-ll-parsing-tutorial.html" rel="noreferrer">recently blogged about resilient parser implementations</a>. The same maintainer has also worked on an incremental computation framework for Rust called <a href="https://salsa-rs.github.io/salsa/" rel="noreferrer">salsa</a>, which could be interesting to look at. <a href="https://ollef.github.io/blog/posts/query-based-compilers.html" rel="noreferrer">Here is a blog post on query-based compiler architectures</a> that makes reference to both salsa and using a compiler in language server tooling.</p>

CC BY-SA 4.0
2023-05-27T23:07:43.450Z
2023-07-02T18:19:07.957Z
2023-07-02T18:19:07.957Z
quiet_laika
CC BY-SA 4.0
2023-05-28T06:44:08.347Z
And thank you so much for the links! I will be reading deep for the next few days.
Gavin D. Howard
<p>That said, after deciding that <a href="https://gavinhoward.com/2023/02/justifying-a-backwards-design-decision-for-yao/" rel="nofollow noreferrer">LSP would essentially be required for my language</a>, I had to figure this out. Here is my list of factors.</p>
<h2>Performance</h2>
<p>A user expects an LSP server to respond after every keypress. Typing 120 words a minute means that, with five characters per word and one space, they type 12 characters per second. Let's round that down to 10.</p>
<p>This means your server has to respond within <em>100 milliseconds</em>.</p>
<p>That's a tall order for a compiler.</p>
<p>This is the biggest reason that compiler-based LSP servers don't really happen; making batch-based compilers fast enough to work that fast is <em>hard</em>.</p>
<p>Why? Because LSP is an entirely different use case than compilation. Compilation is batch; LSP is interactive. Compilation is all-at-once; LSP is incremental.</p>
<p>This mismatch is fundamental.</p>
<p>Of course, the compiler doesn't have to do everything it normally does for LSP; it just has to parse and validate. So it is possible, but it's hard.</p>
<p>Often, based on the design of the language, it makes more sense to have a separate program anyway.</p>
<p>Which brings me to...</p>
<h2>Language Design</h2>
<p>Language design matters a lot.</p>
<p>If your compiler is multi-pass, that's going to hurt in several ways.</p>
<p>First, it's hard to implement incremental compilation in a multi-pass compiler. I tried. My language design changed because of this.</p>
<p>Second, multiple passes means more time taken.</p>
<p>Third, multiple passes means that information is most likely not local.</p>
<h2>How to Do It</h2>
<p>So let's assume that you are unwilling to change your language design for LSP. What do you do?</p>
<p>Easy: <strong>don't do it.</strong></p>
<p>I suggest you actually plan on writing an LSP server that is separate from the compiler. Really. Just plan on it. Their use cases are completely different and require different architectures in most cases.</p>
<p>Save yourself the pain and just write two different programs.</p>
<p>(<strong>Edit</strong>: Of course, you could still do the 20% effort upfront to get the compiler 80% of the way there and then share code between them. But that work does have to happen upfront.)</p>
<p>But if you plan on changing your language design like I did, here's what to do:</p>
<ul>
<li>Make your language single-pass.</li>
<li>Make sure your compiler can continue after parse errors. This means having a way to recover. Usually, this means looking for the next semicolon or right brace and start again from there.</li>
<li>Make sure your compiler can generate semantic information, such as labeling what is a function, what is a type, etc.</li>
<li>Make sure your compiler stores location and span information for every token.</li>
<li>Make it possible for each compilation unit to be standalone and not depend on other compilation units. Yes, this is what C/C++ did better than everyone else. (This has the benefit of preventing your compiler from becoming a build system and making it easier to build in parallel.)</li>
<li>Don't require header files; that's slow. Your compiler should be able to take in that information some other way because in LSP mode, it should store that information and use it as-is without touching disk.</li>
<li>If you can make it possible to restart parsing from random spots, do it. For example, if you can store the order in which the functions were defined, then your server could figure out what functions came before the location of a change, delete them, and reparse from the change.</li>
</ul>

CC BY-SA 4.0
2023-05-21T21:46:21.737Z
2023-07-07T21:05:41.823Z
2023-07-07T21:05:41.823Z
Gavin D. Howard
CC BY-SA 4.0
2023-07-02T18:43:18.157Z
I think “make your language single-pass” is actively counterproductive advice. In addition to simply being an unrealistic goal for any serious modern programming language, it makes things *harder* to incrementalize because you can’t choose to just run some of the passes. You don’t want to be paying for optimization and code generation on every single edit to your program, so it benefits you to split those out into separate passes. That way, you can just run the first few passes (parsing and typechecking) to get results more quickly.
Alexis King
CC BY-SA 4.0
2023-07-02T19:04:36.900Z
@AlexisKing by single pass, I mean one *parsing* pass over the *compilation unit*, not one pass to get the results. In other words, I meant one pass like C is, not one pass like smashing parsing, code gen, and optimization into one mess. As for why it needs to be one pass like C, well, I have the implementation scars of a long-dead multi-pass version that didn't work. The current version has four passes: parsing, type-checking, optimization, and machine code gen. But it only does one *parsing* pass unlike a lot of other languages.
Gavin D. Howard
CC BY-SA 4.0
2023-07-02T19:07:23.250Z
No modern C compilers are one-pass in that sense. It is not a realistic way to engineer a serious modern compiler. I can believe that you managed to get one working for a small hobby project, but I don’t think it is good advice to give to other people.
Alexis King
CC BY-SA 4.0
2023-07-02T19:16:58.243Z
@AlexisKing I appreciate your knowledge and your blog (I'm a reader), so I'd like you to explain why it's not a realistic way to engineer a serious modern compiler since this is the second compiler I've implemented that way. I assume the answer will take more space than a comment, so I'll post a question within an hour or so.
Gavin D. Howard
CC BY-SA 4.0
2023-07-02T20:03:54.463Z
Please do! I’d be happy to answer it.
Alexis King
# How would you design a compiler upfront to be both a compiler and a LSP server?
Posted by: username_0 on 2023-05-21
For most modern programming languages, an LSP ([Language Server Protocol](https://en.wikipedia.org/wiki/Language_Server_Protocol)) server needs to be developed, which requires reusing/rewriting some of the existing code that is already used by the compiler. So, how can a compiler be designed to accommodate this need from the start?

Credit to gavinhoward for [the idea](https://area51.stackexchange.com/proposals/127456/programming-language-design-and-implementation/127493#127493) on Area 51.
**Tags:** compilers language-server-protocol
---
## 3 answers
### Answer by username_3 on 2023-05-21. Score: 11 | Accepted answer
That said, after deciding that [LSP would essentially be required for my language](https://gavinhoward.com/2023/02/justifying-a-backwards-design-decision-for-yao/), I had to figure this out. Here is my list of factors.

## Performance

A user expects an LSP server to respond after every keypress. Typing 120 words a minute means that, with five characters per word and one space, they type 12 characters per second. Let's round that down to 10.

This means your server has to respond within *100 milliseconds*.

That's a tall order for a compiler.

This is the biggest reason that compiler-based LSP servers don't really happen; making batch-based compilers fast enough to work that fast is *hard*.

Why? Because LSP is an entirely different use case than compilation. Compilation is batch; LSP is interactive. Compilation is all-at-once; LSP is incremental.

This mismatch is fundamental.

Of course, the compiler doesn't have to do everything it normally does for LSP; it just has to parse and validate. So it is possible, but it's hard.

Often, based on the design of the language, it makes more sense to have a separate program anyway.

Which brings me to...

## Language Design

Language design matters a lot.

If your compiler is multi-pass, that's going to hurt in several ways.

First, it's hard to implement incremental compilation in a multi-pass compiler. I tried. My language design changed because of this.

Second, multiple passes means more time taken.

Third, multiple passes means that information is most likely not local.

## How to Do It

So let's assume that you are unwilling to change your language design for LSP. What do you do?

Easy: **don't do it.**

I suggest you actually plan on writing an LSP server that is separate from the compiler. Really. Just plan on it. Their use cases are completely different and require different architectures in most cases.

Save yourself the pain and just write two different programs.

(**Edit**: Of course, you could still do the 20% effort upfront to get the compiler 80% of the way there and then share code between them. But that work does have to happen upfront.)

But if you plan on changing your language design like I did, here's what to do:

* Make your language single-pass.
* Make sure your compiler can continue after parse errors. This means having a way to recover. Usually, this means looking for the next semicolon or right brace and start again from there.
* Make sure your compiler can generate semantic information, such as labeling what is a function, what is a type, etc.
* Make sure your compiler stores location and span information for every token.
* Make it possible for each compilation unit to be standalone and not depend on other compilation units. Yes, this is what C/C++ did better than everyone else. (This has the benefit of preventing your compiler from becoming a build system and making it easier to build in parallel.)
* Don't require header files; that's slow. Your compiler should be able to take in that information some other way because in LSP mode, it should store that information and use it as-is without touching disk.
* If you can make it possible to restart parsing from random spots, do it. For example, if you can store the order in which the functions were defined, then your server could figure out what functions came before the location of a change, delete them, and reparse from the change.
> **Comments:**
> **username_4** (2023-07-02): I think “make your language single-pass” is actively counterproductive advice. In addition to simply being an unrealistic goal for any serious modern programming language, it makes things *harder* to incrementalize because you can’t choose to just run some of the passes. You don’t want to be paying for optimization and code generation on every single edit to your program, so it benefits you to split those out into separate passes. That way, you can just run the first few passes (parsing and typechecking) to get results more quickly.  
> **username_3** (2023-07-02): @AlexisKing by single pass, I mean one *parsing* pass over the *compilation unit*, not one pass to get the results. In other words, I meant one pass like C is, not one pass like smashing parsing, code gen, and optimization into one mess. As for why it needs to be one pass like C, well, I have the implementation scars of a long-dead multi-pass version that didn't work. The current version has four passes: parsing, type-checking, optimization, and machine code gen. But it only does one *parsing* pass unlike a lot of other languages.  
> **username_4** (2023-07-02): No modern C compilers are one-pass in that sense. It is not a realistic way to engineer a serious modern compiler. I can believe that you managed to get one working for a small hobby project, but I don’t think it is good advice to give to other people.  
> **username_3** (2023-07-02): @AlexisKing I appreciate your knowledge and your blog (I'm a reader), so I'd like you to explain why it's not a realistic way to engineer a serious modern compiler since this is the second compiler I've implemented that way. I assume the answer will take more space than a comment, so I'll post a question within an hour or so.  
> **username_4** (2023-07-02): Please do! I’d be happy to answer it.  
---
### Answer by username_2 on 2023-05-27. Score: 8
Gavin's answer makes a lot of great points, and I don't disagree with them. Furthermore, I'm going to risk repeating some of his closing advice, but I hope I included at least some context that makes the answer worthwhile to read.

If you end up deciding you want to marry your compiler to the LSP, or if you even just want to make your compiler amenable to being used in editor/IDE tooling, then I would take a look at some of the design and implementation of rust-analyzer. While it isn't actually a compiler itself, it implements a Rust frontend that is both **incremental** and **resilient**. These traits are important because when used in conjunction with an editor we always want our tooling to respond as *quickly* and *as completely as possible*, as opposed to failing to check/format/build later code for the sole reason of an earlier, unrelated piece of code not being well-formed.

[rust-analyzer's dev docs on syntax](https://github.com/rust-lang/rust-analyzer/blob/master/docs/dev/architecture.md) describe some of the design that make it very resilient for parsing. One of the rust-analyzer maintainers has [recently blogged about resilient parser implementations](https://matklad.github.io/2023/05/21/resilient-ll-parsing-tutorial.html). The same maintainer has also worked on an incremental computation framework for Rust called [salsa](https://salsa-rs.github.io/salsa/), which could be interesting to look at. [Here is a blog post on query-based compiler architectures](https://ollef.github.io/blog/posts/query-based-compilers.html) that makes reference to both salsa and using a compiler in language server tooling.
> **Comments:**
> **username_3** (2023-05-28): And thank you so much for the links! I will be reading deep for the next few days.  
---
### Answer by username_1 on 2023-07-02. Score: 4
One thing you can do to save a lot of time on LSP re-parsing and re-compiling your code on frequent changes: use a parser with memoisation. Packrat, for example, or any other similar approach. This way it'll be very easy to invalidate the parsed region affected by an edit and then reparse the entire unit, whatever it is - your parser should know the boundaries of your units, be it a module, a class, a function, an entire file, etc.

Likewise, you can run further lowering, typing passes, etc., only on things that changed and that depend on what you've changed.

It can all be easily made a part of your compiler, because why would you write two different parsers anyway? A side effect of having syntax highlighting, type hints, pretty-printing and all that as a side effect of compilation is that you can add formatted literate code generation to the compiler.
---
