<p>The article <a href="https://www.cs.cmu.edu/%7Erdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf" rel="noreferrer">&quot;Reflections on Trusting Trust&quot;</a>, published in 1984, shows how it's possible to inject a vulnerability into a self-hosting compiler and cover up your tracks. However, self-hosting compilers are not uncommon.</p>
<p>The Swift core team at one point <a href="https://github.com/apple/swift/blob/2c7b0b22831159396fe0e98e5944e64a483c356e/www/FAQ.rst#do-you-plan-to-rewrite-the-swift-compiler-in-swift" rel="noreferrer">explicitly rejected</a> rewriting the compiler in Swift, but <a href="https://forums.swift.org/t/implementing-parts-of-the-swift-compiler-in-swift/59524" rel="noreferrer">they went back on it</a> a few years later. Zig has a <a href="https://github.com/ziglang/zig/issues/853" rel="noreferrer">bootstrapping process</a> involving a multi-stage compiler to avoid the trust issue that even involves rebuilding llvm from source, but most of the compiler is still written in Zig, so clearly there's want for self-hosting. Other examples of self-hosting compilers include <a href="https://github.com/Microsoft/TypeScript/" rel="noreferrer">TypeScript</a> and <a href="https://gcc.gnu.org/install/prerequisites.html" rel="noreferrer">GCC</a>.</p>
<p>Why are self-hosting languages so popular? What advantages are there over writing a compiler in a different language?</p>

<compilers><self-hosting>
What are the benefits to self-hosting compilers?
CC BY-SA 4.0
2023-05-22T13:08:12.433Z
Related: https://stackoverflow.com/questions/1493747/bootstrapping-a-compiler-why
FireTheLost
<p>Just to add another point, writing a compiler in the same language is an example of the concept of <a href="https://en.wikipedia.org/wiki/Eating_your_own_dog_food" rel="nofollow noreferrer">eating your own dog food</a>. This is where you use your own product to demonstrate its merits.</p>
<p>Self hosting allows you to test the language with a non-trival use case which can make it easy to spot issues and iron out problem areas before others use it for their own programs.</p>

CC BY-SA 4.0
2023-05-22T13:10:53.830Z
2023-05-22T13:10:53.830Z
FireTheLost
CC BY-SA 4.0
2024-11-27T22:03:28.597Z
Isn't this essentially the same as @Nuoji's point #1?
Barmar
<p>In order to implement a language, you need to know how it's supposed to work very well. But you also need expertise in the implementation language when you're writing a complex application. So if they're the same language, you only need expertise in one language, not two.</p>
<p>Counter to this is that the kind of people capable of, and interested in, writing language implementations tend to have broad programming expertise. They've probably been exposed to numerous languages, and can program competently in several of them. So they don't <em>need</em> to use the same language they're implementing.</p>
<p>There are some special cases where self-hosting provides specific benefits. For instance, if you write a Lisp compiler or interpreter in Lisp, you don't have to implement a lexer -- it's the built-in <code>READ</code> function.</p>

CC BY-SA 4.0
2024-11-27T22:11:44.033Z
2024-11-27T22:11:44.033Z
Barmar
<p>The claimed benefits are:</p>
<ol>
<li>By building the compiler in the language, it is tested and gets a strong feedback loop into the language design.</li>
<li>The community will be more enthusiastic in wanting to contribute to your compiler if it's self-hosted.</li>
</ol>
<p>In practice, these points don't hold up to scrutiny:</p>
<ol>
<li>A compiler is a very specialized piece of software. Thus, if your main feedback for your language is your compiler then unless the language is geared towards writing compilers, you have the wrong kind of feedback.</li>
<li>Unless your community is large in the first place, you're unlikely to have people help you out anyway, so a large community comes first, which is the actual tricky part. I've encountered a lot of language designers with self-hosted languages, they didn't automatically have communities helping out.</li>
</ol>
<p>Furthermore there are strong drawbacks:</p>
<ol>
<li>A bootstrap process is required, often meaning that you have the language written in 0.x, that then builds the compiler of 0.x+n. Bootstrapping adds another layer of complexity to the toolchain.</li>
<li>When self-hosting <em>before</em> 1.0, this creates a natural inertia against major changes to the language (regardless of how necessary), which in turn is not beneficial for the language's overall design.</li>
<li>If it is early in the design of the compiler, it is harder for contributors to help, as they need to learn about your language at a point where there is relatively little written about the language.</li>
</ol>
<p>Bootstrapping in itself doesn't need to be bad, but doing it before the language has matured to 1.0 is in general a bad idea. There are exceptions like Zig<sup>1</sup> that made it work, but few languages have the luxury of a large community ready to contribute and maintain more than one compiler at the same time.</p>
<hr />
<p><sup>[1]</sup> Also, Zig's bootstrapping was further motivated by architectural problems with the original C++ codebase. So bootstrapping was not the primary goal, but rather this was addressing a need to rewrite the compiler from scratch. So bootstrapping was more of a side-effect of the rewrite.</p>

CC BY-SA 4.0
2023-05-21T18:09:42.257Z
2023-05-21T18:09:42.257Z
Nuoji
<p>I can certainly tell you what were and were not the principal reasons motivating the rewrite of the C# compiler from C++ to C#. A brief history: C# 1.0 through 6.0 compilers were written in C++. From 1.0 to 3.0, the compiler and IDE shared a parser but had siloed semantic analyzers. During the C# 3.0 timeframe we rearchitected the IDE to use the compiler's semantic analyzer. During the C# 6.0 timeframe we split the team in two, one entirely rewriting the compiler in C#, and one continuing 6.0 feature development in the legacy codebase.</p>
<p>This choice was extraordinarily expensive and risky for a major pillar of Microsoft's developer strategy and internal tooling, so we needed surpassingly good reasons to do it. In no particular order:</p>
<ol>
<li>The C# team was already entirely composed of expert C# programmers. There was no question that we could be productive working in C#.</li>
<li>C# is a general-purpose line-of-business language, strongly suited to the developer tools LOB.</li>
<li>C# was specifically designed to be a natural progression from large C++ codebases, but with better safety and productivity.</li>
<li>The design team had a list of &quot;nice to have&quot; features that customers had been requesting for more than a decade, but that we never had time to implement because of architectural limits of the existing C++ codebase. We strongly felt that we could be more productive in the future and implement more small features in each release if we had a C# codebase to work in. (This strategy worked; each release of C# since has had more QOL features than before C# 6.0.)</li>
<li>Our goal was not rearchitecture for its own sake, but rather to continue the process begun in C# 3.0 to make the lexical, syntactic and semantic analyzers in the compiler <em>services</em> that could be consumed by <em>anyone</em>, not just the IDE team. Internal surveys showed that just within Microsoft there was a strong need for a library of compiler services, and we envisioned creating an industry-wide ecosystem for custom IDE code analysis extensions. The natural language to write those extensions in would be C#, so writing the compiler-as-a-service in C# was a natural choice too.</li>
</ol>
<p>There were other reasons as well but those are sufficient. I'll briefly address Nuoji's points in the accepted answer:</p>
<ol>
<li>Building the compiler in its own language is an excellent test case for the compiler.</li>
</ol>
<p>Indeed it is. We very much wanted the compiler-as-a-service to be able to handle multi-million line programs efficiently. Having the compiler as a test case for the compiler was very helpful. Dogfooding the compiler and IDE every day found performance and correctness problems very quickly.  And having an existing test suite with millions upon millions of compliance tests was also crucial.</p>
<p>Self hosting also drove some minor design decisions. C# is general-purpose and we did not wish to fall into the design trap of over-fitting the design to the needs of the compiler team, but we did find some pain points of our own along the way. For example, Anders had always resisted adding exception filters (present in VB but not C#) but the compiler team found some compelling use cases and made a convincing argument that they should be added to C#.</p>
<ol start="2">
<li>The community is more likely to contribute.</li>
</ol>
<p>Correct. C# already had many millions of users and a robust community engagement process. The Roslyn compiler was open-sourced shortly after I left Microsoft in 2012 (as we had always planned it to be) and the community support over the last decade has been very helpful.</p>
<p>Since we started the rewrite in C# 6.0, after there were already millions of C# developers in the world, Nuoji's points against do not apply to our situation.</p>
<p></p>

CC BY-SA 4.0
2024-11-28T16:33:46.880Z
2024-11-28T16:33:46.880Z
Eric Lippert
CC BY-SA 4.0
2024-11-28T22:56:08.240Z
Points 3 and 4 could be loosely paraphrased as "naturally, we think C# is a good language to write in, so we wrote our compiler in it". In hindsight it seems like an obvious (and great) reason, and I'm surprised none of the other answerers (including me) really thought of it.
kaya3
<p>Some of the reason for self-hosting a compiler is marketing: it shows that your language has reached a level of maturity; the language is capable of expressing a significant piece of software; and the language's implementation is &quot;complete&quot; enough to compile a significant piece of software and have it work. A compiler which can compile itself must work well and not be missing any &quot;major&quot; features. And unlike many other marketing claims, having a self-hosted compiler is an objective claim that's relatively easy to independently verify.</p>
<p>Another reason is to encourage your language's community to contribute improvements to the compiler. If it is written in a different language, then contributors must know that language well enough to write in it, in addition to knowing your language well enough to work on its implementation. If the compiler is written in the same language it compiles, relatively more people will be able to contribute to the project.</p>

CC BY-SA 4.0
2023-05-21T17:21:14.087Z
2023-05-21T17:26:56.153Z
2023-05-21T17:26:56.153Z
kaya3
# What are the benefits to self-hosting compilers?
Posted by: username_0 on 2023-05-21
The article ["Reflections on Trusting Trust"](https://www.cs.cmu.edu/%7Erdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf), published in 1984, shows how it's possible to inject a vulnerability into a self-hosting compiler and cover up your tracks. However, self-hosting compilers are not uncommon.

The Swift core team at one point [explicitly rejected](https://github.com/apple/swift/blob/2c7b0b22831159396fe0e98e5944e64a483c356e/www/FAQ.rst#do-you-plan-to-rewrite-the-swift-compiler-in-swift) rewriting the compiler in Swift, but [they went back on it](https://forums.swift.org/t/implementing-parts-of-the-swift-compiler-in-swift/59524) a few years later. Zig has a [bootstrapping process](https://github.com/ziglang/zig/issues/853) involving a multi-stage compiler to avoid the trust issue that even involves rebuilding llvm from source, but most of the compiler is still written in Zig, so clearly there's want for self-hosting. Other examples of self-hosting compilers include [TypeScript](https://github.com/Microsoft/TypeScript/) and [GCC](https://gcc.gnu.org/install/prerequisites.html).

Why are self-hosting languages so popular? What advantages are there over writing a compiler in a different language?
**Tags:** compilers self-hosting
> **Comments:**
> **username_1**: Related: https://stackoverflow.com/questions/1493747/bootstrapping-a-compiler-why  
---
## 5 answers
### Answer by username_3 on 2023-05-21. Score: 11 | Accepted answer
The claimed benefits are:

1. By building the compiler in the language, it is tested and gets a strong feedback loop into the language design.
2. The community will be more enthusiastic in wanting to contribute to your compiler if it's self-hosted.

In practice, these points don't hold up to scrutiny:

1. A compiler is a very specialized piece of software. Thus, if your main feedback for your language is your compiler then unless the language is geared towards writing compilers, you have the wrong kind of feedback.
2. Unless your community is large in the first place, you're unlikely to have people help you out anyway, so a large community comes first, which is the actual tricky part. I've encountered a lot of language designers with self-hosted languages, they didn't automatically have communities helping out.

Furthermore there are strong drawbacks:

1. A bootstrap process is required, often meaning that you have the language written in 0.x, that then builds the compiler of 0.x+n. Bootstrapping adds another layer of complexity to the toolchain.
2. When self-hosting *before* 1.0, this creates a natural inertia against major changes to the language (regardless of how necessary), which in turn is not beneficial for the language's overall design.
3. If it is early in the design of the compiler, it is harder for contributors to help, as they need to learn about your language at a point where there is relatively little written about the language.

Bootstrapping in itself doesn't need to be bad, but doing it before the language has matured to 1.0 is in general a bad idea. There are exceptions like Zig1 that made it work, but few languages have the luxury of a large community ready to contribute and maintain more than one compiler at the same time.

---

[1] Also, Zig's bootstrapping was further motivated by architectural problems with the original C++ codebase. So bootstrapping was not the primary goal, but rather this was addressing a need to rewrite the compiler from scratch. So bootstrapping was more of a side-effect of the rewrite.
---
### Answer by username_4 on 2024-11-28. Score: 11
I can certainly tell you what were and were not the principal reasons motivating the rewrite of the C# compiler from C++ to C#. A brief history: C# 1.0 through 6.0 compilers were written in C++. From 1.0 to 3.0, the compiler and IDE shared a parser but had siloed semantic analyzers. During the C# 3.0 timeframe we rearchitected the IDE to use the compiler's semantic analyzer. During the C# 6.0 timeframe we split the team in two, one entirely rewriting the compiler in C#, and one continuing 6.0 feature development in the legacy codebase.

This choice was extraordinarily expensive and risky for a major pillar of Microsoft's developer strategy and internal tooling, so we needed surpassingly good reasons to do it. In no particular order:

1. The C# team was already entirely composed of expert C# programmers. There was no question that we could be productive working in C#.
2. C# is a general-purpose line-of-business language, strongly suited to the developer tools LOB.
3. C# was specifically designed to be a natural progression from large C++ codebases, but with better safety and productivity.
4. The design team had a list of "nice to have" features that customers had been requesting for more than a decade, but that we never had time to implement because of architectural limits of the existing C++ codebase. We strongly felt that we could be more productive in the future and implement more small features in each release if we had a C# codebase to work in. (This strategy worked; each release of C# since has had more QOL features than before C# 6.0.)
5. Our goal was not rearchitecture for its own sake, but rather to continue the process begun in C# 3.0 to make the lexical, syntactic and semantic analyzers in the compiler *services* that could be consumed by *anyone*, not just the IDE team. Internal surveys showed that just within Microsoft there was a strong need for a library of compiler services, and we envisioned creating an industry-wide ecosystem for custom IDE code analysis extensions. The natural language to write those extensions in would be C#, so writing the compiler-as-a-service in C# was a natural choice too.

There were other reasons as well but those are sufficient. I'll briefly address Nuoji's points in the accepted answer:

1. Building the compiler in its own language is an excellent test case for the compiler.

Indeed it is. We very much wanted the compiler-as-a-service to be able to handle multi-million line programs efficiently. Having the compiler as a test case for the compiler was very helpful. Dogfooding the compiler and IDE every day found performance and correctness problems very quickly. And having an existing test suite with millions upon millions of compliance tests was also crucial.

Self hosting also drove some minor design decisions. C# is general-purpose and we did not wish to fall into the design trap of over-fitting the design to the needs of the compiler team, but we did find some pain points of our own along the way. For example, Anders had always resisted adding exception filters (present in VB but not C#) but the compiler team found some compelling use cases and made a convincing argument that they should be added to C#.

2. The community is more likely to contribute.

Correct. C# already had many millions of users and a robust community engagement process. The Roslyn compiler was open-sourced shortly after I left Microsoft in 2012 (as we had always planned it to be) and the community support over the last decade has been very helpful.

Since we started the rewrite in C# 6.0, after there were already millions of C# developers in the world, Nuoji's points against do not apply to our situation.
> **Comments:**
> **username_5** (2024-11-28): Points 3 and 4 could be loosely paraphrased as "naturally, we think C# is a good language to write in, so we wrote our compiler in it". In hindsight it seems like an obvious (and great) reason, and I'm surprised none of the other answerers (including me) really thought of it.  
---
### Answer by username_5 on 2023-05-21. Score: 6
Some of the reason for self-hosting a compiler is marketing: it shows that your language has reached a level of maturity; the language is capable of expressing a significant piece of software; and the language's implementation is "complete" enough to compile a significant piece of software and have it work. A compiler which can compile itself must work well and not be missing any "major" features. And unlike many other marketing claims, having a self-hosted compiler is an objective claim that's relatively easy to independently verify.

Another reason is to encourage your language's community to contribute improvements to the compiler. If it is written in a different language, then contributors must know that language well enough to write in it, in addition to knowing your language well enough to work on its implementation. If the compiler is written in the same language it compiles, relatively more people will be able to contribute to the project.
---
### Answer by username_1 on 2023-05-22. Score: 1
Just to add another point, writing a compiler in the same language is an example of the concept of [eating your own dog food](https://en.wikipedia.org/wiki/Eating_your_own_dog_food). This is where you use your own product to demonstrate its merits.

Self hosting allows you to test the language with a non-trival use case which can make it easy to spot issues and iron out problem areas before others use it for their own programs.
> **Comments:**
> **username_2** (2024-11-27): Isn't this essentially the same as @Nuoji's point #1?  
---
### Answer by username_2 on 2024-11-27. Score: 0
In order to implement a language, you need to know how it's supposed to work very well. But you also need expertise in the implementation language when you're writing a complex application. So if they're the same language, you only need expertise in one language, not two.

Counter to this is that the kind of people capable of, and interested in, writing language implementations tend to have broad programming expertise. They've probably been exposed to numerous languages, and can program competently in several of them. So they don't *need* to use the same language they're implementing.

There are some special cases where self-hosting provides specific benefits. For instance, if you write a Lisp compiler or interpreter in Lisp, you don't have to implement a lexer -- it's the built-in `READ` function.
---
