<p>What are some testing techniques and strategies (in software engineering sense) that are relevant to compilers/interpreters? For example, having a set of complete programs and verifying their runtime behavior would be an example of an integration test.</p>

<implementation><verification><software-engineering>
How to test a compiler/interpreter?
CC BY-SA 4.0
2023-06-15T23:09:22.383Z
I'm not sure this is on topic. Do we expect testing strategies for compilers to fundamentally differ from testing strategies for non-compiler projects? Either way, this question is a bit broad in scope and could be improved by focusing on a more specific question such as how a particular language was tested or the application of a specific testing technique to programming language projects.
Isaiah
CC BY-SA 4.0
2023-06-15T23:13:43.187Z
@Isaiah Yes, testing strategies for compilers are often significantly different to those for other pieces of software, partly because they are designed as a pipeline of several stages where the intermediate representations are inconvenient to construct for testing purposes, and unstable (due to the expected transformations of them being unspecified), and partly because programming languages moreso than other projects tend to have detailed formal specifications which can be tested against. So I think the question is on-topic, though I agree it might be too broad.
kaya3
CC BY-SA 4.0
2023-06-15T23:24:12.383Z
I think this question is necessarily broad. There's no reason to close it as needing focus; the scope is perfectly clear, it's perfectly answerable, and it's useful and appropriate for the site. Anyone trying to close it is, IMO, closing to close, unless I'm missing something. Voting to leave open.
rydwolf
CC BY-SA 4.0
2023-06-15T23:26:07.520Z
I don't think it would be impossible to make the question narrower. For a start, it could ask about just compilers or just interpreters. Or it could identify a specific issue that makes compilers/interpreters hard to test with conventional strategies, and ask about alternative strategies which address that issue.
kaya3
CC BY-SA 4.0
2023-06-16T01:25:28.457Z
The answer to this would be a whole series of books; it's just too broad to be usefully answerable as currently posed, but compiler-testing techniques are certainly on topic. "What are some ... that are relevant to ..." is an exceptionally wide-ranging structure, though. It could be refined into several more constrained on-topic questions (what do I need to remember to test? how can I test property X? how do I structure a compiler for confirming these properties? specific questions for compilers and interpreters; for (un)typed systems; for paradigms; for diagnostics; for a concrete problem).
Michael Homer
CC BY-SA 4.0
2023-06-16T07:16:37.690Z
I think its fair for a broad question to have a broad answer. The whole series of books is not required. Valid answers need be closer to titles of chapters with a little explanation very much like the answer by Pseudonym.
The broad Q&A informs the discussion and could lead to more specific & narrow questions in the future.
Bruce Adams
CC BY-SA 4.0
2023-06-16T08:52:07.777Z
@Isaiah compiler testing is indeed special - there is a lot of very specific techniques, see CSmith for example of such.
SK-logic
CC BY-SA 4.0
2023-06-18T20:40:49.413Z
See also [How can a compiler's parse errors be tested?](https://languagedesign.stackexchange.com/q/1558/285)
Bruce Adams
<p>I'm going to talk about compilers, although some of this advice also applies to interpreters.</p>
<p>Given a test program, and a potentially buggy compiler, there are essentially four questions that you can ask:</p>
<ol>
<li>Does this piece of test code correctly compile, or correctly fail to compile, under certain circumstances?</li>
<li>Does the compiler produce the correct/expected diagnostic output (i.e. errors or warnings), under certain circumstances?</li>
<li>Does the program, when compiled under certain circumstances, do the right thing when run?</li>
<li>Does the compiler, under certain circumstances, produce <em>expected</em> code?</li>
</ol>
<p>That last one is, by far, the hardest one to test, and you probably shouldn't bother, except for one scenario that I will mention last.</p>
<p>The &quot;under certain circumstances&quot; qualification is important in the presence of flags that control the compiler (e.g. various warnings turned on or off, language variants, specific optimisations turned on or off, debug symbols, etc).</p>
<p>Remember that optimisations, in particular, can interact in nontrivial ways, and you may have written some optimisation pass in such a way that it <em>assumes</em> that a previous pass has already run. This often happens without realising it.</p>
<p>Some specific advice follows.</p>
<h2>Conformance suites</h2>
<p>If your language has some standard or de facto standard conformance tests, then this is obviously the gold standard for testing. But you probably don't need to test against a conformance suite during day-to-day development if the tests are taking a while to run.</p>
<h2>Reported bugs</h2>
<p>This should go without saying, but I'm going to say it anyway: every bug report you get should be turned into part of your compiler's test suite. Of course those tests may need to be maintained as the language changes, errors and warnings get reworded, and so on.</p>
<h2>Test diagnostics too</h2>
<p>I mentioned this above, but I'm making it a section just so you don't forget it. Your tests should include programs that <em>don't</em> compile, or don't compile <em>cleanly</em>.</p>
<p>Error messages and warnings are, to a large extent, the user interface of a compiler. It's what your users will see the most. They will spend much more time looking at diagnostics than they will looking at generated code or even the command-line options of the compiler. Test them.</p>
<h2>Nondeterministic compilers are less testable</h2>
<p>This should be obvious, but you'd be surprised at how often it doesn't happen: The same compiler, with the same command-line flags, compiling the same program, should produce the same output.</p>
<p>Yes, there are some exceptions, such as timestamps, that are unavoidable. We'll assume that you have a way to navigate around these.</p>
<p>This is not only for your benefit, but also for your users' benefit. Programmers tend to like it when their programs are patchable; if they need to ship a bug fix, only the files that changed should need to be shipped. If your compiler produces different binaries on different runs, then any program produced by your compiler is less amenable to being patched.</p>
<p>This doesn't mean you can't, say, use the address of an object as a key within your compiler, as long as that information doesn't &quot;leak&quot; into the object code.</p>
<h2>What a test case looks like</h2>
<p>The three files that make a typical test case are:</p>
<ol>
<li>The program.</li>
<li>The expected output from the <em>compiler</em> (i.e. what it displays to <code>stdout</code>/<code>stderr</code>).</li>
<li>The expected output from the <em>compiled program</em>, when it is run.</li>
</ol>
<p>A tool such as <code>diff</code> is your friend in navigating the result of your test suite.</p>
<p>One thing that is almost <em>never</em> in a test case is the expected generated code. There are several reasons for this, but the most important ones are:</p>
<ul>
<li>It is inherently non-portable across platforms.</li>
<li>Modifying the back end in any non-trivial way way almost always invalidates <em>all</em> such tests. Yes, <em>all</em> of them.</li>
</ul>
<p>However, there is one case where testing the literal generated code makes sense, and that is...</p>
<h2>Bootstrap testing</h2>
<p>If the compiler is self-hosted, then you do have a reference from which you can test the literal compiled code. The procedure goes like this:</p>
<ol>
<li>Compile the compiler. Call the result &quot;A&quot;.</li>
<li>Re-compile the compiler using the compiler &quot;A&quot;. Call the result &quot;B&quot;.</li>
<li>Re-compile the compiler using the compiler &quot;B&quot;. Call the result &quot;C&quot;.</li>
</ol>
<p>Then &quot;B&quot; and &quot;C&quot; should, if your compiler is deterministic, be bit-for-bit identical. You should test this before any release.</p>

CC BY-SA 4.0
2023-06-16T00:34:06.177Z
2023-06-16T00:46:07.910Z
2023-06-16T00:46:07.910Z
Pseudonym
CC BY-SA 4.0
2023-06-16T07:23:40.903Z
Good answer. Two things to add. 
While a complete conformance suite may take a long time to run a well thought out subset probably should be run more frequently but this is the same for any other big testsuite.
Negative test cases are critical. If a system is learning by induction it cannot learn a model form positive test cases alone. This is still true if the system includes a human.
Bruce Adams
# How to test a compiler/interpreter?
Posted by: username_0 on 2023-06-15
What are some testing techniques and strategies (in software engineering sense) that are relevant to compilers/interpreters? For example, having a set of complete programs and verifying their runtime behavior would be an example of an integration test.
**Tags:** implementation verification software-engineering
> **Comments:**
> **username_1**: I'm not sure this is on topic. Do we expect testing strategies for compilers to fundamentally differ from testing strategies for non-compiler projects? Either way, this question is a bit broad in scope and could be improved by focusing on a more specific question such as how a particular language was tested or the application of a specific testing technique to programming language projects.  
> **username_2**: @Isaiah Yes, testing strategies for compilers are often significantly different to those for other pieces of software, partly because they are designed as a pipeline of several stages where the intermediate representations are inconvenient to construct for testing purposes, and unstable (due to the expected transformations of them being unspecified), and partly because programming languages moreso than other projects tend to have detailed formal specifications which can be tested against. So I think the question is on-topic, though I agree it might be too broad.  
> **username_3**: I think this question is necessarily broad. There's no reason to close it as needing focus; the scope is perfectly clear, it's perfectly answerable, and it's useful and appropriate for the site. Anyone trying to close it is, IMO, closing to close, unless I'm missing something. Voting to leave open.  
> **username_2**: I don't think it would be impossible to make the question narrower. For a start, it could ask about just compilers or just interpreters. Or it could identify a specific issue that makes compilers/interpreters hard to test with conventional strategies, and ask about alternative strategies which address that issue.  
> **username_4**: The answer to this would be a whole series of books; it's just too broad to be usefully answerable as currently posed, but compiler-testing techniques are certainly on topic. "What are some ... that are relevant to ..." is an exceptionally wide-ranging structure, though. It could be refined into several more constrained on-topic questions (what do I need to remember to test? how can I test property X? how do I structure a compiler for confirming these properties? specific questions for compilers and interpreters; for (un)typed systems; for paradigms; for diagnostics; for a concrete problem).  
> **username_5**: I think its fair for a broad question to have a broad answer. The whole series of books is not required. Valid answers need be closer to titles of chapters with a little explanation very much like the answer by Pseudonym.
The broad Q&A informs the discussion and could lead to more specific & narrow questions in the future.  
> **username_6**: @Isaiah compiler testing is indeed special - there is a lot of very specific techniques, see CSmith for example of such.  
> **username_5**: See also [How can a compiler's parse errors be tested?](https://languagedesign.stackexchange.com/q/1558/285)  
---
## 1 answers
### Answer by username_7 on 2023-06-16. Score: 11 | Accepted answer
I'm going to talk about compilers, although some of this advice also applies to interpreters.

Given a test program, and a potentially buggy compiler, there are essentially four questions that you can ask:

1. Does this piece of test code correctly compile, or correctly fail to compile, under certain circumstances?
2. Does the compiler produce the correct/expected diagnostic output (i.e. errors or warnings), under certain circumstances?
3. Does the program, when compiled under certain circumstances, do the right thing when run?
4. Does the compiler, under certain circumstances, produce *expected* code?

That last one is, by far, the hardest one to test, and you probably shouldn't bother, except for one scenario that I will mention last.

The "under certain circumstances" qualification is important in the presence of flags that control the compiler (e.g. various warnings turned on or off, language variants, specific optimisations turned on or off, debug symbols, etc).

Remember that optimisations, in particular, can interact in nontrivial ways, and you may have written some optimisation pass in such a way that it *assumes* that a previous pass has already run. This often happens without realising it.

Some specific advice follows.

## Conformance suites

If your language has some standard or de facto standard conformance tests, then this is obviously the gold standard for testing. But you probably don't need to test against a conformance suite during day-to-day development if the tests are taking a while to run.

## Reported bugs

This should go without saying, but I'm going to say it anyway: every bug report you get should be turned into part of your compiler's test suite. Of course those tests may need to be maintained as the language changes, errors and warnings get reworded, and so on.

## Test diagnostics too

I mentioned this above, but I'm making it a section just so you don't forget it. Your tests should include programs that *don't* compile, or don't compile *cleanly*.

Error messages and warnings are, to a large extent, the user interface of a compiler. It's what your users will see the most. They will spend much more time looking at diagnostics than they will looking at generated code or even the command-line options of the compiler. Test them.

## Nondeterministic compilers are less testable

This should be obvious, but you'd be surprised at how often it doesn't happen: The same compiler, with the same command-line flags, compiling the same program, should produce the same output.

Yes, there are some exceptions, such as timestamps, that are unavoidable. We'll assume that you have a way to navigate around these.

This is not only for your benefit, but also for your users' benefit. Programmers tend to like it when their programs are patchable; if they need to ship a bug fix, only the files that changed should need to be shipped. If your compiler produces different binaries on different runs, then any program produced by your compiler is less amenable to being patched.

This doesn't mean you can't, say, use the address of an object as a key within your compiler, as long as that information doesn't "leak" into the object code.

## What a test case looks like

The three files that make a typical test case are:

1. The program.
2. The expected output from the *compiler* (i.e. what it displays to `stdout`/`stderr`).
3. The expected output from the *compiled program*, when it is run.

A tool such as `diff` is your friend in navigating the result of your test suite.

One thing that is almost *never* in a test case is the expected generated code. There are several reasons for this, but the most important ones are:

* It is inherently non-portable across platforms.
* Modifying the back end in any non-trivial way way almost always invalidates *all* such tests. Yes, *all* of them.

However, there is one case where testing the literal generated code makes sense, and that is...

## Bootstrap testing

If the compiler is self-hosted, then you do have a reference from which you can test the literal compiled code. The procedure goes like this:

1. Compile the compiler. Call the result "A".
2. Re-compile the compiler using the compiler "A". Call the result "B".
3. Re-compile the compiler using the compiler "B". Call the result "C".

Then "B" and "C" should, if your compiler is deterministic, be bit-for-bit identical. You should test this before any release.
> **Comments:**
> **username_5** (2023-06-16): Good answer. Two things to add. 
While a complete conformance suite may take a long time to run a well thought out subset probably should be run more frequently but this is the same for any other big testsuite.
Negative test cases are critical. If a system is learning by induction it cannot learn a model form positive test cases alone. This is still true if the system includes a human.  
---
