<p>I've written a compiler, and it currently outputs assembly. I'd like to make it so that when I assemble it into a final binary, I can pass the binary to a debugger and view it with source mappings, so that debugging can be performed line-by-line.</p>
<p>How does the source file content and program structure/semantics get provided to a debugger so that it can properly step through statements, set breakpoints, etc?</p>

<compilers>
When compiling to binary, how do compilers provide debug symbols/source maps and context?
CC BY-SA 4.0
2023-05-22T14:14:58.310Z
Not really applicable to the OP's case, but the Java compiler puts debugging information, etc. into specially tagged parts of the binary
Seggan
CC BY-SA 4.0
2023-05-22T14:23:08.710Z
Yeah, interpreted languages have an easier time of it.
springogeek
CC BY-SA 4.0
2023-05-22T14:32:35.790Z
it works the same in compiled languages (at least in linux ELF). It also has sections specifically for debug-related data. I'm not positive all the debug data is in there or if some is in other locations
mousetail 'he-him'
CC BY-SA 4.0
2023-05-22T14:52:10.327Z
I will die proving Java is not interpreted lol
Seggan
CC BY-SA 4.0
2023-05-22T15:14:01.247Z
@Seggan It isn't really the point of the question so I'd prefer not to debate it here.
springogeek
<p>I'll focus on the linux-specific ELF binary file format, which is what you'll get after assembling. DWARF (<a href="https://dwarfstd.org/" rel="noreferrer">https://dwarfstd.org/</a>) is the name of a very common debugging information format that is typically used with ELF. The goal is to tell a debugger how the binary data relates to the original source data.</p>
<p>You can - of course - investigate how existing compilers emit this data, e.g. <a href="https://godbolt.org/z/GPvv1vT1T" rel="noreferrer">https://godbolt.org/z/GPvv1vT1T</a>
So, you can attach your debugging information by adding the necessary directives in your assembly.
If you use LLVM, the easiest approach would be to use their DBG api: <a href="https://llvm.org/docs/SourceLevelDebugging.html" rel="noreferrer">https://llvm.org/docs/SourceLevelDebugging.html</a></p>

CC BY-SA 4.0
2023-05-23T11:57:58.250Z
2023-05-23T11:57:58.250Z
NaCl
<h1>What &quot;debug-info&quot; to maintain and pass to the debugger?</h1>
<p>Anything which is present in the source but not necessary to run the program. But most commonly:</p>
<ul>
<li><strong>Debug symbols</strong>: maps registers and obfuscated variable/method names to their names in the source code</li>
<li><strong>Source-map</strong>: maps compiled code (<a href="https://en.wikipedia.org/wiki/Bytecode" rel="nofollow noreferrer">bytecode</a>, <a href="https://en.wikipedia.org/wiki/Source-to-source_compiler" rel="nofollow noreferrer">transpiled code</a>, or <a href="https://en.wikipedia.org/wiki/Obfuscation_(software)" rel="nofollow noreferrer">obfuscated code</a>) to the original source code.
<ul>
<li>This can be done at different levels of granularity, but most compilers more or less either map lines of bytecode to every line of source code (such as in C or <a href="https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.7.12" rel="nofollow noreferrer">Java</a>), or map arbitrary ranges (such as in <a href="https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/edit?pli=1" rel="nofollow noreferrer">JavaScript</a>).</li>
<li>A powerful enough source map can completely replace debug symbols, since you can just map every compiled variable to its original name in the source.</li>
<li>However, keep in mind that the more fine-grained the source mapping is, the harder it is to track within the compiler, and often source code doesn't map so close to the compiled output anyways. In particular, release-mode optimizations very often change order of instructions, elide variables, cache operations, and more.</li>
</ul>
</li>
<li><strong>Original source filename</strong>: technically part of the source-map. This along with line numbers are essential for the user to set breakpoints.</li>
<li><strong>Debug-info version</strong></li>
</ul>
<p><em>For a new language, I recommend providing debug symbols, a line number source-map, the original source filename, and a debug-info version</em>. These are both relatively easy to implement in the compiler and relatively easy to use in the debugger, and they enable &quot;breakpoints&quot; and &quot;step in/out/over&quot; within the compiled code (they also enable &quot;watchpoints&quot; if you test a variable on every instruction, but that can be expensive).</p>
<h1>How to maintain debug-info throughout compilation?</h1>
<p>This will depend on your compiler design, but generally, just make sure that your AST and each of your IRs are annotated with source ranges. Any decent parser-generator including <a href="https://www.antlr.org/" rel="nofollow noreferrer">ANTLR4</a> and <a href="https://tree-sitter.github.io/tree-sitter/" rel="nofollow noreferrer">tree-sitter</a> will preserve source ranges on syntax nodes, and it's not hard to do for a hand-rolled parser. Then, when translating the AST into a new representation (for example, bytecode instructions), just make sure to propagate the source ranges. From the source ranges, you can get the original names of variables and line number of instructions.</p>
<h1>Store debug-info in the binary or separately?</h1>
<p>Some compilers add the debug symbols and the source-map to the binary, while others provide them in a separate file. The benefits of embedded debug-info are less clutter and chance of desyncing / losing the debug symbols, the benefits of separate debug-info are that you can withhold the info from end-users resulting in smaller binaries and obfuscation.</p>
<p>In JavaScript, CSS, and some other languages, source-maps are provided a separate <code>.map</code> file (where <code>foo.map</code> is the source map for <code>foo</code>). In Java, the source-map and debug symbols are provided directly in the class file (<a href="https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html" rel="nofollow noreferrer">Java Class File Format</a>). C and C++, and any other language which produces object code supports <em>both</em>: debug symbols and source-maps can be embedded in the compiled output (<code>.o</code>, <code>.a</code>, <code>.dylib</code>, etc.) <em>or</em> in a separate <code>.dSYM</code> file, usually via a compiler flag (read more in <a href="https://stackoverflow.com/a/22460268">this answer</a>)</p>

CC BY-SA 4.0
2023-05-23T12:54:22.517Z
2023-05-23T12:54:22.517Z
tarzh
# When compiling to binary, how do compilers provide debug symbols/source maps and context?
Posted by: username_0 on 2023-05-22
I've written a compiler, and it currently outputs assembly. I'd like to make it so that when I assemble it into a final binary, I can pass the binary to a debugger and view it with source mappings, so that debugging can be performed line-by-line.

How does the source file content and program structure/semantics get provided to a debugger so that it can properly step through statements, set breakpoints, etc?
**Tags:** compilers
> **Comments:**
> **username_1**: Not really applicable to the OP's case, but the Java compiler puts debugging information, etc. into specially tagged parts of the binary  
> **username_0**: Yeah, interpreted languages have an easier time of it.  
> **username_2**: it works the same in compiled languages (at least in linux ELF). It also has sections specifically for debug-related data. I'm not positive all the debug data is in there or if some is in other locations  
> **username_1**: I will die proving Java is not interpreted lol  
> **username_0**: @Seggan It isn't really the point of the question so I'd prefer not to debate it here.  
---
## 2 answers
### Answer by username_3 on 2023-05-23. Score: 5
I'll focus on the linux-specific ELF binary file format, which is what you'll get after assembling. DWARF (<https://dwarfstd.org/>) is the name of a very common debugging information format that is typically used with ELF. The goal is to tell a debugger how the binary data relates to the original source data.

You can - of course - investigate how existing compilers emit this data, e.g. <https://godbolt.org/z/GPvv1vT1T>
So, you can attach your debugging information by adding the necessary directives in your assembly.
If you use LLVM, the easiest approach would be to use their DBG api: <https://llvm.org/docs/SourceLevelDebugging.html>
---
### Answer by username_4 on 2023-05-23. Score: 4
# What "debug-info" to maintain and pass to the debugger?

Anything which is present in the source but not necessary to run the program. But most commonly:

* **Debug symbols**: maps registers and obfuscated variable/method names to their names in the source code
* **Source-map**: maps compiled code ([bytecode](https://en.wikipedia.org/wiki/Bytecode), [transpiled code](https://en.wikipedia.org/wiki/Source-to-source_compiler), or [obfuscated code](https://en.wikipedia.org/wiki/Obfuscation_(software))) to the original source code.
  + This can be done at different levels of granularity, but most compilers more or less either map lines of bytecode to every line of source code (such as in C or [Java](https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.7.12)), or map arbitrary ranges (such as in [JavaScript](https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/edit?pli=1)).
  + A powerful enough source map can completely replace debug symbols, since you can just map every compiled variable to its original name in the source.
  + However, keep in mind that the more fine-grained the source mapping is, the harder it is to track within the compiler, and often source code doesn't map so close to the compiled output anyways. In particular, release-mode optimizations very often change order of instructions, elide variables, cache operations, and more.
* **Original source filename**: technically part of the source-map. This along with line numbers are essential for the user to set breakpoints.
* **Debug-info version**

*For a new language, I recommend providing debug symbols, a line number source-map, the original source filename, and a debug-info version*. These are both relatively easy to implement in the compiler and relatively easy to use in the debugger, and they enable "breakpoints" and "step in/out/over" within the compiled code (they also enable "watchpoints" if you test a variable on every instruction, but that can be expensive).

# How to maintain debug-info throughout compilation?

This will depend on your compiler design, but generally, just make sure that your AST and each of your IRs are annotated with source ranges. Any decent parser-generator including [ANTLR4](https://www.antlr.org/) and [tree-sitter](https://tree-sitter.github.io/tree-sitter/) will preserve source ranges on syntax nodes, and it's not hard to do for a hand-rolled parser. Then, when translating the AST into a new representation (for example, bytecode instructions), just make sure to propagate the source ranges. From the source ranges, you can get the original names of variables and line number of instructions.

# Store debug-info in the binary or separately?

Some compilers add the debug symbols and the source-map to the binary, while others provide them in a separate file. The benefits of embedded debug-info are less clutter and chance of desyncing / losing the debug symbols, the benefits of separate debug-info are that you can withhold the info from end-users resulting in smaller binaries and obfuscation.

In JavaScript, CSS, and some other languages, source-maps are provided a separate `.map` file (where `foo.map` is the source map for `foo`). In Java, the source-map and debug symbols are provided directly in the class file ([Java Class File Format](https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html)). C and C++, and any other language which produces object code supports *both*: debug symbols and source-maps can be embedded in the compiled output (`.o`, `.a`, `.dylib`, etc.) *or* in a separate `.dSYM` file, usually via a compiler flag (read more in [this answer](https://stackoverflow.com/a/22460268))
---
