<p>Let's say I have a bytecode compiler. Even though it doesn't generate native machine code, <strong>is there a simple way for it to produce native executable files</strong> that I can directly run?</p>
<p>One method I've heard of is essentially recompiling the interpreter with the bytecode somehow attached. I think I like this idea, but <strong>it would be ideal if the executable file could be used by the interpreter as well as a standalone program</strong>. That is, the output of the compiler may be executed or loaded as a module by the interpreter when it executes an import.</p>
<p>I confess I don't fully understand modern executable formats, but I have a basic knowledge of ELF. Something I've thought of is using the “interpreter” field to run a program thst loads the bytecode interpreter and feed it the ELF's text. Is this viable? Is it a bad idea? I'm suggesting ELF because it's a feasible format for compiled modules; GNU GUILE <a href="https://www.gnu.org/software/guile/manual/html_node/Object-File-Format.html" rel="noreferrer">uses it</a>.</p>

<compilers><bytecode>
Executable files with a bytecode compiler/interpreter
CC BY-SA 4.0
2024-06-10T23:05:03.270Z
To be clear, by "executable file" do you mean a native binary? On Unix operating systems it is straightforward to make a file executable by giving it a `#!/usr/bin/your-interpreter` line at the start of the file, and setting its `+x` flag. This is how e.g. Python works, though you put the hashbang in the source code rather than the compiled bytecode.
kaya3
CC BY-SA 4.0
2024-06-10T23:10:31.657Z
@kaya3 Yes, I meant native binary.
texdr.aft
CC BY-SA 4.0
2024-06-11T12:24:33.430Z
"*Is this viable?*" - yes. Just put your bytecode in a custom section of the ELF that contains your interpreter.
Bergi
CC BY-SA 4.0
2024-06-11T13:59:08.217Z
What is simple in this context?
feldentm
<p>Here is one way to do it, which is used by the <a href="https://love2d.org/" rel="noreferrer">LÖVE game engine</a>:</p>
<p>The bytecode gets stored in a .zip file which gets concatenated to the interpreter.</p>
<p>When the interpreter starts up, it tries to open itself as a .zip file. If it succeeds, it uses that as the program to be executed, otherwise it uses the regular means of finding the program (usually using command line arguments).</p>
<p>This works because native executables generally have their metadata in the front, and will ignore <em>appended</em> &quot;garbage&quot; data, and .zip files have their metadata at the end of the file, and will ignore <em>prepended</em> &quot;garbage&quot; data.</p>
<p>You don't need to wrap it in a .zip if you design the bytecode format itself in a similar way.</p>
<p>For example something like this:</p>
<pre><code>bytecode is &lt;opcodes&gt;&lt;strings&gt;&lt;debug&gt;&lt;metadata&gt;

where

opcodes is X bytes of byte code instructions
strings is Y bytes of strings and other data used by opcodes
debug is Z bytes of debugging information
metadata is &lt;X&gt;&lt;Y&gt;&lt;Z&gt;&lt;magic number&gt; (with each of those fields a fixed number of bytes)
</code></pre>

CC BY-SA 4.0
2024-06-11T05:12:47.783Z
2024-06-11T05:12:47.783Z
Jasmijn
CC BY-SA 4.0
2024-06-11T07:06:41.393Z
That's clever. I find it entertaining that `cat` is part of the build process.
texdr.aft
CC BY-SA 4.0
2024-06-11T22:23:24.430Z
I believe this is how self-extracting .zip files worked, back when self-extracting archives were a thing.
Mark
CC BY-SA 4.0
2024-06-12T11:43:08.030Z
Python also supports the interpreter executable followed by a ZIP file combo.
pts
<p>As long as you have a bytecode <em>interpreter</em> implemented, all you need to do is generate a program that's the equivalent of</p>
<pre><code>#include &quot;MyBytecodeInterpreter.h&quot;

uint8_t BYTECODE[] = {
   // insert bytecode here
}

int main()
{
    return InterpretBytecode(BYTECODE);
}
</code></pre>
<p>You could even write an actual C program and compile it, if you don't know how to generate executable files directly.  Then just link it (statically or dynamically) to your bytecode interpreter, and you've got an instant “compiled” program.</p>
<p>To ensure that the generated executable can be used both as a standalone program and as a module that the interpreter can import, you can export the <code>BYTECODE</code> symbol from the compiled executable.  Then modify your interpreter to detect if its input is an executable file (e.g., look for the magic number <code>\x7FELF</code> on Linux or <code>MZ</code> on Windows), and if it is, load the bytecode from the symbol's address.</p>
<p>Sure, it may not run as fast as a “real” compiled program, but it will technically satisfy your requirement of having “native executable files that I can directly run”.</p>

CC BY-SA 4.0
2024-06-10T23:45:00.390Z
2024-06-11T15:20:03.130Z
2024-06-11T15:20:03.130Z
dan04
CC BY-SA 4.0
2024-06-11T01:25:50.763Z
From the question: *"One method I've heard of is essentially recompiling the interpreter with the bytecode somehow attached. I think I like this idea, but it would be ideal if the executable file could be used by the interpreter as well as a standalone program."*
kaya3
CC BY-SA 4.0
2024-06-11T12:16:50.113Z
"*it would be ideal if the executable file could be used by the interpreter, that is, the output may be loaded as a module by the interpreter when it executes an import*" - how would that work with this approach?
Bergi
CC BY-SA 4.0
2024-06-11T14:15:25.960Z
As long as the symbol is exported it's relatively straightforward to grab the data with an ELF parsing library.
TLW
<p>It is certainly possible to write a compiler which accepts bytecode as its source and outputs native code. That's what Java's JIT compilers do, after all.</p>
<p>Writing one that produced <em>standalone</em> executables would be a nuisance (among other things, there's the question of whether to start up Java to run  library and utility jarfiles or compiling those too, either inline or linked), but it's really no harder than writing a back-end for other intermediate languages. Though you might actually want a front end which converts it to a more standard IL so you could use existing optimizing back-ends.</p>
<p>If it doesn't already exist (I would expect it does) and you want to attempt it, Go for it. If you're talking about Java bytecodes and are writing your compiler in Java, the BCEL package makes manipulating bytecodes fairly straightforward; that's what we used when implementing compilation <em>to</em> bytecodes in Apache Xalan.</p>
<p>Be aware that, depending on the code involved, this may or may not be able to run faster than what the JIT compiler is already doing for you. That was one of the arguments against offering pre-compilation as a feature of the JDK/JRE.</p>

CC BY-SA 4.0
2024-06-12T16:00:31.960Z
2024-06-12T16:00:31.960Z
keshlam
<p>It depends on what you mean by &quot;native executable&quot;.</p>
<p>If you mean &quot;runs natively on the hardware&quot;, then no: if you don't have machine code, then you don't have machine code, and a runtime is required.</p>
<p>If you mean &quot;can be launched from the operating system in the same way that a machine-code-based executable is&quot;, then yes - perhaps even trivially. But at this point there isn't a clear distinction between &quot;running an 'executable' file that requires a specific runtime&quot; and &quot;using the runtime to open the file&quot;. Which is to say, in Windows I could trivially associate <code>.py</code> or <code>.pyc</code> files with a Python interpreter, type their name as a &quot;command&quot; in CMD or double-click an icon in Explorer, and have the code execute - but few people would call this &quot;making the file executable&quot; <em>even though the user experience may be identical</em> to running a <code>.exe</code>.</p>
<p>Linux, on the other hand, doesn't have the same concept of file association (although a desktop environment might build a reasonable facsimile on top). Instead, you set an executable permissions bit on the file, and then arrange the file format such that it has a <a href="https://en.wikipedia.org/wiki/List_of_file_signatures" rel="noreferrer">signature</a> that is recognized by <a href="https://en.wikipedia.org/wiki/Binfmt_misc" rel="noreferrer"><code>binfmt_misc</code></a>.</p>
<p>If you mean &quot;is stored in a container format that the OS commonly uses for machine-code-based executables&quot; - e.g., as ELF on Linux or PE on Windows - yes, it's certainly possible. The .NET platform works this way: code in .NET languages such as C# compiles to bytecode that may end up stored in a file in the PE format - to my understanding, it's stored as &quot;data&quot;, and the &quot;code&quot; is a hard-coded bootstrap which makes a system call to locate the runtime and feed the bytecode to it. A similar trick should be equally possible with ELF. (Of course, as you've noted, you could also just put the entire interpreter there, and not worry about needing to find it on the system. Many &quot;Python standalone app builder&quot; utilities work this way.)</p>
<p>On Linux, that approach has the advantage that the system is already configured to recognize the ELF format and &quot;execute&quot; it by loading the machine code.</p>
<p>In any event, if you want such an &quot;executable&quot; to also be usable by the interpreter, you just need to arrange for the interpreter to recognize the &quot;executable format&quot;. The trivial example of this is supporting <code>#</code> as a comment character, so that valid source code can have a shebang line; then the interpreter sees ordinary code, while Linux sees an &quot;executable text file&quot; that is &quot;executed&quot; by parsing the shebang to find the interpreter and passing the script to it. A more complex example is Python bytecode: <code>binfmt_misc</code> may be configured to recognize <code>.pyc</code> files according to the header bytes that specify the bytecode version (mapping to a corresponding interpreter).</p>
<p>Related reading: <a href="https://unix.stackexchange.com/questions/71504">File extensions and association with programs in linux</a> on unix.SE</p>

CC BY-SA 4.0
2024-06-11T08:46:29.407Z
2024-06-11T08:46:29.407Z
Karl Knechtel
CC BY-SA 4.0
2024-06-12T02:36:39.040Z
I have to look into the specs more, but `binfmt_misc` has a limit of 128 bytes for the offset of the magic data. Do you know if that's enough to distinguish between normal ELFs and  the compiler's output without deviating from the ELF standard, so that other programs can handle them?
texdr.aft
CC BY-SA 4.0
2024-06-12T04:44:03.620Z
The ELF header is 64 bytes long on 64-bit builds (52 on 32-bit builds) and is *typically* immediately followed by one program header (56/32 bytes long, respectively). That makes worst-case 120 bytes. But the number of program headers, as well as their location, can vary. There are also section headers that could be located somewhere else. So, yes, you should be able to control the contents within the first 128 bytes to allow for your own magic number (probably by making sure a `.rodata` section comes first). You might need custom control over the creation of the ELF file.
Karl Knechtel
CC BY-SA 4.0
2024-06-12T04:45:21.680Z
If I'm thinking clearly: the sort of wrapper program dan04 describes, would effectively result in the C compiler putting the bytecode into an `.rodata` section - but you probably wouldn't be able to control how the C compiler lays out the exectuable. (Probably you should mark the bytecode global as `const` if you do pursue this route.)
Karl Knechtel
CC BY-SA 4.0
2024-06-12T05:17:56.377Z
KarlKnechtel and @texdr.aft: With GNU C `__attribute__((section(".my_custom_name")))` on the array, and a custom linker script, you can get the toolchain to put it where you want, like ahead of the segments that contain the `.text` and `.rodata` sections.  I haven't done much with this; a section name like `.rodata.early` might minimize the amount of modification you have to do to `ld`'s default linker script (in `/usr/lib/ldscripts/` I think).  (But if you want it first, it'll have to be ahead of the .text section, too, not just the start of .rodata.)
Peter Cordes
CC BY-SA 4.0
2024-06-13T01:59:54.080Z
@PeterCordes (and KarlKnechtel): Easier to just generate the file myself, scooting the normal ELF data a few bytes down and placing some magic numbers immediately after the header.
texdr.aft
CC BY-SA 4.0
2024-06-13T02:21:16.250Z
If you prefer. Of course you'll still want to have some kind of automated setup for the user to make the necessary `binfmt_misc` changes and/or file associations. And at that point it's not clear that the ELF wrapper format is actually *doing anything* for you. At which point you basically have my second option instead.
Karl Knechtel
CC BY-SA 4.0
2024-06-15T18:22:41.983Z
Actually the limit recently changed to 256 bytes: https://lore.kernel.org/lkml/20181112160956.GA28472@redhat.com/
texdr.aft
# Executable files with a bytecode compiler/interpreter
Posted by: username_0 on 2024-06-10
Let's say I have a bytecode compiler. Even though it doesn't generate native machine code, **is there a simple way for it to produce native executable files** that I can directly run?

One method I've heard of is essentially recompiling the interpreter with the bytecode somehow attached. I think I like this idea, but **it would be ideal if the executable file could be used by the interpreter as well as a standalone program**. That is, the output of the compiler may be executed or loaded as a module by the interpreter when it executes an import.

I confess I don't fully understand modern executable formats, but I have a basic knowledge of ELF. Something I've thought of is using the “interpreter” field to run a program thst loads the bytecode interpreter and feed it the ELF's text. Is this viable? Is it a bad idea? I'm suggesting ELF because it's a feasible format for compiled modules; GNU GUILE [uses it](https://www.gnu.org/software/guile/manual/html_node/Object-File-Format.html).
**Tags:** compilers bytecode
> **Comments:**
> **username_1**: To be clear, by "executable file" do you mean a native binary? On Unix operating systems it is straightforward to make a file executable by giving it a `#!/usr/bin/your-interpreter` line at the start of the file, and setting its `+x` flag. This is how e.g. Python works, though you put the hashbang in the source code rather than the compiled bytecode.  
> **username_0**: @kaya3 Yes, I meant native binary.  
> **username_2**: "*Is this viable?*" - yes. Just put your bytecode in a custom section of the ELF that contains your interpreter.  
> **username_3**: What is simple in this context?  
---
## 4 answers
### Answer by username_10 on 2024-06-11. Score: 9 | Accepted answer
It depends on what you mean by "native executable".

If you mean "runs natively on the hardware", then no: if you don't have machine code, then you don't have machine code, and a runtime is required.

If you mean "can be launched from the operating system in the same way that a machine-code-based executable is", then yes - perhaps even trivially. But at this point there isn't a clear distinction between "running an 'executable' file that requires a specific runtime" and "using the runtime to open the file". Which is to say, in Windows I could trivially associate `.py` or `.pyc` files with a Python interpreter, type their name as a "command" in CMD or double-click an icon in Explorer, and have the code execute - but few people would call this "making the file executable" *even though the user experience may be identical* to running a `.exe`.

Linux, on the other hand, doesn't have the same concept of file association (although a desktop environment might build a reasonable facsimile on top). Instead, you set an executable permissions bit on the file, and then arrange the file format such that it has a [signature](https://en.wikipedia.org/wiki/List_of_file_signatures) that is recognized by [`binfmt_misc`](https://en.wikipedia.org/wiki/Binfmt_misc).

If you mean "is stored in a container format that the OS commonly uses for machine-code-based executables" - e.g., as ELF on Linux or PE on Windows - yes, it's certainly possible. The .NET platform works this way: code in .NET languages such as C# compiles to bytecode that may end up stored in a file in the PE format - to my understanding, it's stored as "data", and the "code" is a hard-coded bootstrap which makes a system call to locate the runtime and feed the bytecode to it. A similar trick should be equally possible with ELF. (Of course, as you've noted, you could also just put the entire interpreter there, and not worry about needing to find it on the system. Many "Python standalone app builder" utilities work this way.)

On Linux, that approach has the advantage that the system is already configured to recognize the ELF format and "execute" it by loading the machine code.

In any event, if you want such an "executable" to also be usable by the interpreter, you just need to arrange for the interpreter to recognize the "executable format". The trivial example of this is supporting `#` as a comment character, so that valid source code can have a shebang line; then the interpreter sees ordinary code, while Linux sees an "executable text file" that is "executed" by parsing the shebang to find the interpreter and passing the script to it. A more complex example is Python bytecode: `binfmt_misc` may be configured to recognize `.pyc` files according to the header bytes that specify the bytecode version (mapping to a corresponding interpreter).

Related reading: [File extensions and association with programs in linux](https://unix.stackexchange.com/questions/71504) on unix.SE
> **Comments:**
> **username_0** (2024-06-12): I have to look into the specs more, but `binfmt_misc` has a limit of 128 bytes for the offset of the magic data. Do you know if that's enough to distinguish between normal ELFs and  the compiler's output without deviating from the ELF standard, so that other programs can handle them?  
> **username_10** (2024-06-12): The ELF header is 64 bytes long on 64-bit builds (52 on 32-bit builds) and is *typically* immediately followed by one program header (56/32 bytes long, respectively). That makes worst-case 120 bytes. But the number of program headers, as well as their location, can vary. There are also section headers that could be located somewhere else. So, yes, you should be able to control the contents within the first 128 bytes to allow for your own magic number (probably by making sure a `.rodata` section comes first). You might need custom control over the creation of the ELF file.  
> **username_10** (2024-06-12): If I'm thinking clearly: the sort of wrapper program dan04 describes, would effectively result in the C compiler putting the bytecode into an `.rodata` section - but you probably wouldn't be able to control how the C compiler lays out the exectuable. (Probably you should mark the bytecode global as `const` if you do pursue this route.)  
> **username_11** (2024-06-12): KarlKnechtel and @texdr.aft: With GNU C `__attribute__((section(".my_custom_name")))` on the array, and a custom linker script, you can get the toolchain to put it where you want, like ahead of the segments that contain the `.text` and `.rodata` sections.  I haven't done much with this; a section name like `.rodata.early` might minimize the amount of modification you have to do to `ld`'s default linker script (in `/usr/lib/ldscripts/` I think).  (But if you want it first, it'll have to be ahead of the .text section, too, not just the start of .rodata.)  
> **username_0** (2024-06-13): @PeterCordes (and KarlKnechtel): Easier to just generate the file myself, scooting the normal ELF data a few bytes down and placing some magic numbers immediately after the header.  
> **username_10** (2024-06-13): If you prefer. Of course you'll still want to have some kind of automated setup for the user to make the necessary `binfmt_misc` changes and/or file associations. And at that point it's not clear that the ELF wrapper format is actually *doing anything* for you. At which point you basically have my second option instead.  
> **username_0** (2024-06-15): Actually the limit recently changed to 256 bytes: https://lore.kernel.org/lkml/20181112160956.GA28472@redhat.com/  
---
### Answer by username_4 on 2024-06-11. Score: 19
Here is one way to do it, which is used by the LÖVE game engine:
The bytecode gets stored in a .zip file which gets concatenated to the interpreter.
When the interpreter starts up, it tries to open itself as a .zip file. If it succeeds, it uses that as the program to be executed, otherwise it uses the regular means of finding the program (usually using command line arguments).
This works because native executables generally have their metadata in the front, and will ignore appended "garbage" data, and .zip files have their metadata at the end of the file, and will ignore prepended "garbage" data.
You don't need to wrap it in a .zip if you design the bytecode format itself in a similar way.
For example something like this:
bytecode is 

where

opcodes is X bytes of byte code instructions
strings is Y bytes of strings and other data used by opcodes
debug is Z bytes of debugging information
metadata is  (with each of those fields a fixed number of bytes)
> **Comments:**
> **username_0** (2024-06-11): That's clever. I find it entertaining that `cat` is part of the build process.  
> **username_5** (2024-06-11): I believe this is how self-extracting .zip files worked, back when self-extracting archives were a thing.  
> **username_6** (2024-06-12): Python also supports the interpreter executable followed by a ZIP file combo.  
---
### Answer by username_7 on 2024-06-10. Score: 7
As long as you have a bytecode interpreter implemented, all you need to do is generate a program that's the equivalent of
#include "MyBytecodeInterpreter.h"

uint8_t BYTECODE[] = {
   // insert bytecode here
}

int main()
{
    return InterpretBytecode(BYTECODE);
}

You could even write an actual C program and compile it, if you don't know how to generate executable files directly.  Then just link it (statically or dynamically) to your bytecode interpreter, and you've got an instant “compiled” program.
To ensure that the generated executable can be used both as a standalone program and as a module that the interpreter can import, you can export the BYTECODE symbol from the compiled executable.  Then modify your interpreter to detect if its input is an executable file (e.g., look for the magic number \x7FELF on Linux or MZ on Windows), and if it is, load the bytecode from the symbol's address.
Sure, it may not run as fast as a “real” compiled program, but it will technically satisfy your requirement of having “native executable files that I can directly run”.
> **Comments:**
> **username_1** (2024-06-11): From the question: *"One method I've heard of is essentially recompiling the interpreter with the bytecode somehow attached. I think I like this idea, but it would be ideal if the executable file could be used by the interpreter as well as a standalone program."*  
> **username_2** (2024-06-11): "*it would be ideal if the executable file could be used by the interpreter, that is, the output may be loaded as a module by the interpreter when it executes an import*" - how would that work with this approach?  
> **username_8** (2024-06-11): As long as the symbol is exported it's relatively straightforward to grab the data with an ELF parsing library.  
---
### Answer by username_9 on 2024-06-12. Score: 2
It is certainly possible to write a compiler which accepts bytecode as its source and outputs native code. That's what Java's JIT compilers do, after all.

Writing one that produced *standalone* executables would be a nuisance (among other things, there's the question of whether to start up Java to run library and utility jarfiles or compiling those too, either inline or linked), but it's really no harder than writing a back-end for other intermediate languages. Though you might actually want a front end which converts it to a more standard IL so you could use existing optimizing back-ends.

If it doesn't already exist (I would expect it does) and you want to attempt it, Go for it. If you're talking about Java bytecodes and are writing your compiler in Java, the BCEL package makes manipulating bytecodes fairly straightforward; that's what we used when implementing compilation *to* bytecodes in Apache Xalan.

Be aware that, depending on the code involved, this may or may not be able to run faster than what the JIT compiler is already doing for you. That was one of the arguments against offering pre-compilation as a feature of the JDK/JRE.
---
