<p>I think there are languages where most of the standard library is written in themselves, however with most things marked as <code>native</code> or <code>external</code> and imported from a low-level library or module. These things would usually be functions, not variables; though maybe variable declarations can help on describing memory layout, but still maybe dangerous, I suppose.</p>
<p>Suppose you have a class <code>TextMatch</code> that can be inherited, in Java-like terms (where a single class is inherited and multiple protocols (interfaces) implemented). I think that for this to be possible, a <code>super()</code> call would have to take a pre-allocated object and all of the subtype's fields would come after the size of the base memory of <code>TextMatch</code>. For example, given this Java:</p>
<pre class="lang-java prettyprint-override"><code>class C1 {
    long x;
}
class C2 extends C1 {
    long y;
}
</code></pre>
<p>The base structures for these two types (not the garbage-collected counterpart) could be, in Rust:</p>
<pre class="lang-rust prettyprint-override"><code>#[repr(C)]
struct C1Base {
    x: i64;
}
#[repr(C)]
struct C2Base {
    c1: C1Base;
    y: i64;
}
</code></pre>
<p>I guessed of using some decorator in each native non-final class to express its base memory size, but I find it a bit dangerous:</p>
<p>(After that example follows an explanation of <code>memorySize</code>.)</p>
<pre><code>package;
[FFI(memorySize = n)]
public class TextMatch {
    // fooProperty is defined from Rust
    public native function get fooProperty(): FooType;
    /* etc ... */
}
</code></pre>
<p><code>memorySize</code> indicates the byte payload for <code>TextMatch</code> fields as implemented in Rust. It's not necessarily what will be allocated for the instance object: that depends on the subtype being directly constructed. thus, calling <code>new TextMatch()</code> would be similiar to <code>malloc(memorySize)</code> and a call to the constructor code, however <code>malloc()</code> would be a garbage-collected object.</p>
<p>Calling a <code>new SubtypeTextMatch()</code> would construct an object with more memory, where its fields follow the byte payload specified by <code>memorySize</code>.</p>
<p>That will require that, whenever I add or remove new fields to <code>C1Base</code> from Rust, then I've to update the <code>FFI</code> decorator too for that supertype <code>C1</code>.</p>
<p>I've thought of avoiding this by separating all fields of that supertype into another heap object, exactly like <code>Box&lt;C1Base&gt;</code> from Rust:</p>
<h1>Box&lt;C1Base&gt;</h1>
<p>One thing I thought of too is that I can separate the base into another heap object (a pointer), thus allowing the base to be a constant memory size.</p>
<p>So it'd look like:</p>
<pre class="lang-rust prettyprint-override"><code>#[repr(C)]
struct C2Base {
    c1: usize;
    y: i64;
}
</code></pre>
<p>It's worthy remembering that pointers are platform-dependent, so it's best to use <code>u64</code> and cast it to <code>usize</code> as needed, I suppose.</p>
<p>That said, the payload wouldn't change if I take this approach for every supertype. So it's a possible solution in my opinion, but I still am not sure if I'm taking the good practices here, including storing pointer as <code>u64</code>.</p>

<memory-safety><memory-management>
Memory layout from a native class
CC BY-SA 4.0
2023-06-30T18:53:01.697Z
What dangers do you mean? Is it more dangerous for a language to provide a native memory representation of a data structure, than it is to compile user-defined data structures to a memory representation which is also decided by the compiler?
kaya3
CC BY-SA 4.0
2023-06-30T18:56:11.257Z
@kaya3-supportthestrike I wanted to mean that the memory size for a _native_ super type can change as the sources for the runtime are changed too. So if you add, change the type of or remove any field from the base type, you've to update the `FFI` decorator I showed above. It's fine for the compiler to automatically decide the layout as long as there's no native super class inherited.
Hydroper
CC BY-SA 4.0
2023-06-30T21:45:22.243Z
When a library is implemented in a lower-level language, it typically uses macros and language runtime functions to structure the visible parts of the data in the same way as "normal" data. So if it has an integer field, it may need to box it like the language's normal integers. Look at the implementation of Python's `list` methods, for example.
Barmar
CC BY-SA 4.0
2023-06-30T21:47:07.007Z
I’m afraid I also don’t understand the question. What is the semantics of your hypothetical `FFI` decorator? It’s not clear to me what it is intended to allow you to do. Is the idea that you declare `TextMatch` essentially like you’d declare an interface, and the actual implementation is defined elsewhere? Your question discusses inheritance, but it’s not clear to me why inheriting from a class would be useful if it doesn’t have fields and methods accessible from within the language, and those must also conform to some agreed-upon protocol, so I don’t see how `memorySize` could be enough.
Alexis King
CC BY-SA 4.0
2023-06-30T22:12:23.813Z
@AlexisKing Updated a bit. Maybe that clarifies.
Hydroper
CC BY-SA 4.0
2023-06-30T22:15:17.737Z
@Hydroper Can I imagine that your decorator serves the same purpose as a fixed-size byte array private member? That is, it’s just stating that the class contains a payload of fixed size but unspecified layout?
Alexis King
CC BY-SA 4.0
2023-06-30T22:23:37.680Z
@Hydroper I know what a decorator is. I’m asking about the semantics. Is the decorator equivalent to a fixed-size private field of unspecified layout? And if not, how do they differ?
Alexis King
CC BY-SA 4.0
2023-06-30T22:25:23.990Z
@AlexisKing Sorry. I'm not very sure yet how layout will be determined for the fields. I've planned that `FFI(memorySize = n)` will tell how many bytes will be allocated for the enclosing class in the heap. And, yes, it'll be a private information.
Hydroper
CC BY-SA 4.0
2023-06-30T22:28:09.447Z
That would be only used for native (non-final) classes, though (due to `super()` and user type allocation).
Hydroper
CC BY-SA 4.0
2023-06-30T22:33:41.087Z
I ask the question because I don’t know what “how many bytes will be allocated for the class” means. I’ve been assuming it at least means bytes allocated for an *instance* of the class—is that right? You mention a heap, so I assume you are assuming a garbage collector, in which case there must be *some* structure common to all heap objects so the GC can walk them. The value needs a header of some kind, and the GC needs to know which values are pointers. Does the `memorySize` number include the header? Are you assuming none of the values are pointers?
Alexis King
CC BY-SA 4.0
2023-06-30T22:44:49.117Z
@AlexisKing Yes, for the supertype; although it's not necessarily its constructor that will allocate the entire instance object (the bytes given by `FFI` are just for the supertype fields (which, as I updated later in my question, can be put into another heap object)): it may be any subclass too to allocate that object (whose size will depend on the entire inheritance tree.  `memorySize` shouldn't include the header or anything other than the low-level fields (such as `fooProperty` as shown in the OP) of the supertype.
Hydroper
CC BY-SA 4.0
2023-06-30T22:52:35.957Z
Okay, so in that case, it seems like the same effect could be obtained via a fixed-size byte array private field, correct? Both things specify that the objects in question include a payload of arbitrary data of a given size, so I don’t see how they could possibly differ. And if that is correct, I think the question would be clearer if you expressed it that way rather than using a decorator, since it would make it much more obvious what the number means.
Alexis King
CC BY-SA 4.0
2023-06-30T22:56:09.593Z
@AlexisKing Indeed, if I wrap the supertype fields into another heap object, the payload doesn't change. Otherwise, if you alter the fields from the (for ex.) Rust side, then you've to update the supertype's assigned payload size again to reflect it, otherwise the program will corrupt. I might clarify about the decorator though.
Hydroper
CC BY-SA 4.0
2023-06-30T23:14:35.880Z
@AlexisKing I ended up with multiple edits, but guess it got clearer now.
Hydroper
<h2>Preface</h2>
<p>After some discussion in the comments of your question, it is clear that when you write</p>
<pre class="lang-java prettyprint-override"><code>[FFI(memorySize = n)]
public class TextMatch {
   ...
}
</code></pre>
<p>the semantics you intend the <code>FFI</code> annotation to have is entirely interchangeable with defining a class with a single private member of fixed size and unspecified layout:</p>
<pre class="lang-java prettyprint-override"><code>public class TextMatch {
  private byte[n] payload;
  ...
}
</code></pre>
<p>I find this much easier to reason about, so I am going to dispense with your hypothetical <code>FFI</code> annotation and use the version with a private field, instead.</p>
<h2>Answer</h2>
<p>I think what your question is asking about essentially amounts to a matter of <a href="https://en.wikipedia.org/wiki/Application_binary_interface" rel="nofollow noreferrer">ABI</a>.</p>
<p>Whenever code written in different languages must interoperate, it must agree to some protocol. In some cases, this protocol might be quite structured, using something like Protobuf or even <a href="https://en.wikipedia.org/wiki/Component_Object_Model" rel="nofollow noreferrer">COM</a>, but what you have in mind involves significantly less ceremony. You are envisioning a garbage collected, object-oriented programming language implementation with a compiler and runtime that assumes a particular heap layout for objects, and you are supposing that you wish to use some systems programming language, like C or Rust, to create values in memory that happen to have that layout.</p>
<p>In your question, you seem to be very focused on one problem, namely mismatches in the size of an object’s payload:</p>
<blockquote>
<p>I wanted to mean that it's dangerous for native classes to use that <code>FFI</code> decorator. Why? Because if you've written the implementation for <code>TextMatch</code> in Rust or any systems language and everytime you change the fields or memory size of the base type, you've to update that <code>FFI</code> decorator for these <em>native</em> non-final classes.</p>
</blockquote>
<p>I would like to make the case that this is somewhat tunnel-visioned, and ensuring a consistent specification of the sizes of your values is just one of numerous obligations you accept when trying to make languages interoperate in this way. Consider:</p>
<ul>
<li><p>Since your language is garbage collected, it assumes that objects are all allocated from memory originating in its managed heap. Your systems language will therefore need to coordinate with your language’s allocator to ensure the memory comes from the right place.</p>
</li>
<li><p>If your garbage collector is moving, you’ll need to take care to ensure that it never moves the memory while your systems language is using it. Rust’s safety mechanisms assume that it owns the memory it operates on, but if it doesn’t, you’re already in unsafe territory.</p>
</li>
<li><p>A reference to a heap object can’t be represented by a pointer to the start of some opaque payload, it needs some sort of header to identify it. Therefore, your systems language will need to know the size of that header, and where the payload is placed relative to it, in order to know how to access the fields in its custom payload.</p>
</li>
<li><p>If your object’s payload contains any pointers to GC-managed memory, the GC needs to know about them, so the payload can’t truly be opaque. Therefore, you’ll need some way to specify which fields are pointers and which are nonpointers, and you’ll need to ensure that your language’s runtime and your systems language agree on both the order in which the fields are laid out and how they are aligned.</p>
<ul>
<li>If your runtime utilizes a pointer tagging scheme, then any GC pointers in your object’s payload will need to be properly tagged before they are written and untagged before they are read.</li>
</ul>
</li>
<li><p>If your object’s payload contains any resources that are <em>not</em> GC-managed, you’ll need some protocol to arrange for them to be properly disposed of when the object is garbage collected.</p>
</li>
</ul>
<p>When any of these aspects of your language’s ABI changes, the code written in your systems language will have to be changed to reflect it. This is just the nature of working at such a low level: anything that lives outside of the boundaries of your language is also outside the scope of its mechanized integrity checks.</p>
<p>There are of course numerous strategies (of variable effort) for mitigating these risks. One option is to automatically generate type definitions for your systems language from declarations in your managed language (or vice versa), which reduces the surface area for bugs to the code that performs the translation. You could also have some sort of static analyzer that checks that type definitions written in each language match up, with similar results. You could also utilize fuzz testing to search for ways in which the implementations don’t match up, or you could run a handwritten test suite in a special mode that enables lots of internal checks to ensure everything is working properly.</p>
<p>But no matter what you do, some of these requirements are always going to be difficult to guarantee in general. Ultimately, the chain of trust has to start somewhere.</p>

CC BY-SA 4.0
2023-06-30T23:29:01.340Z
2023-06-30T23:34:20.033Z
2023-06-30T23:34:20.033Z
Alexis King
# Memory layout from a native class
Posted by: username_0 on 2023-06-30
I think there are languages where most of the standard library is written in themselves, however with most things marked as native or external and imported from a low-level library or module. These things would usually be functions, not variables; though maybe variable declarations can help on describing memory layout, but still maybe dangerous, I suppose.
Suppose you have a class TextMatch that can be inherited, in Java-like terms (where a single class is inherited and multiple protocols (interfaces) implemented). I think that for this to be possible, a super() call would have to take a pre-allocated object and all of the subtype's fields would come after the size of the base memory of TextMatch. For example, given this Java:
class C1 {
    long x;
}
class C2 extends C1 {
    long y;
}

The base structures for these two types (not the garbage-collected counterpart) could be, in Rust:
#[repr(C)]
struct C1Base {
    x: i64;
}
#[repr(C)]
struct C2Base {
    c1: C1Base;
    y: i64;
}

I guessed of using some decorator in each native non-final class to express its base memory size, but I find it a bit dangerous:
(After that example follows an explanation of memorySize.)
package;
[FFI(memorySize = n)]
public class TextMatch {
    // fooProperty is defined from Rust
    public native function get fooProperty(): FooType;
    /* etc ... */
}

memorySize indicates the byte payload for TextMatch fields as implemented in Rust. It's not necessarily what will be allocated for the instance object: that depends on the subtype being directly constructed. thus, calling new TextMatch() would be similiar to malloc(memorySize) and a call to the constructor code, however malloc() would be a garbage-collected object.
Calling a new SubtypeTextMatch() would construct an object with more memory, where its fields follow the byte payload specified by memorySize.
That will require that, whenever I add or remove new fields to C1Base from Rust, then I've to update the FFI decorator too for that supertype C1.
I've thought of avoiding this by separating all fields of that supertype into another heap object, exactly like Box from Rust:
Box
One thing I thought of too is that I can separate the base into another heap object (a pointer), thus allowing the base to be a constant memory size.
So it'd look like:
#[repr(C)]
struct C2Base {
    c1: usize;
    y: i64;
}

It's worthy remembering that pointers are platform-dependent, so it's best to use u64 and cast it to usize as needed, I suppose.
That said, the payload wouldn't change if I take this approach for every supertype. So it's a possible solution in my opinion, but I still am not sure if I'm taking the good practices here, including storing pointer as u64.
**Tags:** memory-safety memory-management
> **Comments:**
> **username_1**: What dangers do you mean? Is it more dangerous for a language to provide a native memory representation of a data structure, than it is to compile user-defined data structures to a memory representation which is also decided by the compiler?  
> **username_0**: @kaya3-supportthestrike I wanted to mean that the memory size for a _native_ super type can change as the sources for the runtime are changed too. So if you add, change the type of or remove any field from the base type, you've to update the `FFI` decorator I showed above. It's fine for the compiler to automatically decide the layout as long as there's no native super class inherited.  
> **username_2**: When a library is implemented in a lower-level language, it typically uses macros and language runtime functions to structure the visible parts of the data in the same way as "normal" data. So if it has an integer field, it may need to box it like the language's normal integers. Look at the implementation of Python's `list` methods, for example.  
> **username_3**: I’m afraid I also don’t understand the question. What is the semantics of your hypothetical `FFI` decorator? It’s not clear to me what it is intended to allow you to do. Is the idea that you declare `TextMatch` essentially like you’d declare an interface, and the actual implementation is defined elsewhere? Your question discusses inheritance, but it’s not clear to me why inheriting from a class would be useful if it doesn’t have fields and methods accessible from within the language, and those must also conform to some agreed-upon protocol, so I don’t see how `memorySize` could be enough.  
> **username_0**: @AlexisKing Updated a bit. Maybe that clarifies.  
> **username_3**: @Hydroper Can I imagine that your decorator serves the same purpose as a fixed-size byte array private member? That is, it’s just stating that the class contains a payload of fixed size but unspecified layout?  
> **username_3**: @Hydroper I know what a decorator is. I’m asking about the semantics. Is the decorator equivalent to a fixed-size private field of unspecified layout? And if not, how do they differ?  
> **username_0**: @AlexisKing Sorry. I'm not very sure yet how layout will be determined for the fields. I've planned that `FFI(memorySize = n)` will tell how many bytes will be allocated for the enclosing class in the heap. And, yes, it'll be a private information.  
> **username_0**: That would be only used for native (non-final) classes, though (due to `super()` and user type allocation).  
> **username_3**: I ask the question because I don’t know what “how many bytes will be allocated for the class” means. I’ve been assuming it at least means bytes allocated for an *instance* of the class—is that right? You mention a heap, so I assume you are assuming a garbage collector, in which case there must be *some* structure common to all heap objects so the GC can walk them. The value needs a header of some kind, and the GC needs to know which values are pointers. Does the `memorySize` number include the header? Are you assuming none of the values are pointers?  
> **username_0**: @AlexisKing Yes, for the supertype; although it's not necessarily its constructor that will allocate the entire instance object (the bytes given by `FFI` are just for the supertype fields (which, as I updated later in my question, can be put into another heap object)): it may be any subclass too to allocate that object (whose size will depend on the entire inheritance tree.  `memorySize` shouldn't include the header or anything other than the low-level fields (such as `fooProperty` as shown in the OP) of the supertype.  
> **username_3**: Okay, so in that case, it seems like the same effect could be obtained via a fixed-size byte array private field, correct? Both things specify that the objects in question include a payload of arbitrary data of a given size, so I don’t see how they could possibly differ. And if that is correct, I think the question would be clearer if you expressed it that way rather than using a decorator, since it would make it much more obvious what the number means.  
> **username_0**: @AlexisKing Indeed, if I wrap the supertype fields into another heap object, the payload doesn't change. Otherwise, if you alter the fields from the (for ex.) Rust side, then you've to update the supertype's assigned payload size again to reflect it, otherwise the program will corrupt. I might clarify about the decorator though.  
> **username_0**: @AlexisKing I ended up with multiple edits, but guess it got clearer now.  
---
## 1 answers
### Answer by username_3 on 2023-06-30. Score: 3 | Accepted answer
Preface
After some discussion in the comments of your question, it is clear that when you write
[FFI(memorySize = n)]
public class TextMatch {
   ...
}

the semantics you intend the FFI annotation to have is entirely interchangeable with defining a class with a single private member of fixed size and unspecified layout:
public class TextMatch {
  private byte[n] payload;
  ...
}

I find this much easier to reason about, so I am going to dispense with your hypothetical FFI annotation and use the version with a private field, instead.
Answer
I think what your question is asking about essentially amounts to a matter of ABI.
Whenever code written in different languages must interoperate, it must agree to some protocol. In some cases, this protocol might be quite structured, using something like Protobuf or even COM, but what you have in mind involves significantly less ceremony. You are envisioning a garbage collected, object-oriented programming language implementation with a compiler and runtime that assumes a particular heap layout for objects, and you are supposing that you wish to use some systems programming language, like C or Rust, to create values in memory that happen to have that layout.
In your question, you seem to be very focused on one problem, namely mismatches in the size of an object’s payload:

I wanted to mean that it's dangerous for native classes to use that FFI decorator. Why? Because if you've written the implementation for TextMatch in Rust or any systems language and everytime you change the fields or memory size of the base type, you've to update that FFI decorator for these native non-final classes.

I would like to make the case that this is somewhat tunnel-visioned, and ensuring a consistent specification of the sizes of your values is just one of numerous obligations you accept when trying to make languages interoperate in this way. Consider:

Since your language is garbage collected, it assumes that objects are all allocated from memory originating in its managed heap. Your systems language will therefore need to coordinate with your language’s allocator to ensure the memory comes from the right place.

If your garbage collector is moving, you’ll need to take care to ensure that it never moves the memory while your systems language is using it. Rust’s safety mechanisms assume that it owns the memory it operates on, but if it doesn’t, you’re already in unsafe territory.

A reference to a heap object can’t be represented by a pointer to the start of some opaque payload, it needs some sort of header to identify it. Therefore, your systems language will need to know the size of that header, and where the payload is placed relative to it, in order to know how to access the fields in its custom payload.

If your object’s payload contains any pointers to GC-managed memory, the GC needs to know about them, so the payload can’t truly be opaque. Therefore, you’ll need some way to specify which fields are pointers and which are nonpointers, and you’ll need to ensure that your language’s runtime and your systems language agree on both the order in which the fields are laid out and how they are aligned.

If your runtime utilizes a pointer tagging scheme, then any GC pointers in your object’s payload will need to be properly tagged before they are written and untagged before they are read.


If your object’s payload contains any resources that are not GC-managed, you’ll need some protocol to arrange for them to be properly disposed of when the object is garbage collected.


When any of these aspects of your language’s ABI changes, the code written in your systems language will have to be changed to reflect it. This is just the nature of working at such a low level: anything that lives outside of the boundaries of your language is also outside the scope of its mechanized integrity checks.
There are of course numerous strategies (of variable effort) for mitigating these risks. One option is to automatically generate type definitions for your systems language from declarations in your managed language (or vice versa), which reduces the surface area for bugs to the code that performs the translation. You could also have some sort of static analyzer that checks that type definitions written in each language match up, with similar results. You could also utilize fuzz testing to search for ways in which the implementations don’t match up, or you could run a handwritten test suite in a special mode that enables lots of internal checks to ensure everything is working properly.
But no matter what you do, some of these requirements are always going to be difficult to guarantee in general. Ultimately, the chain of trust has to start somewhere.
---
