<p>A part of why Lisp's macros work so well is that the code is just a list of symbols, and it is therefore simple to manipulate using a regular Lisp function. How could a language that uses a syntax more like Rust's or C's implement a way to modify code from code itself?</p>

<syntax><homoiconicity><macro>
How could a language with a C-style syntax implement homoiconicity?
CC BY-SA 4.0
2023-06-26T15:46:11.413Z
@kouta-kun and for the Python-style languages with AST macros I recommend you to take a look at Converge.
SK-logic
CC BY-SA 4.0
2023-05-16T17:56:55.223Z
Rust *does* it. There's a `macro_rules!` macro which is basically `syntax-case`, and a very popular crate provides a `quote!` macro for quasiquoting.
Silvio Mayolo
CC BY-SA 4.0
2023-05-16T17:59:58.323Z
Rust does it, but declarative macros only allow (in practice) a subset of what you can do with Lisp macros. Procedural macros fill the gap, but at the price of a more complicated setup and longer compile times. It's a compromise between power and efficiency
zdimension
CC BY-SA 4.0
2023-05-16T19:22:59.723Z
@kaya3 I've modified for clarification, though I'm also interested in indentation-syntax languages like Python.
kouta-kun
<p>They can use metaprogramming techniques that allow you to do what you want. Some examples:</p>
<ol>
<li>As you said, they can use macros. Pretty similar to Lisp in this regard as they are commonly implemented there too. Some functationally is gone though.</li>
<li>They can use preprocessor directives. Lisp's have more expressive power and enable dynamic code generation. But they lose on simplicity and support.</li>
<li>They can use procedural macros. Lisp's again have more expressive power and are simpler. But Lisp's lose on Integration and complex transformations.</li>
</ol>

CC BY-SA 4.0
2023-05-16T19:27:32.003Z
2023-06-26T21:37:33.317Z
2023-06-26T21:37:33.317Z
Starship
CC BY-SA 4.0
2023-06-29T09:50:27.293Z
How exactly Lisp-style macros lose on complex transformations? You can wrap an entire optimising compiler into  Lisp macro.
SK-logic
<p>The precise meaning of homoiconicity is debated, so I’ll give up front my assumptions about what it means:</p>
<ol>
<li><p>The basic <em>code structure</em> has a direct mapping to a general <em>data structure</em>.</p>
</li>
<li><p>That <em>data structure</em> is a basic <em>type</em> within the language.</p>
</li>
<li><p>Except for some <em>trivial marker</em>, the <em>literal syntax</em> for that type is identical to the <em>code syntax</em>.</p>
</li>
</ol>
<p>So for example, <a href="https://en.wikipedia.org/wiki/Tcl" rel="noreferrer">Tcl</a> is homoiconic in this sense because all code is made of character strings, and not only in the sense that any textual language is, but in a way that’s reflected in the language. The interpreter may store data types and code in a more efficient representation, but this is purely an optimisation—it must retain enough information to transparently convert back to text. And the notations for escaping (<code>&quot;</code> … <code>&quot;</code>, <code>{</code> … <code>}</code>, <code>\</code>) and interpolation (<code>[</code> … <code>]</code>, <code>$</code>) are trivial—their size is a small constant, independent of the size of the escaped or interpolated term.</p>
<p>Likewise, of course, modern Lisp written with S-expressions is homoiconic, because its code is isomorphic to a <a href="https://en.wikipedia.org/wiki/Rose_tree" rel="noreferrer">rose tree</a> of atomic terms, being the main aggregate type in the language, and quotation (<code>quote</code> / <code>'</code>) and splicing (<code>unquote</code> / <code>,</code>, <code>unquote-splicing</code> / <code>,@</code>) are again trivial.</p>
<hr />
<p>But I think that early Lisp, written with M-expressions, also fits the bill. For example, from the <a href="https://web.archive.org/web/20060211020233/http://community.computerhistory.org/scc/projects/LISP/book/LISP%201.5%20Programmers%20Manual.pdf" rel="noreferrer">Lisp 1.5 Programmer’s Manual</a><sup>(PDF)</sup> p. 15, here’s the function <code>member</code> that tests for membership in a list, and how it was desugared in S-expression form.</p>
<pre><code>member[a;x] = [null[x]→F;eq[a;car[x]]→T;T→
      member[a;cdr[x]]]
</code></pre>
<pre><code>define ((
(member (lambda (a x) (cond ((null x) f)
      ( (eq a (car x) ) t) (t (member a (cdr x))) )))
))
</code></pre>
<p>So there are some differences, but they’re entirely superficial. The punctuation is different, and there are a couple of infix operators, but besides that, all of the same nesting structure and ordering of leaf terms is unchanged. As such, even if the programmer doesn’t look at the S syntax, they can easily <em>predict</em> it from the M syntax, so they can read and write code that treats the M form <em>as if</em> it were the S data structure.</p>
<p>Another good example of <em>nontrivial</em> homoiconicity in practice is <a href="https://en.wikipedia.org/wiki/XSLT" rel="noreferrer">XSLT</a>. The program’s input and output are usually both XML, and the program itself is represented as XML, so metaprogramming isn’t appreciably different from ordinary programming, despite the fact that XML has fairly complex syntax.</p>
<hr />
<p>A long time ago, I designed a language where all compound terms had the same form, essentially a series of names followed by bracketed subterms. With some sugar for infix operators, the result was that code could look fairly familiar to a C programmer. For example, here’s the <code>member</code> function from earlier:</p>
<pre><code>define [member](a, x) {
    if (null(x)) {
        false
    } else if (a = head(x)) {
        true
    } else {
        member(a, tail(x))
    }
}
</code></pre>
<p>This desugared to a tree structure very much like the Lisp code, which was accessible to the programmer to a limited extent. For example, <code>member.head[0]</code> is <code>&quot;a&quot;</code>, and <code>member.body[0].name</code> is <code>&quot;_else_&quot;</code>, because <code>else</code> is an infix operator. At the time, in fact my goal wasn’t metaprogramming, but only allowing some customisable syntax and some introspection, while retaining a simple top-down operator-precedence parser. But I hope it’s illustrative nevertheless.</p>
<hr />
<p>I think these points can be assembled into a general method of designing such languages:</p>
<p>First, select a data structure amenable to representing a program. <em>Typically</em> this will be some sort of recursive tree structure with labels at the leaves and/or branches. However, recursion isn’t strictly necessary—you can always flatten the structure of a language. For example, you can prescribe a certain implicit traversal order, such as how prefix/postfix notation can do away with parentheses for operations of fixed arity; or you can require compound terms to be defined out of line and referenced by name. (Labels aren’t strictly necessary either, if a program’s meaning is determined entirely by its structure, but that’s getting into esolang territory.)</p>
<p>Second, reify that data type within the language. Ideally, you should ensure that conversion from data to code (let’s call it <code>parse</code>) and from code to data (<code>format</code>) have various properties you might expect:</p>
<ol>
<li><p><code>parse(format(parse(data)))</code> = <code>parse(data)</code></p>
<p>Formatting preserves all information that parsing does.</p>
</li>
<li><p><code>format(parse(format(code)))</code> = <code>format(code)</code></p>
<p>Parsing preserves all information that formatting does.</p>
</li>
<li><p><code>format(parse(data))</code> = <code>data</code></p>
<p>Parsing preserves formatting.</p>
</li>
<li><p><code>parse(format(code))</code> = <code>code</code></p>
<p>Formatting preserves parsing.</p>
</li>
</ol>
<p>(You could call 1 &amp; 2 “weak roundtripping” and 3 &amp; 4 “strong roundtripping”.)</p>
<p>Third and finally, find a notation for this data structure that’s compatible with C-style syntax, or whatever style your goal may be. The biggest usability constraint I see here is that your choice of data structure might need to forgo some of the compiler-specific organisation that you’d want in an AST, so that it can be <em>conveniently</em> manipulated in a uniform way. The bigger and more complex the language is, the more syntactic cases a programmer will need to consider when metaprogramming.</p>

CC BY-SA 4.0
2023-06-28T23:21:41.570Z
2023-06-28T23:21:41.570Z
Jon Purdy
<p>Homoiconicity is a nice feature but it is not necessary for Common Lisp-style metaprogramming.</p>
<p>All you need is quasiquotation for both constructing ASTs and for pattern-matching ASTs. If you have these two components, you can do everything a homoiconic syntax allows to do. Keep in mind that in some languages, most notably Scheme, homoiconicity is in fact a lie, and ASTs are very special data structures, not the same plain lists that Scheme can operate with (see <code>syntax-&gt;datum</code> and <code>datum-&gt;syntax</code>).</p>
<p>Also, to illustrate how macros in a language with C-like syntax can look like - I <a href="https://github.com/combinatorylogic/clike/blob/master/tests/syntax.c" rel="nofollow noreferrer">experimented</a> with various approaches, from quasiquotation to directly accessing AST to typed macros applied in two phases, before and after typing - all seem to be quite feasible.</p>
<p>E.g., a macro that extends the syntax, adding an upside down assignment operator:</p>
<pre><code>##syntax of pfclike in clcode, inner: ' [clrevset]:v '
+ { clrevset := [clexpr]:r &quot;=:&quot; [cllvalue]:l&quot;;&quot; =&gt; 
            {mode=stmt} set($source(),l,r); } // Forming an AST directly
{
  return v
}
</code></pre>
<p>Which then can be used as</p>
<pre><code>2+2 =: a;
</code></pre>
<p>Here, we're just injecting a new parsing rule into the language parser. See the link above for more elaborate examples and for different approaches to handling C ASTs, both reading and forming them.</p>

CC BY-SA 4.0
2023-06-26T15:45:16.247Z
2023-06-29T09:38:17.650Z
2023-06-29T09:38:17.650Z
SK-logic
<p>Because it is a &quot;how could&quot; question, I'll answer with the concrete design that I am using for my language. However, I highly suggest looking at the other answers, especially the <a href="https://langdev.stackexchange.com/a/1739/134">one by Alexis King</a>, for how other languages do it as well as the general principles.</p>
<p>The compiler for my language has the regular plumbing, but it's also a plugin API.</p>
<p>This API lets users define new keywords and new lexing modes. Basically, users can define functions for parsing and functions for lexing, but those functions will only be called if the right token (keyword) is found.</p>
<p>Surprisingly, this is all that is needed for implementing the equivalent of Lisp macros in a C-like language.</p>
<p>To understand why, first remember that Lisp macros themselves all must begin with a keyword: the name of the macro, so requiring a keyword to start is not a limitation versus Lisp macros.</p>
<p>Second, Lisp macros may define entirely new syntax. This is what lexing modes are for. A user's lexing mode has to handle all possible characters, and it has to turn those into a stream of tokens.</p>
<p>Essentially, since you define the mode, you define how characters map to tokens. That is the entire essence of syntax, so a lexing mode can define new syntax.</p>
<p>And I have done this to embed a JSON-like language inside my main language.</p>
<p>And of course, your parsing function can do anything it wants with that stream of tokens.</p>
<p>The parser for my JSON-like builds up the JSON value using variables that exist in scope and then generates code to make that data available.</p>
<p>Of course, this doesn't work with just <em>any</em> compiler. Your lexer needs to be on-demand. It also should recognize when an identifier is a type, a keyword, or something else. Your parser needs to be recursive descent because that's the easiest for users to understand. Your language must be single-pass. (Good luck on getting user code to work over multiple passes.)</p>
<p>Here are some reasons you don't want this in your language:</p>
<ul>
<li>User code could have bugs and could cause your compiler to misbehave. A lot of the work I've done is the design of the API to prevent this as much as possible.</li>
<li>Your compiler now needs a JIT as well because it has to compile the <em>plugin</em> code. I hope you have a plan for the bootstrap.</li>
<li>Your language could become a bunch of dialects, kind of like Lisp itself. Or C++.</li>
</ul>
<p>But here is why I like it:</p>
<ul>
<li>The language can <a href="https://youtube.com/watch?v=lw6TaiXzHAE" rel="noreferrer">grow</a>, and its growth is not hobbled.</li>
<li>I can define all of the language keywords in the standard library, not the compiler.</li>
<li>It forces my parsing to be <em>consistent.</em> If I can't implement a keyword the usual way, its design is wrong.</li>
<li>Keywords aren't special anymore; they can exist anywhere, not just the standard library. The <code>math</code> package of my language will include keywords for standard units, like meters, and will use that to do algebra on the units and arithmetic with correct units.</li>
<li>Keywords can be fully-qualified: you need to add the package name to access a keyword.</li>
<li>You can see what dialects a file uses by looking at the imports.</li>
<li>I already use this to define DSL's, such as a DSL for a build system.</li>
</ul>

CC BY-SA 4.0
2023-06-28T18:23:07.583Z
2023-06-28T18:23:07.583Z
Gavin D. Howard
<p>It's a very different approach from lisp, but Microsoft's .net has exposed code engines from its earliest days. It does not really allow modification of the running code, but you can generate <em>new</em> code and run that in place of the old.</p>

CC BY-SA 4.0
2024-06-13T20:39:16.153Z
2024-06-13T20:39:16.153Z
SoronelHaetir
<p>I want to preface this by saying that <a href="https://www.expressionsofchange.org/dont-say-homoiconic/" rel="noreferrer">“homoiconicity” is not a useful term</a>. It is <a href="https://twitter.com/ShriramKMurthi/status/1047286807757565953" rel="noreferrer">infamously difficult to define</a>, and no working macro system experts I know choose to use it. In practice, the word “homoiconicity” seems to serve two purposes:</p>
<ol>
<li><p>To vaguely gesture at a collection of features that make compile-time metaprogramming exceptionally pleasant in Lisp-family languages.</p>
</li>
<li><p>To add an air of mysticism to Lisp.</p>
</li>
</ol>
<p>The first of those purposes refers to something very real, but we can use more specific terms to talk about them that have concrete and well-defined meanings. The second is just obscurantist folklore. Fortunately, you do not use the word in your question body, so let us simply dispense with it.</p>
<blockquote>
<p>A part of why Lisp's macros work so well is that the code is just a list of symbols, and it is therefore simple to manipulate using a regular Lisp function.</p>
</blockquote>
<p>This is certainly true in some sense, but it somewhat conflates two distinct properties of Lisp:</p>
<ul>
<li><p>Lisp has a <em>remarkably</em> simple and regular concrete syntax, and that concrete syntax has very little inherent structure. This permits <em>reading</em> a program into a concrete syntax tree (CST) without truly <em>parsing</em> it into the corresponding AST. Reading a program performs delimiter matching, but otherwise little is done beyond tokenization. The process of turning this loose “token tree” into a true AST <em>is</em> macroexpansion, so in a sense, macroexpansion <em>is</em> parsing.</p>
</li>
<li><p>Traditionally, Lisp chooses to represent these syntax trees using very simple, standard data structures like lists and symbols. This means that functions used to manipulate lists can also be used to manipulate concrete syntax trees.</p>
</li>
</ul>
<p>This first property is undoubtedly related to why Lisps are so amenable to macro-based extension, but <a href="https://langdev.stackexchange.com/questions/59/how-could-a-language-with-a-c-style-syntax-implement-homoiconicity/1669#1669">as SK-logic mentions</a>, the second property is not actually essential. It sounds nice on paper, but just spend some time reading traditional Lisp macros of any complexity and you will find yourself quickly getting lost in the sea of <code>map</code>s and <code>caddr</code>s.</p>
<p>While reuse of ordinary data structures does provide a certain economy of implementation, a great deal of real utility is obtained from departing from that model. Using richer data structures allows preserving source location and scoping information through macroexpansion at relatively little cost. A real pattern-matching and templating sublanguage (like, say, <a href="https://www2.ccs.neu.edu/racket/pubs/icfp10-cf.pdf" rel="noreferrer"><code>syntax-parse</code></a>) is exceedingly useful whichever representation you choose, and it abstracts over most of the differences in syntax representation.</p>
<blockquote>
<p>How could a language that uses a syntax more like Rust's or C's implement a way to modify code from code itself?</p>
</blockquote>
<p>Now, finally, to the real question. Recall that the key benefit for macro extensibility of Lisp’s simple syntax is that it is possible to <em>read</em> a program without <em>parsing</em> it, and this reading process produces a loose rose tree of tokens formed simply by matching delimiters. Parsing the program <em>is</em> macroexpansion; they cannot be separated into two separate phases.</p>
<p>Given this perspective, it becomes clear that the challenge of supporting more sophisticated syntax is <em>not</em> how to add macros to a traditional parser, but rather how to parse C-like syntax via macroexpansion! Fortunately, there is prior art on how to do this <strong>even in the context of a rich, procedural macro system</strong> in <a href="https://dl.acm.org/doi/abs/10.1145/2371401.2371420" rel="noreferrer">Honu: syntactic extension for algebraic notation through enforestation</a> (<a href="https://dl.acm.org/doi/pdf/10.1145/2371401.2371420" rel="noreferrer">pdf</a>). The Honu approach, along with <a href="https://www2.ccs.neu.edu/racket/pubs/icfp10-cf.pdf" rel="noreferrer"><code>syntax-parse</code></a>, was a direct inspiration for Rust’s pattern-based macro system, though they chose to implement a simplified version of it.</p>
<p>All that said, there are plenty of challenges with Honu-like approaches that remain open research problems. Certain constructions can be challenging to parse, and this level of macro-extensibility dramatically complicates the story for IDEs, language servers, and other, similar tools. Various languages are actively pursuing different points in that design space, and which approaches are the most pragmatically useful remains to be seen.</p>

CC BY-SA 4.0
2023-06-28T16:44:43.233Z
2023-06-28T16:44:43.233Z
Alexis King
CC BY-SA 4.0
2023-07-19T16:34:43.930Z
Homoiconicity is a decidable property of grammars. Of the languages you mention: CL, Scheme, Racket (and thus Honu), and even Rust are homoiconic; C is not. The E designers had [notes on how to transform C and related languages to be homoiconic](http://erights.org/elang/grammar/quasi-overview.html).
Corbin
CC BY-SA 4.0
2023-07-19T17:52:44.540Z
@Corbin I would love to see your so-called decidable property formally stated.
Alexis King
CC BY-SA 4.0
2023-07-20T00:30:22.163Z
Pick a grammar and a non-terminal production which will serve as the top level. Then the grammar is homoiconic if the production appears immediately as part of its own rule (recursion) and also indirectly via some other rule (quoting). That's it. Any way of *unquoting* or otherwise evaluating quotes is not necessary; that'd be a semantic notion. You can check for yourself that the typical Lisp qualifies, and so does Rust; however, C doesn't.
Corbin
CC BY-SA 4.0
2023-07-20T08:32:32.747Z
@Corbin Doesn’t this mean Haskell also qualifies (since it has Template Haskell)?
Alexis King
CC BY-SA 4.0
2023-07-21T03:44:25.467Z
Isn't homoiconicity partly what makes Racket special?
MicroservicesOnDDD
CC BY-SA 4.0
2023-07-21T14:39:06.730Z
Neither Haskell 98 nor Haskell 2010 qualify. I think Template Haskell qualifies; you can splice an entire module at top-level, right? And TH would only qualify if it has a way to quote modules.
Corbin
CC BY-SA 4.0
2023-07-21T18:38:03.603Z
@Corbin You can’t really “quote modules” or “splice an entire module at the top level” in Racket.
Alexis King
# How could a language with a C-style syntax implement homoiconicity?
Posted by: username_0 on 2023-05-16
A part of why Lisp's macros work so well is that the code is just a list of symbols, and it is therefore simple to manipulate using a regular Lisp function. How could a language that uses a syntax more like Rust's or C's implement a way to modify code from code itself?
**Tags:** syntax homoiconicity macro
> **Comments:**
> **username_2**: Rust *does* it. There's a `macro_rules!` macro which is basically `syntax-case`, and a very popular crate provides a `quote!` macro for quasiquoting.  
> **username_3**: Rust does it, but declarative macros only allow (in practice) a subset of what you can do with Lisp macros. Procedural macros fill the gap, but at the price of a more complicated setup and longer compile times. It's a compromise between power and efficiency  
> **username_0**: @kaya3 I've modified for clarification, though I'm also interested in indentation-syntax languages like Python.  
> **username_1**: @kouta-kun and for the Python-style languages with AST macros I recommend you to take a look at Converge.  
---
## 6 answers
### Answer by username_8 on 2023-06-28. Score: 13 | Accepted answer
I want to preface this by saying that [“homoiconicity” is not a useful term](https://www.expressionsofchange.org/dont-say-homoiconic/). It is [infamously difficult to define](https://twitter.com/ShriramKMurthi/status/1047286807757565953), and no working macro system experts I know choose to use it. In practice, the word “homoiconicity” seems to serve two purposes:

1. To vaguely gesture at a collection of features that make compile-time metaprogramming exceptionally pleasant in Lisp-family languages.
2. To add an air of mysticism to Lisp.

The first of those purposes refers to something very real, but we can use more specific terms to talk about them that have concrete and well-defined meanings. The second is just obscurantist folklore. Fortunately, you do not use the word in your question body, so let us simply dispense with it.

> A part of why Lisp's macros work so well is that the code is just a list of symbols, and it is therefore simple to manipulate using a regular Lisp function.

This is certainly true in some sense, but it somewhat conflates two distinct properties of Lisp:

* Lisp has a *remarkably* simple and regular concrete syntax, and that concrete syntax has very little inherent structure. This permits *reading* a program into a concrete syntax tree (CST) without truly *parsing* it into the corresponding AST. Reading a program performs delimiter matching, but otherwise little is done beyond tokenization. The process of turning this loose “token tree” into a true AST *is* macroexpansion, so in a sense, macroexpansion *is* parsing.
* Traditionally, Lisp chooses to represent these syntax trees using very simple, standard data structures like lists and symbols. This means that functions used to manipulate lists can also be used to manipulate concrete syntax trees.

This first property is undoubtedly related to why Lisps are so amenable to macro-based extension, but [as SK-logic mentions](https://langdev.stackexchange.com/questions/59/how-could-a-language-with-a-c-style-syntax-implement-homoiconicity/1669#1669), the second property is not actually essential. It sounds nice on paper, but just spend some time reading traditional Lisp macros of any complexity and you will find yourself quickly getting lost in the sea of `map`s and `caddr`s.

While reuse of ordinary data structures does provide a certain economy of implementation, a great deal of real utility is obtained from departing from that model. Using richer data structures allows preserving source location and scoping information through macroexpansion at relatively little cost. A real pattern-matching and templating sublanguage (like, say, [`syntax-parse`](https://www2.ccs.neu.edu/racket/pubs/icfp10-cf.pdf)) is exceedingly useful whichever representation you choose, and it abstracts over most of the differences in syntax representation.

> How could a language that uses a syntax more like Rust's or C's implement a way to modify code from code itself?

Now, finally, to the real question. Recall that the key benefit for macro extensibility of Lisp’s simple syntax is that it is possible to *read* a program without *parsing* it, and this reading process produces a loose rose tree of tokens formed simply by matching delimiters. Parsing the program *is* macroexpansion; they cannot be separated into two separate phases.

Given this perspective, it becomes clear that the challenge of supporting more sophisticated syntax is *not* how to add macros to a traditional parser, but rather how to parse C-like syntax via macroexpansion! Fortunately, there is prior art on how to do this **even in the context of a rich, procedural macro system** in [Honu: syntactic extension for algebraic notation through enforestation](https://dl.acm.org/doi/abs/10.1145/2371401.2371420) ([pdf](https://dl.acm.org/doi/pdf/10.1145/2371401.2371420)). The Honu approach, along with [`syntax-parse`](https://www2.ccs.neu.edu/racket/pubs/icfp10-cf.pdf), was a direct inspiration for Rust’s pattern-based macro system, though they chose to implement a simplified version of it.

All that said, there are plenty of challenges with Honu-like approaches that remain open research problems. Certain constructions can be challenging to parse, and this level of macro-extensibility dramatically complicates the story for IDEs, language servers, and other, similar tools. Various languages are actively pursuing different points in that design space, and which approaches are the most pragmatically useful remains to be seen.
> **Comments:**
> **username_9** (2023-07-19): Homoiconicity is a decidable property of grammars. Of the languages you mention: CL, Scheme, Racket (and thus Honu), and even Rust are homoiconic; C is not. The E designers had [notes on how to transform C and related languages to be homoiconic](http://erights.org/elang/grammar/quasi-overview.html).  
> **username_8** (2023-07-19): @Corbin I would love to see your so-called decidable property formally stated.  
> **username_9** (2023-07-20): Pick a grammar and a non-terminal production which will serve as the top level. Then the grammar is homoiconic if the production appears immediately as part of its own rule (recursion) and also indirectly via some other rule (quoting). That's it. Any way of *unquoting* or otherwise evaluating quotes is not necessary; that'd be a semantic notion. You can check for yourself that the typical Lisp qualifies, and so does Rust; however, C doesn't.  
> **username_8** (2023-07-20): @Corbin Doesn’t this mean Haskell also qualifies (since it has Template Haskell)?  
> **username_10** (2023-07-21): Isn't homoiconicity partly what makes Racket special?  
> **username_9** (2023-07-21): Neither Haskell 98 nor Haskell 2010 qualify. I think Template Haskell qualifies; you can splice an entire module at top-level, right? And TH would only qualify if it has a way to quote modules.  
> **username_8** (2023-07-21): @Corbin You can’t really “quote modules” or “splice an entire module at the top level” in Racket.  
---
### Answer by username_4 on 2023-05-16. Score: 6
They can use metaprogramming techniques that allow you to do what you want. Some examples:

1. As you said, they can use macros. Pretty similar to Lisp in this regard as they are commonly implemented there too. Some functationally is gone though.
2. They can use preprocessor directives. Lisp's have more expressive power and enable dynamic code generation. But they lose on simplicity and support.
3. They can use procedural macros. Lisp's again have more expressive power and are simpler. But Lisp's lose on Integration and complex transformations.
> **Comments:**
> **username_1** (2023-06-29): How exactly Lisp-style macros lose on complex transformations? You can wrap an entire optimising compiler into  Lisp macro.  
---
### Answer by username_5 on 2023-06-28. Score: 6
The precise meaning of homoiconicity is debated, so I’ll give up front my assumptions about what it means:

The basic code structure has a direct mapping to a general data structure.

That data structure is a basic type within the language.

Except for some trivial marker, the literal syntax for that type is identical to the code syntax.


So for example, Tcl is homoiconic in this sense because all code is made of character strings, and not only in the sense that any textual language is, but in a way that’s reflected in the language. The interpreter may store data types and code in a more efficient representation, but this is purely an optimisation—it must retain enough information to transparently convert back to text. And the notations for escaping (" … ", { … }, \) and interpolation ([ … ], $) are trivial—their size is a small constant, independent of the size of the escaped or interpolated term.
Likewise, of course, modern Lisp written with S-expressions is homoiconic, because its code is isomorphic to a rose tree of atomic terms, being the main aggregate type in the language, and quotation (quote / ') and splicing (unquote / ,, unquote-splicing / ,@) are again trivial.

But I think that early Lisp, written with M-expressions, also fits the bill. For example, from the Lisp 1.5 Programmer’s Manual(PDF) p. 15, here’s the function member that tests for membership in a list, and how it was desugared in S-expression form.
member[a;x] = [null[x]→F;eq[a;car[x]]→T;T→
      member[a;cdr[x]]]

define ((
(member (lambda (a x) (cond ((null x) f)
      ( (eq a (car x) ) t) (t (member a (cdr x))) )))
))

So there are some differences, but they’re entirely superficial. The punctuation is different, and there are a couple of infix operators, but besides that, all of the same nesting structure and ordering of leaf terms is unchanged. As such, even if the programmer doesn’t look at the S syntax, they can easily predict it from the M syntax, so they can read and write code that treats the M form as if it were the S data structure.
Another good example of nontrivial homoiconicity in practice is XSLT. The program’s input and output are usually both XML, and the program itself is represented as XML, so metaprogramming isn’t appreciably different from ordinary programming, despite the fact that XML has fairly complex syntax.

A long time ago, I designed a language where all compound terms had the same form, essentially a series of names followed by bracketed subterms. With some sugar for infix operators, the result was that code could look fairly familiar to a C programmer. For example, here’s the member function from earlier:
define [member](a, x) {
    if (null(x)) {
        false
    } else if (a = head(x)) {
        true
    } else {
        member(a, tail(x))
    }
}

This desugared to a tree structure very much like the Lisp code, which was accessible to the programmer to a limited extent. For example, member.head[0] is "a", and member.body[0].name is "_else_", because else is an infix operator. At the time, in fact my goal wasn’t metaprogramming, but only allowing some customisable syntax and some introspection, while retaining a simple top-down operator-precedence parser. But I hope it’s illustrative nevertheless.

I think these points can be assembled into a general method of designing such languages:
First, select a data structure amenable to representing a program. Typically this will be some sort of recursive tree structure with labels at the leaves and/or branches. However, recursion isn’t strictly necessary—you can always flatten the structure of a language. For example, you can prescribe a certain implicit traversal order, such as how prefix/postfix notation can do away with parentheses for operations of fixed arity; or you can require compound terms to be defined out of line and referenced by name. (Labels aren’t strictly necessary either, if a program’s meaning is determined entirely by its structure, but that’s getting into esolang territory.)
Second, reify that data type within the language. Ideally, you should ensure that conversion from data to code (let’s call it parse) and from code to data (format) have various properties you might expect:

parse(format(parse(data))) = parse(data)
Formatting preserves all information that parsing does.

format(parse(format(code))) = format(code)
Parsing preserves all information that formatting does.

format(parse(data)) = data
Parsing preserves formatting.

parse(format(code)) = code
Formatting preserves parsing.


(You could call 1 & 2 “weak roundtripping” and 3 & 4 “strong roundtripping”.)
Third and finally, find a notation for this data structure that’s compatible with C-style syntax, or whatever style your goal may be. The biggest usability constraint I see here is that your choice of data structure might need to forgo some of the compiler-specific organisation that you’d want in an AST, so that it can be conveniently manipulated in a uniform way. The bigger and more complex the language is, the more syntactic cases a programmer will need to consider when metaprogramming.
---
### Answer by username_1 on 2023-06-26. Score: 5
Homoiconicity is a nice feature but it is not necessary for Common Lisp-style metaprogramming.
All you need is quasiquotation for both constructing ASTs and for pattern-matching ASTs. If you have these two components, you can do everything a homoiconic syntax allows to do. Keep in mind that in some languages, most notably Scheme, homoiconicity is in fact a lie, and ASTs are very special data structures, not the same plain lists that Scheme can operate with (see syntax->datum and datum->syntax).
Also, to illustrate how macros in a language with C-like syntax can look like - I experimented with various approaches, from quasiquotation to directly accessing AST to typed macros applied in two phases, before and after typing - all seem to be quite feasible.
E.g., a macro that extends the syntax, adding an upside down assignment operator:
##syntax of pfclike in clcode, inner: ' [clrevset]:v '
+ { clrevset := [clexpr]:r "=:" [cllvalue]:l";" => 
            {mode=stmt} set($source(),l,r); } // Forming an AST directly
{
  return v
}

Which then can be used as
2+2 =: a;

Here, we're just injecting a new parsing rule into the language parser. See the link above for more elaborate examples and for different approaches to handling C ASTs, both reading and forming them.
---
### Answer by username_6 on 2023-06-28. Score: 5
Because it is a "how could" question, I'll answer with the concrete design that I am using for my language. However, I highly suggest looking at the other answers, especially the [one by Alexis King](https://langdev.stackexchange.com/a/1739/134), for how other languages do it as well as the general principles.

The compiler for my language has the regular plumbing, but it's also a plugin API.

This API lets users define new keywords and new lexing modes. Basically, users can define functions for parsing and functions for lexing, but those functions will only be called if the right token (keyword) is found.

Surprisingly, this is all that is needed for implementing the equivalent of Lisp macros in a C-like language.

To understand why, first remember that Lisp macros themselves all must begin with a keyword: the name of the macro, so requiring a keyword to start is not a limitation versus Lisp macros.

Second, Lisp macros may define entirely new syntax. This is what lexing modes are for. A user's lexing mode has to handle all possible characters, and it has to turn those into a stream of tokens.

Essentially, since you define the mode, you define how characters map to tokens. That is the entire essence of syntax, so a lexing mode can define new syntax.

And I have done this to embed a JSON-like language inside my main language.

And of course, your parsing function can do anything it wants with that stream of tokens.

The parser for my JSON-like builds up the JSON value using variables that exist in scope and then generates code to make that data available.

Of course, this doesn't work with just *any* compiler. Your lexer needs to be on-demand. It also should recognize when an identifier is a type, a keyword, or something else. Your parser needs to be recursive descent because that's the easiest for users to understand. Your language must be single-pass. (Good luck on getting user code to work over multiple passes.)

Here are some reasons you don't want this in your language:

* User code could have bugs and could cause your compiler to misbehave. A lot of the work I've done is the design of the API to prevent this as much as possible.
* Your compiler now needs a JIT as well because it has to compile the *plugin* code. I hope you have a plan for the bootstrap.
* Your language could become a bunch of dialects, kind of like Lisp itself. Or C++.

But here is why I like it:

* The language can [grow](https://youtube.com/watch?v=lw6TaiXzHAE), and its growth is not hobbled.
* I can define all of the language keywords in the standard library, not the compiler.
* It forces my parsing to be *consistent.* If I can't implement a keyword the usual way, its design is wrong.
* Keywords aren't special anymore; they can exist anywhere, not just the standard library. The `math` package of my language will include keywords for standard units, like meters, and will use that to do algebra on the units and arithmetic with correct units.
* Keywords can be fully-qualified: you need to add the package name to access a keyword.
* You can see what dialects a file uses by looking at the imports.
* I already use this to define DSL's, such as a DSL for a build system.
---
### Answer by username_7 on 2024-06-13. Score: 0
It's a very different approach from lisp, but Microsoft's .net has exposed code engines from its earliest days. It does not really allow modification of the running code, but you can generate *new* code and run that in place of the old.
---
