<p>Are there any languages which allow you to get access to the <a href="https://en.wikipedia.org/wiki/Abstract_syntax_tree" rel="noreferrer">AST</a> during compile time, to use some part of the AST to generate code, or to possibly rewrite the code/AST to optimize it in a heuristic way?</p>
<p>I just found out about <a href="https://en.wikipedia.org/wiki/Rust_%28programming_language%29" rel="noreferrer">Rust</a>'s <code>derive</code> (<a href="https://web.mit.edu/rust-lang_v1.25/arch/amd64_ubuntu1404/share/doc/rust/html/book/first-edition/procedural-macros.html" rel="noreferrer">procedural macros</a>), which gives you the AST that you have to implement a handler for, like in this <a href="https://github.com/diesel-rs/diesel/blob/0f64912293756f8959e1c86d2b1e4c91da64e8ba/diesel_derives/src/lib.rs#L401-L413" rel="noreferrer">example from the diesel crate</a>.</p>
<pre><code>pub fn derive_insertable(input: TokenStream) -&gt; TokenStream {
    insertable::derive(parse_macro_input!(input))
        .unwrap_or_else(syn::Error::into_compile_error)
        .into()
}
</code></pre>
<p>I also feel like I remember reading somewhere in the <a href="https://en.wikipedia.org/wiki/Julia_%28programming_language%29" rel="noreferrer">Julia</a> language documentation that you could get the AST at compile time and rewrite it to get further optimizations, but maybe I misread and I can't find the link.</p>
<p>I am in particular wondering what could be done if you could get access to the AST at compile time, and what that might look like. Are there any other examples of languages doing this? If so, what do they allow you to do (and what is a quick example)?</p>
<p>For the (imperative/object oriented) language I'm working on, I am imagining potentially taking the definition of a recursive function and rewriting that using a flat iterative/stack based approach for optimizations. But not necessarily build this into the main compiler; somehow make it <em>pluggable</em> so you could write your own &quot;function optimizer&quot; and swap it out with a new one, or something like that. I am imagining like how <a href="https://en.wikipedia.org/wiki/ESLint" rel="noreferrer">ESLint</a> for JavaScript allows you to rewrite the AST, maybe there is something like that but for compilers. And how would that look, what an API example might look like.</p>
<p>But main thing I'm wondering about this in summary:</p>
<ul>
<li>What are the major examples of languages giving you access to the AST at compile time?</li>
<li>What might be an API example of how you would use it (and what might be a use case)?</li>
</ul>
<p>I would also be curious as to your perspective on if this would be valuable or a problematic complicated mess of a feature, but perhaps you can leave a comment on that if you've thought about it before or tried this before without success, or otherwise just know that it would be impractical.</p>

<compilers><abstract-syntax-tree>
What languages give you access to the AST to modify during compilation?
CC BY-SA 4.0
2023-07-07T01:39:43.377Z
Note that Rust doesn't give you AST during compilation. It only gives you a stream of tokens, which are then parsed to an AST through a third party crate called `syn`. This is different from the compiler passing its internal AST structure.
Deadbeef
CC BY-SA 4.0
2023-07-07T01:40:20.043Z
@Deadbeef okay good to know, sorry I misinterpreted that.
Lance Pollard
CC BY-SA 4.0
2023-07-07T10:46:42.217Z
Too small a thing an answer, so just leaving as comment - take a look at http://nemerle.org/About and https://convergepl.org/documentation/2.0/ctmp/
SK-logic
CC BY-SA 4.0
2023-07-08T08:41:15.070Z
A discussion in the context of D https://forum.dlang.org/thread/mailman.301.1278636124.24349.digitalmars-d@puremagic.com
Bruce Adams
<p>OCaml has a mechanism called <a href="https://ocaml.org/docs/metaprogramming" rel="nofollow noreferrer">PPX</a> to handle transform ASTs. Note that it also has a now-disfavored Cpp-style preprocessor to modify the code itself <a href="https://github.com/ocaml-community/cppo" rel="nofollow noreferrer">Cppo</a>.</p>
<p>There are two OCaml ASTs, <a href="https://github.com/ocaml/ocaml/blob/trunk/parsing/parsetree.mli" rel="nofollow noreferrer">parsetree</a> and <a href="https://github.com/ocaml/ocaml/blob/trunk/typing/typedtree.mli" rel="nofollow noreferrer">typedtree</a>, and PPXs work on the former.</p>
<p>You can automatically derive useful utilities like printers or equality operators like</p>
<pre class="lang-ocaml prettyprint-override"><code>type expr =
  Val of int
| Add of expr * expr [@@deriving sexp, compare, show]
</code></pre>
<p>and these are called PPX derivers that generate code from items like type declarations, expressions, etc.</p>
<p>Another form of PPX deals with extension nodes, which are directly rewritten. These are extenders.
For example, we can add monadic let-bindings</p>
<pre class="lang-ocaml prettyprint-override"><code>let rec eval = function
  | Val n -&gt; [ n ]
  | Add (x, y) -&gt;
      let open List.Let_syntax in
      let%bind n = eval x in
      let%bind m = eval y in
      return (n + m)
</code></pre>
<p>These PPXs work with <a href="https://blog.janestreet.com/extension-points-or-how-ocaml-is-becoming-more-like-lisp/" rel="nofollow noreferrer">extension points</a>, which allows for adding syntax extensions in the AST level.</p>
<p>More useful explanations are provided <a href="https://ocaml.org/docs/metaprogramming" rel="nofollow noreferrer">here</a>.</p>

CC BY-SA 4.0
2023-07-07T15:21:38.507Z
2024-04-11T03:52:24.543Z
2024-04-11T03:52:24.543Z
Jay Lee
<p>Attached <a href="https://docs.swift.org/swift-book/documentation/the-swift-programming-language/macros/" rel="nofollow noreferrer">Swift macros</a>, which are still in beta, give you access to the AST of the node they are attached to. That is, you can read the AST of the variable, function, or object declaration, and return new additions; you can't change the code that's there. There are also freestanding macros, where you return an AST but only receive the AST of the macro itself. I'd say they're pretty powerful overall, but can get very complicated very quickly as you have to crawl the AST for everything.</p>
<p>Applications are many and varied; I've done a handful for things like adding initializers, adding protocol conformance, even a couple macros to help make other macros.</p>
<p>You can read detailed design docs (<a href="https://github.com/apple/swift-evolution/blob/main/visions/macros.md" rel="nofollow noreferrer">Guiding vision document</a>; <a href="https://github.com/apple/swift-evolution/blob/main/proposals/0382-expression-macros.md" rel="nofollow noreferrer">expression</a>, <a href="https://github.com/apple/swift-evolution/blob/main/proposals/0389-attached-macros.md" rel="nofollow noreferrer">attached macros</a>, <a href="https://github.com/apple/swift-evolution/blob/main/proposals/0397-freestanding-declaration-macros.md" rel="nofollow noreferrer">freestanding macros</a>, <a href="https://github.com/apple/swift-evolution/blob/main/proposals/0402-extension-macros.md" rel="nofollow noreferrer">conformance vs. extension (currently in active review)</a>), see the discussions linked in the proposals, and read a great deal more discussion on the subject on the <a href="https://forums.swift.org/search?q=macros" rel="nofollow noreferrer">Swift evolution forum</a>.</p>
<p>As an example, here's a macro to add an <code>unknown</code> case to a string-based enum:</p>
<pre><code>public struct OpenMacro {}

extension OpenMacro: MemberMacro {
    public static func expansion(
        of node: AttributeSyntax,
        providingMembersOf declaration: some DeclGroupSyntax,
        in context: some MacroExpansionContext
    ) throws -&gt; [DeclSyntax] {
        guard let decl = declaration.as(EnumDeclSyntax.self) else {
            throw OpenError.notAnEnum
        }
        let members = decl.memberBlock.members
        let rawType = &quot;String&quot;
        let caseDecls = members.compactMap { $0.decl.as(EnumCaseDeclSyntax.self) }
        let elements = caseDecls.flatMap { $0.elements }

        let nonnullableInitializer = try InitializerDeclSyntax(&quot;init(_ rawValue: \(raw: rawType))&quot;) {
            try SwitchExprSyntax(&quot;switch rawValue&quot;) {
                for element in elements {
                    SwitchCaseSyntax(
                        &quot;&quot;&quot;
                        case \(raw: element.rawStringValue):
                            self = .\(element.identifier)
                        &quot;&quot;&quot;
                    )
                }
                SwitchCaseSyntax(&quot;default: self = .unknown(rawValue)&quot;)
            }
        }

        let rawValueDecl = try VariableDeclSyntax(&quot;var rawValue: \(raw: rawType)&quot;) {
            try CodeBlockItemListSyntax {
                try SwitchExprSyntax(&quot;switch self&quot;) {
                    for element in elements {
                        SwitchCaseSyntax(
                            &quot;&quot;&quot;

                            case .\(element.identifier.trimmed):
                                return \(raw: element.rawStringValue)
                            &quot;&quot;&quot;
                        )
                    }
                    SwitchCaseSyntax(&quot;\ncase .unknown(let rawValue): return rawValue&quot;)
                }
            }
        }

        return [
            &quot;case unknown(\(raw: rawType))&quot;,
            &quot;init?(rawValue: \(raw: rawType)) { self.init(rawValue) }&quot;,
            DeclSyntax(nonnullableInitializer),
            DeclSyntax(rawValueDecl),
        ]
    }
}

extension OpenMacro: ConformanceMacro {
    public static func expansion(
        of node: AttributeSyntax,
        providingConformancesOf declaration: some DeclGroupSyntax,
        in context: some MacroExpansionContext
    ) throws -&gt; [(TypeSyntax, GenericWhereClauseSyntax?)] {
        return [(&quot;RawRepresentable&quot;, nil)]
    }
}
</code></pre>

CC BY-SA 4.0
2023-07-07T22:59:30.333Z
2023-07-07T22:59:30.333Z
Kevin
<p>Accessing the original AST is a type of metaprogramming, and it is customary to call this ability the <code>macro</code>.</p>
<p>But the word macro is overused, often used to refer to something completely different, like VBA macros, C++ macros, Rust macros, etc. None of this is the kind of macro you want.</p>
<p>To avoid ambiguity, I use the term <code>quasiquote</code> here.</p>
<hr />
<p>In metaprogramming, <a href="https://en.wikipedia.org/wiki/Quasi-quotation" rel="nofollow noreferrer">quasiquote</a> is a feature that allows you to perform template-based code generation.</p>
<h3>Lisp family (e.g. Common Lisp, Scheme, Racket)</h3>
<p>Lisp was one of the first languages to introduce quasiquote, which is implemented through backquote (<code>`</code>) and comma (<code>,</code>).</p>
<p>Backticks are used to quote structures, while comma are used to interpolate variable values.</p>
<p>@Bbrk24 reminds me that there are more on JVM, such as Clojure, ABCL.</p>
<h3>Template Haskell</h3>
<p>Quasiquote blocks are represented by <code>[| ... |]</code>.</p>
<p>In Quasiquote blocks, <code>$</code> can be used to interpolate variables.</p>
<h3>Julia</h3>
<p>Quasiquote blocks are followed by <code>:</code> and <code>$(expression)</code> can be used as interpolate variables.</p>
<h3>Nim</h3>
<p>Metaprogramming in Nim is <a href="https://hookrace.net/blog/introduction-to-metaprogramming-in-nim/" rel="nofollow noreferrer">common</a> and typically done by directly building the abstract syntax tree with methods provided by the <code>std/macros</code> package. The <code>quote</code> routine (itself a macro) along with the <code>do</code> keyword also allows for quoting expressions, with interpolated symbols broken out with backticks <code>``</code>.</p>
<p>Such expressions are of the most use in macros or templates: templates do direct substitution on the AST, while macros allow for arbitrary compile-time operations on the syntax tree. The <code>template</code> keyword can be thought of as <code>macro foo(): quote do: ...</code></p>
<hr />
<p>Some languages also allow access to AST, but the code generation part is relatively primitive, such as concatenating strings.</p>
<p>eg. C# <code>ISourceGenerator</code></p>
<p>Some languages are implemented through third-party libraries.</p>
<p>eg. Rust <code>syn + quote</code></p>
<p>These methods are difficult to check and analyze, and bring greater difficulty to the implementation of IDE/LSP.</p>
<p>If possible, it is better to analyze it with built-in <code>quasiquote</code> within language itself.</p>

CC BY-SA 4.0
2023-07-07T02:37:54.877Z
2024-04-11T21:53:24.113Z
2024-04-11T21:53:24.113Z
Aster
CC BY-SA 4.0
2023-07-07T02:45:37.747Z
I thought that comma was a whitespace character in Lisp? I have the most exposure to Clojure though, which isn't one of the ones you listed.
Bbrk24
CC BY-SA 4.0
2023-07-07T05:04:58.757Z
Can you give links to any relevant Julia docs that explain how this works? I can't seem to find them, other than [this](https://docs.julialang.org/en/v1/devdocs/ast/). Also, could you expand a little on saying what you can accomplish with this sort of feature in the languages you mentioned? That would help me see how it's used.
Lance Pollard
CC BY-SA 4.0
2023-07-07T10:42:25.660Z
Quasiquotation allows to construct AST easily. But there is another side of it - pattern matching. When you can use quasiquotation not just for construction, but as patterns for devonstruction, code rewriting in macros is getting much easier.
SK-logic
CC BY-SA 4.0
2023-07-07T14:08:51.513Z
While Scheme does have quasiquote, it's mostly intended for constructing list structure. Metaprogramming (i.e. macros) is done using higher-level syntax objects, which represent nodes in the AST.
Barmar
CC BY-SA 4.0
2023-07-07T14:12:21.703Z
An additional feature of Common LIsp that enables you to "possibly rewrite the code/AST to optimize it in a heuristic way" is the [compiler macro](http://www.lispworks.com/documentation/lw51/CLHS/Body/m_define.htm), which exists purely to apply conditional optimizations. This is different from normal macros, which are used for source code transformation / generation.
Soupy
CC BY-SA 4.0
2023-07-08T18:46:23.843Z
@Bbrk24 In every Lisp language other than Clojure that I'm familiar with, comma is the unquote operator. I've never used Clojure, but I see that comma actually is considered whitespace in Clojure. I've never seen that before, having used Common Lisp and Scheme before. It looks like the Clojure equivalent is `~`
David Young
<p>I'd like to elaborate some more on LISP's capabilities (in addition to Aster's answer).</p>
<p>LISP is special in that its source code can be understood as being the AST (expressed as nested list structures, written in the generic text syntax for lists).</p>
<p>It has macros in the very generic sense that a &quot;macro-calling&quot; expression gets replaced by some other expression at some stage the compilation process. This is also true for e.g. the good old C language preprocessor macros.</p>
<p>But while C language preprocessor macros are nothing more than text patterns, knowing nothing about the C language syntax, the LISP macros are defined in LISP code themselves. They get the full expression of the &quot;macro call&quot;, in form of the list-structured AST, can analyze and modify anything they want about that expression using every available feature of LISP, and finally emit an expression (a list-structured new AST) that is to be compiled instead of the original one.</p>
<p>In fact, most of the control structures in LISP (e.g. loops) really are macros that typically take some body expressions and wrap them into a different outer expression, one using low-level language features. Sometimes they also look into the body and do replacements there as well. But all this is up to the macro definition, and can be implemented with the full flexibility of the LISP language.</p>
<p>As far as I understand your question, this very much matches what you asked for.</p>

CC BY-SA 4.0
2023-07-07T10:13:24.693Z
2023-07-07T10:13:24.693Z
Ralf Kleberhoff
CC BY-SA 4.0
2023-07-07T10:49:22.870Z
Keep in mind that while some Lisps AST is actually a list, it's now more common to have a completely different representation, which is more rich in metadata (e.g., stores location information), and can be converted to lists when needed. E.g., in Scheme there is `syntax->datum` and `datum->syntax` conversion between the real AST and Scheme-accessible lists.
SK-logic
<h1><a href="https://en.wikipedia.org/wiki/R_(programming_language)" rel="noreferrer">R Language</a></h1>
<p>In R, function arguments aren't evaluated until they are used, and more importantly, they can be quoted (converted into their syntax tree) using <a href="https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/substitute" rel="noreferrer"><code>substitute</code></a>. Once quoted, the argument's syntax tree can be manipulated, and then evaluated in R using <a href="https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/eval" rel="noreferrer"><code>eval</code></a> or interpreted in an entirely different way (e.g. <a href="https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/deparse" rel="noreferrer"><code>deparse</code></a> to turn it into a string).</p>
<pre class="lang-r prettyprint-override"><code>sentence &lt;- function(expr) {
  ast &lt;- substitute(expr)
  literal &lt;- deparse(ast)
  # here we substitute x = 10 and y = 5 in expr
  replace &lt;- function(ast, substs) {
    # switch is an example of a function which takes advantage of fexprs
    # (argument names are interpreted as strings, and only the matched case is evaluated)
    switch(
      typeof(ast),
      language = as.call(lapply(ast, function(child) replace(child, substs))),
      symbol = if (!is.null(substs[[ast]])) {
        substs[[ast]]
      } else {
        ast
      },
      ast
    )
  }
  answer &lt;- eval(replace(ast, list(x = 10, y = 5)))
  paste(literal, &quot;is:&quot;, answer)
}

sentence(x * 50 + y) # [1] &quot;x * 50 + y is: 505&quot;
sentence(x * x + y * y) # &quot;x * x + y * y is: 125&quot;
</code></pre>
<p>These kinds of functions (ones which don't necessarily evaluate their arguments) are known as <a href="https://en.wikipedia.org/wiki/Fexpr" rel="noreferrer"><strong>fexprs</strong></a>. They aren't quite macros because the body is just regular R code and the arguments aren't just syntax (they are <a href="https://adv-r.hadley.nz/functions.html#lazy-evaluation" rel="noreferrer">lazy R expressions</a>, if you assign them they get evaluated normally). But they aren't quite regular functions because, well, the arguments don't necessarily get evaluated and can be quoted.</p>
<p>Fexprs are really powerful, because they work like regular functions 90% of the time, but come in handy the 10% of the time you actually want lazy evaluation or metaprogramming. You can implement all of R's control constructs such as <code>if</code>, <code>while</code>, and <code>&lt;-</code> (assign) using fexprs; in fact, all of R's syntax <em>is</em> just syntactic sugar over fexprs, and <a href="https://www.r-bloggers.com/2019/10/you-can-override-just-about-anything-in-r/" rel="noreferrer">you can override almost any of the builtin language constructs</a>*.</p>
<pre class="lang-r prettyprint-override"><code>old_if &lt;- `if`
# evil if
`if` &lt;- function(cond, then, otherwise) {
  old_if(cond, otherwise, then)
}

if (2 + 2 == 4) {
  print(&quot;obviously&quot;)
} else {
  print(&quot;WHAT&quot;)
}
</code></pre>
<p>Fexprs are used in most of R's popular libraries, like <a href="https://ggplot2.tidyverse.org/" rel="noreferrer">ggplot</a> (<a href="https://raw.githubusercontent.com/rstudio/cheatsheets/master/pngs/thumbnails/data-visualization-cheatsheet-thumbs.png" rel="noreferrer">see cheatsheet</a>) and <a href="https://dplyr.tidyverse.org/" rel="noreferrer">dplyr</a> (<a href="https://raw.githubusercontent.com/rstudio/cheatsheets/main/pngs/thumbnails/data-transformation-cheatsheet-thumbs.png" rel="noreferrer">see cheatsheet</a>). Another example is in <a href="https://adv-r.hadley.nz/translation.html" rel="noreferrer">the &quot;translation&quot; section of <em>Advanced R</em></a>, it uses this mechanism to create an HTML and a LaTeX DSL.</p>
<p>To learn more, check out <a href="https://adv-r.hadley.nz/metaprogramming.html" rel="noreferrer"><em>Advanced R</em>'s metaprogramming section</a> and the <a href="http://adv-r.had.co.nz/Computing-on-the-language.html#nse" rel="noreferrer">Non-standard evaluation section</a> of the first edition. Do note that <em>Advanced R</em> uses a lot of libraries but they're not necessary, all the quoting functionality is in base R. Also check out R's documentation for definitions and examples of common functions, including ones mentioned above as well as <a href="https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/substitute" rel="noreferrer"><code>quote</code>, <code>enquote</code></a>, <a href="https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/expression" rel="noreferrer"><code>expression</code>, <code>as.expression</code></a>, <a href="https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/call" rel="noreferrer"><code>call</code>, <code>as.call</code></a>, and more.</p>
<p>* Actually, the R interpreter / JIT compiler uses hardcoded implementations of the builtins unless they're overridden. Also, byte-compiled packages won't be affected by any changes to the builtin functions, because they are already compiled with the hardcoded variants.</p>

CC BY-SA 4.0
2023-07-07T04:42:32.540Z
2023-07-07T04:42:32.540Z
tarzh
CC BY-SA 4.0
2023-07-07T05:07:24.423Z
How often or when do people use such a feature in R? Is it common or rare?
Lance Pollard
CC BY-SA 4.0
2023-07-07T15:47:47.377Z
Rare. I believe the main usage is to `deparse` the function argument and set its literal text as the table or plot label.
tarzh
<p>Groovy allows you to edit the AST, but only to the degree the built in annotations allow. For example, you can add an annotation to inherit constructors, or for creating a data class. See <a href="https://groovy-lang.org/metaprogramming.html" rel="nofollow noreferrer">this link</a> for more info.</p>

CC BY-SA 4.0
2023-07-07T02:25:25.757Z
2023-07-07T12:44:03.823Z
2023-07-07T12:44:03.823Z
Seggan
<p>In Java, an <a href="https://www.baeldung.com/java-annotation-processing-builder" rel="nofollow noreferrer">annotation processor</a> can inspect the AST and generate new code.</p>
<p>Unfortunately, its API is rather painful to use, and therefore not used often, but several popular metaprogramming libraries use this to add new syntactic sugar to the Java language.</p>
<p>While the annotation processing API is not intended to modify the AST, most compilers pass their actual AST elements to annotation processors, and with some ugly type casting, a processor <em>can</em> modify the AST. One library which does this is <a href="https://github.com/projectlombok/lombok" rel="nofollow noreferrer">Lombok</a>.</p>

CC BY-SA 4.0
2023-07-08T07:55:19.873Z
2023-07-08T07:55:19.873Z
meriton
# What languages give you access to the AST to modify during compilation?
Posted by: username_0 on 2023-07-07
Are there any languages which allow you to get access to the AST during compile time, to use some part of the AST to generate code, or to possibly rewrite the code/AST to optimize it in a heuristic way?
I just found out about Rust's derive (procedural macros), which gives you the AST that you have to implement a handler for, like in this example from the diesel crate.
pub fn derive_insertable(input: TokenStream) -> TokenStream {
    insertable::derive(parse_macro_input!(input))
        .unwrap_or_else(syn::Error::into_compile_error)
        .into()
}

I also feel like I remember reading somewhere in the Julia language documentation that you could get the AST at compile time and rewrite it to get further optimizations, but maybe I misread and I can't find the link.
I am in particular wondering what could be done if you could get access to the AST at compile time, and what that might look like. Are there any other examples of languages doing this? If so, what do they allow you to do (and what is a quick example)?
For the (imperative/object oriented) language I'm working on, I am imagining potentially taking the definition of a recursive function and rewriting that using a flat iterative/stack based approach for optimizations. But not necessarily build this into the main compiler; somehow make it pluggable so you could write your own "function optimizer" and swap it out with a new one, or something like that. I am imagining like how ESLint for JavaScript allows you to rewrite the AST, maybe there is something like that but for compilers. And how would that look, what an API example might look like.
But main thing I'm wondering about this in summary:

What are the major examples of languages giving you access to the AST at compile time?
What might be an API example of how you would use it (and what might be a use case)?

I would also be curious as to your perspective on if this would be valuable or a problematic complicated mess of a feature, but perhaps you can leave a comment on that if you've thought about it before or tried this before without success, or otherwise just know that it would be impractical.
**Tags:** compilers abstract-syntax-tree
> **Comments:**
> **username_1**: Note that Rust doesn't give you AST during compilation. It only gives you a stream of tokens, which are then parsed to an AST through a third party crate called `syn`. This is different from the compiler passing its internal AST structure.  
> **username_0**: @Deadbeef okay good to know, sorry I misinterpreted that.  
> **username_2**: Too small a thing an answer, so just leaving as comment - take a look at http://nemerle.org/About and https://convergepl.org/documentation/2.0/ctmp/  
> **username_3**: A discussion in the context of D https://forum.dlang.org/thread/mailman.301.1278636124.24349.digitalmars-d@puremagic.com  
---
## 7 answers
### Answer by username_6 on 2023-07-07. Score: 19 | Accepted answer
Accessing the original AST is a type of metaprogramming, and it is customary to call this ability the `macro`.

But the word macro is overused, often used to refer to something completely different, like VBA macros, C++ macros, Rust macros, etc. None of this is the kind of macro you want.

To avoid ambiguity, I use the term `quasiquote` here.

---

In metaprogramming, [quasiquote](https://en.wikipedia.org/wiki/Quasi-quotation) is a feature that allows you to perform template-based code generation.

### Lisp family (e.g. Common Lisp, Scheme, Racket)

Lisp was one of the first languages to introduce quasiquote, which is implemented through backquote (```) and comma (`,`).

Backticks are used to quote structures, while comma are used to interpolate variable values.

@Bbrk24 reminds me that there are more on JVM, such as Clojure, ABCL.

### Template Haskell

Quasiquote blocks are represented by `[| ... |]`.

In Quasiquote blocks, `$` can be used to interpolate variables.

### Julia

Quasiquote blocks are followed by `:` and `$(expression)` can be used as interpolate variables.

### Nim

Metaprogramming in Nim is [common](https://hookrace.net/blog/introduction-to-metaprogramming-in-nim/) and typically done by directly building the abstract syntax tree with methods provided by the `std/macros` package. The `quote` routine (itself a macro) along with the `do` keyword also allows for quoting expressions, with interpolated symbols broken out with backticks ````.

Such expressions are of the most use in macros or templates: templates do direct substitution on the AST, while macros allow for arbitrary compile-time operations on the syntax tree. The `template` keyword can be thought of as `macro foo(): quote do: ...`

---

Some languages also allow access to AST, but the code generation part is relatively primitive, such as concatenating strings.

eg. C# `ISourceGenerator`

Some languages are implemented through third-party libraries.

eg. Rust `syn + quote`

These methods are difficult to check and analyze, and bring greater difficulty to the implementation of IDE/LSP.

If possible, it is better to analyze it with built-in `quasiquote` within language itself.
> **Comments:**
> **username_7** (2023-07-07): I thought that comma was a whitespace character in Lisp? I have the most exposure to Clojure though, which isn't one of the ones you listed.  
> **username_0** (2023-07-07): Can you give links to any relevant Julia docs that explain how this works? I can't seem to find them, other than [this](https://docs.julialang.org/en/v1/devdocs/ast/). Also, could you expand a little on saying what you can accomplish with this sort of feature in the languages you mentioned? That would help me see how it's used.  
> **username_2** (2023-07-07): Quasiquotation allows to construct AST easily. But there is another side of it - pattern matching. When you can use quasiquotation not just for construction, but as patterns for devonstruction, code rewriting in macros is getting much easier.  
> **username_8** (2023-07-07): While Scheme does have quasiquote, it's mostly intended for constructing list structure. Metaprogramming (i.e. macros) is done using higher-level syntax objects, which represent nodes in the AST.  
> **username_9** (2023-07-07): An additional feature of Common LIsp that enables you to "possibly rewrite the code/AST to optimize it in a heuristic way" is the [compiler macro](http://www.lispworks.com/documentation/lw51/CLHS/Body/m_define.htm), which exists purely to apply conditional optimizations. This is different from normal macros, which are used for source code transformation / generation.  
> **username_10** (2023-07-08): @Bbrk24 In every Lisp language other than Clojure that I'm familiar with, comma is the unquote operator. I've never used Clojure, but I see that comma actually is considered whitespace in Clojure. I've never seen that before, having used Common Lisp and Scheme before. It looks like the Clojure equivalent is `~`  
---
### Answer by username_11 on 2023-07-07. Score: 9
I'd like to elaborate some more on LISP's capabilities (in addition to Aster's answer).

LISP is special in that its source code can be understood as being the AST (expressed as nested list structures, written in the generic text syntax for lists).

It has macros in the very generic sense that a "macro-calling" expression gets replaced by some other expression at some stage the compilation process. This is also true for e.g. the good old C language preprocessor macros.

But while C language preprocessor macros are nothing more than text patterns, knowing nothing about the C language syntax, the LISP macros are defined in LISP code themselves. They get the full expression of the "macro call", in form of the list-structured AST, can analyze and modify anything they want about that expression using every available feature of LISP, and finally emit an expression (a list-structured new AST) that is to be compiled instead of the original one.

In fact, most of the control structures in LISP (e.g. loops) really are macros that typically take some body expressions and wrap them into a different outer expression, one using low-level language features. Sometimes they also look into the body and do replacements there as well. But all this is up to the macro definition, and can be implemented with the full flexibility of the LISP language.

As far as I understand your question, this very much matches what you asked for.
> **Comments:**
> **username_2** (2023-07-07): Keep in mind that while some Lisps AST is actually a list, it's now more common to have a completely different representation, which is more rich in metadata (e.g., stores location information), and can be converted to lists when needed. E.g., in Scheme there is `syntax->datum` and `datum->syntax` conversion between the real AST and Scheme-accessible lists.  
---
### Answer by username_12 on 2023-07-07. Score: 7
R Language
In R, function arguments aren't evaluated until they are used, and more importantly, they can be quoted (converted into their syntax tree) using substitute. Once quoted, the argument's syntax tree can be manipulated, and then evaluated in R using eval or interpreted in an entirely different way (e.g. deparse to turn it into a string).
sentence <- function(expr) {
  ast <- substitute(expr)
  literal <- deparse(ast)
  # here we substitute x = 10 and y = 5 in expr
  replace <- function(ast, substs) {
    # switch is an example of a function which takes advantage of fexprs
    # (argument names are interpreted as strings, and only the matched case is evaluated)
    switch(
      typeof(ast),
      language = as.call(lapply(ast, function(child) replace(child, substs))),
      symbol = if (!is.null(substs[[ast]])) {
        substs[[ast]]
      } else {
        ast
      },
      ast
    )
  }
  answer <- eval(replace(ast, list(x = 10, y = 5)))
  paste(literal, "is:", answer)
}

sentence(x * 50 + y) # [1] "x * 50 + y is: 505"
sentence(x * x + y * y) # "x * x + y * y is: 125"

These kinds of functions (ones which don't necessarily evaluate their arguments) are known as fexprs. They aren't quite macros because the body is just regular R code and the arguments aren't just syntax (they are lazy R expressions, if you assign them they get evaluated normally). But they aren't quite regular functions because, well, the arguments don't necessarily get evaluated and can be quoted.
Fexprs are really powerful, because they work like regular functions 90% of the time, but come in handy the 10% of the time you actually want lazy evaluation or metaprogramming. You can implement all of R's control constructs such as if, while, and <- (assign) using fexprs; in fact, all of R's syntax is just syntactic sugar over fexprs, and you can override almost any of the builtin language constructs*.
old_if <- `if`
# evil if
`if` <- function(cond, then, otherwise) {
  old_if(cond, otherwise, then)
}

if (2 + 2 == 4) {
  print("obviously")
} else {
  print("WHAT")
}

Fexprs are used in most of R's popular libraries, like ggplot (see cheatsheet) and dplyr (see cheatsheet). Another example is in the "translation" section of Advanced R, it uses this mechanism to create an HTML and a LaTeX DSL.
To learn more, check out Advanced R's metaprogramming section and the Non-standard evaluation section of the first edition. Do note that Advanced R uses a lot of libraries but they're not necessary, all the quoting functionality is in base R. Also check out R's documentation for definitions and examples of common functions, including ones mentioned above as well as quote, enquote, expression, as.expression, call, as.call, and more.
* Actually, the R interpreter / JIT compiler uses hardcoded implementations of the builtins unless they're overridden. Also, byte-compiled packages won't be affected by any changes to the builtin functions, because they are already compiled with the hardcoded variants.
> **Comments:**
> **username_0** (2023-07-07): How often or when do people use such a feature in R? Is it common or rare?  
> **username_12** (2023-07-07): Rare. I believe the main usage is to `deparse` the function argument and set its literal text as the table or plot label.  
---
### Answer by username_13 on 2023-07-07. Score: 4
Groovy allows you to edit the AST, but only to the degree the built in annotations allow. For example, you can add an annotation to inherit constructors, or for creating a data class. See [this link](https://groovy-lang.org/metaprogramming.html) for more info.
---
### Answer by username_14 on 2023-07-08. Score: 4
In Java, an [annotation processor](https://www.baeldung.com/java-annotation-processing-builder) can inspect the AST and generate new code.

Unfortunately, its API is rather painful to use, and therefore not used often, but several popular metaprogramming libraries use this to add new syntactic sugar to the Java language.

While the annotation processing API is not intended to modify the AST, most compilers pass their actual AST elements to annotation processors, and with some ugly type casting, a processor *can* modify the AST. One library which does this is [Lombok](https://github.com/projectlombok/lombok).
---
### Answer by username_5 on 2023-07-07. Score: 3
Attached Swift macros, which are still in beta, give you access to the AST of the node they are attached to. That is, you can read the AST of the variable, function, or object declaration, and return new additions; you can't change the code that's there. There are also freestanding macros, where you return an AST but only receive the AST of the macro itself. I'd say they're pretty powerful overall, but can get very complicated very quickly as you have to crawl the AST for everything.
Applications are many and varied; I've done a handful for things like adding initializers, adding protocol conformance, even a couple macros to help make other macros.
You can read detailed design docs (Guiding vision document; expression, attached macros, freestanding macros, conformance vs. extension (currently in active review)), see the discussions linked in the proposals, and read a great deal more discussion on the subject on the Swift evolution forum.
As an example, here's a macro to add an unknown case to a string-based enum:
public struct OpenMacro {}

extension OpenMacro: MemberMacro {
    public static func expansion(
        of node: AttributeSyntax,
        providingMembersOf declaration: some DeclGroupSyntax,
        in context: some MacroExpansionContext
    ) throws -> [DeclSyntax] {
        guard let decl = declaration.as(EnumDeclSyntax.self) else {
            throw OpenError.notAnEnum
        }
        let members = decl.memberBlock.members
        let rawType = "String"
        let caseDecls = members.compactMap { $0.decl.as(EnumCaseDeclSyntax.self) }
        let elements = caseDecls.flatMap { $0.elements }

        let nonnullableInitializer = try InitializerDeclSyntax("init(_ rawValue: \(raw: rawType))") {
            try SwitchExprSyntax("switch rawValue") {
                for element in elements {
                    SwitchCaseSyntax(
                        """
                        case \(raw: element.rawStringValue):
                            self = .\(element.identifier)
                        """
                    )
                }
                SwitchCaseSyntax("default: self = .unknown(rawValue)")
            }
        }

        let rawValueDecl = try VariableDeclSyntax("var rawValue: \(raw: rawType)") {
            try CodeBlockItemListSyntax {
                try SwitchExprSyntax("switch self") {
                    for element in elements {
                        SwitchCaseSyntax(
                            """

                            case .\(element.identifier.trimmed):
                                return \(raw: element.rawStringValue)
                            """
                        )
                    }
                    SwitchCaseSyntax("\ncase .unknown(let rawValue): return rawValue")
                }
            }
        }

        return [
            "case unknown(\(raw: rawType))",
            "init?(rawValue: \(raw: rawType)) { self.init(rawValue) }",
            DeclSyntax(nonnullableInitializer),
            DeclSyntax(rawValueDecl),
        ]
    }
}

extension OpenMacro: ConformanceMacro {
    public static func expansion(
        of node: AttributeSyntax,
        providingConformancesOf declaration: some DeclGroupSyntax,
        in context: some MacroExpansionContext
    ) throws -> [(TypeSyntax, GenericWhereClauseSyntax?)] {
        return [("RawRepresentable", nil)]
    }
}
---
### Answer by username_4 on 2023-07-07. Score: 2
OCaml has a mechanism called PPX to handle transform ASTs. Note that it also has a now-disfavored Cpp-style preprocessor to modify the code itself Cppo.
There are two OCaml ASTs, parsetree and typedtree, and PPXs work on the former.
You can automatically derive useful utilities like printers or equality operators like
type expr =
  Val of int
| Add of expr * expr [@@deriving sexp, compare, show]

and these are called PPX derivers that generate code from items like type declarations, expressions, etc.
Another form of PPX deals with extension nodes, which are directly rewritten. These are extenders.
For example, we can add monadic let-bindings
let rec eval = function
  | Val n -> [ n ]
  | Add (x, y) ->
      let open List.Let_syntax in
      let%bind n = eval x in
      let%bind m = eval y in
      return (n + m)

These PPXs work with extension points, which allows for adding syntax extensions in the AST level.
More useful explanations are provided here.
---
