<p>In template languages, most of the content is written out as-is, with only certain special markers for expression evaluation or flow control. In my attempts to develop a language of this style, I've used regex and enforced that flow control (if, for, etc) must be on a line of their own to simplify detection of structures. Is there a better way to do so? I would expect that Jinja/Freemarker/other languages don't use regex and somehow parse the template. Do they just generate huge tokens for whatever is literal text until a control structure is found?</p>

<parsing><template-languages>
How do different template languages handle parsing?
<p>My programmable markup language <a href="https://kaya3.github.io/papyri/" rel="nofollow noreferrer">Papyri</a> is not a templating language, but it has the same property that literal content can be mixed with &quot;code&quot;.</p>
<p>Papyri doesn't have &quot;open-code&quot; and &quot;close-code&quot; tags like <code>&lt;%</code> ... <code>%&gt;</code> or <code>&lt;?php</code> ... <code>?&gt;</code>; this would create too much syntactic noise for how Papyri is meant to be used. Instead, each token is parsed depending on whether &quot;literal content&quot; or &quot;code&quot; (i.e. an expression) is syntactically expected in the current position. So for example the token <code>.</code> may be parsed as text or the &quot;empty&quot; value.</p>
<p>Since the parsing context isn't fed back into the lexer, this means that a sequence of text like <code>foo bar.</code> must be lexed as <code>NAME WHITESPACE NAME DOT</code>. If the parser expects literal content then this would be parsed as a sequence of text elements, whereas if an expression is expected then the token <code>foo</code> would be consumed (and treated as a bare string) and then parsing of the expression is complete, so the remaining tokens (including the whitespace) are consumed by the next rule up (which would typically expect content, but not necessarily).</p>
<p>This makes lexing simpler compared to sometimes treating the source string <code>foo bar.</code> as a single token, which would require feedback from the parser to the lexer. As a bonus, it is straightforward to apply some useful text transformations such as collapsing sequences of whitespace, or replacing <code>-&gt;</code> with a Unicode arrow symbol.</p>

CC BY-SA 4.0
2023-11-12T17:33:25.343Z
2023-11-12T17:33:25.343Z
kaya3
<p>One option if your &quot;start code&quot; token is of length n &gt; 1 is to only check every nth character and see if it is part of the start token, if not continue checking.</p>
<p>If the character is part of the start token, then check whether it is actually part of the token. Dump whatever came before verbatim to the output and then swap over to the parser/interpreter for the actual language.</p>
<p>When compiling down a template to code to be executed then everything between code sections is going to get converted to a write equivalent with the bit between code sections as the string to be written to the output.</p>
<p>Making the part between code section one giant token is a valid option when dealing with the tokenized source. Keep in mind that you don't need to make a copy of the text the token refers to when tokenizing, instead you can make the token have an offset and length referring to the part of the file it is from, while keeping the file in memory as you deal with it.</p>

CC BY-SA 4.0
2023-11-11T19:18:33.197Z
2023-11-11T19:18:33.197Z
ratchet freak
<p>This is a fairly broad category and there certainly are (limited) systems that use regular-expression matching or string-splitting, but yes, of course any serious system uses a real parser. It's necessary to do so if, for example, the closing delimiter might appear <em>within</em> a code block as part of a string literal, because regular expression captures can't handle that.</p>
<p>Whether they &quot;generate huge tokens&quot; is more of an implementation choice; you don't <em>have</em> to have a token phase at all, they can break into smaller tokens arbitrarily, or may be passing through any literal text immediately as it's seen, but most do. Producing a long token is not an issue, and no different to a long string literal. They're not really going to be <em>that</em> long to be a problem, and this all don't present any sort of parsing difficulty.</p>
<p>For example, some production systems:</p>
<ul>
<li><a href="https://cs.opensource.google/go/go/+/refs/tags/go1.21.4:src/text/template/parse/lex.go;l=270" rel="noreferrer">Here</a> is the code of the lexer for the Go template library (as used in Hugo), emitting an <code>itemText</code> token for literal text outside template tags. It absorbs all text up until seeing <code>{{</code>.</li>
<li><a href="https://heap.space/xref/PHP-8.2/Zend/zend_language_scanner.l?r=08b75395#2264" rel="noreferrer">Here</a> is the same for PHP, producing a <code>T_INLINE_HTML</code> token. It's interwoven with some logic for different delimiters and some PHP filtering features.</li>
<li><a href="https://github.com/pallets/jinja/blob/main/src/jinja2/lexer.py#L524" rel="noreferrer">Jinja</a> calls this <code>TOKEN_DATA</code> as the default state, and deletes empty instances.</li>
<li><a href="https://github.com/apache/freemarker/blob/37b5b161a4ce5f2acdc3dac5c0beeade625048ba/src/main/javacc/FTL.jj#L4238" rel="noreferrer">Freemarker</a> uses a few separate <code>STATIC_TEXT_*</code> token types, including a &quot;false alarm&quot; token when a partial delimiter was seen, and parses all of these using a <code>PCDATA</code> production.</li>
</ul>
<p>The parsers for each of these handle these tokens as passthrough data, with trivial handling and the bulk of their work going into handling the delimited code. They expect to see interleaved literal tokens and sequences of code tokens, with delimiter tokens between them.</p>
<p>You will find similar arrangements in most comparable template languages too. They do less work than backtracking regular expressions, and parsers are generally trivial systems that don't take up a lot of the developers' effort, so they're not going to be worth avoiding. The task isn't meaningfully different than a typical parser for any other language, despite looking like it's inverted somehow.</p>

CC BY-SA 4.0
2023-11-11T19:59:48.463Z
2023-11-11T20:35:17.100Z
2023-11-11T20:35:17.100Z
Michael Homer
# How do different template languages handle parsing?
Posted by: username_0 on 2023-11-11
In template languages, most of the content is written out as-is, with only certain special markers for expression evaluation or flow control. In my attempts to develop a language of this style, I've used regex and enforced that flow control (if, for, etc) must be on a line of their own to simplify detection of structures. Is there a better way to do so? I would expect that Jinja/Freemarker/other languages don't use regex and somehow parse the template. Do they just generate huge tokens for whatever is literal text until a control structure is found?
**Tags:** parsing template-languages
---
## 3 answers
### Answer by username_3 on 2023-11-11. Score: 6 | Accepted answer
This is a fairly broad category and there certainly are (limited) systems that use regular-expression matching or string-splitting, but yes, of course any serious system uses a real parser. It's necessary to do so if, for example, the closing delimiter might appear *within* a code block as part of a string literal, because regular expression captures can't handle that.

Whether they "generate huge tokens" is more of an implementation choice; you don't *have* to have a token phase at all, they can break into smaller tokens arbitrarily, or may be passing through any literal text immediately as it's seen, but most do. Producing a long token is not an issue, and no different to a long string literal. They're not really going to be *that* long to be a problem, and this all don't present any sort of parsing difficulty.

For example, some production systems:

* [Here](https://cs.opensource.google/go/go/+/refs/tags/go1.21.4:src/text/template/parse/lex.go;l=270) is the code of the lexer for the Go template library (as used in Hugo), emitting an `itemText` token for literal text outside template tags. It absorbs all text up until seeing `{{`.
* [Here](https://heap.space/xref/PHP-8.2/Zend/zend_language_scanner.l?r=08b75395#2264) is the same for PHP, producing a `T_INLINE_HTML` token. It's interwoven with some logic for different delimiters and some PHP filtering features.
* [Jinja](https://github.com/pallets/jinja/blob/main/src/jinja2/lexer.py#L524) calls this `TOKEN_DATA` as the default state, and deletes empty instances.
* [Freemarker](https://github.com/apache/freemarker/blob/37b5b161a4ce5f2acdc3dac5c0beeade625048ba/src/main/javacc/FTL.jj#L4238) uses a few separate `STATIC_TEXT_*` token types, including a "false alarm" token when a partial delimiter was seen, and parses all of these using a `PCDATA` production.

The parsers for each of these handle these tokens as passthrough data, with trivial handling and the bulk of their work going into handling the delimited code. They expect to see interleaved literal tokens and sequences of code tokens, with delimiter tokens between them.

You will find similar arrangements in most comparable template languages too. They do less work than backtracking regular expressions, and parsers are generally trivial systems that don't take up a lot of the developers' effort, so they're not going to be worth avoiding. The task isn't meaningfully different than a typical parser for any other language, despite looking like it's inverted somehow.
---
### Answer by username_1 on 2023-11-12. Score: 1
My programmable markup language [Papyri](https://kaya3.github.io/papyri/) is not a templating language, but it has the same property that literal content can be mixed with "code".

Papyri doesn't have "open-code" and "close-code" tags like `<%` ... `%>` or `php</code ... ?>; this would create too much syntactic noise for how Papyri is meant to be used. Instead, each token is parsed depending on whether "literal content" or "code" (i.e. an expression) is syntactically expected in the current position. So for example the token . may be parsed as text or the "empty" value.`

Since the parsing context isn't fed back into the lexer, this means that a sequence of text like `foo bar.` must be lexed as `NAME WHITESPACE NAME DOT`. If the parser expects literal content then this would be parsed as a sequence of text elements, whereas if an expression is expected then the token `foo` would be consumed (and treated as a bare string) and then parsing of the expression is complete, so the remaining tokens (including the whitespace) are consumed by the next rule up (which would typically expect content, but not necessarily).

This makes lexing simpler compared to sometimes treating the source string `foo bar.` as a single token, which would require feedback from the parser to the lexer. As a bonus, it is straightforward to apply some useful text transformations such as collapsing sequences of whitespace, or replacing `->` with a Unicode arrow symbol.
---
### Answer by username_2 on 2023-11-11. Score: 1
One option if your "start code" token is of length n > 1 is to only check every nth character and see if it is part of the start token, if not continue checking.

If the character is part of the start token, then check whether it is actually part of the token. Dump whatever came before verbatim to the output and then swap over to the parser/interpreter for the actual language.

When compiling down a template to code to be executed then everything between code sections is going to get converted to a write equivalent with the bit between code sections as the string to be written to the output.

Making the part between code section one giant token is a valid option when dealing with the tokenized source. Keep in mind that you don't need to make a copy of the text the token refers to when tokenizing, instead you can make the token have an offset and length referring to the part of the file it is from, while keeping the file in memory as you deal with it.
---
