<p>Suppose that there was a programming language in which there were two different syntaxes for function calls.</p>
<p>An example of this is shown below:</p>
<pre><code>print(x)
print x

mkdir &quot;reports/january/amy&quot;
mkdir(&quot;reports/january/amy&quot;)
</code></pre>
<hr />
<blockquote>
<ul>
<li><p>We consider one or more letters <code>[A-Z]</code> followed by a space <code> </code> to represent an <em><strong>opening delimiter</strong></em>.</p>
</li>
<li><p>We consider a <code>\n\r</code> and <code>\r\n</code> and <code>\n</code> and <code>\r</code> to represent a <em><strong>closing delimiter</strong></em>. That is, the end of a line is delimiter.</p>
</li>
<li><p>We consider <code>(</code> to represent an <em><strong>opening delimiter</strong></em>.</p>
</li>
<li><p>We consider <code>)</code> to represent a <em><strong>closing delimiter</strong></em>.</p>
</li>
</ul>
</blockquote>
<p>How might you tokenize such a language in light of the fact that the language allows two different syntaxes for function calls?</p>

<compilers><interpreters><lexing><tokens><preprocessor>
How would you implement a language in which the function-name could be separated from function arguments two different ways?
CC BY-SA 4.0
2023-12-17T04:08:34.690Z
Lua does something like this for single argument functions being passed literals
Seggan
CC BY-SA 4.0
2023-12-17T04:54:46.190Z
How does this affect *tokenization*?
Unrelated String
CC BY-SA 4.0
2023-12-17T15:10:27.507Z
Surprised I haven't seen Visual Basic (i.e. classic VB) mentioned as an example of a popular language having this syntax. Calls where there is no return value (or an unused and unassigned return value), can be made without brackets - like `Add a, b` to represent `Add(a, b)`. I've never liked the feature myself, but I guess it was supposed to resemble assembly language instructions (hence why it isn't compatible with return values).
Steve
CC BY-SA 4.0
2023-12-17T15:16:35.703Z
@Steve: you beat me to it…
jmoreno
CC BY-SA 4.0
2023-12-19T22:00:20.200Z
Perl also has syntax like this, doesn't it?
Barmar
<p>What you’re really facing here is a parsing issue, rather than a tokenising one.  You would <em>tokenise</em> this language the same way that you tokenise any other language — give each of those distinct classes of source element a distinct token class, and hand them over to the parser. The two cases will be handled and distinguished during parsing.</p>
<p>When parsing an expression that could be a function call, upon seeing an identifier there are four things that could follow it, identified by token classes:</p>
<ul>
<li>A left parenthesis: this puts us in the classic parenthesised argument list.</li>
<li>A right parenthesis or the end of the line; this is either a variable name or a zero-argument function call, and needs no further handling.</li>
<li>An infix operator: again, this is a variable/nullary call, but there is ordinary expression-parsing still to come (if these exist in the language; more below).</li>
<li>Anything else: this is an argument to an unparenthesised function call.</li>
</ul>
<p>In both function-call cases, standard argument-parsing occurs until the respective closing delimiter. You haven’t indicated how that works beyond these examples, but it could either be different or the same for the two cases at your option. This will be similar with a hand-written parser or a defined grammar, and both will support either using the same or different syntax for the arguments.</p>
<hr />
<p>It is possible that there are some ambiguities that can arise if you do have infix operators or complex argument lists. In those cases, it’s possible that <em>multiple</em> argument lists are closed by the same end-of-line token. For a hand-written parser, just closing any outstanding lists at the end of the line is fairly straightforward, but for a standard grammar to be consumed by a parser generator it could be a little more complicated.</p>
<p>For example, it seems like you want</p>
<pre class="lang-none prettyprint-override"><code>a b + c d
</code></pre>
<p>To mean</p>
<pre class="lang-none prettyprint-override"><code>a(b + c(d))
</code></pre>
<p>given those delimiting rules. The end of the line delimits both argument lists. This is really the only point where any significant complexity arises in the parsing, and it’s something you can choose as part of the language design. If you don’t, this one is clear.</p>
<p>You’ve suggested there can be multiple arguments, but not shown an example of how that would look. There is again a potential ambiguity when a function call with its own arguments is given as an argument to another call, and neither uses parentheses. You can decide how to handle this (e.g. prohibit it, consume greedily). This is a language design choice again.</p>
<hr />
<p>As you’ve described it so far, this is not a highly-unusual design, and a number of languages do allow at least single-argument function calls with literal arguments to be written this way. You could look to some of those &amp; their implementations for inspiration as well. Lua has been mentioned in the comments; I worked on one called Grace (<a href="https://github.com/mwh/kernan/blob/a176ef4998c9b7541f3f5fb444d6be5e64036ce7/GraceLanguage/Parsing/Parser.cs#L1201" rel="nofollow noreferrer">see parsing starting here</a>); Perl has <a href="https://perldoc.perl.org/perlsub#SYNOPSIS" rel="nofollow noreferrer">long supported multiple comma-separated arguments</a>; there are numerous others. All of these make slightly different <em>design</em> trade-offs, but the parser that follows usually isn’t any more complex than those designs entail.</p>

CC BY-SA 4.0
2023-12-17T04:56:41.847Z
2023-12-17T04:56:41.847Z
Michael Homer
# How would you implement a language in which the function-name could be separated from function arguments two different ways?
Posted by: username_0 on 2023-12-17
Suppose that there was a programming language in which there were two different syntaxes for function calls.
An example of this is shown below:
print(x)
print x

mkdir "reports/january/amy"
mkdir("reports/january/amy")




We consider one or more letters [A-Z] followed by a space   to represent an opening delimiter.

We consider a \n\r and \r\n and \n and \r to represent a closing delimiter. That is, the end of a line is delimiter.

We consider ( to represent an opening delimiter.

We consider ) to represent a closing delimiter.



How might you tokenize such a language in light of the fact that the language allows two different syntaxes for function calls?
**Tags:** compilers interpreters lexing tokens preprocessor
> **Comments:**
> **username_1**: Lua does something like this for single argument functions being passed literals  
> **username_2**: How does this affect *tokenization*?  
> **username_3**: Surprised I haven't seen Visual Basic (i.e. classic VB) mentioned as an example of a popular language having this syntax. Calls where there is no return value (or an unused and unassigned return value), can be made without brackets - like `Add a, b` to represent `Add(a, b)`. I've never liked the feature myself, but I guess it was supposed to resemble assembly language instructions (hence why it isn't compatible with return values).  
> **username_4**: @Steve: you beat me to it…  
> **username_5**: Perl also has syntax like this, doesn't it?  
---
## 1 answers
### Answer by username_6 on 2023-12-17. Score: 4
What you’re really facing here is a parsing issue, rather than a tokenising one.  You would tokenise this language the same way that you tokenise any other language — give each of those distinct classes of source element a distinct token class, and hand them over to the parser. The two cases will be handled and distinguished during parsing.
When parsing an expression that could be a function call, upon seeing an identifier there are four things that could follow it, identified by token classes:

A left parenthesis: this puts us in the classic parenthesised argument list.
A right parenthesis or the end of the line; this is either a variable name or a zero-argument function call, and needs no further handling.
An infix operator: again, this is a variable/nullary call, but there is ordinary expression-parsing still to come (if these exist in the language; more below).
Anything else: this is an argument to an unparenthesised function call.

In both function-call cases, standard argument-parsing occurs until the respective closing delimiter. You haven’t indicated how that works beyond these examples, but it could either be different or the same for the two cases at your option. This will be similar with a hand-written parser or a defined grammar, and both will support either using the same or different syntax for the arguments.

It is possible that there are some ambiguities that can arise if you do have infix operators or complex argument lists. In those cases, it’s possible that multiple argument lists are closed by the same end-of-line token. For a hand-written parser, just closing any outstanding lists at the end of the line is fairly straightforward, but for a standard grammar to be consumed by a parser generator it could be a little more complicated.
For example, it seems like you want
a b + c d

To mean
a(b + c(d))

given those delimiting rules. The end of the line delimits both argument lists. This is really the only point where any significant complexity arises in the parsing, and it’s something you can choose as part of the language design. If you don’t, this one is clear.
You’ve suggested there can be multiple arguments, but not shown an example of how that would look. There is again a potential ambiguity when a function call with its own arguments is given as an argument to another call, and neither uses parentheses. You can decide how to handle this (e.g. prohibit it, consume greedily). This is a language design choice again.

As you’ve described it so far, this is not a highly-unusual design, and a number of languages do allow at least single-argument function calls with literal arguments to be written this way. You could look to some of those & their implementations for inspiration as well. Lua has been mentioned in the comments; I worked on one called Grace (see parsing starting here); Perl has long supported multiple comma-separated arguments; there are numerous others. All of these make slightly different design trade-offs, but the parser that follows usually isn’t any more complex than those designs entail.
---
