<p>Parsing shell scripts is notoriously challenging. There is a long history of subtle parsing bugs in implementations (<a href="https://unix.stackexchange.com/questions/340718/how-do-i-bring-heredoc-text-into-a-shell-script-variable">e.g.</a>), some of them long-lived, and of static analysis tools that settle for very partial coverage or that just delegate to an existing (probably also buggy) implementation.</p>
<p>What is it about the shell language syntax (and/or semantics) that makes this so hard? What would a language need to <em>avoid</em> in order to escape these difficulties?</p>

<syntax><parsing><shell-script>
What is it that makes parsing shell so hard?
CC BY-SA 4.0
2023-06-03T08:32:25.583Z
Are you thinking about a particualr shell script dialect?
Bruce Adams
CC BY-SA 4.0
2023-06-03T08:37:49.130Z
I think this is true for any of the supersets of POSIX sh and shells before that, but if there are peculiarities in one that make for a meaningful difference that would be a great basis for an answer.
Michael Homer
<p><strong>Disclaimer</strong>: I haven't implemented a shell, but I am working on <em>some</em> shell parsing.</p>
<p>The problem is that shell is a string that has to parse to lists of strings.</p>
<p>Why is it a string? Because users type a pure string at their shell, and each invocation is separate.</p>
<p>Okay, so we just need to parse a string into a list of strings. Seems easy.</p>
<p>So you can start by saying, &quot;Okay, I'll just do the equivalent of <code>str.split(' ')</code> on a command-line and drop empty strings.&quot;</p>
<p>And then users and their desires get involved.</p>
<p><strong>Users want arguments with spaces.</strong> We'll let them wrap arguments in quotes.</p>
<p><strong>Now they want quotes.</strong> Fine, we'll add escape sequences.</p>
<p><strong>But wait, if the shell can only use a constant string for a command, that's incredibly limiting.</strong> Okay, we'll add <code>$var</code> expansion syntax.</p>
<p><strong>How does expansion work if <code>$var</code> has spaces? Is it one or multiple arguments?</strong> Uh, we'll define <em>something</em>.</p>
<p><strong><code>$var</code> has quotes itself.</strong> Okay, we'll add another set of behaviors.</p>
<p><strong>Uh, oh; someone put <code>$var</code> in quotes!</strong> Well, we guess that should be fine, so we'll make it work.</p>
<p><strong>Hang on; if a command just produces a string, why can't users put commands into other commands?</strong> Sigh...we'll add command expansion.</p>
<p><strong>Some user tried to write the output of a command to a file.</strong> We hate you now, but we'll add a special character for that.</p>
<p><strong>Pipes! Everybody loves pipes!</strong> Except us parser programmers, and you all can go jump in a...ahem, we'll use a special character again.</p>
<p><strong>Hey! I want to use a string for <code>stdin</code>!</strong> For the love of peace, fine!</p>
<p><strong>Shell must be Turing-complete, or I have to use something else.</strong> What are we, magicians? Whatever, here's your <code>if</code>, <code>while</code>, and <code>for</code>.</p>
<p><strong>Are <code>if</code>, <code>while</code>, and <code>for</code> commands? Can I treat them like commands?</strong> Why in tarnation would you <em>want</em> to? And no!</p>
<p><strong>But I want them to take strings as input like <code>stdin</code> and stuff...</strong> <a href="https://www.youtube.com/watch?v=86jbnMOAcy0" rel="noreferrer">I HATE YOU!</a></p>
<p>It all comes down to taking strings from various sources and breaking them into a list of strings.</p>
<p>Other languages don't have this problem because they can directly represent lists of strings, or they have structure to make parsing easier.</p>
<p>Shells started out as pure strings, and had to stay pure strings, while adding some semblance of sensible behavior around all of the features above and more.</p>

CC BY-SA 4.0
2023-06-03T04:46:21.893Z
2023-06-03T04:46:21.893Z
Gavin D. Howard
CC BY-SA 4.0
2023-06-03T08:26:43.123Z
Entertaining answer but you could be equally scathing about the evolution of any language.
Bruce Adams
CC BY-SA 4.0
2023-06-03T13:49:32.817Z
I definitely could. Don't point me at C++.
Gavin D. Howard
<p>The answer by @gavin-d-howard gives a good list of some of the problems.</p>
<p>I just want to add some points:</p>
<ul>
<li><p>There are many different shells. Not all of them are equally hard to parse.</p>
<p>Do you mean bash, zsh, csg, dash or the posix version of sh?</p>
</li>
<li><p>The early shells did not start out with the intention of becoming programming languages. They evolved to become them over time with use.</p>
</li>
<li><p>Competition between shells meant that there was an incentive to add features from one shell to another. These weren't necessarily a good fit.</p>
</li>
<li><p>Shell scripting has been around a long time and so they have evolved.
By this I mean that evolutionary changes rather than overall design has been dominant.</p>
</li>
<li><p>Generally shells started off with line oriented parsers rather than by lexing tokens and stringing them together with grammar rules like say the ALGOL family.</p>
</li>
<li><p>When re-implementing a shell you have to take all this history into account. Fitting it into the lexer/parser mold of 'normal' programming language is challenging. There are some notable attempts.</p>
</li>
<li><p>&quot;Good enough&quot;. Shell scripting are a tool and if you use them in a normal way following the hard learned guidelines (and lint with <a href="https://www.shellcheck.net/" rel="nofollow noreferrer">ShellCheck</a> which will explain some of these rules) you won't often run into any hard to parse corner cases. How often do you use <a href="https://en.wikipedia.org/wiki/Duff%27s_device" rel="nofollow noreferrer">Duff's device</a> in C? If a compiler didn't parse it you might not notice for a long time.</p>
</li>
<li><p>If you stretch your definition of shell script you will find some that are not so hard to parse. For example <a href="https://www.tutorialspoint.com/tcl-tk/tcl_basic_syntax.htm" rel="nofollow noreferrer">TCL</a> has a very simple syntax.</p>
</li>
</ul>

CC BY-SA 4.0
2023-06-03T08:51:22.207Z
2023-06-03T08:51:22.207Z
Bruce Adams
# What is it that makes parsing shell so hard?
Posted by: username_0 on 2023-06-03
Parsing shell scripts is notoriously challenging. There is a long history of subtle parsing bugs in implementations ([e.g.](https://unix.stackexchange.com/questions/340718/how-do-i-bring-heredoc-text-into-a-shell-script-variable)), some of them long-lived, and of static analysis tools that settle for very partial coverage or that just delegate to an existing (probably also buggy) implementation.

What is it about the shell language syntax (and/or semantics) that makes this so hard? What would a language need to *avoid* in order to escape these difficulties?
**Tags:** syntax parsing shell-script
> **Comments:**
> **username_1**: Are you thinking about a particualr shell script dialect?  
> **username_0**: I think this is true for any of the supersets of POSIX sh and shells before that, but if there are peculiarities in one that make for a meaningful difference that would be a great basis for an answer.  
---
## 2 answers
### Answer by username_2 on 2023-06-03. Score: 6
**Disclaimer**: I haven't implemented a shell, but I am working on *some* shell parsing.

The problem is that shell is a string that has to parse to lists of strings.

Why is it a string? Because users type a pure string at their shell, and each invocation is separate.

Okay, so we just need to parse a string into a list of strings. Seems easy.

So you can start by saying, "Okay, I'll just do the equivalent of `str.split(' ')` on a command-line and drop empty strings."

And then users and their desires get involved.

**Users want arguments with spaces.** We'll let them wrap arguments in quotes.

**Now they want quotes.** Fine, we'll add escape sequences.

**But wait, if the shell can only use a constant string for a command, that's incredibly limiting.** Okay, we'll add `$var` expansion syntax.

**How does expansion work if `$var` has spaces? Is it one or multiple arguments?** Uh, we'll define *something*.

**`$var` has quotes itself.** Okay, we'll add another set of behaviors.

**Uh, oh; someone put `$var` in quotes!** Well, we guess that should be fine, so we'll make it work.

**Hang on; if a command just produces a string, why can't users put commands into other commands?** Sigh...we'll add command expansion.

**Some user tried to write the output of a command to a file.** We hate you now, but we'll add a special character for that.

**Pipes! Everybody loves pipes!** Except us parser programmers, and you all can go jump in a...ahem, we'll use a special character again.

**Hey! I want to use a string for `stdin`!** For the love of peace, fine!

**Shell must be Turing-complete, or I have to use something else.** What are we, magicians? Whatever, here's your `if`, `while`, and `for`.

**Are `if`, `while`, and `for` commands? Can I treat them like commands?** Why in tarnation would you *want* to? And no!

**But I want them to take strings as input like `stdin` and stuff...** [I HATE YOU!](https://www.youtube.com/watch?v=86jbnMOAcy0)

It all comes down to taking strings from various sources and breaking them into a list of strings.

Other languages don't have this problem because they can directly represent lists of strings, or they have structure to make parsing easier.

Shells started out as pure strings, and had to stay pure strings, while adding some semblance of sensible behavior around all of the features above and more.
> **Comments:**
> **username_1** (2023-06-03): Entertaining answer but you could be equally scathing about the evolution of any language.  
> **username_2** (2023-06-03): I definitely could. Don't point me at C++.  
---
### Answer by username_1 on 2023-06-03. Score: 3
The answer by @gavin-d-howard gives a good list of some of the problems.

I just want to add some points:

* There are many different shells. Not all of them are equally hard to parse.

  Do you mean bash, zsh, csg, dash or the posix version of sh?
* The early shells did not start out with the intention of becoming programming languages. They evolved to become them over time with use.
* Competition between shells meant that there was an incentive to add features from one shell to another. These weren't necessarily a good fit.
* Shell scripting has been around a long time and so they have evolved.
  By this I mean that evolutionary changes rather than overall design has been dominant.
* Generally shells started off with line oriented parsers rather than by lexing tokens and stringing them together with grammar rules like say the ALGOL family.
* When re-implementing a shell you have to take all this history into account. Fitting it into the lexer/parser mold of 'normal' programming language is challenging. There are some notable attempts.
* "Good enough". Shell scripting are a tool and if you use them in a normal way following the hard learned guidelines (and lint with [ShellCheck](https://www.shellcheck.net/) which will explain some of these rules) you won't often run into any hard to parse corner cases. How often do you use [Duff's device](https://en.wikipedia.org/wiki/Duff%27s_device) in C? If a compiler didn't parse it you might not notice for a long time.
* If you stretch your definition of shell script you will find some that are not so hard to parse. For example [TCL](https://www.tutorialspoint.com/tcl-tk/tcl_basic_syntax.htm) has a very simple syntax.
---
