<p>As a followup to PLDI's <a href="https://languagedesign.stackexchange.com/questions/1/which-horizontal-whitespace-should-be-supported">first question</a>, about horizontal whitespace, what vertical whitespace (newlines) should be supported? I know of at least the following:</p>
<ul>
<li>Line feed (<code>\n</code>)</li>
<li>CRLF (<code>\r\n</code>)</li>
<li>Carriage return only (<code>\r</code>)</li>
<li>Form feed (<code>\f</code>, U+000C)</li>
<li>Next line (U+0085)</li>
<li>Vertical tab (<code>\v</code>, U+000B)</li>
</ul>
<p>For many languages newlines are just treated like other whitespace, while for others they have special syntactical meaning (typically those which don't require line-ending characters like <code>;</code>). What non-horizontal-whitespace characters should be considered whitespace, and what are the pros and cons of allowing each as newlines when this has a special syntactic meaning (or for counting line numbers)?</p>

<syntax><unicode><whitespace><lexing>
What vertical whitespace should be supported?
CC BY-SA 4.0
2023-05-18T18:47:41.810Z
One thing which may be worth taking into consideration is IDEs and code editors: How do they count lines? If edge cases like NELs or form feeds are counted differently in editors vs. interpreters/compilers, debugging could get annoying (this could maybe even be used maliciously somehow?)
rydwolf
CC BY-SA 4.0
2023-05-18T19:39:12.893Z
Voting to leave open, it seems to me that this is a real issue since AFAIK Unicode doesn't have a categorization for vertical separators.
kouta-kun
<p>Given that <code>\r</code> and other line endings now only exists as a cultural artifacts, there is only really <code>\r\n</code> and <code>\n</code> to consider. If we now decide to filter <code>\r</code> we end up with a single character that represents going to a new line, namely <code>\n</code>. If you pre-filter all other vertical whitespace, you'll end up with a much simpler line counting (and debug printing).</p>
<p>Whether this is the best method or not depends on language syntax (for example, does that language encourage Unicode characters in the source code?)</p>

CC BY-SA 4.0
2023-05-18T20:50:28.967Z
2023-05-18T20:51:02.333Z
2023-05-18T20:51:02.333Z
Nuoji
<p>Also consider LS    (Line Separator, U+2028) and PS    (Paragraph Separator, U+2029) supported by JavaScript. These, in combination with the ones you listed, are matched by PCRE's <code>\v</code> (vertical white space character).</p>
<p>QNX used to use RS (Record Separator, U+1E)</p>
<p>That said, <code>\r?\n</code> is the only thing that matters today, so unless you need compatibility with old or strange systems, just go with that.</p>

CC BY-SA 4.0
2023-05-18T18:46:04.857Z
2023-05-18T23:00:48.583Z
2023-05-18T23:00:48.583Z
Adám
CC BY-SA 4.0
2023-05-18T19:19:24.010Z
Is this meant to be an answer to the question, or just a **comment** about some missing items in the question's body?
Nike Dattani
CC BY-SA 4.0
2023-05-18T19:23:01.937Z
@NikeDattani For now, it answers the first half of "What non-horizontal-whitespace characters should be considered whitespace" which is also the title.
Adám
CC BY-SA 4.0
2023-05-18T19:27:10.320Z
Okay, but the answer is not clear to me. Are you saying that everything listed in the question should be supported, as well as the two examples in your first paragraph? What about U+1E? That seems like an incomplete paragraph. Also, what is meant by "For now?"
Nike Dattani
CC BY-SA 4.0
2023-05-18T23:01:26.953Z
@NikeDattani Somtimes the baby on my arm causes me to leave an MVP answer and come back later to explain futher.
Adám
CC BY-SA 4.0
2023-05-19T01:21:34.550Z
Congratulations on the baby :) Mine turned 5 months old this week!
Nike Dattani
CC BY-SA 4.0
2023-05-19T05:42:46.407Z
@NikeDattani Thank you, and to you. Mine turned 4 months old this week.
Adám
<p>Unicode fully defines the set of characters that should be regarded as terminating lines in <a href="https://www.unicode.org/reports/tr14/tr14-32.html#BK" rel="noreferrer">Annex 14</a>, with more in <a href="https://www.unicode.org/versions/Unicode15.0.0/ch05.pdf" rel="noreferrer">section 5.8 of the standard</a>. These are</p>
<ul>
<li>U+000A Line Feed (LF)</li>
<li>U+000D Carriage Return (CR) alone</li>
<li>CRLF as <a href="https://www.unicode.org/reports/tr14/tr14-32.html#LB5" rel="noreferrer">one indivisible sequence</a></li>
<li>U+000B Line Tabulation (VT) — supporting this is explicitly <em>optional</em>, and the main standard's newline function definition <em>does not include it</em> (so probably don't)</li>
<li>U+000C Form Feed (FF)</li>
<li>U+0085 Next Line (NEL), an EBCDIC round-trip compatibility character</li>
<li>U+2028 Line Separator (LS), not on your list, nor</li>
<li>U+2029 Paragraph Separator (PS)</li>
</ul>
<p>As a practical matter, however, most of these will never be encountered in source text for a language developed today — they are backwards-compatibility encodings for EBCDIC and physical teleprinters.</p>
<p>The most inclusive approach is to take all of these, but there's unlikely to be any ill effect from taking <strong>only the standard (CR)?LF pairing</strong>, and editors generally treat the rest as control characters, not line endings.</p>
<p>The only other one you'll plausibly encounter is LS, which is very occasionally used for unambiguous line marking. Even that is generally ignored — VS Code actually <em>warns you when you load the file</em> and offers to remove (not replace) them.</p>
<p><em>If</em> you're working on an EBCDIC machine then you can take NEL too, but line endings are a more complex question on those platforms and so something more tailored will be better.</p>
<p>Of special note is bare CR. Most code editors will still accept this, but don't create new files in that format, so code in a language you're creating now isn't going to use it. The most common way to end up with a lone carriage return these days is a LF-based editor that has mangled CRLF and inserted something in the middle. Treating it as a line increment only compounds the problem here, as the line number won't match how the file is displayed anywhere. Sniffing for line endings to guess is not constructive for a programming language; if you're targetting classic Mac OS or Oberon, you'll know it already.</p>
<hr />
<p>My suggested line endings are <strong>those that match <code>/\r?\n|\u2028/</code> at most</strong>, or even just <code>/\r?\n/</code>, and to make the presence of any of the other characters in the file an immediate error. If it's to be any broader than that, the only consistent choice is to match the full Unicode list, but for human-edited source code they won't actually appear.</p>

CC BY-SA 4.0
2023-05-18T21:54:55.083Z
2023-05-18T21:54:55.083Z
Michael Homer
# What vertical whitespace should be supported?
Posted by: username_0 on 2023-05-18
As a followup to PLDI's [first question](https://languagedesign.stackexchange.com/questions/1/which-horizontal-whitespace-should-be-supported), about horizontal whitespace, what vertical whitespace (newlines) should be supported? I know of at least the following:

* Line feed (`\n`)
* CRLF (`\r\n`)
* Carriage return only (`\r`)
* Form feed (`\f`, U+000C)
* Next line (U+0085)
* Vertical tab (`\v`, U+000B)

For many languages newlines are just treated like other whitespace, while for others they have special syntactical meaning (typically those which don't require line-ending characters like `;`). What non-horizontal-whitespace characters should be considered whitespace, and what are the pros and cons of allowing each as newlines when this has a special syntactic meaning (or for counting line numbers)?
**Tags:** syntax unicode whitespace lexing
> **Comments:**
> **username_0**: One thing which may be worth taking into consideration is IDEs and code editors: How do they count lines? If edge cases like NELs or form feeds are counted differently in editors vs. interpreters/compilers, debugging could get annoying (this could maybe even be used maliciously somehow?)  
> **username_1**: Voting to leave open, it seems to me that this is a real issue since AFAIK Unicode doesn't have a categorization for vertical separators.  
---
## 3 answers
### Answer by username_5 on 2023-05-18. Score: 8
Unicode fully defines the set of characters that should be regarded as terminating lines in [Annex 14](https://www.unicode.org/reports/tr14/tr14-32.html#BK), with more in [section 5.8 of the standard](https://www.unicode.org/versions/Unicode15.0.0/ch05.pdf). These are

* U+000A Line Feed (LF)
* U+000D Carriage Return (CR) alone
* CRLF as [one indivisible sequence](https://www.unicode.org/reports/tr14/tr14-32.html#LB5)
* U+000B Line Tabulation (VT) — supporting this is explicitly *optional*, and the main standard's newline function definition *does not include it* (so probably don't)
* U+000C Form Feed (FF)
* U+0085 Next Line (NEL), an EBCDIC round-trip compatibility character
* U+2028 Line Separator (LS), not on your list, nor
* U+2029 Paragraph Separator (PS)

As a practical matter, however, most of these will never be encountered in source text for a language developed today — they are backwards-compatibility encodings for EBCDIC and physical teleprinters.

The most inclusive approach is to take all of these, but there's unlikely to be any ill effect from taking **only the standard (CR)?LF pairing**, and editors generally treat the rest as control characters, not line endings.

The only other one you'll plausibly encounter is LS, which is very occasionally used for unambiguous line marking. Even that is generally ignored — VS Code actually *warns you when you load the file* and offers to remove (not replace) them.

*If* you're working on an EBCDIC machine then you can take NEL too, but line endings are a more complex question on those platforms and so something more tailored will be better.

Of special note is bare CR. Most code editors will still accept this, but don't create new files in that format, so code in a language you're creating now isn't going to use it. The most common way to end up with a lone carriage return these days is a LF-based editor that has mangled CRLF and inserted something in the middle. Treating it as a line increment only compounds the problem here, as the line number won't match how the file is displayed anywhere. Sniffing for line endings to guess is not constructive for a programming language; if you're targetting classic Mac OS or Oberon, you'll know it already.

---

My suggested line endings are **those that match `/\r?\n|\u2028/` at most**, or even just `/\r?\n/`, and to make the presence of any of the other characters in the file an immediate error. If it's to be any broader than that, the only consistent choice is to match the full Unicode list, but for human-edited source code they won't actually appear.
---
### Answer by username_2 on 2023-05-18. Score: 5
Given that `\r` and other line endings now only exists as a cultural artifacts, there is only really `\r\n` and `\n` to consider. If we now decide to filter `\r` we end up with a single character that represents going to a new line, namely `\n`. If you pre-filter all other vertical whitespace, you'll end up with a much simpler line counting (and debug printing).

Whether this is the best method or not depends on language syntax (for example, does that language encourage Unicode characters in the source code?)
---
### Answer by username_3 on 2023-05-18. Score: 5
Also consider LS (Line Separator, U+2028) and PS (Paragraph Separator, U+2029) supported by JavaScript. These, in combination with the ones you listed, are matched by PCRE's `\v` (vertical white space character).

QNX used to use RS (Record Separator, U+1E)

That said, `\r?\n` is the only thing that matters today, so unless you need compatibility with old or strange systems, just go with that.
> **Comments:**
> **username_4** (2023-05-18): Is this meant to be an answer to the question, or just a **comment** about some missing items in the question's body?  
> **username_3** (2023-05-18): @NikeDattani For now, it answers the first half of "What non-horizontal-whitespace characters should be considered whitespace" which is also the title.  
> **username_4** (2023-05-18): Okay, but the answer is not clear to me. Are you saying that everything listed in the question should be supported, as well as the two examples in your first paragraph? What about U+1E? That seems like an incomplete paragraph. Also, what is meant by "For now?"  
> **username_3** (2023-05-18): @NikeDattani Somtimes the baby on my arm causes me to leave an MVP answer and come back later to explain futher.  
> **username_4** (2023-05-19): Congratulations on the baby :) Mine turned 5 months old this week!  
> **username_3** (2023-05-19): @NikeDattani Thank you, and to you. Mine turned 4 months old this week.  
---
