<p>Inspired by <a href="https://langdev.stackexchange.com/questions/2128/why-do-keywords-have-to-be-reserved-words">Why do keywords have to be reserved words?</a></p>
<p>Suppose that you're the <a href="https://en.wikipedia.org/wiki/Benevolent_dictator_for_life" rel="noreferrer">BDFL</a> of a programming language.  Version 1 of the language becomes decently popular.  A few years later, you decide to make Version 2 with a bunch of new features.  This requires new keywords to be added to the language.  For the sake of making the example concrete, let's suppose that the new keywords are <a href="https://en.wikipedia.org/wiki/Async/await" rel="noreferrer"><code>async</code> and <code>await</code></a>.</p>
<p>Problem is, there are programs written in Version 1 of the language that happen to use <code>async</code> or <code>await</code> as the name of a variable or function.  And now that these are reserved words, the existing code no longer compiles.</p>
<p>What approaches do existing programming languages take to minimize the hassle of renaming identifiers when they conflict with newly-added keywords?  How would you choose to handle this situation?</p>

<keywords><backwards-compatibility>
Preserving backwards compatibility when adding new keywords
CC BY-SA 4.0
2023-07-12T02:26:21.953Z
Another option is not adding new keywords and *'reusing'* keywords instead. Hence so many different meanings of `static` in C.
CPlus
CC BY-SA 4.0
2023-07-13T09:19:58.097Z
My first thought was that the language should require variables to have some special signifier - for example $variable. The $ sign would not be present in keywords. That way, any new keyword introduced could not possibly be conflated with an existing variable.
Chris Melville
CC BY-SA 4.0
2023-07-27T01:15:30.057Z
The question assumes that keywords are necessarily reserved words. This assumption isn't always true.  E.g FORTRAN has many keywords, but *no* reserved words.
Ray Butterworth
<h2>Different lexical forms for keywords and identifiers</h2>
<p>In my current design, identifier names must contain a capital letter, digit or underscore; anything that is purely lowercase letters is inherently a keyword. There is thus no risk of a variable name becoming a reserved word in a future version, because it can't be a valid variable name in the first place.</p>

CC BY-SA 4.0
2023-07-12T23:04:42.260Z
2023-07-27T11:48:00.120Z
2023-07-27T11:48:00.120Z
Karl Knechtel
CC BY-SA 4.0
2023-07-13T11:36:02.890Z
I would do it the other way around, using all uppercase for keywords and otherwise reserved words. Uppercase is harder to type and keywords should be very short and you need to type more letter for non-keywords. It makes it also the same as in SQL convention (but not enforced). Uppercase is also harder to read, which is better suited for something you know than something you have read completely first. It is also easer to search inside the code without accidentally finding too many matches in the comments.
12431234123412341234123
CC BY-SA 4.0
2023-07-13T11:38:45.577Z
Good answer but under "namespace" i understand something different (like C++ namespace).
12431234123412341234123
CC BY-SA 4.0
2023-07-13T19:10:34.950Z
I feel like uppercase keywords would be too visually distracting due to their frequency. For me, all-uppercase should be reserved for broad-scope constant values.
Karl Knechtel
CC BY-SA 4.0
2023-07-27T08:06:05.853Z
This idea, user23013's, user16217248's and Krazy Glew's are all quite similar, really. Arguably, different implementations of the same underlying idea.
Karl Knechtel
CC BY-SA 4.0
2023-10-13T06:00:08.880Z
@KarlKnechtel: yes, all of these syntactic distinctions expressed in the characters that are typed by and visible to the user are similar.   IMHO where it gets interesting is when there is hidden metadata with the constructs to solve many of the other problems with compatibility.  Then you don't need to have arbitrary restrictions like "keywords must be lowercase", or "keywords must be uppercase.
Krazy Glew
CC BY-SA 4.0
2023-10-13T09:51:17.417Z
@KrazyGlew I don't think I understand what you mean by "hidden" here.
Karl Knechtel
CC BY-SA 4.0
2023-10-18T22:35:09.063Z
@KarlKnechtel: fair enough.  First, if your storage format is XML I mean that the user types "thread", the IDE might convert it into <identifier text="thread"/> if there is no thread keyword - but if the keyword is later defined , stored code with the <identifier...> storage form will still work, but the IDE might choose to emit <keyword text="thread"/> or <identifier...>  depending on policy settings.  The user doesn't see the storage syntax, although the user might see different coloring or fonts.  So I guess the EFFECT of the metadata is visible, but not the exact metadata.
Krazy Glew
CC BY-SA 4.0
2023-10-18T22:45:37.800Z
@KarlKnechtel: or the example from where "await" might be a variable name in some early code, then becomes a keyword, and then later a different language feature uses the "await" keyword in a way sufficiently different that you want to distinguish <keyword text="await" version="1"/> from <keyword text="await" version="2"/>. You are correct in that the distinction will have to be made visible the user in some way - perhaps color/font, but possible just a flyover or other query.  And the user does NOT need to type or see the XML crap.
Krazy Glew
<p>You can add syntax to tell the parser which language standard to use. <a href="https://gcc.gnu.org/onlinedocs/gcc-4.3.6/gnat_rm/Pragma-Ada_005f05.html" rel="nofollow noreferrer">Ada has that</a>.</p>
<p>But, in most cases it is fine to warn the user upfront and use keywords that most people expect to be keywords anyway.</p>

CC BY-SA 4.0
2023-07-12T07:00:17.047Z
2023-07-17T19:16:50.283Z
2023-07-17T19:16:50.283Z
feldentm
<p>Python had exactly this issue (specifically for <code>async</code>/<code>await</code>), and ended up introducing those as 'soft keywords'. Essentially, there's no special case for them in the lexer, so they are treated as identifiers like any other, but there are also grammatic rules that expect these specific identifiers in specific positions. Since the new grammatic constructs would've been illegal previously anyway, no collisions take place. The only (slight) downside to this approach is performance of parsing, but the effect is miniscule</p>

CC BY-SA 4.0
2023-07-12T07:04:15.007Z
2023-07-12T07:04:15.007Z
abel1502
<p>I think that the possibility of adding new keywords reinforces the idea that they should either be a property of the base language ab initio, or should be identified in some way. As somebody else has mentioned, ALGOL mandated or at least permitted stropping, while in Modula-2 Wirth mandated that keywords were capitalised while at least preferring that non-keywords (i.e. imported from libraries etc.) were not.</p>
<p>I would suggest designing the base language such that a spurious non-keyword identifier can always be recognised as an error, and that pragmata are used to indicate e.g. storage class rather than keywords.</p>
<p>From there, I would suggest first providing an intermediate version of the language, where any existing usage of the new keyword is flagged as an error which either has to be fixed or blessed by a pragma indicating that it's been reviewed.</p>
<p>Putting keywords in their own namespace, or requiring that a program unit be imported to make them available, is one approach. But my own feeling is that they should be minimised, in order- apart from anything else- to reduce lexing problems caused by the demands of internationalisation.</p>

CC BY-SA 4.0
2023-07-13T07:29:47.577Z
2023-07-13T07:29:47.577Z
Mark Morgan Lloyd
<p>Make sure those keywords can only appear in places where program-specific identifiers can not. That way there can be an identifier that is spelled like a keyword without their being any ambiguity. This can be difficult in a C-style language where identifiers can appear in all kinds of places. But it is far more achievable in a COBOL-style language where every line begins with a keyword that says how the rest of the line is to be parsed. For example, when your syntax for calling a function looks like that:</p>
<pre><code>FUNCTION async TAKING input TYPE string
   WRITE input
ENDFUNCTION

CALL FUNCTION async PASSING await
</code></pre>
<p>then you could in the next vesion allow this:</p>
<pre><code>ASYNC FUNCTION async TAKING input TYPE string
   WRITE input
ENDFUNCTION

AWAIT FUNCTION async PASSING await
</code></pre>
<p>When every line must start with a keyword, then you can add as many more line-starter keywords as you want without there ever being a conflict.</p>

CC BY-SA 4.0
2023-07-14T08:23:24.317Z
2023-07-14T08:30:11.207Z
2023-07-14T08:30:11.207Z
Philipp
CC BY-SA 4.0
2023-07-15T15:11:30.290Z
This doesn't work for all changes, though, as some may require a new keyword to appear in a place where it could previously have been used.  See above for Eric Lippert's example of "yield" for C#.  And there's a reason nobody writes new programs in COBOL any more -- it is so ridiculously verbose that you just can't write moderately complex programs so that they're readably small.  How anyone could stand using it on *punched cards* is beyond me....
occipita
<h2>Preprocessor magic</h2>
<p>One approach that can be used if your language has a token stream to token stream preprocessor, is to utilize it in the keyword-or-identifier decision making process.</p>
<p>The basic idea is to change the lexer to always output keyword or identifiers as identifier tokens<sup>1</sup>, leaving it up to the preprocessor to replace identifier tokens representing a keyword with a keyword token.<sup>2</sup> This makes it possible to have preprocessor directives that can affect the identifier or reserved word decision. For example, you could have a preprocessor directive that turns on or off a set of keywords, or even an individual keyword. Doing it as a matched set of directives (akin to <code>#if</code> and <code>#endif</code>) can be especially useful for scoping these changes, especially for code that may be <code>#include</code>d and not know what state to return to.</p>
<p>The main advantage of this versus a language version compiler flag, is that you can write code that uses the new keyword for its new purpose in the same file, simply by wrapping the section that uses it with a preprocessor directive.</p>
<p>The only language I am aware of that does this is Verilog, which supports a <code>`begin_keywords</code> preprocessor-like directive, and a corresponding <code>`end_keywords</code>. The <code>`begin_keywords</code> directive takes an argument that indicates the standard revision that defines the keywords to support.</p>
<hr />
<p>Footnotes:</p>
<p><sup>1</sup> Traditionally a lexer will output special tokens for keywords, making them trivial for the parser to identify. Having to look inside a token to decide the parse path is more complicated, and often not supported by parser generators if your language is using one of those.</p>
<p><sup>2</sup> Obviously, there are some variations possible, like always output all reserved words as their own tokens in lexer, and having preprocessor conditionally convert them back to identifiers, etc.</p>

CC BY-SA 4.0
2023-07-26T17:30:04.150Z
2023-07-26T19:22:47.413Z
2023-07-26T19:22:47.413Z
Kevin Cathcart
<h2>Reserve unused keywords for future use</h2>
<p>Most of the time a new keyword is added to a language after its initial release, the keyword is one which exists in other languages already. The obvious examples are <code>async</code>, <code>await</code> and <code>yield</code>, but e.g. <code>const</code> and <code>let</code> were added to Javascript, <code>case</code> and <code>match</code> were added to Python, and so on.</p>
<p>This means that with some foresight, it's possible to reserve some keywords before the associated features are added to your language; then when it comes to add those features, there is no backwards compatibility problem because the keywords you need have already been reserved from day one.</p>
<p>Examples:</p>
<ul>
<li><p>In <a href="https://doc.rust-lang.org/reference/keywords.html#reserved-keywords" rel="nofollow noreferrer">Rust</a>, the keywords <code>abstract</code>, <code>become</code>, <code>box</code>, <code>do</code>, <code>final</code>, <code>macro</code>, <code>override</code>, <code>priv</code>, <code>typeof</code>, <code>unsized</code>, <code>virtual</code> and <code>yield</code> are currently reserved for future use.</p>
</li>
<li><p>In <a href="https://docs.oracle.com/javase/specs/jls/se19/html/jls-3.html#jls-3.9" rel="nofollow noreferrer">Java</a>, the keyword <code>_</code> is reserved for &quot;possible future use in parameter declarations&quot;; and <code>const</code> and <code>goto</code> are reserved to allow for better error messages in case C++ programmers use these by mistake, but it would also be possible for Java to use them for new features in the future.</p>
</li>
<li><p><a href="https://nim-lang.org/docs/manual.html#lexical-analysis-identifiers-amp-keywords" rel="nofollow noreferrer">Nim</a> likewise has some unused keywords reserved for future use, but I didn't find a comprehensive list of which ones are currently unused.</p>
</li>
</ul>

CC BY-SA 4.0
2023-07-27T11:45:06.123Z
2023-07-27T11:45:06.123Z
kaya3
CC BY-SA 4.0
2023-07-27T15:23:57.260Z
For Nim, I think the only unused keyword is `interface`.
xigoi
<p><strong><a href="https://en.wikipedia.org/wiki/Stropping_(syntax)" rel="noreferrer">Stropping</a></strong> was the approach taken in some early programming languages like Algol-60 and Algol-W.  The Wikipedia Article says</p>
<blockquote>
<p>In computer language design, stropping is a method of explicitly marking letter sequences as having a special property, such as being a keyword, or a certain type of variable or storage location, and thus inhabiting a different namespace from ordinary names (&quot;identifiers&quot;), in order to avoid clashes. Stropping is not used in most modern languages – instead, keywords are reserved words and cannot be used as identifiers. Stropping allows the same letter sequence to be used both as a keyword and as an identifier, and simplifies parsing in that case – for example allowing a variable named if without clashing with the keyword if.</p>
</blockquote>
<p>Algol-60 implementations frequently used  single quotes around a keyword, e.g. 'BEGIN'. On some machines it used underlining.  And in the language specification this was typically represented as bolding.</p>
<p>Algol 68  used several alternate strapping conventions, that could be specified by a compiler directive, like .FOR, 'FOR,  upper/lowercase...  plus they had  the option of not requiring strapping at all, and just requiring reserved words like most modern languages. In publication  keywords were typically represented as underlined or bold.</p>
<p>FORTRAN has limited stropping  for reserved words or operators like .EQ. and .AND.</p>
<p>Stropping went out of fashion  as programming language portability became  more and more important,  since the least common denominator  for composing programs was in  plain ASCII text with little or no formatting.  The usual  mechanisms of displaying stropped keywords  via fonts like <strong>bold</strong>  could not be relied on  to be portable.</p>
<p>Stropping avoids breaking existing programs if new keywords collide with  identifiers in old code.  Stropping  avoids the contortions that languages like C++ need to provide for extensibility, avoiding such conflicts. Nearly all modern programming is done using IDEs or editors that can do syntax coloring.  Syntax coloring is almost exactly what you need to do to make stropping user-friendly.  You don't need to require the user to type the stropping syntax:  the user can type in plaintext, and the editor/IDE can apply stropping automatically,  and save the stropped format in a file. If new syntax is added  with a new keyword <strong>foo</strong>  that happened to be used  as an identifier <em>foo</em>  in old programs, no problem.  There are several  portable formats that can save such stropped program representations,  if necessary with more metadata than simple .stropping.  can represent.</p>
<p>However, stropping does not completely eliminate problems with extensibility.  E.g.  if you wish to design a language that allows new language constructs, syntax,  and keywords to be added to your language.   If there are multiple such extension packages, they can conflict.  Ultimately you will need to have some sort of namespace indication  for such  syntactic extensions. So you might want to go beyond simple .stropping.,  who have representations that can  store more  metadata.</p>

CC BY-SA 4.0
2023-07-12T21:18:09.957Z
2023-07-12T21:29:08.543Z
2023-07-12T21:29:08.543Z
Krazy Glew
CC BY-SA 4.0
2023-07-13T17:47:27.343Z
A language could require that keywords be punctuated a certain way using ASCII characters in a way which may look ugly if printed in a single font without highlighting, but also specify a preferred visual representation which would look much more attractive (but be easily recognizable as mapping to the original ASCII construct).
supercat
CC BY-SA 4.0
2023-07-13T17:50:14.287Z
For example, it could specify that a Euclidian division operator be written as ` .ediv. ` in text (with white space to either side), but that editors should when practical show alphanumeric strings delimited by periods and whitespace in such fashion as italics with the periods replaced by micro-spaces, e.g. *ediv*.
supercat
CC BY-SA 4.0
2023-07-15T15:00:35.577Z
Huh.  All this time, I thought all those old papers I read with Algol code in them were just nicely formatted to make them easy to read.  Never realised that highlighting the keywords was a required feature of the language. :)
occipita
CC BY-SA 4.0
2023-10-13T05:09:34.707Z
@occipta: stropping was not a "required" feature of the language.  As I mentioned there were several different stropping conventions, and some compilers could support several, including the "no-stropping" version where things were just you were in whatever character set you had.  However, I am 99.9% certain that various Algol 68 programs had some variable names that were the same as keywords. And I think I remember this also in either Algol 58 or Algol 60. There were various tools to convert from one's dropping representation to another, which is trivial, except for keyword collision.
Krazy Glew
CC BY-SA 4.0
2023-10-13T05:18:55.223Z
Now something controversial: IMHO we should use the moral equivalent of syntax coloring or stropping to distinguish keywords from variable names, etc. It should be extensible. XML is a perfectly good exchange format. The low-level representation of your code might look like <if> if == then <then> then() </if>.   XML is that it is future extendable.  A compiler might not understand a new keyword, but it least it can recognize the code that uses it and report a meaningful error, re-synchronizing at the end of the construct </if>, rather than going wildly crazy as so often happens with C++ code.
Krazy Glew
CC BY-SA 4.0
2023-10-13T05:46:37.743Z
Stropping (XML or otherwise) means old code runs if included by new code that uses new keywords. You don't need language versions. You need not reserve keywords for the future. You don't need _crap_. You need not deprecate anything: XML stropping can have versions like <await ver="1"> and <await ver="2">.   You don't need a  verbose form like C#2 'yield return whatever' because 'yield return' has a conflict. But you need hidden metadata in the stropping format to get these benefits. XML already has this. If you don't like XML, define your own new metadata stropping format.
Krazy Glew
CC BY-SA 4.0
2023-10-13T05:48:59.513Z
Let me emphasize: a normal human would never need to know what the XML format looks like. Your IDE would hide it from you. You can type plain old ASCII, and as long as there are no conflicts or ambiguities it just gets colorized and XML stropped invisibly.
Krazy Glew
<h1>Use namespaces everywhere</h1>
<p>This could mean many things. You may choose what is appropriate in your language:</p>
<ol>
<li>A user-defined identifier could override the meaning of a keyword.</li>
<li>You could use the original keyword by explicitly specifying the namespace, for example <code>system::await</code>.</li>
<li>You could alias a keyword to a different name in a program.</li>
<li>You could import the namespace only if you need the keyword in your program.</li>
</ol>
<p>It might be easiest to implement only #4, in which case the namespace may not be called a namespace, but a feature option that you could turn on and off. But there would be problems if it exported identifiers that code using new features would use. You could support aliases in the import, but that wouldn't work well for members in a structure. But there is an easy solution, by using a special grammar for forcing the use of an identifier that is the same as a keyword. You could turn off the option in old source code files, and when you want to upgrade a file, change the identifiers with conflicting names to the new grammar. C# uses <code>@x</code>. Some SQL dialects use <code>`x`</code> or <code>[x]</code>. Shell scripts, say bash, uses <code>'x'</code> or any other quotes or escapes, and also <code>command x</code> which also avoids functions. PHP uses <code>${'x'}</code> for names that is not in the format of an identifier, but may not be the best example because variables and keywords are not in the same namespace in the first place.</p>

CC BY-SA 4.0
2023-07-12T09:38:09.137Z
2023-07-17T18:17:04.407Z
2023-07-17T18:17:04.407Z
user23013
CC BY-SA 4.0
2023-07-12T14:07:06.993Z
I do like option #4: it's similar to @ATaco's version opt-in, but more fine-grained; in particular it avoids the issue that you can't just `require` the latest version in all of your code, because that could unnecessarily break backwards compatibility.
G. Sliepen
<p>A quick and dirty approach C uses is by reserving all identifiers starting with <code>_</code>, and using <code>_Keywords</code> as built-in keywords and only having the <code>keywords</code> as convenience macros in header files.</p>

CC BY-SA 4.0
2023-07-12T02:25:18.900Z
2023-07-12T02:25:18.900Z
CPlus
<h1>Require explicit opt-in to language version</h1>
<p>Taking inspiration from C#, one can require programs declare which version of the language they are using. Then, the compiler/interpreter may choose to only handle select newer features depending on declared version.</p>
<p>An example is the <a href="https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/required" rel="noreferrer"><code>required</code></a> keyword, which in programs written for &lt;=C#10 will warn when attempting to use it, and &gt;=C#11 will refuse to accept as an identifier.</p>
<p>This allows older programs to compile and run fine on a newer compiler/interpreter,  but newer programs may opt in.</p>
<p>If version is declarable at the file or scope level, then new keywords can be used in an older program without a total rewrite.</p>

CC BY-SA 4.0
2023-07-12T02:43:31.740Z
2023-07-12T02:43:31.740Z
ATaco
CC BY-SA 4.0
2023-07-12T20:24:15.733Z
I don't know why more languages don't do this, since it simultaneously accommodates an unlimited range of language extensions while providing 100% compatibility with programs that specify what language version they require.
supercat
CC BY-SA 4.0
2023-07-13T06:50:08.947Z
@supercat Because at some point you just have several different languages instead.
Passer By
CC BY-SA 4.0
2023-07-13T15:03:59.733Z
@PasserBy: One will end up with different dialects, whether one officially recognizes the distinctions or not.  If the overall language includes feature-test and feature-demanding directives, however, then it will be possible to have different parts of a program each use whatever dialect is more appropriate for what that part of the program needs to do.
supercat
CC BY-SA 4.0
2023-07-13T17:39:01.770Z
@PasserBy: Because implementations are often specialized for different kinds of tasks which are apt to rely upon different features, recognizing a feature as "optional" would allow compiler writers whose customers wouldn't need that feature to accomplish what they need to do to direct their efforts in ways that would actually benefit their customers, rather than wasting time on support for features none of their customers would care about.  People wishing to claim their compilers are suitable for tasks should support features that are useful for those tasks...
supercat
CC BY-SA 4.0
2023-07-13T17:40:59.163Z
...but if properly-written compilers and compilers would be required to refuse compilation when a program requires features a compiler doesn't support, the Standard wouldn't otherwise have to worry about defining features that might be unsupportable on 1% of implementations.
supercat
CC BY-SA 4.0
2023-07-14T00:49:30.453Z
@supercat this is also how ultra-legacy systems are created because companies will simply put zero resources into upgrading, since "it works".
I've worked on MSSQL database stuck on some archaic  2003 compatibility level simply because the development company couldn't be arsed to do proper testing and actually use newer versions. This level is missing some critical stuff such as the `string_agg` function, and it's just terrible.
Nelson
CC BY-SA 4.0
2023-07-14T07:31:45.867Z
@supercat The point of language design is to accommodate general requirements. The small minority that really want something else needs to work out a solution themselves.
Passer By
CC BY-SA 4.0
2023-07-14T14:47:17.297Z
@PasserBy: The C Standard has forever been blocked from providing features that would be useful for many applications because a few otherwise-useful implementations would be unable to support them.  Recognizing the existence of constructs that implementations should support *when practical* would allow the Standard to recognize features that are common to implementations designed to be suitable for low-level programming on commonplace platforms.
supercat
CC BY-SA 4.0
2023-07-14T16:24:14.850Z
@PasserBy: Further, it is useful simultaneously for a language to have the notion "The type `unsigned char` is 8 bits on all conventional platforms", but also to say "The TI 32050 compiler processes a language that's just like 'ordinary C', except that character types are 16 bits", or "The Atari 400 C compiler behaves as specified by the C Standard, except that [list of features] are omitted to allow operation with 16K of RAM, and programs requiring those features will be rejected".  In general, a compiler that supports full-precision `double` may be more useful than...
supercat
CC BY-SA 4.0
2023-07-14T16:26:51.217Z
...one that lacks floating-point support, but given a choice between a compiler that supports `double` but can only handle tiny programs, or one that lacks support for `long long` and `double`, but can support larger programs, the latter may be preferable, and allowing compiler writers to omit features for which they judge support to be genuinely impractical wouldn't weaken the language processed by implementations that aren't subject to unusual resource or hardware constraints.
supercat
CC BY-SA 4.0
2023-07-24T15:28:15.823Z
@Nelson: If a language handles features that are expensive to implement as specified by offering easier-to-process ways *of accomplishing the same tasks* as well or better than the old ways, then after while the only code which uses the old ways will generally be code that was written to work with machines that are much slower than present machines; if it's necessary to migrate the machines to newer machines because the old hardware isn't available, new machines wouldn't need to run the code very efficiently to satisfy any performance requirements that had been satisfied by the old machines.
supercat
CC BY-SA 4.0
2023-07-24T15:29:44.323Z
@Nelson: Of course, if people keep using the older constructs because the maintainers of the language refuse to offer any practical alternative, then an ability to efficiently process such constructs will remain important, but instead of blaming the programmers one should blame the language maintainers for forcing programmers to continue using the old constructs.
supercat
<blockquote>
<p>Let's suppose that the new keywords are async and await. What approaches do existing programming languages take to minimize the hassle of renaming identifiers when they conflict with newly-added keywords? How would you choose to handle this situation?</p>
</blockquote>
<p>I <em>did</em> have to handle that situation, so I can tell you exactly what we did.</p>
<p>In C# 6 <code>await</code> means the operator only in methods that are declared with the <code>async</code> modifier. In C# 5 and before it was a syntax error to use <code>async</code> as a modifier, so there was no back compat issue; it is not a breaking change to turn a program that never compiled into one which does. Moreover, <code>async</code> as a modifier is legal only in a declaration; every other usage can be treated as an identifier, so again, no back compat issues.</p>
<p>That is, <code>async</code> had no <em>semantic</em> purpose in C# 6. It is only to signal to the compiler that <code>await</code> means the operator in what follows.</p>
<p>More generally, we used several different tricks throughout the evolution of C# to add new keywords. First off, it's important to realize in C# that there are &quot;reserved&quot; and &quot;contextual&quot; keywords. C# has only added contextual keywords since C# 2.  (And C# 1 started with contextual keywords; <code>get</code> and <code>set</code> are contextual for example.) Also, C# allows any identifier to be preceded by <code>@</code> to indicate that it is to be treated as an identifier, not a keyword, even if it is reserved.</p>
<p>Every time a new keyword was added the grammar and/or semantic analysis pass was carefully designed so that we could determine whether the keyword was being used in its new sense, or was an identifier.  For example when <code>yield</code> was added to C# 2 the original design was just <code>yield whatever;</code> as a statement, but that is grammatically ambiguous with declaring local <code>whatever</code> of type <code>yield</code>, so the design was modified to require <code>yield return whatever;</code>, which was always a syntax error in C# 1.</p>
<p>You've probably noticed that this choice for C# 2 was different than the choice for C# 6; we could have in C# 6 chosen a two word phrase for the operator instead of requiring a marker modifier. It could have been <code>async wait whatever;</code> and that would have been a syntax error in previous versions of the language. We considered many options like that before arriving on the marker modifier solution; consistency with <code>yield return</code> was attractive but ultimately was considered too wordy. (Whether the C# 2 design team considered a marker modifier for iterator blocks, I do not know offhand.)</p>
<p>When designing LINQ for C# 3 there were a great many contextual keywords added and it was not clear what query syntax we'd arrive at. To facilitate finding problems early one of our developers with expertise in parser theory wrote a grammar analyzer which could produce examples of expressions that parsed ambiguously for a given query grammar proposal. That certainly helped us find lurking problems early in the design process.</p>
<p>In C# 4, <code>dynamic</code> was easy. There's no special grammar logic needed, and the semantic analyzer just checks to see if <code>dynamic</code> already has a meaning in the program and if it does, it uses that, otherwise it falls back to the special type. Same for <code>var</code> in C# 3.</p>
<p>I hope that helps; let me know if you want more details on any of these techniques.</p>

CC BY-SA 4.0
2023-07-13T07:11:27.723Z
2023-07-13T08:09:10.310Z
2023-07-13T08:09:10.310Z
Eric Lippert
CC BY-SA 4.0
2023-07-15T17:57:18.140Z
Wow, one of the actual designers of a programming language I use answered my question.  I feel honored ☺️
dan04
CC BY-SA 4.0
2023-07-16T18:21:12.507Z
"yield return" is easyer to understand then "yield" for someone who have not seen it before.
Ian Ringrose
CC BY-SA 4.0
2023-07-17T19:43:19.547Z
@dan04: You're very welcome but I assure you also, I'm just some guy.
Eric Lippert
CC BY-SA 4.0
2023-07-22T20:54:19.980Z
The point about `yield return` in C# is really interesting; I have sometimes wondered why C# uses this instead of the simpler `yield`, but now of course I completely understand the "backwards-compatibility" reason behind this!
printf
<p>Depending on the syntax specifics, a new keyword can be contextual at the expense of a new ambiguity the parser must handle.</p>
<p>For example, let's say you add a prefix unary expression operator <code>await</code> to your language, and there's a program with an <code>await</code> identifier:</p>
<pre><code>var await = 4;
</code></pre>
<p>This need not be an error since it is not a legal use of <code>await</code> anyway. In fact, in most positions where an identifier is declared, a keyword is not going to be ambiguous. Later, in other expressions, when the parser encounters a sequence like</p>
<pre><code>var x = await
</code></pre>
<p>it needs to do one (or possibly more) token of lookahead to see what's after it. For example, you might have</p>
<pre><code>var x = await * 2;
</code></pre>
<p>The <code>*</code> token is enough to disambiguate.</p>
<p>Problems may arise if a certain token forms both unary and binary expressions, for example</p>
<pre><code>var x = await + await;
</code></pre>
<p>This might be <code>await (+await)</code>, or <code>(await) + (await)</code>, depending on your rules. In this situation you might choose to simply require parens to disambiguate, choose the new or old behavior consistently, or defer to other strategies as described in other answers.</p>

CC BY-SA 4.0
2023-07-12T03:37:29.210Z
2023-07-12T03:37:29.210Z
Ryan Cavanaugh
CC BY-SA 4.0
2023-07-12T11:39:07.167Z
This is why the Swift 5.9 keywords `discard`, `borrow`, and `consume` must be applied directly to an identifier rather than an expression in parentheses — `discard(self)` is just a regular function call in 5.8, while `discard self` would be illegal.
Bbrk24
<h3>Provide an upgrade tool</h3>
<p>Another approach is to not preserve backwards compatibility and instead provide tools to upgrade to the new version.</p>
<p>In the case of adding a couple of new keywords all the upgade tool would have to do is rename variables using the string that is now a reserved word.</p>
<h3>Make it possible to identify the language dialect in use</h3>
<p>This procedure is facilitated if you have something in the source code that can identify the version in use. Otherwise your tool may incorrectly treat the new keywords as variables.</p>
<h3>Example</h3>
<p>For example running <code>foo1to2</code> on:</p>
<pre><code>dialect(foolang,1)

var IamAKeyword : integer = 2;
</code></pre>
<p>It becomes:</p>
<pre><code>dialect(foolang,2)

var IamNotAKeyword : integer = 2;
</code></pre>
<p>Upgrade tools can be a pain to write if the nature of the change is complex or the syntax is complex. You can make your life easier if you consider this issue upfront.</p>
<h3>Consider context</h3>
<p>As others have said you can somewhat avoid the issue if your new syntax is illegal in the previous language version/dialect.</p>
<p>Beware though that you are trading convenience of upgrade for clarity in the language.</p>
<p>An interesting example of this is C++.
Relatively early on for pure methods the <code>= 0</code> syntax <a href="https://softwareengineering.stackexchange.com/questions/284412/why-does-c-not-have-a-pure-keyword-for-virtual-functions">was introduced</a> rather than a <code>pure</code> keyword</p>
<pre><code>class Foo
{
   virtual void someMethod() = 0;
};
</code></pre>
<p>assignment of methods was not a legal syntax.
Much later when override and final were introduced they appear in the same place. So instead of:</p>
<pre><code>class Foo
{
   override void someMethod();
};
</code></pre>
<p>Where <code>override</code> could seem to be the name of the return type say
You write:</p>
<pre><code>class Foo
{
   void someMethod() override;
};
</code></pre>
<p>There was a good link about this which I cannot find but <a href="https://stackoverflow.com/questions/32757571/why-do-the-c-language-designers-keep-re-using-keywords">https://stackoverflow.com/questions/32757571/why-do-the-c-language-designers-keep-re-using-keywords</a> is very relevant.</p>
<h3>Don't bother - just document</h3>
<p>A final option is to not to anything other than document carefully the changes.</p>
<p>You leave the community deal with the issue.
They may well come up with upgrade tool on your behalf.</p>
<p>This risks alienating your user base, particularly if you do it too often.
But if your language is good and your community loyal you can get away with it.
This is really trading user friendliness for development resources.</p>
<p>There are quite a few languages that are guilty of this (despite having ample developement budget) and remain popular.</p>
<h3>Deprecate</h3>
<p>When evolving a language you could consider formally deprecating interfaces in one version before removing or changing them in a subsequent version. This slows down changes to your langauge but makes it easier for users to manage the transition.</p>
<p>This is particularly useful in 'enterprise' environments.</p>
<p>You can also consider having a syntactic or semantic way to mark something as deprecated in the language itself.
For example an attribute like <a href="https://en.cppreference.com/w/cpp/language/attributes/deprecated" rel="nofollow noreferrer">[[deprecated]]</a> in C++ can be used to indicate a function is to be retired.</p>
<p>This may or may not be harder to do for a keyword. In some languages keywords are a lot like functions or even exactly the same (e.g. <a href="https://stackoverflow.com/questions/44407332/checking-whether-a-word-is-tcl-keyword-or-not">TCL</a>.</p>
<p>My personal preference is both:</p>
<ol>
<li>a clear way to the dialect/version used.</li>
<li>an upgrade tool</li>
</ol>

CC BY-SA 4.0
2023-07-12T08:26:16.837Z
2023-08-01T15:54:49.090Z
2023-08-01T15:54:49.090Z
Bruce Adams
CC BY-SA 4.0
2023-07-12T15:44:50.930Z
Along with the "don't bother" option should be "warn ahead of time". This is similar to deprecating features -- you warn about it during release N-1, then implement the breaking change in release N.
Barmar
CC BY-SA 4.0
2023-07-13T17:43:33.090Z
@Barmar: An important aspect of deprecating features is ensuring that anything that could be done using a deprecated feature can be done at least as well--if not better--without it.
supercat
CC BY-SA 4.0
2023-07-14T02:40:05.897Z
In this case, you just have to rename any conflicting variables. @supercat The warning gives you time to do that.
Barmar
CC BY-SA 4.0
2023-07-14T04:50:44.317Z
In this particular case, that would be true unless one needs to link with code outside one's control.  But in the more general case, depecation requires ensuring that replacements exist--a concept that maintainers of the C language ignore.
supercat
CC BY-SA 4.0
2023-07-24T12:56:57.623Z
@supercat Java deprecated `sun.misc.Unsafe` without providing a replacement or alternative. If they had provided a replacement then it would defeat the purpose of deprecating it, which was to remove something that was never supposed to be available to users in the first place. I think you are using the word "requires" to mean that this is what you think language maintainers *should* do, but AFAIK this is not part of the definition of the word "deprecate".
kaya3
CC BY-SA 4.0
2023-07-24T15:01:07.663Z
@kaya3-supportthestrike: An action which removes a language construct that is necessary for some tasks without providing a viable replacement will split the language into a family of dialects that remain suitable for the aforementioned tasks, and a family that are suitable only for tasks not involving that construct, but might process those tasks better than the earlier language did.  Such a split is fine if one recognizes that the more limited dialects are't the same language as the one from which the aformentioned constructs are removed.  BTW, an important thing to note with C...
supercat
CC BY-SA 4.0
2023-07-24T15:10:57.207Z
...is that the Standard was chartered to describe a family of *pre-existing dialects*, but rather than describing features that were and should have remained common to 90%+ of them, it deliberately waived jurisdiction over features that couldn't be practically supported on 100% of them.  The only way the C Standard could be viewed as ever having accurately describing the popular language is if it was expected that implementations that processed more behaviors than mandated would continue to do so *as the authors have said in the published Rationale that they intended*.
supercat
# Preserving backwards compatibility when adding new keywords
Posted by: username_0 on 2023-07-12
Inspired by [Why do keywords have to be reserved words?](https://langdev.stackexchange.com/questions/2128/why-do-keywords-have-to-be-reserved-words)

Suppose that you're the [BDFL](https://en.wikipedia.org/wiki/Benevolent_dictator_for_life) of a programming language. Version 1 of the language becomes decently popular. A few years later, you decide to make Version 2 with a bunch of new features. This requires new keywords to be added to the language. For the sake of making the example concrete, let's suppose that the new keywords are [`async` and `await`](https://en.wikipedia.org/wiki/Async/await).

Problem is, there are programs written in Version 1 of the language that happen to use `async` or `await` as the name of a variable or function. And now that these are reserved words, the existing code no longer compiles.

What approaches do existing programming languages take to minimize the hassle of renaming identifiers when they conflict with newly-added keywords? How would you choose to handle this situation?
**Tags:** keywords backwards-compatibility
> **Comments:**
> **username_1**: Another option is not adding new keywords and *'reusing'* keywords instead. Hence so many different meanings of `static` in C.  
> **username_2**: My first thought was that the language should require variables to have some special signifier - for example $variable. The $ sign would not be present in keywords. That way, any new keyword introduced could not possibly be conflated with an existing variable.  
> **username_3**: The question assumes that keywords are necessarily reserved words. This assumption isn't always true.  E.g FORTRAN has many keywords, but *no* reserved words.  
---
## 14 answers
### Answer by username_18 on 2023-07-12. Score: 25
# Require explicit opt-in to language version

Taking inspiration from C#, one can require programs declare which version of the language they are using. Then, the compiler/interpreter may choose to only handle select newer features depending on declared version.

An example is the [`required`](https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/required) keyword, which in programs written for <=C#10 will warn when attempting to use it, and >=C#11 will refuse to accept as an identifier.

This allows older programs to compile and run fine on a newer compiler/interpreter, but newer programs may opt in.

If version is declarable at the file or scope level, then new keywords can be used in an older program without a total rewrite.
> **Comments:**
> **username_15** (2023-07-12): I don't know why more languages don't do this, since it simultaneously accommodates an unlimited range of language extensions while providing 100% compatibility with programs that specify what language version they require.  
> **username_19** (2023-07-13): @supercat Because at some point you just have several different languages instead.  
> **username_15** (2023-07-13): @PasserBy: One will end up with different dialects, whether one officially recognizes the distinctions or not.  If the overall language includes feature-test and feature-demanding directives, however, then it will be possible to have different parts of a program each use whatever dialect is more appropriate for what that part of the program needs to do.  
> **username_15** (2023-07-13): @PasserBy: Because implementations are often specialized for different kinds of tasks which are apt to rely upon different features, recognizing a feature as "optional" would allow compiler writers whose customers wouldn't need that feature to accomplish what they need to do to direct their efforts in ways that would actually benefit their customers, rather than wasting time on support for features none of their customers would care about.  People wishing to claim their compilers are suitable for tasks should support features that are useful for those tasks...  
> **username_15** (2023-07-13): ...but if properly-written compilers and compilers would be required to refuse compilation when a program requires features a compiler doesn't support, the Standard wouldn't otherwise have to worry about defining features that might be unsupportable on 1% of implementations.  
> **username_20** (2023-07-14): @supercat this is also how ultra-legacy systems are created because companies will simply put zero resources into upgrading, since "it works".
I've worked on MSSQL database stuck on some archaic  2003 compatibility level simply because the development company couldn't be arsed to do proper testing and actually use newer versions. This level is missing some critical stuff such as the `string_agg` function, and it's just terrible.  
> **username_19** (2023-07-14): @supercat The point of language design is to accommodate general requirements. The small minority that really want something else needs to work out a solution themselves.  
> **username_15** (2023-07-14): @PasserBy: The C Standard has forever been blocked from providing features that would be useful for many applications because a few otherwise-useful implementations would be unable to support them.  Recognizing the existence of constructs that implementations should support *when practical* would allow the Standard to recognize features that are common to implementations designed to be suitable for low-level programming on commonplace platforms.  
> **username_15** (2023-07-14): @PasserBy: Further, it is useful simultaneously for a language to have the notion "The type `unsigned char` is 8 bits on all conventional platforms", but also to say "The TI 32050 compiler processes a language that's just like 'ordinary C', except that character types are 16 bits", or "The Atari 400 C compiler behaves as specified by the C Standard, except that [list of features] are omitted to allow operation with 16K of RAM, and programs requiring those features will be rejected".  In general, a compiler that supports full-precision `double` may be more useful than...  
> **username_15** (2023-07-14): ...one that lacks floating-point support, but given a choice between a compiler that supports `double` but can only handle tiny programs, or one that lacks support for `long long` and `double`, but can support larger programs, the latter may be preferable, and allowing compiler writers to omit features for which they judge support to be genuinely impractical wouldn't weaken the language processed by implementations that aren't subject to unusual resource or hardware constraints.  
> **username_15** (2023-07-24): @Nelson: If a language handles features that are expensive to implement as specified by offering easier-to-process ways *of accomplishing the same tasks* as well or better than the old ways, then after while the only code which uses the old ways will generally be code that was written to work with machines that are much slower than present machines; if it's necessary to migrate the machines to newer machines because the old hardware isn't available, new machines wouldn't need to run the code very efficiently to satisfy any performance requirements that had been satisfied by the old machines.  
> **username_15** (2023-07-24): @Nelson: Of course, if people keep using the older constructs because the maintainers of the language refuse to offer any practical alternative, then an ability to efficiently process such constructs will remain important, but instead of blaming the programmers one should blame the language maintainers for forcing programmers to continue using the old constructs.  
---
### Answer by username_21 on 2023-07-13. Score: 17
> Let's suppose that the new keywords are async and await. What approaches do existing programming languages take to minimize the hassle of renaming identifiers when they conflict with newly-added keywords? How would you choose to handle this situation?

I *did* have to handle that situation, so I can tell you exactly what we did.

In C# 6 `await` means the operator only in methods that are declared with the `async` modifier. In C# 5 and before it was a syntax error to use `async` as a modifier, so there was no back compat issue; it is not a breaking change to turn a program that never compiled into one which does. Moreover, `async` as a modifier is legal only in a declaration; every other usage can be treated as an identifier, so again, no back compat issues.

That is, `async` had no *semantic* purpose in C# 6. It is only to signal to the compiler that `await` means the operator in what follows.

More generally, we used several different tricks throughout the evolution of C# to add new keywords. First off, it's important to realize in C# that there are "reserved" and "contextual" keywords. C# has only added contextual keywords since C# 2. (And C# 1 started with contextual keywords; `get` and `set` are contextual for example.) Also, C# allows any identifier to be preceded by `@` to indicate that it is to be treated as an identifier, not a keyword, even if it is reserved.

Every time a new keyword was added the grammar and/or semantic analysis pass was carefully designed so that we could determine whether the keyword was being used in its new sense, or was an identifier. For example when `yield` was added to C# 2 the original design was just `yield whatever;` as a statement, but that is grammatically ambiguous with declaring local `whatever` of type `yield`, so the design was modified to require `yield return whatever;`, which was always a syntax error in C# 1.

You've probably noticed that this choice for C# 2 was different than the choice for C# 6; we could have in C# 6 chosen a two word phrase for the operator instead of requiring a marker modifier. It could have been `async wait whatever;` and that would have been a syntax error in previous versions of the language. We considered many options like that before arriving on the marker modifier solution; consistency with `yield return` was attractive but ultimately was considered too wordy. (Whether the C# 2 design team considered a marker modifier for iterator blocks, I do not know offhand.)

When designing LINQ for C# 3 there were a great many contextual keywords added and it was not clear what query syntax we'd arrive at. To facilitate finding problems early one of our developers with expertise in parser theory wrote a grammar analyzer which could produce examples of expressions that parsed ambiguously for a given query grammar proposal. That certainly helped us find lurking problems early in the design process.

In C# 4, `dynamic` was easy. There's no special grammar logic needed, and the semantic analyzer just checks to see if `dynamic` already has a meaning in the program and if it does, it uses that, otherwise it falls back to the special type. Same for `var` in C# 3.

I hope that helps; let me know if you want more details on any of these techniques.
> **Comments:**
> **username_0** (2023-07-15): Wow, one of the actual designers of a programming language I use answered my question.  I feel honored ☺️  
> **username_22** (2023-07-16): "yield return" is easyer to understand then "yield" for someone who have not seen it before.  
> **username_21** (2023-07-17): @dan04: You're very welcome but I assure you also, I'm just some guy.  
> **username_23** (2023-07-22): The point about `yield return` in C# is really interesting; I have sometimes wondered why C# uses this instead of the simpler `yield`, but now of course I completely understand the "backwards-compatibility" reason behind this!  
---
### Answer by username_24 on 2023-07-12. Score: 16
Depending on the syntax specifics, a new keyword can be contextual at the expense of a new ambiguity the parser must handle.
For example, let's say you add a prefix unary expression operator await to your language, and there's a program with an await identifier:
var await = 4;

This need not be an error since it is not a legal use of await anyway. In fact, in most positions where an identifier is declared, a keyword is not going to be ambiguous. Later, in other expressions, when the parser encounters a sequence like
var x = await

it needs to do one (or possibly more) token of lookahead to see what's after it. For example, you might have
var x = await * 2;

The * token is enough to disambiguate.
Problems may arise if a certain token forms both unary and binary expressions, for example
var x = await + await;

This might be await (+await), or (await) + (await), depending on your rules. In this situation you might choose to simply require parens to disambiguate, choose the new or old behavior consistently, or defer to other strategies as described in other answers.
> **Comments:**
> **username_25** (2023-07-12): This is why the Swift 5.9 keywords `discard`, `borrow`, and `consume` must be applied directly to an identifier rather than an expression in parentheses — `discard(self)` is just a regular function call in 5.8, while `discard self` would be illegal.  
---
### Answer by username_26 on 2023-07-12. Score: 14
Provide an upgrade tool
Another approach is to not preserve backwards compatibility and instead provide tools to upgrade to the new version.
In the case of adding a couple of new keywords all the upgade tool would have to do is rename variables using the string that is now a reserved word.
Make it possible to identify the language dialect in use
This procedure is facilitated if you have something in the source code that can identify the version in use. Otherwise your tool may incorrectly treat the new keywords as variables.
Example
For example running foo1to2 on:
dialect(foolang,1)

var IamAKeyword : integer = 2;

It becomes:
dialect(foolang,2)

var IamNotAKeyword : integer = 2;

Upgrade tools can be a pain to write if the nature of the change is complex or the syntax is complex. You can make your life easier if you consider this issue upfront.
Consider context
As others have said you can somewhat avoid the issue if your new syntax is illegal in the previous language version/dialect.
Beware though that you are trading convenience of upgrade for clarity in the language.
An interesting example of this is C++.
Relatively early on for pure methods the = 0 syntax was introduced rather than a pure keyword
class Foo
{
   virtual void someMethod() = 0;
};

assignment of methods was not a legal syntax.
Much later when override and final were introduced they appear in the same place. So instead of:
class Foo
{
   override void someMethod();
};

Where override could seem to be the name of the return type say
You write:
class Foo
{
   void someMethod() override;
};

There was a good link about this which I cannot find but https://stackoverflow.com/questions/32757571/why-do-the-c-language-designers-keep-re-using-keywords is very relevant.
Don't bother - just document
A final option is to not to anything other than document carefully the changes.
You leave the community deal with the issue.
They may well come up with upgrade tool on your behalf.
This risks alienating your user base, particularly if you do it too often.
But if your language is good and your community loyal you can get away with it.
This is really trading user friendliness for development resources.
There are quite a few languages that are guilty of this (despite having ample developement budget) and remain popular.
Deprecate
When evolving a language you could consider formally deprecating interfaces in one version before removing or changing them in a subsequent version. This slows down changes to your langauge but makes it easier for users to manage the transition.
This is particularly useful in 'enterprise' environments.
You can also consider having a syntactic or semantic way to mark something as deprecated in the language itself.
For example an attribute like [[deprecated]] in C++ can be used to indicate a function is to be retired.
This may or may not be harder to do for a keyword. In some languages keywords are a lot like functions or even exactly the same (e.g. TCL.
My personal preference is both:

a clear way to the dialect/version used.
an upgrade tool
> **Comments:**
> **username_27** (2023-07-12): Along with the "don't bother" option should be "warn ahead of time". This is similar to deprecating features -- you warn about it during release N-1, then implement the breaking change in release N.  
> **username_15** (2023-07-13): @Barmar: An important aspect of deprecating features is ensuring that anything that could be done using a deprecated feature can be done at least as well--if not better--without it.  
> **username_27** (2023-07-14): In this case, you just have to rename any conflicting variables. @supercat The warning gives you time to do that.  
> **username_15** (2023-07-14): In this particular case, that would be true unless one needs to link with code outside one's control.  But in the more general case, depecation requires ensuring that replacements exist--a concept that maintainers of the C language ignore.  
> **username_13** (2023-07-24): @supercat Java deprecated `sun.misc.Unsafe` without providing a replacement or alternative. If they had provided a replacement then it would defeat the purpose of deprecating it, which was to remove something that was never supposed to be available to users in the first place. I think you are using the word "requires" to mean that this is what you think language maintainers *should* do, but AFAIK this is not part of the definition of the word "deprecate".  
> **username_15** (2023-07-24): @kaya3-supportthestrike: An action which removes a language construct that is necessary for some tasks without providing a viable replacement will split the language into a family of dialects that remain suitable for the aforementioned tasks, and a family that are suitable only for tasks not involving that construct, but might process those tasks better than the earlier language did.  Such a split is fine if one recognizes that the more limited dialects are't the same language as the one from which the aformentioned constructs are removed.  BTW, an important thing to note with C...  
> **username_15** (2023-07-24): ...is that the Standard was chartered to describe a family of *pre-existing dialects*, but rather than describing features that were and should have remained common to 90%+ of them, it deliberately waived jurisdiction over features that couldn't be practically supported on 100% of them.  The only way the C Standard could be viewed as ever having accurately describing the popular language is if it was expected that implementations that processed more behaviors than mandated would continue to do so *as the authors have said in the published Rationale that they intended*.  
---
### Answer by username_6 on 2023-07-12. Score: 11
**[Stropping](https://en.wikipedia.org/wiki/Stropping_(syntax))** was the approach taken in some early programming languages like Algol-60 and Algol-W. The Wikipedia Article says

> In computer language design, stropping is a method of explicitly marking letter sequences as having a special property, such as being a keyword, or a certain type of variable or storage location, and thus inhabiting a different namespace from ordinary names ("identifiers"), in order to avoid clashes. Stropping is not used in most modern languages – instead, keywords are reserved words and cannot be used as identifiers. Stropping allows the same letter sequence to be used both as a keyword and as an identifier, and simplifies parsing in that case – for example allowing a variable named if without clashing with the keyword if.

Algol-60 implementations frequently used single quotes around a keyword, e.g. 'BEGIN'. On some machines it used underlining. And in the language specification this was typically represented as bolding.

Algol 68 used several alternate strapping conventions, that could be specified by a compiler directive, like .FOR, 'FOR, upper/lowercase... plus they had the option of not requiring strapping at all, and just requiring reserved words like most modern languages. In publication keywords were typically represented as underlined or bold.

FORTRAN has limited stropping for reserved words or operators like .EQ. and .AND.

Stropping went out of fashion as programming language portability became more and more important, since the least common denominator for composing programs was in plain ASCII text with little or no formatting. The usual mechanisms of displaying stropped keywords via fonts like **bold** could not be relied on to be portable.

Stropping avoids breaking existing programs if new keywords collide with identifiers in old code. Stropping avoids the contortions that languages like C++ need to provide for extensibility, avoiding such conflicts. Nearly all modern programming is done using IDEs or editors that can do syntax coloring. Syntax coloring is almost exactly what you need to do to make stropping user-friendly. You don't need to require the user to type the stropping syntax: the user can type in plaintext, and the editor/IDE can apply stropping automatically, and save the stropped format in a file. If new syntax is added with a new keyword **foo** that happened to be used as an identifier *foo* in old programs, no problem. There are several portable formats that can save such stropped program representations, if necessary with more metadata than simple .stropping. can represent.

However, stropping does not completely eliminate problems with extensibility. E.g. if you wish to design a language that allows new language constructs, syntax, and keywords to be added to your language. If there are multiple such extension packages, they can conflict. Ultimately you will need to have some sort of namespace indication for such syntactic extensions. So you might want to go beyond simple .stropping., who have representations that can store more metadata.
> **Comments:**
> **username_15** (2023-07-13): A language could require that keywords be punctuated a certain way using ASCII characters in a way which may look ugly if printed in a single font without highlighting, but also specify a preferred visual representation which would look much more attractive (but be easily recognizable as mapping to the original ASCII construct).  
> **username_15** (2023-07-13): For example, it could specify that a Euclidian division operator be written as ` .ediv. ` in text (with white space to either side), but that editors should when practical show alphanumeric strings delimited by periods and whitespace in such fashion as italics with the periods replaced by micro-spaces, e.g. *ediv*.  
> **username_11** (2023-07-15): Huh.  All this time, I thought all those old papers I read with Algol code in them were just nicely formatted to make them easy to read.  Never realised that highlighting the keywords was a required feature of the language. :)  
> **username_6** (2023-10-13): @occipta: stropping was not a "required" feature of the language.  As I mentioned there were several different stropping conventions, and some compilers could support several, including the "no-stropping" version where things were just you were in whatever character set you had.  However, I am 99.9% certain that various Algol 68 programs had some variable names that were the same as keywords. And I think I remember this also in either Algol 58 or Algol 60. There were various tools to convert from one's dropping representation to another, which is trivial, except for keyword collision.  
> **username_6** (2023-10-13): Now something controversial: IMHO we should use the moral equivalent of syntax coloring or stropping to distinguish keywords from variable names, etc. It should be extensible. XML is a perfectly good exchange format. The low-level representation of your code might look like <if> if == then <then> then() </if>.   XML is that it is future extendable.  A compiler might not understand a new keyword, but it least it can recognize the code that uses it and report a meaningful error, re-synchronizing at the end of the construct </if>, rather than going wildly crazy as so often happens with C++ code.  
> **username_6** (2023-10-13): Stropping (XML or otherwise) means old code runs if included by new code that uses new keywords. You don't need language versions. You need not reserve keywords for the future. You don't need _crap_. You need not deprecate anything: XML stropping can have versions like <await ver="1"> and <await ver="2">.   You don't need a  verbose form like C#2 'yield return whatever' because 'yield return' has a conflict. But you need hidden metadata in the stropping format to get these benefits. XML already has this. If you don't like XML, define your own new metadata stropping format.  
> **username_6** (2023-10-13): Let me emphasize: a normal human would never need to know what the XML format looks like. Your IDE would hide it from you. You can type plain old ASCII, and as long as there are no conflicts or ambiguities it just gets colorized and XML stropped invisibly.  
---
### Answer by username_16 on 2023-07-12. Score: 9
# Use namespaces everywhere

This could mean many things. You may choose what is appropriate in your language:

1. A user-defined identifier could override the meaning of a keyword.
2. You could use the original keyword by explicitly specifying the namespace, for example `system::await`.
3. You could alias a keyword to a different name in a program.
4. You could import the namespace only if you need the keyword in your program.

It might be easiest to implement only #4, in which case the namespace may not be called a namespace, but a feature option that you could turn on and off. But there would be problems if it exported identifiers that code using new features would use. You could support aliases in the import, but that wouldn't work well for members in a structure. But there is an easy solution, by using a special grammar for forcing the use of an identifier that is the same as a keyword. You could turn off the option in old source code files, and when you want to upgrade a file, change the identifiers with conflicting names to the new grammar. C# uses `@x`. Some SQL dialects use ``x`` or `[x]`. Shell scripts, say bash, uses `'x'` or any other quotes or escapes, and also `command x` which also avoids functions. PHP uses `${'x'}` for names that is not in the format of an identifier, but may not be the best example because variables and keywords are not in the same namespace in the first place.
> **Comments:**
> **username_17** (2023-07-12): I do like option #4: it's similar to @ATaco's version opt-in, but more fine-grained; in particular it avoids the issue that you can't just `require` the latest version in all of your code, because that could unnecessarily break backwards compatibility.  
---
### Answer by username_1 on 2023-07-12. Score: 9
A quick and dirty approach C uses is by reserving all identifiers starting with `_`, and using `_Keywords` as built-in keywords and only having the `keywords` as convenience macros in header files.
---
### Answer by username_4 on 2023-07-12. Score: 7
## Different lexical forms for keywords and identifiers

In my current design, identifier names must contain a capital letter, digit or underscore; anything that is purely lowercase letters is inherently a keyword. There is thus no risk of a variable name becoming a reserved word in a future version, because it can't be a valid variable name in the first place.
> **Comments:**
> **username_5** (2023-07-13): I would do it the other way around, using all uppercase for keywords and otherwise reserved words. Uppercase is harder to type and keywords should be very short and you need to type more letter for non-keywords. It makes it also the same as in SQL convention (but not enforced). Uppercase is also harder to read, which is better suited for something you know than something you have read completely first. It is also easer to search inside the code without accidentally finding too many matches in the comments.  
> **username_5** (2023-07-13): Good answer but under "namespace" i understand something different (like C++ namespace).  
> **username_4** (2023-07-13): I feel like uppercase keywords would be too visually distracting due to their frequency. For me, all-uppercase should be reserved for broad-scope constant values.  
> **username_4** (2023-07-27): This idea, user23013's, user16217248's and Krazy Glew's are all quite similar, really. Arguably, different implementations of the same underlying idea.  
> **username_6** (2023-10-13): @KarlKnechtel: yes, all of these syntactic distinctions expressed in the characters that are typed by and visible to the user are similar.   IMHO where it gets interesting is when there is hidden metadata with the constructs to solve many of the other problems with compatibility.  Then you don't need to have arbitrary restrictions like "keywords must be lowercase", or "keywords must be uppercase.  
> **username_4** (2023-10-13): @KrazyGlew I don't think I understand what you mean by "hidden" here.  
> **username_6** (2023-10-18): @KarlKnechtel: fair enough.  First, if your storage format is XML I mean that the user types "thread", the IDE might convert it into <identifier text="thread"/> if there is no thread keyword - but if the keyword is later defined , stored code with the <identifier...> storage form will still work, but the IDE might choose to emit <keyword text="thread"/> or <identifier...>  depending on policy settings.  The user doesn't see the storage syntax, although the user might see different coloring or fonts.  So I guess the EFFECT of the metadata is visible, but not the exact metadata.  
> **username_6** (2023-10-18): @KarlKnechtel: or the example from where "await" might be a variable name in some early code, then becomes a keyword, and then later a different language feature uses the "await" keyword in a way sufficiently different that you want to distinguish <keyword text="await" version="1"/> from <keyword text="await" version="2"/>. You are correct in that the distinction will have to be made visible the user in some way - perhaps color/font, but possible just a flyover or other query.  And the user does NOT need to type or see the XML crap.  
---
### Answer by username_7 on 2023-07-12. Score: 6
You can add syntax to tell the parser which language standard to use. [Ada has that](https://gcc.gnu.org/onlinedocs/gcc-4.3.6/gnat_rm/Pragma-Ada_005f05.html).

But, in most cases it is fine to warn the user upfront and use keywords that most people expect to be keywords anyway.
---
### Answer by username_8 on 2023-07-12. Score: 5
Python had exactly this issue (specifically for `async`/`await`), and ended up introducing those as 'soft keywords'. Essentially, there's no special case for them in the lexer, so they are treated as identifiers like any other, but there are also grammatic rules that expect these specific identifiers in specific positions. Since the new grammatic constructs would've been illegal previously anyway, no collisions take place. The only (slight) downside to this approach is performance of parsing, but the effect is miniscule
---
### Answer by username_9 on 2023-07-13. Score: 5
I think that the possibility of adding new keywords reinforces the idea that they should either be a property of the base language ab initio, or should be identified in some way. As somebody else has mentioned, ALGOL mandated or at least permitted stropping, while in Modula-2 Wirth mandated that keywords were capitalised while at least preferring that non-keywords (i.e. imported from libraries etc.) were not.

I would suggest designing the base language such that a spurious non-keyword identifier can always be recognised as an error, and that pragmata are used to indicate e.g. storage class rather than keywords.

From there, I would suggest first providing an intermediate version of the language, where any existing usage of the new keyword is flagged as an error which either has to be fixed or blessed by a pragma indicating that it's been reviewed.

Putting keywords in their own namespace, or requiring that a program unit be imported to make them available, is one approach. But my own feeling is that they should be minimised, in order- apart from anything else- to reduce lexing problems caused by the demands of internationalisation.
---
### Answer by username_10 on 2023-07-14. Score: 5
Make sure those keywords can only appear in places where program-specific identifiers can not. That way there can be an identifier that is spelled like a keyword without their being any ambiguity. This can be difficult in a C-style language where identifiers can appear in all kinds of places. But it is far more achievable in a COBOL-style language where every line begins with a keyword that says how the rest of the line is to be parsed. For example, when your syntax for calling a function looks like that:
FUNCTION async TAKING input TYPE string
   WRITE input
ENDFUNCTION

CALL FUNCTION async PASSING await

then you could in the next vesion allow this:
ASYNC FUNCTION async TAKING input TYPE string
   WRITE input
ENDFUNCTION

AWAIT FUNCTION async PASSING await

When every line must start with a keyword, then you can add as many more line-starter keywords as you want without there ever being a conflict.
> **Comments:**
> **username_11** (2023-07-15): This doesn't work for all changes, though, as some may require a new keyword to appear in a place where it could previously have been used.  See above for Eric Lippert's example of "yield" for C#.  And there's a reason nobody writes new programs in COBOL any more -- it is so ridiculously verbose that you just can't write moderately complex programs so that they're readably small.  How anyone could stand using it on *punched cards* is beyond me....  
---
### Answer by username_12 on 2023-07-26. Score: 3
## Preprocessor magic

One approach that can be used if your language has a token stream to token stream preprocessor, is to utilize it in the keyword-or-identifier decision making process.

The basic idea is to change the lexer to always output keyword or identifiers as identifier tokens1, leaving it up to the preprocessor to replace identifier tokens representing a keyword with a keyword token.2 This makes it possible to have preprocessor directives that can affect the identifier or reserved word decision. For example, you could have a preprocessor directive that turns on or off a set of keywords, or even an individual keyword. Doing it as a matched set of directives (akin to `#if` and `#endif`) can be especially useful for scoping these changes, especially for code that may be `#include`d and not know what state to return to.

The main advantage of this versus a language version compiler flag, is that you can write code that uses the new keyword for its new purpose in the same file, simply by wrapping the section that uses it with a preprocessor directive.

The only language I am aware of that does this is Verilog, which supports a ``begin_keywords` preprocessor-like directive, and a corresponding ``end_keywords`. The ``begin_keywords` directive takes an argument that indicates the standard revision that defines the keywords to support.

---

Footnotes:

1 Traditionally a lexer will output special tokens for keywords, making them trivial for the parser to identify. Having to look inside a token to decide the parse path is more complicated, and often not supported by parser generators if your language is using one of those.

2 Obviously, there are some variations possible, like always output all reserved words as their own tokens in lexer, and having preprocessor conditionally convert them back to identifiers, etc.
---
### Answer by username_13 on 2023-07-27. Score: 2
## Reserve unused keywords for future use

Most of the time a new keyword is added to a language after its initial release, the keyword is one which exists in other languages already. The obvious examples are `async`, `await` and `yield`, but e.g. `const` and `let` were added to Javascript, `case` and `match` were added to Python, and so on.

This means that with some foresight, it's possible to reserve some keywords before the associated features are added to your language; then when it comes to add those features, there is no backwards compatibility problem because the keywords you need have already been reserved from day one.

Examples:

* In [Rust](https://doc.rust-lang.org/reference/keywords.html#reserved-keywords), the keywords `abstract`, `become`, `box`, `do`, `final`, `macro`, `override`, `priv`, `typeof`, `unsized`, `virtual` and `yield` are currently reserved for future use.
* In [Java](https://docs.oracle.com/javase/specs/jls/se19/html/jls-3.html#jls-3.9), the keyword `_` is reserved for "possible future use in parameter declarations"; and `const` and `goto` are reserved to allow for better error messages in case C++ programmers use these by mistake, but it would also be possible for Java to use them for new features in the future.
* [Nim](https://nim-lang.org/docs/manual.html#lexical-analysis-identifiers-amp-keywords) likewise has some unused keywords reserved for future use, but I didn't find a comprehensive list of which ones are currently unused.
> **Comments:**
> **username_14** (2023-07-27): For Nim, I think the only unused keyword is `interface`.  
---
