<p>Languages developed over the last fifteen years or so have been within the era where Unicode is ubiquitous, and so could design their core string types accordingly. There are a lot of new issues that Unicode brings up, and I'm interested in how they've been traded off in different designs.
I've listed some of the dimensions of those design choices below the line.</p>
<p>I'm looking for instances of different <strong>real-world general-purpose languages</strong> released from <strong>around 2008 onwards</strong> addressing Unicode in their strings. Languages that I know released fitting in that group and made relevant design choices include Rust, Swift, Go, Raku, and Python 3, but there are surely others too. <strong><em>What</em> did these languages decide to do with their string type(s), and if applicable <em>how</em> has that been received by programmers?</strong></p>
<p>References to design documentation, rationale, or notable reception are welcome.</p>
<hr />
<p>Unicode raises a number of questions and design choices that the preceding history of languages leaning on predominantly byte or 16-bit strings in platform-dependent encodings didn't. For example:</p>
<ul>
<li>Encoding: Are strings stored in UTF-8, UTF-16, UTF-32, a combination, or something else? Is that made visible to the user?</li>
<li>Characters: What is the atomic unit of a string, if anything? It may be a byte, a code unit, a code point, a grapheme cluster, or something else.</li>
<li>Indexing: Possibly related, what are the semantics and ergonomics of indexing into a string (&quot;give me character 5&quot;, or from 5 on), if there is any? Iteration may also be relevant.</li>
<li>Normalisation: Are strings automatically normalised internally into some canonical form?</li>
<li>Equality: When are two strings considered equal? Encoding or normalisation of the value may matter here, or may be abstracted away.</li>
<li>Ordering: When does one string precede another for comparison or sorting operations?</li>
</ul>
<p>Any given language might deliberately deal with only some of these. A variety of different reasonable designs are available for each, with tradeoffs for different combinations. I'm interested in what design choices were made and what the implications and reception of those choices by users of the language were, if that's notable.</p>

<p>I'm thinking about the run-time string values here, so language <em>syntax</em> for strings is relevant only as far as it's directly supporting the semantics (for example, syntactic support for different forms of indexing or specifying different encodings on a string literal).</p>

<language-design><strings><design-choice-comparison><unicode>
How have modern language designs dealt with Unicode strings?
CC BY-SA 4.0
2023-10-25T16:23:37.633Z
I can't help but say this to prospective language designers as a grumpy user of programming languages: if your core string type is UTF-8 (which is fine) you better give your users good tools in your standard library to manipulate common things (like file paths) that may not be valid UTF-8. Lookin at you golang/node.js.
Jared Smith
<h1>Rust</h1>
<p>The default string types <code>(&amp;)str</code> and <code>String</code> are both UTF-8, ignoring native encoding. There are various other types and conversion methods to convert between native rust strings and other encodings, such as <code>UTF-32</code> (the <code>widestring</code> stdlib crate) and native strings (<code>OsString</code>)</p>
<p>The &quot;atomic unit&quot; of a string is defined to be a <code>char</code>, representing a single UTF-8 codepoint - a <code>Unicode scalar value</code>.</p>
<p>You cannot directly index a string without converting it to a byte-like Rust-level representation and therefore losing the invariant that the string is correct UTF. You can, however, access the codepoint <code>n</code> (which is of course a <code>O(n)</code> operation).</p>
<p>As far as I am aware, everything else just uses the UTF-8 byte values in terms of ordering, encoding, and equality.</p>
<p>Also as far as I know, this approach has been pretty well received in the linux community, as UTF-8 is prevalent there. I cannot speak for the Windows community.</p>

CC BY-SA 4.0
2023-06-13T05:04:53.287Z
2023-06-13T05:20:21.743Z
2023-06-13T05:20:21.743Z
blueberry
CC BY-SA 4.0
2023-06-13T05:08:02.163Z
When you say "standard for UTF-8 in terms of ordering, encoding, and equality", you mean it just uses byte ordering and equality (like a C string)?
Michael Homer
CC BY-SA 4.0
2023-06-13T05:14:07.243Z
@MichaelHomer Yes, as confirmed by the source code here: https://doc.rust-lang.org/src/alloc/string.rs.html#362
blueberry
CC BY-SA 4.0
2023-07-03T13:39:39.030Z
A few issues in this answer... First a codepoint is independent from an encoding, hence `UTF-8 codepoint` has no meaning. Second you can definitely [index a string](https://doc.rust-lang.org/std/primitive.str.html#impl-Index%3CI%3E-for-str) in Rust, though you do have to be aware that shall indexes fall on non codepoint boundaries, a panic will ensue. Thirdly, it may be important to point out that Rust aims for a _minimal_ standard library, so that relegating more Unicode functionality to 3rd-party crates is not shocking to Rust users... though it may be to newcomers.
Matthieu M.
CC BY-SA 4.0
2023-07-05T04:26:40.130Z
this is fairly similar to what Julia does with strings
Oscar Smith
CC BY-SA 4.0
2023-07-07T16:58:05.927Z
Note that UTF-8 byte order is the same as Unicode code point order.  This is *not* the case for UTF-16, due to the "surrogate" code units not being at the end of the 16-bit code space.
dan04
<h1>Swift</h1>
<p>Swift strings are <em>complicated</em>. At the time of writing, the implementation of <code>Swift.String</code> is split across more than 30 files in the standard library, and that's not counting things that need to be handled in the compiler (like string literals) or Objective-C (like <code>NSString</code>).</p>
<p>In Swift, the stack-/register-allocated storage of a <code>String</code> is 16 bytes. It's a tagged union of one of a few things:</p>
<ul>
<li>An <code>NSString</code>, which is an abstract class from Objective-C. Usually this stores immutable UTF-16, but <a href="https://developer.apple.com/documentation/foundation/nsstring#1666323" rel="noreferrer">it can use</a> a number of other encodings such as ASCII or UTF-32. This member of the union only exists on Apple's platforms, since Objective-C interop <a href="https://developer.apple.com/documentation/objectivec" rel="noreferrer">is not supported</a> on Windows or Linux.</li>
<li>If the string is no more than 15 bytes in UTF-8 with no null terminator, the first <em>n</em> bytes are its UTF-8 representation, the next 15 - <em>n</em> bytes are all zeroes, and the low nibble of the last byte is the string's length (<a href="https://github.com/apple/swift/blob/1532fb188c55f29de7bf8aaee94896557b3a3db1/stdlib/public/core/SmallString.swift#L18" rel="noreferrer">source</a>).</li>
<li>A pointer to a copy-on-write, resizable, null-terminated UTF-8 buffer (<a href="https://developer.apple.com/documentation/swift/string#Performance-Optimizations" rel="noreferrer">docs</a> and <a href="https://github.com/apple/swift/blob/1532fb188c55f29de7bf8aaee94896557b3a3db1/stdlib/public/core/StringStorage.swift#L99" rel="noreferrer">source</a>). Sometimes, there's a &quot;breadcrumbs&quot; pointer after the null terminator, but more on that later.</li>
<li>If the optimizer can prove the string does not grow arbitrarily large or escape the current stack frame, it uses an owned stack buffer rather than a CoW heap buffer (<a href="https://forums.swift.org/t/string-pooling-interning/67969/2" rel="noreferrer">forum post</a>).</li>
<li>A pointer to an &quot;immortal&quot; UTF-8 buffer that the Swift runtime can't modify even to resize, i.e. a string literal (<a href="https://github.com/apple/swift/blob/1532fb188c55f29de7bf8aaee94896557b3a3db1/stdlib/public/core/StringObject.swift#L43" rel="noreferrer">source</a>).</li>
</ul>
<p>But, we're here to talk Unicode, so let's talk Unicode.</p>
<h2><code>Character</code></h2>
<p>A Character in Swift is an <em>extended grapheme cluster</em> (<a href="https://developer.apple.com/documentation/swift/character" rel="noreferrer">docs</a>). This can be more than one Unicode codepoint, but it always looks like a single character. Examples of multi-codepoint characters include <code>\r\n</code> (yes, I know) and <code>é</code> ('e' + U+0301). If you want a single Unicode codepoint, that type is <code>Unicode.Scalar</code>, which is just a wrapper around <code>UInt32</code> (<a href="https://github.com/apple/swift/blob/1532fb188c55f29de7bf8aaee94896557b3a3db1/stdlib/public/core/UnicodeScalar.swift#L35" rel="noreferrer">source</a>).</p>
<p>In terms of storage, a Character is stored exactly the same way as a String! (<a href="https://github.com/apple/swift/blob/1532fb188c55f29de7bf8aaee94896557b3a3db1/stdlib/public/core/Character.swift#L64" rel="noreferrer">source</a>)</p>
<pre class="lang-swift prettyprint-override"><code>@frozen
public struct Character: Sendable {
  @usableFromInline
  internal var _str: String

  @inlinable @inline(__always)
  internal init(unchecked str: String) {
    self._str = str
    _invariantCheck()
  }
}
</code></pre>
<h2><code>count</code></h2>
<p>In Swift 3, <code>String.count</code> gave a very specific error message (<a href="https://swiftfiddle.com/3yrvvxaax5eyjcjyjdilsasmvy" rel="noreferrer">example</a>):</p>
<blockquote>
<p>'count' is unavailable: there is no universally good answer, see the documentation comment for discussion</p>
</blockquote>
<p>However, starting in Swift 4, it is available. <code>.count</code> is an O(n) operation that counts the number of extended grapheme clusters in the string (<a href="https://developer.apple.com/documentation/swift/string#Measuring-the-Length-of-a-String" rel="noreferrer">docs</a>). This is consistent with the fact that <code>String</code> implements <code>Sequence&lt;Character&gt;</code> -- <code>String.count</code> is the number of Characters in the String.</p>
<h2>String Views</h2>
<p>Swift offers a number of views on a string that don't allocate new storage:</p>
<ul>
<li><code>String.unicodeScalars</code> is a <code>Sequence&lt;Unicode.Scalar&gt;</code> rather than a <code>Sequence&lt;Character&gt;</code>, but otherwise operates in much the same way as the original String (<a href="https://developer.apple.com/documentation/swift/string#Unicode-Scalar-View" rel="noreferrer">docs</a>). General-purpose string algorithms are intentionally not available for this type.</li>
<li><code>String.utf16</code> is a <code>Sequence&lt;UInt16&gt;</code> (<a href="https://developer.apple.com/documentation/swift/string#UTF-16-View" rel="noreferrer">docs</a>). If the string is stored in another format, this converts one value at a time as you iterate over it (<a href="https://github.com/apple/swift/blob/1532fb188c55f29de7bf8aaee94896557b3a3db1/stdlib/public/core/StringUTF16View.swift#L466" rel="noreferrer">source</a>).</li>
<li><code>String.utf8</code> is a <code>Sequence&lt;UInt8&gt;</code> (<a href="https://developer.apple.com/documentation/swift/string#UTF-8-View" rel="noreferrer">docs</a>). It offers fast access to the raw bytes for a String that's already in UTF-8, and converts an <code>NSString</code> one value at a time as needed (<a href="https://github.com/apple/swift/blob/1532fb188c55f29de7bf8aaee94896557b3a3db1/stdlib/public/core/StringUTF8View.swift#L501" rel="noreferrer">source</a>).</li>
</ul>
<h3>Breadcrumbs</h3>
<p>In order to speed up repeated accesses to <code>utf16</code>, UTF-8 strings may put a <em>breadcrumbs</em> pointer after the null terminator (<a href="https://github.com/apple/swift/blob/1532fb188c55f29de7bf8aaee94896557b3a3db1/stdlib/public/core/StringStorage.swift#L157" rel="noreferrer">source</a>). The breadcrumbs are a list of the UTF-8 offsets of every 64th UTF-16 code unit (<a href="https://github.com/apple/swift/blob/483087a47dfb56e78fcc20ef2b43085ebfb48ea0/stdlib/public/core/StringBreadcrumbs.swift" rel="noreferrer">source</a>). A string stores whether it has breadcrumbs in an unused bit in its capacity field (<a href="https://github.com/apple/swift/blob/1532fb188c55f29de7bf8aaee94896557b3a3db1/stdlib/public/core/StringStorage.swift#L45" rel="noreferrer">source</a>).</p>
<h2>Indexing</h2>
<p>Swift Strings aren't indexable by an integer, the way an Array is. Instead, there's a specific type <code>String.Index</code> that's used to avoid the O(n) index access UTF-8 would normally impose. To summarize, it's the following 64-bit bitfield (listed in order from MSb to LSb; <a href="https://github.com/apple/swift/blob/18798b6cdf6189f1c87ca2e48b7660995a84e1f1/stdlib/public/core/StringIndex.swift#L17" rel="noreferrer">source</a>):</p>
<ul>
<li>a 48-bit number (&quot;position&quot; or &quot;encoded offset&quot;) that is the raw offset in the underlying buffer;</li>
<li>a 2-bit number (&quot;transcoded offset&quot;) that's the location within the Unicode scalar, when applicable (e.g. if it points to the third byte of a 4-byte UTF-8 character);</li>
<li>ten bits that are purely private and not even used by the ABI (the &quot;resilient slice&quot;) -- at the time of writing, only six of them are used, and it stores the distance to the next grapheme cluster, if known; and</li>
<li>four single-bit flags, for which '1' is true and '0' is either false or unknown:
<ul>
<li>the underlying buffer is UTF-16,</li>
<li>the underlying buffer is UTF-8,</li>
<li>the index points to a boundary between Characters (<code>_isCharacterAligned</code>), and</li>
<li>the index points to a boundary between Unicode scalars (<code>_isScalarAligned</code>).</li>
</ul>
</li>
</ul>
<p>Before Swift 5.7, the resilient slice was thirteen bits, and included what is now the first three of the four flags.</p>
<p>This layout reduces the overhead of advancing the index forwards by spreading out when the UTF-8 is interpreted. The resilient slice also allows some room for the implementation to improve while still being ABI-stable.</p>
<h2>String Literals</h2>
<p>A literal like <code>&quot;a&quot;</code> could be any of the following types:</p>
<ul>
<li><code>String</code>, obviously.</li>
<li><code>Substring</code>, less obviously (<a href="https://developer.apple.com/documentation/swift/expressiblebystringliteral#conforming-types" rel="noreferrer">docs</a>). A Substring is just a slice of a String, but it isn't an unowned slice: it keeps the original String alive (<a href="https://developer.apple.com/documentation/swift/substring#Converting-a-Substring-to-a-String" rel="noreferrer">docs</a>). This allows Substrings to be expressed with string literals.</li>
<li><code>NSString</code> (<a href="https://developer.apple.com/documentation/foundation/nsstring#conforms-to" rel="noreferrer">docs</a>).</li>
<li><code>Character</code>, if it's only one grapheme cluster (<a href="https://developer.apple.com/documentation/swift/expressiblebyextendedgraphemeclusterliteral#overview" rel="noreferrer">docs</a>).</li>
<li><code>Unicode.Scalar</code>, if it's only one Unicode codepoint (<a href="https://developer.apple.com/documentation/swift/unicode/scalar#overview" rel="noreferrer">docs</a>). This is converted to UTF-32 so that it's always four bytes.</li>
<li><code>StaticString</code>, which is either a pointer to some constant UTF-8 data in the binary, or in some rare cases, a single character in UTF-32 (<a href="https://developer.apple.com/documentation/swift/staticstring" rel="noreferrer">docs</a>). This is generally used for things you want to know at compile-time, like the file name passed to <code>assert(_:_:line:file:)</code>.</li>
</ul>
<h2>Equality and Normalization</h2>
<p>String equality is elementwise equality of its Characters*. Character equality <a href="https://developer.apple.com/documentation/swift/string#Modifying-and-Comparing-Strings" rel="noreferrer">is not bytewise equality</a>. Character equality uses the canonical normalized form, but Characters are not automatically normalized (<a href="https://swiftfiddle.com/tml3ainsoje67hg3ghsq5spxyu" rel="noreferrer">example</a>):</p>
<pre class="lang-swift prettyprint-override"><code>import Foundation

func printRawBytes(_ c: Character) {
  let ints = unsafeBitCast(c, to: (UInt64, UInt64).self)
  print(String(format: &quot;0x%.16llX%.16llX&quot;, ints.0.bigEndian, ints.1.bigEndian))
}

let precomposed: Character = &quot;é&quot;
printRawBytes(precomposed) // 0xC3A900000000000000000000000000A2

let combining: Character = &quot;é&quot;
printRawBytes(combining) // 0x65CC81000000000000000000000000A3

print(precomposed == combining) // true
</code></pre>
<p>Strings that use an external buffer set a bit in the class header when they are known to already be NFC normalized (<a href="https://github.com/apple/swift/blob/1532fb188c55f29de7bf8aaee94896557b3a3db1/stdlib/public/core/StringObject.swift#L688" rel="noreferrer">source</a>). This can speed up certain operations: for example, two strings that use the same encoding and are both normalized can be compared bytewise.</p>
<p>*While this is semantically true, that's not how the Swift stdlib actually implements it. Instead, <a href="https://github.com/apple/swift/blob/cb61903d8ca9b6424d058029294ec6a53e714c42/stdlib/public/core/Character.swift#L215" rel="noreferrer">Character equality delegates to String equality</a>, and String equality is <a href="https://github.com/apple/swift/blob/c9115e1cec13830d2534769030e10b8d4863f7f7/stdlib/public/core/StringComparison.swift" rel="noreferrer">its own mess</a>.</p>
<h2>Ordering</h2>
<p>String equality is relatively straightforward: do the strings represent the same text? However, Strings can also be compared for order: <code>str1 &lt; str2</code> is legal. While it's generally preferred to use methods like <a href="https://developer.apple.com/documentation/foundation/nsstring/1409742-localizedstandardcompare" rel="noreferrer"><code>localizedStandardCompare</code></a> for user-facing text, for some cases it's better to be locale-independent.</p>
<p>For NFC normalized UTF-8 text, <a href="https://github.com/apple/swift/blob/c9115e1cec13830d2534769030e10b8d4863f7f7/stdlib/public/core/StringComparison.swift#L296" rel="noreferrer">it just uses <code>memcmp</code></a>. Other formats are converted to that and compared one Character at a time. The use of <code>memcmp</code> here sounds strange at first, but remember that this is for providing a consistent ordering across versions and locales, not for alphabetizing user-facing text.</p>
<h2>Performance Issues</h2>
<p>As was recently pointed out <a href="https://forums.swift.org/t/is-strings-subscript-o-n-when-given-a-lot-of-combining-code-points/67860" rel="noreferrer">on the Swift forums</a>, there is no limit to how large a <code>Character</code> can be. Consider the following:</p>
<pre class="lang-swift prettyprint-override"><code>let problematic = &quot;a&quot; + String(repeating: &quot;\u{0301}&quot;, count: 20_000_000) + &quot;x&quot;
// `problematic` is 20,000,002 Unicode scalars, but only 2 grapheme clusters!

func thisWillTakeAWhile() -&gt; Character {
  problematic.first!
}
</code></pre>
<p>Because a <code>Character</code> is an owned (often unique) <code>String</code>, never a <code>Substring</code>, that has to copy the 20,000,001-scalar grapheme cluster into a new string. This forms pathological cases for a lot of functions that assume materializing a <code>Character</code> takes negligible time, such as <code>haystack.contains(needle)</code>. That function, at the very least, compares the first character of <code>needle</code> to each character in <code>haystack</code> until it finds a match. When retrieving the first character of <code>needle</code> takes a noticeable amount of time, even obviously (to us) false ones like <code>&quot;Hello, World!&quot;.contains(problematic)</code> take several seconds to execute (<a href="https://swiftfiddle.com/y65ph25ofjd2haqzgimgs4gmse" rel="noreferrer">example</a>, though I'm seeing around 1.3 seconds, rather than the 10 seconds reported in the forum post).</p>

CC BY-SA 4.0
2023-06-13T12:07:45.880Z
2023-10-24T13:33:36.620Z
2023-10-24T13:33:36.620Z
Bbrk24
<p>This does not address 2008 onwards (which seems a pretty arbitrary date) but provides some background instead.</p>
<h1>Treat it as a (mostly) library issue</h1>
<h2><em>A <s><a href="https://stackoverflow.com/questions/3870088/a-monad-is-just-a-monoid-in-the-category-of-endofunctors-whats-the-problem">monad</a></s> string is just an array of bytes interpreted as UTF-8 what's the problem?</em></h2>
<p>A fair few languages treat it as a library issue.</p>
<p>Users thinking about Unicode know:</p>
<ul>
<li>UTF-8 is a variable length encoding</li>
<li>The difference between indexing a byte and a character</li>
</ul>
<p>So they expect operations to distinguish:</p>
<ul>
<li>arrays of bytes</li>
<li>arrays of characters/code points (i.e. strings of text)</li>
</ul>
<p>And provide different sets of operations for each and conversions from one to the other.</p>
<p>Getting this right can be hard but you can farm it out.
Rather it is easy to miss a corner case and introduce bugs which is unfortunate in a core library.</p>
<p>A string is of course a core type but starting with the <strong>faulty</strong> assumption that most users are using Western scripts it is sufficient to:</p>
<h2>Use UTF-8 everywhere</h2>
<ul>
<li><p>Source files are encoded as UTF-8</p>
<ul>
<li>so you can write comments natively</li>
<li>possibly use non-western letters as variable names</li>
</ul>
</li>
<li><p>The core string type is an array of bytes ideally with a length rather than null terminated</p>
<ul>
<li>So you can embed nulls.</li>
<li>So inserting a null does cannot be used as an attack vector to truncate strings</li>
</ul>
</li>
</ul>
<p>So:</p>
<ul>
<li>C having null-terminated strings gets this wrong.</li>
<li>C++ follows C, so <code>std::string</code> originally got this wrong too (and zero-terminated strings are still needed in places).</li>
<li><a href="https://en.wikipedia.org/wiki/D_(programming_language)" rel="nofollow noreferrer">D</a> from 2001 gets this right - as do most other newer languages</li>
</ul>
<h2>Leave it to a (possibly third party) library</h2>
<p>For example:</p>
<ul>
<li><p><a href="https://icu.unicode.org" rel="nofollow noreferrer">ICU</a> - International Components for Unicode.</p>
</li>
<li><p>Users of the Qt library will use <a href="https://doc.qt.io/qt-6/qstring.html" rel="nofollow noreferrer"><code>QString</code></a>.</p>
<p><code>QString</code> is an array of UTF-16 so there are potential issues with code points above 0xFFFF which require 2 <code>QChar</code>'s to represent - see <a href="//stackoverflow.com/q/6972883"><em>Retrieve Unicode code points &gt; U+FFFF from QChar</em></a>.</p>
<p>This is a good example of how subtle bugs remain even in implementations we might hope are Unicode safe.</p>
</li>
</ul>
<h2>But sometimes it's an OS issue...</h2>
<p>The Windows API is quite fond of UTF-16</p>
<p>Because Windows is popular, this leads to abominations like <a href="https://stackoverflow.com/questions/402283/stdwstring-vs-stdstring"><code>std::wstring</code></a> becoming standard library elements in some languages.
It's an abomination because:</p>
<ul>
<li>It should not be necessary to make your code generic over more than one string representation.</li>
<li>Because 2 bytes will be more than enough for everyone (oops - no it won't).</li>
</ul>
<p>See also <a href="//stackoverflow.com/q/496321"><em>UTF-8, UTF-16, and UTF-32</em></a>.</p>
<h2>Maturity brings wisdom</h2>
<p>With enough maturity a good Unicode library should become a de-facto or blessed standard or a core library.</p>
<p>Putting it into a clever type system and getting it right early on would be great though so I look forward to reading the other answers.</p>

CC BY-SA 4.0
2023-06-13T08:36:45.963Z
2023-08-16T15:02:37.003Z
2023-08-16T15:02:37.003Z
Bruce Adams
CC BY-SA 4.0
2023-10-18T19:55:08.350Z
Many aspects of Unicode should be OS issues, save for the lack of a useful specification for a set of primitives an OS should support.  Changes to the Unicode standard should not require rebuilding programs that might be affected thereby, but should instead be accommodated via OS update which would then have the appropriate effect on applications that use the OS functions.  Unfortunately, there is at present no way to avoid baking changeable aspects of the Unicode standard into the application layer.
supercat
CC BY-SA 4.0
2023-10-19T17:02:35.057Z
If your interpretation is UTF-8 in-memory then you give up on random-access character indexing.
Karl Knechtel
CC BY-SA 4.0
2023-10-19T18:08:23.703Z
That is still true in systems that use UTF-16. It is the nature of the beast that characters, graphemes, code point and what not do not have to match up anymore.
Bruce Adams
CC BY-SA 4.0
2023-10-24T15:10:00.213Z
@KarlKnechtel: Unicode traded encoding density for the ability to extract code points in context-free fashion, but then specified the behavior of grapheme clusters in a manner that would make it impossible to, in the fully general case, determine whehter a grapheme cluster boundaries exists between two bytes near the end of a billion-character string without scanning every single byte preceding them.
supercat
CC BY-SA 4.0
2023-10-24T15:12:44.017Z
@KarlKnechtel: Outside of the specific task of converting strings between UTF-8, UTF-16, and UTF-32, there aren't many situations where code-point boundaries are significant, so I'm not sure why the inability to retrieve the Nth code point given a string and N would be a problem.
supercat
CC BY-SA 4.0
2023-10-25T13:21:40.293Z
...Because people want to be able to splice strings by index?
Karl Knechtel
CC BY-SA 4.0
2023-10-26T15:32:10.203Z
@KarlKnechtel They do, but why? Even fully understanding all Unicode complexities, i18n, blah, blah? What are the use cases? (I can think 2, where perhaps some other data type may be better: file system paths or web URIs, and strings of symbols such as DNA, number/license plates, product codes (e.g. EAN), etc.)
Pablo H
CC BY-SA 4.0
2023-10-26T22:18:04.123Z
@PabloH You'd have to ask them, honestly. Modern language designers don't seem brave enough to drop that support.
Karl Knechtel
CC BY-SA 4.0
2024-10-15T20:29:04.307Z
@PabloH: Common use cases involve (1) working with text data that is intended primarily to be read by machines rather than humans, rendering internationalization issues irrelevant, and (2) being able to convert a (needle, haystack) string into combination of things that can be used to derive the part of the haystack before or after the needle, or one of the above with the needle included.  UTF8 with byte indexing is fine for any of those tasks.
supercat
CC BY-SA 4.0
2023-08-16T14:50:37.987Z
Yeah, it's a shame that crippled implementations of `std::wstring` are out there.  I get used to assuming that `wchar_t` can hold _any_ code point - thankfully I don't have to target Windows platforms!
Toby Speight
<h1>Python 3</h1>
<p>Conceptually, strings are immutable sequences of code points.</p>
<p>Indexing strings results in a string containing a single code point. Characters not exposed as a separate type.</p>
<p>Equality is based on equality of code points, and ordering is based on the value of the code points. Normalisation is not done implicitly.</p>
<p>The representation in memory is more of an implementation detail, but in order to keep indexing as fast as expected by users, strings in CPython are represented as arrays of code points, either using 1 byte (Latin-1), 2 bytes (UCS-2, aka UTF-16 not containing surrogate pairs) or 4 bytes (UTF-32), using the smallest representation that fits all code points in the string. This is not exposed to Python users.</p>
<p>Compare and contrast <code>bytes</code>, Python 3's immutable byte sequence:</p>
<p>Indexing <code>bytes</code> gives an <code>int</code>.</p>
<p>Equality is based on byte values, and ordering is based on the value of the bytes. Normalisation is not relevant to byte sequences.</p>

CC BY-SA 4.0
2023-06-13T08:02:32.400Z
2023-06-13T08:02:32.400Z
Jasmijn
CC BY-SA 4.0
2023-07-03T15:45:26.313Z
What about Ordering?
Pablo H
CC BY-SA 4.0
2023-07-03T16:26:37.217Z
I'm sorry, what is your question exactly?
Jasmijn
CC BY-SA 4.0
2023-07-03T17:10:46.080Z
Sorry, I failed to read the last sentence of your answer! :-(
Pablo H
<h1>Nim</h1>
<p>All strings are UTF-8 strings by convention, but it is not enforced.</p>
<p>Strings are char arrays, where <code>char</code> is a single byte. Indexing into a string is <em>safe</em> in the sense that Nim will not panic upon getting a char &gt; 127 representing a fraction of a codepoint, but it will just treat it as a byte.
Strings are conventionally treated as no more than a char array. There is a commonly used special <code>openarray[T]</code> type that abstracts over lists (<code>seq[T]</code>), arrays of various sizes, and strings.</p>
<p>There is a separate <code>std/unicode</code> module in the standard library that provides a (four-byte) <code>Rune</code> type representing a Unicode character, various associated functions, and a <code>.runes()</code> iterator.</p>
<p>Strings are both length-prefixed and null-terminated. Nim also has the <code>cstring</code> type for compatibility with C, which behaves the same as the <code>string</code> type, but lacks a length field and is instead only null-terminated. <code>string</code> is implicitly convertible to <code>cstring</code>. It doesn't do anything terribly different wrt. Unicode.</p>

CC BY-SA 4.0
2023-10-26T23:52:32.000Z
2023-10-26T23:52:32.000Z
apropos
<blockquote>
<p>Languages developed over the last fifteen years or so have been within the era where Unicode is ubiquitous, and so could design their core string types accordingly.</p>
</blockquote>
<p>There is a lot of history, and because of that, there is a lot of <em>backward compatibility</em> that may need to be made avaliable in new languages. The considerations below focus in this extra requisite.</p>
<p>This answer is not specific to a previous language, but considerations for a possible future language that tries to avoid a lot of pitfalls in string design. It's heavly inspired in Swift design, as mentioned in other answer, and provides some design alternatives (naming, sequential/proximal indexing, opaque positions and SSO)</p>
<h3>Encoding</h3>
<blockquote>
<p>Encoding: Are strings stored in UTF-8, UTF-16, UTF-32, a combination, or something else?</p>
</blockquote>
<p>Make the internal encoding a pure implementation detail. It's the only way to ensure you can <em>change</em> it, globally or on a per case basis. Python's experience in changing strings between versions 2 and 3 showed how disastrous exposing <em>any</em> internal encoding can cause afterwards.</p>
<h3>Views</h3>
<blockquote>
<p>Is that made visible to the user?</p>
</blockquote>
<p>Expose the popular encoding via non allocating views.</p>
<p>Say:</p>
<pre class="lang-js prettyprint-override"><code>str s = &quot;...&quot;;
foreach (a in s.Utf8)  {} // a is u8
foreach (b in s.Utf16) {} // b is u16
foreach (c in s.Utf32) {} // c is u32
foreach (d in s.CChar) {} // machine dependent sized type
</code></pre>
<p>These views should be implemented like like C#'s structs or non-<code>malloc()</code>ed C's structs. That is, objects allocated in directly stack frames.</p>
<h3>Character</h3>
<blockquote>
<p>Characters: What is the atomic unit of a string, if anything? It may be a byte, a code unit, a code point, a grapheme cluster, or something else.</p>
</blockquote>
<p>Offer all, explicitly, and avoid all confusion.</p>
<pre class="lang-js prettyprint-override"><code>str s = &quot;...&quot;;
foreach (a in s.Bytes)     {} // a is u8
foreach (b in s.Runes)     {} // b is u32
foreach (c in s.Graphemes) {} // c is str
</code></pre>
<p><code>Rune</code> is a common name for code points. But if it is the same as <code>.Utf32</code> view, it may be omitted.</p>
<h3>Indexing</h3>
<blockquote>
<p>Indexing: Possibly related, what are the semantics and ergonomics of indexing into a string (&quot;give me character 5&quot;, or from 5 on), if there is any? Iteration may also be relevant.</p>
</blockquote>
<p>There is no indexing or counting on string objects, only in views.</p>
<p>That is, the indexing by <code>.Bytes</code> accesses the underlying byte sequence, no transformation is applied. Indexing by <code>.Utf8</code> is <span class="math-container">$O(1)$</span> complexity in strings stored as UTF-8 strings, and <span class="math-container">$O(n)$</span> complexity for strings in other encoding, and so on.</p>
<p>With this, it is obvious that some users will <em>want</em> to make certain strings in certain encoding sometimes, to obtain the <span class="math-container">$O(1)$</span> complexity. So the string type will then gain some methods as:</p>
<pre class="lang-js prettyprint-override"><code>// Create new arrays in these specific encoding
// or return the same string, if already internally
// on requested encoding

str s = &quot;...&quot;;
var a = s.ToUtf8();
var b = s.ToUtf16();
var c = s.ToUtf32();
</code></pre>
<p>And in cases the <a href="https://stackoverflow.com/questions/35252033/why-is-string-charatx-much-slower-than-stringx">performance</a> really matters, and can be traded for memory:</p>
<pre class="lang-js prettyprint-override"><code>var d = s.Utf8.ToArray();
var e = s.Utf16.ToArray();
var f = s.Utf32.ToArray();
</code></pre>
<h3><em>Sequential</em> indexing</h3>
<p>The most accessed method on Java's <code>string</code> is <code>charAt()</code>. That is no mistake. Every XML or JSON parser over there will use <code>charAt()</code> or <a href="https://github.com/openjdk/jdk/blob/9864951dceb0ddc4479ced04b6d5a2363f1e307d/src/java.xml/share/classes/com/sun/org/apache/xml/internal/utils/FastStringBuffer.java#L950" rel="nofollow noreferrer">make around it</a>.</p>
<p>There is something to consider here. Parsers in general will use <code>charAt()</code> in <strong>every code unit.</strong> These accesses are not arbitrary or random, as one <code>charAt()</code> call almost always is performed in an index very next to a previous <code>charAt()</code> call.</p>
<p>That is, indexing is almost always sequential or proximal.</p>
<p>The <span class="math-container">$O(n)$</span> complexity on UTF-8 indexing occurs on a random access basis and <em>no memorization</em>. But if one specific string object &quot;remembers&quot; the last byte to code unit mapping, it can then use another algorithm to navigate around the byte stream, instead of starting every time from beginning.</p>
<p>So a new string object may reserve space for this memorization, or, even better, make this sequential/proximal indexing very visible.</p>
<pre class="lang-js prettyprint-override"><code>str s = &quot;...&quot;;

var walker = s.Utf16.Walker;

var p = walker.Prev; // Previous code unit or zero
var t = walker.Code; // Actual code unit or zero
var n = walker.Next; // Actual code unit or zero

if ( walker.HasNext ) {};
if ( walker.HasPrev ) {};
if ( walker.HasCode ) {};

walker.Advance();
walker.Advance(n);
walker.Rewind();
walker.Rewind(n);
</code></pre>
<h3>Indexing and Moore's law</h3>
<p>It's important to take a hint of history of C's <code>fgetpos()</code>/<code>fsetpos()</code>, and make all positioning on string to use <code>WalkerPosition</code>/<code>fpos_t</code>/<code>nint</code>/<code>u0</code>, instead of built-in integers.</p>
<p>Built-in integers can be used in indexing and advancing/rewind as usual, but absolute positioning should be opaque, as <code>fpos_t</code>.</p>
<h3>Normalisation</h3>
<blockquote>
<p>Normalisation: Are strings automatically normalised internally into some canonical form?</p>
</blockquote>
<p>As above, system configurable and explicit invoked methods and views.</p>
<pre class="lang-js prettyprint-override"><code>var b = File.ReadAllBytes(filename);
var r = new str(b);

var s = r.Normalize(); // System default, configurable

var nfd = s.NormalizeFormD();
var nfc = s.NormalizeFormC();
var nfkd = s.NormalizeFormD();
var nfkc = s.NormalizeFormC();
</code></pre>
<blockquote>
<p>Equality: When are two strings considered equal? Encoding or normalisation of the value may matter here, or may be abstracted away.</p>
</blockquote>
<p>As strings may have may have different encoding and freely convertible between these encoding, without changing their meanings, then:</p>
<ul>
<li><p>Strings with same encoding compare equals if are byte identical (fast path);</p>
</li>
<li><p>Strings with different encoding compare equals if they have the same code points. That is, in general:</p>
</li>
</ul>
<pre class="lang-js prettyprint-override"><code>def Equals( this? , other: str ) :bool
    if ( this == null &amp;&amp; other == null )
        return true;
    if ( this == null &amp;&amp; other != null )
        return false;
    if ( this != null &amp;&amp; other == null )
        return false;
    var s1 = this.Utf32.Walker;
    var s2 = other.Uft32.Walker;
    for ( ; s1.HasNext , s2.HasNext ; s1.Advance(), s2.Advance() )
        if ( s1.Code != s2.Code )
            return false;
    if ( s1.SamePosition(s2) )
        return true;
    return false;
</code></pre>
<h3>Ordering</h3>
<blockquote>
<p>Ordering: When does one string precede another for comparison or sorting operations?</p>
</blockquote>
<p>As with Normalization, make configurable if the string uses a &quot;binary fast&quot; collation, to compare code points directly, uses a specific collation, or tries to detect and select a collation from systems configuration.</p>
<p>Also, is sometimes necessary to compare string with different collations, so a explicit API is in order:</p>
<pre><code>var cmp = new String.Collation(code);
cmp.Compare(str1, str2);
</code></pre>
<h3>Small string optimization</h3>
<blockquote>
<p>I'm interested in what design choices were made and what the implications and reception of those choices by users of the language were, if that's notable.</p>
</blockquote>
<p>There is a notable, somewhat recent development. <a href="https://stackoverflow.com/questions/21694302/what-are-the-mechanics-of-short-string-optimization-in-libc">Small string optimization</a> (SSO) is an implementation that boosts performance of systems dramatically.</p>
<p>Designing your strings API to make it possible to implement SSO is important.</p>

CC BY-SA 4.0
2023-10-26T16:12:23.150Z
2023-10-26T16:31:02.420Z
2023-10-26T16:31:02.420Z
André L F S Bacci
# How have modern language designs dealt with Unicode strings?
Posted by: username_0 on 2023-06-13
Languages developed over the last fifteen years or so have been within the era where Unicode is ubiquitous, and so could design their core string types accordingly. There are a lot of new issues that Unicode brings up, and I'm interested in how they've been traded off in different designs.
I've listed some of the dimensions of those design choices below the line.

I'm looking for instances of different **real-world general-purpose languages** released from **around 2008 onwards** addressing Unicode in their strings. Languages that I know released fitting in that group and made relevant design choices include Rust, Swift, Go, Raku, and Python 3, but there are surely others too. ***What* did these languages decide to do with their string type(s), and if applicable *how* has that been received by programmers?**

References to design documentation, rationale, or notable reception are welcome.

---

Unicode raises a number of questions and design choices that the preceding history of languages leaning on predominantly byte or 16-bit strings in platform-dependent encodings didn't. For example:

* Encoding: Are strings stored in UTF-8, UTF-16, UTF-32, a combination, or something else? Is that made visible to the user?
* Characters: What is the atomic unit of a string, if anything? It may be a byte, a code unit, a code point, a grapheme cluster, or something else.
* Indexing: Possibly related, what are the semantics and ergonomics of indexing into a string ("give me character 5", or from 5 on), if there is any? Iteration may also be relevant.
* Normalisation: Are strings automatically normalised internally into some canonical form?
* Equality: When are two strings considered equal? Encoding or normalisation of the value may matter here, or may be abstracted away.
* Ordering: When does one string precede another for comparison or sorting operations?

Any given language might deliberately deal with only some of these. A variety of different reasonable designs are available for each, with tradeoffs for different combinations. I'm interested in what design choices were made and what the implications and reception of those choices by users of the language were, if that's notable.

I'm thinking about the run-time string values here, so language *syntax* for strings is relevant only as far as it's directly supporting the semantics (for example, syntactic support for different forms of indexing or specifying different encodings on a string literal).
**Tags:** language-design strings design-choice-comparison unicode
> **Comments:**
> **username_1**: I can't help but say this to prospective language designers as a grumpy user of programming languages: if your core string type is UTF-8 (which is fine) you better give your users good tools in your standard library to manipulate common things (like file paths) that may not be valid UTF-8. Lookin at you golang/node.js.  
---
## 6 answers
### Answer by username_6 on 2023-06-13. Score: 27
Swift
Swift strings are complicated. At the time of writing, the implementation of Swift.String is split across more than 30 files in the standard library, and that's not counting things that need to be handled in the compiler (like string literals) or Objective-C (like NSString).
In Swift, the stack-/register-allocated storage of a String is 16 bytes. It's a tagged union of one of a few things:

An NSString, which is an abstract class from Objective-C. Usually this stores immutable UTF-16, but it can use a number of other encodings such as ASCII or UTF-32. This member of the union only exists on Apple's platforms, since Objective-C interop is not supported on Windows or Linux.
If the string is no more than 15 bytes in UTF-8 with no null terminator, the first n bytes are its UTF-8 representation, the next 15 - n bytes are all zeroes, and the low nibble of the last byte is the string's length (source).
A pointer to a copy-on-write, resizable, null-terminated UTF-8 buffer (docs and source). Sometimes, there's a "breadcrumbs" pointer after the null terminator, but more on that later.
If the optimizer can prove the string does not grow arbitrarily large or escape the current stack frame, it uses an owned stack buffer rather than a CoW heap buffer (forum post).
A pointer to an "immortal" UTF-8 buffer that the Swift runtime can't modify even to resize, i.e. a string literal (source).

But, we're here to talk Unicode, so let's talk Unicode.
Character
A Character in Swift is an extended grapheme cluster (docs). This can be more than one Unicode codepoint, but it always looks like a single character. Examples of multi-codepoint characters include \r\n (yes, I know) and é ('e' + U+0301). If you want a single Unicode codepoint, that type is Unicode.Scalar, which is just a wrapper around UInt32 (source).
In terms of storage, a Character is stored exactly the same way as a String! (source)
@frozen
public struct Character: Sendable {
  @usableFromInline
  internal var _str: String

  @inlinable @inline(__always)
  internal init(unchecked str: String) {
    self._str = str
    _invariantCheck()
  }
}

count
In Swift 3, String.count gave a very specific error message (example):

'count' is unavailable: there is no universally good answer, see the documentation comment for discussion

However, starting in Swift 4, it is available. .count is an O(n) operation that counts the number of extended grapheme clusters in the string (docs). This is consistent with the fact that String implements Sequence -- String.count is the number of Characters in the String.
String Views
Swift offers a number of views on a string that don't allocate new storage:

String.unicodeScalars is a Sequence rather than a Sequence, but otherwise operates in much the same way as the original String (docs). General-purpose string algorithms are intentionally not available for this type.
String.utf16 is a Sequence (docs). If the string is stored in another format, this converts one value at a time as you iterate over it (source).
String.utf8 is a Sequence (docs). It offers fast access to the raw bytes for a String that's already in UTF-8, and converts an NSString one value at a time as needed (source).

Breadcrumbs
In order to speed up repeated accesses to utf16, UTF-8 strings may put a breadcrumbs pointer after the null terminator (source). The breadcrumbs are a list of the UTF-8 offsets of every 64th UTF-16 code unit (source). A string stores whether it has breadcrumbs in an unused bit in its capacity field (source).
Indexing
Swift Strings aren't indexable by an integer, the way an Array is. Instead, there's a specific type String.Index that's used to avoid the O(n) index access UTF-8 would normally impose. To summarize, it's the following 64-bit bitfield (listed in order from MSb to LSb; source):

a 48-bit number ("position" or "encoded offset") that is the raw offset in the underlying buffer;
a 2-bit number ("transcoded offset") that's the location within the Unicode scalar, when applicable (e.g. if it points to the third byte of a 4-byte UTF-8 character);
ten bits that are purely private and not even used by the ABI (the "resilient slice") -- at the time of writing, only six of them are used, and it stores the distance to the next grapheme cluster, if known; and
four single-bit flags, for which '1' is true and '0' is either false or unknown:

the underlying buffer is UTF-16,
the underlying buffer is UTF-8,
the index points to a boundary between Characters (_isCharacterAligned), and
the index points to a boundary between Unicode scalars (_isScalarAligned).



Before Swift 5.7, the resilient slice was thirteen bits, and included what is now the first three of the four flags.
This layout reduces the overhead of advancing the index forwards by spreading out when the UTF-8 is interpreted. The resilient slice also allows some room for the implementation to improve while still being ABI-stable.
String Literals
A literal like "a" could be any of the following types:

String, obviously.
Substring, less obviously (docs). A Substring is just a slice of a String, but it isn't an unowned slice: it keeps the original String alive (docs). This allows Substrings to be expressed with string literals.
NSString (docs).
Character, if it's only one grapheme cluster (docs).
Unicode.Scalar, if it's only one Unicode codepoint (docs). This is converted to UTF-32 so that it's always four bytes.
StaticString, which is either a pointer to some constant UTF-8 data in the binary, or in some rare cases, a single character in UTF-32 (docs). This is generally used for things you want to know at compile-time, like the file name passed to assert(_:_:line:file:).

Equality and Normalization
String equality is elementwise equality of its Characters*. Character equality is not bytewise equality. Character equality uses the canonical normalized form, but Characters are not automatically normalized (example):
import Foundation

func printRawBytes(_ c: Character) {
  let ints = unsafeBitCast(c, to: (UInt64, UInt64).self)
  print(String(format: "0x%.16llX%.16llX", ints.0.bigEndian, ints.1.bigEndian))
}

let precomposed: Character = "é"
printRawBytes(precomposed) // 0xC3A900000000000000000000000000A2

let combining: Character = "é"
printRawBytes(combining) // 0x65CC81000000000000000000000000A3

print(precomposed == combining) // true

Strings that use an external buffer set a bit in the class header when they are known to already be NFC normalized (source). This can speed up certain operations: for example, two strings that use the same encoding and are both normalized can be compared bytewise.
*While this is semantically true, that's not how the Swift stdlib actually implements it. Instead, Character equality delegates to String equality, and String equality is its own mess.
Ordering
String equality is relatively straightforward: do the strings represent the same text? However, Strings can also be compared for order: str1 < str2 is legal. While it's generally preferred to use methods like localizedStandardCompare for user-facing text, for some cases it's better to be locale-independent.
For NFC normalized UTF-8 text, it just uses memcmp. Other formats are converted to that and compared one Character at a time. The use of memcmp here sounds strange at first, but remember that this is for providing a consistent ordering across versions and locales, not for alphabetizing user-facing text.
Performance Issues
As was recently pointed out on the Swift forums, there is no limit to how large a Character can be. Consider the following:
let problematic = "a" + String(repeating: "\u{0301}", count: 20_000_000) + "x"
// `problematic` is 20,000,002 Unicode scalars, but only 2 grapheme clusters!

func thisWillTakeAWhile() -> Character {
  problematic.first!
}

Because a Character is an owned (often unique) String, never a Substring, that has to copy the 20,000,001-scalar grapheme cluster into a new string. This forms pathological cases for a lot of functions that assume materializing a Character takes negligible time, such as haystack.contains(needle). That function, at the very least, compares the first character of needle to each character in haystack until it finds a match. When retrieving the first character of needle takes a noticeable amount of time, even obviously (to us) false ones like "Hello, World!".contains(problematic) take several seconds to execute (example, though I'm seeing around 1.3 seconds, rather than the 10 seconds reported in the forum post).
---
### Answer by username_2 on 2023-06-13. Score: 13
# Rust

The default string types `(&)str` and `String` are both UTF-8, ignoring native encoding. There are various other types and conversion methods to convert between native rust strings and other encodings, such as `UTF-32` (the `widestring` stdlib crate) and native strings (`OsString`)

The "atomic unit" of a string is defined to be a `char`, representing a single UTF-8 codepoint - a `Unicode scalar value`.

You cannot directly index a string without converting it to a byte-like Rust-level representation and therefore losing the invariant that the string is correct UTF. You can, however, access the codepoint `n` (which is of course a `O(n)` operation).

As far as I am aware, everything else just uses the UTF-8 byte values in terms of ordering, encoding, and equality.

Also as far as I know, this approach has been pretty well received in the linux community, as UTF-8 is prevalent there. I cannot speak for the Windows community.
> **Comments:**
> **username_0** (2023-06-13): When you say "standard for UTF-8 in terms of ordering, encoding, and equality", you mean it just uses byte ordering and equality (like a C string)?  
> **username_2** (2023-06-13): @MichaelHomer Yes, as confirmed by the source code here: https://doc.rust-lang.org/src/alloc/string.rs.html#362  
> **username_3** (2023-07-03): A few issues in this answer... First a codepoint is independent from an encoding, hence `UTF-8 codepoint` has no meaning. Second you can definitely [index a string](https://doc.rust-lang.org/std/primitive.str.html#impl-Index%3CI%3E-for-str) in Rust, though you do have to be aware that shall indexes fall on non codepoint boundaries, a panic will ensue. Thirdly, it may be important to point out that Rust aims for a _minimal_ standard library, so that relegating more Unicode functionality to 3rd-party crates is not shocking to Rust users... though it may be to newcomers.  
> **username_4** (2023-07-05): this is fairly similar to what Julia does with strings  
> **username_5** (2023-07-07): Note that UTF-8 byte order is the same as Unicode code point order.  This is *not* the case for UTF-16, due to the "surrogate" code units not being at the end of the 16-bit code space.  
---
### Answer by username_12 on 2023-06-13. Score: 8
# Python 3

Conceptually, strings are immutable sequences of code points.

Indexing strings results in a string containing a single code point. Characters not exposed as a separate type.

Equality is based on equality of code points, and ordering is based on the value of the code points. Normalisation is not done implicitly.

The representation in memory is more of an implementation detail, but in order to keep indexing as fast as expected by users, strings in CPython are represented as arrays of code points, either using 1 byte (Latin-1), 2 bytes (UCS-2, aka UTF-16 not containing surrogate pairs) or 4 bytes (UTF-32), using the smallest representation that fits all code points in the string. This is not exposed to Python users.

Compare and contrast `bytes`, Python 3's immutable byte sequence:

Indexing `bytes` gives an `int`.

Equality is based on byte values, and ordering is based on the value of the bytes. Normalisation is not relevant to byte sequences.
> **Comments:**
> **username_10** (2023-07-03): What about Ordering?  
> **username_12** (2023-07-03): I'm sorry, what is your question exactly?  
> **username_10** (2023-07-03): Sorry, I failed to read the last sentence of your answer! :-(  
---
### Answer by username_7 on 2023-06-13. Score: 4
This does not address 2008 onwards (which seems a pretty arbitrary date) but provides some background instead.

# Treat it as a (mostly) library issue

## *A ~~[monad](https://stackoverflow.com/questions/3870088/a-monad-is-just-a-monoid-in-the-category-of-endofunctors-whats-the-problem)~~ string is just an array of bytes interpreted as UTF-8 what's the problem?*

A fair few languages treat it as a library issue.

Users thinking about Unicode know:

* UTF-8 is a variable length encoding
* The difference between indexing a byte and a character

So they expect operations to distinguish:

* arrays of bytes
* arrays of characters/code points (i.e. strings of text)

And provide different sets of operations for each and conversions from one to the other.

Getting this right can be hard but you can farm it out.
Rather it is easy to miss a corner case and introduce bugs which is unfortunate in a core library.

A string is of course a core type but starting with the **faulty** assumption that most users are using Western scripts it is sufficient to:

## Use UTF-8 everywhere

* Source files are encoded as UTF-8

  + so you can write comments natively
  + possibly use non-western letters as variable names
* The core string type is an array of bytes ideally with a length rather than null terminated

  + So you can embed nulls.
  + So inserting a null does cannot be used as an attack vector to truncate strings

So:

* C having null-terminated strings gets this wrong.
* C++ follows C, so `std::string` originally got this wrong too (and zero-terminated strings are still needed in places).
* [D](https://en.wikipedia.org/wiki/D_(programming_language)) from 2001 gets this right - as do most other newer languages

## Leave it to a (possibly third party) library

For example:

* [ICU](https://icu.unicode.org) - International Components for Unicode.
* Users of the Qt library will use [`QString`](https://doc.qt.io/qt-6/qstring.html).

  `QString` is an array of UTF-16 so there are potential issues with code points above 0xFFFF which require 2 `QChar`'s to represent - see [*Retrieve Unicode code points > U+FFFF from QChar*](//stackoverflow.com/q/6972883).

  This is a good example of how subtle bugs remain even in implementations we might hope are Unicode safe.

## But sometimes it's an OS issue...

The Windows API is quite fond of UTF-16

Because Windows is popular, this leads to abominations like [`std::wstring`](https://stackoverflow.com/questions/402283/stdwstring-vs-stdstring) becoming standard library elements in some languages.
It's an abomination because:

* It should not be necessary to make your code generic over more than one string representation.
* Because 2 bytes will be more than enough for everyone (oops - no it won't).

See also [*UTF-8, UTF-16, and UTF-32*](//stackoverflow.com/q/496321).

## Maturity brings wisdom

With enough maturity a good Unicode library should become a de-facto or blessed standard or a core library.

Putting it into a clever type system and getting it right early on would be great though so I look forward to reading the other answers.
> **Comments:**
> **username_11** (2023-08-16): Yeah, it's a shame that crippled implementations of `std::wstring` are out there.  I get used to assuming that `wchar_t` can hold _any_ code point - thankfully I don't have to target Windows platforms!  
> **username_8** (2023-10-18): Many aspects of Unicode should be OS issues, save for the lack of a useful specification for a set of primitives an OS should support.  Changes to the Unicode standard should not require rebuilding programs that might be affected thereby, but should instead be accommodated via OS update which would then have the appropriate effect on applications that use the OS functions.  Unfortunately, there is at present no way to avoid baking changeable aspects of the Unicode standard into the application layer.  
> **username_9** (2023-10-19): If your interpretation is UTF-8 in-memory then you give up on random-access character indexing.  
> **username_7** (2023-10-19): That is still true in systems that use UTF-16. It is the nature of the beast that characters, graphemes, code point and what not do not have to match up anymore.  
> **username_8** (2023-10-24): @KarlKnechtel: Unicode traded encoding density for the ability to extract code points in context-free fashion, but then specified the behavior of grapheme clusters in a manner that would make it impossible to, in the fully general case, determine whehter a grapheme cluster boundaries exists between two bytes near the end of a billion-character string without scanning every single byte preceding them.  
> **username_8** (2023-10-24): @KarlKnechtel: Outside of the specific task of converting strings between UTF-8, UTF-16, and UTF-32, there aren't many situations where code-point boundaries are significant, so I'm not sure why the inability to retrieve the Nth code point given a string and N would be a problem.  
> **username_9** (2023-10-25): ...Because people want to be able to splice strings by index?  
> **username_10** (2023-10-26): @KarlKnechtel They do, but why? Even fully understanding all Unicode complexities, i18n, blah, blah? What are the use cases? (I can think 2, where perhaps some other data type may be better: file system paths or web URIs, and strings of symbols such as DNA, number/license plates, product codes (e.g. EAN), etc.)  
> **username_9** (2023-10-26): @PabloH You'd have to ask them, honestly. Modern language designers don't seem brave enough to drop that support.  
> **username_8** (2024-10-15): @PabloH: Common use cases involve (1) working with text data that is intended primarily to be read by machines rather than humans, rendering internationalization issues irrelevant, and (2) being able to convert a (needle, haystack) string into combination of things that can be used to derive the part of the haystack before or after the needle, or one of the above with the needle included.  UTF8 with byte indexing is fine for any of those tasks.  
---
### Answer by username_13 on 2023-10-26. Score: 1
# Nim

All strings are UTF-8 strings by convention, but it is not enforced.

Strings are char arrays, where `char` is a single byte. Indexing into a string is *safe* in the sense that Nim will not panic upon getting a char > 127 representing a fraction of a codepoint, but it will just treat it as a byte.
Strings are conventionally treated as no more than a char array. There is a commonly used special `openarray[T]` type that abstracts over lists (`seq[T]`), arrays of various sizes, and strings.

There is a separate `std/unicode` module in the standard library that provides a (four-byte) `Rune` type representing a Unicode character, various associated functions, and a `.runes()` iterator.

Strings are both length-prefixed and null-terminated. Nim also has the `cstring` type for compatibility with C, which behaves the same as the `string` type, but lacks a length field and is instead only null-terminated. `string` is implicitly convertible to `cstring`. It doesn't do anything terribly different wrt. Unicode.
---
### Answer by username_14 on 2023-10-26. Score: -1
Languages developed over the last fifteen years or so have been within the era where Unicode is ubiquitous, and so could design their core string types accordingly.

There is a lot of history, and because of that, there is a lot of backward compatibility that may need to be made avaliable in new languages. The considerations below focus in this extra requisite.
This answer is not specific to a previous language, but considerations for a possible future language that tries to avoid a lot of pitfalls in string design. It's heavly inspired in Swift design, as mentioned in other answer, and provides some design alternatives (naming, sequential/proximal indexing, opaque positions and SSO)
Encoding

Encoding: Are strings stored in UTF-8, UTF-16, UTF-32, a combination, or something else?

Make the internal encoding a pure implementation detail. It's the only way to ensure you can change it, globally or on a per case basis. Python's experience in changing strings between versions 2 and 3 showed how disastrous exposing any internal encoding can cause afterwards.
Views

Is that made visible to the user?

Expose the popular encoding via non allocating views.
Say:
str s = "...";
foreach (a in s.Utf8)  {} // a is u8
foreach (b in s.Utf16) {} // b is u16
foreach (c in s.Utf32) {} // c is u32
foreach (d in s.CChar) {} // machine dependent sized type

These views should be implemented like like C#'s structs or non-malloc()ed C's structs. That is, objects allocated in directly stack frames.
Character

Characters: What is the atomic unit of a string, if anything? It may be a byte, a code unit, a code point, a grapheme cluster, or something else.

Offer all, explicitly, and avoid all confusion.
str s = "...";
foreach (a in s.Bytes)     {} // a is u8
foreach (b in s.Runes)     {} // b is u32
foreach (c in s.Graphemes) {} // c is str

Rune is a common name for code points. But if it is the same as .Utf32 view, it may be omitted.
Indexing

Indexing: Possibly related, what are the semantics and ergonomics of indexing into a string ("give me character 5", or from 5 on), if there is any? Iteration may also be relevant.

There is no indexing or counting on string objects, only in views.
That is, the indexing by .Bytes accesses the underlying byte sequence, no transformation is applied. Indexing by .Utf8 is $O(1)$ complexity in strings stored as UTF-8 strings, and $O(n)$ complexity for strings in other encoding, and so on.
With this, it is obvious that some users will want to make certain strings in certain encoding sometimes, to obtain the $O(1)$ complexity. So the string type will then gain some methods as:
// Create new arrays in these specific encoding
// or return the same string, if already internally
// on requested encoding

str s = "...";
var a = s.ToUtf8();
var b = s.ToUtf16();
var c = s.ToUtf32();

And in cases the performance really matters, and can be traded for memory:
var d = s.Utf8.ToArray();
var e = s.Utf16.ToArray();
var f = s.Utf32.ToArray();

Sequential indexing
The most accessed method on Java's string is charAt(). That is no mistake. Every XML or JSON parser over there will use charAt() or make around it.
There is something to consider here. Parsers in general will use charAt() in every code unit. These accesses are not arbitrary or random, as one charAt() call almost always is performed in an index very next to a previous charAt() call.
That is, indexing is almost always sequential or proximal.
The $O(n)$ complexity on UTF-8 indexing occurs on a random access basis and no memorization. But if one specific string object "remembers" the last byte to code unit mapping, it can then use another algorithm to navigate around the byte stream, instead of starting every time from beginning.
So a new string object may reserve space for this memorization, or, even better, make this sequential/proximal indexing very visible.
str s = "...";

var walker = s.Utf16.Walker;

var p = walker.Prev; // Previous code unit or zero
var t = walker.Code; // Actual code unit or zero
var n = walker.Next; // Actual code unit or zero

if ( walker.HasNext ) {};
if ( walker.HasPrev ) {};
if ( walker.HasCode ) {};

walker.Advance();
walker.Advance(n);
walker.Rewind();
walker.Rewind(n);

Indexing and Moore's law
It's important to take a hint of history of C's fgetpos()/fsetpos(), and make all positioning on string to use WalkerPosition/fpos_t/nint/u0, instead of built-in integers.
Built-in integers can be used in indexing and advancing/rewind as usual, but absolute positioning should be opaque, as fpos_t.
Normalisation

Normalisation: Are strings automatically normalised internally into some canonical form?

As above, system configurable and explicit invoked methods and views.
var b = File.ReadAllBytes(filename);
var r = new str(b);

var s = r.Normalize(); // System default, configurable

var nfd = s.NormalizeFormD();
var nfc = s.NormalizeFormC();
var nfkd = s.NormalizeFormD();
var nfkc = s.NormalizeFormC();


Equality: When are two strings considered equal? Encoding or normalisation of the value may matter here, or may be abstracted away.

As strings may have may have different encoding and freely convertible between these encoding, without changing their meanings, then:

Strings with same encoding compare equals if are byte identical (fast path);

Strings with different encoding compare equals if they have the same code points. That is, in general:


def Equals( this? , other: str ) :bool
    if ( this == null && other == null )
        return true;
    if ( this == null && other != null )
        return false;
    if ( this != null && other == null )
        return false;
    var s1 = this.Utf32.Walker;
    var s2 = other.Uft32.Walker;
    for ( ; s1.HasNext , s2.HasNext ; s1.Advance(), s2.Advance() )
        if ( s1.Code != s2.Code )
            return false;
    if ( s1.SamePosition(s2) )
        return true;
    return false;

Ordering

Ordering: When does one string precede another for comparison or sorting operations?

As with Normalization, make configurable if the string uses a "binary fast" collation, to compare code points directly, uses a specific collation, or tries to detect and select a collation from systems configuration.
Also, is sometimes necessary to compare string with different collations, so a explicit API is in order:
var cmp = new String.Collation(code);
cmp.Compare(str1, str2);

Small string optimization

I'm interested in what design choices were made and what the implications and reception of those choices by users of the language were, if that's notable.

There is a notable, somewhat recent development. Small string optimization (SSO) is an implementation that boosts performance of systems dramatically.
Designing your strings API to make it possible to implement SSO is important.
---
