<p>In many languages such as Java, strings and character arrays are distinct types. In Java one must use <code>toCharArray()</code> in order to use array semantics on strings.</p>
<p>In C, strings and character arrays are more or less the same. A C string is just a character array with a null terminator. Otherwise there is no internal distinction between them.</p>
<p>What are the advantages or disadvantages of strings being distinct types from character arrays?</p>

<c><strings><arrays><characters>
What are the advantages of strings and character arrays being different?
<h1>Separate string methods or going via arrays</h1>
<p>A disadvantage of a string type is that the methods available to manipulate arrays cannot just be used for strings. Instead, the language must either provide a separate set of functionality (which means more to document, learn, etc.) or force the programmer to split a string into an array whenever character manipulations are needed.</p>
<p>Array programming languages like APL, J, and K, make extensive use of array functions to manipulate strings, with great effect.</p>

CC BY-SA 4.0
2023-05-16T18:21:02.087Z
2023-05-16T18:21:02.087Z
Adám
CC BY-SA 4.0
2023-07-03T18:48:47.153Z
Ideally I'd like a string _not_ to share methods with an array. What is `"\U0001F1F8\u200d\U0001F1FA".map(println)` supposed to do? Query the OS' default font to figure it out at runtime?
Longinus
CC BY-SA 4.0
2023-07-03T18:50:01.253Z
@Longinus I'd expect it to loop over Unicode codepoints. That's what [Scala does](https://scastie.scala-lang.org/ltLr73CJRxqE6lOQWeQu4Q).
user
CC BY-SA 4.0
2023-07-03T18:51:23.923Z
@user That would break any text editor GUI using that code.
Longinus
CC BY-SA 4.0
2023-07-03T18:53:47.437Z
@Longinus I don't see why it would. A text editor specifically working with Unicode would use a separate method that's aware of the fact that the "array" passed into it is really a string in UTF-8 or whatever, to be displayed in a specific font.
user
CC BY-SA 4.0
2023-07-03T19:04:28.573Z
@user Unless you are implementing a compiler for a particular system, Unicode is the default, not a specificity the user should give particular care to enable. And the Standard defers the meaning of `"\U0001F1F8\u200d\U0001F1FA"` to the ISO 3166 Maintenance Agency which observes _geopolitics_. To ascribe some other API behaviour requires you to ignore ISO. Also why the Standard says a "character" is "what the user sees."
Longinus
CC BY-SA 4.0
2023-07-03T20:17:51.823Z
@Longinus That is true, but I don't see what stops you from making a method that works on arrays of `Character`s with the same behavior as a method working on strings in a language with a separate string type
user
CC BY-SA 4.0
2023-07-03T21:03:26.200Z
@user We can, but when `map`ping over an `Array<T>` we know we deal with `T`s, whereas when `map`ping over a `String`, nothing tells us what we're actually doing - code units? Code points? Graphemes? Grapheme clusters? Glyphs? Composed emojis? Emoji components? ZWJ sequence components? All of which fit one of the **many** definitions of the word "character" - so I don't think we should. Also whether one option is more "obvious" than another depends on whether you're dealing with text (which is graphical) or memory (which is not).
Longinus
CC BY-SA 4.0
2023-07-10T15:24:53.117Z
As a counter-example, C++ has string types that can be manipulated using the `<algorithm>` functions that work equally well on arrays or other containers.  (Yes, C++ strings are somewhat limited in that they tend to hold only a single codepoint at each position, but the language would allow a more sophisticated grapheme-cluster representation to be implemented and still satisfy the Container interface.)
Toby Speight
CC BY-SA 4.0
2023-07-24T01:39:50.730Z
@Longinus I wrote an answer that gives some focus to this issue. Would appreciate any additional technical insights.
Karl Knechtel
<h2>Encapsulation</h2>
<p>Treating strings as character arrays entails a belief that a character array is an adequate <em>model for</em> a string - not just good enough now, but good enough that you won't want to be able to swap it out later while keeping existing interfaces.</p>
<p>In reality, there are many operations on strings that don't make a lot of sense on character arrays, and vice-versa. It's questionable, for example, whether strings should provide random access at all. If you index into a string, should this produce a &quot;character&quot; that is an instance of a separate type? What should one be able to <strong>do with</strong> that object? Different languages have different answers to this question.</p>
<p>Another problem is that characters are completely different from <em>graphemes</em>. Consider this Python example:</p>
<pre class="lang-python prettyprint-override"><code>&gt;&gt;&gt; 'fiancée'[::-1]
'eécnaif'
&gt;&gt;&gt; 'fiancée'[::-1]
'éecnaif'
</code></pre>
<p>The first string uses a precomposed character to represent the grapheme é with a single code point:</p>
<pre><code>&gt;&gt;&gt; import unicodedata
&gt;&gt;&gt; unicodedata.name('fiancée'[-2])
'LATIN SMALL LETTER E WITH ACUTE'
</code></pre>
<p>The second &quot;composes&quot; the symbol using two separate characters:</p>
<pre><code>&gt;&gt;&gt; unicodedata.name('fiancée'[-2])
'COMBINING ACUTE ACCENT'
&gt;&gt;&gt; unicodedata.name('fiancée'[-3])
'LATIN SMALL LETTER E'
</code></pre>
<p>Similarly with flags:</p>
<pre><code>&gt;&gt;&gt; ''[::-1]
''
&gt;&gt;&gt; ''[::-1]
''
</code></pre>
<p>Your terminal (like mine) might not support the composition of flag emojis from their components (<a href="https://en.wikipedia.org/wiki/Regional_indicator_symbol" rel="nofollow noreferrer">regional indicator symbols</a>), so you might see something more like a fancy <code>AU</code> turning into a fancy <code>UA</code>, or <code>BG</code> into <code>GB</code> - which goes a long way towards explaining the result.</p>
<p>There are much stranger phenomena than that:</p>
<pre class="lang-python prettyprint-override"><code>&gt;&gt;&gt; calendars = &quot;️&quot; + &quot;&quot;
&gt;&gt;&gt; len(calendars)
3
</code></pre>
<p><strong>Depending on your environment</strong>, the two calendars might look the same or different - for me, they are different in my browser, but the same in my terminal. The length is <code>3</code> because the first one is composed of two characters: the calendar emoji followed by a &quot;variation selector&quot;. There isn't a &quot;precomposed version&quot; of that; the variation selector is something like a control character that tells the text engine to draw a different glyph (image) for the previous character, while still treating it as the same character.</p>
<p>It is left as a (hopefully simple) exercise to explain why <code>calendars[::-1]</code> will produce a result that looks the same as <code>calendars</code>, regardless of the environment.</p>
<p>Don't even get me <em>started</em> on the <a href="https://www.unicode.org/reports/tr9/" rel="nofollow noreferrer">bidirectional text layout algorithm and associated control codes</a>. You understood correctly: the people responsible for standardizing Unicode are currently on revision 46 of a document about 18 thousand words long just to explain <em>how to decide when to write text left to right and when to write it right to left</em>.</p>
<p>But, sure, reversing a string is a toy problem that also turns out not to be very important in realistic applications. What about the real meat and potatoes of string manipulation?</p>
<ul>
<li><p>Did you know that some languages have a concept of uppercase and lowercase, but don't have separate uppercase and lowercase versions of every letter? For example, the <a href="https://en.wikipedia.org/wiki/%C3%9F" rel="nofollow noreferrer">German letter ß</a> is considered lowercase; when a word containing it is set in uppercase, the ß is supposed to be replaced with SS (two uppercase Ses)? So it's not actually possible to uppercase words in a locale-aware way by simply iterating over characters and replacing them separately. The length of the string might need to change. I'm not even sure that each character can be considered independently, for every language.</p>
</li>
<li><p>Did you know that many widely-used written languages are only ever intended to appear in a &quot;cursive&quot; form (i.e. there are strokes that visually continue between letters), so there are corresponding variant characters with different appearances (assigned separate code points - not distinguished with variation selectors - perhaps because they had to solve the problem before anyone came up with the idea of variation selectors)? Did you know that you might be expected to substitute the glyphs from those variant characters to render a string, while still treating the string as being composed of whatever was actually put into it? And, if you're the poor sap tasked with implementing text rendering for an operating system, you might be expected to ensure that the characters have their variant appearances when selected, but copying and pasting them gives the original &quot;general&quot; characters?</p>
<pre class="lang-python prettyprint-override"><code>&gt;&gt;&gt; # Try highlighting, copying and pasting individual letters of the output.
&gt;&gt;&gt; # (You might find it tricky because of the LTR rendering.)
&gt;&gt;&gt; print('\u0628\u0628\u0628')
ببب
</code></pre>
</li>
<li><p>Did you know that the order in which people expect strings to be sorted, is not necessarily based on Unicode code point order at all? And that it might be different for different languages (that could validly have the same two strings in them)? And that it might not be possible to apply naive lexicographical ordering at all (i.e. considering the first character first, breaking ties with the second character etc.)? And that this will get completely broken by combining characters anyway? And that you can't actually necessarily determine the language of text just by examining some characters (<code>chat</code> can validly be an English verb or a French noun)? And that there's a <a href="https://www.unicode.org/reports/tr10/" rel="nofollow noreferrer">whole other essay</a> on Unicode.org about how to do it properly? (About 30 thousand words, by the way.)</p>
</li>
<li><p>Did you know that for the East Asian languages that use ideographs and/or logographs (Chinese, Japanese and <a href="https://en.wikipedia.org/wiki/Hanja" rel="nofollow noreferrer">occasionally</a> Korean - and even <a href="https://en.wikipedia.org/wiki/Ch%E1%BB%AF_H%C3%A1n" rel="nofollow noreferrer">Vietnamese</a>, which I didn't know before writing this answer), Unicode provides <a href="https://en.wikipedia.org/wiki/CJK_Unified_Ideographs" rel="nofollow noreferrer">&quot;unified&quot; characters</a> - such that a symbol might be considered &quot;the same character&quot; in different languages, but is supposed to be displayed differently depending on what language is being written? And that you can't rely on variation selectors to tell you which glyph to use? (And, again, for short excerpts, you can't reliably determine the language by examining the characters - after all, it often takes only a single character to express an entire word/concept?) And that the <a href="https://en.wikipedia.org/wiki/Han_unification" rel="nofollow noreferrer">process</a> of doing this was an absolute ton of work, and the results are still contentious in some places? And that sometimes &quot;traditional Chinese&quot; characters and their &quot;simplified Chinese&quot; counterparts use the same code point (as in the table at the previous link), and <a href="https://en.wikipedia.org/wiki/Simplified_Chinese_characters" rel="nofollow noreferrer">sometimes they're separate</a>?</p>
<ul>
<li>And that the entire topic can be <em>really politically sensitive</em>?</li>
</ul>
</li>
<li><p>Did you know that the text in a string could be in more than one language? That it could be in <a href="https://en.wikipedia.org/wiki/Dingbat" rel="nofollow noreferrer"><em>no</em> languages</a>? That it could try to <a href="https://en.wikipedia.org/wiki/Emoji" rel="nofollow noreferrer">express an idea, despite</a> not consisting of words in a language? That it could try to express an idea that <a href="https://en.wikipedia.org/wiki/Musical_Symbols_(Unicode_block)" rel="nofollow noreferrer">isn't linguistic</a>? That it could try to <a href="https://en.wikipedia.org/wiki/Zalgo_text" rel="nofollow noreferrer">decorate meaningful text with meaningless symbols</a>?</p>
</li>
<li><p>Did you know about <a href="https://en.wikipedia.org/wiki/Universal_Character_Set_characters#Surrogates" rel="nofollow noreferrer">the hack used to cram the Unicode code space into 16-bit encodings</a>? (Or, relatedly, the reason why the allowed number of distinct code points <em>isn't a power of 2</em>?)</p>
</li>
</ul>
<p>Text is <strong>really, really</strong> complicated. Attempting to model it as merely an <em>array</em> of characters is hopelessly naive. While, yes, it is conceptually composed of a sequence of characters (in the only models of it we have that are viable to work with on computers), it makes next to no sense to expose a plain array interface to that sequence (whether or not mutable). Arrays have the conceptual affordance of algorithms that just don't work on text.</p>
<hr />
<p>This stands <em>completely independent</em> of the question of whether a <em>byte</em> is an adequate model for a <em>character</em>. Yes, it's true that UTF-8 uses a varying number of bytes to encode each character, such that a string implementation relying on a storage buffer in UTF-8 encoding could not be random access. But this is an absolutely <em>trifling</em> issue compared to ones created by naive ideas about what a &quot;character&quot; actually is, and about how text is written, and about what manipulations make sense for that data.</p>
<p>Sure, we have a standard that precisely, unambiguously and permanently maps abstract numbers to &quot;characters&quot;. We have a virtual &quot;code space&quot; for those numbers, a commitment not to ever reassign them even if it would make things look much more elegant, and a reasonable assurance that we won't overflow that space for a very, very long time, even as we strain to come up with bizarre pseudo-textual things to fill that space with.</p>
<p>And, sure, there are many plausible ways to represent &quot;a character&quot;, depending on system requirements:</p>
<ul>
<li><p>A byte whose values correspond to the first 256 Unicode code points (obviously restricted to text that only uses those characters)</p>
</li>
<li><p>A byte whose values are mapped to Unicode code points in an instance-specific manner, according to encoding metadata stored along with the array (restricted to text that only uses characters from one of such single-byte encodings - which can only exist for languages that fit in such a character limit, and don't deal well with mixed-language text)</p>
</li>
<li><p>An unsigned 16-bit value representing a Unicode code point from the <a href="https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane" rel="nofollow noreferrer">Basic Multilingual Plane</a> (only represents those characters - so no emoji, very limited ideographic support, many world languages missing, no <a href="https://en.wikipedia.org/wiki/Private_Use_Areas" rel="nofollow noreferrer">private use area</a>)</p>
</li>
<li><p>The same, but using surrogate pairs to represent characters outside the BMP</p>
</li>
<li><p>An 32-bit value, which might as well be signed I suppose</p>
</li>
<li><p>A 21-bit value, bit-packed three to each 64-bit machine word (now that 64-bit architectures are the norm)</p>
</li>
</ul>
<p>And, sure, there might not be any need to use a linked list, or other data structure, to store the underlying character data. Simply writing each character's corresponding number sequentially in memory - i.e., using an array - is perfectly adequate for the simple task of remembering which characters are in the string, in what order.</p>
<p>But... <strong>none of these things actually address the fundamental complexity of text</strong>. While it's easy to laugh at the previous generation of Western programmers and their naive faith placed in &quot;code pages&quot;, the &quot;array&quot; part of &quot;array of characters&quot; is also problematic - perhaps much more so.</p>
<p>Exposing such an interface to the raw storage raises many questions. Simply equating the string type with an array-of-characters type - leaving all that string functionality outside the type itself - causes innumerable problems. Please don't do it in your language.</p>

CC BY-SA 4.0
2023-07-24T01:16:15.183Z
2024-05-30T19:06:25.807Z
2024-05-30T19:06:25.807Z
Karl Knechtel
CC BY-SA 4.0
2023-08-25T17:09:20.553Z
+1000 Excellent answer addressing the (huge) gap from arrays to human language text. And you didn't even get to normalization. :-)
Pablo H
CC BY-SA 4.0
2023-08-25T17:42:41.100Z
Although I didn't explicitly mention normalization, I touch on it in a few places.
Karl Knechtel
<p>An array must have a fixed stride, so that the memory address of the <em>i</em>th element equals the address of the start of the array plus <em>i</em> times the stride. If you really want an array of <em>characters</em>, so that each logical character (i.e. each Unicode symbol) occupies one slot in the array, then the array's stride would need to be 32 bits.</p>
<p>That is a waste of space compared to UTF-8 encoding, in which the most common characters take 8 or 16 bits each. On the other hand, if you encode strings as UTF-8 then it's no longer true that the <em>i</em>th character occurs at a predictable index, so what you have is not an array of characters. (Instead, it is an array of bytes, but that's the internal implementation ─ the API need not expose it to the user as such.)</p>

CC BY-SA 4.0
2023-07-03T18:25:17.503Z
2023-07-03T18:25:17.503Z
kaya3
CC BY-SA 4.0
2023-07-03T18:34:03.723Z
_Technically_, you only need 21 bits. That doesn't exactly optimize well, though, as loading an odd number of bits is even worse than a regular unaligned load.
Bbrk24
CC BY-SA 4.0
2023-07-03T18:35:41.827Z
A "Unicode symbol" can be more than 32bits. National flags are 64bits, and both combining characters and ZWJ sequences allow for "symbols" to be arbitrarily long.
Longinus
CC BY-SA 4.0
2023-07-03T18:43:40.120Z
A Unicode *code point* must be between 0 and 0x10FFFF, hence "21 bits".  It's true that a user-perceived "character" can contain multiple code points, though.
dan04
CC BY-SA 4.0
2023-07-03T18:55:59.583Z
Arrays cannot have a stride of 21 bits; it would be rounded up to 32.
kaya3
CC BY-SA 4.0
2023-07-03T19:00:47.850Z
@Longinus To my knowledge, the national flag symbols are not "Unicode symbols" ─ they are "individual symbols" because applications render them as such, but Unicode doesn't define them as far as I know. Really they are sequences of Unicode symbols which are interpreted as national flags by convention rather than by the Unicode standard.
kaya3
CC BY-SA 4.0
2023-07-03T19:20:11.290Z
@kaya3-supportthestrike Unicode describes national flags as "components" of "emojis," and all emojis as "symbols." Then, TR51 explicitly mentions that there are concurrent ways to "count emojis" including delegating work to the system's font and the fact that this is a platform-specific problem.
Longinus
CC BY-SA 4.0
2023-07-07T18:07:01.853Z
@kaya3-supportthestrike Would 24 be possible? Or do arrays of three-byte objects (e.g., a tuple of `i8`s or an `Option<i16>`) get rounded up to 32 for some sort of alignment reason?
rydwolf
CC BY-SA 4.0
2023-07-10T15:28:44.160Z
Why can't arrays have a stride of 21 bits?  That seems quite reasonable on hardware that has a 7-bit word size (obviously that's one that can't run C, but we haven't chosen our target yet).
Toby Speight
CC BY-SA 4.0
2023-07-24T01:42:17.587Z
@TobySpeight but if we are considering impractical and/or theoretical machines, why do we still care about the Unicode standard?
Karl Knechtel
CC BY-SA 4.0
2023-07-27T07:31:53.033Z
@Karl,  I'm merely referring to [5.2.4.2.1](https://web.archive.org/web/20181230041359if_/http%3a//www.open-std.org/jtc1/sc22/wg14/www/abq/c17_updated_proposed_fdis.pdf#subsection.5.2.4), which specifies that `CHAR_BIT` must be at least 8.  BTW, that doesn't mean that these older machines are necessarily _impractical_, but that that their implementation of C needs a little care.
Toby Speight
CC BY-SA 4.0
2023-07-27T07:40:32.110Z
I understood what you meant by "can't run C" (as you clarify, need special considerations in order to run C). My point is that 21-bit strides make no sense on *ordinary* hardware *in common use* that follow contemporary *standards*; and if those standards and expectations are out the window, then I don't see why respecting the Unicode standard would still be important.
Karl Knechtel
CC BY-SA 4.0
2023-07-27T11:16:45.063Z
This discussion is all moot; even if you had an exotic CPU architecture where a character array could have a stride of 21 bits, it would still be a waste of space compared to a string encoded in UTF-8 where most characters take only 8 or 16 bits. The conclusion of the answer still holds.
kaya3
<h1>Different definitions of “character”</h1>
<p>In Swift, the <code>Character</code> type isn’t a single Unicode codepoint (that’s <code>UnicodeScalar</code>), nor is it a single byte (that’s <code>CChar</code>). Instead, it represents an “extended grapheme cluster”, which includes combining diacritics. This makes some things more straightforward — <code>é</code> is one Unicode codepoint and <code>é</code> is two, but they’re both a single character. However, it means that characters aren’t a fixed size, and for <code>String</code> to just be <code>[Character]</code> would be a massive waste of memory. Don’t get me wrong, it is a <code>Collection&lt;Character&gt;</code>, just not a random-access collection the way an array is.</p>

CC BY-SA 4.0
2023-05-16T18:07:45.643Z
2023-05-16T18:07:45.643Z
Bbrk24
<h1>Handling Encoding</h1>
<p>Arrays of raw data don't, generally speaking, have an encoding. Strings can be in UTF-8 or any number of other formats. In Ruby, a <code>String</code> object is an array of bytes together with its <code>Encoding</code>. If you're handling data in different formats (such as data coming in from the network and then entering your main program's file system or database), it can be invaluable having the language keep track of this for you.</p>

CC BY-SA 4.0
2023-05-16T18:03:09.157Z
2023-05-16T18:03:09.157Z
Silvio Mayolo
CC BY-SA 4.0
2023-05-16T18:17:24.063Z
In Dyalog APL, there are only character arrays, but they *are* stored with their encoding, namely 1-, 2-, or 4-byte unsigned integer Unicode code points.
Adám
<h2>Resizing with Bounds Checking and Constant-Time Size Lookup</h2>
<p>The most common alternatives do not have all three.  A fixed-length array is not resizable.  And C-style arbitrary-length, null-terminated strings are the biggest bug attractor in history.  Anyone reading this far down on this site knows why.</p>
<p>It’s also faster on modern vector CPUs to copy or scan a memory range whose length is known in advance than to search for the terminating character.</p>
<h2>Short-String Optimization</h2>
<p>Most strings are short (less than 32 bytes or so), and <code>String</code> implementations almost always optimize them so they can be statically-allocated, and not use the heap.</p>
<p>Standard libraries want this optimization to be transparent, and they do <em>not</em> want this optimization for a generic <code>Vector</code>, which would meet the first three requirements.</p>
<h2>Variable-Width Encodings</h2>
<p>Several other answers have mentioned this, but the specific problem with making a <code>String</code> an array of <code>Char</code> is that arbitrary indexing of a variable-width encoding is both unsafe and useless.  It might point to a location that is a UTF-8 continuation byte or UTF-16 surrogate (or, previously, turn Shift-JIS into mojibake).</p>
<p>Changing arbitrary bytes could produce an invalid string.  Even the “wide-character strings” that were promised to be fixed-width in the ’90s have turned out not to be.  The C and C++ standards could say how wide strings “MUST” be a fixed-width encoding all they wanted.  It didn’t get Microsoft to break the Windows API.  They made their wide encoding variable-width anyway, and that was that.</p>
<p>Even changing one codepoint in the middle of a string to another valid codepoint could force the entire tail of the string to shift in memory, a linear-time operation that might even force the entire string to be reallocated and invalidate all current pointers to its data, so using array notation for replacing characters with other characters is actively misleading about the semantics.</p>
<p>It would be possible to use arrays of UCS-4, but that wastes too much memory when storing English, and often the user will want to iterate over graphemes instead of codepoints anyway.  Some implementations add “breadcrumbs” (the offsets of the characters at certain positions within the string) to speed up linear-time operations, which also requires a special type.</p>
<h2>Validation</h2>
<p>Related to the above, a blob of character data is not necessarily a valid Unicode string.  And in some languages, it’s common to use strings as blobs of arbitrary bytes of data.  But if a <code>String</code> class validates its input, the type system can guarantee that any <code>String</code> object always contains a valid string.  It could also perform normalization, if desired.  Some algorithms run faster on canonically-decomposed Unicode.</p>
<h2>Arrays are a Leaky Abstraction</h2>
<p>If you look at the history of the C++ <code>std::string</code>, you see it starting out trying to offer implementers some flexibility.  That didn’t work, because the API exposed a pointer to the bytes of a C-style, null-terminated array of <code>char</code>.</p>
<p>So, it became official: <code>std::string</code> <em>must</em> be implemented that way.  It could not, for example, be a rope, because it must be laid out in memory as a single contiguous array with random access.  It could not be a copy-on-write class, because writing through a pointer within the array doesn’t invalidate pointers to the data.  It could not save a byte of memory by removing the redundant terminating null when it already stored the length, because you had to be able to pass the address of the first <code>char</code> to a function that expected a C-style <code>char*</code>.</p>
<p>A class that did a better job of hiding its implementation details would have much more freedom to implement something stringy in a way that suits the application better.</p>

CC BY-SA 4.0
2023-07-10T18:10:52.030Z
2023-07-10T18:56:31.150Z
2023-07-10T18:56:31.150Z
Davislor
CC BY-SA 4.0
2024-12-15T00:03:29.713Z
@Davislor: Other than code that converts between UTF-8 and UTF-16, what kinds of function would need to care how many code points are in a blob of UTF-8 or UTF-16 text?  Being able to define grapheme clusters or direction switching points would seem much more useful, but code to do either of those things would be bigger than some entire ASCII-based text-rendering engines.
supercat
CC BY-SA 4.0
2024-12-15T05:00:08.477Z
@supercat A lot of traditional string algorithms break on variable-width encodings. A string is no longer an array of characters. For example, every function  like `ispunct` or `toupper` in the C and C++ standard libraries becomes useless outside the BMP, because all of them work on a single wide character. You can’t split a string in half to divide-and-conquer unless you check that you aren’t breaking apart a surrogate pair. Vectorizing string transformations with branchless SIMD instructions becomes difficult or impossible.
Davislor
CC BY-SA 4.0
2024-12-15T07:01:27.310Z
@Davislor: Why would one *care* about whether something that wasn't in any interesting subsets of characters was half of a surrogate pair, and for what purpose would one be attempting to perform a "divide and conquer" operation on a string, in a manner that couldn't be applied in context-agnostic fashion to a binary blob?
supercat
CC BY-SA 4.0
2024-12-15T16:47:50.683Z
@supercat How about: it is no longer possible to normalize a string in place.
Davislor
CC BY-SA 4.0
2024-12-15T19:36:06.180Z
@Davislor: Normalizing a string with a divide-and-conquer approach requires a normalizing routine that can recgonize a tail portion of a string as "pending".  I don't see how the scenario where the first part of a string ends with a combining accent is any easier than scenarios where it ends with a partial code point for something that might be a combining accent, or where it ends with a combining accent and a partial codepoint for the thing the accent combines with.  Actually, later versions of the Unicode standard completely broke divide-and-conquer processing by adding an "infix"...
supercat
CC BY-SA 4.0
2024-12-15T19:39:13.947Z
...zero-width joiner, meaning that a string might appear to be complete if chopped just before a zero-width jointer character.  I suppose a text measurement function could accept an argument to say whether more text would follow, and leave any portion of the text whose width could be affected by what follows be left as an "unprocessed tail", but unfortunately there's no standardized set of useful primitives that can be used by something like a text formatting engine to break up text into pieces that can be treated separately.
supercat
CC BY-SA 4.0
2023-08-02T05:28:47.757Z
`It didn’t get Microsoft to break the Windows API. They made their wide encoding variable-width anyway, and that was that.` I'm no MS fan but this is not the sequence of events as I recall it. It actually took MS an annoyingly long time to switch Windows from UCS-2 to UTF-16. It looks like you're suggesting they forced that change on everyone else. If I'm wrong please point me somewhere I can read up and learn.
hippietrail
CC BY-SA 4.0
2023-08-03T17:20:36.383Z
@hippietrail All I’m saying is that their switching `wchar_t` from UCS-2 to UTF-16 ignored the part of the Standard that said `wchar_t` must be a fixed-width encoding. The Windows API was defined back when UCS-2 was, and MS refused to make breaking changes, but this meant that `wchar_t`  became a variable-width encoding on Windows. This made all code that uses `wchar_t` non-portable, and not all compilers support `std::ctype<char32_t>`.
Davislor
<p>If Java had included a family of immutable-array and immutable-array-segment types, each with a constructor that could accept a single array (mutable or not) and construct an immutable array (copying the data if the original was mutable), a constructor that could accept an array of arrays (mutable or not) and produce a composite immutable array, and a factory method to produce an immutable array segment from an immutable array, then it would make sense to use immutable arrays and array segments of <code>char</code> to represent strings.  Having an immutable string type is extremely useful in a language that's based around a tracing garbage collector, and if there are no other immutable array types, making strings immutable would make it necessary for them to be distinct from other array types.</p>
<p>Note that in scenarios where an immutable array of <code>char</code>, <code>short</code>, or <code>int</code> would often not contain any values outside the range of a signed or unsigned byte, it may be worthwhile to store instances meeting that criterion using one byte per element.  Code using elements of such arrays would be more complicated, but the reduction in cache footprint would often overcome that disadvantage.</p>

CC BY-SA 4.0
2023-07-03T18:05:12.427Z
2023-07-03T18:05:12.427Z
supercat
<h1>Performance</h1>
<p>In Java, a char array can be expensive. If what we need is a pure ASCII string, then using a char array will waste most of the memory. We may want to use a byte array, but it's not the same type as char arrays.</p>
<h1>Encodings</h1>
<p>In Java, strings are stored in UTF-16 encoding, but it is explicitly known by all users. If we use a char array, it's less clear which encoding it is using.</p>

CC BY-SA 4.0
2023-05-16T18:01:32.780Z
2023-05-16T18:01:32.780Z
ice1000
<blockquote>
<p>In C, strings and character arrays are more or less the same. A C string is just a character array with a null terminator. Otherwise there is no internal distinction between them.</p>
</blockquote>
<p>The only distinction between a string and a character array is whether the data is handled as a scalar or not.</p>
<p>C basically handles all strings usually as variable-length arrays, and C has so little facility for working with VLAs that its treatment of strings (whilst very simple to grasp in principle) tends to be regarded as hard work compared to more modern languages.</p>
<blockquote>
<p>What are the advantages or disadvantages of strings being distinct types from character arrays?</p>
</blockquote>
<p>Alternatively put, you are asking <em>what is the advantage of arrays being treated as scalars?</em></p>
<p>The advantage of treating arrays as scalars is that you don't have to treat everything constantly as an array of bits! The bit being the only fundamental scalar in computing, with everything else being an array of bits.</p>
<p>The main problem with strings is that they tend to be variable length. Other bit arrays which are treated as scalars - like integers - were reducible to fixed-width forms at a relatively early stage of electronic computing, so that all <em>hardware</em> treats them as scalars even. Never so with strings, not even today.</p>
<p>At the time of C's design, and still today, there isn't a well worked-out scheme for how type systems cope with arrays in a fully general, systematic, and happy manner.</p>
<p>Even in APL, where arrays are king (and already were in 1972), the problem is solved largely by dispensing with types and a variety of other extravagances which make it completely unsuitable to the same purposes as C.</p>
<p>Nevertheless, most languages more modern than C, have implemented some kind of special treatment for string types and for operators which manipulate strings as if they are scalars, because there aren't any disadvantages to being able to treat strings as scalars, none whatsoever.</p>

CC BY-SA 4.0
2023-07-09T11:12:56.213Z
2023-07-10T15:47:36.150Z
2023-07-10T15:47:36.150Z
Steve
<h2>UTF-8 Packing</h2>
<p>So, what C calls a &quot;character&quot; isn't really a character in the typical Unicode sense. It is an 8-bit integer: a byte. The character 'c' takes one byte to represent, but the character 'ć' is not representable - it is too large to fit in C's <code>char</code> type. It is instead encoded as <em>multiple</em> bytes:</p>
<ul>
<li><code>&quot;c&quot; = [0b01100011, 0x0]</code></li>
<li><code>&quot;ć&quot; = [0b11000100, 0b10000111, 0b0]</code></li>
</ul>
<p>So a string in C is perhaps better thought of as an <strong>array of bytes</strong>.</p>
<p>In the original design of UTF-8, characters may be represented with any number of bytes: the number of 1s before the first <code>0</code> read in the first byte indicates the number of subsequent bytes to read. I believe subsequent iterations restricted the maximum size of a character to four bytes.</p>
<div class="s-table-container"><table class="s-table">
<thead>
<tr>
<th>First code point</th>
<th>Last code point</th>
<th>Byte 1</th>
<th>Byte 2</th>
<th>Byte 3</th>
<th>Byte 4</th>
</tr>
</thead>
<tbody>
<tr>
<td>U+0000</td>
<td>U+007F</td>
<td>0xxxxxxx</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>U+0080</td>
<td>U+07FF</td>
<td>110xxxxx</td>
<td>10xxxxxx</td>
<td></td>
<td></td>
</tr>
<tr>
<td>U+0800</td>
<td>U+FFFF</td>
<td>1110xxxx</td>
<td>10xxxxxx</td>
<td>10xxxxxx</td>
<td></td>
</tr>
<tr>
<td>U+10000</td>
<td>U+1FFFFF</td>
<td>11110xxx</td>
<td>10xxxxxx</td>
<td>10xxxxxx</td>
<td>10xxxxxx</td>
</tr>
</tbody>
</table></div>
<p>(chart adapted from <a href="https://en.wikipedia.org/wiki/UTF-8" rel="nofollow noreferrer">Wikipedia</a>)</p>
<p>This design allows for characters of different sizes to be placed contiguously in a file, and is conveniently backwards-compatible with ASCII (ASCII characters are 7-bit: all of the form <code>0xxxxxxx</code>). It does, however, mean that arbitrarily splitting a UTF-8 encoded sequence at a byte boundary can produce invalid UTF-8. This, and subsequently that individual bytes do not necessarily correspond to a printable notion of a character, is why a representation of (UTF-8 encoded) strings as an array of bytes is undesirable.</p>
<p>Naively implementing strings as an array of characters, however, has <strong>considerable overhead</strong>. Arrays require the types of their elements to have a <strong>known size</strong> - as UTF-8 characters can vary from one to four bytes, typical modern representations of <code>char</code> types must compromise on this and represent all Unicode characters with a consistent size of four bytes. This is incredibly inefficient for characters in the ASCII range - taking up four times as much space! Implementing strings as arrays of chars only compounds this inefficiency.</p>
<p>Further, while all Unicode characters can be represented by four bytes - not all sequences of four bytes are valid characters (in any encoding). An array-based implementation may allow direct access to the individual characters as bytes: which gets rid of any invariants you may like a string type to hold, like being valid UTF-8. A <em>distinct</em> <code>char</code> type that is not convertable to from bytes can uphold them, but is yet another design consideration to watch out for.</p>
<p>This is not without tradeoffs, of course - but they're mostly very much worth it. Modification of a char is no longer an O(1) operation, as it has to shift every subsequent byte to make room if the char takes more bytes to represent. Any sort of indexing is also necessarily O(n), even if you store the length of the array. All methods defined on arrays no longer carry over to strings - though many would say this is a benefit, despite some code duplication, as strings <em>are</em> semantically distinct from character arrays.</p>

CC BY-SA 4.0
2024-05-30T19:19:54.813Z
2024-05-31T03:19:05.420Z
2024-05-31T03:19:05.420Z
apropos
CC BY-SA 4.0
2024-05-30T19:33:19.160Z
Surely nobody is using arrays of 4-byte UTF-8, but UTF-32? There’s no packing involved there. The last two paragraphs I don’t understand: the answer to that point has wanted fixed-size characters to support indexing, but now individual bytes are indexed. Maybe there’s a transition missing.  The last paragraph on tradeoffs maybe is missing a negation somewhere, though I’m not sure where (and it’s not clear what the comparison point where insertion was O(1) is).
Michael Homer
CC BY-SA 4.0
2024-05-30T20:57:06.030Z
A char array in Rust (and any other language where chars are four bytes) is an array of 4-byte UTF-8 characters. "Insertation (or removal)" is wrong and should read "modification", thanks for the catch. The point in the last two paragraphs is that representation as a char array does support indexing - but the individual chars themselves can be incorrectly modified, which is undesirable.
apropos
CC BY-SA 4.0
2024-05-30T21:13:32.457Z
That is certainly not true; every single implementation I can think of with 4-byte chars uses UTF-32, not UTF-8, for all the obvious reasons that make processing much easier. I can’t imagine why anyone would use UTF-8 there, although maybe Rust does (but the [documentation implies](https://doc.rust-lang.org/std/primitive.char.html) but doesn’t outright state otherwise). I think the overarching answer here needs some structuring to set out when these considerations apply, sourcing ideally.
Michael Homer
CC BY-SA 4.0
2024-05-31T02:58:55.533Z
Ah, I see. I suppose "4-byte UTF-8 characters" is a misnomer to begin with since it's variable-length... I've updated my answer to be more pedantic on those fronts and elaborated on what C's actually doing. Hopefully that's clearer.
apropos
<p>While there may be some common features of strings and arrays, there are also some differences.</p>
<p>Rather than making strings a subtype of arrays, it would make more sense for them both to be a subtype of an abstract type (e.g. &quot;indexed sequence&quot;) that represents that commonality. This would allow you to use the same syntax and methods for accessing elements and slices.</p>
<p>But they can then each have unique methods for aspects that don't make sense for the other.</p>
<p>Forcing strings to be a type of array makes it difficult to enforce these disjoint aspects.</p>

CC BY-SA 4.0
2024-05-30T17:32:16.397Z
2024-05-30T17:32:16.397Z
Barmar
<p>Java strings are immutable, making possible to share them. But this is very bad for security, as data like passwords may stay in the memory for very long with no way to clear them. Due that, character arrays must be used for all sensitive data that are discarded (overwritten) asap after using them. <a href="https://stackoverflow.com/questions/8881291/why-is-char-preferred-over-string-for-passwords">See here</a> for more on this topic.</p>
<p>Immutable strings also mean that a new string is created each time you modify it. A heavily multi-threaded applications avoided touching the heap as much as possible because the new object allocation scales very badly with the number of threads: the memory pool is shared and all access there must be synchronized. I had applications where adding more threads after the first 20 or about actually brought the performance down, even if it was on the 64 core high performance computer.</p>

CC BY-SA 4.0
2023-07-10T08:20:19.790Z
2023-07-10T15:05:07.803Z
2023-07-10T15:05:07.803Z
Audrius Meškauskas
CC BY-SA 4.0
2023-07-10T14:07:06.657Z
"Due that, character arrays must be used for all sensitive data that are discarded asap after using them." Explicitly cleared, not necessarily "discarded" (i.e. abandoned to the garbage collector, which will only "deallocate" memory but not actually make it unreadable). On the other hand, this is only important in a context where an attacker can already read process memory. Also, last I checked the String type is still immutable (it would be a huge, breaking change), it's just that there are also other textual types that aren't.
Karl Knechtel
CC BY-SA 4.0
2023-07-10T16:17:24.907Z
It is commonly seen as a security flaw to use strings for passwords in Java. Please read the provided source for details.
Audrius Meškauskas
# What are the advantages of strings and character arrays being different?
Posted by: username_0 on 2023-05-16
In many languages such as Java, strings and character arrays are distinct types. In Java one must use `toCharArray()` in order to use array semantics on strings.

In C, strings and character arrays are more or less the same. A C string is just a character array with a null terminator. Otherwise there is no internal distinction between them.

What are the advantages or disadvantages of strings being distinct types from character arrays?
**Tags:** c strings arrays characters
---
## 12 answers
### Answer by username_1 on 2023-05-16. Score: 8 | Accepted answer
# Separate string methods or going via arrays

A disadvantage of a string type is that the methods available to manipulate arrays cannot just be used for strings. Instead, the language must either provide a separate set of functionality (which means more to document, learn, etc.) or force the programmer to split a string into an array whenever character manipulations are needed.

Array programming languages like APL, J, and K, make extensive use of array functions to manipulate strings, with great effect.
> **Comments:**
> **username_2** (2023-07-03): Ideally I'd like a string _not_ to share methods with an array. What is `"\U0001F1F8\u200d\U0001F1FA".map(println)` supposed to do? Query the OS' default font to figure it out at runtime?  
> **username_3** (2023-07-03): @Longinus I'd expect it to loop over Unicode codepoints. That's what [Scala does](https://scastie.scala-lang.org/ltLr73CJRxqE6lOQWeQu4Q).  
> **username_2** (2023-07-03): @user That would break any text editor GUI using that code.  
> **username_3** (2023-07-03): @Longinus I don't see why it would. A text editor specifically working with Unicode would use a separate method that's aware of the fact that the "array" passed into it is really a string in UTF-8 or whatever, to be displayed in a specific font.  
> **username_2** (2023-07-03): @user Unless you are implementing a compiler for a particular system, Unicode is the default, not a specificity the user should give particular care to enable. And the Standard defers the meaning of `"\U0001F1F8\u200d\U0001F1FA"` to the ISO 3166 Maintenance Agency which observes _geopolitics_. To ascribe some other API behaviour requires you to ignore ISO. Also why the Standard says a "character" is "what the user sees."  
> **username_3** (2023-07-03): @Longinus That is true, but I don't see what stops you from making a method that works on arrays of `Character`s with the same behavior as a method working on strings in a language with a separate string type  
> **username_2** (2023-07-03): @user We can, but when `map`ping over an `Array<T>` we know we deal with `T`s, whereas when `map`ping over a `String`, nothing tells us what we're actually doing - code units? Code points? Graphemes? Grapheme clusters? Glyphs? Composed emojis? Emoji components? ZWJ sequence components? All of which fit one of the **many** definitions of the word "character" - so I don't think we should. Also whether one option is more "obvious" than another depends on whether you're dealing with text (which is graphical) or memory (which is not).  
> **username_4** (2023-07-10): As a counter-example, C++ has string types that can be manipulated using the `<algorithm>` functions that work equally well on arrays or other containers.  (Yes, C++ strings are somewhat limited in that they tend to hold only a single codepoint at each position, but the language would allow a more sophisticated grapheme-cluster representation to be implemented and still satisfy the Container interface.)  
> **username_5** (2023-07-24): @Longinus I wrote an answer that gives some focus to this issue. Would appreciate any additional technical insights.  
---
### Answer by username_8 on 2023-05-16. Score: 20
# Different definitions of “character”

In Swift, the `Character` type isn’t a single Unicode codepoint (that’s `UnicodeScalar`), nor is it a single byte (that’s `CChar`). Instead, it represents an “extended grapheme cluster”, which includes combining diacritics. This makes some things more straightforward — `é` is one Unicode codepoint and `é` is two, but they’re both a single character. However, it means that characters aren’t a fixed size, and for `String` to just be `[Character]` would be a massive waste of memory. Don’t get me wrong, it is a `Collection`, just not a random-access collection the way an array is.
---
### Answer by username_5 on 2023-07-24. Score: 11
Encapsulation
Treating strings as character arrays entails a belief that a character array is an adequate model for a string - not just good enough now, but good enough that you won't want to be able to swap it out later while keeping existing interfaces.
In reality, there are many operations on strings that don't make a lot of sense on character arrays, and vice-versa. It's questionable, for example, whether strings should provide random access at all. If you index into a string, should this produce a "character" that is an instance of a separate type? What should one be able to do with that object? Different languages have different answers to this question.
Another problem is that characters are completely different from graphemes. Consider this Python example:
>>> 'fiancée'[::-1]
'eécnaif'
>>> 'fiancée'[::-1]
'éecnaif'

The first string uses a precomposed character to represent the grapheme é with a single code point:
>>> import unicodedata
>>> unicodedata.name('fiancée'[-2])
'LATIN SMALL LETTER E WITH ACUTE'

The second "composes" the symbol using two separate characters:
>>> unicodedata.name('fiancée'[-2])
'COMBINING ACUTE ACCENT'
>>> unicodedata.name('fiancée'[-3])
'LATIN SMALL LETTER E'

Similarly with flags:
>>> ''[::-1]
''
>>> ''[::-1]
''

Your terminal (like mine) might not support the composition of flag emojis from their components (regional indicator symbols), so you might see something more like a fancy AU turning into a fancy UA, or BG into GB - which goes a long way towards explaining the result.
There are much stranger phenomena than that:
>>> calendars = "️" + ""
>>> len(calendars)
3

Depending on your environment, the two calendars might look the same or different - for me, they are different in my browser, but the same in my terminal. The length is 3 because the first one is composed of two characters: the calendar emoji followed by a "variation selector". There isn't a "precomposed version" of that; the variation selector is something like a control character that tells the text engine to draw a different glyph (image) for the previous character, while still treating it as the same character.
It is left as a (hopefully simple) exercise to explain why calendars[::-1] will produce a result that looks the same as calendars, regardless of the environment.
Don't even get me started on the bidirectional text layout algorithm and associated control codes. You understood correctly: the people responsible for standardizing Unicode are currently on revision 46 of a document about 18 thousand words long just to explain how to decide when to write text left to right and when to write it right to left.
But, sure, reversing a string is a toy problem that also turns out not to be very important in realistic applications. What about the real meat and potatoes of string manipulation?

Did you know that some languages have a concept of uppercase and lowercase, but don't have separate uppercase and lowercase versions of every letter? For example, the German letter ß is considered lowercase; when a word containing it is set in uppercase, the ß is supposed to be replaced with SS (two uppercase Ses)? So it's not actually possible to uppercase words in a locale-aware way by simply iterating over characters and replacing them separately. The length of the string might need to change. I'm not even sure that each character can be considered independently, for every language.

Did you know that many widely-used written languages are only ever intended to appear in a "cursive" form (i.e. there are strokes that visually continue between letters), so there are corresponding variant characters with different appearances (assigned separate code points - not distinguished with variation selectors - perhaps because they had to solve the problem before anyone came up with the idea of variation selectors)? Did you know that you might be expected to substitute the glyphs from those variant characters to render a string, while still treating the string as being composed of whatever was actually put into it? And, if you're the poor sap tasked with implementing text rendering for an operating system, you might be expected to ensure that the characters have their variant appearances when selected, but copying and pasting them gives the original "general" characters?
>>> # Try highlighting, copying and pasting individual letters of the output.
>>> # (You might find it tricky because of the LTR rendering.)
>>> print('\u0628\u0628\u0628')
ببب


Did you know that the order in which people expect strings to be sorted, is not necessarily based on Unicode code point order at all? And that it might be different for different languages (that could validly have the same two strings in them)? And that it might not be possible to apply naive lexicographical ordering at all (i.e. considering the first character first, breaking ties with the second character etc.)? And that this will get completely broken by combining characters anyway? And that you can't actually necessarily determine the language of text just by examining some characters (chat can validly be an English verb or a French noun)? And that there's a whole other essay on Unicode.org about how to do it properly? (About 30 thousand words, by the way.)

Did you know that for the East Asian languages that use ideographs and/or logographs (Chinese, Japanese and occasionally Korean - and even Vietnamese, which I didn't know before writing this answer), Unicode provides "unified" characters - such that a symbol might be considered "the same character" in different languages, but is supposed to be displayed differently depending on what language is being written? And that you can't rely on variation selectors to tell you which glyph to use? (And, again, for short excerpts, you can't reliably determine the language by examining the characters - after all, it often takes only a single character to express an entire word/concept?) And that the process of doing this was an absolute ton of work, and the results are still contentious in some places? And that sometimes "traditional Chinese" characters and their "simplified Chinese" counterparts use the same code point (as in the table at the previous link), and sometimes they're separate?

And that the entire topic can be really politically sensitive?


Did you know that the text in a string could be in more than one language? That it could be in no languages? That it could try to express an idea, despite not consisting of words in a language? That it could try to express an idea that isn't linguistic? That it could try to decorate meaningful text with meaningless symbols?

Did you know about the hack used to cram the Unicode code space into 16-bit encodings? (Or, relatedly, the reason why the allowed number of distinct code points isn't a power of 2?)


Text is really, really complicated. Attempting to model it as merely an array of characters is hopelessly naive. While, yes, it is conceptually composed of a sequence of characters (in the only models of it we have that are viable to work with on computers), it makes next to no sense to expose a plain array interface to that sequence (whether or not mutable). Arrays have the conceptual affordance of algorithms that just don't work on text.

This stands completely independent of the question of whether a byte is an adequate model for a character. Yes, it's true that UTF-8 uses a varying number of bytes to encode each character, such that a string implementation relying on a storage buffer in UTF-8 encoding could not be random access. But this is an absolutely trifling issue compared to ones created by naive ideas about what a "character" actually is, and about how text is written, and about what manipulations make sense for that data.
Sure, we have a standard that precisely, unambiguously and permanently maps abstract numbers to "characters". We have a virtual "code space" for those numbers, a commitment not to ever reassign them even if it would make things look much more elegant, and a reasonable assurance that we won't overflow that space for a very, very long time, even as we strain to come up with bizarre pseudo-textual things to fill that space with.
And, sure, there are many plausible ways to represent "a character", depending on system requirements:

A byte whose values correspond to the first 256 Unicode code points (obviously restricted to text that only uses those characters)

A byte whose values are mapped to Unicode code points in an instance-specific manner, according to encoding metadata stored along with the array (restricted to text that only uses characters from one of such single-byte encodings - which can only exist for languages that fit in such a character limit, and don't deal well with mixed-language text)

An unsigned 16-bit value representing a Unicode code point from the Basic Multilingual Plane (only represents those characters - so no emoji, very limited ideographic support, many world languages missing, no private use area)

The same, but using surrogate pairs to represent characters outside the BMP

An 32-bit value, which might as well be signed I suppose

A 21-bit value, bit-packed three to each 64-bit machine word (now that 64-bit architectures are the norm)


And, sure, there might not be any need to use a linked list, or other data structure, to store the underlying character data. Simply writing each character's corresponding number sequentially in memory - i.e., using an array - is perfectly adequate for the simple task of remembering which characters are in the string, in what order.
But... none of these things actually address the fundamental complexity of text. While it's easy to laugh at the previous generation of Western programmers and their naive faith placed in "code pages", the "array" part of "array of characters" is also problematic - perhaps much more so.
Exposing such an interface to the raw storage raises many questions. Simply equating the string type with an array-of-characters type - leaving all that string functionality outside the type itself - causes innumerable problems. Please don't do it in your language.
> **Comments:**
> **username_6** (2023-08-25): +1000 Excellent answer addressing the (huge) gap from arrays to human language text. And you didn't even get to normalization. :-)  
> **username_5** (2023-08-25): Although I didn't explicitly mention normalization, I touch on it in a few places.  
---
### Answer by username_7 on 2023-07-03. Score: 10
An array must have a fixed stride, so that the memory address of the *i*th element equals the address of the start of the array plus *i* times the stride. If you really want an array of *characters*, so that each logical character (i.e. each Unicode symbol) occupies one slot in the array, then the array's stride would need to be 32 bits.

That is a waste of space compared to UTF-8 encoding, in which the most common characters take 8 or 16 bits each. On the other hand, if you encode strings as UTF-8 then it's no longer true that the *i*th character occurs at a predictable index, so what you have is not an array of characters. (Instead, it is an array of bytes, but that's the internal implementation ─ the API need not expose it to the user as such.)
> **Comments:**
> **username_8** (2023-07-03): _Technically_, you only need 21 bits. That doesn't exactly optimize well, though, as loading an odd number of bits is even worse than a regular unaligned load.  
> **username_2** (2023-07-03): A "Unicode symbol" can be more than 32bits. National flags are 64bits, and both combining characters and ZWJ sequences allow for "symbols" to be arbitrarily long.  
> **username_9** (2023-07-03): A Unicode *code point* must be between 0 and 0x10FFFF, hence "21 bits".  It's true that a user-perceived "character" can contain multiple code points, though.  
> **username_7** (2023-07-03): Arrays cannot have a stride of 21 bits; it would be rounded up to 32.  
> **username_7** (2023-07-03): @Longinus To my knowledge, the national flag symbols are not "Unicode symbols" ─ they are "individual symbols" because applications render them as such, but Unicode doesn't define them as far as I know. Really they are sequences of Unicode symbols which are interpreted as national flags by convention rather than by the Unicode standard.  
> **username_2** (2023-07-03): @kaya3-supportthestrike Unicode describes national flags as "components" of "emojis," and all emojis as "symbols." Then, TR51 explicitly mentions that there are concurrent ways to "count emojis" including delegating work to the system's font and the fact that this is a platform-specific problem.  
> **username_10** (2023-07-07): @kaya3-supportthestrike Would 24 be possible? Or do arrays of three-byte objects (e.g., a tuple of `i8`s or an `Option<i16>`) get rounded up to 32 for some sort of alignment reason?  
> **username_4** (2023-07-10): Why can't arrays have a stride of 21 bits?  That seems quite reasonable on hardware that has a 7-bit word size (obviously that's one that can't run C, but we haven't chosen our target yet).  
> **username_5** (2023-07-24): @TobySpeight but if we are considering impractical and/or theoretical machines, why do we still care about the Unicode standard?  
> **username_4** (2023-07-27): @Karl,  I'm merely referring to [5.2.4.2.1](https://web.archive.org/web/20181230041359if_/http%3a//www.open-std.org/jtc1/sc22/wg14/www/abq/c17_updated_proposed_fdis.pdf#subsection.5.2.4), which specifies that `CHAR_BIT` must be at least 8.  BTW, that doesn't mean that these older machines are necessarily _impractical_, but that that their implementation of C needs a little care.  
> **username_5** (2023-07-27): I understood what you meant by "can't run C" (as you clarify, need special considerations in order to run C). My point is that 21-bit strides make no sense on *ordinary* hardware *in common use* that follow contemporary *standards*; and if those standards and expectations are out the window, then I don't see why respecting the Unicode standard would still be important.  
> **username_7** (2023-07-27): This discussion is all moot; even if you had an exotic CPU architecture where a character array could have a stride of 21 bits, it would still be a waste of space compared to a string encoded in UTF-8 where most characters take only 8 or 16 bits. The conclusion of the answer still holds.  
---
### Answer by username_12 on 2023-07-10. Score: 8
## Resizing with Bounds Checking and Constant-Time Size Lookup

The most common alternatives do not have all three. A fixed-length array is not resizable. And C-style arbitrary-length, null-terminated strings are the biggest bug attractor in history. Anyone reading this far down on this site knows why.

It’s also faster on modern vector CPUs to copy or scan a memory range whose length is known in advance than to search for the terminating character.

## Short-String Optimization

Most strings are short (less than 32 bytes or so), and `String` implementations almost always optimize them so they can be statically-allocated, and not use the heap.

Standard libraries want this optimization to be transparent, and they do *not* want this optimization for a generic `Vector`, which would meet the first three requirements.

## Variable-Width Encodings

Several other answers have mentioned this, but the specific problem with making a `String` an array of `Char` is that arbitrary indexing of a variable-width encoding is both unsafe and useless. It might point to a location that is a UTF-8 continuation byte or UTF-16 surrogate (or, previously, turn Shift-JIS into mojibake).

Changing arbitrary bytes could produce an invalid string. Even the “wide-character strings” that were promised to be fixed-width in the ’90s have turned out not to be. The C and C++ standards could say how wide strings “MUST” be a fixed-width encoding all they wanted. It didn’t get Microsoft to break the Windows API. They made their wide encoding variable-width anyway, and that was that.

Even changing one codepoint in the middle of a string to another valid codepoint could force the entire tail of the string to shift in memory, a linear-time operation that might even force the entire string to be reallocated and invalidate all current pointers to its data, so using array notation for replacing characters with other characters is actively misleading about the semantics.

It would be possible to use arrays of UCS-4, but that wastes too much memory when storing English, and often the user will want to iterate over graphemes instead of codepoints anyway. Some implementations add “breadcrumbs” (the offsets of the characters at certain positions within the string) to speed up linear-time operations, which also requires a special type.

## Validation

Related to the above, a blob of character data is not necessarily a valid Unicode string. And in some languages, it’s common to use strings as blobs of arbitrary bytes of data. But if a `String` class validates its input, the type system can guarantee that any `String` object always contains a valid string. It could also perform normalization, if desired. Some algorithms run faster on canonically-decomposed Unicode.

## Arrays are a Leaky Abstraction

If you look at the history of the C++ `std::string`, you see it starting out trying to offer implementers some flexibility. That didn’t work, because the API exposed a pointer to the bytes of a C-style, null-terminated array of `char`.

So, it became official: `std::string` *must* be implemented that way. It could not, for example, be a rope, because it must be laid out in memory as a single contiguous array with random access. It could not be a copy-on-write class, because writing through a pointer within the array doesn’t invalidate pointers to the data. It could not save a byte of memory by removing the redundant terminating null when it already stored the length, because you had to be able to pass the address of the first `char` to a function that expected a C-style `char*`.

A class that did a better job of hiding its implementation details would have much more freedom to implement something stringy in a way that suits the application better.
> **Comments:**
> **username_14** (2023-08-02): `It didn’t get Microsoft to break the Windows API. They made their wide encoding variable-width anyway, and that was that.` I'm no MS fan but this is not the sequence of events as I recall it. It actually took MS an annoyingly long time to switch Windows from UCS-2 to UTF-16. It looks like you're suggesting they forced that change on everyone else. If I'm wrong please point me somewhere I can read up and learn.  
> **username_12** (2023-08-03): @hippietrail All I’m saying is that their switching `wchar_t` from UCS-2 to UTF-16 ignored the part of the Standard that said `wchar_t` must be a fixed-width encoding. The Windows API was defined back when UCS-2 was, and MS refused to make breaking changes, but this meant that `wchar_t`  became a variable-width encoding on Windows. This made all code that uses `wchar_t` non-portable, and not all compilers support `std::ctype<char32_t>`.  
> **username_13** (2024-12-15): @Davislor: Other than code that converts between UTF-8 and UTF-16, what kinds of function would need to care how many code points are in a blob of UTF-8 or UTF-16 text?  Being able to define grapheme clusters or direction switching points would seem much more useful, but code to do either of those things would be bigger than some entire ASCII-based text-rendering engines.  
> **username_12** (2024-12-15): @supercat A lot of traditional string algorithms break on variable-width encodings. A string is no longer an array of characters. For example, every function  like `ispunct` or `toupper` in the C and C++ standard libraries becomes useless outside the BMP, because all of them work on a single wide character. You can’t split a string in half to divide-and-conquer unless you check that you aren’t breaking apart a surrogate pair. Vectorizing string transformations with branchless SIMD instructions becomes difficult or impossible.  
> **username_13** (2024-12-15): @Davislor: Why would one *care* about whether something that wasn't in any interesting subsets of characters was half of a surrogate pair, and for what purpose would one be attempting to perform a "divide and conquer" operation on a string, in a manner that couldn't be applied in context-agnostic fashion to a binary blob?  
> **username_12** (2024-12-15): @supercat How about: it is no longer possible to normalize a string in place.  
> **username_13** (2024-12-15): @Davislor: Normalizing a string with a divide-and-conquer approach requires a normalizing routine that can recgonize a tail portion of a string as "pending".  I don't see how the scenario where the first part of a string ends with a combining accent is any easier than scenarios where it ends with a partial code point for something that might be a combining accent, or where it ends with a combining accent and a partial codepoint for the thing the accent combines with.  Actually, later versions of the Unicode standard completely broke divide-and-conquer processing by adding an "infix"...  
> **username_13** (2024-12-15): ...zero-width joiner, meaning that a string might appear to be complete if chopped just before a zero-width jointer character.  I suppose a text measurement function could accept an argument to say whether more text would follow, and leave any portion of the text whose width could be affected by what follows be left as an "unprocessed tail", but unfortunately there's no standardized set of useful primitives that can be used by something like a text formatting engine to break up text into pieces that can be treated separately.  
---
### Answer by username_11 on 2023-05-16. Score: 7
# Handling Encoding

Arrays of raw data don't, generally speaking, have an encoding. Strings can be in UTF-8 or any number of other formats. In Ruby, a `String` object is an array of bytes together with its `Encoding`. If you're handling data in different formats (such as data coming in from the network and then entering your main program's file system or database), it can be invaluable having the language keep track of this for you.
> **Comments:**
> **username_1** (2023-05-16): In Dyalog APL, there are only character arrays, but they *are* stored with their encoding, namely 1-, 2-, or 4-byte unsigned integer Unicode code points.  
---
### Answer by username_13 on 2023-07-03. Score: 2
If Java had included a family of immutable-array and immutable-array-segment types, each with a constructor that could accept a single array (mutable or not) and construct an immutable array (copying the data if the original was mutable), a constructor that could accept an array of arrays (mutable or not) and produce a composite immutable array, and a factory method to produce an immutable array segment from an immutable array, then it would make sense to use immutable arrays and array segments of `char` to represent strings. Having an immutable string type is extremely useful in a language that's based around a tracing garbage collector, and if there are no other immutable array types, making strings immutable would make it necessary for them to be distinct from other array types.

Note that in scenarios where an immutable array of `char`, `short`, or `int` would often not contain any values outside the range of a signed or unsigned byte, it may be worthwhile to store instances meeting that criterion using one byte per element. Code using elements of such arrays would be more complicated, but the reduction in cache footprint would often overcome that disadvantage.
---
### Answer by username_15 on 2023-05-16. Score: 2
# Performance

In Java, a char array can be expensive. If what we need is a pure ASCII string, then using a char array will waste most of the memory. We may want to use a byte array, but it's not the same type as char arrays.

# Encodings

In Java, strings are stored in UTF-16 encoding, but it is explicitly known by all users. If we use a char array, it's less clear which encoding it is using.
---
### Answer by username_16 on 2023-07-09. Score: 1
> In C, strings and character arrays are more or less the same. A C string is just a character array with a null terminator. Otherwise there is no internal distinction between them.

The only distinction between a string and a character array is whether the data is handled as a scalar or not.

C basically handles all strings usually as variable-length arrays, and C has so little facility for working with VLAs that its treatment of strings (whilst very simple to grasp in principle) tends to be regarded as hard work compared to more modern languages.

> What are the advantages or disadvantages of strings being distinct types from character arrays?

Alternatively put, you are asking *what is the advantage of arrays being treated as scalars?*

The advantage of treating arrays as scalars is that you don't have to treat everything constantly as an array of bits! The bit being the only fundamental scalar in computing, with everything else being an array of bits.

The main problem with strings is that they tend to be variable length. Other bit arrays which are treated as scalars - like integers - were reducible to fixed-width forms at a relatively early stage of electronic computing, so that all *hardware* treats them as scalars even. Never so with strings, not even today.

At the time of C's design, and still today, there isn't a well worked-out scheme for how type systems cope with arrays in a fully general, systematic, and happy manner.

Even in APL, where arrays are king (and already were in 1972), the problem is solved largely by dispensing with types and a variety of other extravagances which make it completely unsuitable to the same purposes as C.

Nevertheless, most languages more modern than C, have implemented some kind of special treatment for string types and for operators which manipulate strings as if they are scalars, because there aren't any disadvantages to being able to treat strings as scalars, none whatsoever.
---
### Answer by username_17 on 2024-05-30. Score: 1
## UTF-8 Packing

So, what C calls a "character" isn't really a character in the typical Unicode sense. It is an 8-bit integer: a byte. The character 'c' takes one byte to represent, but the character 'ć' is not representable - it is too large to fit in C's `char` type. It is instead encoded as *multiple* bytes:

* `"c" = [0b01100011, 0x0]`
* `"ć" = [0b11000100, 0b10000111, 0b0]`

So a string in C is perhaps better thought of as an **array of bytes**.

In the original design of UTF-8, characters may be represented with any number of bytes: the number of 1s before the first `0` read in the first byte indicates the number of subsequent bytes to read. I believe subsequent iterations restricted the maximum size of a character to four bytes.

| First code point | Last code point | Byte 1 | Byte 2 | Byte 3 | Byte 4 |
| --- | --- | --- | --- | --- | --- |
| U+0000 | U+007F | 0xxxxxxx |  |  |  |
| U+0080 | U+07FF | 110xxxxx | 10xxxxxx |  |  |
| U+0800 | U+FFFF | 1110xxxx | 10xxxxxx | 10xxxxxx |  |
| U+10000 | U+1FFFFF | 11110xxx | 10xxxxxx | 10xxxxxx | 10xxxxxx |

(chart adapted from [Wikipedia](https://en.wikipedia.org/wiki/UTF-8))

This design allows for characters of different sizes to be placed contiguously in a file, and is conveniently backwards-compatible with ASCII (ASCII characters are 7-bit: all of the form `0xxxxxxx`). It does, however, mean that arbitrarily splitting a UTF-8 encoded sequence at a byte boundary can produce invalid UTF-8. This, and subsequently that individual bytes do not necessarily correspond to a printable notion of a character, is why a representation of (UTF-8 encoded) strings as an array of bytes is undesirable.

Naively implementing strings as an array of characters, however, has **considerable overhead**. Arrays require the types of their elements to have a **known size** - as UTF-8 characters can vary from one to four bytes, typical modern representations of `char` types must compromise on this and represent all Unicode characters with a consistent size of four bytes. This is incredibly inefficient for characters in the ASCII range - taking up four times as much space! Implementing strings as arrays of chars only compounds this inefficiency.

Further, while all Unicode characters can be represented by four bytes - not all sequences of four bytes are valid characters (in any encoding). An array-based implementation may allow direct access to the individual characters as bytes: which gets rid of any invariants you may like a string type to hold, like being valid UTF-8. A *distinct* `char` type that is not convertable to from bytes can uphold them, but is yet another design consideration to watch out for.

This is not without tradeoffs, of course - but they're mostly very much worth it. Modification of a char is no longer an O(1) operation, as it has to shift every subsequent byte to make room if the char takes more bytes to represent. Any sort of indexing is also necessarily O(n), even if you store the length of the array. All methods defined on arrays no longer carry over to strings - though many would say this is a benefit, despite some code duplication, as strings *are* semantically distinct from character arrays.
> **Comments:**
> **username_18** (2024-05-30): Surely nobody is using arrays of 4-byte UTF-8, but UTF-32? There’s no packing involved there. The last two paragraphs I don’t understand: the answer to that point has wanted fixed-size characters to support indexing, but now individual bytes are indexed. Maybe there’s a transition missing.  The last paragraph on tradeoffs maybe is missing a negation somewhere, though I’m not sure where (and it’s not clear what the comparison point where insertion was O(1) is).  
> **username_17** (2024-05-30): A char array in Rust (and any other language where chars are four bytes) is an array of 4-byte UTF-8 characters. "Insertation (or removal)" is wrong and should read "modification", thanks for the catch. The point in the last two paragraphs is that representation as a char array does support indexing - but the individual chars themselves can be incorrectly modified, which is undesirable.  
> **username_18** (2024-05-30): That is certainly not true; every single implementation I can think of with 4-byte chars uses UTF-32, not UTF-8, for all the obvious reasons that make processing much easier. I can’t imagine why anyone would use UTF-8 there, although maybe Rust does (but the [documentation implies](https://doc.rust-lang.org/std/primitive.char.html) but doesn’t outright state otherwise). I think the overarching answer here needs some structuring to set out when these considerations apply, sourcing ideally.  
> **username_17** (2024-05-31): Ah, I see. I suppose "4-byte UTF-8 characters" is a misnomer to begin with since it's variable-length... I've updated my answer to be more pedantic on those fronts and elaborated on what C's actually doing. Hopefully that's clearer.  
---
### Answer by username_19 on 2024-05-30. Score: 0
While there may be some common features of strings and arrays, there are also some differences.

Rather than making strings a subtype of arrays, it would make more sense for them both to be a subtype of an abstract type (e.g. "indexed sequence") that represents that commonality. This would allow you to use the same syntax and methods for accessing elements and slices.

But they can then each have unique methods for aspects that don't make sense for the other.

Forcing strings to be a type of array makes it difficult to enforce these disjoint aspects.
---
### Answer by username_20 on 2023-07-10. Score: 0
Java strings are immutable, making possible to share them. But this is very bad for security, as data like passwords may stay in the memory for very long with no way to clear them. Due that, character arrays must be used for all sensitive data that are discarded (overwritten) asap after using them. [See here](https://stackoverflow.com/questions/8881291/why-is-char-preferred-over-string-for-passwords) for more on this topic.

Immutable strings also mean that a new string is created each time you modify it. A heavily multi-threaded applications avoided touching the heap as much as possible because the new object allocation scales very badly with the number of threads: the memory pool is shared and all access there must be synchronized. I had applications where adding more threads after the first 20 or about actually brought the performance down, even if it was on the 64 core high performance computer.
> **Comments:**
> **username_5** (2023-07-10): "Due that, character arrays must be used for all sensitive data that are discarded asap after using them." Explicitly cleared, not necessarily "discarded" (i.e. abandoned to the garbage collector, which will only "deallocate" memory but not actually make it unreadable). On the other hand, this is only important in a context where an attacker can already read process memory. Also, last I checked the String type is still immutable (it would be a huge, breaking change), it's just that there are also other textual types that aren't.  
> **username_20** (2023-07-10): It is commonly seen as a security flaw to use strings for passwords in Java. Please read the provided source for details.  
---
