<p>To make this question concrete and draw out depth and objectivity in answers, here are some specific points you may want to address:</p>
<ul>
<li>What <em>measurable, concrete</em> qualities are useful for capturing mental overhead / cognitive load in language feature learning and memorability?</li>
<li>Can it be quantified by any absolute methods? Or measured relatively against other language features?</li>
<li>What are the limitations of those approaches?- in particular, are there any limitations that are not very obvious?</li>
</ul>
<p>The purpose of this question is inform how to evaluate the usability of a language's design / features. I think evaluating a design's success in achieving its goals is important to the language design process, and to a degree, as far as I know, usability is a consideration in many languages' design.</p>

<usability>
Is the mental overhead / cognitive load of learning and remembering how a language's features work measurable? If so, how?
CC BY-SA 4.0
2023-05-21T20:30:50.243Z
This is probably more suited for https://cseducators.stackexchange.com/ ─ there are academic papers on this in CS education research, because it's relevant for teaching programming and choosing suitable languages for beginners. "Cognitive load" is a concept in psychology, and the methodologies for measuring it are the domain of psychology research, so https://psychology.stackexchange.com/ is another potential place to ask.
kaya3
CC BY-SA 4.0
2023-05-21T20:34:16.257Z
@kaya3 ok. I've never visited there, so it'd take me some time to personally learn about their on-topicality and search for possible existing duplicates before posting / moving this there. In the meantime, I'm going to leave this here. I'm curious what other people think of having such questions on this site.
starball
CC BY-SA 4.0
2023-05-21T21:21:02.037Z
While CSED will have relevant literature and experience and there's a lot more work in this space with novices, it's not exclusively a beginner's issue and there has been at least some study for experienced programmers also (probably more than I'm aware of, but I'm pretty sure still less). It's also fairly common for work with visual languages or novel programming interfaces to use NASA-TLX or a simpler systems as used in HCI - would they be in scope for this question?
Michael Homer
CC BY-SA 4.0
2023-05-21T21:25:28.020Z
@MichaelHomer I have never heard of NASA-TLX, but from a cursory glance at its wikipedia page, yes, it does sound in-scope for this question. as for visual languages, I didn't originally imagine for this question's answers to focus on particular types of languages, or specific individual languages, but if you think such an answer would answer the question as posed, and are willing to give such an answer some depth, that sounds fine to me.
starball
CC BY-SA 4.0
2023-05-23T05:19:52.850Z
related on meta: [Is it on-topic to ask about techniques for empirically evaluating a language's success at achieving a specific, commonly-sought design goal?](https://languagedesign.meta.stackexchange.com/q/227/251)
starball
<p><em>User experiments</em> with participants working in programming languages (attempt to) measure these things in a few different ways. A lot of them are proxies that seem reasonable, some objective and some with more subjective aspects. It's not firmly-established that these proxies are correctly measuring these things, but they are at least respected as appropriate attempts.</p>
<p>It is sometimes possible to compare meaningfully between languages, but quite fraught. Even with different variants of the <em>same</em> language it can be hard to disentangle where the difficult is from learning to use the test language and when it is from the test language itself, and many experiments don't try to do so, treating both as part of the same usability/effectiveness issue.</p>
<p>Most work on this sort of thing is in the education space and with learners of programming, and some of the results may be less applicable to current programmers, or even to experienced programmers learning a new language. In more HCI-adjacent parts of the field such as visual programming languages, other approaches in common use in HCI are also common.</p>
<p>Common metrics include:</p>
<ul>
<li><p><strong>Time</strong>: given a specific task, do participants complete it more quickly in a version of the language with feature X, or one without? The one that took longer is presumed to have a higher cognitive load.</p>
<p>The idea behind this is that taking longer means more thought must have been required to reason about the task, when the two conditions are otherwise of very similar scale. There are a number of confounds here, including whether the load is due to learning the new element or to the effects of the element itself.</p>
<p>An example of this metric in use is in a number of Stefan Hanenberg studies on static typing, including <em><a href="https://doi.org/10.1007/s10664-013-9289-1" rel="nofollow noreferrer">An empirical study on the impact of static typing on software maintainability</a></em>.</p>
</li>
<li><p><strong>Error rate</strong>: again performing the same task, are there more errors in one condition than the other? The one with more errors is considered to have a higher load.</p>
<p>An example of this metric in use is in the Quorum/Randomo study by Andreas Stefik and Susanna Siebert, <em><a href="https://doi.org/10.1145/2534973" rel="nofollow noreferrer">An Empirical Investigation into Programming Language Syntax</a></em>. This was attempting to demonstrate that the &quot;empirically-designed&quot; Quorum language was more learnable than a number of existing languages and one with randomly-chosen syntax, and is the origin of the famous &quot;Perl is less comprehensible than a randomly-generated programming language&quot; story.</p>
</li>
<li><p><strong>Scaled load questionnaires</strong> such as <a href="https://en.wikipedia.org/wiki/NASA-TLX" rel="nofollow noreferrer">NASA-TLX</a>: these are normed questionnaires that ask participants for <em>subjective</em> ratings on dimensions of mental, physical, temporal, performance, effort, and frustration, and then pairwise to rate each of these as more or less important than another; a standard formula then weights to obtain a final score.</p>
<p>While inherently subjective, it is very widely used, and in a within-subjects experiment where the same participant makes multiple attempts under different conditions a direct comparison can be made, and a more objective metric of which had higher perceived workload derived. As physical demand is mostly not relevant to programming languages, the remaining dimensions amount to mental overhead.</p>
</li>
<li><p><strong>Recall</strong>: after a first round of experimentation, a second round is organised with the same subjects after a period of delay (say a few weeks), and parallel prompts are given in both. This may be combined with other metrics, with the idea that performance will be more similar if the concept is retained.</p>
<p>This may not be suitably representative for systems that participants are not actively using or expecting to use again. It is a main approach for measuring <em>remembering</em> specifically as opposed to continual reinforcement.</p>
</li>
</ul>
<p>User experiments are resource-intensive and not always practical, and designing fair tasks/variants is difficult. Some language variants are so substantive that disentangling different effects is also hard. These do give fairly empirical measurements when they are executed, however, and have been used in language design in practice. Determining which facets of performance in these metrics are learning-remembering and which are in the thing that is being learned itself is sometimes not possible at all, but experiments with true novices are able to combine these together reasonably at least.</p>

CC BY-SA 4.0
2023-05-21T22:38:33.560Z
2023-05-21T22:43:33.910Z
2023-05-21T22:43:33.910Z
Michael Homer
# Is the mental overhead / cognitive load of learning and remembering how a language's features work measurable? If so, how?
Posted by: username_0 on 2023-05-21
To make this question concrete and draw out depth and objectivity in answers, here are some specific points you may want to address:

* What *measurable, concrete* qualities are useful for capturing mental overhead / cognitive load in language feature learning and memorability?
* Can it be quantified by any absolute methods? Or measured relatively against other language features?
* What are the limitations of those approaches?- in particular, are there any limitations that are not very obvious?

The purpose of this question is inform how to evaluate the usability of a language's design / features. I think evaluating a design's success in achieving its goals is important to the language design process, and to a degree, as far as I know, usability is a consideration in many languages' design.
**Tags:** usability
> **Comments:**
> **username_1**: This is probably more suited for https://cseducators.stackexchange.com/ ─ there are academic papers on this in CS education research, because it's relevant for teaching programming and choosing suitable languages for beginners. "Cognitive load" is a concept in psychology, and the methodologies for measuring it are the domain of psychology research, so https://psychology.stackexchange.com/ is another potential place to ask.  
> **username_0**: @kaya3 ok. I've never visited there, so it'd take me some time to personally learn about their on-topicality and search for possible existing duplicates before posting / moving this there. In the meantime, I'm going to leave this here. I'm curious what other people think of having such questions on this site.  
> **username_2**: While CSED will have relevant literature and experience and there's a lot more work in this space with novices, it's not exclusively a beginner's issue and there has been at least some study for experienced programmers also (probably more than I'm aware of, but I'm pretty sure still less). It's also fairly common for work with visual languages or novel programming interfaces to use NASA-TLX or a simpler systems as used in HCI - would they be in scope for this question?  
> **username_0**: @MichaelHomer I have never heard of NASA-TLX, but from a cursory glance at its wikipedia page, yes, it does sound in-scope for this question. as for visual languages, I didn't originally imagine for this question's answers to focus on particular types of languages, or specific individual languages, but if you think such an answer would answer the question as posed, and are willing to give such an answer some depth, that sounds fine to me.  
> **username_0**: related on meta: [Is it on-topic to ask about techniques for empirically evaluating a language's success at achieving a specific, commonly-sought design goal?](https://languagedesign.meta.stackexchange.com/q/227/251)  
---
## 1 answers
### Answer by username_2 on 2023-05-21. Score: 2
*User experiments* with participants working in programming languages (attempt to) measure these things in a few different ways. A lot of them are proxies that seem reasonable, some objective and some with more subjective aspects. It's not firmly-established that these proxies are correctly measuring these things, but they are at least respected as appropriate attempts.

It is sometimes possible to compare meaningfully between languages, but quite fraught. Even with different variants of the *same* language it can be hard to disentangle where the difficult is from learning to use the test language and when it is from the test language itself, and many experiments don't try to do so, treating both as part of the same usability/effectiveness issue.

Most work on this sort of thing is in the education space and with learners of programming, and some of the results may be less applicable to current programmers, or even to experienced programmers learning a new language. In more HCI-adjacent parts of the field such as visual programming languages, other approaches in common use in HCI are also common.

Common metrics include:

* **Time**: given a specific task, do participants complete it more quickly in a version of the language with feature X, or one without? The one that took longer is presumed to have a higher cognitive load.

  The idea behind this is that taking longer means more thought must have been required to reason about the task, when the two conditions are otherwise of very similar scale. There are a number of confounds here, including whether the load is due to learning the new element or to the effects of the element itself.

  An example of this metric in use is in a number of Stefan Hanenberg studies on static typing, including *[An empirical study on the impact of static typing on software maintainability](https://doi.org/10.1007/s10664-013-9289-1)*.
* **Error rate**: again performing the same task, are there more errors in one condition than the other? The one with more errors is considered to have a higher load.

  An example of this metric in use is in the Quorum/Randomo study by Andreas Stefik and Susanna Siebert, *[An Empirical Investigation into Programming Language Syntax](https://doi.org/10.1145/2534973)*. This was attempting to demonstrate that the "empirically-designed" Quorum language was more learnable than a number of existing languages and one with randomly-chosen syntax, and is the origin of the famous "Perl is less comprehensible than a randomly-generated programming language" story.
* **Scaled load questionnaires** such as [NASA-TLX](https://en.wikipedia.org/wiki/NASA-TLX): these are normed questionnaires that ask participants for *subjective* ratings on dimensions of mental, physical, temporal, performance, effort, and frustration, and then pairwise to rate each of these as more or less important than another; a standard formula then weights to obtain a final score.

  While inherently subjective, it is very widely used, and in a within-subjects experiment where the same participant makes multiple attempts under different conditions a direct comparison can be made, and a more objective metric of which had higher perceived workload derived. As physical demand is mostly not relevant to programming languages, the remaining dimensions amount to mental overhead.
* **Recall**: after a first round of experimentation, a second round is organised with the same subjects after a period of delay (say a few weeks), and parallel prompts are given in both. This may be combined with other metrics, with the idea that performance will be more similar if the concept is retained.

  This may not be suitably representative for systems that participants are not actively using or expecting to use again. It is a main approach for measuring *remembering* specifically as opposed to continual reinforcement.

User experiments are resource-intensive and not always practical, and designing fair tasks/variants is difficult. Some language variants are so substantive that disentangling different effects is also hard. These do give fairly empirical measurements when they are executed, however, and have been used in language design in practice. Determining which facets of performance in these metrics are learning-remembering and which are in the thing that is being learned itself is sometimes not possible at all, but experiments with true novices are able to combine these together reasonably at least.
---
