text:
cross-national and cross-cultural academic collaboration
Are there online avenue(s) for would-be cross-national and cross-cultural academic researcher to meet? There are times where one would want to broaden the scope of one's research.

Could you add some more detail? As it stands, the question is quite unclear: what do you mean by "meet online"? To do what? Who is a "cross-national and cross-cultural researcher"? Or do you mean "for researchers of different nations and cultures to meet"?

What I meant was that is there an online platform whereby academic researchers from different nationals or culture could meet to co-author a paper they both have similar interest on. I might be writing on private tutoring in Nigeria and I would want to  have another quantitative view on the said subject from another researcher in Saudi Arabia. If I do not have the opportunity to attend  international conference where I could interact with such researcher, is there any online platform available for cross-national researchers to interact on the same theme of research?

What does culture have to do with all this? 2. Can't you simply send them an e-mail, or Skype with them, if you already know the person you want to work with?

I don't know if there is such a platform.  It would be a good idea, but a lot of work to create one.
If one does not exist, I suggest you

make a website where you post about your own work, and comments about others' work.  My partner is a physicist and made a primitive website. Believe it or not, strangers find this website and make contact to initiate collaborations.
look on Google Scholar for articles that interest you.  Write to the authors of the articles that really intrigue you.  I have done that.  Rarely, there's no answer.  Sometimes, there's a one-line email and an attached article that is exactly what I was hoping to find.  Sometimes, there is an invitation to collaborate on a specific project!


text:
What tradeoffs are there between internal and external validity?
I was reading through an article about research design. The authors gave an example on the relation between internal and external validity which I thought might be an important area of consideration when designing research proposals.

"A restricted study population (and exclusion criteria) may limit bias
  and increase the internal validity of the study; however, this
  approach will limit external validity of the study and, thus, the
  generalizability of the findings to the practical clinical setting.
  Conversely, a broadly defined study population and inclusion criteria
  may be representative of practical clinical practice but may increase
  bias and reduce the internal validity of the study."

Broadly speaking, what questions can be asked to help decide one should strike a balance between the internal and external validity in our research proposal designs? Is it necessary then to carry out a pilot study first to establish some level of internal validity before moving on to establish external validity in a much bigger and formal follow-up study?
Crosspost: Quora

Good question (+1). I think that you would have better chances of getting good answers to such focused research methodology question on Cross Validated (you might have some luck on Quora as well, so crossposting was a good idea).

For anyone tracking, it ended up here

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
Should examination questions be ordered in the same way as they are presented?
I'm writing my midterm and am currently coming up with questions that relate to important course concepts that I want to test. This is being done in a somewhat ad-lib manner, and my questions are out of order in terms of how the concepts were presented in class.   Is there a compelling argument to ensure that the order is the same as that in class?

I can't imagine a single reason.

As long as the course concepts are independent, no.

Mixing the order may be more effective in testing whether the students can decide for themselves which concepts to use in answering each question. Is that a bug or a feature?

If the questions are not dependent on one another, then in many cases it is better to mix-up the order. Student responses will be less likely do to basic memorization of their notes/material and more likely to reflect higher-level learning.

The only consideration in ordering (beyond randomly) would be where some questions are intended to be harder than others. As students normally answer front-to-back, it may be appropriate to put the harder questions later such that students don't get "put off" by the hard questions early on.

I deliberately order my exam questions so that they do not increase in difficulty--and I tell students that--because I want students to read all the questions before deciding for themselves which they find easier. (But I do try to put the easiest question first.)

It depends on the style of exam as well. While choosing questions is customary in my university I feel that it is invalid if the exam is intended as a random sample. In a choose your question exam all questions should be equal in intended difficulty but sub questions can be of different levels.

In a choose your question exam all questions should be equal in intended difficulty — I strongly disagree.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Planning a course: lecture-by-lecture or topic-by-topic?
I'm planning my first course, and I'm starting to realize that it's actually broken into a long string of smaller topics, and that each smaller topic is unlikely to fit into a single lecture.  Teaching each sub-topic as a lecture would either make my lectures run short or too long.  I'm considering planning out my lectures in a continuous way: rather than write up a plan for lecture 1, 2, and so on, I'm thinking of just writing out all of what I want to say for each topic, all of the examples and so forth.  Then, I just start at the beginning and keep going, ending each lecture at an appropriate place in the notes.
Is that a bad idea?  Are there any best practices I should follow?

Best practices: Be careful not to get too attached to your planned schedule, lest you end up going too fast and leaving your students behind in order to "cover" everything you had planned. See the joke at the end of this answer.

Yeah, you definitely want more content than you need.  Classes have a pesky way of not going the way the teacher expects.  You might go too fast and you might go too slow.  Also, give yourself a few semesters to get it right.

There is an "easy" way to make sure lectures are neither too short nor too long. If you finish a topic early, let everyone go early. If the class period ends before you finish the day's topic, refer everyone to the book/notes, and start the next topic next time. Plan accordingly.

Planning it in a continuous way and not lecture by lecture gives you more flexibility in the classroom as questions or other issues might interfere with your planning and therefore your planning doesn't work out anymore. On the other case, there is a chance that you won't plan enough for a lecture, so a continuous plan enables you to "fill" a hole, in case you'd advance faster than you have planned.
A lecture by lecture plan may be helpful as well, maybe not on a very detailed level. But writing down which topics you'd like to cover within which lectures helps in setting up an underlying structure of the course, showing the coherence of the smaller topics.
Going further, I'd like to add, that if you plan to write down what you are going to say, you might ask yourself the question if it is helpful for your teaching or the students learning if you hand out some form of lecture notes at the beginning of the course. I, as a student, enjoy having to take less notes and being able to focus more on what the teacher is saying. As a teacher, with lecture notes, I know for myself, that my lesson is carefully planned and that I have already done a good amount of work for next year. If you are concerned, that with lecture notes students attention may decline, I suggest that you leave out certain type of content, in mathematics e.g. examples.

"Planning it in a continuous way and not lecture by lecture gives you more flexibility" because "questions or other issues might interfere with your planning and therefore your planning doesn't work out anymore." Huh? The conjecture of your opening sentence argues against itself.

@8protons My planning rarely works out, having it planned in a continuous way, leaves me with less / smaller problems.

I would recommend a very firm day-to-day schedule in advance, and (as in software) a "top-down" approach to developing it. E.g.:
Will you have a final exam? Then the skills desired on the final should be drafted (or obtained from the department) first. Will you have midterm exams and assignments? Then these are your "tentpoles"; ideally they should be scheduled before the semester starts, and the necessary skills must be covered before the exam dates are reached. Assignments need the skill presentations complete beforehand; and also time to submit, grade, and give feedback before associated tests. If you give weekly or daily quizzes, then the required skill scheduling gets even more granular.
Establish coherent topical "chunks" of your semester (a few weeks between exams or assignments), and then make a schedule filling in between them day-by-day. If you find that there's an extra or missing part of a day in a block, then commit to cutting or expanding the subject within that topical block. (Sometimes an extra 30 minute span provides the best opportunity for an extra related topic that tickles your fancy and no one else covers.)
It's far easier to squeeze or expand lectures by 5 minutes each day than it is to get to the last week of the semester and realize that you haven't gotten to 5 chapters of the book. Krantz in How to Teach Mathematics recommends having lectures that can be adjusted by a few minutes each day (I do this with a list of in-class exercises).
As most of us know, time-management is among the top priorities working in academia, and this is a reflection of that. My guiding principle is:   schedule uber alles; after a few semesters of teaching a class, I can usually dictate from memory what's happening in every 30-minute time period all semester long.
I haven't taken education courses, but the "top-down" design method works for me, and I've tangentially heard that this is what's recommended in courses on pedagogy.

What does schedule uber alles mean?

@ILiketoCode: "Schedule over everything" [as a priority]

There is one very big drawback with the continuous approach as you've described it: the students get no direct positive feedback from attending a lecture. That means that the effort of turning up to a lecture goes unrewarded for a long time, which may seriously hurt the attendance and motivation.
I like to spend at least a little time considering for each lecture how the students will feel afterward. I remember best the lectures where I really felt like I had learned something. A kind of feeling of exhaustion and exhilaration at the same time. The sensation that you now know more cool stuff. I feel if I can achieve that feeling in my students I can keep them coming back.
Moreover, people have a natural attention span of about 20 minutes. It's best to plan your material in chunks of 20 minutes, and plan for a small distraction in between those chunks (some interactivity, or a light video).
Finally, it really helps students to give a layout of the lecture beforehand, so they can plan their attention levels. If they have no idea what's coming, how long you're going to be talking about X, and why it's important to know before you move on to Y, they'll lose focus. It's very hard to maintain your focus when you don't know what you're supposed to be focusing on.
If you don't give some thought to where you want to end up after each 20 minute chunk in your lectures, there no way to achieve this, and most likely you will just drone on for an hour, eat in to the break, and end up losing student's attention, and then students.
Of course, the flip-side of a rigorously planned lecture series, as others have pointed out, is that you lose the flexibility to adjust to how well the students are absorbing the material. Here's how I would start:

Make a list of learning goals: what should the students absolutely learn, what would be nice for them to learn.
Plan your course requirements (exam, report, practicals, everything that adds to the grade) to reflect these learning goals and their relative priorities.
Create a preliminary series of topics on the basis of the learning goals. Make sure to focus on the information required to achieve the requirements. Chunk each lecture into 4 20-minute sub-topics.
Make sure to check at every opportunity whether the students are following along, or falling behind. Ask plenty of questions during the lectures (for instance, between these 20 minute chunks), and try to talk to students one-on-one whenever possible.
If you find out that people are falling behind, re-adjust between lectures. Adjust the planning, scratch some of the minor learning goals. But change the plan you had, rather than not having a plan to begin with.

We have focused on very different parts of his notion of "continuous". I assumed that, by "writing out all of what I want to say", he doesn't imply saying exactly what he wrote. My notion of continuous planning includes flexibility in wording as well. And I really like your list!

Thanks! I didn't imagine the OP would fully write out a series of lectures in advance (that would be pretty heroic). I interpreted the "continuous plan" as dealing with each topic one after the other in one stream, and just stopping  whenever the time runs out in picking back up the next lecture. Kind of one big lecture in 10 two-hour chunks. I may have misinterpreted though.

No, you didn't.  I've done a lot of tutorials, not many lectures, but I tend to have points that I elaborate on.  Usually I have worked examples set up completely, but other than that it's point form.  I'm not writing a speech for every lecture :-)


text:
Bad experiences at conferences, worth going anyway?
I went to a conference I my field for the first time in the last month. I have severe social anxiety but can usually hide it. I was there for learning and didn’t have a paper.
I expected some difficulties but there were more than I thought. I had my anxiety flare up when asking a question in a crowded Q&A and froze up, tried to finish anyway, probably ended up looking like I was clueless and trying to sound smart. The audience looked more concerned than disapproving, but I'm pretty sure that they would remember me if we met again, and that would reflect badly on me.
Many people at poster sessions didn't seem interested in discussing or answering questions, some acted like they did their presentation out of politeness only. I usually left and looked for something else when I realised that, but I wonder what made them react that way.
Overall, I'm questioning whether it's worth going to more conferences without having a paper to present. I did have interesting conversations with some people, some of them even suggested collaboration, but the experience as someone at non-PhD student level was overall just stressful. I didn't feel like I belonged there.
From a career point of view, would it be advisable to wait until I actually have a reason to go? Should I just seek advice for social anxiety and go anyway? I'm interested in networking and collaboration but this didn't seem like the right approach.

"Overall, I'm questioning whether it's worth going to more conferences without having a paper to present." Maybe, maybe not. What's certain is that letting your social anxiety make decisions for you is going to impact your career negatively in the long term.

"I'm pretty sure that they would remember me if we met again, and that would reflect badly on me". I really, really doubt anyone would recognize you, most probably won't even remember the incident. I also really doubt the people in poster sessions who were disinterested, would've been interested talking to any other grad student (besides people they know). You're overestimating how much others scrutinize you; I know it's hard (from someone else with social anxiety), but these awkward moments in regular interactions with randoms, are simply forgotten by everyone else.

50% of them will literally have forgotten your face the moment they stopped looking in your direction, the other 50% the next day. For good and bad, people don't care about random strangers a lot. You'd have to dance naked on the table and they probably still wouldn't recognize you a year later. Don't worry about such things, I cannot remember a single person ever in my life who did something I considered silly, ridiculous or the like (unless I knew then before).

I don't know if you are familiar with the "illusion of transparency", it seems to me that it could explain (and help you deal with) your experience at the first conference https://en.wikipedia.org/wiki/Illusion_of_transparency#Public_speaking_and_stage_fright

If you "lock up" in public situations you should probably talk to a mental health professional or counsellor about it. It once helped me greatly.
And, one of the best ways to overcome such feelings of dread and such is to intentionally put yourself in such situations and practice being more open and self-confident. Extreme introversion and even being "on the spectrum" can be a serious detriment to a career. Much worse than feelings of "looking bad" in a particular situation.
But it is safer to practice in smaller, less intense, situations until you begin to get the confidence to speak publicly in general.
One person I know, who is a superstar in CS, overcame such things by joining an acting group and learning to "play a part" and separate his own internal aversion from the need to speak. He is considered a great conference speaker.
So, yes, by all means, go to such conferences and speak with people. Small groups, perhaps, at first, working up to bigger things.
You might also consider writing a few notes on a note card before you speak up at a large gathering, noting the main points you wish to make so that you have a "crutch" if you start to freeze up.
But, talk to someone who is trained to give personal advice about such things. Many universities will have an office that can provide help.

Yes, "learning to play a part/role"...

I like to think of the secret as becoming comfortable with being uncomfortable and making public mistakes. @OP, it will take time and patience, but if you dedicate yourself to following this path, I think you'll be surprised at how much progress you've made one day.

I don't think freezing up like that will reflect badly on you.
It sounds like some people were interested in talking to you and some were not. I would say that is fine and normal - don't waste time wondering about the people who were not interested in talking, and perhaps they don't have good social skills. Some people even suggesting collaboration is a positively good result.
It would be natural to feel slightly out of place if you are a non-PhD student. But don't worry about it. Plenty of people feel out of place at conferences, especially conferences where lots of people all seem to have known each other for years.
On the other hand, perhaps it would be advisable to wait till you actually have a reason. Are you planning to apply for a PhD course? If so, your advisor will give you advice on when and where to go.
Seeking advice for social anxiety sounds like a good idea. If you only have it mildly, no harm will have been done.

After 30 years of going to several conferences per year, the handful of askers I remember were all people I knew previously. People I have shared afternoons with don't seem to remember me years later. No one will remember you for tripping while asking a question.
The benefits of going to conferences are to get awareness of who is who, and what people are working on. Eventually (so, not necessarily after a single conference) people will start to know you. As a graduate student, you will hang out with other graduate students. Twenty years from now those graduate students will be leading committees that decide on positions and grants. And they will be more willing to share research idea with "a pal from the graduate student days" than with someone they have just met.
Eventually you will have to show your value research-wise if you really want to belong to the community, but the steps from the previous paragraph can start way before.

It is perfectly normal to be anxious in a place with lots of strangers.
Here's what helps: understand that they are all there because of a common interest, in this case, a scientific field. You are all in for a particular common cause.
Secondly, one tip that works surprisingly well: identify people that look like they are also lonely and strike up a conversation, about the field, their work etc. Not only will this warm you up and give you people to talk to, you also will have done your good deed of the day. Plus, in a way, you become to some degree "the host" to those people. Sometimes you may be able to introduce such people to each other.
As for the freezing - many people have been in challenging social situations, they are probably more compassionate than judgemental. Be more compassionate to yourself, and cut yourself slack.
The idea mentioned elsewhere comes to the fore: when speaking to a large audience (whether as presenter, or as question-asker), always prepare. It can be just a sheet of notes or a rehearsed talk, but prepare. If necessary, rehearse the question, in your room, a bathroom stall (silently) or elsewhere. This will give you more confidence in asking it.
Finally: remember why you are there - you are there to see the latest trends, to see what is going on, to listen to people. If you are to anxious to speak, listening is always possible, and is a great way to learn.

Can you go to a conference with a more senior person like a post-doc, your advisor, or some professor?
They might show you around, introduce you to some people, and most important, you can just walk along with him/her. Whenever they speak to someone, discuss a poster, or are approached for their last paper, you stand next to him/her. The advantages are, that nobody expects anything from you, you can speak up if you want, the senior person can involve you in the conversion ("my PhD student here is actually working on a similar problem"), and can help you out, if you are blocked.
This needs some trust to the senior person and some willingness to help you out ("babysit" might be a negative term from someone not willing to do so).

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Switching between postdoc fellowships
I am non-EU citizen working as postdoc at a university in Denmark. Not so long ago I won two postdoc fellowships, of which I accepted one, and that's where I am currently getting my salary from. I can still accept the other (at least try), but I am struggling to understand whether it is a good idea.
Arguments in favor of accepting:

my PI tells me it may be possible to use this new fellowship to promote me to Assistant Professor
new fellowship means a longer period with guaranteed income and residence rights
the project for which I won that fellowship is more interesting (though also more challenging) than what I am currently working on

Arguments against:

I don't know if switching to a different fellowship and leaving a research project unfinished won't be a spot on my CV; it's unlikely I will be able to finish the "old" project while working on the "new" one, and also not possible to finish the "old" project before it's time to switch
I don't know what will happen if my PI's smart plan to promote me fails to work for some reason; it won't be nice if it turns out I cannot be promoted but also can't continue working as a postdoc because I have used up my allowed number of prolongations of my postdoc position

Would be interested in hearing any feedback.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Publishing an incomplete idea
I am working on an extension to my thesis work in robotics.  I believe that, in order to complete the result I'm moving towards, I'll need at least two, probably more, papers to provide a full solution.  The problem itself can be broken into several pieces that clearly build off of each other.  I'd like to publish each piece separately in order to keep the size of the paper reasonable.  Also, the first piece is actually of limited technical use but introduces the concepts, notations, and methods required for the further pieces.  Has anyone had any success trying to publish work that stands alone but is very obviously incomplete?

Will each paper clearly present a method, results, and discussion that tells a complete story?  If so, I think it is definitely reasonable to split your thesis into multiple papers.  It will also depend on the journal you are submitting to as some are more open to introducing concepts rather than analyzing hard data.

Maybe this varies by field, but I'm not sure why you are so concerned with keeping the size of the paper "reasonable".  If you need more space to present your work, and it can't be broken into pieces that stand alone, well, you write a longer paper.  What's the point of having 2-3 shorter papers if a reader has to read all of them anyway in order to get value?

Primarily, papers in robotics are often limited to under 12 pages, with huge page charges being imposed around the 8 page mark.  I feel that to explain the whole piece of work, probably around 20 pages would be necessary.

Also, the first piece is actually of limited technical use but introduces the concepts, notations, and methods required for the further pieces.

The main question is whether this work is of independent interest or use to anybody but you.  All research is incomplete in some sense, and that doesn't bother people (everyone likes opportunities for further work).  So the issue is whether your paper is incomplete in the sense of not accomplishing anything interesting yet, or incomplete in the sense that there is still much more to do.
If there are people who want to read it now, on its own, then that's an argument for publishing it now.  If it has little value except as preparation for the next paper, and that paper is already written, then it might be OK, but getting it accepted on its own may be tough.  If it has little value except as preparation for a future paper that does not yet exist and might not pan out, then it will be really difficult to publish it.

That makes sense.  I would say that the first piece is of independent interest since it solves a problem in the field.  However, the problem that it solves is a limiting case that is rarely considered in practice.  That being said, this limiting case forms the basis for the more complicated cases that are considered, so perhaps it is relevant...

Sounds like a scientific contribution. But be careful not to under-sell your case. In the paper, be as precise as you can in describing how the limited case will be helpful when dealing with more complicated cases.


text:
Course gamification: rewards that aren't bonus marks?
I am experimenting with a game mechanism in my course in which students can progress through 4 core competencies, A, B, C, and D.  Each competency has 3 levels, 1, 2, and 3.  The game component is optional.  The idea is that each level of each competency will be linked to a few homework activities that, once completed, achieves the level.  When all levels are achieved for a given competency, the competency itself is achieved.
In my department, homework assignments are not graded (projects and so forth are, but I can't assign a few questions for grading).  As a result, I want to ensure that my students are actually going to do the homework I've assigned.  I can give bonus marks.
I'd like to give rewards for completing these competencies, but I don't want the course so overloaded with bonus marks that students can too easily get 100% without doing well on the final/midterm/projects.  So far, the ideas I have are:

Complete level 1 for any competency - earn 1 grace day for handing in an assignment late without penalty
Complete level 1 for all competencies - earn 5 grace days, in addition to the 1 you gain for doing the above
Complete all levels for any competency - bonus quizzes now worth 7% (originally they were worth 5%)
Complete all competencies - 2% course bonus

I'm not a big fan of all of this bonus, but other than bonus marks and late days I'm not sure what other options I have.  Any ideas?

You should probably check whether distinguishing students on the basis of bonuses is permitted by your institute.

Yes, bonus marks are allowed.

One idea is to include some problems/tasks in the final exam related to those competencies. This might not be possible.
A student who achieved the highest level in that competency automatically gets full grade on that question, and doesn't have to solve it (thus getting extra time for the rest of the exam).
For levels 1 and 2 you can award free part marks, and the student gets either the free marks or his grade on the problem (whichever which is larger).
Added
Depending on how long/short it needs to be, something like below might work (or could be terrible):
Exam:
Part A (short questions testing the competency parts)
Question 1 (competency A): bla bla bla
Question 2 (competency B): bla bla bla
Question 3 (competency C): bla bla bla
Question 4 (competency D): bla bla bla
Part B Actual exam, Questions 5- ...

I was thinking that.  What might it look like?  For instance, I should identify which competency relates to which exam question, I suppose?

@MichaelStachowsky See the addon. A student solving all competencies would only need to solve part B, while one not doing anything would need to solve both parts.

If the school has said that homework can not be graded then it is in reality optional. lets say the 4 competencies areas take 20 mins on the exam. You are giving an extra 20 mins on the exam for people who did the optional homework. You should really look into this before trying this one. I can almost guarantee a student will complain about this when they have to do those questions and others do not, and the administration may have some rule that everyone has to get the same exam.

I'm not sure it is really gamified if they have to wait for the reward.

My answer is based on the following points:

In theory, if a student has achieved a competency at a certain skill they should be able to repeatedly demonstrate said skill.
Students at the same proficiency levels should received the same grade.
Your gamification approach exists to help them achieve mastery in competencies A, B, C and D. It is not the only way of doing so and it shouldn't matter how they get there, as long as they do get there.
Your extra activities are well thought and helpful for students to achieve mastery in competencies A, B, C and D.
Students can see value in achieving the course's competencies.

Given these assumptions, an alternative idea is to not give bonus marks at all, but instead make the exam (or other high stakes assessment activities) an opportunity for students to demonstrate their mastery of competencies A, B, C and D. The questions posed should relate to all the optional activities and you can even make some a little more complicated to gauge proficiency in higher skills levels.
For those who have joined the gamified part of the course the exam should look like  just another activity they have already completed. We expect their performance to match their level of skill obtained in the gamified part and students should be able to self-assess their performance with reasonable precision.
For those who do not join the gamified part of the course the exam might be their first opportunity to demonstrate the competencies. They may or may not do well, as they might not have prepared themselves for the exam. It is their risk in doing so, as joining the gamified part would have made the exam much easier (or at least much more predictable).

Note that points 3 and 4 are essential for this to work. If achieving the course's competencies is not seen by students as interesting/challenging/useful then the gamified part risks becoming just pointless homework. The same goes for the actual tasks done during homework: if they are not properly contextualized and connected with the course's competencies they might also become pointless homework.

I suggest a food reward.
A badge or star is another option - physical or virtual.

I think that food rewards would work better for rewarding in-class participation than things like homework - e.g. bringing a box of prepackaged little chocolates to your lectures and tossing one to anyone who asks a good question, or answers a question you pose to the class (right or wrong).


text:
Are course grade distributions supposed to be bell shaped?
I just got done teaching my first course, and my grades are heavily skewed:

I consider my assessments to be fair and have had them vetted by more experienced faculty.  Also, the course averages are usually high in this course (mine was 83.5 or so).  However, it seems like I have a strange distribution of grades.  What might have caused it?

To answer the titular question: no.  And if you think your grade distribution is a problem, try asking senior faculty.

It's not that any distribution in the world should be bell shaped, and grades can have any sort of distributions. The few times I had a look at the distribution in my exams they were either bimodal or loosely exponential.

Note that there's a hard cap on grades at 100% - and that's where you seem to have a second peak. You might be essentially squeezing everyone that would have been on the high tail into one bucket.

What @Walt says. This is a classical ceiling effect. A psychometrist's first impulse would be to add a couple more difficult items to the test, to better distinguish very good from stellar students. YMMV.

Note that philosophically, a bell-shaped distribution also implies things like "As many students should fail my class as should get A's." Whether this is desirable at all is something you should probably decide.

Also see What does an optimal grading distribution look like?

The thing about this course is that the grades are already about this high. I'm not sure what the distribution looks like for other instructors though

In a perfectly bell-shaped grade distribution, you would generally expect as many As as Fs. But if your class is run in the typical US-style fashion, the vast majority of Fs should be eliminated as students are allowed to drop the course well into the semester - usually having plenty of time to realize they are not ready for the class. This alone will produce a strong skew towards high achievement, and that's just a single reason. Overall I don't see anything obviously wrong based on the distribution alone.

@Walt good Observation. Just adding that this may be wanted.

Why would you want your grades to come out as a bell curve? Wouldn't that signify a failing class?

I suppose pre-course exams are expected to obtain something close to a bell curve. You could expect half the people to know half the stuff, and the rest know more or less, before having walked into any course.

What type of course this this?

@anonymous: it is a second year multivariable calculus course

@Fomite That's not necessarily true -- the bell can be (and usually is) centered significantly above the pass/fail mark. The detail of an optimal shape could probably be argued forever, but the implied truncation in this distribution would concern me. I'd go with Stephan's suggestion and add some more challenging elements at the ends of the questions to distinguish between all those students in the top couple of bins.

The distribution doesn't have to be of bell-shape. In very large scale open exams it may be reasonable to assume a bell curve. In many other situations, the distributions can be affected by the class size, mix of the students, objectives of the course, validity of the exam questions, and difficulty of the exam questions, etc.
Class size: The smaller the class, the harder to observe any discernible distributions, such as a bell-shape normal distribution.
Mix of students: If you're teaching a quantitative class and there are two streams of students from i) art programs and ii) engineer programs, you may see some other distribution like a double bell-shape bimodal distribution.
Objectives of the course: Some courses can be designed based on a fixed and stringent set of standards. For instance, if you teach anesthesiology and the passing grade for the students to take the board exam is 90%, the end distribution is unlikely to be bell-shape.
Validity and difficulty of the exam: Invalid questions may lower the accuracy which prohibits you to see the true distribution; overly easy or hard exam can move the curve towards high- and low-boundary, causing truncated bell-shape distributions.
If I have to give an assessment I'd first suggest removing the 0% as it's a special case and yet tilting the impression of the curve quite badly. For the rest, I'd say if you're teaching an introductory course in which students are expected to gain a good foundation, this is not a bad distribution. If you're teaching a very advanced course, with nearly 20% getting close to full mark then the assessment may benefit from a re-tuning.
Generally, I'd advise:

Analyze the grades and your course objectives in tandem. Just grade distribution itself does not tell if you're doing a good job.
Accumulate more data across cohorts of students. I find that after 3-4 times teaching the same course the patterns would start to emerge.
Compare to historic grade distribution (just a few years before you picked this course up) to make sure you are not way off. Consult the appropriate dean if they are.
If so inclined, try to analyze your exam items. There are special statistics to check if your exam questions are "good" questions. Most academic institutes should have an education affair office that can help you.

To this very thorough answer, I'd add that a bimodal distribution such as OP observes is, in my experience, pretty usual.  And, if you must use a single measure of central tendency, median, not mean, is the right one.

The shape seems less important than the whether or not the shape can be explained. After all, what good are observations without a hypothesis?

@corsiKa When I was very new to teaching I showed a senior member of the faculty my bimodal grade distribution and asked what it meant.  "It means," he said, tapping the right hump on the graph, "that these kids get it and," tapping the left hump, "these kids don't."  That has been my hypothesis ever since.

This looks to me like the sort of distribution you would expect from an exam that is simply too easy, and fails to distinguish at the top end.
You probably do have a roughly bell-shaped distribution of student abilities, but since in your exam the middle of the bell is at 85% or so, all of the high-ability tail inevitably get lumped together in the 95-100% bar.

Interesting.  Totally opinion-based follow-up - is that a bad thing?

It depends what the point of the exam is. If you just want to test that students have attained at least a certain standard, this is fine. If you also want to be able to tell who is merely good and who is outstanding, then it is a bad thing. Usually in a university context you do want to do both.

@Michael Stachowsky As a student, I always hate easy exams. Even if you are still very generous with grades, making exams push the mental limits of the students makes it way more fun. It also drives home that there is much more to learn.

@A.W.Grossbard Yes, strong students often enjoy challenging exams. But exams that challenge the top students can easily lead to (literal) tears at the other end of the class.

@A.W.Grossbard The point of an exam is not necessarily to challenge students. Especially in lower level classes it is to determine whether they have met a certain level of comprehension that is adequate to move on to the next course without being totally lost. A hard exam would be inappropriate in such a class.

@A.W.Grossbard: As an instructor at a large urban community college, I've learned the following criterion: If I find any test question to be "interesting", then that's catastrophic (i.e., the entire class but one will fail it badly), and it must be stricken out.

Remember that your primary task is not to accurately tell better students from worse, but to make sure they learn what they are supposed to. This is also the role of the test: for students passing a test is supposed to be a fixed, specific goal to achieve, not a competition.
Therefore, the proper question here is: do you actually believe all of your students except for those unlucky few deserve passing your class? Do they learned what they were supposed to learn? If so, this is fine. The scores do reflect good on you that you taught your students well. This might also be a sign that it's not the test that requires adjustments, but the curriculum—you could probably teach more material in that class, and it will also indirectly lead to the exam being more difficult.
If not, then you should definitely adjust the test.
Some time ago I collected graphs from some documents from Polish Ministry of Education on the high-school exam. This is a huge sample (around 300k yearly), and you can see that scores do not always take a bell shape. Some useful discussion on interpretation of these graphs is in a reddit thread, especially explanation on the peak around 30% (the passing point) of the language exam.

That's great. Thanks so much for sharing that link.

"Remember that your primary task is not to accurately tell better students from worse, but to make sure they learn what they are supposed to." This is clearly not the case in many circumstances.

Senior faculty here...
Ask yourself if you are grading your students against each other, against the material, or a combination of both. If you are grading against the material then the shape of distribution is not relevant. For example, if the first exam is to write the "Hello World" program and everyone aces it, then you would have a very skewed distribution but you have the optimal result for the class as a whole.
If your learning outcomes are well-defined, as they should be, and your curriculum addresses those outcomes, then you are doing your job when everyone earns an 'A'. When I teach programming I am careful to grade students against the material, not against each other. I have a set of outcomes and I teach to those outcomes. On the first day I tell students "I hope you all earn an 'A.' I am not pitting them against each other, grade-wise. There are other ways to reward high-achieving students than imposing a bell curve or any other distribution on a class.

Indeed, material learned should not be represented by a Gaussian curve. Heights of your students, yes.

Generally grading distributions are analyzed like this:

If too many students get above the bell curve, that means tests are too easy. The tests are not separating "outstanding" students from "average" students.
If too many students get below the bell curve, that means the tests are too hard.

However, I am not sure that looking at a grade distribution is the best way to get a feel for how students are reacting to the coursework. If you just look at the grades you are only really looking at the sum of all of the results, and it may be hard for you to identify the best thing you can do to improve your couse.
As an instructor remember that there are more resources available to you than just an overall grade on a test (or an overall class grade). I recommend breaking down an exam or class by topic and evaluating how well students did with each topic.

If a large percentage of students answered questions on a certain topic correctly, that is great! Your choice of textbook combined with your presentation is making it so that they can absorb the information very well. If you are in this scenario, and you have already covered the requirements of the course you may want to consider going into more detail on the material or bringing up additional related topics that you think would help the students. You have a huge opportunity to go beyond the basics, if you have the time. If you can't go into more details, you could also go into a more practical direction and talk about how to apply these concepts.
If a large percentage of students seem to be struggling with a topic, if it's not a crucial topic you may want to consider dropping it. If it is a crucial topic consider reviewing it again in class (possibly using a different approach), putting it on the next exam, or assigning a paper on the topic. You may also want to consider revising the slides on the topic for future classes or watching presentations by other people on the topic to give you ideas for other ways you can present it.

Some answers and comments have mentioned the competitive aspect of grades that is sometimes present. I would like to mention this in my answer as well.

First, I am not convinced that every class needs to be competitive, carefully consider whether it will be useful for your students to add some healthy and constructive competition to the classroom experience before jumping into that.
If you want to seek out and reward the outstanding students, I recommend doing so by assigning projects and research papers. Challenging students to do research on their own and present their work will naturally bring out the best in talented students, and it could be very valuable practice for their careers (and grad school too).
Encourage students to publish their work if their work is at that level. This may also be a great way to find grad students if you are looking for them. At worst it will introduce students to what a career in research would be like.
Please don't assume that students are not interested in research, there was one professor I had who assigned "realistic" research papers (intended to resemble the requirements of publishing in a journal as closely as possible). It was a great experience, and based on that I ended up doing a thesis instead of a pure "class based" master's degree.

NOTE: My experience is primarily in STEM fields.

The theory is, given a random sample of students who have met the course pre-requisites, you should see something resembling a bell curve in your grade results.
In practice, there are a number of things that skew this. Sample size is a big one. For smaller classes, the sample size of any one course is just too small. Self-selection is another. At the higher-ed level, students have self-selected for courses in their major, such that they take more courses to match their interest and (supposedly) ability. Additionally, the entire college admissions process should pre-select for students who have at least some academic ability. This is part of why many colleges and universities in the US have a graduation requirement that mandates students maintain a B average for courses within their major (or some variation of this).
Another flaw in the bell curve is that it only looks at final outcomes. The full bell curve should include a so-called "long tail" to the left for lower grades, indicating students that withdraw before completion of the course.
There are many other human elements that can also distort a "pure" bell curve. Teaching ability can be one of these. As a new instructor, you may need to develop some experience for how to better grade and measure the performance of your students. But with only one class under your belt, it's just too soon to say if this is the big factor for that specific set of data.

And what theory says that one should see something resembling a bell curve in the grade results?

If you class size is large enough and you test isn't too easy or too difficult (with respect to the capabilities of your students), you should get an approximate bell curve.
Your distribution is bimodal. Either the students cheated(they had the questions before the exam) or the paper was too easy.

I've seen bimodal distributions considerably more extreme than this one. In some cases, it just means that some students understood the material and others didn't.


text:
Correct anonymization of submission using Latex
For a double blind review process, the venue (CHI) requires complete anonymization of submission files. Does this include the source files? More specifically, at the top in latex, you would normally specify authors as follows:
\author{Author name}
    \affiliation{Author affiliation}
\email{Author mail}

The template allows for "anonymous" keyword in the documentclass, hiding authors in the produced pdf. However, information would still be in the source files. So should one leave out these headers completely?

Are you required to submit the source file or just the pdf output?

@LazzaroCampeotti actually, this depends upon what the journal shares with reviewers. Usually, they just share a PDF with reviewers even if you upload the LaTeX files (but I have seen some journals that share TeX code with reviewers).

I would ask the journal. It all depends upon their workflow. If you want to be safe, replace your information with generic placeholders.

It is not a journal but a conference, CHI is the ACM Conference on Human Factors in Computing Systems.

I don't think it's an entirely unreasonable question. But isn't it just faster to go ahead and remove these from the source while you're thinking about it than worry?

If only I would have read the comments in the submission portal... There it states:

Please upload the source files of the paper in a zip format. The zip file must be under 300 MB. The source may be used if an accessible version of your PDF is not available. You do not need to perform anonymization in your source files. The files uploaded through this field are visible only to the paper chairs and accessibility chairs to create a more accessible version of your submission if needed by a reviewer.

Consequently, no anonymization is required. However, confirming with paper chairs, I was suggested to anonymize source files anyways, just to be safe. From that I understand that it doesn't matter whether there is author information in the source files or not.

In general the answer is no, you do not have to anonymize latex source files when submitting a paper to a conference using a blind process, because what you submit is the PDF file produced by latex, not the source files.
In the particular case of CHI 2025, the website states in Step 4:

Main Paper Deadline: Authors must submit the pdf of their paper, and related form data, before this deadline. No extensions will be granted. Any submissions showing ‘incomplete’ after the deadline will be deleted.

Which confirms that only the PDF file has to be submitted, not the latex source files.


text:
General Relativity Book Publication
I am writing a general relativity book for the high school grads and above. I expect my readers to have basics of calculus and vectors and Newtonian Mechanics - I am covering the necessary Special Relativity, Differential Geometry, Tensor Calculus and other basics here. Then I will explain the necessary tensors of Riemannian Geometry, then finally derive EFE, then dive into black holes and cosmology. The book will contain countess graphs and discrete 2D and 1 + 1d examples and EinsteinPy codes (I am a CS grad) to make it as visual as possible - while not sacrificing the necessary maths. TBH, part 1 and 2 of this are already published in Bangla, right now I am rewriting it in English for an international publication.
I need help about the publication process, steps of it.

What specifically is your question? "I need help" is not something specific enough for any of us to answer.

Go to your preferred publisher's webpage and submit a publication proposal.
Example for Springer. For this publisher you can also contact one of their Publishing Editors (here's the page for the ones in physics). Springer also has a guide for what to put in the publication proposal. You'll need to think carefully about the scope of your book, its length, its intended market, etc.


text:
Manuscript in 'Decision in Process' for 6 weeks and no response to email requesting update
I seem to be having a recurring issue with a journal. I submitted a manuscript in April of this year that was sent out to peer-review and the reviews were completed relatively quickly. The manuscript then sat in 'Decision in Process' for 4 weeks before I sent an email politely requesting an update, after which within 2 hours I got the reviews back with a request to submit corrections within 3 weeks. I completed the corrections and submitted the updated manuscript. Again, the reviews were completed relatively quickly on the corrected manuscript and the manuscript then sat in 'Decision in Process' for another 4 weeks at which point I sent an email, again politely asking for an update. Unlike the previous time however, I have received no response to this request and it has been 2 weeks since I sent the email asking for an update on my manuscript following my corrections. Is it appropriate to send another email requesting an update? Or do I continue waiting? Is this unusual or am I unreasonable to think that these decisions following peer-review shouldn't take this long?

https://academia.stackexchange.com/questions/88355/is-my-paper-under-review-or-similar-for-too-long-and-if-yes-how-should-i-reac Does this answer your question?

One of the answers states that a decision is typically taken within a week, so I suppose it is not unreasonable of me to think that decisions shouldn't take this long once peer-review has been completed (which answers that question). However, what that thread does not address is whether it would be appropriate to send another email requesting update, after I have already requested one when the manuscript is in 'Decision in Process'.

This one, then? https://academia.stackexchange.com/questions/26857/is-it-okay-to-resend-an-email-to-a-journal-asking-for-the-status-of-a-submitted

You can send an email, and you might get some information, but it is unlikely to change or shorten the decision process.
The canonical question on the editorial process provides some information, but there are a couple of other factors that can delay decisions.
If the editor handling your paper is not a full time employee of the journal, which is typical in some fields, their work for the journal must be balanced against other duties. Late Summer and early Fall can be busy periods for academics.
Moreover, if the journal is print, rather than online, then page limits for the currently planned issues must be taken into account, and if the more-or-less acceptable submissions exceed this limit there can be delays while the editor(s) work out the best use of available space, as well as, in some cases, providing some coherence to an individual issue.
I suggest emailing only if you have a need to know, rather than a desire to know. For example, if you have some decision pending or a deadline of sorts, then knowing might be necessary. Likewise if you are considering withdrawing the paper for any reason. Otherwise it is just anxiety producing. You can reduce that, perhaps by getting down (harder) to work on the next paper.

I guess I'm just anxious because I will be submitting my thesis within the next couple of months, and would like to have the paper (hopefully) accepted before then. Plus the need to factor in time for any further corrections while I write my thesis. Thanks for your response.

NGH, that probably qualifies as a need to know.


text:
What is the procedure for citing a source with incomplete information?
If it matters, I am specifically asking about the IEEE referencing style, but I'm happy to learn about other styles as well.
One of the more frustrating parts of referencing is that every book publisher and journal has their own standards for what information they give you. I've come across some books that don't have a city of publication, for example, and some journal articles where the specific volume is not clear.  When citing web pages this is a particular challenge, since sometimes it is not possible to find, say, when the web page itself was published.
In cases where information is simply not available, how do you cite a source correctly?  Is there a way to indicate that the information was not available?

For web page dates, use the date you access the page. It may have changed since original publication. For city for books, use the publishers headquarters if nothing else is available. I don't know why you can't find the journal volume, though, unless you are relying on secondary sources. Some journals and other publications may not have formal volume numbers. Use the date of publication. In general, though, don't try to cite something that you don't have "in hand".

If you are in computer science (or anything IT related or IEEE), bibtex download (from ACM or IEEE sites) will be your friends.
Apart from that, if you still need more information, try your best (include everything you know) and it's ok.


text:
How to reward students for learning from mistakes without penalizing those who didn't make mistakes in the first place?
We're thinking of implementing a policy where if a student asks a question/makes a mistake in a lab, they are required to write down what that question/mistake was and how it was resolved.  These "lab notebooks" (for want of a better word) will be evaluated based on the quality of the responses to the problems, not the problems themselves.  For instance, if the problem was relatively trivial ("We didn't turn the device on") but the response is great ("we came up with a checklist that we will follow to ensure that the basic setup is done every time") then the submission would be evaluated with a high mark.
For the majority of our students, there will be more than enough stuff in each lab to write about.   However, this isn't universally true and some groups will do the lab just fine without asking any questions.  Whether they make mistakes or not is an open question, but there really are some groups who just get what's going on, take their data and do very well without needing to interact with us at all.  And there's the problem - these students would have an empty notebook because they didn't need to go back and check things out.  Would they get a 0 on that part?  That doesn't sound fair at all.
How can I reward the students who learned from their mistakes without penalizing those who don't make any?

Just grade them based on outcomes.  In my opinion, the issues you're running into are a clear sign that you're assigning a task that is not worthwhile ("busywork").  If they are graded based on results, they will be motivated to learn from their mistakes in order to produce good results.

I'm willing to concede that after a bit of discussion.  We are noticing a few things.  First, students are asking the same questions over and over without really learning from it, and second students' first reaction is to ask us before trying to solve the problem themselves.   We feel that the best way to address that is to have students record their issues and write the responses in their own terms.  I don't necessarily feel that's busywork, but I'm willing to consider other options to my problem

This sounds like « partial marking » ie awardings points for application of method even if the final answer was incorrect.

@SolarMike it goes a little bit beyond that.  In our labs, currently the only thing we mark is the end results.  Nearly every group gets these results "correct" (ie: they get reasonably good marks, not necessarily through good experimentation).  We want to assign grades, ideally, to what they learned of the process.  I don't want to penalize a student for bad process if they learned from it, but I do want to reward them for thinking about their process as much as their results

Give 100 bonus points to those that don't need help/make no mistake. Those who make mistakes receive 0-99 bonus points depending on this "lab notebook". Wouldn't this be an easy solution?

@DavidKetcheson If that "busywork" teaches them how to better document, journal and problem solve? Especially if the "journey" is more important than the "destination". I'd consider this a variation of "Show your work". 8x + 7 = 47... x = 5? Great! How'd you get there? Just because you "know" the answer doesn't mean you don't get a pass because you can just write it down...

students' first reaction is to ask us before trying to solve the problem themselves Not an answer, but taking a page out of Stack Exchange, ask them: What have you tried? If their effort was insignificant or null,  look at them with a dead stare and answer them, in a robotic voice: "Your question has been closed due to evident lack of minimum research"

@xDaizu: we're going to be doing something really similar.  The TAs are only going to show them where in the lab manuals to look.  The lab manuals are pretty extensive but still under testing, so I expect that the first time we run this (next semester) we may have to be less strict

"First, students are asking the same questions over and over without really learning from it, and second students' first reaction is to ask us before trying to solve the problem themselves." This gives me the impression that when they ask a question, you give the answer. Try to lead them to the answer instead.

A different idea: if you want to reward them for learning from their mistakes, how about giving them extra points the first time they correctly answer a question that's of the same type as one they previously got wrong?  As long as the bonus points are less than what they lost in the first place, there's no way to game the system by selectively getting things wrong.

@BlueRaja-DannyPflughoeft that's a good idea, actually.  It might take some effort to set up in terms of grading time, though.  Any thoughts on how to get around that?  We do use an online tool for some grading but I don't think it's sophisticated enough.  For what it's worth, we have about 400 students and 12 TAs

There's tacit assumption that rewarding students for learning and penalizing them for not learning is the proper business of those who teach. If that assumption gets explicitly rejected by students and teachers, that would avoid some avoidable problems.

Urgh. I remember this from my Chemistry coursework for GCSE.  As one of those who actually "measured twice, cut once" there was often very little that went wrong.  As this was part of the marking criteria, I actually had to make up mistakes, to which I could provide corrections to get the full marks.   - and I wonder why I don't like writing up results these days.

r.e. the above, my teacher actually instructed me to do this, along with a small group of others.

And that's what I'm trying to avoid.  I don't want to get students to make fake mistakes in order to get marks

"How to reward students (increase their grade, spend more time teaching/mentoring them) for learning from mistakes without penalizing (not lowering their grade, not taking time away) those who didn't make mistakes in the first place?"  Not possible without grade inflation or you working more.  I see no true answer to the title question.

I might have to agree.  Great points have been raised but it's probable that the question itself is not the right one to ask

To be fair (and to avoid encouraging students to intentionally make mistakes), the marks awarded for correcting an error should never exceed the marks lost by making that error in the first place.

At face value, it seems clear that the issue is that you're trying to mix measurements of two very different kinds of performance. I think the REAL issue is deeper than this, but let's start with the two things you're mixing up:

You want to measure outcomes. Did the group end up with "correct" results from the lab?
You want to measure a specific (learning) process. Did the group learn from mistakes and questions?

The first measurement is usually the default in classroom settings because it's the most obvious. The second measurement has value, though: You're trying to prepare students for life, not just hand them a set of answers. Measuring their ability to learn, and encouraging them to be learners, is important.
That said, measuring outcomes has the advantage that it's not dependent on the process. It's easy to see if the lab result is correct or not. I don't think that's the focus of your question, so let's concentrate on the second concept - measuring the learning itself.
That brings us to the core issue with your proposed process: Measuring learning from mistakes and questions is very, very different from measuring learning in general. You hinted at this when you said,

For the majority of our students, there will be more than enough stuff in each lab to write about. However, this isn't universally true and some groups will do the lab just fine without asking any questions.

The issue here is that you've singled out a single type of learning to measure. The students who learned by failure and questioning are correctly being evaluated by your grading approach. This is good - you're encouraging that type of student to hone their skills. So let's look at the second group - the students who "did the lab just fine without asking any questions." The important thing to remember is that they weren't born knowing how to do the lab - they learned it too, just in a different way than the question-askers and mistake-makers. The reason why your new approach lacks fairness (as you identified yourself) is that it's only attributing value to one type of learning.
So - is it a bad idea to grade your students based on the questions they ask and the mistakes they make? If you're trying to grade them on their learning process, which is what I think you're trying to do, you need to make sure you're accounting for all types of learning not just the one style. This will be really challenging - people learn in multiple ways and transition between learning styles in a fluid manner, and many learning styles don't leave behind a "paper trail" you can measure after the fact. So let's get to your specific question:

How can I reward the students who learned from their mistakes without penalizing those who don't make any?

I may be frame-challenging your question, but if you allow me that liberty, I would rewrite it as:

How can I encourage students to be aware of, and work on improving, their learning skills, versus just rewarding them for getting the right answer to a specific assignment?

I think the answer to this will be much broader than a single stack exchange question, but some ideas to think about:

Include a discussion/presentation during/after the lab where a selected group walks the other groups through how they approached the lab, with a focus on how they prepared or learned the material beforehand and how they dealt with issues/challenges.
When a group asks questions during the lab, let the other lab groups answer it (instead of you). When an answer is given, have the answering group provide explanation of where they got the answer from.
Include an entire lab session, or elements of a lab, that are not part of the graded solution and are deliberately "unanswerable" in that they cover material you haven't taught or are otherwise deliberately difficult. Have the students come up with their own way to resolve this. This could even be a lab where the process intentionally causes failure, which would "force" the students to use your "learning notebook" as a tool to document and explain the failure and their approach to solving it. Purposefully cause them to go off-script and respond to the failure with creativity.

Basically, you need something that accounts for multiple learning styles, and something that encourages accountability instead of just allowing students to "cheat the system" by pencil-whipping busywork in order to get a grade. Notice I left your word choice of "reward" out of my re-worded version of your question - I think awareness and demonstration are probably more important than a reward - at the least because the reward just causes students to focus on the end, rather than the means, which totally dodges your goal of getting them to focus on learning skills versus just a final result.
Edited to add:
Let me further add: I don't think the "learning notebook" is a bad idea. It actually sounds like a great way to encourage people to recognize failure and accept it as part of a (positive) learning process, versus being afraid of failure. It would be great to include along with my three suggestions above. That said, I do think it becomes cumbersome to try to include it as a graded item, and more importantly, regardless of whether you grade it or not, focusing on it (as a graded item) misses the larger opportunity to consider and encourage other learning styles.

Very good points and a lot to think about. Thank you!

As a student, I would have been really frustrated by your proposed approach. I didn't like to ask questions or show failure in a lab setting. I needed to learn it upfront and then just execute - I needed to be an expert. My suggestions 1 and 2 would have been really rewarding and reinforcing for me, and my third suggestion would have caused me to face my own weakness of getting caught off guard by issues. Having a mix like that should encourage and challenge everyone versus just a specific subset of the students.

Students need to become comfortable with failure—and the earlier the better. The longer a student goes without learning from failure the harder it will be for them to deal with once it comes up. If you have students who never fail, I would simply remove that potential score from their grade. Oversimplified, students who fail, but don't learn from it (or don't demonstrate how they learned from it) would get, say, 0/100, students who showed superb learning from failure would get 100/100, students who continue not to fail would get 0/0. That makes their other work more heavily weighted.

@RubelliteFae - I agree that students need to learn from failure - but I don't agree that a grading approach should depend on it. Mainly as I said because it doesn't incent students to develop other learning styles. To put it differently: it's unfair to students who "fail" or learn outside of the graded setting. As a student, I had lab classes where I got perfect grades on every lab. What would my score have been with your approach? Should I not get credit for the work I put in to learning and preparing for those labs?

I tried to indirectly say this in my answer: I think it's appropriate to grade outcomes. I don't think it's appropriate to grade learning processes (mainly because it's hard to come up with an approach that's equal and fair to everyone) - you need to find other ways to incent or encourage development of learning processes. That's why none of my three suggestions at the end of the answer focus on assigning a grade, but rather just on direct participation/encouragement of learning.

@dwizum As stated, under the scenario you presented you would receive for example, 250/250 for the work proper and 0/0 for the "learning notebook." It wouldn't harm your grade, it would just more heavily weight your other grades.

@RubelliteFae My point is: You're giving me a null grade because I learned in a different way than some others. Yes, it doesn't directly bring down my average, but it changes weighting significantly, which can definitely influence the final grade. Should the students who needed the notebook get a 0/100 or 0/0 for "not studying ahead of time?" For me, you're lumping learning and outcome together. For others, you're evaluating those separately. That doesn't feel right to me. And at the end, I don't think the grade should be what matters, but rather recognizing and encouraging ALL styles.

Let me add - I appreciate the comments and I'm going to edit my answer to add some clarity to my thoughts around the "learning notebook."

I am okay with lumping those two together. After all, if the student did their lab perfectly, the extra weight can't lower their grade. If they got an A, but not a perfect A, it would, but very slightly. As long as it is clear at the beginning of the course it seems perfectly fair—and honestly is something teachers should be doing from a very young age. Alternative styles are fine until they aren't. So, those who experiment w/ different styles will find out what works best in which situations & become comfortable learning from failure. IMO, failure shouldn't have negative connotation.

I've framed the lab notebook in my own mind as a deviations notebook. You're given essentially an SOP on how to get from raw materials to a product in student labs and the "answers" are the result of in-process testing to ensure you obtained the right product. Deviations state that some step(s) of the SOP weren't followed normally, and include an impact analysis, corrective/preventative actions and a final disposition or resolution. It's a great way to look at how a mistake was made, whether it's your fault or not, and determine how it was handled. I'd encourage group discussion on deviations.

It's a great way to highlight good documentation practices in the laboratory setting, laboratory writing, and the importance of quality control/assurance without making anyone feel smaller for it. It's also a great way to get students thinking critically about the experiment: if you can discuss as a group how a mistake at step A, B or C might change the product, or ways to prevent or correct future mistakes, they're also thinking outside the comfort zone of the typical lab handout packet.

@CKM Great comments. It also strikes me that a prof could introduce the idea of such journaling by making the first assignment one that will always "fail."

Yes! That's the sort of thing I was thinking of with my third suggestion above. That way, it's not "optional" and it's part of a bigger approach that might encourage lots of styles versus focused on "learn by failure." I will edit the answer to reflect this.

You could use a "point recovery" system. I haven't seen anyone else describe it, and I don't know if there is a formal name for it. Anton's answer is somewhat similar in practice.
After each lab, students are offered an opportunity to recover a portion of the missed points from that lab. This opportunity can be a notebook review, a test/quiz on the lab material, a followup analysis on their previous lab work, or an optional lab exercise. It could even be a group project, similar to Michael's suggestions.
You can choose a proportionate cap (max XX% of missed points), an absolute cap (max score of XX), or both. Setting an absolute cap will deter stronger students from participating in the optional work, as they have less to gain. You decide whether that is desirable.
Setting the "right" caps on point recovery is a bit of trial and error. The best values will depend on how stringently your labs are scored and how much you want to encourage participation. If you chose to allow "up to 85% of missed points" with "no maximum on final score", you would allow any student to achieve a B with excellent followup work---even if they missed the lab entirely. (A reasonable way to accommodate absences, too.) In this case, even an A-level student could accrue a few extra points by participating.
Ideally, you would look for some evidence that the student grasps material which was not evident from their performance in the lab. This makes notebook reviews or self-analysis of their lab work better options; you can award points specifically for addressing issues with their original lab work. The self-comparison is key for improvement. Otherwise, you can expect to see a lot of students with 50-60% lab scores come in and get around 60% on a followup quiz without much real improvement. If you want improvement, you need to measure exactly that, and only that.
The biggest difference between this method and Anton's approach is that this method favors students who mastered the material initially---they will end up with the highest scores in the group most of the time. As before, you would have to decide whether this is desirable. Should their faster uptake or lesser need for instruction be a factor in their final grade? Is this program or field highly competitive in nature?

This is what I was thinking, basically not doing the notebook doesn't hurt you if you've done the lab flawlessly but having the notebook can save you from a less than flawless performance grade.

I had a professor that offered something similar: the "revised" version of your lab would be graded and weighted at 50% of the difference: 50% on the original, if revised to 100%, would cap at 75% final, but an 80% could be increased to a 90% with a solid revision. From what I recall, >50% of students opted to submit revised versions of their reports.

"Should their faster uptake or lesser need for instruction be a factor in their final grade? Is this program or field highly competitive in nature?" Realistically, when you have a job, your won't get many do-overs. Even in an academic position, getting stuff wrong is not good for anyone. Someone who gets things right the first time is more valuable to whomever they serve, and there's nothing wrong with grades reflecting that. Love the idea. +1

The confusion i see with this is that different groups are going to have different problems. This makes comparisonsense of responses difficult because of the variety in the responses. This is in addition to people being able to skip this altogether. I recommend the following.
** have a set of common problems that every group responds too. There are probably common mistakes  that happen every year. Make everyone explain  these whether they made the mistake or not. If they didn't make the mistake it reinforces proper behavior while preventing others from making the same mistake.
** for those who do make a real mistake set parameters for what to report. For example maybe they only have to share 2-3 mistakes rather than report everything. A criteria for judging the significance would be useful as well. For example if someone forgets to turn the power on maybe that shouldn't require an explanation.
** lastly a rubric that delineates exactly what is expected would eliminate the confusion and laziness. An actual example would further help the students as well.

I'm answering this from the point of view of, that in professional software development a log book is an important tool for the long term maintenance of a product.
Instead of focusing just on "learning from mistakes", which potentially alienates students who feel they have not made any mistakes. You want to have students built the log book as their "bible of information for the next developer".
This means documenting:

Potential pitfalls that were discovered or researched
Solutions that were ruled out for one reason or another
Areas that can be looked into further, if more time was allocated
Thoughts on potential improvements that could be made
Mistakes and major learning items that were found
Any thoughts or remarks about the work done
Justifications on why an approach was taken, over another one

Importantly, not everybody's log book will contain all of these. But these are the kind of elements that can provide a real value to another developer on a real world problem. The idea is to be tracking everything that might be valuable in the future - to avoid duplicated work and to help communicate potential ideas for where things could be taken.
Marking is of course difficult, but I'm sure you can see that from the scope of the log book; no student should have an empty one. Students who struggle, will be able to detail what problems they faced, while students who succeed will be able to note down potential improvements or alternate solutions they avoided (and why).
Log books are definitely a valuable professional tool, and I'd suggest approaching it from that angle - rather than trying to gauge a student's learning from their book (and grading them on that).

One option would be to do the following:
The grade for each lab consists of two components:

X% for the actual lab report (that would have been submitted without introducing the "new system"
(100-X)% for the "Q&A lab notebook" part of the report.

Now, if the Q&A lab notebook is not present (because students completed the lab without questions/mistakes), they automatically get full (100-X)% portion of the grade. If the Q&A lab notebook is present, it is graded accordingly.
With that system (in my opinion):

Students are not discouraged to ask questions because they can get a full mark even if "everything falls apart" with a proper explanation.
Not asking a question when the problem is not resolved within the group is not an option – as then the repercussions will be on the X% of the grade.
Students are encouraged to check obvious things themselves (like turning on the equipment) because otherwise, they will be required to write an additional paragraph in the report.
Those who silently finish the lab get an opportunity for a full grade
anyway (provided they did everything correctly).

Selecting X is definitely an art and highly depends on the structure of the lab and the existing reports. But 75% (actual report)-25% (Q&A) seems a reasonable initial guess to me.

It's an interesting idea, but what's stopping groups from just not submitting anything?  By your grading scheme a group that doesn't submit anything is assumed to have done the lab without any difficulty.

@MichaelStachowsky does this imply that by default a group does not submit any report on the lab?

@MichaelStachowsky I would say, that lab assistants (who are supposedly the graders) should just mark somewhere that a certain question has been asked.

Basically, all groups will submit a report, but some groups may not submit the question-report (the 100-X part).  Some of those groups genuinely had no questions, some did but want the automatic 100% on that part.  I agree, though, that if our lab assistants marked it down, then we can just compare.  That's definitely possible

Then you need to ensure the lab assistants do their job of marking the asked questions. You can mention to them (if you are not the one and not present during each lab) - that this, technically, reduces the amount of questions/work they are required to do DURING the lab. However, making sure the lab assistants are following the instructions is a totally different question.

Why not have two different 'rubrics' (aka marking schemes in the UK) - mark students against both and keep only their highest mark.
One of the rubrics could for example focused on those students how perform in the actual lab or how well they described what they did and why they did it.
The other could focus on Q&A part as described in this question.
This would not penalize those who don't ask questions as there marks would come from the first  rubric whilst still rewarding those who do as you describe - as there marks would come from the second rubric.

The groups that don't make mistakes will arrange their reward:

Think of a great response
Make a trivial mistake that would be fixed by that response
Get a high mark

A mistakes quota is easily gamed.

Agreed, that was one of my issues with this.  I think the main answer, about assessing all learning and not just mistakes, will be the way to go

A fundamental principle I've absorbed into my philosophy of life is "one can learn more from being wrong than being right".  If students are in a course because they want to learn (as opposed to wanting to get credit for having completed the course well), the reward for getting a lab done quickly and accurately should be the ability to take on something they're less certain about and may thus find more interesting.
Another principle that is all too often ignored (by myself as well as others, I'll admit) is that documentation about why various ideas that seems like they should be good, actually aren't, may be more valuable than documentation that seemingly-good ideas actually work.  If an idea seems like it should be good, it's likely other people will try it whether one documents it or not.  If it actually is good, those other people won't need one's documentation to benefit from it.  If it turns out not to be good, documenting that fact may benefit those who would otherwise have wasted time trying it for themselves.

Build the follow up lab assignment in a way that the experience from mistakes gives the student an advantage (eg knowing your way around certain equipment better due to having had to dismantle it to clean up a mistake).

Just have multiple labs (or whatever) and grade on results.  You are making it too complicated otherwise.  There is limited time to do coaching and assessment.  If I mess something up once and learn from it, it actually shows up in future work.  It's even better if I have to live with the bad grade versus getting a makeup. (We are not chopping off body parts...who cares if you take a bad grade once or twice...it's a kick in the butt.)
The ONE key help that you can do is having several labs, tests, etc.  Not single chances for failure, performance.  But if you do this...no makeups (other than perhaps the final).

Actually we are seeing students not learning from mistakes,  that's the problem


text:
Designing the "user experience" for labs without giving too much away
I have a rather unusual UX to design.  I'm writing a laboratory manual for engineering students learning how to build electric circuits.  The manual will be an interactive webpage.  One of our biggest complaints that we get from students is that the labs are too "cookie cutter", meaning that we often just list the exact steps they need to take in order to complete the lab.  Students would prefer to explore more on their own.
My idea is to design the lab manual in such a way that tasks that are not relevant to the learning objectives, but which must be done in order to achieve those learning objectives, are presented in the cookie cutter way.  Tasks that are relevant to the learning objectives are described differently.
An example of the first type of task would be: "Plug the red cable into the red jack on the DMM", accompanied by a picture.
An example of the second type of task might be: "Taking as many measurements as you think you need, generate a data set you can use to determine a model of the sensor".
Here is my problem: I want to create this lab manual and test it on real students.  However, I don't know which actions students will take while working towards the goals of the lab.  I know what I would do, but that's only one possible way to achieve the goals.  If I observe students who are working on the lab to determine the steps that they take, I can determine which tasks are frustrating them.  I then want to take those frustrating tasks and determine which ones are not really relevant to the learning objectives, so I'll just make them cookie cutter.  However, how do I generate that list of tasks?  I can think of two ways, neither of which are good:

Make the first version totally cookie cutter and observe student frustration
Make the first version completely not cookie cutter and observe student frustration

My issue with 1. is this imposes my method on the students, which is exactly what I'm trying to avoid.  It won't give me any information regarding which tasks need to be more open-ended.  My issue with 2. is that there are some tasks that are not intuitive and just need to be explained quickly. It won't give me any information regarding which tasks are really frustrating for the right reasons, since students will be frustrated by irrelevant tasks as well.
Any thoughts?

In my experience in designing electronic labs for EE the main factor is the time constraint:more freedom to explore requires (much) more time. How much flexibility do you have on this?

At the moment this is a completely exploratory phase, so I've got quite a bit of freedom.  I envision creating a bunch of activities, testing them all out, and seeing which ones fit both in terms of learning objectives and other constraints

Students are not lab mice. Make a best effort at educated guessing.

As @MassimoOrtolano mentioned in a comment (and I think this extends to all lab courses, not just EE), "the main factor is the time constraint." Clearly your suggestion to make the preparatory/emphasized steps "cookie cutter" and the learning objectives more open-ended is a way to approach this time constraint with balance.
It sounds like your main concern is in determining which of these steps should be cookie cutter and which should not. You give two approaches: start from the beginning and work forwards, or start from the end and work backwards.
However, that assumes you are going into this with no information. That might be a principled approach in some cases, but if I can be a Bayesian for a moment, you are missing a crucial piece of prior information: you should have some idea of what your learning objectives are before you even start.
The approach I would take is to first identify learning goals, then sketch out the procedure and identify which steps are relevant to the learning goals and which are not. Cookie cutter those that aren't, leave open-ended those that are. Surely, this won't produce a perfect lab course on first pass, and it might take some tweaking over time, but it seems like you expect this and it is a better approach than either extreme.
Second, involve your students in your goals. I think it's best to think of students as if they are quite capable and smart. Let them know: "I designed these labs so that I give you a step-by-step through some of the things that aren't relevant to the learning goals of the day. At the start of each lab, I will provide you with a list of the intended learning goals. I am looking for feedback if you either a) feel like you are spending too much time on something that isn't a learning goal, or b) feel like the answers are being handed to you for the things that are."
When in doubt, it seems like it makes some sense to bias towards cookie-cutter towards the beginning of the course and towards more freedom in the end, but that depends on how well the different pieces fit together and how complex the problems get by the end.

@MichaelStachowsky I appreciate the upvote, and the acceptance, but it's often better to hold off accepting an answer for a bit, to encourage other opinions/approaches. Of course it's totally up to you as the asker, just a minor piece of advice. :)

That's a fair point, but it is a good answer


text:
How to set goals when the challenge is to continually, gradually improve?
I  work in engineering education in Canada.  One of our biggest challenges is to instill the students with a desire to become "lifelong learners".  This isn't just an academic ideal, either - we are required by our accrediting body to ensure that students learn to learn, and engage with learning throughout their life.  This is notoriously difficult to do - how do you ensure that a student learned such things?
We often teach students to set goals.  However, I'm not convinced that goal setting is the right thing to do when the purpose is to continually, gradually improve.  Goals are, by definition, attainable milestones.  I feel that they are useful tools, especially in the short term, of course.  I set goals for myself all the time.  However, setting goals for the long term (as in, lifelong) is much more nebulous.
How can I encourage my students to continually monitor their own progress in the long term, without being concerned about the "end result"?

The best engineers don't know everything they need to know to do their job. They look things up. Try explaining to your students that it is perfectly normal to look things up. Hence, it is perfectly normal to learn on the job. Better still, invite some of the best engineers to explain how they learn on the job.

I think this is a very good question. Goal setting - as you mentioned - is a very useful tool to achieve milestones. The smaller the goals ("microgoals"), the more these follow the characteristics of a continuous growth curve. However, if I understand your question correctly, you might be looking more at something like "values" that you want to teach.
If lifelong learning is part of the curriculum, you face the question how to evaluate the process (?) and maybe more meaningful, how to teach something that ends up as an understanding of a useful value in the arsenal of a modern-day student. I'd suggest, the best way to achieve this is to 1) lead by example and to 2) provide further examples. Examples should come from someone whose achievements can be traced back to them learning specific new skills throughout life without which they could not have achieved what we currently perceive as their main achievement. You should by now have a few examples in mind of people in your specific niche that would be examples for this. Talk about them and how their willingness to continually grow and learn enabled them to achieve what they did. Doesn't have to take long, but should stick in students' minds.
Hope this helps!

To be an effective lifelong learner, you need to set yourself goals that are relevant at the time. Your students can start to do that now. The goals they set for themselves need not remain static all their lives. Examples of goals for a particular point in time:

balance reading papers and meeting coursework deadlines;
skim five papers per month and read one paper per month more carefully;
ask an average of one question for every three talks attended.

Do you see that some of these might disappear over time, or the numbers might change?


text:
Study design for best teaching methods - how to avoid overly specific results?
I would like to do a study with a class in which the students are split into two groups, A and B.  They will each receive a tutorial on the same topic, but group A will get a different tutorial than group B.  I will then give a post-assessment and a week after post-assessment, to explore retention rates among other things.  The goal is to compare the methods for tutorial delivery.  Specifically, group A will get a tutorial that is modeled on the traditional "expert solving problems at the board" while group B will get a tutorial that requires the students to do self-directed problem solving, with scaffolding.
My concern is that I won't be able to distinguish between expert solved problems vs. student solved problems. Rather, I will be able to distinguish between "when an expert solves THESE problems in THIS topic" compared to when students do so.
In short, I'm worried that my study results will only be valid for a very specific set of circumstances, not generally applicable even to other institutions that teach a similar but not identical course.  What are some strategies, if any, I can use to avoid overly specific results?

I think this is not just a question of statistics, but also of research methodology. I agree with CJ59's suggestion, but I would add that your school probably has a department that studies education/pedagogy: find them and make friends with them. This sort of question is unlikely to be a truly novel one, and there could be a lot of existing scholarly work in the field for you to lean on. Also, make sure you're complying with your institutions rules for conduct of research with human subjects and students more specifically.

No study in only a single class will be able to definitively answer such a question. The answer can only come from either a very large study or an accumulation of results over several small similar studies.
If it is an attempt to get at the truth it will tell you very little. If it is an attempt to learn/teach something about experimentation it can have value.
In the US, you would need to run this by your IRB. I don't know if Canada has similar regulations. But you are treating two parts of a class (human beings) differently and need to assure that neither group is disadvantaged in any way. It is unethical to do certain things when you strongly suspect that they are not effective, though I don't think that is a particular issue given your description.
On the other hand, there is already a lot of research on learning that addresses just this question. If you haven't already explored it, you really should.

yes, we will need to get research ethics approval no worries :-).  I've seen some interesting literature, but it appears that there are three camps that are at odds with each other.  I am interested in studying under which condition each conclusion can be made.  As is always the case, it is impossible to explain more in a 500 character comment :-P.  Good answer, though, thank you.