text:
How can I create multiple running threads?
Is there a way I can have multiple parts of the program running together without doing multiple things in the same code block?
One thread waiting for an external device while also blinking a LED in another thread.

You should probably first ask yourself if you really need threads. Timers may be OK for your needs already and they are natively supported on Arduino.

You may want to check out the Uzebox too.  It's a two chip homebrew video game console.  So while it isn't exactly an Arduino, the entire system is built on interrupts.  So audio, video, controls, etc. are all interrupt driven while the main program doesn't have to worry about any of it.  May be good reference.

Use the Arduino library NonBlockingSequence. You can define more than one sequence. Each sequence can describe a specific thread.

There is no multi-process, nor multi-threading, support on the Arduino. You can do something close to multiple threads with some software though.
You want to look at Protothreads:

Protothreads are extremely lightweight stackless threads designed for
  severely memory constrained systems, such as small embedded systems or
  wireless sensor network nodes. Protothreads provide linear code
  execution for event-driven systems implemented in C. Protothreads can
  be used with or without an underlying operating system to provide
  blocking event-handlers. Protothreads provide sequential flow of
  control without complex state machines or full multi-threading.

Of course, there is an Arduino example here with example code. This SO question might be useful, too.
ArduinoThread is a good one too.

Note that the Arduino DUE has an exception to this, with multiple control loops: https://www.arduino.cc/en/Tutorial/MultipleBlinks

AVR based Arduino's do not support (hardware) threading, I am unfamiliar with the ARM based Arduino's. One way around this limitation is the use of interrupts, especially timed interrupts. You can program a timer to interrupt the main routine every so many microseconds, to run a specific other routine.
https://www.arduino.cc/reference/en/language/functions/interrupts/interrupts/

It is possible to do software side multi-threading on the Uno. Hardware level threading is not supported.
To achieve multithreading, it will require the implementation of a basic scheduler and maintaining a process or task list to track the different tasks that need to be run.
The structure of a very simple non-preemptive scheduler would be like:
//Pseudocode
void loop()
{

for(i=o; i<n; i++)
run(tasklist[i] for timelimit):

}

Here, tasklist can be an array of function pointers.
tasklist [] = {function1, function2, function3, ...}

With each function of the form:
int function1(long time_available)
{
   top:
   //Do short task
   if (run_time<time_available)
   goto top;
}

Each function can perform a separate task such as function1 performing LED manipulations, and function2 doing float calculations. It will be the responsibility of each task(function) to adhere to the time allocated to it.
Hopefully, this should be enough to get you started.

I am not sure I would talk about "threads" when using a non preemptive scheduler. By the way, such a scheduler already exists as an arduino library: http://arduino.cc/en/Reference/Scheduler

@jfpoilpret - Cooperative multithreading is a real thing.

Yes you're right! My mistake; it had been so long ago I had not faced cooperative multithreading that in my mind, multithreading had to be preemptive.

As per the description of your requirements:

one thread waiting for an external device
one thread blinking a LED

It seems you could use one Arduino interrupt for the first "thread" (I would rather call it "task" in fact).
Arduino interrupts can call one function (your code) based on an external event (voltage level or level change on a digital input pin), that will trigger your function immediately.
However, one important point to keep in mind with interrupts is that the called function should be as fast as possible (typically, there should be no delay() call or any other API that would depend on delay()).
If you have a long task to activate upon external event trigger, then you could potentially use a cooperative scheduler and add a new task to it from your interrupt function.
A second important point about interrupts is that their number is limited (e.g. only 2 on UNO). So if you start tohave more external events, you would need to implement some kind of multiplexing all inputs into one, and have your interrupt function determine what multiplexed inut was the actual trigger.

A simple solution is to use a Scheduler. There are several implementations. This describes shortly one that is available for AVR and SAM based boards. Basically a single call will start a task; "sketch within a sketch".
#include <Scheduler.h>
....
void setup()
{
  ...
  Scheduler.start(taskSetup, taskLoop);
}

Scheduler.start() will add a new task that will run the taskSetup once and then repeatedly call taskLoop just as the Arduino sketch works. The task has its own stack. The size of the stack is an optional parameter. Default stack size is 128 bytes.
To allow context switching the tasks need to call yield() or delay(). There is also a support macro for waiting for a condition.
await(Serial.available());

The macro is syntactic sugar for the following:
while (!(Serial.available())) yield();

Await can also be used to synchronize tasks. Below is an example snippet:
volatile int taskEvent = 0;
#define signal(evt) do { await(taskEvent == 0); taskEvent = evt; } while (0)
...
void taskLoop()
{
  await(taskEvent);
  switch (taskEvent) {
  case 1:
  ...
  }
  taskEvent = 0;
}
...
void loop()
{
  ...
  signal(1);
}

For further details see the examples. There are examples from multiple LED blink to debounce button and a simple shell with non-blocking command line read. Templates and namespaces can be used to help structure and reduce the source code. Below sketch shows how to use template functions for multi-blink. It is sufficient with 64 bytes for the stack.
#include <Scheduler.h>

template<int pin> void setupBlink()
{
  pinMode(pin, OUTPUT);
}

template<int pin, unsigned int ms> void loopBlink()
{
  digitalWrite(pin, HIGH);
  delay(ms);
  digitalWrite(pin, LOW);
  delay(ms);
}

void setup()
{
  Scheduler.start(setupBlink<11>, loopBlink<11,500>, 64);
  Scheduler.start(setupBlink<12>, loopBlink<12,250>, 64);
  Scheduler.start(setupBlink<13>, loopBlink<13,1000>, 64);
}

void loop()
{
  yield();
}

There is also a benchmark to give some idea of the performance, i.e. time to start task, context switch, etc.
Last, there are a few support classes for task level synchronization and communication; Queue and Semaphore.

I also came to this topic while implementing a matrix LED display.
In one word, you may build a polling scheduler by using millis() function and timer interrupt in Arduino.
I suggest the following articles from Bill Earl:
https://learn.adafruit.com/multi-tasking-the-arduino-part-1/overview
https://learn.adafruit.com/multi-tasking-the-arduino-part-2/overview
https://learn.adafruit.com/multi-tasking-the-arduino-part-3/overview

From a previous incantation of this forum, the following question/answer was moved to Electrical Engineering.  It has sample arduino code to blink an LED using a timer interrupt while using the main loop to do serial IO.
https://electronics.stackexchange.com/questions/67089/how-can-i-control-things-without-using-delay/67091#67091
Repost:
Interrupts are a common way to get things done while something else is going on.  In the example below, the LED is blinking without using delay().  Whenever Timer1 fires, the interrupt service routine (ISR) isrBlinker() is called.  It switches the LED on/off.
To show that other things can simultaneously happen, loop() repeatedly writes foo/bar to the serial port independent of the LED blinking.
#include "TimerOne.h"

int led = 13;

void isrBlinker()
{
  static bool on = false;
  digitalWrite( led, on ? HIGH : LOW );
  on = !on;
}

void setup() {
  Serial.begin(9600);
  Serial.flush();
  Serial.println("Serial initialized");

  pinMode(led, OUTPUT);

  // initialize the ISR blinker
  Timer1.initialize(1000000);
  Timer1.attachInterrupt( isrBlinker );
}

void loop() {
  Serial.println("foo");
  delay(1000);
  Serial.println("bar");
  delay(1000);
}

This is a very simple demo.  ISRs can be much more complex and can be triggered by timers and external events (pins).  Many of the common libraries are implemented using ISRs.

You could also give my ThreadHandler library a try
https://bitbucket.org/adamb3_14/threadhandler/src/master/
It uses an interrupting scheduler to allow context switching without relaying on yield() or delay().
I created the library because I needed three threads and I needed two of them to run at a precise time no matter what the others were doing. The first thread handled serial communication. The second was running a Kalman filter using float matrix multiplication with the Eigen library. And the third was a fast current control loop thread which had to be able to interrupt the matrix calculations.
How it works
Each cyclic thread has a priority and a period. If a thread, with higher priority than the current executing thread, reaches its next execution time the scheduler will pause the current thread and switch to the higher priority one. Once the high priority thread completes its execution the scheduler switches back to the previous thread.
Scheduling rules
The scheduling scheme of the ThreadHandler library is as follows:

Highest priority first.
If the priority is the same then the thread with the earliest deadline is executed first.
If two threads have the same deadline then the first created thread will execute first.
A thread can only be interrupted by threads with higher priority.
Once a thread is executing it will block execution for all threads with lower priority until the run function returns.
The loop function has priority -128 compared to ThreadHandler threads.

How to use
Threads can be created via c++ inheritance
class MyThread : public Thread
{
public:
    MyThread() : Thread(priority, period, offset){}

    virtual ~MyThread(){}

    virtual void run()
    {
        //code to run
    }
};

MyThread* threadObj = new MyThread();

Or via createThread and a lambda function
Thread* myThread = createThread(priority, period, offset,
    []()
    {
        //code to run
    });

Thread objects automatically connect to the ThreadHandler when they are created.
To start execution of created thread objects call:
ThreadHandler::getInstance()->enableThreadExecution();

And here is yet another microprocessor cooperative multitasking library – PQRST: a Priority Queue for Running Simple Tasks.

Home page
Class-level documentation
Repository, with downloads and issues list

In this model, a thread is implemented as a subclass of a Task, which is scheduled for some future time (and possibly rescheduled at regular intervals, if, as is common, it subclasses LoopTask instead).  The run() method of the object is called when the task becomes due.  The run() method does some due work, and then returns (this is the cooperative bit); it'll typically maintain some sort of state machine to manage its actions on successive invocations (a trivial example is the light_on_p_ variable in the example below).  It requires a slight rethinking of how you organise your code, but has proven very flexible and robust in fairly intensive use.
It's agnostic about the time units, so it is as happy running in units of millis() as micros(), or any other tick that is convenient.
Here is the ‘blink’ program implemented using this library.  This shows only a single task running: other tasks would typically be created, and started within setup().
#include "pqrst.h"

class BlinkTask : public LoopTask {
private:
    int my_pin_;
    bool light_on_p_;
public:
    BlinkTask(int pin, ms_t cadence);
    void run(ms_t) override;
};

BlinkTask::BlinkTask(int pin, ms_t cadence)
    : LoopTask(cadence),
      my_pin_(pin),
      light_on_p_(false)
{
    // empty
}
void BlinkTask::run(ms_t t)
{
    // toggle the LED state every time we are called
    light_on_p_ = !light_on_p_;
    digitalWrite(my_pin_, light_on_p_);
}

// flash the built-in LED at a 500ms cadence
BlinkTask flasher(LED_BUILTIN, 500);

void setup()
{
    pinMode(LED_BUILTIN, OUTPUT);
    flasher.start(2000);  // start after 2000ms (=2s)
}

void loop()
{
    Queue.run_ready(millis());
}

These are “run-to-completion” tasks, right?

@EdgarBonet I'm not sure quite what you mean. After the run() method is called, it is not interrupted, so it has the responsibility to finish reasonably promptly. Typically, however, it'll do its work then reschedule itself (possibly automatically, in the case of a subclass of LoopTask) for some future time.  A common pattern is for the task to maintain some internal state machine (a trivial example is the light_on_p_ state above) so that it behaves suitably when it is next due.

So yes, those are run-to-completion (RtC) tasks: no task can run before the current one completes its execution by returning from run(). This is in contrast with cooperative threads, that can yield the CPU by, e.g., calling yield() or delay(). Or preemptive threads, that can be scheduled-out at any time. I feel the distinction is important, as I have seen that many people who come around here searching for threads do so because they prefer writing blocking code rather than state machines. Blocking real threads that yield the CPU is fine. Blocking RtC tasks is not.

@EdgarBonet It's a useful distinction, yes. I would regard both this style, and yield-style threads, as simply different styles of cooperative thread, as opposed to preemptive threads, but it's true that they require a different approach to coding them.  It would be interesting to see a thoughtful and in-depth comparison of the various approaches mentioned here; one nice library not mentioned above is protothreads. I find things to criticise in both, but also to praise. I (of course) prefer my approach, because it seems most explicit and needs no extra stacks.

(correction: protothreads was mentioned, in @sachleen's answer)


text:
Logging multiple gigabytes of data with Arduino
I have Arduino Uno with SD card shield. I'm planning to log a huge amount of data from various sensors - including accelerometers and 3D compass - to the SD card.
However, I have understood that Arduino, 64GB SD, FAT and large amount of data are not working too well.
Does Arduino work well with this amount of data on FAT? There's no need to read any data from SD on Arduino. If not, is there a better approach? Writing directly to SD without FAT?
This data logger will be on the environment where no computer or network connection is available. Performance is not an issue - around 8kB/s (over three months) is already enough.

I'm not sure where you have the impression it won't work. I haven't used 64GB cards, but have used 32GB cards and they work fine in terms of FAT32 and writing files.
As you have already worked out, reading or searching the file system can get very slow as size increases.
That said, I have found that larger capacity cards are more likely to have problems with the SPI interface being poor quality. Consumer devices do not use the SPI interface, and it seems that some manufacturers are willing to skimp on this side of it. The SPI interface can be buggy or unstable as a result. Genuine name-brand cards should work fine though.

You should consider using sdfatlib. This is currently supported version. Where an early version of this was the predecessor the IDE's stock SD library. SdFat better supports newer and larger SdCards, along with more features.

Updating with new location of sdfatlib

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
How to initialize digital output pin as low
I need to define a pin as output for relay module but when i do it in setup section, relay is activated (low - 0) automatically then i deactivated immediately with coding digitalWrite(pin, 1) but this issue results the relay is activated then deactivated within miliseconds. But I want to know that is it possible to define a pin as output but low in default.
void setup(){
  pinMode(pin,OUTPUT);
  digitalWrite(pin,1);
}

void loop(){
  if(start_relay){
    digitalWrite(pin, 0);
  }
}

you have a relay module which disconnects if signal is HIGH. the AVR MCU starts with pins in INPUT mode so the relay is not controlled by Arduino until setup()

You mean you want OUTPUT HIGH without an OUTPUT LOW transient?

There are two solutions:

Use the pin as a open-collector pin as in the answer of VE7JRO.
Switch the pin between input (high) and output with low. I prefer
to use the INPUT_PULLUP.
Write the output value before setting the pinMode.

All the microcontrollers of the AVR family (ATtiny and ATmega chips) allow to set the output value before setting the pin as output. This will enable the pullup resistor, but when it is done in the right order, that is fine.
Because this is a common way to use the microcontrollers and processors, the Arduino libraries allow to use that. The Arduino libraries for the SAMD processors (Arduino Zero, M0 and MKR) are written to allow that as well.
void setup() {
  digitalWrite(pin,HIGH);     // pre-set output value to HIGH
  pinMode(pin,OUTPUT);        // set to output, it will be HIGH
}

void loop() {
  digitalWrite(pin,LOW);
  delay(1000);
  digitalWrite(pin,HIGH);
  delay(1000);
}

When the relay module has floating inputs (as far as I know they do not) then you need extra hardware to pull the signal high with a pullup resistor to keep the relays disabled during reset.

If you have a relay that is activated by a LOW or ground path, and is deactivated
by an open circuit or HIGH signal applied, then you could try switching the pinMode
to act as the HIGH or open circuit.
const byte pin = 2;

void setup(){

  // Define the pin as INPUT_PULLUP until you are ready to use it.
  pinMode(pin, INPUT_PULLUP);

}

void loop(){

  delay(1000);

  // Activate the relay. Supplying a low impedance
  // path to ground turns the relay on.
  pinMode(pin, OUTPUT);
  digitalWrite(pin, LOW);

  delay(1000);

  // Deactivate the relay.
  pinMode(pin, INPUT_PULLUP);

  // Using digitalWrite(pin, HIGH); is an alternative to using
  // pinMode(pin, INPUT_PULLUP); If your relay module is the
  // same as the one shown below, then it doesn't matter which
  // method you choose.

}

I'm not sure which relay module you are using, but if it is the same as this type, using INPUT_PULLUP will work.

As Edgar Bonet mentions in his comment to kot's answer, using pinMode(pin, INPUT_PULLUP); is clearer than using digitalWrite(pin, HIGH);.
This code does 2 things:
digitalWrite(pin,HIGH);
pinMode(pin,OUTPUT);

First it sets the pin to INPUT_PULLUP, then it sets it to VCC (for lack of a better term). If you connect a modern LED with a 1k ohm resistor to pin 2, and use this sketch, you can see the LED "dimly" lit for 5 seconds, then it is "brightly" lit.
const byte pin = 2;

void setup() {

  // pre-set output value to HIGH
  // The internal pull up resistor is now connected to the pin.
  digitalWrite(pin,HIGH);

  // Wait for 5 seconds.
  delay(5000);

  // set to output, it will be HIGH
  // The pin is now connected to VCC (for lack of a better term).
  pinMode(pin,OUTPUT);

}

void loop(){}

As fas as I know, by default digital pins are set as inputs. Arduino has internally only pull-up resistors, which you can use in that way:
digitalWrite(pinOut[i], HIGH);

in first setup() line (without pinMode(pinOut[i], OUTPUT);).
If you need LOW state on start-up (pull-down resistors), you have to use an external resistor.

If you want to activate the internal pull-up, pinMode(pin, INPUT_PULLUP); is clearer (because more explicit) than digitalWrite(pin, HIGH);.

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Disassembling a RC car for a Arduino Robot
I am looking at making an Arduino robot car. I have a RC car( photos below). I was wondering if I could take the chassis of the car and modify it to make it Arduino controlled, if so how could I do that?(I have an Arduino UNO R3). I would like to make it into a wall avoiding robot. I have never soldered a thing in my life, and I don't own a soldering iron.

Depends what you want to do? But if you're good with soldering and electrics it should be easy... Tell us what you want your robot to do, and we will try to help..

Someone has done exactly this, and written detailed instructions at http://www.instructables.com/id/Autonomous-Control-of-RC-Car-Using-Arduino/
Some remote control devices (usually not the really cheap ones, like the one shown here) will have a standard transmitter/receiver. See http://www.alibaba.com/product-detail/3-channel-radio-control-receiver-box_311659816.html for a picture of the receiver. If you can find this in your remote control device, you should be able to unplug it, and replace it with an Arduino. You will need the "Servo" library - see https://www.arduino.cc/en/reference/servo
You can probably get away without soldering - you will probably need to strip some wires, and you can use crocodile clips, and/or breadboard jumpers to connect everything up. It won't be pretty.
If you want to learn to solder (I know it's intimidating, but it's not so bad) I recommend that you find a local hackspace/makerspace, and find someone there - they will probably be more than happy to teach you. They probably have everything you need, be nice to them, and if you want them to continue, join up & pay regular subs. And take every opportunity to teach the skills you have to anyone else who asks for help.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
Arduino: Put Data in Other File
The below code starts off with lots of data in the beginning, distracting the reader from the actual code.  What is the best practice to clean this up:
#include <Wire.h>
// For LED Backpack
#include "Adafruit_LEDBackpack.h"
#include "Adafruit_GFX.h"
// For LCD Shield
#include <Adafruit_MCP23017.h>
#include <Adafruit_RGBLCDShield.h>

// For pixel Units, such as hero
#include <Unit.h>

Adafruit_BicolorMatrix matrix = Adafruit_BicolorMatrix();
Adafruit_RGBLCDShield lcd = Adafruit_RGBLCDShield();

Unit *hero;
boolean doesNeedRedraw = true;
boolean textNeedsRedraw = true;
int8_t lastButtonPress = 0;
int8_t frame = 0;

void setup() {
  /* Important Setup Routine */
}

static const uint8_t PROGMEM
  smile_bmp[] =
  { B00111100,
    B01000010,
    B10100101,
    B10000001,
    B10100101,
    B10011001,
    B01000010,
    B00111100 },
  neutral_bmp[] =
  { B00111100,
    B01000010,
    B10100101,
    B10000001,
    B10111101,
    B10000001,
    B01000010,
    B00111100 },
  frown_bmp[] =
  { B00111100,
    B01000010,
    B10100101,
    B10000001,
    B10011001,
    B10100101,
    B01000010,
    B00111100 },
  walls_bmp[] =
  { B10100000,
    B10110000,
    B10010000,
    B11011100,
    B01000100,
    B01110100,
    B00010100,
    B00011100 };

void loop() {
  /* Important Main Loop Routine */
}

Who is your reader? What is the propose of cleaning your code? What is the skill level of the reader?

One way to clean that up is to create a "Globals.inc" file and cut and paste all of your globals into there.  After that's done, simply include the globals file from your main .ino file with #include "Globals.inc"
(Main.ino)
#include <Wire.h>
// For LED Backpack
#include "Adafruit_LEDBackpack.h"
#include "Adafruit_GFX.h"
// For LCD Shield
#include <Adafruit_MCP23017.h>
#include <Adafruit_RGBLCDShield.h>

// For pixel Units, such as hero
#include <Unit.h>

#include "Globals.inc"

void setup() {
  /* Important Setup Routine */
}

void loop() {
  /* Important Main Loop Routine */
}

(Globals.inc)
Adafruit_BicolorMatrix matrix = Adafruit_BicolorMatrix();
Adafruit_RGBLCDShield lcd = Adafruit_RGBLCDShield();

Unit *hero;
boolean doesNeedRedraw = true;
boolean textNeedsRedraw = true;
int8_t lastButtonPress = 0;
// 0-255
uint8_t frame = 0;

static const uint8_t PROGMEM
  smile_bmp[] =
  { B00111100,
    B01000010,
    B10100101,
    B10000001,
    B10100101,
    B10011001,
    B01000010,
    B00111100 },
  neutral_bmp[] =
  { B00111100,
    B01000010,
    B10100101,
    B10000001,
    B10111101,
    B10000001,
    B01000010,
    B00111100 },
  frown_bmp[] =
  { B00111100,
    B01000010,
    B10100101,
    B10000001,
    B10011001,
    B10100101,
    B01000010,
    B00111100 },
  walls_bmp[] =
  { B10100000,
    B10110000,
    B10010000,
    B11011100,
    B01000100,
    B01110100,
    B00010100,
    B00011100 };

I wouldn't use the .h extension, as this is usually used for header files which have a connection to a .cpp or .c file with the same name.

Thanks, I will edit it the answer.  I take it that it should be a .cpp extension instead of .h extension?

.cpp is a possibility. Personally I prefer .inc


text:
Why is Serial.Write that slower when writing x+1 characters than when it is writing x characters?
Using Serial.Write() in a time sensitive application. I realize that the number of characters being written not only had an impact on the time spent writing in the buffer, but actually had a bigger impact than expected.
Consider the following sketch, which measures the time it takes to write NB_CHARS characters, using Serial.Write(). I sampled 1000 times in order to get a more reliable measure.
#define NB_CHARS 100

void setup()
{
  Serial.begin(115200);

  uint8_t chars[NB_CHARS];
  for (int i=0; i<NB_CHARS; i++)
  {
    chars[i] = 66;
  }

  unsigned long time1, time2;
  unsigned long totalTime = 0;

  delay(100);

  for (int i=0; i<1000; i++)
  {
    time1 = micros();
    Serial.write(chars, NB_CHARS);
    time2 = micros();
    totalTime += time2 - time1;

    delay(10);
    Serial.println("");
    delay(10);
  }

  float meanTime = (float)totalTime / 1000.0f;
  Serial.print("meanTime = ");
  Serial.println(meanTime);
}

void loop()
{

}

Here are some results for different NB_CHARS values:
NB_CHARS: Time (microseconds)
1: 12
2: 18
3: 24
[...]
99: 2912
100: 2997
101: 3082
[...]
199: 11412
200: 11497
201: 11582

It seems that at first, each new character adds around 6 microseconds of delay, but afterward it becomes 85 microseconds. I tested every values from 1 to 100 to find where it changes and it seems to be right after 70:

It seems weird to me.
A quick Google search gets me to this 2013 post by Tom Carpenter. I am not sure how true this is or if it is still relevant.

Part of the trouble is that the Hardware Serial library is so
  incredibly inefficient (I don't think they could have made it less
  efficient if they tried!). After each byte is sent, there is an
  interrupt called which spends a lifetime loading the next byte into
  the buffer to send. The write function also wastes lots of time doing
  calculations as to where to put a character in a ring buffer.
[...]

Why is there such a "long" delay for each character and is there a way to make it faster?
What happens after 70 characters that slows it down even further?

set higher baud rate. at least 460800 baud should go with good connections.

the quote is not true anymore. at bit rates over 500kbit/s the interrupt is not used if the buffer is empty. (bit rate, not baud rate)

The Arduino core has a 64 byte transmission buffer.  Characters are sent out of that buffer by the hardware.
When you blast lots of characters out of serial the first 64 just get put into that buffer - almost instantly.
Once that buffer is full your sketch blocks until the hardware has sent a character to make room.
Since it's sending right from the moment you place the first character into the buffer you get a little more than 64 bytes worth of buffering time (64 + the number that have been sent by the time the buffer fills).
From that point on you are entirely at the mercy of the baud rate you have chosen.
To get more predictable throughput (and maximise the speed) you should bypass the Ardiuno core and manipulate the UART registers directly.

"you should bypass the Ardiuno core and manipulate the UART registers directly". Any good library for this?

@AlexMillette Ummm... no.  That's the whole point.  You will be avoiding any library / core functions and using the registers directly. Read the datasheet.

I think it would be handy @Majenko if you could point Alex at some example code....

There is a fair chance that once you bypass the core, your code will run as slow as the right part, and not as fast as the left one.

Yes. At best, more efficient code will let you do more stuff in between sending the chars. But you can't speed up sending the chars. But maybe "do more stuff in between" is what you need.

For smooth serial communication you really want an MCU with DMA to do the transfers for you...

Your "long" delay is not too long at all.
If you measure the throughput of the right part, you have about 30 characters sent in 2500 microseconds. This means the rate is roughly (30 * 8 * 1,000,000) / 2500, which is 96,000 bits/sec.
It's a bit less than 115,200 you specified, but serial protocols use service bits, so in fact you'd have to replace 8 with 10. This gives 120,000 baud which is within measurement error.
The left part is actually much faster than you required, but as @Majenko correctly noticed, this is thanks to buffering. Apparently, you can increase the buffer size up to 256 bytes. If you need consistency instead of maximum speed, you may decrease the buffer size instead.
Finally, you should test performance in a realistic scenario. In your example you just keep sending data in a loop; obviously you soon fill up the buffer and run into port speed limit. However, in real life your loop would probably read some input, do some calculations, then send the result. In this case, you'd be using the buffer more effectively.

Majenko already gave the right answer: as soon as the buffer is full, everything falls apart (for a time sensitive application).
It is possible to enlarge the serial buffer for your project. However, in the future, with a new Arduino version, you might have forgotten that you have changed a library and the project does no longer work that well.
I want to point out that you can use the usb serial port that is inside some microcontrollers and processors. That is not a real serial uart with a baudrate and a clock, but a usb CDC device.
A atmega32u4 and the arm m0+ have such a usb CDC serial port.
The arduino leonardo, micro and pro micro use the atmega32u4.
The arduino zero, m0, mkr series use the arm m0+ processor.
What about these numbers:
NB_CHARS: Time (microsec)
1: 6.93
2: 7.60
3: 8.06
[...]
99: 153
100: 154
101: 152
[...]
199: 1685
200: 1624
201: 1705

That is with a arm m0+ processor. I used the same sketch as you, but I changed the Serial to SerialUSB.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why is it considered bad practice to use the 'new' keyword in Arduino?
I previously asked this question:
Is it required to delete variables before going to sleep?
On that question, @Delta_G posted this comment:

... Really on a microcontroller I would create the object in a smaller scope and try to do everything in my power to avoid having to use new or any other form of dynamic allocation. .... etc.

That comment got three likes and when I google about dynamic allocation using Arduino, everyone also tries to stay away from that. In summary from all the research I did, my conclusion is now Do not allocate memory unless you really really have to.
I am using the Visual Studio IDE to create my C++ libraries that I intend to use with Arduino. On the Arduino IDE I just reference those libraries and the code compiles great. Visual Studio is very powerful and it enables me to create really nice code, because I can test it on my computer before running it on Arduino. For example, I created this library:
// MyQueue.h

typedef struct QueueItem
{
    void* item;

    QueueItem* next;

    QueueItem()
    {
        item = nullptr;
        next = nullptr;
    }

} QueueItem;

class Queue
{
public:
    unsigned char count;        /* Number of items on queue */
    QueueItem* first;           /* Points to first item on the queue */

    Queue()                     /* Constructor */
    {
        count = 0;
        first = nullptr;
    }

    void enqueue(void* item)    /* Enqueue an object into the queue */
    {
        count++;

        if (first == nullptr)
        {
            first = new QueueItem();
            first->item = item;

            // Log message because we are using the "new" keword. We need to make sure we dispose QueueItem later

            #ifdef windows

            std::cout << "Creating " << first << endl;

            #endif // windows
        }
        else {

            // Find last item
            QueueItem* current = first;
            while (current->next != NULL)
            {
                current = current->next;
            }
            QueueItem* newItem = new QueueItem();
            newItem->item = item;

            // Log message because we are using the "new" keyword. We need to make sure we dispose QueueItem later
            #ifdef windows
            std::cout << "Creating " << newItem << endl;
            #endif // windows

            current->next = newItem;
        }
    }

    void* dequeue()
    {
        if (count == 0)
            return nullptr;

        QueueItem* newFirst = first->next;

        void* pointerToItem = first->item;

        // Log message we are deleting an object because we created it with the 'new' keyword
        #ifdef windows
        std::cout << "Deleting " << first << endl;
        #endif // windows

        delete first;
        first = newFirst;
        count--;

        return pointerToItem;
    }

    void clear()                /* Empty queue */
    {
        while (count > 0)
        {
            dequeue();
        }
    }

    ~Queue()                    /* Destructor. Dispose everything */
    {
        clear();
    }
};

Now on my Arduino sketch, I can have the following code if I reference that header file.
typedef struct Foo
{
    int id;
} Foo;

void someMethod()
{

    Queue q;

    // Create items
    Foo a;
    a.id = 1;

    Foo b;
    b.id = 2;

    // Enqueue a,b and c
    q.enqueue(&a);
    q.enqueue(&b);

    // Deque
    Foo * pointerTo_a = (Foo*)q.dequeue();
    int x = pointerTo_a->id; // =1

    Foo * pointerTo_b = (Foo*)q.dequeue();
    int y = pointerTo_b->id; // =2

    // Error
    Foo * test = (Foo*)q.dequeue();
    // test == null pointer
}

Most people say do not use void pointers. Why!? Because I am using void pointers I can now use this queue class with whatever object I want!
So I guess my question is: Why is everyone trying to stay away and avoid code like this?
I am using the NRF24L01 radio module to send messages to several Arduinos. It is convenient to have a queue of messages to be sent. I would be able to code the same program without allocating memory and avoiding the new keyword. But that code will look ugly in my opinion.
In this quarantine I decided to learn C++ and that has changed the way I code Arduino. The moment I learned C++ I stopped using the Arduino IDE. I been a backed developer for 12 years, and that is the reason why I learned C++ in a couple of months. Arduino is just a hobby for me. I am still very new to microcontrollers and I will like to understand why people stay away from the full power of C++ when it comes to microcontrollers. I know I have only 2 kilobytes of RAM. I will not be allocating that much memory. I still want to take advantage of the C++ programming language by using the new , delete , poineters and destructors`. I want to keep using Visual Studio to write powerful C++ libraries.
In C++ I write interfaces like this
// Note I use uint32_t instead of 'unsigned long' because an unsigned long is different size on Windows than on Arduino. Also I use an unsigned short instead of an int because an unsigned short is the same size on Windows and Arduino.

class IArduinoMethods
{
public:

    // Unsigned long in Arduino
    virtual void delay(uint32_t delayInMilliseconds) = 0;

    virtual void print(const char* text) = 0;

    virtual uint32_t millis() = 0; // Get elapsed time in milliseconds
};

And I then implement the classes like this. For example, this is the class that I will use when testing my code on a windows computer:
// Class to be run on Windows.
class ArduinoMockWindows : public IArduinoMethods
{
public:

    // Inherited via IArduinoMethods
    virtual void delay(uint32_t delayInMilliseconds) override
    {
        // This code will be different on Arduino, and that is why I need this dependency
        Sleep(delayInMilliseconds); // Windows
    }

    virtual uint32_t millis()
    {
        //clock_begin = std::chrono::steady_clock::now();

        std::chrono::steady_clock::time_point now = std::chrono::steady_clock::now();
        auto duration = now.time_since_epoch();
        // etc..
        return someDuration;

    }

};

Because a Windows computer cannot send NRF24 radio messages, I can implement an interface (dependency) that will write to a file, for example instead of sending a real radio packet just for testing.
The caveat is that my libraries will require these dependencies. For my library to work, I will have to pass it an object of type IArduinoMethods and INrfRadio. If I am running my code on Windows I will pass it a class that will implement those methods that can run on windows. Anyways the point is not to show how C++ works. I am just showing how I use pointers and allocate memory for a lot of things.
Because I allocated memory I was able to test my library on Windows and on Arduino for example. I may also create unit tests. I see so many benefits by allocating memory. If I am organized and remember to free the objects I no longer use, I can gain all this benefits. Why people do not code like this when it comes to Arduino?

Edit 1

Now that I understand how heap fragmentation works, I know I have to be careful when using the new keyword.
I hate when people do what they are told to do without understanding how things work. For example, the answer https://arduino.stackexchange.com/a/77078/51226 from Why is the queue library in this question for starters?. There are going to be times when a ring buffer works better and other times when the new keyword works better. Probably the ring buffer will work best for most cases.
Take the following scenario where you only have 1 KB of memory left.

There is a hierarchy of nodes where a node have a child and a sibling. For example, node A can have child B and sibling C. Then child B can have another child, etc.

(I will be storing this in memory)

I have a queue of work that needs to be done.

(I will have to store this work somewhere)

I will have a queue of events

(I will have to store this somewhere)
If I use what most people say I should do then I will be have to:

Reserve 500 kB to be able to store nodes (I will be limited to n number of nodes)

Reserve 250 kB for the queue of work that needs to be done.

Reserve 250 kB for the queue of events.

This is what most people will do and it will work great with no problems of heap fragmentation.
Now this is what I will do

Ensure that everything that I allocate is of size 12 bytes. A node only has its id (unsigned int), child (pointer), type (unsigned char), etc.. with a total of 12 bytes.

Ensure that all the work that will be enqueued is of size 12 bytes as well.

Ensure that all the events that will be enqueued is of size 12 bytes as well.

Now if I have more work than events, this will work. I just have to program in my code that I never allocate more than 70 items. I will have a global variable that has that count of allocations. My code will be more flexible. I will not have to be stuck with strictly 20 events, 20 work and 30 nodes. If I have fewer nodes then I will be able to have more events. **Anyways my point is that one solution is not better than the other. There are going to be scenarios when one solution is better.
In conclusion, just understand how heap fragmentation works and you will gain a lot of power by using the new keyword. Do not be a sheep and do what people tell you to do without understanding how things work.**.

Edit 2

Thanks to @EdgarBonet, I ended up storing Nodes on the stack. Here is why:
I have a hierarchy of nodes that can be represented as:
typedef struct Node
{
   unsigned short id;
   Node * sibling;
   Node * child;
} Node;

As you can see every node is only 6 bytes. That is another reason why I did not care to much about allocating Nodes at the beginning. If I allocate this node on the heap I will be losing 2 more bytes (33%) for every allocation because on every allocation the size of the node has to be stored. As a result I created these two methods and a buffer:
// For this to work a node can never have an id of 0 !!!

Node nodeBuffer[50];                     /* Buffer to store nodes on stack */

Node* allocateNode(Node nodeToAllocate)  /* Method to store a node */
{
    // Find first available spot where a node can be saved
    for (char i = 0; i < 50; i++)
    {
        if (nodeBuffer[i].id == 0)
        {
            nodeBuffer[i] = nodeToAllocate;
            return & nodeBuffer[i];
        }
    }
    return nullptr;
}

void freeNode(Node* nodeToFree)          /* Method to delete a node */
{
    nodeToFree->id = 0; // If the id of a node is 0 this is my convention of knowing it is deleted.
}

And on my code I used to have things like:
Node * a = new Node();
a->id = 234423;
// ....
// .. etc
// ..
delete a;

Now I just have to replace that code with:
Node * a = allocateNode({});
a->id = 234423;
// ....
// .. etc
// ..
freeNode(a);

And my code works exactly the same without having to use the new keyword. I thought it was going to be complicated to refactor the code and create a buffer.
I made this modification because I wanted to be able to store more nodes on my code. By loosing that 33% I was not going to be able to create that many. If I only allocate objects of the same size and I do not allocate that many it is perfectly fine to use the new keyword. > Also in the case of the queue I will allocate and delete objects very fast. Because the objects will not persist on memory for too long, and the chances of having heap fragmentation are very low.

short answer: there is no heap management to defragament it. so you can allocate at setup or create a pool of objects, but don't delete/free the memory.

Because you have very limited memory space and no operating system to clean up behind you.  That means you have to do a lot of things manually that you wouldn’t ordinarily have to do like watch for heap fragmentation.  Add to that the fact that an Arduino is a one program at a time device and that sort of eliminates most of the advantage to using heap memory.  You got nobody else to share with so it’s ok to be stingy.  But it’s mostly about not having the OS to keep things clean for you.

This question looks like a duplicate of Is using malloc() and free() a really bad idea on Arduino?. 2. Before dereferencing the pointer returned by new, you should check it is not nullptr. Otherwise your code is bound to crash on memory exhaustion. 3. For this particular use case, a ring buffer is presumably the easiest, memory-safe option. This is what Serial uses to enqueue the bytes.

-1 for "Do not be a sheep and do what people tell you to do."

You can be a sheep then @null and do what people tell you to do ;) being very null .

I can't call this an answer, because it's not targeted at the Arduino, but on some embedded platforms with very little memory for things like heap management, maloc is implemented as uint8_t* rval = __frontier; __frontier += sz; return rval and free is a no-op.  Needless to say, on such a platform, using heap memory this way goes poorly!

Of course you are right, but as shown by your edit, using dynamic allocation safely brigs some uncomfortable constraints. I would not recommend it to a beginner (i.e. most of those who ask questions here). It also has its own memory cost: two bytes per allocated chunk, plus some padding, because it is more than unlikely that all your objects happen to naturally have the same size.

@CortAmmon: Do you have an example? The avr-libc's malloc() does heap management, and it runs on devices as small as the ATtiny13A (1 KiB flash, 64 bytes of RAM). Not that it would be very sensible to use it on such device though...

@EdgarBonet: yes I am wasting 2 bytes for every allocation I perform :/ . I wish the heap stored the sizes in a char instead than using 2 bytes since I am never storing anything larger than 255. Because I am storing very few objects I will use this keep using this approach. Is there a way to instruct the compiler to store the sizes of heap objects using one byte instead of two?

@TonoNam You could compress that by using simple segregated storage like boost.pool does.  That lets you use arbitrarially sized chunks (at the cost of fragmenting if you don't free them uniformly, of course).  The extreme version of this, with 1 large chunck, of course, is the typical storage reservation approach you document above.

@CortAmmon thanks that will be great to know but I thing I am complicated thins a lot. I just ended up storing the items on a buffer. I thought it was going to be hard to refactor when it was not. Thanks for the help

@TonoNam You got my downvote for that last paragraph of your first edit. You are aggressively defending your original solution, that there are scenarios in which it can be the better solution - as if anybody claimed that this wasn't the case. I find this sort of  "I could have gotten away with what I had in the first place" attitude somewhat childish. As you have done what people told you (and rightfully so, because you got some good advice), your recommendation is plain false: doing what people tell you to do is actually key to understanding things.

Re:  " If I only allocate objects of the same size and I do not allocate that many it is perfectly fine to use the new keyword".   But what advantage does it gain you at that point?  If the objects are a set size and you have room for a set number of them then why would you not just statically allocate that much?   It's about more than just that you could, think about the consequences of it going wrong.  I guess you don't really learn that until it bites you.  But one day you'll be telling people the same thing we're telling you because you'll have been bit by it.

Re:  "In conclusion, just understand how heap fragmentation works and you will gain a lot of power by using the new keyword. Do not be a sheep and do what people tell you to do without understanding how things work.".  It's not about being a sheep.  You say you hate when people blindly follow rules.  I hate when someone who has to ask this kind of question thinks I'm just a sheep following.  I know what I'm doing.  I know where the danger is.  You think you've outsmarted it.  I've seen a thousand before you who thought the same.  They all learn the same hard lesson eventually.  You will too.

Most Arduinos (like the Uno or Nano) have very few RAM, thus you first need to make sure that you never allocate too much memory. Dynamically allocating memory can also lead to heap fragmentation (heap being the part of memory where dynamic allocation happens).
In most cases you would want to allocate memory of different sizes (for example arrays of different sizes) or just different objects (with each having it's own size) (!!! This is the key point here). Then you are going to delete some of these objects which will create holes inside the memory. They can be filled again with objects with the same or less size. As time passes and more allocation and deleting happens these holes tend to get smaller, up to the point where none of your new to allocate objects can fit in there. That memory then is unusable. This phenomenon is called heap fragmentation.
These holes appear naturally also on a PC but there are 2 key differences:

The Arduino has such little RAM that the holes can fill up your memory very very fast.

While the PC has an operating system, which manages the RAM (defragmenting it or putting unused stuff away into a paging/swap file), the Arduino does not have an OS. So noone keeps an eye on the real available RAM and noone tidies up the memory once in a while.

That does not mean that you cannot use dynamic allocation on an Arduino, but that is very risky depending on what exactly you are doing and how long the program should work without failing.
Considering this big caveat, you are very limited on how to use dynamic allocation. Doing it too much will result in very unstable code. The remaining possibilities, where it might be safe to use it, can also be easily done with static allocation. For example take your queue, which is basically a linked list. Where is the problem with allocating an array of QueueItems at the start. Each item gets a way to determine if it is valid. When creating a new item you just pick the first element in the array, which has a non-valid item, and set it to the desired value. You still can use the data via the pointers, just as before, but now you have it with static allocation.
You might find that the code looks uglier that way but you need to adapt to the platform that you use.
Note that this does not apply when you are going to create only objects with the same size. Any deleted object will leave a hole where any new object can fit into. The compiler uses that fact. So in that case you are safe. Just every object that you dynamically create in your program needs to be the exact same size. That of course also includes objects that are created inside different libraries or classes. For this reason it can still be a bad design choice, as you or others (if you want to publish your code), may want to pair your library with other code.
Another way to be safe is to only create and delete objects in closed cycles, meaning, that a created object needs to be deleted before the next object is created. Though that is not fitting for your application.

'Bigger' microcontrollers, for example the non-Arduino boards with the ESP32, have much more memory. Thus the use of dynamic allocation is not that bad on them. Though you still don't have an OS to manage the RAM.

Thanks a lot for the explanation. I already coded a lot of libraries using pointers. So I guess that now I have to be careful on the number of objects I alocate correct? The code works now because I am allocating very few objects and there is always room in the heap for new objects. As long as my allocations are small size and I do not allocate to many objects I should be good correct?

is there any way to de-fragment memory manually?

@TonoNam Yes, that is correct. You need to consider the size, the rate of allocation and how long the code should run. Assuming a constant rate of allocation, the code might run good for a day. But if you want it to run like a month, that could be way different. If you want to use dynamic allocation you will need to just test this out.

@dandavis I don't know of a way, other than just deleting all dynamic objects and recreating them. If you do this, I would guess, that then you again have a linear bunch of memory without holes in it (effectively defragmented). Though I'm not enough of an expert in the compiler, that I could guarantee that. Also I don't think, that this is really an option most of the time.

"Most Arduinos (like the Uno or Nano) have very few RAM"  Plus, Arduino programs are typically controllers that are expected to run unattended; they don't run to completion and get re-started some other time like most desktop programs. So even a large-memory PC using new/delete allocation would eventually fragment its memory, too. It would just take (a lot) longer.

@dandavis: The only way to defrag new/delete allocated memory is to don't. ;-) What I do instead, is to declare an array of buffers of a fixed size, and hand out one of those, no matter how much the caller asked for (up to the buffer size, of course). That mechanism won't cause fragmentation because you never split blocks, so every hole is re-usable. You do have to know - or guess well! - the maximum number of buffers your application will ever need and declare a few more than that for safety.

@dandavis Yes, you can de-fragment memory. It's called "memory management," and while an OS will have a sophisticated program for this, you can write a simple one. The biggest caveat is, you have to update all pointers to an object once you move it, so you have to be able to find every pointer to it from your memory management routine. I don't know whether such a program would fit reasonably on an Arduino. Haven't tried.

I would suggest using the ETL, which is like the STL but for embedded programming. It provides all the vectors, queues, linked lists, maps, iterators, algorithms, and other useful stuff from the C++ standard library, without using dynamic allocation at all. (You have to specify the maximum size of your structure, and everything will be handled automatically inside it). Note: I have no affiliation with the ETL, I'm just a satisfied user of it.

For a Queue implementation, it's probably better to have a pointers or indexes of the first and last item, than giving each item a valid flag. That way access becomes O(1), it's much easier to wrap around the array to reuse the memory and there are no weird situations like "holes" you weren't trying to express.

I would add to this that dynamic allocation is often slow and somwhat unpredictable in terms of timings (due to fragmentation and other effects), so if you're writing performance or timing sensitive code, it's often best to avoid it even if memory usage is not an issue.

Dynamic allocation is generally discouraged in embedded applications because you cannot guarantee that you do not exceed (attempt to allocate more than) the available memory. Static allocation will generally have this guarantee although out-of-memory bugs may still be possible.
Additionally, far fewer services or tools are available to automatically manage and mind the memory for you. Any service that does so will consume computational resources.
This means that you inherently create a mechanism in your device that would cause a memory (heap) overflow and possible undefined behavior (UB). This is true even if your code is bug-free and has no memory leaks.
In non-critical, exploration, learning, and prototype applications this may not be important.
Consider that without careful consideration undefined behavior can result in hardware failures and unsafe performance, for example if the device reconfigures GPIO through an errant write to the correct registers during a crash.

Yes Dynamic allocation is generally discouraged. +1 for using the word generally. In some cases it might work. I will just have to keep track of how many objects I allocate and ensure that all of them are of the same size.

For starters, fix your library
As noted by @crasic, dynamic memory allocation is generally not recommended for embedded systems.  It may be acceptable for embedded devices which have a larger amount of free memory - embedded Linux is commonly used, for example, and all Linux apps/services will tend to use dynamic memory allocation - but on small devices such as an Arduino there simply is no guarantee that this will work.
Your library illustrates one common reason why this is a problem.  Your enqueue() function creates a new QueueItem() but does not check that the allocation succeeded.  The result of failed allocation may either be a C++ bad_alloc exception, or it may be returning a null pointer, which when you reference it will give a system memory access exception (SIGSEGV signal in Linux, for example).  It is nearly universal in Linux and Windows programming to ignore memory allocation failure (as encouraged by most textbooks), because the massive amount of free RAM and the existence of virtual memory makes this very unlikely, but this is unacceptable in embedded programming.
More generally though, as @crasic says, memory fragmentation can leave even non-buggy code unable to allocate memory.  The result will be a failure to allocate memory, but the code will at least know this has happened and will probably be able to continue.
But better, use a fixed-size FIFO queue instead
Your code relies on dynamic allocation to add and remove elements in a queue.  It is perfectly possible (and equally easy coding-wise) to create a fixed-size array for the queue, so the various failure modes of dynamic allocation simply do not apply.  An item to be queued is simply copied into the next free queue slot, and a queue slot is marked free when it has been used.  (Don't forget to use a mutex when adding and removing items from the queue, because adding and removing will often be called from different places.)
The queue can be made whatever size you feel is appropriate (allowing for how much RAM you have).  With a fixed size, you are forced to make a design decision on what should happen if the queue overflows - do you delete the oldest data to make room for the new value, or do you ignore the new value?  This may seem an unwelcome new feature, but it is a good thing, because the third option which you've currently got is that your code goes "Aaaarrggghhh I don't know what to do!" and crashes fatally, and we don't really want that.

A fixed-size FIFO queue is typically implemented as a ring buffer.

@EdgarBonet Yes it is.  The OP should be able to find plenty of existing code for FIFOs online though, so I guess I'm more interested in helping them getting the concept than exactly how they do the implementation.

The FIFO fix sized as ring buffer is a great sollution. There are going to be times when the ring buffer is better (probably most of the time) and there are others when it is not. Take a look at the edit of the question. If you keep track of allocations and you never exceed the count of allocations all of the same size you should have no problem allocating a new queue item.

Exactly graham your answer is probably the best solution for most of the cases and also the safest. But for cases when I am building a hierarchy of nodes it will be harder to create with a buffer. If I am already allocating Nodes I guess I can keep allocating them using the queue. My point is that if you know how things work it is ok to not go with the convention. Also the probability of an alloc not working is very small since I am not using interupts or running my code in different places. For the rest of my projects I will probably use your solution.

@TonoNam You are assuming there that your allocation granularity goes down to at least 4 bytes. If the granularity is 8 bytes at best, then each 12-byte allocation will actually occupy 16 bytes of RAM. So your heap-based code would only be able to store 3/4 of what could be stored with a statically-allocated queue. Plus whatever overheads are involved in running the heap, which can be significant when you're allocating small amounts of data (and 12 bytes is usually "small"). You can't just divide RAM length by structure length and assume that's how many you get.

That is the reason why I made edit number 2 on the question Graham. Thanks for all the help.

I'm adding this not so much to add to the answer as to add some real world implications for those that may be down this particular rabbit hole.  It's nice to talk about what could happen theoretically, but a new programmer may still be tempted to think that he can out-think these limitation and still do something useful.  Here is some real world talk about why that is foolish even if you are capable.
Let's say we're developing code on an Arduino UNO.  We've got 2K of RAM to work with.  We have a class that loads a list of names, maybe it's a building access device or something.  Anyways, this hypothetical class has a name field to store someone's name.  And we decide to use the String class to hold the name as a String.
Let's say after our program is all there and doing what it's doing that there is 500 bytes left for this list of objects, each with a name field that could be of varying length.  So we run along nicely for years with a crew of 14 or 15 people with an average name length of 30 characters or so.
But one day a new guy signs up.  And his name is really long.  Let's say it takes 100 characters.  Even if we're smart coders and only have one copy of this String in memory, we've got it there and suddenly it doesn't fit.  Now a program that has worked for years suddenly fails.  And nobody knows why.
So the solution is easy right?  Enforce a maximum limit on the length of the name.  Some simple code that checks the name length and we can write a great piece that still allows the variable name length and won't let you create a new user if there's less than that much left.  Seems simple enough, but then Becky in accounting gets married and her last name changes from Smith to Wolfeschlegelsteinhausenbergerdorff and suddenly our program breaks again for no reason.
So the solution is simple right?  We'll enforce a maximum length and we'll be sure to reserve enough memory for each object that it can afford to have the maximum length name.  And that's something that we do most efficiently without dynamic allocation since we already know the size of the object.
I hear what you're saying, "but Delta-G, if we have all those short names in there, why are we going to waste all that memory on them when we don't have to.  Let's only save space for a long one if we have a long one."  And I like your thinking, this is good thinking.  But it doesn't help anything to save that memory.  If you do save some, what are you going to do with it?  If your program uses it, then there is no longer room for the longer case and suddenly you have to enforce an even shorter maximum length to accommodate that usage.
Say for instance we have 500 bytes of room and we enforce a maximum length of 50 bytes for 10 users.  And let's say that when the names are short we want to let the program use some of that saved space.  If the program can encroach 100 bytes into that space when the names are short, then why wouldn't the same situation happen with long names?  So really since the program can use all but 400 bytes then there's really only 400 bytes of room and we have to enforce a 40 byte maximum for 10 users or 50 bytes for 8.
Since you're going to have to make that sacrifice anyway, then it only makes sense to remove the work of dynamic allocation and just make things fixed sizes or use fixed size buffers.
If we had a PC with gigabytes of memory, we wouldn't even be thinking about this.  But on an Arduino UNO with 2K bytes of memory it can become a big problem really fast.
The other problem is that these bugs are so insidious.  If out of memory bugs just caused a simple crash and the thing no longer worked then it wouldn't be nearly so scary.  But that's not how out of memory bugs work on a microcontroller.  It all depends on how things are arranged in memory by the compiler.
These bugs often manifest as something that seems to work most of the time but has some funny bugs nobody can explain.  Maybe there's only a problem if someone has a name that is exactly 26 characters long and tries to open the door on a Wednesday.  Or maybe the issue will only arise if Becky logs in immediately after Bob.  Maybe it just garbles three letters on the screen but other than that everything works.  Maybe with a different name changed that suddenly turns into our lock opens for anyone.  There's no guessing or explaining out of memory bugs.  So we have to be super careful to avoid even the remote possibility of running into one.
And that's why we avoid using dynamic allocation on small microcontrollers.  At the end of the day, there's nothing you can save with it, and even if you could the consequences of getting something a little wrong are terribly frustrating.  With these types of programs you almost always have to end up enforcing some sort of limit on everything and once you're enforcing limits there's no use for dynamic allocation anymore.

Again in your real life example you are allocating objects of different sizes (names). That will lead to heap fragmentation. If you where to allocate names of the same size and very few of them it is perfectly fine to use heap allocation.

But like I said in the example, if you’re allocating a fixed number of things of a fixed size then why do you want to use dynamic allocation to do that?  What would be the advantage at that point?

That if I have multiple queues I do not have to create one buffer for each. There might be cases when allocation is helpful. My point is that if you understand how it works it is ok to use it. Probably most of the time it makes no sense to use it but not never.

Yeah not never, but not for what you're thinking either.  You run into the same issue.  If you have multiple queues and you want them to be able to share a buffer space then it would certainly be easier to have one buffer allocated and a single class that is controlling adding and removing.  I'm thinking probably anything that can add to the queue needs to inherit some base class that allows you to have code in one place doing it.  That is exactly the use case I'm on about where you think you know what you're doing and one day some corner case you never suspected bites you.

I guess it's one of those things that you just can't truly learn any way but the hard way.  But at the end of the day, if you want shared buffer space you must put a limit on its size and if you're going to do that then you might as well allocate it all and let your code control putting things in and out.

The one good exception I can think of is when you have a class you need to instantiate later in the code.  Maybe you need some user input before you can build the instance.  So you just have a pointer and then later in code you can construct a new object with new and set that pointer.  But in that case the plan is to create the object early on in the program and leave it there for the duration.  In this case you never plan to ever free or delete the object and create another.


text:
Serial printing from flash memory (F() macro, PROGMEM, sprintf_P, SPTR)
I have been testing various techniques on an Arduino Uno  for serial printing from flash memory instead of RAM. At the same time, I have been using the freeMemory() function from Adafruit.
I created a program that prints the word "test", with various methods:
Simple print:
Serial.print("test");
Serial.print(freeMemory());

This call reports free RAM = 2267
Using the F() Macro:
Serial.print(F("test"));
Serial.print(freeMemory());

This call reports free RAM = 2299
Using sprintf_P()
char buf[100];
sprintf_P(buf,PSTR("test"));
Serial.print(buf);
Serial.print(freeMemory());

This call reports free RAM = 2267
Using strcpy_P()
 char buf[100];
 strcpy_P(buf, (PGM_P)F("test"));
 Serial.print(buf);
Serial.print(freeMemory());

This call reports free RAM = 2267
Using strcpy_P and const char[] PROGMEM
char buf[100];
const char buff[] PROGMEM = "test";
strcpy_P(buf, buff);
Serial.print(buf);
Serial.print(freeMemory());

This call reports free RAM = 2267
The basic point of this question is to gather all the possible flash memory printing techniques. Also it would be really useful if you could explain in detail what exactly each of the keywords F(), (PGM_P)F, PSTR, const PROGMEM does.
IMPORTANT EDIT, 1:
Depending on where and how I use the call Serial.print(freeMemory()); the results change. For example, at the last example (PROGMEM, strcpy_P) when I included another Serial.print(freeMemory()); before the call of the printing functions, then both freeMemory() calls report 2299!
Another interesting problem is that the freeMemory() reports 2299 as freeRam, while the SRAM of my Arduino Uno is only 2 kbytes.
EDIT 2:
Following the suggestion in comments, I changed "test" to "testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttest". Now all different methods report a free ram of 2299, including the simple Serial.print().
freeMemory():
#ifdef __arm__
// should use uinstd.h to define sbrk but Due causes a conflict
extern "C" char* sbrk(int incr);
#else  // __ARM__
extern char *__brkval;
#endif  // __arm__
int freeMemory() {
  char top;
#ifdef __arm__
  return &top - reinterpret_cast<char*>(sbrk(0));
#elif defined(CORE_TEENSY) || (ARDUINO > 103 && ARDUINO != 151)
  return &top - __brkval;
#else  // __arm__
  return __brkval ? &top - __brkval : &top - __malloc_heap_start;
#endif  // __arm__
}

Nice report, but I miss a question; I guess you don't use a Uno as that model has 2 KB only.

All those routines use 100 bytes on the stack and nothing else, except maybe transient space within the functions you call which is also on the stack. None of that is relevant to storing data in and reading data from flash. All the things you ask about are macros. You can find what they expand to in the source code of either the core or the compiler.

I suggest you try with a message much longer than "test". There are many details in the compiler optimization process that can make small variations in the memory consumption. Those can make your measurements hard to interpret with such a small string.

Will try immediately, and report back with the results. @MichelKeijzers Strangely enough, I am using an UNO. Any ideas?

@EdgarBonet your suggestion really seems to have an impact. I changed "test" to "testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttest". However the results seem to be even weirder, as all different methods (including the simple Serial.print) report a free RAM of 2299. Let me also point out that this is the freeRam I get when I only call the freeMemory() in my code.

First, the freeMemory() function that you used from Adafruit was originated from a GitHub repository.
I don't know about the __arm__ implementation, but for the AVR one, it is incomplete for handling the corner case when the program does not use the malloc() function. If your program never uses malloc(), the formula used in freeMemory() will produce the wrong result. A simple corrected version for AVR that I used is here:
extern unsigned int __heap_start;
extern char *__brkval;

int freeMemory() {
  char top_of_stack;

  if (__brkval == 0) {
    Serial.println(((int)&top_of_stack - (int)&__heap_start));
  } else {
    Serial.println((int)&top_of_stack - (int)__brkval);
  }
}

explain in detail what exactly does each of the keywords F(), (PGM_P)F, PSTR, const PROGMEM.

PSTR("string literal")
PSTR came into Arduino from part of avr-libs (a C library) defined in avr/pgmspace.h. It is actually a macro, not a function as many think it was, and is defined as:
#define     PSTR(s)   ((const PROGMEM char *)(s))

It looks a little bit intimidating, but it is actually quite simple. It not only tells the avr-gcc that the string literal s should be kept in program memory (i.e., flash memory in the case of Arduino), but it allows the program to convert it into a const char *s PROGMEM variable, so that it can be pass around into some PROGMEM-aware functions as a parameter.
 // This is a PROGMEM-aware function in C
 printMsg_P(const char* str) {
     char buf[strlen_P(str)+1];
     strcpy_P(buf, str);
     puts(buf);
 }

 // This handles the normal string array function
 printMsg(const char* str) {
     puts(str);
 }

F("string literal")
F() is also a macro, but it is not part of the avr/progmem.h. It is actually part of the String class defined in WString.h together with a definition of a class called __FlashStringHelper:
class __FlashStringHelper;
#define F(string_literal) (reinterpret_cast<const __FlashStringHelper *>(PSTR(string_literal)))

To understand the F() macro and the difference between F() and PSTR() take a little bit explanation.
As you can see from the definition of F(), it casts a string literal into a const char* variable using PSTR(), and then recast it into a class __FlashStrngHelper. If you further look at the __FlashStringHelper, it is actually an empty class without any construct or method. It is literally empty, so why is this useful? And why can't it just pass the string literal around by just using PSTR()?
This has to do with C++'s function overload. Function overload is a feature of C++ where multiple methods could have the same function name, but each with different parameters. Within all the Serial.print() methods, there is a method Serial.print(const char* str) that will take in a string literal and print it to serial monitor, but it is not PROGMEM-aware. If you pass in a PROGMEM string literal Serial.print(PSTR("string literal"));, the same method will accept what you pass in because it meet the type checking (remember PSTR is actually const char*), but it will not print out the correct string for PSTR("string literal"), because it simply expecting a normal string.
In C, as we see before, we solve the problem by using two functions with different names for handling normal string and PROGMEM string, but Serial is a C++ class. It needs another function overloaded method to be PROGMEM-aware, and PSTR() is clearly not the solution as we just mentioned.
WSting.h and Print.cpp create an overload method that is PROGMEM-aware by accepting a class as parameter, so that you can pass a F() wrapped string literal which has a data type of class __FlashStringHelper into the Serial.print() and get the print out correctly from program memory. See the source code yourself.
Serial.print(const __FlashStringHelper *ifsh)

So in summary, the F() and PSTR() are alike. Both are telling the compiler to keep the string literal in the program memory, and allow the string literal to be pass around into some function as parameter. But it was designed for different purpose from different programming paradigm (C versus C++). F() is Serial.print()-friendly, while PSTR() is not. If you are getting PSTR() in a function and yet you want to send it to Serial.print(), you will have to explicitly cast it into __FlashStringHelper like this:
Serial.print((__FlashStringHelper*)PSTR("test"));

PGM_P
PGM_P is a macro defined as:
#define     PGM_P   const char *

Whether you want to use PGM_P or const char * is a personal preference. For example, this is copy from PROGMEM (Arduino reference):
const char *const string_table[] PROGMEM = {string_0, string_1, string_2, string_3, string_4, string_5};

can be written as:
PGM_P const string_table[] PROGMEM = {string_0, string_1, string_2, string_3, string_4, string_5};

This is a lengthy explanation, but I hope this will help with better understanding the subject and therefore be able to use the PROGMEM and all the associated macros effectively.

Re “PSTR [...] is defined as [...]”: This definition is introduced by the comment “The #define below is just a dummy that serves documentation purposes only.” The actual definition is somewhat more convoluted.

explain in detail what exactly does each of the keywords F(), (PGM_P)F, PSTR, const PROGMEM.

F() wraps the string literal in PSTR(...) and then casts it to __FlashStringHelper.  This forces it to remain in flash and also gives it a type that the Arduino core can identify as a string in flash for overloading.
PGM_P is a "Pointer to ProGraM mememory". This is the type that all the _P variants of functions require. It's basically const char * provided by avr/pgmspace.h
const PROGMEM is actually two things. const means you're not changing the value, and PROGMEM is another name for __ATTR_PROGMEM__ which itself is an alias for __attribute__((__progmem__)) which is a flag for the compiler to tell it to keep the variable in flash and not copy it to RAM.

So F() and PGM_P are used when you are passing a string literal to a function that expects one of those formats. PROGMEM is used when you're setting up a constant "variable" (oxymoron there, I know...) that points to data in flash.

Could you please explain: sprintf_P(buf,PSTR("test")); ? Is this wrong? Should it be like this: sprintf_P(buf,PGM_P("test")); ? Also what about this command: strcpy_P(buf, (PGM_P)F("test")); which combines both PGM_P and F()?

(PGM_P)F("test") is what I usually use.

(PGM_P)F("test") is equivalent to PSTR("test"). What happen is that F("test") cast the string literal into a class  __FlashStringHelper, and then recast it to PGM_P which basically is a macro as const char*. This is the same as PSTR("test") that converts a string literal s[] to `const char* s variable.


text:
Issues with Stacking Ethernet shield and USB Host shield
I want to stack Ethernet shield and USB host together with my Arduino MEGA 2560.
Each of these shields works fine separately.
Is it necessary to contact bootloader pins with each shield and arduino board?
I have connected and tried, but the problem exists.
only one shield works fine at once.(always the sheild that directly connect to the Arduino board).
How do I resolve this problem ? I want to work all 2 shields at once.
these are the sheilds I have used.

Circuits At Home USB Host Shield v2.0
Ethernet Shield v5.0

Thank you

Both shields are SPI devices, and both shields seem to be hard-wired to use D10 as the chip select line.
In order for you to use both shields at once you will have to either modify one of them or connect the shield via jumper wires.
You will need to change the chip select connection on one of the shields from D10 to some other pin that is not currently used, and then modify your software (or the library) accordingly.

Do I have to Modify D10 from one of the shields and as well as change the library? can't I resolve it just by changing the library.

You HAVE to somehow re-route D10 on one of the shields.  You also HAVE to inform the software that it's no longer using D10 but some other pin.  Some libraries provide the ability to pass the CS (or SS if you prefer) pin when constructing the object, and some are hard coded.  If the former you just change the pin number in your sketch. If the latter then you need to change the pin number in the library itself.

Also, I think that there is an hardware bug on the USB Master shield: IC3, the 3-state buffer which converts MISO to 5V, is always active... The enable pin of this 74LVC1G125 (pin 1) should be connected to the pin ss_3V3. Then you can stack this shield with another one.


text:
I2C with an arduino, a PI and an IO expander
im working on a project which requires alo i/o pins. For that reason i bought a pcf8574P 8bit I/O expander which uses the I2C protocol. The problem is how to combine the Rasp PI and the arduino.
For example:
I want the Rasp to send data to the Arduino. Currently i'm using serial but i wanted to switch over to I2C because it better, simpler, and faster however how can i use 2 I2C when the Atmega 328P supports only 1 I2C bus?
The plan goes like this:
The PI is master of the arduino and it sends the data. Arduino is the slave
Arduino is master of the IO expander and the expander is slave
Possible solutions:

Use the Serial protocol to send the data and use the I2C for the expander
Make the arduino master of the PI and the expander which i'm not sure if its possible

http://playground.arduino.cc/Main/SoftwareI2CLibrary

You can have multiple masters on one I2C bus (http://electronics.stackexchange.com/questions/115577/how-does-multimaster-i2c-work-and-its-significance)


text:
Ethernet Shield on Arduino Mini
I would like to connect my custom-made Arduino with an Ethernet shield but I'm not so sure I can.
The problem
My board is using the design from the Arduino Mini which lacks the ATmega16U2 which does the USB-to-serial conversion and adds the ICSP pins. I think you get the point... I have only the ATmega328P on the board. Nothing else.
Is it even possible to work my way around it without adding the ATmega16U2? I can't find the chip anywhere on Ebay.

Are you using an IEthernet shield (using a W5200, W5500 or such like), or do you mean a generic Ethernet shield? Your title and body are inconsistent. Not that important, but a little confusing :-)

Im using the shield with the W5500. Is there any difference with the other chips?

There may be some differences, but I was just trying to clarify whether the "IEthernet", in your title, was a typo or not. I understand now, that it wasn't :-)

You can just connect the pins used on the shield the the corresponding pins on your custom board. Check the Schematic to see what pins are used for what (and which can be left unconnected)
The only thing you need to lookout for is, that the ethernets shield seems to require 3.3v (the shield has it's own 3.3v regulator, so you only need to connect 5v)
That 16u2 doesn't do anything but usb-to-serial. It's not needed for the ethernet shield.
The ISP header is just 5v, GND, and the 3 SPI pins and reset. But those SPI pins are also broken out to the pins 11, 12 and 13.

So basicaly im just using the pins 11, 12 and 13 for the shield?

Indeed those, and 10 for the ChipSelect (SS). Pin 4, if you want to use the SD card functionality. See my edit concerning the 3.3v.


text:
IR not working on Attiny85
After countless attemps on making this thing work i'm not sure what the hell is going on. First of all you can't use the normal IR library because Attiny85 is crap and due to timer isses it wont work. So basicaly you need just the specific part which gets the data from the remote and thats all.
This is what i did but i failed badly again.
This time i used the simplest possible code which works perfectly on the Arduino UNO but it wont do anything on the Attiny85
The code
int irPin     = 4;
int tipPin    = 3;
int start_bit = 2200;    //Start bit threshold (Microseconds)
int bin_1     = 1000;    //Binary 1 threshold (Microseconds)
int bin_0     = 400;     //Binary 0 threshold (Microseconds)

boolean state = 0x0;

void setup() {
   pinMode(tipPin, OUTPUT);
   pinMode(irPin, INPUT);
   //Serial.begin(9600);
   //Serial.println("IR/Serial Initialized: ");
}

void loop() {
   int key = getIRKey();   //Fetch the key
   //Serial.println(key);
   if (key != 0) {
     state != state;
     digitalWrite(tipPin, state);
   }
   delay(400);    // avoid double key logging (adjustable)
}

int getIRKey() {
  int data[12];
  int i;

while(pulseIn(irPin, LOW) < start_bit); //Wait for a start bit

for(i = 0 ; i < 11 ; i++)
  data[i] = pulseIn(irPin, LOW);      //Start measuring bits, I only want   low pulses

  for(i = 0 ; i < 11 ; i++)             //Parse them
  {
    if(data[i] > bin_1)                 //is it a 1?
data[i] = 1;
else if(data[i] > bin_0)            //is it a 0?
data[i] = 0;
else
return -1;                        //Flag the data as invalid; I don't know what it is! Return -1 on invalid data
  }

  int result = 0;
  for(i = 0 ; i < 11 ; i++)             //Convert data bits to integer
    if(data[i] == 1) result |= (1<<i);

  return result;                        //Return key number
}

The ouput values are 3-digit numbers which work OK and i'm not even checking the value here. Just if something came in. And it doesn't bloody work!!!!
Stuff i've already checked

The circuit
If the pin works
If the timers are set correctly
Using adafruits library
Changing every possible core
If the receiver works
The wiring

If you are trying to toggle the LED when a signal comes in, then there is a glitch in the code and I have no idea how the UNO worked for you as I tried and it only spat out Serial, but didn't do anything with the LED.
Try changing this line of code:
state != state;
to
state = !state
you are not actually toggling the LED state with the other line, but instead doing a logical not equal to which won't do what you want.

Oh my god.... Now i see it. It worked on the arduino because i wrote state != state; the first time then i deleted it to make it blink to test the pin and then i typed it manualy instead of pressing ctrl+z

@MeletisFlevarakis - thought it odd for a mistake like that anyway, done it myself a few times.


text:
How to convert the 0-5V to a 4-20mA current loop?
I'm pretty clueless about electronics so everything has to be explained in simple terms to me.
I have an Arduino with a program which provides a value via the Analog output, so that is anything from 0-5V.
On the other side I have a PLC (programmable logic controller) with an IO card which allows to connect to a (4-20mA) current loop.
How do I convert from lets say 1-5V to the 4-20mA in a somewhat cheap and safe way (I don't want to destroy the PLC [and possibly the Arduino])?
Of course the Arduino and the PLC have separate power sources.

Most arduino's don't have analog outputs (DACs), only PWM outputs.

Volts (V) and Milliamps (mA) are not units you can convert between.  for an analogy using water flowing in a pipe Volts would measure the water pressure, while mA or A measures how much water is going thru the pipe.  As you can see one can effect the other, but they are not measuring the same thing at all.   Also be certain to bridge the grounds (or common) of the two power supplies.

Can you link to some documentation for the particular PLC you have?

Usually PLC are 24v be careful with that

If all you need is analog 4-20mA loop transmitter, then you need an op-amp, the transistors that its output drives, a sense resistor, and a few other resistors. Note that these are at least 1% resistors. This needs to be built on a printed circuit board, not a solderless breadboard.  Two of my coworkers at Maxim Integrated wrote an applications note about a 4-20mA current-loop transmitter: http://www.maximintegrated.com/en/app-notes/index.mvp/id/5610  see this figure:

Since you are already using a PWM (hopefully with an RC low-pass filter), you just need everything after the DAC output. (Performance will not be as good with an Arduino PWM as it would be with a 16-bit voltage-output DAC, but the general idea would still work.) For your purpose, you can also ignore all of the stuff about the HART modem -- that's an extra feature for sending digital information along the same loop. Only relevant if your PLC receiver was expecting it.
There are also numerous prebuilt 4-20mA transmitter boards available, this has been around many decades in the industrial control market. Since Maxim Integrated makes precision DAC chips, we don't make reference designs that take 0-5V (especially from a PWM!) and convert to 4-20mA -- but we do sell reference design boards that include the DAC. Such as our Monterey system board (MAXREFDES15#). http://www.maximintegrated.com/en/app-notes/index.mvp/id/5683
There's also a 4-20mA Current-Loop Transmitter from ti.com, XTR117: http://www.ti.com/lit/ds/symlink/xtr117.pdf
There's also a board from mikroElektronika, MIKROE-1296 (available through mouser.com) that uses an Analog Devices DAC (through SPI interface) to drive the 4-20mA transmit circuit. Again, this is an idea that's been around many decades, so a search for 4-20mA transmitter should yield lots of useful information.
So your best bet is probably to purchase one of these prebuilt 4-20mA transmitter boards, which most likely will include its own precision DAC. You'll have to modify your Arduino code to drive the SPI interface, specific to whichever DAC you end up using. Or, if you really want to use the PWM to drive 4-20mA (at reduced performance), be sure to at least use an RC low-pass filter to get it as close as possible to a stable DC level.

Here is an article which explain how you can use arduino and a 4-20mA current loop transmitter to generate 4-20mA signal. Its using a DAC and 4-20mA current generator.
https://ncd.io/interfacing-isolated-4-20ma-current-loop-transmitter-arduino/
So this is how it works
in this article they are using a 4-20mA current transmitter board with arduino, You can build it by yourself as well as you can get it from it from https://store.ncd.io/shop/?fwp_product_type=4-20ma-transmitter. it has a DAC and a XTR ( its a TI part which is used to generate 4-20mA signal) and signal and power isolator. In these application it is very important that you use isolation between device and your arduino or any other master device. Once you have all the hardware setup, you can send i2c commands and it will generate 4-20mA current loop signal. Lets say you have a device which provides 0-5V and you want to control a different device which needs 4-20mA signal in that case you can use arduino analog input to read the 0-5V and you can use this 4-20mA transmitter to convert that 0-5V into 4-20mA.
For more info checkout the above mentioned article.


text:
Why doesn't my code work in a function, yet works inline?
I'm trying to write a faster shiftOut function, which does not use the slow digitalWrite. In my code I have the original shiftOut, the new shiftOutFast, and the same as shiftOutFast, but inline. The three blocks are separated by delays to tell them apart on my scope. Here's the code:
#define CLR(x,y) (x &= (~(1 << y)) )
#define SET(x,y) (x |= (1 << y) )

void shiftOutFast(uint8_t dataPort, uint8_t dataBit, uint8_t clkPort,
                  uint8_t clkBit, uint8_t bitOrder, uint8_t val) {
  for (uint8_t i = 0; i < 8; i++)  {
    if (bitOrder == LSBFIRST) {
      if (val & 0x01) {
        SET(dataPort, dataBit);
      } else {
        CLR(dataPort, dataBit);
      }
      val >>= 1;
    } else {
      if (val & 0x80) {
        SET(dataPort, dataBit);
      } else {
        CLR(dataPort, dataBit);
      }
      val <<= 1;
    }

    CLR(clkPort, clkBit);
    SET(clkPort, clkBit);
  }
}

#define dataPin 4
#define clockPin 5
#define dataPort PORTD
#define dataBit PORTD4
#define clockPort PORTD
#define clockBit PORTD5

void setup() {
  pinMode(dataPin, OUTPUT);
  pinMode(clockPin, OUTPUT);
}

void loop() {
  shiftOut(dataPin, clockPin, LSBFIRST, 0x55);
  delay(1);

  shiftOutFast(dataPort, dataBit, clockPort, clockBit, LSBFIRST, 0x5C);
  delay(1);

  bool bitOrder = LSBFIRST;
  uint8_t val = 0x5C;

  for (uint8_t i = 0; i < 8; i++)  {
    if (bitOrder == LSBFIRST) {
      if (val & 0x01) {
        SET(dataPort, dataBit);
      } else {
        CLR(dataPort, dataBit);
      }
      val >>= 1;
    } else {
      if (val & 0x80) {
        SET(dataPort, dataBit);
      } else {
        CLR(dataPort, dataBit);
      }
      val <<= 1;
    }

    CLR(clockPort, clockBit);
    SET(clockPort, clockBit);
  }
  delay(1);
}

The problem is that calling shiftOutFast doesn't do zilch. Absolutely nothing. It doesn't seem to be the logic, because when I do the same inline it works. Any ideas?

The problem is that you are passing the value of the ports to the function, not the address of the ports.
You need to either pass "by reference" or as a pointer, so that modifying the port variable modifies the actual port variable not the copy of the value that you pass.
The simplest way is to modify your function to be:
void shiftOutFast(volatile uint8_t &dataPort, uint8_t dataBit,
                  volatile uint8_t &clkPort, uint8_t clkBit,
                  uint8_t bitOrder, uint8_t val) {
....
}

That way, instead of passing whatever is in the port at the time to the function (pass by value) it instead creates a new variable which is located at the same address as the variable that you are passing (pass by reference) so modifications of one variable modifies the other.
Note that only the "port" variables need to be passed by reference, all the others remain as pass by value.  Also they should be, as Edgar says, flagged as volatile.

You should declare volatile the port parameters of your function. Otherwise, to the compiler, their write accesses seem useless, and they get optimized out.

You mean void shiftOutFast(volatile uint8_t dataPort, volatile uint8_t dataBit, volatile uint8_t clkPort, etc..? I tried it but it doesn't solve the problem.

only the ports, and as @Majenko said, you should pass them by reference.


text:
Why use a module rather than sensor directly?
I'm trying to understand when one would use an Arduino module rather than directly a sensor.
Here's an example to contextualize my question: I need to use a Hall sensor to measure the number of rotations per minute of a magnet mounted on a computer fan. I see that I can either buy the Hall sensor and wire it to the Arduino (Hall-Effect Sensor Analog 49E), or alternatively I could buy a sensor module (e.g., Digital Hall Effect Sensor Module or Linear Hall Effect Sensor Module). What is the advantage of buying modules, rather than the sensors directly?

do you not see a difference between the bare sensor and the sensor module?

The same reason why ready made meals exist when you can just buy individual ingredients.

@Gerben you don't see the advantage of having a power circuit and a schmitt trigger wired up and properly tuned for a range of magnetic fields?

The only difference i see between the sensor and the first module is the LED with its resistor. Not much of an advantage i think.

I usually use a module first and then, when I'm sure everything works as I intended, I might make a PCB that carries the sensor directly (and usually also the MCU).

One other way to look at it - it's the same benefit as using an Arduino (microcontroller module) rather than directly using an atmega328p integrated circuit (the main chip of the Arduino)

The modules are typically easier to work with, particularly for DIY projects. They have standardized connectors, have their connections labeled and can be mounted into wiring boards. The particular sensor you have there does have quite big legs, so this may work fine without the board, but this is often not true. Consider sensors such as the BME/BMP types (BME680 data sheet). They're far to small to be used directly without a PCB. And they require some external components to work correctly, which are already included on the module. So for experimental purposes, the module is the way to go.

It's a question of convenience and application use-case specific. Sometimes a module will give you additional functionality and/or structural stability, just by providing a mini-PCB (maybe with screw holes) for a low price overhead.
For example, compared to the bare element the module you posted has additional components (LED or A0,D0 on the pricier one) is easily mountable and provides standarized connection pins.
If you already have a PCB (or other circuitry) the module could be superfluous, maybe you just need the sensor itself, but often those modules come as neat packages, sometimes with controller chips speaking I²C and comparable protocols (like the DHT11/DHT22 and others).

Since nobody else has addressed your example directly, and to use it as an example, in addition to the other observations about these two items, let's also take note of :
Hall-Effect Sensor Analog 49E

The 49E sensor can detect the magnetic N/S pole and relative strength of a magnetic field and outputs an analog voltage

Vcc Range    2.3 – 10V
Output : Analog 1.9mV/GS (± 0.2mV/GS)

Digital Hall Effect Sensor Module

3144 sensor detects presence of magnetic field and outputs logic level.

Vcc Range    4.5 – 24V
Output :  Open Collector

Linear Hall Effect Sensor Module

Detects N/S poles and relative strength of a magnetic field.

Vcc Range 3.3 – 5V
Output : Analog 2.5mV / G PLUS digital output
Includes 49E sensor plus everything in the schematic below :

So the sensor you've linked provides an analog signal, allowing you measure the magnetic field strength.  The digital module provides only an open-collector output (ie: ON/OFF) which indicates either the absence or presence of a magnetic field above some threshold of intensity. These are entirely different functions which you would use for entirely different applications.  The digital module also works with VCC up to 24V, which makes it handy to integrate with 24V automation systems.  The linear module provides an analog output as well as some indicators and an additional digital output, but is capped at 5V Vcc (whereas the raw module accepts up to 10V).
Why would you choose either?  What do you need to do?  Do you want to check for the presence of a magnetic field only, or do you also want to measure its intensity?  Or both?  The modules do some of the work for you and, in the case of the linear module, actually give you an additional output that the raw sensor does not.  It's up to you to decide which of these serves your needs.

+1 for nice markup, but I think OPs question was more general, why one would chose some module over a component at all, rather than the differences in those specific modules/components (which are obvious, hopefully)

@scitor, I think this answer explains the difference in general using a specific example brought by OP

In your case it only makes sense if you want to put the module on a breadboard and don't have the other components on the board at hand.
Example: for a DS18B20 one wire temperature sensor you need a pullup resistor. If you have this resistor at hand there is no sense to buy a module. But if you have to buy one resistor it is more easy to buy a module.
For experimental setups, it is often easier to use a module. If you want to build the circuit permanently later, the modules are rather annoying. Because the modules are often very bulky. The direct mounting of the components on a perfboard is then more space-saving.

It's a tradeoff between convenience and your time, money and expertise. You could design the module's circuitry and function into your circuit board, or, at slightly more expense, you could solder a module to your circuit board, with the designing and functionality already built-in. It might cost a little bit more and maybe be somewhat larger, than if you did it yourself. Ultimately it's up to you, which way works better for you in your particular circumstance.