text:
Output set to 'low' when new variable in buffer
I would like to turn on and off the output of two pins on my arduino independently from each other. However, so far all my attempts have failed.
My current idea is to save the status (which I chose to be 0 as 'low' and 'high' as 1) of each of the pins in an integer array and then change the status of the pin depending of the entry in the variable. However, the entries in the variable are always set back to 0 and I do not know why this is happening.
This is my arduino code
const byte numChars = 32;
byte receivedChars[numChars];
boolean newData = false;
const int num_of_pins=2;
int output_pins[num_of_pins]={22,26};
int status_pins[num_of_pins]={0,0};

void setup() {
    int ii=0;
    Serial.begin(9600);
    for(ii=0; ii<num_of_pins; ii++){
        pinMode(output_pins[ii],OUTPUT);
    }

}

void loop() {
    recvWithStartEndMarkers();
    chooseFunction();
}

void chooseFunction(){
  if (receivedChars[0]==0){
    shutter();
    }
}

void shutter(){
        // get number of pin
        int pin_number_rcvd;
        int ii;
        int pin_status_rcvd;
        pin_number_rcvd=receivedChars[1];
        pin_status_rcvd=receivedChars[2];
        //check all status of pins and turn on and off
        for (ii=0; ii<num_of_pins; ii++){
          if (pin_number_rcvd==output_pins[ii]){
            status_pins[ii]=pin_status_rcvd;
          }
          if (status_pins[ii]==0){
            digitalWrite(output_pins[ii],LOW);
          }
          if (status_pins[ii]==1){
            digitalWrite(output_pins[ii],HIGH);
          }
        }
}

void recvWithStartEndMarkers() {
    static boolean recvInProgress = false;
    static byte ndx = 0;
    byte startMarker = 8;
    byte endMarker = 9;
    byte rc;

    while (Serial.available() > 0 && newData == false) {
        rc = Serial.read();

        if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars) {
                    ndx = numChars - 1;
                }
            }
            else {
                receivedChars[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }
        }

        else if (rc == startMarker) {
            recvInProgress = true;
        }
    }
}

I use pyserial to send the input data to the arduino. my python code:
import serial
import time, struct, sys
ser=serial.Serial('COM17', 9600, timeout=10)
time.sleep(2)
ser.reset_input_buffer()
ser.reset_output_buffer()
pin_number= int(sys.argv[1])
high_or_low=int(sys.argv[2])

ser.write(struct.pack('>5B',8,1,pin_number,high_or_low,9))

I am using an ArduinoDue together with the arduino IDE.

Maybe it isn't it, but you are expecting:  receivedChars[0]==0  but  ser.write(struct.pack('>5B',8,1,pin_number,high_or_low,9)) is sent (eg it's 1, not 0)

thanks for your answer. Unfortunately, it does not change anything

you left  our some information .... the following description does not explain why python script is involved .... I would like to turn on and off the output of two pins on my arduino independently from each other.

Thats true. Sorry for that. I want to switch the output on the board on and off using python, e.g. I want to send data to the arduino with pyserial telling the arduino to switch on or off the output on a certain pin.

I think I figured iut the problem, but I still do not know how to solve it. The problem is that whenever I want to switch the output on or off, I restart the python script which completely resets the arduino and therefore the output on all the other channels is switched off

Your problem is in one tricky option for arduino's com connection. When you reopen COM port where arduino is connected your arduino resets automatically.
Here's workaround from arduino.cc:

DisablingAutoResetOnSerialConnection
The simple way that doesn't require
  any permanent modifying of your hardware or software configuration
  changes:
Stick a 120 ohm resistor in the headers between 5v and reset (you can
  find these on the isp connector too). 120 is hard to find so just
  combine resistors. Don't go below 110 ohms or above 124 ohms, and
  don't do this with an isp programmer attached. You can just pull out
  the resistor when you want auto-reset back.

http://playground.arduino.cc/Main/DisablingAutoResetOnSerialConnection
Or you can just modify you script to be able to start over without reopening it.
For example use a loop there and some exit command to end that loop:
try:
    while True:
        do_something()
except KeyboardInterrupt:
    pass

https://stackoverflow.com/questions/13180941/how-to-kill-a-while-loop-with-a-keystroke
Replace do_something() with this part of your code:
ser.reset_input_buffer()
ser.reset_output_buffer()
pin_number= int(sys.argv[1])
high_or_low=int(sys.argv[2])

ser.write(struct.pack('>5B',8,1,pin_number,high_or_low,9))

I hope I was clear enough.


text:
LED on Arduino does not blink
I successful installed arduino, added me to uucp and lock and loaded cdc_acm module.
I use Arduino Uno R3, so I also installed arduino-avr-core.
I have Arduino on /dev/ttyUSB0.
I configured with:
stty -F /dev/ttyUSB0 cs8 9600 ignbrk -brkint -imaxbel -opost -onlcr -isig -icanon -iexten -echo -echoe -echok -echoctl -echoke noflsh -ixon -crtscts

I have this folder with these files:
~/Documents/Arduino
build.options.json
------------------------------------------------------------
{
    "fqbn": "archlinux-arduino:avr:uno",
    "hardwareFolders": "/usr/share/arduino/hardware",
    "toolsFolders": "/usr/bin"
}

sketch.ino
------------------------------------------------------------
void setup() {
    pinMode(13, OUTPUT);
}

void loop() {
    digitalWrite(13, HIGH);
    delay(1000);
    digitalWrite(13, LOW);
    delay(1000);
}

After this commands I got output:
arduino-builder -build-options-file build.options.json sketch.ino

Sketch uses 914 bytes (2%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.

But nothing happened on my Arduino, what's my problem?
Thanks.
P.S. I don't have external LED, I try to use bultin LED.

did you try Arduino IDE?

I don't wanna use Arduino IDE. And when I tried it, I got error with avrdude.

there is no way around avrdude

You invoked the builder, but not an upload to the board. The builder created a hex file.You should use avrdude for upload.
Or use Arduino IDE which does it with a click on a button

I don't have hex file. I try to search, but I cannot find it.

turn on verbose output of the builder

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Correlated signal from two sensors in one circuit
I have water level/humidity sensor and a LDR diode hooked to arduino like this (the resistor is 10k)

On A0 I can read signal from water sensor (blue)
On A1 I can read signal from light sensor (yellow)

The problem is that both signals seem to be correlated. When I dip the water sensor to water and keep the light levels the same, the signal from light goes up as well (dramatically).
I suspect that this is due to Ohm's law. The voltage in a circuit depends on the total resistance of the circuit. Both sensors act as a sort of resistors so when one's resistance changes the voltage in the circuit changes as well.
How can I wire the sensors such that the change in resistance of either sensor doesn't affect the second? Or do I need a separate power supply for each sensor?

is the light sensor wiring right? shouldn't the resistor connect to ground?

I get some signal, so I guess it is (row 37)

You can try to read another channel between them - connected to ground to discharge sampling capacitor.

@KIIV I am not sure if I follow (still learning the lingo). You mean like connecting a simple resistor to GND and A3 (same as the LDR is wired). and read A0 (water) -> A3 (discharge)  --> A1 (light)?

Yes, pretty much. But you can short A3 to the ground directly. And sequence would be:  A3 (discharge), A0 (water), A3 (discharge), A1 (light). Now it shouldn't be crosstalking between A0 and A1 as it starts the measure on completely discharged capacitor. Now it'll depend on output impedance of sensors only.

short A3 to the ground directly You mean a jumper from row 45 GND to A3 directly? just making sure

https://electronics.stackexchange.com/questions/213851/arduino-analogread-neighbor-pin-noise-on-adc-even-with-big-delay

I still think the problem is the pullup resistor

I've unplugged the whole thing and rebuilt from scratch. That didn't work initally, so I wired A3 to GND directly and read a value from that. Then I unplugged the A3-GND and it seems to be working just fine even without it. Thanks to both of you! @Juraj, if you compile the A3-GND tip to an answer I will happily accept it, since that seemed to help imo

@KIIV knew what to do, answer is his

of course (sorry @KIIV !) just swapped the two names. Thanks to both of you anyway

If I've found the same humidity sensor, it has pretty big output impedance and the maximum input impedance for the ADC input is 10kOhms. According the the datasheet the recommended minimum load for the sensor is about 80kOhms (?). That means readings of humidity sensor will be highly affected by sampling capacitor initial charge (= the value of previous reading).
Basically reading GND between sensor readings helps to separate the results from each other, but the humidity sensor won't be able to charge sampling capacitor to its output voltage without amplifying the current (you'll get values much lower than it should be).
Or you can repeat the readings on humidity sensor channel several times until the value is more "stable".

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Could I Connect many Arduino Nano boards together?
I want to know if I could connect many Arduino Nano  boards together (Example : 10 boards ) , Also i want to know  if there is any problem in Communication between them  using I2C , could this affects processing speed on every board ,  and  if there's going to be any kind of delay when communicating?

It is a good (but broad) question. Can you add more information to your question? Do you really need arduino nano boards? Other boards have a extra serial port to be able to connect all boards with serial ports. How far is the distance? What kind of cable do you want to use? Is is okay to use wireless communication? Why do you worry about processing speed and delay? What else do you want to run on the arduino board? The i2c bus might work for short distances, but requires good programming skills. Using the i2c does not work very well with libraries that turn off the interrupts.

I want to know if I could connect many Arduino Nano boards together...

Yes, you can read about it here at arduino.cc. Where is says: "...it can be helpful to set up two (or more!) Arduino ... boards to share information with each other..."

... i want to know if there is any problem in Communication between
  them using I2C ...

This feature, according to the above link, is expected to work.  (Can you explain the type of problems you are avoiding?)

...could [i2c communication] affect processing speed on every board ...

Most likely it will.  Most Arduino platforms do not run operating systems.  So your program is all that is running.  If your program is busy sampling sensors, it will be difficult (not impossible) to concurrently handle i2c communications.

I am going to connect 4 DC motors  encoders and some other sensors but the encoders must be on interrupt pins for best performance , I tried to connect them on an Arduino Mega but it has only 3 pairs of interrupt pins and I need 4 pairs , so the problem I am trying to Avoid is : The interruption of every encoder to the other one .

Consider what your objective is.  Can you process the interrupt on an individual embedded processor in such a way as to contain the burden of timing to within that processor?  Sometimes the answer is clearer when you refactor the question instead of looking for hard to find hardware.

Yes it certainly is possible. And can work well if done properly. The concern that I have is when longer wiring lengths are used. I2C is not intended for extended distributions. And if that were the plan then I STRONGLY advise using proper I2C buffer chips designed for that purpose.
The more devices, the more wire, all adds to the capacitance on the lines. Because I2C uses pull up resistors to produce a high the added capacitance limits either the length of wire that can be used, or the transmission frequency. I have used I2C buffers in may products and it is very useful when extended length is required. I use both I2C buffers, and also I2C isolated buffers. The company I work for manufactures distributed control electronics. I am one of the hardware designers. I have been using I2C like this for about 15 years.
For now have a look at this web page.  http://www.techbitar.com/how-to-network-many-arduinos-with-sensors-using-i2c.html
I'm sure it will spur some questions. It makes things simpler if there is only one master controlling the communications bus. Multimast is possible but can have more conflicts and you then need to deal with that issue. Best not to go there.
I2C is not a very robust protocol but still can be usable between processors if handled carefully. Keep it simple.


text:
Midi controlled LED by USB
I am trying to create something that will allow me to control an individually adressable LED strip using an electric piano using its midi output.
Some things to note:

The piano can output midi either via USB or via standard midi connectors.
I have an Arduino Uno and an Arduino Mega available
I'm using a WS2812B strip with +ve, ground and data cables

I don't believe there's a way to read midi from my piano direct to the arduinos without reflashing the USB controller. For this reason I'm connecting via USB to my mac.  On my mac I've written a program which reads the midi input and can understand it and send it on to somewhere else.
The arduino for the light strip I think I can program pretty easily but I have some questions:

If I need to power the LED strip, should I be powering it from the arduino power or giving it its own power source with common ground?
Is splitting it up into midi -> computer, computer -> arduino, arduino -> LED strip sensible? If not how should I be going about it?
Am I making life difficult by going the USB route for Midi or should I switch to midi connectors and connect direct to the arduino somehow?

the Internet is full of Arduino and MIDI. you can connect it directly. the MIDI protocol is very simple. https://www.arduino.cc/en/Tutorial/Midi

@Juraj you'd suggest going the route of using 5 pole MIDI cables then rather than USB?

of course. you glue the LED strip to a Jan Hammer style piano and where do you put then the MacBook?

@Juraj Little bit confused.  As I said in the question I'm currently connecting the piano via USB to the macbook and using software I've written on there to read the MIDI data.  Just trying to figure out the best way to connect it all up given I have very little knowledge of electronics.  I'm normally just a software developer.

but you know the Keytar? if you programmed the midi for computer, it will be simpler to write it for Arduino

I did some time ago something very similar, except not using USB, and on an STM32 (Arduino compatible STM32F103C8T6), but the principle is similar.
Regarding another MIDI test I did on an Arduino, I used the 47 Effects library and for the WS2813 (software-compatible with WS2812B) the FastLed library.
One MIDI port is easy to use (supported by 47 Effects), which uses the single UART of the Arduino UNO.
You have to write a small program to convert the MIDI notes to WS2812 LED data, which is not much more than filling in a structure. Mapping 61 or 88 notes to 30, 60 or 144 LED notes is not so easy, just clip the notes, or use some mapping factor.
You cannot use the Arduino for powering the LED strip, use a separate adapter for that. Only the line signal comes from the Arduino, and connect GNDs together.
The Arduino should be fast enough to both read MIDI, translate it to LED data (unless you make some very time consuming algorithm) and send the data.


text:
5V MCP2515 to 3.3V Teensy. How to reduce the MISO voltage
So to try and speed up my project, I have moved from using two UNO's to a UNO on the receive side, and teensy 3.6 on the sender side. However, no dice. The UNO  doesn't receive any data or even try to print it. I have found it gets to the line;
if(!digitalRead(CAN0_INT))

and then the program won't proceed. I have swapped out the teensy and run the same code on an UNO again on the sender to check all the connections work and it is fine, as soon as I plug in the teensy no messages are received.
As far as I can tell the MCP2515 module does not work on 3.3V, so I'm wondering if it is that. So, my plan is to power the MCP2515 from a 5V source, and drop the voltage on the interrupt and MISO line to 3.3V so the teensy is safe.
However I am unsure on how to do this, and I also need a quick fix as i'm away this afternoon until January and don't have a logic converter to hand. I have seen you can use a voltage divider to reduce voltage but will that be an issue as that solution is apparently only for slow signals?
Any other thoughts on why this setup won't work are welcome. So just incase I have added my code that works on UNO's but not on teensy. My only other thought why this won't work is that teensy 3.6 already has CAN bus and the code is not compatible?
Sender
// CAN Send Example
//

#include <mcp_can.h>
#include <SPI.h>
#include <Wire.h>

const int MPU_addr = 0x68; // I2C address of the MPU-6050

float lastTimeSent;
const float intervalSendMs = 0.000001; // Send period in milliseconds
float currMillis;

int16_t AccX, AccY, AccZ, Tmp, GyX, GyY, GyZ;
uint8_t dataArray[32];
uint8_t data1[8], data2[8];
int Switch = 7;
const int AS_Pin = 10;

#define CAN0_INT 2    // Set INT to pin 2
MCP_CAN CAN0(9);     // Set CS to pin 10

void setup()
{
    Serial.begin(115200);

    initializeMCP2515();

    pinMode(Switch, OUTPUT);
    digitalWrite(Switch, HIGH);

    initializeMPU9250();

}

void loop()
{
    currMillis = millis();

    if ((currMillis - lastTimeSent) >= intervalSendMs)
    {
        lastTimeSent += intervalSendMs;

        readFromMPU9250();

        //MPU_Data_to_CAN_Array();

        convertDataToPackets();

        // Send the two frames; if the first sending is successful send also the second, otherwise abort
        // send data frame 1:  ID = 0x100, Standard CAN Frame, Data length = 8 bytes, 'data' = array of data bytes to send
        // send data frame 2:  ID = 0x101, Standard CAN Frame, Data length = 8 bytes, 'data' = array of data bytes to send
        if (CAN0.sendMsgBuf(0x100, 0, 8, data1) == CAN_OK)
            CAN0.sendMsgBuf(0x101, 0, 8, data2);
    }

                  Serial.print(currMillis);
                  Serial.print("\t");
                  Serial.print(AccX);
                  Serial.print("\t");
                  Serial.print(AccY);
                  Serial.print("\t");
                  Serial.print(AccZ);
                  Serial.print("\t");
                  Serial.print(GyX);
                  Serial.print("\t");
                  Serial.print(GyY);
                  Serial.print("\t");
                  Serial.println(GyZ);

}

//===================================================================================
//                                  FUNCTIONS
//===================================================================================
void initializeMCP2515()
{
    // Initialize MCP2515 running at 16MHz with a baudrate of 500kb/s and the masks and filters disabled.
    if(CAN0.begin(MCP_ANY, CAN_1000KBPS, MCP_16MHZ) == CAN_OK)
        Serial.println("MCP2515 Initialized Successfully!");
    else
        Serial.println("Error Initializing MCP2515...");

    CAN0.setMode(MCP_NORMAL);   // Change to normal mode to allow messages to be transmitted
}

void initializeMPU9250()
{
  pinMode(AS_Pin, OUTPUT);
  digitalWrite(AS_Pin, HIGH);
  SPI.setClockDivider(SPI_CLOCK_DIV4);
  SPI.setDataMode(SPI_MODE0);
  SPI.begin();
}

void readFromMPU9250()
{
      digitalWrite(AS_Pin, LOW);
  SPI.transfer(0x80|0x3B);

  for(int i = 0; i < 14; i++){
  dataArray[i] = SPI.transfer(0);
  }
  digitalWrite(AS_Pin, HIGH);
delay(50);

AccX = dataArray[0]<<8|dataArray[1];
AccY = dataArray[2]<<8|dataArray[3];
AccZ = dataArray[4]<<8|dataArray[5];
GyX = dataArray[8]<<8|dataArray[9];
GyY = dataArray[10]<<8|dataArray[11];
GyZ = dataArray[12]<<8|dataArray[13];
}

void convertDataToPackets()
{
    long2Byte(currMillis, 0, data1);
    int2Byte(AccX, 4, data1);
    int2Byte(AccY, 6, data1);

    int2Byte(AccZ, 0, data2);
    int2Byte(GyX, 2, data2);
    int2Byte(GyY, 4, data2);
    int2Byte(GyZ, 6, data2);
}

void int2Byte(int16_t input, int pos, uint8_t *data)
{
    for (int i = 0; i < 2; i++)
    {
        data[i + pos] = ((input >> (i * 8)) & 0xff);
    }
}

void long2Byte(uint32_t input, int pos, uint8_t *data)
{
    for (int i = 0; i < 4; i++)
    {
        data[i + pos] = ((input >> (i * 8)) & 0xff);
    }
}

/*********************************************************************************************************
  END FILE
*********************************************************************************************************/

Receiver
// CAN Receive Example
//

#include <mcp_can.h>
#include <SPI.h>

long unsigned int rxId;
unsigned char len = 0;
unsigned char rxBuf[8];
uint8_t receivedFirstFrame;

char msgString[128];                        // Array to store serial string
uint32_t TimeMicro;
int16_t AccX, AccY, AccZ, GyX, GyY, GyZ;

#define CAN0_INT 2  // Set INT to pin 2
MCP_CAN CAN0(10);   // Set CS to pin 10

void setup()
{
    Serial.begin(115200);

    initializeMCP2515();

    Serial.println("MCP2515 Library Receive Example...");
    Serial.println("waiting for data");
    receivedFirstFrame = 0;
    Serial.println("data recived?");
}

void loop()
{
    if(!digitalRead(CAN0_INT))
    {
        // Received a message
        if (CAN0.readMsgBuf(&rxId, &len, rxBuf) == CAN_OK)
        {
            // Note: the above line is not like in the examples, but looking at the code the
            // function may return if no message was in the queue, so I think this is safer

            // Let's check if it is one of the known messages
            switch (rxId)
            {
            case 0x100:
                { // Received first frame
                    // Decode the content and store that the first frame was received
                    decodeFirstFrame(rxBuf);
                    receivedFirstFrame = 1;
                    Serial.print(TimeMicro);
                    Serial.print("\t");
                    Serial.print(AccX);
                    Serial.print("\t");
                    Serial.print(AccY);

                }
                break;
            case 0x101:
                if (receivedFirstFrame)
                { // Received second frame after a first frame
                    // Decode the content
                    decodeSecondFrame(rxBuf);

                  Serial.print("\t");
                  Serial.print(AccZ);
                  Serial.print("\t");
                  Serial.print(GyX);
                  Serial.print("\t");
                  Serial.print(GyY);
                  Serial.print("\t");
                  Serial.println(GyZ);

                    //printDataToSerial();

                    receivedFirstFrame = 0;
                }
                break;
            }
        }
    }

    TimeMicro = 0;
    AccX = 0;
    AccY = 0;
    AccZ = 0;
    GyX = 0;
    GyY = 0;
    GyZ = 0;
}

//===================================================================================
//                                  FUNCTIONS
//===================================================================================

void initializeMCP2515()
{
    // Initialize MCP2515 running at 16MHz with a baudrate of 500kb/s and the masks and filters disabled.
    if(CAN0.begin(MCP_ANY, CAN_1000KBPS, MCP_16MHZ) == CAN_OK)
        Serial.println("MCP2515 Initialized Successfully!");
    else
        Serial.println("Error Initializing MCP2515...");

    CAN0.setMode(MCP_NORMAL);   // Change to normal mode to allow messages to be transmitted

    pinMode(CAN0_INT, INPUT);                            // Configuring pin for /INT input
}

void decodeFirstFrame(uint8_t *rxBuff)
{
    TimeMicro = byte2Double(TimeMicro, 0, rxBuff);
    AccX = byte2Int(AccX, 4, rxBuff);
    AccY = byte2Int(AccY, 6, rxBuff);
}

void decodeSecondFrame(uint8_t *rxBuff)
{
    AccZ = byte2Int(AccZ, 0, rxBuff);
    GyX = byte2Int(GyX, 2, rxBuff);
    GyY = byte2Int(GyY, 4, rxBuff);
    GyZ = byte2Int(GyZ, 6, rxBuff);
}

uint32_t byte2Double(uint32_t output, int pos, uint8_t data[4]){
      for (int i = 0; i < 4; i++)
    {
      output += ((uint32_t)data[i + pos]) << (i * 8);
    }
    return output;
}

int16_t byte2Int(int16_t output, int pos, uint8_t data[2]){
      for (int i = 0; i < 2; i++)
    {
      output += ((int16_t)data[i + pos]) << (i * 8);
    }
    return output;
}
/*********************************************************************************************************
  END FILE
*********************************************************************************************************/

Thanks

search for arduino logic level converter

That is my plan for when I'm back but I wanted to test the setup before I left. Is there no quick fixes just so I can test for 10 seconds?

Why should a voltage divider only for slow signals? Is the parasite capacitance of the lines really that big?

The MCP2515 works fine at 3.3V. Check the Electrical Characteristics in the datasheet,  VDD: 2.7V to 5.5V

Thanks majenko, the seller I bought it from had it down as 5V only so must have made a mistake. The voltage divider question I'm not sure on. There seems to be a lot of conflicting views on methods for reducing voltage for sensors. I had read somewhere for things like spi voltage dividers weren't fast enough


text:
What could cause data to be printed correctly, then incorrectly and then repeated? CAN bus / general transmission
I am attempting to send data from an MPU6050 connected to an arduino uno to another uno via CAN bus, however i don't think this is a CAN bus specific issue.
In short, i am sending two packets of 8 bytes in the following formats
Packet 1 - (uint32_t/ int16_t/ int16_t)
Packet 2 - (int16_t/ int16_t/ int16_t/ int16_t)

all converted into bytes in this sort of fashion;
int16_t byte2Int(int16_t output, int pos, uint8_t data[2]){
      for (int i = 0; i < 2; i++)
    {
      output += ((int16_t)data[i + pos]) << (i * 8);
    }
    return output;
}

where everything should make sense apart from the "int pos" part which is just where in the buffer i want to start putting the data as packet 1 and 2 are different structures.
My issue is that with just one packet, i can print the values just fine, with the second packet added in i get the following on the serial monitor;
AccX    AccY    AccZ  GyX    GyY     GyZ

16832   -816    928 -440    16832   -816
16984   -940    992 -449    16984   -940
16896   -804    988 -462    16896   -804
209 55  0   0   209 55
16900   -852    1080    -471    16900   -852
16900   -852    1080    -471    16900   -852
16984   -760    948 -481    16984   -760
16880   -824    896 -459    16880   -824
230 102 0   0   230 102
16928   -860    996 -422    16928   -860
16928   -860    996 -422    16928   -860
17004   -856    864 -466    17004   -856
17004   -856    864 -466    17004   -856
214 105 0   0   214 105
16964   -828    944 -446    16964   -828
16964   -828    944 -446    16964   -828

As you can see i get some coherent messages, then some scrambled which don't make sense, then coherent again and the cycle continues. Not sure of relevance, but when i first start the serial monitor this doesn't happen, then after varying time, but around 5 seconds or so, it starts going amiss.
I believe this is down to my programming, and me not being able to code in a reliable way to wait for the first message to complete, then send the next one and them being read correctly. Below is my embarrassing attempt at sending an ACK message to the sender to say that the message has been received and to progress to the next message. I am not a programmer by nature and i am putting up this shoddy code that i've spent days trying to sort in the hope someone can point me in the right direction.
Apologies in advance.
// CAN Send Example
//

#include <mcp_can.h>
#include <SPI.h>
#include <Wire.h>

const int MPU_addr = 0x68; // I2C address of the MPU-6050
uint32_t TimeMicro;
int16_t AccX, AccY, AccZ, Tmp, GyX, GyY, GyZ;
int8_t data1[8], data2[8], rxBuf[8];
int Switch = 7;
long unsigned int rxId;
unsigned char len = 0;

#define CAN0_INT 2                              // Set INT to pin 2
MCP_CAN CAN0(10);     // Set CS to pin 10

void setup()
{
  Serial.begin(115200);

  // Initialize MCP2515 running at 16MHz with a baudrate of 500kb/s and the masks and filters disabled.
  if(CAN0.begin(MCP_ANY, CAN_500KBPS, MCP_16MHZ) == CAN_OK) Serial.println("MCP2515 Initialized Successfully!");
  else Serial.println("Error Initializing MCP2515...");

  CAN0.setMode(MCP_NORMAL);   // Change to normal mode to allow messages to be transmitted

  pinMode(Switch, OUTPUT);
  digitalWrite(Switch, HIGH);

  Wire.begin();
  Wire.beginTransmission(MPU_addr);
  Wire.write(0x6B);  // PWR_MGMT_1 register
  Wire.write(0);     // set to zero (wakes up the MPU-6050)
  Wire.endTransmission(true);

}

void loop()
{

  TimeMicro = millis();

  Wire.beginTransmission(MPU_addr);
  Wire.write(0x3B);  // starting with register 0x3B (ACCEL_XOUT_H)
  Wire.endTransmission(false);
  Wire.requestFrom(MPU_addr, 14, true); // request a total of 14 registers
  AccX = Wire.read() << 8 | Wire.read(); // 0x3B (ACCEL_XOUT_H) & 0x3C (ACCEL_XOUT_L)
  AccY = Wire.read() << 8 | Wire.read(); // 0x3D (ACCEL_YOUT_H) & 0x3E (ACCEL_YOUT_L)
  AccZ = Wire.read() << 8 | Wire.read(); // 0x3F (ACCEL_ZOUT_H) & 0x40 (ACCEL_ZOUT_L)
  Tmp = Wire.read() << 8 | Wire.read();
  GyX = Wire.read() << 8 | Wire.read(); // 0x43 (GYRO_XOUT_H) & 0x44 (GYRO_XOUT_L)
  GyY = Wire.read() << 8 | Wire.read(); // 0x45 (GYRO_YOUT_H) & 0x46 (GYRO_YOUT_L)
  GyZ = Wire.read() << 8 | Wire.read(); // 0x47 (GYRO_ZOUT_H) & 0x48 (GYRO_ZOUT_L)

  double2Byte(TimeMicro, 0, data1);
  int2Byte(AccX, 4, data1);
  int2Byte(AccY, 6, data1);

  int2Byte(AccZ, 0, data2);
  int2Byte(GyX, 2, data2);
  int2Byte(GyY, 4, data2);
  int2Byte(GyZ, 6, data2);

  // send data:  ID = 0x100, Standard CAN Frame, Data length = 8 bytes, 'data' = array of data bytes to send
  byte sndStat = CAN0.sendMsgBuf(0x100, 0, 8, data1);

  if(!digitalRead(CAN0_INT))                         // If CAN0_INT pin is low, read receive buffer
  {
    CAN0.readMsgBuf(&rxId, &len, rxBuf);      // Read data: len = data length, buf = data byte(s)

    while(rxBuf[0] != 0x0A)
    {
      continue;
    }

  }
  delay(10);
  byte sndStat2 = CAN0.sendMsgBuf(0x100, 0, 8, data2);

    while(rxBuf[0] != 0x0A)
    {
      continue;
    }

  for(int i = 0; i < 8; i++)
  {
    data1[i] = 0;
    data2[i] = 0;
  }

}

//===================================================================================
//                                  FUNCTIONS
//===================================================================================
uint8_t int2Byte(int16_t input, int pos, uint8_t data[2]){
  for (int i = 0; i < 2; i++)
  {
    data[i+ pos] = ((input >> (i * 8)) & 0xff);
  }
return data[2];
}

uint8_t double2Byte(uint32_t input, int pos, uint8_t data[4]){
  for (int i = 0; i < 4; i++)
  {
    data[i+ pos] = ((input >> (i * 8)) & 0xff);
  }
return data[4];
}

/*********************************************************************************************************
  END FILE
*********************************************************************************************************/

Receiver
// CAN Receive Example
//

#include <mcp_can.h>
#include <SPI.h>

long unsigned int rxId;
unsigned char len = 0;
unsigned char rxBuf1[8];
unsigned char rxBuf2[8];
unsigned char txBuf[8];
char msgString[128];                        // Array to store serial string
uint32_t TimeMicro;
int16_t AccX, AccY, AccZ, GyX, GyY, GyZ;

#define CAN0_INT 2                              // Set INT to pin 2
MCP_CAN CAN0(10);                               // Set CS to pin 10

void setup()
{
  Serial.begin(115200);

  // Initialize MCP2515 running at 16MHz with a baudrate of 500kb/s and the masks and filters disabled.
  if(CAN0.begin(MCP_ANY, CAN_500KBPS, MCP_16MHZ) == CAN_OK)
    Serial.println("MCP2515 Initialized Successfully!");
  else
    Serial.println("Error Initializing MCP2515...");

  CAN0.setMode(MCP_NORMAL);                     // Set operation mode to normal so the MCP2515 sends acks to received data.

  pinMode(CAN0_INT, INPUT);                            // Configuring pin for /INT input

  Serial.println("MCP2515 Library Receive Example...");
}

void loop()
{
  if(!digitalRead(CAN0_INT))                         // If CAN0_INT pin is low, read receive buffer
  {
    CAN0.readMsgBuf(&rxId, &len, rxBuf1);      // Read data: len = data length, buf = data byte(s)
  }

  for(int i; i < 8; i++){
    txBuf[i] = 0x0A;
  }

  byte sndStat = CAN0.sendMsgBuf(0x101, 0, 8, txBuf);

  if(!digitalRead(CAN0_INT))                         // If CAN0_INT pin is low, read receive buffer
  {
  CAN0.readMsgBuf(&rxId, &len, rxBuf1);      // Read data: len = data length, buf = data byte(s)
  }

  byte sndStat2 = CAN0.sendMsgBuf(0x101, 0, 8, txBuf);

  TimeMicro = byte2Double(TimeMicro, 0, rxBuf1);
  AccX = byte2Int(AccX, 4, rxBuf1);
  AccY = byte2Int(AccY, 6, rxBuf1);
  AccZ = byte2Int(AccZ, 0, rxBuf2);
  GyX = byte2Int(GyX, 2, rxBuf2);
  GyY = byte2Int(GyY, 4, rxBuf2);
  GyZ = byte2Int(GyZ, 6, rxBuf2);

  Serial.print(TimeMicro);
  Serial.print("\t");
  Serial.print(AccX);
  Serial.print("\t");
  Serial.print(AccY);
  Serial.print("\t");
  Serial.print(AccZ);
  Serial.print("\t");
  Serial.print(GyX);
  Serial.print("\t");
  Serial.print(GyY);
  Serial.print("\t");
  Serial.println(GyZ);

  TimeMicro = 0;
  AccX = 0;
  AccY = 0;
  AccZ = 0;
  GyX = 0;
  GyY = 0;
  GyZ = 0;

}

//===================================================================================
//                                  FUNCTIONS
//===================================================================================

uint32_t byte2Double(uint32_t output, int pos, uint8_t data[4]){
      for (int i = 0; i < 4; i++)
    {
      output += ((uint32_t)rxBuf1[i + pos]) << (i * 8);
    }
    return output;
}

int16_t byte2Int(int16_t output, int pos, uint8_t data[2]){
      for (int i = 0; i < 2; i++)
    {
      output += ((int16_t)rxBuf1[i + pos]) << (i * 8);
    }
    return output;
}
/*********************************************************************************************************
  END FILE
*********************************************************************************************************/

I've tried debugging this by the following;

check that the MPU is sending correctly (Fine)
check if it happens with just one packet (no problems)
check if the packets are assembled correctly on the sender ( got the data,
broke it into bytes and reassembled and printed that result. did what i
expected)
removed time from being sent incase that was causing an issue (same problem
as illustrated above)
removed all serial prints and just did one value (still an issue)

So i believe that it is down to a terrible and non working attempt at sending an acknowledgment message. I have tried researching but i couldn't find an example for CAN bus, and all the others were using protocol specific interrupt routines which i tried to implement but didn't work. obviously.
Thankyou for your patience

Did you check that you have always enough available RAM? I had some crazy, seemingly random issues, and it was connected with running out of RAM from time to time.

send fixed bytes for both packets .... we have no way of knowing if the data is actually corrupted by the transmission ..... gyro could be producing corrupted  values

DISCLAIMER: Ok, I never worked with CAN on arduino, so I may be terribly wrong, nor I have the required boards to test this, so I may make a lot of errors; if you find any issue, please point them out, so I can fix them (or if you fixed them notify me, so I can update the reply for the future readers).
Ok, from what I can read you are have two packets, but only one CAN frame. This seems to me a huge inefficiency, because how can you know which frame have you just received?
To fix this, you can either use two frames or use something in the payload that informs you which frame you have received. In my opinion, since there are a lot of free IDs, you can use the first approach.
Moreover you are performing some checks in the processing (e.g. "did I receive an ack?"),  but if this fails you go on without further checking. For instance, when in the sender you write:
byte sndStat = CAN0.sendMsgBuf(0x100, 0, 8, data1);
if(!digitalRead(CAN0_INT))
{
  ...
}
delay(10);
byte sndStat2 = CAN0.sendMsgBuf(0x100, 0, 8, data2);

what happens when CAN0_INT is still high? Yes, it sends the data immediately.
I could not understand the while(rxBuf[0] != 0x0A) part; it will block your program forever...
Finally when you receive a message you never check the ID; this means you will check all the messages on CAN, and when you add another device on the net you will not be able to send any other message.
Ok, so this is what I'd try for your needs:

The TX board will always transmit data, while the RX board will always receive it. No ack frames is needed
There will be two frames on the network: frame 0x100 will have packet 1, while frame 0x101 will have packet 2
The TX board will send one message every 100ms; this can be reduced, but specify a time rather than having the part send frames "continuously"
The RX board will consider the data valid only after a 0x101 packet is received and a 0x100 packet was received before.

So, here is the code. Please note that I refactored some parts for easier understanding (putting some code in functions to have a smaller "higher level" setup and loop functions.
BTW what is the "Switch" in the sender? It is never used...
Sender:
// CAN Send Example
//

#include <mcp_can.h>
#include <SPI.h>
#include <Wire.h>

const int MPU_addr = 0x68; // I2C address of the MPU-6050

uint32_t lastTimeSent;
uint32_t currMillis;
const uint32_t intervalSendMs = 100; // Send period in milliseconds

int16_t AccX, AccY, AccZ, Tmp, GyX, GyY, GyZ;
uint8_t data1[8], data2[8];
int Switch = 7;

#define CAN0_INT 2    // Set INT to pin 2
MCP_CAN CAN0(10);     // Set CS to pin 10

void setup()
{
    Serial.begin(115200);

    initializeMCP2515();

    pinMode(Switch, OUTPUT);
    digitalWrite(Switch, HIGH);

    initializeMPU6050();
}

void loop()
{
    currMillis = millis();

    if ((currMillis - lastTimeSent) >= intervalSendMs)
    {
        lastTimeSent += intervalSendMs;

        readFromMPU6050();

        convertDataToPackets();

        // Send the two frames; if the first sending is successful send also the second, otherwise abort
        // send data frame 1:  ID = 0x100, Standard CAN Frame, Data length = 8 bytes, 'data' = array of data bytes to send
        // send data frame 2:  ID = 0x101, Standard CAN Frame, Data length = 8 bytes, 'data' = array of data bytes to send
        if (CAN0.sendMsgBuf(0x100, 0, 8, data1) == CAN_OK)
            CAN0.sendMsgBuf(0x101, 0, 8, data2);
    }
}

//===================================================================================
//                                  FUNCTIONS
//===================================================================================
void initializeMCP2515()
{
    // Initialize MCP2515 running at 16MHz with a baudrate of 500kb/s and the masks and filters disabled.
    if(CAN0.begin(MCP_ANY, CAN_500KBPS, MCP_16MHZ) == CAN_OK)
        Serial.println("MCP2515 Initialized Successfully!");
    else
        Serial.println("Error Initializing MCP2515...");

    CAN0.setMode(MCP_NORMAL);   // Change to normal mode to allow messages to be transmitted
}

void initializeMPU6050()
{
    Wire.begin();
    Wire.beginTransmission(MPU_addr);
    Wire.write(0x6B);  // PWR_MGMT_1 register
    Wire.write(0);     // set to zero (wakes up the MPU-6050)
    Wire.endTransmission(true);
}

void readFromMPU6050()
{
    Wire.beginTransmission(MPU_addr);
    Wire.write(0x3B);  // starting with register 0x3B (ACCEL_XOUT_H)
    Wire.endTransmission(false);
    Wire.requestFrom(MPU_addr, 14, true); // request a total of 14 registers
    AccX = Wire.read() << 8 | Wire.read(); // 0x3B (ACCEL_XOUT_H) & 0x3C (ACCEL_XOUT_L)
    AccY = Wire.read() << 8 | Wire.read(); // 0x3D (ACCEL_YOUT_H) & 0x3E (ACCEL_YOUT_L)
    AccZ = Wire.read() << 8 | Wire.read(); // 0x3F (ACCEL_ZOUT_H) & 0x40 (ACCEL_ZOUT_L)
    Tmp = Wire.read() << 8 | Wire.read();
    GyX = Wire.read() << 8 | Wire.read(); // 0x43 (GYRO_XOUT_H) & 0x44 (GYRO_XOUT_L)
    GyY = Wire.read() << 8 | Wire.read(); // 0x45 (GYRO_YOUT_H) & 0x46 (GYRO_YOUT_L)
    GyZ = Wire.read() << 8 | Wire.read(); // 0x47 (GYRO_ZOUT_H) & 0x48 (GYRO_ZOUT_L)
}

void convertDataToPackets()
{
    long2Byte(currMillis, 0, data1);
    int2Byte(AccX, 4, data1);
    int2Byte(AccY, 6, data1);

    int2Byte(AccZ, 0, data2);
    int2Byte(GyX, 2, data2);
    int2Byte(GyY, 4, data2);
    int2Byte(GyZ, 6, data2);
}

void int2Byte(int16_t input, int pos, uint8_t *data)
{
    for (int i = 0; i < 2; i++)
    {
        data[i + pos] = ((input >> (i * 8)) & 0xff);
    }
}

void long2Byte(uint32_t input, int pos, uint8_t *data)
{
    for (int i = 0; i < 4; i++)
    {
        data[i + pos] = ((input >> (i * 8)) & 0xff);
    }
}

/*********************************************************************************************************
  END FILE
*********************************************************************************************************/

Receiver:
// CAN Receive Example
//

#include <mcp_can.h>
#include <SPI.h>

long unsigned int rxId;
unsigned char len = 0;
unsigned char rxBuf[8];
uint8_t receivedFirstFrame;

char msgString[128];                        // Array to store serial string
uint32_t TimeMicro;
int16_t AccX, AccY, AccZ, GyX, GyY, GyZ;

#define CAN0_INT 2  // Set INT to pin 2
MCP_CAN CAN0(10);   // Set CS to pin 10

void setup()
{
    Serial.begin(115200);

    initializeMCP2515();

    Serial.println("MCP2515 Library Receive Example...");
    receivedFirstFrame = 0;
}

void loop()
{
    if(!digitalRead(CAN0_INT))
    {
        // Received a message
        if (CAN0.readMsgBuf(&rxId, &len, rxBuf) == CAN_OK)
        {
            // Note: the above line is not like in the examples, but looking at the code the
            // function may return if no message was in the queue, so I think this is safer

            // Let's check if it is one of the known messages
            switch (rxId)
            {
            case 0x100:
                { // Received first frame
                    // Decode the content and store that the first frame was received
                    decodeFirstFrame(rxBuf);
                    receivedFirstFrame = 1;
                }
                break;
            case 0x101:
                if (receivedFirstFrame)
                { // Received second frame after a first frame
                    // Decode the content
                    decodeSecondFrame(rxBuf);

                    printDataToSerial();

                    receivedFirstFrame = 0;
                }
                break;
            }
        }
    }
}

//===================================================================================
//                                  FUNCTIONS
//===================================================================================

void initializeMCP2515()
{
    // Initialize MCP2515 running at 16MHz with a baudrate of 500kb/s and the masks and filters disabled.
    if(CAN0.begin(MCP_ANY, CAN_500KBPS, MCP_16MHZ) == CAN_OK)
        Serial.println("MCP2515 Initialized Successfully!");
    else
        Serial.println("Error Initializing MCP2515...");

    CAN0.setMode(MCP_NORMAL); // Change to normal mode to allow messages to be transmitted

    pinMode(CAN0_INT, INPUT); // Configuring pin for /INT input
}

void decodeFirstFrame(uint8_t *buffer)
{
    TimeMicro = byte2Double(0, buffer);
    AccX = byte2Int(4, buffer);
    AccY = byte2Int(6, buffer);
}

void decodeSecondFrame(uint8_t *buffer)
{
    AccZ = byte2Int(0, buffer);
    GyX = byte2Int(2, buffer);
    GyY = byte2Int(4, buffer);
    GyZ = byte2Int(6, buffer);
}

void printDataToSerial()
{
    Serial.print(TimeMicro);
    Serial.print("\t");
    Serial.print(AccX);
    Serial.print("\t");
    Serial.print(AccY);
    Serial.print("\t");
    Serial.print(AccZ);
    Serial.print("\t");
    Serial.print(GyX);
    Serial.print("\t");
    Serial.print(GyY);
    Serial.print("\t");
    Serial.println(GyZ);
}

uint32_t byte2Double(int pos, uint8_t *data)
{
    uint32_t output = 0;
    for (int i = 0; i < 4; i++)
    {
      output += ((uint32_t)data[i + pos]) << (i * 8);
    }
    return output;
}

int16_t byte2Int(int pos, uint8_t *data)
{
    int16_t output = 0;
    for (int i = 0; i < 2; i++)
    {
      output += ((int16_t)data[i + pos]) << (i * 8);
    }
    return output;
}
/*********************************************************************************************************
  END FILE
*********************************************************************************************************/

Please let me know if you have any comments or find some issues with this.

Code works perfectly after a few compiling issues. Only interesting one is that I had to serial. Print the data directly after the "CAN0. readMsgBuf" function. Could it be because the data is being overwritten in rxBuf after the "decodeSecondFrame"? I've now got the file to work though. Thankyou for taking the time to help teach me. And what should I do with the code that works? Put it at the bottom of the question?

@RossHanna If you tell me the compiling issues I can fix the code so that future readers don't have to figure out what is wrong ;) About the issue you are having, it's my fault. I should have learned not to call variables in similar way. Please correct the code with the two decode functions I edited above, and put back the print in the position it was before (after the readMsgBuf is wrong, since it does not print valid data but also partial). As for the code, it is enough to accept this answer like you did

1 - changed data 1 to variable type to uint8_t. 2 - define currentMillis in "convertDataToPackets". 3 - remove data array size "[2][i + pos]". 4 - set all variables to zero at end of main loop or they just increment. then move serial prints to after decode functions.

For the sake of fixing the code: the functions byte2Double and byte2Int in the receiver sketch still contain the original code with the error as mentioned in my answer.

@RossHanna I should have fixed the code for issues 1, 2 and (if I understood correctly) 3. The point 3 was related to data[4] and data[2] in the byte2Double and byte2Int functions? Point 4 is also solved, but not in the way you think (the old functions were wrong; I fixed them in the "correct" - IMHO - way)

@oh.dae.su Thank you, I missed that point (fixed it in the sender, not in the receiver); I integrated this in the code in the answer

There are indeed a few coding issues in your sketch, especially in the receiver part. I will try to list them and suggest changes.

If you look closely to your received data, you will see that AccX==GyY and AccY==GyZ on every line of your output. This is because you use the global variable rxBuf1 in your byte2Int and byte2Double functions on the receiver side. This causes the output to consume rxBuf1 also for the second CAN frame. Instead you should use the pointer to data that you have prepared in the inputs for this purpose.
In the receiver sketch, for both times you read in the CAN message with  CAN0.readMsgBuf you write the result to rxBuf1, although for the second read you most likely intended to write to rxBuf2. Now you simply overwrite rxBuf1 and therefore the content of the first received CAN message in the loop is never evaluated.
The switched outputs every few lines, in my opinion, are most likely due to a mixed order of both messages on the receiver side, when you receive CAN message 2 before CAN message 1. In order to avoid this, please follow the nice suggestions by @frarugi87 to send both messages with separate CAN IDs and to check for receiption of both before printing them.
I also agree that you can get rid of the acknowledge reply from the receiver. The can protocol already contains an acknowledge mechanism on the CAN transceiver that will notify the sender that at least one listening node has correctly received the message. If the sender does not get this acknowledge indication, it will repeat to send the message for some time.

I used frarugi87's code changes and it works great. My only remaining question. Using two separate ID's for each packet? is that how it would be done in a real system? i was attempting to get all the data in one transmission as i thought that is the way it would be done in a real system?

Typically, one CAN-ID is used for a fixed payload of up to 8 byte. So normally when you retransmit a CAN-ID it contains the same variables (however with updated values). When you have more than 8 byte of information that you want to transmit regularly, you either split it up into several CAN-IDs or you use a transport protocol such as SAE J1939 on top of the regular CAN-protocol in the data-link layer.


text:
send more than one byte at a time SPI Arduino
I have recently been trying to send integers via SPI between two arduino Uno's. I am using the first example of Nick Gammons tutorial, and trying to modify it to get a basic understanding of how to communicate via SPI.
http://www.gammon.com.au/spi
MASTER
// Written by Nick Gammon
// February 2011

#include <SPI.h>

void setup (void)
{

  digitalWrite(SS, HIGH);  // ensure SS stays high for now

  // Put SCK, MOSI, SS pins into output mode
  // also put SCK, MOSI into LOW state, and SS into HIGH state.
  // Then put SPI hardware into Master mode and turn SPI on
  SPI.begin ();

  // Slow down the master a bit
  SPI.setClockDivider(SPI_CLOCK_DIV8);

}  // end of setup

void loop (void)
{

  char c;

  // enable Slave Select
  digitalWrite(SS, LOW);    // SS is pin 10

  // send test string
  for (const char * p = "Hello, world!\n" ; c = *p; p++)
    SPI.transfer (c);

  // disable Slave Select
  digitalWrite(SS, HIGH);

  delay (1000);  // 1 seconds delay
}  // end of loop

SLAVE - WITHOUT INTERRUPTS
// Written by Nick Gammon
// February 2011

#include <SPI.h>

char buf [100];
volatile byte pos;
volatile boolean process_it;

void setup (void)
{
  Serial.begin (115200);   // debugging

  // turn on SPI in slave mode
  SPCR |= bit (SPE);

  // have to send on master in, *slave out*
  pinMode(MISO, OUTPUT);

  // get ready for an interrupt
  pos = 0;   // buffer empty
  process_it = false;

  // now turn on interrupts
  SPI.attachInterrupt();

}  // end of setup

// SPI interrupt routine
ISR (SPI_STC_vect)
{
byte c = SPDR;  // grab byte from SPI Data Register

  // add to buffer if room
  if (pos < (sizeof (buf) - 1))
    buf [pos++] = c;

  // example: newline means time to process buffer
  if (c == '\n')
    process_it = true;

}  // end of interrupt routine SPI_STC_vect

// main loop - wait for flag set in interrupt routine
void loop (void)
{
  if (process_it)
    {
    buf [pos] = 0;
    Serial.println (buf);
    pos = 0;
    process_it = false;
    }  // end of flag set

}  // end of loop

SLAVE - WITH INTERRUPTS
// Written by Nick Gammon
// April 2011

// what to do with incoming data
byte command = 0;

// start of transaction, no command yet
void ss_falling ()
{
  command = 0;
}  // end of interrupt service routine (ISR) ss_falling

void setup (void)
{

  // have to send on master in, *slave out*
  pinMode(MISO, OUTPUT);

  // turn on SPI in slave mode
  SPCR |= _BV(SPE);

  // turn on interrupts
  SPCR |= _BV(SPIE);

  // interrupt for SS falling edge
  attachInterrupt (0, ss_falling, FALLING);

}  // end of setup

// SPI interrupt routine
ISR (SPI_STC_vect)
{
  byte c = SPDR;

  switch (command)
  {
  // no command? then this is the command
  case 0:
    command = c;
    SPDR = 0;
    break;

  // add to incoming byte, return result
  case 'a':
    SPDR = c + 15;  // add 15
    break;

  // subtract from incoming byte, return result
  case 's':
    SPDR = c - 8;  // subtract 8
    break;

  } // end of switch

}  // end of interrupt service routine (ISR) SPI_STC_vect

void loop (void)
{
// all done with interrupts
}  // end of loop

ORIGINAL ISSUE - I am able to get this working, so the wiring is correct, however i am unable to get it to send anything other than a string. I can change the string and that works, but i am getting confused with pointers in C. Any code i have tried doesn't compile correctly, or doesn't send at all if it does compile. Could someone give me a clue as to where to start with this. My best effort is below
MASTER
#include <SPI.h>

int x = 10000;
int y = 20000;
int z = 30000;
byte High_x = highByte(x);
byte Low_x = lowByte(x);
byte High_y = highByte(y);
byte Low_y = lowByte(y);
byte High_z = highByte(z);
byte Low_z = lowByte(z);

int X =  word(High_x<<8|Low_x);
int Y =  word(High_y<<8|Low_y);
int Z =  word(High_z<<8|Low_z);

  byte myArray[7];

void setup (void)
{

  Serial.begin(115200);
  digitalWrite(SS, HIGH);  // ensure SS stays high for now

  // Put SCK, MOSI, SS pins into output mode
  // also put SCK, MOSI into LOW state, and SS into HIGH state.
  // Then put SPI hardware into Master mode and turn SPI on
  SPI.begin ();

  // Slow down the master a bit
  SPI.setClockDivider(SPI_CLOCK_DIV8);

 myArray[0] = High_x;
 myArray[1] = Low_x;
 myArray[2] = High_y;
 myArray[3] = Low_y;
 myArray[4] = High_z;
 myArray[5] = Low_z;
 myArray[6] = 0;

  Serial.println(X);
  Serial.println(Y);
  Serial.println(Z);
  delay(2000);
}

void loop (void)
{

  byte data;

  // enable Slave Select
  digitalWrite(SS, LOW);    // SS is pin 10

  // send test string
  for (byte * p =  myArray[0]; data = *p; p++)
    SPI.transfer (data);
    Serial.println("sending data");

  // disable Slave Select
  digitalWrite(SS, HIGH);

  delay (1000);  // 1 seconds delay
}  // end of loop

SLAVE
#include <SPI.h>

byte buf [100];
volatile byte pos;
volatile boolean process_it;

void setup (void)
{
  Serial.begin (115200);   // debugging

  // turn on SPI in slave mode
  SPCR |= bit (SPE);

  // have to send on master in, *slave out*
  pinMode(MISO, OUTPUT);

  // get ready for an interrupt
  pos = 0;   // buffer empty
  process_it = false;

  // now turn on interrupts
  SPI.attachInterrupt();
Serial.println("End of Setup");
}  // end of setup

// SPI interrupt routine
ISR (SPI_STC_vect)
{
  Serial.println("Beginning of Interrupt Routine");
byte data = SPDR;  // grab byte from SPI Data Register

  // add to buffer if room
  if (pos < (sizeof (buf) - 1))
    buf [pos++] = data;

  // example: newline means time to process buffer
  if (data == '0')
    process_it = true;
      Serial.println("End of Interrupt Routine");
}  // end of interrupt routine SPI_STC_vect

// main loop - wait for flag set in interrupt routine
void loop (void)
{
    Serial.println("Beggining of Loop");

  for(int i = 0; i < 100; i++)
  if (process_it)
    {
    buf [pos] = 0;
    Serial.println (buf[i]);
    pos = 0;
    process_it = false;
    }  // end of flag set
    Serial.println("End of Loop");
}  // end of loop

In the future i hope to send 10 16-bit integers over SPI from an arduino to an arduino or maybe arduino to a raspberry Pi.
Thanks
EDIT -I have started using the code that uses interrupts and managed to get the data i need sent across, however, i can only get the correct data by setting both the first and second element of the array to "High_x", otherwise it skips the first part of data?
NEW MASTER
#include <SPI.h>

int x = 10000;
int y = 20000;
int z = 30000;
byte High_x = highByte(x);
byte Low_x = lowByte(x);
byte High_y = highByte(y);
byte Low_y = lowByte(y);
byte High_z = highByte(z);
byte Low_z = lowByte(z);

int X =  word(High_x<<8|Low_x);
int Y =  word(High_y<<8|Low_y);
int Z =  word(High_z<<8|Low_z);

  byte myArray[8];

void setup (void)
{

  digitalWrite(SS, HIGH);  // ensure SS stays high for now

  // Put SCK, MOSI, SS pins into output mode
  // also put SCK, MOSI into LOW state, and SS into HIGH state.
  // Then put SPI hardware into Master mode and turn SPI on
  SPI.begin ();

  // Slow down the master a bit
  SPI.setClockDivider(SPI_CLOCK_DIV8);

 myArray[0] = High_x;
 myArray[1] = High_x;
 myArray[2] = Low_x;
 myArray[3] = High_y;
 myArray[4] = Low_y;
 myArray[5] = High_z;
 myArray[6] = Low_z;
 myArray[7] = 0;

  Serial.print("wait for loop");
  //Serial.println(X);
  //Serial.println(Y);
  //Serial.println(Z);
  delay(2000);
}

void loop (void)
{

  byte data;

  // enable Slave Select
  digitalWrite(SS, LOW);    // SS is pin 10

  // send test string
  for (byte * p =  &myArray[0]; data = *p; p++){
    SPI.transfer (data);
  }
  // disable Slave Select
  digitalWrite(SS, HIGH);

  delay (1000);  // 1 seconds delay
}  // end of loop

NEW SLAVE
byte buf [100];
volatile byte pos;
// what to do with incoming data
byte command = 0;

// start of transaction, no command yet
void ss_falling ()
{
  command = 0;
}  // end of interrupt service routine (ISR) ss_falling

void setup (void)
{

  Serial.begin(9600);

  // have to send on master in, *slave out*
  pinMode(MISO, OUTPUT);

  // turn on SPI in slave mode
  SPCR |= _BV(SPE);

  // turn on interrupts
  SPCR |= _BV(SPIE);

  // interrupt for SS falling edge
  attachInterrupt (0, ss_falling, FALLING);

}  // end of setup

// SPI interrupt routine
ISR (SPI_STC_vect)
{
  byte data = SPDR;

  // add to buffer if room
  if (pos < (sizeof (buf) - 1))
    buf [pos++] = data;

}  // end of interrupt service routine (ISR) SPI_STC_vect

void loop (void)
{

  for(int i = 0; i < 100; i++){

    Serial.println (buf[i]);
    pos = 0;
    delay(1000);
  }
}  // end of loop

Any suggestions on why this is necessary? Is there some basic SPI idiosyncrasy I'm missing?
Thanks all
Ross

You have a couple of issues going on here.
First of all, Nick Gammon's example is transmitting strings. Strings have the property that a particular value, '\0' (null), cannot normally appear in the string itself, and it can therefore be used as an "end flag" to mark the end of the string. This is why the for loop on the master side works correctly.
Note that the null itself is NOT transmitted over the SPI link. Instead, Nick shows an example of using another character ('\n' - newline) to trigger processing on the slave side.
If you want to transmit binary data, which could easily contain bytes of 0x00 (null), you need to find a different way to delimit the data on both the sending side and the receiving side. Your example data (16-bit integers of 10000, 20000 and 30000) doesn't contain any null bytes, but you can't rely on that in general. Also, because the null doesn't actually get transmitted, you can't use it on the receiving side to trigger processing.
There are a couple of approaches that are frequently used to send binary data:

Send the number of bytes explicitly at the beginning of the transaction, and trigger receive-side processing once that number of bytes has been received.
Use a "special" value (e.g., null) to mark the end of the data, but then you have to "escape" that value in some way if it ever appears in the actual data.

In addition to all of that, you have a bug on your master side code. The initialization of your for loop, byte * p =  myArray[0] does not set the pointer p to the address of myArray as intended; instead, it sets it to the value of the first byte in the array, which is 0x27 (High_x). You need to say either byte * p =  &myArray[0] or simply byte * p =  myArray.

If it were me, I would probably use SS to delimit the data explicitly. Nick shows examples of this using either polling or interrupts.

No, he doesn't collect it on the receive side in that example; he just processes it directly in the ISR and sends it out again. But there's no reason you can't put the data into a buffer just like in the earlier slave-side examples.

Edit your question, and replace your code with the latest version. Note that I removed Nick Gammon's name from your code. I doubt that he would want to be associated with code that he didn't write.


text:
Analog vs Digital Accelerometers - pro's and con's?
For my project i need to measure accelerations up to 16g. My options are either digital accelerometers such as the MPU6050 or MPU9250 or an analog accelerometer such as the ADXL326.
What are any major pro's and con's of each. number of wires would be almost equal. the digital accelerometer is less pins on the board but i could easily use analog to digital convertors to bypass that. sampling rate is lower at 550Hz for the analog vs 4000Hz for the MPU9250. However i believe 550Hz would be more than adequate. And the biggest difference for me. £2 for an MPU9250 and £12 for the ADXL326!
Does anyone have any general advice on the main differences when choosing between analog and digital? I am measuring suspension motion on a motorcycle and need 5 of these accelerometers.

Is the motorcycle supposed to be driving while you take your measurements? I suspect you would only capture vibrations rather than the (relatively slow) movement of the suspension components relative to each other. Hall sensors with a mechanical linkage are traditionally used to measure suspension movements on cars.
IR distance sensor may also be an option.

Very interesting point. I successfully recorded suspension motion of a motorcycle moving over a specially designed obstacle and compared to a numerical simulation and the answers were relatively close. I've never heard of hall sensors being used. Would you mind sharing a link to an example of such a system or device? I've only seen linear potentiometers used.

It's mostly used in finished products, but both my BMW E39 and MB W211 use hall based sensors for regulation the air suspension (and automatic headlight adjustment) functions. Have a look for "W211 Höhenstandgeber" to see an example, I'm not sure about the licences on those pictures.

Thanks, i had a look at the sensor, and it seems that it uses a linkage system with a hall sensor on a rotary shaft or similar? The type of system i was looking at was more similar to a suspension potentiometer. However i might start a new thread for people to comment on alternatives, but i will still test the accelerometer method as it is an interesting proposal in my mind if the small accelerations from vibration could be filtered out.

The biggest difference is where the measurement (the conversion to digital) is done.  On the digital accelerometer, it is done in the accelerometer itself. On the analog one it is done at the MCU.
For simple arrangements where the accelerometer is near the MCU that makes no difference. However the analog one is more susceptible to noise. With longer wires / traces between the accelerometer and the MCU you can get less accurate results with an analog one since it picks up noise which you are then sampling and including in your results.
The digital one is more immune to that kind of interference - instead it is susceptible to digital noise - corruption of the data as it traverses long wires. But that is less of a problem since it takes far more induced voltage to corrupt a digital signal than an analog one.
One other main difference is that you are limited to the ADC that is in the digital accelerometer. If you want a different resolution you have to use a different accelerometer. With analog, since it has no inherent resolution, you sample at the resolution of your MCU's ADC.
Digital is also easier to interface in modern systems.  Many SoCs don't have analog inputs. So adding an analog accelerometer would mean adding an ADC chip as well - and you don't really want to do that in something like a mobile phone.
So in summary:

Digital is the most convenient, reliable and suitable for current basic uses
Analog is more of a niche product that is rarely used in consumer electronics

The ADXL326 is only an accelerometer.
When you have one, you can measure the output of the sensor with a oscilloscope. For the final project a digital sensor is preferred as @Majenko has explained in his answer.
The MPU-9250 is an accelerometer, gyro and magnetometer. That makes it possible to use sensor fusion (combining the sensors with software calculations for a much improved result).
Are there vibrations involved? An accelerometer is sensitive for vibrations. I doubt if an accelerometer can be used to measure the suspension motion.
By the way, I'm ignoring the MPU-6050 and so should you. The MPU-9250 is less noisy.


text:
Should I buy Arduino as a Computer Engineering Student?
I'm going to be in the third year of university in a month. I'm satisfied with the software part of my bachelor but I feel like I lack in practice in practical implementation. Would getting an Arduino board and playing around with it worth it and useful for me?

Depends om how much time you want to spend and what are your goals. You can learn a lot about resource constraint software system development from simple boards such as the Arduino. But it is all about how much time you have to spend. The board(s) is cheap in comparison. You could just start off by reading and trying to understand some of the source code that builds the Arduino core and libraries. Often university curriculum's miss teaching how to read large source code systems.

You don't mention your level of experience working with hardware. Do you have experience working with electronic circuits?

Mikael Patel, you should post this as an answer.

It depends. If you don't have any knowledge about microcontrollers it might be useful.
However, Arduino hides a lot of concepts in their libraries. If you want to go more indepth check other systems like STM32 or PIC.
You also can do both, start with Arduino to get a rough feeling, continue with another.
More important is WHAT you try, don't stop with LEDs/switches, but check communication protocols like SPI, I2c, UART etc. And learn what timers, interrupts etc. are and how to use them.

and networking for IoT

The original Arduino boards are based on 8 bit  AVR microcontrollers, and using Arduino libraries with them is entirely optional. There is a lot of information about programming AVR microcontrollers in C and assembly, and the datasheets are quite readable. With STM32, low-level programming is a lot more difficult to learn because various HALs are preferred by developers. Even finding a basic register/pin definition file is a challenge if you do not wish to use any HAL. For this reason I would argue against STM32 for learning low-level programming.

@Im_Int not sure if I agree, about 99% using Arduino use the IDE which hides a lot of the internals (which is ok for most people). With STM32 the HAL is used, but not as often as the IDE for Arduino (relatively). There are many different STM32s/IDEs and ways to program (HAL, CMSIS, StdLib). I didn't mean he needs to go so low level, but the Arduin commands hide so much it's barely needed to understand how a microcontroller works.

As already stated in previous answers, there is no clear-cut answer to
this kind of question. It all depends on what you want to learn.
If that can help, I will share my personal experience. I studied
computer science for only one year at university. Then I stopped to
continue on my physics major. But as I never lost interest in computers,
I continued learning on my own. I started playing with Arduinos just
because it looked like a fun way to learn some microcontroller stuff and
low-level programming. The barrier to entry is extremely low: in no time
you get your LED blinking, and from there you can start exploring plenty
of possibilities. The Arduino IDE and core library make it extremely
easy to make simple programs, but they are quite limited when you want
to get more serious. For many people, this environment, plus extra
downloadable libraries, is all they need to achieve their goals. For me,
it was a stepping stone to learn lower level stuff, like direct hardware
access, use of the avr-libc, and even assembly programming.
Today, what I love most about Arduinos (or, rather, the AVR
microcontrollers at their core), is their simplicity and
understandability. Today, when I disassemble a program I have written,
I can understand basically everything: the C runtime initialization
code, the memory allocation, the way the compiler translates my own
code... I find it extremely rewarding to reach such an understanding of
a (kind of) computer system.
The same level of understanding would be (for me) impossible to achieve
on a typical PC. Even though I use Linux, which is Open Source, and I
have access to the full source code, the whole software stack involved
in running even the simplest program is overwhelming. The kernel itself
is like millions of lines of code. Even the assembly language of an x86
(or x86_64) is something I find way too complex for my taste.
In conclusion, the Arduino has been for me a very rewarding learning
experience. If you want to get your hands dirty with bare metal
programming, I definitely recommend you get an Uno, or something
similar, and start playing with it. Once you get your first program
running (and that takes no time!) start looking under the hood: read the
source of the Arduino core (it's short and very readable), read the
datasheet of the microcontroller, start poking the
hardware, take a look at how the compiler allocates
memory, how it translates your source code...

I don't think there is a should or should not about it, do you want a low cost experimental setup using the C language to toy with ideas and concepts without a massive outlay or proprietary software or NDA's to get chip data that ties you to a single company then why not? You can get Arduino clones very cheaply from the likes of aliexpress etc as well as prototyping kits. While the Arduino IDE is a relatively primitive editor you can use any programmers editor you like to compose your C files and use the Arduino IDE just to do the final compile and linking and upload/burn them to the target device. It can be done so cheaply that even if you decide it does not inspire your imagination you won't have wasted hundreds of $! Just spent a couple of tens of $ to accumulate a significant micro-controller playground! Assuming you already have some kind of personal computer or easy access to one.


text:
Am I at the limit of UART transmission time?
I'm currently using an STM32F103C8 (With an arduino core) to read 4 sensors at 1kHz, then send that data via UART (at 115200 baud) to another device. Each sensor returns a float, I also include an unsigned long representing milliseconds since boot time. This adds up to be a 20 byte package, that I attempt to send at 1kHz. Sensor sampling is carried out via a timer interrupt, as it is crucial the time between samples is constant.
The data transmission itself (If I send it slowly) takes approx. 70 microseconds. When sped up to 1kHz, it takes approx. 2500. I assume this is because I am filling a serial buffer of some sort.
Is this about the limit of the communication method? If it takes longer than 1000 microseconds to send the data, then presumably the interrupt will trigger before the data is sent and it will have to start again?

You could try a higher baudrate. Though that depends on how high a baudrate the other device can handle

At 115200 baud you can send 11520 characters per second. At 20 bytes per packet you can send 576 packets per second. That's 1,736µs per packet.
That's assuming zero wait between bytes. That's not going to happen.
Your measurement of 70µs will just be the time taken to place data into the transmit buffer, and has nothing to do with how long the transmission itself takes.
Roll all that together and the value of 2,500µs per packet is not unreasonable.
To transmit faster you will simply have to run at a faster baud rate.  If you're getting 2500µs per packet currently and a packet takes 1736µs (call it 1,750 for simplicity) then there's about a 750µs overhead per packet to do the actual work of sending the data from the buffer. I'll assume this is a fixed value, though with no knowledge of how the STM32F1 UART works it may not be.
If you want <1000µs then that leaves you about 250µs of time to fit the packet into. That means an equivalent of 4,000 packets per second, or 80,000 bytes per second, or a baud rate of 800,000baud.
1 MBaud is more than possible, and is a good standard baud rate number that most things can cope with. So raising your baud rate to 1,000,000 should allow the communication to happen fast enough.

Interesting! I wasn't really aware such high baud rates were feasible! I have had some success with a rate of 921600 rather than 1000,000, so presumably something to do with a multiple of crystal frequencies

The closer you are to an integer factor of your clock speed the more accurate the baud will be. But if both ends of the connection use the same clock the error difference between the two ends will be 0 so is meaningless.

Of course at higher baud rates you're more at the mercy of the clock accuracy, as any slight deviation in clock will cause a bigger percentage of difference in the baud rate.

I tested a lot of baud rates on Arduino Unos, including cheap Chinese models, and decided to use a default 115200 for my DaqPort system (https://www.daqarta.com/dw_rraa.htm).  I had found problems, like the system hanging after a second or so at 150000 baud, and didn't want to have my users run into issues that would be hard to troubleshoot.
So I'm wondering if you can reduce the bytes you need to send.  Do you really need to send floats?  I'm thinking 4 bytes for the data and 4 for the time would give ample resolution for almost anything, and I can't (off the top of my head anyway) think of a sensor type with more than 32-bit resolution. Also, 32 bits of msec is nearly 50 days; if you need more than that it should be pretty easy to unwrap the times after the fact.
Just a thought...

I had honestly completely forgotten about shorts as a datatype, my sensors are only 12 bit anyway so that's going to help a huge amount. I should be able to just store a raw sensor reading as a short and post-process everything to floating point later. I'm using microseconds rather than millis for time, so the unsigned long is only 70 minutes - my max sampling duration is 20 minutes, so too long for an unsigned int.


text:
Best buffering practice for continuous SD card writing
In a simplified example, I have a microcontroller functioning as a data acquisition device - sensor data is received over UART, and then saved to an SD card.
This received data is in the form of a 12 byte struct, received at 1kHz. The microcontroller receiving the data is an esp32, with an SD card connected over spi. Not an uncommon issue, but if a typical SD card write is triggered everytime new data is received, it will take longer than the gap between samples to  write the data.
What is typically considered the best practice for this type of system, presuming the data will always be received faster than it can be written. I have seen examples of circular buffers, but presumably these will eventually fill completely and samples will be missed.
In my particular example, I have the bonus second cpu core in the esp32. Therefore, my thinking is to use 2 alternating buffers. Data will be received on core 1, and placed into the active buffer. When the buffer is full, incoming data will be written to the second buffer, and the first will be written to the SD card by core 2.
Is this a sensible solution to the problem? I am hesitant to implement anything that simply relies on an improvement in SD write speed, as I will likely need to increase the size of the data payload in the future and don't want to run into this issue again later on.

data will always be received faster than it can be written ... that means that you will lose data ... data compression may help .... please provide few examples of the 12 bytes of data

SDCards can be written over a 1 bit bus or 4 bit bus.  The latter is faster and, as I recall, requires a license to implement.  Also, the speed at which SDCards work varies greatly.  If your driver can take advantage of this, try a faster SDCard.  Finally, all SDCards contain a propriety controller.  So picking the right OEM for your needs may help.  (I'd go w/@jsotola recommendation 1st, try losses compression if you can.  Simple is best.)

if a typical SD card write is triggered everytime new data is received, -- It isn't. An internal 512 byte buffer is maintained by the SD library to store an entire data block. When that is filled it will flush it to the SD card in one go and get (or allocate) the next.

I suppose the question is not so much how the SD writing process itself can be sped up, but how incoming data can be preserved while the write is taking place

Is this a sensible solution to the problem?

No, unfortunately not. The size of the buffer is limited by the available RAM on your chip. Jumping between multiple buffers or cores will not change that, since it is a hardware limitation.

I am hesitant to implement anything that simply relies on an improvement in SD write speed, as I will likely need to increase the size of the data payload in the future and don't want to run into this issue again later on.

When you are continuously receiving faster than you can write, you only have 2 options:

Improve write speed
Reduce the needed data size

There is no other way, and that's a logical conclusion, not so much a programmical or hardware conclusion. It that situation the amount of "buffered" data will increase indefinitely, so you would need an infinite amount of memory. And that doesn't exist in real world. The same principle would apply to a high end PC with lots of memory, it will just be full after longer time, than with an ESP.

Improving write speed:

On various sites I have read, that you can use the SD card with different SPI frequencies. A higher frequency would mean a higher transaction speed. You can investigate further at this point.
Writing a full 512 byte page at once and only closing and opening the file again, if really needed, will improve the overall speed.

Reduce the needed data size

Maybe you can compromise with your data resolution and then pack those values as dense as possible (like packing two 4-bit values into a single byte of transmission).
You can research for compression code, so that you can compress your data lossless before transmitting it. Though the result will always depend on the kind of your data. Also mind, that in this case you are trading transmission size for computation time and RAM.

About buffers:
I would say, a normal circular buffer does not bring you much advantages here, since with the SD card you don't want to do bytewise data transmission. The 2 buffer approach, that you mentioned, seems good to me. Though doing handling with different cores does not make much sense to me. The UART receiving itself is done by hardware and some ISRs. You can do all the SD card transmissions in your main code, just making sure, that you regularly copy the received data from the Serial buffer into your buffer.

Buffers will help you if at some point the input stream stops, so the "slower" device (e.g. the SD card) can catchup. If the data will always be received faster than it can be written (note always) then no buffer will help you.
But I assume you meant that there times where you get bursts of input data at 1Khz, then it will stop. So, as long as the average input data rate is lower than the ability for the SD to persist the data, you will be fine.
The SD card library already has a buffer (as @Majenko pointed out), so most likely you won't need to do anything. Seem like something you can test relatively simply.
The 2 core approach sounds somewhat overkill (not to mention some synchronization issues you will have to deal with)

Unfortunately in this case the input data is at a constant rate, start to finish with no interruptions. There's also up to a maximum of 30 minutes of data, meaning it's a bit outside the bounds of just writing to onboard memory and dumping to the SD card at the end, hence the 2 core idea. If a full SD 512 byte write only takes 3000us, data can be written at an average speed higher than its coming in - but what to do with the data received while the SD write is occurring?

One approach would be to use Circular Buffering. It involves using a fixed-size buffer that wraps around itself, allowing data to be continuously written and overwritten. This approach eliminates the need for costly memory reallocation or shifting operations. As new data is written, older data is overwritten in a circular fashion. But, you will need to choose the appropriate buffer size. A larger buffer size can help handle bursts of data However, larger buffers also consume more memory.
If one buffer isn't enough, then you could use Two buffers instead. While one buffer is being written to the SD card, the other buffer can be used to receive new incoming data. Once the write operation is completed, the roles of the buffers are swapped. This technique can help improve efficiency by minimizing data transfer and write latency, which is the goal you aim at. I've used this type of buffer for audio data acquisition, and it worked well, got rid of the buffer overflow issues.
The other method would be to use DMA,  if your microcontroller or system supports it, which I think it does, using DMA for SD card writing can offload the data transfer task from the CPU. DMA allows direct memory access without CPU intervention, enabling faster and more efficient data transfers. DMA can significantly reduce the overhead associated with continuous SD card writing.
For DMA, you might want to check this:
https://github.com/RobTillaart/DAC8552/issues/8 and this: https://github.com/espressif/esp-idf/blob/master/examples/peripherals/spi_master/lcd/main/spi_master_example_main.c

Forgive my non-technical analogy, but this is how my brain works.   Hope you can get past the conclusion that "this isn't a technical answer", until you at least see it from this different perspective.
Buffers, input streams, and sampling rates, storage rates all relate strongly to filling up a bath.   Taps, water, baths and plug holes?   WTF?
Try slightly blocking the plug hole in your bath, and turn both taps on full.
The water that's drained away is the rate of which data is stored to the SD card.  That's the storage-rate.
The remaining water which doesn't drain away fast enough causes the bath the fill up.   The water is the sampling rate, the bath is the buffer.
If the taps remain on, and water is leaving the taps faster than the water can escape from the partially blocked hole will cause the bath to fill up with water.
So, if the blockage isn't cleared, the bath will eventually overflow.
Either you turn the taps down, use a bigger bath, or increase the drainage rate.


text:
Differing Serial Values Between Arduino Uno and TTGO ESP32 SX1276
I'm wanting to analogRead the value off a MQ-135 gas sensor. I have tired this with both an Arduino Uno and an TTGO ESP32 SX1276.
When reading the value of the sensor off the Uno, I get values approximately equal to 400 which is expected, as shown in this video. When doing the same on the TTGO, I get values close to 2000. For some reason, this person gets a similar value range as shown in this video.
What I don't understand is how these values are different when only the board is being changed. Do different boards analogRead differently?
More or less, how do I get the TTGO to read the same values as the Uno?
Below is my code for both boards. Nothing is changed between the boards:
#define sensor 2 //sensor pin

int gasLevel = 0; //int variable for gas level

void setup() {
  Serial.begin(9600); //start serial comms
  pinMode(sensor,INPUT); //set sensor for input
}

void loop() {
  gasLevel = analogRead(sensor);
  Serial.println(gasLevel);
  delay(1000);
}

The following things I have tested:

Multiple boards of both the Arduino Uno and TTGO
Different pins, both analog and digital

I also am powering the sensor using 3.3V pin, so there is no variation in the voltage the sensor is receiving.

Do different boards analogRead differently?

Yes.
Your typical Arduino has a 10-bit ADC. That can give values between 0 and 1023 (210-1), with 400 falling just short of half way.
The ESP32 has a 12-bit ADC. That gives values between 0 and 4095 (212-1), which means that your 2000 falls, yes, just short of half way.
It is not the value from the ADC that is important - it is what the value represents. It is meaningless until you convert it into something meaningful, like a voltage.
Like asking "How heavy is it?" -- "It's 23." Completely meaningless, until you qualify it with units, and the units are different because of the different resolutions, so you need to "convert" it into units that have a meaning.
Typically the value represents a proportion of the supply voltage (but not necessarily: it's actually a proportion of the ADC reference voltage, which is often the same thing as the supply voltage).  On the Arduino the value is 1/1024th of the supply voltage. On the ESP32 it's 1/4096th.

That makes complete sense. Thanks for the clarification.


text:
Elements excluded from buffer array output after given structure (ESP8266 WifiSniffer) (snifferPacket)
I am currently trying to determine how the ESP8266 WifiSniffer program by Kalanda works.
Here is the source code:
#include <Arduino.h>

extern "C" {
  #include <user_interface.h>
}

#define DATA_LENGTH           112

#define TYPE_MANAGEMENT       0x00
#define TYPE_CONTROL          0x01
#define TYPE_DATA             0x02
#define SUBTYPE_PROBE_REQUEST 0x04

struct RxControl {
//THESE HAVE AN EFFECT
// signed rssi:8; // signal intensity of packet
// unsigned rate:4;
// unsigned is_group:1;
// unsigned:1;
// unsigned legacy_length:12; // if not 11n packet, shows length of packet. NOT THIS
// unsigned MCS:7; // if is 11n packet, shows the modulation and code used (range from 0 to 76)
// unsigned Smoothing:1;
// unsigned Not_Sounding:1;
// unsigned:1;
// unsigned Aggregation:1;
// unsigned STBC:2;
// unsigned rxend_state:8;
// unsigned ampdu_cnt:8;
// unsigned channel:4; //which channel this packet in.

//THESE HAVE NO EFFECT
// unsigned sig_mode:2; // 0:is 11n packet; 1:is not 11n packet;
// unsigned damatch0:1;
// unsigned damatch1:1;
// unsigned bssidmatch0:1;
// unsigned bssidmatch1:1;
// unsigned CWB:1; // if is 11n packet, shows if is HT40 packet or not
// unsigned HT_length:16;// if is 11n packet, shows length of packet. NOT THIS
 //unsigned FEC_CODING:1; // if is 11n packet, shows if is LDPC packet or not.
 //unsigned SGI:1;
 //unsigned:12; //HAS NO EFFECT
};

struct SnifferPacket{
    struct RxControl rx_ctrl; //RxControl is just a name, can be named anything (as long as you change other things)
    uint8_t data[DATA_LENGTH]; //DOES NOT HAVE EFFECT
//    uint16_t cnt; //THIS HAS NO EFFECT
//    uint16_t len; //THIS HAS NO EFFECT
};

// Declare each custom function (excluding built-in, such as setup and loop) before it will be called.
// https://docs.platformio.org/en/latest/faq.html#convert-arduino-file-to-c-manually
//static void showMetadata(SnifferPacket *snifferPacket);
//static void sniffer_callback(uint8_t *buffer, uint16_t length);
static void printDataSpan(uint16_t start, uint16_t size, uint8_t* data);
static void getMAC(char *addr, uint8_t* data, uint16_t offset);
void channelHop();

/**
 * Callback for promiscuous mode
 */
static void sniffer_callback(uint8_t *buffer, uint16_t length) {

  struct SnifferPacket *snifferPacket;
  snifferPacket = (struct SnifferPacket*) buffer;

  Serial.print("buffer size: ");
  Serial.print(sizeof(buffer));

  Serial.print(" length size: ");
  Serial.print(sizeof(length));

  Serial.print(" snifferPacket size: ");
  Serial.print(sizeof(snifferPacket));

  Serial.print(" START OF BUFFER OG: ");
  for (int i = 0; i < 20; i++){
    Serial.print((unsigned int) buffer[i], BIN);
    Serial.print(" ");
  }
  Serial.print(" END OF BUFFER OG ");

  showMetadata(snifferPacket);
}

static void showMetadata(SnifferPacket *snifferPacket) {

  unsigned int frameControl = ((unsigned int)snifferPacket->data[1] << 8) + snifferPacket->data[0];

  uint8_t version        = (frameControl & 0b0000000000000011) >> 0;
  uint8_t frameType      = (frameControl & 0b0000000000001100) >> 2;
  uint8_t frameSubType   = (frameControl & 0b0000000011110000) >> 4;
  uint8_t toDS           = (frameControl & 0b0000000100000000) >> 8;
  uint8_t fromDS         = (frameControl & 0b0000001000000000) >> 9;
  uint8_t moreFragments  = (frameControl & 0b0000010000000000) >> 10;
  uint8_t retry          = (frameControl & 0b0000100000000000) >> 11;
  uint8_t powerMgmt      = (frameControl & 0b0001000000000000) >> 12;
  uint8_t moreData       = (frameControl & 0b0010000000000000) >> 13;
  uint8_t protectedFrame = (frameControl & 0b0100000000000000) >> 14;
  uint8_t htcOrder       = (frameControl & 0b1000000000000000) >> 15;

  Serial.print(" START OF BUFFER: ");
  for (int i = 0; i < 20; i++){
    Serial.print((unsigned int)snifferPacket->data[i], BIN);
    Serial.print(" ");
  }
  Serial.print(" END OF BUFFER ");

  Serial.print(" DATASPAN START: ");
  printDataSpan;
  Serial.print(" DATASPAN END ");

  Serial.print(" frameControl: ");
  Serial.print(frameControl);

  Serial.print(" version: ");
  Serial.print(version);

  Serial.print(" Type: ");
  Serial.print(frameType);

  Serial.print(" frameSubType: ");
  Serial.print(frameSubType);

  Serial.print(" toDS: ");
  Serial.print(toDS);

  Serial.print(" fromDS: ");
  Serial.print(fromDS);

  Serial.print(" moreFragments: ");
  Serial.print(moreFragments);

  Serial.print(" retry: ");
  Serial.print(retry);

  Serial.print(" powerMgmt: ");
  Serial.print(powerMgmt);

  Serial.print(" moreData: ");
  Serial.print(moreData);

  Serial.print(" protectedFrame: ");
  Serial.print(protectedFrame);

  Serial.print(" htcOrder: ");
  Serial.print(htcOrder);

  Serial.print(" Ch: ");
  Serial.print(wifi_get_channel());

//  Serial.print(" rssi: ");
//  Serial.print(snifferPacket->rx_ctrl.rssi, BIN);

//  Serial.print(" RSSI: ");
//  Serial.print(snifferPacket->rx_ctrl.rssi, DEC);

  //Only look for probe request packets
//  if (frameType != TYPE_MANAGEMENT ||
//      frameSubType != SUBTYPE_PROBE_REQUEST)
//        //Serial.println();
//        return;

//  Serial.print(" RSSI: ");
//  Serial.print(snifferPacket->rx_ctrl.rssi, DEC);

  Serial.print(" Ch: ");
  Serial.print(wifi_get_channel());

  char addr[] = "00:00:00:00:00:00";
  getMAC(addr, snifferPacket->data, 10);
  Serial.print(" Peer MAC: ");
  Serial.print(addr);

//  uint8_t SSID_length = snifferPacket->data[25];
//  Serial.print(" SSID: ");printDataSpan
//  printDataSpan(26, SSID_length, snifferPacket->data);

  Serial.println();
}

static void printDataSpan(uint16_t start, uint16_t size, uint8_t* data) {
  for(uint16_t x = start; x < DATA_LENGTH && x < start+size; x++) {
    Serial.print(data[x]);
    //Serial.write(data[x]);
  }
}

static void getMAC(char *addr, uint8_t* data, uint16_t offset) {
  sprintf(addr, "%02x:%02x:%02x:%02x:%02x:%02x", data[offset+0], data[offset+1], data[offset+2], data[offset+3], data[offset+4], data[offset+5]);
}

#define CHANNEL_HOP_INTERVAL_MS   100
static os_timer_t channelHop_timer;

/**
 * Callback for channel hoping
 */
void channelHop()
{
  // hoping channels 1-13
  uint8 new_channel = wifi_get_channel() + 1;
  if (new_channel > 13) {
    new_channel = 1;
  }
  wifi_set_channel(new_channel);
}

#define DISABLE 0
#define ENABLE  1

void setup() {
  // set the WiFi chip to "promiscuous" mode aka monitor mode
  Serial.begin(115200);
  //delay(10);
  wifi_set_opmode(STATION_MODE);
  wifi_set_channel(1);
  wifi_promiscuous_enable(DISABLE);
  //delay(10);
  wifi_set_promiscuous_rx_cb(sniffer_callback);
  //delay(10);
  wifi_promiscuous_enable(ENABLE);

  // setup the channel hoping callback timer
  //os_timer_disarm(&channelHop_timer);
  //os_timer_setfn(&channelHop_timer, (os_timer_func_t *) channelHop, NULL);
  //os_timer_arm(&channelHop_timer, CHANNEL_HOP_INTERVAL_MS, 1);
}

void loop() {
  //delay(10);
}

I have noticed the Serial.print output directly from the buffer differs to the Serial.print output for the variable snifferPacket which is given attributes of the struct RxControl.
When the code is configured as normal, the raw buffer output (BUFFER OG) has 12 additional elements compared to the snifferPacket (BUFFER) output.
Here is an example:
buffer size: 4 length size: 2 snifferPacket size: 4 START OF BUFFER OG: 10100010 10000 11100110 1010000 0 0 0 0 0 0 1 0 10000000 0 0 0 11111111 11111111 11111111 11111111  END OF BUFFER OG  START OF BUFFER: 10000000 0 0 0 11111111 11111111 11111111 11111111 11111111 11111111 11011000 111 10110110 11010 10100010 1000001 11011000 111 10110110 11010  END OF BUFFER  DATASPAN START:  DATASPAN END  frameControl: 128 version: 0 Type: 0 frameSubType: 8 toDS: 0 fromDS: 0 moreFragments: 0 retry: 0 powerMgmt: 0 moreData: 0 protectedFrame: 0 htcOrder: 0 Ch: 1 Ch: 1 Peer MAC: d8:07:b6:1a:a2:41

Comparing just the two buffer outputs:
 - BUFFER OG: 10100010 10000 11100110 1010000 0 0 0 0 0 0 1 0 10000000 0
   0 0 11111111 11111111 11111111 11111111
 - BUFFER: 10000000 0 0 0 11111111 11111111 11111111 11111111...

I've been able to identify parts of the first 12 elements:
1.  10100010 -> RCPI (RSSI) in two's complement format
2.  10000    -> Unknown
3.  11100110 -> Unknown
4.  1010000  -> Unknown
5.  0        -> Unknown, but constant
6.  0        -> Unknown, but constant
7.  0        -> Unknown, but constant
8.  0        -> Unknown, but constant
9.  0        -> Unknown, but constant
10. 0        -> Unknown, but constant
11. 1        -> Wi-Fi channel (in this example it's channel 1)
12. 0        -> Unknown, but constant

These 12 elements come before the 802.11 packet (starting with the frame header which begins at 10000000)
Through trial and error, I've been able to the identify parts of the code that effect the snifferPacket output. These are:

14 attributes in the RxControl struct, titled as THESE HAVE AN EFFECT.
a. Commenting between 1-8 of any attributes: The last 4 elements appear in the snifferPacket output (0 0 1 0)
b. Commenting between 9-13 of any attributes: The last 8 elements appear in the snifferPacket output (0 0 0 0 0 0 1 0)
c. Commenting all 14 attributes: The last 11 elements appear in the snifferPacket output (10000 11100110 1010000 0 0 0 0 0 0 1 0)

struct RxControl rx_ctrl;, under struct SnifferPacket
a. Commenting out: All 12 elements appear in the snifferPacket output (10100010 10000 11100110 1010000 0 0 0 0 0 0 1 0)

So it's clear that there is an association between defining attributes and their exclusion from the snifferPacket output.
Note that leading 0s are part of an element are ignored are part of the output. For example:
 - 0100 = 100
 - 0000 = 0
 - 1000 = 1000

So my questions are:

What do each of the 12 elements represent?
Why is there a difference in the output between the output of the raw buffer and snifferPacket?
Why do only certain attributes effect the output of snifferPacket?
Why does commenting a greater number of attributes result in a greater number of elements reappearing, no matter which attributes are commented?

Thanks in advance.

https://en.cppreference.com/w/cpp/language/bit_field

What do each of the 12 elements represent?

Things in the header. That is off-topic for here.

Why is there a difference in the output between the output of the raw buffer and snifferPacket?

snifferPacket is overlaid over the incoming buffer. That includes the "data" buffer which you are printing. The data buffer is only a portion of the snifferPacket.

Why do only certain attributes effect the output of snifferPacket?

They are bit fields, and they are packed into bytes or words. The number and size of bit fields affects the number of bytes needed to contain them, and thus where in the structure the data array comes.

Why does commenting a greater number of attributes result in a greater number of elements reappearing, no matter which attributes are commented?

See above. Also, the attributes must be there. Removing them will mean that other attributes don't point to the right bits in memory.
You should learn how C bit fields work. You can read more at the C++ reference here.