Magnetometer x, y, z overflows in CALIBRATED_SAMPLE and BLE characteristics

[martinwork]

As I understand it, the MAG3110 and LSM303 ranges are +/- 10 and nearly +/- 50 Gauss. They return 16bit signed integers with nominal units of 1 and 1.5 milliGauss. I think that's +/- 1000 and +/- 5000 microTesla with units 100 and 150 nanotesla.

The DAL normalises to nanotesla units in 32bit integers, with ranges +/- one and five million.

The magnetometer service assigns the 32bit numbers to its 16bit data characteristic buffers
https://github.com/lancaster-university/microbit-dal/blob/master/inc/bluetooth/MicroBitMagnetometerService.h#L96
https://github.com/lancaster-university/microbit-dal/blob/master/source/bluetooth/MicroBitMagnetometerService.cpp#L151

It looks like the data characteristics will overflow outside +/- 32 microtesla. Have I got my millis, micros, and nanos mixed up?!

[martinwork]

It looks like there is also a problem in the CALIBRATED_SAMPLE calculation.

When passing a fridge magnet near the micro:bit, a small increase in the raw reading causes the result of CALIBRATED_SAMPLE to flip sign. It seems, because the raw readings are millions, multiplying by calibration.scale (1024) causes an overflow in the 32bit arithmetic.

With both micro:bits I tested, calibration.scale.y was 1024. Is that expected? If calibration.scale were another value, would the values still be nanotesla?

Test details...

I modified MicroBitCompass::update() to dump values to serial. My main.cpp calls uBit.compass.heading() to trigger the compass calibration routine, then uBit.compass.getX() in a while loop. I was concentrating on the final X reading which comes from the raw Y reading.

The readings below were from a micro:bit v1.5. When I tested a micro:bit 1.3 it also produced raw readings outside +/-2 million, so high enough to trigger the overflow.

First reading:
[ r = raw reading; c = calibrated sample; t = transformed values]

calY = (( 2007150 - -12424) * 1024) >> 10
calY = ( 2019574 * 1024) >> 10
calY = 2068043776 >> 10
calY = 2019574
r = 733050, 2007150, 469500
c = 747086, 2019574, 492218
t = -2019574, 747086, 492218

Next reading:

calY = (( 2103450 - -12424) * 1024) >> 10
calY = ( 2115874 * 1024) >> 10
calY = -2128312320 >> 10
calY = -2078430
r = 745950, 2103450, 518400
c = 759986, -2078430, 541118
t = 2078430, 759986, 541118

Added trace

int MicroBitCompass::update()
{
  SERIAL_DEBUG->printf( "calY = (( %d - %d) * %d) >> 10\n", sampleENU.y, calibration.centre.y, calibration.scale.y);
  int calY = sampleENU.y - calibration.centre.y;
  SERIAL_DEBUG->printf( "calY = ( %d * %d) >> 10\n", calY, calibration.scale.y);
  calY = calY * calibration.scale.y;
  SERIAL_DEBUG->printf( "calY = %d >> 10\n", calY);
  calY = calY >> 10;
  SERIAL_DEBUG->printf( "calY = %d\n", calY);

    SERIAL_DEBUG->printf( "r = %d, %d, %d\n", sampleENU.x, sampleENU.y, sampleENU.z);
  
    // Store the raw data, and apply any calibration data we have.
    sampleENU.x = CALIBRATED_SAMPLE(sampleENU, x);
    sampleENU.y = CALIBRATED_SAMPLE(sampleENU, y);
    sampleENU.z = CALIBRATED_SAMPLE(sampleENU, z);

    SERIAL_DEBUG->printf( "c = %d, %d, %d\n", sampleENU.x, sampleENU.y, sampleENU.z);
  
    // Store the user accessible data, in the requested coordinate space, and taking into account component placement of the sensor.
    sample = coordinateSpace.transform(sampleENU);

    SERIAL_DEBUG->printf( "t = %d, %d, %d\n", sample.x, sample.y, sample.z);

    // Indicate that a new sample is available
    MicroBitEvent e(id, MICROBIT_COMPASS_EVT_DATA_UPDATE);

    return MICROBIT_OK;
};