discussing about the I2C hanging issues #72
Comments
|
I closed issues #68 and #69 in favor of this discussion. I spent quite a bit of time investigating these issues, so I thought I would give a small update here. That may be useful for other users who hit the same issues. Because of some issues with querying several data feeds with the current SF code (see #70 ), I use the Adafruit code. There is a bit of a hack to get it to work on the Artemis board, see the recipe https://github.com/jerabaul29/Artemis_MbedOS_recipes/blob/main/recipes/recipe_IMU_BNO_Adafruit/recipe_IMU_BNO_Adafruit.ino comment at the start to explain how to get it to work. I use this script to follow the advices of @PaulZC and restart the board if connection fails (of course the issue is that I then loose all the RAM information; a solution would be to dump user data to Flash through the "EEPROM" library): #include "Arduino.h"
#include "WDT.h"
#include <Adafruit_BNO08x.h>
Adafruit_BNO08x bno08x{};
sh2_SensorValue_t sensorValue;
unsigned long micros_last;
unsigned long micros_crrt;
void setReports(void) {
if (!bno08x.enableReport(SH2_ROTATION_VECTOR, 500000UL)) {
Serial.println("Could not enable rotation vector");
}
}
APM3_WDT wdt;
void resetArtemis(void)
{
Wire.end(); //Power down I2C
SPI.end(); //Power down SPI
power_adc_disable(); //Power down ADC. It it started by default before setup().
Serial.end(); //Power down UART
//Force the peripherals off
am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM0);
am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM1);
am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM2);
am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM3);
am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM4);
am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM5);
am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_ADC);
am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_UART0);
am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_UART1);
while (1) // That's all folks! Artemis will watchdog reset in 1.25 seconds
;
}
void setup(void) {
wdt.start();
Serial.begin(115200);
while (!Serial){
delay(10);
}
Serial.println();
Serial.println("------ booted ------");
Serial.println();
// initialize
for (int i=0; i<60;i++){
wdt.restart();
if (bno08x.begin_I2C(0x4B, &Wire, 0)){
Serial.println(F("found the Artemis!"));
break;
}
else{
Serial.print("Failed to find BNO08x chip "); Serial.println(i+1);
if (i == 19){
Serial.println(F("reset Artemis"));
resetArtemis();
}
Wire.end();
delay(50);
Wire.begin();
delay(50);
delay(500);
}
}
Wire.setClock(400000);
delay(50);
setReports();
micros_last = micros();
Serial.println("----- Reading events -----");
delay(100);
}
void loop() {
Serial.println(F("start reading"));
while(true){
if (bno08x.wasReset()) {
Serial.print("sensor was reset ");
setReports();
continue;
}
if (!bno08x.getSensorEvent(&sensorValue)) {
continue;
}
else{
wdt.restart();
switch (sensorValue.sensorId) {
case SH2_ROTATION_VECTOR:
micros_crrt = micros();
Serial.print(F("micros: ")); Serial.print(micros_crrt); Serial.println(F(" | micros elapsed: ")); Serial.println(micros_crrt-micros_last);
micros_last = micros_crrt;
Serial.print("Rotation Vector - r: ");
Serial.print(sensorValue.un.rotationVector.real);
Serial.print(" i: ");
Serial.print(sensorValue.un.rotationVector.i);
Serial.print(" j: ");
Serial.print(sensorValue.un.rotationVector.j);
Serial.print(" k: ");
Serial.println(sensorValue.un.rotationVector.k);
// TODO: may want to put a small delay here
break;
}
}
}
}I noticed a few interesting things. I think there may be 2 different bugs that cause hanging. Indeed, I get 2 "flavors" of hanging.
|
|
@PaulZC I was thinking more about that. Any way that we could restart the I2C port, i.e. power it up and down, from the watchdog, without the need to restart the whole board? That may be a workaround to reset the port without loosing the data in RAM, right? Any idea this could be doable / is there a HAL call for that? |
|
I am trying to reset the I2C hardware from within the interrupt with something like this: volatile bool force_wait {true}; // no delay() in interrupt; force reading and writing a volatile variable
volatile int remaining_watchdog_tries {10};
// Interrupt handler for the watchdog
extern "C" void am_watchdog_isr(void)
{
wdt.clear();
if (remaining_watchdog_tries > 0){
wdt.restart();
remaining_watchdog_tries -= 1;
Serial.print(F("retry I2C "));
Wire.end();
am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM0);
// hand-made delay, to make sure power has time to go down
for (int i=0; i<1000000; i++){
force_wait = !force_wait;
}
am_hal_pwrctrl_periph_enable(AM_HAL_PWRCTRL_PERIPH_IOM0);
Wire.begin();
}
else{
// reset
while (true){;}
}
} |
|
I even tried to disable and enable all the periphs (IOM0 to end), but that does not help. I do not know / have no way to know as far as I know (some method to get this information?) where the code hangs. I would say that now in around 90% of the cases I manage with the looping to get contact with the BNO, and in around 1 in 10, the code hangs when trying to begin the BNO, the watchdog kicks in, tries 10 times to restart the I2C, but with no success, leading to a full reset. If there was a way to trigger a reset of the right I2C stuff from the watchdog, rather than by doing a full chip reset, that would practically solve the problem. I do not have the JTAG or similar debugging tool nor the experience to debug down to this level / get this to work. If you have any advices @PaulZC that would be amazing. |
|
Hi JR, |
|
@PaulZC just FYI: it looks like I "empirically" found a recipe that works all the time at starting the BNO with Artemis, at least on the Artemis RedBoard and AGT I have here for testing, and with the couple of BNOs I have here too ("works on my computer" symptom). This recipe: With in particular this snippet: Has been working without any issues now for me several tens of times. Other similar snippets with different timing or checks fail regularly, but just this one seems to work. I would be interested to hear if other users are able to use this snippet to get a reliable startup. |
|
(you really do not want to know how many hours I spent having trial and errors to get to this one...). |
|
Hi JR, |
|
Yes, there is something really fishy. I have been "test trying" for around 1 hours to get this snippet to crash in the same way other similar snippets do, "luckily without" success. I think there is a broken piece of code somewhere and that somehow having just this combination of delays allows to avoid it. |
|
For the records, Teensy forum has a loooong discussion around that: https://forum.pjrc.com/threads/58268-Sparkfun-BNO080-Teensy-3-6-gt-gt-4-0 Read through it now, and not completely sure what the conclusion was ^^ . |
|
Hi folks, I just received my SparkFun BNO080 boards yesterday and spent a couple of hours trying to troubleshoot why they wouldn't initialize with Artemis-based boards before I found these issues on GitHub. I'm still catching up, but it sounds like the main takeaway is that the Artemis and BNO080 are, for the moment, incompatible over I2C? Thanks, @jerabaul29 and @PaulZC for your work in looking into this. Cheers, |
|
Hi @adamgarbo , Yes, had the same kind of troubles unfortunately :( . A few notes:
So my advice would be, if it is ok for your application, to use the RVC mode. As far as I found out, the only negative points for my use were that:
So it is in general quite ok to use, but it will make for a good story in my department, when I explain to my boss and colleagues that our instruments believe that they are robot cleaners ^^ . These problems are very sad, because this is an amazing sensor. I would be very curious @PaulZC to know if using the BNO085 could fix this issue, and if Sparkfun would be ok shifting to these in the future :) . PS: my current snippet for the I2C. Ugly, ad-hoc, hacky, but seems to do the trick around 90% of the time in my tests: bool BNO080_Manager::_power_start(void){
turn_bno_off();
delay(1000);
Wire1.begin();
delay(50);
Wire1.setClock(400000);
delay(50);
wdt.restart();
delay(2000);
wdt.restart();
turn_bno_on();
delay(5000);
wdt.restart();
bno080_imu.begin(BNO080_DEFAULT_ADDRESS, Wire1);
delay(50);
bno080_imu.enableDebugging(Serial);
delay(50);
wdt.restart();
bno080_imu.enableRotationVector(imu_output_period);
// bno080_imu.enableAccelerometer(imu_output_period);
wdt.restart();
delay(5000);
wdt.restart();
while (!bno080_imu.begin(BNO080_DEFAULT_ADDRESS, Wire1)){
Serial.println(F("try to begin"));
delay(100);
}
delay(5000);
bno080_imu.begin(BNO080_DEFAULT_ADDRESS, Wire1);
wdt.restart();
Serial.println(F("bno set up, start measuring"));
wdt.restart();
return true;
}
bool BNO080_Manager::_first_measurement_start(void){
//--------------------------------------------------
Serial.println(F("wait for the first BNO output"));
bool got_first_reading {false};
// TODO: protect, this can hang
for (int i=0; i<15;i++){
if (bno080_imu.getReadings() != 0){
got_first_reading = true;
break;
}
Serial.println(F("receiving nothing, try to set output"));
bno080_imu.enableRotationVector(imu_output_period);
delay(50);
bno080_imu.enableAccelerometer(imu_output_period);
delay(50);
}
wdt.restart();
if (!got_first_reading){
Serial.println(F("could not get a reading"));
return false;
}
// bno080_imu.enableAccelerometer(imu_output_period);
bno080_imu.enableRotationVector(imu_output_period);
wdt.restart();
Serial.println(F("got first measurement, check has both quat and accel"));
//--------------------------------------------------
// check that we can get both a quat and an accel reading
delay(100);
Serial.println(F("get accel and quat"));
while (true){
if (bno080_imu.getReadings() == SENSOR_REPORTID_ACCELEROMETER){
Serial.println("accel ok");
break;
}
delay(1);
// bno080_imu.enableAccelerometer(imu_output_period);
}
while (true){
if (bno080_imu.getReadings() == SENSOR_REPORTID_ROTATION_VECTOR){
Serial.println(F("quat ok"));
break;
}
delay(1);
bno080_imu.enableRotationVector(imu_output_period/2);
}
//--------------------------------------------------
Serial.println(F("ignore first BNO measurements"));
Serial.flush();
delay(5000);
wdt.restart();
for (int i=0; i<50; i++){
while (true){
if (_perform_one_reading() != 0){
break;
}
delay(1);
wdt.restart();
}
}
wdt.restart();
Serial.println(F("ready"));
Serial.flush();
return true;
}
bool BNO080_Manager::start(void){
bool start_succes {false};
for (int i=0; i<5; i++){
if (!_power_start()){
stop();
delay(1000);
continue;
};
wdt.restart();
if (_first_measurement_start()){
start_succes = true;
break;
}
else{
stop();
delay(1000);
}
}
if (start_succes){
Serial.println(F("success starting bno"));
}
else{
Serial.println(F("no success starting bno"));
stop();
Serial.println(F("we reboot"));
while(true){;}
}
return start_succes;
} |
|
Hi folks, I tested the majority of BNO080 examples using v2.0.5 of the Apollo3 Core and a SparkFun Redboard Artemis/Redboard Artemis Nano. I would have also included theMicroMod Artemis, but v2.0.5 doesn't have this board definition yet. I am happy to report that I was not able to reproduce any of the I2C hangs or sensor initialization failures with the examples I tested. Perhaps this bug is specific only to v1.2.x? @jerabaul29, with SparkFun announcing the EOL of v1.x of the Apollo3 Core yesterday, I believe it makes sense for us to focus our efforts on v2.x. User contributions will be a big help in getting it to where it needs to be. Edit: I may have spoken too soon. I left Example 17 running and it crashed after ~20 minutes multiple times. I was then unable to re-initialize the sensor until I reset/unplugged it. Yikes. I wonder if the Hillcrest Labs/CEVA would have any insight. Cheers, |
|
Ok, interesting. Yes, there are reports of issues with the BNO080 all over the internet with a variety of boards. For my part, very curious to know if the BNO085 fixes these issues. |
|
@PaulZC @nseidle I am planning to build a new batch of between 5 to 10 instruments based on Artemis Global Tracker and some variation of BNO IMU in the course of the next few months. Any hope you can check if upgrading to the BNO085 can offer a fix to this problem, or if there is another way to fix this issue? That would be extremely helpful. |
|
Hi JR, |
|
Thanks for the detailed explanations :) . If I understand right, the BNO085 is virtually exactly the same chip as the BNO080, except from a few fixes. It looks like for example Adafruit just swapped the BNO080 for a BNO085 in production, using the exact same breakout board and libraries. I will definitely buy one of their BNO085 to test a bit and see if it helps. The fact that these 2 chips seems to be very easily swappable, with the BNO085 being a drop-out backwards-compatible replacement, is what would make me very interested in seeing Sparkfun offering it. |
|
Another thing is that I am interested a bit in the power consumption, but most important for me is accuracy. It seems that, on this point, the BNO08x is the best, right? Or do you know @PaulZC of an accelerometer with even better performance? That starts to drift a bit from the original discussion, but if you have recommendation about what would be the "best possible accelerometer under at or under 50 to 100USD" in the sense of accuracy, I would be very interested; may be possible to have both an high accuracy accelerometer and an IMU to get acceleration and orientation on the same board for my part, i.e. would not be a worry to have 2 chips with a bit of overlap, if that would be the way to get best accuracy. |
|
Hi JR, |
|
Thanks for the link :) . I think this will become much broader discussion, would you @PaulZC @adamgarbo be ok to continue discussing here? https://forum.sparkfun.com/viewtopic.php?f=139&t=55182&p=223524#p223524 |
|
Hi JR (@jerabaul29 ), |
|
Hi Paul, That sounds amazing, many thanks :) . I am exhausted after the fieldwork of last month and taking a bit of holidays. @adamgarbo do you think you could have a look at this? :) . |
|
Morning, I just tested the BNO080 with a SparkFun Redboard Artemis on v2.0.5. It appears that if the sensor and/or system is power cycled, the initialization failures will still manifest themselves. I also tried testing the Qwiic Power Switch with the Redboard Artemis but I'm not sure if this combination is compatible. I can observe the softReset forcing the QPS to shut down (LED turns off)? I haven't looked at the changes to the library yet, but is this causing a hard reset to the I2C bus? Cheers, |
|
Hi Adam, |
|
Hi Paul, There is a 1-second delay after enabling the QPS and initializing the BNO080. QPS "IN" pullups enabled, "OUT" disabled. I just ran another series of tests with the SparkFun Redboard Artemis and QPS and had better results. The QPS was able to repeatably power cycle the BNO080. I'll run a longer test to see how it performs. I did encounter a strange Mbed OS error on more than one occasion (shown below), but I don't know if it is related. It occurs when powering up the Redboard Artemis with the QPS/BNO080 attached or when uploading new code. With the BNO080 connected directly to the Redboard Artemis, it does appear to be more stable, though pressing the "RESET" button will still produce the usual I2C initialization errors: Cheers, |
|
Hi Paul, I forgot to mention that the LED on the QPS turning off was due to a coding error on my part. I just started a power-cycle test with the QPS and have already been able to observe some strange behaviour with the BNO080. The test cycles between 10 seconds ON/OFF. Below, you can see that the sensor fails to reinitialize several times, but then magically starts to work again. This seems to be consistent behaviour the longer the test runs. Very strange! |
|
Well, it's progress. Kind of... ;-) |
|
Hi Paul, To confirm, are you using Apollo3 Core v2.0.5? I've found that when the BNO080 is connected directly to a Redboard Artemis on v2.0.5, there is a significant improvement to stability. The BNO080 seems to be able to recover from multiple hard resets to the board or unplugging/plugging back in the USB-C cable. However, when the QPS is added back, the BNO080 will crash after a number of power cycles and never recover. The QPS continues to operate (see below). Also, no improvements with SAMD-based boards, though I'm not sure if the changes would affect this platform. Cheers, |
|
I’m still using 1.2.1. Haven’t been brave enough to upgrade yet. But that time is coming soon! |
|
Interesting! I found that when I downgraded to v1.2.1 and conducted the same tests, the sensor was significantly less stable than on v2.0.5. What instrument are you using to monitor the I2C data? It might be something worth investing in. Cheers, |
|
Just picking things up: what is the current status on this? :) . @adamgarbo are you using the BNO in 'production' and does it work fine, or did you give up and go for another IMU? |
|
Hi @jerabaul29, Unfortunately, I haven't had time to troubleshoot the BNO080 further. With how unreliable it's been, I think it'll be too risky to include in the instrumentation I'll be deploying this year, especially considering it can crash the entire I2C bus. I'm currently exploring alternative IMU sensor options. Jim Remington has written a library for the LSM9DS1 that enables tilt-compensated compass measurements and sensor fusion (https://github.com/jremington/LSM9DS1-AHRS). I've been waiting for SparkFun to get the sensor back in stock (https://www.sparkfun.com/products/13944). Please let me know if you have any suggestions. |
|
Ok, thanks for the update! I just ordered / will be looking at replacing it with the ICM-20948 . |
|
Fair warning that the ICM-20948 is a black box and has its own issues. Paul has done lots of great work with the library, but I haven't been able to get reliable measurements from it yet. It also has a high quiescent draw of ~4 mA due to the switching circuitry. |
|
Ok, thanks for the warning. This is quite annoying - there are so many IMUs that 'should be a perfect fit' but looks like none of them is truly working flawlessly :( . |
|
Hi JR, |
|
That sounds great @PaulZC :) . So do you mean this new fix on dev branch is working 100% of the time? @adamgarbo you were naming some issues even with the newer fixes, does that apply to this dev branch code? |
|
To continue this discussion @PaulZC , from my 'small egoistic' perspective I am only interested in using the BNO with the Artemis platform, but I really need high reliability, so Portenta is not a big deal for me :) Would even say that if there were an 'artemis tuned' branch that would work perfectly with specifically the Artemis, that would be excellent as far as I am concerned :) |
|
The Portenta issue was - I think - a silly debug print mistake which I've just corrected: |
|
Sounds good, I want do some detailed testing of the updated branch, hope to have a bit of time for that next week :) . |
|
Hi @jerabaul29, I quickly tested the release-candidate branch with the Artemis on v1.2.1 and it appears to still have issues with initialization and power cycling (reset button). I purchased a logic analyzer from SparkFun but don't have the time to work on this at the moment. Please let me know what you find! Cheers, |
|
Ok, thanks, sad to hear that, will do some tests later this week (I hope, busy week). |
|
Hi @jerabaul29, I have a test running with the BNO080 and MicroMod Artemis Processor + Data Logging Carrier Board. I am power cycling the sensor 5 seconds ON and 5 seconds OFF, reinitializing each time. So far, it does appear to be more stable, but the important thing was to remove the When you enable debugging, you can observe there is a timeout, but the sensor does eventually initialize properly: When I upload new code the sensor does still fail to initialize, frequently, but it appears you can just ignore this. @PaulZC have you found the same? Cheers, |
|
Many thanks Adam, |
|
@adamgarbo : when doing some testing a couple of months ago I had had similar results that ignoring initialization was helping. However, I was then hitting another problem: I needed to enable BOTH the quaternion and the IMU output, and when some initialization was failing, I was sometimes not able to ask the chip for both of them, and I was getting either the quaternion of the IMU but not both. Not sure if this works now or not, but that may be a tips for testing: to be really exhaustive, may be a good idea during test to check that you can both:
|
|
@adamgarbo regarding your last test, was it well with the core v1.2.1? Or with the 2.x? We received a bit of extra funding for building some instruments, but I will have very little time to debug this in the months to come unfortunately. If @PaulZC you have the time to validate how things go with power cycling it would be very helpful. |
|
Hi @jerabaul29, I was using Apollo3 Core v1.2.1. The power cycle tests with the MicroMod Artemis Processor and Data Logging Carrier Board were promising but initialization failures and other errors did still regularly occur. However, in most cases, I don't think this prevents the sensor from outputting data. My main concerns are that the sensor has been shown to crash the entire I2C bus and I don't think it's compatible yet with SAMD-based processors. |
|
Ok, thanks. I do not think we need to care too much about SAMD processors, we only plan to use Artemis as far as I know. I know, the I2C crashing is quite annoying. But I think that worst case we will just accept to have some watchog full-reboot. I think that to survive reboot without issue I will move "important data that should persist and is not used too often" to the flash memory, that will solve all problems. |
|
I'll continue my testing the BNO080 with SAMD-based processors, as I rely on them for many projects the Artemis is not suitable for (e.g. the Artemis can not make 0-3.3 V analog measurements). In my opinion, relying on the watchdog timer just to ensure normal operation is a big red flag. The watchdog should really only be used as a last resort to prevent the system from freezing, but of course, it's all a matter of what is acceptable to the end-user. I will also order an Adafruit BNO085 for testing and continue to explore other IMU options. Cheers, |
|
Ok, will be very interesting to hear your experience :) . |
|
I get the same behaviour reported in this issue with a BNO086 and a micromod artemis running FW 2.1 in platformio: no IMU detected e.g. after uploading some code or after using the IMU for some (short) time. Before reading this thread I was "restoring" the BNO086 by power-cycling the micromod ATP board (unplug-replug the USB), but the work by @jerabaul29 and suggestions from @PaulZC point toward alternative options that I may try in the near future. |
|
(a note @diplodocuslongus : I have spent a lot of time a few years ago trying to get the BNO080 working with the artemis global tracker, and never got it to work reliably; I ended up moving to the ISM330DHCX - a really nice little chip, working very well, available from SF and Adafruit, using much less power, only drawback is you need to run the Kalman filtering yourself, but there are packets for this: see the AHRS library https://github.com/adafruit/Adafruit_AHRS for example; you can see for example how my specific project developed (not that it is "perfect" in any way of course") by looking into: https://github.com/jerabaul29/OpenMetBuoy-v2021a ). |
|
@diplodocuslongus I concur with @jerabaul29. If you wish to continue to use the SparkFun Artemis, save yourself the time and effort and switch to a different IMU. Those are hours, days and weeks of your life you'll never get back! 😉 |
|
The previous discussions stopped in 2021 but I see that you guys are still around the corner! Thanks @jerabaul29 for the IMU recommendation, I'll definitely look into it as well as your OpenMetBuoy project (looks very interesting!). Implementing a Kalman Filter on the host MCU is ok. I've tested a few different IMUs over the past 18 months and have been experimenting with the latest BNO086 from CEVA for a month or so, the specs looked impressive. It's still a nice IMU but, well... paired with the artemis results in the same troubles you've experienced. |
This is moslty just for discussing / brainstorming.
There is this really annoying issue with the BNO080 hanging when put on I2C to the Artemis.
Any feedback you received about connecting the BNO080 through UART or SPI instead? Does that work without hanging? Of course I would prefer I2C, but if needed, I can use something else.
It is very late, will try tomorrow, but would be great to know if there are some known issues there too / if investigating this would waste my time. In this case, I would stick to I2C and the hard reset solution ^^ .
The text was updated successfully, but these errors were encountered: