Panic reset when scanning and several concurrent http requests occur

Working with others on a project which leverages NimBLE as a client only receiving advertisements (ADV_NONCONN_IND) from up to 8 devices.  We have recently converted the project over to use ESPAsyncWebserver and have started experiencing sporadic PANIC resets.  A reset can be fairly reliably triggered by hard reloading one of the pages with several concurrent jQuery requests repeatedly.  We suspect this relates to competition for radio time between WiFi and BLE but are unsure about this aspect. Have done some troubleshooting such as trying to adjust the Scan Interval and Scan Window but have not been able to reliably eliminate the issue and am looking for ideas.  The backtrace is always the same and appears to indicate that the PANIC is being triggered in NimBLE related code.
Any guidance on whether this likely points to a problem with our implementation or whether we have uncovered a library issue is much appreciated!
Logging output:
ble_hs_hci_cmd_send: ogf=0x08 ocf=0x000c len=2
0x0c 0x20 0x02 0x00 0x00
Command complete: cmd_pkts=5 ogf=0x8 ocf=0xc status=0
GAP procedure initiated: discovery; own_addr_type=0 filter_policy=0 passive=1 limited=0 filter_duplicates=0 duration=3000ms
ble_hs_hci_cmd_send: ogf=0x08 ocf=0x000b len=7
0x0b 0x20 0x07 0x00 0x54 0x00 0x3b 0x00 0x00 0x00
Command complete: cmd_pkts=5 ogf=0x8 ocf=0xb status=0
ble_hs_hci_cmd_send: ogf=0x08 ocf=0x000c len=2
0x0c 0x20 0x02 0x01 0x00
Backtrace: 0x40091fa4:0x3ffbdc20 0x400921d5:0x3ffbdc40 0x40085a9d:0x3ffbdc60 0x40085bc9:0x3ffbdc90 0x40180377:0x3ffbdcb0 0x4017ad15:0x3ffbdf70 0x40177de9:0x3ffbdfc0 0x400ee0e2:0x3ffbdff0 0x401c75ea:0x3ffbe010 0x401bba2a:0x3ffbe030 0x4001791d:0x3ffbe050 0x401bc029:0x3ffbe070 0x401bc106:0x3ffbe0b0 0x400188f5:0x3ffbe0e0 0x400175ee:0x3ffbe120 0x40017688:0x3ffbe140 0x4008cb36:0x3ffbe170 0x4008cd0b:0x3ffbe190 0x401bb9d7:0x3ffbe1b0 0x4008cd28:0x3ffbe1d0 0x40085025:0x3ffbe1f0 0x4008583e:0x3ffb9d00 0x4008cadd:0x3ffb9d20 0x401c7789:0x3ffb9d40 0x401aa088:0x3ffb9d60 0x400edd8d:0x3ffb9d80 0x40082b53:0x3ffb9da0 0x4008e33d:0x3ffb9dc0
#0  0x40091fa4:0x3ffbdc20 in invoke_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:707
#1  0x400921d5:0x3ffbdc40 in abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:707
#2  0x40085a9d:0x3ffbdc60 in lock_acquire_generic at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/locks.c:143
#3  0x40085bc9:0x3ffbdc90 in _lock_acquire_recursive at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/locks.c:171
#4  0x40180377:0x3ffbdcb0 in _vfiprintf_r at /Users/ivan/e/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdio/../../../.././newlib/libc/stdio/vfprintf.c:860 (discriminator 2)
#5  0x4017ad15:0x3ffbdf70 in fiprintf at /Users/ivan/e/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdio/../../../.././newlib/libc/stdio/fiprintf.c:50
#6  0x40177de9:0x3ffbdfc0 in __assert_func at /Users/ivan/e/newlib_xtensa-2.2.0-bin/newlib_xtensa-2.2.0/xtensa-esp32-elf/newlib/libc/stdlib/../../../.././newlib/libc/stdlib/assert.c:59 (discriminator 8)
#7  0x400ee0e2:0x3ffbdff0 in host_rcv_pkt at .pio\libdeps\lcd_ssd1306\NimBLE-Arduino@src-e248a3b4ca528fb2d5b4751077939ebd\src\esp-hci\src/esp_nimble_hci.c:365
#8  0x401c75ea:0x3ffbe010 in vhci_send at ??:?
#9  0x401bba2a:0x3ffbe030 in r_eif_send at ??:?
#10 0x4001791d:0x3ffbe050 in ?? ??:0
#11 0x401bc029:0x3ffbe070 in hci_tx_start at hci_tl.c:?
#12 0x401bc106:0x3ffbe0b0 in r_hci_tl_send at ??:?
#13 0x400188f5:0x3ffbe0e0 in ?? ??:0
#14 0x400175ee:0x3ffbe120 in ?? ??:0
#15 0x40017688:0x3ffbe140 in ?? ??:0
#16 0x4008cb36:0x3ffbe170 in vhci_recv_end at vhci.c:?
#17 0x4008cd0b:0x3ffbe190 in r_vhci_isr at ??:?
#18 0x401bb9d7:0x3ffbe1b0 in r_eif_isr at ??:?
#19 0x4008cd28:0x3ffbe1d0 in r_eif_isr_wrapper at intc.c:?
#20 0x40085025:0x3ffbe1f0 in _xt_lowint1 at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/xtensa_vectors.S:1154
#21 0x4008583e:0x3ffb9d00 in cause_sw_intr_to_core_wrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/bt/bt.c:1322
#22 0x4008cadd:0x3ffb9d20 in vhci_set_interrupt at vhci.c:?
#23 0x401c7789:0x3ffb9d40 in API_vhci_host_send_packet at ??:?
#24 0x401aa088:0x3ffb9d60 in esp_vhci_host_send_packet at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/bt/bt.c:1322
#25 0x400edd8d:0x3ffb9d80 in ble_hci_trans_hs_cmd_tx_on_core_0 at .pio\libdeps\lcd_ssd1306\NimBLE-Arduino@src-e248a3b4ca528fb2d5b4751077939ebd\src\esp-hci\src/esp_nimble_hci.c:171
#26 0x40082b53:0x3ffb9da0 in ipc_task at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/ipc.c:62
#27 0x4008e33d:0x3ffb9dc0 in vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c:355 (discriminator 1)

I can only add that I fought the exact same issue for a couple weeks with espasyncwebserver and the old Bluetooth library. I ended up dropping back to the regular webserver and using client side JavaScript to smooth things over.

I can only add that I fought the exact same issue for a couple weeks with espasyncwebserver and the old Bluetooth library. I ended up dropping back to the regular webserver and using client side JavaScript to smooth things over.

Well I cannot say for sure what is happening here but I found this interesting, in 7 this assert is triggered. This is a message from the controller that it has encountered a hardware error.
Not much to work with here, except perhaps a power issue. If not that, then perhaps a bug in the controller, guessing by the previous area of the backtrace (who knows here, it's black magic).
One other thing it could be that could be tested however, is the commands being sent from core 0. This is a modification made to the esp-nimble files to address delay issues when sending commands from core 1. If you revert this commit and test I'd be very curious of the results.

Well I cannot say for sure what is happening here but I found this interesting, in 7 this assert is triggered. This is a message from the controller that it has encountered a hardware error.
Not much to work with here, except perhaps a power issue. If not that, then perhaps a bug in the controller, guessing by the previous area of the backtrace (who knows here, it's black magic).
One other thing it could be that could be tested however, is the commands being sent from core 0. This is a modification made to the esp-nimble files to address delay issues when sending commands from core 1. If you revert this commit and test I'd be very curious of the results.

Hi. I have run into the same issue running an http server using ESPAsyncWebserver and scanning BLE - my understanding is that this is hardware problem in that the WiFi and BLE share the same hardware radio. Meaning that you have a harder software problem to balance scanning BLE for devices and servicing Wifi packets as they share the data pipeline. I've not found a solution yet and may have to make a separate controller for BLE scanning as the ESP32 crashes often or the http server takes as long to respond as the shortest scanning cycle. It seems while BLE scan is in progress you can't service http requests.
I'll share my findings if I manage to come up with a workable approach.

Hi. I have run into the same issue running an http server using ESPAsyncWebserver and scanning BLE - my understanding is that this is hardware problem in that the WiFi and BLE share the same hardware radio. Meaning that you have a harder software problem to balance scanning BLE for devices and servicing Wifi packets as they share the data pipeline. I've not found a solution yet and may have to make a separate controller for BLE scanning as the ESP32 crashes often or the http server takes as long to respond as the shortest scanning cycle. It seems while BLE scan is in progress you can't service http requests.
I'll share my findings if I manage to come up with a workable approach.

Backtrace: 0x40091fa4:0x3ffbdc20 0x400921d5:0x3ffbdc40 0x40085a9d:0x3ffbdc60 0x40085bc9:0x3ffbdc90 0x40180377:0x3ffbdcb0 0x4017ad15:0x3ffbdf70 0x40177de9:0x3ffbdfc0 0x400ee0e2:0x3ffbdff0 0x401c75ea:0x3ffbe010 0x401bba2a:0x3ffbe030 0x4001791d:0x3ffbe050 0x401bc029:0x3ffbe070 0x401bc106:0x3ffbe0b0 0x400188f5:0x3ffbe0e0 0x400175ee:0x3ffbe120 0x40017688:0x3ffbe140 0x4008cb36:0x3ffbe170 0x4008cd0b:0x3ffbe190 0x401bb9d7:0x3ffbe1b0 0x4008cd28:0x3ffbe1d0 0x40085025:0x3ffbe1f0 0x4008583e:0x3ffb9d00 0x4008cadd:0x3ffb9d20 0x401c7789:0x3ffb9d40 0x401aa088:0x3ffb9d60 0x400edd8d:0x3ffb9d80 0x40082b53:0x3ffb9da0 0x4008e33d:0x3ffb9dc0

I've put that through an Exception Decoder. It looks like its related to the WiFi data, DNS and so on - I get the same sort of issues. If I find a workable approach I'll share it.

Backtrace: 0x40091fa4:0x3ffbdc20 0x400921d5:0x3ffbdc40 0x40085a9d:0x3ffbdc60 0x40085bc9:0x3ffbdc90 0x40180377:0x3ffbdcb0 0x4017ad15:0x3ffbdf70 0x40177de9:0x3ffbdfc0 0x400ee0e2:0x3ffbdff0 0x401c75ea:0x3ffbe010 0x401bba2a:0x3ffbe030 0x4001791d:0x3ffbe050 0x401bc029:0x3ffbe070 0x401bc106:0x3ffbe0b0 0x400188f5:0x3ffbe0e0 0x400175ee:0x3ffbe120 0x40017688:0x3ffbe140 0x4008cb36:0x3ffbe170 0x4008cd0b:0x3ffbe190 0x401bb9d7:0x3ffbe1b0 0x4008cd28:0x3ffbe1d0 0x40085025:0x3ffbe1f0 0x4008583e:0x3ffb9d00 0x4008cadd:0x3ffb9d20 0x401c7789:0x3ffb9d40 0x401aa088:0x3ffb9d60 0x400edd8d:0x3ffb9d80 0x40082b53:0x3ffb9da0 0x4008e33d:0x3ffb9dc0

I've put that through an Exception Decoder. It looks like its related to the WiFi data, DNS and so on - I get the same sort of issues. If I find a workable approach I'll share it.

Nimble works perfectly fine with me with all of these libraries, even during scan. It's just the espasync .  You can however spin off the regular webserver below on its own task and it works reasonably well. Good luck!
#include <WebServer.h>
#include <HTTPClient.h>
#include <HTTPUpdate.h>
#include <SPIFFS.h>
#include <ESPmDNS.h>
#include <WiFiClientSecure.h>
#include <Update.h>
#include <DNSServer.h>

Nimble works perfectly fine with me with all of these libraries, even during scan. It's just the espasync .  You can however spin off the regular webserver below on its own task and it works reasonably well. Good luck!
#include <WebServer.h>
#include <HTTPClient.h>
#include <HTTPUpdate.h>
#include <SPIFFS.h>
#include <ESPmDNS.h>
#include <WiFiClientSecure.h>
#include <Update.h>
#include <DNSServer.h>

Someone did mention power supply maybe needing a capacitor to deal with extra power needed during scan. Has anyone tried the setActiveScan with false or observed anything related to this setting and stability with Wifi and BLE. I'll try this myself anyway and see if this helps.

Someone did mention power supply maybe needing a capacitor to deal with extra power needed during scan. Has anyone tried the setActiveScan with false or observed anything related to this setting and stability with Wifi and BLE. I'll try this myself anyway and see if this helps.

I'm one of the unnamed co-conspirators.

Not much to work with here, except perhaps a power issue.

I can help rule-out the power unless it's a design flaw in the dev board itself (Lolin D32 Pro.)  There's three of us working on this with similar results, it seems unlikely all of us have flakey power.  I'd also expect to see a reset caused by brownout, not a panic.  Unless you see something I do not?

Has anyone tried the setActiveScan with false or observed anything related to this setting and stability with Wifi and BLE.

setActiveScan is set to false.  We are just using the iBeacon in this project.
@justoke which exception decoder are you using?  I've had horrible results trying to consistently use those, but that would be a nice thing to figure out.  I am using the esp32_exception_decoder built in to PIO and I get this which does not show the same information:
Backtrace: 0x40091978:0x3ffbe190 0x40091ba9:0x3ffbe1b0 0x40121f58:0x3ffbe1d0 0x400847ed:0x3ffbe1f0 0x401c4ecf:0x3ffb5cf0 0x401c65c5:0x3ffb5d60 0x401c3adb:0x3ffb5db0 0x401c106e:0x3ffb5e00 0x40019d11:0x3ffb5e40 0x40055b4d:0x3ffb5e60 0x401bb1e7:0x3ffb5e80 0x401bb731:0x3ffb5ea0 0x4008dd45:0x3ffb5ed0
  #0  0x40091978:0x3ffbe190 in invoke_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:707
  #1  0x40091ba9:0x3ffbe1b0 in abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:707
  #2  0x40121f58:0x3ffbe1d0 in task_wdt_isr at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/task_wdt.c:252
  #3  0x400847ed:0x3ffbe1f0 in _xt_lowint1 at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/xtensa_vectors.S:1154
  #4  0x401c4ecf:0x3ffb5cf0 in r_llm_le_adv_report_ind at ??:?
  #5  0x401c65c5:0x3ffb5d60 in r_llm_pdu_defer at ??:?
  #6  0x401c3adb:0x3ffb5db0 in r_lld_pdu_check at ??:?
  #7  0x401c106e:0x3ffb5e00 in r_lld_evt_deffered_elt_handler at ??:?
  #8  0x40019d11:0x3ffb5e40 in ?? ??:0
  #9  0x40055b4d:0x3ffb5e60 in ?? ??:0
  #10 0x401bb1e7:0x3ffb5e80 in r_rw_schedule at ??:?
  #11 0x401bb731:0x3ffb5ea0 in btdm_controller_task at ??:?
  #12 0x4008dd45:0x3ffb5ed0 in vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c:355 (discriminator 1)

I ended up dropping back to the regular webserver

Rolling back to use the synchronous web server is an undesirable outcome.  The performance difference between the two is quite pronounced for this application.  Unless I inline all of the code (undesirable from a re-use and maintenance standpoint), a single connection would be quite bad.

One other thing it could be that could be tested however, is the commands being sent from core 0. This is a modification made to the esp-nimble files to address delay issues when sending commands from core 1. If you revert this commit and test I'd be very curious of the results.

I noticed the core - the web server (async specifically) is pinned to core 1 per this note.  Removing that makes things quite a bit worse,  I did attempt to pin NimBLE to core 1 as well with no improvement.
Reverting that commit has proved challenging for me, and I'm sure it's something I am forgetting but I don't see it:
git log --oneline
bc52eac (grafted, HEAD -> master, origin/master, origin/HEAD) Fix attributes not found with 16/32bit UUIDs.

I appreciate the folks here willing to help out.  If it's something we need to bring upstream we can obviously do that.  It's just quite a bit more challenging to work there.

I'm one of the unnamed co-conspirators.

Not much to work with here, except perhaps a power issue.

I can help rule-out the power unless it's a design flaw in the dev board itself (Lolin D32 Pro.)  There's three of us working on this with similar results, it seems unlikely all of us have flakey power.  I'd also expect to see a reset caused by brownout, not a panic.  Unless you see something I do not?

Has anyone tried the setActiveScan with false or observed anything related to this setting and stability with Wifi and BLE.

setActiveScan is set to false.  We are just using the iBeacon in this project.
@justoke which exception decoder are you using?  I've had horrible results trying to consistently use those, but that would be a nice thing to figure out.  I am using the esp32_exception_decoder built in to PIO and I get this which does not show the same information:
Backtrace: 0x40091978:0x3ffbe190 0x40091ba9:0x3ffbe1b0 0x40121f58:0x3ffbe1d0 0x400847ed:0x3ffbe1f0 0x401c4ecf:0x3ffb5cf0 0x401c65c5:0x3ffb5d60 0x401c3adb:0x3ffb5db0 0x401c106e:0x3ffb5e00 0x40019d11:0x3ffb5e40 0x40055b4d:0x3ffb5e60 0x401bb1e7:0x3ffb5e80 0x401bb731:0x3ffb5ea0 0x4008dd45:0x3ffb5ed0
  #0  0x40091978:0x3ffbe190 in invoke_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:707
  #1  0x40091ba9:0x3ffbe1b0 in abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:707
  #2  0x40121f58:0x3ffbe1d0 in task_wdt_isr at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/task_wdt.c:252
  #3  0x400847ed:0x3ffbe1f0 in _xt_lowint1 at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/xtensa_vectors.S:1154
  #4  0x401c4ecf:0x3ffb5cf0 in r_llm_le_adv_report_ind at ??:?
  #5  0x401c65c5:0x3ffb5d60 in r_llm_pdu_defer at ??:?
  #6  0x401c3adb:0x3ffb5db0 in r_lld_pdu_check at ??:?
  #7  0x401c106e:0x3ffb5e00 in r_lld_evt_deffered_elt_handler at ??:?
  #8  0x40019d11:0x3ffb5e40 in ?? ??:0
  #9  0x40055b4d:0x3ffb5e60 in ?? ??:0
  #10 0x401bb1e7:0x3ffb5e80 in r_rw_schedule at ??:?
  #11 0x401bb731:0x3ffb5ea0 in btdm_controller_task at ??:?
  #12 0x4008dd45:0x3ffb5ed0 in vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c:355 (discriminator 1)

I ended up dropping back to the regular webserver

Rolling back to use the synchronous web server is an undesirable outcome.  The performance difference between the two is quite pronounced for this application.  Unless I inline all of the code (undesirable from a re-use and maintenance standpoint), a single connection would be quite bad.

One other thing it could be that could be tested however, is the commands being sent from core 0. This is a modification made to the esp-nimble files to address delay issues when sending commands from core 1. If you revert this commit and test I'd be very curious of the results.

I noticed the core - the web server (async specifically) is pinned to core 1 per this note.  Removing that makes things quite a bit worse,  I did attempt to pin NimBLE to core 1 as well with no improvement.
Reverting that commit has proved challenging for me, and I'm sure it's something I am forgetting but I don't see it:
git log --oneline
bc52eac (grafted, HEAD -> master, origin/master, origin/HEAD) Fix attributes not found with 16/32bit UUIDs.

I appreciate the folks here willing to help out.  If it's something we need to bring upstream we can obviously do that.  It's just quite a bit more challenging to work there.

I think this might resolve your issue on ESP32. Add this define you your code - in main.cpp if you are using Platform IO
#define CONFIG_ASYNC_TCP_RUNNING_CORE 1
This will pin the AsyncWebserver which runs as a FreeRTOS task on CORE 1. I tried this and although the web server can still get blocked by the BLE(apparent because the web server response times are longer when the scan is in progress), it has not crashed since I tried this and often it responds in sub 100ms times which is as good as I could hope for. I implemented some other things specific to my use case - like blocking my loop() task logic while updating data from the BLE onResult() event.

I think this might resolve your issue on ESP32. Add this define you your code - in main.cpp if you are using Platform IO
#define CONFIG_ASYNC_TCP_RUNNING_CORE 1
This will pin the AsyncWebserver which runs as a FreeRTOS task on CORE 1. I tried this and although the web server can still get blocked by the BLE(apparent because the web server response times are longer when the scan is in progress), it has not crashed since I tried this and often it responds in sub 100ms times which is as good as I could hope for. I implemented some other things specific to my use case - like blocking my loop() task logic while updating data from the BLE onResult() event.

I think this might resolve your issue on ESP32.

Yes, sir - already doing that.  My comment about it was not horribly clear in retrospect.  I've edited my post above.

I think this might resolve your issue on ESP32.

Yes, sir - already doing that.  My comment about it was not horribly clear in retrospect.  I've edited my post above.

@lbussy I used this decode the exceptions https://github.com/me-no-dev/EspExceptionDecoder. Although I mainly use PIO, this is the only thing I use Arduino IDE  for.

@lbussy I used this decode the exceptions https://github.com/me-no-dev/EspExceptionDecoder. Although I mainly use PIO, this is the only thing I use Arduino IDE  for.

I think this might resolve your issue on ESP32.

Yes, sir - already doing that. My comment about it was not horribly clear in retrospect. I've edited my post above.

I had replied before I saw your comment. I'm happy to try help you. I was definitely in a world of pain before I made the core assignment. Of course my use case will differ from yours, but maybe there is an approach to what I have done that might help you.

I think this might resolve your issue on ESP32.

Yes, sir - already doing that. My comment about it was not horribly clear in retrospect. I've edited my post above.

I had replied before I saw your comment. I'm happy to try help you. I was definitely in a world of pain before I made the core assignment. Of course my use case will differ from yours, but maybe there is an approach to what I have done that might help you.

Thanks - I really hate going back to the Arduino IDE after PIO.
So here's something interesting.  I have a fork of ESPAsynchWebServer I was using elsewhere and I pulled that in instead.  The behavior is much improved (but not perfect).  Here's the PR I merged in my fork:
https://github.com/me-no-dev/ESPAsyncWebServer/pull/836/files
As you can see the difference is basically cleaning up connections.  It's not making perfect sense yet as I've not had my coffee - but you are 5 hours ahead so maybe it strikes a chord with you.

Thanks - I really hate going back to the Arduino IDE after PIO.
So here's something interesting.  I have a fork of ESPAsynchWebServer I was using elsewhere and I pulled that in instead.  The behavior is much improved (but not perfect).  Here's the PR I merged in my fork:
https://github.com/me-no-dev/ESPAsyncWebServer/pull/836/files
As you can see the difference is basically cleaning up connections.  It's not making perfect sense yet as I've not had my coffee - but you are 5 hours ahead so maybe it strikes a chord with you.

I was definitely in a world of pain before I made the core assignment. Of course my use case will differ from yours, but maybe there is an approach to what I have done that might help you.

I agree - and we're using that:
    -D CONFIG_ASYNC_TCP_RUNNING_CORE=1  ; per: https://github.com/me-no-dev/ESPAsyncWebServer/issues/731#issuecomment-628163515
    -D CONFIG_ASYNC_TCP_USE_WDT=1

So aside from the PR up there having a net positive impact, if I remove the CONFIG_ASYNC_TCP_USE_WDT=1 define I get no crashes, but eventually the webserver stops responding.
We're poking in the right place for sure.

I was definitely in a world of pain before I made the core assignment. Of course my use case will differ from yours, but maybe there is an approach to what I have done that might help you.

I agree - and we're using that:
    -D CONFIG_ASYNC_TCP_RUNNING_CORE=1  ; per: https://github.com/me-no-dev/ESPAsyncWebServer/issues/731#issuecomment-628163515
    -D CONFIG_ASYNC_TCP_USE_WDT=1

So aside from the PR up there having a net positive impact, if I remove the CONFIG_ASYNC_TCP_USE_WDT=1 define I get no crashes, but eventually the webserver stops responding.
We're poking in the right place for sure.

I can help rule-out the power unless it's a design flaw in the dev board itself (Lolin D32 Pro.) There's three of us working on this with similar results, it seems unlikely all of us have flakey power. I'd also expect to see a reset caused by brownout, not a panic. Unless you see something I do not?

If the regulator is weak, which it may be on this board, the voltage might dip low enough to cause an error but not low enough to trigger the brownout detection (2.43v default in arduino). So if your regulator dropped to, say, 2.9v, that might be enough to trigger this. Just speculating here though, you would have to measure with a scope or meter to detect min levels.
Hopefully your PR there will lead to a software resolution though.

I can help rule-out the power unless it's a design flaw in the dev board itself (Lolin D32 Pro.) There's three of us working on this with similar results, it seems unlikely all of us have flakey power. I'd also expect to see a reset caused by brownout, not a panic. Unless you see something I do not?

If the regulator is weak, which it may be on this board, the voltage might dip low enough to cause an error but not low enough to trigger the brownout detection (2.43v default in arduino). So if your regulator dropped to, say, 2.9v, that might be enough to trigger this. Just speculating here though, you would have to measure with a scope or meter to detect min levels.
Hopefully your PR there will lead to a software resolution though.

Thanks - I really hate going back to the Arduino IDE after PIO.
So here's something interesting.  I have a fork of ESPAsynchWebServer I was using elsewhere and I pulled that in instead.  The behavior is much improved (but not perfect).  Here's the PR I merged in my fork:
https://github.com/me-no-dev/ESPAsyncWebServer/pull/836/files
As you can see the difference is basically cleaning up connections.  It's not making perfect sense yet as I've not had my coffee - but you are 5 hours ahead so maybe it strikes a chord with you.

FYI, you can still work in PIO and just use the trace debugger
In Arduino IDE separately. When you select it from the menu, it asks for your firmware.elf file (grab it from your .pio folder) and paste the debug message into the window that pops up.

Thanks - I really hate going back to the Arduino IDE after PIO.
So here's something interesting.  I have a fork of ESPAsynchWebServer I was using elsewhere and I pulled that in instead.  The behavior is much improved (but not perfect).  Here's the PR I merged in my fork:
https://github.com/me-no-dev/ESPAsyncWebServer/pull/836/files
As you can see the difference is basically cleaning up connections.  It's not making perfect sense yet as I've not had my coffee - but you are 5 hours ahead so maybe it strikes a chord with you.

FYI, you can still work in PIO and just use the trace debugger
In Arduino IDE separately. When you select it from the menu, it asks for your firmware.elf file (grab it from your .pio folder) and paste the debug message into the window that pops up.

I was definitely in a world of pain before I made the core assignment. Of course my use case will differ from yours, but maybe there is an approach to what I have done that might help you.

I agree - and we're using that:
    -D CONFIG_ASYNC_TCP_RUNNING_CORE=1  ; per: https://github.com/me-no-dev/ESPAsyncWebServer/issues/731#issuecomment-628163515
    -D CONFIG_ASYNC_TCP_USE_WDT=1

So aside from the PR up there having a net positive impact, if I remove the CONFIG_ASYNC_TCP_USE_WDT=1 define I get no crashes, but eventually the webserver stops responding.
We're poking in the right place for sure.

So I've had 5 hrs of up time on my ESP32 and not a single panic crash. The basic app use case is as follows:
Scan BLE every 2 seconds and read sensor values from detected beacons/sensors
Manage thermostat task when sensor readings change
When sensor  values changes a flag is set and this prevents further BLE scanning until the thermostat task has responded and then the state machine repeats the cycle.
The loop() function simply services the web server - again while scan results in sensor changes, the web server is not serviced.
To test the web api response, I fired up Postman and fired requests at 500 millisecond intervals for a few minutes. What is apparent is during the BLE scan response time goes up to 4s, otherwise typical response is sub 200 millisecond with 80 to 90 ms being fairly average.

I was definitely in a world of pain before I made the core assignment. Of course my use case will differ from yours, but maybe there is an approach to what I have done that might help you.

I agree - and we're using that:
    -D CONFIG_ASYNC_TCP_RUNNING_CORE=1  ; per: https://github.com/me-no-dev/ESPAsyncWebServer/issues/731#issuecomment-628163515
    -D CONFIG_ASYNC_TCP_USE_WDT=1

So aside from the PR up there having a net positive impact, if I remove the CONFIG_ASYNC_TCP_USE_WDT=1 define I get no crashes, but eventually the webserver stops responding.
We're poking in the right place for sure.

So I've had 5 hrs of up time on my ESP32 and not a single panic crash. The basic app use case is as follows:
Scan BLE every 2 seconds and read sensor values from detected beacons/sensors
Manage thermostat task when sensor readings change
When sensor  values changes a flag is set and this prevents further BLE scanning until the thermostat task has responded and then the state machine repeats the cycle.
The loop() function simply services the web server - again while scan results in sensor changes, the web server is not serviced.
To test the web api response, I fired up Postman and fired requests at 500 millisecond intervals for a few minutes. What is apparent is during the BLE scan response time goes up to 4s, otherwise typical response is sub 200 millisecond with 80 to 90 ms being fairly average.

Stopping the BLE scans while servicing the jquery (which are JSON requests) is a possibility.  That might help.  Thank you for your ideas - there might be something in there we can use!
I wonder if the AsyncWebServer can be paused during BLE scans - sort of the opposite.

Stopping the BLE scans while servicing the jquery (which are JSON requests) is a possibility.  That might help.  Thank you for your ideas - there might be something in there we can use!
I wonder if the AsyncWebServer can be paused during BLE scans - sort of the opposite.

Hopefully your PR there will lead to a software resolution though.

He has 40 open PRs going back to 2017.  I'm not going to hold my breath.

Hopefully your PR there will lead to a software resolution though.

He has 40 open PRs going back to 2017.  I'm not going to hold my breath.

Stopping the BLE scans while servicing the jquery (which are JSON requests) is a possibility. That might help. Thank you for your ideas - there might be something in there we can use!
I wonder if the AsyncWebServer can be paused during BLE scans - sort of the opposite.

I do this - so when BLE onResult() event finds a matching device that I am after, it reads the sensor data and sets HasChanges to true. In that state I don't process web requests until the changes have been processed by the relevant observers. In practice this doesn't take long and the stability gained is worth the slightly slower request response.

Stopping the BLE scans while servicing the jquery (which are JSON requests) is a possibility. That might help. Thank you for your ideas - there might be something in there we can use!
I wonder if the AsyncWebServer can be paused during BLE scans - sort of the opposite.

I do this - so when BLE onResult() event finds a matching device that I am after, it reads the sensor data and sets HasChanges to true. In that state I don't process web requests until the changes have been processed by the relevant observers. In practice this doesn't take long and the stability gained is worth the slightly slower request response.

Hopefully your PR there will lead to a software resolution though.

He has 40 open PRs going back to 2017. I'm not going to hold my breath.

I'm using the latest version and haven't had to rely on any PRs which is preferred in my case as I don't want to have to keep track of such nuances in my projects

Hopefully your PR there will lead to a software resolution though.

He has 40 open PRs going back to 2017. I'm not going to hold my breath.

I'm using the latest version and haven't had to rely on any PRs which is preferred in my case as I don't want to have to keep track of such nuances in my projects

Yeah, that would be fine if I was synchronous, I'm not.  I hope to not have to go back to the other webserver.  If I can somehow "pause" the async, that might help.

Yeah, that would be fine if I was synchronous, I'm not.  I hope to not have to go back to the other webserver.  If I can somehow "pause" the async, that might help.

Are you able to completely shut down the async webserver then restart it?
Last I tried, it led to a kernel panic but that may have been my error.
On Thu, Jan 7, 2021 at 1:25 PM Lee Bussy<EMAIL_ADDRESS>wrote:

Yeah, that would be fine if I was synchronous, I'm not. I hope to not
have to go back to the other webserver. If I can somehow "pause" the async,
that might help.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/h2zero/NimBLE-Arduino/issues/167#issuecomment-756327616,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEEO2KFZQQDLYJRUBPP4YXLSYYDCFANCNFSM4VYJK3BA
.

--
----------------------
Anthony Doud

Are you able to completely shut down the async webserver then restart it?
Last I tried, it led to a kernel panic but that may have been my error.
On Thu, Jan 7, 2021 at 1:25 PM Lee Bussy<EMAIL_ADDRESS>wrote:

Yeah, that would be fine if I was synchronous, I'm not. I hope to not
have to go back to the other webserver. If I can somehow "pause" the async,
that might help.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/h2zero/NimBLE-Arduino/issues/167#issuecomment-756327616,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEEO2KFZQQDLYJRUBPP4YXLSYYDCFANCNFSM4VYJK3BA
.

--
----------------------
Anthony Doud

I reverted the mentioned commit and it did not appear to make much difference as both myself and my fellow conspirators continue to experience the PANIC.   Thank you for providing the suggestion.  We'll continue to brainstorm ideas.

One other thing it could be that could be tested however, is the commands being sent from core 0. This is a modification made to the esp-nimble files to address delay issues when sending commands from core 1. If you revert this commit and test I'd be very curious of the results.

Are you able to completely shut down the async webserver then restart it?

I do this in a different project (ESPAsyncWebServer::stop() if I remember correctly) and it works fine.  It seems heavy-handed here, it takes a sec to start and stop, so I'm more looking for a "hold up a sec" method. :)

Thanks for letting me know.
Could I ask what your scan interval and window values are? I know you've tried adjusting them I'm just curious.if I get time tonight I'll load up my async server BLE project and see if I can reproduce.

Here's our init:
void tiltScanner::init()
{
    NimBLEDevice::init("");
    pBLEScan = NimBLEDevice::getScan();
    pBLEScan->setAdvertisedDeviceCallbacks(callbacks);

    pBLEScan->setActiveScan(false);
    pBLEScan->setInterval(53);
    pBLEScan->setWindow(37);
}

Those values look pretty tight in reality if you are sharing the antenna with a webserver.
You've got 20ms open for the webserver to do its job between scan windows. If your latency to the server is more than that you'll have issues. Try increasing the interval by 200, you'll get less responses on BLE but it may solve the abort.
You can work your way up or down from there with those values to find the sweet spot.

Those values look pretty tight in reality if you are sharing the antenna with a webserver.
You've got 20ms open for the webserver to do its job between scan windows. If your latency to the server is more than that you'll have issues. Try increasing the interval by 200, you'll get less responses on BLE but it may solve the abort.
You can work your way up or down from there with those values to find the sweet spot.

I had randomly chosen some values with (as expected) no improvement.  Can you recommend what might work?
For reference, we're working with up to eight devices that are BLEacon-only.

I had randomly chosen some values with (as expected) no improvement.  Can you recommend what might work?
For reference, we're working with up to eight devices that are BLEacon-only.

Only values I can suggest would be setting the interval much higher and keeping the window as is. Something extreme, say 5000 interval.
This is not an easy issue to solve for sure and the feedback we have is not ideal. We don't know where or why the problem occurs but we can try.
If the extreme interval value does not improve the results we need to look elsewhere. Let me know what happens if you try that.

Only values I can suggest would be setting the interval much higher and keeping the window as is. Something extreme, say 5000 interval.
This is not an easy issue to solve for sure and the feedback we have is not ideal. We don't know where or why the problem occurs but we can try.
If the extreme interval value does not improve the results we need to look elsewhere. Let me know what happens if you try that.

Well, it's consistent anyway :) :
Backtrace: 0x40091978:0x3ffbe190 0x40091ba9:0x3ffbe1b0 0x40122074:0x3ffbe1d0 0x400847ed:0x3ffbe1f0 0x401d6617:0x3ffbd8e0 0x40123626:0x3ffbd900 0x4008f539:0x3ffbd920 0x4008dd45:0x3ffbd940
  #0  0x40091978:0x3ffbe190 in invoke_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:707
  #1  0x40091ba9:0x3ffbe1b0 in abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:707
  #2  0x40122074:0x3ffbe1d0 in task_wdt_isr at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/task_wdt.c:252
  #3  0x400847ed:0x3ffbe1f0 in _xt_lowint1 at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/xtensa_vectors.S:1154
  #4  0x401d6617:0x3ffbd8e0 in esp_pm_impl_waiti at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/pm_esp32.c:492
  #5  0x40123626:0x3ffbd900 in esp_vApplicationIdleHook at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/freertos_hooks.c:108
  #6  0x4008f539:0x3ffbd920 in prvIdleTask at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:4813
  #7  0x4008dd45:0x3ffbd940 in vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c:355 (discriminator 1)

Rebooting...

Well, it's consistent anyway :) :
Backtrace: 0x40091978:0x3ffbe190 0x40091ba9:0x3ffbe1b0 0x40122074:0x3ffbe1d0 0x400847ed:0x3ffbe1f0 0x401d6617:0x3ffbd8e0 0x40123626:0x3ffbd900 0x4008f539:0x3ffbd920 0x4008dd45:0x3ffbd940
  #0  0x40091978:0x3ffbe190 in invoke_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:707
  #1  0x40091ba9:0x3ffbe1b0 in abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c:707
  #2  0x40122074:0x3ffbe1d0 in task_wdt_isr at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/task_wdt.c:252
  #3  0x400847ed:0x3ffbe1f0 in _xt_lowint1 at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/xtensa_vectors.S:1154
  #4  0x401d6617:0x3ffbd8e0 in esp_pm_impl_waiti at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/pm_esp32.c:492
  #5  0x40123626:0x3ffbd900 in esp_vApplicationIdleHook at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/freertos_hooks.c:108
  #6  0x4008f539:0x3ffbd920 in prvIdleTask at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:4813
  #7  0x4008dd45:0x3ffbd940 in vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c:355 (discriminator 1)

Rebooting...

That looks to be a watchdog timeout. I'd like to see the actual error message, not just the backtrace if possible.

That looks to be a watchdog timeout. I'd like to see the actual error message, not just the backtrace if possible.

Our project is Tiltbridge and the relevant branch is devel_ardjson.  So very much appreciate you dedicating some time to look at this!

Our project is Tiltbridge and the relevant branch is devel_ardjson.  So very much appreciate you dedicating some time to look at this!

Ahh, a repo I can appreciate for sure, BLE beer brewing, LOVE IT! 😄
Looks like very nice work, I'll see if I can come up with anything.

Ahh, a repo I can appreciate for sure, BLE beer brewing, LOVE IT! 😄
Looks like very nice work, I'll see if I can come up with anything.

That's what I was thinking....
Can I give up on this BLE fitness stuff yet and just drink beer?
😁🍻
----------------------
Anthony Doud
On Thu, Jan 7, 2021 at 9:53 PM h2zero<EMAIL_ADDRESS>wrote:

Ahh, a repo I can appreciate for sure, BLE beer brewing, LOVE IT! 😄
Looks like very nice work, I'll see if I can come up with anything.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/h2zero/NimBLE-Arduino/issues/167#issuecomment-756530108,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEEO2KHZDHWX5X36NPOEF2LSYZ6R7ANCNFSM4VYJK3BA
.

That's what I was thinking....
Can I give up on this BLE fitness stuff yet and just drink beer?
😁🍻
----------------------
Anthony Doud
On Thu, Jan 7, 2021 at 9:53 PM h2zero<EMAIL_ADDRESS>wrote:

Ahh, a repo I can appreciate for sure, BLE beer brewing, LOVE IT! 😄
Looks like very nice work, I'll see if I can come up with anything.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/h2zero/NimBLE-Arduino/issues/167#issuecomment-756530108,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEEO2KHZDHWX5X36NPOEF2LSYZ6R7ANCNFSM4VYJK3BA
.

If you do get it compiled and loaded, we have generally found it to be easiest to trigger the PANIC reset by repeated hard-reloading of the 'about' tab on the web interface.  This page has several jquery json requests being simultaneously requested.

If you do get it compiled and loaded, we have generally found it to be easiest to trigger the PANIC reset by repeated hard-reloading of the 'about' tab on the web interface.  This page has several jquery json requests being simultaneously requested.

Can I give up on this BLE fitness stuff yet and just drink beer?
😁🍻

Join the club!

If you do get it compiled and loaded, we have generally found it to be easiest to trigger the PANIC reset by repeated hard-reloading of the 'about' tab on the web interface.

Going to give it a shot now.

Can I give up on this BLE fitness stuff yet and just drink beer?
😁🍻

Join the club!

If you do get it compiled and loaded, we have generally found it to be easiest to trigger the PANIC reset by repeated hard-reloading of the 'about' tab on the web interface.

Going to give it a shot now.

I'd like to see the actual error message, not just the backtrace if possible.

That's all of it.  I am using the esp32_exception_decoder monitor filter so maybe it eats it?

I'd like to see the actual error message, not just the backtrace if possible.

That's all of it.  I am using the esp32_exception_decoder monitor filter so maybe it eats it?

I've had a very stable ESP32 and reliable web server and BLE scans for around 16 hrs now. To recap I added this
#define CONFIG_ASYNC_TCP_RUNNING_CORE 1
and used the values from the original scanner example for the BLE:

I've had a very stable ESP32 and reliable web server and BLE scans for around 16 hrs now. To recap I added this
#define CONFIG_ASYNC_TCP_RUNNING_CORE 1
and used the values from the original scanner example for the BLE:

We had not used the active scan previously.  Using it, I can still get it to crash quite reliably (but maybe it's my imagination but the scans seem quicker?)  My best fail case is reloading this "About" page, which has four JQuery calls - still crashes.

We had not used the active scan previously.  Using it, I can still get it to crash quite reliably (but maybe it's my imagination but the scans seem quicker?)  My best fail case is reloading this "About" page, which has four JQuery calls - still crashes.

Sorry I could not get this loaded last night, seems I may need to use a WROVER module and didn't have any on hand, I'll bring one home from work tonight and give it a try then.

Sorry I could not get this loaded last night, seems I may need to use a WROVER module and didn't have any on hand, I'll bring one home from work tonight and give it a try then.

We had not used the active scan previously. Using it, I can still get it to crash quite reliably (but maybe it's my imagination but the scans seem quicker?) My best fail case is reloading this "About" page, which has four JQuery calls - still crashes.

This may seem obvious, probably you have already tried this. But can you disable the BLE scan and cause the ESP32 to crash just by hammering the web server concurrently from the About page?

We had not used the active scan previously. Using it, I can still get it to crash quite reliably (but maybe it's my imagination but the scans seem quicker?) My best fail case is reloading this "About" page, which has four JQuery calls - still crashes.

This may seem obvious, probably you have already tried this. But can you disable the BLE scan and cause the ESP32 to crash just by hammering the web server concurrently from the About page?

This may seem obvious, probably you have already tried this. But can you disable the BLE scan and cause the ESP32 to crash just by hammering the web server concurrently from the About page?

Nothing wrong with making sure I'm not overlooking the obvious.  While I hate when that happens, it happens. :)
I did disable BLE, and the crashing stopped.  That doesn't mean it's in the scan, it just means it's involved.

This may seem obvious, probably you have already tried this. But can you disable the BLE scan and cause the ESP32 to crash just by hammering the web server concurrently from the About page?

Nothing wrong with making sure I'm not overlooking the obvious.  While I hate when that happens, it happens. :)
I did disable BLE, and the crashing stopped.  That doesn't mean it's in the scan, it just means it's involved.

I got a chance to get this loaded and I cannot access the webserver, browser complains of too many redirects... so I am unfortunately unable to test this locally.

I got a chance to get this loaded and I cannot access the webserver, browser complains of too many redirects... so I am unfortunately unable to test this locally.

Nevermind, works now. I realized I had not got the data folder uploaded into SPIFFS. Just got the error to occur on my board as well.

Nevermind, works now. I realized I had not got the data folder uploaded into SPIFFS. Just got the error to occur on my board as well.

Well after some testing and removing the assert in nimble I found the errors after were many watchdog timeout issues. After looking at the relevant code in the backtrace I cannot find anything wifi or ble related. My only thought then was there is a controller code problem (closed source :( ). So I updated my IDF and looked at the update logs:
Submodule components/bt/controller/lib 0fd5fffa0..a799b1910 (rewind):
  < bugfix: clear the timer after remote reject when respond the parameters update
  < fix exl200, add btdm_disable_adv_delay, fix RPA addr update error(c03ddb37)
  < Add ACL-U flow on & off when slave accept packet type request (4ae2b95)
  < Free EM ACL RX buffer in disconnection procudure (a61cd5d)
  < Add protection for interval_min when it comes to zero (6bf0910)
  < Remove dummy assert when receive passkey requst (f060711)
  < Fix two bugs:(cfa40b27)
  < fix hci timeout, add timer to monitor scan evt(9fd95e0)
  < Fix controller dead when clkn overflow (1c5361f)
  < fix assert(10, 9), when the AFPM happens(019fd74b)
  < fixed tx_prog insert error(7760a76)
  < Fix waking up fail while sleeping(330943b)
  < make sleep avaliable in hli(for future use)(845e59c)
  < btdm fix cache disable crash(b73c48e)
  < Some bugfix about ble dynamic prio(3901b15)
  < Insert the llcp packet to the top of the tx_preg linked list(153fac6c)
  < refactor coexist event in order to adapter to new chips(07adeadf)
  < Confirmed bugfix from Baidu Project (4bb1a62)

Seems to be a few fixes in there that may be related to this issue, I will load this using the IDF master branch and check again.

Well after some testing and removing the assert in nimble I found the errors after were many watchdog timeout issues. After looking at the relevant code in the backtrace I cannot find anything wifi or ble related. My only thought then was there is a controller code problem (closed source :( ). So I updated my IDF and looked at the update logs:
Submodule components/bt/controller/lib 0fd5fffa0..a799b1910 (rewind):
  < bugfix: clear the timer after remote reject when respond the parameters update
  < fix exl200, add btdm_disable_adv_delay, fix RPA addr update error(c03ddb37)
  < Add ACL-U flow on & off when slave accept packet type request (4ae2b95)
  < Free EM ACL RX buffer in disconnection procudure (a61cd5d)
  < Add protection for interval_min when it comes to zero (6bf0910)
  < Remove dummy assert when receive passkey requst (f060711)
  < Fix two bugs:(cfa40b27)
  < fix hci timeout, add timer to monitor scan evt(9fd95e0)
  < Fix controller dead when clkn overflow (1c5361f)
  < fix assert(10, 9), when the AFPM happens(019fd74b)
  < fixed tx_prog insert error(7760a76)
  < Fix waking up fail while sleeping(330943b)
  < make sleep avaliable in hli(for future use)(845e59c)
  < btdm fix cache disable crash(b73c48e)
  < Some bugfix about ble dynamic prio(3901b15)
  < Insert the llcp packet to the top of the tx_preg linked list(153fac6c)
  < refactor coexist event in order to adapter to new chips(07adeadf)
  < Confirmed bugfix from Baidu Project (4bb1a62)

Seems to be a few fixes in there that may be related to this issue, I will load this using the IDF master branch and check again.

It is great to hear that you got it running.
After further troubleshooting, I think we may have a couple of issues at play which is why the backtraces are slightly different between @lbussy and myself.

The "normal" configuration for this project is supposed to be a Lolin D32 Pro and Lolin TFT (d32_pro_tft firmware build which is what @lbussy is using). This build will not work if trying to run the ESP32 without a screen attached.  When using a TFT display, we are seeing watchdog trigger errors and this appears to relate to interaction with the TFT during several simultaneous requests to the Async Webserver.

If running the OLED (lcd_ssd1306 firmware) configuration without a screen (which is what I was initially using), the watchdog timers from the main loop are no longer an issue but we were still receiving sporadic RESETs corresponding to the backtrace I posted in the original message.  Leaving the scan Window at 37 and widening the Interval to 197 seemed to eliminate these resets. However, it would be great to be able to tighten the Interval back to improve probability of receiving advertisements if possible so please do let us know if the IDF fixes appear to improve things on the BLE/Wifi side.

It is great to hear that you got it running.
After further troubleshooting, I think we may have a couple of issues at play which is why the backtraces are slightly different between @lbussy and myself.

The "normal" configuration for this project is supposed to be a Lolin D32 Pro and Lolin TFT (d32_pro_tft firmware build which is what @lbussy is using). This build will not work if trying to run the ESP32 without a screen attached.  When using a TFT display, we are seeing watchdog trigger errors and this appears to relate to interaction with the TFT during several simultaneous requests to the Async Webserver.

If running the OLED (lcd_ssd1306 firmware) configuration without a screen (which is what I was initially using), the watchdog timers from the main loop are no longer an issue but we were still receiving sporadic RESETs corresponding to the backtrace I posted in the original message.  Leaving the scan Window at 37 and widening the Interval to 197 seemed to eliminate these resets. However, it would be great to be able to tighten the Interval back to improve probability of receiving advertisements if possible so please do let us know if the IDF fixes appear to improve things on the BLE/Wifi side.

I updated my IDF and looked at the update logs:

Some of those look real promising.  Also makes me wonder where all the baling wire and bubble gun is coming from. :)

I updated my IDF and looked at the update logs:

Some of those look real promising.  Also makes me wonder where all the baling wire and bubble gun is coming from. :)

Ok, so my results are sadly non-existent. I had hoped to find out if the controller was the cause of this but after a couple hours futzing around and modifying all the libraries to get them to compile with IDF4.2 I ran into a roadblock where the webserver would say it start and I can see it connected on my router but there is no response in the browser.
So, I cannot say for sure that is the cause, if I had the time to go about updating a few libraries (already did asyncWifiManager) I might have an answer. I don't think asyncWebServer plays nice with IDF4.2 yet, and I got hung up on the data_sender.init(); call in your repo.
If you want to invest the time to try this, I can help and give you the changes I made.
@pletch I did use the lcd_ssd1306 config and confirmed the exceptions both of you experienced with the changes I made in NimBLE, both lead to the same place. I wish I could confirm the fix :(

Ok, so my results are sadly non-existent. I had hoped to find out if the controller was the cause of this but after a couple hours futzing around and modifying all the libraries to get them to compile with IDF4.2 I ran into a roadblock where the webserver would say it start and I can see it connected on my router but there is no response in the browser.
So, I cannot say for sure that is the cause, if I had the time to go about updating a few libraries (already did asyncWifiManager) I might have an answer. I don't think asyncWebServer plays nice with IDF4.2 yet, and I got hung up on the data_sender.init(); call in your repo.
If you want to invest the time to try this, I can help and give you the changes I made.
@pletch I did use the lcd_ssd1306 config and confirmed the exceptions both of you experienced with the changes I made in NimBLE, both lead to the same place. I wish I could confirm the fix :(

I really appreciate you trying it @h2zero.  At least determining the exceptions with the 1306 and TFT led to the same place was more information than we had.  It means that whether we make the jQuery calls "kinder" or not, we still have the potential to hit them in other areas.
Yes, I'd love to see the work you did to get that far.  You never know.

I really appreciate you trying it @h2zero.  At least determining the exceptions with the 1306 and TFT led to the same place was more information than we had.  It means that whether we make the jQuery calls "kinder" or not, we still have the potential to hit them in other areas.
Yes, I'd love to see the work you did to get that far.  You never know.

Found an interesting post here https://www.esp32.com/viewtopic.php?f=19&t=16545 that describes a similar problem. Seems possible the issue is triggered by ESPAsyncWebServer taking too long to do something with the radio.
Here is the modified wifiManager.
AsyncWiFiManager.zip
And the platformio.ini
platformio.ini.txt
That will use the idf4.2 branch for the ardruino-esp32 core. You will need to run the get.py or get.exe after it clones the repo and copy the contents from the platformio/packages/framework-arduinoespressif32/tools/xtensa-esp32-elf folder to platformio/packages/toolchain-xtensa32.
This is what I did to get it compiling, I'm not a platformio user so I don't know it well, there is probably a better way.

Found an interesting post here https://www.esp32.com/viewtopic.php?f=19&t=16545 that describes a similar problem. Seems possible the issue is triggered by ESPAsyncWebServer taking too long to do something with the radio.
Here is the modified wifiManager.
AsyncWiFiManager.zip
And the platformio.ini
platformio.ini.txt
That will use the idf4.2 branch for the ardruino-esp32 core. You will need to run the get.py or get.exe after it clones the repo and copy the contents from the platformio/packages/framework-arduinoespressif32/tools/xtensa-esp32-elf folder to platformio/packages/toolchain-xtensa32.
This is what I did to get it compiling, I'm not a platformio user so I don't know it well, there is probably a better way.

Thank you very much for that.  It gives me a great starting place to possibly keep prodding this.

Thank you very much for that.  It gives me a great starting place to possibly keep prodding this.

@lbussy I had so good results in testing against https://github.com/espressif/arduino-esp32/tree/idf-release/v3.3 while investigating other bugs. That branch has the controller updates and seems more reliable, not sure how well it will work for you but it should be easy enough to test. You will need to use NimBLE with the commit I mentioned above reverted as that now causes stack overflow exceptions on that branch.

@lbussy I had so good results in testing against https://github.com/espressif/arduino-esp32/tree/idf-release/v3.3 while investigating other bugs. That branch has the controller updates and seems more reliable, not sure how well it will work for you but it should be easy enough to test. You will need to use NimBLE with the commit I mentioned above reverted as that now causes stack overflow exceptions on that branch.

Just to update this, the branch mentioned above is now merged into the arduino-esp32 master.
Updating your sources to use that should provide the fix for this issue. It has the latest controller bug fixes, at least for IDF 3.3 anyway.

Just to update this, the branch mentioned above is now merged into the arduino-esp32 master.
Updating your sources to use that should provide the fix for this issue. It has the latest controller bug fixes, at least for IDF 3.3 anyway.

Just to update this, the branch mentioned above is now merged into the arduino-esp32 master.
Updating your sources to use that should provide the fix for this issue. It has the latest controller bug fixes, at least for IDF 3.3 anyway.

Would you mind just outlining how one would use a specific branch as opposed to whatever the latest is in Platform IO? I can't see how the release versions are related to the idf and arduino parts. Thank you.

Just to update this, the branch mentioned above is now merged into the arduino-esp32 master.
Updating your sources to use that should provide the fix for this issue. It has the latest controller bug fixes, at least for IDF 3.3 anyway.

Would you mind just outlining how one would use a specific branch as opposed to whatever the latest is in Platform IO? I can't see how the release versions are related to the idf and arduino parts. Thank you.

You can just change it in the platformio.ini:
platform_packages = platformio/framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#2452c1fb539246e47a715b74a3ad25b8a7ebbec7

You can just change it in the platformio.ini:
platform_packages = platformio/framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#2452c1fb539246e47a715b74a3ad25b8a7ebbec7

I am WAY behind on stuff but I just wanted to answer and thank you for following up.
I decided to move my domain registrars, hosting, etc., and let's just say I provided myself the opportunity to screw up a number of things. :)
As soon as I have a chance I'll take a look and see if things are better.

I am WAY behind on stuff but I just wanted to answer and thank you for following up.
I decided to move my domain registrars, hosting, etc., and let's just say I provided myself the opportunity to screw up a number of things. :)
As soon as I have a chance I'll take a look and see if things are better.

I'm putting some finishing touches on a couple patches, after that, hopefully I'll get a chance to test this out as well.

I'm putting some finishing touches on a couple patches, after that, hopefully I'll get a chance to test this out as well.

You can just change it in the platformio.ini:
platform_packages = platformio/framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#2452c1fb539246e47a715b74a3ad25b8a7ebbec7

I have managed to reference the framework as you suggested. I get this compile error, so is there a specific version of NimBLE that I should reference in my lib_deps:

Thank you.

You can just change it in the platformio.ini:
platform_packages = platformio/framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#2452c1fb539246e47a715b74a3ad25b8a7ebbec7

I have managed to reference the framework as you suggested. I get this compile error, so is there a specific version of NimBLE that I should reference in my lib_deps:

Thank you.

Not to worry, I grabbed master as I see a fix for this was pushed recently

Not to worry, I grabbed master as I see a fix for this was pushed recently

I'm trying to push a nimble core update later tonight that works better with the latest arduino master. There are a few core file patches as well as undoing some workarounds for older arduino versions.
My guess is that it will fix the issues reported here and I will test it once done.

I'm trying to push a nimble core update later tonight that works better with the latest arduino master. There are a few core file patches as well as undoing some workarounds for older arduino versions.
My guess is that it will fix the issues reported here and I will test it once done.

Here's hoping, sir!

Here's hoping, sir!

No luck so far, compiled and loaded fine but hangs at 312 V: Initializing WiFi. no time to investigate tonight. For reference the only change I made was in your (devel branch) platformio.ini @ line 72 I have:
platform_packages = platformio/framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#d2530850a38af18ada5e805f0f4539e0048c43b3 no other changes.

No luck so far, compiled and loaded fine but hangs at 312 V: Initializing WiFi. no time to investigate tonight. For reference the only change I made was in your (devel branch) platformio.ini @ line 72 I have:
platform_packages = platformio/framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#d2530850a38af18ada5e805f0f4539e0048c43b3 no other changes.

No luck so far, compiled and loaded fine but hangs at 312 V: Initializing WiFi. no time to investigate tonight. For reference the only change I made was in your (devel branch) platformio.ini @ line 72 I have:
platform_packages = platformio/framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#d2530850a38af18ada5e805f0f4539e0048c43b3 no other changes.

I tried these two things:
platform_packages = framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#2452c1fb539246e47a715b74a3ad25b8a7ebbec7
NimBLE latest master
I still experience ESP32 crashes pretty consistently. The present state of affairs seems that BLE + WiFi has some issues which make using things like BLE scanning and a web server challenging. You'd certainly have to be defensive of how your  system would deal with crashing randomly. For some use cases this would just be unacceptable.

No luck so far, compiled and loaded fine but hangs at 312 V: Initializing WiFi. no time to investigate tonight. For reference the only change I made was in your (devel branch) platformio.ini @ line 72 I have:
platform_packages = platformio/framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#d2530850a38af18ada5e805f0f4539e0048c43b3 no other changes.

I tried these two things:
platform_packages = framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#2452c1fb539246e47a715b74a3ad25b8a7ebbec7
NimBLE latest master
I still experience ESP32 crashes pretty consistently. The present state of affairs seems that BLE + WiFi has some issues which make using things like BLE scanning and a web server challenging. You'd certainly have to be defensive of how your  system would deal with crashing randomly. For some use cases this would just be unacceptable.

That's unfortunate. Are the crashes the same as before or different?

That's unfortunate. Are the crashes the same as before or different?

Found this when looking at  async issues https://github.com/me-no-dev/ESPAsyncWebServer/issues/900#issuecomment-748505931
might be worth giving that a shot.

Found this when looking at  async issues https://github.com/me-no-dev/ESPAsyncWebServer/issues/900#issuecomment-748505931
might be worth giving that a shot.

Found this when looking at  async issues https://github.com/me-no-dev/ESPAsyncWebServer/issues/900#issuecomment-748505931
might be worth giving that a shot.

Found this when looking at  async issues https://github.com/me-no-dev/ESPAsyncWebServer/issues/900#issuecomment-748505931
might be worth giving that a shot.

Found this when looking at async issues me-no-dev/ESPAsyncWebServer#900 (comment)
might be worth giving that a shot.

Thank you. I've tried to reference that branch as  it does sound promising. Unfortunately there are a lot of compile errors which I don't really understand related to things like this:

I'll keep trying.

Found this when looking at async issues me-no-dev/ESPAsyncWebServer#900 (comment)
might be worth giving that a shot.

Thank you. I've tried to reference that branch as  it does sound promising. Unfortunately there are a lot of compile errors which I don't really understand related to things like this:

I'll keep trying.

I have recently achieved very good stability and performance.  I attribute the improvements to the following:

using the latest arduino-espressif platform packages
revising event queue management in AsyncTCP after several hints in some of the ESPAsyncWebserver issue threads
implementing better parallelism and task flow control using FreeRTOS tasks to distribute work between the two ESP32 cores.

WDT errors are completely gone and I haven't been able to produce any other crashes for several days now under heavy load.  I am closing this issue as I consider it resolved from my perspective.  If others wish to continue the discussion, feel free to reopen.

I have recently achieved very good stability and performance.  I attribute the improvements to the following:

using the latest arduino-espressif platform packages
revising event queue management in AsyncTCP after several hints in some of the ESPAsyncWebserver issue threads
implementing better parallelism and task flow control using FreeRTOS tasks to distribute work between the two ESP32 cores.

WDT errors are completely gone and I haven't been able to produce any other crashes for several days now under heavy load.  I am closing this issue as I consider it resolved from my perspective.  If others wish to continue the discussion, feel free to reopen.

Glad you got it stable!

Glad you got it stable!

I attribute the improvements to the following:

Would you mind outlining what you did in more detail? When you say latest arduino-espressif are you adding anything specific to the platformio.ini file or you are using latest PlatformIO Espressif 2.1? I pretty much followed the last 2 of your list and got to a much more stable place, but have not been able to eliminate the crashes all together. My impression is that while scanning is in progress and a WifiUDP::parsePacket is processed - in my case as part of OTA handler(note OTA is not in progress, just the handler is polled by the loop function). I run the scan in a Freertos task on core 0, and as part of a state machine so that tasks run according to state and not concurrently. That said I'm happy for the issue to be closed as I don't think there is anything wrong with the NimBLE library, rather it is not easy to get BLE and WiFi working in the scenario described. Running BLE on it's own is fine, I run into heap running out and not recovering when I combine everything together, even with a state machine.
Thanks to @h2zero for finding this https://github.com/me-no-dev/ESPAsyncWebServer/issues/876 - I modified the AsynTCP.cpp according to the suggestions there and the web server responds near instantly most of the time. I was getting responses taking up to five seconds, now it is typically under 200ms.

I attribute the improvements to the following:

Would you mind outlining what you did in more detail? When you say latest arduino-espressif are you adding anything specific to the platformio.ini file or you are using latest PlatformIO Espressif 2.1? I pretty much followed the last 2 of your list and got to a much more stable place, but have not been able to eliminate the crashes all together. My impression is that while scanning is in progress and a WifiUDP::parsePacket is processed - in my case as part of OTA handler(note OTA is not in progress, just the handler is polled by the loop function). I run the scan in a Freertos task on core 0, and as part of a state machine so that tasks run according to state and not concurrently. That said I'm happy for the issue to be closed as I don't think there is anything wrong with the NimBLE library, rather it is not easy to get BLE and WiFi working in the scenario described. Running BLE on it's own is fine, I run into heap running out and not recovering when I combine everything together, even with a state machine.
Thanks to @h2zero for finding this https://github.com/me-no-dev/ESPAsyncWebServer/issues/876 - I modified the AsynTCP.cpp according to the suggestions there and the web server responds near instantly most of the time. I was getting responses taking up to five seconds, now it is typically under 200ms.

I attribute the improvements to the following:

Would you mind outlining what you did in more detail? When you say latest arduino-espressif are you adding anything specific to the platformio.ini file or you are using latest PlatformIO Espressif 2.1? I pretty much followed the last 2 of your list and got to a much more stable place, but have not been able to eliminate the crashes all together. My impression is that while scanning is in progress and a WifiUDP::parsePacket is processed - in my case as part of OTA handler(note OTA is not in progress, just the handler is polled by the loop function) - a crash often occurs. I run the scan in a Freertos task on core 0, and as part of a state machine so that tasks run according to state and not concurrently. That said I'm happy for the issue to be closed as I don't think there is anything wrong with the NimBLE library, rather it is not easy to get BLE and WiFi working in the scenario described. Running BLE on it's own is fine, I run into heap running out and not recovering when I combine everything together, even with a state machine.
Thanks to @h2zero for finding this me-no-dev/ESPAsyncWebServer#876 - I modified the AsynTCP.cpp according to the suggestions there and the web server responds near instantly most of the time. I was getting responses taking up to five seconds, now it is typically under 200ms.

I can't answer for him but you can use:
platform_packages = framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#master
in platformio.ini.
It's a blessing and a curse though. Small things they change have caused issues for me that made it past my initial testing and into binaries.

I attribute the improvements to the following:

Would you mind outlining what you did in more detail? When you say latest arduino-espressif are you adding anything specific to the platformio.ini file or you are using latest PlatformIO Espressif 2.1? I pretty much followed the last 2 of your list and got to a much more stable place, but have not been able to eliminate the crashes all together. My impression is that while scanning is in progress and a WifiUDP::parsePacket is processed - in my case as part of OTA handler(note OTA is not in progress, just the handler is polled by the loop function) - a crash often occurs. I run the scan in a Freertos task on core 0, and as part of a state machine so that tasks run according to state and not concurrently. That said I'm happy for the issue to be closed as I don't think there is anything wrong with the NimBLE library, rather it is not easy to get BLE and WiFi working in the scenario described. Running BLE on it's own is fine, I run into heap running out and not recovering when I combine everything together, even with a state machine.
Thanks to @h2zero for finding this me-no-dev/ESPAsyncWebServer#876 - I modified the AsynTCP.cpp according to the suggestions there and the web server responds near instantly most of the time. I was getting responses taking up to five seconds, now it is typically under 200ms.

I can't answer for him but you can use:
platform_packages = framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32.git#master
in platformio.ini.
It's a blessing and a curse though. Small things they change have caused issues for me that made it past my initial testing and into binaries.

@lbussy, @pletch I have not tested this with tiltbridge but I made some significant changes to NimBLEScan and NimBLEAdvertised device that significantly increase performance and may alleviate some of the watchdog timeouts. If you're interested in testing this please checkout this branch.
Some notes for your use: in the scan setup call pBLEScan->setMaxResults(0); (you also will not need to call clearResults() when the scan stops with this setting).

@lbussy, @pletch I have not tested this with tiltbridge but I made some significant changes to NimBLEScan and NimBLEAdvertised device that significantly increase performance and may alleviate some of the watchdog timeouts. If you're interested in testing this please checkout this branch.
Some notes for your use: in the scan setup call pBLEScan->setMaxResults(0); (you also will not need to call clearResults() when the scan stops with this setting).

We'll give it a go and let you know how it works out, thank you!

We'll give it a go and let you know how it works out, thank you!

Some notes for your use: in the scan setup call pBLEScan->setMaxResults(0);

I added this to my platformio.ini file lib_deps:
https://github.com/h2zero/NimBLE-Arduino.git#refactor-scan
But after building it I don't see the setMaxResults() method. Am I missing something? Thanks.

Some notes for your use: in the scan setup call pBLEScan->setMaxResults(0);

I added this to my platformio.ini file lib_deps:
https://github.com/h2zero/NimBLE-Arduino.git#refactor-scan
But after building it I don't see the setMaxResults() method. Am I missing something? Thanks.

I managed to get it to compile - had to clean out all the environments and used this reference:
https://github.com/h2zero/NimBLE-Arduino.git#79cd9464b2c98f6ade1d1ec1fc01959cd16521a1

I managed to get it to compile - had to clean out all the environments and used this reference:
https://github.com/h2zero/NimBLE-Arduino.git#79cd9464b2c98f6ade1d1ec1fc01959cd16521a1

@lbussy, @pletch I have not tested this with tiltbridge but I made some significant changes to NimBLEScan and NimBLEAdvertised device that significantly increase performance and may alleviate some of the watchdog timeouts. If you're interested in testing this please checkout this branch.
Some notes for your use: in the scan setup call pBLEScan->setMaxResults(0); (you also will not need to call clearResults() when the scan stops with this setting).

I've given this a try. It works fine, most notably for me is the heap seems to be almost entirely preserved. I would often see the heap go from 80k down to 35k(the danger zone from my experience). So far it has been fluctuating around only a few hundred bytes. This is also probably because I am now specifying max results to 3 as I only have 2 devices of interest. This is anecdotal and my use case is likely to differ from others, but still a good improvement and I see the payload is now complete which was another issue that seems to be rolled into this branch,
@h2zero Excellent work here - this project is so useful and was much needed. Thank you to you and everyone else who is making this better and better!

@lbussy, @pletch I have not tested this with tiltbridge but I made some significant changes to NimBLEScan and NimBLEAdvertised device that significantly increase performance and may alleviate some of the watchdog timeouts. If you're interested in testing this please checkout this branch.
Some notes for your use: in the scan setup call pBLEScan->setMaxResults(0); (you also will not need to call clearResults() when the scan stops with this setting).

I've given this a try. It works fine, most notably for me is the heap seems to be almost entirely preserved. I would often see the heap go from 80k down to 35k(the danger zone from my experience). So far it has been fluctuating around only a few hundred bytes. This is also probably because I am now specifying max results to 3 as I only have 2 devices of interest. This is anecdotal and my use case is likely to differ from others, but still a good improvement and I see the payload is now complete which was another issue that seems to be rolled into this branch,
@h2zero Excellent work here - this project is so useful and was much needed. Thank you to you and everyone else who is making this better and better!

I don't have time to "dig in" and learn what may be beneficial, I slapped it in just to see what happens.  Unfortunately, the heap usage went up significantly.
Before:
Free Heap: 163544, Max Allocated: 93028, Frag: 44

After:
Free Heap: 135536, Max Allocated: 97392, Frag: 29

Fortunately (but unfortunately I suppose) we are using a deinit() before calling some memory intensive functions.  Therefore the delta is not impactful right now.

I don't have time to "dig in" and learn what may be beneficial, I slapped it in just to see what happens.  Unfortunately, the heap usage went up significantly.
Before:
Free Heap: 163544, Max Allocated: 93028, Frag: 44

After:
Free Heap: 135536, Max Allocated: 97392, Frag: 29

Fortunately (but unfortunately I suppose) we are using a deinit() before calling some memory intensive functions.  Therefore the delta is not impactful right now.

@lbussy As I said, my use case will differ from others. Nonetheless I would say the heap is very healthy from my point of view. I'm pegged at around 85k. I expect that it is not unusual for it to drop, but I've experienced that once you get to the <35k it is likely to crash the ESP32.

@lbussy As I said, my use case will differ from others. Nonetheless I would say the heap is very healthy from my point of view. I'm pegged at around 85k. I expect that it is not unusual for it to drop, but I've experienced that once you get to the <35k it is likely to crash the ESP32.

I should add that I have now had a stable web server and BLE running without crashes for 7hrs now - not related to this branch of NimBLE directly, but mainly around the way the BLE and web server and other tasks are orchestrated using Freertos. I had 2 weeks of pain trying to get to this stage. I need to get through 24 hrs straight to be satisfied that I have indeed cracked it and that the pattern will be fit for a MVP.

I should add that I have now had a stable web server and BLE running without crashes for 7hrs now - not related to this branch of NimBLE directly, but mainly around the way the BLE and web server and other tasks are orchestrated using Freertos. I had 2 weeks of pain trying to get to this stage. I need to get through 24 hrs straight to be satisfied that I have indeed cracked it and that the pattern will be fit for a MVP.

I don't have time to "dig in" and learn what may be beneficial, I slapped it in just to see what happens. Unfortunately, the heap usage went up significantly.

Before:

Free Heap: 163544, Max Allocated: 93028, Frag: 44
After:

Free Heap: 135536, Max Allocated: 97392, Frag: 29

Yikes, that shouldn't happen, I'll have to look into this. Thanks for letting me know.
@justoke Awesome!, glad to hear it's working out for you.

I don't have time to "dig in" and learn what may be beneficial, I slapped it in just to see what happens. Unfortunately, the heap usage went up significantly.

Before:

Free Heap: 163544, Max Allocated: 93028, Frag: 44
After:

Free Heap: 135536, Max Allocated: 97392, Frag: 29

Yikes, that shouldn't happen, I'll have to look into this. Thanks for letting me know.
@justoke Awesome!, glad to hear it's working out for you.

I have a VERY chatty BLE environment.  Is that a factor?

I have a VERY chatty BLE environment.  Is that a factor?

I have a VERY chatty BLE environment. Is that a factor?

That's what this branch was intended to help with, if max results is set to 0 then no advertised devices will be stored unless done by copy in the callback in the user code. It's quite shocking that you are seeing the opposite of the intended effect.

I have a VERY chatty BLE environment. Is that a factor?

That's what this branch was intended to help with, if max results is set to 0 then no advertised devices will be stored unless done by copy in the callback in the user code. It's quite shocking that you are seeing the opposite of the intended effect.

@lbussy, I'm not able to reproduce the heap use you see.
This is master branch:

34527 V: Free Heap: 132928, Max Allocated: 95112, Frag: 29

This is refactor-scan:

34508 V: Free Heap: 133252, Max Allocated: 95524, Frag: 29

@lbussy, I'm not able to reproduce the heap use you see.
This is master branch:

34527 V: Free Heap: 132928, Max Allocated: 95112, Frag: 29

This is refactor-scan:

34508 V: Free Heap: 133252, Max Allocated: 95524, Frag: 29

Well after testing this branch for a while and spamming the about page like a mad man I have not been able to reproduce the crash originally discussed in this thread 😄 . Whatever changes you all made, good work!

Well after testing this branch for a while and spamming the about page like a mad man I have not been able to reproduce the crash originally discussed in this thread 😄 . Whatever changes you all made, good work!

Maybe we need to get you to compile and send it to me?  I can still kill it. :)
Which branch are you testing with?

Maybe we need to get you to compile and send it to me?  I can still kill it. :)
Which branch are you testing with?

I'm working with this https://github.com/thorrak/tiltbridge/commit/8c2c81f96f51be6298fd76a07f28ac6046095eb0
using lcd_ssd1306 env

I'm working with this https://github.com/thorrak/tiltbridge/commit/8c2c81f96f51be6298fd76a07f28ac6046095eb0
using lcd_ssd1306 env

Ah yeah.  The 1306 is quite a bit more stable for some reason.

Ah yeah.  The 1306 is quite a bit more stable for some reason.

Interesting, sounds like the display updates may be a contributing factor then. I'm not sure how they are handled though so just thinking out loud.

Interesting, sounds like the display updates may be a contributing factor then. I'm not sure how they are handled though so just thinking out loud.

We've been leaning in that direction, but we're not able to prove it (yet).

We've been leaning in that direction, but we're not able to prove it (yet).

Are you still seeing a big difference in heap consumption with the refactor-scan branch?

Are you still seeing a big difference in heap consumption with the refactor-scan branch?

Are you still seeing a big difference in heap consumption with the refactor-scan branch?

It's looking good so far.

Are you still seeing a big difference in heap consumption with the refactor-scan branch?

It's looking good so far.


Media: Only transcode 1 resource at a time

Context

Summary

Relevant Technical Choices

To-do

User-facing changes

Testing Instructions

[ ] This is a non-user-facing change and requires no QA

This PR can be tested by following these steps:

Reviews
Does this PR have a security-related impact?

Does this PR change what data or activity we track or use?

Does this PR have a legal-related impact?

Checklist

[x] This PR addresses an existing issue and I have linked this PR to it in ZenHub
[ ] I have tested this code to the best of my abilities
[ ] I have verified accessibility to the best of my abilities (docs)
[ ] I have verified i18n and l10n (translation, right-to-left layout) to the best of my abilities
[ ] This code is covered by automated tests (unit, integration, and/or e2e) to verify it works as intended (docs)
[ ] I have added documentation where necessary
[x] I have added a matching Type: XYZ label to the PR

Fixes #8779

Might worth doing a merge from main. This PR is 27 days old.

One thing to note, the pending videos, do not have a messaging on them on hover.
Hoving over trancoding video

Hoving over pending video, shows no message at all.

Not sure if it should say, pending processing or just processing through.

https://user-images.githubusercontent.com/237508/148792273-55e5ebef-89f0-4dea-aceb-f9a9281d7884.mov
I am seeing a browser crash here.
I believe what is happening here, is that the first video is processed, then when it finishes, all the other video all start transcoding at once.

Sounds like I need to come up with some good unit tests to cover this 🤔

is there any worth in extracting the simple async task queue from this PR and using it here?
https://github.com/google/web-stories-wp/pull/9584

I've been considering re-using that one for some other tasks where we'd needed to perform 1 thing at a time (hence my comment on your PR last month), but I'm not sure it's a good fit here.
This one here is part of a larger media upload queue, where all steps should run in parallel except this one task should be sequentially. Adopting your work would basically mean creating a separate queue just for this, which might make things unnecessarily complex. But I'll take another look for sure.

I am still seeing the issue locally. Anything more than 7 videos being uploaded at once, will still see the browser crash.
https://user-images.githubusercontent.com/237508/149515332-14addd37-885a-4e0d-bf55-ce033f26ece5.mov

The following lines
https://github.com/google/web-stories-wp/blob/7440358cb9cd4115550a5b2f51319fdfba747e6a/packages/story-editor/src/app/media/utils/useMediaUploadQueue/index.js#L145-L192
Should also be limited to one at a time. This code generates the poster image using ffmpeg for files that need to be transcoded. This spins up ffmpeg to get a poster image. But this would be done for all uploading video. Which maybe why the above issue is happening. Sadly, if don't generate poster image, the preview will not look great, but I don't see a way around this.

I have confirmed, if you drag 7 mp4 into the canvas, browser does not crash. 7 mov files and it does. It must be poster generation via FFMPeg is the issue.

Interesting! Totally forgot about the ffmpeg poster generation.
Should be easy to address that. Will take a look.

Also not seeing any changes related to this comment https://github.com/google/web-stories-wp/pull/9999#issuecomment-1008985240
Is this going to actioned as part of this PR or not?

Hoving over pending video, shows no message at all.

Pending items are just pending and we don't know whether they are going to need transcoding or not.
We've never shown a tooltip for pending items, so this is more a feature request. But I suppose it makes sense.
I'm going to generalize the tooltip text a bit for it to make more sense.

There different now, is that pending items used to just be pending for half a second while the join the queue. Now a pending item could be queued for a while, minutes while other video transcode in front of it. If a video has a progress bar for 5 minute with the tooltip, might be confusing. Maybe the message should be like this.

Pending - "Video is await processing".
Trasncoding - "Video is being processed"
Uploading - "Video is uploading".

That way it clear to the user

https://user-images.githubusercontent.com/237508/149549539-c40fb997-1ee5-4108-aeb5-1d14d99e70af.mov
I am not longer seeing the crashing, which is good. However, I am seeing some poster images not generating at all. There are some weird gaps.

Not at all or just one after another? To me it looks like the latter.

After a couple of tries. There is no order I can see. It is not the first / last element. It can be any of them.

I don‘t think an order is (or should be) guaranteed for the ffmpeg poster generation.

@swissspidy I am not sure I understand your last two comments. The video I uploaded, shows a regression in functionality. This change messages that some poster images not loading at all. Can you take a look?

So I found one sorta related bug where the temporary poster image generated by ffmpeg was later dropped after uploading. I fixed this in 55f5a8b5bbee6384300559c31b8bb71aedfecffc. But worth noting that this was also a bug on main and not introduced here. Maybe it was just made more obvious.
Apart from that, in my testing the ffmpeg poster generation works fine with 1 poster being generated at a time.
I'll do some more testing though.

In my testing, it seems like it only one or two of files that are failing. Example.
Otherwise this is working well.

That's actually the file I was testing with.

Approved!
This needs QA instructions, as this testing here will be valuable.


Responsive Accordion Tabs bug  in Foundation v6.5.0

What happen?
When use the Responsive Accordion Tabs function, every another Foundation functions will be shutdown, I can't open a Reveal,  Off-Canvas,...etc. And the Responsive Accordion Tabs function doesn't work well with responsive when I change the width of screen. But no error is printed in console of browser.
You can check the codepen here https://codepen.io/anvucodespot/pen/YvPxqj
I use your official example, your original CSS and your original JS, so you can check it easier.
I will publish my project in the nex few days, so hope you fix it as soon as you can.
Thank you.
Test Case and/or Steps to Reproduce

Test Case:

[ ] Try to open a Reveal, Off-Canvas, Dropdown,...etc
[ ] Check the responsive fucntion

How to reproduce:

Console the error if anything is wrong
Prevent function conflict
Make sure it works with responsive

Your Environment

Foundation version(s) used: 6.5.0 from npm
Browser(s) name and version(s): Google Chrome v67.0.3396.99
Operating System and version (desktop or mobile): Windows 10, iOS 10, Android 7

Checklist (all required):

[x] I have read and follow the CONTRIBUTING.md document.
[x] There are no other issues similar to this one.
[x] The issue title is descriptive.

Probably a duplicate of https://github.com/zurb/foundation-sites/issues/11360. I'll double-check that tomorrow.

Note: This issue uses v6.5.0-rc.1, not v6.5.0

1+
the same error with offcanvas + accordion, offcanvas not work.

I can also create a patch for patch-package if this is needed.

Duplicate of #11360. Already resolved on develop and develop-6.5.


fix(csp): ensure charset meta at top of head

Closes #444
Types of changes

[x] Bug fix (a non-breaking change which fixes an issue)
[ ] New feature (a non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Description

Currently we insert the CSP meta tag as the first element, to make sure it is read by the UA as soon as possible.
However the W3C spec for HTML5 mentions that the charset meta tag should be the first in head
This PR ensures that the CSP meta tag is inserted right after the charset meta tag, if it exists.
Checklist:

[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have added tests to cover my changes (if not applicable, please state why)

I'm closing #445 in favour of this version, which is v2-compatible

uses the recombineHtml step to generate meta instead of the spSsgHashes step
uses regex instead of cheerio, in anticipation of completely dropping cheerio in v2

Thank you very much! ♥


Number the Source File Listing results in the output.html file

It takes my eyes a moment to figure out where individual links end and the next links begin.  It would also be helpful to be able to say which link one is referring to using a number, such as "link number three".
Suggest numbering each of the the Source File Listing results as shown in the image I've added to this Issue. Doesn't need to be red font or super big font. 😄

Thanks for the report. I can appreciate that this is a bit unclear. I think we can add numbering here without much trouble.


[ADP-1963] factor out ApiDecodedTransaction type

I have

[x] factor out ApiDecodedTransaction from the Api.Types module
[x] created ApiT, ApiAsArray, ExtendedObject, Options as API  libraries
[x] factor out Certificate and Key API types with relatives closures
[x] factor out Primitive for all ApiT instances that just wrap types in the Primitive hierarchy
[x] moved out a couple of coin functions to the Coin primitive

Comments

This work is preliminary to ApiDecodedTransaction reuse in the ADP work, but independent from CBOR PRs
Issue Number
ADP-1963

CI failed:
To rerun use: --match "/Cardano.Wallet.Api.Server.Tls/TLS Client Authentication/Deny client with wrong certificate if TLS is enabled/"

Randomized with seed<PHONE_NUMBER>

bors try

bors r+

bors r+


Extract host profiling data using csvexport and capture tool from tracy

ZoneText and ZoneName dynamic data is not reported by csv export. Only static ZoneScoped or ZoneScopedN data are presented on the name column.


Make JointAccessor a variant on its own and a Derived Joint

JointAccessor now inherits both from JointBase and JointVariant. It is thus now a variant respecting the Base interface, hiding the calls to visitor.
Added some copy operator in order to allow to build a vector of JointAccessor.
Related to #131

Coverage remained the same at 83.56% when pulling af5b7c5422f80a9380cb692ba201503be68f14cc on fvalenza:devel into 6a26544561e1b46fe6e8a895e782832d696c5de7 on stack-of-tasks:devel.


I should be able to define a latency in ms (milliseconds) before the search function is executed.

It would be helpful if i can add a latency in milliseconds before the search is executed. Currently i want to avoid searching for every key pressed.

Hi Asad,
if you want to debounce the search you are able to use the ng-model-options. You need to use the master and you can check how to do it here: https://github.com/guylabs/ion-autocomplete#debouncing
With this you are able to exactly what you want.
Thanks and regards,
Guy


Infinite loop in files_to_save

What happened?
I think this is an infinite loop.
What did you expect to happen?
save a shapefile
Minimal Complete Verifiable Example
import tempfile
from pathlib import Path
import pywbt
bbox = (-2.66969, 53.22417, -1.86813, 53.75173)
temp_dir = tempfile.TemporaryDirectory()
src_dir = Path(temp_dir.name)
save_dir = src_dir
pywbt.dem_utils.get_nasadem(bbox, src_dir / "dem.tif", to_utm=True)
wbt_args = {
    "BreachDepressions": ["-i=dem.tif", "--fill_pits", "-o=dem_corr.tif"],
    "D8Pointer": ["-i=dem_corr.tif", "-o=fdir.tif"],
    "D8FlowAccumulation": ["-i=fdir.tif", "--pntr", "-o=d8accum.tif"],
    "ExtractStreams": ["--flow_accum=d8accum.tif", "--threshold=600.0", "-o=streams.tif"],
    "RasterToVectorLines" : ["-i=streams.tif", "-o=streams.shp"],
    "IsoBasins": ["-i=dem_corr.tif", "--size=40000", "-o=basins.tif", "--connections=True"],
}
pywbt.whitebox_tools(src_dir, wbt_args, save_dir=save_dir,verbose=True, files_to_save=["streams.shp"])

MVCE confirmation

[X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue.
[X] Complete example — the example is self-contained, including all data and the text of any traceback.
[X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output
No response
Anything else we need to know?
No response
Environment

pywbt==0.2.6

Speedy - thanks so much.
Thanks for the heads up RE cluster, I will be rerunning some form of my experiments at some point in the next few months (ICL HPC is still not at full operation if you can believe it, absolute disgrace!). I'll try it without my hack workarounds next time and see what happens.

Thanks, awesome. I hope the HPC gets back online soon, it must have been a nasty maintenance!


Wrong answer

Hi. These tests return wrong values
var isJSON = require('is-json');
console.log(isJSON('["el1","el2"]')); // false instead true

JSON must be an object at its root, not an array. That said I've found a similar issue. '{"fush": "carp", "row": ["1","2",]}' should fail due to the training coma but passes. not really a huge issue for me as it gets caught by another function in the curry process but still a false positive. Probably because this is a style used for configurations that is ignoring rules of the JSON spec.


Cached mDNS record not updated

I'm not sure if this is a Fermentrack or Avahi issue but it seems that my temperature controller is correctly reporting its new IP address but fermentrack is still using the old one.
The results of debugging the connection show it is still trying to use the cached IP rather than the new one:

Test
Parameter
Pass/Fail
Output

Device Status Test
active
Passed
Device active & managed by Circus

DNS Lookup Test
esp14510865.local
Passed
<IP_ADDRESS>

Connection Test
esp14510865.local
Passed
Connected

Controller Response Test
esp14510865.local
Passed
b'N:{"v":"0.2.4","n":0,"c":"00000000","s":0,"y":0,"b":"e","l":"3"}'

Cached IP Test
<IP_ADDRESS>
Passed
Available

Connection Test
<IP_ADDRESS>
Failed
Unable to connect

I was able to manually update the cached IP via the Django admin panel but this doesn't seem intuitive.  It seems that when DNS Lookup returns a new IP that should replace the cached address.

You shouldn't need to update the cached IP (though it should update automatically). I'll take a look to see what is going on.
Were you having issues actually reaching/using your controller?

Yes the connection failed, admittedly I was editing my network and changing the subnet that these devices were on so I expected a little bit of resetting but ideally mDNS should be able to cope with changes like this.
Interestingly, after manually updating the cached address in Django, the controller came online as expected and then a short time later seemed to reset its EEPROM and lost all sensor and pin allocations.  Haven't tried troubleshooting what caused this but was surprising.  I've managed to replicate this entire behaviour twice when changing subnet and updating the cached IP.

I think that this is fixed with the BrewPi script refactoring that took place alongside #724 . If not, then it’s a networking issue up the stack which makes it a bit more difficult to fix. Going to tag this against the PR for now — if it reoccurs after that PR is merged, we can revisit.