Can you switch the codes from ESP-IDF to ESP32 libraries in Arduino IDE?

Hi There,
Thank you so much for your codes for ESP32 MCU in ESP-IDF, I have lots of ESP camera boards and a Hologram Nova that has Sara-R410 and I want to connect ESP32 to it.
But the problem is all of my previous application codings are in Arduino IDE as most ESP32 codes are written in Arduino IDE!
So would you please modify your code so that we be able to use ubxlib library in Arduino IDE platform instead of IDF, like what Sparkfun people did?
Thank you so much!

Hi there, and thanks for you interest.  We've never looked at the ESP32 Arduino IDE, it might not be an easy thing to do.  I'll have a look at it tomorrow and comment again here.

Thanks, Rob for the quick response, if you look at something like Sparkfun LTE libs that would be great too;
https://github.com/sparkfun/SparkFun_LTE_Shield_Arduino_Library
https://learn.sparkfun.com/tutorials/lte-cat-m1nb-iot-shield-hookup-guide/all
Details:
My project has some image classification with ESP32, take pictures, and send it with LTE (I am familiar with Python not much C++, but working on it!), I was able to send images with Pycom Gpy recently, using micropython!
Gpy is ESP32 plus Sequans Monarch, but they do not have good library support as you do, so that is why we switched to SaraR410M-2B and we wanna integrate it with our ESP custom-designed board. So I recently bought some Sparkfun level-shifters to connect ESP32 3.3 volts to Sara modem 1.8 V Uart pins, similar to Sparkfun Arduino shield Eagle design, but I need codes to send a message(eventually image) from ESP to our website using Sara cellular modem, unfortunately, ESP-IDF is not that optimal, however, I am looking at some easier ways like using Visual studio IDE for IDF integration now, but if you could migrate to Arduino Libs, that would be gold!
Thanks,
MoZen

Hi again.  I started writing a comment about how I didn't understand the relationship between Arduino and ESP-IDF but then I found this article which suggests that ESP-IDF is already there, inside the pre-built libraries included with the Arduino IDE.  In other words, you should be able to take the ubxlib code, the existing esp-idf porting layer for ubxlib and simply persuade the Arduino IDE to build and link them against the pre-built ESP-IDF libraries it already has.
Then you would have the ubxlib APIs as well as the Arduino APIs, though of course you should probably NOT use the Arduino serial or GPIO APIs as ubxlib will be using that HW via the ESP-IDF APIs.
Have you tried doing anything like this?

Hi Rob,
Thank you, I looked at my Arduino lib folder, I found IDF-related files, so the integration sounds cool!
Yes, I assume you mean ESP is using UART0 with Arduino IDE and ubxlib uses UART0 with Sara modem too which might cause problems.
I used Sparkfun Sara-r4 shield eagle files and added your LTE modem schematic to our ESP integration board in Eagle but I am worried on software side; It is a little vague how to use ESP-IDF ubxlib APIs for me (I worked with ESP-IDF previously, I need to install it again), For example, which ESP32 pins to connect to Sara-R4? I assume Uart0 to UART pins of Sara-r4 using a level shifter... I have components, but honestly, I could not understand how your cellular example work, what should we set when uploading the code, where to add APN? what your example will print in the terminal of ESP-IDF?
My goal (and I assume every developer's first goal) is to send a "Hello World" message using Sara LTE with ESP-IDF, how can I do that with ubxlib Github examples with ESP32? Then I will modify that to send my images...
Some LTE messaging examples;
GPY messaging example: https://docs.pycom.io/tutorials/networks/lte/
Sparkfun: https://learn.sparkfun.com/tutorials/lte-cat-m1nb-iot-shield-hookup-guide/all#example-2-send-a-hologram-message
Hologram: https://www.hologram.io/references/python-sdk
I know you guys are busy but if one of your engineers could add an LTE messaging Github example or record a video on how to use ubxlib/GPIO APIs with ESP-IDF on your ublox1 YouTube channel that would be great.
Thanks,
MoZen

Having checked other priorities we will be able to make time to have a poke at this in the next few weeks.  I can't promise that it will work, there might be impossibilities or too-hard-for-us-to-solve things hiding within it but I'm hoping this will be just another set of build metadata, no code required.  If that is true it will indeed be a valuable addition.
I will report back.

Sounds good, Rob, really appreciate it!  Also, adding more info on how to use the current cellular API examples will help too, I will dig more on cellular documentation too!

On the issue of documenting how to use the cellular API examples, can you provide some hints as to what needs to be clarified? Since we know how to use it and have made an attempt to explain we need your help to figure out where we can improve.

Sure, I think first I need to install IDF and run it asap to get a better user experience, However,  today I checked the runner folder, it gave more clues on how it might work;
https://github.com/u-blox/ubxlib/tree/master/port/platform/esp-idf/mcu/esp32/runner
It looks like you made some "general" examples here; https://github.com/u-blox/ubxlib/tree/master/example
Where users can access them for all boards, previously I was looking for a cellular example in your ESP-IDF folder!
Also, the hardware part is not clear, if there was a schematic picture (fritzing, or eagle) that would help a lot;
something like Sparkfun schematics; https://www.sparkfun.com/products/14997
My question is which esp32 and sarar410-2B UART ports are connected, or how do they do Uart handshake.
Thanks

Hi again.  We have implemented Arduino support and have pushed a preview branch here, see the port/platform/arduino directory for more information.  This is for you to have a look at, maybe try out, see if it is what you are after.  Note that it is (a) still subject to change (it has not been reviewed internally yet) and (b) this preview branch will be DELETED when the feature is merged into master.
On the documentation, what would usually happen concerning hardware is that you would consult the u-blox integration manual for your chosen module and go from there. I will see if we can provide some form of "HW hints and tips" README.md in ubxlib which might point you in the right direction.

Hi Rob!
Thank you so much for the great explanation. I was able to copy the files using your Python code in the Arduino Library folder and build the ubxlib src folder, then include and compile a sketch in Arduino IDE successfully, more details after my question. Now I want to write the sketch code there...
My question:

I am not sure where to find these API functions: xxxInit() ,xxxAdd(), xxxConnect().

Assuming my first goal is to send an SMS like this code from Sparkfun (my end goal is to send the camera board picture);
https://github.com/sparkfun/SparkFun_LTE_Shield_Arduino_Library/blob/master/examples/01_SMS_Send/01_SMS_Send.ino
In the u_blox repo, which example should I look at?

There are two cell resources in ubxlib;
Cell API:https://github.com/u-blox/ubxlib/tree/preview_feature_arduino/cell
And
https://github.com/u-blox/ubxlib/tree/preview_feature_arduino/example/cell/lte_cfg
My Updates;
I am connecting 4 UART wires; from ESPCAM RX, TX, VCC, and Ground, to Sparkfun Arduino Shield I received yesterday; https://learn.sparkfun.com/tutorials/lte-cat-m1nb-iot-shield-hookup-guide/all
Things I need to do;

configuring the pins for ESPCAM in u_cfg_app_platform_specific.h

Need to install ESP-IDF and look at how to integrate it with Arduino from the medium article you shared again

Then, if it works, we can integrate u-blox SaraR410M-02B-01 to our boards using Eagle.

Thanks,
MoZen

That's great progress, thanks for looking at the preview branch.  If you were able to include and compile/link the ubxlib Arduino library for ESP32 then you must already have the ESP-IDF stuff inside your Arduino environment - maybe ESP-IDF comes with Arduino by default anyway? I only followed the instructions on the Espressif site because that seemed the obvious route, since I don't really know very much about Arduino and I do know about Espsressif.
On the examples, the cell examples are just a few small cellular examples, we try to direct you away from those towards the general examples which use the common APIs.  The common APIs will work across our cellular and Wifi/BLE modules and are intended to be simpler to use so it is much better to start there, only go down to the cellular API if you need something slightly special.
So, if you want to open a socket to a server, look at the sockets examples, if you want to use MQTT look at the mqtt_client examples, if you want to do location stuff look at the location examples, etc.  These examples each include all of the xxxInit() and xxxAdd() functions you need so you should be able to understand what's going on from there.
You mention SMS above: is that a requirement?  I imagine you aren't going to send a picture over SMS?  If it is a requirement then we will need to add SMS support on cellular (not something anyone has asked us for yet).  Not a huge issue, it is on the TODO list, just needs to be moved up.

Aloha and Thanks, Rob, I will check the TCP example, that's what I want.
Oh, I thought you mean installing esp-idf independently, I did that previously not sure if I deleted it...
But yes, I have esp stuff installed inside Arduino following the espressif link long ago (esp support inside Arduino you shared).
My goal is to send ESPCAM pictures using sockets, so I was thinking at SMS as a starting point that I know hardware works!, but thinking about it again... more relevant thing I see in Gpy or OpenMV codes as the first step is getting a website IP address using TCP (like Google!).
Check GPu example at bottom of this page;
https://docs.pycom.io/firmwareapi/micropython/usocket/
To send a picture I used this code first to know cellular module, antenna, hologram Simcard,... works(Sequans module inside Gpy);
socket.getaddrinfo()
So putting get address info from TCP work in Arduino(which is essentially C or C++) is a great TODO thing I think for many users, including me! Thanks!
So if you write a code that gets an IP address using TCP in Arduino using the ubxlib lib we have, that is great!
Arduino coding is mostly the same. I will check the TCP example you shared too. I have direct access to WiFi from ESP, we do not use GPS and BLuthotth now, first priority is making sarar4 work using my Sparkfun sarar4 breakout board using ESPCAM(wanted to share a picture of my espcam and sparkfun on my table now but probably sending picture is not supported in Github!
Thanks,
Mohsen

Sounds like you're having fun!  For TCP sockets (assuming ignoring TLS security for now) this is the example you want.  The entry point, I suppose the start of setup() in our terms, is this line, then you have to fit the following calls however you would like into the Arduino setup()/loop() world.  I'd guess that up to this line you'd probably put in setup(), then you have to decide whether you want bringing the network up (i.e. registering with the cellular network and gaining an IP address) to be part of 'setup()' or you might prefer to put that whole section (i.e. all of lines 212 to 281) in 'loop()', it entirely depends on how you would like your application to behave; does it connect to the network once, in setup() and stay on forever or does it connect only while it is in loop() and disconnect when it leaves loop()?  Only you can decide :-).
I've never used socket.getaddrinfo() in Python, and you need to remember that, despite C being there, hiding, underneath Python, Python is almost the exact opposite of C in almost every way, but it looks like all it effectively does is return the IP address of the device, which you could get, in the context of that example, by calling uCellNetGetIpAddressStr() with the network handle after you've done uNetworkUp(), so something like:
uCellNetGetIpAddressStr(networkHandle, buffer);

...just re-using the buffer that is already there as it is big enough (it just happens to be U_CELL_NET_IP_ADDRESS_SIZE characters).  The IP address is just a string in the buffer, so if you like you could print it with:
uPortLog("The IP address of this device is %s\n", buffer);

The thing to remember with the ubxlib C code is that everything is done using handles and all of the sockets stuff is pure BSD sockets, there should be no surprises to a BSD sockets user (which is pretty much all sockets users I think).
Looking forward to seeing a picture of our ESPCAM :-).

Good morning Rob!
Thanks for your patience, I know it's hard dealing with non-computer engineers, it took me a while as an electronics engineer to learn Python from Coursera (I knew C to some extent). Anyways, I tried copying the parts you mentioned in Arduino and call the functions, I got this error;  exit status 1, 'networkHandle' was not declared in this scope
I was not sure what to include and what to ignore at the beginning of the sketch based on the codes...
I think if you write a sample .ino file in your arduino sketch example folder, that is great.
Here are two examples I was talking about, tested in OpenMV which I found useful for wireless projects (fetching IP address of a website and photo upload);
https://github.com/openmv/openmv/blob/master/scripts/examples/OpenMV/14-WiFi-Shield/http_client.py
https://github.com/openmv/openmv/blob/master/scripts/examples/OpenMV/14-WiFi-Shield/http_post.py
Ok, since you said you know IDF more, I will start with it to make sure the sarar4 hardware is working, then I will go back to Arduino for the photo upload project. I have the ESPCAM photo upload code in Arduino already.
Ok I installed ESP-IDF again, and I went to configure these files (pins for ESPCAM);
https://github.com/u-blox/ubxlib/tree/master/port/platform/esp-idf/mcu/esp32/cfg
I think you mentioned,  u_cfg_app_platform_specific.h should be modified more based on ESP format which in my case is ESPCAM(ESP32S), there were many RX and TX to be set! so I panicked a bit, btw what is the short-range module?
It looks like I should change most of them to -1 as they are not connected to sara r4, I am trying to mimic Sparkfun shield wiring, they just connect rx and tx, and I will just connect rx,tx,vcc,gnd from espcam to sparkfun.
4 pins in ESPCAM UART are shown here; https://randomnerdtutorials.com/program-upload-code-esp32-cam/
In Sparkfun SaraR4 shield, they mention they connect all uart handshake signals to ground as recommended for a UART "with NO FLOW CONTROL" mentioned at the lower part of page two of their schematic; https://cdn.sparkfun.com/assets/3/9/d/5/f/lte_cat_m1_shield_schematic.pdf
I assume I should do the same.
Hopefully, when I am able to run Sarar4 with ESPCAM, I will definitely send its success picture, haha!
Thanks,
MoZen

Understood, in that case I will see if I can write an Arduino example, making some assumptions about how the application should behave, which does the same send/receive to a TCP echo server as the sockets example.
On the setting of pins etc., yes, you either edit the u_cfg_app_platform_specific.h under the ESP-IDF folder ('cos that's the chipset you're building for) or you override the values, or just set them, in the application itself, it is entirely up to you.  All of the pins that might be required are present in the file; you might be using one of our Wi-Fi/BLE modules or GNSS modules etc., we can't know.  But if you're only using a cellular module you only need to set the things with "CELL" in the name and if you're not using flow control you should set the CTS/RTS ones to -1 since they are not connected.
I won't close this issue until I've seen the picture :-).

Hi MoZen: for neatness, could you possibly open a new issues, maybe entitled "config settings" and put your last two points above in that?  Easier to manage then, and people will be able to find the answers later more easily.

Hi Rob, Sure!

Hi again: I have pushed an updated version of the Arduino preview branch, called preview_feature_arduino_updated (the previous preview branch is still there, unchanged, in case you need to refer to it for some reason).
Please delete any stuff that was created last time by running the Python script, read the revised README.md (some of the script names and certainly the script behaviours have changed).  The most important thing about this update is that it includes an example sketch, an Arduino version of the sockets example.
This branch has also passed testing on our test system and so is likely quite close to the version I will merge.  When I have merged Arduino support into master I will delete both of these preview branches.

Hi Rob, it is awesome to test the socket example in Arduino, I deleted my previous libraries to try the preview version.
However, I get the following error when trying to run u_arduino.py;
No such file or directory: 'source_lib.txt'
I guess it is because in the code you have added;
UBXLIB_LIB_POSTFIX = "_lib"
Source file listing file
SOURCE_FILES = "source" + UBXLIB_LIB_POSTFIX + ".txt"
But there is no "source_lib.txt" file in your Arduino folder...

Comparing to the previous Arduino preview folder; https://github.com/u-blox/ubxlib/tree/preview_feature_arduino/port/platform/arduino
It looks like you probably forgot to add these files in the new Arduino preview folder; source_lib.txt and include_lib.txt, so I will just copy those from the previous preview and hopefully it will work!

Sorry, my fault, I was testing with the automated test system which always specifies the names of those text files on the command-line to u_arduino.py rather than using the default names; it's just the name in u_arduino.py that is wrong, UBXLIB_LIB_POSTFIX should be "" not _"lib".  Don't use the old files, the contents are different and it probably won't compile.
I have update the branch so that UBXLIB_LIB_POSTFIX is now correct.

No problem my friend,
Yeah, it was not compiling, but it successfully copied the ublox in the Arduino Library using old files...
Anyways, I will delete them and use your updated version now, thanks for the quick reply.
Btw, I am curious how you test the code, what hardware do you use? Probably Nina...

We test ubxlib in general on the things listed here, where the Arduino ones are instances 11.1 and 22, so an ESP32-DevkitC and a NINA-W10.

Thanks. It copied the library with updated code, it worked.
Unfortunately I got error running empty sketch(just #include <ubxlib.h>) in Arduino IDE;
Library ubxlib has been declared precompiled:
Precompiled library in "C:\Users\AI_Solutions\Documents\Arduino\libraries\ubxlib\src\esp32" not found
C:\Users\AI_Solutions\Documents\Arduino\libraries\ubxlib\src\u_port_uart.c: In function 'uPortUartOpen':
C:\Users\AI_Solutions\Documents\Arduino\libraries\ubxlib\src\u_port_uart.c:275:19: error: 'uart_config_t {aka struct }' has no member named 'source_clk'
config.source_clk = UART_SCLK_REF_TICK;
^
C:\Users\AI_Solutions\Documents\Arduino\libraries\ubxlib\src\u_port_uart.c:275:33: error: 'UART_SCLK_REF_TICK' undeclared (first use in this function)
config.source_clk = UART_SCLK_REF_TICK;
^
C:\Users\AI_Solutions\Documents\Arduino\libraries\ubxlib\src\u_port_uart.c:275:33: note: each undeclared identifier is reported only once for each function it appears in
exit status 1
Error compiling for board Adafruit ESP32 Feather.
Thanks,
MoZen

Library ubxlib has been declared precompiled:
Precompiled library in "C:\Users\myuser\Documents\Arduino\libraries\ubxlib\src\esp32" not found
C:\Users\myuser\Documents\Arduino\libraries\ubxlib\src\u_port_uart.c: In function 'uPortUartOpen':
C:\Users\myuser\Documents\Arduino\libraries\ubxlib\src\u_port_uart.c:275:19: error: 'uart_config_t {aka struct }' has no member named 'source_clk'
config.source_clk = UART_SCLK_REF_TICK;
^
C:\Users\myuser\Documents\Arduino\libraries\ubxlib\src\u_port_uart.c:275:33: error: 'UART_SCLK_REF_TICK' undeclared (first use in this function)
config.source_clk = UART_SCLK_REF_TICK;
^
C:\Users\myuser\Documents\Arduino\libraries\ubxlib\src\u_port_uart.c:275:33: note: each undeclared identifier is reported only once for each function it appears in
exit status 1
Error compiling for board Adafruit ESP32 Feather.

Thanks.
I get the following error compiling empty sketch example in Arduino IDE;
Library ubxlib has been declared precompiled:
Precompiled library in "C:\Users\myuser\Documents\Arduino\libraries\ubxlib\src\esp32" not found
C:\Users\myuser\Documents\Arduino\libraries\ubxlib\src\u_port_uart.c: In function 'uPortUartOpen':
C:\Users\myuser\Documents\Arduino\libraries\ubxlib\src\u_port_uart.c:275:19: error: 'uart_config_t {aka struct }' has no member named 'source_clk'
config.source_clk = UART_SCLK_REF_TICK;
^
C:\Users\myuser\Documents\Arduino\libraries\ubxlib\src\u_port_uart.c:275:33: error: 'UART_SCLK_REF_TICK' undeclared (first use in this function)
config.source_clk = UART_SCLK_REF_TICK;
^
C:\Users\myuser\Documents\Arduino\libraries\ubxlib\src\u_port_uart.c:275:33: note: each undeclared identifier is reported only once for each function it appears in
exit status 1
Error compiling for board Adafruit ESP32 Feather.

That's strange: it looks like the version of the underlying ESP-IDF code is different in your case, their definition of uart_config_t has changed.  Do you know what ESP-IDF version you have?

idf4.2_py3.8_env but I can install a version that worked for you...

All I did was follow Espressif's instructions; it compiled   for you yesterday and none of that code has changed.  Don't understand what's different.

Assuming you are Windows, can you look at the file:
C:\Users\<your_username>\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.0\tools\sdk\esp32\include\hal\include\hal\uart_types.h
...and tell me what the uart_config_t structure looks like?  I have:
typedef struct {
    int baud_rate;                      /*!< UART baud rate*/
    uart_word_length_t data_bits;       /*!< UART byte size*/
    uart_parity_t parity;               /*!< UART parity mode*/
    uart_stop_bits_t stop_bits;         /*!< UART stop bits*/
    uart_hw_flowcontrol_t flow_ctrl;    /*!< UART HW flow control mode (cts/rts)*/
    uint8_t rx_flow_ctrl_thresh;        /*!< UART HW RTS threshold*/
    union {
        uart_sclk_t source_clk;         /*!< UART source clock selection */
        bool use_ref_tick  __attribute__((deprecated)); /*!< Deprecated method to select ref tick clock source, set source_clk field instead */
    };
} uart_config_t;

...i.e. there is a source_clk field in the version I have under Arduino.

Ok, it looks like my esp32 is 1.0.6! instead of 2.0.0;
C:\Users\my_user\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\tools\sdk
From there I do not see hal folder...
I guess I need to update it...

Oh!  That's probably a good idea :-).

Ya thanks, it's updating now in Arduino IDE following the Espressif link, it was 1.0.6!

Cool, it compiled! Thanks!
I have questions about gpio pins, will ask in the other thread now.
God Bless you!

I get this warning too;
Library ubxlib has been declared precompiled:
Precompiled library in "C:\Users\myuser\Documents\Arduino\libraries\ubxlib\src\esp32" not found

That is intentional: the library.properties file is set up so that, if you wish, you may copy the .a file for the library once you have compiled it into the libraries\ubxlib\src\esp32 and it will use that instead of re-compiling the source code of the library each time.  We use this internally so that we don't re-compile the library for each test.  However, note that if you do this and make a change in the library source code you MUST delete the .a file from libraries\ubxlib\src\esp32 as your changed code is not going to be compiled/used otherwise (since it's using the compiled code from the .a file).

The board setup:

The stuff that is present on the preview_arduino and preview_arduino_updated branches has now been pushed to master in commit c1dfbf01ba45eaa10550b676ca9dbdb6d3ebb1e8, so I will delete those branches.
How are you doing with your development?

Aloha and Thanks Rob!,
Sorry, I was busy with class finals, but recently we soldered our Sara-R4 module, I was not able to turn the module on yet!, 2 updates;

I get this message in Arduino IDE ;
Done compiling,
Library ubxlib has been declared precompiled:
Precompiled library in "C:\Users\myusername\Documents\Arduino\libraries\ubxlib\src\esp32" not found.

To turn the Sara module ON, we use a P-channel MOSFET at pin VCC of SARA module, so in the other thread you said and I quote; "U_CFG_APP_PIN_CELL_ENABLE_POWER: if you have a relay or some such gating power to the cellular module then you need to set this to the MCU pin which controls that relay. If you don't you should set this to -1."

So using Adafruit ESP32  feather board, I have set it to
define U_CFG_APP_PIN_CELL_ENABLE_POWER      32
#endif
This means I have connected pin 32 of it to the gate of P-MOS,
and 3.3 volts voltage supply of feather board is connected to its sources,
so the idea is the Drain of this P-MOS that is connected to VCC pin of Sara-R4, will have 3.3 volts only when the P-MOS Gate is pulled down(opposite of N-MOS as you know that works when Gate is Pulled Up), so our approach is similar to having a Relay you mentioned to provide power to the Sara module but NOT of it, so my question is how can I NOT the U_CFG_APP_PIN_CELL_ENABLE_POWER in the program, i.e. when it is zero volts? then the Sara module will have power at pin VCC and vice versa.
To test it(without having NOT, and running socket example), I put an LED at pin 32 of feather board, but it does not turn ON currently when I run the socket.ino code, so I will try ESP IDF today too...
Happy Holidays!
MoZen

Hi there, hope finals went OK.

Precompiled library in "C:\Users\myusername\Documents\Arduino\libraries\ubxlib\src\esp32" not found.

That's fine; it's not there yet, you're compiling it for the first time.  Setting it as pre-compiled just means it won't keep recompiling the library.
On inverting the state of the U_CFG_APP_PIN_CELL_ENABLE_POWER pin you'll have to modify the code to do that for you.  Search for pinEnablePower in u_cell.c and u_cell_pwr.c and you should find the bits you'll need to change.

Yeah, One of the finals was promising, not sure about the other yet, haha.
Cool, I was not sure where to modify the code,  thanks for the big help!
By the way, we developed this module for ESPCAM and OPENMV H7, I guess with ESP32 it should be fine, what do you think about making ublox library work for OPENMV boards too?

OpenMV is quite a beast: looks as though they intend you to run MicroPython not C.

Hi Rob,

In this Eagle picture of our prototype module, Sara's VCC is energized as I explained in the thread; pin 32 of Adafruit ESP Feather is connected to PWRNOT in the picture and 3 volts pin of ESP feather is connected to 3.3V-MAIN of the picture.
define U_CFG_APP_PIN_CELL_ENABLE_POWER 32
it looks like pinEnablePower is repeated 19 times in ucellpower and 14 in ucell
Sorry I am not proficient in c++, today I learned
pInstance->pinEnablePower = pinEnablePower;
is kinda similar to
pInstance.pinEnablePower = pinEnablePower;
in the cplusplus pointer world.
Next thing is to decipher the following!
uPortGpioSet(pInstance->pinEnablePower, 0);
It looks like it is a function;
uPortGpioSet with two variables;
pInstance.pinEnablePower and 0 but I do not know what it means.
Also, I was hoping to toggle zeros to ones and vice versa to achieve NOT condition I am after, But I am not sure why we have equal sign here;
if (pinEnablePower >= 0) { ....
Or
if ((platformError == 0) && (pinEnablePower >= 0)) .....
I mean ESP32 pin will have two conditions either it is 3 volts or zero volts, so probably we want the condition to be >0 but here we have >=0
Also I am not sure what platformError is here;
if (platformError == 0) {
enablePowerAtStart = uPortGpioGet(pinEnablePower);
I really appreciate your help.
Thanks

To go around the mentioned software problem, I connected PWRNOT pin of my Eagle Schematic image to GND so this provides 3.3 Volts to Sara module, and I measured 1.8 Volts on Sara 1.8V pin which is a good sign ...
But it still says Bringing up the network...
The network is not available!
it's kinda weird; it does not wait for a few seconds to tell it is not available, and as I push the reset button on ESP feather Arduino IDE Serial monitor says the network is not available.

Aloha Rob,
I just connected the P-MOS input to the ground of the ESP feather(act like a relay), good news is I have 1.8 volts from Sara r410m-02B at 1.8V pin so ESP feather is supplying it properly.
Next as I am using hologram simcard, so in Arduino code I set it as 'hologram' in Honolulu!;
static const uNetworkConfigurationCell_t configCell = {U_NETWORK_TYPE_CELL,
U_CFG_TEST_CELL_MODULE_TYPE,
NULL, /* SIM pin, must be NULL /
"hologram", / APN: NULL to accept default.  If using a Thingstream SIM enter "tsiot" here /
240, / Connection timeout in seconds */
U_CFG_APP_CELL_UART,
...
I receive this message in the Arduino terminal;
rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1284
load:0x40078000,len:12836
load:0x40080400,len:3032
entry 0x400805e4
Added network with handle -5.
Bringing up the network...
Network is not available!
0 Tests 0 Failures 0 Ignored
So in u-cfg-platform specific I only have defined and connected 3 pins;
define U_CFG_APP_PIN_CELL_PWR_ON            14
define U_CFG_APP_PIN_CELL_TXD              17
define U_CFG_APP_PIN_CELL_RXD              16
Looks like the power-on pin is working as it turns on the module as I read 1.8V on the multimeter
and I double-checked TX and RX pins, are you suspicious of anything else?
Thanks,
MoZen

Hi,
Is there any way you can check to see if the Arduino is able to send AT commands through the UART and you get "OK" in response?
Until you confirm the AT interface is working, I'd ignore the "Network is not available!" message. This could simply mean there is no communication to the SARA module.
Phil.

Hi @Paryavi: for this project you will really need to understand embedded C in some considerable depth.  I would seriously suggest taking a course in the subject, maybe alongside doing the project, as you will need it when you come to write your application.  Making use of ubxlib requires good knowledge of embedded C.
But to your point, your general approach when debugging must be:

look at each piece of debug information you have,
trace that information back to the code to determine what has happened.

The debug information has a lot of clues in it, you just need to consider each one with care and then follow it through to the code.  It is easy to gloss over debug messages, not read them carefully, not look at the sequence, I make that mistake myself all the time, I think it is why "rubber ducking" is so effective, as it makes you think about what your, ummm, thinking.  But quite a lot of the time you'll find that the information is there in front of you, it just requires logical following-through.
The first debug print after the ESP32 boot prints above is:
Added network with handle -5.
You can search the code for the text Added network with handle but there's no need as it is front-and-centre in the top-level sockets.ino file:
https://github.com/u-blox/ubxlib/blob/master/example/sockets/sockets.ino#L144
So clearly uNetworkAdd() is returning -5.  A function starting with uNetwork is going to be in the common network API, so you can find its definition, with all the documentation about it, in common/network/api; it is here. Even if you didn't know that, a text editor recursively searching for uNetworkAdd in *.h from the root of the ubxlib directory will find it very quickly.
However it is simpler than that: as you will see from the definition of pretty much all ubxlib functions, a negative number indicates an error.  The generic error codes are in common/error/api/u_error_common.h and you can see that -5 is U_ERROR_COMMON_INVALID_PARAMETER.
There are only two parameters to uNetworkAdd(): U_NETWORK_TYPE_CELL and the address of configCell.  You are obviously opening a cellular network type and so the most likely candidate for a problem is configCell, since it is the large complicated thing.
The structure type for configCell is named uNetworkConfigurationCell_t, so again it is part of the common network API in common/network/api and its definition, with all the documentation about it, can be found there (or you can simply search the .h files for uNetworkConfigurationCell_t and you'll find its header file soon enough).
Does your structure contain valid values according to those definitions for each field?  For instance, can you confirm that you have edited the .ino file to provided a value for U_CFG_TEST_CELL_MODULE_TYPE?  The instructions just above the structure in the .ino file tell you how to do this:
https://github.com/u-blox/ubxlib/blob/master/example/sockets/sockets.ino#L31
I think you are using SARA-R410M-02B, so the line would need to be:
#define U_CFG_TEST_CELL_MODULE_TYPE U_CELL_MODULE_TYPE_SARA_R410M_02B

Thanks, Rob, I knew it is about -5 error but thanks for helping me understand how to look at the API and its different components, I really liked the rubber ducking concept and it looks like Coursera has a nice embedded programming course that I should check out during the hollydays as it is my blindspot currently! I should learn about CMake, compiling details, ...;
https://www.coursera.org/learn/introduction-embedded-systems#syllabus
You were right, previously, long ago I did set sarar4-2b in the code, but this time I forgot about it, so setting the Sara module config properly, it starts the AT commands but ends with error -7;
https://github.com/u-blox/ubxlib/blob/master/common/error/api/u_error_common.h#L55
U_ERROR_COMMON_NOT_RESPONDING -7
Not sure where to find more about this error,I got suspicious of my RX, TX pins, and level shifting transistors voltages which look fine, also I looked at hologram cloud documentation, and I am not sure if I need to set the SIM pin in arduino code;
NULL, /* SIM pin, must be NULL */
In Hologram API, the have device Key which I did set with hologram Nova and Raspb pi ;
https://support.hologram.io/hc/en-us/articles/360035212714
But I guess it is not relevant here, I can text them to get my sim Pin if needed.
Hologram Help Centr for Sarar410 modules;
https://support.hologram.io/hc/en-us/articles/360038810214?input_string=not+sure+if+pin+is+required+for+turning+sara+module+on
Btw, thank you, we've had good progress so far and here is the serial response, not sure how to confront the common not responding error(-7)....
U_CELL_PWR: powering on.
AT
AT
AT
AT
AT
AT
AT
AT
AT
AT
Added network with handle -7.
Bringing up the network...
The network is not available!
Thank you so much,
P.S. Yes I am a Ph.D. student at UHM, Honolulu, Hawaii, there is an invasive insect, Coconut Rhino Beetle, attacking its Palms and I am hired as a graduate student to design Software and Hardware to delineate their infestation, we previously used WiFi to send our integrated camera board images to a server with ESP32 and OpenMV, so I did some Tensorflow CRB insect detection on OpenMV (using TFLite) and recently doing inference on Flask/Heroku using TensorFlow.
Traps are spread around Oahu, a few thousands, and we do not have WiFi on the road! So currently we soldered 2 Sara-R4 boards, this one turns on, and I am working on it, hopefully, this will be part of my Ph.D. thesis and if we detect CRB in traps quickly, the team can respond quickly!
Also, My advisor found another Sara-R4 with GPS capabilities and we will work with that design too but it is not available currently due to chip shortage!
Yes, I loved Hawaii, and happy you had a great time here. Feel free to visit again, I'd be here at least in the next two years, we can hang out!
Mahalo,
MoZen

Good stuff, so the issue is serial comms; you appear to be sending AT commands to the module but either they are not getting to the module or it is not responding.  A few things to check:

Remember that the module uses the DCE convention, i.e. TXD is the input pin at the module, it is where data to be transmitted goes, and RXD is the output pin at the module.  People often get this mixed up and think the TXD pin of the module is an output.
If you are not using flow control. is the RTS pin of the module grounded?  The module will come up with flow control on and so if RTS is not grounded it will never send anything back to you.

Ignore the SIM, that's irrelevant until we're talking to the module and it is very unlikely to be an issue anyway, just make sure no PIN is set.
Sounds like a very interesting project indeed, let's get it working!  Seems as though your supervisor has found SARA-R422M8S: that will work out of the box with ubxlib and give you various location establishment options.

Cool,
Yes in eagle design I had grounded RTS and DTR, pin 9 and 10 of Sara module are physically grounded under the ublox shield and I can't access those pins to test them, also I have set RTS and CTS to -1 in u-cfg-app-platform-specific, not sure if -1 is proper as it is grounded internally...
Today I checked the DCE convention and our logic shifting transistors are working, our setup looks like Figure 45 in "the Sara-r4 integration manual"; Figure 45-UART interface application circuit with a partial V.24 link instead of voltage translator we use level shifters, other difference is on the application processor side(ESP32) I did not connect RTS to CTS or DTR to DSR....
I think we are not using Flow control

Btw, the labels RX-Cell and TX-Cell are misleading in this image, they are connected to proper level shifters, so RX of ESP goes to RXD, and TX of it goes to TXD of Sara!

I think you need to monitor the dynamic behaviour of the PWR_ON, TXD and RXD lines, as close to the module as possible, during the initial power-on sequence, using something like a storage scope or a Salaea logic probe.  Need to check that PWR_ON is actually toggling, that it is the right way up, whether TXD is getting to the module and whether RXD is coming back.

Sure, I will check them out, can it be the level shifters I used instead of Texas Instrument unidirectional voltage translators suggested by the Sara datasheet(Fig 45)? I mean I am not sure our N-MOS transistors switching is fast enough (115200 bps?)

.

Another difference is our level shifter is bidirectional but Fig 45 required unidirectional logic shifting...
Here Adafruit people say;
While we designed it for use with I2C, this works as well for  TTL Serial,  slow <2MHz SPI, and any other digital interface both uni-directional and bidirectional.
https://www.adafruit.com/product/757
So probably 115200 baudrate probably might be fine! but as you said we should test it with a probe!

I use the Sparkfun bidirectional level-shifters routinely, they're fine at 115,200, just don't try to use bi-directional level-shifters in flow control lines, that doesn't work.
Your O6/BSS138 (the thing between the MCU's U_CFG_APP_PIN_CELL_PWR_ON pin and the module's  PWR_ON pin); is that inverting or not?  If it is inverting then you will need to pass the conditional compilation flag U_CELL_PWR_ON_PIN_INVERTED into the build to get everything the right way up again.
If it is NOT inverting then there still may be an issue: the default drive for the U_CFG_APP_PIN_CELL_PWR_ON pin is open drain, intended to pull the module's PWR_ON pin low; will open drain drive work with your O6/BSS138?  If not then it may be necessary to set the conditional compilation flag U_CELL_PWR_ON_PIN_DRIVE_MODE=U_PORT_GPIO_DRIVE_MODE_NORMAL for the build; this is set anyway if U_CELL_PWR_ON_PIN_INVERTED is set since there is plainly a driver transistor in the way.

FYI, assuming you are using arduino-cli, you would pass conditional compilation flags in with an additional command-line switch to arduino-cli compile as follows:
--build-property "compiler.cpp.extra_flags=STUFF_HERE"
...so maybe:
--build-property "compiler.cpp.extra_flags=\"-DU_CELL_PWR_ON_PIN_INVERTED\""
...or
--build-property "compiler.cpp.extra_flags=\"-DU_CELL_PWR_ON_PIN_DRIVE_MODE=U_PORT_GPIO_DRIVE_MODE_NORMAL\""

Actually, I used the Sparkfun level shifter Eagle files to implement RX and TX in my eagle files with Sara module, as seen in shared pictures, so as you see MAIN RX and TX comes from ESP Feather and go to Sara directly through RX-Cell and TX-Cell labels in the two pictures above. I think it is non-inverting, similar to Sparkfun level-shifter. so I will follow your Not inverting comment now...
I think the Power-ON is working because when I connect its pin and press boot on ESP feather I will have 1.8 volts, so looks like the Power_On pin is turning the module on, so I follow your not inverting comment now and see how it goes.

FYI, in commit bb05f55583e90105758c90febcebc293fa21005f I've added the facility for you to have an inverted enable-power pin.  You just need to pass the conditional compilation flag U_CELL_ENABLE_POWER_PIN_INVERTED into the build.

Hi Rob,
Thank you!
My Yesterday Updates:
As you know I was working with Arduino IDE, but yesterday I installed arduino cli, and it was not recognizing Adafruit ESP feather, but I got its --fqbn from list of boards in cli and I compiled the sockets.ino;
cd C:\Users\myUser\Documents\Arduino\libraries\ubxlib\examples\sockets
Then for compile;
arduino-cli upload -p COM5 --fqbn esp32:esp32:featheresp32 sockets
Then right after that for upload I used;
arduino-cli upload --build-property "compiler.cpp.extra_flags="-DU_CELL_PWR_ON_PIN_INVERTED"" --fqbn esp32:esp32:featheresp32 sockets
I was able to upload it to ESP feather but was getting error handle -5 again.
Also, I watched the Harvard CS50 C Youtube video to understand make file and compiling details(I took an embedded C from Udemy to be watched...)... I cant use the make command on windows yet, not sure I need it because Arduino cli already compiles?
So you changed these 3 C files in new commit, How can I download this commit(bb05f55), do you suggest I edit these parts in these 3 files in my current ublox folder? I guess there should be a git command to download a specific commit; currently, I say git commit https://github.com/u-blox/ubxlib.git when right-clicking my git Bash in Windows.
And lastly, previously as you know I was reading 1.8 V from micro with the multimeter, we know that our transistor inverts the logic at Power_On pin, but how come it was turning Sara module ON although it was inverted?
Thanks,
MoZen

I will try git clone the master to check if I see your new committed changes in those 3 files...

To get the new files, git pull is what you want.
On the compilation mechanism, use whatever it is your platform uses, so for ESP32 you'd use CMake, for Arduino you use Arduino's home grown machenaism.  arduino-cli is the most recent flavour of the Arduino thing and seems relatively usable. If you had a platform that used make then you'd use make but Arduino doesn't so you don't, if you see what I mean; no point in causing extra pain (which make certainly will on Windows).
Do make sure, though, that you do a clean build; above you say "I compiled the sockets.ino" and then "for compile", which makes it look as though you may have already compiled the files once before you add the compilation flag in the second step.
Putting it another way: you want to start of with no build products, so delete any build directory etc.  and then do one arduino-cli compilation of the whole lot, along the lines of:
arduino-cli compile --fqbn esp32:esp32:featheresp32 --clean -v --build-property compiler.c.extra_flags="-DU_CELL_PWR_ON_PIN_INVERTED" --build-property compiler.cpp.extra_flags="-DU_CELL_PWR_ON_PIN_INVERTED" sockets.ino

This has the -v verbose flag so that you can see exactly how each file is being compiled, has --clean for good measure and adds the compile flags both for .c and .cpp 'cos I'm not sure whether an .ino file is considered to be C or C++.
On the powering on and off thing, it might work from a button, everything depends on timing and which way up the edge is: mashing a button with a finger can do things (bounces, multiple tries, etc.) which no machine-driven pin would do.

To be quite sure that the compilation flag is being passed through to the compilation process, edit u_cell.h to add a #error as blow:
#ifndef U_CELL_ENABLE_POWER_PIN_INVERTED
# define U_CELL_ENABLE_POWER_PIN_ON_STATE 1
#else
# error U_CELL_ENABLE_POWER_PIN_INVERTED is most definitely defined!
# define U_CELL_ENABLE_POWER_PIN_ON_STATE 0
#endif

It the compilation processes fails with that #error then you know the compilation flag is being used.

above you say "I compiled the sockets.ino" and then "for compile"
Sorry I meant; I changed the directory to the sockets.ino and then for compile... I modified my comment. Thank You, I will implement the changes and see what happens!

Aloha Rob,
Ok, looks like we have some progress iwth AT commands.
First, I connected pin 32 of ESP feather to PWR NOT in my shared Eagle image(cell enable power) so pin 32 of ESP is now connected to Gate of the PMOS which previously I was connecting the Gate to ground to provide 3.3 volts constantly) and modified u-cfg-app-platform specific;
define U_CFG_APP_PIN_CELL_ENABLE_POWER      32
And like before I have;
define U_CFG_APP_PIN_CELL_PWR_ON            14
define U_CFG_APP_PIN_CELL_TXD              17
define U_CFG_APP_PIN_CELL_RXD              16
It is interesting that whenever the following AT commands are sent to the board, the Red LED is OFF, ReD LED in our board shows if 3V3 is provided for the board(when PWR NOT is Zero), and as you know previously I would physically connect the gate of P-MOS to ground to have 3V3 provided for Sara VCC pin and hence having RED LED ON constantly, now I connected it to pin 32 of ESP and red LED is OFF most of the time
In Arduino IDE serial monitor I have;
U_CELL: initialising with enable power pin 32 (0x20) (where 1 is on), PWR_ON pin 14 (0x0e) (and is toggled from 0 to 1) and VInt pin not connected.
AT
AT
AT
AT
AT
U_CELL_PWR: powering on.
[ff][ff][ff][ff][fb][ff][fb][ff][ff][df][fb][ff][fb][ff][fb][fb][ff][ff][fb][ff][ff][ff][ff][ff][ff][df][ff][fb][ff][ff][fb][ff][fe][ff][ff][ff][ff][ff][fb][ff][df][fb][df][7f][ff][ff][ff][fb][ff][fd][df][e7][ff][7f][de][ff][df][bf][ff][ff][fd][df][fd][bf][fd][dd][ff][f7][eb][db][fe][dd][fb][ff][dc][ea][ff][ba][7f][f3][99][f8][df]}[d6][ba][01]p[d8][b8][1f][00][97]R[a5]=[92]<'[00]$[00][00]Dd[92][88][ff][db]bA[84][09]H[02][00][84]HD[04][98][01][00]"[00]A"[00] [90]A [09][00][08][00][88][04][00][04][00][00][01][04][90][01][00][80][08][04][00] [00][00][01][04][01][00][00][00][00][08][04][08][04]@[00][00][05][00][00][10][00][00][00][00][00][00][00][08][00][00][00][00][00][00][00][00][00][00][08][00][00][04][00][00][00][00][00][00]@[00][00][00][00][00][00][00][00][00][01][01][00][00][00][01][00][00][00][08][00]@[08][00][00][00]@[00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][04][90][00][00][00]$[90][00][00][00][00][00][00][00][00]$[90][00][00][00]$[00][00][00][01][00][08][00]H[00][00][00][00][00][08][00][00][01][04][00]@[00][00][00][00][04][00][00][00] [00] [00][00][00][00][00][00][00][00][00][01][00][00][00][00][00][04][00][00][00][00][00][00][01][00][00][00][00][00][00][04][00][00][00][01][00][00][00][00][00][00][00][00] [00][a0][80][01][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00]@[00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][01][00][08][00][00][00][00][04][00][00][00][00][80][10][00][00][00][00][00][00][00][00][03][00][00][00][01][00][00][00][00][00]$[00][00][01][00]@[00][00][00][00][00][00][00][00][00][00][00][00][00][01][00] [00][00][00][00][00][01][00][00][00]@[00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][08][00][00][00][00][00][00][00][00][00][00][00][04][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00] [00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][80][02][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00]AT
AT
AT
AT
AT
AT
AT
AT
[ff][fb][ff]{[ff][fb][ff][f3]O[1f][83]@[00][07][02][00][00][00][00][00][00][01][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00][00]AT
AT
AT
AT
AT
AT commands do not stop in the serial monitor...
Also in order to have compile-upload consecutively in Arduino CLI, I just added -u for upload and it needs port number too; -u -p COM5
arduino-cli compile --fqbn esp32:esp32:featheresp32 --clean -v -u -p COM5 --build-property compiler.c.extra_flags="-DU_CELL_PWR_ON_PIN_INVERTED" --build-property compiler.cpp.extra_flags="-DU_CELL_PWR_ON_PIN_INVERTED" sockets.ino
But adding the error where you mentioned did not stop compiling in Arduino CLI. So I am not sure if the toggling flag is working or not.
Btw, I can't compile it from Arduino IDE, which is not that important;
In file included from C:\Users\myUser\Documents\Arduino\libraries\ubxlib\src\u_at_client.c:43:
C:\Users\myUser\Documents\Arduino\libraries\ubxlib\src\u_at_client.c: In function 'uAtClientInit':
C:\Users\myUser\Documents\Arduino\libraries\ubxlib\src\u_cfg_os_platform_specific.h:97:36: error: 'CONFIG_ESP32_PTHREAD_TASK_PRIO_DEFAULT' undeclared (first use in this function)
#define U_CFG_OS_APP_TASK_PRIORITY CONFIG_ESP32_PTHREAD_TASK_PRIO_DEFAULT
Thanks

GPIO 32 of the ESP32 chip is, unfortunately, not an entirely normal GPIO, it has a dual purpose to act as a 32 kHz crystal input on the RTC GPIO bus.  So it will come up at boot as an RTC GPIO and an input, not a normal GPIO and an output, which is what you want.
So somewhere in your code, before any ubxlib stuff (e.g. at the very start of setup()), I think you will need to call something like:
rtc_gpio_deinit(U_CFG_APP_PIN_CELL_ENABLE_POWER);
...to make that pin a normal GPIO; the ubxlib code will be able to handle it from there.
On the #defines, I think passing in U_CELL_PWR_ON_PIN_INVERTED is working because the debug print from U_CELL says:
U_CELL: initialising with enable power pin 32 (0x20) (where 1 is on), PWR_ON pin 14 (0x0e) (and is toggled from 0 to 1) and VInt pin not connected.
For the PWR_ON pin it says is toggled from 0 to 1, which is the opposite of what it would usually say.  I think you might have edited the wrong u_cell.h when adding the #error to check this out: remember that, because Arduino insists on files being in a specific directory structure, our u_arduino.py script copies the files into that required directory structure; you either need to edit the copied file or you can edit the original file and run the u_arduino.py script again to copy the edited version into place.
Of course, you have not passed U_CELL_ENABLE_POWER_PIN_INVERTED into the build, so that would still be the wrong way up, even if you had added rtc_gpio_deinit() in the start of your application.
Just to show you how it should work, I have gone through all the build steps from scratch myself and a text file is attached which shows what I did and the full output from the Arduino tools; search the text file for lines which commence >>### containing my comments.  You can see the build fail because it hits the #error and then, when I remove the #error, it builds to completion.
esp32_build_output_rob.txt

Thanks, Rob, you are right the error works as I followed the temp folder ubxlib copy;

I also connected V_int "inverted" to pin 18 of ESP!
Also the build works with cpp not c, which is shorter in text now;
arduino-cli compile --fqbn esp32:esp32:featheresp32 --clean -v -u -p COM5 --libraries . --build-property compiler.cpp.extra_flags="-DU_CELL_VINT_PIN_INVERTED -DU_CELL_ENABLE_POWR_PIN_INVERTED -DU_CFG_TEST_CELL_MODULE_TYPE=U_CELL_MODULE_TYPE_SARA_R410M_02B -DU_CFG_APP_PIN_CELL_ENABLE_POWER=19 -DU_CFG_APP_PIN_CELL_VINT=18 -DU_CFG_APP_PIN_CELL_PWR_ON=14 -DU_CFG_APP_PIN_CELL_TXD=17 -DU_CFG_APP_PIN_CELL_RXD=16" ubxlib\examples\sockets\sockets.ino
If you like I can send our board schematic PDF to your email provided on meades.org or to your LinkedIn!
right inside setup()), after its curly brace, I added:
rtc_gpio_deinit(U_CFG_APP_PIN_CELL_ENABLE_POWER);
But if you try that it does not compile so I removed that and just changed pin from 32 to Pin19 and it compiles with our arduino-cli code!
Arduino serial result;
U_CELL: initialising with enable power pin 19 (0x13) (where 1 is on), PWR_ON pin 14 (0x0e) (and is toggled from 1 to 0) and VInt pin 18 (0x12) (and is 1 when module is on).
U_CELL_PWR: powering on, module is already on.
[00]ATE0
AT
AT
AT
AT
ATE0
U_CELL_PWR: powering on.
[fb][fe][ff][ff][ff][ff][fb][fb][fb][ff][ff][ff][ff][ff][ff][ff][ff][ff][fb][ff][ff][ff][fb][ff][ff][ff][ff][ff][fe][ff][ff][ff][ff][ff][f7]
Unable to add network, error -7!
More details on Serial;
arduino_serial_Mohsen.txt
Thanks,
MoZen

Hmm, OK, you say "the build works with cpp not c, which is shorter in text now", but it would be really surprising if that were the case; u_cell.c, u_cell_pwr.c etc. should be treated as .c files and not .cpp files.  It looks to me as though the flags are being applied to the .ino file (because the pin numbers are coming out correct and they take effect in the .ino file), which I guess is being turned into a .cpp file, but I suspect they are NOT being applied to any of the .c files, which is where the inversion takes effect.
I note that you are no longer inverting the PWR_ON pin.  Is that intentional?  Also note a spelling mistake above U_CELL_ENABLE_POWR_PIN_INVERTED: should be U_CELL_ENABLE_POWER_PIN_INVERTED (missing the E in POWER).
Assuming you did still intend to invert all three pins then I think if you try:
arduino-cli compile --fqbn esp32:esp32:featheresp32 --clean -v -u -p COM5 --libraries . --build-property compiler.cpp.extra_flags="-DU_CFG_TEST_CELL_MODULE_TYPE=U_CELL_MODULE_TYPE_SARA_R410M_02B -DU_CFG_APP_PIN_CELL_ENABLE_POWER=19 -DU_CFG_APP_PIN_CELL_VINT=18 -DU_CFG_APP_PIN_CELL_PWR_ON=14 -DU_CFG_APP_PIN_CELL_TXD=17 -DU_CFG_APP_PIN_CELL_RXD=16" --build-property compiler.c.extra_flags="-DU_CELL_PWR_ON_PIN_INVERTED -DU_CELL_ENABLE_POWER_PIN_INVERTED -DU_CELL_VINT_PIN_INVERTED" ubxlib\examples\sockets\sockets.ino
...there's a chance things will start working.  Note that this applies some flags only to the .ino file and some flags only to the .c files: I would normally apply all the flags to all files, basically typing them all in twice, since that is more straightforward; I've only done the above to save you typing.

Thanks for correcting my error on the build side,also on typo since in some places like in socket.ino U_CFG_APP_PIN_CELL_PWR_ON was used I used it as PWR.
Yes, I tried to tweak the pins inverting and not inverting them to see if it will work! but it was probably not a good idea, haha!
Using your code, andwith no deinitiaizing pins in sockets.ino code as before, Now we have;
U_CELL: initialising with enable power pin 19 (0x13) (where 0 is on), PWR_ON pin 14 (0x0e) (and is toggled from 0 to 1) and VInt pin 18 (0x12) (and is 0 when module is on).
U_CELL_PWR: powering on, module is already on.
[00][e0]ATE0
[fb][ff][ff][ff][ff][ff][ff][ff][ff][fb][ff][fb][ff][fb][ff][ff][ff][ff][df][ff][ff][ff][ff][ff][ff][ff][ff][ff][ff][ff][ff][ff][ff][ff][ff][7f][ff][ff][ff][ff][ff][bf][ff][f7][ff][ff][ff][ff][ff][fb][ff][ff][fb][ff][fb][fb][ff][fb][df][7f][f9][fb][fb][ff][ff][ff][ff][fb][ff]7[fb][ff][ff][ff][df][ff][ff][7f][fa][ff][ff][ff][ff][fb][df][de][df]w[fb][ff][df][ff][ff][ff][df][fb][af][bf][f7][ff][fb][be][ff][ff][ff][ff][ff][ff][7f]/[ff][fe][ff][ef][ff][fd][b3][eb][7f][ff][ff][ef][fe][fb];[fe]][dd][b6][bb][bd]?[be],[ff][8f]?[fe]~[8a][f5][df]Y[da][b7][d4][d2][cc][0c][f2]<[db][9b][1e][14][HDfW[14]0[d5][f0][08]PE$[a3][1f][00][c4][10]C[07][aa][0c]7[fa][80]&[00][00]![95] [12][00][12]P[81][00]!D [01][00][12]@[00]$[04][80][00][01][00][05][00][05][92]A"[10] [00][02][08][0c][00][01][90][80][00][00][11][01] [00][00][00][00]P[00][00][05][04][10][00][10][00]@[00][08][00][00]%[00][00][04][00][00][00][00][00][00][04][00]a[00][00][00][02][08][00][00]@[00][00][00][00][00][00][00][10][00][00][00][01][00][00][00][01][00][10][00][00][08][00][02][08][00]@[00][08][00]X[00][00]A[00][00]"[00][00][00][00][00][00][02][00][00][00][08]@[00][00][00][00][00] [00]@[00][00]H[00][00][00][00][00][00][00][00][00][02][00][00][00][00][00][00][82]@[00][00][00][00][00][09][00][00][00][00][00][00][00][00][00][00][00][00][00]AT
AT
Unable to add network, error -7!
Network is not available!
1 Tests 1 Failures 0 Ignored
Is it my board design problem? Is the board working with the serial reports? Should I solder a new board?

Also Header Pins;
Feather Pin 14 is connected to Sara-module Pin Start which goes to the gate of Power_On pin with  Q6 transistor.
Feather Pin 18 is connected to Sara-module Pin SARA_R4_ON Not which is the inverse of V_INT.
Feather Pin 19 is connected to Sara-module Pin PWR Not which is the inverse of Power ON Enable; when it is connected to GND, Gate of Q5 tranisitor which is a PMOS will conduct 3 volts from feather to Sara.
Feather Pin RX and TX have also connected accordingly.

ESP Feather Pins:

Thanks

I think I probably should use the oscilloscope to see if the Uart transmission is working!

Yes, I think that's the next step.
As far as I can see, assuming that the lines from the MCU to the module for enable power, PWR_ON, and VINT are all inverting, then the code is being built correctly for that (from the U_CELL print), which only leaves the level shifting of the UART.  A storage scope or Saleae probe, something that can show you the dynamic behaviour over time, is what you need to check that the UART data is good, both at the module in the TXD direction and at the MCU in the RXD direction.

I found a Tektronix TDS2012 Digital storage scope in our lab, and it looks like the 115200 baud rate is equal to 86 microseconds, so I tune the sec/dv on the scope to be around 100 microseconds, but I have not seen anything interesting to or from RX and TX so far, just a DC shift when it sends AT commands, not sure whats wrong with it...
The pattern of received UART data being printed out above would suggest that something slowly slewing from 0xff to 0 rather than being proper crisp received bytes.
Do you have a successful UART reference for sent and received through RXD and TXD? Since you said you use Sparkfun level shifter do you think our design for level shifting will work? Do you add capacitors in the way?
Thanks

For the TXD/RXD UART stuff it is not something I've ever had to worry about, I usually just connect-up via the Sparkfun bidirectional level shifters and that works for me (with VH connected to the MCU's 3.3V and VL connected to the module's VINT).
Here are a couple of screen-shots I have taken of a Saleae probe output in the past: one of the power on process and another of just AT being sent to the module.  In the images CP_ON is another name for PWR_ON; all traces were taken on the module-side of the level shifters.

FYI, you see RXD activity in the first image above before any TXD activity only because these traces were taken with a SARA-R5 module that had a development build on it which spits out a lot of identification information at power-on.  Normally you wouldn't see RXD activity until after the TXD activity unless a greeting message had been set in the module.

FYI, you see RXD activity in the first image above before any TXD activity only because these traces were taken with a SARA-R5 module that had a development build on it which spits out a lot of identification information at power-on.  Normally you wouldn't see RXD activity until after the TXD activity unless a greeting message had been set in the module.

In the screenshots, you have 1.8 volts for V_int and since we have populated the V-int a 3rd header(where I just added a 100 nF=0.1 uF capacitor between it and the ground), currently I do not see any voltage on it!; when AT commands are sent in terminal or after it gets -7 error, so I might try connecting cell enable power to ground again to see how it might change the situation!

In the screenshots, you have 1.8 volts for V_int and since we have populated the V-int a 3rd header(where I just added a 100 nF=0.1 uF capacitor between it and the ground), currently I do not see any voltage on it!; when AT commands are sent in terminal or after it gets -7 error, so I might try connecting cell enable power to ground again to see how it might change the situation!

Yes: if the module is not driving its VINT pin to 1.8V then the module is off.
In the AT print-out above it says U_CELL_PWR: powering on, the module is already on, which is happening since you have defined U_CELL_VINT_PIN_INVERTED and you have connected the module's VINT to MCU pin 18 via an inverter; the MCU is measuring that pin as 0 which means it must be being driven high by the module.

Yes: if the module is not driving its VINT pin to 1.8V then the module is off.
In the AT print-out above it says U_CELL_PWR: powering on, the module is already on, which is happening since you have defined U_CELL_VINT_PIN_INVERTED and you have connected the module's VINT to MCU pin 18 via an inverter; the MCU is measuring that pin as 0 which means it must be being driven high by the module.

Hi Rob,
We just bought the GPS version of the cellular modules as the backlog is over on some vendors, and we will solder a new board with a better regulator design, we might buy Salea probe, I was wondering if logic 8 is enough compared to logic 8 pro. Since u-blox UART uses a 115,200 baud rate, do you think it will work with logic 8? The sample rate is 100 Mega samples/ sec, vs 8 pro which is 500 MS/s. (I suspect since 115,200 is less than 1 Mega samples per second, so logic 8 might work).
Thanks,
Mohsen

Hi Mohsen: any Saleae probe will work very well on any normal UART, no problem.  In my view it is an indispensable tool for the price: I bought one for myself after using one at work.

Hi Rob,
Wassup?
I bought the Salea Probe, have not used it yet will test it soon!
Anyhow,  we recently soldered a new board having both ESP MCU and SaraModule, MCU works fine so I loaded ublox code on it and we have connected GPIO 19 of it to CFG_APP_PIN_CELL_ENABLE_POWER=19 using the same Q5 PMOS so when GPIO 19 is zero volts, PMOS Source to drain conduct and Sara module is energized;
Q5 Image: https://github.com/u-blox/ubxlib/issues/29#issuecomment-1001098138
When I reset the MCU, for a while, GPIO 19 is zero so Sara module gets 3.3V but after a while GPIO 19 switches to 3.3Vs, hence no power to Sara module, and when I leave it for a long time, I see the Gate is zero again hence 3.3V is provided to Sara module. Any reason why the code does that?
Also, the board does not provide 1.8V voltage as an internal Pullup, I guess we need to solder another board...
Thanks,
Mohsen

Hi Rob,
Wassup?
We recently soldered a new board having both ESP MCU and SaraModule, MCU works fine so I loaded ublox code on it and we have connected GPIO 19 of it to CFG_APP_PIN_CELL_ENABLE_POWER=19 using the same Q5 PMOS so when GPIO 19 is zero volts, PMOS Source to drain conduct and Sara module is energized;
Q5 Image: #29 (comment)
I guess the inverting of cell enable power is not working properly;
When I reset the MCU, for a while, GPIO 19 is zero so Sara module gets 3.3V but after a while GPIO 19 switches to 3.3Vs, hence no power to Sara module, and when I leave it for a long time, I see the Gate is zero again hence 3.3V is provided to Sara module. Any reason why the code does that?
Also, the board does not provide 1.8V voltage as an internal Pullup, I guess we need to solder another board...
I use dev module for our MCU(ESP32_WROOM-32UE (has UFL antenna), so I set its esp32 fqbn simply as esp32; esp32:esp32:esp32
arduino-cli compile --fqbn esp32:esp32:esp32 --clean -v -u -p COM3 --libraries . --build-property compiler.c.extra_flags="-DU_CELL_PWR_ON_PIN_INVERTED -DU_CELL_ENABLE_POWR_PIN_INVERTED -DU_CFG_TEST_CELL_MODULE_TYPE=U_CELL_MODULE_TYPE_SARA_R410M_02B -DU_CFG_APP_PIN_CELL_ENABLE_POWER=19 -DU_CFG_APP_PIN_CELL_PWR_ON=0 -DU_CFG_APP_PIN_CELL_RESET=2 -DU_CFG_APP_PIN_CELL_TXD=1 -DU_CFG_APP_PIN_CELL_RXD=3" --build-property compiler.cpp.extra_flags="-DU_CELL_PWR_ON_PIN_INVERTED -DU_CELL_ENABLE_POWR_PIN_INVERTED -DU_CFG_TEST_CELL_MODULE_TYPE=U_CELL_MODULE_TYPE_SARA_R410M_02B -DU_CFG_APP_PIN_CELL_ENABLE_POWER=19 -DU_CFG_APP_PIN_CELL_PWR_ON=0 -DU_CFG_APP_PIN_CELL_RESET=2 -DU_CFG_APP_PIN_CELL_TXD=1 -DU_CFG_APP_PIN_CELL_RXD=3" ubxlib\examples\sockets\sockets.ino
Thanks,
Mohsen

I am running socket.ino mentioned at the end of the arduino-cli command I shared;
ubxlib\examples\sockets\sockets.ino
The only thing I modified in .ino file is adding "hologram" here;
"hologram", /* APN: NULL to accept default.  If using a Thingstream SIM enter "tsiot" here */
I did not even provide the U_CFG_TEST_CELL_MODULE_TYPE U_CELL as I am defining it in cli.
"The board does not provide 1.8V voltage as an internal Pullup" Sorry, I meant the hologram module does not turn on!
"The 'E' is missing"; Oh, OK I will modify and upload it shortly.
I will upload to it with U_CELL_ENABLE_POWER_PIN_INVERTED and see what happens...
Thanks

Ah, yes, if you are running the sockets example then that will power the module on at the start and off at the end, so that is fine, though if you have an inverter in the line and the flag passed to the CLI was misspelled the code will powering the module off at the start and on at the end!

Thanks, I see, now it is fixed, I have stable 3.3 Volts on Drain of Q5, however looking at Arduino Serial Monitor, It is just writing ATATAT...
I changed that APN from "hologram" to NULL still showing ATATAT on the monitor. Also on SARA pin 4 (V_int where we connected it to a resistor head) I do not see 1.8V! Which tells me Sara is OFF.
Btw, we bought the Saleae Logic 8 Probe, what do you use to connect its wires to let's say a few SMD resistor/transistor contacts?

One more thing: for pins where you have set the value you desire correctly in the u_cfg_app_platform_specific.h you don't also need to pass that flag on the compiler command-line; the compiler command-line is only for defining values that are not set anywhere in the code or a header file or for overriding values that are already set in the code or a header file.

Thanks, Rob, I've added an inversion flags for Reset to compiler.c and cpp; -DU_CELL_RESET_PIN_INVERTED
I will work with Salea Probe next week! Cheers and have a great weekend!

Thanks, Rob,

I have tested the Saleae probe with my scope signal generator, it works! Saleae 8 did not come with connection clips, it has the harness wires though, So I will solder instead.

For the PCB, we connected RTS and DTR to the ground, and we did not populate the connection for Sara-R4 modules' CTS, DSR, DCD, and VINT to the MCU.

Best Regards

Oh, strange, my Saleae 8 probe came with those clips, they must have changed the package of stuff they provide.

Oh, actually, I did not know I can remove the 16 clippers from the package!
I thought these two extra black boxes are clamps for holding the harness wires!
Also, the startup guide does not talk about it assuming everyone is smart, haha!
https://support.saleae.com/getting-started/setup

Oooo, weird: mine came in a little plastic bag, which is a tad more obvious!

Hi Rob,
I hope all is well and thanks for inspiring me to search for and find the Saleae clips!
My goal is to have Module Power_ON(looking at 1.6.1 on integration Manual).

I set Saleae to record when it sees a rising age trigger on GPIO0_LTE_START MCU signal applied to the Gate of an NMOS that when is high it will conduct hence provides Low_level on PWR_ON pin(all transistors are NMOS).
Only the cell_enable_power signal goes to a PMOS and it will be active when the gate is Zero volts, in that case, it will conduct 3.3V to VCC pin.
I followed your suggestion for having RXD as close to the module as possible but for now, I was curious to see what MCU is sending on the Gates.
My question is, I do not understand why U_cell_enable_power(channel zero) toggles! I think MCU should always provide zero volts on this pin(GPIO19 of MCU which is LTE_START_NOT), similar to Fig.9 of the manual PDF where there is a stable VCC and Power_ON signal toggle to a LOW_Pulse to turn the module ON.
In Channel D0 we see Low on both ends of the signal; which provides a stable Low voltage on the gate of PMOS which is good which keeps providing VCC to the module, but channel D0 shows that, at the crucial important time when Power_ON is Low_Level, we see Gate of PMOS  jumps to High so actually in that toggle time, the PMOS transistor is OFF, so module does not get any VCC during that time. and UBLOX RXD when it does not receive VCC is sending a Pulse on D4!
Our flags are:
U_CELL_ENABLE_POWER_PIN_INVERTED
-DU_CFG_APP_PIN_CELL_ENABLE_POWER=19

Thanks,
Mohsen

Since I assume the previous message signals were probably when I hit our reset button on the board, now I just connected the Lipo battery to the board and looked for trigger on Power_ON without pressing any button, it resulted in the following signals;
Also, like previous message, I have connected Ground of Saleae to the ground of the HDMI programmer (I use it to load CLI code from PC) and D3 signal which goes to GPIO3/RX MCU(comes from RXD from level shifter) is connected to HDMI programmer RX pin.

I think we need to zoom out a little here: there is nothing in the ubxlib code that operates on the micro-second timeframe.  The RTOS tick on Arduino is 1 ms, nothing software-driven is going to happen faster than that, and very likely 10 times slower than that.
For instance, for the UART data lines, running at 115,200 and driven by the UART HW block (not SW), one bit is 8 us, one byte is 80 us (1 start bit, 8 data bits, 1 stop bit), so we need to be looking here at many 100's of ms to see what's going on in SW terms.  Anything less than that is HW which I can't really be of much help with!

Yes, you are right I was uploading the zoom out that you messaged!

Can you attach a probe to the TXD line as well as the RXD line, in the Saleae SW attach a serial analyzer to both displaying both ASCII and Hex, then do another trace with a timespan of 15 seconds from power-on and post the fully zoomed-out view first?

Ok, will do, I will edit this message with new trace accordingly (got another class now!).

Aloha Rob, I have two 15 seconds traces; When everything is Off and I insert the Lipo battery to energize the MCU.
Also when Lipo is there and I hit Reset button connected to EN of MCU.

Inserting Lipo:

D0 channel: On second 6.5 : GPIO19 connected to gate of PMOS goes Low, hence 3.3V goes to SARA VCC pin.
D1 shows: at 6.7 sec, MCU GPIO0 applies high voltage to NMOS Q6 Gate, so SARA Power_ON pin should see toggle from 1 to zero (If there was 1.8V pullup internal there).
D6: MCU TX output(AT command signals?)

2. Lipo was connected so MCU is ON, I started recording with Saleae, then I hit Reset button on the board; kinda same thing happens;

SARA looks OFF and since I use level shifter transistors in our design to shift 3.3 to 1.8V (same as Sparkfun level shifter you use; https://www.sparkfun.com/products/12009).
Our Level_shifter Gates are energized with  V_int:1.8V, I guess since I do not have 1.8V when measuring V_int signal with multimeter, TX handshake from MCU can not be converted to 1.8V equivalent and reach TXD on SARA (level shifter transistors are OFF as their Gate is connected to V_int).
Thanks

Great, so those traces all look sensible: the Enable Power line is being brought up, the TXD line is the MCU sending "AT" to see if the module is awake, the PWR_ON line is toggling to power it on and to power it off.
Can we now add two more probes: one connected to the 3V3 power supply pin of the module and another connected to the V_INT output from the module, both as near to the module as possible, and take another trace?  Let's make it 10 seconds this time.

Thanks for interpreting the results, before adding more probes, with your explanation I might know what is wrong; Enable Power is not inverted here;
At second 2: You say; the Enable Power line is being brought up;
Actually, As I mentioned we only have one PMOS transistor in our design, Q5 (and on D0 Saleae we are looking at its Gate! signal coming from MCU GPIO19);
https://github.com/u-blox/ubxlib/issues/29#issuecomment-1040801905
I know PMOS looks different, in conventional NMOS, as you know, when we apply High on Gate, it will switch(source to drain), in PMOS it is the opposite, when we have Low on Gate, it will switch(here in Eagle schematic, when it sees Low on its gate, it will switch 3.3V to VDD which is Sara VCC). So in order to Enable power to SARA, we need to have the opposite waveform. right?
I think that is why you were suspicious of 3.3 V and want me to add probes there, I will do shortly.
Thanks for helping me debug!

Ummm, could be, I'm not really a HW guy I'm afraid, you know best!

Yeah, I added Saleae Probe D7 is VDD which goes to SARA after it has seen PMOS, and is an inversion of what we want(D0, Cell_enable power).
Btw, sorry it is not clean VDD or VCC I will work on that.

Well U_CELL_ENABLE_POWER_PIN_INVERTED is definitely the correct #define and you have corrected the spelling and, I guess, done a clean build; please do so if you haven't; remember that changing a #define passed to the compiler on the command-line hasn't changed the code so no "make" process will know to re-build the files in question.

Ok, I just rechecked the spelling, it is fine, and recompiled it from the Arduino command-line interface to the board, the same issue exist.
remember that changing a #define passed to the compiler on the command-line hasn't changed the code so no "make" process will know to re-build the files in question.
So what do you recommend we do to change the code and make it re-build?

Delete the build directory is the usual way, then you can see that all the object files have gone.

Just to be complete sure, you do have these lines in cell.h:

Yes, I was looking at them just now, I am deleting the temp folder content now...

In every example and every test, when the cellular instance is created, it prints out what the "on" state of the Enable Power pin is, so you could check that.

Not sure where can I check the print of cellular instance(uCellDeinit())? Normally I use Serial.print() and read the serial monitor.
Btw, in Arduino IDE, Can I add #define U_CELL_ENABLE_POWER_PIN_ON_STATE 1?

Not sure I understand: can you not just look at the debug serial port output of the ESP32 chip?

Ok, will look it up, thanks.

Hi Rob, Platform IO has a great debugging environment, it supports Arduino and esp32, I was able to almost build ublox sockets example there, I get one error referring to u_cfg_os_platform_specific.h lines 97; 'CONFIG_ESP32_PTHREAD_TASK_PRIO_DEFAULT' undeclared (first use in this function)

Are you sure you are connected to the correct COM port of the FTDI chip on the board?  What you're showing there looks like the port that is connected to the cellular module rather than the trace UART.  When you boot the ESP32 chip I would expect to see a splurge of its boot output, e.g.
boot: ESP-IDF v5.0-dev-1554-g20847eeb96 2nd stage bootloader
boot: compile time 23:14:24
boot: chip revision: 1
boot_comm: chip revision: 1, min. bootloader chip revision: 0
boot.esp32: SPI Speed      : 40MHz
boot.esp32: SPI Mode       : DIO
boot.esp32: SPI Flash Size : 2MB
boot: Enabling RNG early entropy source...
boot: Partition Table:
boot: ## Label            Usage          Type ST Offset   Length
boot:  0 nvs              WiFi data        01 02 00009000 00006000
boot:  1 phy_init         RF data          01 01 0000f000 00001000
boot:  2 factory          factory app      00 00 00010000 00100000
...

Do you see that in your trace output?
As to the Platform IO thingy, looks like they are using a different version of ESP-IDF, I have a feeling that ESP-IDF changed their #defines for things at some point.

What you're showing there looks like the port that is connected to the cellular module rather than the trace UART.
Thanks, Rob, you are right, between RX and TX pins of MCU and Module we have an FTDI program port to program the MCU, we did so since we needed many pins of ESP32 for driving the camera so we wanted to conserve some GPIOs. So the ATATAT commands I was seeing in Serial Plotter of IDE were coming from MCU after being programmed!
To access the state of the Enable Power pin, today I uploaded the same code on my feather board connected to a Ublox shield I designed for OpenMV, where RX and TX are connected to Module and the board is programmed and debugged using a micro-USB port. It shows, enable power pin is inverting actually;
"U_CELL: initialising with enable power pin 32 (0x20) (where 0 is on)"
The interesting thing is it will U_CELL_PWR: powering on after 4 or 5 seconds;

We also see the boot splurge data too.
So we see the inversion signal from MCU 4 or 5 seconds after MCU Reset button is pushed;

So actually we were wrong thinking AT commands last shortly(4 seconds between two Power_ON pulse signals, the board energizes the module after 4 seconds, Therefore I do not know what is the first Power_ON pulse(right after Reset) is for.

OK: it might put a bit of a crimp in your development, not being able to see any debug output from the MCU on your actual target, but that's your choice of course.
As to that initial pulse over on the left, I guess that's simply due to the default state of a pin 19 for ESP32.  The first thing the code does when it initialises the cellular instance is to put all of the IO lines in the correct default statues.  For the Enable Power pin, with the inversion in place, the code would set its default state to high, so that it can be set low to do the Enable Power part.  So I assume all your seeing there is that the ESP32, at boot, has pin 19 low, then when the code starts running it sets pin 19 high as the requested default state (i.e. "off"). It's a transitory state.

I'm not sure I'm really following this but I think you're saying that even with your simple code you can't get the module into a state where VINT remains high.  All I can do is point out the PWR_ON-low pulse timings from the the SARA-R4 series data-sheet:

Also, it is worth double-checking that your board can supply both the average and the peak current required by the SARA-R410:

Aloha Rob,
I used a power supply with 750 mA limitation and 4.2V instead of battery port, however, it was a soldering issue, now I can turn the module ON and OFF with my simple code on a newly soldered board.
When the module is ON, I have  V_INT and RESET_N pin pulled up internally to1.8V and 0.8V on PWR_ON.
Then I loaded sockets.ino code; I get the following AT commands. I assume MCU tries to set for the Sara Module.
So when resetting the MCU, the sockets code quickly tries to boot up the module twice(I will add a delay there for a few seconds for it to start a bit later), and then it gives up by turning the signal to SARA VCC pins to low (all these are done in 20 to 30 seconds, in contrast, Sparkfun Ublox shield guide says turning a module ON in the USA needs sometimes up to 3 minutes);
ATE0
AT+CMEE=2
AT+UDCONF=1,0
ATI9
AT&C1
AT&D0
AT+UCGED=5
AT+CPSMS=0
AT+CPSMS=0
AT+CPSMS=0
AT
AT
AT
AT
ATE0
AT+CMEE=2           //Report mobile termination error
AT+UDCONF=1,0  // Enables/disables the HEX mode
ATI9
AT&C1
AT&D0
AT+UCGED=5     //  Channel and network environment description
AT+CPSMS=0
AT+CPSMS=0
AT+CPSMS=0
AT
AT
Looking at AT commands PDF, it looks like Ublox sockets.ino code is trying to set some values but does not probably get back from module anything. (I should check with Saleae!)
I just pinged the "ubxlib.it-sgn.u-blox.com", and it return an IP address. I will send you the Saleae Probe results later today.  Thank you for your help!


Support deeper subdomains in domain validation

The regex used for domain validation doesn't support 4th-level or greater subdomains.
The SO post that had the original regex used here has an update with a more complete pattern, so I've updated the pattern here to match.

This is a good catch! I'm not sure but I also found the host_re regex in Django's built-in URLValidator class: django.core.validators | Django documentation | Django. Maybe could we use this regex here? It seems to cover more comprehensive patterns.

Oh yes, we should totally use the Django one if we can - it's been fixed over many years of bug reports.

I've updated this to use a new DomainValidator based on the regex from Django's URLValidator and added a few tests.

I think it might be better to reference the regex already present in the Django code rather than duplicate the regex with the risk that they soon become not aligned.

I felt like pulling undocumented attributes directly off URLValidator has the potential to break with Django upgrades, and subclassing URLValidator and trying to 'undo' all the bits we don't need feels fragile and unnecessarily complicated - but I'm happy to rework this if needed.

Yeah, I don't mind vendoring (copying) here - it's small enough and the license is directly compatible that we're fine, and that regex doesn't exactly change every month!

Thanks for this!


Fixed doc referece to job cprofile file location

Closes N/A
What's Changed
Reflected location of docs: https://github.com/nautobot/nautobot/blob/c6ce7dbd4b98aa776fbf630f733ce44f48f313f7/nautobot/extras/jobs.py#L138C60-L138C78
Screenshots
TODO

This should get back ported to ltm as well.


Should a comment reference on a type alias be able to refer to members of the aliased type?

Today we place a declaration's members into the resolvable scope of the declaration's doc comment, including inherited ones:
class C {
  int foo = 1;
}

/// Refer to [foo].
abstract class D implements C {}

That comment reference resolves to C.foo.
But we don't do anything similar for type aliases. For example, should the reference here resolve to int.isEven?
/// Can I refer to [isEven]?
typedef T = int;

There are several attempts to reference like this in the Dart SDK, e.g. on SetMixin (the comments resolve in dartdoc because it has a separate resolution mechanism).

I think we should support referencing the members of the aliased type. It seems sensible, given someone wants to document a type alias.

I'd say "no". The rule I use to understand what's in scope is that it's what's in scope inside the commented declaration (where "in scope" really means "can be referenced by a raw identifier" rather than the actual lexical scope).
The members of the aliased type is not in scope "inside" the type alias.
That said, aliases that allow static member access could be considered "special", and provide access to at least those static members. Could say that they give access to the instance members too, for good measure. But it's because they're special, not because it's obvious.

As a concrete example, if we wanted to keep a doc comment on SetMixin and refer to members of SetBase (the aliased type), like add and length, should they just be written as [SetBase.add] and [SetBase.length]? (They would appear at api.dart.dev with that qualified name, 'SetBase.add' and 'SetBase.length'.)

The language specification unambiguously defines the current scope for a documentation comment:

Documentation comments are comments that begin with the tokens /// or /**. Documentation comments are intended to be processed by a tool that produces human readable documentation. The current scope for a documentation comment immediately preceding the declaration of a class C is the body scope of C. The current scope for a documentation comment immediately preceding the declaration of a non-redirecting generative constructor k with initializing formals is the formal parameter initializer scope of k. Otherwise, the current scope for a documentation comment immediately preceding the declaration of a function f is the formal parameter scope of f.

This basically implies that a documentation comment on a class can "see" the members declared in the body of that class, the formal type parameters of the class, if any, and anything in the enclosing library scope (including imported stuff).
We should probably make this specification slightly more detailed and say that it is possible to refer to an identifier id in this documentation comment if it can be resolved using lexical lookup (as defined in the section 'Lexical Lookup'), such that we have access to members of the interface of the class that are inherited from the superclass or "promised" by the implements clause, not just the members that are declared in the class itself.
However, it is not obvious to me that the relevant references in the documentation of a declaration are exactly the ones that are the result of resolving an identifier in the body of said declaration, even with that clarification about this.id.
You could also say that the members of the interface of a type are relevant to the documentation of a declaration that introduces said type into the enclosing scope. Some declarations introducing a type will also have formal type parameters, and they'd be relevant as well.
I think this means "no change" to the current behavior, except that members of a type which is aliased in a typedef should be available as well. (But there could be additional type-introducing declarations in the future).
An obvious corner case to consider would be typedef F<X extends Widget> = X; which would consider the members of F to be the members of Widget, ignoring the fact that F<MySpecialWidget> has a different set of members (some members in addition to the ones that Widget has, and some members shared by both, but with different signatures).
I think it would make sense to generalize the rules about documentation comments along these lines. Presumably, it would be a non-breaking change.


Update couldBuy to reflect number of possible tickets left in window

Fixes #9

I also wonder if it should check the number of tickets in the mempool first, and have some kind of special behaviour depending on the fullness. Otherwise, fix those issues and it's OK.


Unpooled CompositeByteBufs log leak detection errors

If you create a CompositeByteBuf via Unpooled.wrappedBuffer(buffers...) or via Unpooled.compositeBuffer that itself wraps unpooled buffers, the resulting buffer will log a leak error on GC.
This seems like incorrect behavior...I'd expect only pooled buffers within a CompositeByteBuf or a CompositeByteBuf created via a pooled allocator to log leak errors.

Unpooled != not reference counted

Regardless of how many unpooled bytebufs "leak", the error is never logged, only with unpooled composite bytebufs. So either all unpooled bytebufs should warn, or unpooled composite bytebufs should not.

Will look into it.

Just checked and this was fixed already. Please reopen if you still see the problem

Thanks, will do.


Need to set Interpolator and duration

Thanks for such a nice library!
Is there a way to change the interpolator and duration? It would be really nice, and just a few lines of code I believe. Thanks.

Hi, there is no way to set that currently, unless you take a fork of the code and and a setter.


mutate no longer works with next version of dplyr

devtools::install_github(`hadley/dplyr`)
library(dplyr)

sc <- spark_connect(master = "local")
iris_tbl <- copy_to(sc, iris)
iris_tbl %>% mutate(S = Species)

Already addressed in: https://github.com/rstudio/sparklyr/commits/bugfix/hotfix-0.5.3

Fixed with the dplyr 0.6 upgrade.


[Testerina] Implement assertion equality diff for Xml

Description:
Implement assertion equality diff for Xml
Steps to reproduce:
Affected Versions:
OS, DB, other environment details and versions:
Related Issues (optional):

Suggested Labels (optional):

Suggested Assignees (optional):

There hasn't been a real requirement for this. Hence closing.


[Story] Print container logs with Scorecard test output

It would be useful to see the logs for each container running in the Scorecard tests, whether the tests pass or fail. It will be noisy, but I believe the only test output for Scorecard tests is the one log. Perhaps we can format these nicely with section headers for each container, with some whitespace before and after.
There's a Kubernetes API you can use to get the logs for each container:
https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#get-read-log-of-the-specified-pod

I wonder how about the scorecard can create a ConfigMap (with a well-known prefix/suffix) and put the logs with entries for each container in the cryostat pod.
At the end of tests, we can check if any "log" configmap exists, then dump it to local files for inspection. What do you think?

I was just thinking of reading the logs into memory while running the tests. There is a follow parameter that works like kubectl logs -f, and we could create a goroutine for each container to read them asynchronously. What do you think of that approach?

I was just thinking of reading the logs into memory while running the tests. There is a follow parameter that works like kubectl logs -f, and we could create a goroutine for each container to read them asynchronously. What do you think of that approach?

Right. That sounds good to me! I guess I was a bit concerned about large quantity of logs being stored in memory, especially in a more resource-limit cluster. Maybe, it's not too large to be concerned about.
Otherwise, I am thinking that goroutines can write to files on the pod's ephemeral storage and operator-sdk scorecard can default to run with:
-x, --skip-cleanup             Disable resource cleanup after tests are run

Then, before running scorecard tests, we can run clean up first with a prompt to confirm if any resources exist (or skipped if certain env var is set).  Logs can then access by exec into the scorecard pod.

Maybe I am overthinking. I can help with this and see it how it goes^^


Catalyst and Apple silicon support?

The build script lack of arm macOS, arm iOS simulator and Catalyst intel/arm support.

I've been able to build moltenvk on apple m1 mac mini. I've actually gone through a crazy process to recreate the lunarG sdk. I've managed to run the cube app from Vulkan-Tools and it runs, but vulkan info fails....
ERROR at /Users/danielelliott/Documents/projects/SDk/Vulkan-Tools/vulkaninfo/vulkaninfo.h:248:vkEnumerateInstanceExtensionProperties failed with ERROR_OUT_OF_HOST_MEMORY
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

not sure what that is about.
Also, I'm not sure how to build a universal app, all the builds seem to be Apple Silicon only.

So while vulkaninfo and the cube apps run, I'm getting a crash on my minimal triangle example...
https://github.com/dokipen3d/vulkanHppMinimalExample/edit/master/main.cpp
(Yes I know the validation layers is deprecated, but after moving it to the new name, it still crashes).

The build script lack of arm macOS, arm iOS simulator and Catalyst intel/arm support.

If you run fetchDependencies --all and make, it will build MoltenVK with Apple Silicon and Simulator support in the resulting XCFramework.
We're working on Mac Catalyst support.

Yes building is working and moltenvk is working with cube apps and vulkan info, so the seg faults in seeing g might be to do with the loader, or layers lib. I will wait for the official SDK. I wish there was a build script for the sdk (which I've been trying to put together, but it's a bit error prone). Its all amazing stuff though. Great work everyone.

Close as already supported on the latest release.

This works for me.

Close as already supported on the latest release.

This works for me!
TY


Verbose flag not outputting anything other than Entering chain and Finished chain

System Info
langchain==0.0.216 (I have had this since i started with Langchain (198)
python 3.10
Who can help?
No response
Information

[ ] The official example notebooks/scripts
[X] My own modified scripts

Related Components

[X] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[ ] Document Loaders
[ ] Vector Stores / Retrievers
[ ] Memory
[ ] Agents / Agent Executors
[ ] Tools / Toolkits
[ ] Chains
[x] Callbacks/Tracing
[ ] Async

Reproduction
I run this function with
{
 "prompt": "some prompt",
 "temperature": 0.2
}

from typing import Any

from langchain.chains import RetrievalQAWithSourcesChain
from langchain.chat_models import ChatOpenAI
from langchain.embeddings import OpenAIEmbeddings
from langchain.prompts import PromptTemplate

from app.config.api_keys import ApiKeys
from app.repositories.pgvector_repository import PGVectorRepository

def completions_service(data) -> dict[str, Any]:
    pgvector = PGVectorRepository().instance

    tech_template = """Some prompt info (redacted)

        {summaries}

        Q: {question}
        A: """
    prompt_template = PromptTemplate(
        template=tech_template, input_variables=["summaries", "question"]
    )

    qa = RetrievalQAWithSourcesChain.from_chain_type(llm=ChatOpenAI(temperature=data['temperature'],
                                                                    model_name="gpt-3.5-turbo",
                                                                    openai_api_key=ApiKeys.openai),
                                                     chain_type="stuff",
                                                     retriever=pgvector.as_retriever(),
                                                     chain_type_kwargs={"prompt": prompt_template},
                                                     return_source_documents=True,
                                                     verbose=True,
                                                     )

    output = qa({"question": data['prompt']})

    return output

I get this in the command line

> Entering new  chain...

> Finished chain.

For some reason, there is a double space in the Entering line (as if there is meant to be something there) and then nothing until it says finish, I have set verbose = True but not luck
Expected behavior
I would expect to see the embeddings, the full prompt being sent to openai etc. but I get nothing.

Work around...
I added , callbacks=[ConsoleCallbackHandler()] to this line
    output= qa({"question": data['prompt']}, callbacks=[ConsoleCallbackHandler()])

But not it outputs everything even with Verbose set to false...

Try,
import langchain
langchain.verbose = True