Common

ESP8266: Offline Debugging with the Platformio Environment

Daniel Eichhorn on
4
Having tools is great but you also need to know how to use them. Being a native Java speaker using a debugger is a very natural and easy thing. With the ESP8266 on the other hand this looks a bit different. I struggled now for a half day to figure out how to use the debugger to resolve a nasty crash that was happening if I would use a NFC card reader several times in a row. Most of the time I tried to use life debugging as I was used from the Java world and it still doesn’t work. The debugger connects perfectly fine to the serial console (using the tcp_serial_redirect.py script) but then just waits after printing Remote debugging using localhost:9980. Please let me know if you have an idea what might be the problem.


But it turns out that life debugging wasn’t even necessary: you can use the crash information also offline to figure out what function was responsible for the crash!

Preparing the binary

These days I am using the platform.io environment with the Atom editor to do my ESP8266  development. In order to have enough information in the binary you have to add some flags to the platformio.ini. In the section for your target platform add:

 

build_flags = -Og -ggdb -DDEBUG_ESP_PORT=Serial



which will cause the compiler to create a binary containing the symbol information required for debugging. Save it and make sure that the whole workspace gets re-compiled.. Upload it, start the serial console and provoke the crash:

 

Fatal exception 0(IllegalInstructionCause):
epc1=0x4020188c, epc2=0x00000000

 

The interesting part here is the epc1 address, which is the program counter.

Finding the culprit

The next step was to figure out where the debugger was installed for the platformio environment.
On a Mac you can (currently) find it here:

~/.platformio/packages/toolchain-xtensa/bin/xtensa-lx106-elf-gdb

 Now you can start it by telling it where your binary is:

 

~/.platformio/packages/toolchain-xtensa/bin/xtensa-lx106-elf-gdb ./.pioenvs/nodemcuv2/firmware.elf

(assuming that you are in the root of your project folder). The debugger should great you with:

Reading symbols from .pioenvs/nodemcuv2/firmware.elf...done. (gdb)

Now you can enter the epc1 pointer from above and get a pretty clear picture where the problem occured:

(gdb) info line *0x4020188c Line 116 of ".pioenvs/nodemcuv2/FrameworkArduino/Tone.cpp" starts at address 0x4020188c and ends at 0x40201891

Tone.cpp was the culprit! I quickly headed over to the Arduino/ESP8266 github repo and searched for problems with Tone and found the following issue:  https://github.com/esp8266/Arduino/issues/1761

Even better was that there was already a solution, even if not released yet: you just have to add ICACHE_RAM_ATTR  to a method in Tone.cpp. Since I cannot wait for the next release to fix this I did it myself and all the problems went away!

Summary

Even (or even more) on embedded devices debugging is a very powerful tool. But it might take some time to set it up properly. While I still did not succeed in doing operation on the open heart (life debugging) it wasn’t even necessary in this case. Having a program counter pointing me to the location of the crash helped already a lot. After debugging, don’t forget to remove the flags in platformio or you will miss compiler optimizations…

Posted by Daniel Eichhorn

Daniel Eichhorn is a software engineer and an enthusiastic maker. He loves working on projects related to the Internet of Things, electronics, and embedded software. He owns two 3D printers: a Creality Ender 3 V2 and an Elegoo Mars 3. In 2018, he co-founded ThingPulse along with Marcel Stör. Together, they develop IoT hardware and distribute it to various locations around the world.

All Posts Website

4 comments

  1. For live debugging, you need to include GDBStub library and open gdb session once crash happens. Gdb should then present you with a prompt where you can type backtrace command, for example.

    Reply

    • Hi Ivan. Thanks for your post and your great work for the ESP8266 community! I tried exactly that: Included GDBStub, added #define DEBUG_ESP_PORT Serial (is this necessary), forwarded the serial port to an IP port, listened with nc on that port and when the crash happened tried to connect with gdb to the ESP. But gdb stops with "Remote debugging using :9980" without presenting the (gdb) prompt. With this crash the Watchdog always restarted the ESP, so I tried to add the ((int*)0) = 0; in the suggested method. But the ESP kept restarting. Couldn't I connect because of the restarting? Or do I have to turn off Serial logging to free the Serial channel for gdb connection?

      Reply

  2. Knowing EPC isn’t enough in general case – it may point into some standard library (f.i., NULL pointer is passed incorrectly in some function).
    For the proper investigation one needs to know the stack, which is usually dumped to the serial during the crash.
    There is a tool named Exception Stack Trace Decoder (https://github.com/me-no-dev/EspExceptionDecoder) for Arduino IDE. It helps to decode the stack trace, showing functions (in a human-readable format) for each address in it.

    Reply

  3. I get the following:
    (gdb) info line *0x4000e1b2
    No line number information available for address 0x4000e1b2

    Any idea maybe? I’m kind of stuck 🙁

    Reply

Leave a Reply