ESP8266: Continuous Delivery Pipeline – Push To Production

ESP32 Development Board WiFi+Bluetooth Ultra-Low Power Consumption Dual Cores ESP-32 ESP-32S Board
Integrated antenna and RF balun, power amplifier, low-noise amplifiers, filters, and power management module. The entire solution takes up the least amount of printed circuit board area. This board is used with 2.4 GHz dual-mode Wi-Fi and Bluetooth chips by TSMC 40nm low power technology, power and RF properties best, which is safe, reliable, and scalable to a variety of applications.

The integration of the ESP8266 framework into the Arduino IDE brought ease of use for development. This was further improved when firmwares could be pushed over the local network to the test devices with the direct over-the-air feature. But what if your ESP8266 devices are no longer in your own network? How can you then update all the devices which are running an older version of your firmware?

The Arduino/ESP8266 crew also has an answer for that: the ESPhttpUpdate class lets you download the latest firmware from a web server and replace the old one, required that the available flash memory is twice the size of the firmware. That’s great, but how does the firmware get to the web server? And how do I make sure that only changed firmware gets updated? In the following post I will describe a workflow which will let you automagically update your devices with just a few clicks, once you have working setup.

NodeMCU_DEVKIT_1.0

 

What is Continuous Delivery?

That’s a good question. But lets start one step earlier: do you know what Continuous Integration is? In my day job I work as a Java developer and technical project lead in projects for big corporate companies in Switzerland. There we use systems that automatically build the code, run tests and create artefacts with every change of the source code in the code repository. This process is called Continuous Integration or CI, since it validates the integration of all code changes continuously. In the last 15 years this has become very much the standard in the Java world. The next step in the work pipeline evolution after CI is called Continuous Delivery (or CD). With a CD pipeline a team tries to treat every build as a potential candidate for release into production if it passes the build, the automatic unit tests, the automatic integration and acceptance tests. The tests must be so good that if an artefact passes all these tests nobody has to hesitate to push the last button and deploy them into production. Scary? Not there yet. It gets even more scary! There also a workflow called Continuous Deployment where the manual push of a button is not even needed anymore. Every build that passes the tests goes into production.

continuous-delivery-deployment-sm
Image from blog.crisp.se

 

Continuous Delivery for the ESP8266

We won’t go as far as Continuous Deployment in this post but I want to show you a nice workflow for Continuous Delivery. The workflow I’m describing here is using freely available tools assuming that you are working on an open source project. You still might use the ideas described here for closed source projects but then the tools are not free anymore. So what are the components in the workflow?

drawit-diagram
The Continuous Delivery process – Developer commits to Github. Travis builds commits. If the developer tags a build as a release Travis will upload the firmware to Github. The ESP’s will frequently check with the PHP script if they are up-to-date. The PHP script on the other hand will use the Github API to compare the ESP tag version with the latest release

 

I have to admit, it is a relatively complicated workflow. Especially the PHP script doesn’t look too pretty in the picture but a deficiency in the ESPhttpUpdate class made it necessary. So here are the different stages during development:

Stage 1 – Development – The developer on his computer writes the code (e.g. in the platformio IDE) and tests it locally either by uploading it over the serial connection or by the “local” or direct OTA update mechanism. You can by the way combine the two update mechanisms without problem but you might consider disabling it for the production release or your devices will be more vulnerable for attacks. When developer is happy with his work he commits the changes to Github.

Stage 2 – Continuous Integration – Now Travis will start doing his work and build the last commit. If you have written tests Travis execute them and improve your confidence that your code is working as expected.

Stage 3 – Tagging – Once the developer team is confident that the current version is ready for deployment to the production devices the last commit will be tagged as a release.

Stage 4 – Release Build – Tagging the branch caused Travis to start a new build. But this time Travis will create a binary artefact (the firmware) and upload it to Github and add it to the release

Stage 5 – ESP8266 Update Check – During all this time the devices where hungry for a new release. They frequently asked the PHP script if there was a version different from the one they are running on. Every single time the PHP script had checked with the Github API if there was a newer version and every time the answer was NO. But now there is a new release, a new tag and hence  a new firmware to be installed. The ESP’s download the firmware, replace the old one and do a restart. If everything went well they will be running the latest firmware until a newer version is found.

 

How to set up the workflow?

In this post I won’t explain you how to set up a Github account or how to connect Travis with your Github repository. You can find links for that at the end of the post. I chose to use the Platformio IDE but you could also use the Standard Arduino IDE. But for the example project I recommend to use the Platformio IDE.

As a first challenge the firmware has to be able to tell the PHP script with what version it was built. So how does the version get into tagged firmware? There are two build profiles in platformio.ini. One for the normal (local) nodemcu build and one for the travis build. The local one sets a content of the BUILD_TAG macro to “0.0.0”:

build_flags = -DBUILD_TAG=0.0.0

while the travis profile uses the $TRAVIS_TAG environment variable if available to pass the tag into the build or set it to “0.0.0” for a non-release build:

build_flags = !echo ‘-DBUILD_TAG=’${TRAVIS_TAG:-“0.0.0”}

This happens with every commit but only for release builds the $TRAVIS_TAG environment variable is set and only then the firmware will be uploaded to Github. The following line in .travis.yml makes sure that only tagged versions will be deployed back to Github:

deploy:
on:
repo: squix78/esp8266-ci-ota
all_branches: true
condition: $TRAVIS_TAG =~ ^[0-9]+\.[0-9]+\.[0-9]+$

The condition attribute checks that the tag version matches the regular expression. For a not very experienced C(++) developer I had a hard time to figure out how to pass in a String literal as macro during expansion phase. This code does it by using the # operator:

#define TEXTIFY(A) #A
#define ESCAPEQUOTE(A) TEXTIFY(A)

String buildTag = ESCAPEQUOTE(BUILD_TAG);

Let me know if you have more simple solution…

 

How to create a release?

On your Github repository root page (e.g. https://github.com/squix78/esp8266-ci-ota) you click on Releases > Draft New Release. Now in the Tag Version drop down you enter a version number matching the above condition, e.g. 0.0.15. You can either do this form master version or from one of the last commits. Then publish the Release.

Release Screen Github
Release Screen Github

 

The PHP script

It would have been nice if the production ESPs could have contacted the Github API directly. But there are two issues that made be use the intermediate PHP script:

  1. The ESPhttpUpdate currently cannot follow redirects. This is important since github hosts the release artefacts on Amazon AWS. But in the API JSON object the address points to github, so the http client has to follow a redirect to download the artefact.
  2. Github uses https for its API and will redirect you to it if you are trying plain HTTP. This means that you would have to know the SSL fingerprints of the github API server and the AWS hosting instance since this is required by the ESPs secure client interface. After all the ESPs don’t have a chain of trusted certificates stored somewhere. While the fingerprint of the github API might be stable, the redirection on Amazon AWS might not always use the same certificate.

So what does the script do? It connects to the github API and fetches a JSON object describing the latest release. I’m currently only interested in the tag name and the firmware download URL. The ESPs will send their firmware tag version with the update request and if the latest tag on Github and the one from the request are identical nothing will happen. But if they are different the script will fetch the binary from github (with a hidden redirection to Amazon) and proxy it to the ESP’s requesting it. You can find the php script in the server folder of the Github project.

 

The ESP8266 code

Have a look at the main class of the example directory. One important guide line for this kind of deployment is that your code should be free of secrets (e.g. for Wifi) or environment specific variables. Because the devices should be configurable in a end-user environment I’m using tzapu’s excellent WifiManager library which starts a captive portal behind a Wifi AP if the device cannot connect to an access point. This way the Wifi credentials are not part of the firmware but stored in the flash memory.

Then you need to include many header files for the OTA update to work. My example code doesn’t do anything but keeping itself up-to-date with the latest release version on github by checking every 60 seconds (yes, I’m very impatient;-)) if there is no update. In a real project you might consider doing these checks less frequently or even use MQTT to get a notification once a new version is available. If you cannot wait for the 60 seconds to pass you can also push the “Flash” button on a NodeMCU to trigger the check.

UPDATE 2016/06/05: Ivan from Platformio was so kind to show me some simplifications which are now part of the example project.

Conclusion

This post together with the example project (see resources below) shows you how you can set up a continuous delivery workflow for your ESP8266 devices. There are several steps involved in making this work but afterwards you will have a semi-automatic deployment to all the devices running your firmware. It took me many hours to get the workflow to work and to write this post. If you feel like I deserve a reward for it consider Teleporting a Beer  to me(see box below) or use one of the affiliate links around my blog. Thank you!

 

A word of warning: during my testing the NodeMCU devices sometimes hung after an update. This is a known issue of the NodeMCUs and should only happen after a serial upload and not after OTA. But if you plan to use this workflow or the OTA feature in general for production be sure that you do some testing.

 

Hardware Resources

 

Software Resources

Posted by squix78

26 comments

  1. Hello Dani,
    cool thing and a rather fascinating story! If I did not already have one kit I would order one…
    But one thing: On the label stuck on the bag is written “NodeMCU V3” but in the text description you write “NodeMCU V1”.
    Perhaps you can sync this?
    Best regards and successful business!
    Thomas

  2. Hi Daniel;

    Awesome! I got my kit and I loaded up the WeatherStation sketch and everything was working 🙂

    I then went on to add the DHT11 code and that’s when things stopped working. I suspect the firmware was over written or something bad … bricked I suspect.

    From the Arduino IDE …

    warning: espcomm_send_command: didn’t receive command response
    warning: espcomm_send_command(FLASH_DOWNLOAD_BEGIN) failed
    error: espcomm_upload_mem failed
    error: espcomm_upload_mem failed

    I then went on to try and reflash the NodeMCU firmware using ESP8266Flasher.exe … from there I get ..

    Note:Serial port connected.
    Note:Begin find ESP8266.
    Note:ESP8266 ACK success.
    Note:ESP8266 ACK success.
    Error:Set ESP8266 Address timeout.

    Similar problems directly from the esptool

    c:\Python27\Scripts>esptool.py –port com5 write_flash -fm dio 0x00000 nodemcu_integer_0.9.5_20150318.bin
    esptool.py v1.2.1
    Connecting…

    A fatal error occurred: Timed out waiting for packet header

    … in fiddling around the board is alive .. it does send this out the serial port at 75K baud… could it be a baud rate issue? I assume not as I understand it’s autobaud detecting.

    ets Jan 8 2013,rst cause:2, boot mode:(3,7)

    Questions –

    o Do you have any suggestions on next steps?

    o Could it be power? I gather power is important for FTDI boards but the NodeMCU has the 3v3 regulator and it’s running off a PC with “good” USB current .. and it did work fine before it did not 😉

    o I’ve had no success in pressing the RST and FLASH buttons — I as I understand them, the RST/CTS lines put the board into to bootloader mode at any rate .. thoughts?

    … thoughts?

    1. Hi David

      Here sone thoughts
      – try another USB cable, the can be reason for your symptoms
      – try another USB port or restart your PC (or Mac?) I had situations where my Mac would only show the device after a restart
      – unplug the DHT11. Maybe your wiring is wrong and you accidentally switched VCC and Ground. Not sure about the longterm effects of such a switch. Such a switch was actually the only way I could ever brick an ESP8266

      If these tips don’t help feel free to get back to me

      Cheers, Dani

      1. Thanks Daniel —

        I’ve tried multiple USB cables and multiple USB ports (Windows).

        I did go back to just the bare board .. no display and no DHT11.

        I’m now trying to see it the ESP AT command set even works … and no luck there yet. I do get this when I press the reset button so it is alive a least a bit 😉

        ets Jan 8 2013,rst cause:2, boot mode:(3,7)

        but nothing in response to AT ….

        ??

        Cheers David

        1. According to this post it might have to do something with your reset pin: http://www.esp8266.com/viewtopic.php?p=2096#p2112
          I’m not sure what that means;-).
          With which baud rate are you trying to flash the ESP? I your example from your last comment you did auto-sensing, right? Try to use a fixed lower rate like like 115200 or 57600. It often is something like the “-b” parameter.

          When you say you went back to the AT firmware: so did you manage to flash the AT firmware? After flashing the ESP8266 with the new firmware the AT binary has been replaced…

  3. Thanks Daniel — I just received another NodeMCU board as I’ve wasted enough time on the dead one 😉 If I manage to get the dead one working again, I’ll let you know.

    HNY, David

  4. Thanks Daniel, You sent me a trial kit and I purchased 2 others from Amazon. Some minor problems (outlined in Bryan Miller’s comments on Amazon). it works great but I still could use a fix to be able to get the screen to reverse from top to bottom your current variable didn’t work.

    I also located a free case plan that I had 3D printed and it fits and looks good. I found a local printer and picked it up today ($9) and it was a good fit, BUT the pins to hold the OLED were slightly out of register. I think I’ll have to cut them off and tape or glue the display in place. There is a ledge for it to fit in so i don’t think that will be a problem. I’m going to get another one printed for my local (Grand Rapids , MI USA) and I sent the plans off to my son in Atlanta, GA USA for him to use. I built the Squix kit with them over our Thanksgiving Holiday and now it will have a much better case than the plastic box it came in. Thanks for your good work and I did send off a tip. This case fits the slightly smaller SQUIX version of ESP8266 -12E.

    http://www.thingiverse.com/thing:1720314 for the plans

  5. I have my kit, and am following the directions, but even after 4 cable swaps, verifying all setting from the getting started guide, and checking my IDE for errors, I cannot seem to get a valid up load.

    The error I encounter is espcomm_upload_mem failed;

    Now I verified that the chip on the board shows a CP2102, so that is the USB driver I installed;
    I restarted my computer (windows 10) to ensure driver is loaded;
    I see a com3 show up when I apply power with the USB cable, so that is the port i chose;
    I loaded the libraries as indicated, and select the board nodeMCU 1.0

    I also updated from the 1.6 to the 1.8 arduino IDE too;

    I am lost to what else I can do to get the base sketch uploaded?

    1. Hi. From distance it indeed sounds like you did everything fine. On one hand it is a good sign that the COM port shows up is a very good sign, it means that you have the correct driver installed. On the other hand there are reports of problems with the CP2102 on Windows 10. Could you check please if you have the latest driver? See these forum posts:
      http://community.silabs.com/t5/Interface/cp210x-USB-to-UART-driver-problems-windows-10-code-31/td-p/146924
      http://community.silabs.com/t5/Interface/Windows-10-VCP-Driver-Now-Available/m-p/161629
      Feel free to contact me again if that should not solve the problem, please also let me know if it does so I can adjust the instructions…

      Regards,
      Daniel

      1. Daniel,

        I did download from that site the latest drivers, then I un-installed and re-installed just to be safe. That did not cure it, but I also think you have pointed me in the right direction, I duplicated the installation on an adjacent windows 10 computer and have the same result. Both computers can reach an Arduino UNO with no problem (from the same USB port), so I will continue to explore the device drivers. Thanks for your response! Once I resolve this, I will advise you for your documentation.

        Regards,

        Bruce

        1. Daniel, happy to report that the drivers with Windows 10were fine. Unfortunately the nodeMCU appears to be bad. I ordered a couple more of these and they came in todays’ mail; I plugged one in to the windows 10 system and it fired right up tonight. Now I can add the sensor and the oled and play, woo-hoo!

          Thanks again for your quick response!

          Bruce
          Kentucky, USA

  6. Only ever been using 8266 as a wifi board for an arduino. This sounds much more interesting.
    I really want to buy your handy box, but I live in Sweden. Any chance of finding way for us Europeans?

    1. Hi again,
      I now noticed that you also had your own shop. Order is now completed. Please disregard the above. Thanks.

    1. Very nice. I was hoping that the winds could be included and looks like you nailed it with an arrow and value. Still waiting for my color display in the mail so can’t wait to see how it works.

    2. Am having trouble compiling your code, keep getting the following error:
      Arduino: 1.8.1 (Windows 10), Board: “NodeMCU 1.0 (ESP-12E Module), 160 MHz, 115200, 4M (3M SPIFFS)”

      Build options changed, rebuilding all
      C:\Users\Danny\Documents\Weather Station Bodmer\esp8266-weather-station-color-masterbodmer\esp8266-weather-station-color-master\esp8266-weather-station-color\esp8266-weather-station-color.ino:44:58: fatal error: TFT_ILI9341_ESP.h: No such file or directory

      #include // Hardware-specific library

      ^

      compilation terminated.

      exit status 1
      Error compiling for board NodeMCU 1.0 (ESP-12E Module).

      This report would have more information with
      “Show verbose output during compilation”
      option enabled in File -> Preferences.

      I have all the libraries loaded, but still cannot determine why I am getting this compile error. I have set up the display wiring according to your example.
      Thanks,
      Danny

      1. Bodmer thanks for your post on the fork of Daniel weather station color. I must have had some garbage in my IDE folder, have corrected the problem and have your version up and running. Thanks again to you and Daniel.

  7. Hi, there! considering the shortage on IL9431 display (even Banggood) and some vendors sending IL9225 (176×220) to anyone, is someone worked/working or willing to work on porting the code to use such display even sacrificing some graphic features? I can start to work from zero but my time availability is so short that to the time of progress it will be too late or other more thrilling things will be out there.

    1. This is an issue introduced with an update to the Adafruit ILI9341 library I believe. Try reverting back to a previous version (1.0.2 worked I think). I ran into this a month or so ago and it took the better part of a day to track it down. The issue may have been fixed recently, but I haven’t tried the most recent release of the library.

  8. Funny. I just two boards of which I think are ESP-01s. I can’t understand the comment about shorts in a breadboard. Their 8-pin connectors have 2.54mm or 0.1″ pitch. So they’ll fit well into a common breadboard.

  9. I uploaded the World Clock demo an hour ago and noticed at 10:30pm CDT it was showing New York time as 15:30. What caused this? Where is the time zone (+ or – hours from UTC, and DST enable/disable) information set? Now as I type this at 11:40pm, it is now showing NY time as 00:40. What happened to correct the NY time (I assume the other world times were and still are correct)?

  10. Hello, I’ve tried the trick to power the screen through the digital pin and I’ve run into trouble : my wemos was not able to communicate by serial link anymore. Moreover, the final size is so small that the USB cable is heavier than it and I’ve difficulties to maintain the whole assembly on my desk, turned to my direction.

Leave a Reply