ESP8266: Continuous Delivery Pipeline – Push To Production

ESP32 Development Board WiFi+Bluetooth Ultra-Low Power Consumption Dual Cores ESP-32 ESP-32S Board
Integrated antenna and RF balun, power amplifier, low-noise amplifiers, filters, and power management module. The entire solution takes up the least amount of printed circuit board area. This board is used with 2.4 GHz dual-mode Wi-Fi and Bluetooth chips by TSMC 40nm low power technology, power and RF properties best, which is safe, reliable, and scalable to a variety of applications.

The integration of the ESP8266 framework into the Arduino IDE brought ease of use for development. This was further improved when firmwares could be pushed over the local network to the test devices with the direct over-the-air feature. But what if your ESP8266 devices are no longer in your own network? How can you then update all the devices which are running an older version of your firmware?

The Arduino/ESP8266 crew also has an answer for that: the ESPhttpUpdate class lets you download the latest firmware from a web server and replace the old one, required that the available flash memory is twice the size of the firmware. That’s great, but how does the firmware get to the web server? And how do I make sure that only changed firmware gets updated? In the following post I will describe a workflow which will let you automagically update your devices with just a few clicks, once you have working setup.

NodeMCU_DEVKIT_1.0

 

What is Continuous Delivery?

That’s a good question. But lets start one step earlier: do you know what Continuous Integration is? In my day job I work as a Java developer and technical project lead in projects for big corporate companies in Switzerland. There we use systems that automatically build the code, run tests and create artefacts with every change of the source code in the code repository. This process is called Continuous Integration or CI, since it validates the integration of all code changes continuously. In the last 15 years this has become very much the standard in the Java world. The next step in the work pipeline evolution after CI is called Continuous Delivery (or CD). With a CD pipeline a team tries to treat every build as a potential candidate for release into production if it passes the build, the automatic unit tests, the automatic integration and acceptance tests. The tests must be so good that if an artefact passes all these tests nobody has to hesitate to push the last button and deploy them into production. Scary? Not there yet. It gets even more scary! There also a workflow called Continuous Deployment where the manual push of a button is not even needed anymore. Every build that passes the tests goes into production.

continuous-delivery-deployment-sm
Image from blog.crisp.se

 

Continuous Delivery for the ESP8266

We won’t go as far as Continuous Deployment in this post but I want to show you a nice workflow for Continuous Delivery. The workflow I’m describing here is using freely available tools assuming that you are working on an open source project. You still might use the ideas described here for closed source projects but then the tools are not free anymore. So what are the components in the workflow?

drawit-diagram
The Continuous Delivery process – Developer commits to Github. Travis builds commits. If the developer tags a build as a release Travis will upload the firmware to Github. The ESP’s will frequently check with the PHP script if they are up-to-date. The PHP script on the other hand will use the Github API to compare the ESP tag version with the latest release

 

I have to admit, it is a relatively complicated workflow. Especially the PHP script doesn’t look too pretty in the picture but a deficiency in the ESPhttpUpdate class made it necessary. So here are the different stages during development:

Stage 1 – Development – The developer on his computer writes the code (e.g. in the platformio IDE) and tests it locally either by uploading it over the serial connection or by the “local” or direct OTA update mechanism. You can by the way combine the two update mechanisms without problem but you might consider disabling it for the production release or your devices will be more vulnerable for attacks. When developer is happy with his work he commits the changes to Github.

Stage 2 – Continuous Integration – Now Travis will start doing his work and build the last commit. If you have written tests Travis execute them and improve your confidence that your code is working as expected.

Stage 3 – Tagging – Once the developer team is confident that the current version is ready for deployment to the production devices the last commit will be tagged as a release.

Stage 4 – Release Build – Tagging the branch caused Travis to start a new build. But this time Travis will create a binary artefact (the firmware) and upload it to Github and add it to the release

Stage 5 – ESP8266 Update Check – During all this time the devices where hungry for a new release. They frequently asked the PHP script if there was a version different from the one they are running on. Every single time the PHP script had checked with the Github API if there was a newer version and every time the answer was NO. But now there is a new release, a new tag and hence  a new firmware to be installed. The ESP’s download the firmware, replace the old one and do a restart. If everything went well they will be running the latest firmware until a newer version is found.

 

How to set up the workflow?

In this post I won’t explain you how to set up a Github account or how to connect Travis with your Github repository. You can find links for that at the end of the post. I chose to use the Platformio IDE but you could also use the Standard Arduino IDE. But for the example project I recommend to use the Platformio IDE.

As a first challenge the firmware has to be able to tell the PHP script with what version it was built. So how does the version get into tagged firmware? There are two build profiles in platformio.ini. One for the normal (local) nodemcu build and one for the travis build. The local one sets a content of the BUILD_TAG macro to “0.0.0”:

build_flags = -DBUILD_TAG=0.0.0

while the travis profile uses the $TRAVIS_TAG environment variable if available to pass the tag into the build or set it to “0.0.0” for a non-release build:

build_flags = !echo ‘-DBUILD_TAG=’${TRAVIS_TAG:-“0.0.0”}

This happens with every commit but only for release builds the $TRAVIS_TAG environment variable is set and only then the firmware will be uploaded to Github. The following line in .travis.yml makes sure that only tagged versions will be deployed back to Github:

deploy:
on:
repo: squix78/esp8266-ci-ota
all_branches: true
condition: $TRAVIS_TAG =~ ^[0-9]+\.[0-9]+\.[0-9]+$

The condition attribute checks that the tag version matches the regular expression. For a not very experienced C(++) developer I had a hard time to figure out how to pass in a String literal as macro during expansion phase. This code does it by using the # operator:

#define TEXTIFY(A) #A
#define ESCAPEQUOTE(A) TEXTIFY(A)

String buildTag = ESCAPEQUOTE(BUILD_TAG);

Let me know if you have more simple solution…

 

How to create a release?

On your Github repository root page (e.g. https://github.com/squix78/esp8266-ci-ota) you click on Releases > Draft New Release. Now in the Tag Version drop down you enter a version number matching the above condition, e.g. 0.0.15. You can either do this form master version or from one of the last commits. Then publish the Release.

Release Screen Github
Release Screen Github

 

The PHP script

It would have been nice if the production ESPs could have contacted the Github API directly. But there are two issues that made be use the intermediate PHP script:

  1. The ESPhttpUpdate currently cannot follow redirects. This is important since github hosts the release artefacts on Amazon AWS. But in the API JSON object the address points to github, so the http client has to follow a redirect to download the artefact.
  2. Github uses https for its API and will redirect you to it if you are trying plain HTTP. This means that you would have to know the SSL fingerprints of the github API server and the AWS hosting instance since this is required by the ESPs secure client interface. After all the ESPs don’t have a chain of trusted certificates stored somewhere. While the fingerprint of the github API might be stable, the redirection on Amazon AWS might not always use the same certificate.

So what does the script do? It connects to the github API and fetches a JSON object describing the latest release. I’m currently only interested in the tag name and the firmware download URL. The ESPs will send their firmware tag version with the update request and if the latest tag on Github and the one from the request are identical nothing will happen. But if they are different the script will fetch the binary from github (with a hidden redirection to Amazon) and proxy it to the ESP’s requesting it. You can find the php script in the server folder of the Github project.

 

The ESP8266 code

Have a look at the main class of the example directory. One important guide line for this kind of deployment is that your code should be free of secrets (e.g. for Wifi) or environment specific variables. Because the devices should be configurable in a end-user environment I’m using tzapu’s excellent WifiManager library which starts a captive portal behind a Wifi AP if the device cannot connect to an access point. This way the Wifi credentials are not part of the firmware but stored in the flash memory.

Then you need to include many header files for the OTA update to work. My example code doesn’t do anything but keeping itself up-to-date with the latest release version on github by checking every 60 seconds (yes, I’m very impatient;-)) if there is no update. In a real project you might consider doing these checks less frequently or even use MQTT to get a notification once a new version is available. If you cannot wait for the 60 seconds to pass you can also push the “Flash” button on a NodeMCU to trigger the check.

UPDATE 2016/06/05: Ivan from Platformio was so kind to show me some simplifications which are now part of the example project.

Conclusion

This post together with the example project (see resources below) shows you how you can set up a continuous delivery workflow for your ESP8266 devices. There are several steps involved in making this work but afterwards you will have a semi-automatic deployment to all the devices running your firmware. It took me many hours to get the workflow to work and to write this post. If you feel like I deserve a reward for it consider Teleporting a Beer  to me(see box below) or use one of the affiliate links around my blog. Thank you!

 

A word of warning: during my testing the NodeMCU devices sometimes hung after an update. This is a known issue of the NodeMCUs and should only happen after a serial upload and not after OTA. But if you plan to use this workflow or the OTA feature in general for production be sure that you do some testing.

 

Hardware Resources

 

Software Resources

Posted by squix78

40 comments

  1. Observation, you could pull the weather icons from the Weatherunderground server rather than host them yourself. I think they do-not apply any read quotas.

    Did embedding the icons in flash prove to be too great an overhead?

    1. Hi David. If I’m not mistaken these icons are from a 3rd party (which reminds me to put their credits up). So I’d have to host it somewhere anyway. I currently also only use uncompressed bmp format and I doubt that any reasonable person still has them around for normal use;-). And then: downloading has the advantage to be easier to change compared to embedding in the firmware… But it was a design decision. Comparing pros and cons…

    1. Hi Stephen. I plan to upgrade the plane spotter project to the color screen, so yes, it’s definitely possible. Just a question of time:-)

  2. Hi,

    I have a working esp8266 weather but the small LCD only.
    im trying this one as i have the exact LCD lying around. i saw the “settings.h” where values need to be changed. (i.e. wundeground API, country, city, etc).. the only one that i was not sure and I did not see any explanation was the ThingSpeak value.

    May i ask what is the purpose of the Thingspeak value? i tried to view it but I think its a private channel.

    Anyway, i uploaded it on NodeMCU and when i inspect the Serial monitor, i can get to the part where it downloads the pictures.
    Here is the last bottom lines on the Serial.
    There is a line that says “connection failed.”…. Im guessing it has something to do with the Thingspeak..or maybe something else..

    …….(lots of downloding lines here…………..)
    Downloading http://www.squix.org/blog/moonphase_L19.bmp and saving as /moon19.bmp
    Downloading http://www.squix.org/blog/moonphase_L20.bmp and saving as /moon20.bmp
    Downloading http://www.squix.org/blog/moonphase_L21.bmp and saving as /moon21.bmp
    Downloading http://www.squix.org/blog/moonphase_L22.bmp and saving as /moon22.bmp
    connection failed
    connection failed
    connection failed
    connection failed

    Loading image ‘unknown.bmp’
    File not found
    Loading image ‘/mini/unknown.bmp’
    File not found
    Loading image ‘/mini/unknown.bmp’
    File not found
    Loading image ‘/mini/unknown.bmp’
    File not found
    Loading image ‘/moon0.bmp’
    File not foundconnection failed
    connection failed
    connection failed
    connection failed

    1. Hi Glenn,
      The purpose of the Thingspeak channel is to display (say) your own local temperature and humidity. So I have an ESP8266 and BME280 sensor running that reports every 10-mins to thingspeak the current temperature and humidity in my garden. Therefore I provide the keys for my channel and it gets reported on the weather display. Ideally there would be a software (compile time) switch to remove that option.

  3. Please ignore the comment below.
    I was able to solve it. Apparently, the wifimanager didn’t connect successfully to my router, so i had to manually connect and put the static SSID and password.

    So its working now, yey! i can see the pictures and the icons properly.

    im still dumbfounded with the Thingspeak channel in the settings.h.
    May i know if that is really needed?

    1. Hi Glenn, I’m a program newbie and met same message from serial port, but I struggled for long time but cant overcome this issue, do you mind give me some suggestion?

      Thanks!

  4. Hi,
    Is there any chance regarding tutorial how to call bitmaps using WiFi connection?
    I want to prepare own set of icons but I don’t know how to use it in your script 🙁
    As well I have a question reagrding screen refreshing, as I see that is the issue when the time is updated and some pixels are not removed?
    Thanks in advance for help.
    Piotr

  5. Very impressive weather station, works great!
    If it could automatically update summer/winter-time, it would be absolutely perfect.
    Have fun with the beer 😉

  6. I was able to get this running, but wanted to mention a small issue I ran into. (Sounds like the same one Mike McRoberts references.) The bitmaps require more than 1M of SPIFFS, so if you were like me and had selected only 1M at compile time, you run out of space to download all of the bitmaps, and many of the moon-phase bitmaps will fail to download, or will be perhaps truncated or corrupted. I went back and changed to 4M (3M SPIFFS) in the IDE and was able to solve this. Still having a bit of overlay issues with the time updates, but as Daniel mentioned, the code may still need a little work.

    Thanks for all the effort on this…now to add some of my own tweaks!

  7. Hi Daniel,
    i tried to flash the script but i will receiving all the time the following error:
    *******************************
    In file included from esp8266-weather-station-color.ino:34:0:
    C:\Users\Jens\Documents\Arduino\libraries\esp8266-weather-station-color-master/WebResource.h:25:31: fatal error: ESP8266HTTPClient.h: No such file or directory
    #include
    ^
    compilation terminated.
    Fehler beim Kompilieren.
    *******************************
    I added the ESP8266HTTPClient-library to the Arduino 1.6.5 but there is still the same error.
    Do you have any idea what i can make that the script will runing?
    Jens

  8. Unfortunately Banggood.com are no longer stocking the 2.2 Inch Serial TFT SPI Display ILI9341 display! Can it be obtained from another supplier? Or can an alternative be used?

    I would like a bigger screen for my PlaneSpotter project so that I can enclose everything in a case with the lcd screen acting as a box top : )

  9. Dani, excellent! Built this over the weekend (preparing to teach another IoT class this week) and almost worked flawlessly (2.4″ TFT + Wemos D1 Mini). Couldn’t get the WifiManager to connect, tried a few different things then finally went manual mode. The 199: timeClient.updateTime(); would sometimes hang, changed the .h to use “us.pool.ntp.org” instead; didn’t do a lot of troubleshooting, was more in quick-fix mode. Maybe later this month I’ll try incorporating Neptune’s daylight savings code back into this (thanks btw for the link to that). PS will try to get some pictures with the kids doing projects over the session the next two weeks. Regards, Mike

    1. Mike, you stated that you changed the .h to use “us.pool.ntp.org”. What is the path/filename to that .h file?

  10. Dani,

    I’ve been adding some of my own enhancements to your original code for several weeks now, and thought I’d share some of my changes in case anyone was interested. I’ve posted these on GitHub if you’re interested…https://github.com/fowlerk/ESP8266-Color-Weather-Station-v8 .
    This version includes many enhancements to the displays (see the README on GitHub), along with Neptune’s enhancements for daylight savings time. The code could stand some clean-up, but otherwise seems to work well.

    Let me know what you think…I’ll post some photos of the displays when I get a chance.

    Keith Fowler

    1. Hi Keith

      You put some really nice things into the code! I knew that the WifiManager had this configuration UI feature but never got around to use it in my code. Now with your code I have a very nice example to use it;-). Thank you a lot for sharing! I just posted your fotos on Twitter and linked to your repository, I hope you don’t mind.

      I am working ATM on a 2.4″ display with touch. This would allow to switch screens or similar…

      Dani

      1. Dani…I don’t mind at all, in fact I’m honored. One thing to note, which I pointed out in the README on the repo, is that I have added a few fields over the past few months that may (or may not) have made it into the Weather Station library for the Weather Underground functions. Not sure how exactly to go about getting those into the master. (You may even be able to spot these on some of the displays.) I did not put those library changes up on the repo.

        I’d also had my eyes on those touch-screens and had similar thoughts on using them to control the screen displays. I’m anxious to see what you end up doing with it. Your projects always give me more ideas to try ;).

  11. Hey Dani and Keith, I just tried this out a few days ago and love it. Just wanted to confirm that when I tried to use Keiths’ version, it failed to compile due to the changes in the Weather_Station library. I would really like to try out this version if one of you could point me at the correct library.
    This was my first project with an esp8266 and I am really surprised at how powerful and easy it is to use.

    Kevin

    1. Kevin…yes, sorry, I knew that would bite someone. I really wanted to get the changes merged into Dani’s master library rather than start off on another branch. In the interim, I have uploaded the two changed library files for the Weather Underground components to a separate folder on the repo if you want to download those to (temporarily) replace the original ones in the Weather Station library. That should allow it to compile. Let me know if that doesn’t work for you. Hopefully I’ll figure out the right (permanent) solution to address the changes soon.

      Keith

      1. Thanks for the update Keith,

        I got the new code and everything works great. Now I just have to figure out how to make it work with a bigger screen, maybe 7 inch. My eyes ain’t what they used to be. 🙂

        Kevin

        1. Glad you got it working Kevin…and I know what you mean about the old eyes! I’m right there with you. I’m torn between the compact size and being able to read all the data displayed without my glasses ;). Let me know if you find a bigger option that works well for a reasonable cost.

  12. Dani…just made a pull request on your repository to add some of the changes I had made to your Weather Underground library. Let me know if you have questions or issues.

  13. Dani, I just made another pull request for some new updates to the Weather Underground client library. These include changes to support some additional enhancements I have been working on recently that display weather alerts (as per the WU API call “/alerts”), as well as the addition of probability of precipitation (PoP). I have uploaded a new version to my repository on GitHub that shows these additional enhancements (see the README for further details), along with screenshots.

    As always, let me know if you have any questions or issues, and best wishes for the upcoming holidays!

  14. Hi Daniel. I don’t like the Ardunid IDE because of the lack of debugging with breakpoints. Therefore, I’ve just started with VisualGDB and JLink debugger and ESP8266. I have quite some difficulties.

    Do you have any experiences with VisualGDB?

    By the way: I am situated near Zurich and I like your projects! Great work!

  15. I have been having a lot of fun working through the examples with your weatherstation library. I bought your kit, which was an easy way to get started with the ESP8266.
    I am working through the WeatherStationDemoExtendedDST, for some reason updateData is not firing every 10 minutes. updateDHT is working fine, fires every minute.
    I think this is where the problem is, it isn’t setting readyForWeatherUpdate to true. However, it looks exactly the same as your other examples.
    if (readyForWeatherUpdate && ui.getUiState()->frameState == FIXED) {
    updateData(&display);
    }
    Thanks for any help.

  16. Figured it out. Apparently you can’t attach two events to a single ticker? I created two tickers, and now it works.
    tickerUpdateData.attach(UPDATE_INTERVAL_SECS, setReadyForWeatherUpdate);
    tickerUpdateSensor.attach(60, setReadyForDHTUpdate);

Leave a Reply