ESP8266: Continuous Delivery Pipeline – Push To Production

ESP32 Development Board WiFi+Bluetooth Ultra-Low Power Consumption Dual Cores ESP-32 ESP-32S Board
Integrated antenna and RF balun, power amplifier, low-noise amplifiers, filters, and power management module. The entire solution takes up the least amount of printed circuit board area. This board is used with 2.4 GHz dual-mode Wi-Fi and Bluetooth chips by TSMC 40nm low power technology, power and RF properties best, which is safe, reliable, and scalable to a variety of applications.

The integration of the ESP8266 framework into the Arduino IDE brought ease of use for development. This was further improved when firmwares could be pushed over the local network to the test devices with the direct over-the-air feature. But what if your ESP8266 devices are no longer in your own network? How can you then update all the devices which are running an older version of your firmware?

The Arduino/ESP8266 crew also has an answer for that: the ESPhttpUpdate class lets you download the latest firmware from a web server and replace the old one, required that the available flash memory is twice the size of the firmware. That’s great, but how does the firmware get to the web server? And how do I make sure that only changed firmware gets updated? In the following post I will describe a workflow which will let you automagically update your devices with just a few clicks, once you have working setup.

NodeMCU_DEVKIT_1.0

 

What is Continuous Delivery?

That’s a good question. But lets start one step earlier: do you know what Continuous Integration is? In my day job I work as a Java developer and technical project lead in projects for big corporate companies in Switzerland. There we use systems that automatically build the code, run tests and create artefacts with every change of the source code in the code repository. This process is called Continuous Integration or CI, since it validates the integration of all code changes continuously. In the last 15 years this has become very much the standard in the Java world. The next step in the work pipeline evolution after CI is called Continuous Delivery (or CD). With a CD pipeline a team tries to treat every build as a potential candidate for release into production if it passes the build, the automatic unit tests, the automatic integration and acceptance tests. The tests must be so good that if an artefact passes all these tests nobody has to hesitate to push the last button and deploy them into production. Scary? Not there yet. It gets even more scary! There also a workflow called Continuous Deployment where the manual push of a button is not even needed anymore. Every build that passes the tests goes into production.

continuous-delivery-deployment-sm
Image from blog.crisp.se

 

Continuous Delivery for the ESP8266

We won’t go as far as Continuous Deployment in this post but I want to show you a nice workflow for Continuous Delivery. The workflow I’m describing here is using freely available tools assuming that you are working on an open source project. You still might use the ideas described here for closed source projects but then the tools are not free anymore. So what are the components in the workflow?

drawit-diagram
The Continuous Delivery process – Developer commits to Github. Travis builds commits. If the developer tags a build as a release Travis will upload the firmware to Github. The ESP’s will frequently check with the PHP script if they are up-to-date. The PHP script on the other hand will use the Github API to compare the ESP tag version with the latest release

 

I have to admit, it is a relatively complicated workflow. Especially the PHP script doesn’t look too pretty in the picture but a deficiency in the ESPhttpUpdate class made it necessary. So here are the different stages during development:

Stage 1 – Development – The developer on his computer writes the code (e.g. in the platformio IDE) and tests it locally either by uploading it over the serial connection or by the “local” or direct OTA update mechanism. You can by the way combine the two update mechanisms without problem but you might consider disabling it for the production release or your devices will be more vulnerable for attacks. When developer is happy with his work he commits the changes to Github.

Stage 2 – Continuous Integration – Now Travis will start doing his work and build the last commit. If you have written tests Travis execute them and improve your confidence that your code is working as expected.

Stage 3 – Tagging – Once the developer team is confident that the current version is ready for deployment to the production devices the last commit will be tagged as a release.

Stage 4 – Release Build – Tagging the branch caused Travis to start a new build. But this time Travis will create a binary artefact (the firmware) and upload it to Github and add it to the release

Stage 5 – ESP8266 Update Check – During all this time the devices where hungry for a new release. They frequently asked the PHP script if there was a version different from the one they are running on. Every single time the PHP script had checked with the Github API if there was a newer version and every time the answer was NO. But now there is a new release, a new tag and hence  a new firmware to be installed. The ESP’s download the firmware, replace the old one and do a restart. If everything went well they will be running the latest firmware until a newer version is found.

 

How to set up the workflow?

In this post I won’t explain you how to set up a Github account or how to connect Travis with your Github repository. You can find links for that at the end of the post. I chose to use the Platformio IDE but you could also use the Standard Arduino IDE. But for the example project I recommend to use the Platformio IDE.

As a first challenge the firmware has to be able to tell the PHP script with what version it was built. So how does the version get into tagged firmware? There are two build profiles in platformio.ini. One for the normal (local) nodemcu build and one for the travis build. The local one sets a content of the BUILD_TAG macro to “0.0.0”:

build_flags = -DBUILD_TAG=0.0.0

while the travis profile uses the $TRAVIS_TAG environment variable if available to pass the tag into the build or set it to “0.0.0” for a non-release build:

build_flags = !echo ‘-DBUILD_TAG=’${TRAVIS_TAG:-“0.0.0”}

This happens with every commit but only for release builds the $TRAVIS_TAG environment variable is set and only then the firmware will be uploaded to Github. The following line in .travis.yml makes sure that only tagged versions will be deployed back to Github:

deploy:
on:
repo: squix78/esp8266-ci-ota
all_branches: true
condition: $TRAVIS_TAG =~ ^[0-9]+\.[0-9]+\.[0-9]+$

The condition attribute checks that the tag version matches the regular expression. For a not very experienced C(++) developer I had a hard time to figure out how to pass in a String literal as macro during expansion phase. This code does it by using the # operator:

#define TEXTIFY(A) #A
#define ESCAPEQUOTE(A) TEXTIFY(A)

String buildTag = ESCAPEQUOTE(BUILD_TAG);

Let me know if you have more simple solution…

 

How to create a release?

On your Github repository root page (e.g. https://github.com/squix78/esp8266-ci-ota) you click on Releases > Draft New Release. Now in the Tag Version drop down you enter a version number matching the above condition, e.g. 0.0.15. You can either do this form master version or from one of the last commits. Then publish the Release.

Release Screen Github
Release Screen Github

 

The PHP script

It would have been nice if the production ESPs could have contacted the Github API directly. But there are two issues that made be use the intermediate PHP script:

  1. The ESPhttpUpdate currently cannot follow redirects. This is important since github hosts the release artefacts on Amazon AWS. But in the API JSON object the address points to github, so the http client has to follow a redirect to download the artefact.
  2. Github uses https for its API and will redirect you to it if you are trying plain HTTP. This means that you would have to know the SSL fingerprints of the github API server and the AWS hosting instance since this is required by the ESPs secure client interface. After all the ESPs don’t have a chain of trusted certificates stored somewhere. While the fingerprint of the github API might be stable, the redirection on Amazon AWS might not always use the same certificate.

So what does the script do? It connects to the github API and fetches a JSON object describing the latest release. I’m currently only interested in the tag name and the firmware download URL. The ESPs will send their firmware tag version with the update request and if the latest tag on Github and the one from the request are identical nothing will happen. But if they are different the script will fetch the binary from github (with a hidden redirection to Amazon) and proxy it to the ESP’s requesting it. You can find the php script in the server folder of the Github project.

 

The ESP8266 code

Have a look at the main class of the example directory. One important guide line for this kind of deployment is that your code should be free of secrets (e.g. for Wifi) or environment specific variables. Because the devices should be configurable in a end-user environment I’m using tzapu’s excellent WifiManager library which starts a captive portal behind a Wifi AP if the device cannot connect to an access point. This way the Wifi credentials are not part of the firmware but stored in the flash memory.

Then you need to include many header files for the OTA update to work. My example code doesn’t do anything but keeping itself up-to-date with the latest release version on github by checking every 60 seconds (yes, I’m very impatient;-)) if there is no update. In a real project you might consider doing these checks less frequently or even use MQTT to get a notification once a new version is available. If you cannot wait for the 60 seconds to pass you can also push the “Flash” button on a NodeMCU to trigger the check.

UPDATE 2016/06/05: Ivan from Platformio was so kind to show me some simplifications which are now part of the example project.

Conclusion

This post together with the example project (see resources below) shows you how you can set up a continuous delivery workflow for your ESP8266 devices. There are several steps involved in making this work but afterwards you will have a semi-automatic deployment to all the devices running your firmware. It took me many hours to get the workflow to work and to write this post. If you feel like I deserve a reward for it consider Teleporting a Beer  to me(see box below) or use one of the affiliate links around my blog. Thank you!

 

A word of warning: during my testing the NodeMCU devices sometimes hung after an update. This is a known issue of the NodeMCUs and should only happen after a serial upload and not after OTA. But if you plan to use this workflow or the OTA feature in general for production be sure that you do some testing.

 

Hardware Resources

 

Software Resources

Posted by squix78

26 comments

  1. Hi Daniel,

    Firstly, thanks a lot for the great article!!! 🙂

    A few comments:

    1. You can use `platformio run` command instead of `platformio ci`. In this case, the firmware will be located in `.pioenvs` directory.

    2. Instead of these lines:

    > – platformio lib install 567 562
    > – export PLATFORMIO_BUILD_FLAGS=”-D BUILD_TIMESTAMP=”`date +%s`
    > – echo “${TRAVIS_TAG}”
    >- echo “const char * BUILD_TAG = \”${TRAVIS_TAG}\”;” > src/version.h

    Just create environments in `platformio.ini`:

    “`ini

    [env:nodemcuv2]
    platform = espressif
    framework = arduino
    board = nodemcuv2
    upload_speed = 921600
    lib_install = 567, 562
    build_flags = -DBUILD_TIMESTAMP=$UNIX_TIME -DBUILD_TAG=0.0.0

    [env:nodemcuv2_deploy]
    platform = espressif
    framework = arduino
    board = nodemcuv2
    lib_install = 567, 562
    build_flags = !echo ‘-DBUILD_TIMESTAMP=$UNIX_TIME -DBUILD_TAG=’$TRAVIS_TAG
    “`

    The final .travis.yml
    “`yml
    language: python
    python:
    – ‘2.7’
    sudo: false
    cache:
    directories:
    – ~/.platformio

    install:
    – pip install -U platformio
    script:
    – platformio run -e nodemcuv2_deploy

    deploy:
    on:
    repo: squix78/esp8266-ci-ota
    all_branches: true
    condition: $TRAVIS_TAG =~ ^[0-9]+\.[0-9]+\.[0-9]+$

    skip_cleanup: true
    provider: releases
    overwrite: true
    api_key:
    secure: DeLq8sczTsEAo3qi2ztSYgvaeidenvJherPCrceoknNuBs8SEsb8czGxBDPKt7lzK155Kcsfq/zo1VtUpVQ8HsalMQJABv9Li6t1uBc0m+mSLGggxw9woGzHv8FfnJdqeBD3YXgX8Qrmlnqp5QdIDZuyRfOxeF4NFwWZ89aOxMO8Kk363ykvQg5mq8mbqzHJOtCQCkHOEjklhcMTaHZgW/mlt7jBbuX/RM5jGTTAEOIKU/gmVvLo1LQMgsvJ2BCSLMrkEZmfxPncI1xTf26X7aSu01krn6oaROgJtFpTWEmSWHDHOJwngPC36SuaVtyd5WHE8u6+ZkDlfZsvDShq7Iap6vzqZwrTM3NvNyAnLWbE4YLLBDkuHMojyhy5MkvFLjVfEd4wkvy4JOnCL7IhJjcM4OWEXykZv80X/XBqT3XcZYYaorXs1x7rVUCsMYIxythVRaBzsP61NMFIEbLxV0maTYbuLMWX2HmForumMkk6SD7NdbaONZUaKRaG5G/QGWGHrOfVwBWCL/0KEZl6mKqkfadAQI4GWD77MOWn5JNmbM6bVE2hD/oVkSoao1XdY2N0flDEQQuVu3GHIweN3f5kudJ/bS04US1iyWw3t3PYtNOaQrUFtoUMd88/Q6hzzbdYuVVvjBH4Lsni4i/svMu//7URWXGgmcjm14Fr8xU=
    file: “.pioenvs/nodemcuv2_deploy/firmware.bin”
    “`

    P.S: I didn’t test it. However, I hope you understand my idea.

    • Hi Ivan. Thanks a lot for the hints! I will try to integrate them. Your support is excellent like always!

  2. Hi,
    how do you write back to the version.h file?
    As far as I know travis has no permissions to write back to the repo.

    • You are right, I’m not writing the changes in the version file back to the repo and in that matter it is a bit ugly, since the released version is not 100% identical to the actual firmware. The version.h file only gets changed in the checkout in the Travis environment. I decided that this compromise was acceptable…

      UPDATE: With Ivan’s suggestions I could get rid of that contamination of the source code by using the BUILD_TAG macro. This means that the source code now really remains as committed, but the build is parametrized by feeding in the BUILD_TAG

  3. Please, I would like to ask some questions about the PlatformIO:

    1. Try to put libraries in the lib folder and does not recognize. It only works if put in the same folder as the source.
    2. How to go saving multiple versions of the same project in the same folder? It gives error to let more than one.

    Thanks

  4. Nice work! I am trying to replicate your Travis build and deploy mechanism.

    Here is my travis file: https://github.com/openenergymonitor/emonpi/blob/master/.travis.yml

    I think I’m almost there but I’m getting error in the travis log:

    /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/octokit-4.3.0/lib/octokit/response/raise_error.rb:16:in `on_complete’: GET https://api.github.com/user: 401 – Bad credentials // See: https://developer.github.com/v3 (Octokit::Unauthorized)

    https://travis-ci.org/openenergymonitor/emonpi/builds/136591535

    I have double checked that the token I’m using is valid created here: https://help.github.com/articles/creating-an-access-token-for-command-line-use/

    then hashed using ruby gem $ travis encrypt GH_TOKEN=XXXX

    Please could you provide some more details as to how you generated your token and if any extra steps are required? E.g. what access scopes are required

    Thanks a lot, amazing guide.

    • Good point. I guess you are the first who is actually trying:-). From the top of my head I used “travis setup releases” which overwrote the deployment section in the travis file. Take the key from there then revert to the default…

  5. Hi Squix78,
    I have a OLED 1.3″ SH1106 chip display and if I change the commands in the display routine, your weather station works perfectly:

    void SSD1306::display(void) {
    for (uint8_t k=0; k<8; k++) {
    sendCommand(0xB0+k);//set page addressSSD_Data_Mode;
    sendCommand(0x02) ;//set lower column address
    sendCommand(0x10) ;//set higher column address

    for (uint16_t i=0; i< 128; i++) {
    // send a bunch of data in one xmission
    //Wire.begin(mySda, mySdc);
    Wire.beginTransmission(myI2cAddress);
    Wire.write(0x40);
    for (uint8_t x=0; x<16; x++) {
    Wire.write(buffer[128*k+i]);
    i++;
    }
    i–;
    yield();
    Wire.endTransmission();
    }
    }
    }

    Is there a way that the SSD1306 library can be modified to include this fix for the SH1106 chip (I2C)?

  6. YEAH! Thanks a lot. $ travis setup releases did the trick 😀

    However I’ve now got an error ` No such file or directory – .pioenvs/emonpi_deploy/firmware.hex `. However I can see that this file is created by travis. I double checked that skip_cleanup: true.

    Are there any further steps required?

    https://travis-ci.org/openenergymonitor/emonpi/builds/136748825

    avr-objcopy -O ihex -R .eeprom .pioenvs/emonpi_deploy/firmware.elf .pioenvs/emonpi_deploy/firmware.hex
    AVR Memory Usage
    —————-
    Device: atmega328p
    Program: 18888 bytes (57.6% Full)
    (.text + .data + .bootloader)
    Data: 947 bytes (46.2% Full)
    (.data + .bss + .noinit)
    ========================= [SUCCESS] Took 3.39 seconds =========================
    The command “platformio run -d firmware -e emonpi_deploy” exited with 0.
    cache.2
    store build cache
    0.00s
    0.87snothing changed, not updating cache
    dpl.0
    Fetching: dpl-1.8.16.gem (100%)
    Successfully installed dpl-1.8.16
    1 gem installed
    dpl.1
    Installing deploy dependencies
    Fetching: addressable-2.4.0.gem (100%)
    Successfully installed addressable-2.4.0
    Fetching: multipart-post-2.0.0.gem (100%)
    Successfully installed multipart-post-2.0.0
    Fetching: faraday-0.9.2.gem (100%)
    Successfully installed faraday-0.9.2
    Fetching: sawyer-0.7.0.gem (100%)
    Successfully installed sawyer-0.7.0
    Fetching: octokit-4.3.0.gem (100%)
    Successfully installed octokit-4.3.0
    5 gems installed
    Fetching: mime-types-2.99.2.gem (100%)
    Successfully installed mime-types-2.99.2
    1 gem installed
    dpl.2
    Preparing deploy
    Logged in as Glyn Hudson
    Deploying to repo: openenergymonitor/emonpi
    Current tag is: 0.1.1
    dpl.3
    Deploying application
    /home/travis/.rvm/gems/ruby-1.9.3-p551/gems/octokit-4.3.0/lib/octokit/client/releases.rb:86:in `initialize’: No such file or directory – .pioenvs/emonpi_deploy/firmware.hex (Errno::ENOENT)

    • I don’t see a platformio.ini in your repo. I’m just checking this on my smartphone but I think there is some magic happening in that file…

  7. Platformio.ini is in the firmware folder, that’s the missing part, I just needed to add firmware into the hex file path. Working now 😀

    Thanks again for your awesome blog. Now to setup php script and esp…. Exciting weekend ahead 🙂

  8. I think I’ve almost managed to re-create your continuous delivery workflow, however I’m getting a
    “Verify bin header failed” error from the ESP when update is attempted. I hope you don’t mind, I’ve posted some further info as an issue on your project GittHub: https://github.com/squix78/esp8266-ci-ota/issues/2. Although I’m sure the ‘issue’ is my fault rather than your code! Any insight would be apprichated. Thanks! Have a good weekend 🙂

    • Hi Glyn. Could you find an answer to your issues in the meantime? I would guess that the message means that your downloaded file doesn’t look like a firmware. I tried to find the build on travis that uploads the firmware to see if you are choosing the correct file but I couldn’t find it. Maybe your uploading the wrong file? I compared the bin downloaded by your php script and downloaded from the release and from distance they look the same…

    • Hi Glyn. For the record and to help others (as duplicate to my answer on the github issue): I think I found the problem: your php script contains an empty line at the beginning of the script. Remove it and you should be good to go…
      Kind regards,
      Daniel

Leave a Reply