The integration of the ESP8266 framework into the Arduino IDE brought ease of use for development. This was further improved when firmwares could be pushed over the local network to the test devices with the direct over-the-air feature. But what if your ESP8266 devices are no longer in your own network? How can you then update all the devices which are running an older version of your firmware?
The Arduino/ESP8266 crew also has an answer for that: the ESPhttpUpdate class lets you download the latest firmware from a web server and replace the old one, required that the available flash memory is twice the size of the firmware. That’s great, but how does the firmware get to the web server? And how do I make sure that only changed firmware gets updated? In the following post I will describe a workflow which will let you automagically update your devices with just a few clicks, once you have working setup.
What is Continuous Delivery?
That’s a good question. But lets start one step earlier: do you know what Continuous Integration is? In my day job I work as a Java developer and technical project lead in projects for big corporate companies in Switzerland. There we use systems that automatically build the code, run tests and create artefacts with every change of the source code in the code repository. This process is called Continuous Integration or CI, since it validates the integration of all code changes continuously. In the last 15 years this has become very much the standard in the Java world. The next step in the work pipeline evolution after CI is called Continuous Delivery (or CD). With a CD pipeline a team tries to treat every build as a potential candidate for release into production if it passes the build, the automatic unit tests, the automatic integration and acceptance tests. The tests must be so good that if an artefact passes all these tests nobody has to hesitate to push the last button and deploy them into production. Scary? Not there yet. It gets even more scary! There also a workflow called Continuous Deployment where the manual push of a button is not even needed anymore. Every build that passes the tests goes into production.
Continuous Delivery for the ESP8266
We won’t go as far as Continuous Deployment in this post but I want to show you a nice workflow for Continuous Delivery. The workflow I’m describing here is using freely available tools assuming that you are working on an open source project. You still might use the ideas described here for closed source projects but then the tools are not free anymore. So what are the components in the workflow?
I have to admit, it is a relatively complicated workflow. Especially the PHP script doesn’t look too pretty in the picture but a deficiency in the ESPhttpUpdate class made it necessary. So here are the different stages during development:
Stage 1 – Development – The developer on his computer writes the code (e.g. in the platformio IDE) and tests it locally either by uploading it over the serial connection or by the “local” or direct OTA update mechanism. You can by the way combine the two update mechanisms without problem but you might consider disabling it for the production release or your devices will be more vulnerable for attacks. When developer is happy with his work he commits the changes to Github.
Stage 2 – Continuous Integration – Now Travis will start doing his work and build the last commit. If you have written tests Travis execute them and improve your confidence that your code is working as expected.
Stage 3 – Tagging – Once the developer team is confident that the current version is ready for deployment to the production devices the last commit will be tagged as a release.
Stage 4 – Release Build – Tagging the branch caused Travis to start a new build. But this time Travis will create a binary artefact (the firmware) and upload it to Github and add it to the release
Stage 5 – ESP8266 Update Check – During all this time the devices where hungry for a new release. They frequently asked the PHP script if there was a version different from the one they are running on. Every single time the PHP script had checked with the Github API if there was a newer version and every time the answer was NO. But now there is a new release, a new tag and hence a new firmware to be installed. The ESP’s download the firmware, replace the old one and do a restart. If everything went well they will be running the latest firmware until a newer version is found.
How to set up the workflow?
In this post I won’t explain you how to set up a Github account or how to connect Travis with your Github repository. You can find links for that at the end of the post. I chose to use the Platformio IDE but you could also use the Standard Arduino IDE. But for the example project I recommend to use the Platformio IDE.
As a first challenge the firmware has to be able to tell the PHP script with what version it was built. So how does the version get into tagged firmware? There are two build profiles in platformio.ini. One for the normal (local) nodemcu build and one for the travis build. The local one sets a content of the BUILD_TAG macro to “0.0.0”:
build_flags = -DBUILD_TAG=0.0.0
while the travis profile uses the $TRAVIS_TAG environment variable if available to pass the tag into the build or set it to “0.0.0” for a non-release build:
build_flags = !echo ‘-DBUILD_TAG=’${TRAVIS_TAG:-“0.0.0”}
This happens with every commit but only for release builds the $TRAVIS_TAG environment variable is set and only then the firmware will be uploaded to Github. The following line in .travis.yml makes sure that only tagged versions will be deployed back to Github:
deploy:
on:
repo: squix78/esp8266-ci-ota
all_branches: true
condition: $TRAVIS_TAG =~ ^[0-9]+\.[0-9]+\.[0-9]+$
The condition attribute checks that the tag version matches the regular expression. For a not very experienced C(++) developer I had a hard time to figure out how to pass in a String literal as macro during expansion phase. This code does it by using the # operator:
#define TEXTIFY(A) #A
#define ESCAPEQUOTE(A) TEXTIFY(A)String buildTag = ESCAPEQUOTE(BUILD_TAG);
Let me know if you have more simple solution…
How to create a release?
On your Github repository root page (e.g. https://github.com/squix78/esp8266-ci-ota) you click on Releases > Draft New Release. Now in the Tag Version drop down you enter a version number matching the above condition, e.g. 0.0.15. You can either do this form master version or from one of the last commits. Then publish the Release.
The PHP script
It would have been nice if the production ESPs could have contacted the Github API directly. But there are two issues that made be use the intermediate PHP script:
- The ESPhttpUpdate currently cannot follow redirects. This is important since github hosts the release artefacts on Amazon AWS. But in the API JSON object the address points to github, so the http client has to follow a redirect to download the artefact.
- Github uses https for its API and will redirect you to it if you are trying plain HTTP. This means that you would have to know the SSL fingerprints of the github API server and the AWS hosting instance since this is required by the ESPs secure client interface. After all the ESPs don’t have a chain of trusted certificates stored somewhere. While the fingerprint of the github API might be stable, the redirection on Amazon AWS might not always use the same certificate.
So what does the script do? It connects to the github API and fetches a JSON object describing the latest release. I’m currently only interested in the tag name and the firmware download URL. The ESPs will send their firmware tag version with the update request and if the latest tag on Github and the one from the request are identical nothing will happen. But if they are different the script will fetch the binary from github (with a hidden redirection to Amazon) and proxy it to the ESP’s requesting it. You can find the php script in the server folder of the Github project.
The ESP8266 code
Have a look at the main class of the example directory. One important guide line for this kind of deployment is that your code should be free of secrets (e.g. for Wifi) or environment specific variables. Because the devices should be configurable in a end-user environment I’m using tzapu’s excellent WifiManager library which starts a captive portal behind a Wifi AP if the device cannot connect to an access point. This way the Wifi credentials are not part of the firmware but stored in the flash memory.
Then you need to include many header files for the OTA update to work. My example code doesn’t do anything but keeping itself up-to-date with the latest release version on github by checking every 60 seconds (yes, I’m very impatient;-)) if there is no update. In a real project you might consider doing these checks less frequently or even use MQTT to get a notification once a new version is available. If you cannot wait for the 60 seconds to pass you can also push the “Flash” button on a NodeMCU to trigger the check.
UPDATE 2016/06/05: Ivan from Platformio was so kind to show me some simplifications which are now part of the example project.
Conclusion
This post together with the example project (see resources below) shows you how you can set up a continuous delivery workflow for your ESP8266 devices. There are several steps involved in making this work but afterwards you will have a semi-automatic deployment to all the devices running your firmware. It took me many hours to get the workflow to work and to write this post. If you feel like I deserve a reward for it consider Teleporting a Beer to me(see box below) or use one of the affiliate links around my blog. Thank you!
A word of warning: during my testing the NodeMCU devices sometimes hung after an update. This is a known issue of the NodeMCUs and should only happen after a serial upload and not after OTA. But if you plan to use this workflow or the OTA feature in general for production be sure that you do some testing.
Hardware Resources
Software Resources
Hi Daniel,
Firstly, thanks a lot for the great article!!! 🙂
A few comments:
1. You can use `platformio run` command instead of `platformio ci`. In this case, the firmware will be located in `.pioenvs` directory.
2. Instead of these lines:
> – platformio lib install 567 562
> – export PLATFORMIO_BUILD_FLAGS=”-D BUILD_TIMESTAMP=”`date +%s`
> – echo “${TRAVIS_TAG}”
>- echo “const char * BUILD_TAG = \”${TRAVIS_TAG}\”;” > src/version.h
Just create environments in `platformio.ini`:
“`ini
[env:nodemcuv2]
platform = espressif
framework = arduino
board = nodemcuv2
upload_speed = 921600
lib_install = 567, 562
build_flags = -DBUILD_TIMESTAMP=$UNIX_TIME -DBUILD_TAG=0.0.0
[env:nodemcuv2_deploy]
platform = espressif
framework = arduino
board = nodemcuv2
lib_install = 567, 562
build_flags = !echo ‘-DBUILD_TIMESTAMP=$UNIX_TIME -DBUILD_TAG=’$TRAVIS_TAG
“`
The final .travis.yml
“`yml
language: python
python:
– ‘2.7’
sudo: false
cache:
directories:
– ~/.platformio
install:
– pip install -U platformio
script:
– platformio run -e nodemcuv2_deploy
deploy:
on:
repo: squix78/esp8266-ci-ota
all_branches: true
condition: $TRAVIS_TAG =~ ^[0-9]+\.[0-9]+\.[0-9]+$
skip_cleanup: true
provider: releases
overwrite: true
api_key:
secure: DeLq8sczTsEAo3qi2ztSYgvaeidenvJherPCrceoknNuBs8SEsb8czGxBDPKt7lzK155Kcsfq/zo1VtUpVQ8HsalMQJABv9Li6t1uBc0m+mSLGggxw9woGzHv8FfnJdqeBD3YXgX8Qrmlnqp5QdIDZuyRfOxeF4NFwWZ89aOxMO8Kk363ykvQg5mq8mbqzHJOtCQCkHOEjklhcMTaHZgW/mlt7jBbuX/RM5jGTTAEOIKU/gmVvLo1LQMgsvJ2BCSLMrkEZmfxPncI1xTf26X7aSu01krn6oaROgJtFpTWEmSWHDHOJwngPC36SuaVtyd5WHE8u6+ZkDlfZsvDShq7Iap6vzqZwrTM3NvNyAnLWbE4YLLBDkuHMojyhy5MkvFLjVfEd4wkvy4JOnCL7IhJjcM4OWEXykZv80X/XBqT3XcZYYaorXs1x7rVUCsMYIxythVRaBzsP61NMFIEbLxV0maTYbuLMWX2HmForumMkk6SD7NdbaONZUaKRaG5G/QGWGHrOfVwBWCL/0KEZl6mKqkfadAQI4GWD77MOWn5JNmbM6bVE2hD/oVkSoao1XdY2N0flDEQQuVu3GHIweN3f5kudJ/bS04US1iyWw3t3PYtNOaQrUFtoUMd88/Q6hzzbdYuVVvjBH4Lsni4i/svMu//7URWXGgmcjm14Fr8xU=
file: “.pioenvs/nodemcuv2_deploy/firmware.bin”
“`
P.S: I didn’t test it. However, I hope you understand my idea.
Hi Ivan. Thanks a lot for the hints! I will try to integrate them. Your support is excellent like always!
Very informative. thank you
Hi,
how do you write back to the version.h file?
As far as I know travis has no permissions to write back to the repo.
You are right, I’m not writing the changes in the version file back to the repo and in that matter it is a bit ugly, since the released version is not 100% identical to the actual firmware. The version.h file only gets changed in the checkout in the Travis environment. I decided that this compromise was acceptable…
UPDATE: With Ivan’s suggestions I could get rid of that contamination of the source code by using the BUILD_TAG macro. This means that the source code now really remains as committed, but the build is parametrized by feeding in the BUILD_TAG
Please, I would like to ask some questions about the PlatformIO:
1. Try to put libraries in the lib folder and does not recognize. It only works if put in the same folder as the source.
2. How to go saving multiple versions of the same project in the same folder? It gives error to let more than one.
Thanks
Great Post many thanks!
Nice work! I am trying to replicate your Travis build and deploy mechanism.
Here is my travis file: https://github.com/openenergymonitor/emonpi/blob/master/.travis.yml
I think I’m almost there but I’m getting error in the travis log:
/home/travis/.rvm/gems/ruby-1.9.3-p551/gems/octokit-4.3.0/lib/octokit/response/raise_error.rb:16:in `on_complete’: GET https://api.github.com/user: 401 – Bad credentials // See: https://developer.github.com/v3 (Octokit::Unauthorized)
https://travis-ci.org/openenergymonitor/emonpi/builds/136591535
I have double checked that the token I’m using is valid created here: https://help.github.com/articles/creating-an-access-token-for-command-line-use/
then hashed using ruby gem $ travis encrypt GH_TOKEN=XXXX
Please could you provide some more details as to how you generated your token and if any extra steps are required? E.g. what access scopes are required
Thanks a lot, amazing guide.
Good point. I guess you are the first who is actually trying:-). From the top of my head I used “travis setup releases” which overwrote the deployment section in the travis file. Take the key from there then revert to the default…
Hi Squix78,
I have a OLED 1.3″ SH1106 chip display and if I change the commands in the display routine, your weather station works perfectly:
void SSD1306::display(void) {
for (uint8_t k=0; k<8; k++) {
sendCommand(0xB0+k);//set page addressSSD_Data_Mode;
sendCommand(0x02) ;//set lower column address
sendCommand(0x10) ;//set higher column address
for (uint16_t i=0; i< 128; i++) {
// send a bunch of data in one xmission
//Wire.begin(mySda, mySdc);
Wire.beginTransmission(myI2cAddress);
Wire.write(0x40);
for (uint8_t x=0; x<16; x++) {
Wire.write(buffer[128*k+i]);
i++;
}
i–;
yield();
Wire.endTransmission();
}
}
}
Is there a way that the SSD1306 library can be modified to include this fix for the SH1106 chip (I2C)?
Another thought: Perhaps a SSH1106 library so there are two distinct libraries depending on display type
YEAH! Thanks a lot. $ travis setup releases did the trick 😀
However I’ve now got an error ` No such file or directory – .pioenvs/emonpi_deploy/firmware.hex `. However I can see that this file is created by travis. I double checked that skip_cleanup: true.
Are there any further steps required?
https://travis-ci.org/openenergymonitor/emonpi/builds/136748825
avr-objcopy -O ihex -R .eeprom .pioenvs/emonpi_deploy/firmware.elf .pioenvs/emonpi_deploy/firmware.hex
AVR Memory Usage
—————-
Device: atmega328p
Program: 18888 bytes (57.6% Full)
(.text + .data + .bootloader)
Data: 947 bytes (46.2% Full)
(.data + .bss + .noinit)
========================= [SUCCESS] Took 3.39 seconds =========================
The command “platformio run -d firmware -e emonpi_deploy” exited with 0.
cache.2
store build cache
0.00s
0.87snothing changed, not updating cache
dpl.0
Fetching: dpl-1.8.16.gem (100%)
Successfully installed dpl-1.8.16
1 gem installed
dpl.1
Installing deploy dependencies
Fetching: addressable-2.4.0.gem (100%)
Successfully installed addressable-2.4.0
Fetching: multipart-post-2.0.0.gem (100%)
Successfully installed multipart-post-2.0.0
Fetching: faraday-0.9.2.gem (100%)
Successfully installed faraday-0.9.2
Fetching: sawyer-0.7.0.gem (100%)
Successfully installed sawyer-0.7.0
Fetching: octokit-4.3.0.gem (100%)
Successfully installed octokit-4.3.0
5 gems installed
Fetching: mime-types-2.99.2.gem (100%)
Successfully installed mime-types-2.99.2
1 gem installed
dpl.2
Preparing deploy
Logged in as Glyn Hudson
Deploying to repo: openenergymonitor/emonpi
Current tag is: 0.1.1
dpl.3
Deploying application
/home/travis/.rvm/gems/ruby-1.9.3-p551/gems/octokit-4.3.0/lib/octokit/client/releases.rb:86:in `initialize’: No such file or directory – .pioenvs/emonpi_deploy/firmware.hex (Errno::ENOENT)
I don’t see a platformio.ini in your repo. I’m just checking this on my smartphone but I think there is some magic happening in that file…
Platformio.ini is in the firmware folder, that’s the missing part, I just needed to add firmware into the hex file path. Working now 😀
Thanks again for your awesome blog. Now to setup php script and esp…. Exciting weekend ahead 🙂
I think I’ve almost managed to re-create your continuous delivery workflow, however I’m getting a
“Verify bin header failed” error from the ESP when update is attempted. I hope you don’t mind, I’ve posted some further info as an issue on your project GittHub: https://github.com/squix78/esp8266-ci-ota/issues/2. Although I’m sure the ‘issue’ is my fault rather than your code! Any insight would be apprichated. Thanks! Have a good weekend 🙂
Hi Glyn. Could you find an answer to your issues in the meantime? I would guess that the message means that your downloaded file doesn’t look like a firmware. I tried to find the build on travis that uploads the firmware to see if you are choosing the correct file but I couldn’t find it. Maybe your uploading the wrong file? I compared the bin downloaded by your php script and downloaded from the release and from distance they look the same…
Not managed to solve yet, I’ll try pointing the updater directly to the .bin or flashing manually to test. Here an example of a travis release that uploads the file https://travis-ci.org/openenergymonitor/EmonESP/builds/138772263. Yes the md5sum of the binary from the php matched the release
Hi Glyn. For the record and to help others (as duplicate to my answer on the github issue): I think I found the problem: your php script contains an empty line at the beginning of the script. Remove it and you should be good to go…
Kind regards,
Daniel