ESP8266/Arduino: Playing around with the upcoming filesystem feature

Do you like this post? Writing posts, developing libraries and programming online font creators is actually hard work. I order most of my electronics from a Chinese website called Banggood. It is save and the items usually arrive in good quality. With every order you do there by following this link you are supporting the blog. Thank you!

I recently stumbled on a feature that hopefully soon will be integrated into the stable release of the ESP8266/Arduino environment: a file system interface.

Low-Level: EEPROM

Up until now and currently your best bet for data persistence in the stable release you could use the EEPROM interface to write and read your data so it would be available between reboots of the device. This persistence interface is pretty low-level: you have to define the range of the memory you want to use and then it’s all on you how to organize this stream of bytes. Of course, you could write yourself some code to write and read the data in a more accessible way but this is cumbersome and error-prone. Especially if the stored data has different formats, lets say one part is a JSON object and the next is HTML you have to create a table of content (TOC) and keep track of the sizes and start points of these blocks.

New: SPIFFS

But why re-invent the wheel? This organization of all the single blocks in a TOC is generally called a file system and the team around the ESP8266/Arduino environment is currently integrating the SPIFFS into their development branch. SPIFFS stands for SPI flash file system and is especially targeted for embedded devices such as the ESP8266. According to the Github page of the SPIFFS project the file system has the following features:

  •  Specifically designed for low ram usage
  •  Uses statically sized ram buffers, independent of number of files
  •  Posix-like api: open, close, read, write, seek, stat, etc
  •  Implements static wear leveling
  •  Built in file system consistency checks

There are also a few limitations but hey, what do you expect from anything especially targeted for small and embedded devices?

 

Hands-on

I couldn’t resist and wanted to try out this new feature. As a preparation you’ll have to install the staging version of the ESP8266/Arduino environment (as the time of writing – 2015-08-16 the stable version does not include this feature yet). To do that follow the instructions on the following link and use the reference to the staging version instead of stable: https://github.com/esp8266/Arduino#available-versions
Once you have done that you should be able to run the following code:
#include "FS.h"

void setup() {
  Serial.begin(115200);

  // always use this to "mount" the filesystem
  bool result = SPIFFS.begin();
  Serial.println("SPIFFS opened: " + result);

  // this opens the file "f.txt" in read-mode
  File f = SPIFFS.open("/f.txt", "r");
  
  if (!f) {
    Serial.println("File doesn't exist yet. Creating it");

    // open the file in write mode
    File f = SPIFFS.open("/f.txt", "w");
    if (!f) {
      Serial.println("file creation failed");
    }
    // now write two lines in key/value style with  end-of-line characters
    f.println("ssid=abc");
    f.println("password=123455secret");
  } else {
    // we could open the file
    while(f.available()) {
      //Lets read line by line from the file
      String line = f.readStringUntil('n');
      Serial.println(line);
    }

  }
  file.close();
}

void loop() {
  // nothing to do for now, this is just a simple test

}

You can also find the example here:
https://github.com/squix78/esp8266-projects/tree/master/arduino-ide/filesystem-example

Explanation

The above code tries first to open a file called f.txt. If this succeeds it will read from it line by line and print the result to the serial console. If it cannot open the file it probably doesn’t exist yet. The code will then create the file and fill it with some imaginary key/value pairs, one pair per line.

Summary

While other ESP8266 firmwares (NodeMCU, smart.js and frankenstein) already come with a filesystem it is great to see that the Arduino/ESP8266 environment soon will be blessed with this important feature. It is easy to apply and a huge improvement over the very low-level EEPROM persistence interface. Directories let you structure your files and you can freely create new files, delete or rename them.
Compared to the NodeMCU firmware you have worry how you can get the files onto the device. If you want to run a web server you cannot just transfer the html, css and js files from your PC to the ESP8266. There is no interface for such an operation yet. But you could store them on another webserver and store/update them in flash when the device connects to WiFi.

 

Documentation

Posted by squix78

26 comments

  1. Hello Dani,
    cool thing and a rather fascinating story! If I did not already have one kit I would order one…
    But one thing: On the label stuck on the bag is written “NodeMCU V3” but in the text description you write “NodeMCU V1”.
    Perhaps you can sync this?
    Best regards and successful business!
    Thomas

  2. Hi Daniel;

    Awesome! I got my kit and I loaded up the WeatherStation sketch and everything was working 🙂

    I then went on to add the DHT11 code and that’s when things stopped working. I suspect the firmware was over written or something bad … bricked I suspect.

    From the Arduino IDE …

    warning: espcomm_send_command: didn’t receive command response
    warning: espcomm_send_command(FLASH_DOWNLOAD_BEGIN) failed
    error: espcomm_upload_mem failed
    error: espcomm_upload_mem failed

    I then went on to try and reflash the NodeMCU firmware using ESP8266Flasher.exe … from there I get ..

    Note:Serial port connected.
    Note:Begin find ESP8266.
    Note:ESP8266 ACK success.
    Note:ESP8266 ACK success.
    Error:Set ESP8266 Address timeout.

    Similar problems directly from the esptool

    c:\Python27\Scripts>esptool.py –port com5 write_flash -fm dio 0x00000 nodemcu_integer_0.9.5_20150318.bin
    esptool.py v1.2.1
    Connecting…

    A fatal error occurred: Timed out waiting for packet header

    … in fiddling around the board is alive .. it does send this out the serial port at 75K baud… could it be a baud rate issue? I assume not as I understand it’s autobaud detecting.

    ets Jan 8 2013,rst cause:2, boot mode:(3,7)

    Questions –

    o Do you have any suggestions on next steps?

    o Could it be power? I gather power is important for FTDI boards but the NodeMCU has the 3v3 regulator and it’s running off a PC with “good” USB current .. and it did work fine before it did not 😉

    o I’ve had no success in pressing the RST and FLASH buttons — I as I understand them, the RST/CTS lines put the board into to bootloader mode at any rate .. thoughts?

    … thoughts?

    1. Hi David

      Here sone thoughts
      – try another USB cable, the can be reason for your symptoms
      – try another USB port or restart your PC (or Mac?) I had situations where my Mac would only show the device after a restart
      – unplug the DHT11. Maybe your wiring is wrong and you accidentally switched VCC and Ground. Not sure about the longterm effects of such a switch. Such a switch was actually the only way I could ever brick an ESP8266

      If these tips don’t help feel free to get back to me

      Cheers, Dani

      1. Thanks Daniel —

        I’ve tried multiple USB cables and multiple USB ports (Windows).

        I did go back to just the bare board .. no display and no DHT11.

        I’m now trying to see it the ESP AT command set even works … and no luck there yet. I do get this when I press the reset button so it is alive a least a bit 😉

        ets Jan 8 2013,rst cause:2, boot mode:(3,7)

        but nothing in response to AT ….

        ??

        Cheers David

        1. According to this post it might have to do something with your reset pin: http://www.esp8266.com/viewtopic.php?p=2096#p2112
          I’m not sure what that means;-).
          With which baud rate are you trying to flash the ESP? I your example from your last comment you did auto-sensing, right? Try to use a fixed lower rate like like 115200 or 57600. It often is something like the “-b” parameter.

          When you say you went back to the AT firmware: so did you manage to flash the AT firmware? After flashing the ESP8266 with the new firmware the AT binary has been replaced…

  3. Thanks Daniel — I just received another NodeMCU board as I’ve wasted enough time on the dead one 😉 If I manage to get the dead one working again, I’ll let you know.

    HNY, David

  4. Thanks Daniel, You sent me a trial kit and I purchased 2 others from Amazon. Some minor problems (outlined in Bryan Miller’s comments on Amazon). it works great but I still could use a fix to be able to get the screen to reverse from top to bottom your current variable didn’t work.

    I also located a free case plan that I had 3D printed and it fits and looks good. I found a local printer and picked it up today ($9) and it was a good fit, BUT the pins to hold the OLED were slightly out of register. I think I’ll have to cut them off and tape or glue the display in place. There is a ledge for it to fit in so i don’t think that will be a problem. I’m going to get another one printed for my local (Grand Rapids , MI USA) and I sent the plans off to my son in Atlanta, GA USA for him to use. I built the Squix kit with them over our Thanksgiving Holiday and now it will have a much better case than the plastic box it came in. Thanks for your good work and I did send off a tip. This case fits the slightly smaller SQUIX version of ESP8266 -12E.

    http://www.thingiverse.com/thing:1720314 for the plans

  5. I have my kit, and am following the directions, but even after 4 cable swaps, verifying all setting from the getting started guide, and checking my IDE for errors, I cannot seem to get a valid up load.

    The error I encounter is espcomm_upload_mem failed;

    Now I verified that the chip on the board shows a CP2102, so that is the USB driver I installed;
    I restarted my computer (windows 10) to ensure driver is loaded;
    I see a com3 show up when I apply power with the USB cable, so that is the port i chose;
    I loaded the libraries as indicated, and select the board nodeMCU 1.0

    I also updated from the 1.6 to the 1.8 arduino IDE too;

    I am lost to what else I can do to get the base sketch uploaded?

    1. Hi. From distance it indeed sounds like you did everything fine. On one hand it is a good sign that the COM port shows up is a very good sign, it means that you have the correct driver installed. On the other hand there are reports of problems with the CP2102 on Windows 10. Could you check please if you have the latest driver? See these forum posts:
      http://community.silabs.com/t5/Interface/cp210x-USB-to-UART-driver-problems-windows-10-code-31/td-p/146924
      http://community.silabs.com/t5/Interface/Windows-10-VCP-Driver-Now-Available/m-p/161629
      Feel free to contact me again if that should not solve the problem, please also let me know if it does so I can adjust the instructions…

      Regards,
      Daniel

      1. Daniel,

        I did download from that site the latest drivers, then I un-installed and re-installed just to be safe. That did not cure it, but I also think you have pointed me in the right direction, I duplicated the installation on an adjacent windows 10 computer and have the same result. Both computers can reach an Arduino UNO with no problem (from the same USB port), so I will continue to explore the device drivers. Thanks for your response! Once I resolve this, I will advise you for your documentation.

        Regards,

        Bruce

        1. Daniel, happy to report that the drivers with Windows 10were fine. Unfortunately the nodeMCU appears to be bad. I ordered a couple more of these and they came in todays’ mail; I plugged one in to the windows 10 system and it fired right up tonight. Now I can add the sensor and the oled and play, woo-hoo!

          Thanks again for your quick response!

          Bruce
          Kentucky, USA

  6. Only ever been using 8266 as a wifi board for an arduino. This sounds much more interesting.
    I really want to buy your handy box, but I live in Sweden. Any chance of finding way for us Europeans?

    1. Hi again,
      I now noticed that you also had your own shop. Order is now completed. Please disregard the above. Thanks.

    1. Very nice. I was hoping that the winds could be included and looks like you nailed it with an arrow and value. Still waiting for my color display in the mail so can’t wait to see how it works.

    2. Am having trouble compiling your code, keep getting the following error:
      Arduino: 1.8.1 (Windows 10), Board: “NodeMCU 1.0 (ESP-12E Module), 160 MHz, 115200, 4M (3M SPIFFS)”

      Build options changed, rebuilding all
      C:\Users\Danny\Documents\Weather Station Bodmer\esp8266-weather-station-color-masterbodmer\esp8266-weather-station-color-master\esp8266-weather-station-color\esp8266-weather-station-color.ino:44:58: fatal error: TFT_ILI9341_ESP.h: No such file or directory

      #include // Hardware-specific library

      ^

      compilation terminated.

      exit status 1
      Error compiling for board NodeMCU 1.0 (ESP-12E Module).

      This report would have more information with
      “Show verbose output during compilation”
      option enabled in File -> Preferences.

      I have all the libraries loaded, but still cannot determine why I am getting this compile error. I have set up the display wiring according to your example.
      Thanks,
      Danny

      1. Bodmer thanks for your post on the fork of Daniel weather station color. I must have had some garbage in my IDE folder, have corrected the problem and have your version up and running. Thanks again to you and Daniel.

  7. Hi, there! considering the shortage on IL9431 display (even Banggood) and some vendors sending IL9225 (176×220) to anyone, is someone worked/working or willing to work on porting the code to use such display even sacrificing some graphic features? I can start to work from zero but my time availability is so short that to the time of progress it will be too late or other more thrilling things will be out there.

    1. This is an issue introduced with an update to the Adafruit ILI9341 library I believe. Try reverting back to a previous version (1.0.2 worked I think). I ran into this a month or so ago and it took the better part of a day to track it down. The issue may have been fixed recently, but I haven’t tried the most recent release of the library.

  8. Funny. I just two boards of which I think are ESP-01s. I can’t understand the comment about shorts in a breadboard. Their 8-pin connectors have 2.54mm or 0.1″ pitch. So they’ll fit well into a common breadboard.

  9. I uploaded the World Clock demo an hour ago and noticed at 10:30pm CDT it was showing New York time as 15:30. What caused this? Where is the time zone (+ or – hours from UTC, and DST enable/disable) information set? Now as I type this at 11:40pm, it is now showing NY time as 00:40. What happened to correct the NY time (I assume the other world times were and still are correct)?

  10. Hello, I’ve tried the trick to power the screen through the digital pin and I’ve run into trouble : my wemos was not able to communicate by serial link anymore. Moreover, the final size is so small that the USB cable is heavier than it and I’ve difficulties to maintain the whole assembly on my desk, turned to my direction.

Leave a Reply