So I decided to tackle one problem at a time and looked at the big JSON object issue first. If you want to parse big documents you can either build a representation of the object in memory (DOM) or you can write a event based parser. This means that your parser is processing the document as it comes in as a stream, character by character and that it will notify you with different event types once portions of the document were parsed successfully. This is very helpful since you don’t have to keep the whole  document in memory and you can even start parsing the stream almost immediately. You might be familiar with the SAX parser for XML which also follows this concept.

This concept is very useful on devices with little memory as the ESP8266. Looking for a start I found such a stream based JSON parser written in PHP and decided to port it to C++.

The result looks very promising: I wrote a little example program for the ESP8266 which downloads a sample JSON object of 1MB size. In the sketch I’m only interested in one of the 5000 sub-objects and print it out as soon as I see it. During processing I print out the heap size with ESP.getFreeHeap() and the available heap size remains stable at around 32KB!

If you like to try it out you can find the library here: https://github.com/squix78/json-streaming-parser. In theory it should work for any Arduino target device but I only tested it with the ESP8266.

Happy hacking!