The ESP32 platform offers a wide range of functionality almost out of the box. For instance you can turn a ESP32 with little effort into a Bluetooth Audio speaker. In this article I describing how I tapped into the digital data stream coming from a smartphone to visualize the music with a 8 band VU meter. The code is also a nice example how to you can use the full power of the two Tensilica Xtensa cores in the ESP32.

The ESP32 is an amazing piece of hardware. It is very affordable, easy to use and comes with great libraries. One of those is the esp32-a2dp library by fellow Swiss Phil Schatzmann. With this library you can turn any ESP32 into a bluetooth sink or source. Using it as a sink effectively turns it into a Bluetooth speaker, using it as a source you can send music to a Bluetooth speaker.

Depending on your hardware, this is the minimal code to use the ESP32 as a bluetooth speaker, announcing its services as “MyMusic”:

#include "BluetoothA2DPSink.h"

BluetoothA2DPSink a2dp_sink;

void setup() {
    a2dp_sink.start("MyMusic");
}

void loop() {
}

Wouldn’t it be nice to visualize the music you receive over Bluetooth? I always loved the spectrum analyzers which split the available frequency spectrum in several bands and displayed the intensity in those bands. In August 2019 I wrote a post about how to do that in the browser with sound recorded on the ESP32.

Eavesdropping

Is it possible to get access the digital data stream received over Bluetooth? The A2DP library has a hook to register on events like “stream started” or “stream paused”. But there is no callback to tap into the audio data.

The library is actually a conveniency wrapper around the A2DP functionality in Espressif’s Audio Development Framework (ADF). It uses the esp_a2d_sink_register_data_callback(..) method to get the Bluetooth data stream.

By registering my own callback I could “slip in” between ADF and the A2DP library! Now my method with the signature

void audio_data_callback(const uint8_t *data, uint32_t len)

will be called from time to time with a pointer to the current music data package.

Not Enough Horsepower!

A common method to visualize music is to take slices and convert those into the frequency domain. Blocks with a length of a couple of milliseconds will then show which frequencies are currently active. The process to convert data from the time domain into the frequency domain requires quite a lot of calculations. One algorithm which can do this relatively efficiently is the Fast Fourier Transformation or short FFT.

So just like in my previous project I wrote code which would collect music data until a buffer was full. Then the code would run the FFT algorithm to get the content of the buffer as an array of discrete frequencies. This worked, but the music started to sound awful! Apparently the FFT algorithm took too much time to process the data and the output would stutter.

If One Horse Can’t Do It…

…two can! The ESP32 has two Tensilica Xtensa cores. Until now I never really had a use case where I needed to use the other core. I was also under the impression that you should leave that core alone since all the WiFi processing and other system tasks run on it. But why not give it a shot and do the Fast Fourier processing on the other core?

The revised code fills the buffer as before on core 0. But once the buffer is full it stops collecting data and notifies a task running on core 1 that it can start running the FFT processing on the buffer. Once the FFT processing is completed the code clusters the frequency information into 8 bands.

At this time the task on core 1 will tell the other task on core 0 that it can start again to fill the buffer. Core 1 doesn’t need the data in the buffer anymore. Now core 1 only needs to visualize the intensity of the 8 bands.

Multicore Programming

The data collecting task and the FFT processing task have to communicate with each other. This has to happen thread-safe and if possible without blocking each other unnecessarily. I used a RTOS queue where the data collecting task places a message when the buffer is ready for processing. The FFT task on core 1 starts once it detects a message in the queue. When the buffer is ready for more data it removes the message from the queue.

I have to admit that I’m not sure if my solution is the correct or best way to solve this problem in RTOS. I tried with semaphores but always ended up that all tasks were blocked. Please let me know in the comments below how you would have implemented this.

Hardware

The code I wrote should theoretically run with all DACs supported by the esp32-a2dp library with minor changes, even the internal ESP32 DAC. I wrote the code as a stock firmware for the ThingPulse Icon64 device. You can see it in action in the video below.

Get the Code

I published the code for this project on Github. I hope you can profit from it! As I mentioned before, if you have suggestions how to improve the multi-tasking code or other aspects please let me know either in the comments below or on github!