There were many talks about low latency on Android, the most recent “big thing” that happend was probably the Talk about High Performance Audio at the Google IO 2013 were Raph Levien was talking in-depth with Glenn Kasten about the current state of development and tips for developers to help bringing low latency audio to Android.
Of course, I did everything they mentioned to bring low latency to my music app Heat Synthesizer and released version 0.9.0 in the Play Store
What Heat Synthesizer did previously was not that bad at all and required only a few changes.
Heat always already calculated everything in the OpenSL ES (the audio interface of Android on the C++/native side) callback, just because that was always my opinion how to achieve low latency at and that is also how other big systems work, for example the VST standard, you get called directly to create your samples. It makes sense, if it would have been done in a separate thread, you would always increase the latency at least by one block size.
Additionally on Android, the audio callback thread runs with higher priority, so you want to take an advantage from this of course. Any other thread that is created by yourself cannot get the priority raised above “normal” if the device isn’t rooted.
I also used two buffers only, so one is playing while the second one is computed.
Heat always got positive reviews regarding its latency, so far so good.
What I also did was always calculating at 44100hz. As Glenn Kasten and Raph Levien pointed out, if the native samplerate may differ from device to device and if it isn’t matched, the so-called “Fast Mixer” can’t be used, which means that latency is increased drastically by the Android internal audo drivers, because they need to resample the data before sending it to the output.
So I changed the sample rate of OpenSL to match the native one. To prevent that the CPU usage is increased, I added a resampler if the native sample rate is higher than 44100hz and everything seemed to be fine.
I then released version 0.9.0 because everything seemed to be okay, but a few days later while playing on my Nexus 4 running Android 4.4 KitKat, I noticed audio crackles even on sounds that are not that complicated. The trouble began:
Having heared about systrace and all that at Google I/O, I started investigating what the root cause is. That wasn’t that easy because some things in my systrace were different than in the Google I/O, for example the fRdy2 parameter was always 240 instead of the mentioned 480. Also unclear was the fact that my OpenSL ES callback occasionally required a lot more time than it should.
Somehow I needed to reach some android experts, but I didn’t find them on sites like stackoverflow and I wondered where to ask. Luckily by googling around, I found the google group android-ndk that seemed to be the right place.
So I started a thread about possible android bugs, problems with my app and technical questions. The thread is still running. Although the feedback was far better than anywhere else, the most important person, Glenn Kasten, didn’t answer but only bumped my second thread that I created by accident and said that only one of them should be used for discussion. Glenn, it would’ve been nice if you shared your thoughts instead 🙂
What can I say, none of them has the time to help even though I’d say that Heat is one of the specific application types that will get into trouble.
The problem in comparison to the synthesizer used for the Google I/O is, that my math is more complex and my calculations need more time. If everything would run as expected on the Android side, this wouldn’t be a problem, because I’m not too expensive, the CPU core that Heat is running on is only at 50% load. But in comparison to the DX7 emulation of Raph Levien, it’s a lot.
Although not all points have been confirmed by officials there (they are all so busy…), the following are, imho, the remaining problems that Android itself has to solve until low latency audio is really possible:
- CPU Core clock: Heat suffers from the fact that Android decides to lower the clock of the core Heat is running on just because Android guesses that the core is under-utilized. But what happens here is that the time frame at which Heat has to return a new buffer to OpenSL ES is no longer met, which results in crackling. I did a screenshot of the systrace where the problem is very obvious: http://heatvst.com/temp/systrace4_clockfreqproblems.png
You can clearly see that everything is working as soon as the core clock is raised.
- Thread Priority: Although the audio callback thread has higher priority, it is still not enough for smooth audio playback. It may happen that services, such as the sensor service, gets enqueued on the same core that the audio callback / Heat is running on. As Heat also uses sensors as modulation sources, I cannot just disable them.
What then happens is, Heat can start processing a new audio block lately only and the time frame isn’t met at which OpenSL ES expects to receive the next block of audio data. Crackles in the audio occur.
As there are CPU cores left that are disabled completely, this is even more ridiculous.
- Wierd audio buffering callback behaviour: To reduce the introduced jitter in the audio, I also experimented with using more than 2 blocks, I tried to use three and I even tried to use 16 blocks to be sure to prevent ANY kind of crackles, just for testing.
You won’t expect it, but that doesn’t work. The callback to fill a new buffer isn’t called when the first of the 16 buffers has been played, but when 15 buffers have played! For me this is a huge bug, because that means that there is no way to make it smooth on lower end devices by adding more buffers.
It could be achieved if running in another thread. Infact, I can do this by a simple compile-time switch. But as my own thread then does no longer has higher priority, this causes even more problems. And it just isn’t “how it should be done” anyway.
- Block Size Madness: I also tried increasing the block size, even the user can do this in Heat in the options. But as soon as the device supports low latency audio, wierd things are going on:
If you enqueue 2048 samples on a Nexus 4, which has a native blocksize of 240, you get called if the device is playing its last remaining 240 samples. That means that you have to calculate 2048 samples but in a time frame of 240 samples only!
Someone in the android-ndk pointed out that this is normal. But sorry guys, for me this is the same bug as with the buffering, mentioned above.
No other audio device does it this way, it just doesn’t make sense at all. If they’re two buffers of 2048 in size, I have to get called when the first 2048 have been played, not after 3856 samples already being played and only 240 left until audio will start crackling.
Thinking positive, I assume that the last two points are bugs and will get addressed. If not, the first two points should be addressed and a lot of music app developers would still be happy.
So what I’ve done with Heat? Having spent days reducing the crackling on my Nexus 4, the current version in the Play Store, 0.9.2 does the following:
- If the requested block size is the native block size or is smaller, two audio buffers are used, one buffer is filled inside of the callback while the other one is playing. This is because I do not want to introduce extra latency if the user doesn’t request itThe same applies if the device has no low latency audio support, two audio buffers are created with the requested size selected by the user and one is filled while the other one is played.
- If the device has low latency audio support and the requested size is too large, additional buffers are used and the calculation is moved to a separate thread. This gives the advantage to compensate against jittering of the processing code
I really hope that the discussion in the android-ndk group will give help or at least clarifiy if there is anything I can do against the problems. I’ll keep you updated.
Please share this article, spread it! If more people know about it, hopefully Google will react faster!
