While working on Heat Synthesizer 0.9.4, I created a song to test the VST integration and to show some of Heat Synthesizers capabilities. More information can be found here:
Heat Synthesizer version 0.9.3 has been released. The following features have been added and/or fixed:
- NEW: Heat App now keeps the screen on to prevent the device from turning off while using an external MIDI device
- FIX: VST plugin didn’t save patches with project
- NEW: The VST plugin can now be given a name. This is useful to prevent confusion on the Android device when using multiple plugin instances at the same time
There were many talks about low latency on Android, the most recent “big thing” that happend was probably the Talk about High Performance Audio at the Google IO 2013 were Raph Levien was talking in-depth with Glenn Kasten about the current state of development and tips for developers to help bringing low latency audio to Android.
Of course, I did everything they mentioned to bring low latency to my music app Heat Synthesizer and released version 0.9.0 in the Play Store
What Heat Synthesizer did previously was not that bad at all and required only a few changes.
Heat always already calculated everything in the OpenSL ES (the audio interface of Android on the C++/native side) callback, just because that was always my opinion how to achieve low latency at and that is also how other big systems work, for example the VST standard, you get called directly to create your samples. It makes sense, if it would have been done in a separate thread, you would always increase the latency at least by one block size.
Additionally on Android, the audio callback thread runs with higher priority, so you want to take an advantage from this of course. Any other thread that is created by yourself cannot get the priority raised above “normal” if the device isn’t rooted.
I also used two buffers only, so one is playing while the second one is computed.
Heat always got positive reviews regarding its latency, so far so good.
What I also did was always calculating at 44100hz. As Glenn Kasten and Raph Levien pointed out, if the native samplerate may differ from device to device and if it isn’t matched, the so-called “Fast Mixer” can’t be used, which means that latency is increased drastically by the Android internal audo drivers, because they need to resample the data before sending it to the output.
So I changed the sample rate of OpenSL to match the native one. To prevent that the CPU usage is increased, I added a resampler if the native sample rate is higher than 44100hz and everything seemed to be fine.
I then released version 0.9.0 because everything seemed to be okay, but a few days later while playing on my Nexus 4 running Android 4.4 KitKat, I noticed audio crackles even on sounds that are not that complicated. The trouble began:
Having heared about systrace and all that at Google I/O, I started investigating what the root cause is. That wasn’t that easy because some things in my systrace were different than in the Google I/O, for example the fRdy2 parameter was always 240 instead of the mentioned 480. Also unclear was the fact that my OpenSL ES callback occasionally required a lot more time than it should.
Somehow I needed to reach some android experts, but I didn’t find them on sites like stackoverflow and I wondered where to ask. Luckily by googling around, I found the google group android-ndk that seemed to be the right place.
So I started a thread about possible android bugs, problems with my app and technical questions. The thread is still running. Although the feedback was far better than anywhere else, the most important person, Glenn Kasten, didn’t answer but only bumped my second thread that I created by accident and said that only one of them should be used for discussion. Glenn, it would’ve been nice if you shared your thoughts instead
What can I say, none of them has the time to help even though I’d say that Heat is one of the specific application types that will get into trouble.
The problem in comparison to the synthesizer used for the Google I/O is, that my math is more complex and my calculations need more time. If everything would run as expected on the Android side, this wouldn’t be a problem, because I’m not too expensive, the CPU core that Heat is running on is only at 50% load. But in comparison to the DX7 emulation of Raph Levien, it’s a lot.
Although not all points have been confirmed by officials there (they are all so busy…), the following are, imho, the remaining problems that Android itself has to solve until low latency audio is really possible:
- CPU Core clock: Heat suffers from the fact that Android decides to lower the clock of the core Heat is running on just because Android guesses that the core is under-utilized. But what happens here is that the time frame at which Heat has to return a new buffer to OpenSL ES is no longer met, which results in crackling. I did a screenshot of the systrace where the problem is very obvious: http://heatvst.com/temp/systrace4_clockfreqproblems.png
You can clearly see that everything is working as soon as the core clock is raised.
- Thread Priority: Although the audio callback thread has higher priority, it is still not enough for smooth audio playback. It may happen that services, such as the sensor service, gets enqueued on the same core that the audio callback / Heat is running on. As Heat also uses sensors as modulation sources, I cannot just disable them.
What then happens is, Heat can start processing a new audio block lately only and the time frame isn’t met at which OpenSL ES expects to receive the next block of audio data. Crackles in the audio occur.
As there are CPU cores left that are disabled completely, this is even more ridiculous.
- Wierd audio buffering callback behaviour: To reduce the introduced jitter in the audio, I also experimented with using more than 2 blocks, I tried to use three and I even tried to use 16 blocks to be sure to prevent ANY kind of crackles, just for testing.
You won’t expect it, but that doesn’t work. The callback to fill a new buffer isn’t called when the first of the 16 buffers has been played, but when 15 buffers have played! For me this is a huge bug, because that means that there is no way to make it smooth on lower end devices by adding more buffers.
It could be achieved if running in another thread. Infact, I can do this by a simple compile-time switch. But as my own thread then does no longer has higher priority, this causes even more problems. And it just isn’t “how it should be done” anyway.
- Block Size Madness: I also tried increasing the block size, even the user can do this in Heat in the options. But as soon as the device supports low latency audio, wierd things are going on:
If you enqueue 2048 samples on a Nexus 4, which has a native blocksize of 240, you get called if the device is playing its last remaining 240 samples. That means that you have to calculate 2048 samples but in a time frame of 240 samples only!
Someone in the android-ndk pointed out that this is normal. But sorry guys, for me this is the same bug as with the buffering, mentioned above.
No other audio device does it this way, it just doesn’t make sense at all. If they’re two buffers of 2048 in size, I have to get called when the first 2048 have been played, not after 3856 samples already being played and only 240 left until audio will start crackling.
Thinking positive, I assume that the last two points are bugs and will get addressed. If not, the first two points should be addressed and a lot of music app developers would still be happy.
So what I’ve done with Heat? Having spent days reducing the crackling on my Nexus 4, the current version in the Play Store, 0.9.2 does the following:
- If the requested block size is the native block size or is smaller, two audio buffers are used, one buffer is filled inside of the callback while the other one is playing. This is because I do not want to introduce extra latency if the user doesn’t request itThe same applies if the device has no low latency audio support, two audio buffers are created with the requested size selected by the user and one is filled while the other one is played.
- If the device has low latency audio support and the requested size is too large, additional buffers are used and the calculation is moved to a separate thread. This gives the advantage to compensate against jittering of the processing code
I really hope that the discussion in the android-ndk group will give help or at least clarifiy if there is anything I can do against the problems. I’ll keep you updated.
Please share this article, spread it! If more people know about it, hopefully Google will react faster!
Heat Synthesizer version 0.8.9 has been released. The following features have been added and/or fixed:
NEW: Optional 2 octaves keyboard can be enabled in the options. For tablets with 7 inches screen size or higher, this is the new default, but can be reverted to 1 octave if preferred in the options
FIX/NEW: Fix data loss of locally stored sounds. The system has been rewritten completely, sounds are now stored as files on the internal storage of the device. They can even be shared/copied/whatever by just copying the files.The location of the sounds is <internal storage>/HeatSynthesizer/Presets
NEW: Every logged in user now has its own category “My Favourites” and sounds can now be added as favourites using the long-tap context menu
NEW: Improved performance of wavetable oscillators and the modulation matrix
FIX: “Best presets” were sorted incorrectly
FIX: arp mode “as played” was the same as arp mode “up”
FIX: Sound couldn’t be stored online if it was previously added but then deleted already
Heat Synthesizer version 0.8.8 has been released. The following new features have been added:
- New: Added support for class-compliant USB Midi Interfaces to be able to play Heat with a Midi keyboard
- New: Added support for Bluetooth keyboards (Octave 1: Z-M; Octave 2: Q-U; left/right cursors for octave shift)
Android App changes in version 0.8.7
- New: Allow to adjust the BPM / Beats per Minute on the new “Global” page
- New: Wave Recorder: Allows to save the output of Heat as wave file
Android App changes in version 0.8.6
- New: As the Preset database keeps growing, you can now search for Presets by name
- New: Share Presets with others, send them by mail, Facebook, Twitter, whatever you want. To Listen to shared Presets, open the links on your Android device with Heat installed, Heat will automatically start and load the linked sound.
- Fix: Rare crash in Arpeggiator when touching outside of the note grid bounds
- Fix: Modulation source was not visible on all devices once selected
VST Plugin changes
- Fix invalid plugin window size in some hosts
Heat Synthesizer 0.8.5 has been released, this is a maintenance release with the following changes:
- Fix crash when clicking near the ‘Processing Mode’ setting in settings page
- Fix filters envelope was exponential, is now linear. Please review your Presets, but this change shouldn’t be noticed most of the time.
- Fix adding presets with invalid names could stop Preset Browser to display any Presets
- Fix “Store Online Terms” box was not scrollable / unable to click OK on small screens
- Removed DSP processing mode, WiFi is just not always stable enough to handle it and it makes no sense anyway as GUI mode behaves identical but VST is used to process sound
Please also download new VST plugins or your Presets may sound slightly different because of the filter envelope change.
As version 0.8.4 has been released in the Play Store now, I totally forgot to update the VST plugin downloads aswell.
They have been updated now, to get them please visit the Download page.
If you had issues with freezing of the host when loading a preset on your Android Device into your connected VST, this is fixed now.
Heat Synthesizer 0.8.4 has been released. The following changes have been made:
- New soft-piano design by my friend and co-worker phoenix, thank you! (You can expect more UI enhancements from him in the future)
- New FX added: Phaser
- Audio block size can be adjusted in the options between 128 samples and 2048 samples (512 was and is still the default). Please note that lowering the block size raises CPU usage and reduces the number of voices
- Support upside-down display if device is rotated 180°
- Fix rare crash when rotating the device while a Preset is loaded
- Delay FX unit knobs have been reordered to have matching left/right knobs
- Do no longer show “Store presets online terms” until user really tries to save a preset online
- Chorus FX is now processed before Delay FX. Full FX chain is now Phaser => Chorus => Delay
- VST plugin: Fix crash on plugin unload, some hosts refused to load the plugin because of this error. For all those that had problems before, the update is recommended.
- Server code: Replacing existing presets was not possible if the initial preset was not dependant on another one