Win a copy of Murach's Java Programming this week in the Beginning Java forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

RxJS in Action: is a custom RxJS scheduler required for accurate music playback?  RSS feed

 
Mvsica Ficta
Greenhorn
Posts: 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Really looking forward to RxJS in Action, which I think includes something on RxJS schedulers?

I recently saw the following video about making music with RxJS and was surprised that it was not scheduling notes using the Web Audio API clock. Rather, it was just playing audio directly using the System based timers with the standard schedulers in RxJS: https://www.youtube.com/watch?v=Wa5-DePTWdA&feature=youtu.be&t=1074

The Web Audio clock has a much higher resolution than the System clock, which is suitable for sample accurate playback (see info on clock differences below). The common setInterval and timeOut functions typically used are based on the System clock, these functions are known to be imprecise and these happen to be what the current RxJS schedulers all use! I'm aware that Tone.js can be used for playback but it comes with a large API and it's own idea of a music timeline that is limiting and not well suited for playing back arbitrary note structures. Using RxJS music can be processed in many ways so it makes sense to have the right scheduler as well.

From my research it looks like a custom RxJS scheduler is required that uses the Web Audio clock as a source. This way events scheduled on an Observable become queued actions that get scheduled directly to the Web Audio context (thread). How do I go about creating such a custom scheduler, there does not seem to be much info about this topic? For most tasks the standard RxJS schedulers would suffice but for accurate music playback I see no other way.

CLOCK INFO

The Web Audio API has an extremely high-resolution and reliable timing system which differs in many ways from the JavaScript timing system of which the maximum resolution is only 1 millisecond, 0.001. The Web Audio API timing system has a resolution of 1/1000 of a picosecond, 0.000000000000001, or 1 femtosecond.  This resolution is designed to be able to specify alignment on the sample-level, even with high-resolution audio. The timing system is always relative to the creation of the AudioContext, always starting at 0, and progressing in high-resolution seconds. The value can be accessed at any time by calling the ‘currentTime’ parameter of the AudioContext. This timing system makes it possible to build accurate sequencers, drum-machines, or time-reliant effects.

        var audioContext = new AudioContext;
audioContext.currentTime; // example: 507.570793650793631

Music notes or any numerical AudioParam can be automated and scheduled. For instance, you could schedule the cutoff frequency of a filter to change over time, creating a filter sweep, or fadeout a sound over time.

 
Pau Daniels
Author
Greenhorn
Posts: 8
5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for sharing such a cool demonstration of the power of RxJS.

In early versions of the book we had a chapter that was to be devoted exclusively to Schedulers, however, upon revision we realized that for the most part Schedulers are meant to be a near invisible part of the library.
And when they are used the built-in ones are usually all anybody would need. That's why in the final version of the text we instead focused our talk of schedulers exclusively around the TestScheduler which is probably one scheduler that will explicitly be used the most.

Using the different timing mechanisms for schedulers is certainly not unheard of (the AnimationFrameScheduler is one such example), but from my initial look I'm not sure if the WebAudio API is sufficient to create a complete scheduler.
Since Schedulers need to be able to run arbitrary units of work.

For reference this is the interface that needs to be fulfilled



and this is an "Action"



Where an action essentially needs to take an arbitrary unit of work and schedule it for future execution. I suspect that the WebAudio's precision is because it focuses so narrowly on Audio components that it wouldn't be able to execute generic JavaScript code.

I would suggest an approach more in line with how the demo video works, where you build the appropriate abstractions over the Audio API and use RxJS to wire everything up and create a nice declarative syntax.
 
Mvsica Ficta
Greenhorn
Posts: 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for your reply as this is an important case that needs a solid solution. I do understand that schedulers in RxJS are meant to be transparent for the most part and I also do not like to be the messenger pointing out this gap.

The "arbitrary units of work" that a Web Audio scheduler would run would be the music notes and audio params. To picture this lets say we have a frame/slice of notes pulled out of a score of data using RxJS pipeline operators. Each note in the quantized time frame/slice still has it's own fine durational timing. Each note event then needs to be scheduled as an "action" in the Web Audio scheduler. The scheduler then pushes out each scheduled note to the Web Audio context/thread at the scheduled time (or maybe it flushes all the notes instead?).

What is key is to switch to the Web Audio scheduler (maybe using observeOn) just as the notes are scheduled and sent to the Web Audio context. The essential thing is to use the Web Audio clock to know when to schedule each note. So in the over all pipeline this vital aspect has to be done outside the standard RxJS schedulers to avoid the jitter and low-resolution that comes from using setInterval and setTimeout, which are based on the System clock not the high-resolution Web Audio clock.

The Web Audio world has had an imperative workaround solution to setInterval and setTimeout for a few years now using a hack. Now in this new functional and reactive age we have a great opportunity to fix this hack. What I and the rest of the musical world would really appreciate is if RxJS experts like yourself would take a good look at the following article and it's demo. Please note also that the demo schedules visuals using RAF and then separately schedules the audio notes using the workaround hack. It is this hack that a new RxJS scheduler for Web Audio would improve upon.

https://www.html5rocks.com/en/tutorials/audio/scheduling/#disqus_thread
http://webaudiodemos.appspot.com/metronome/index.html
https://github.com/cwilso/metronome/




 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!