There is Max/MSP/Jitter patch behind all this.
The loaded videoclip is forwarded to 3 instances, so that 3 different moments can be triggered at the same time. As the sound processing depends on the original track of the quicktime movie I needed the audio to go into MSP. This should have been done with jitters [spigot~] object but that never worked really well. I happily found a workaround [spigoty] in the cycling74.com user forum that uses a [buffer~] [groove] combination, driven with [sig~]. The number of frames is recalculated the 0.0 – 1.0 float of the [sig]. Then it is easy to do the further processing with granulation.
The visuals are being mangled through Jitters openGL objects basically a combination of [co.accum.jxs] and [co.additive.jxs]. and repositioning. Search the mentioned forum, there is a lot of information on how to do that.
The letters and written words are nutured by a patcher that loads a preselection of phrases into eight [buffer~] objects and in parallel the respective text archives into [coll] object and are displayed with a [jit.gl.text2d]
For the sequencer, to send the trigger messages for all that, I first wanted to write my own, but soon found out that Abletons Live is easier to use and fare more versatile that everything I could have come up with … So be it.
All this is controlled with Behringers a kind-of-heavyweight FCB1010 … I need to have my hands free, for playing bass.
Here is a screenshot of the patch. Lots of numbers, uh.