Hi all,

eversince I saw that for the first time some ten years ago I was more than fascinated. It's an artistic realtime audio
analysis/remapping engine made with PD and python(?).

The creator mentioned back then that he wants to release it - which hasn't happened in more than 10 years. So I am wondering how hard it is to recreate that for studying and fun purposes.

I'm at an experience level where I can imagine creating a link between an external database and Pd but I would still have to research the details.

Hence my questions:

1. How do you think the audio is beeing analysed in the beginning? What are the criteria for discriminating individual samples? he speaks of ~100 dimensional vectors for analysis - that's beyond my capacity. I don't even know how to research that but feel I only need good directions in order to proceed extending my understanding.

2. When I have chunks of audio plus their analysed properties stored externally - What, generally speaking, are the options for connecting to an external database from Pd? Writing an external for the live communication? Or are there other techniques I might have missed so far?

Thanks for any thoughts. I'm hoping not only to get deeper insights for myself but also to trigger a learning discussion among fellow enthusiasts.

Kind regards,
oyster

@oystersauce very nice project. it seems to be made with the timbreID library.
i would also like to know how to do something like that.
regarding your questions:
1:
the patch is mentioned in this paper: http://williambrent.conflations.com/papers/timbreID.pdf
"4.2. Target-based Concatenative Synthesis
Some new challenges arise in the case of comparing a con-
stant stream of input features against a large database in
real-time. The feature database in the vowel recognition
example only requires about 20 instances. To obtain inter-
esting results from target-based concatenative synthesis, the
database must be much larger, with thousands rather than
dozens of instances. This type of synthesis can be achieved
using the systems mentioned in section 1, and is practiced
live by the artist sCrAmBlEd?HaCkZ! using his own soft-
ware design [5]. The technique is to analyze short, overlap-
ping frames of an input signal, find the most similar sound-
ing audio frame in a pre-analyzed corpus of unrelated audio,
and output a stream of the best-matching frames at the same
rate and overlap as the input."
2:
just discovered that pure data can read csv files with binfile from mrpeach.
perhaps that is a way to read an external database?

@Jona thanks for your reply. the paper you linked is gold worth. nails it. hopefully I can soon find the time to get into it more deeply and understand it. I will share any results here.

concerning the the database: having more of a diy-webdesigner background I always think of dedicated database software (like SQL or MySQL) as soon as someone says "database". now I assume that the scrambled hackz man, koenig, and also the paper you posted use the term in a broader sense, i.e. basically any organized collection of data, which could as well just be a list of values (like properties of audio snippets for example). and pd can do that internally. I also assume that an external database would result im immense and unneccessary overhead.

I played a bit with the settings from the concatenative example from the timbreId library. Really fascinating, even if the result is far from perfect. I used a snippet from a speech from Bercow as the source and used a snippet from Adorno as realtime input. The voice from Adorno is synthesized with the voice from Bercow if I understand it right? Would be nice to improve the result.
And it would be really nice if it is possible to recognize a whole word or a sentence, jump to the most similar position in a movie and play from there on. But I do not really understand how it works yet.
Here is the modified example:
concatenative_video_mod.zip

@Jona this is so nice, nice video! makes me really excited to try it out myself. right now I'm just coming back from a travel and hope I still get to it later.

concerning your aim to recognize words and sentences - as far as I understood it, that's beyond the functionality of the timbreID library modules. the source audio signals (both carrier (preanalysed audio) and modulator (live audio input or another audio sample)) are divided into and analysed by in fixed, very small (some ~30ms) segments. each of these small segments of the live input is compared to the ones already preanalysed and the most fitting is selected and chosen for playback. in order to detect words or even phrases it would require additional logic (far beyond my knowledge) that would somehow detect a word's beginning and end and use that as the window size(?). I can't say if implementing it at all in Pd is possible or not. also sceptical if timbreID would even be a good basis to implement such an algorithm upon. if you find something in that regard let me know.