Hey all-

I've been using pd to mix audio, provide sound effects, and record audio for my dissertation project -- a serious game for teaching team members to coordinate. As the next step in this research, I need to be able to detect when each player is speaking to the others -- voice activity detection. Essentially logging the times that participants start and stop talking.

Each player has a bluetooth headset with a noise-cancelling microphone. Looking at the audio I've recorded directly from the headset, it's pretty obvious to human eyes when a participant is speaking, and when they are not, as the SNR is fairly high.

I am trying to use a combination of an envelope follower ([env~]) and a thresholder ([threshold~]) to create the start and stop events. I am having some difficulty in figuring out what the proper settings for the thresholder should be, and I'm also wondering if there are any other pd components that might be useful. One big problem that I'm encountering is that the thresholder catches the quiet portions of speech as turnoffs (generally followed almost immediately by another turnon). Does anyone out there have any experience with this? Or any advice?

Thanks!

-Zach

i would think that fiddle~ would be a good source of information about the audio

in the help file there is a bng that the fiddle~ object send an on and off signal to so you could use a delay on that bng signal so you could set a length of time before it would get shut off from recording

or you could use a spigot object on the amplitude readout from the fiddle~ object
and set up the math so that only if it dropped below a certain apmlitude would the spigot be activated or turned off

or you could use keyup (32 for example (space bar)) possibly so that a person could hold down the space bar for as long as they wanted to talk
unless that would be to much to do while playing the game

i hope at least the fiddle~ object will help a little with what your doing

or if all you are doing is logging the time they talk use the fiddle object and use like a realtime to get the readout in ms between bngs
i put something together real quick but this would be more just logging the face that there is activity and you could weed out shorter blips and longer more obvious speaking readouts
im not sure if this is what you need or not hope it helps

EDIT***
didn't post the patch sorry

http://www.pdpatchrepo.info/hurleur/fiddly.pd.zip

Thanks, Wednesday! I'll look into it.

I've been reading a set of papers on VAD, and it looks like a lot of the detection is based not only on amplitude, but also on zero crossing rate.

Right now, I think I'm going to look into adding a click track, so that the audio can be re-synched later and then processed offline.

Thanks for pointing out the fiddle object, I think it will help!

-Zach

Hi, guys. I am looking for something similar... I need PD to detect my voice and shoot a bang when I start speaking, and another bang (to another part of the patch) when I stop doing it. But... I think I need something that "smooth" the amplitude to avoid starting and stopping on every vocal. Also, I think that the attack detected by fiddle~ must bang the first AND THEN, the smoother will start working. The starting shouldn't be softened.

I'm new on this... please, explain me step by step. If you can draw the patch, better. If you build the patch, more better. I don't want to spend so much time learning to use PD... I have to start modeling and animating in Blender (*).

What I need is something to use in a project like this . I wrote this guy and he respond at the beggining... but not now. I don't know anything about him.

(*) This is my project:

Thanks!

Hi MarioMey,

With [iemlib/peakenv~] or [iemlib/peakenv_hold~] you can make your own attack-detector, and the smoother is built into these objects, you just need to set decay time or hold time. See attached.

http://www.pdpatchrepo.info/hurleur/peakenv~attackdetect.pd.zip

Thanks Katjav, but these patch only detects peaks... Let me explain with the PNG I've attached.

When I'm not speaking, my puppet doesn't move the mouth. The "voice detect number" (VDN), would be at 0 (zero). When I start speaking, a peak detect would lift the VDN to 100, for example. There would be a minimun limit to know when I am speaking. Then, a smoother would smooth the VDN to avoid go down below the minimum. But if I stop speaking, the VDN go to zero.

When VDN is more than 50, I am speaking. A bang is shooted to a "start moving the mouth" command sent by OSC. Below that, I'm not speaking... another bang is shooted to a "stop moving the mouth" command.

I think this patch would have the fiddle~ object.

http://www.pdpatchrepo.info/hurleur/voice_detect.png

If you're not doing pitch detection, then [fiddle~] is too much. I think katjav is right. [peakenv~] and/or [peakenv_hold~] is all you really need. Try the attached.

http://www.pdpatchrepo.info/hurleur/vdn.mmb.pd

I don't understand the most boxes... but it works! THANKS, MAELSTORM! I gave a threshold of 70 and decay, for the moment, it's ok on 250. When I test working together with Blender, maybe I'll change those numbers.

Now... I have to bang the OSC messages. I need something like this... but in PD:

Min = minimum #when I am speaking
NTP = numbers.to.process #gotten from...? snapshot~?
mouth_open = False

if not mouth_open and NTP > Min:
mouth_open = True
bang(open_it)
elif mouth_open and NTP < Min:
mouth_open = False
bang(close_it)

How...?

Just stick a [snapshot~] and [!= 0] at the end of the patch.

Maelstorm... finally, I could make it work. Some day, I will show you.

I have a doubt about the patch you did for me. I know that there is a metro that activate the array every 300ms. Why, if I change that number to other different from 300, the array stops? Does it affect the entire voice detection system? How can I change to 100ms? There's something else to change?

Thank you very much.

When you change the argument of [metro], it re-instantiates itself, and by default [metro] is turned off. So you have to turn it back on by banging it or sending [1( to it.

Maelstorm... there's something strange. I have tryied what you told me, before I posted. I knew that... but I just try different numbers and I discover that the minimun is 227 and the graph update is slower than 300! But, If I put 230, it's faster than 300. A lower value doesn't work.

Another question: Is any relationship between peakenv_hold~ and the pulse of metro? They were at 300. Peakenv_hold~ is the time of holding the peaks, isn't it?

The visualization consume process, doesn't it?

I think the problem actually has to do with the size of the array. The array isn't redrawn until [tabwrite~] finishes writing all the samples to it. So if you set the [metro] to fast, it will retrigger [tabwrite~] before it's finished, and you won't see the updated display.

And, yes, I set [peakenv_hold~] to have the same time as [metro] to make sure that every time [snapshot~] received a bang it would show the highest value between updates.

I'll check what you told me. For the moment, check the patch working: