Hello,

this is going to be a long one.
After years of using PD, I am still confused about its' timing and schedueling.

I have collected many snippets from here and there about this topic,
-wich all together are really confusing to me.

*I think it is very important to understand how timing works in detail for low-level programming … *

(For example the number of heavy jittering sequencers in hard and software make me wonder what sequencers are made actually for ? lol )

This is a collection of my findings regarding this topic, a bit messy and with confused questions.

I hope we can shed some light on this.

• a)
  The first time, I had issues with the PD-scheduler vs. how I thought my patch should work is described here:
  https://forum.pdpatchrepo.info/topic/11615/bang-bug-when-block-1-1-1-bang-on-every-sample

The answers where:

„
[...] it's just that messages actually only process every 64 samples at the least. You can get a bang every sample with [metro 1 1 samp] but it should be noted that most pd message objects only interact with each other at 64-sample boundaries, there are some that use the elapsed logical time to get times in between though (like vsnapshot~)
also this seems like a very inefficient way to do per-sample processing..
https://github.com/sebshader/shadylib http://www.openprocessing.org/user/29118
seb-harmonik.ar posted about a year ago , last edited by seb-harmonik.ar about a year ago 
• 1 
 whale-av 
@lacuna An excellent simple explanation from @seb-harmonik.ar.
Chapter 2.5 onwards for more info....... http://puredata.info/docs/manuals/pd/x2.htm
David.
“

There is written: http://puredata.info/docs/manuals/pd/x2.htm

„2.5. scheduling 
Pd uses 64-bit floating point numbers to represent time, providing sample accuracy and essentially never overflowing. Time appears to the user in milliseconds. 
2.5.1. audio and messages 
Audio and message processing are interleaved in Pd. Audio processing is scheduled every 64 samples at Pd's sample rate; at 44100 Hz. this gives a period of 1.45 milliseconds. You may turn DSP computation on and off by sending the "pd" object the messages "dsp 1" and "dsp 0." 
In the intervals between, delays might time out or external conditions might arise (incoming MIDI, mouse clicks, or whatnot). These may cause a cascade of depth-first message passing; each such message cascade is completely run out before the next message or DSP tick is computed. Messages are never passed to objects during a DSP tick; the ticks are atomic and parameter changes sent to different objects in any given message cascade take effect simultaneously. 
In the middle of a message cascade you may schedule another one at a delay of zero. This delayed cascade happens after the present cascade has finished, but at the same logical time. 
2.5.2. computation load 
The Pd scheduler maintains a (user-specified) lead on its computations; that is, it tries to keep ahead of real time by a small amount in order to be able to absorb unpredictable, momentary increases in computation time. This is specified using the "audiobuffer" or "frags" command line flags (see getting Pd to run ). 
If Pd gets late with respect to real time, gaps (either occasional or frequent) will appear in both the input and output audio streams. On the other hand, disk strewaming objects will work correctly, so that you may use Pd as a batch program with soundfile input and/or output. The "-nogui" and "-send" startup flags are provided to aid in doing this. 
Pd's "realtime" computations compete for CPU time with its own GUI, which runs as a separate process. A flow control mechanism will be provided someday to prevent this from causing trouble, but it is in any case wise to avoid having too much drawing going on while Pd is trying to make sound. If a subwindow is closed, Pd suspends sending the GUI update messages for it; but not so for miniaturized windows as of version 0.32. You should really close them when you aren't using them. 
2.5.3. determinism 
All message cascades that are scheduled (via "delay" and its relatives) to happen before a given audio tick will happen as scheduled regardless of whether Pd as a whole is running on time; in other words, calculation is never reordered for any real-time considerations. This is done in order to make Pd's operation deterministic. 
If a message cascade is started by an external event, a time tag is given it. These time tags are guaranteed to be consistent with the times at which timeouts are scheduled and DSP ticks are computed; i.e., time never decreases. (However, either Pd or a hardware driver may lie about the physical time an input arrives; this depends on the operating system.) "Timer" objects which meaure time intervals measure them in terms of the logical time stamps of the message cascades, so that timing a "delay" object always gives exactly the theoretical value. (There is, however, a "realtime" object that measures real time, with nondeterministic results.) 
If two message cascades are scheduled for the same logical time, they are carried out in the order they were scheduled. 
“ 

[block~ smaller then 64] doesn't change the interval of message-control-domain-calculation?,
Only the size of the audio-samples calculated at once is decreased?
Is this the reason [block~] should always be … 128 64 32 16 8 4 2 1, nothing inbetween, because else it would mess with the calculation every 64 samples?
How do I know which messages are handeled inbetween smaller blocksizes the 64 and which are not?

How does [vline~] execute?
Does it calculate between sample 64 and 65 a ramp of samples with a delay beforehand, calculated in samples, too - running like a "stupid array" in audio-rate?

While sample 1-64 are running, PD does audio only?

[metro 1 1 samp]  

How could I have known that? The helpfile doesn't mention this. EDIT: yes, it does.
(Offtopic: actually the whole forum is full of pd-vocabular-questions)
How is this calculation being done?
But you can „use“ the metro counts every 64 samples only, don't you?

Is the timing of [metro] exact? Will the milliseconds dialed in be on point or jittering with the 64 samples interval?
Even if it is exact the upcoming calculation will happen in that 64 sample frame!?

• b )
  
  There are [phasor~], [vphasor~] and [vphasor2~] … and [vsamphold~]
  https://forum.pdpatchrepo.info/topic/10192/vphasor-and-vphasor2-subsample-accurate-phasors

“Ive been getting back into Pd lately and have been messing around with some granular stuff. A few years ago I posted a [vphasor.mmb~] abstraction that made the phase reset of [phasor~] sample-accurate using vanilla objects. Unfortunately, I'm finding that with pitch-synchronous granular synthesis, sample accuracy isn't accurate enough. There's still a little jitter that causes a little bit of noise. So I went ahead and made an external to fix this issue, and I know a lot of people have wanted this so I thought I'd share. 
[vphasor~] acts just like [phasor~], except the phase resets with subsample accuracy at the moment the message is sent. I think it's about as accurate as Pd will allow, though I don't pretend to be an expert C programmer or know Pd's api that well. But it seems to be about as accurate as [vline~]. (Actually, I've found that [vline~] starts its ramp a sample early, which is some unexpected behavior.)
[…]
“

• c)
  
  Later I discovered that PD has jittery Midi because it doesn't handle Midi at a higher priority then everything else (GUI, OSC, message-domain ect.)

EDIT:
Tryed roundtrip-midi-messages with -nogui flag:
still some jitter.
Didn't try -nosleep flag yet (see below)

• d)
  
  So I looked into the sources of PD:

scheduler with m_mainloop()
https://github.com/pure-data/pure-data/blob/master/src/m_sched.c

And found this paper

Scheduler explained (in German): 

https://iaem.at/kurse/ss19/iaa/pdscheduler.pdf/view

wich explains the interleaving of control and audio domain as in the text of @seb-harmonik.ar with some drawings
plus the distinction between the two (control vs audio / realtime vs logical time / xruns vs burst batch processing).
And the "timestamping objects" listed below.

And the mainloop:

Loop
- messages (var.duration)
- dsp (rel.const.duration)
- sleep

With
[block~ 1 1 1]
calculations in the control-domain are done between every sample? But there is still a 64 sample interval somehow?
Why is [block~ 1 1 1] more expensive? The amount of data is the same!? Is this the overhead which makes the difference? Calling up operations ect.?

Timing-relevant objects
from iemlib:

[...]
iem_blocksize~   blocksize of a window in samples
iem_samplerate~  samplerate of a window in Hertz 


------------------ t3~ - time-tagged-trigger --------------------
-- inputmessages allow a sample-accurate access to signalshape --
t3_sig~          time tagged trigger sig~
t3_line~         time tagged trigger line~


--------------- t3 - time-tagged-trigger  ---------------------
----------- a time-tag is prepended to each message -----------
----- so these objects allow a sample-accurate access to ------
---------- the signal-objects t3_sig~ and t3_line~ ------------
t3_bpe           time tagged trigger break point envelope
t3_delay         time tagged trigger delay
t3_metro         time tagged trigger metronom
t3_timer         time tagged trigger timer

[...]

What are different use-cases of [line~] [vline~] and [t3_line~]?
And of [phasor~] [vphasor~] and [vphasor2~]?

When should I use [block~ 1 1 1] and when shouldn't I?

[line~] starts at block boundaries defined with [block~] and ends in exact timing?
[vline~] starts the line within the block?
and [t3_line~]???? Are they some kind of interrupt? Shortcutting within sheduling???

• c) again)
  

https://forum.pdpatchrepo.info/topic/1114/smooth-midi-clock-jitter/2

I read this in the html help for Pd: 
„
MIDI and sleepgrain
In Linux, if you ask for "pd -midioutdev 1" for instance, you get /dev/midi0 or /dev/midi00 (or even /dev/midi). "-midioutdev 45" would be /dev/midi44. In NT, device number 0 is the "MIDI mapper", which is the default MIDI device you selected from the control panel; counting from one, the device numbers are card numbers as listed by "pd -listdev."
The "sleepgrain" controls how long (in milliseconds) Pd sleeps between periods of computation. This is normally the audio buffer divided by 4, but no less than 0.1 and no more than 5. On most OSes, ingoing and outgoing MIDI is quantized to this value, so if you care about MIDI timing, reduce this to 1 or less.
„

Why is there the „sleep-time“ of PD? For energy-saving??????
This seems to slow down the whole process-chain?

Can I control this with a startup flag or from withing PD? Or only in the sources?

There is a startup-flag for loading a different scheduler, wich is not documented how to use.

• e)
  
  [pd~] helpfile says:

ATTENTION: DSP must be running in this process for the sub-process to run. This is because its clock is slaved to audio I/O it gets from us!

Doesn't [pd~] work within a Camomile plugin!?

How are things scheduled in Camomile? How is the communication with the DAW handled?

• f)
  
  and slightly off-topic:
  There is a batch mode:
  https://forum.pdpatchrepo.info/topic/11776/sigmund-fiddle-or-helmholtz-faster-than-realtime/9

EDIT:

• g)
  I didn't look into it, but there is:
  https://grrrr.org/research/software/

clk – Syncable clocking objects for Pure Data and Max
This library implements a number of objects for highly precise and persistently stable timing, e.g. for the control of long-lasting sound installations or other complex time-related processes.

Sorry for the mess!
Could you please help me to sort things a bit? Mabye some real-world examples would help, too.

As already noticed, proceed the control messages and compute audio vectors of the DSP graph are interleaved operations. The internal audio vector size is 64.

[64][control][64][control][64][control][64][control]

What's happen in a 1 sample reblocked subpatch? In short, instead of compute 1 vector of 64 samples, it computes 64 times following a vector of 1 sample.

 [1][1][1]...[1][1][1][control][1][1][1]...[1][1][1][control]

AFAIK, no messages is trigger faster than (very) roughly 1 ms (and that is true for the metro object also).

The tricky stuff with real time audio is not to do the things fast, but do things fast at the right time. Wait the sound in, deliver the sound out, compute the next sound and wait. When i benchmark my fork for instance, most of the time is spent in sleeping!

@Nicolas-Danet said:

control messages and compute audio vectors of the DSP graph are interleaved operations. The internal audio vector size is 64.

[64][control][64][control][64][control][64][control]

Ok, i see.
But I read control messages are first, then audio. f.e. [loadbang] is proceeded before an upcoming audio-block.
And [vline~] is calculated and "drawn" before and "modulating" the *following * audio blocks.

[control][64][control][64][control][64][control][64]

What's happen in a 1 sample reblocked subpatch? In short, instead of compute 1 vector of 64 samples, it computes 64 times following a vector of 1 sample.

And there is no way to change this interval rate of 64?
Is upsampling in a subpatch increasing the computation time-interval of control-messages?

The tricky stuff with real time audio is not to do the things fast, but do things fast at the right time. Wait the sound in, deliver the sound out, compute the next sound and wait. When i benchmark my fork for instance, most of the time is spent in sleeping!

I see. In the analog world it is very different. This is why we have the buffer-latency in digital everywhere:

...incoming audio-samples in blocks, computation, audio out, and again...

And the control-domain every 64 samples.
For me as "user" of PD this is confusing.
So every object with a [v ...] will be sampleaccurate on point in upcoming audio-blocks, as long as it is not needed more often then 64 samples later!??? i.e. as long it is not starting several times in a smaller interval then 64 samples?

@lacuna The whole patch is recompiled within Pd and I think that although the data flow model is fantastic it makes it harder to understand the workings.
The blocks (of audio) are read, or generated, and all of the stuff that the patch needs to do to the block is done all at once to every sample in the block, and then the block is sent onwards.
So if you put [x~ 2] >> [/~ 2] then nothing is done..... the code that Pd is running has done the math and the result is "multiply sample values by one".......... so "do nothing". A complex patch will have been boiled down to "subtract x from sample1" "add y to sample2" etc...... up to sample 64, rinse, calculate the next set of additions and subtractions to apply, and do it to the next block.

Those operations..... add to sample value... or subtract from sample value.... are the only possible operations on a sample value.......
Interpolation uses adjacent sample values for the calculation, but adding or subtracting to / from the sample values is what happens when the calculations have been done.

Some objects like [x~] can be controlled by a control signal, and so the new value can only be applied at block boundaries as the control calculations are done between boundaries. The addition will be the same for every sample in the block. Pd didn't know in advance what it's next value might be, so a ramp cannot be applied across the samples in this block.
Some objects though, like [vline~] are scheduling changes of value that will happen across the block, and future blocks, and may finish at sample 43 within a block. Programmatically it is saying, as part of the whole patch "add a bit to sample 1 (if it has a +ve value or subtract if -ve)) and a bit more to sample 2 etc..... etc... and then for the next block, when the audio program runs again add even more to the 1st sample etc..... until.
So it is sample accurate.
And of course if [x~] is controlled by [vline~] it will do as it is told and be sample accurate too.
You can add a start delay to [vline~] so that it's start point is sample accurate too.

So every object with a [v ...] will be sampleaccurate on point in upcoming audio-blocks, as long as it is not needed more often then 64 samples later!??? i.e. as long it is not starting several times in a smaller interval then 64 samples?

I guess, but i'm not 100% sure (tests are required). I didn't remenber precisely how [vline~] is implemented thus its behaviour in reblocked cases. IMHO if you want sample accurate operations, everything should be stay in the DSP context as far as possible. I mean use only DSP objects and all is always in sync.

@Nicolas-Danet [vline~] schedules its operations. But it has a control rate inlet, so it can only schedudule at block boundaries. But once it has it's schedule it will add that to the program and can be starting and stopping many times within blocks. That is my understanding.
It could be tested to see whether its control rate message inlet will take (in msecs) small enough float values to make that happen.
See C04.control.to.signal.pd in the doc.
You will need then to calculate how many blocks you need to schedule for until the end of the last schedule matches a breakpoint so that you can safely send in a new control message at that point.
David.

@whale-av I think you are right. IIRC [vline~] accept a list of breakpoint that could be used to make multiple ramps inside the 64 sample range. Need to be tested.

@whale-av said:

It could be tested to see whether its control rate message inlet will take (in msecs) small enough float values to make that happen.

here is a test patch vline-test.pd. vline~ accepts messages at any time with sub-sample accuracy. This is also documented in the help-file.

@lacuna said:

[metro 1 1 samp]  

How could I have known that? The helpfile doesn't mention this. (Offtopic: actually the whole forum is full of pd-vocabular-questions)

It is documented in the metro-helpfile since pd-0.48 i guess. But i agree, there is not much redundant information in pd's helpfiles, to put it mildly

But you can „use“ the metro counts every 64 samples only, don't you?

yes, except you use it with vline~ (see test-patch)

When should I use [block~ 1 1 1] and when shouldn't I?

whenever you need delay~ times smaller than 64 samples.

merci beaucoup!
I just edited my first post and added clk lib.