Let's have a sharing thread -- "things I wish somebody had told me when I was first starting with Pd."

My #1: Basic list operations of iteration and incremental "fill." It took me probably half a year to discover the [list] operators and be able to use them reliably for these two absolutely essential procedures. (It turns out that these operations aren't so difficult after all, but it took far too long to figure it out.)

hjh

@ddw_music That I could become as dangerously addicted as to some very powerful drugs.

@ddw_music That all list objects other than [list store] iterate through the entire list every time and that most things can be done more efficiently with [list store]

Edit: First one is thanks too @ingox
Edit Again; Actually I do not know that [list store] does not iterate through the entire list, it might just be more efficient because it can often do the work of multiple list object all by its lonesome so you only iterate the list once instead of twice in my vs your examples. Hopefully someone will chime in here.

@oid Hm, yeah, list store looks useful. The first one avoids making multiple list instances, should be more efficient.

My examples don't iterate twice though.

I wonder where I would have found these templates in Pd documentation. Taking apart lists is pretty basic, warrants more attention than community forum folklore.

hjh

@ddw_music From what I understand, every time you send to the hot inlet of the standard list objects it iterates through the full list. In your list iterate example the [list] and [list split 1] each iterate through the full list, making two iterations. Your second does not, I did not look close enough on my first glance, but it does iterate through the list once more than [list store] would (assuming [list store] does iterate), 5 times for [list prepend] and once for [list]. For short lists this would notf matter, but could make a big difference on large lists, and a huge difference if [list store] prepends/appends without iteration, we will have to wait for someone more knowledgeable to answer that or just accept that it is faster and be happy at that.

@oid said:

@ddw_music From what I understand, every time you send to the hot inlet of the standard list objects it iterates through the full list.

So mine look like O(n) but they're really O(n^2). I see... you're talking about internals.

This should all be documented... List operations are fundamental (and, even after a year, the solution I arrived at is not optimal).

hjh

@ddw_music Documentation is not one of PureData's strong points. Sometimes I think it's only real strong point is the one @whale-av alluded too and all of us have completely lost perspective.

@ddw_music https://forum.pdpatchrepo.info/topic/11122/faster-list-drip-with-list-store-pd-0-48
Arrays could be quicker for some purposes.
The zexy library has a range of externals for arrays/tables.
Probably cyclone/zl is worth a look for lists as externals are often more efficient.

I was pretty sure that [list foreach] was already there to iterate, but nope ...

.... just an old PR < https://github.com/pure-data/pure-data/pull/595 > never merged!

That using vanilla is not actually tying one hand behind your back. For years I used extended/pile of externals and that kept me from really learning pd, I just found an external to do what I needed. The reality is far more nuanced, vanilla is very capable, the gains offered by externals are often purely academic, and the ability to interact with the internal logic of an abstraction vs an external can make the vanilla abstraction the more efficient option in the end. I think the programmers in the community often forget that many who come to pd are not programmers or especially logical and the nay-saying against vanilla can result in dependence upon externals instead of understanding of the language.

This should all be documented... List operations are fundamental (and, even after a year, the solution I arrived at is not optimal).

Generally, I'd advise you to be cautious about any non-signal class in Pd where you aren't pre-allocating the number of items at object instantiation time.

Even if [list store] were improved so that it doesn't allocate memory on each call, that doesn't improve the worst case performance. And if you are trying to hit soft realtime deadlines with a chain of signal objects, worst case is all you should care about.

Also, notice that the use of alloca in the [list] family objects doesn't help you here because the implementation silently shifts between stack and heap allocation on an arbitrary size boundary. You can hack your way around that as I'm sure most Pd users do, but it's just not a great interface for a realtime environment.

@jancsika Are you saying that in situations where timing is critical a fixed sized array should be used instead of [list] objects? If not, can you give the laymen version for us non-programmers?

Edit; I assume a list created with [pack] would be an example of a list whose number of items are pre-allocated at instantiation time and would be as good as an array in this case?

Are you saying that in situations where timing is critical a fixed sized array should be used instead of [list] objects? If not, can you give the laymen version for us non-programmers?

Edit; I assume a list created with [pack] would be an example of a list whose number of items are pre-allocated at instantiation time and would be as good as an array in this case?

Yes, that's correct.

Now-- you can run into a situation where you need to manipulate a message that is large enough to be cumbersome for [pack] yet still small enough to process reliably using [list] objects. My point is that the interface of [list] doesn't tell you when you've gone from stack allocation-- which is predictable-- to heap allocation-- which isn't as predictable. It's hard-coded to make the shift at lists greater than 100 elements, but even knowing that it's hard to keep track in a non-trivial chain of objects.

Furthermore, it's not clear to the user which [list] objects actually need to copy lists in the first place. For example-- [list split] does not copy the list. That means that the size of the list doesn't affect the time the object takes to compute its output.

@oid said:

.... the gains offered by externals are often purely academic, and the ability to interact with the internal logic of an abstraction vs an external can make the vanilla abstraction the more efficient option in the end. I think the programmers in the community often forget that many who come to pd are not programmers or especially logical and the nay-saying against vanilla can result in dependence upon externals instead of understanding of the language..... <

I think there is a little too much vanilla evangelism.
Almost everyone using Pd is trying to get a job done.
I want to plonk down my laptop for a show and know that all my effort, and that of my clients, will not have been ruined by failure. So patches are tested to destruction and use any tool that makes success more likely.
That means I don't move on as quickly as some.
My laptops have not been connected to the internet for years....... have had no updates since they were proven fool proof.... and so the externals from extended still play a huge part.
For a long time I tried to go pure brown envelope. But it was hard. While extended was said to be redundant vanilla really didn't have the tools to replace it.
That changes slowly of course, but the 12 year period leading up to the release of extended was as explosive as Nasa going to the moon and throwing all that work and knowledge away to reinvent the wheel seems pointless to me.
Look back in time through this forum and you will almost always find the same discussion from long ago.
That is the real "wiki" "knowledge base" "documentation" problem with Pd....... that the information is almost always already available but no moderator has the time to research and step in with......
"Sorry.... this question has already been answered"....

And all those externals were written to solve problems for which Vanilla had no efficient solution.
Or no solution at all.
David.

@oid well abstractions are good for understanding the language and often good for quick development. However, c externals are almost always far more efficient/fast if the same general algorithm is used. Just for the simplest example, getting a "float" in pd requires a few function calls: sending a "bang" message to an object that calls its outlet function, which finds the connection and calls the "bang" method for float which then calls the outlet function after accessing the data itself (the actual float value). in c all that would be is 1 memory lookup, and would require the memory of just 1 float instead of an entire pd object. And that isn't even counting all of the extra space and processing it takes to manage the object and class.
You can do a lot in vanilla but there are definitely things that are way easier to do even in c. A good example is any list processing, or any searching or sorting algorithm, or any manipulations of more complex data structures.
Personally I like to use pd for signal-graphs and some message glue, with lua doing the structure and sequencing (with pdlua) and c doing the complex low-level or data structure stuff.

you should be able to make almost anything in vanilla, but for practical use it makes more sense to use compiled externals where possible. I wouldn't necessarily call the gains "purely academic" anyways.
of course you also have to consider how many dependencies you want your patches to rely on..

@oid In the audio domain..... sub sample process etc. externals are useful if not inevitable.
But even in patching efficiency (not processing efficiency).
With an external I can make a mixer.......

and this is what I had to do with dynamic patching in Vanilla to replace it.....
matrix~.zip
Plus a dynamic patching headache to connect it to the rest of the patch once it has built.

The ones I never managed to replace were [lowshelf] and [highshelf] and [equalizer] (/ggee) ..... simple math functions that my simple brain never managed to replicate in Vanilla.
Their absence from the Vanilla distribution makes [biquad~]...... that should be the most useful filter in Pd..... obsolete for any but the most initiated........ unless you actually need a 5512.5 Hz notch filter @44.1k......
David.

((( P.S. [equalizer] was included in Vanilla up to Pd 0.47.1...... and then in 0.48 it disappeared.
So it's absence was recognised as a problem..... and then it was dropped for "purity"..???.... or simply forgotten.)))

@jancsika So my standard practice of using list objects for things I pack and unpack and arrays for everything else is a good way to go. Don't think I have ever made a list of 100 items, at least not on purpose. Thanks.

@whale-av @seb-harmonik.ar To quote myself from that post

The reality is far more nuanced

Note that I did not say that reality was black and white and that externals are evil and should always be avoided. The point being that neither extremes are all that great, if you do not have a good foundation in vanilla you will likely not have much success in puredata., giving a beginner without that foundation an external to do the work for them can be detrimental to their learning and act as a crutch. I gave an example of when abstractions can be better and why, you get access to the internal logic of the abstraction, you do not get that with an external, if you need access to that logic the abstraction will often win compared to the external and the extra logic needed to fake having access to the internals. Between the poor documentation and the tendency to answer questions with externals it can be very difficult to really learn pd for the non-programmer. What really made it work for me this time when I decided to return to pd was that I requested vanilla answers to my questions and all of a sudden I got all sorts of in depth explanations, many ways accomplishing the same task, and links to other threads that go deeper instead of the names of some externals. Things I wish I had known when I started Pd.

@oid I agree...... except for the "difficult for a non-programmer" where I know from my experience that the opposite can be true.
David.

@whale-av That is why I said " it CAN be very difficult" instead of "it IS very difficult"

@oid Ahhh... sorry. I thought you were hedging your bets......

@seb-harmonik.ar

You can do a lot in vanilla but there are definitely things that are way easier to do even in c. A good example is any list processing, or any searching or sorting algorithm, or any manipulations of more complex data structures.
Personally I like to use pd for signal-graphs and some message glue, with lua doing the structure and sequencing (with pdlua) and c doing the complex low-level or data structure stuff.

This point shouldn't be overlooked.

There's a reason why graphical patchers never took over the programming world: they are actually quite bad at a lot of the things that typical programs have to do. (MSP himself admits as much.) Note, I'm not saying that graphical patchers can't, only that it's likely to be faster and clearer to express in other languages. Sufficiency != usability.

Pd: t b f b --> initialize float box to 0, then "until" --> float box --> t f f --> increment counter, and use the index into something else --> final bang to output the result

C: for(i = 0; i < n; i++) ... A bit ugly but already more concise.

SuperCollider: n.do { |i| ... } or if n is the size of an array, simply array.do or array.collect with a function body.

I recently watched a student struggle with the concept of recursive function calls in Lua, which showed me something about the way that patching wires can help visualize the process. So I'm not ignoring the appeal of patchers. But I don't want to use patchers for complex algorithms. (One concrete point: it's painful when you have to add an outlet in the middle of a t object. Really awful.)

I do understand the point of the discipline of doing it with vanilla. I'd suggest that there's always going to be a dividing line between implementation details you need to know and those that can stay under the hood, and that the position of this dividing line may be different for different people or circumstances. Coming to grips with pd list iteration helps me feel better about patching; but most of the students in my class couldn't care less. They have no commitment at all to this way of doing things (and the ones who are interested have already jumped ship and picked up Max). It would be silly to force them. If there's just an object they can use to pick a list apart, that's fine for them (and if somebody else built that object, then maybe I have time for my own projects).

hjh