Following up on some posts from the "Symbols explicit" thread, where there were some questions about concrete use cases.

To be honest, I'm not sure how clear this is going to be, because the best way to make it clear is to build it in Pd, and I don't have time to do something complicated.

Anyway... consider a "data stream" interface. Some part of the code asks the stream for the next value, and -- in text languages -- the value is returned to the caller. There may be dozens of places that need to access the stream, and the rule is, whoever asked gets the result.

I had mentioned parsing. If the input string is held in a stream wrapper (in SC, CollStream), then this object can be passed around the system as a function/method argument, and everywhere that needs the next character can just ask for it.

The challenge, then, in graphical patchers is that there is no concept of returning a value to the caller. A message traversing a patch cable is calling into the next object. "Returning from a function" is, in patching terms, backtracking to a prior object to see if it has more output cables that haven't been handled yet. But data don't travel backward up the stack, only down.

A file handle can be treated as a stream. Here, Pd recognizes the issue and allows shared file handles to be "define"-d: [file define xyz], and then every "caller" can have its own [file handle xyz].

If the data source is in memory, though, then the current position in the stream has to be maintained within the abstraction -- and there can be only one of those, because Pd/Max don't have a concept of a pointer to an abstraction that's loaded somewhere else. E.g., a really simple data queue (didn't implement "rewind" here) -- as an abstraction, everything would have to go through this.

So then there has to be a way to request a value from a specific part of the patch and get back to that part of the patch.

... where oid said, "In that situation I would say that [pd my-stateful-thing] should have a send method" -- seems to me that's what the above is (just not folded into the subpatch).

Regarding your live-coding dialect, would be curious to see it if you shared it anywhere.

https://github.com/jamshark70/ddwChucklib-livecode -- see https://github.com/jamshark70/ddwChucklib-livecode/blob/master/cl-manual.pdf for full documentation.

The parser (with heavy use of CollStream throughout) is at https://github.com/jamshark70/ddwChucklib-livecode/blob/master/parsenodes.sc

Example:

Most of what I do in pd is designing languages

Quite curious about that. Anything public to see?

hjh

@oid the [v] for the index is interesting. I wouldn't have thought of that.

(I guess?) I still lean slightly toward the targeted-send approach, over the spray-everywhere approach. One reason is, if you're using lists, sending a sub-list to one place is likely to be more efficient than sending it to potentially dozens of places. It's a valid concern, about redundant list allocations; one way to deal with that is to be less promiscuous about sending data to places where they won't be used.

I do appreciate everyone's indulgence. I've certainly gotten more comfortable with patching over the years, but I'm never quite sure if what I'm doing is really idiomatic.

hjh

@ddw_music Value has been mentioned multiple times in both threads It has been fun. I took a break from my projects and threw together asStream, it will be awhile before I get back around to it but here it 99% complete, there is a stupid error on the append so it is actually insert and for some reason I just can not figure out it is not resetting the [v] when it completes. You can solve those if you care. Oh, and it spits out an extra repeated number second to last, at least with my simple test, not sure why but worst case a [change] between the [array get] and [s] in stream.pd will fix that. Report back if you figure out why it is not resetting to zero, driving me nuts but I have other things to do. Patch in next post.

Not sure what you mean by "spray everywhere," these are targeted approaches and there is no copying. Does reallocate on resizing the array but that could be avoided easily enough, just use an array bigger than you will ever need and only use what you need.

Realized my errors finally, extra output was because I forgot to add a [- 1] after a [list length]. The [v ] not resetting was order of operations but decided that the logic of automatically resetting to zero was flawed so now it resets only if you send [start( when it is at the end up the array. The stupidest mistakes are always the hardest to see. Fixed and added an optional delay before the metro so they don't have to all start at the same time. Fixed [append( as well and give it a right inlet for appending without the [append list( which is probably how it should be? Still could use some logic to make sure you don't duplicate streams and perhaps the ability to configure individual streams but that is for someone else or some other day. Added some usage notes in the patch.

Improved version below.

@oid said:

Added some usage notes in the patch

don't see it, where are the help files?

@oid said:

@ddw_music Value has been mentioned multiple times in both threads

I know, and I've used value in other places. I hadn't thought to connect it to the current index, though.

Not sure what you mean by "spray everywhere," these are targeted approaches and there is no copying.

Sure, that's a misreading. I see also (on closer inspection) that folding the metro into the stream abstraction makes the stream effectively its own caller, sidestepping the return to caller problem. Though this means that your approach could be used only in self-timed situations, where SC streams and Python iterators are more general.

Haskell people like to say that a programming language isn't worth learning if it doesn't change the way you think. Pd has actually done that for me, to some extent.

hjh

@porres No help file, usage notes in asStream.pd, this thread is the documentation. If I ever get it to the point where I don't think there will be any major changes I will upload it with help files over in Abstractions. But a new version, reworked it as an array object since that is what it really is. Also added looping. Still needs lots of work to be a useful abstraction for more than learning from, it is kind of fun though and eventually I will develop it more.

asStream.pd
stream.pd

@ddw_music said:

Though this means that your approach could be used only in self-timed situations,

Just need to change [obj 10 10 stream $1 $2 $3...( to [obj 10 10 $1 $2 $3...( and add in some simple logic and now asStream can load the abstraction you specify through methods/arguments. As I said earlier, you design the interface and the protocol which is what those sends and receives and values and dollar arguments in asStream are, an interface and protocol for communicating with the stream.pd abstraction or any other abstraction designed to work with that interface. Object names themselves can also be filled in through arguments, [$1 $2] can be either a metro or a receive or anything else but [$1 $2 ....] are actually [list]s so they only give you two inlets and one outlet, if you need more than that you need to resort to editing the abstraction in a text editor, patch it together with any object with suitable io and then open it in a text editor and edit that object into $1 $2... And there is plain old dynamic patching to programmatically create the required patch. Lots of option and they can all be mixed.

@oid said:

@porres No help file, usage notes in asStream.pd, this thread is the documentation.

You see, I'm interested in this thread, I suggested that it would be created. I'm interested in the problem and how to make it work in Pd. I read the thread. This that you're sharing is not enough for me. I get that you may be satisfied and that you worked a lot and just about as much as you care on this, but I just wanted to make this clear. Ok, I know, I could do more of my homework, but I'm born and bred in Pd-land, all this just sounds way too complicated and like explaining 'snow' to me, who's also born and bred in a tropical country (I was at the beach yesterday by the way), so more help would be appreciated.

I have brought myself into such Pd x SC challenge other times, it's not the first time that things don't get too conclusive. I hear that it's impossible to do something "x", or too hard, and I hear about complex programming paradigms, and then it comes down to a simple solution like using [value]. My idea is to maybe try and expand Pd to improve it and open it up to what some people that come from elsewhere struggle with, and maybe offer a new object in my ELSE library... but all I'm getting so far from this thread is that you can share the index of an array with [value] in multiple subpatches or abstractions.

I was interested in studying this abstraction and see if maybe I could add it if I really got what it did and what it is for, but for now it's just quite abstract and over my head.

cheers

@porres I am more than happy to answer any questions you have about using it or how it works. As I said in the other thread, I hate doing documentation, it tends to turn into a rabbit hole for me and eat up a good deal of time; once I start thinking about usage I start seeing how some things are not right in the abstraction and I start fiddling which means a change to the docs and it quickly turns into a feedback loop. That small change to the manual I suggested in the other thread caused/required me to reread much of section 2 and reread a few sections many times even though I had it all typed out after reading two short section once.

I don't think extending pd would solve this issue, the question was not really how to do this in an isolated situation but about the methodology. How do you do something like functionA (functionB (1 2) 849 functionC ($1 $2) asdf) in pd? In a one off situation it is generally a fairly simple problem to solve but to make it so at anytime you can do the equivalent of [abstractionA [abstractionB 1 2] 849 [abstractionC $1 $2] asdf] is a much more complicated issue and one which externals don't solve and often make it seem more complicated than it actually is because they have a tendency to make pd feel like a higher level language than it is, keep the user from ever really learning the nuances and techniques of the low level dataflow paradigm that is pd. The one off solutions which often come up in these threads result in what I think @ddw_music refers to as "spray everywhere," to try and build that solution into every abstraction so you can use it everywhere but to get the equivalent of abstractions calling abstractions would require you to fill your abstractions with [v ]s and [send]s and [receives]s in the vain attempt to cover every possible situation and very complicated/convoluted bit of inter abstraction communication. All my posts since the first asStream screenshot post have been about methodology, how to build higher level paradigms in a low level language like pd and avoid spraying everywhere, this is what asStream is meant to show. Like any well founded low level language pd is quite good about giving you the tools to build out higher level functionality and adapt it into your own language. At least that is what I got from this thread.

asStream has the potential to be a really useful abstraction but still has some issues, I can help you understand it and flesh it out, just ask your questions, they will probably make it a better abstraction than either of us could make on our own. As an exercise I will attempt to repatch it in your heavily subdivided patching style on my next break from my projects, I will probably fail but it will be interesting, I get the logic of the style but it obfuscates the dataflow for me and anytime I want to understand the work of a subdivider I have to remove all that subdivision, I can't even see how it could be subdivided into discrete, well named subpatches of highly singular/specialized function, for me it already is that.

FWIW, I interpreted "spray everywhere" to mean that Pd "functions" spray their return values to every place that could potentially need it, depending on the control flow. That's a common way of dealing with Pd's lack of intrisic "return to caller" when there are just a few callers.

@porres

for now it's just quite abstract and over my head

I made a concrete example -- with scores! If the programming-ese terminology is too difficult, maybe focus on that example.

I hear that it's impossible to do something "x", or too hard, and I hear about complex programming paradigms, and then it comes down to a simple solution like using [value].

That's a good point. Sometimes it takes effort to find the simplest way.

@oid

I don't think extending pd would solve this issue

Yeah, the core of the issue is that Pd's stack works differently from most text languages since Fortran.

Tbh I think variable-target [send] is fine, once it's been established that [send] is going to be involved in pretty much any Pd solution. "Return to caller" as in any structured or object-oriented language is nicer, but [send] is not bad.

what I think @ddw_music refers to as "spray everywhere,"

Here, I'm referring specifically to https://forum.pdpatchrepo.info/topic/15085/return-to-caller-followup-from-symbols-explicit/8 , where the [send] pushes the value out to all matching receivers -- to every potential caller -- where it's swallowed by an [f]'s cold inlet, and only the specific caller pushes the value forward. If you have 50 potential callers, then the data have to be replicated 50 times.

If the data are sub-lists instead of single floats, that would be a lot of multiple allocation. My/jameslo's solution doesn't do that; nor does yours.

to get the equivalent of abstractions calling abstractions would require you to fill your abstractions with [v ]s and [send]s and [receives]s in the vain attempt to cover every possible situation

[list-map] in list-abs shows a nice approach, though, decoupling the iteration logic from the user-operation: the abstraction outputs a trigger to perform the operation, and then the result of that operation is put back into the abstraction to finish the job. That could inspire a super-minimalist approach (which has the benefit of assuming nothing about the returned value's type, whereas most other solutions posted here seem to assume it will always be a single float):

Now, I didn't think of this until the discussion went through a few iterations... see porres's comment about the eventual solution often being much simpler than the problem seems at first.

hjh

@ddw_music said:

"Return to caller" as in any structured or object-oriented language is nicer

I think this thread may have convinced me that the idea of return to caller is antithetical to dataflow, it goes against the flow of data and my solution which you said side stepped it is just restating the problem into dataflow. Not that this was my intent, didn't even realize I sidestepped it until you mentioned it but it got me thinking. Not sure if it is even possible or practical to avoid but I think I am going to try and be more rigid with the flow of data for awhile and see what happens.

Here, I'm referring specifically to

That is what I originally thought but the previous post confused me, I now realize that you weren't addressing my patch directly but offering a summation of what you got from the thread in your reply to me.

whereas most other solutions posted here seem to assume it will always be a single float):

Just replace [array] with [text] and make a tiny change in stream.pd/add other abstractions asStream can load/change the arguments it is loaded with. I switched from [text] to [array] because you used an array in your SC example and I switched tactics to confront your example head on. Originally I thought your ignoring the suggestion of [value] was that you wanted to use things other than floats and then thought that what you were having issues with is the methodology. This is probably a part of the problem porres mentioned about people having issues transitioning to pd from another language, you have a good grasp on programming and when you ignore a solution which seems the easy/obvious solution to us it is easy to assume you have a good reason for ignoring it. These sorts of assumptions are probably reasonably common in such situations, we all gave up on the suggestion of [value] early on and it came back into the discussion purely on accident.

It has been an interesting thread, the failures in communication ended up surprisingly insightful on many fronts.

I'm sorry I don't read the full thread, English is not my natural language and it's a bit technical…
Nevertheless I'd like to note that even in written languages the return to caller process at low level is coded by giving the called method an address to store the result in and then the caller can read it.

Pseudo assembly code

MAIN:
give me address to store N bytes
push this address to the return register
call the method
METHOD:
do stuff
get the address from the return register
write N bytes at this address
:END 
read N bytes at the address
do stuff with this result

The full code is unfoldable as a control flow.
The N bytes can be a pointer to an address the method allocates so the return data size can be set by the method.

The easiest way to simulate this in puredata is to create a memory address as [value] do for float. The caller can then give the address to the method with an argument or a message.
As data structures are pointer-accessible they can certainly help to create such a feature.

@FFW said:

Nevertheless I'd like to note that even in written languages the return to caller process at low level is coded by giving the called method an address to store the result in and then the caller can read it.

I think (not a major expert here) that most text languages do this on the stack: an operator or function leaves its result on the stack, and returns to the point in the code that called it. Nested calls like myFunction(random(20) * 2, random(20) * 3) are easy for an interpreter to handle if they're:

1. Push 20
2. Do "random" on the top value.
3. Push 2.
4. Do * on the top 2 values.
5. 6, 7, 8. Similar 4 steps.
6. Do "myFunction" on the 2 values left on the stack (and leave its result there).

In that sense, high level languages are just a less mind-bending way to write FORTH.

(oid is right that Pd uses a stack for something quite different, such that returning data back upstream really can't be a thing. But workarounds aren't all that painful in the end.)

hjh

@ddw_music said:

I made a concrete example -- with scores!

which one exactly do you mean?

Probably best to skip this post and just read the next, will leave it here since it could result in useful replies from those who understand the internals of pd and perhaps some people here are as interested in these sorts of details as I am,

@ddw_music I think pd uses the stack in exactly the same way but the difference here is that pd lacks a return stack (and stack/instruction pointer manipulation); abstractions are actually macros which are expanded, they are not functions which are pointed to and returned from. The unique ID given to an abstraction when it is expanded allows us to use them sort of like functions and mimic scope but pd is a very simple globally scoped imperative language.

Edit: I guess pd does have very basic IP manipulation, [send] is essentially a goto. The big difference with pd and dataflow is that the IP carries a bit of our data with it as it moves through the patch. The result of an operation does not go onto the stack but is carried with the IP. Or maybe data is only carried with the IP in a conceptual sense and result actually is TOS? Think I will have to investigate pd's stack when I have the time.

Edit2: I guess wires are also gotos then. I suppose what this really is is that pd is a 2d language, [send] and wires only seem like gotos if we try and think of pd as a 1d text language.

Edit3: (last one) Pd would have a return stack, but the returns always point to branches. A [t b b b b] is essentially push and drop commands for the return stack, rightmost bang pushes, leftmost drops. End of the branch having an explicit pop so it can return to the [trigger] or where ever the code branched. More like other languages than I thought in stacks, assuming my reasoning is sound, the big difference being that pd has no concept of anything remotely function like. IP probably carries the pointer as well as the data, would save popping everytime it return which means no return stack and back to my original point of pd lacking a return stack (subrosa edit4.) But that would mean the IP caries a return stack in a rather funge like way, which it might, the return stack is part of the IP object and not the interpreter proper (subrosa edit5.)

I guess I went off on a tangent there, went into implementation and lost focus. The important thing there is that abstractions are not functions which you return from or objects you point to, they are macros which expand in place. We can sort of use them like objects or functions but they are macros from the users perspective and that is what we need to keep in mind.

@oid said:

Edit2: I guess wires are also gotos then.

I realized a bit later what the difference is -- In Pd, wires are procedure calls, not gotos.

Continuing the tangent, feel free to disregard

Pascal distinguishes between procedures and functions, where a function has a return value and a procedure does not. (C gets rid of the term "procedure," but you can still have void functions.) In SmallTalk (and SC inherits this), there is always a return value, which may be ignored -- hence no such thing as a procedure. The last thing a function or method does is to return a result and drop its own execution frame off the stack. In rand(20) * 2, after rand returns the random number, there is only the result -- the fact that rand was called at all is forgotten. (Returning from a procedure just drops the execution frame, with no return value.)

In dataflow, there are only procedures and the last thing an object does is to procedure-call the next object -- the entire history stays on the stack. In [random 20] --> [* 2] --> [print], "random" is more like: 1. Get from the RNG; 2. Call descendants with this result; 3. Then backtrack up the stack.

It's kind of like, Random(20, descendants: [Multiply(_, 2, descendants: [Print(_)])]). I could hypothetically create these objects in SC and it would work (but, not much point lol).

The two approaches are mirror images: lisp would write (print (* (random 20) 2)) where the last operation to be performed ("print") is at the head of the tree. Structurally SC is the same. Syntactically it may be different, since a method selector may be either pre- or post-fixed, but (rand(20) * 2).postln still parses as follows. Then it renders bytecodes depth first! So internally it ends up like FORTH ("push 20, random, push 2, *, postln").

   postln
     |
     *
    / \
rand   2
 |
 20

But in Pd, the last operation is structurally the innermost leaf, not the head.

I think this explains a bit why it took me a good couple of years to wrap my head around dataflow, even though (I thought) I knew a thing or two. It's literally upside-down world for me. Still struggle with that sometimes.

hjh

@porres said:

which one exactly do you mean?

https://forum.pdpatchrepo.info/topic/15085/return-to-caller-followup-from-symbols-explicit/9

With a bit of clarification at https://forum.pdpatchrepo.info/topic/15085/return-to-caller-followup-from-symbols-explicit/13

hjh

@ddw_music said:

In Pd, wires are procedure calls, not gotos.

Are they actually procedures or just conceptually? I would assume the latter but if they actually are procedures can you point me where to look in the source? I get the depth first stuff, I just turned myself around in those edits, hence the follow up post and the disclaimer added to the original post. I did a bit of digging in the source and I think my original point was correct, pd does not do anything weird with how it uses the stack, it is just that without a return stack we can't actually return which means we have to go about/think about things in a different way.

lisp would write (print (* (random 20) 2)) where the last operation to be performed ("print") is at the head of the tree

But there is no tree in lisp, right? it is just a list data structure for the REPL to evaluate. I can't quite grasp how lisp works under the hood, I poisoned my brain with one of those build your own lisp tutorials which I later learned did not actually function in anyway like lisp proper, it was just an easy way to do a lisp like language and I have been lost ever since. Lisp is one of the languages I really want to learn and understand because it is supposedly great for building languages but there is something about it which I fail to grasp.

Regarding Forth since you keep bringing it up, it is no where near as mind bending as people think, in practice you don't spend much time doing that low level RPN stuff; you define some words and use those words to define other words and before you know it you have a purpose built high level DSL for your task. But you have to learn all that low level RPN pushing and popping stuff first. I am a big fan of Forth and what got me interested in language design/implementation was the whim of implementing a language in pd, so I found a language which would seem suitable for that whim and it was Forth, I learned Forth by implementing it in pd.

@ddw_music said:

@oid said:

Edit2: I guess wires are also gotos then.

I realized a bit later what the difference is -- In Pd, wires are procedure calls, not gotos.

In my mind wires (and hidden wires [s] [r] [v]) are variables and boxes are procedures…
You share data throw wires as you transmit them with variables or stacks or registers. The order you link the boxes give you the order the data stack is filled.
By the way, it's also the order the process stack is filled because the full tree is flatten before it computes.