I am trying to send real time audio data from Python to PD for processing and then send the processed data back to Python. I previously used connecting python and PD with Soundflower virtual sound card, but there was certain latency and distortion that didnt work very well for me. In order to achieve higher transfer rate and lower latency I thought of using UDP. I tried, however PD receives UDP data (bit number integer) in the format of two integers (eg. 520 is sent as 8 2 - 2x 256 + 8 ), which I send in chunks of 1024. Processing the data back into plausible sound data in range of -1 to 1 and sending them out using [sig~] seems to be beyond PDs processing capability, PD seems to output only the first integer of the chunk. Is there any more direct way to capture audio thru UDP? thanks

@whale-av My goal is to write a program that can encode an image/video into sound format (data bending), send the data in real time to Pd, where I can process the "audio" consisting of the raw image data using Pd effects and then send it back to Python where can I convert that to live video again.
I managed to do this already, using virtual sound card (Soundflower) as the connection between python and Pd. This worked, although the image seems a bit scrambled and distorted when send thru this "chain" uneffected,. and the frame rate, at which the image can be processed (and thus the video streamed) is very low, because it depends on the maximum sample rate, which is 192kHz for Soundflower(that is roughly 2,3 FPS for image size of 1000x800).
My former professor of programming suggested to use UDP or TCP for faster and more reliable transfer of data, so now I am trying to do this.
But let me know, if this is something that can be done purely in Pd (GEM). I have no experience with GEM though, thats also why i chose Python.

@Schlamborius anything gem has with a ~ is similar to analog video synthesizer art. There's also a three dimensional scope.

I haven't tried video or maybe just flipping frames / animation yet. But here I am sending a block of video information out as audio for R G and B (ignoring alpha), running it through low pass and high pass filters, then turning it back in to video

In order to hear it (it sounds bad but you probably knew that) you have to put it in a [pd subpatch] with an [outlet~] because [dac~] has to live on the ground floor of a patch and [dac~] also can't be reblocked.

I really had trouble getting an image format it was happy with, ended up with uncompressed .tiff using GIMP. Also it helps to load the image first before opening the gem window.

pix_pix2sig~-helpIVEBEENSHREKED.pd

@Schlamborius As you are comfortable with code, and GEM has problems currently on some platforms, I suggest that you take a look at Ofelia....... https://github.com/cuinjune/Ofelia
Based on Lua, you might find it very easy to script.
The library for Pd for your platform can be found on Deken....... Pd menu... help... find externals... ofelia...
You should be able to do everything then within Pd.

But there are a lot of options you could consider and........ https://videogameaudio.com/SFCM-Apr2020/SFCM2020-PureDataAndGameAudio-LPaul.pdf .... is well worth reading.....

As you are familiar with Python you could consider embedding your Pd patch...... https://github.com/enzienaudio/hvcc
David.
P.S. I don't really know what I am talking about with programming languages (my last experience was Fortran 74), but hope my gibberish will help.

Since @whale-av suggested embedding Python into Pd, I'll mention Pyo again, as it includes a Pd external to load Pyo code, so Python actually I to a Pd patch, and output its audio output straight into your Pd patch.

@alexandros thank you. I have PYO but I cant find any pd externals for that. Do you possibly have any info about that?

In Pyo's sources, in the embedded/puredata directory. https://github.com/belangeo/pyo/tree/master/embedded
You'll probably have to compile yourself.

@alexandros thanks, i tried to do it, but it seems to be obsolete now. when compiling (make command in puredata folder), terminal shows:
clang: error: unknown argument: '-ftree-vectorizer-verbose=2' which is obsolete flag according to stack overflow.
I tried to delete line '-ftree-vectorizer-verbose=2' from Makefile, but the error persists. Is there anything i can do about this? I am afraid that I am reaching the limits of my sparse programming knowledge by now, so sorry for stupid question.

@Schlamborius I'm not sure. I can compile the object with a simple make, but I can't load it, as I get some error about _Py_NoneStruct, which the developer of Pyo can't reproduce, so I have abandoned this for now, and every now and then I keep thinking that I have to retry.
How did you compile? Which Pyo version do you have?

I have latest version of Pyo, I compiled using make command.
I am afraid that I cant make it work. But I am still thinking - there MUST be a way how to tranfer data from python to PD and simultaneously back to python without being limited with limited sample rate and latency and other problems of virtual audio devices(the only way I managed it to work so far). I was now again trying to use mrpeach's [udpreceive~] and [udpsend~], using now 3channel 8bit communication to send each color channel separately and at the same time not be limited with the fact that [udpreceive~] receives correctly only the first byte of the python format (as @fishcrystals states, it receives correctly only values 0-255, which is 8 bit format). But the format that [udpreceive~] requires seems totally cryptic to me when it comes to multichannel audio and I didn't manage to reverse engineer that using [udpsend~] either. There is no documentation for this and I didn't even find any contact on Martin Peach himself. So I am completely lost now. @alexandros let me know if you managed to make the Pyo pd embedded work. And if somebody has any ideas how I could continue, or at least a contact to mrpeach, that would be very appreciated and helpful, because I really feel there has to be a solution to this fairly simple question.

you can see the udpreceive~ source here: https://github.com/pd-externals/mrpeach/blob/main/net/udpreceive~.c
if each pixel is a sample, wouldn't the samplerate be 800k*framerate for 1000X800? that's way more than 192k...
fundamentally your biggest issue is transferring that much data across a network, even a local one. (even if you do pack each color channel as 8-bit values into 1 24-bit int though it might get you closer if you separate the data over multiple channels)
Anyways I'm kind of confused: how is the source 'audio'? it might make sense to think of it as sending video, then convert to/from audio in pd
There might be some GEM-based objects to do that part.. maybe you can stream to/from vlc or something
you could also consider using libpd to use pd from within python, but then externals might be tricky to get working idk

I will again mention the Jack audio server here, as it can do what you want without breaking a sweat. If you have Jack support in Pd and Python (you can have that with Pyo), then you just make the connections in Jack's graph, and that's all.
As for mrpeach, unfortunately, Martin Peach passed away about a year ago. I don't know if someone has taken over the maintenance of his library though.