@kristofferlislegaard
Just using the -rt switch when starting Pd could help.
What does
$ aplay -L --dump-hw-params
tell you about the soundcard?
Check the Man page for aplay first though, as I am a Linux noob.
Somewhere the Pd settings do not match the driver settings, or the driver itself has problems, or other devices are delaying the audio processing by interrupts.
Are you running a 64-bit os and have the correct driver?
It could be a USB hub speed problem...... https://puredata.info/docs/raspberry-pi
You should have Alsa installed..... and select the alsa drivers for Pd I believe.
You could try using Jack (see below).
This is necessary reading...... https://wiki.linuxaudio.org/wiki/raspberrypi .... that should help you solve the problem.
David.