I'm trying to use [array random] to create a random number generator that outputs numbers in a similar distribution to a real-world process. In the snapshot above, the "distribution" table is a histogram of 285 points of real data, and I'm speculating that if I had access to more data, the curve would become smoother.

The "simDist" table is just my eyeball attempt to make the idealized version of the real histogram. In tests, it works fairly well, but I'm wondering if it would be better if it was arithmetically derived from the "distribution" table, e.g. uh, I dunno, a moving N-point RMS? I'm sure a data scientist would say "it depends what you mean by 'better'", but I'd be interested to know what options there are and what each is good for. Somehow, low pass filtering doesn't feel right to me because it's a histogram.

@jameslo What's wrong with low-pass filtering? Not being a data-scientist I would have thought that this is what they are doing all the time.

But yes, of course, there's a problem with low-pass filtering that you have to be aware of (and that you are maybe refering to): If the output of your filter only depends on the current input and previous input or output values, then you will always have some phase shift, which is certainly not what you want when dealing with a histogram! So you have to use a zero-phase filter.

Here's how I would do it ... filtered-histogram.pd

@manuels Yes, it was the phase shift I was referring to, and I wasn't aware that one could low-pass filter without phase shift, although now that I see your patch it seems obvious. It even looks like what I was proposing, a moving RMS window, might be another instance of a zero-phase lowpass filter:

I wish I knew more about filters, statistics, probability....

@jameslo said:

It even looks like what I was proposing, a moving RMS window, might be another instance of a zero-phase lowpass filter

Yes, indeed! Or to be exact, that's only true as long as there are data points available for the whole RMS window. At the beginning and end of your dataset, you still have a phase shift (of half the effective window size). But that's always a problem, and I don't have a solution for that ...

BTW you could also try using a weighted average. I did that indirectly by repeating the process many times (which has the effect of approaching a Gaussian filter).

I find the idea of using a zero-phase filter or a moving RMS window really interesting for histogram smoothing. I'm curious how would a weighted average approach compare to something like kernel density estimation for smoothing?

@spoidy23 Interesting ... I didn't know about KDE, but it seems to be more or less the same thing. The iterative approach I proposed could be seen as a way to find the "right" bandwidth (whatever that is). To illustrate this, I added a plot for the corresponding kernel, which is just the impulse response of the filter: filtered-histogram-kernel.pd

Thanks for explaining.
The idea of using the filter to find the 'right' bandwidth is really interesting. I’ll take a look at the plot you shared. Do you think this method works well with data that has different densities?

@spoidy23 You mean something like a variable kernel density estimation (with adaptive bandwidth)? I guess, that would be difficult ...

@manuels OK Thankyou for suggestion.