Out of curiosity and to learn about data structures i build this vanilla xy-pad.
In principle it works very well, but when the pad gets bigger than about 100 x 100 pixels, it becomes very, very heavy. This means that clearing the subpatch would take a long time and it isn't even possible to close the window until the patch is cleared. In full size (256 x 640 pixels) it took 9 minutes on my computer to clear the patch.
Also, on my linux machine it crashed my file browser, so this comes with a warning. Better save any open projects before trying this.