Looking at the DV original code, you can see that a transparent panel is the one getting the "touch" . This panel, in turns, drags the underlying view (in your case your panel). You could try to transfer the draggable functionality to your panel, read a "click", verify it's on the button and the act properly.
ps: I'm away from my PC so I can't read your code right now. Surely there will be easier ways to accomplish what you need