I fixed it by using a boolean variable on the Click event.
When pressing down, it would always go DOWN/UP/DOWN. Whilst pressing and releasing was DOWN/UP/CLICK.
So using Click, I managed a variable so down only ran 1 time. I needed down instead of click because if someone put the finger on the button for a bit too long, you lose time. No good for a stopwatch situation. Gotta trigger on the Down event.