only 5v to 3.3 or regulators which only can handle 5amp
If the 5v to 3.3v regulator says it can only handle 5 amps, what that means is that it can "output" up to 5amps at 3.3v. It doesn't mean something bad will happen if you connect it to a 10amp 5v source.
But what is strange is that 5 amps seems like a lot of output current capability from what I imagine is a small "on-board" DC-to-DC converter (the 5v to 3.3v converter). The 5 amp rating could very well be accurate, but it just seems like such a heavy duty converter would be an overkill for the small ESP32 board.
Now, getting back to your 9amp requirement for the leds and that you think your power supply is not very regulated. Well the way to tell if the power supply is doing a good job of regulation is to measure the voltage at the 5v terminals with no load (none of the LEDs on) - it should be 5.00vdc +/- 5%. Then turn on all the LEDs, and measure it again. If it's stays near 5.00 +/- 5%, then its doing a very good job regulating the voltage and it's safe to use with the 5 to 3.3 converter. But if the voltage varies more then +/- 5%, then you should get a more regulated power supply.
The other issue is that if all the LED's are on and it does in fact draw 9amps, then you might want to consider getting a larger 12 or 15 amp power supply because if the LEDs draw 9amps and the ESP32 draws lets say only 250ma, that's 9.25amps total, and you typically want more then just .75amp (10 max amps rating of PS minus max load of 9.25amps) overhead room because the normal load of a project should not be very close to the max load of the power supply. Thus, if you instead use a 12amp power supply, then you would have 2.75 amps of headroom which will have a much lower chance of stressing out the power supply constantly.