You are not using ButtonsState correctly. Use SetButtonState instead.
Use this code:
B4X:
Dim rs As ResumableSub = Dialog.Show("aaa", "OK", "", "")
AfterShow
Wait For (rs) Complete (Result As Int)
Sub AfterShow
Dim jo As JavaObject = Dialog.Base
jo.RunMethod("setClipToOutline", Array(True))
End Sub