It is not possible to open custom dialogs. What you can try is to open a new (invisible) activity which immediately opens a custom dialog. On exiting the dialog you can close the activity then.
You can use the AddIntent() method to open an activity:
...
And add this to the manifest so the activity listens to the action:
AddActivityText(MyActivity, <intent-filter>
<action android:name="your.package.name.action.OPENACTIVITY" />
<category android:name="android.intent.category.DEFAULT" />
</intent-filter>)
you can use any string for the action.