I'm not sure I understand correctly.
If you don't call the MainForm.Show method, the window won't be visible.
Otherwise you can make a non-ui app by clicking File>New>Non-UI
Once you compile your app in Release mode, you can open your jar file in the objects folder, and there won't be any DOS window appearing.