@MrKim your ploblem sounds like the same I had just now with one of my relatively old B4J projects.
You may open the designer from the IDE Menu, this will open the B4X Designer. Because this was developed using JavaFX Scene Builder before B4X Designer existence, it use fxml layout instead of bjl layout expected by B4X Designer.
To solve it ensure you have JavaFX Scene Builder 1.0 or better JavaFX Scene Builder 2.0 installed, then go to FileManager in the IDE and click on blue Open Designer and scene builder should be launched with your layout.
My personal opinion (that have nothing to do with your error) is that there has been a great loss by removing the possibility of managing layouts created with JavaFX Scene Builder from the IDE, which in my personal opinion is better than B4X Designer and I still only use the latter for very simple projects.
Please try my solution and post results to ensure the problem was solved.