I'm just doing something similar and wondered the same thing. in the end I went for activities but using the same layout file (just changing the data that is loaded into listview. I chose this because the back button works automatically. With panels you would have to trap the back buttons trigger event and also remember the 'path back' is that makes sense. Which panels to go back to.
I suggest you activities too.
That's how Android is designed.
Another advantage is when you turn the device you don't need to manage what Panel is active and show that one.
Just tried 2 activities (one for each listview), each with their own layout, using StartActivity and Activity.Finish to move back and forth.
Works well, keeps its state properly without fiddling, and I even get the left/ride slide for free (I assume it's the Android default for switching activities using these commands.)