It might be worth you reading about ULV... or thinking about bump scrolling or paging... e.g. holding 2-3 'pages' in memory and loading these. As the User moves to a new 'page' you grab another one in that direction.
e.g.
Load pages 1, 2 and 3.
User moves to Page 2, nothing.
User moves to Page 3, Load Page 4, unload Page 1
User moves to Page 4, Load Page 5, unload Page 2
User moves back to Page 3, unload Page 5, Load Page 2
etc...
ETA : Actually, how are you populating the list and what are you displaying/listing? Is there any way to improve the data retrieval?