I have faced this problem a few years ago.
If you request the list of contacts without images is fast, while if you request the image of the profile everything becomes very slow, especially in older devices.
The solution is to load the list of contacts and with an asyncron method add the images, which can also be a timer.
WhatsApp uses just this method, in fact sometimes you notice that the images appear progressively.
You can get this with a ScrollView or custom ListView.