Yes, you are correct. It doesn't work if you set the direction to horizontal.
It seems to be something to do with the PreoptimizedCLV.
I notice that the scrollbar is always a vertical one.
I have added another clv without the preoptimizedCLV addition but with lazy loading and this works fine.