Yes I understand that, but it's easy to remove entire libraries when switching build configurations prior to compiling.
I agree that in an ideal world it would be great to be able to conditionally compile in/out libraries, but it doesn't take much to select/deselect them in the libraries list - & if you conditionally include/exclude the related code you don't have to change anything else.