You can modify AndroidManifest.xml and remove the READ_PHONE_STATE permission. It is near the bottom of the text. Afterward change the file to be read-only so it will not be overwritten. AndroidManifest is located under the Objects folder.
You will need to keep the VIBRATE permission.
No permission is added when you add a reference to the phone library.
For each object type there is a list of related permissions that this object requires. It appears in the documentation of each object.
Only when you declare an object of this type, the permissions get added.