Yes and no.
If they download it from your website (or get mailed directly by you) their concern wouldn't be so high. Afterall they trust you for what the app does once installed..
I generally instruct them to allow "unknow sources" just for the time to download my app, then set it back to its "not allowed" state.
Yes, there are risks involved when you leave the perfect(?), safe and fantastic close-world solutions offered by Google and Apple. But I value my freedom most.