Yes, app permissions can cause trouble as some apps may secretly access your memory card or contact data. It is important to be aware of the permissions that an app is requesting before you install it on your device. You can also review the permissions of apps that are already installed on your device and revoke any permissions that you are not comfortable with.
If you are concerned about the security of your data, you can also use a mobile security app to help protect your device from malware and other threats.