This is a note for me and all of you having more than one profile photos and missed the option to delete them.
- You sign into plus.google.com
- then click on your ‘Profile’ (in the sidebar)
- Click your profile photo, it opens it in preview mode.
- Use the right arrow to see older profile photos.
- Click the ‘three vertical bots’ button
- and select ‘Delete’
- Repeat step 4 to 6 for older photos.
Source: Google Productforums
PS: Should you want to disable the profile photo altogether, this is also possible. Just proceed as explained here.