This is a note for me and all of you having more than one profile photos and missed the option to delete them.

  1. You sign into plus.google.com
  2. then click on your ‘Profile’ (in the sidebar)
  3. Click your profile photo, it opens it in preview mode.
  4. Use the right arrow to see older profile photos.
  5. Click the ‘three vertical bots’ button
  6.  and select ‘Delete’
  7. Repeat step 4 to 6 for older photos.

Source: Google Productforums

PS: Should you want to disable the profile photo altogether, this is also possible. Just proceed as explained here.