'Cursed' island once inhabited by hermit 'wizard' left to ro