Sadness over 'Loss of culture' as flats to be built on site