Ashfield Labour party hit out at Tory MP rival for boycottin