'He stood with party, won't leave': Digvijaya downplays spec