Minister refuses to say if he has confidence in prison direc