Anupam Kher had earlier refused Manmohan Singh’s biopic, told what was most difficult in the character