If your answer is No, politely say No. You don’t have to explain or justify your answer. You were asked a Yes or No question; you gave your answer. Now move on.