The flower pot is not in a room! Its outside next to the door mat and the milk bottles. If it was inside a room then why would you need a door key!!
But the easiest way, if the milk bottles are full then the wife is at home so bang harder on the door! If the milk bottles are empty then the milkman is still doing the deed with your wife and hasnt filled the bottles yet!
He will fill them with milk only when he is leaving!!