Although Voldemort was unwilling to admit it from the bottom of his heart, he was indeed at a disadvantage in the confrontation with Albert. The young guy in front of him was even more dangerous than Dumbledore made him feel back then. If this continues, defeat will happen sooner or later. Voldemort didn't think he would lose anyway, because he could apparate and leave here at any time, and Albert Anderson couldn't stop him at all.