I think the answer will depend on which philosopher of math you ask. As you seem to recognize, some philosophers of math deny that numbers exist independently of us in such a way that their existence is genuinely discovered by us. Even philosophers of math who think that numbers are discovered might say that your question -- "What happened?" -- is an empirical historical or psychological question rather than a philosophical one. In any case, you'll find relevant material in the SEP entry on "Philosophy of Mathematics" at this link.
You are reasoning correctly--mathematics deals with possibilities and physics with actualities (even though in quantum mechanics these are probabilistic). Theory in physics is often expressed mathematically, but that does not make it mathematical knowledge. Some theoretical advances in physics can come from working in an armchair and extending the mathematical implications of (already accepted and contingently true) theory. The actual status of mathematics (a priori or not) is debatable (Quine etc claiming that mathematics is an empirical theory like any other). But you are correct that physics is not mathematics, and the sort of evidence that confirms physical theory is not (or perhaps, not entirely) the evidence (or other considerations) needed to confirm mathematics.