I'm pretty sure that mathematicians and physicists would both reject the way you've described them.
Mathematics not only accepts the concept of infinity but has a great deal to say about it. To take just one example: Cantor proved in the 19th century that not all infinite sets are of the same size. In particular, he showed that whereas the counting numbers and the rational numbers can be paired up one-for-one, there's no such pairing between the counting numbers and the full set of real numbers. Thus, he proved that in a well-defined sense, there are more real numbers than integers, even though in that same sense there are not more rational numbers than integers.
Now of course, we sometimes talk about certain functions going to infinity in a certain limit. For example: as x goes to 0, 1/x goes to infinity, even though there is no value of x for which the value of 1/x is infinity. Rather, we say that at 0, the function is not defined. There are good reasons why we say that, though this isn't the place to spell them out. But examples like this don't show that mathematics rejects the concept of infinity.
As for physics, consider a very basic physical concept: an inertial system. The net force on an inertial system is 0. Physics assumes that the concept of an inertial system is a perfectly good one and indeed, a deeply important one. Whether any actual physical systems literally experience a net force of zero is another question; it depends on how things are actually arranged in the world. But physics doesn't rule it out a priori.