When you major in physics, you are basically double majoring in math as well. Math is more an integral (no puns please) part of physics than any other science. The most basic aspects of physics require nothing more than algebra and geometry.
When you want to find the force on a object, it's simply force equals mass times acceleration. You learn a lot of equations during your first year and use algebra to rearrange the equations, so you can find the appropriate variable.
If a force is applied at an angle, then you use sine and cosine in the equation as well. Many of the expressions learned can be rewritten in derivative an integral form. This is important because as you advance in physics you take more variables into consideration. As a beginner, you simply assume the force of friction is zero, but we all know that in reality every surface has some sort of fiction.
The real work begins once you start learning about electricity and magnetism, quantum mechanics and general relativity. They almost use high level calculus and differential equations exclusively. The days of easy algebra are over.
If you choose to go beyond the four-year degree, the math becomes like a tidal wave. The real world is full of variables and that goes for constructing a buildings to modeling a supernova. Math is crucial for understanding the most basic and far reaching aspects of our universe and physics is how we do it.