I prefer to answer this Question with some real application scenarios.

Transformation: we just mentioned that
every 3D object is a series of numerical coordinates. So what if we want to
move (translate) this object or rotate it to another direction? Well all these
are based on Vectors. And to move an object we have to pass over every point
and apply a translate vector to move the whole object for a specific distance.

Collision detection: how will we detect
if ball touched the ground? Or when any object touches other object? Its again
math equation to calculate the distance between point & plane.

Physical simulations: after we detect the
collision between a ball and the ground, what is spouse to happen? How should the
ball or the ground react? Following the physics rules and base on the (Speed,
gravity, …) which we have to set manually.

Do not be scared. You do not have to be that good in math
& physics. Unless you want to build your own Game or physics engine. Because
usually Game Engine takes care of these details for you. (like the Physics
engine: Havok) Many big titles used that physics engine i.e (The Elder Scrolls
online, Devil May Cry, Half-life 2, F.E.A.R 2,…).