We report theoretically on the interplay between polarization switching and bifurcations to nonlinear dynamics in a vertical-cavity surface-emitting laser (VCSEL) subject to orthogonal optical injection. Qualitatively different bifurcation scenarios leading to polarization switching are found and mapped out in the plane of the injection parameters, i.e., the frequency detuning vs injection strength plane. A Hopf bifurcation mechanism on the two-polarization-mode solution determines the injection-locking boundaries and influences polarization switching induced by optical injection. We furthermore report on a torus bifurcation emerging from a two-linearly polarized (LP) mode time-periodic dynamics before polarization switching and injection locking appear. It corresponds to an interesting combination of relaxation oscillation dynamics in the x-LP mode together with wave mixing dynamics in the injected y-LP mode. In agreement with recent experiments, we unveil a period-doubling route to chaos that involves both VCSEL orthogonal LP modes. The corresponding region of chaotic dynamics coincides with abrupt changes in the polarization switching boundaries in the plane of the injection parameters.