Experiments on quantum well and recently quantum dot VCSELs have shown that the increase of injection current may lead to a transition from a linearly polarized light emission at threshold to a region of nonlinear dynamics (self-pulsing) accompanying polarization switching between either two orthogonal linearly polarized states or non-orthogonal elliptically polarized states. The dynamics occurs on a nanosecond time-scale and relates either to the birefringence induced frequency splitting or to the relaxation oscillation frequency. In this contribution, we bring new light into the bifurcation mechanisms explaining the occurrence of deterministic self-pulsing accompanying polarization switching. We demonstrate theoretically that depending on the laser parameters, different polarization switching scenarios may be observed with self-pulsing dynamics at a dominant time-scale related to either linear cavity birefringence or relaxation oscillation, and with additional period doubling or quasiperiodicity. Our work therefore not only reconciles previous experiments with different conclusions on dynamical states, but also provides an improved understanding of the bifurcations underlying the commonly used spin flip model for VCSEL - hence motivating new in depth experiments of polarization dynamics at nanosecond time scale.