We demonstrate the physical generation of random bits at high bit rates (> 100 Gb/s) using optical chaos from a solitary laser diode and therefore without the complex addition of either external optical feedback or injection. This striking result is obtained despite the low dimension and relatively small bandwidth of the laser chaos, i.e. two characteristics that have been so far considered as limiting the performances of optical chaos-based applications. We unambiguously attribute the successful randomness at high speed to the physics of the laser chaotic polarization dynamics and the resulting growth rate of the dynamical entropy.