During the last five years, optical chaos-based random bit generators (RBGs) attracted a lot of attention and demonstrated impressive performances with bit rates up to hundreds of Gbps. However all the suggested schemes use external injection schemes (optical injection or feedback) to turn the lasers into chaos, hence strongly increasing setup complexity. On the other hand, we reported that a laser diode can generate a chaotic output without the need for external perturbation or forcing, hence unveiling a highly simplified way to generate an optical chaos at high frequency. However the low dimension and limited number of positive Lyapunov exponent casted doubts about its direct use for chaos-based applications. Here we make a proof-of-concept demonstration for a Random Bit Generator based on polarization chaos. We therefore suggest a highly simplified RBG scheme using only a free-running laser and small-bandwidth acquisition electronics and demonstrate convincing performances with bit rates up to 100 Gbps without unusual or complex post-processing methods. We link these performances to the double-scroll structure of the chaotic attractor rather than the bandwidth of the dynamics, hence bringing new light on the importance of chaos topology for chaos-based applications. In addition our scheme exhibit a strong potential as it enables a low-cost and/or integrated in parallel on-chip scheme.