This work proposes a design methodology for an observer-based impact detection with serial robot manipulators in presence of modeling uncertainties and using only proprioceptive sensors. After expressing modeling errors as the contribution of both dynamic parameters uncertainties and numerical differentiation errors, a Kalman filter is designed based on the inverse dynamic model with process and measurement power spectral densities explicitly depending on characterized uncertainties. The influence of the design parameters on the quality of the external torque estimation is studied in simulation and guidelines for tuning the Kalman filter are provided.