In the inelastic time history analyses of structures in seismic motion, part of the seismic energy that is imparted to the structure is absorbed by the inelastic structural model, and Rayleigh damping is commonly used in practice as an additional energy dissipation source. It has been acknowledged that Rayleigh damping models lack physical consistency and that, in turn, it must be carefully used to avoid encountering unintended consequences as the appearance of artificial damping. There are concerns raised by the mass proportional part of Rayleigh damping, but they are not considered in this paper. As far as the stiffness proportional part of Rayleigh damping is concerned, either the initial structural stiffness or the updated tangent stiffness can be used. The objective of this paper is to provide a comprehensive comparison of these two types of Rayleigh damping models so that a practitioner (i) can objectively choose the type of Rayleigh damping model that best fits her/his needs and (ii) is provided with useful analytical tools to design Rayleigh damping model with good control on the damping ratios throughout inelastic analysis. To that end, a review of the literature dedicated to Rayleigh damping within these last two decades is first presented; then, practical tools to control the modal damping ratios throughout the time history analysis are developed; a simple example is finally used to illustrate the differences resulting from the use of either initial or tangent stiffness-based Rayleigh damping model.