In this paper, we consider a smooth connected finite-dimensional manifold M, an affine connection a double dagger with holonomy group H (a double dagger) and Delta a smooth completely non integrable distribution. We define the Delta-horizontal holonomy group as the subgroup of H (a double dagger) obtained by a double dagger-parallel transporting frames only along loops tangent to Delta. We first set elementary properties of and show how to study it using the rolling formalism Chitour and Kokkonen (2011). In particular, it is shown that is a Lie group. Moreover, we study an explicit example where M is a free step-two homogeneous Carnot group with m >= 2 generators, and a double dagger is the Levi-Civita connection associated to a Riemannian metric on M, and show in this particular case that is compact and strictly included in H (a double dagger) as soon as m >= 3.