Exploring and identifying a good feature representation to describe large-scale datasets is one of the main problems of machine learning algorithms. However, plenty of feature selection techniques and distance metrics with very different properties exist, which entails an intricacy for identifying the proper method. This paper provides a general algorithm to design a high-order distance metric over a sparse selection of features dedicated to semi-supervised clustering and classification. We extend usual learning methods to design a metric accounting for patterns over sets of objects. Our approach is based on Conditional Random Field (CRF) energy minimization and Dual Decomposition, which allow efficiency and great flexibility in the features to consider. In particular, it enables to leverage the higher-order graph structures information efficiently. The optimization technique employed ensures the tractability of very high dimensionality problems using hundreds of features and samples. On a challenging task of Covid-19 patients stratification, we compare the classification results between state of the art baselines and our proposed classifier, relying on the higher-order distance learned to prove this metric formulation's relevance.