Generating realistic datasets with fine-grained control over their properties can help overcome challenges linked to the scarcity of data in many domains, such as medical applications. To that end, we extend Variational Autoencoders by using a hierarchical and topological prior consisting of a sequence of Self-Organizing Maps (SOM), which are stacked in the latent space and learned without supervision, jointly with the parameters of the variational autoencoder. We induce a hierarchy between the codes of the SOM sequence, each SOM corresponding to a different hierarchical level and learning increasingly finer-grained representations of the data. Our model combines the power of deep learning with the interpretability of hierarchical and topological clustering and produces competitive results when evaluated on three well-known computer vision benchmarks and a custom medical dataset.