We propose a class of identifiable deep generative models for very flexible data types. The key features of the proposed models include (a) discrete latent layers and (b) a shrinking pyramid- or ladder-shaped deep architecture. We establish model identifiability by developing transparent conditions on the sparsity structure of the deep generative graph. The proposed identifiability conditions can ensure estimation consistency in both the Bayesian and frequentist senses. As an illustration, we consider the two-latent-layer model and propose shrinkage estimation methods to recover the latent structure and model parameters. Simulation results corroborate the identifiability of the model, and also demonstrates the excellent empirical performance of our estimation algorithm. Applications of the methodology to a DNA nucleotide sequence dataset and an educational assessment response time dataset both give interpretable results. The proposed framework provides a recipe for identifiable, interpretable, and reliable deep generative modeling