We use a combination of symmetry analysis, phenomenological modeling, and first-principles density functional theory to explore the interplay between the magnetic ground state and the detailed atomic structure in the hexagonal rare-earth manganites. We find that the magnetic ordering is sensitive to a breathing mode distortion of the Mn and O ions in the ab plane, which is described by the K1 mode of the high-symmetry structure. Our density functional calculations of the magnetic interactions indicate that this mode particularly affects the single-ion anisotropy and the interplanar symmetric exchanges. By extracting the parameters of a magnetic model Hamiltonian from our first-principles results, we develop a phase diagram to describe the magnetic structure as a function of the anisotropy and exchange interactions. This in turn allows us to explain the dependence of the magnetic ground state on the identity of the rare-earth ion and on the K1 mode. The attached files contain VASP input files and bash scripts for running all the DFT calculations in the manuscript.