Exploiting and Controlling Lattice Symmetry for Infrared Nanophotonics and Ultrafast Thermal Transport