The decay of Taylor-Couette turbulence, i.e., the flow between two coaxial and independently rotating cylinders, is numerically studied by instantaneously stopping the forcing from an initially statistically stationary flow field at a Reynolds number of Re=3.5×104. The effect of wall friction is analyzed by comparing three separate cases, in which the cylinders are either suddenly made no-slip or stress-free. Different life stages are observed during the decay. In the first stage, the decay is dominated by large-scale rolls. Counterintuitively, when these rolls fade away, if the flow inertia is small a redistribution of energy occurs and the energy of the azimuthal velocity behaves nonmonotonically, first decreasing by almost two orders of magnitude and then increasing during the redistribution. The second stage is dominated by non-normal transient growth of perturbations in the axial (spanwise) direction. Once this mechanism is exhausted, the flow enters the final life stage, viscous decay, which is dominated by wall friction. We show that this stage can be modeled by a one-dimensional heat equation, and that self-similar velocity profiles collapse onto the theoretical solution.