The axisymmetric collapse of a cylindrical air cavity in water follows a universal power law with logarithmic corrections. Nonetheless, it has been suggested that the introduction of a small azimuthal disturbance induces a long-term memory effect, reflecting in oscillations which are no longer universal but remember the initial condition. In this work, we create non-axisymmetric air cavities by driving a metal disc through an initially quiescent water surface and observe their subsequent gravity-induced collapse. The cavities are characterized by azimuthal harmonic disturbances with a single mode number and amplitude . For small initial distortion amplitude (1 or 2 % of the mean disc radius), the cavity walls oscillate linearly during collapse, with nearly constant amplitude and increasing frequency. As the amplitude is increased, higher harmonics are triggered in the oscillations and we observe more complex pinch-off modes. For small-amplitude disturbances we compare our experimental results with the model for the amplitude of the oscillations by Schmidt et al. (Nature Phys., vol. 5, 2009, pp. 343¿346) and the model for the collapse of an axisymmetric impact-created cavity previously proposed by Bergmann et al. (J. Fluid Mech., vol. 633, 2009b, pp. 381¿409). By combining these two models we can reconstruct the three-dimensional shape of the cavity at any time before pinch-off.