The nonlinear instability of the density-inverted granular Leidenfrost state and the resulting convective motion in strongly shaken granular matter are analysed via a weakly nonlinear analysis of the hydrodynamic equations. The base state is assumed to be quasi-steady and the effect of harmonic shaking is incorporated by specifying a constant granular temperature at the vibrating plate. Under these mean-field assumptions, the base-state temperature decreases with increasing height away from the vibrating plate, but the density profile consists of three distinct regions: (i) a collisional dilute layer at the bottom, (ii) a levitated dense layer at some intermediate height and (iii) a ballistic dilute layer at the top of the granular bed. For the nonlinear stability analysis (Shukla & Alam, J. Fluid Mech., vol. 672, 2011b, pp. 147–195), the nonlinearities up to cubic order in the perturbation amplitude are retained, leading to the Landau equation, and the related adjoint stability problem is formulated taking into account appropriate boundary conditions. The first Landau coefficient and the related modal eigenfunctions (the fundamental mode and its adjoint, the second harmonic and the base-flow distortion, and the third harmonic and the cubic-order distortion to the fundamental mode) are calculated using a spectral-based numerical method. The genesis of granular convection is shown to be tied to a supercritical pitchfork bifurcation from the density-inverted Leidenfrost state. Near the bifurcation point the equilibrium amplitude (Ae) is found to follow a square-root scaling law, Ae∼Δ−−√, with the distance Δ from the bifurcation point. We show that the strength of convection (measured in terms of velocity circulation) is maximal at some intermediate value of the shaking strength, with weaker convection at both weaker and stronger shaking. Our theory predicts that at very strong shaking the convective motion remains concentrated only near the top surface, with the bulk of the expanded granular bed resembling the conduction state of a granular gas, dubbed as a floating-convection state. The linear and nonlinear patterns of the density and velocity fields are analysed and compared with experiments qualitatively. Evidence of 2:1 resonance is shown for certain parameter combinations. The influences of bulk viscosity, effective Prandtl number, shear work and free-surface boundary conditions on nonlinear equilibrium states are critically assessed.