We study orbital excitations in the optical absorption spectra of YVO3 and HoVO3. We focus on an orbital absorption band observed at 0.4 eV for polarization E∥c. This feature is only observed in the intermediate, monoclinic phase. By comparison with the local crystal-field (CF) excitations in VOCl and with recent theoretical predictions for the CF levels we show that this absorption band cannot be interpreted in terms of a local CF excitation. We discuss a microscopic model which attributes this absorption band to the exchange of two orbitals on adjacent sites, i.e. to the direct excitation of two orbitons. This model is strongly supported by the observed dependence on polarization and temperature. Moreover, the calculated spectral weight is in good agreement with the experimental result.