Objective: This paper introduces a generalized coherence framework for detecting and characterizing nonlinear interactions in the nervous system, namely cross-spectral coherence (CSC). CSC can detect different types of nonlinear interactions including harmonic and intermodulation coupling as present in static nonlinearities and also subharmonic coupling, which only occurs with dynamic nonlinearities. Methods: We verified the performance of CSC in model simulations with both static and dynamic nonlinear systems. We applied CSC to investigate nonlinear stimulus-response interactions in the human proprioceptive system. A periodic movement perturbation was imposed to the wrist when the subjects performed an isotonic wrist flexion. CSC analysis was performed between the perturbation and brain responses (electroencephalogram, EEG). Results: Both the simulation and the application demonstrated that CSC successfully detected different types of nonlinear interactions. High-order nonlinearities were revealed in the proprioceptive system, shown in harmonic and intermodulation coupling between the perturbation and EEG for all subjects. Subharmonic coupling was found in some subjects but not all. Conclusion: This paper provides a general tool to detect and characterize nonlinear nature and dynamics of the nervous system. The application of CSC on the experimental dataset indicates a complex nonlinear dynamics in the proprioceptive system. Significance: This novel framework 1) unveils the nonlinear neural dynamics in a more complete way than the existing coherence measures, and 2) is more suitable for estimating the input-output relation regarding both phase and amplitude compared to phase synchrony measures (which only consider phase coupling). Subharmonic coupling is reported in human proprioceptive system for the first time.