Abstract: Based on the extended nonlinear Schrdinger equation, the gain spectrum of crossphase modulation (XPM) instability in fiber was investigated with highorder dispersion and highorder nonlinearity considered. The effects of fourthorder dispersion, quintic nonlinearity and fiber loss on the gain spectra were analyzed. The results showed that because of fourthorder dispersion, XPM instability occured at two spectrum regions in both the normal and the anomalous dispersion regimes of fiber. The positive quintic nonlinearity would make the width and the peak value of the gain spectra larger. The negative quintic nonlinearity would make the width and peak value of the gain spectra smaller. Results also showed that fiber loss reduced the frequency range of the gain spectrum, and the frequency range became smaller with the propagation distance increasing.