Transmission system of optical fiber communication has a tendency of developing high speed, large-capacity and long-haul fiber transmission system.With the advantanges in spectral efficiency, integration and OAM cost, 40Gb/s high rate optical transmission system will gradually replace 10Gb/s system. However, with the improvement of transmission rate and channel capacity, transmission system meet transmission impairments caused by group-velocity dispersion(GVD), polarization mode dispersion(PMD), fiber nonlinear effects and so on. As one of the key technologies of optical transmission system, advanced modulation format code could decrease transmission impairments without great changes in the existing transmission system.This thesis focus on the study of two main modulation formats i.e. amplitude and phase modulation formats in theory for the 40Gb/s transmission system. Main achievements are listed as follow.1. Based on the theories of modulation and demodulation processes of different modulation formats, the demodulation process of DPSK is analyzed in detail.The eye diagrams and modulated spectrum are obtained by simulating system model of amplitude and phase modulation formats.
