BICM capacity analysis of 8QAM-alternative 2D/4D modulation formats in nonlinear fiber transmission

8-ary quadrature-amplitude modulation (8QAM) plays an important role in filling the gap between quaternary phaseshift keying (QPSK) and 16QAM in terms of bit rates and reach. To achieve the same spectral efficiency with improved sensitivity, quaternary code and sphere-cut lattice codes, 4D honeycomb lattice codes, Circular-8QAM, and 4D 2-ary amplitude 8-ary phase-shift keying (4D-2A8PSK) have been proposed. 4D-2A8PSK is especially attractive among these 3-/s/Hz/pol formats, since it has the properties of Gray coding and constant modulus. Current optical communication systems usually rely on soft-decision (SD) forward error correction (FEC) coding based on bit-interleaved coded modulation (BICM). Therefore, using the BICM mutual information, also called generalized mutual information (GMI), is a better metric for comparing multiple modulation formats. Several 8QAM constellations were compared using GMI in the linear region and the nonlinear region. More recently, two new modulation formats, (4D) 64 set-partitioned (SP-) 12QAM and DP-Circular-8QAM with optimized inner/outer circle radius ratio, which are shown to have better performances than some previous modulation formats. However, the direct comparison among all the latest modulation formats have not been done yet. In this work, we compare the nonlinear transmission characteristics of four 2D/4D modulation formats, i.e., 4D- 2A8PSK, 4D-64SP-12QAM, optimized DP-Circular-8QAM, and the conventional DP-Star-8QAM using GMI.