Optical fiber nonlinearity is the main factor limiting the transmission distance of coherent optical communications. We give an overview of several modulation formats which are intrinsically tolerant to fiber nonlinearity. We then describe in detail the recently proposed four-dimensional modulation format family based on 2-ary amplitude 8-ary phase-shift keying (2A8PSK), supporting spectral efficiencies of 5, 6, and 7 bits/symbol. These formats nicely fill the spectral efficiency gap between dual-polarization quadrature PSK (DP-QPSK) and DP 16-ary quadrature-amplitude modulation (DP-16QAM), with excellent linear and nonlinear performance. Since these modulation formats just use different parity bit expressions in the same constellation, similar digital signal processing can be seamlessly used for different spectral efficiency. A series of nonlinear transmission simulation results shows that this modulation format family outperforms the conventional modulation formats at the corresponding spectral efficiency. We also review DSP algorithms and the experimental results. Their application to time-domain hybrid (TDH) modulation is also reviewed.