This paper considers target detection in distributed multiple-input multiple-output (MIMO) radar with imperfect waveform separation at local receivers. The problem is formulated as a binary composite hypothesis testing problem, where target residuals due to imperfect waveform separation are explicitly modeled as a subspace component in the alternative hypothesis, while disturbances including the clutter and thermal noise are present under both hypotheses. Under assumptions of fluctuating and non-fluctuating target amplitude over a scan, e.g., Swerling models, we particularly consider a distributed hybrid-order Gaussian (DHOG) signal model and develop the generalized likelihood ratio test (GLRT) which relies on the maximum likelihood (ML) estimation of the target amplitude and the residual covariance matrix under the alternative hypothesis. The Cramer-Rao bounds (CRBs) on estimating the target amplitude and residual subspace covariance matrix are derived. Simulation results in both local and distributed scenarios confirm the effectiveness of the proposed GLRT and show improved performance in terms of receiver operating characteristic (ROC) by exploiting the existence of target residual component.