Abstract: An improved direction-finding algorithm for noncircular sources was proposed in the presence of angularly dependent gain and phase errors, which is called NC-ISM based on the fundamental principle of ISM (instrumental sensor method). Through the application of the extended data model of non-circular signals with maximum rate, the proposed algorithm enhances the information utilization leading to advancement of estimation accuracy and twice the number of sources that can be distinguished. The performance study on NC-ISM proved the statistical consistency of the parameter estimation, and a theoretical derivation for the closed-form expression of the mean square error (MSE) of NC-ISM estimation was presented under the influence of modeling errors of instrumental sensors by the first-order analysis. Therefore, the analysis can provide theoretical support for practical applications. The simulation results not only verify the effectiveness of theoretical derivation, but also illustrate that NC-ISM is more robust than ISM with respect to signal-to-noise ratio as well as modeling errors.