Multiple-input multiple-output (MIMO) systems improve the bit error rate (BER) performance, meanwhile,in order to reduce the impact to system capacity, a novel antenna selection algorithm for space-time block coding (STBC) and vertical Bell laboratories layered space time (VBLAST) hybrid coding systems was proposed. Employing the Alamouti STBC code for some VBLAST layers, the STBC and VBLAST hybrid coding MIMO system has the advan-tages of both STBC and VBLSAT. On the criterion of maximal antenna-subset capacity, the proposed antenna selection algorithm selects those antennas with less contribution to the system capacity to send multilayer STBC signals, so that the system capacity loss by STBC coding can be reduced. Both theoretical analysis and simulation results indicate that the proposed antenna selection algorithm increases the capacity of the STBC and VBLAST hybrid system effectively, and improves the BER remarkably comparing with a VBLAST encoding MIMO system.