Abstract: The features of unknown network applications can be extracted using its traffic data. However, the sample traffic in network engineering is usually a mixed traffic generated by several unknown applications. The separation of the mixed traffic by applications an unsolved problem presently. A clustering method for traffic classification was proposed based on payload information. The proposed method can firstly encode certain bytes of message payload, then separate and classify the unknown mixed traffic using an extended ROCK algorithm. The experiment results reveal that compared with the clustering method based on statistics character of traffic, the proposed method has higher accuracy.