Abstract: Pay close attention to the protocol’s abnormal behavior, and takes the message raw data and the protocol binary code both as the analysis objects. The proposed method uses dynamic taint analysis combined with static analysis, firstly monitor and analyze the process of protocol program parses the message in our developed virtual platform Abnormal Disc prototype system, and record the protocol’s public behavior; then based on the proposed abnormal behavior perception and mining algorithm, static analyze the protocol’s abnormal behavior trigger conditions and abnormal behavior instruction sequences. Finally, generate the new protocol messages with the sensitive information according to the abnormal behavior trigger conditions, and dynamic trigger the abnormal behaviors execute. Abnormal Disc prototype system can perceive, trigger and analyze the protocol’s abnormal behaviors. According to the statistical analysis results, the evaluation method of protocol execution security was proposed. The experimental results show that the method can accurately mine the protocol’s abnormal behavior, and evaluate the protocol’s execution security.

Key words: protocol reverse analysis; protocol’s abnormal behavior; protocol message; protocol software