Loading...
Empirical Evaluation of Extended Finite State Machine Test Suites
Hasan, Faiz Mohammed
Hasan, Faiz Mohammed
Files
Description
A Master of Science thesis in Computer Engineering by Faiz Mohammed Hasan entitled, “Empirical Evaluation of Extended Finite State Machine Test Suites”, submitted in May 2022. Thesis advisor is Dr. Khaled El-Fakih. Soft copy is available (Thesis, Completion Certificate, Approval Signatures, and AUS Archives Consent Form).
Abstract
We present novel empirical assessments of prominent finite state machine (FSM)-based test derivation methods against their coverage of code faults. In particular, we consider many realistic extended FSM examples and their related Java implementations and then derive for these examples complete test suites using the W method and its HSI and H derivatives in addition to singe transfer fault test suites. The W, HSI, and H test suites are derived considering the case when the implementation under test (IUT) has the same number of states as the specification FSM, and we also consider W++, HSI++, and H++ test suites derived when the IUT can have one state more than the specification machine. An assessment is then provided to determine for each pair of considered test suites if there is a difference between the pair in covering the implementations faults. If the difference is found significant, we determine which test suite outperforms the other. According to conducted experiments, the W method and its derivatives do not provide the same fault coverage of code faults; in contrast to their similar coverage of FSM-based faults. We further show that the results of this result are not due to size nor length of the test suites. In addition, as W tests outperformed both HSI and H tests, we further investigated whether these differences were due to certain types of faults or are spread over all classes of faults. Besides, another assessment is provided to determine whether certain test suites have better coverage of some classes/types of code faults than others. The empirical assessments use proper data transformation of mutation scores and p-value adjustments for controlling Type I error due to multiple tests. Results and outcomes of conducted experiments are summarized.