Questions and issues to Walter Cazzola.
A Brief Introduction to FLiRTS 2
Modern software development processes often use UML models to plan and manage the evolution of software systems. Regression testing of evolving software systems is important but can be expensive. Regression test selection (RTS) approaches are used to reduce the cost. Existing RTS approaches based on UML models use behavioral diagrams or a combination of structural and behavioral diagrams. Behavioral diagrams are necessary to perform impact analysis and obtain traceability links between model elements and test cases. However, in practice, behavioral diagrams are rarely used to design and maintain an application, and are incomplete even when used, which limits the applicability of the existing UML-based RTS approaches.
In previous work, we proposed a fuzzy logic approach called FLiRTS that supports RTS based on UML sequence and activity diagrams. In this work, we introduce FLiRTS 2, which enhances FLiRTS by dropping the need for behavioral diagrams and relying only on the presence of UML class diagrams describing the application. The class diagram is the most widely used type of UML diagram in practice. Relying only on class diagrams frees FLiRTS 2 from the applicability limitation that other model-based RTS approaches have.
Getting FLiRTS 2
Download and save the FLiRTS 2 zip file. It contains:
│ ├── configs.txt
│ └── fuzzylogic.fcl
│ └── flirts2.jar
2 directories, 5 files
- flirts2.jar contains the implementation of FLiRTS 2
- flirts2.sh is a script to run FLiRTS 2
- README.md describes how to install, configure and run FLiRTS 2.
Demonstrating FLiRTS 2 with Sample Subjects
To demonstrate how FLiRTS 2 works we prepared two sample subjects: Commons-compress and LectureReservationSystem. The former is a Java project freely available on GitHub and we extracted the class diagram through the RSA software. We developed the latter drawing the model with RSA and then coding it in Java.
Download the demos zip file. It contains:
│ ├── AdaptedClasses.txt
│ ├── commons-compress_687129547ae314a6b286553ad59af88a7a88d492.emx
│ ├── demo.sh
│ ├── DirectInvocationsFromTestClasses.txt
│ └── output
│ ├── AdaptedClasses.txt
│ ├── demo.sh
│ ├── LectureReservationSystem.emx
│ └── output
4 directories, 8 files
- Commons-compress and LectureReservationSystem contain the sample subjects with their class diagram (the .emx files), the list of direct invocations from the test classes and a script (named demo.sh) to run FLiRTS 2 on them
- README.md describes the subjects and how to run the scripts.
Evaluation of FLiRTS 2
We evaluated FLiRTS 2 over several metrics using 21 open-source Java projects with more than 8000 revisions considered.
Getting the Scripts and Revisions used in the Experiment
Download and save the experiment zip file. It contains:
│ ├── calculateSPR.sh
│ └── lib
│ └── utils.jar
│ ├── download.sh
│ ├── lib
│ │ ├── bcel-6.2.jar
│ │ ├── formatStartsDiff.py
│ │ └── utils.jar
│ └── repositories.txt
│ └── finalizeData.py
│ └── pit.sh
│ ├── asterisk-java.txt
│ ├── commons-cli.txt
│ ├── commons-codec.txt
│ ├── commons-collections.txt
│ ├── commons-compress.txt
│ ├── commons-configuration.txt
│ ├── commons-dbcp.txt
│ ├── commons-dbutils.txt
│ ├── commons-email.txt
│ ├── commons-fileupload.txt
│ ├── commons-imaging.txt
│ ├── commons-io.txt
│ ├── commons-jxpath.txt
│ ├── commons-lang.txt
│ ├── commons-net.txt
│ ├── commons-pool.txt
│ ├── commons-text.txt
│ ├── commons-validator.txt
│ ├── HikariCP.txt
│ ├── invokebinder.txt
│ └── stream-lib.txt
│ └── runFLiRTS2.sh
9 directories, 33 files
- calculateSPR contains a script (calculateSPR.sh) to calculate test suite reduction, and safety and precision violations of FLiRTS 2 with respect to Ekstazi and STARTS.
- downloadRevisions contains a script (download.sh) to download the used revisions from GitHub, to run Ekstazi and STARTS on them, and record the direct invocations from the test classes, as well as the names of the adapted classes to be used with FLiRTS 2.
- finalizeData contains a python script (finalizeData.py) to aggregate all the data from the experiments, computing means and medians and generating csv files.
- pit contains a script (pit.sh) to run PIT mutation testing tool with the code-level revisions using the full test suites, and the reduced test suites achieved by FLiRTS 2, Ekstazi, and STARTS.
- revisionsSHAs contains the SHAs for the used revisions divided per subject.
- runFLiRTS2 contains a script (runFLiRTS2.sh) to run FLiRTS 2 on the downloaded revisions of the subjects.
- timeTests contains a python script (timeTests.py) to execute and time the test cases selected by each tool.
- README.md provides information about the revisions, the inputs and outputs of each script, and the instructions to run the provided scripts.
Note that, downloading the revisions and running the whole experiment can take days and a lot of storage. It is advisable to run the experiment only on one (or few) subject(s). To adapt the experiment:
- keep in revisionsSHAs only the files named after the subjects you want to exercise.
- edit the file downloadRevisions/repositories.txt to keep only the GitHub URLs for the subjects you want to exercise.
- follow the instructions in the README.md file to run the experiment with the subject.
Reproducibility of the SoSyM Experiments.
In the experiment reported in our SoSyM submission to avoid any bias we restricted the experiment to separate the subjects used for FLiRTS 2 tuning from those used to evaluate it. It still uses 13 subjects with more than 3000 revisions. If you are interested in reproducing such an experiment download experiment sosym zip file. The zip file content is still the same as described in the general experiment only the used set of subjects (and the corresponding SHAs) is reduced. The same recommendations still hold.
UML Class Diagrams Generated from each Subject Revision
We generated the class diagrams from the code-level revisions of each subject using the Rational Software Architect Java to UML transformation. The following links can be used to download the zip files containing the class diagrams for the revisions of each subject. The SHA of the corresponding code-level revision is appended to the name of each class diagram.
- Walter Cazzola, Sudipto Ghosh, Mohammed Al-Refai, and Gabriele Maurina, Bridging the Model-to-Code Abstraction Gap with Fuzzy Logic in Model-Based Regression Test Selection, Software and Systems Modeling, July 2021. [DOI, PDF (CC)]