4th Workshop on
Validation, Analysis and Evolution of Software Tests

Abstract - Test cases are meant to form the first line of defence against the introduction of faults in a software product, especially when re-testing modified code. Therefore, assessing the quality of a test suite is crucial to ensure that system under test is correctly implemented, which can help to decrease the overall development and maintenance costs. Test quality can be considered in different perspectives such as adequacy, correctness, and runtime performance of the test suite. Currently, most test analysis techniques consider test methods as the smallest unit of interest. However, in order to effectively analyze tests, we need to look inside tests. In this talk, we will explore "fine-grained test analysis", which dissects test methods and examines them at the test statement level. This also enables us to focus on test assertions, which are the heart of test methods by performing the actual checking. We will discuss how fine-grained analysis can facilitate various test analysis tasks such as test effectiveness assessment, test minimization, and test repair.

Bio - Ali Mesbah is an Associate Professor at the University of British Columbia (UBC) in Vancouver, Canada. During 2017-2018, he was a Visiting Research Scientist at Google. His main area of research is software engineering, with emphasis on software analysis and testing, software evolution, and fault localization and repair. He is the recipient of numerous awards including the Killam Faculty Research Prize at UBC, a Discovery Accelerator award from NSERC, two ACM Distinguished Paper Awards at ICSE, an IEEE Distinguished Paper Award at ICST, a Best Paper Award at ESEM, and a Best Paper Award at ICWE. He is an Associate Editor of the IEEE Transactions on Software Engineering and regularly serves on program committees of top software engineering conferences such as ICSE, FSE, and ISSTA.

Abstract - In this paper, we present a coverage-based regression test selection (RTS) approach and a developed tool for Python. The tool can be used either on a developer's machine or on build servers. A special characteristic of the tool is the attention to easy integration to continuous integration and deployment. To evaluate the performance of the proposed approach, several small industry projects are used to evaluate time savings in DevOps/MLOps process. Furthermore, to study precision, mutation testing is applied to three open-source projects, and the results of the execution of full test suites are compared to the execution of a set of tests selected by the tool. It is shown that the missed fault rate of the test selection at file-level and line-level granularities vary between 0-2 % and 16-24 %, respectively. For internal projects, in best cases, testing time was decreased by a factor of four. The high missed fault rate at the line-level granularity is related to the selected basic mutation approach and the result could be improved with advanced mutation techniques. Depending on the target optimization metric (time or precision) in DevOps/MLOps process the error rate could be acceptable or further improved by using file-level granularity based test selection.

Abstract - Automated Driving System refers to a vehicle system where hardware and software are collectively capable of on-road operational and tactical functions and such functions involve the detection, recognition, classification of objects and response to events. Many automotive companies are incorporating automated driving into their current R&D and are transforming their business models. To both conventional and disruptive manufacturers, safety is always one of the top priorities. Appropriate verification and validation procedures are needed and should be followed to mitigate unreliability and hazardousness. Sufficient testing scenario should be considered and planned to simulate and cover functional and non-functional requirements. Disengagement ratio serves as an indicator during performance evaluations because analyzing root causes of both technical and non-technical disengagements is pivotal especially during testing strategy planning. Autonomous Vehicle Disengagement Reports and Autonomous Vehicle Collision Reports from the Department of Motor Vehicles (DMV), California, USA are collectively used for the purpose of this research. And the analytical result shows there is no clear relationship between mileage and disengagements. Influencing factors are generated and consolidated from the mentioned reports and are proposed in addition to a Society of Automotive Engineers International (SAE International) standard. Stakeholders will benefit from the presented rationales and consider the suggestive parameters throughout their developing and testing activities. This paper further recommends testing management for automated driving systems, especially test driver management and test routes planning. And the recommendations are in accordance with the analytical results and feedback from KPMG Global Automotive Executive Surveys.

Abstract - This paper presents an envisioned approach to AI-based enhancements of test models in the industrial model-based testing tool TEMPPO Designer. Based on an overview of the tool, we present the required data collector, the AI-based data analyzer and ways to integrate the results into TEMPPO Designer.

Abstract - Evolving configurable software poses unique challenges, due to dependencies between the configuration knowledge and the assets that implement the system. Despite a lot of research in the area of configurable software evolution many open challenges subsist, which hamper industrial adoption of different techniques. One such challenge is that the configuration space is often not clearly defined and reasoning over its evolutions and the implication from it is difficult. In this paper, we study the configuration space of the web-based bug tracker Bugzilla, a highly configurable system, replicating the analysis of previous research. In addition, we analyze the evolution of the configuration space over several versions of the system to investigate the changes that were made over the revisions spanning the development of several years and their implications and open challenges.

Abstract - Automated test case generation has been one of the most active research topics in software testing for several decades. Many different approaches and tools have been proposed. In this talk, we present a tool under development for software test case generation, Symflower. Symflower employs advanced language independent symbolic execution to generate test data and analyse programs for potential problems. It can be used locally or as a cloud solution. All that is required is the location of the source code repository and Symflower can fully automatically test the software. The goal of Symflower is to generate a small, comprehensible test suite with high code coverage.

Abstract - Fuzzing (or Fuzz Testing) is a technique to verify the robustness of a program-under-test. Valid input is replaced by random values with the goal to force the program-under-test into unresponsive states. In this position paper, we propose a white box Fuzzing approach by transforming (mutating) existing test methods. We adopt the mechanisms used for test amplification to generate crash inducing tests, which developers can reproduce later. We provide anecdotal evidence that our approach towards Fuzzing reveals crashing issues in the Pharo environment.