Non-regression testing
Non-Regression Testing (NRT) is an approach of software testing which verifies if the desired previous functionalities of a software application (SW) were not compromised after the introduction of new procedures or functional modifications in a new release. The observation of undesired behaviours generally indicates a software regression, i.e. the new features added to the SW introduced software bugs in the functionalities that were working in the right way in its previous version.
Introduction: Development of a Software
The development process of a software application can be divided in several steps, where each one culminates in a new SW version containing a certain number of new features until the last release, when all the contents that satisfy the customer’s requirements are available. As the complexity of SW architecture grows up, step by step, higher is the probability of occurrence of bugs during this development process. Bugs can occur when the software code is modified mainly for two reasons:
- new procedure in conflict with an old one;
- modification to upgrade a pre-existing procedure.
Usually, the occurrence of a software bug can result in an unexpected delay to the project. Due to time-to-market restrictions, the validation phase of software functionalities must be well organized and efficient so that it lasts as short as possible. In this context, Non-Regression testing provides a systematic procedure to have a fast and efficient validation phase and discover as soon as possible the software bugs hidden in the architecture.
How to perform a Non-Regression Test
A Non-Regression test can be performed according the following steps:
- Define a benchmark SW release;
- Define a set of routines able to stimulate as many functionalities as possible of the SW;
- Launch these routines on both SW (the benchmark and the new release under test) and acquire data which represents their behaviour;
- Analyse this data with a post-processing tool, able to provide statistic results;
- Report the outcome.
Exploratory testing is performed following similar steps, but it differs from NRT concerning their analysis phase and focus, searching, hence, for different results and conclusions. The NRT aims to check if any undesired behaviour comes out after the last modifications applied to the software. There, the new behaviour of the application is previously known, making possible the identification of an eventual regression (bug). The exploratory testing, on the other hand, seeks to find out how the software actually works, conciliating simultaneous testing and learning, and stimulating testers to create new test cases.[1]
Regression and Non-Regression Testing
While the intent of regression testing is to assure that a bug fix has been successfully corrected, based on the error that was found, by retesting the modified SW, the Non-Regression Testing aims to verify the no introduction of software bugs due to the introduction of new features.[2] In general, this difference of definitions can be assumed as based on the outcome of each test, as follows.
When a new software version is released without any new feature regarding to the previous one (i.e. the differences between them are restricted to bug fixes or software optimization), both the releases are expected to present the same functionalities. In this case, the tests applied to both versions are not expected to result in different behaviours, but only assure the fixing and/or no introduction of new bugs. This testing methodology characterizes the regression testing.
On the other hand, when the new release presents new functionalities or improvements that lead the SW to behaviour in a different way, then the tests performed on the previous and new version can result in:
- desired differences, related to an expected new behaviour; and
- undesired differences, which indicate a software regression generally caused by a side-effect bug.
In this case we can talk about Non-Regression Testing.
Who performs the Non-Regression Testing
Once the customer has set all the requirements, the supplier will introduce all the contents, release by release, until the final one. In this context, NRT can be performed by both the customer and the supplier.
It can be made by the supplier as a beta testing service to guarantee a higher quality product with a very low percentage probability of bugs. Basically the client is equipped with simulation environment that enables an easy way to perform routines and acquire data. In case of regression the supplier, owing the know-how, can quickly solve the problem and avoid releasing a malfunction SW version to the customer.
On the other hand, NRT can be performed by the customer as an acceptance testing in order to prevent his final product from damages and eventually charge the supplier for the mismatch with requirements. Moreover, the customer, having a reduced know-how about the software structure, can perform the NRT as a black-box testing and, after meeting a regression, refuse the new SW release.
NRT Automotive Applications
Throughout the years Engine Control Unit (ECU) SW requirements are getting more complex and harder to reach due to the more and more stringent emission norms and ambitious performance in terms of fuel consumption and power request, which also increase the demand and complexity of in-vehicle driving tests and diagnostic functionalities. As a consequence, along engine control systems development, each new SW release results from a sequence of many others, each one introducing new functions seeking to satisfy, time after time, the crescent demands. In this context, Non-Regression testing is useful to verify that the performance and robustness of each SW release does not decrease in relation with the previous one, or, in other terms, does not introduce regression.
NRT is applied, along each SW release testing phase, as the final stage during integration testing, right before the execution of system testing and after the module testing (or unit testing) phase.[3]
In the module testing phase, single SW modules are evaluated individually, which allows the identification of elementary errors like overflow, underflow, round-off, and also discrepancies between algorithm model simulation results and the signals coming from the Engine Management System (EMS). The integration testing phase, performed afterwards, aims to verify if the tested module is correctly integrated in the overall SW system. Finally, functional testing (also called validation testing) is applied to validate the algorithms concerning functional requirements. This stage is usually performed after the calibration phase and characterizes an overall system testing, concluding the new SW testing phase, and allowing, therefore, its release.
In automotive applications, non-regression testing is performed as follows:
- Selection of test manoeuvres and definition of engine parameters to be monitored;
- Execution of the selected manoeuvres on benchmark SW and the SW under test;
- Post-processing and analysis of data acquired during these tests.
The selected test manoeuvres must be able to stimulate as many algorithms implemented in the SW as possible. Cold start, Overshoot of Rounds per minute, and ECE cycle (a standard manoeuvre used to calibrate On-Board diagnosis) are relevant examples. In addition, the engine parameters selected to be monitored must represent the engine global operating state along the manoeuvres executed, such as Accelerator pedal deflection, Engine speed, Vehicle speed, Engine temperature and Throttle body opening percentage. It’s also necessary to monitor the mean variables of the control chain of Air and Torque estimation. All these diagnostic variables must be kept under control during the execution of the manoeuvres.
The tests are performed in simulation environments such as Hardware-in-the-loop (HIL) simulators or Micro HIL (feed-forward systems that work as downsized simulators of the previous ones), which support the drawing and execution of complex manoeuvres usually very difficult to perform on a real engine or car (mainly because of time, cost and equipment restrictions).[4]
Afterwards, a post-processing tool is required to process the acquired data, offering graphic analysis and statistical data, generally addressed to skilled personal able to identify possible regression on the SW. This kind of tool can also be endowed with an automatic report generator, which gathers in a single document all the analysis results and conclusions from the comparison between the two SW releases during the NRT.
See Also
References
- ^
Kaner, Cem (2001). Lessons Learned in Software Testing. John Wiley & Sons. ISBN 0471081124.
{{cite book}}: Unknown parameter|coauthors=ignored (|author=suggested) (help) - ^ T. Jéron, J.M. Jézéquel, Y. Le Traon and P. Morel, "Efficient Strategies for Integration and Regression Testing of OO Systems", 10th IEEE International Symposium on Software Reliability Engineering (ISSRE'99), Florida, p. 260-269, November 1999. By VerTeCs (accessed on 01/10/10).
- ^ A. Palladino, G. Fiengo, F. Giovagnini, and D. Lanzo, "A Micro Hardware-In-the-Loop Test System", IEEE European Control Conference, 2009. Abstract (accessed on 01/10/10).
- ^ S.Raman, N. Sivashankar, W. Milam, W. Stuart, and S. Nabi, "Design and Implementation of HIL Simulators for Powertrain Control System Software Development", Proceedings of the American Control Conference, 1999.