Draft:Test Maturity Model integration

Test Maturity Model Integration

1. Introduction

In the ever-changing landscape of software development, ensuring product quality is paramount to delivering functioning software and satisfied customers. As software systems are becoming increasingly complex, the need for robust testing processes has never been more critical. The Test Maturity Model integration (TMMi) serves as a structured framework for organizations aiming to improve their software testing capabilities in a systematic manner. This article presents the TMMi framework, providing information about its history, as well as some basic guidance for using it. Visit tmmi.org to discover more details on how TMMi can transform a company's approach to software testing, fostering a culture of continuous improvement and excellence.

2. Overview

The Test Maturity Model integration (TMMi) framework is aimed at enhancing and evaluating software testing processes for improved quality and efficiency. Developed by the TMMi Foundation, it addresses the gaps left by broader models like CMMI, providing focused guidance specifically for testing.

TMMi includes five maturity levels, starting with chaotic, unstructured testing (Level 1) and progressing toward optimization with continuous improvement and defect prevention (Level 5). Its structured approach, built on process areas, goals, and practices, mirrors CMMI while zeroing in on software testing excellence.

TMMi v1.3 updates the framework by aligning with CMMI V2 terminology, and adhering to latest testing standards. TMMi also works effectively within Agile and DevOps environments, ensuring rigorous testing in iterative development cycles.  

Organizations can adopt TMMi through training programs, including the TMMi Professional certification, and perform assessments to identify strengths and areas for improvement. The model’s global adoption across industries highlights its effectiveness in enhancing software quality and operational efficiency.

The TMMi Foundation oversees the model's development and provides resources to support its implementation. By following TMMi, organizations systematically advance their testing processes, leading to higher-quality software and streamlined development workflows.

3. Background and History

Starting in 2005 a group of volunteers who were testing professionals and organisations with expertise in software testing and quality assurance got together to create the TMMi Foundation initiative. Brian Wells then officially launched the TMMi Foundation, a non-profit organisation at the ICSTest-UK conference in 2005. The conference dedicated a track to TMM (since TMMi did not exist then), including two case studies presented by the retailer Marks and Spencer and IT service provider Improve QS and an afternoon tutorial on TMM.

The TMMi model released in 2010 is aligned with international testing standards, and the syllabi and terminology of the International Software Testing Qualifications Board [lSTQB]. The TMMi Foundation has consciously not introduced new or their own terminology but reuses the ISTQB terminology. TMMi is an objective and business-driven model. Testing is never an activity on its own and the purpose behind TMMi was to introduce a structured staged approach to improve testing practices within software development by providing detailed guidance on test processes, techniques and management. The model contains stages or levels through which an organisation passes as its testing processes evolves. It was evident that whilst software development maturity models like CMMi existed, there was no such maturity model for software testing and its processes. Organisations using the TMMi framework could make incremental improvements by following the five maturity levels when assessing the test organisations maturity. TMMi rapidly gained acceptance within the industry mainly due to its alignment with the CMMi model and the model's ability to address software testing challenges. The TMMi model is considered a complementary practice to CMMi and has been translated into many languages. Over the years the foundation has benefited from a broad contribution from members and organisations to develop the Foundation based on the following principles:

1. Develop a model that is
   • Progressive and data-driven
   • Enables continious improvement
   • Inspired by CMMi
   • Integrated with project lifecycle
   • Measurable outcomes
   • Encourages institutionalised learning
2. The model was made available so that anyone could download and use the model as a reference for making improvements to testing.

5. Scope of the TMMi v1.3

The Test Maturity Model integration (TMMi) is a detailed and comprehensive framework for assessing and improving the test process maturity of an organization. The scope of the TMMi model encompasses various aspects of testing processes, aiming to provide a structured path for organizations to follow in order to achieve higher levels of testing maturity.

TMMi supports testing and test process improvement in systems and software engineering, encompassing both hardware and software aspects.

TMMi addresses all test levels, from static to dynamic, including component, integration, system, and acceptance tests. It covers the four key areas of structured testing: lifecycle, techniques, infrastructure, and organization.

While TMMi functions as an independent test improvement model, it can complement CMMI. It references CMMI practices, especially for supporting and management processes (like Configuration Management), without duplicating details.

TMMi is lifecycle-independent and can be applied to sequential, Agile, and DevOps models. A survey shows 78% of TMMi users utilize it in Agile contexts, and 57% in DevOps. The TMMi Foundation provides a detailed guideline to support Agile use, offering Agile alternatives for traditional practices. Additionally, a white paper explains how TMMi supports the DevOps approach . See https://www.tmmi.org/tmmi-documents/ for details. .

Organizations use TMMi for test process assessments to benchmark and identify improvements. The TMMi framework is a reference for these assessments, guided by the TMMi Assessment Method Application Requirements based on ISO 15504. TMMi can also complement CMMI assessments by integrating development and testing evaluations.

While TMMi offers a framework for test process improvement, it does not prescribe a specific approach like the IDEAL model. Successful improvement often starts with strong organizational support and the formation of a skilled test process group.

6. TMMi Maturity Levels

A maturity level within the TMMi represents as a degree of organizational test process quality. It is defined as an evolutionary plateau of test process improvement. Each level progressively develops a critical aspect of the organization’s test processes. TMMi framework has five maturity levels. Each maturity level defines what to implement in order to achieve the given level. Note that all organizations doing testing usually posess a minimum set of activities required by the model already on TMMi level 1, even if this level does not contain any goals that must be satisfied. The higher the maturity level the organization achieves, the more mature the test processes of organization are.

5 Maturity Levels of TMMi

The Process Areas associated to the maturity levels are the following:

Maturity Level 2 – Managed

• 2.1 Test Policy and Strategy
• 2.2 Test Planning
• 2.3 Test Monitoring and Control
• 2.4 Test Design and Execution
• 2.5 Test Environment

Maturity Level 3 – Defined

• 3.1 Test Organization
• 3.2 Test Training Program
• 3.3 Test Lifecycle and Integration
• 3.4 Non-functional Testing
• 3.5 Peer Reviews

Maturity Level 4 – Measured

• 4.1 Test Measurement
• 4.2 Product Quality Evaluation
• 4.3 Advanced Reviews

Maturity Level 5 – Optimization

• 5.1 Defect Prevention
• 5.2 Quality Control
• 5.3 Test Process Optimization

7. Structure of TMMi

The Test Maturity Model Integration (TMMi) is structured similarly to the Capability Maturity Model Integration (CMMI) v1.3, ensuring familiarity for organizations that have already adopted process improvement models. The model consists of multiple components categorized into required, expected, and informative elements, which guide organizations in improving their testing processes.

Key Components

TMMi is composed of the following structured elements:

• Maturity Levels: A progressive framework defining the test process maturity of an organization
• Process Areas: Key focus areas for achieving maturity at different levels.
• Specific and Generic Goals: Mandatory targets an organization must achieve to satisfy a process area
• Specific and Generic Practices: Recommended activities to help organizations meet the associated goals
• Supporting Informative Components: Guidance materials such as sub-practices, examples, and references.

Process Areas and Practices

Each maturity level (except Level 1) contains process areas, which are clusters of related activities essential for improving test maturity. Specific and generic practices provide implementation guidance. Some key process areas include:

• Test Planning – Developing a structured approach to testing.
• Test Monitoring & Control – Measuring and managing test execution.
• Test Lifecycle & Integration – Aligning testing within the Software Development Lifecycle (SDLC).
• Test Measurement – Establishing metrics for process evaluation.

Relationship with CMMI

Originally aligned with CMMI v1.3, TMMi has incorporated industry best practices while evolving independently. Despite structural similarities, TMMi focuses exclusively on software testing, complementing broader process improvement models like CMMI.

Future Structural Considerations

With CMMI transitioning to newer versions (V2 and V3), discussions within the TMMi Foundation have explored whether to align future TMMi versions with updated CMMI structures. However, TMMi continues to maintain its independent identity while considering best practices from modern software engineering frameworks.

This structured approach ensures that organizations can systematically assess, improve, and optimize their testing processes within the broader context of software quality assurance.

8. TMMi and CMMi

The TMMi model has positioned itself as being complementary to CMMI (Capability Maturity Model Integration ¹). While TMMi concentrates on testing processes, CMMI was conceived to help improve the entire software development process.  

Testing within the SDLC (Software Development Life Cycle)  is an important activity, emphasized by CMMI as needed during the entire software development. [5]  However, testing appears explicitly in the CMMI model in a few Practice Areas, and these provide  a high-level guidance, without entering into details. Practice areas in CMMI connected to testing are: VER (Verification) and VAL (Validation). Elements of the integration testing can be found in PI (Product Integration).  Peer Reviews, being a separate Process Area previously, became a Specific Practice of the VER Process Area.  

As testing is addressed at a rather high level in CMMI, organizations that wish to concentrate on improving their testing processes benefit from using TMMi.    

TMMi v1.3 shares a common structure with CMMI v1.3.  

The basic elements of the two  models are the same: Specific Goals and Generic Goals are required model components, Specific Practices and Generic Practices are expected components, and both models have informative components (sub-practices, example work products, notes, examples, and references).

Although declaring complementarity with CMMI (having both staged and continuous representations),  TMMi was developed basically as a staged model. The Five Maturity Levels of TMMi are the same as CMMI Maturity Levels.  

Although TMMi can be used in isolation for the testing processes within a software development organization, it is often used as a complementary model to the CMMI.  

Practice areas from CMMI are generally not repeated within TMMi model description, but they are often referenced. Entire chapters from TMMi model description point out the interconnections between CMMI and TMMi practices. For details see TMMi model description ( TMMi Reference model v1.3)

The world-wide surveys from 2021([10]) and 2023 ([11]) indicate that while initially positioned as a complementary model to CMMI, today TMMi is also widely used independently: around 40 %. of its users do not use CMMI - taking the average of both user surveys.

The actual version of TMMi is v1.3, which used the concepts of CMMI V1.3.  

In the meantime CMMI had new version release: CMMI V2.0 – with a major structural change- released on March 28. 2018, and CMMI  V3.0 (minor updates compared to CMMI V2.0), released on 6 April 2023.  

9. TMMi in the Agile environment

The TMMi framework and Agile approaches can be used together, and provide substantial benefits when integrated effectively. TMMi emphasizes the integration of testing within the software development process rather than treating it as a separate activity. While the Agile approach focuses on testing practices such as unit testing, test automation, and exploratory testing, TMMi can complement these by highlighting important , locate the specific testing areas where change can provide value given the business objectives.

TMMi should not be seen as a heavy set of practices to be followed, but rather as a flexible model that helps identify valuable changes in testing processes. Agile practices often achieve the intent of TMMi practices through alternative means, reflecting the importance of aligning practices with business needs rather than strict adherence to predefined standards or contract.

The Agile mindsets emphasizes simplicity, which can be challenge. TMMi implementation can support to analyse the process and reduce the overall risks. Agile improvements often occur through small, empowered teams that make quick decisions, offering further benefits to TMMi practices.

Using the TMMi model in an Agile context provides reminders of critical testing practices that are often “forgotten”. The document TMMi in the Agile World provides detailed examples of how the TMMi framework and Agile methods can work together successfully.

10. Trainings

One main mission of the TMMi Foundation  is to ensure professional training connected to the model and its assessment. Exams can certify the knowledge gained on the trainings.

TMMi Foundation has developed a network training providers (https://www.tmmi.org/tmmi-professional-training-providers/) and  accredits them to ensure the uniform quality  of the trainings around the world. The training materials of these training providers are developed based on the actual Professional Training Syllabus  (https://www.tmmi.org/tmmi-documents/) and is aided by sample exam papers freely downloadable.

TMMi Professional Training

The TMMi Professional qualification course is designed for individual or group who involved in using the TMMi model for the test process improvement implementation. This includes people in roles such as test process improvers, test consultants, TMMi (lead-)assessors, business stakeholders, test managers, and members of a Test Process Group.

This TMMi Professional qualification is appropriate for anyone who is seeking an understanding of the TMMi model. Holders of the TMMi Professional Certificate will be able to demonstrate a level of knowledge required as a prerequisite to become an accredited TMMi lead-assessor or assessor.

The TMMi Professional training course has eight chapters.

• Context of Test Improvement
• Introduction to the TMMi Model
• TMMi Maturity Levels
• Structure of the TMMi
• TMMi Model
• TMMi in an Agile context
• TMMi Assessments
• Implementing TMMi

The training is typically run as two days. It will explain to management the business importance of test process improvement. Guide and advice an organization or project when using the TMMi model as a basis for their test process improvement, both in traditional sequential lifecycle environment as Agile context. Providing support in the interpretation and understanding of the TMMi model, including the relationship of the TMMi to the CMMI model.  Act as a co-assessor in informal TMMi assessments. Participate in programs for improving the test process within an organization or project and can identify critical success factors.

One can get TMMi Professional certified by achieving 65% on the one-hour, 40-question multiple-choice exam offered by the recognized exam providers.

Training can be completed by joining one of the TMMi Foundation’s recognized training providers or professionals can prepare by themselves for the certification exam using the documents provided on the TMMi website.

TMMi (Lead) Assessor Training

The TMMi (Lead) Assessor course is designed for individuals who want to become an accredited TMMi lead-assessor or assessor. It provides an overview and statement of what can be expected from a TMMi assessor, in terms of the added value and skills they will bring to the business.

The TMMi (Lead) Assessor course has eight chapters.

• TMMi Model
• TMMi Documents and Processes
• Introduction to TMMi Assessments
• Assessment Planning and Preparation
• Data Collection
• Process Component Rating
• Assessment Reporting
• Assessment Closure

The training is typically a  three-days one. It will teach the expected performance of the assessments, meeting the highest level of professional integrity.  One can join a training from one of the TMMi Foundation’s recognized TMMi assessor training provider.

11. TMMi Assessment

Using TMMi model offers the possibility to a testing organization of obtaining a certification about organization’s maturity. Based on TMMi, organizations can have their test processes objectively evaluated by accredited Lead Assessors and Assessors, and improve their test processes based on the results. Having a certification, organizations can demonstrate to their customers that their testing processes reach a certain level of capability and maturity, thus ensuring that the testing organization will meet customer requirements. The framework of training, accreditation of assessors and assessment of organizations’ test processes is ensured and maintained by TMMi Foundation (see details at tmmi.org / Assessments).

There are 2 types of assessments defined by the TMMi Foundation: formal assessments and informal assessments.

A formal assessment is recommended if the organization requires a detailed assessment and / or a certification is desired. This type of assessment provides a gap analysis together with the improvements required to achieve the desired level of maturity. In this case the assessment team must be led by an Accredited Lead Assessor and must contain at least 2 people. The assessment will consist of document review and interviews, and corroboration of all evidence gathered must be verified. Each Process Area will be rated as Satisfied of Not satisfied, based on practice and goal characterization. The result is a Maturity Level.

An informal assessment can be considered if an organization wishes to have indicative results of its testing processes and organizational maturity. In this case the assessment team can consist of one Accredited Assessor, and one type of objective evidence is enough (interviews or documents). No formal rating will be generated, but a gap analysis will be done and a direction for improvement will be set.

TMMi Foundation ensures Lead Assessor and Assessor training, accreditation, and maintains the list of persons entitled by TMMi Foundation to conduct assessments  (https://www.tmmi.org/leadassessors/, https://www.tmmi.org/assessors/).

An organization can require TMMi assessment through the accredited Assessment Service Providers or using independently accredited TMMi (lead) assessors. Any organization can apply to become an accredited Assessment Service Provider. TMMi Assessment Method Application Requirements (TAMAR) R1.1 (see [1]) defines the requirements for performing an informal or formal TMMi assessment and developing a TMMi assessment method. Anyone can develop their own assessment method based on these requirements. Requirements conform to ISO 33002 ([3]). There are more assessment  methods usable by the assessors.  Here we list them based on TMMi Foundation home page (https://www.tmmi.org/about-tmmi-assessment-method-tam/ and https://www.tmmi.org/assessment-options/). The TMMi Assessment Method (TAM) is an accredited assessment method for conducting test process assessments against the TMMi Model. TAM has been developed by the TMMi Foundation and is available for licensing. The supporting tool is available in multiple languages. Actually (August 2024) there are accredited TMMi (Lead) Assessors who are using the SCAMPI (Standard CMMI Appraisal Method for Process Improvement)   for assessments. A specific supporting guidelines document has been created for SCAMPI Appraisers that also perform TMMi assessments.

Likewise, a number of accredited TMMi (Lead) Assessors use ASPICE (Automotive SPICE) Software Process Improvement & Capability dEtermination. ASPICE (V1.3) has now been recognized as a valid method to perform TMMi assessments. It is derived from the generic SPICE (ISO/IEC 15504) standard.)

It is possible to do a Self Assessment using an accredited assessment method, and it involves having at least one TMMi accredited Assessor within the organization and using an accredited TMMi assessment method.

Self Assessment without an accredited assessment method means that TMMi model is freely available to download to use it as a guide for an informal assessment approach. The resulting assessment report and recommendations may not be as detailed or accurate as using an accredited TMMi assessment method.

It is important to mention the TMMi Lightning Scan Tool v2.4 , which is available in different languages (English, Dutch, French, Italian, Chinese).  The TMMi Lightning Scan has been developed to get a sense of test maturity in an organization, compared to the TMMi.  The tool is easy to use, and it is available in the languages English, Chinese, French, Italian and Dutch.

When an assessment is done, a template can be used for submitting TMMi assessment results to the TMMi Foundation. This template, the  Data Submission Form (DSF) (see [2]), is available on TMMi web page. TMMi Foundation will be able to maintain only those data connected to TMMi assessment that were submitted to them via DSR.

According to TMMi Foundation official page, there are actually (January 2024) 333 official TMMi certifications worldwide. However, the world-wide surveys from 2021( 5])  and 2023 ( [6])   clearly indicate that there is a broad set of companies using TMMi concepts without being ever assessed, certified or undergoing an informal assessment (which does not appear with the previously referenced certifications).  Based on an extensive IEEE study ([4]) the estimated number of TMMi users world-wide is now around 9.000.

It is worth to note that TMMi can be used in conjunction with the CMMI (staged version) to do combined  assessments (software development processes are addressed within CMMI, and testing processes are addressed in more detail using TMMi).

TMMi is needed to understand and asses testing processes, as CMMI addresses testing at a rather high level, having Verification, Validation and Product Integration Process Areas.  

In 2018 CMMI Version 2 (CMMI v2) was released by the ISACA, the new owner of CMMI. Together with CMMI v2 the new Appraisal Method Definition Document (further: MDD) with requirements for CMMI V2 appraisals was released. The question whether TMMi Assessment method(s) can make use of best practices from MDD is being an important subject of analysis to the TMMi community (see ([7])

12. TMMi in the World

TMMi is used in many countries of the worls. TMMi Foundation established the system of Local Chapters to ensure the uptake of TMMi in different regions. In the beginning of 2025 the 30^th Local Chapter was created. The complete list of Local Chapters can be found at https://www.tmmi.org/tmmi-local-chapters/.  

TMMi Foundation recently performed two world-wide surveys to understand  who is using TMMi, the costs and benefits associated with applying TMMi.  Details are described in [3] and [4]. It resulted that today, TMMi is world-leading model for test process improvement, as described in [5] and emphasized by  [6] and [7] .  

Among the advantages of TMMi model we can mention the possibility of obtaining a certification about test organization’s maturity. Based on this model organizations can have their test processes objectively evaluated by accredited assessors and improve their test processes based on the results.

There are more than 200 companies officially assessed to be compliant to a certain TMMi Maturitly Level (https://www.tmmi.org/accredited-certifications/).  

However, the world-wide surveys from 2021( [3]) and 2023 ( [4]) clearly indicate that there is a broad set of companies using

TMMi concepts without being ever assessed, certified or undergoing an informal assessment.  Further information about TMMi cost, benefits and motivation to use TMMi can be found in [5], [6], [7].

One reason for becoming a world-leading model can be the strength of the TMMi model that its terminology is synchronized with other test-related approaches and standards.  TMMi is fully aligned with Glossary and Syllabi of International Software Testing Qualifications Board (ISTQB), which is a leading organization in certifying testers world-wide. (www.istqb.org). Besides ISTQB, TMMi is also aligned with a set of ISO standards connected to testing: ISO 20246, ISO 25010, ISO 25040, ISO 29119-1, ISO 29119-3 and ISO 33061. TMMi follows the newest technology trends;  Agile, DevOps and SAFE principles are incorporated within the practices of the model. The document TMMi in the Agile world - V1.4 ( [2])   describes how to apply and use TMMi in an Agile context.  

3. TMMi World-Wide User Survey 2021. Produced by TMMi Foundation. V1.1. https://www.tmmi.org/tmmidocuments/

( 2021).

4. TMMi 2nd World-Wide User Survey 2022- 2023 - V1.0. Produced by TMMi Foundation. /www.tmmi.org/tmmi-documents// ( 2023)

5. Erik van Veenendaal: Costs and Benefits of the TMMi– Results of the 2nd TMMi World-Wide User Survey Quality Matters, Issue No. 16, July 2023 . www.tmmi.org/download// ( 2022)

6. Erik van Veenendaal, Vahid Garousi and Michael Felderer: Motivations for and Benefits of Adopting the Test Maturity Model integration (TMMi) Software Quality – The Next Big Thing in Software Engineering and Quality, May 2022, Springer Publishing (2022)

7. Vahid Garousi, Michael Felderer, and Tuna Hacalo˘glu: What we know about software test maturity and test process improvement IEEE Software, January/ February 2018 (2018)

13. TMMi Foundation

TMMi Foundation is a non-profit organization, established in 2005, with the goal to create an up-to-date Test Maturity Model, based on the previously existing approaches (see Background and History ). TMMi Foundation has a well-defined organizational structureⁱ   and it operates based on a professionally developed Code of Conduct  (see https://www.tmmi.org/code-of-conduct/)    

TMMi Foundation maintains the TMMi model and it ensures that the model description is freely downloadable.   It also ensures model translations into a continuously growing number of other languages (for instance: Chinese, French, Spanish, Brazilian, Polish, Japanese…).  

The framework of training, accreditation of assessors and assessments of organizations’ test processes is also ensured and maintained by TMMi Foundation. Lists of Accredited Training providers, Accredited TMMi Service Providers, Accredited Lead Assessors and Accredited Assessors are displayed on TMMi Foundation webpage.  

TMMi Foundation has a strategic alliance with ISTQB^[ii], ensuring this was that the set of terms used in testing are common for the two world-wide organizations, forming the software testing community’s body of knowledge.

TMMi Foundation official page contains information about official TMMi certifications worldwide https://www.tmmi.org/accredited-certifications/).

In the beginning of 2025 there are more than 200 companies officially assessed according to TMMi requirements. Majority of the companies are ML3 compliant, but a considerable number of ML5 certifications are also mentioned.