ModelOps

ModelOps (model operations) is a discipline related to DevOps, designed for managing the lifecycle of a wide array of complex models, including machine learning (ML), artificial intelligence (AI), and decision optimization models. ModelOps coordinates the activities between application and model development. It orchestrates the flow of operations from training a model to putting a model into production, then evaluating and updating the resulting application according to a set of governance rules. It uses AutoAI and technologies such as decision optimization for model building, in concert with continuous integration and continuous deployment (CI/CD) to update the model on a regular basis. ModelOps defines the cadences between model behavior over time and the processes used to build, evaluate, and consume the models within the application development lifecycle. It grants business domain experts the capability to evaluate AI models in production, independent of data scientists.

ModelOps uses the pipeline concept prevalent in computing processes, such as application development and artificial intelligence (AI) model development. In standard application development, a pipeline is a sequence of processes, from collecting the source data to training the model, and then to deploying the model to predict results when presented with fresh data. In AI, a pipeline moves data from the ingest and processing phase through several transformation stages that might include machine learning, deep learning model training, evaluation, and retraining. ModelOps extends the development pipeline so it includes data processing and preparation, model training and retraining, and continuous integration and deployment to a production environment. The result is an application that, if managed properly, improves over time.

History

In a 2018 Gartner survey, 37% of respondents reported that they had deployed AI in some form; however, Gartner pointed out that enterprises were still far from implementing AI, citing deployment challenges. Enterprises were accumulating undeployed, unused, and unrefreshed models. Independent analyst firm Forrester also covered this topic in a 2018 report on machine learning and predictive analytics vendors: “Data scientists regularly complain that their models are only sometimes or never deployed. A big part of the problem is organizational chaos in understanding how to apply and design models into applications. But another big part of the problem is technology. Models aren’t like software code because they need model management.”

In December 2018, Waldemar Hummer and Vinod Muthusamy of IBM Research AI, proposed ModelOps as “a programming model for reusable, platform-independent, and composable AI workflows” on IBM Programming Languages Day. In their presentation, they noted the difference between the application development lifecycle, represented by DevOps, and the AI application lifecycle.

Traditional application lifecycle	AI application lifecycle
Requires DevOps skills	Involves more diverse skill sets
Relatively short running	Long-running, resource-intensive
Human speed (low change frequency)	Continuous training/retraining
Few versions of software artifacts	Huge number of models
Linear evolution of artifacts	Specialized models coexist
Configurations applied at runtime	Parameters tuned at training time
Codebase changes trigger new builds	Data/code changes trigger model retraining
Deterministic testing	Statistical/probabilistic testing
Monitoring of application performance & KPIs	Monitoring of model accuracy, drift, and KPIs

The goal for developing ModelOps was to address the gap between model deployment and DevOps with a programming model that would result in AI-powered business apps. In their presentation, Hummer and Muthusamy described a programmatic solution for AI-aware staged deployment and reusable components that would enable model versions to match business apps, and which would include AI model concepts such as model monitoring, drift detection, and active learning. The solution would also address the tension between model performance and business key performance indicators (KPIs), application and model logs, and model proxies and evolving policies. Various cloud platforms were part of the proposal. In June 2019, Hummer, Muthusamy, Thomas Rausch, Parijat Dube, and Kaoutar El Maghraoui presented a paper at the 2019 IEEE International Conference on Cloud Engineering (IC2E). The paper expanded on their 2018 presentation, proposing ModelOps as a cloud-based framework and platform for end-to-end development and lifecycle management of artificial intelligence (AI) applications. In the abstract, they stated that the framework would show how it is possible to extend the principles of software lifecycle management to enable automation, trust, reliability, traceability, quality control, and reproducibility of AI model pipelines. In March 2020, ModelOp, Inc. published the first comprehensive guide to ModelOps methodology. The objective of this publication was to provide an overview of the capabilities of ModelOps, as well as the technical and organizational requirements for implementing ModelOps practices.

Use cases

One typical use case for ModelOps is in the financial services sector, where hundreds of time-series models are used to focus on strict rules for bias and auditability. In these cases, model fairness and robustness are critical, meaning the models have to be fair and accurate, and they have to run reliably. ModelOps automates the entire pipeline of models used for data governance. Such automation includes data preparation, training the models, monitoring the model as it runs, monitoring the models for bias and other data anomalies, and updating the model as needed without disrupting the applications. ModelOps is the dispatcher that keeps all of the trains running on time and on the right track.

Another use case is the monitoring of a diabetic's blood sugar levels based on a patient's real-time data. The model that can predict hypoglycemia must be constantly refreshed with the current data and must be available in a distributed environment, so the information is available on a mobile device as well as reporting to a larger system. The coordination, retraining, monitoring, and refreshing is done with ModelOps.

The ModelOps process

The ModelOps process focuses on domain abstractions for platform-independent pipelines, enabling AI and application developers to easily plug in lifecycle capabilities (such as bias-detection, robustness and reliability, or drift detection) for building AI models and then putting them into production as business applications. The process starts with a generic AI pipeline representation that includes a metamodel (the model specification) with all of the component and dependent pieces that go into building the model, such as the data, the hardware and software environments, the classifiers, and code plug-ins. There are entities implemented as Python classes, with attributes defined in JSON schema, and stored in markup files like YAML and JSON. The language features include plug-in imports, variable placeholders and control flow. Extensibility is possible with plugins that can be referenced in the configuration. The ModelOps process coordinates the following sequence:

• A pipeline executor generates code from a specification and deploys the resulting pipeline to the target platform. Logs and metrics are retrieved from the platform. Pipeline templates, which allow for the inclusion of parameters, or options, to create templates of common pipeline patterns, can process training data, train a model, and deploy pipelines (model candidates). The templates allow for easily running experiments using default values or custom configurations.
• Pipeline transformers optimize the base configurations with additional features, such as adding deployment safeguards and plugging in cross-cutting features for trusted AI. Reusable domain abstractions that seamlessly move between a public and private cloud environment build intelligence into the pipeline, automatically adjusting the pipeline at runtime while also managing all the dependent assets required to run the pipeline.
• Classifiers, which are algorithms that sort data, are tested and fine-tuned. Model training entails multiple specialized stages, and each stage requires custom configuration, input-output mapping, and more, which can be tedious when done manually. ModelOps annotates the model entity with desired features and automates the process.
• Staged deployment controls the deployment of the model to a production environment, compares and updates model versions, and hides new model deployments in order to gradually roll out the change. Because models operate in dynamic environments, data and conditions tend to change over time, creating incorrect or inaccurate predictions. The term for this is model drift. In the ModelOps process, model drift detection monitors runtime traffic and raises an alert if drift is discovered and crosses a set threshold.

ModelOps: An evolution of MLOps

MLOps (machine learning operations) is a discipline that enables data scientists and IT professionals to collaborate and communicate while automating machine learning algorithms. It extends and expands on the principles of DevOps to support the automation of developing and deploying machine learning models and applications. As a practice, MLOps involves routine machine learning (ML) models. However, the variety and uses of models have changed to include decision optimization models, optimization models, and transformational models that are added to applications. ModelOps is an evolution of MLOps that expands its principles to include not just the routine deployment of machine learning models but also the continuous retraining, automated updating, and synchronized development and deployment of more complex machine learning models. ModelOps refers to the operationalization of all AI models, including the machine learning models with which MLOps is concerned.