GxP-Blog

For some years now, AI has been making its way into many areas of our lives, including in regulated industries such as medical devices and pharmaceutical manufacturing. In the future, AI systems can lead to better treatment outcomes when incorporated into medical devices, for example in neurology, cardiology, oncology, psychiatry or rehabilitative robotics in prevention, early diagnosis and patient-friendly therapies. Since 2014, the FDA has approved more than 50 algorithm-based medical devices (3).

In pharmaceutical manufacturing, many application scenarios for AI applications may also be foreseen. For example, AI systems could be used to optimize manufacturing processes in production facilities, increase efficiency and reduce downtime. Use of AI to monitor manufacturing parameters or in optical quality control would be conceivable, e.g. by replacing a second pair of eyes with the 4-eyes principle. Yet the use of AI applications entails additional new risks inherent in AI: AI applications are considered “black boxes”, which present problems in terms of control and traceability.

While these risks generally apply to the use of AI, there is also a great deal of uncertainty for the pharmaceutical industry: To date, there are few mandatory regulatory requirements for validating AI / ML-based systems.

The FDA issued a "Discussion paper” in April 2019 for AI software defined as medical devices, but such initiatives for the application of AI/ML in pharmaceutical manufacturing are still largely absent.

In this area, the current GMP principles, which do not always explicitly take into account the specific nature of AI applications, should continue to be followed.

A lot of pioneering work still needs to be done at present to exploit the potential of AI applications in the pharmaceutical industry. Willingness to do so is steadily increasing in the industry.

AI based applications are IT applications designed to demonstrate intelligent behavior. Today's AI systems mean that they can also be trained and thus learn over time to achieve ever better results. This process is called machine learning (ML). Computer algorithms learn from data such as identifying patterns, automatically classifying images or showing desired behavior, without each individual case being explicitly programmed in advance. Machine learning is often synonymous with AI, although it is only one of many possible AI methods. Machine learning with large neural networks is called deep learning

Another common term from the AI environment, Big Data, describes the processing of very large and very different amounts of data, which are no longer manageable with conventional methods of data processing.

While traditional IT applications are based on rigid, hard-coded rules and cannot automatically change their behavior or calculation results over time, AI systems are able to dynamically adapt to changing environmental conditions or input data and deliver better results over time without explicitly programming system behavior in advance.

This blur or non-determinism of AI models is their inherent basic property, since the calculation or decision is not preprogrammed deterministically, but arises dynamically over the term of the model and can also be different with identical input information. In AI-based applications, it is therefore difficult to understand why a system makes a particular suggestion that is even more difficult to correct or prevent it in the future.

This means that, on average, the better calculation results of the AI systems come at the expense of correctness and reliability of an individual result, which can also be faulty within a certain probability scale. Reliability, traceability and the ability to reproduce results is an inherent problem in fully automated machine learning.

In the GxP environment, however, it is a basic requirement of the computer-aided systems that the results generated by these systems- e.g. B. calculations, forecasts and resulting follow-up processes or decisions - be transparent, understandable and reproducible. For the AI-based systems, which often appear as black boxes, this requirement is particularly difficult to implement.

Previously approved AI systems are the so-called "closed systems”: The learning process of the system is completed even before approval. If the AI system learns in use  after approval, the problem of traceability of results becomes more acute. The system then learns without human supervision. The software uses a continuously learning algorithm and processes new input data during operation – so the AI system is constantly changing and therefore no longer corresponds to the approval status. Current legislation does not permit the use of AI-based medical devices that continue to learn without supervision.

In validating AI systems, data are more important than knowledge from conventional IT systems.  One of the main causes of systematic errors in AI systems is an imbalance in the amount of data: Records can have a bias, i.e. contain bias effects such as errors in the sample selection, and as such, not be representative.  If the training data set contains too few or no examples of some classes for a classification algorithm, e.g. if, say only a few images of defects are required for an AI-based optical quality control, the algorithm will rarely be able to correctly assign images from these classes. An unconscious bias in the selection of training data can also lead to the algorithm not taking into account, for example, diseases that occur more frequently in certain ethnic groups. If while evaluating such algorithms explicit attention is not paid to the balance and representativeness of the training and validation data, it is very likely that the problem will only become apparent after commissioning. For this reason, procedures should be developed in the validation process of AI systems through which the balance of training and validation data sets can be systematically demonstrated.

Since AI applications exhibit non-deterministic behavior, and errors cannot be completely eliminated by additional testing, the GxP risk for product quality or for patients must be minimized as far as possible through organizational rules in the process. Particularly critical decision-making processes would have to be designed in such a way that the final decision-making authority remains with responsible stakeholders until AI governance quality reaches a level acceptable to all parties involved. This means that autonomy will only increase by small steps within the approval processes. Confidence in AI - like trust in people - is not achieved by unconditional traceability of methods, but by testing the algorithm and verifying its results by a human expert. "Adequate care must be taken in the documenting and quality assurance phases when introducing AI-based decision-making processes. In the case of critical processes, people should remain the final decision-making authority”(2).

Choosing the appropriate AI method can in some cases simplify the validation of the AI application.  For example, the "Reinforcement learning" approach,where a human trains the AI algorithm, is very helpful in some application scenarios to circumvent one of the biggest problems with machine intelligence: If the machine does not trust its own answers, it will ask the human expert for advice. On the one hand, the current business case is successfully concluded. On the other hand, human expertise supports the machine to improve the algorithms for the next opportunity. In contrast, this method is not very suitable for AI applications in the process industry, because here the failures are relatively rare and very individual, i.e. there is not enough data to train the algorithm to detect failure conditions automatically.

For AI applications in critical application areas like medicine or pharmaceutical manufacturing, traceability of functionalities, predictability of fundamental decisions and prevention of simple manipulation of an AI application are the fundamental requirements for safety and reliability (2).

Recent research has already developed procedures to make decisions comprehensible for some applications, such as the use of neural networks in image and text processing. Other approaches tending toward the "White Box AI" try to approximate an AI model locally by means of a simple, comprehensible representation in order to enable interpretation. These approaches, summarized under the term "explainable AI“, could make a valuable contribution to the validation of AI applications by clarifying how a data-driven algorithm reached a decision or recommendation, thereby ensuring the transparency of AI-based decisions.

Procedures to improve the evaluation of models should also be considered in this context The use of performance metrics or the calculation of statistics of such metrics can help to better understand models and, for example, check their robustness.  When selecting the final model, it is important to take a closer look at the rejected models as well as data points that could not be properly modeled. Both can help in better understanding how developed models perform.

The use of artificial intelligence holds enormous potential for pharmaceutical companies. However, in order to exploit this potential, some pioneering work still needs to be done at present, particularly in the area of setting up a regulatory basis and developing a specific methodology for validating of AI systems.

1. “Secure AI Systems for Medicine” White Paper of the IT Security, Privacy, Legal and Ethics Working Group of the Learning Systems Platform, Prof. Dr. Jörn Müller-Quade et al, April 2020
2. “Decision support with artificial intelligence: economic significance, societal challenges, human responsibility”, Bitkom/DFKI position paper
3. Peix Helth Group, Artificial Intelligence in Use for Pharma Targets. www.peix.de/2017/08/kuenstliche-intelligenz-im-einsatz-fuer-pharma-ziele/17.08.2017