Bhusan Chettri summarises different Interpretable Machine Learning methods

 

 

 

This tutorial from Bhusan Chettri provides an overview of different methods of interpretable machine learning (IML) a.k.a explainable AI (xAI) framework. This tutorial is the third installment of the interpretable AI tutorial series by Dr Bhusan Chettri a PhD graduate in AI and Voice Technology from Queen Mary University of London. The tutorial explains different approaches towards explaining or understanding the working phenomenon of data-driven machine learning models. The methods for interpreting AI models falls usually in two categories: (1) aiming to design inherently interpretable models that are fairly easy and straightforward to understand; (2) and, devising specialized algorithms and methods to analyze or unbox a per-trained black-box machine learning models (usually deep learning based). This second category is often referred as post-hoc interpretability methods that take into account a per-trained model rather than aiming to incorporate various conditioning during model training as done in case of approach 1. As the two topics are quite vast to cover in a single tutorial, more focus has been put on first part in this tutorial. The follow-up on this tutorial will focus more on post-hoc methods of interpretability, the second part. 

 

 

 

 

                            

 

 

 

 

However, before getting deeper into today's topic, Bhusan Chettri briefly revisits the previous two installments of this tutorial series. Part1 focused on providing an overview of AI, Machine learning, Data, Big-Data and Interpretability. It is a well known fact that ‘Data’ has been the driving fuel behind the success of every machine learning and AI applications. The first part discussed how vast amount of data are produced every single minute from different mediums (online transactions, sensors, surveillance, and social media). It talked about how today’s fast growing digital age that leads to generation of such massive data, commonly referred as Big Data, has been one of the key factor towards the apparent success of current AI systems across different sectors. The tutorial also highlighted how AI, Machine Learning and Deep Learning are inter-related: deep learning is a subset of machine learning and machine learning is a subset of AI. In other words, AI is a general terminology that encompasses both machine learning and deep learning. The tutorial also briefly explained back-propagation, the engine of neural networks. Finally, it provided a basic overview of IML stressing their need and importance towards understanding how model makes a judgement about a particular outcome. Please read part1-tutorial for more details. Part2 of the series, then provided insights on xAI and IML taking into consideration safe-critical application domains such as medicine, finance and security where deployment of ML or AI requires satisfaction of certain criteria (such as fairness, trustworthiness, reliability etc). The tutorial explained the need for interpretability on today’s state-of-the-art ML models that offer impressive results as governed by a single evaluation metric (e.g., accuracy). Bhusan Chettri considered: Wild-life monitoring and automated tuberculosis detector as two use cases to elaborate the need of xAI in detail. Furthermore, how dataset biases can impact adoption of machine learning models in real-world scenarios and how crucial is understanding training data were discussed in the tutorial. Please read part2 tutorial  for details. 

 

Interpretability methods

This tutorial is focused on explaining different interpretability methods for understanding the behavior of machine learning models. There have been tremendous research work on IML and researchers have proposed several methods to explain the working phenomenon of ML models. Different taxonomies of IML methods can also be found in the literature but with a lack in the consistency of taxonomies. Thus, for simplicity this tutorial summarize IML methods in two broad categories. The first involves designing ML models that are implicitly interpretable. These class of models are simple models such as decision trees which in itself is easy to interpret. The second method includes attempting to understand what a pre-trained model has learned from the underlying data to form a particular outcome or decision. This is called post-hoc analysis that takes a pre-trained model which is often black-box in nature, for example deep neural networks.

Towards designing interpretable models

 

In this approach, researchers aim to build solution to a given problem using ML models that do not require use of any post-hoc analysis once the model is trained, rather it focuses on building models in such a way that they are easy to interpret in themselves. Although these form of methods offer a good degree of explanability, which is encoded into the model itself, they often suffer in terms of performance due to the underlying simplicity of the model architecture that often fails to learn the underlying complex data distribution. This off-course depends and varies across different problem domains. Nonetheless, they are easy to understand which is a key to many safety-critical application domains, for example finance and medicine. During model training these form of models are conditioned to satisfy certain criterion in order to maintain interpretability. These conditions (for example sparsity) may take different forms depending upon the nature of the problem. They are often referred as white-boxes, intrinsic explainable models or transparent boxes. To derive an understanding of their working phenomenon, one can inspect different model components directly. For example, inspecting the different nodes visited from the root to the leaf node in a decision tree. Such analysis provides enough insights about why and how a model made a certain decision.

 

Approach 1: Rule-based models 

 

The first category of methods aim at applying a predefined set of rules that are often mutually exclusive or dependent while training the models. One well known example of such model class is decision tree model which comprises set of if-else rules. Because of simplicity of if-else rules it becomes very easier to get an idea of how the model is forming a particular prediction. Researchers have proposed an extension to decision tree which is called as decision lists that comprises of an ordered set of if-then-else statements and these models take a decision whenever a particular rule holds true.

  

Approach 2: Case-based reasoning and prototype selection

 

In this approach, prototype selection and case-based reasoning are applied towards designing interpretable ML models. Here, prototype can mean different for various application and therefore it is application specific. For example, an average of N training examples from a particular class in the training dataset can be regarded as a prototype. Once trained, such model perform inference (or prediction) by computing the similarity of a test example with every element in the prototype set. Unsupervised clustering followed by prototype and subspaces learning have been performed by researchers to learn an interpretable Bayesian case model where each subspace is defined as a subset of features characterizing a prototype. Learning such prototypes and low-dimensional subspaces helps promote interpretability and generating explanations from the learned model.

Approach 3: towards building inherently interpretable models 

 

In this approach, researchers aim at developing training algorithms and often defines dedicated model architecture in a way to bring interpretability in black-box machine learning models (especially the deep learning based). In that direction, one common and quite popular method used in the literature to promote interpretability is through use of attention algorithms during model training. Through such attention mechanism one can encode some degree of explainability in the training process itself. In other words, it provides a way to weigh feature components in the input (that can be eventually visualized) to understand what part of the input is being utilized most heavily by the model in forming a particular prediction in contrast to other feature components. On the other note, researchers have also encapsulated a special layer within the deep neural network (DNN) architecture to train the model in an interpretable way for different machine learning tasks. The output from such a layer that provides different information (for example different parts of input) can later be utilized during inference time for explaining or understanding different class category.

Furthermore, use of some training tricks such as network regularization has also been performed in the literature to make convolutional neural network models more interpretable. Such a regularization guides the training algorithm in learning disentangled representation from the input which eventually helps model learn the weights (i.e the filters) that eventually learns more meaningful features. Some other line of work can be found where self-explainable DNNs have been proposed. This model architecture comprises of a encoder module, a parameterizer module and an aggregation function module.

It is to be noted, however, that the design of interpretable models is not favorable under every situations. While it is true that they provide inherent explainability due to their design choices but there are limitations or challenges with this approach. One challenge is the use of input features. What if the input features used in itself is hard for humans to understand? For example, Mel Frequency Cepstral Coefficients is one of the state-of-the-art features used in automatic speech recognition systems, and is not easily interpretable. This implies that the obtained explanations from the trained interpretable model would lack interpretability because of the choice of input features. Thus, as highlighted earlier, there is always a tradeoff between model complexity and model interpretability. Lower the model complexity, higher is the interpretability but lower would be model performance. In contrast, higher the model complexity lower is the interpretability (but generally offers better performance on a test dataset). In almost every domain applications (audio, video, text, images etc) high accuracy showing models are complex in nature. It is hard to achieve state-of-the-art performance on a given task using simplistic interpretable models for example a linear regression model because of its simplicity as it fails to learn the complex data distribution in the training dataset, and hence shows poor performance on a test set. Thus post-hoc methods have evolved and explored by researchers across many domains to understand what complex machine learning models are capturing from the input data to make predictions. The next section provides a brief introduction on post-hoc methods of interpretability.

 

 

Post-hoc interpretability methods 

 

This class of interpretability method works on a pre-trained machine learning model. Here, the post-hoc interpretability methods aims at investigating the behavior of a pre-trained models using specially devised algorithms to perform explainability study. This means that this class of methods do not put any conditioning with regard to interpretability during the model training. Thus the models that are being investigated to understand their behavior using post-hoc approaches are usually complex deep learning models which are black-box in nature. These methods are broadly grouped into two parts.

First class of methods aim at understanding the global or overall behavior of machine learning models (deep learning models in particular). The second class of methods focus on understanding the local behavior of the models. For example, producing explanations to understand which different features (among N set of features) contributed most to a particular prediction. It should also be noted that these post-hoc methods can be applicable to any machine learning model (so called model agnostic types) or it can be designed specifically for a particular class of machine learning models (so called model specific). The next tutorial will be discussing more on the post-hoc methods of model interpretability.

 

Relevant Links:                 ISSUU         DEEPAI      ORCID      GITHUB        QUORA

 

Explainable AI in Safe-Critical Applications by Bhusan Chettri

 

This tutorial from Bhusan Chettri provides an overview of Interpretable Machine Learning (IML) a.k.a Explainable AI (xAI) taking into account safety-critical application domains such as medicine, finance and security. This tutorial is the second installment of the interpretable AI tutorial series by Dr Bhusan Chettri a PhD graduate in AI and Voice Technology from Queen Mary University of London. The tutorial talks about the need for explanations from AI and Machine Learning (ML) models by providing two examples in order to provide a good context about the IML topic. Finally, it describes some of the important criteria that any ML/AI model in safe-critical applications must satisfy for their successful adoption in real-world setting. But, before getting deeper into this edition, it is worth revisiting briefly the first part of this tutorial series. Part1 mainly focused on providing an overview about various aspects related to AI, Machine Learning, Data, Big-Data and Interpretability. It is a well known fact that data is the driving fuel behind the success of every machine learning and AI applications. The first part described how vast amount of data are generated (and recorded) every single minute from different mediums such as online transactions, use of different sensors, video surveillance applications and social media such as Twitter, Instagram, Facebook etc. Today’s fast growing digital age that leads to generation of such massive data, commonly referred as Big Data, has been one of the key factor towards the apparent success of current AI systems across different sectors.

                                    

The tutorial also provided a brief overview of AI, Machine Learning, Deep Learning and highlighted their relationship: deep learning is a form of machine learning which involves use of artificial neural network with more than one hidden layers for solving a problem by learning patterns from training data; machine learning involves solving a given problem by discovering patterns within the training data but it does not involve use of neural networks (PS: machine learning using neural networks is simply referred as deep learning); AI is a general terminology that encompasses both machine learning and deep learning. For example, a simple chess program which involves a sequence of hard-coded if-else rules defined by a programmer can be regarded as an AI which does not involve use of data i.e there is no data-driven learning paradigm. To put it in simple terms deep learning is a subset of machine learning and machine learning is a subset of AI.

The tutorial also briefly talked about back-propagation algorithm which is the engine of neural networks and deep learning models. Finally, it provided a basic overview of IML stressing their need and importance towards understanding how model makes a judgement about a particular outcome. It also briefly discussed a Post-hoc IML framework (that takes a per-trained model to understand their behavior) showcasing an ideal scenario with a human in a loop for making final decision of whether to accept or reject the model prediction or a particular outcome. Please read part1-tutorial for more details.

The need for explainable AI: Why?

One important question that arises is why do we need interpretability or explanations from ML models at all? Well, Bhusan Chettri explains that the need arises when an AI or ML model needs to satisfy some criteria (for example trustworthiness) for its successful adoption in real-world setting or business applications. Before elaborating further, Bhusan Chettri considers following two examples to build up the context on this topic. More specifically, the examples below aim to reflect how interpretability helps discover problems caused by biases in dataset that influence model decisions which can be catastrophic in safety-critical applications.

Automatic wildlife monitoring system

Example1 – Consider an automatic system trained to detect between a dog and a wolf, a wolf monitoring system, which is to be deployed in some neighborhood. Whenever the system detects wolf it trigger’s an alarm to raise awareness to the people in that neighborhood for taking safety precautions. However, when it detects dog (for example a husky) the system just continues its normal functioning i.e keeps scanning the images around the neighborhood in real time. Now, what happens if the training data used for training such automatic system contained biases? Dataset biases can occur from a variety of sources for example during data collection, compilation and post processing. Often the underlying data collection process involved is automatic with little or no human involvement in post verification. One reason for this is due to enormous dataset sizes. Thus errors are prone to occur at various stages of data collection, for instance a wolf image mistakenly labelled as dog and vice-versa in the training set from crowd source workers hired for data annotation. Or, lets say images in training dataset for a particular class (say Dog or Wolf class) contained some cues when images were recorded?

On a similar note, as illustrated in this research article from Washington University, the researchers found that the classifier (dog vs wolf detector) had learned to detect wolf based on other contents within the image. In simple words, the model had learned to use presence/absence of snow (white patches) as a potential cue for detecting wolf. This suggests that most of the wolf pictures in training set contained white patches of snow or ice and the model had used this as a cue which is actually not relevant to the problem. To put it another way, as there is a very tiny difference between a wolf and a husky dog, ML model may have found it quite hard to exploit that tiny cue for making the decision in a right way. Hence ML models exploited a back-door as a way to provide right answers but for wrong reasons. This means if a dog image contained some snow in the background, then the model could classify it as a wolf with a high degree of confidence. Similarly, if a wolf picture is not having any snow in the background, then this model may classify it as a dog with a high probability. Now, imagine the consequence for such a false acceptance in real world scenario. The system would not trigger alarm and public may find wolf roaming freely around the streets – and the damage it may cause would be costly. This model, therefore, is not trustworthy. It is heavily biased to irrelevant cues in the training data. It is worth noting that it would be quite challenging for a normal human being to think that context in the image (snow in this case) could have been used as a potential cue for class discrimination. However, it is very possible for machines. In-fact, machines can do things that are way beyond human expectations, imaginations and understanding. Thus interpretation and reasoning behind why and how a ML model comes to a particular outcome or decision is very important to ensure trustworthiness and its reliability.

Automatic Tuberculosis detector

Example2 – Medicine: Consider an automatic Tuberculosis (TB) detector. The machine learning model has been trained to automatically detect if a person has TB or not. It has been trained using both a healthy person’s data and a person’s with TB. Let’s imagine that a large sample of data has been collected for this. Raw data collected from each person contains various information collected in a form of a questionnaire related to TB such as duration of cough, body temperature, presence of blood samples in the cough, etc. The model would simply learn to make judgement about whether a person has a TB by exploiting correlation between input features and the answers: TB or no TB. This model might show good performance on a held-out test sets also. But can such model be deployed in hospitals in real time for making automatic decisions without any human involvement? The answer is No. It is quite risky to just solely make a judgement if a person has a TB or not on this model outcome itself. Such safety-critical domain requires a certain level of trust and assurance that these models do behave as they should when deployed in real-time applications. Thus these models need to satisfy certain criteria for their safe adoption in real-time use cases. For this human-in-the-loop is often used and appears to be an important part in assessing their trustworthiness before making final decisions.

Thus, for this particular problem, IML framework can be used in the following way. First, the automatic detector generates the prediction for the new input (i.e the new patient details) by extracting relevant features just the same way as done during model training. An output value closer to 1 would indicate the presence of TB while a value closer to 0 indicates the patient is not having TB with high likelihood. A value of 1 indicates that the model is highly confident that the patient has TB while a value of 0 indicates otherwise with high certainty. Now to produce an explanation for this prediction, the obtained prediction/outcome along with the features extracted for the input instance are usually passed to the IML framework i.e the explainer in a post-hoc setting (where IML framework operates on per-trained models). The explainer then returns an explanation for the prediction of this input instance which could be a list of words highlighting important features (among all the features) that the model used for making the particular decision. For example, it may assign large weights to input features such as blood in cough, continuous coughing for more than 2 weeks. With such explanations which directly correlates to TB, it becomes easier for a doctor (human-in-the-loop) to make final decision and produce a report with high confidence whether the patient has TB or not. Without such explanations, it is difficult to make decisions based solely on models prediction.

Next, the article will discuss some of the important criteria that any AI or ML model needs to satisfy for its successful adoption in real-time usage. The article will further describe how IML can help in meeting these criterion.

Trust 

Imagine the degree of trust and faith a patient has on Doctor; the trust an individual has on the Pilot flying the plane being boarded to travel to a new destination. In a similar context, for application domain such as medicine, finance or security it is very important that humans (or users) have the similar degree of faith and trust (just the way humans do in real-word scenarios like the case of a doctor or a pilot as highlighted earlier) on the AI or ML models that shall be deployed for use in real time. Now to achieve this, one simple yet crucial step in order to promote trust to ML users is by making use of IML methods to ensure that data used to train such models are free from inherent biases or con-founders that influence model decisions. This in turn can often be achieved by performing data exploratory analysis using initially trained models and analyzing data for test instances of confidently classified data samples. With such approaches one can confirm whether models are exploiting the right features from the input data towards making decisions. Check out this related research article for more on this.

Robustness

It is very important that AI & ML models are robust against small perturbation on input features. Their behavior must not change when bad actors present manipulated input instances to the model to manipulate model decisions for their personal gain. Research has demonstrated that today’s high-performance showing AI are often easily fooled when crafted input instances imperceptible to human eyes or ears are presented to them. The systems produce completely different output class label (in case of an image classifier) or might produce a different audio (in case of Speech synthesis systems). Such carefully crafted instances are called adversarial input and the field is referred as adversarial machine learning, a very hot topic in the field of machine learning but beyond the scope of this article. Although, many different techniques to combat adversarial inputs have been proposed by ML researchers, none of them offers a full-proof solution. Therefore, lack of robustness in AI/ML models remains a pressing concern when they are deployed in safe-critical applications such as medicine, finance or autonomous vehicles. One way to tackle this issue and make ML models more robust could be to use IML framework for analyzing such adversarial examples to derive better understanding about how and why models show different behavior on such input instances. This in turn may help build robust ML models.

Reliability

The behavior of ML models must remain similar when tested across multiple instances. Often ML models fail to show good generalization or performs unreliably when they get deployed in real world setting. In other words, their behavior is often different in real-world usage in comparison to the one found in laboratory setting (where they are trained and tested). Thus reliability of ML models is also one key factor in safe-critical applications to ensure that models behave in a similar way as they behaved in laboratory conditions. One approach towards building reliable ML models could be to ensure that test dataset mimic real-world setting where those models are likely to be deployed.

Legal requirements

As per GDPR – General Data Protection Regulation – a European Union regulation, every users of automatic systems possess a right to explanation of the systems decisions. By incorporating an IML framework in building such automatic system, it becomes easier to explain the reasons behind an outcome or predictions in meaningful and human-friendly way.

Fairness

ML model must make decisions by exploiting the right cues from the training data. However, it is often very challenging to ensure that training dataset are clean and bias free. Biases may occur during data compilation, post processing or even during data collection through crowd sourcing platforms (for example Amazon Mechanical Turk). Therefore, it becomes difficult to often avoid such ML models to be unfair when the underlying data used for model training is biased already. Therefore, such ML models often behave wrongly or show discriminatory behavior all because of biased training data being used to learn model parameters. One approach towards avoiding this problem could be to not use certain features from the input data that show such biased behavior. However, it is again difficult to detect such implicit correlations within the training data before training a ML model. Thus, IML framework can be utilized in such situations to detect if the ML models under question is unfair by analyzing whether discriminating features (right cues/features) have been utilized by models in making predictions.

 

Summary

In this tutorial, Bhusan Chettri provided an insight on xAI and IML taking into consideration safe-critical application domains such as medicine, finance and security where deployment of ML or AI requires satisfaction of certain criteria (such as fairness, trustworthiness, reliability etc). To that end, Dr Bhusan Chettri who earned his PhD in Machine Learning and AI for Voice Technology from QMUL, London described why there is a need for interpretability on today’s state-of-the-art ML models that offer impressive results as governed by a single evaluation metric (for example classification accuracy). Bhusan Chettri elaborated this in detail by taking two simple use cases of AI systems: wild-life monitoring (a case of dog vs wolf detector) and automatic tuberculosis detector. He further detailed how biases in training data can affect models from being adopted in real-world scenarios and that understanding training data and performing initial data exploratory analysis is equally crucial so as to ensure models behave reliably in the end during deployment. Stay tuned for more on the topics of explainable AI. The next edition of this series shall discuss different taxonomies of interpretable machine learning. Furthermore, various methods of opening black-boxes: towards explaining behavior of ML models shall be described.

 

Bhusan Chettri describes AI and its need for explainability

 

 

This tutorial from Bhusan Chettri is focused on providing an in-depth understanding of interpretable machine learning (IML) taking into consideration different use-cases, various application domains and emphasizing why it is important to understand how a machine learning model demonstrating impressive results make their decisions. The tutorial also discusses if such impressive results are trustworthy to be adopted by humans for use in various safety-critical businesses for example: medicine, finance and security. Dr Bhusan Chettri who earned his PhD from Queen Mary University of London aims at providing an overview of Machine Learning and AI interpretability. Now, before diving into the topic of IML or explainable AI (xAI), it is worth revisiting some of the vital concepts revolving around data, Machine Learning, AI and how these systems function. With a good grasp of such concepts it becomes easier for readers from non-technical or non CS background to gain a better insight about the underlying phenomenon related to model interpretability. For advanced technical readers (CS or AI background) these may be too basic.

  

Data and Big data

Data is the driving fuel behind the success of every data-driven AI and Machine Learning (ML) based application. Do you agree? Okay, so what is data? Anything that exists in nature; represents some facts or figures is regarded as data. An image of a dog; numbers between 1 to 10 are all examples of data. Data is often available in raw format and they need to be corrected before using them in a Data Science or ML pipeline. For example, errors can occur during data collection and processing such as labeling of a data sample to a certain class (image of a Cat being labelled as Dog or an image of a Horse labelled as Cat) or recording errors made by an automatic device (reading taken from a faulty thermometer under different weather conditions). Furthermore, in the current era of “Big Data” where data is produced and collected on a massive scale world-wide, there is an enormous amount of data available digitally online. To summarize, some of the sources of data are: retail and wholesale transactions; data collected from the use of various sensors; data collected through video surveillance; population census data; social media such as Facebook, Twitter, YouTube and blogging etc. A post published in 2018 at Forbes.com (here) regarding how much data is produced on the internet through various mediums (like social media platforms, online transactions etc) revealed interesting or rather shocking reports. For instance, as per their analysis, in just a minute time interval snap chat users were found to share 527,760 photos, Instagram users were found to post around 46,740 photos. If these numbers of image data are uploaded in a single minute, then just imagine how many photos are produced/uploaded in a single day, a month, and a year time? 

Although there is a massive amount of digital data available, they are not in a structured form and therefore hinders using them directly in a downstream AI pipeline. But, what does it mean by structured and unstructured data? Data prepared and arranged in some particular format, for example multidimensional arrays; tabular or spreadsheet-like data where each column holds values of different types (numeric, string etc), CSV (comma separated value) files are all examples of structured data. On the contrary, contents of web pages, documents, multimedia data etc are unstructured as they need some form of processing and cannot be used directly. Even though it may not always be obvious, a large percentage of such data can be transformed into a format that is more suitable for analysis and modelling. For instance, a collection of news articles could be processed into a word frequency table which could then be used to perform sentiment analysis. Big Data, therefore, is often not immediately useful. It needs to be examined, aggregated, to make sense of the endless stream of bytes. Thus engineers/researchers often spend a considerable amount of time performing initial data exploration (data cleansing, aggregation etc) before being able to use them in an AI/ML pipeline.

 

Artificial Intelligence (AI), Machine Learning (ML) and Deep Learning

AI can be defined as a general field which involves rules, logic and data (to some extent) and has evolved over time encompassing both ML and deep learning as its sub-fields. Bhusan Chettri explained in other words a simple computer program consisting of several if-else decision making rules can be regarded as an AI where there is no data-driven learning involved. It consists of a large set of rules transforming inputs to a corresponding decision (or the output). A classic example of this is chess programs in early days (before deep learning kicked off) which had only a set of hard-coded rules defined by a designer or programmer and this does not qualify as machine learning as there is no learning involved. However, ML and deep learning are completely data-driven learning and make massive use of data. They do not use hand-crafted rules to define an answer or outcome to a problem being aimed to solve. During the period between 1950 - 1980, which was often called as an era of symbolic AI there was a belief among researchers/scientists that human-level AI was achievable by making use of a large set of explicit carefully handcrafted rules for manipulating and representing knowledge base. Interestingly, the symbolic AI methodology worked quite well for tasks such as playing chess - which were of course very simple but these approaches failed dramatically for other complex problems such as speech recognition, image classification, language translation, language recognition etc. Thus, this aspect led to the invention or rise of what we call Machine Learning - a fully data driven mechanism that aims at learning patterns within the data to discover correlation between the data and the answer (or label of data).

In traditional AI (also called symbolic AI) which is based on classical programming the input rules (set of instructions to a computer) and data is provided as input to such a system. These predefined rules are applied on the data to produce answers. On the contrary, users provide both data and expected outcome/answers from data to produce the rules as output. A Machine Learning system does this by exploiting patterns within the data and associated class/answers. Then these rules can be then applied to new data to produce answers. A Machine learning system is trained unlike symbolic AI which is explicitly programmed. It is presented with many examples relevant to the task, and it finds statistical structure in these examples that eventually allows the system to come up with rules for automating the task. For example, in order to build a voice authentication system using machine learning, a lot of speech samples spoken from multiple speakers are collected to train the system. The system is trained to learn a pattern that differentiates one speaker from another while making less errors in detecting true speakers.

Deep learning is a sub-field of Machine Learning which is also data-driven but makes use of neural networks in solving a problem. In simple words, deep learning refers to a neural network with more than one hidden layer for learning representations. The learning process primarily aims to mimic human brains in performing various tasks through a series of non-linear transformations of input data to achieve the desired output. Usually, the first layer of the neural network is called the input layer that encodes the input data and the last layer of the network is referred to as the output layer which comprises a set of units (or neurons) representing the desired outputs. The remaining layers in the network are often referred to as hidden layers. As humans or programmers do not have direct control on these layers, deep neural network models are often called as black box models. For example, consider the design of an automatic gender classification system using deep neural networks. The input layer will have N (this will depend upon the size of actual input features used by the programmer) number of neurons that will take various attributes of person’s as input. These attributes might include voice pitch (also called fundamental frequency), body weight, height etc. Here, the output layer will have two neurons (or output units) where one neuron denotes Male class and the other Female class. The network can have M number of hidden layers (which again depends upon the programmer/engineer/researcher’s design choice) which will take the input data and apply a series of non-linear transformations in order to learn the mapping function from input to output.

During the network training the algorithm responsible for providing such capability to learn in an iterative fashion is called back-propagation. In-fact, back-propagation is actually the driving engine of every neural network that undergoes a continual learning process by adjusting the parameters often called weights of the neural network. In layman terms these weights can be thought of as a bunch of numbers organized in a table or matrix form where every number has a special purpose or meaning. There will be one such matrix learned for each of the hidden layers of the network so that the learned transformation function is correctly mapping every input to its correct class.

There are two terms associated with the use of artificial neural networks: shallow learning and deep learning. So what is the difference between the two? Shallow learning simply refers to the use of artificial neural networks with a single hidden layer. On the contrary, deep learning basically refers to the use of more than one hidden layer in learning the transformation function towards solving a given problem. The ability to perform large matrix multiplication between input data and the weight matrices during the learning process (training phase) has been possible today because of high-end computing infrastructure and advancement in algorithms. Thus deep learning is more popular in comparison to shallow learning nowadays.

 

Understanding predictions of ML models: Explainable AI (xAI)

Interpretable machine learning, often referred to as explainable AI (xAI), is a field of machine learning aimed at understanding how a model makes a decision of an outcome. Is the model exploiting cues from the underlying data that is relevant to the problem? What parts of input are more contributing towards a prediction? These are some of the questions that IML or xAI aim to address. And, there has been significant demand and interest in research on interpretability in recent years by academics and tech companies such as Meta, Google, IBM, Microsoft and more around the globe.

Fig 1: A basic overview of an Interpretable Machine Learning framework where humans verify the explanations to make final decisions by Bhusan Chettri.

 

Figure 1 by Bhusan Chettri illustrates an interpretable machine learning (IML) framework. The framework showcases an ideal scenario where a human is involved in order to verify the explanations provided by an explainer a.k.a IML algorithm. Here the IML algorithm shown is a post-hoc interpretability method which operates on per-trained ML models. This means that once a ML model has been trained, an IML algorithm can be applied on it. For this, the framework takes both the feature extractor and the ML model ensuring that features are extracted in a similar way as done during training and then prediction is obtained for an input test example. This process is illustrated in the figure. The output from an explainer is an explanation which could be, for example, a list of words highlighting the significance of various words in a given input sentence that was highly contributing towards that particular prediction.

The rise of Artificial Intelligence which can be attributed to the successful use of deep learning algorithms, in particular, has brought a revolution in almost every domain and business showing impressive results. The two main factors contributing to this success are: (1) Big data and, (2) Computational resources. As explained earlier in the post, every business is now able to collect a vast amount of data through digitization of their business process, which is used in training their AI systems. Furthermore, with the advancement in Computational hardware, for example, availability of Graphical Processing Units (GPUs) have enabled high-demanding mathematical computations much faster thus allowing researchers to run experiments on large datasets. However, several safety-critical applications cannot simply use them without understanding how it forms decisions under the hood. For example, domains such as finance, medicine/healthcare and security (to name a few) require sufficient explanations to ensure fairness, reliability and trustworthiness of decisions they make. These models are black boxes. All that is known about them is that it takes certain input data and produces the output. Bhusan further say's Yes, in many applications explanations may not be that important and all that matters would be just numbers - good results or say accuracy. Such businesses don’t care how their model arrived at such a decision because the result is pretty good and most importantly their client is happy. So all sorted. However, this is not true with every domain as discussed earlier. There is a danger in using such black boxes without understanding their working mechanism. Also, research has shown that deep learning models can be fooled very easily by just making a tiny perturbation to its input data (which to humans is undetectable), yet the machine learning system produces completely different results. This is very dangerous. This means that an attacker can easily manipulate the input data passed to a neural network to produce the output they desire. Chettri gives an example, imagine a scenario where a Text-to-Speech system takes the input text “I will call you tonight” but the system gets manipulated by an attacker to produce the speech that sounds “I will kill you tonight”. This field, not within the scope of today’s discussion however, is called Adversarial Machine Learning, a very hot topic in the field of AI that is actively being studied by researchers around the globe.

Machine learning and deep learning models are heavily data-driven. This means that they learn to perform task discovering/learning patterns within the training data fed to the learning algorithm. Thus, the quality of data is one important factor that needs to be considered before using them in building ML models. The training data must be balanced across different classes of interest. For example, while building a gender classification system if the training data across two classes is imbalanced then there is a high probability that the model will pickup this imbalance factor and favor the gender class having more training samples (which is quite obvious). Bhusan Chettri gives another example to illustrate such imbalance data bias could be training an image classifier to recognize ten different classes of animals using per-captured images with following data distribution per class: 80% of training samples contain images of Cat; 5% samples are of Horses; and remaining 15% constitutes other 8 classes. What do you think would such a model learn? Do you think such a model would perform fairly in real-world applications if deployed? The answer is No. This model no doubt may show 100% detection accuracy for Cat images but shall perform miserably for other categories of images. Due to an imbalance in training data across different classes, the model in question is heavily biased to detect one class while showing poor performance in other classes. This model is not trustworthy to be deployed in the real world. Likewise, training data must not have any other kind of biases resulting from initial data collection and post processing (it often happens while using automatic pipelines for data collection). Thus data used must be free of biases to ensure that the final model is reliable and unbiased. Drop your questions.