Most machine learning practitioners start experimenting with the established scikit-learn library, but there's an easier and more approachable alternative, named PyCaret.

This library has many advantages compared to scikit-learn, especially for people with limited experience. This tutorial will provide an overview of the main features of PyCaret, as well as a case study focusing on regression. I suggest installing the latest version of Anaconda on Windows/macOS/Linux to follow this tutorial, but it is also compatible with Google Colab. You can either execute the code in a Jupyter notebook or use your preferred IDE.

First of all, we have to install PyCaret by running the following command in an Anaconda terminal:

PyCaret should then be installed on your machine and any other required dependencies that might be missing.

Regression is a basic supervised machine learning task which estimates the relationship between a dependent variable $y$ (known as the target) and independent variables (known as features).

Regression can be used to predict continuous values such as the value of a house instead of classification, which is used for discrete values known as classes. The PyCaret regression module, which uses sklearn under the hood, lets you create and test regression models with a few lines of code. It includes a variety of algorithms, as well as the ability to plot and do hyperparameter tuning.

We are now going to examine a regression case study based on that module.

The basis of every machine learning project is the acquisition or creation of an appropriate dataset. PyCaret includes a variety of example datasets for different kinds of machine learning tasks, and in this project we will use the medical insurance dataset.

This dataset originates from the book Machine Learning with R by Brett Lantz, and contains health insurance information. The target variable, $y$, represents the insurance charges for each person, and the features are properties, such as age, sex, and Body Mass Index (BMI).

Real-world data is rarely that simple, but working with toy datasets helps us understand the concepts and methodology before moving on to more complex cases.

Here is a description for each dataset variable:

• age: age of the primary beneficiary
• sex: insurance contractor gender - female, male
• bmi: Body mass index, providing an understanding of body weights that are relatively high or low relative to height. An objective index of body weight ($kg / m ^ 2$) using the ratio of height to weight, ideally 18.5 to 24.9
• children: Number of children covered by health insurance / Number of dependents
• smoker: Smoking
• region: the beneficiary's residential area in the US, northeast, southeast, southwest, northwest.
• charges: Individual medical costs billed by health insurance

To get the data, we'll use the get_data function from pycaret:

The get_data function returns a pandas dataframe, so we can use the info() function from pandas to get some details about the dataset.

As we can see, there are 1338 records, and zero null values. Most real-world datasets have some null values and may require some feature engineering, but in this case we don't have to deal with that.

After loading the dataset, we will normally need to examine and understand its basic properties. This is known as Exploratory Data Analysis, and can be accomplished with various tools and methods, such as plotting.

We start by plotting the histograms of the numerical variables.

Here, we're using the built-in hist() function from pandas to plot a histogram for age, BMI, and charges. This helps us better understand the distribution of values for these numerical variables.

The BMI variable has a distribution close to normal, while the charges variable is right-skewed. Skewed distributions can be a problem for machine learning algorithms, so we will deal with that later.

Now, we'll get a bit creative by plotting the histogram of the target variable — i.e., the insurance charges — with stacked bars that represent different categories of the categorical variables. We accomplish this by using the histplot() function of the seaborn library:

Smokers have significantly higher charges, and we can see that men have higher medical costs more often than women.

Now that we've gleaned some useful insight from EDA, let's begin the PyCaret process for regression on this data.

The setup() function of PyCaret initializes the environment and prepares the machine learning modeling data and deployment. There are two necessary parameters, a dataset, and the target variable. After executing the function, each feature's type is inferred, and several pre-processing tasks are performed on the data.

After running the setup() function on our data, the results display the pre-processing pipeline applied to the dataset. Some highlights of this pipeline are:

1. Inferred data types

We can see that four features have been correctly identified as categorical, and the rest as numerical. In case PyCaret fails to do that correctly, we can define them in the setup() function ourselves, using the categorical_features and numeric_features parameters.

2. Train/Test Split The dataset has been split into a train and test set, as it is standard practice in machine learning. The train set size has been set to 80% of the original dataset, meaning that 80% of the data will be used to train the machine learning model and the rest for testing its accuracy.

3. Normalization of Numerical Features Many regression algorithms that require the features to be normalized for them to work as expected. Normalized features have $μ = 0$ and $σ = 1$. The standard method to accomplish that is to replace each value with its associated z-score, which is defined as $z = \frac{x-μ}{σ}$ .

4. One-Hot Encoding of Categorical Features Some machine learning algorithms that accept categorical features and some that don't, so it is best to convert them to numerical features using one-hot encoding. One-hot encoding removes the categorical features and replaces them with additional binary variables, one for each category, minus one (to avoid the dummy variable trap).

5. Target Transformation As we've noticed in the EDA section, the target variable is right-skewed. This could cause problems as many regression algorithms expect the data to have a normal distribution to perform optimally. The setup() function includes the option to transform the target to have a distribution close to normal. Transformations can also be applied to the features if needed, but it was unnecessary in this case.

There are various other advanced parameters in the setup() function, so if you're curious, feel free to check out the relevant section of their docs that goes over each piece in detail.

The get_config('X') function returns the features dataset after the pre-processing pipeline has been applied to it:

We can see that the numerical features have been normalized with the z-score method, and the categorical features have been encoded with one-hot encoding. It is important to verify that the pre-processing has been completed successfully, as in some cases, our dataset might not be as clean as the one used in this example. In case the pre-processing pipeline fails, we may get incorrect and unexpected results from the machine learning models.

There are numerous regression algorithms available, and it is not always obvious which one is optimal for our dataset. The only way to find the best model is to test a number of them and compare the results. Fortunately, PyCaret provides the compare_models() function, which compares a variety of different models easily:

After running the compare_models() function, the results are displayed. This table may seem intimidating, but it's actually fairly simple to understand. The first column contains each model's name, and the rest of the columns are various metrics.

You can focus on RMSE for now, which stands for Root Mean Squared Error. RMSE is a widely used metric for regression, and it is defined as the square root of the averaged squared difference between the actual value and the one predicted by the model:

$RMSE = \sqrt{ \frac{1}{N}\sum_{i=1}^{N} ( x_{i} - \hat{x_{i}} )^2 }$

The lower the RMSE value, the more accurate our model is. In this case, the best model is the Gradient Boosting Regressor model, with an RMSE value of 4368.4047.

The create_model() function lets you create a regression model based on the algorithm of your preference. In this case, we'll use Gradient Boosting Regressor since it had the best performance from compare_models() above.

The create_model() function uses k-fold cross-validation to evaluate the model accuracy. In this method, the dataset is first partitioned into $k$ subsamples, one subsample is retained for validation, and the rest is used to train the model. This process is repeated $k$ times, and each subsample is used only once as validation data.

After training the model, the cross-validation results are displayed. We set folds ($k$) to 10, so in this case, we have a ten-fold cross-validation. We can see the metrics for every fold, and the mean and standard deviation of all steps.

If you've used sklearn before, you'll notice that one line of code with PyCaret is equivalent to several lines with sklearn.

The tune_model() function tunes the hyperparameters of a given model and outputs the results. Hyperparameters are model settings that can be modified and can have either a positive or negative effect in their accuracy.

tune_model() uses the Random Grid Search method to tune and optimize the model by testing a random sample of the hyperparameters. We can define a grid with specific values for the hyperparameters by using the custom_grid parameter.

We can also define the number of iterations with the n_iter parameter. A random value from the defined grid of hyperparameters is selected for every iteration and tested using k-fold cross-validation.

As we can see from the cross-validation results, the hyperparameter tuning slightly increased the model's accuracy. The improvement is small, but experimenting with a higher iteration number or a grid with different hyperparameter values may lead to better results.

PyCaret includes a plot_model() function that lets us visualize our model's accuracy and other properties. The function includes a variety of plots that help us evaluate and understand our model better. Compared to the underlying libraries used to generate these plots — sklearn, pandas, and matplotlib — using PyCaret is significantly quicker and simpler to work with.

First, we'll plot the error of the predictions on the test set:

Second, we'll plot the importance of each feature:

In the EDA section above, we saw that being a smoker leads to significantly higher insurance charges, and now from the feature importance chart we see that being a smoker has the highest predictive value. Furthermore, we can also see that age and BMI seem to play an important role as well.

Every real-world machine learning project's ultimate goal is to make predictions on new data, where the target variable is unknown. You can accomplish that by using the predict_model() function, which returns a pandas dataframe with predictions.

We are going to create a small synthetic dataset and test our model and see how it predicts insurance charges:

We can see that young non-smokers with a low BMI are predicted to have the lowest charges by our model. On the other hand, those who are older, obese, and smoke are predicted to be charged ten times as much. Those results are in line with the EDA and the feature importance plot.

The ability to interpret a machine learning model's results allows you to avoid relying on a "black box model," where you don't understand how it exactly works.

PyCaret includes the interpret_model() function that provides an interpretation plot for a given model. This function requires the SHAP (SHapley Additive exPlanations) library to work, so we'll have to install it first.

After installing the SHAP library, we can create an interpretation plot for our model. The Gradient Boosting Regressor isn't supported by the interpret_model() function, so we will create another model based on the XGBoost algorithm and interpret that model instead.

To interpret the model, we'll use the "reason" plot type:

Above the plot, you'll notice the "base value," which is defined as the mean predicted target, and f(x), which is the prediction for a selected observation. The red-colored features increased the predicted value, while the blue-colored features decreased it.

The size of each feature indicates the impact it has on the model. In this case, not being a smoker and having zero children had a positive effect, and as a result, decreased the predicted insurance charges below the mean value.

We have pre-processed our data, compared a variety of regression models, and tuned the model of our preference, all in a few lines of code. Using scikit-learn for regression is, of course, an option, but the time and effort required are significantly higher. PyCaret lets us create machine learning models quickly and easily, making it an ideal choice for beginners. Furthermore, PyCaret can also be used by experienced data scientists who want to reduce the time needed to complete machine learning projects.

There's many other machine learning tasks you can accomplish with PyCaret, so definitely check out their docs.