Use PyTorch Deep Learning Models with scikit-learn

by

The leading deep learning libraries in Python for research and development are TensorFlow/Keras and PyTorch, largely due to their user-friendly design. Conversely, Scikit-learn remains the most popular library for general machine learning tasks in Python. This post will demonstrate how to integrate deep learning models from PyTorch with the Scikit-learn library, enabling you to harness Scikit-learn’s capabilities for model evaluation and hyperparameter optimization. After completing this lesson, you will understand:

  • How to wrap a PyTorch model for use with the Scikit-learn library.
  • How to evaluate PyTorch models using cross-validation in Scikit-learn.
  • How to tune hyperparameters of PyTorch models using grid search in Scikit-learn.

Let’s Get Started

Overview

This article is divided into four sections:

  1. Introduction to skorch
  2. Evaluating Deep Learning Models with Cross-Validation
  3. Implementing k-Fold Cross-Validation with Scikit-learn
  4. Performing Grid Search on Deep Learning Model Parameters

Introduction to skorch

While PyTorch excels as a deep learning library, its primary focus is not general-purpose machine learning. PyTorch’s design prioritizes minimalism, concentrating on the essentials needed to define and build deep learning models quickly. On the other hand, Scikit-learn is built on the SciPy ecosystem for efficient numerical computations and offers extensive functionality for various machine learning tasks.

Key benefits of combining Scikit-learn with PyTorch include:

  • Model evaluation using resampling methods such as k-fold cross-validation.
  • Efficient search and evaluation of hyperparameters.
  • Integration of multiple steps within a machine learning pipeline.

Since PyTorch and Scikit-learn are inherently different, direct compatibility is not feasible. However, the versatility of Python allows for easy adaptation of PyTorch models for use with Scikit-learn, thanks to the skorch module. With skorch, you can utilize your PyTorch model as a Scikit-learn model, allowing for smoother integration.

In the sections that follow, you will work with the NeuralNetClassifier wrapper to create a classification neural network using the Sonar dataset, an accessible dataset consisting entirely of numerical attributes.

Before beginning, ensure you have installed PyTorch, skorch, and Scikit-learn. Use the following command:

pip install torch skorch scikit-learn

Evaluating Deep Learning Models with Cross-Validation

The NeuralNet classes in skorch, such as NeuralNetClassifier, NeuralNetBinaryClassifier, and NeuralNetRegressor, act as factory wrappers for your PyTorch models. These classes facilitate easy model creation by allowing you to specify your model class while managing the training loop and other boilerplate tasks.

Here’s an example of training a binary classifier on the Sonar dataset:

import pandas as pd
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.preprocessing import LabelEncoder
from skorch import NeuralNetBinaryClassifier

# Load data
data = pd.read_csv("sonar.csv", header=None)
X = data.iloc[:, 0:60]
y = data.iloc[:, 60]

# Binary encoding of labels
encoder = LabelEncoder()
encoder.fit(y)
y = encoder.transform(y)

# Convert to PyTorch tensors
X = torch.tensor(X.values, dtype=torch.float32)
y = torch.tensor(y, dtype=torch.float32)

# Define the model
class SonarClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(60, 60)
        self.act1 = nn.ReLU()
        self.layer2 = nn.Linear(60, 60)
        self.act2 = nn.ReLU()
        self.layer3 = nn.Linear(60, 60)
        self.act3 = nn.ReLU()
        self.output = nn.Linear(60, 1)

    def forward(self, x):
        x = self.act1(self.layer1(x))
        x = self.act2(self.layer2(x))
        x = self.act3(self.layer3(x))
        x = self.output(x)
        return x

# Create the skorch wrapper
model = NeuralNetBinaryClassifier(
    SonarClassifier,
    criterion=torch.nn.BCEWithLogitsLoss,
    optimizer=torch.optim.Adam,
    lr=0.0001,
    max_epochs=150,
    batch_size=10
)

# Train the model
model.fit(X, y)

In this example, we use torch.nn.BCEWithLogitsLoss as the loss function, which effectively incorporates the sigmoid function and binary cross-entropy loss, ensuring greater numerical stability by avoiding the need for a separate sigmoid activation at the output.

After running the model, you will see progress metrics for each epoch, including loss and accuracy.

Running k-Fold Cross-validation with Scikit-learn

Utilizing the PyTorch model wrapper significantly reduces the boilerplate code required for building custom training loops. The extensive functionalities of Scikit-learn, such as k-fold cross-validation, further enhance productivity.

To perform k-fold cross-validation, you can use Scikit-learn’s built-in functions:

from sklearn.model_selection import StratifiedKFold, cross_val_score

# Create the model
model = NeuralNetBinaryClassifier(
    SonarClassifier,
    criterion=torch.nn.BCEWithLogitsLoss,
    optimizer=torch.optim.Adam,
    lr=0.0001,
    max_epochs=150,
    batch_size=10,
    verbose=False
)

# Set up k-fold cross-validation
kfold = StratifiedKFold(n_splits=5, shuffle=True)
results = cross_val_score(model, X, y, cv=kfold)
print(results)

The default configuration provides validation scores, and you can assess performance with the mean and standard deviation:

print("mean = %.3f; std = %.3f" % (results.mean(), results.std()))

This measurement provides insights into model consistency across different test sets.

Grid Search Deep Learning Model Parameters

Now that you’re familiar with using PyTorch models in Scikit-learn, let’s take it a step further by adjusting hyperparameters through grid search.

With the skorch wrapper, you can customize the NeuralNetBinaryClassifier by passing various parameters to optimize model performance. Here’s how to define the model and include a parameter for the number of hidden layers:

class SonarClassifier(nn.Module):
    def __init__(self, n_layers=3):
        super().__init__()
        self.layers = nn.ModuleList()
        for _ in range(n_layers):
            self.layers.append(nn.Linear(60, 60))
        self.output = nn.Linear(60, 1)

    def forward(self, x):
        for layer in self.layers:
            x = nn.ReLU()(layer(x))
        return self.output(x)

# Create the skorch model
model = NeuralNetBinaryClassifier(
    SonarClassifier,
    criterion=torch.nn.BCEWithLogitsLoss,
    optimizer=torch.optim.Adam,
    lr=0.0001,
    max_epochs=150,
    batch_size=10,
    verbose=False
)

# Define the grid search parameters
param_grid = {
    'module__n_layers': [1, 3, 5],
    'lr': [0.1, 0.01, 0.001, 0.0001],
    'max_epochs': [100, 150],
}

grid_search = GridSearchCV(model, param_grid, scoring='accuracy', verbose=1, cv=3)
result = grid_search.fit(X, y)

# Summarize results
print("Best: %f using %s" % (result.best_score_, result.best_params_))
means = result.cv_results_['mean_test_score']
stds = result.cv_results_['std_test_score']
params = result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))

Putting It All Together

This complete code snippet demonstrates how to leverage PyTorch models within Scikit-learn effectively:

import numpy as np
import pandas as pd
import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.model_selection import GridSearchCV, StratifiedKFold, cross_val_score
from sklearn.preprocessing import LabelEncoder
from skorch import NeuralNetBinaryClassifier

# Load data
data = pd.read_csv("sonar.csv", header=None)
X = data.iloc[:, 0:60]
y = data.iloc[:, 60]

# Binary encoding of labels
encoder = LabelEncoder()
encoder.fit(y)
y = encoder.transform(y)

# Convert to PyTorch tensors
X = torch.tensor(X.values, dtype=torch.float32)
y = torch.tensor(y, dtype=torch.float32).reshape(-1, 1)

class SonarClassifier(nn.Module):
    def __init__(self, n_layers=3):
        super().__init__()
        self.layers = nn.ModuleList()
        for _ in range(n_layers):
            self.layers.append(nn.Linear(60, 60))
        self.output = nn.Linear(60, 1)

    def forward(self, x):
        for layer in self.layers:
            x = nn.ReLU()(layer(x))
        return x

model = NeuralNetBinaryClassifier(
    SonarClassifier,
    criterion=torch.nn.BCEWithLogitsLoss,
    optimizer=torch.optim.Adam,
    lr=0.0001,
    max_epochs=150,
    batch_size=10,
    verbose=False
)

param_grid = {
    'module__n_layers': [1, 3, 5],
    'lr': [0.1, 0.01, 0.001, 0.0001],
    'max_epochs': [100, 150],
}

grid_search = GridSearchCV(model, param_grid, scoring='accuracy', verbose=1, cv=3)
result = grid_search.fit(X, y)

print("Best: %f using %s" % (result.best_score_, result.best_params_))

Further Reading

For additional resources on deep learning with PyTorch and its integration with Scikit-learn, check out:

Summary

In this chapter, you learned how to wrap your PyTorch deep learning models for use in Scikit-learn. Specifically, you discovered:

  • How to adapt PyTorch models for the Scikit-learn environment.
  • The procedures for evaluating model performance using cross-validation.
  • The intricacies of hyperparameter tuning with grid search techniques.

By utilizing Scikit-learn’s capabilities, you can significantly enhance the standard machine learning workflow by integrating deep learning models, simplifying processes like evaluation and hyperparameter optimization.

This rewritten version captures the essential content and structure of the original article while improving clarity and readability.

Leave a Comment