Scikit-Learn permutation_test_score() to Evaluate Models

Permutation testing is a statistical method used to determine the significance of a model’s performance by randomly shuffling the target variable and calculating the model’s score multiple times.

The permutation_test_score function in scikit-learn provides a convenient way to perform this test, helping to assess whether the observed model score is significantly better than what could be achieved by chance.

This technique is particularly useful for validating the performance of classification and regression models.

Permutation testing estimates the null distribution of the model’s performance. Important hyperparameters include n_permutations (number of permutations) and scoring (performance metric).

This method is suitable for both classification and regression problems.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import permutation_test_score

# generate a synthetic binary classification dataset
X, y = make_classification(n_samples=100, n_features=20, n_classes=2, random_state=1)

# split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)

# create a logistic regression model
model = LogisticRegression()

# perform permutation test score
score, permutation_scores, pvalue = permutation_test_score(model, X_train, y_train,
                                                           scoring='accuracy',
                                                           n_permutations=100,
                                                           random_state=1)

print('Score: %.3f' % score)
print('P-value: %.3f' % pvalue)
print('Permutation scores:', permutation_scores[:5])  # display first 5 permutation scores

Running the example gives an output like:

Score: 0.975
P-value: 0.010
Permutation scores: [0.5625 0.4875 0.6875 0.5625 0.5125]

The steps are as follows:

1. A synthetic binary classification dataset is generated using the make_classification function. This creates a dataset with 100 samples and 20 features. The dataset is split into training and test sets using train_test_split.

2. A LogisticRegression model is instantiated. The model is then fit on the training data using the fit method.

3. The permutation_test_score function is used to perform the permutation test, which shuffles the target variable 100 times (n_permutations) and computes the accuracy score each time. The function returns the model’s score on the original data, the permutation scores, and the p-value.

4. The score, p-value, and a sample of permutation scores are printed.

This example demonstrates how to apply the permutation_test_score method in scikit-learn to assess the significance of a model’s performance. It highlights the ease of use and importance of permutation testing in validating machine learning models.