How to Use PolynomialFeatures in Scikit-Learn

Polynomial regression is an extension of linear regression that allows for modeling non-linear relationships by introducing polynomial terms of the input features. Scikit-learn’s PolynomialFeatures class enables the transformation of input features into higher-degree polynomial terms, making it possible to fit non-linear patterns in data using linear models.

This article provides a detailed guide on how to use PolynomialFeatures in scikit-learn, including fundamental concepts, implementation steps, hyperparameter tuning, and practical examples.

Understanding Polynomial Features

What Are Polynomial Features?

Polynomial features are derived by raising the original features to different powers and including interaction terms. For example, if we have a single input feature x, the second-degree polynomial transformation expands it to:

For multiple features x₁ and x₂, the transformation expands as:

This expansion allows linear regression models to capture higher-order relationships between variables, improving predictive performance for non-linear datasets.

When to Use PolynomialFeatures?

• When there is non-linearity in the data, and a simple linear regression model does not fit well.
• When interactions between different input features may improve model performance.
• When you want to enhance feature representation for a linear model without switching to more complex non-linear models like decision trees or neural networks.

How to Use PolynomialFeatures in Scikit-Learn

Scikit-learn’s PolynomialFeatures class is a transformation tool that enables the expansion of input features into higher-degree polynomial terms, helping linear models capture non-linear relationships in data. This section will explore practical implementations, key considerations, and advanced techniques for working with PolynomialFeatures effectively.

Applying PolynomialFeatures for Model Improvement

The transformation process involves generating new features based on polynomial expansions and interactions. In practice, using this feature transformation can significantly impact model performance, but choosing the appropriate degree is crucial to prevent overfitting.

1. Feature Expansion Process
   • The PolynomialFeatures transformer creates new features by raising each input variable to specified polynomial degrees.
   • For example, a second-degree transformation of a dataset with two features x₁,x₂ results in:

1. Higher-degree polynomials introduce additional terms, capturing complex interactions but at the cost of increased computational complexity.
2. Using PolynomialFeatures for Regression Models
   • Polynomial regression can be applied using LinearRegression combined with PolynomialFeatures.
   • Example:

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline

poly_pipeline = Pipeline([
    ('poly_features', PolynomialFeatures(degree=3)),
    ('linear_regression', LinearRegression())
])

poly_pipeline.fit(x, y)
y_pred = poly_pipeline.predict(x_test)

3. The pipeline efficiently transforms features and trains the model in one step.

Key Considerations When Using PolynomialFeatures

1. Feature Scaling
   • High-degree polynomial features can result in large numerical values, leading to instability in the model.
   • Solution: Apply standardization using StandardScaler before training the model.

from sklearn.preprocessing import StandardScaler

poly_pipeline = Pipeline([
    ('poly_features', PolynomialFeatures(degree=3)),
    ('scaler', StandardScaler()),
    ('linear_regression', LinearRegression())
])

2. Overfitting Risk with High-Degree Polynomials
   • Excessive polynomial terms may lead to models that fit the training data perfectly but perform poorly on unseen data.
   • Solution: Use cross-validation to select an appropriate degree and apply regularization (Lasso or Ridge regression).

from sklearn.linear_model import Ridge

ridge_pipeline = Pipeline([
    ('poly_features', PolynomialFeatures(degree=4)),
    ('ridge_regression', Ridge(alpha=1.0))
])
ridge_pipeline.fit(x, y)

3. Computational Cost for High-Degree Polynomials
   • Feature explosion: A dataset with 5 features and a degree of 4 results in 126 transformed features.
   • Solution: Use feature selection techniques such as Lasso regression to eliminate redundant polynomial terms.

Advanced Techniques: Interaction-Only Features

• Sometimes, we may only need interaction terms without additional polynomial powers.
• Setting interaction_only=True in PolynomialFeatures excludes squared terms and higher powers:

poly = PolynomialFeatures(degree=2, interaction_only=True)
X_poly = poly.fit_transform(X)

This approach reduces complexity while still capturing relationships between features.

By implementing these strategies, PolynomialFeatures can enhance model flexibility while minimizing overfitting and computational inefficiencies.

Choosing the Right Degree for Polynomial Features

Impact of Different Polynomial Degrees

Selecting the right degree is crucial:

• Degree 1 (Linear Regression): The model remains a simple straight-line fit.
• Degree 2-3: Can capture common non-linear trends without excessive complexity.
• Degree 4 or higher: May introduce overfitting, leading to poor generalization.

Using Cross-Validation to Choose the Best Degree

To avoid overfitting, we use cross-validation to find the optimal polynomial degree:

from sklearn.model_selection import cross_val_score

for d in range(1, 6):
    poly = PolynomialFeatures(degree=d)
    X_poly = poly.fit_transform(x)
    model = LinearRegression()
    scores = cross_val_score(model, X_poly, y, cv=5, scoring='neg_mean_squared_error')
    print(f"Degree {d}: Mean Squared Error = {-scores.mean():.4f}")

This approach ensures that we select the polynomial degree that minimizes prediction error on unseen data.

Using PolynomialFeatures with Pipelines

Scikit-learn’s Pipeline simplifies the process of transforming features and training models:

poly_pipeline = Pipeline([
    ('poly_features', PolynomialFeatures(degree=2)),
    ('linear_regression', LinearRegression())
])

poly_pipeline.fit(x, y)
y_pred_pipeline = poly_pipeline.predict(x_test)

This ensures that feature transformation and model training happen in a streamlined workflow.

Advantages and Disadvantages of PolynomialFeatures

Advantages

✔ Allows linear models to capture non-linearity. ✔ Can be useful for small datasets where complex models are unnecessary. ✔ Works well with regularization techniques to control complexity.

Disadvantages

✖ Can lead to overfitting if degree is too high. ✖ Computationally expensive for very high-degree polynomials. ✖ May introduce multicollinearity, requiring regularization.

Regularization Techniques to Improve Polynomial Regression

High-degree polynomial models often suffer from overfitting. Ridge regression (L2 regularization) can help prevent this:

from sklearn.linear_model import Ridge

ridge_pipeline = Pipeline([
    ('poly_features', PolynomialFeatures(degree=3)),
    ('ridge_regression', Ridge(alpha=1.0))  # Adding regularization
])

ridge_pipeline.fit(x, y)
y_pred_ridge = ridge_pipeline.predict(x_test)

Regularization ensures better generalization while maintaining the benefits of polynomial feature transformation.

Conclusion

Polynomial regression using PolynomialFeatures in scikit-learn is a powerful technique for modeling non-linear relationships. By carefully selecting the polynomial degree, using cross-validation, and applying regularization techniques, we can achieve optimal model performance while avoiding overfitting.

Key takeaways:

• Use PolynomialFeatures to transform input features into polynomial terms.
• Choose an appropriate polynomial degree to balance bias and variance.
• Leverage Pipelines to streamline preprocessing and model training.
• Apply Ridge regularization to prevent overfitting in high-degree models.

With these techniques, you can effectively apply polynomial regression in real-world scenarios, enhancing the predictive capabilities of machine learning models.