하이퍼 파라미터 튜닝 & 교차 검증¶

궁극적인 목표: 좋은 머신러닝 모델을 만들자!

• 모델을 선택했으면, 하이퍼 파라미터를 튜닝해서 더 좋은 모델을 만들어봅시다.
• 더 좋은 모델은 어떻게 평가해야 제일 좋을까요?

1. Hyper Parameter¶

• 하이퍼 파라미터 = 모델이 만들어지는 데에 연구자가 변경할 수 있는 요소들 모두 다.
• Decision Tree 의 예시에서는 깊이. 최대 노드의 갯수, 사용할 feature의 갯수, ...

하이퍼 파라미터를 변경하면서 직접 모델을 튜닝해 봅시다!

In [2]:

# Insuarance Data - https://github.com/stedy/Machine-Learning-with-R-datasets/blob/master/insurance.csv

import pandas as pd

df = pd.read_csv("../../data/insurance.csv")
df

# Insuarance Data - https://github.com/stedy/Machine-Learning-with-R-datasets/blob/master/insurance.csv import pandas as pd df = pd.read_csv("../../data/insurance.csv") df

Out[2]:

	age	sex	bmi	children	smoker	region	charges
0	19	female	27.900	0	yes	southwest	16884.92400
1	18	male	33.770	1	no	southeast	1725.55230
2	28	male	33.000	3	no	southeast	4449.46200
3	33	male	22.705	0	no	northwest	21984.47061
4	32	male	28.880	0	no	northwest	3866.85520
...	...	...	...	...	...	...	...
1333	50	male	30.970	3	no	northwest	10600.54830
1334	18	female	31.920	0	no	northeast	2205.98080
1335	18	female	36.850	0	no	southeast	1629.83350
1336	21	female	25.800	0	no	southwest	2007.94500
1337	61	female	29.070	0	yes	northwest	29141.36030

1338 rows × 7 columns

In [3]:

# VISUALIZE!!
import seaborn as sns
import matplotlib.pyplot as plt

# 1. Pairplot for numerical relationships, highlighting smoker status
sns.set(style="whitegrid")
sns.pairplot(
    df, 
    hue="smoker", 
    vars=["age", "bmi", "children", "charges"], 
    diag_kind="hist",
    palette="muted"
)
plt.suptitle("Pairplot of Key Features (Colored by Smoker)", y=1.02)
plt.show()

# 2. Barplot for charges by region and smoker status
plt.figure(figsize=(10, 6))
sns.barplot(
    x="region", 
    y="charges", 
    hue="smoker", 
    data=df, 
    ci=None, 
    palette="pastel"
)
plt.title("Average Charges by Region and Smoking Status")
plt.show()

# 3. Distribution of charges with smoker categorization
plt.figure(figsize=(10, 6))
sns.histplot(
    df, 
    x="charges", 
    hue="smoker", 
    kde=True, 
    palette="coolwarm", 
    bins=30
)
plt.title("Distribution of Charges (Smoker vs Non-Smoker)")
plt.show()

# 4. Scatterplot: BMI vs. Charges with regression and smoker distinction
sns.lmplot(
    x="bmi", 
    y="charges", 
    hue="smoker", 
    data=df, 
    height=6, 
    aspect=1.5, 
    scatter_kws={"alpha": 0.5}
)
plt.title("BMI vs. Charges (Colored by Smoker)")
plt.show()

# VISUALIZE!! import seaborn as sns import matplotlib.pyplot as plt # 1. Pairplot for numerical relationships, highlighting smoker status sns.set(style="whitegrid") sns.pairplot( df, hue="smoker", vars=["age", "bmi", "children", "charges"], diag_kind="hist", palette="muted" ) plt.suptitle("Pairplot of Key Features (Colored by Smoker)", y=1.02) plt.show() # 2. Barplot for charges by region and smoker status plt.figure(figsize=(10, 6)) sns.barplot( x="region", y="charges", hue="smoker", data=df, ci=None, palette="pastel" ) plt.title("Average Charges by Region and Smoking Status") plt.show() # 3. Distribution of charges with smoker categorization plt.figure(figsize=(10, 6)) sns.histplot( df, x="charges", hue="smoker", kde=True, palette="coolwarm", bins=30 ) plt.title("Distribution of Charges (Smoker vs Non-Smoker)") plt.show() # 4. Scatterplot: BMI vs. Charges with regression and smoker distinction sns.lmplot( x="bmi", y="charges", hue="smoker", data=df, height=6, aspect=1.5, scatter_kws={"alpha": 0.5} ) plt.title("BMI vs. Charges (Colored by Smoker)") plt.show()

/var/folders/kx/c6xk17ln6blbs7f3nf_p1str0000gn/T/ipykernel_33742/3579438954.py:19: FutureWarning: 

The `ci` parameter is deprecated. Use `errorbar=None` for the same effect.

  sns.barplot(

In [4]:

from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error, r2_score

# Prepare the data for the Decision Tree Regressor
# Convert categorical variables into dummy variables
insurance_data_encoded = pd.get_dummies(df, drop_first=True)

# Define features (X) and target (y)
X = insurance_data_encoded.drop("charges", axis=1)
y = insurance_data_encoded["charges"]

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Create and train the Decision Tree Regressor
decision_tree_model = DecisionTreeRegressor(random_state=42)
decision_tree_model.fit(X_train, y_train)

# Make predictions on the test set
y_pred = decision_tree_model.predict(X_test)

# Evaluate the model
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

mse, r2

from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeRegressor from sklearn.metrics import mean_squared_error, r2_score # Prepare the data for the Decision Tree Regressor # Convert categorical variables into dummy variables insurance_data_encoded = pd.get_dummies(df, drop_first=True) # Define features (X) and target (y) X = insurance_data_encoded.drop("charges", axis=1) y = insurance_data_encoded["charges"] # Split the data into training and testing sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) # Create and train the Decision Tree Regressor decision_tree_model = DecisionTreeRegressor(random_state=42) decision_tree_model.fit(X_train, y_train) # Make predictions on the test set y_pred = decision_tree_model.predict(X_test) # Evaluate the model mse = mean_squared_error(y_test, y_pred) r2 = r2_score(y_test, y_pred) mse, r2

Out[4]:

(44727632.28565265, 0.672863521755465)

하이퍼 파라미터 튜닝 하기¶

특히 Tree Based model은 가능한 파라미터 조합따라서 많은 성능차이를 보이곤 합니다.
다양한 파라미터들을 실험해 보면서 제일 좋은 모델을 찾아봅시다.

In [5]:

from sklearn.model_selection import GridSearchCV

# Define a reduced parameter grid for efficient tuning
param_grid = {
    'max_depth': [None, 10, 20, 30],
    'min_samples_split': [2, 10, 20],
    'min_samples_leaf': [1, 5, 10]
}

# Perform hyperparameter tuning using GridSearchCV
grid_search = GridSearchCV(
    DecisionTreeRegressor(),
    param_grid,
    cv=3,
    verbose=1,
    n_jobs=-1
)

grid_search.fit(X_train, y_train)

# Best parameters and results
best_params = grid_search.best_params_
best_model = grid_search.best_estimator_

# Evaluate the tuned model on the test set
y_pred = best_model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

best_params, mse, r2

from sklearn.model_selection import GridSearchCV # Define a reduced parameter grid for efficient tuning param_grid = { 'max_depth': [None, 10, 20, 30], 'min_samples_split': [2, 10, 20], 'min_samples_leaf': [1, 5, 10] } # Perform hyperparameter tuning using GridSearchCV grid_search = GridSearchCV( DecisionTreeRegressor(), param_grid, cv=3, verbose=1, n_jobs=-1 ) grid_search.fit(X_train, y_train) # Best parameters and results best_params = grid_search.best_params_ best_model = grid_search.best_estimator_ # Evaluate the tuned model on the test set y_pred = best_model.predict(X_test) mse = mean_squared_error(y_test, y_pred) r2 = r2_score(y_test, y_pred) best_params, mse, r2

Fitting 3 folds for each of 36 candidates, totalling 108 fits

Out[5]:

({'max_depth': None, 'min_samples_leaf': 10, 'min_samples_split': 2},
 24087554.05824817,
 0.823824396654508)

In [7]:

import matplotlib.pyplot as plt
from sklearn.tree import plot_tree

plot_tree(decision_tree_model, feature_names=X.columns, filled=True, fontsize=5)
plt.title("Default Decision Tree")
plt.show()


plot_tree(best_model, feature_names=X.columns, filled=True, fontsize=5)
plt.title("Best Decision Tree")
plt.show()

import matplotlib.pyplot as plt from sklearn.tree import plot_tree plot_tree(decision_tree_model, feature_names=X.columns, filled=True, fontsize=5) plt.title("Default Decision Tree") plt.show() plot_tree(best_model, feature_names=X.columns, filled=True, fontsize=5) plt.title("Best Decision Tree") plt.show()

In [ ]:

교차 검증 (Cross Validation)¶

학습시에는 k-fold, N 더미로 데이터를 나눠서 학습/실험을 반복합니다. 왜냐하면, 랜덤하게 나눠진 데이터에서 우연히 모델이 좋을 수도 있기 때문이죠. 엄청나게 많은 파라미터를 시험하게 되면, overfit 가능성이 있기 때문에 이런 교차 검증 과정을 거칩니다.

딥러닝 세상으로 가게 되면, 보통 CV 는 하지 않고, train / valid / test 로 나눠서 한번만 합니다.
왜냐하면 학습이 너무 오래걸리고 비싸니까요...