In this python project, I’ll use fictive customer data from a bank to construct a predictive model for the likely churn clients.
Machine Learning Bank Customer Churn Prediction Project1) Data Preprocessing¶
1) Data Preprocessing¶
Imported All the necessary libraries:
In [1]:
import pandas as pd
import numpy as np
import random
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.model_selection import train_test_split,cross_val_score,GridSearchCV
Read the Data set and store it in a pandas Data Frame:
In [2]:
df = pd.read_csv('.\datata/Churn_Modelling.csv')
Pass the Warnings:
In [3]:
def warn(*args, **kwargs):
pass
import warnings
warnings.warn = warn
Explore the names of columns in the data frame:
In [4]:
df.columns
Out[4]:
How many rows and columns in the data frame :
In [5]:
df.shape
Out[5]:
General Information About the Columns of the Data Frame :
what are the data types of the column, missing values, and the memory taken by the data setIn [6]:
df.info()
What are the data types of the columns in the data frame:
In [7]:
df.dtypes
Out[7]:
Display the first 10 rows of data frame:
In [8]:
df.head(10)
Out[8]:
Count the missing values in the columns of the data frame:
In [9]:
df.isnull().sum()
Out[9]:
Unique values in categorical and numeric variable:
In [10]:
df["Geography"].unique(),df["Gender"].unique(), df.NumOfProducts.unique(), df.HasCrCard.unique(), df.IsActiveMember.unique()
Out[10]:
Drop the unnecessary columns from data frame:
In [11]:
df = df.drop(['CustomerId','RowNumber','Surname'], axis = "columns")
df.head()
Out[11]:
General statistic of the data: (count, mean, std, min, q1, q2, q3, max):
In [12]:
df.describe().T
Out[12]:
Value counts for Class column(Exited):
In [13]:
ValueCounts = df['Exited'].value_counts()
print(ValueCounts)
A bar graph to show the Value counts for Class column(Exited):
we observe that we have a imbalanced dataset, to handle imbalance data we may need to have resampling.
In [14]:
ax = ValueCounts.plot(kind='bar',figsize=(14,8), width=0.40 ,color=['lightblue','green'])
ax.set_xlabel("Exited values",fontsize=15)
ax.set_ylabel("Frequency Count",fontsize=15)
ax.set_title( 'A Bar graph showing values Class variable (Exited)' ,fontsize = 15)
plt.show()
Convert the categorical columns to numeric using get dummies:
In [15]:
df1 = pd.get_dummies(df,columns = ['Geography','Gender'])
df1.head()
Out[15]:
Heat map (Correlation Matrix) before Resampling of imbalanced data:
In [16]:
plt.figure(figsize=(16,10))
sns.heatmap(df.corr(),annot=True,linewidths=.5, cmap="RdYlGn")
plt.title('Heatmap showing correlations among columns',fontsize = 15)
plt.show()
Storing Features into "X" matrix and Response Class into "y" vector:
In [17]:
#Features
X = df1.loc[:,df1.columns != 'Exited']
#Response
y = df1['Exited']
Import SMOTE and fit it on X and y to resample the response classes,Remove under sampling :
In [18]:
from imblearn.over_sampling import SMOTE
sm = SMOTE(random_state=42)
X1, y1 = sm.fit_sample(X, y)
A bar graph to show the Value counts for Exite column After Resampling:
We can observe that now we have equal number of Exite and non Exite values in the data set.In [19]:
ValueCounts = pd.Series(np.array(y1)).value_counts()
ax = ValueCounts.plot(kind='bar',figsize=(14,8), width=0.40 ,fontsize=15,color=['lightblue','green'], title=' Graph After Resampling Exite Column Values' )
ax.set_xlabel("Exite values",fontsize=15)
ax.set_ylabel("Frequency Count",fontsize=15)
Out[19]:
Combine Resampled Features and response:
In [20]:
df2 = pd.concat([pd.DataFrame(X1), pd.Series(y1)], axis=1)
df2.columns = df1.columns
df2.head()
Out[20]:
In [21]:
plt.figure(figsize=(15,10))
sns.heatmap(df2.corr(),annot=True,linewidths=.5, cmap="RdYlGn")
plt.title('Heatmap After Resampling')
plt.show()
2) Data Evaluation and Exploratory Data Analysis (EDA)¶
Explore the distribution of columns and visualize them:
In [22]:
fig, axes = plt.subplots(2,4,figsize=(12,6))
feats = ['CreditScore', 'Age', 'Tenure', 'Balance', 'NumOfProducts', 'HasCrCard', 'IsActiveMember', 'EstimatedSalary']
for i, ax in enumerate(axes.flatten()):
ax.hist(df[feats[i]], bins=25, color='green')
ax.set_title(str(feats[i])+' Distribution', color='brown')
ax.set_yscale('log')
plt.tight_layout()
A Count plot to show the Comparison of Geography and Exited columns :
In [23]:
plt.figure(figsize=(14,8))
ax = sns.countplot(x="Geography", hue="Exited", data=df)
ax.set_xlabel("Geography",fontsize=15)
ax.set_ylabel("Count",fontsize=15)
ax.set_title('Comparison of Geography and Exited columns ',fontsize=15)
plt.show()
A Count plot to show the Comparison of Gender and Exited columns :
In [24]:
plt.figure(figsize=(14,8))
ax = sns.countplot(x="Gender", hue="Exited", data=df)
ax.set_xlabel("Gender",fontsize=15)
ax.set_ylabel("Count",fontsize=15)
ax.set_title('Comparison of Gender and Exited columns ',fontsize=15)
plt.show()
A Count plot to show the Comparison of HasCrCard and Exited columns :
In [25]:
plt.figure(figsize=(14,8))
ax = sns.countplot(x="HasCrCard", hue="Exited", data=df)
ax.set_xlabel("HasCrCard",fontsize=15)
ax.set_ylabel("Count",fontsize=15)
ax.set_title('Comparison of HasCrCard and Exited columns ',fontsize=15)
plt.show()
A Count plot to show the Comparison of NumOfProducts and Exited columns :
In [26]:
plt.figure(figsize=(14,8))
ax = sns.countplot(x="NumOfProducts", hue="Exited", data=df)
ax.set_xlabel("NumOfProducts",fontsize=15)
ax.set_ylabel("Count",fontsize=15)
ax.set_title('Comparison of NumOfProducts and Exited columns ',fontsize=15)
plt.show()
A Bar graph showing Exite Rate for each gender :
In [27]:
Gender_exite_rate = df.groupby('Gender').Exited.mean()
Gender_exite_rate
Out[27]:
In [28]:
ax = Gender_exite_rate.plot(kind='bar',figsize=(14,8), width=0.40 ,color=['gray','brown'])
ax.set_xlabel("Gender",fontsize=15)
ax.set_ylabel("Exited Rate",fontsize=15)
ax.set_title( 'A Bar graph showing Exite Rate for each gender' ,fontsize = 15)
plt.show()
A Bar graph showing Exite Rate for different Tenure :
In [29]:
Tenure_exite_rate = df.groupby('Tenure').Exited.mean()
Tenure_exite_rate
Out[29]:
In [30]:
ax = Tenure_exite_rate.plot(kind='bar',figsize=(14,8), width=0.40 ,color=['gray','brown'])
ax.set_xlabel("Tenure",fontsize=15)
ax.set_ylabel("Exited Rate",fontsize=15)
ax.set_title( 'A Bar graph showing Exite Rate for each Tenure' ,fontsize = 15)
plt.show()
A Bar graph showing Exite Rate for different Credit Scores :
In [31]:
CreditScore_Exite_rate = df.groupby('CreditScore').Exited.mean()
CreditScore_Exite_rate
Out[31]:
In [32]:
plt.figure(figsize = (14,7))
plt.scatter(x=CreditScore_Exite_rate.index , y= CreditScore_Exite_rate.values,color = 'blue',marker = 'o',)
plt.xlabel('CreditScore',fontsize = 15)
plt.ylabel('Exite Rate',fontsize = 15)
plt.title('Scatter plot showing Exite Rate for each CreditScore',fontsize = 15)
plt.show()
A Bar graph showing Exite Rate for different age range :
In [33]:
Age_Exite_rate = df.groupby('Age').Exited.mean()
Age_Exite_rate
Out[33]:
In [34]:
plt.figure(figsize = (14,7))
plt.scatter(x=Age_Exite_rate.index , y= Age_Exite_rate.values,color = 'blue',marker = 'o',)
plt.xlabel('Age',fontsize = 15)
plt.ylabel('Exite Rate',fontsize = 15)
plt.title('Scatter plot showing Exite Rate for Age',fontsize = 15)
plt.show()
General Statistics for the data before modelling:
In [35]:
df1.describe()
Out[35]:
3) Model Selection¶
Divide the dataset into train set and test set (80 percent train and 20 percent test):
In [36]:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X1, y1, test_size=0.20, random_state=0)
Shapes for training and testing sets:
In [37]:
X_train.shape, X_test.shape, y_train.shape, y_test.shape
Out[37]:
Baseline Models:¶
Model 1 : Decision Tree
Importing DecisionTreeClassifier from tree in sklearn:
In [38]:
from sklearn.tree import DecisionTreeClassifier
Build Decision Tree classifier using default parameters :
In [39]:
Dt_classifier = DecisionTreeClassifier()
Call 'fit' function of the created DT model:
In [40]:
Dt_classifier.fit(X_train, y_train)
Out[40]:
Make predictions using test data on Decision Tree model:
In [41]:
y_pred = Dt_classifier.predict(X_test)
Accuracy and Confusion matrix of Decision Tree Model:
In [42]:
from sklearn.metrics import confusion_matrix, accuracy_score
print('Confusion Matrix Decision Tree:\n',confusion_matrix(y_test,y_pred))
accu_dt = accuracy_score(y_test,y_pred)
print(' Accuracy Decision Tree:\n',accu_dt,'\n')
Model 2 : Sopport Vector Machine Model:
Importing SVC from sklearn.svm class:
In [43]:
from sklearn.svm import SVC
Build SVM classifier using rbf kernal as parameter:
In [44]:
Svc_classifier = SVC()
Call 'fit' function of the SVM model to train it:
In [45]:
Svc_classifier.fit(X_train, y_train)
Out[45]:
Make predictions using test data on SVM model:
In [46]:
y_pred = Svc_classifier.predict(X_test)
Accuracy and Confusion matrix of SVM Model:
In [47]:
from sklearn.metrics import confusion_matrix, accuracy_score
print('Confusion Matrix SVM:\n',confusion_matrix(y_test,y_pred))
accu_svm = accuracy_score(y_test,y_pred)
print(' Accuracy SVM:\n',accu_svm,'\n')
Model 3 : Random Forest
Importing RandomForestClassifier from ensemble class in sklearn:
In [48]:
from sklearn.ensemble import RandomForestClassifier
Build RandomForestClassifier using default parameters :
In [49]:
Rf_classifier = RandomForestClassifier()
Train the random forest on training data:
In [50]:
Rf_classifier.fit(X_train, y_train)
Out[50]:
Make predictions using test data on Random forest model:
In [51]:
y_pred = Rf_classifier.predict(X_test)
Accuracy and Confusion matrix of random forest Model:
In [52]:
from sklearn.metrics import confusion_matrix, accuracy_score
print('Confusion Matrix Random Forest:\n',confusion_matrix(y_test,y_pred))
accu_rf = accuracy_score(y_test,y_pred)
print(' Accuracy Random Forest:\n',accu_rf,'\n')
Model 4 : Logistic Regression
Importing LogisticRegression from sklearn linear_model:
In [53]:
from sklearn.linear_model import LogisticRegression
Build Logistic Regression classifier using default parameters:
In [54]:
Lg_classifier = LogisticRegression()
Call 'fit' function of the created model:
In [55]:
Lg_classifier.fit(X_train, y_train)
Out[55]:
Create predictions by calling 'predict' function of the fitted model:
In [56]:
y_pred = Lg_classifier.predict(X_test)
y_pred[0:30]
Out[56]:
Accuracy and Confusion matrix of Logistic Regression Model:
In [57]:
from sklearn.metrics import confusion_matrix, accuracy_score
print('Confusion Matrix Logistic Regression:\n',confusion_matrix(y_test,y_pred))
accu_lg = accuracy_score(y_test,y_pred)
print(' Accuracy Logistic Regression:\n',accu_lg,'\n')
Model 5 : K-Nearest Neighbor Model
Importing KNeighborsClassifier from sklearn neighbors class:
In [58]:
from sklearn.neighbors import KNeighborsClassifier
Build KNeighborsClassifier using n_neighbors = 5 parameter:
In [59]:
knn_classifier = KNeighborsClassifier(n_neighbors=5)
Call 'fit' function of the created KNN model:
In [60]:
knn_classifier.fit(X_train, y_train)
Out[60]:
Make predictions by calling 'predict' function of the fitted KNN model:
In [61]:
y_pred = knn_classifier.predict(X_test)
Accuracy and Confusion matrix of KNN Model:
In [62]:
from sklearn.metrics import confusion_matrix, accuracy_score
print('Confusion Matrix KNN Model:\n',confusion_matrix(y_test,y_pred))
accu_knn = accuracy_score(y_test,y_pred)
print(' Accuracy KNN Model:\n',accu_knn,'\n')
Model 6 : Naive Bayes Model:
Importing GaussianNB from sklearn naive_bayes class:
In [63]:
from sklearn.naive_bayes import GaussianNB
Build GaussianNB using default parameters:
In [64]:
NB_classifier = GaussianNB()
Call 'fit' function of the created Naive bayes model:
In [65]:
NB_classifier.fit(X_train, y_train)
Out[65]:
Make predictions by calling 'predict' function of the fitted Naive Bayes model:
In [66]:
y_pred = NB_classifier.predict(X_test)
Accuracy and Confusion matrix of NaiveBayes Model:
In [67]:
from sklearn.metrics import confusion_matrix, accuracy_score
print('Confusion Matrix Naive Bayes:\n',confusion_matrix(y_test,y_pred))
accu_nb = accuracy_score(y_test,y_pred)
print(' Accuracy Naive Bayes Model:\n',accu_nb,'\n')
Comparing different machine learning models (model selection) in terms of accuracy:
In [68]:
results = pd.Series([accu_dt , accu_svm, accu_rf, accu_lg, accu_knn, accu_nb ])
names = ['Decision Tree','SVm','Random Forest','Logistic Regression','KNN','Naive Bayes']
ax = results.plot(kind = 'bar',figsize=(13,7),color=['black','gray','brown','blue','pink','green'])
ax.set_title('Comparision of Models',fontsize=15)
ax.set_yticks([0.0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9])
ax.set_xticklabels(names ,fontsize=15,rotation = 45)
ax.set_xlabel("Models",fontsize=15)
ax.set_ylabel("Accuracy",fontsize=15)
Out[68]:
From the above comparison we see that Random Forest gave the highest Performance in terms of accuracy hence we select Random Forest Model as our selecton.¶
4) Model Evaluation¶
Train & Evaluate Chosen Model:
Fit the selected model (Random Forest in this case) on the training dataset and evaluate the results.In [69]:
Selected_classifier = RandomForestClassifier(random_state = 0)
Selected_classifier.fit(X_train, y_train)
# Predict the Test set results
y_pred = Selected_classifier.predict(X_test)
#Evaluate Model Results on Test Set:
from sklearn.metrics import precision_score,recall_score,f1_score
acc = accuracy_score(y_test, y_pred )
prec = precision_score(y_test, y_pred )
rec = recall_score(y_test, y_pred )
f1 = f1_score(y_test, y_pred )
results = pd.DataFrame([['Random Forest', acc, prec, rec, f1, ]],columns = ['Model', 'Accuracy', 'Precision', 'Recall', 'F1 Score',])
print (results)
k-Fold Cross-Validation for Random Forest:
In [70]:
from sklearn.model_selection import cross_val_score
accuracies = cross_val_score(estimator = Selected_classifier, X = X_train, y = y_train, cv = 10)
print("Random Forest Classifier Accuracy: %0.2f (+/- %0.2f)" % (accuracies.mean(), accuracies.std() * 2))
Therefore, our k-fold Cross Validation results indicate that we would have an accuracy anywhere between 89% to 91% while running this model on any test set.¶
Evaluate with Confusion Matrix:
In [71]:
cm = confusion_matrix(y_test,y_pred)
print(cm)
In [72]:
plt.figure(figsize=(5,5))
sns.heatmap(data=cm, linewidths=.5,annot=True,square=True, cmap='Blues')
plt.xlabel('Predicted Values')
plt.ylabel('Actual Values')
plt.show()
We have got an accuracy of 90%; signalling the characteristics of a reasonably good model.¶
5) Model Improvement¶
Prepare the grid values for GridSearchCV for Random Forest Classifier:
In [73]:
# input different # of trees n_estimators for random forrest to GridSearchCV
grid_values = {'n_estimators': [5, 10,15,20,25,30,35,40,45,50,55,60],
# input two different criterion'gini','entropy' for random forrest to GridSearchCV
'criterion':['gini','entropy'],
# input different # of min_samples_split for random forrest to GridSearchCV
'min_samples_split': [1e-20, 5e-20, 1e-10, 5e-10, 1e-5, 5e-5, 1e-2, 5e-2],
}
#INitializing GridSearchCV using Random forrest Classifier, CV=5 and roc_auc as metrics
Grid_classifier_rf = GridSearchCV(Selected_classifier,grid_values,cv=10, scoring='accuracy')
Fit the Grid_classifier_rf with features and responses:
In [74]:
Grid_classifier_rf.fit(X_train,y_train)
Out[74]:
The best parameters given by GridSearchCV on Random Forest Model:
In [75]:
Grid_classifier_rf.best_params_
Out[75]:
The best accuracy score given by GridSearchCV on Random Forest Model:
In [76]:
Grid_classifier_rf.best_score_
Out[76]:
The best estimator given by GridSearchCV :
In [77]:
Grid_classifier_rf.best_estimator_
Out[77]:
Build the Random Forest model again with the best parameters given by GridSearchCV (tuned model):
In [78]:
Tuned_classifier = RandomForestClassifier(criterion = 'gini', min_samples_split = 1e-20, n_estimators = 55)
Fit the model with best parameters:
In [79]:
Tuned_classifier.fit(X_train,y_train)
Out[79]:
6) Future Predictions¶
Make predictions on test data and show first 60 predicted values:
In [80]:
y_pred = Tuned_classifier.predict(X_test)
y_pred[0:60]
Out[80]:
Compare the true values and predicted values of Exited column for tuned model:
In [81]:
df_comp = pd.DataFrame({'Actual': y_test, 'Predicted': y_pred})
df_comp.head(20)
Out[81]:
Accuracy and Confusion matrix for Tuned Random Forest Model:
In [82]:
from sklearn.metrics import confusion_matrix, accuracy_score
print('Confusion Matrix Tuned Random Forest:\n',confusion_matrix(y_test,y_pred))
tuned_accu = accuracy_score(y_test,y_pred)
print(' Accuracy Tuned Random Forest:\n',tuned_accu,'\n')
7) Model Deployment¶
Deploy the model to a server using ‘joblib’ library so that we can productionize the end-to-end machine learning framework. Later we can run the model over any new dataset to predict the probability of any customer to churn in months to come.
In [ ]:
# pip install joblib
#to install the package
filename = 'final_model.model'
i = [Tuned_classifier]
joblib.dump(i,filename)
In [ ]:
Related Posts
