Last active
June 23, 2022 04:17
findCorrelation in python
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import pandas as pd | |
import numpy as np | |
def find_correlation(df, thresh=0.9): | |
""" | |
Given a numeric pd.DataFrame, this will find highly correlated features, | |
and return a list of features to remove | |
params: | |
- df : pd.DataFrame | |
- thresh : correlation threshold, will remove one of pairs of features with | |
a correlation greater than this value | |
""" | |
corrMatrix = df.corr() | |
corrMatrix.loc[:,:] = np.tril(corrMatrix, k=-1) | |
already_in = set() | |
result = [] | |
for col in corrMatrix: | |
perfect_corr = corrMatrix[col][corrMatrix[col] > thresh].index.tolist() | |
if perfect_corr and col not in already_in: | |
already_in.update(set(perfect_corr)) | |
perfect_corr.append(col) | |
result.append(perfect_corr) | |
select_nested = [f[1:] for f in result] | |
select_flat = [i for j in select_nested for i in j] | |
return select_flat |
Here is a website that I often refer to, for categorical columns and ways to change them:
https://pbpython.com/categorical-encoding.html
Hope that helps.
Very specific and useful, thanks!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
hey Hi ,
Nice to see. this it was useful. ''
This will be for only numerical columns.
How to do it for categorical columns?