Select categorical features with RFE in sklearn - scikit-learn

Question

I want to select the most important features out of my data containing categorical and numerical features. I tried SelectFromModel and RFE.
As a preprocessing step, I have transform my categorical features via OHE into multiple features. (e.g. weekdays -> monday, tuesday, wednesday...)
The above mentioned methods now select only parts of categorical features (e.g. only monday). Is there any other way to either select entire categorical features or drop them completely than brute force all combinations of categorical features?

Answer 1

There are many other ways to do this.
A simple solution is using Label Encoder, which maps your categories to integers. For example, 'Monday' turns into 0, 'Tuesday' turns into 1, 'Wednseday' into 2, etc.
The problem with this is that it will trick your random forest into thinking that Tuesday (1) is greater than Monday (0), which may or may not be a reasonable thing to assume. Label Encoder is therefore only recommended when your categories have an intrinsic ranking (such as bad < neutral < good < awesome).
Another approach would be to group by the categorical column and get the mean of the target column. You can then assign numeric values to each category depending on how they relate to the target. If you want more details on this second option, post a more detailed question.
Finally, I recommend using catboost. It is a gradient boosting algorithm that can handle categorical data natively.

Answer 2

If you want Random Forests or a RFE algorithms to consider the categorical variable as a whole, then one hot encoding is not the way forward.
You can encode the variable with integers using the OrdinalEncoder transformers available in any of the open source libraries Sklearn, Category_encoders or Feature-engine.
Random forests should be able to capture non-linear relationships, hence it is not super important if the Tuesday > Monday relationship is lost in the encoding.
You can of course, replace the categories by integers of your choosing with mappings, for example [df\[my_var\].map({'Monday':1, 'Tuesday':2 ... })][3]. If you do this, you could keep those intrinsic ordinal relationships.
There are also alternative encodings that create a monotonic relationship between your categorical variable and the target, like MeanEncoding or Ordinal encoding following the target mean. These are available in the libraries Feature-engine and Category encoders. But for tree based models, monotonic encodings are not always helpful. The advantage of these encodings is that they return 1 variable per categorical variable, instead of the dummy variables, which suits your problem.
Creating the variables you mention, IsMonday, IsTuesday and so on, is also a suitable option if your willing to ignore the other labels in the variable. You can always select with RFE the most predictive variables and then train the final model using the selected ones plus the ignored categories (i.e., IsWed, IsThu, etc). If the model uses regularization, this should not influence the final performance.

Answer 3

Just set the estimator of the selector object to be a pipeline (and/or ColumnTransformer) that includes the OneHotEncoder.