Error during conversion column pandas data frame python 3 - python-3.x

Question

I have a big problem with pandas. I have an important data frame containing
Ref_id PRICE YEAR MONTH BRAND
100000 '5000' '2012' '4' 'FORD'
100001 '10000' '2015' '5' 'MERCEDES'
...
I want to convert my PRICE, YEAR and MONTH columns but when I use .astype(int) or .apply(lambda x : int(x)) on the column I receive an ValueError. The length of my data frame is 1.8 million on rows.
ValueError: invalid literal for int() with base 10: 'PRICE'
So I don't understand why pandas wants to convert the name of the column.
Could you explain me why ?
Best,
C.

Answer 1

Try this:
In [59]: cols = 'PRICE YEAR MONTH'.split()
In [60]: cols
Out[60]: ['PRICE', 'YEAR', 'MONTH']
In [61]: for c in cols:
...: df[c] = pd.to_numeric(df[c], errors='coerce')
...:
In [62]: df
Out[62]:
Ref_id PRICE YEAR MONTH BRAND
0 100000 5000.0 2012 4 FORD
1 100001 10000.0 2015 5 MERCEDES
2 100002 NaN 2016 6 AUDI
Reproducing your error:
In [65]: df
Out[65]:
Ref_id PRICE YEAR MONTH BRAND
0 100000 5000 2012 4 FORD
1 100001 10000 2015 5 MERCEDES
2 100002 PRICE 2016 6 AUDI # pay attention at `PRICE` value !!!
In [66]: df['PRICE'].astype(int)
...
skipped
...
ValueError: invalid literal for int() with base 10: 'PRICE'
As #jezrael has added in this comment most probably you have "bad" (unexpected) values in your data set.
You can use one of the following techniques in order clean it up:
In [155]: df
Out[155]:
Ref_id PRICE YEAR MONTH BRAND
0 100000 5000 2012 4 FORD
1 100001 10000 2015 5 MERCEDES
2 Ref_id PRICE YEAR MONTH BRAND
3 100002 15000 2016 5 AUDI
In [156]: df.dtypes
Out[156]:
Ref_id object
PRICE object
YEAR object
MONTH object
BRAND object
dtype: object
In [157]: df = df.drop(df.loc[df.PRICE == 'PRICE'].index)
In [158]: df
Out[158]:
Ref_id PRICE YEAR MONTH BRAND
0 100000 5000 2012 4 FORD
1 100001 10000 2015 5 MERCEDES
3 100002 15000 2016 5 AUDI
In [159]: for c in cols:
...: df[c] = pd.to_numeric(df[c], errors='coerce')
...:
In [160]: df
Out[160]:
Ref_id PRICE YEAR MONTH BRAND
0 100000 5000 2012 4 FORD
1 100001 10000 2015 5 MERCEDES
3 100002 15000 2016 5 AUDI
In [161]: df.dtypes
Out[161]:
Ref_id object
PRICE int64
YEAR int64
MONTH int64
BRAND object
dtype: object
or simply:
In [159]: for c in cols:
...: df[c] = pd.to_numeric(df[c], errors='coerce')
...:
In [165]: df
Out[165]:
Ref_id PRICE YEAR MONTH BRAND
0 100000 5000.0 2012.0 4.0 FORD
1 100001 10000.0 2015.0 5.0 MERCEDES
2 Ref_id NaN NaN NaN BRAND
3 100002 15000.0 2016.0 5.0 AUDI
and then .dropna(how='any') if you know that there were no NaN's in your original data set:
In [166]: df = df.dropna(how='any')
In [167]: df
Out[167]:
Ref_id PRICE YEAR MONTH BRAND
0 100000 5000.0 2012.0 4.0 FORD
1 100001 10000.0 2015.0 5.0 MERCEDES
3 100002 15000.0 2016.0 5.0 AUDI