read_csv giving error for movielens 20M dataset - python-3.x

Question

I am using pandas for the first time and wanted to do some data analysis for Movielens dataset. After running my code for 1M dataset, I wanted to experiment with Movielens 20M
I am only reading one file i.e ratings.csv
However, I faced multiple problems with 20M dataset, and after spending much time I realized that this is because the dtypes of columns being read are not as expected.
The following code(where path is path of ratings.csv file)
import pandas as pd
import numpy as np
df = pd.read_csv('../data/ml-20m/ratings.csv',sep=',',names=
['userId','movieId','rating','timestamp'],engine='python', dtype=
{'userId':np.int32, 'movieId':np.int32, 'rating':np.float64,
'timestamp':np.int64}, skipinitialspace=True, error_bad_lines=False)
is giving me the following error :-
Traceback (most recent call last): File
"/home/sahildeep/.local/lib/python3.5/site-packages/pandas/io/parsers.py",
line 1663, in _cast_types
values = astype_nansafe(values, cast_type, copy=True) File "/home/sahildeep/.local/lib/python3.5/site-packages/pandas/core/dtypes/cast.py",
line 709, in astype_nansafe
return lib.astype_intsafe(arr.ravel(), dtype).reshape(arr.shape) File "pandas/_libs/lib.pyx", line 456, in
pandas._libs.lib.astype_intsafe File "pandas/_libs/src/util.pxd",
line 142, in util.set_value_at_unsafe ValueError: invalid literal for
int() with base 10: 'movieId'
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "test.py", line 4, in
df = pd.read_csv('../data/ml-20m/ratings.csv',sep=',',names= ['userId','movieId','rating','timestamp'],engine='python', dtype=
{'userId':np.int32, 'movieId':np.int32, 'rating':np.float64,
'timestamp':np.int64}, skipinitialspace=True, error_bad_lines=False)
File
"/home/sahildeep/.local/lib/python3.5/site-packages/pandas/io/parsers.py",
line 678, in parser_f
return _read(filepath_or_buffer, kwds) File "/home/sahildeep/.local/lib/python3.5/site-packages/pandas/io/parsers.py",
line 446, in _read
data = parser.read(nrows) File "/home/sahildeep/.local/lib/python3.5/site-packages/pandas/io/parsers.py",
line 1036, in read
ret = self._engine.read(nrows) File "/home/sahildeep/.local/lib/python3.5/site-packages/pandas/io/parsers.py",
line 2272, in read
data = self._convert_data(data) File "/home/sahildeep/.local/lib/python3.5/site-packages/pandas/io/parsers.py",
line 2338, in _convert_data
clean_conv, clean_dtypes) File "/home/sahildeep/.local/lib/python3.5/site-packages/pandas/io/parsers.py",
line 1574, in _convert_to_ndarrays
cvals = self._cast_types(cvals, cast_type, c) File "/home/sahildeep/.local/lib/python3.5/site-packages/pandas/io/parsers.py",
line 1666, in _cast_types
"type %s" % (column, cast_type)) ValueError: Unable to convert column movieId to type
Basically I want to skip all those lines whose datatype doesn't conform to dictionary
{'userId':np.int32, 'movieId':np.int32, 'rating':np.float64,
'timestamp':np.int64}
If I don't give the dtype argument to read_csv, then all four columns turn out to be of type "object" which is not what I want.
I searched on google and found noone facing this problem. Can you help me?
I am using python3

Answer 1

Problem is you define columns names, but csv have header, so first row of DataFrame is same like columns names, so all rows are converted to strings:
df = pd.read_csv('ratings.csv',
names= ['userId','movieId','rating','timestamp'])
print (df.head())
userId movieId rating timestamp
0 user_id movie_id rating timestamp
1 1 1193 5 978300760
2 1 661 3 978302109
3 1 914 3 978301968
4 1 3408 4 978300275
Solution is use parameter skiprows=1 or header=0 for rename columns names by names parameter:
df = pd.read_csv('ratings.csv',
dtype= {'userId':np.int32,
'movieId':np.int32,
'rating':np.float64,
'timestamp':np.int64},
header=0, #skiprows=1
names= ['userId','movieId','rating','timestamp'])
print (df.head())
userId movieId rating timestamp
0 1 1193 5.0 978300760
1 1 661 3.0 978302109
2 1 914 3.0 978301968
3 1 3408 4.0 978300275
4 1 2355 5.0 978824291
If dont want rename column names:
df = pd.read_csv('ratings.csv',
dtype= {'userId':np.int32,
'movieId':np.int32,
'rating':np.float64,
'timestamp':np.int64})
print (df.head())
user_id movie_id rating timestamp
0 1 1193 5.0 978300760
1 1 661 3.0 978302109
2 1 914 3.0 978301968
3 1 3408 4.0 978300275
4 1 2355 5.0 978824291