Issue with using .take() function with spark 2+ pyspark - python-3.x

Question

This is the code I am using. Here it runs fine without data.take but gives error when using it in pyspark python
from pyspark.mllib.recommendation import ALS, MatrixFactorizationModel, Rating
data = sc.textFile("re_u.data")
pData=data.take(2000)
ratings = pData.map(lambda l: l.split(','))\
.map(lambda l: Rating(int(l[0]), int(l[1]), float(l[2])))
Gives Error
AttributeError Traceback (most recent call last)
<ipython-input-12-c9c51af1b2e9> in <module>
2 data = sc.textFile("re_u.data")
3 pData=data.take(2000)
----> 4 ratings = pData.map(lambda l: l.split(','))\
5 .map(lambda l: Rating(int(l[0]), int(l[1]), float(l[2])))
AttributeError: 'list' object has no attribute 'map'
Update:
After using your change #Hristo Iliev it helped but encountered another issue that followed with ratings as a list. Thank you for your help!
from pyspark.mllib.recommendation import ALS, MatrixFactorizationModel, Rating
data = sc.textFile("re_u.data")
ratings = data.map(lambda l: l.split(','))\
.map(lambda l: Rating(int(l[0]), int(l[1]), float(l[2])))\
.take(2000)
rank = 20
numIterations = 20
model = ALS.train(ratings, rank, numIterations)
Gives error
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-24-7e35afff970b> in <module>
1 rank = 20
2 numIterations = 20
----> 3 model = ALS.train(ratings, rank, numIterations)
C:\spark\spark-3.0.0-preview2-bin-hadoop2.7\python\pyspark\mllib\recommendation.py in train(cls, ratings, rank, iterations, lambda_, blocks, nonnegative, seed)
271 (default: None)
272 """
--> 273 model = callMLlibFunc("trainALSModel", cls._prepare(ratings), rank, iterations,
274 lambda_, blocks, nonnegative, seed)
275 return MatrixFactorizationModel(model)
C:\spark\spark-3.0.0-preview2-bin-hadoop2.7\python\pyspark\mllib\recommendation.py in _prepare(cls, ratings)
227 else:
228 raise TypeError("Ratings should be represented by either an RDD or a DataFrame, "
--> 229 "but got %s." % type(ratings))
230 first = ratings.first()
231 if isinstance(first, Rating):
TypeError: Ratings should be represented by either an RDD or a DataFrame, but got <class 'list'>.
Please help!

Answer 1

take() is an action that takes the specified number elements from the top of an RDD and transfers them to the driver program. What you get from it is a Python list with the requested elements, which is:
local to the driver, hence you should not take too many elements
doesn't have a map() method simply because Python list class has no map() method
What you most likely want to do is to first apply the transformations to the data RDD and take() from the transformed RDD:
data = sc.textFile("re_u.data")
ratings = data.map(lambda l: l.split(','))\
.map(lambda l: Rating(int(l[0]), int(l[1]), float(l[2])))\
.take(2000)
You'll get a list of Rating instances.
Since you pass the data further down to ALS, which takes distributed data, i.e., an RDD, and not driver-local list, you have three choices:
Parallelise again the list, turning it into an RDD:
data = sc.textFile("re_u.data")
ratings = data.map(lambda l: l.split(','))\
.map(lambda l: Rating(int(l[0]), int(l[1]), float(l[2])))\
.take(2000)
ratingsRDD = sc.parallelize(ratings)
rank = 20
numIterations = 20
model = ALS.train(ratingsRDD, rank, numIterations)
Use the sample() method to sample a subset of the data in the RDD:
data = sc.textFile("re_u.data")
ratings = data.map(lambda l: l.split(','))\
.map(lambda l: Rating(int(l[0]), int(l[1]), float(l[2])))\
.sample(False, 0.1, 42)
rank = 20
numIterations = 20
model = ALS.train(ratings, rank, numIterations)
Here sample(False, 0.1, 42) means take approximately 10% of the original data and use 42 as the seed of the pseudorandom number generator. Fixing the seed will allow for reproducibility while testing. You should adjust 0.1 to the proper value so you get about 2000 samples. Notice that those samples will be taken from random places inside the RDD and will most likely not be the first 2000.
Emulate take() while staying withing the realm of RDDs:
data = sc.textFile("re_u.data")
ratings = data.map(lambda l: l.split(','))\
.map(lambda l: Rating(int(l[0]), int(l[1]), float(l[2])))\
.zipWithIndex()\
.filter(lambda l: l[1] < 2000)\
.map(lambda l: l[0])
rank = 20
numIterations = 20
model = ALS.train(ratings, rank, numIterations)
zipWithIndex() creates tuples of the RDD content where the first element comes from the RDD and the second one is the index in the RDD (essentially, the line number). You can then filter only elements with index less than 2000 and then get rid of the index using map(lambda l: l[0]).
Method 2 is probably the best one.