So assume ive got an rdd with 3000 rows. The 2000 first rows are of class 1 and the 1000 last rows are of class2.
The RDD is partitioned across 100 partitions.
When calling RDD.randomSplit(0.8,0.2)
Does the function also shuffle the rdd? Our does the splitting simply sample 20% continuously of the rdd? Or does it select 20% of the partitions randomly?
Ideally does the resulting split have the same class distribution as the original RDD. (i.e. 2:1)
Thanks
For each range defined by weights array there is a separate mapPartitionsWithIndex transformation which preserves partitioning.
Each partition is sampled using a set of BernoulliCellSamplers. For each split it iterates over the elements of a given partition and selects item if value of the next random Double is in a given range defined by normalized weights. All samplers for a given partition use the same RNG seed. It means it:
doesn't shuffle a RDD
doesn't take continuous blocks other than by chance
takes a random sample from each partition
takes non-overlapping samples
require n-splits passes over data
Related
[New to Spark] Language - Scala
As per docs, RangePartitioner sorts and divides the elements into chunks and distributes the chunks to different machines. How would it work for below example.
Let's say we have a dataframe with 2 columns and one column (say 'A') has continuous values from 1 to 1000. There is another dataframe with same schema but the corresponding column has only 4 values 30, 250, 500, 900. (These could be any values, randomly selected from 1 to 1000)
If I partition both using RangePartitioner,
df_a.partitionByRange($"A")
df_b.partitionByRange($"A")
how will the data from both the dataframes be distributed across nodes ?
Assuming that the number of partitions is 5.
Also, if I know that second DataFrame has less number of values then will reducing number of partitions for it make any difference ?
What I am struggling to understand is that how Spark maps one partition of df_a to a partition of df_b and how it sends (if it does) both those partitions to same machine for processing.
A very detailed explanation of how RangePartitioner works internally is described here
Specific to your question, RangePartitioner samples the RDD at runtime, collects the statistics, and only then are the ranges (limits) evaluated. Note that there are 2 parameters here - ranges (logical), and partitions (physical). The number of partitions can be affected by many factors - number of input files, inherited number from parent RDD, 'spark.sql.shuffle.partitions' in case of shuffling, etc. The ranges evaluated according to the sampling. In any case, RangePartitioner ensures every range is contained in single partition.
how will the data from both the dataframes be distributed across nodes ? how Spark maps one partition of df_a to a partition of df_b
I assume you implicitly mean joining 'A' and 'B', otherwise the question does not make any sense. In that case, Spark would make sure to match partitions with ranges on both DataFrames, according to their statistics.
Suppose there is a dataset with some number of rows.
I need to find out the Heterogeneity i.e.
distinct number of rows divide by total number of rows.
Please help me with spark query to execute the same.
Dataset and dataframe supports distinct function which finds distinct rows in the dataset.
So essentially you need to do
val heterogeneity = dataset.distinct.count / dataset.count
Only thing is if the dataset is big the distinct could be expensive and you might need to set the spark shuffle partition correctly.
I've done this in PySpark:
Created a DataFrame using a SELECT statement to get asset data ordered by asset serial number and then time.
Used DataFrame.map() to convert the DataFrame to an RDD.
Used RDD.combineByKey() to collate all the data for each asset, using the asset's serial number as the key.
Question: Can I be certain that the data for each asset will still be sorted in time order in the RDD resulting from the last step?
Time order is crucial for me (I need to calculate statistics over a moving time window across the data for each asset). When RDD.combineByKey() combines data from different nodes in the Spark cluster for a given key, is any order in that key's data retained? Or is the data from the different nodes combined in no particular order for a given key?
Can I be certain that the data for each asset will still be sorted in time order in the RDD resulting from the last step?
You cannot. When you apply sort across multiple dimensions (data ordered by asset serial number and then time) records for a single asset can be spread across multiple partitions. combineByKey will require a shuffle and the order in which these parts are combined is not guaranteed.
You can try with repartition and sortWithinPartitions (or its equivalent on RDDs):
df.repartition("asset").sortWithinPartitions("time")
or
df.repartition("asset").sortWithinPartitions("asset", "time")
or window functions with frame definition as follows:
w = Window.partitionBy("asset").orderBy("time")
In Spark >= 2.0 window functions can be used with UserDefinedFunctions so if you're fine with writing your own SQL extensions in Scala you can skip conversion to RDD completely.
What is the Big-O time complexity for Apache Spark RDD sortByKey?
I am trying to assign row numbers to an RDD based on a particular order.
Say I have a {K,V} pair RDD and I wish to perform an order by key using
myRDD.sortByKey(true).zipWithIndex
What is the time complexity for this operation, in big-O form?
And what is happening under-the-covers? Bubble sort? I hope not! My dataset is very large and runs across partitions, so I'm curious whether the sortByKey function is optimal, or does some kind of intermediate data structure within a partition and then something else across partitions to optimize message passing, or what.
A quick look at the code shows that a RangePartitioner is being used under the covers. The docs say:
partitions sortable records by range into roughly
* equal ranges. The ranges are determined by sampling the content of the RDD passed in
So in essence your data is sampled (O[n]), then only the unique sample keys (m) are sorted are sorted (O[m log(m)]) and ranges of keys determined, then the entire data is shuffled around (O[n], but costly), then the data sorted internally for the range of keys received on a given partition (O[p log[p)).
zipWithIndex probably uses local sizes to assign numbers, using the partition number, so it is likely that partition metadata is stored for this effect:
Zips this RDD with its element indices. The ordering is first based on the partition index
* and then the ordering of items within each partition. So the first item in the first
* partition gets index 0, and the last item in the last partition receives the largest index.
If I create two rdds like these:
a = sc.parallelize([[1 for j in range(3)] for i in xrange(10**9)])
b = sc.parallelize([[1 for j in xrange(10**9)] for i in range(3)])
When you think about it partitioning first one is intuitive, billion rows are partitioned around workers.
But for the second one there are 3 rows and for each row there are billion item.
My question is: For the second line, if I have 2 workers does one row goes to one worker, and the other two rows goes to the other worker?
Data distribution in Spark is limited to the top level sequence you use to create a RDD.
Depending on a configuration in the second case you'll get at most three non-empty partitions, each assigned to a single worker so in the second scenario 1-2 split is a likely outcome.
Generally speaking small number of elements, especially very large, doesn't fit well into Spark processing model.