This question already has answers here:
Overwrite specific partitions in spark dataframe write method
(14 answers)
Overwrite only some partitions in a partitioned spark Dataset
(3 answers)
Closed 4 years ago.
I have a large table and in which I would like overwrite certain top level partitions. for e.g. I have table which is partitioned based on year and month, and I would like to overwrite partitions say from year 2000 to 2018.
How I can do that.
Note : I would not like to delete the previous table and overwrite entire table with new data.
Related
This question already has answers here:
How to perform union on two DataFrames with different amounts of columns in Spark?
(22 answers)
Closed 4 years ago.
I have ‘n’ number of delimited data sets, CSVs may be. But one of them might have a few extra columns. I am trying to read all of them as dataframes and put them in one. How can I merge them as an unionAll and make them a single dataframe ?
P.S: I can do this when I know what is ‘n’. And, it’s a simple unionAll when the column counts are equal.
There is another approach other than the solutions mentioned in first two comments.
Read all CSV files to a single RDD producing RDD[String].
Map to create Rdd[Row] with appropriate length while filling missing values with null or any suitable values.
Create dataFrame schema.
Create DataFrame from RDD[Row] using created Schema.
This may not be a good approach if the CSVs has large number of columns.
Hope this helps
This question already has answers here:
Spark dataframe write method writing many small files
(6 answers)
Closed 4 years ago.
I have a data frame df, I want to partition it by date (a column in the df).
I have the code below:
df.write.partitionBy('date').mode(overwrite').orc('path')
Then under the path above, there are bunch folders, e.g. date=2018-10-08 etc...
But under the folder date=2018-10-08, there are 5 files, what I want is to reduce to only one file inside the date=2018-10-08 folder. How to do that? I still want it partitioned by date.
Thank you in advance!
In order to have 1 file per partition folder you will need to repartition the data by the partition column before writing. This will shuffle the data so the dates are in the same DataFrame/RDD partitions:
df.repartition('date').write.partitionBy('date').mode(overwrite').orc('path')
This question already has answers here:
Concatenate columns in Apache Spark DataFrame
(18 answers)
How to add a constant column in a Spark DataFrame?
(3 answers)
Closed 4 years ago.
I have several dataframes and I want to uniquely identify each row in each dataframe. Hence I want to use personalized Ids .
I am using the monotonically_increasing_id() built-in function in spark as follows:
import org.apache.spark.sql.functions._
val dfWithId = trzuCom.withColumn("UniqueID", monotonically_increasing_id)
The problem is when I try to personalize it as follows :
val dfWithId = trzuCom.withColumn("UniqueID", "TB1_" + monotonically_increasing_id)
I get errors.
Actually I want to have TB1_ID for dataframe 1, TB2_ID and so one . Any I dea how to do this please.
Best Regards
This question already has answers here:
How to select the first row of each group?
(9 answers)
Closed 4 years ago.
I have a Hive table with the schema:
id bigint
name string
updated_dt bigint
There are many records having same id, but different name and updated_dt. For each id, I want to return the record (whole row) with the largest updated_dt.
My current approach is:
After reading data from Hive, I can use case class to convert data to RDD, and then use groupBy() to group by all the records with the same id together, and later picks the one with the largest updated_dt. Something like:
dataRdd.groupBy(_.id).map(x => x._2.toSeq.maxBy(_.updated_dt))
However, since I use Spark 2.1, it first convert data to dataset using case class, and then the above approach coverts data to RDD in order to use groupBy(). There may be some overhead converting dataset to RDD. So I was wondering if I can achieve this at the dataset level without converting to RDD?
Thanks a lot
Here is how you can do it using Dataset:
data.groupBy($"id").agg(max($"updated_dt") as "Max")
There is not much overhead if you convert it to RDD. If you choose to do using RDD, It can be more optimized by using .reduceByKey() instead of using .groupBy():
dataRdd.keyBy(_.id).reduceByKey((a,b) => if(a.updated_dt > b.updated_dt) a else b).values
This question already has answers here:
How do I delete all data in a Cassandra column family?
(6 answers)
Closed 6 years ago.
Is there a command to all the rows present in a cql table in cassandra like the one in sql?
delete from TABLE
Going by the documentation, I don't find any way to perform delete operation without a where condition.
DELETE col1 FROM SomeTable WHERE userID = 'some_key_value';
To remove all rows from a CQL Table, you can use the TRUNCATE command:
TRUNCATE keyspace_name.table_name;
Or if you are already using the keyspace that contains your target table:
TRUNCATE table_name;
Important to note, but by default Cassandra creates a snapshot of the table just prior to TRUNCATE. Be sure to clean up old snapshots, or set auto_snapshot: false in your cassandra.yaml.