apache-spark

apache-spark

PySpark error - Cannot resolve column name "age" among (_c0, _c1) [closed]

How to set the date to UTC with PySpark when converting from Epoch Time without UDF?

SPARK encoding issue while reading a csv with multiline=true option

Why does spark create only one part file despite not having coalesce/repartition?

PySpark- How to output csv/parquet file with the sequential records?

Function on top of grouped values in pyspark [closed]

Spark : 100 GB file OutOfMemoryError: Requested array size exceeds VM limit [closed]

How is it in Spark, reduce and aggregate are the same

Issue in Pyspark code when running Glue Script

update nested struct with null values

java.lang.ClassCastException while saving delta-lake data to minio

spark auc and pr-auc not stable

Get the Query stored in Database and then collect information after running query

Physical mem used % and Physical Vcores Used % in spark 3 yarn

Understand the difference between pyspark.ml.regression 'IsotonicRegression' vs 'IsotonicRegressionModel'

How to make Spark skip sort in merge-join?

spark repartition assigning same container to every element in the rdd

Problems configuring LivyOperator in Airflow

java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found

Can there be conflicts between HADOOP_CONF_DIR and SPARK_CONF_DIR?

passing array into isin() function in Databricks

Returning a default value when looking up a pyspark map Column

How to display streaming computation in Google Colab?

How to properly load 10 milion json files in dataframe uning PySpark

Where is yarn.nodemanager.log-dirs in spark?

How do I drop a Cassandra table with Spark JDBC?

java.io.FileNotFoundException error in Apache Spark even though my file exists

Row identifier in sparklyr ml_kmeans

Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging

Determining Partitioning in spark

Can an AWS Glue Job Script handle multiple streaming sources in the same job?

Spark Performance on AWS EMR

pyspark-How to submit Spark SQL in parrallism?

KAFKA's topic message's timestamp cannot merge with the timestamp value within the Kafka's value

How can I set a value of Location with CustomDataSource?

Spark Aggregator on sorted Window never uses merge - is this reliable?

Can you tell Spark to calculate `when` function's arguments lazily? [duplicate]

pySpark Context Listeners

PySpark is raising error (ORA-00933) when fetching data from Oracle Database

How to get the size of a list returned by column in pyspark

Dataproc pyspark on yarn cluster not starting with specified number of cores per executor. Or, maybe it is?

Reading from Azure Event hub with Kafka driver doesn't seem to get any data

Hudi Job failing with ExecutorDeadException in spark

Pivoting streaming dataframes without aggregation in pyspark

AnalysisException: ambiguous reference error when trying to replace withColumn() with select()

Spark crashing with any python UDF, but not if I execute locally

How to write to a JSON Column in BigQuery using Spark Scala

Spark data volume, parallelisation trade-off

Spark: Join two equally large dataframes

Improving performance for Spark with a large number of small files?

Azure Synapse Studio: VS Code usage [closed]

How to merge dataframes with the closest timestamps

How Repartitioning of a data frame on frequently used filter column can be helpful in Spark?

Spark 3.3.1 is throwing error could not find file

Saving JSON DataFrame from Spark Standalone Cluster

HADOOP_CONF_DIR seems to overrule SPARK_CONF_DIR for log4j configuration

Pyspark extracting exactly 4 consecutive numeric digit from a column and return it in a new column

how to get the latest version of hudi table

How to use pyspark to efficiently keep only those groupbs from dataframe that satisfy a certain group specific filter?

Why Apache Spark Structured Streaming job get stuck on writing to checkpoint when using DBFS / S3

page:1 of 351 next page main page

Categories

bitbucket-pipelines

modulenotfounderror

android-collapsingtool...

kendo-ui-angular2

javaapplicationstub

nlp-question-answering

selenium-jupiter

spring-data-ldap

perforce-integrate

haskell-snap-framework

installshield-2010

lync-client-sdk

android-ibeacon

azure-role-environment

Resources