How to read a csv file from unix server in pyspark - apache-spark

Question

I need to create spark dataframe from csv file which is located in my UNIX server.
I tried like below,
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local").appName("demo").getOrCreate()
df = spark.read.format('csv').option('header','True'). \
load("ftp://USER:PASSWORD#UNIX_IP/home/user/sample.csv")
df.show(10)
But its throwing the error as,
pyspark.sql.utils.IllegalArgumentException: java.net.URISyntaxException: Illegal character in user info at index 32
Could anyone help me to resolve this. How we need to refer the ftp location in pyspark? Do we need to include any other library for this?

Answer 1

You need to use the addFile method like this:
import org.apache.spark.SparkFiles
sc.addFile("ftp://user:pwd#host:port/home/user/sample.csv")
spark.read.csv(SparkFiles.get("sample.csv")).show()
To test it, you could use a public ftp like:
sc.addFile("ftp://anonymous:anonymous#ftp.gnu.org/README")
spark.read.csv(SparkFiles.get("README")).show(2)
+--------------------+--------------------+
| _c0| _c1|
+--------------------+--------------------+
| This is ftp.gnu.org| the FTP server o...|
|NOTICE (Updated O...| null|
+--------------------+--------------------+
In python:
from pyspark import SparkFiles
sc.addFile('ftp://user:pwd#host:port/home/user/sample.csv')
spark.read.csv(SparkFiles.get('sample.csv')).show()