Azure Databricks - Cannot export results from Databricks to blob - azure

Question

I want to export my data from Databricks to Azure blob. My Databricks commands select some pdf from my blob, run Form Recognizer and export the output results in my blob.
Here is my code:
%pip install azure.storage.blob
%pip install azure.ai.formrecognizer
from azure.storage.blob import ContainerClient
container_url = "https://mystorageaccount.blob.core.windows.net/pdf-raw"
container = ContainerClient.from_container_url(container_url)
for blob in container.list_blobs():
blob_url = container_url + "/" + blob.name
print(blob_url)
import requests
from azure.ai.formrecognizer import FormRecognizerClient
from azure.core.credentials import AzureKeyCredential
endpoint = "https://myendpoint.cognitiveservices.azure.com/"
key = "mykeynumber"
form_recognizer_client = FormRecognizerClient(endpoint, credential=AzureKeyCredential(key))
import pandas as pd
field_list = ["InvoiceDate","InvoiceID","Items","VendorName"]
df = pd.DataFrame(columns=field_list)
for blob in container.list_blobs():
blob_url = container_url + "/" + blob.name
poller = form_recognizer_client.begin_recognize_invoices_from_url(invoice_url=blob_url)
invoices = poller.result()
print("Scanning " + blob.name + "...")
for idx, invoice in enumerate(invoices):
single_df = pd.DataFrame(columns=field_list)
for field in field_list:
entry = invoice.fields.get(field)
if entry:
single_df[field] = [entry.value]
single_df['FileName'] = blob.name
df = df.append(single_df)
df = df.reset_index(drop=True)
df
account_name = "mystorageaccount"
account_key = "fs.azure.account.key." + account_name + ".blob.core.windows.net"
try:
dbutils.fs.mount(
source = "wasbs://pdf-recognized#mystorageaccount.blob.core.windows.net",
mount_point = "/mnt/pdf-recognized",
extra_configs = {account_key: dbutils.secrets.get(scope ="formrec", key="formreckey")} )
except:
print('Directory already mounted or error')
df.to_csv(r"/dbfs/mnt/pdf-recognized/output.csv", index=False)
The code runs fine until the very last line. I get the following error message:
Directory already mounted or error. FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/mnt/pdf-recognized/output.csv'.
I tried using /dbfs:/ instead of /dbfs/ but I don't know what I am doing wrong.
How can I export my Databricks results to the blob?
Thank you

Answer 1

It looks like that you're trying to mount the storage that was already mounted. Really, mount operation should be done only once, and not done dynamically. You have several choices to implement it correctly:
unmount before mouting using the dbutils.fs.unmount("/mnt/pdf-recognized")
check if storage already mounted & only run mount if it's not mounted. Something like this (not tested)
mounts = [mount for mount in dbutils.fs.mounts()
if mount.mountPoint == "/mnt/pdf-recognized"]
if len(mounts) == 0:
dbutils.fs.mount(....)
you don't really need a mount - it has the "bad" property that anyone in the workspace can use it with permissions that was used for mounting. It could be just simpler to write results to local disk, and then copy file to necessary location using dbutils.fs.cp with wasbs protocol. Something like this:
df.to_csv(r"/tmp/my-output.csv", index=False)
spark.conf.set(account_key, dbutils.secrets.get(scope ="formrec", key="formreckey"))
dbutils.fs.cp("file:///tmp/my-output.csv"),
"wasbs://pdf-recognized#mystorageaccount.blob.core.windows.net/output.csv")