I'm using Databricks on Azure and am using a library called OpenPyXl.
I'm running the sameple cosde shown here: and the last line of the code is:
wb.save('document.xlsx', as_template=False)
The code seems to run so I'm guessing it's storing the file somewhere on the cluster. Does anyone know where so that I can then transfer it to BLOB?
To save a file to the FileStore, put it in the /FileStore directory in DBFS:
dbutils.fs.put("/FileStore/my-stuff/my-file.txt", "Contents of my
file")
Note: The FileStore is a special folder within Databricks File System - DBFS where you can save files and have them accessible to your web browser. You can use the File Store to:
For more detials, refer "Databricks - The FileStore".
Hope this helps.
Related
We have a full fledge Spark Application that is taking a lot off parameter from properties file. Now we want move the application to Azure notebook format .Entire code is working fine and giving expected result with hard coded parameter. But is it possible to use external properties file in Azure Databricks Notebook also ??If we can, then where we need to place properties file??
You may utilize the Databricks DBFS Filestore, Azure Databricks note books can access user's files from here.
To Upload the properties file you have, you can use 2 options
Using wget,
import sys
"wget -P /tmp/ http://<your-repo>/<path>/app1.properties"
dbutils.fs.cp("file:/tmp/app1.properties", "dbfs:/FileStore/configs/app1/")
Using dbfs.fs.put, (may be an one-time activity to create this file)
dbutils.fs.put("FileStore/configs/app1/app1.properties", "prop1=val1\nprop2=val2")
To import the properties file values,
properties = dict(line.strip().split('=') for line in open('/dbfs/FileStore/configs/app1/app1.properties'))
Hope this helps!!
There's a possibility of providing/returning arguments with use of Databricks Jobs REST API, more information can be found e.g. here: https://docs.databricks.com/dev-tools/api/latest/examples.html#jobs-api-example
I have some dynamically created files in a blob storage container. I want to send it through email as a single attachment.
The total file size is less than 5 MB.
But here the difficulty I am facing is, when I try to compress the file using CopyData options, the compressed/zipped file not creating properly with multiple files.
If I try to zip a single file by giving its full path and filename, it is working fine. But when I give a folder name to compress all the files in that folder, it is not working correctly.
Please note that here I am not using any kind of external C# code or libraries.
Any help appreciated
Thank you
You can reference my settings in Data Factory Copy active:
Source settings:
Source dataset settings:
Sink settings:
Sink dataset settings:
Pipeline works ok:
Check the zip file in contianer containerleon:
Hope this helps.
We have few .py files on my local needs to stored/saved on fileStore path on dbfs. How can I achieve this?
Tried with dbUtils.fs module copy actions.
I tried the below code but did not work, I know something is not right with my source path. Or is there any better way of doing this? please advise
'''
dbUtils.fs.cp ("c:\\file.py", "dbfs/filestore/file.py")
'''
It sounds like you want to copy a file on local to the dbfs path of servers of Azure Databricks. However, due to the interactive interface of Notebook of Azure Databricks based on browser, it could not directly operate the files on local by programming on cloud.
So the solutions as below that you can try.
As #Jon said in the comment, you can follow the offical document Databricks CLI to install the databricks CLI via Python tool command pip install databricks-cli on local and then copy a file to dbfs.
Follow the offical document Accessing Data to import data via Drop files into or browse to files in the Import & Explore Data box on the landing page, but also recommended to use CLI, as the figure below.
Upload your specified files to Azure Blob Storage, then follow the offical document Data sources / Azure Blob Storage to do the operations include dbutils.fs.cp.
Hope it helps.
I have not been able to find a solution to this so will ask the experts.
A co-worker has a .txt file on his laptop that we want to load into Azure SQL DB using SSMS and Bulk Insert. We can open the local file easily enough but we don't know how to reference this file in FROM clause.
Assuming a file named myData.txt is saved to
c:\Users\Someone
how do we tell Azure SQL DB where that file is?
You don't. :) You have to upload a file to an Azure Blob Store and then, from there, you can use BULK INSERT or OPENROWSET to open the file.
https://learn.microsoft.com/en-us/sql/t-sql/statements/bulk-insert-transact-sql?view=sql-server-2017
I've written an article that describes the steps to open a JSON file here:
https://medium.com/#mauridb/work-with-json-files-with-azure-sql-8946f066ddd4
I fixed this problem by uploading the file to a local database and then use a linked server to my Azure db to insert or update the record. Much easier than creating a Blob Storage. However, if the file is very big or you have a lot of files to upload you might not want to use my method as linked servers is not the quickest connection.
I have a required to download a file from SFTP server and the file downloaded is stored to local folder say "D:\Data\tempData.csv"
I have to read the data from local file and consume in my application for other data manipulation.
This job is created using web hooks scheduler in Azure Web Jobs.
I am unable to download file to azure and then read from there.
Can some one help me to use a location for temp data which is equivalent to "D:\Data\tempData.csv" in local system in the azure environment.
Suggest a place in azure where can I download file and then to read from there.
Thanks in Advance.
What I tried?
Tried using SSH.NET dll to download file from SFTP to local folder
Again to read from local folder to my application
Tried looking at BLOB storage usage, which was not approved Tech Arch.
In an Azure Web App, you can create files anywhere under d:\home (for persistent files) or under d:\local (temporary files). See this page for more details on the file system. Try using Kudu Console to see those locations.
How you get the file in that location sounds mostly unrelated to your primary question about what location you can use.
In Azure Environment, the "Web-Jobs" are stored in its local folder where known as "D:\home" and "D:\local" is the local folder used by the Web-hooks.
I was in need to use a folder for temporary usage of downloading a file from SFTP server and again read the file from that local temporary location file and consume it in my application.
I have used the "D:\local\Temp" as the temporary folder which is created by the code after checking the folder existence, then after creating the folder the code will download a file from server and store to this location and then read from the same location and delete the file from that temporary folder.
Thanks all for your help, #David Ebbo Thanks.