Copying files from FTP to Azure Blob Storage - azure

I have created my FTP (ftp://xyz.in) with user id and credentials.
I have created an asp.net core API application that will copy files from FTP to Azure blob storage.
I have my API solution placed in C://Test2/Test2 folder.
Now below is my code :
FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp:/xyz.in");
request.Method = WebRequestMethods.Ftp.UploadFile;
// This example assumes the FTP site uses anonymous logon.
request.Credentials = new NetworkCredential("pqr#efg.com", "lmn");
// Copy the contents of the file to the request stream.
byte[] fileContents;
// Getting error in below line.
using (StreamReader sourceStream = new StreamReader("ftp://xyz.in/abc.txt"))
{
fileContents = Encoding.UTF8.GetBytes(sourceStream.ReadToEnd());
}
request.ContentLength = fileContents.Length;
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(fileContents, 0, fileContents.Length);
}
using (FtpWebResponse response = (FtpWebResponse)request.GetResponse())
{
Console.WriteLine($"Upload File Complete, status {response.StatusDescription}");
}
But on line
using (StreamReader sourceStream = new StreamReader("ftp://xyz.in/abc.txt"))
I am getting error : System.IO.IOException: 'The filename, directory name, or volume label syntax is incorrect : 'C:\Test2\Test2\ftp:\xyz.in\abc.txt''
I am not able to understand from where does 'C:\Test2\Test2' string gets append to my FTP.
Test2 is a folder where my .Net Core application is placed.

StreamReader() doesn't take a URL/URI, it takes a file path on your local system: (read the doco):
https://learn.microsoft.com/en-us/dotnet/api/system.io.streamreader.-ctor?view=net-5.0
StreamReader is interpurting the string you've supplied as a filename ("ftp://xyz.in/abc.txt"), and it's looking for it in the current running folder "C:\Test2\Test2". If your string was "abc.txt", it would look for a file called "abc.txt" in the current folder, e.g. C:\Test2\Test2\abc.txt.
What you want is to get the file using WebClient or something similar:
WebClient request = new WebClient();
string url = "ftp://xyz.in/abc.txt";
request.Credentials = new NetworkCredential("username", "password");
try
{
byte[] fileContents = request.DownloadData(url);
// Do Something...
}

Related

How to upload file to FTP/webdav generated by c# Azure Function? [duplicate]

This question already has an answer here:
Upload a streamable in-memory document (.docx) to FTP with C#?
(1 answer)
Closed 1 year ago.
What I'm trying to do, is create an Azure Function that generates certain files.
I then have to:
upload these files to an FTP server or Webdav server
archive them to a storage account.
What solution do you recommend? How can I store these temp files while I perform the operations above? (Sorry for this n00bish question, this is my first Azure Function that I'm working on.)
Thank you Martin-Prikryl. Posting your suggestion as an answer to help the other community members.
upload these files to an FTP server or Webdav server
Below is the example code on how to upload .docx to FTP server using memorystream.
Just copy your stream to the FTP request stream:
Stream requestStream = ftpRequest.GetRequestStream();
stream.CopyTo(requestStream);
requestStream.Close();
For a string (assuming the contents is a text):
byte[] bytes = Encoding.UTF8.GetBytes(data);
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(bytes, 0, bytes.Length);
}
Or even better use the StreamWriter
using (Stream requestStream = request.GetRequestStream())
using (StreamWriter writer = new StreamWriter(requestStream, Encoding.UTF8))
{
writer.Write(data);
}
If the contents is a text, you should use the text mode:
request.UseBinary = false;
Check the SO for More Information.
archive them to a storage account.
Below is the example on how to archive files
using (Stream memoryStream = new MemoryStream())
{
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
foreach (string path in Directory.EnumerateFiles(#"C:\source\directory"))
{
ZipArchiveEntry entry = archive.CreateEntry(Path.GetFileName(path));
using (Stream entryStream = entry.Open())
using (Stream fileStream = File.OpenRead(path))
{
fileStream.CopyTo(entryStream);
}
}
}
memoryStream.Seek(0, SeekOrigin.Begin);
var request =
WebRequest.Create("ftp://ftp.example.com/remote/path/archive.zip");
request.Credentials = new NetworkCredential("username", "password");
request.Method = WebRequestMethods.Ftp.UploadFile;
using (Stream ftpStream = request.GetRequestStream())
{
memoryStream.CopyTo(ftpStream);
}
}
Check the SO for complete information.

Duplicating File Uploading Process - Asp.net WebApi

I created a web API which allows users to send files and upload to Azure Storage. The way it works is, the client app will connect to API to send one or more files to the file upload controller and controller will take care of rest such as
Upload file to Azure storage
Update database
Works great but I don't think it is the right way to do this because now I can see there are two different processes
Upload file from the client's file system to my web API (server)
Upload file to the Azure storage from API (server)
It gives me the feeling that I am duplicating the upload process as the same file first travels to API (server) and then Azure (destination) from the client (file system). I feel the need of showing two progress-bars to the client for file upload progress (from client to server and then the server to Azure) - That just doesn't make sense to me and I feel that my approach is incorrect.
My API accepts up to 250MBs so you can imagine the overload.
What do you guys think?
//// API Controller
if (!Request.Content.IsMimeMultipartContent("form-data"))
{
throw new HttpResponseException(HttpStatusCode.UnsupportedMediaType);
}
var provider = new RestrictiveMultipartMemoryStreamProvider();
var contents = await Request.Content.ReadAsMultipartAsync(provider);
int Total_Files = contents.Contents.Count();
foreach (HttpContent ctnt in contents.Contents)
{
await storageManager.AddBlob(ctnt)
}
////// Stream
#region SteamHelper
public class RestrictiveMultipartMemoryStreamProvider : MultipartMemoryStreamProvider
{
public override Stream GetStream(HttpContent parent, HttpContentHeaders headers)
{
var extensions = new[] { "pdf", "doc", "docx", "cab", "zip" };
var filename = headers.ContentDisposition.FileName.Replace("\"", string.Empty);
if (filename.IndexOf('.') < 0)
return Stream.Null;
var extension = filename.Split('.').Last();
return extensions.Any(i => i.Equals(extension, StringComparison.InvariantCultureIgnoreCase))
? base.GetStream(parent, headers)
: Stream.Null;
}
}
#endregion SteamHelper
///// AddBlob
public async Task<string> AddBlob(HttpContent _Payload)
{
CloudStorageAccount cloudStorageAccount = KeyVault.AzureStorage.GetConnectionString();
CloudBlobClient cloudBlobClient = cloudStorageAccount.CreateCloudBlobClient();
CloudBlobContainer cloudBlobContainer = cloudBlobClient.GetContainerReference("SomeContainer");
cloudBlobContainer.CreateIfNotExists();
try
{
byte[] fileContentBytes = _Payload.ReadAsByteArrayAsync().Result;
CloudBlockBlob blob = cloudBlobContainer.GetBlockBlobReference("SomeBlob");
blob.Properties.ContentType = _Payload.Headers.ContentType.MediaType;
blob.UploadFromByteArray(fileContentBytes, 0, fileContentBytes.Length);
var B = await blob.CreateSnapshotAsync();
B.FetchAttributes();
return "Snapshot ETAG: " + B.Properties.ETag.Replace("\"", "");
}
catch (Exception X)
{
return ($"Error : " + X.Message);
}
}
It gives me the feeling that I am duplicating the upload process as
the same file first travels to API (server) and then Azure
(destination) from the client (file system).
I think you're correct. One possible solution would be have your API generate a Shared Access Signature (SAS) token and return that SAS token/URI to the client whenever a client wishes to upload a file.
Using this SAS URI your client can directly upload the file to Azure Storage without sending it to your API first. Once the file is uploaded successfully by the client, it can send a message to the API to update the database.
You can read more about SAS here: https://learn.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1.
I have also written a blog post long time back on using SAS that you may find useful: https://gauravmantri.com/2013/02/13/revisiting-windows-azure-shared-access-signature/.

Why can't I download an Azure Blob using an asp.net core application published to Azure server

I am trying to download a Blob from an Azure storage account container. When I run the application locally, I get the correct "Download" folder C:\Users\xxxx\Downloads. When I publish the application to Azure and try to download the file, I get an error. I have tried various "Knownfolders", and some return empty strings, others return the folders on the Azure server. I am able to upload files fine, list the files in a container, but am struggling with downloading a file.
string conn =
configuration.GetValue<string>"AppSettings:AzureContainerConn");
CloudStorageAccount storageAcct = CloudStorageAccount.Parse(conn);
CloudBlobClient blobClient = storageAcct.CreateCloudBlobClient();
CloudBlobContainer container =
blobClient.GetContainerReference(containerName);
Uri uriObj = new Uri(uri);
string filename = Path.GetFileName(uriObj.LocalPath);
// get block blob reference
CloudBlockBlob blockBlob = container.GetBlockBlobReference(filename);
Stream blobStream = await blockBlob.OpenReadAsync();
string _filepath = _knownfolder.Path + "\\projectfiles\\";
Directory.CreateDirectory(_filepath);
_filepath = _filepath + filename;
Stream _file = new MemoryStream();
try
{
_file = File.Open(_filepath, FileMode.Create, FileAccess.Write);
await blobStream.CopyToAsync(_file);
}
finally
{
_file.Dispose();
}
The expected end result is the file ends up in the folder within the users "Downloads" folder.
Since you're talking about publishing to Azure, the code is probably from a web application, right? And the code for the web application runs on the server. Which means the code is trying to download the blob to the server running the web application.
To present a downloadlink to the user to enable them to download the file, use the FileStreamResult which
Represents an ActionResult that when executed will write a file from a stream to the response.
A (pseudo code) example:
[HttpGet]
public FileStreamResult GetFile()
{
var stream = new MemoryStream();
CloudBlockBlob blockBlob = container.GetBlockBlobReference(filename);
blockBlob.DownloadToStream(stream);
blockBlob.Seek(0, SeekOrigin.Begin);
return new FileStreamResult(stream, new MediaTypeHeaderValue("text/plain"))
{
FileDownloadName = "someFile.txt"
};
}

How can i delete directory in Azure datalake Store using httpclient API

I can delete a file in Azure using the httpclient API , But how can I delete a directory in Azure's datalake Store using the httpclient API?
File Delete Code
private const string DeleteUrl = "https://{0}.azuredatalakestore.net/webhdfs/v1/{1}?op=DELETE";
public string DeleteFile(string path)
{
var deleteUrl = string.Format(DeleteUrl, _datalakeAccountName, path);
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", _accesstoken.access_token);
var res = client.DeleteAsync(deleteUrl).Result;
var data = res.Content.ReadAsStringAsync().Result;
return data;
}
}
But how i How can i delete directory in Azure datalake Store using httpclient API
Please have try a try use the following URL to delete the directory
private const string DeleteUrl = "https://{0}.azuredatalakestore.net/webhdfs/v1/{1}?api-version=2017-08-01&op=DELETE&recursive=true";
If recursive=true then will delete all of the files in the directory include the directory itself.
If recursive=false and there are files in the directory will get 403 forbidden error.

NPOI writing XLS file converting to Azure Blob

I'm trying to convert current application that uses NPOI for creating xls document on the server to Azure hosted application. I have little experience with NPOI and Azure so 2 strikes right there. I have the app uploading the xls to Blob container however it is always blank (9 bytes). From what I understand NPOI uses filestream to write to the file so I just changed that to write to the blob container.
Here is what i think are the relevant portions:
internal void GenerateExcel(DataSet ds, int QuoteID, string ReportFileName)
{
string ExcelFileName = string.Format("{0}_{1}.xls",ReportFileName,QuoteID);
try
{
//these 2 strings will get deleted but left here for now to run side by side at the moment
string ReportDirectoryPath = HttpContext.Current.Server.MapPath(".") + "\\Reports";
if (!Directory.Exists(ReportDirectoryPath))
{
Directory.CreateDirectory(ReportDirectoryPath);
}
string ExcelReportFullPath = ReportDirectoryPath + "\\" + ExcelFileName;
if (File.Exists(ExcelReportFullPath))
{
File.Delete(ExcelReportFullPath);
}
// Create a new workbook.
var workbook = new HSSFWorkbook();
//Rest of the NPOI XLS rows cells etc. etc. all works fine when writing to disk////////////////
// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve a reference to a container.
CloudBlobContainer container = blobClient.GetContainerReference("pricingappreports");
// Create the container if it doesn't already exist.
if (container.CreateIfNotExists())
{
container.SetPermissions(new BlobContainerPermissions { PublicAccess = BlobContainerPublicAccessType.Blob });
}
// Retrieve reference to a blob with the same name.
CloudBlockBlob blockBlob = container.GetBlockBlobReference(ExcelFileName);
// Write the output to a file on the server
String file = ExcelReportFullPath;
using (FileStream fs = new FileStream(file, FileMode.Create))
{
workbook.Write(fs);
fs.Close();
}
// Write the output to a file on Azure Storage
String Blobfile = ExcelFileName;
using (FileStream fs = new FileStream(Blobfile, FileMode.Create))
{
workbook.Write(fs);
blockBlob.UploadFromStream(fs);
fs.Close();
}
}
I'm uploading to the Blob and the file exists, why doesn't the data get written to the xls?
Any help would be appreciated.
Update: I think I found the problem. Doesn't look like you can write to a file in Blob Storage. Found this Blog which pretty much answers my questions: it doesn't use NPOI but the concept is the same. http://debugmode.net/2011/08/28/creating-and-updating-excel-file-in-windows-azure-web-role-using-open-xml-sdk/
Thanks
Can you install fiddler and check the request and the response packets? You may also need to seek back to 0 between two writes . So the correct code here could be to add the below before trying to write the stream to blob.
workbook.Write(fs);
fs.Seek(0, SeekOrigin.Begin);
blockBlob.UploadFromStream(fs);
fs.Close();
I also noticed that you are using String Blobfile = ExcelFileName instead of String Blobfile = ExcelReportFullPath.

Resources