I am trying to gzip and upload a static js file to an Azure blob, but I end up with a 0 byte blob. Can someone tell me what I'm doing wrong?
var filePath = "C:\test.js";
using (var compressed = new MemoryStream()) {
using (var gzip = new GZipStream(compressed, CompressionMode.Compress)) {
var bytes = File.ReadAllBytes(filePath);
gzip.Write(bytes, 0, bytes.Length);
var account = CloudStorageAccount.Parse("...");
var blobClient = new CloudBlobClient(account.BlobEndpoint, account.Credentials);
var blobContainer = blobClient.GetContainerReference("temp");
var blob = blobContainer.GetBlockBlobReference(Path.GetFileName(filePath));
blob.Properties.ContentEncoding = "gzip";
blob.Properties.ContentType = "text/javascript";
blob.Properties.CacheControl = "public, max-age=3600";
blob.UploadFromStream(compressed);
}
}
You need to reset the stream position to zero on the compressed stream. You've written data into the stream, but when you go to upload from the stream on the last line the position of the stream is at the end of the data you have written, so the blob.UploadFromStream starts from the current position in the stream, which has nothing after it.
add the following before you perform the upload:
compressed.Position = 0;
That should upload the full contents of the stream.
This isn't necessarily an Azure thing, most code that works with streams operate of the current position for the stream. I've been burned by it several times.
Related
This question already has an answer here:
Upload a streamable in-memory document (.docx) to FTP with C#?
(1 answer)
Closed 1 year ago.
What I'm trying to do, is create an Azure Function that generates certain files.
I then have to:
upload these files to an FTP server or Webdav server
archive them to a storage account.
What solution do you recommend? How can I store these temp files while I perform the operations above? (Sorry for this n00bish question, this is my first Azure Function that I'm working on.)
Thank you Martin-Prikryl. Posting your suggestion as an answer to help the other community members.
upload these files to an FTP server or Webdav server
Below is the example code on how to upload .docx to FTP server using memorystream.
Just copy your stream to the FTP request stream:
Stream requestStream = ftpRequest.GetRequestStream();
stream.CopyTo(requestStream);
requestStream.Close();
For a string (assuming the contents is a text):
byte[] bytes = Encoding.UTF8.GetBytes(data);
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(bytes, 0, bytes.Length);
}
Or even better use the StreamWriter
using (Stream requestStream = request.GetRequestStream())
using (StreamWriter writer = new StreamWriter(requestStream, Encoding.UTF8))
{
writer.Write(data);
}
If the contents is a text, you should use the text mode:
request.UseBinary = false;
Check the SO for More Information.
archive them to a storage account.
Below is the example on how to archive files
using (Stream memoryStream = new MemoryStream())
{
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
foreach (string path in Directory.EnumerateFiles(#"C:\source\directory"))
{
ZipArchiveEntry entry = archive.CreateEntry(Path.GetFileName(path));
using (Stream entryStream = entry.Open())
using (Stream fileStream = File.OpenRead(path))
{
fileStream.CopyTo(entryStream);
}
}
}
memoryStream.Seek(0, SeekOrigin.Begin);
var request =
WebRequest.Create("ftp://ftp.example.com/remote/path/archive.zip");
request.Credentials = new NetworkCredential("username", "password");
request.Method = WebRequestMethods.Ftp.UploadFile;
using (Stream ftpStream = request.GetRequestStream())
{
memoryStream.CopyTo(ftpStream);
}
}
Check the SO for complete information.
I'm currently using v2 of Azure Function Apps. I've set the environment to be 64 bit and am compiling to .Net Standard 2.0. Host Json specifies version 2.
I'm reading in a .csv and it works fine for smaller files. But when I read in a 180MB .csv into a List of string[] it's ballooning to over a GB on read and when I try to parse it, it's up over 2 GB but then throws the 'Out of Memory' Exception. Even running on an app service plan with more than 3.5 GB hasn't solved the issue.
Edit:
I'm using this:
Uri blobUri = AppendSasOnUri(blobName); _webClient = new WebClient();
Stream sourceStream = _webClient.OpenRead(blobUri);
_reader = new StreamReader(sourceStream);
However, since It's a csv, I'm splitting out entire columns of data. It's pretty hard to get away from this:
internal async Task<List<string[]>> ReadCsvAsync() {
while (!_reader.EndOfStream) {
string[] currentCsvRow = await ReadCsvRowAsync();
_fullBlobCsv.Add(currentCsvRow);
}
return _fullBlobCsv; }
Goal is to store json into blob when alls said and done.
Try using stream (StreamReader) to read the input .csv file and process one line at a time.
I'm able to parse 300mb files on consumption plan with streams. My use-case may not be same but similar. Parse a large concatenated pdf file and separate it to 5000+ smaller files and store the separated files into blob container. Below is my code for reference.
For your use case you may want to use CloudAppendBlob instead of CloudBlockBlob if you're pushing all parsed data into single blob.
public async static void ExtractSmallerFiles(CloudBlockBlob myBlob, string fileDate, ILogger log)
{
using (var reader = new StreamReader(await myBlob.OpenReadAsync()))
{
CloudBlockBlob blockBlob = null;
var fileContents = new StringBuilder(string.Empty);
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
if (line.StartsWith("%%MS_SKEY_0000_000_PDF:"))
{
var matches = Regex.Match(line, #"%%MS_SKEY_0000_000_PDF: A(\d+)_SMFL_B1234_D(\d{8})_A\d+_M(\d{15}) _N\d+");
var smallFileDate = matches.Groups[2];
var accountNumber = matches.Groups[3];
var fileName = $"SmallerFiles/{smallFileDate}/{accountNumber}.pdf";
blockBlob = myBlob.Container.GetBlockBlobReference(fileName);
}
fileContents.AppendLine(line);
if (line.Equals("%%EOF"))
{
log.LogInformation($"Uploading {fileContents.Length} bytes to {blockBlob.Name}");
await blockBlob.UploadTextAsync(fileContents.ToString());
fileContents = new StringBuilder(string.Empty);
}
}
await myBlob.DeleteAsync();
log.LogInformation("Extracted Smaller files");
}
}
I need to update orientation tag(EXIF data) for the uploaded image. I am using "PIEXIF" for this. I am not using express but swagger. The code I've written is:
//Get the uploaded buffer
var _originalBuffer = req.swagger.params.uploadedFile.value.buffer;
let Duplex = require('stream').Duplex;
//Create stream from buffer. This stream is required later to send to cloud.
let _uploadedFileStream = new Duplex();
_uploadedFileStream.push(_originalBuffer);
_uploadedFileStream.push(null);
//Create base 64 string so that "PIEXIF" can read exif data from it.
const jpegData = "data:image/jpeg;base64, " + createStringFromBuffer(_originalBuffer, 'base64');
//Read exif data.
var _exifData = piexif.load(jpegData);
//Create a copy of exif data. Will be used to create a new image with updated orientation tag.
var _exifDataCopied = {};
for (var key in _exifData) {
_exifDataCopied[key] = _exifData[key];
}
//Update orientation tag.
if (_exifDataCopied["0th"][piexif.ImageIFD.Orientation])
_exifDataCopied["0th"][piexif.ImageIFD.Orientation] = 1;
//Example taken from https://www.npmjs.com/package/piexifjs
//From here onwards, there seems to be an issue.
var exifbytes = piexif.dump(_exifDataCopied);
var newData = piexif.insert(exifbytes, createStringFromBuffer(_originalBuffer, 'binary'));
var newJpeg = new Buffer(newData);
//Create a new stream and save it as image back.
let _updatedFileStream = new Duplex();
_updatedFileStream.push(newJpeg);
_updatedFileStream.push(null);
var fs = require('fs');
var writeStream = fs.createWriteStream("./uploads/" + "Whatever.jpg")
The issue is there is no error thrown by the code. The image is also getting saved in the directory but it is corrupted. I can not preview it. Since, the code does not breaks anywhere, I am confused what could be the issue? The function to convert buffer to string with different encoding(since I need it a lot) is:
var createStringFromBuffer = function(_buffer, _encoding) {
return Buffer.from(_buffer).toString(_encoding);
}
Can someone point out where I am mistaking? I am using the example given Here
On a node server I would like to save uploaded datauri data as an image. To do this I've tried decoding the content of this png-
data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAMAAAACCAIAAAFlEcHbAAAAB3RJTUUH1gMWFjk7nUWcXQAAAAlwSFlzAABOIAAATiABFn2Z3gAAAARnQU1BAACxjwv8YQUAAAAeSURBVHjaY7h79y7DhAkTGIA04/Tp0xkYGJ49ewYAgYwLV/R7bDQAAAAASUVORK5CYII=
And saving it as a .png extension. Looks like there is more too it than that. How do I decode the datauri and save it as a file?
I've created a library to be used with Node.js that helps with encoding and decoding of data URI schemes. I believe it can help you, check:
https://github.com/DiegoZoracKy/image-data-uri
Using this library, in your case, the code would be:
'use strict';
const ImageDataURI = require('image-data-uri');
const dataURI = 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAMAAAACCAIAAAFlEcHbAAAAB3RJTUUH1gMWFjk7nUWcXQAAAAlwSFlzAABOIAAATiABFn2Z3gAAAARnQU1BAACxjwv8YQUAAAAeSURBVHjaY7h79y7DhAkTGIA04/Tp0xkYGJ49ewYAgYwLV/R7bDQAAAAASUVORK5CYII=';
const fileName = 'decoded-image.png';
ImageDataURI.outputFile(dataURI, filePath);
I was trying to decode the data using atob and saving this as a png file. I'm instead saving it base64 encoded but specifying the encoding in the write buffer.
fs.writeFileSync('tmp/myfile.png', new Buffer(data, 'base64'));
You can convert your data uri to a blob using below code:
function dataURItoBlob(dataURI) {
var byteStr;
if (dataURI.split(',')[0].indexOf('base64') >= 0)
byteStr = atob(dataURI.split(',')[1]);
else
byteStr = unescape(dataURI.split(',')[1]);
var mimeStr = dataURI.split(',')[0].split(':')[1].split(';')[0];
var arr= new Uint8Array(byteStr.length);
for (var i = 0; i < byteStr.length; i++) {
arr[i] = byteStr.charCodeAt(i);
}
return new Blob([arr], {type:mimeStr});
}
and then you can append this blob data to from data and upload it as a file:
var blob = dataURItoBlob(dataURI);
var fd = new FormData(document.forms[0]);
fd.append("image", blob);
I'm trying to convert current application that uses NPOI for creating xls document on the server to Azure hosted application. I have little experience with NPOI and Azure so 2 strikes right there. I have the app uploading the xls to Blob container however it is always blank (9 bytes). From what I understand NPOI uses filestream to write to the file so I just changed that to write to the blob container.
Here is what i think are the relevant portions:
internal void GenerateExcel(DataSet ds, int QuoteID, string ReportFileName)
{
string ExcelFileName = string.Format("{0}_{1}.xls",ReportFileName,QuoteID);
try
{
//these 2 strings will get deleted but left here for now to run side by side at the moment
string ReportDirectoryPath = HttpContext.Current.Server.MapPath(".") + "\\Reports";
if (!Directory.Exists(ReportDirectoryPath))
{
Directory.CreateDirectory(ReportDirectoryPath);
}
string ExcelReportFullPath = ReportDirectoryPath + "\\" + ExcelFileName;
if (File.Exists(ExcelReportFullPath))
{
File.Delete(ExcelReportFullPath);
}
// Create a new workbook.
var workbook = new HSSFWorkbook();
//Rest of the NPOI XLS rows cells etc. etc. all works fine when writing to disk////////////////
// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve a reference to a container.
CloudBlobContainer container = blobClient.GetContainerReference("pricingappreports");
// Create the container if it doesn't already exist.
if (container.CreateIfNotExists())
{
container.SetPermissions(new BlobContainerPermissions { PublicAccess = BlobContainerPublicAccessType.Blob });
}
// Retrieve reference to a blob with the same name.
CloudBlockBlob blockBlob = container.GetBlockBlobReference(ExcelFileName);
// Write the output to a file on the server
String file = ExcelReportFullPath;
using (FileStream fs = new FileStream(file, FileMode.Create))
{
workbook.Write(fs);
fs.Close();
}
// Write the output to a file on Azure Storage
String Blobfile = ExcelFileName;
using (FileStream fs = new FileStream(Blobfile, FileMode.Create))
{
workbook.Write(fs);
blockBlob.UploadFromStream(fs);
fs.Close();
}
}
I'm uploading to the Blob and the file exists, why doesn't the data get written to the xls?
Any help would be appreciated.
Update: I think I found the problem. Doesn't look like you can write to a file in Blob Storage. Found this Blog which pretty much answers my questions: it doesn't use NPOI but the concept is the same. http://debugmode.net/2011/08/28/creating-and-updating-excel-file-in-windows-azure-web-role-using-open-xml-sdk/
Thanks
Can you install fiddler and check the request and the response packets? You may also need to seek back to 0 between two writes . So the correct code here could be to add the below before trying to write the stream to blob.
workbook.Write(fs);
fs.Seek(0, SeekOrigin.Begin);
blockBlob.UploadFromStream(fs);
fs.Close();
I also noticed that you are using String Blobfile = ExcelFileName instead of String Blobfile = ExcelReportFullPath.