Azure Blob Storage: Download all logs from Azure Storage Container "$logs" using C# - azure

I am trying to download all the logs from the Container "$logs", but it always throws a Exception -
"Could not find a part of the path 'C:\logs\blob\2020\05\24\2300\000000.log'"
public static void GetAnalyticsLogs(CloudBlobClient blobClient, CloudTableClient tableClient)
{
try
{
DateTime time = DateTime.UtcNow;
CloudAnalyticsClient analyticsClient = new CloudAnalyticsClient(blobClient.StorageUri, tableClient.StorageUri, tableClient.Credentials);
IEnumerable<ICloudBlob> results = analyticsClient.ListLogs(StorageService.Blob, time.AddDays(-30), null, LoggingOperations.All, BlobListingDetails.Metadata, null, null);
List<ICloudBlob> logs = results.ToList();
foreach (var item in logs)
{
string name = ((CloudBlockBlob)item).Name;
CloudBlobContainer container = blobClient.GetContainerReference("$logs");
CloudBlockBlob blockBlob = container.GetBlockBlobReference(name);
string path = (#"C:/logs/" + name);
using (var fileStream = System.IO.File.Create(path))
{
blockBlob.DownloadToStream(fileStream);
}
}
}
catch (Exception e)
{
Console.WriteLine(e);
}
}
How can we solve this error?

The reason is that the path contains directories, but the File.Create() method cannot create a file inside a directory if the directory does not exist. So you should create the directory first, then create the file using File.Create() method.
The code below works fine at my side:
public static void GetAnalyticsLogs(CloudBlobClient blobClient, CloudTableClient tableClient)
{
try
{
DateTime time = DateTime.UtcNow;
CloudAnalyticsClient analyticsClient = new CloudAnalyticsClient(blobClient.StorageUri, tableClient.StorageUri, tableClient.Credentials);
IEnumerable<ICloudBlob> results = analyticsClient.ListLogs(StorageService.Blob, time.AddDays(-30), null, LoggingOperations.All, BlobListingDetails.Metadata, null, null);
List<ICloudBlob> logs = results.ToList();
foreach (var item in logs)
{
string name = ((CloudBlockBlob)item).Name;
CloudBlobContainer container = blobClient.GetContainerReference("$logs");
CloudBlockBlob blockBlob = container.GetBlockBlobReference(name);
//specify the directory without file name
string sub_folder = name.Remove(name.LastIndexOf("/") + 1);
string path = (#"C:/logs/" + sub_folder);
//create the directory if it does not exist.
Directory.CreateDirectory(path);
//specify the file full path
string file_path= (#"C:/logs/" + name);
using (var fileStream = File.Create(file_path))
{
blockBlob.DownloadToStream(fileStream);
}
}
}
catch (Exception e)
{
Console.WriteLine(e);
}
}

Related

zip all files in azure blob container

I am trying to use azure function to zip all files inside a blob container using System.IO.Compression.
I could list all files inside the container using the below CloudBlob code
CloudStorageAccount storageAccount = CloudStorageAccount.Parse (storageConn);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("<container>");
BlobContinuationToken blobToken = null;
var blobs = await container.ListBlobsSegmentedAsync(blobToken);
var fileList = new List<string>();
var blobpath1 = #"https://<pathtocontainer>/test.zip";
foreach (var blbitem in blobs.Results)
{
if (blbitem is CloudBlockBlob)
{
var blobFileName = blbitem.Uri.Segments.Last().Replace("%20", " ");
var blobFilePath = blbitem.Uri.AbsolutePath.Replace(blbitem.Container.Uri.AbsolutePath + "/", "").Replace("%20", " ");
var blobPath = blobFilePath.Replace("/" + blobFileName, "");
log.LogInformation("blob path : " + blbitem.Uri.ToString());
fileList.Add(blbitem.Uri.ToString());
string rootpath = #"D:\home\site\wwwroot\ZipandSendFile\temp\";
string path = rootpath + blobPath;
log.LogInformation("saving in " + path);
//Add to zip
/*
CloudBlobContainer container = cloudBlobClient.GetContainerReference("<container>");
CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
using (FileStream fs = new FileStream(rootpath, FileMode.Create))
{
blob. DownloadToStream(fs);
}
*/
}
}
</code>
After getting each file details inside the blob , I am trying to add them into zip archive
using the below System.IO.Compression package
My attempt to download file
<code>
public static void AddFilesToZip(string zipPath, string[] files,ILogger log)
{
if (files == null || files.Length == 0)
{
return;
}
log.LogInformation("Executing add files to zip");
log.LogInformation(zipPath);
using (var zipArchive = ZipFile.Open(zipPath, ZipArchiveMode.Update))
{
log.LogInformation("in Zip archive");
foreach (var file in files)
{
var fileInfo = new FileInfo(file);
log.LogInformation(fileInfo.FullName);
zipArchive.CreateEntryFromFile(fileInfo.FullName, fileInfo.Name);
}
}
}
</code>
But I am getting access denied. Any pointers on this ?
Resolved issue by logging into kudo cmdshell and cd into directory and change file attribute
with attrib +A .

Email Attachments missing on Azure, works locally

I have implemented something similar to the following question and I can get it working locally on my server, but when I deploy to Azure it doesn't work. I don't get any errors: just an email without the attachment.
Sending attachments using Azure blobs
Are there restrictions on what types of files can be sent with SendGrid (file is only 56k)?
Does the Azure App service have to be at a particular level or can it be done on Basic?
The blob URL definitley exists and I am setting the stream to zero as suggested in that previous question.
MailMessage mm = new MailMessage("receiver address", "someone");
mm.From = new MailAddress("myAddress", "My Name");
mm.Subject = content.Subject;
mm.Body = content.Body;
mm.IsBodyHtml = true;
mm.BodyEncoding = UTF8Encoding.UTF8;
mm.DeliveryNotificationOptions = DeliveryNotificationOptions.OnFailure;
var attachmentAsStream = _storageAccessService.GetAssetAsStreamForEmail("myContainer", "fileThatExists.pdf");
var attachment = new Attachment(attachmentAsStream, "File.pdf", MediaTypeNames.Application.Pdf);
mm.Attachments.Add(attachment);
public MemoryStream GetAssetAsStreamForEmail(string containerName, string fileName)
{
// Create the blob client.
CloudBlobClient blobClient = StorageAccountReference().CreateCloudBlobClient();
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference(containerName);
CloudBlockBlob blob = container.GetBlockBlobReference(fileName);
var memoryStream = new MemoryStream();
try
{
using (var stream = new MemoryStream())
{
blob.DownloadToStream(memoryStream);
memoryStream.Seek(0, SeekOrigin.Begin);
}
}
catch (Exception ex)
{
Elmah.ErrorSignal.FromCurrentContext().Raise(new Exception("Failed to download Email Atatchment: "+ ex.Message));
}
memoryStream.Position = 0;
return memoryStream;
}
using (SmtpClient client = new SmtpClient())
{
client.Port = 587;
client.Host = "smtp.sendgrid.net";
client.EnableSsl = true;
client.Timeout = 10000;
client.DeliveryMethod = SmtpDeliveryMethod.Network;
client.UseDefaultCredentials = false;
client.Credentials = new System.Net.NetworkCredential("hidden", "hidden");
await client.SendMailAsync(message);
}
Update
Update after Yasir's suggestion below. Downloading the blob from Azure as a Stream only seems to work locally. But if I change to download as a ByteArray then it works everywhere, nonetheless...
public MemoryStream GetAssetAsStreamForEmail(string containerName, string fileName)
{
// Create the blob client.
CloudBlobClient blobClient = StorageAccountReference().CreateCloudBlobClient();
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference(containerName);
CloudBlockBlob blob = container.GetBlockBlobReference(fileName);
try
{
blob.FetchAttributes();
var fileStream = new byte[blob.Properties.Length];
for (int i = 0; i < blob.Properties.Length; i++)
{
fileStream[i] = 0x20;
}
blob.DownloadToByteArray(fileStream, 0) ;
MemoryStream bufferStream = new MemoryStream(fileStream);
}
catch (Exception ex)
{
Elmah.ErrorSignal.FromCurrentContext().Raise(new Exception("Failed to download Email Atatchment: " + ex.Message));
}
return null;
}
The following code works for me for sending emails using Sendgrid from Azure functions. I attach a CSV file from Blob Storage. It should work for you as well. All you will have to do is ensure that you read the pdf file as a byte[].
public interface IEmailAttachment
{
string Name { get; }
byte[] FileData { get; }
}
public static void Send(MailMessage mailMessage, IEnumerable<IEmailAttachment> attachments)
{
try
{
// Get the configuration data
string server = ConfigReader.EmailServer;
int port = ConfigReader.EmailPort;
string username = ConfigReader.SendGridUserName;
string password = ConfigReader.SendGridPassword;
smtpClient.EnableSsl = false;
smtpClient.Credentials = new NetworkCredential(username, password);
// Create the SMTP Client
SmtpClient smtpClient = new SmtpClient(server, port);
// Prepare the MailMessage
mailMessage.From = new MailAddress(ConfigReader.FromEmail);
var toEmails = ConfigReader.ToEmail.Split(',');
foreach (var toEmail in toEmails)
{
mailMessage.To.Add(toEmail);
}
var ccEmails = ConfigReader.EmailCc.Split(',');
foreach (var ccEmail in ccEmails)
{
mailMessage.CC.Add(ccEmail);
}
// Add attachments
List<MemoryStream> files = new List<MemoryStream>();
if (attachments != null)
{
foreach (IEmailAttachment file in attachments)
{
MemoryStream bufferStream = new MemoryStream(file.FileData);
files.Add(bufferStream);
Attachment attachment = new Attachment(bufferStream, file.Name);
mailMessage.Attachments.Add(attachment);
}
}
mailMessage.IsBodyHtml = true;
// Send the email
smtpClient.Send(mailMessage);
foreach (MemoryStream stream in files)
{
stream.Dispose();
}
}
catch (Exception)
{
throw;
}
}

How to download image to Xamarin.forms listview

I have a image url in list view and after click to listview image should download on same row.
How can I achieve on Xamarin.forms?
I got image from Azure blob
I'd recommend checking this blog post using Xamarin Forms Listview with SQL and Blob storage. They basically used SQL to store the images from Blob storage using the bellow code:
public class AzureMediaStorageService : IMediaStorageService
{
private CloudBlobContainer MediaContainer { get; set; }
public void InitializeStorage(string settings = null)
{
if (MediaContainer != null)
return;
if (string.IsNullOrEmpty(settings))
throw new ArgumentNullException(nameof(settings), "Azure Storage Media Service needs a connection string name as \"settings\" parameter");
var storageAccount = CloudStorageAccount.Parse(ConfigurationManager.ConnectionStrings[settings].ConnectionString);
var blobClient = storageAccount.CreateCloudBlobClient();
MediaContainer = blobClient.GetContainerReference("medias");
MediaContainer.CreateIfNotExists();
MediaContainer.SetPermissions(new BlobContainerPermissions { PublicAccess = BlobContainerPublicAccessType.Blob });
}
public async Task<Uri> UploadBytesAsync(byte[] bytes, string uniqueId, MediaType mediaType)
{
var blockBlob = MediaContainer.GetBlockBlobReference(uniqueId);
blockBlob.Properties.ContentType = mediaType == MediaType.Picture ? "image/jpeg" : "video/mp4";
await blockBlob.UploadFromByteArrayAsync(bytes, 0, bytes.Length);
return blockBlob.Uri;
}
public async Task<byte[]> DownloadBytesAsync(Uri uri)
{
using (var httpClient = new HttpClient())
{
var response = await httpClient.GetAsync(uri, HttpCompletionOption.ResponseContentRead);
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsByteArrayAsync();
}
}
public string GetSecureClientUploadUrl(string uniqueId, DateTimeOffset expirationDateTime)
{
if (expirationDateTime <= DateTimeOffset.Now)
throw new ArgumentOutOfRangeException(nameof(expirationDateTime), expirationDateTime, null);
var blob = MediaContainer.GetBlockBlobReference(uniqueId);
var sasPolicy = new SharedAccessBlobPolicy
{
SharedAccessExpiryTime = expirationDateTime,
Permissions = SharedAccessBlobPermissions.Create
};
return MediaContainer.Uri + blob.GetSharedAccessSignature(sasPolicy);
}
}
The blog post can be found here: https://forums.xamarin.com/discussion/98784/how-to-load-images-in-xamarin-forms-listview-with-asp-net-web-api-from-sql-azure-database

How to list blobs from virtual directory without prefix?

I have the following list of blobs:
VirtualDirectroy1/VirtualSubDirectory1/Blob1
VirtualDirectroy2/VirtualSubDirectory2/Blob2
VirtualDirectroy3/VirtualSubDirectory3/Blob3
I need to list Blob1, Blob2 and Blob3, so that when accessing the CloudBlockBlob.Name property it returns just Blob1, Blob2 or Blob 3 WITHOUT prefix of virtual directories.
How can I archive this?
Best Wishes, Oleg
If you are using the Azure storage .Net client library (I am using version 3.0.3 in which these methods/overloads are available), you could do something this:
var container = GetBlobContainer();
foreach (var blobItem in container.ListBlobs(useFlatBlobListing: true))
{
Console.WriteLine(blob.Parent.Uri.MakeRelativeUri(blob.Uri));
}
I found you can do this:
var storageAccountConnectionString = "...";
var storageAccount = CloudStorageAccount.Parse(storageAccountConnectionString);
var cloudBlobClient = storageAccount.CreateCloudBlobClient();
var cloudBlobContainer = cloudBlobClient.GetContainerReference("containerName");
foreach (var blob in cloudBlobContainer.ListBlobs())
{
Console.WriteLine(blob.Uri.Segments.Last());
}
class Program
{
const string _conStr = "storage connection string";
const string _container = "container name";
static void Main(string[] args)
{
var names = new Program().GetBlobNames();
Console.ReadKey();
}
private List<string> GetBlobNames()
{
CloudStorageAccount acc = CloudStorageAccount.Parse(_conStr);
CloudBlobClient blobClient = acc.CreateCloudBlobClient();
CloudBlobContainer cntnr = blobClient.GetContainerReference(_container);
List<IListBlobItem> blobList = cntnr.ListBlobs("").ToList();
List<string> flatList = new List<string>();
List<IListBlobItem> blobItems = new List<IListBlobItem>();
foreach (IListBlobItem blobItem in blobList)
{
//If it is cloudblob directory
if (blobItem.GetType() == typeof(CloudBlobDirectory))
{
CloudBlobDirectory dir = blobItem as CloudBlobDirectory;
GetFilesInDirectory(dir, blobItems);
}
}
return blobItems.Select(b => b.Parent.Uri.MakeRelativeUri(b.Uri).ToString()).ToList();
}
private void GetFilesInDirectory(CloudBlobDirectory directory, List<IListBlobItem> fileList)
{
foreach (var blobItem in directory.ListBlobs())
{
if (blobItem.GetType() == typeof(CloudBlockBlob))
{
CloudBlockBlob blob = (CloudBlockBlob)blobItem;
fileList.Add(blob);
}
else if (blobItem.GetType() == typeof(CloudPageBlob))
{
CloudPageBlob blob = (CloudPageBlob)blobItem;
fileList.Add(blob);
}
else if (blobItem.GetType() == typeof(CloudBlobDirectory))
{
//Fetch recursively all the blobs
CloudBlobDirectory blob = (CloudBlobDirectory)blobItem;
GetFilesInDirectory(blob, fileList);
}
}
}
}

Web API : Read the image send from the app

I need to read the file been send from the Ios app to the Azure web api.
I need to upload that to the blob and keep the Uri.
Can anybody suggest me the code to accept the file been send.
/// <summary>
///This is to upload the image file for the user profile.
/// </summary>
///<param name="userRegId"> </param>
///<returns></returns>
[HttpPost]
public Response ImageUpload(int userRegId)
{
try
{
var response = new Response();
if (!Request.Content.IsMimeMultipartContent())
{
response.status = CommonHandler.GetInvalidStatus();
}
else
{
//here i want to read the file and upload it to blob.
string fileName =Request.Files[0].FileName;//unable to do this. Here i want to know ho to read the file
string uniqueBlobName = string.Format("{0}/{1}", constants.pdf, fileName);
CloudBlobClient blobStorage = cloudClasses.CreateOrGetReferenceOfBlobStorage(constants.pdf);
CloudBlockBlob blob = cloudClasses.ClouBblockBlobPropertySetting(blobStorage, uniqueBlobName, ".pdf");
blob.UploadFromStream(Request.Files[0].InputStream);//unable to do this. Here i want to know ho to read the file
response.status = CommonHandler.GetSuccessStatus();
}
return response;
}
catch (Exception ex)
{
_logger.LogError(Log4NetLogger.Category.Exception, "Error in : APIController.ImageUpload :>> Exception message: " + ex.Message);
return new Response { status = CommonHandler.GetFailedStatus(ex.Message) };
}
}
Here is my code,
var streamProvider = new MultipartMemoryStreamProvider();
Request.Content.ReadAsMultipartAsync(streamProvider);
foreach (var content in streamProvider.Contents)
{
if (content != null)
{
if (content.Headers.ContentDisposition.FileName != null)
{
var fileName =
content.Headers.ContentDisposition.FileName.Replace("\"", string.
Empty);
Stream stream = content.ReadAsStreamAsync().Result;
CloudBlobContainer blobStorage = BlobHandler.GetBlobStorage("profileimage");
CloudBlockBlob blob = BlobHandler.BlobPropertySetting(blobStorage,
Guid.NewGuid().ToString().ToLower() +
fileName);
blob.UploadFromStream(stream);
response = ProfileHandler.ImageUpdate(userRegId, blob.Uri);
}
}
}
For the blobHandler
public static CloudBlobContainer GetBlobStorage(string cloudBlobContainerName)
{
CloudBlobContainer container;
try
{
var storageAccount = CloudStorageAccount.FromConfigurationSetting("StorageConnectionString");
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
container = blobClient.GetContainerReference(cloudBlobContainerName); //profile
container.CreateIfNotExist();
var permissions = container.GetPermissions();
permissions.PublicAccess = BlobContainerPublicAccessType.Container;
container.SetPermissions(permissions);
}
catch (Exception ex)
{
Logger.LogError(Log4NetLogger.Category.Exception, "Error in : BlobHandler.GetBlobStorage :>> Exception message: " + ex.Message);
throw;
}
return container;
}
public static CloudBlockBlob BlobPropertySetting(CloudBlobContainer cloudBlobClientReferenceName, string blobContentName)
{
CloudBlockBlob blob = cloudBlobClientReferenceName.GetBlockBlobReference(blobContentName);
//blob.Properties.ContentType = contentType;
return blob;
}
Hope this will help someone..
Yao Huang Lin wrote a blog post on how to produce a Web API file service backed by Azure blob storage. You can find it here.

Resources