'AdlsClient' does not contain a definition for 'CreateDocumentAsync' - azure

TARGET:
Create Console Application, which 1) Read json from Azure Data Lake Store 2) Store data to Cosmos DB as json.
PROBLEM:
I can read file (1), but cannot stores data to Cosmos. See error.
ERROR:
Severity Code Description Project File Line Suppression State
Error CS1061 'AdlsClient' does not contain a definition for 'CreateDocumentAsync' and no extension method 'CreateDocumentAsync' accepting a first argument of type 'AdlsClient' could be found (are you missing a using directive or an assembly reference?) CircleCustomActivityCosmosDB C:\AzureCoding\CustomActivityCosmosDB\Program.cs
CODE:
private async Task CreateDocumentsAsync()
{
string fileName = "/myjsonfile.json";
// Obtain AAD token for ADLS
var creds = new ClientCredential(applicationId, clientSecret);
var clientCreds = ApplicationTokenProvider.LoginSilentAsync(tenantId, creds).GetAwaiter().GetResult();
// Create ADLS client object
AdlsClient client = AdlsClient.CreateClient(adlsAccountFQDN, clientCreds);
String json = "";
//Read file contents
using (var readStream = new StreamReader(client.GetReadStream(fileName)))
{
string line;
while ((line = readStream.ReadLine()) != null)
{
Console.WriteLine("Read file Line: " + line);
json += line;
}
}
//Read file to json
JsonTextReader reader = new JsonTextReader(new StringReader(json));
//Storing json to CosmosDB
Uri collectionUri = UriFactory.CreateDocumentCollectionUri(databaseName, collectionName);
//ERROR HAPPENS HERE - 'AdlsClient' does not contain a definition for 'CreateDocumentAsync' and no extension method 'CreateDocumentAsync'
await client.CreateDocumentAsync(collectionUri, reader);
}
}

There is no CreateDocumentAsync method defined for the class AdlsClient, see the docs
You need a CosmosDB client to create the document there, I think you want to use DocumentClient.CreateDocument instead.
The AdlsClient class gives to methods to access the Azure Data Lake Store only, not the CosmosDB datastore. For that, you need the DocumentClient class, see the docs.
So summarized:
TARGET: Create Console Application, which 1) Read json from Azure Data Lake Store 2) Store data to Cosmos DB as json.
for 1) you need AdlsClient (to access adls)
for 2) you need DocumentClient (to access cosmosdb)

Related

Azure Blob Change Feed missing blob append events

I'm trying out Azure Blob Change Feed feature and it behaves strange to me with Append Blobs: append events are missing in the feed.
My scenario is:
Create storage account, enable change feed feature:
Change feed enabled
Create Append Blob if not exists (1) and appending some input into it (2).
private void WriteBlob(string input)
{
MemoryStream stream = new MemoryStream(Encoding.UTF8.GetBytes(input));
try
{
if (client == null)
{
var credential = new ClientSecretCredential("...", "...");
client = new AppendBlobClient(new Uri("..."), credential);
}
client.CreateIfNotExists(); // (1)
client.AppendBlock(stream); // (2)
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
Fetch Change Feed entries in separate console app.
public static List<BlobChangeFeedEvent> GetChanges()
{
var credential = new ClientSecretCredential("...", "...");
BlobChangeFeedClient blobChangeFeedClient = new BlobChangeFeedClient(new Uri("..."), credential);
List<BlobChangeFeedEvent> events = new List<BlobChangeFeedEvent>();
foreach (BlobChangeFeedEvent changeFeedEvent in blobChangeFeedClient.GetChanges())
{
events.Add(changeFeedEvent);
}
return events;
}
The problem is that after a few runs of WriteBlob method I only get single change feed event that corresponds to the blob creation, and subsequent appends are missing in the feed, however inputs are being appended successfully to the blob resource.
The question is why it is working this way? I didn't find anything special about Append Blob blob type regarding Change feed in docs.
Currently, the append event for an append blob is not supported.
As per this doc, only the following event types are supported:
BlobCreated
BlobDeleted
BlobPropertiesUpdated
BlobSnapshotCreated
And in the source code of Azure.Storage.Blobs.ChangeFeed package, there is no append event type.
A feature request of this is submitted, hope it can be added in the future release.

How to Read a file from Azure Data Lake Storage with file Url?

Is there a way to read files from the Azure Data Lake. I have the Http url of the file. I want to read it direclty. How can i acheive it because I don't see a way to do it via the SDK.
Thanks for your help.
Regards
Did you check docs?
public async Task ListFilesInDirectory(DataLakeFileSystemClient fileSystemClient)
{
IAsyncEnumerator<PathItem> enumerator =
fileSystemClient.GetPathsAsync("my-directory").GetAsyncEnumerator();
await enumerator.MoveNextAsync();
PathItem item = enumerator.Current;
while (item != null)
{
Console.WriteLine(item.Name);
if (!await enumerator.MoveNextAsync())
{
break;
}
item = enumerator.Current;
}
}
You can also use ADLS Gen2 rest api ,
For example, you can write code like below with sas token authentication(or you can also use the shared key authentication):
string sasToken = "?sv=2018-03-28&ss=b&srt=sco&sp=rwdl&st=2019-04-15T08%3A07%3A49Z&se=2019-04-16T08%3A07%3A49Z&sig=xxxx";
string url = "https://xxxx.dfs.core.windows.net/myfilesys1/app.JPG" + sasToken;
var req = (HttpWebRequest)WebRequest.CreateDefault(new Uri(url));
//you can specify the Method as per your operation as per the api doc
req.Method = "HEAD";
var res = (HttpWebResponse)req.GetResponse();
If you know Blob APIs and Data Lake Storage Gen2 APIs can operate on the same data, then you can directly use the azure blob storage SDK to read file from ADLS Gen2.
First, install this nuget package: Microsoft.Azure.Storage.Blob, version 11.1.6.
Note that, in this case, you should use this kind of url "https://xxx.blob.core.windows.net/mycontainer/myfolder/test.txt" instead of that kind of url "https://xxx.dfs.core.windows.net/mycontainer/myfolder/test.txt".
Here is the sample code which is used to read a .txt file in ADLS Gen2:
var blob_url = "https://xxx.blob.core.windows.net/mycontainer/myfolder/test.txt";
//var blob_url = "https://xxx.dfs.core.windows.net/mycontainer/myfolder/test.txt";
var username = "xxxx";
var password = "xxxx";
StorageCredentials credentials = new StorageCredentials(username, password);
var blob = new CloudBlockBlob(new Uri(blob_url),credentials);
var mystream = blob.OpenRead();
using (StreamReader reader = new StreamReader(mystream))
{
Console.WriteLine("Read file content: " + reader.ReadToEnd());
}
//you can also use other method like below
//string text = blob.DownloadText();
//Console.WriteLine($"the text is: {text}");
The test result:

Azure Storage Search Blobs by Metadata

I have CloudBlockBlobs that have metadata.
CloudBlockBlob blockBlob = container.GetBlockBlobReference("myblob.jpg");
using (var fileStream = System.IO.File.OpenRead(filePath))
{
blockBlob.UploadFromStream(fileStream);
blockBlob.Properties.ContentType = "image/jpg";
blockBlob.Metadata.Add("Title", "Yellow Pear");
blockBlob.SetProperties();
}
I see the Metadata is there:
Debug.WriteLine(blockBlob.Metadata["Title"]);
Now later if I query from storage I see the blobs but the Metadata is missing:
(in the below I know blobItems[0] had Metadata when uploaded but now blobItems[0].Metadata.Count == 0)
var blobItems = container.ListBlobs(
null, false, BlobListingDetails.Metadata);
I also noticed the Metadata is not available when I obtain the blob by itself:
CloudBlockBlob a = container.GetBlockBlobReference("myblob.jpg");
//Below throws an exception
var b = a.Metadata["Title"];
Thank you!
There are some issues with your code :(.
The blob doesn't have any metadata set actually. After setting the metadata, you're calling blob.SetProperties() method which only sets the blob's properties (ContentType in your example). To set the metadata, you would actually need to call blob.SetMetadata() method.
Your upload code is currently making 2 calls to storage service: 1) upload blob and 2) set properties. If you call SetMetadata then it would be 3 calls. IMHO, these can be combined in just 1 call to storage service by doing something like below:
using (var fileStream = System.IO.File.OpenRead(filePath))
{
blockBlob.Properties.ContentType = "image/jpg";
blockBlob.Metadata.Add("Title", "Yellow Pear");
blockBlob.UploadFromStream(fileStream);
}
This will not only upload the blob but also set it's properties and metadata in a single call to storage service.
Regarding
I also noticed the Metadata is not available when I obtain the blob by
itself:
CloudBlockBlob a = container.GetBlockBlobReference("myblob.jpg");
//Below throws an exception
var b = a.Metadata["Title"];
Basically the code above is just creating an instance of the blob on the client side. It doesn't actually fetch the properties (and metadata) of the blob. To fetch details about the blob, you would need to call FetchAttributes method on the blob. Something like:
CloudBlockBlob a = container.GetBlockBlobReference("myblob.jpg");
a.FetchAttributes();
If after that you retrieve blob's metadata, you should be able to see it (provided metadata was created properly).

Windows Azure Blob

I've been trying to create a Windows Azure Blob containing an image file. I followed these tutorials: http://www.nickharris.net/2012/11/how-to-upload-an-image-to-windows-azure-storage-using-mobile-services/ and http://www.windowsazure.com/en-us/develop/mobile/tutorials/upload-images-to-storage-dotnet/. Finally the following code represents a merging of them. On the last line, however, an exception is raised:
An exception of type 'System.TypeLoadException' occurred in
mscorlib.ni.dll but was not handled in user code
Additional information: A binding for the specified type name was not
found. (Exception from HRESULT: 0x80132005)
Even the container is created the table, but It doesn't work properly.
private async void SendPicture()
{
StorageFile media = await StorageFile.GetFileFromPathAsync("fanny.jpg");
if (media != null)
{
//add todo item to trigger insert operation which returns item.SAS
var todoItem = new Imagem()
{
ContainerName = "mypics",
ResourceName = "Fanny",
ImageUri = "uri"
};
await imagemTable.InsertAsync(todoItem);
//Upload image direct to blob storage using SAS and the Storage Client library for Windows CTP
//Get a stream of the image just taken
using (var fileStream = await media.OpenStreamForReadAsync())
{
//Our credential for the upload is our SAS token
StorageCredentials cred = new StorageCredentials(todoItem.SasQueryString);
var imageUri = new Uri(todoItem.SasQueryString);
// Instantiate a Blob store container based on the info in the returned item.
CloudBlobContainer container = new CloudBlobContainer(
new Uri(string.Format("https://{0}/{1}",
imageUri.Host, todoItem.ContainerName)), cred);
// Upload the new image as a BLOB from the stream.
CloudBlockBlob blobFromSASCredential =
container.GetBlockBlobReference(todoItem.ResourceName);
await blobFromSASCredential.UploadFromStreamAsync(fileStream.AsInputStream());
}
}
}
Please use Assembly Binding Log Viewer to see which load is failing. As also mentioned in the article, the common language runtime's failure to locate an assembly typically shows up as a TypeLoadException in your application.

Azure Storage CloudBlob.Properties are not initialized when using GetBlobReference()

I'm trying to get some information about Azure blob (last modified UTC date time). This information is stored CloudBlob.Properties.LastModifiedUtc property.
If I use method GetBlobReference() or GetBlockBlobReference(), the Properties of the blob are not initialized (LastModifiedUtc is DateTime.MinDate). If I use ListBlobs() the Properties are initialized correctly (LastModifiedUtc has correct value).
Am I doing something wrong when using GetBlobReference function? Is there some way how to get CloudBlob instance just for one specific blob? I know I can missue ListBlobs() and filter just the blob I'm interested in, or use ListBlobsWithPrefix() from class CloudBlobClient, but I would expect to get all the metadata when I ask for specific Blob Reference.
Code showing how I'm working with Azure blobs:
string storageAccountName = "test";
string storageAccountKey = #"testkey";
string blobUrl = "https://test.blob.core.windows.net";
string containerName = "testcontainer";
string blobName = "testbontainer";
var credentials = new StorageCredentialsAccountAndKey(storageAccountName, storageAccountKey);
var cloudBlobClient = new CloudBlobClient(blobUrl, credentials);
var containerReference = cloudBlobClient.GetContainerReference(string.Format("{0}/{1}", blobUrl, containerName));
// OK - Result is of type CloudBlockBlob, cloudBlob_ListBlobs.Properties.LastModifiedUtc > DateTime.MinValue
var cloudBlob_ListBlobs = containerReference.ListBlobs().Where(i => i is CloudBlob && ((CloudBlob)i).Name == blobName).FirstOrDefault() as CloudBlob;
// WRONG - Result is of type CloudBlob, cloudBlob_GetBlobReference.Properties.LastModifiedUtc == DateTime.MinValue
var cloudBlob_GetBlobReference = containerReference.GetBlobReference(string.Format("{0}/{1}/{2}", blobUrl, containerName, blobName));
// WRONG - Result is of type CloudBlockBlob, cloudBlob_GetBlockBlobReference.Properties.LastModifiedUtc == DateTime.MinValue
var cloudBlob_GetBlockBlobReference = containerReference.GetBlockBlobReference(string.Format("{0}/{1}/{2}", blobUrl, containerName, blobName));
I believe you have to make a seperate call to fetch the attributes/metadata. After you have the blob referrence, issue the following line to retrieve the attributes.
cloudBlob_GetBlobReference.FetchAttributes();
This relates to the Java SDK. But having a CloudBlob derived CloudBlockBlob object, you may need the CloudBlob.downloadAttributes() call.

Resources