Hi Guys I am building a Client which Interact with Azure Storage Rest API.
I was going through documentation https://learn.microsoft.com/ru-ru/rest/api/storageservices/fileservices/list-containers2:
And didn't understood the use of parameter prefix and marker which can be send along with Azure request.
It says:
prefix
Optional. Filters the results to return only containers whose name
begins with the specified prefix.
marker
Optional. A string value that identifies the portion of the list of
containers to be returned with the next listing operation. The
operation returns the NextMarker value within the response body if the
listing operation did not return all containers remaining to be listed
with the current page. The NextMarker value can be used as the value
for the marker parameter in a subsequent call to request the next page
of list items.
The marker value is opaque to the client.
With Prefix, I think:
If i have dir structure:
file01.txt
images/image01.jpg
images/folder/image001.jpg
fightVideo/subFolder/current/video001.mpg
fightVideo/subFolder/current/video002.mpg
If I give prefix container name as "fight". It should return
fightVideo.
But I am not sure.
And for Marker I don't understand whats its use?
Please can someone explain the use of Prefix and Marker with examples?
In context of listing containers, if you specify prefix parameter it will list the containers names of which start with that prefix value. It has nothing to do with listing blobs.
List blobs operation also supports this prefix parameter and when you specify this parameter, it will list the blobs names of which start with that prefix value.
So the example you have given is for listing blobs and when you specify flight as prefix there, you will get back fightVideo/subFolder/current/video001.mpg and fightVideo/subFolder/current/video002.mpg in response but not when you call list containers with this prefix.
Regarding marker, Kalyan's explanation is correct but let me add a little bit more to that.
Essentially Azure Storage Service is a shared service and you simply can't ask it to return all the results in one go (if we were to take an analogy from SQL world, you simply can't do SELECT * FROM TABLE kind of thing). Each request to the service is assigned a predefined timeout and the response would include the items fetched in that time + optionally a token if the service thinks that there's more data available. This token is called continuation token. In order to get the next set of items, you would need to pass this continuation token in the marker parameter in your next request.
Each call to storage service will try to return a predefined maximum number of items. For listing blob containers/blobs, this limit is 5000 items. For listing tables/entities, this limit is 1000 items. If there are more items in your account, then apart from this data storage service returns you a continuation token which tells you that there's more data available.
Please note that even though the limit is there but you can't always assume that you will get these number of records. Based on a number of conditions, it is quite possible that you don't get back any data but still receive a continuation token. So your code need to handle this condition as well.
If there are too many blobs to be listed, then the response contains the NextMarker element.
<?xml version="1.0" encoding="utf-8"?>
<EnumerationResults ServiceEndpoint="https://myaccount.blob.core.windows.net">
<Prefix>string-value</Prefix>
<Marker>string-value</Marker>
<MaxResults>int-value</MaxResults>
<Containers>
<Container>
<Name>container-name</Name>
<Properties>
<Last-Modified>date/time-value</Last-Modified>
<Etag>etag</Etag>
<LeaseStatus>locked | unlocked</LeaseStatus>
<LeaseState>available | leased | expired | breaking | broken</LeaseState>
<LeaseDuration>infinite | fixed</LeaseDuration>
<PublicAccess>container | blob</PublicAccess>
</Properties>
<Metadata>
<metadata-name>value</metadata-name>
</Metadata>
</Container>
</Containers>
<NextMarker>marker-value</NextMarker>
</EnumerationResults>
The REST API documentation mentions that the marker value can be used in a subsequent call to request the next set of list items.
You can imagine marker as a paginatator index.
Related
I called below AZURE API from postman.
https://management.azure.com/subscriptions/{SubscriptionID}/providers/Microsoft.Commerce/UsageAggregates?api-version=2015-06-01-preview&reportedstartTime=2019-12-29T00%3a00%3a00%2b00%3a00&reportedEndTime=2019-12-30T00%3a00%3a00%2b00%3a00&$top=1
I got a response with empty value field along with some nextlink. when I again called the API with Nextlink URL response was having blank value field.
{
"value": [],
"nextLink": somelink
}
I am able to get proper response using same API for some different subscription.
Empty Array shows that it doesn;t have any usage details available for requested time frame.
Firstly as suggested by Tony, check if you have the usage resource for the requested date.
**Important**
Please note that the dateTime value format must be URL encoded as ISO 8601 format, and non-numeric characters must use escape codes (i.e. colon is escaped to %3a, plus sign is escaped to %2b) so that it is URI friendly. These refer to the start and end time ranges of your query. This dateTime parameter must also be specified in Universal Time Coordinated (UTC).
Secondly , Set {aggregationGranularity} to ‘Hourly’. This is an optional parameter with two discrete potential values: Daily and Hourly. As the values suggest, the former one returns the data in daily granularity whereas the latter one is hourly resolution. Daily is the default.
Sample URI:
**https://management.azure.com/subscriptions/{subscription-Id}/providers/Microsoft.Commerce/UsageAggregates?api-version=2015-06-01-preview\&reportedStartTime=2014-05-01T00%3a00%3a00%2b00%3a00\&reportedEndTime=2015-06-01T00%3a00%3a00%2b00%3a00\&aggregationGranularity=Hourly\&showDetails=f**
Hope it helps.
Make sure that you have enough permissions for the subscription in question.
Empty result [] can be returned when the permissions are missing. There is no error shown in such case.
The easiest way to ensure the permissions are available is to use Contributor or Owner role.
Additionally, Get-UsageAggregates -Debug (from Az PowerShell module) can be used to check how the request URL should look like.
Get-UsageAggregates -ReportedStartTime "2022-05-24 10:00" -ReportedEndTime "2022-05-24 15:00" -Debug -AggregationGranularity Hourly
I currently have a Timer triggered Azure Function that checks a data endpoint to determine if any new data has been added. If new data has been added, then I generate an output blob (which I return).
However, returning output appears to be mandatory. Whereas I'd only like to generate an output blob under specific conditions, I must do it all of the time, clogging up my storage.
Is there any way to generate output only under specified conditions?
If you have the blob output binding set to your return value, but you do not want to generate a blob, simply return null to ensure the blob is not created.
You're free to execute whatever logic you want in your functions. You may need to remove the output binding from your function (this is what is making the output required) and construct the connection to blob storage in your function instead. Then you can conditionally create and save the blob.
I'm writing a REST api in node js that will execute a sql query and send the results;
in the request I need to send the WHERE conditions; ex:
GET 127.0.0.1:5007/users //gets the list of users
GET 127.0.0.1:5007/users
id = 1 //gets the user with id 1
Right now the conditions are passed from the client to the rest api in the request's headers.
In the API I'm using sequelize, an ORM that needs to receive WHERE conditions in a particular form (an object); ex: having the condition:
(x=1 AND (y=2 OR z=3)) OR (x=3 AND y=1)
this needs to be formatted as a nested object:
-- x=1
-- AND -| -- y=2
| -- OR ----|
| -- z=3
-- OR -|
|
| -- x=3
-- AND -|
-- y=1
so the object would be:
Sequelize.or (
Sequelize.and (
{x=1},
Sequelize.or(
{y=2},
{z=3}
)
),
Sequelize.and (
{x=3},
{y=1}
)
)
Now I'm trying to pass a simple string (like "(x=1 AND (y=2 OR z=3)) OR (x=3 AND y=1)"), but then I will need a function on the server that can convert the string in the needed object (this method in my opinion has the advantage that the developer writing the client, can pass the where conditions in a simple way, like using sql, and this method is also indipendent from the used ORM, with no need to change the client if we need to change the server or use a different ORM);
The function to read and convert the conditions' string into an object is giving me headache (I'm trying to write one without success, so if you have some examples about how to do something like this...)
What I would like to get is a route capable of executing almost any kind of sql query and give the results:
now I have a different route for everything:
127.0.0.1:5007/users //to get all users
127.0.0.1:5007/users/1 //to get a single user
127.0.0.1:5007/lastusers //to get user registered in the last month
and so on for the other tables i need to query (one route for every kind of request I need in the client);
instead I would like to have only one route, something like:
127.0.0.1:5007/request
(when calling this route I will pass the table name and the conditions' string)
Do you think this solution would be a good solution or you generally use other ways to handle this kind of things?
Do you have any idea on how to write a function to convert the conditions' string into the desired object?
Any suggestion would be appreciated ;)
I would strongly advise you not to expose any part of your database model to your clients. Doing so means you can't change anything you expose without the risk of breaking the clients. One suggestion as far as what you've supplied is that you can and should use query parameters to cut down on the number of endpoints you've got.
GET /users //to get all users
GET /users?registeredInPastDays=30 //to get user registered in the last month
GET /users/1 //to get a single user
Obviously "registeredInPastDays" should be renamed to something less clumsy .. it's just an example.
As far as the conditions string, there ought to be plenty of parsers available online. The grammar looks very straightforward.
IMHO the main disadvantage of your solution is that you are creating just another API for quering data. Why create sthm from scratch if it is already created? You should use existing mature query API and focus on your business logic rather then inventing sthm new.
For example, you can take query syntax from Odata. Many people have been developing that standard for a long time. They have already considered different use cases and obstacles for query API.
Resources are located with a URI. You can use or mix three ways to address them:
Hierarchically with a sequence of path segments:
/users/john/posts/4711
Non hierarchically with query parameters:
/users/john/posts?minVotes=10&minViews=1000&tags=java
With matrix parameters which affect only one path segment:
/users;country=ukraine/posts
This is normally sufficient enough but it has limitations like the maximum length. In your case a problem is that you can't easily describe and and or conjunctions with query parameters. But you can use a custom or standard query syntax. For instance if you want to find all cars or vehicles from Ford except the Capri with a price between $10000 and $20000 Google uses the search parameter
q=cars+OR+vehicles+%22ford%22+-capri+%2410000..%2420000
(the %22 is a escaped ", the %24 a escaped $).
If this does not work for your case and you want to pass data outside of the URI the format is just a matter of your taste. Adding a custom header like X-Filter may be a valid approach. I would tend to use a POST. Although you just want to query data this is still RESTful if you treat your request as the creation of a search result resource:
POST /search HTTP/1.1
your query-data
Your server should return the newly created resource in the Location header:
HTTP/1.1 201 Created
Location: /search/3
The result can still be cached and you can bookmark it or send the link. The downside is that you need an additional POST.
I have a continuous Azure WebJob that is running off of a QueueInput, generating a report, and outputting a file to a BlobOutput. This job will run for differing sets of data, each requiring a unique output file. (The number of inputs is guaranteed to scale significantly over time, so I cannot write a single job per input.) I would like to be able to run this off of a QueueInput, but I cannot find a way to set the output based on the QueueInput value, or any value except for a blob input name.
As an example, this is basically what I want to do, though it is invalid code and will fail.
public static void Job([QueueInput("inputqueue")] InputItem input, [BlobOutput("fileoutput/{input.Name}")] Stream output)
{
//job work here
}
I know I could do something similar if I used BlobInput instead of QueueInput, but I would prefer to use a queue for this job. Am I missing something or is generating a unique output from a QueueInput just not possible?
There are two alternatives:
Use IBInder to generate the blob name. Like shown in these samples
Have an autogenerated in the queue message object and bind the blob name to that property. See here (the BlobNameFromQueueMessage method) how to bind a queue message property to a blob name
Found the solution at Advanced bindings with the Windows Azure Web Jobs SDK via Curah's Complete List of Web Jobs Tutorials and Videos.
Quote for posterity:
One approach is to use the IBinder interface to bind the output blob and specify the name that equals the order id. The better and simpler approach (SimpleBatch) is to bind the blob name placeholder to the queue message properties:
public static void ProcessOrder(
[QueueInput("orders")] Order newOrder,
[BlobOutput("invoices/{OrderId}")] TextWriter invoice)
{
// Code that creates the invoice
}
The {OrderId} placeholder from the blob name gets its value from the OrderId property of the newOrder object. For example, newOrder is (JSON): {"CustomerName":"Victor","OrderId":"abc42"} then the output blob name is “invoices/abc42″. The placeholder is case-sensitive.
So, you can reference individual properties from the QueueInput object in the BlobOutput string and they will be populated correctly.
I am storing image files as blobs in Azure Storage with the following naming convention:
directory/image-name
When trying to retrieve the blobs using BlobService.listBlobs(container, options, callback) in Javascript on the server, I use:
var options = { "prefix":directory }
and it gets back only blobs that start with the directory name, as I expect, but I thought I would also be able to use:
var options = { "delimiter":"/", "prefix":directory }
and get back the same blobs, perhaps without the prefix in their names. Instead I get back nothing at all. What is the correct way to use the delimiter? What's the point in having it if you get the items that you want with only using the prefix?
I've not used the REST APIs from JavaScript, but I think what you are missing is a trailing slash after the directory name, so I suggest:
var options = { "delimiter":"/", "prefix":directory+"/" }
Windows Azure Storage doesn't really have directories, in the underlying implementation all the blobs in a container are just flat list, and blob names (not container names) may contain slashes. The delimiter is an option when calling the ListBlobs REST API that allows you to simulate directory-like behavior. If the delimiter option is enabled, and the part of the blob name past the prefix contains the delimiter, the reply will omit that blob.
To illustrate, lets name some blobs, assuming all of them in the same container https://myaccount.blob.core.windows.net/mycontainer":
a/b/extra.txt
a/bloba.txt
a/blobb.txt
other.txt
So then if you invoke listBlobs on that container with the prefix "a/" and without specifying the delimiter, it will return the first three names, because they all have the "a/" prefix.
If instead you invoke listBlobs with the same "a/" prefix and set the delimiter to "/", you only get the middle two names; the service leaves out a/b/extra.txt because it's in a (simulated) sub-directory "b".