So I'm trying to do something I expected to be simple - using Logic App to insert a json object into Cosmos DB.
I've created a CosmosDB (based on Core SQL API) and created a container called Archive with the partition key /username (which is in my json object).
Now to Logic App.
First, I YOLO'ed the simple approach.
Which gave me error: "The input content is invalid because the required properties - 'id; ' - are missing"
But my json object doesnt have an id field. Maybe use the IsUpsert parameter? I dont feel like manipulating my input document to add a field called 'id'.
Which gave me error: "One of the specified inputs is invalid" Okay - feels even worse.
I now tried to use the other logic app connector (Not V2, but the original).
which gave me error: "The partition key supplied in x-ms-partitionkey header has fewer components than defined in the the collection."
I saw that this connector(unlike the V2 one) has a Partition Key Value parameter from UI, which I added to pass the value of my username
which gave me error "The input content is invalid because the required properties - 'id; ' - are missing".
At this point I thought, let me just give the bloody machine what it wants, and so I added "id" to my json object.
and yes that actually worked.
So questions are
With Logic Apps connectors, are you only able to insert json objects into Cosmos DB without that magic field "id" in the message payload?
If the partitioning key is required. Why is it not available from the V2 connector parameter UI?
Thanks.
The v1 version of this thing doesn't have partition key because it's so old, Cosmos didn't have partitioned containers. You will want to use the v2 preview. I think that will go GA at some point here soon because it's been in preview for a while now.
And yes, With Cosmos DB you can't insert anything without it's id so you will need to generate from whatever calls your endpoint and pass it in the body of your http request.
Related
While calling user delta API(beta version) for Azure
https://learn.microsoft.com/en-gb/graph/api/user-list?view=graph-rest-beta&tabs=http
I am getting the following fields,
"assignedPlans"
"assignedLicenses"
"licenseAssignmentStates". These fields are changing way too frequently because of which delta API keeps on getting more and more pages even for a small number of users.
https://graph.microsoft.com/beta/users/delta
I wanted to know what does these fields mean, however I couldn't find it in the API documentation
List Users operation returns a list of user resource type. You can find the properties of a user resource type here: https://learn.microsoft.com/en-gb/graph/api/resources/user?view=graph-rest-beta#properties.
Your read about the resources you mentioned in your question here:
assignedPlans: https://learn.microsoft.com/en-gb/graph/api/resources/assignedplan?view=graph-rest-beta.
assignedLicenses: https://learn.microsoft.com/en-gb/graph/api/resources/assignedlicense?view=graph-rest-beta
licenseAssignmentStates: https://learn.microsoft.com/en-gb/graph/api/resources/licenseassignmentstate?view=graph-rest-beta
I am using an Azure Function kicked by an HTTPTrigger. In that function, I am using the CosmosClient from azure-cosmos.
client = CosmosClient(endpoint, key)
database = client.get_database_client(database-name)
container = database.get_container_client(container-name)
container.create_item(body=req_body)
With the last line, I am trying to create an item with the object that I receive in the body of the HTTP request. I thought that Cosmos DB would automatically generate the id of the item. However, I am getting the following error:
Exception: CosmosHttpResponseError: (BadRequest) Message: {"Errors":["The input content is invalid because the required properties - 'id; ' - are missing"]}
Do I really need to assign an id to the item or am I missing anything? According to the documentation, the id can either be "System-generated or user-configurable"
id is a required attribute. Some older SDKs had the option to autogenerate it as a Guid, but newer SDKs opt for transparency and giving the user the choice explicitly. You could use a Guid or any other value that is relevant for you.
I am creating an extensive data factory work flow that will create and fill a data warehouse for multiple customers automatic, however i'm running into an error. I am going to post the questions first, since the remaining info is a bit long. Keep in mind i'm new to data factory and JSON coding.
Questions & comments
How do i correctly pass the parameter through to an Execute Pipeline activity?
How do i add said parameter to an Azure Function activity?
The issue may lie with correctly passing the parameter through, or it may lie in picking it up - i can't seem to determine which one. If you spot an error with the current setup, dont hesitate to let me know - all help is appreciated
The Error
{
"errorCode": "BadRequest",
"message": "Operation on target FetchEntries failed: Call to provided Azure function
'' failed with status-'BadRequest' and message -
'{\"Message\":\"Please pass 'customerId' on the query string or in the request body\"}'.",
"failureType": "UserError",
"target": "ExecuteFullLoad"
}
The Setup:
The whole setup starts with a function call to get new customers from an online economic platform. It the writes them to a SQL table, from which they are processed and loaded into the final table, after which a new pipeline is executed. This process works perfectly. From there the following pipeline is executed:
As you can see it all works well until the ForEach loop tries to execute another pipeline, that contains an azure function that calls a .NET scripted function that fills said warehouse (complex i know). This azure function needs a customerid to retrieve tokens and load the data into the warehouse. I'm trying to pass those tokens from the InternalCustomerID lookup through the ForEach into the pipeline and into the function. The ForEach works actually, but fails "Because an inner activity failed".
The Execute Pipeline task contains the following settings, where i'm trying to pass the parameter through which comes from the foreach loop. This part of the process also works, since it executes twice (as it should in this test phase):
I dont know if it doesn't successfully pass the parameter through or it fails at adding it to the body of the azure function.
The child pipeline (FullLoad) contains the following parameters. I'm not sure if i should set a default value to be overwritten or how that actually works. The guides i've look at on the internet havent had a default value.
Finally there is the settings for the Azure function. I'm not sure what i need to write in order to correctly capture the parameter and/or what to fill in - if it's the header or the body regarding the error message. I know a post cannot be executed without a body.
If i run this specific funtion by hand (using the Function App part of portal.azure.com) it works fine, by using the following settings:
I viewed all of your detailed question and I think the key of the issue is the format of Azure Function Request Body.
I'm afraid this is incorrect. Please see my below steps based on your description:
Work Flow:
Inside ForEach Activity, only one Azure Function Activity:
The preview data of LookUp Activity:
Then the configuration of ForEach Activity: #activity('Lookup1').output.value
The configuration of Azure Function Activity: #json(concat('{"name":"',item().name,'"}'))
From the azure function, I only output the input data. Sample Output as below:
Tips: I saw your step is executing azure function in another pipeline and using Execute Pipeline Activity, (I don't know why you have to follow such steps), but I think it doesn't matter because you only need to focus on the Body format, if your acceptable format is JSON, you could use #json(....),if the acceptable format is String, you could use #cancat(....). Besides, you could check the sample from the ADF UI portal which uses pipeline().parameters
This is my scenario:
I want to have one single collection where I can insert and query documents using SqlApi and create vertices and edges using Graph Api, all in the same collection.
I believed that that was possible taken into account what I've read on the documentations.
My first try was using Microsoft.Azure.Graphs.dll
With this approach I was able to do CRUD operations with the Sql Api and execute gremlin scripts against the collection.
It is important to note that the documents created with the Sql api Insert command where used as vertices. Then I created edges using the graph api connecting the documents created with the Sql Api. This works as expected.
The only problem I had was that if the document contains an array property the graph Api returned and error : Invalid cast from 'System.String' to 'Newtonsoft.Json.Linq.JObject'.
After investigating I was told that the Microsoft.Azure.Graphs.dll was deprecated and I should use Gremlin.Net instead.
I have not read in any place that this assembly is deprecated and even in the official documentation and examples I can see that this assembly is being used.
Is it really deprecated? Should I not use it?
Then this was my second try:
I moved to Gremlin.net.
First issue: I was connecting to a collection created originally for Sql Api. When I tried to connect with Gremlin.Net Client it tolds me that can not connect to the server.
When I created another database and collection for graph Api I was able to connect.
Conclusion: It's not possible to use Gremlin.net with a collection created with Sql Api.
Or is it possible to activate the gremlin endpoint in a database with Sql Api?
Using the new collection for Graph Api I tried again to create documents with the Sql Api and then connect using Graph Api. I see that in this case both endpoints are created: SqlApi+Gremlin.
I had a few problems making this works. For example I had to set the GraphSon writer and reader to version 2, if not I received a null reference exception.
Second issue: I was able to create documents with Sql Api but I had the same problem with the array property (Example Document { "id":"aaa" "Prop": [ "1", "2"] } )
I get the same error when I query with gremlin: g.V('aaa') => Invalid cast from 'System.String' to 'Newtonsoft.Json.Linq.JObject'.
Conclusion: My first issue with previous library was not solved with the new one.
Third issue: The json returned when I query with SqlApi the edges or vertices are different from received when I used Microsoft.Azure.Graphs.dll. It looks like the cosmos db engine handles differently the gremlin scripts depending on the assembly. Which the json format I should expect?
Notes:
-Why I need to create vertices using SqlApi?
Because I want to create properties with custom objects and I'm not able to do it with graphApi:
Example: { "Id": "aaa", "Custom" : { "Id": 456, "Code" : { "Id": 555, "Name": "Hi"} }
-Why I want to query graph using SqlApi?
Because I want to access my custom properties.
Because I want to paginate using tokens. (Range gremlin function it is not performant. It traverse all the vertex/edges from 0 to the last page I want)
Has someone some information about this situations?
Help will be appreciated.
I know this post is a bit older, but it looks like it is possible to use graph and sql api. The key is in adopting the gremlin's underlying storage format when using the sql api for querying/manipulating. Please check this post out:
https://vincentlauzon.com/2017/09/05/hacking-accessing-a-graph-in-cosmos-db-with-sql-documentdb-api/
I'm using DocumentDB Data Migration Tool to migrate a documentDB db to a newly created documentDB db. The connectionStrings verify say it is ok.
It doesn't work (no data transferred (=0) but not failure written in the log file (Failed = 0).
Here is what is done :
I've tried many things such as :
migrate / transfer a collection to a json file
migrate to partitionned / non partitionned documentdb db
for the target indexing policy I've taken the source indexing policy (json got from azure, documentdb db collection settings).
...
Actually nothing's working, but I have no error logs, maybe a problem of documentdb version ?
Thanx in advance for your help.
After debugging the solution from the tool's repo I figure the tools fail silently if you mistyped the database's name like I did.
DocumentDBClient just returns an empty async enumerator.
var database = await TryGetDatabase(databaseName, cancellation);
if (database == null)
return EmptyAsyncEnumerator<IReadOnlyDictionary<string, object>>.Instance;
I can import from an Azure Cosmos DB DocumentDB API collection using DocumentDB Data Migration tool.
Besides, based on my test, if the name of the collection that we specify for Source DocumentDB is not existing, no data will be transferred and no error logs is written.
Import result
Please make sure the source collection that you specified is existing. And if possible, you can try to create a new collection and import data from this new collection, and check if data can be transferred.
I've faced same problem and after some investigation found that internal document structure was changed. Therefor after migration with with tool documents are present but couldn't be found with data explorer (but with query explorer using select * they are visible)
I've migrated collection through mongo api using Mongichef
#fguigui: To help troubleshoot this, could you please re-rerun the same data migration operation using the command line option? Just launch dt.exe from the same folder as Data Migration Tool for syntax required. Then after you launch it with required parameters, please paste the output here and I'll take a look what's broken.