I need to use "GROUP BY" clause on Azure Data Explorer but I think it is unsupported.
Someone have any idea to solve or avoid group by?
Best regards,
Finally, Azure Cosmos DB currently supports GROUP BY in .NET SDK 3.3 or later. Support for other language SDK's and the Azure Portal is not currently available but is planned.
<group_by_clause> ::= GROUP BY <scalar_expression_list>
<scalar_expression_list> ::=
<scalar_expression>
| <scalar_expression_list>, <scalar_expression>
There is no group by in Azure Data Explorer as of now. However, there is the summarize operator that can help achieve many of the things that you would use GROUP BY for in SQL.
You can find it at https://learn.microsoft.com/en-us/azure/kusto/query/summarizeoperator
Cosmos DB doesn't support group by feature,you could vote up this if you have urgent need.
Provide a third-party package documentdb-lumenize for your reference here which supports group by feature,it has .net example:
string configString = #"{
cubeConfig: {
groupBy: 'state',
field: 'points',
f: 'sum'
},
filterQuery: 'SELECT * FROM c'
}";
Object config = JsonConvert.DeserializeObject<Object>(configString);
dynamic result = await client.ExecuteStoredProcedureAsync<dynamic>("dbs/db1/colls/coll1/sprocs/cube", config);
Console.WriteLine(result.Response);
You could group by assetId column and get the max timestamp.
Besides,you could refer to my previous case:how to count distinct value in cosmos DB to use stored procedure to implement some aggregation features.
Related
Is there a simple way to find out all the tables referenced in a stored procedure in azure analytics data warehouse other than parsing the stored procedure code? I tried few commands like sp_tables, sp_depends but none seems to be working in azure data warehouse.
sys.sql_expression_dependencies is supported in Azure Synapse Analytics, dedicated SQL pools, but only supports tables, views and functions at this time. A simple example:
SELECT * FROM sys.sql_expression_dependencies;
So you are left either parsing sys.sql_modules. Something like this is imperfect (ie doesn't deal with schema name, square brackets, partial matches etc) but could server as a starting point:
SELECT
sm.[definition],
OBJECT_SCHEMA_NAME(t.object_id) schemaName,
OBJECT_NAME(t.object_id) tableName
FROM sys.sql_modules sm
CROSS JOIN sys.tables t
WHERE sm.object_id = OBJECT_ID('dbo.usp_test')
AND sm.[definition] Like '%' + t.name + '%';
I actually use SQL Server Data Tools (SSDT) with dedicated SQL pools so your dependencies can't get out of step and are trackable via the project.
I'm working with the Azure Core API. Using the web based data explorer, I'm able to successfully run a query that has multiple aggregations across multiple partitions. Yet when I attempt to run such a query from my shell using the Azure client library for Python (pip install azure-cosmos==4.0.0), I get an error message. I've tried two variations of the query, where one included the partition key and one didn't. Both queries returned the same error message.
container = database.get_container_client('some_container')
query1 = "select c.fmonth, c.fquarter, c.fyear, sum(c.revenue) as actual_revenue__sum, sum(c.predicted_revenue_m1) as predicted_revenue__sum from c where c.fyear=2020 group by c.fmonth, c.fquarter, c.fyear"
query2 = "select c.fmonth, c.fquarter, c.fyear, sum(c.revenue) as actual_revenue__sum, sum(c.predicted_revenue_m1) as predicted_revenue__sum from c where c.date_start >='2020-01-01' and c.date_start < '2021-01-01' group by c.fmonth, c.fquarter, c.fyear"
res = container.query_items(query1, enable_cross_partition_query=True)
Error Message Returned:
CosmosHttpResponseError: (BadRequest) Message: {"Errors":["Cross partition query only supports 'VALUE ' for aggregates."]}
ActivityId: 4961b99e-7032-4eac-ae84-2c8cab03a496, Microsoft.Azure.Documents.Common/2.11.0
Errors out for aggregates on multiple partitions, with enable cross
partition query set to true, but no "value" keyword present
If you want to use aggregates on multiple partitions,select value count(1) from c,this will work.
But python sdk doesn't support group by now.
Refer to this document,.net sdk and js sdk support group by.Other sdk will support later.
Hope this can help you.
I am trying to get the count of all records present in cosmos db in a lookup activity of azure data factory. I need this value to do a comparison with other value activity outputs.
The query I used is SELECT VALUE count(1) from c
When I try to preview the data after inserting this query I get an error saying
One or more errors occurred. Unable to cast object of type
'Newtonsoft.Json.Linq.JValue' to type 'Newtonsoft.Json.Linq.JObject'
as shown in the below image:
snapshot of my azure lookup activity settings
Could someone help me in resolving this error and if this is the limitation of azure data factory how can I get the count of all the rows of the cosmos db document using some other ways inside azure data factory?
I reproduce your issue on my side exactly.
I think the count result can't be mapped as normal JsonObject. As workaround,i think you could just use Azure Function Activity(Inside Azure Function method ,you could use SDK to execute any sql as you want) to output your desired result: {"number":10}.Then bind the Azure Function Activity with other activities in ADF.
Here is contradiction right now:
The query sql outputs a scalar array,not other things like jsonObject,or even jsonstring.
However, ADF Look Up Activity only accepts JObject,not JValue. I can't use any convert built-in function here because the query sql need to be produced with correct syntax anyway. I already submitted a ticket to MS support team,but get no luck with this limitation.
I also tried select count(1) as num from c which works in the cosmos db portal. But it still has limitation because the sql crosses partitions.
So,all i can do here is trying to explain the root cause of issue,but can't change the product behaviours.
2 rough ideas:
1.Try no-partitioned collection to execute above sql to produce json output.
2.If the count is not large,try to query columns from db and loop the result with ForEach Activity.
You can use:
select top 1 column from c order by column desc
I want to retrieve data from Cosmos DB with the following query:
SELECT * FROM c WHERE c.pi like '09%001'
(This is a SQL query, which I can use in MySQL)
Here, pi is a string value, which can be 09001001 or 09025001.
Is there a way to use a LIKE command in Cosmos DB?
I know that cosmos DB uses CONTAINS, but this cannot be used when you want to match specifically the beginning or end of the string.
UPDATE :
You can now use the LIKE keyword to do text searches in Azure Cosmos DB SQL (core) API!
EXAMPLE:
SELECT *
FROM c
WHERE c.description LIKE "%cereal%"
OLD Answer:
This can be achieved in 2 ways
(i) Currently Azure Cosmosdb supports the CONTAINS, STARTSWITH, and ENDSWITH built-in functions which are equivalent to LIKE.
The keyword for LIKE in Cosmosdb is Contains .
SELECT * FROM c WHERE CONTAINS(c.pi, '09')
So, in your case if you want to match the pattern 09%001, you need to use:
SELECT * FROM c WHERE STARTSWITH(c.pi, '09') AND ENDSWITH(c.pi, '001')
(ii) As 404 mentioned, Use SQL API User Defined Functions which supports regex :
function executeRegex(str, pattern) {
let regex=RegExp(pattern);
return regex.test(str);
}
SELECT udf.EXECUTE_REGEX("foobar", ".*bar")
Another possibility is creating your own User Defined Function. As example here's a regex check:
function matchRegex(str, pattern) {
let regex=RegExp(pattern);
return regex.test(str);
}
Created under the name MATCH_REGEX it can then be used like:
SELECT udf.MATCH_REGEX("09001001", "^09.*001$")
As note: it'll kill any index optimization that for instance STARTSWITH would have. Although allows for much more complex patterns. It can therefor be beneficial to use additional filters that are capable of using the index to narrow down the search. E.g. using StartsWith(c.property1, '09') as addition to the above example in a WHERE clause.
UPDATE:
Cosmos now has a RegexMatch function that can do the same. While the documentation for the MongoApi mentions the $regex can use the index to optimize your query if it adheres to certain rules this does not seem to be the case for the SqlApi (at this moment).
I'm querying Azure table storage using the Azure Storage Explorer. I want to find all messages that contain the given text, like this in T-SQL:
message like '%SysFn%'
Executing the T-SQL gives "An error occurred while processing this request"
What is the equivalent of this query in Azure?
There's no direct equivalent, as there is no wildcard searching. All supported operations are listed here. You'll see eq, gt, ge, lt, le, etc. You could make use of these, perhaps, to look for specific ranges.
Depending on your partitioning scheme, you may be able to select a subset of entities based on specific partition key, and then scan through each entity, examining message to find the specific ones you need (basically a partial partition scan).
While an advanced wildcard search isn't strictly possible in Azure Table Storage, you can use a combination of the "ge" and "lt" operators to achieve a "prefix" search. This process is explained in a blog post by Scott Helme here.
Essentially this method uses ASCII incrementing to query Azure Table Storage for any rows whose property begins with a certain string of text. I've written a small Powershell function that generates the custom filter needed to do a prefix search.
Function Get-AzTableWildcardFilter {
param (
[Parameter(Mandatory=$true)]
[string]$FilterProperty,
[Parameter(Mandatory=$true)]
[string]$FilterText
)
Begin {}
Process {
$SearchArray = ([char[]]$FilterText)
$SearchArray[-1] = [char](([int]$SearchArray[-1]) + 1)
$SearchString = ($SearchArray -join '')
}
End {
Write-Output "($($FilterProperty) ge '$($FilterText)') and ($($FilterProperty) lt '$($SearchString)')"
}
}
You could then use this function with Get-AzTableRow like this (where $CloudTable is your Microsoft.Azure.Cosmos.Table.CloudTable object):
Get-AzTableRow -Table $CloudTable -CustomFilter (Get-AzTableWildcardFilter -FilterProperty 'RowKey' -FilterText 'foo')
Another option would be export the logs from Azure Table storage to csv. Once you have the csv you can open this in excel or any other app and search for the text.
You can export table storage data using TableXplorer (http://clumsyleaf.com/products/tablexplorer). In this there is an option to export the filtered data to csv.