How to render JSON using Stream Analytics Query - azure

I have Inputs in the form of JSON stored in Blob Storage
I have Output in the form of SQL Azure table.
My wrote query and successfully moving value of specific property in JSON to corresponding Column of SQL Azure table.
Now for one column I want to copy entire JSON payload as Serialized string in one sql column , I am not getting proper library function to do that.
SELECT
CASE
WHEN GetArrayLength(E.event) > 0
THEN GetRecordPropertyValue(GetArrayElement(E.event, 0), 'name')
ELSE ''
END AS EventName
,E.internal.data.id as DataId
,E.internal.data.documentVersion as DocVersion
,E.context.custom As CustomDimensionsPayload
Into OutputTblEvents
FROM InputBlobEvents E
This CustomDimensionsPayload should be a JSON actually

I made a user defined function which did the job for me:
function main(InputJSON) {
var InputJSONString = JSON.stringify(InputJSON);
return InputJSONString;
}
Then, inside the Query, I used the function like this:
SELECT udf.ConvertToJSONString(COLLECT()) AS InputJSON
INTO outputX
FROM inputY

You need to just reference the input object itself instead of COLLECT() if you want the entire payload to be converted. I was trying to do this also so figured I'd add what i did.
I used the same function suggested by PerSchjetne, query then becomes
SELECT udf.JSONToString(IoTInputStream)
INTO [SQLTelemetry]
FROM [IoTInputStream]
Your output will now be the full JSON string, including all the metadata extras that IOT hub adds on.

Related

Azure Cognitive Search int to Edm.String field issues

I'm having trouble trying to added data to my Azure Cognitive Search index. The data is being read from SQL Server tables with a python script. The script sends it to the index using the SearchIndexClient from the azure search sdk.
The problem is when sending Python "int" values into a search index field of type Edm.String. The link below seems to indicate that this should be possible. Any number type is allowed to go into a Edm.String.
https://learn.microsoft.com/en-us/rest/api/searchservice/data-type-map-for-indexers-in-azure-search#bkmk_sql_search
However I get this error:
Cannot convert the literal '0' to the expected type 'Edm.String'.
Am I misunderstanding the docs? Is the python int different than the SQL Server int through the Azure Search SDK?
I'm using pyodbc to connect to an Azure Synapse db. Retrieving the rows with cursor loop. This is basically what I'm doing...
search_client = SearchIndexClient(env.search_endpoint,
env.search_index,
SearchApiKeyCredential(env.search_api_key),
logging_enable=True)
conn = pyodbc.connect(env.sqlconnstr_synapse_connstr, autocommit=True)
query = f"SELECT * FROM [{env.source_schema}].[{source_table}]"
cursor = conn.cursor()
cursor.execute(query)
source_table_columns = [source_table_column[0] for source_table_column in cursor.description]
rows = []
for source_table_values in cursor.fetchmany(MAX_ROWS_TO_FETCH):
source_table_row = dict(zip(source_table_columns,
source_table_values))
rows.append(source_table_row)
upload = search_client.upload_documents(documents=rows)
If the row contains a row with an int value and the search index table field is Edm.String, we get the error.
Cannot convert the literal '0' to the expected type 'Edm.String'.
Thank you for providing the code snippet. The data type mapping link is applicable when using an Indexer to populate an Index.
Indexers provide a convenient mechanism to load documents into an Index from a source datasource. They perform the mapping outlined here by default or can take in an optional fieldMappings.
In the case of the code snippet where an index is being updated manually, when there is a type mismatch between source & target, that would be handled by casting/converting etc. by the user. In the code snippet after you have the dictionary, you can convert the int into a string using str() before uploading the batch in to the Index
source_table_row[column_name] = str(source_table_row[column_name])
This is a python sample that creates an indexer to update an index

How to query fields with multiple values in Azure Cognitive Search

Working on Azure Cognitive Search with backend as MS SQL table, have some scenarios where need help to define a query.
Sample table structure and data :
Scenarios 1 : Need to define a query which will return data based on category.
I have tied query using search.ismatch but its uses prefix search and matches other categories as well with similar kind of values i.e. "Embedded" and "Embedded Vision"
$filter=Region eq 'AA' and search.ismatch('Embedded*','Category')
https://{AZ_RESOURCE_NAME}.search.windows.net/indexes/{INDEX_NAME}/docs?api-version=2020-06-30-Preview&$count=true&$filter=Region eq 'AA' and search.ismatch('Embedded*','Category')
And it will response with below result, where it include "Embedded" and "Embedded Vision" both categories.
But my expectation is to fetch data only if it match "Embedded" category, as highlighted below
Scenario 2: For the above Scenario 1, Need little enhancement to find records with multiple category
For example if I pass multiple categories (i.e. "Embedded" , "Automation") need below highlighted output
you'll need to use a different analyzer which will break the tokens on every ';' just for the category field rather than 'whitespaces'.
You should first ensure your Category data is populated as a Collection(Edm.String) in the index. See Supported Data Types in the official documentation. Each of your semicolon-separated values should be separate values in the collection, in a property called Category (or similar).
You can then filter by string values in the collection. See rules for filtering string collections. Assuming that your index contains a string collection field called Category, you can filter by categories containing Embedded like this:
Category/any(c: c eq 'Embedded')
You can filter by multiple values like this:
Category/any(c: search.in(c, 'Embedded, Automation'))
Start with clean data in your index using proper types for the data you have. This allows you to implement proper facets and you can utilize the syntax made specifically for this. Trying to work around this with wildcards is a hack that should be avoided.
To solve above mention problem used a below SQL function which will convert category to a json string array supported by Collection(Edm.String) data type in Azure Search.
Sql Function
CREATE FUNCTION dbo.GetCategoryAsArray
(
#ID VARCHAR(20)
)
RETURNS NVARCHAR(MAX)
AS
BEGIN
DECLARE #result NVARCHAR(MAX) = ''
SET #result = REPLACE(
STUFF(
(SELECT
','''+ TRIM(Value) + ''''
FROM dbo.TABLEA p
CROSS APPLY STRING_SPLIT (Category, ';')
WHERE p.ID = #ID
FOR XML PATH('')
),1,1,''),'&','&')
RETURN '[' + #result + ']'
END
GO
View to use function and return desired data
CREATE View dbo.TABLEA_VIEW AS
select
id
,dbo. GetCategoryAsArray(id) as CategoryArr
,type
,region
,Category
from dbo.TABLEA
Defined a new Azure Search Index using above SQL View as data source and during Index column mapping defined CategoryArr column as Collection(Edm.String) data type
Query to use to achieve expected output from Azure Search
$filter=Region eq 'AA' and CategoryArr/any(c: search.in(c, 'Embedded, Automation'))

Convert AutoQuery query string to SqlExpression

I am trying to re-create AutoQuery queries outside of a service request. I am doing this because I give user option to save a request and then use that data elsewhere. I save the query string data so I am trying to create a query from the saved query string.
I need 2 things.
1) query that returns the complete data not limited by default autoquery page size
2) query that returns the count
I tried making the query like this:
IAutoQueryDb _autoQuery = HostContext.TryResolve<IAutoQueryDb>();
var dto = new MyQueryDbClass();
Dictionary<string, string> pars = GetParameters();
var query = _autoQuery.CreateQuery(dto, pars);
The problem with this is that the query generated has the table name of the response object and not the actual table so it doesn't work. Also I am unable to call ToCountSatement() on it. It is also limited by my default page size.
Is there a way to convert the AutoQuery query string to a SqlExpression so I can execute it and also get count statement?
The CreateQuery() API returns a populated SqlExpression<Table> similar to what would have been created if manually constructing the query yourself, e.g:
SqlExpression<Table> query = _autoQuery.CreateQuery(dto, pars);
To clear the paging info you can call .Limit() without arguments which will clear any populated Offset/Rows values:
query.Limit();
The Custom AutoQuery Implementations docs shows an example of AutoQuery executes the query behind the scenes, e.g. you can get the total with:
var total = Db.Count(query);

Generate table in hubot brain and insert values

I'm trying to make a table of values inside hubot and he pass it's values to redis-brain.coffee but i just know a way: robot.brain.get("blablabla").
This will get a string from redis-brain and i need some kind of table.
How I'll use it:
At first call of this function, hubot will load the full database to the memory, then, if there's
robot.catchAll (msg) ->
if not quiet
text = msg.message.text
ector.setUser msg.message.user.name
if not loaded_brain
ector_brain = robot.brain.get('ector_brain') #need to be some type of table - In mysql should be like a select
ector.addEntry ector_brain
loaded_brain = true
else
ector.addEntry text
ector_brain = ector_brain+text #this line should insert the value of text inside ector_brain table. -- In mysql shoud be like an insert into
ector.linkNodesToLastSentence previousResponseNodes
response = ector.generateResponse()
previousResponseNodes = response.nodes
msg.reply response.sentence
So, how do I create a table in redis from hubot?
robot.brain.get and robot.brain.set operates with JSON objects, not only strings. Just place an object with your data structure of choice in the brain and get it back when necessary.

Pass table name a parameter to the Data Context method using LINQ to SQL in C#

I have created a method which delete data from the database by joining a db column with a csv file, I have a list of tables in a Data Context which I need to iterate, i means that I need to change the table name by passing it as parameter to the method.
Basically I need to use this method for all the table present in the Data Context
Current hard coded Table name in a GetTable method: dw_Job
DataContext Object: job
Here is my code
var query = from line in File.ReadAllLines(file, Encoding.GetEncoding(1252))
//let jobrecord = line.Split(',')
join j in job.GetTable<dw_Job>() on
Convert.ToInt32(line.Split(',').GetValue(factColPos))
equals j.JobID
select j;
foreach (var deletejob in query)
{
Console.WriteLine(deletejob);
job.dw_Jobs.DeleteOnSubmit(deletejob);
}
job.SubmitChanges();
I can able to provide if you need any further details of my code. Please correct me on this
Thanks in advance

Resources