I'm trying to create index which using import data tool.
The datasource is from azure sql's view.
SELECT
b.Name,
b.ID
(SELECT
'[' + STUFF((
SELECT
',{"name":"' + p.Name + '"}'
FROM Product p WHERE p.Brand = b.ID
FOR XML PATH (''), TYPE)
.value('.', 'nvarchar(max)'), 1, 1, '') + ']') AS TAry,
b.IsDelete,
b.ModifyDatetime
from Brand b
Column with TAry will return JSon format string like:
[{"name":"Test1"},{"name":"Test2"}]
In Indexder properties with field TAry Chose the type Collection(Edm.String)
After create , It's return error , the message below:
"The data field 'TAry' has an invalid value. The expected type was 'Collection(Edm.String)'."
Thank for your reply.
I have try this kind format :[Test1","Test2"] still not work.
To do this, you need to use Azure Search REST API to set up a field mapping with jsonArrayToStringCollection function. Take a look at this article for detailed instructions.
Related
I'm having trouble trying to added data to my Azure Cognitive Search index. The data is being read from SQL Server tables with a python script. The script sends it to the index using the SearchIndexClient from the azure search sdk.
The problem is when sending Python "int" values into a search index field of type Edm.String. The link below seems to indicate that this should be possible. Any number type is allowed to go into a Edm.String.
https://learn.microsoft.com/en-us/rest/api/searchservice/data-type-map-for-indexers-in-azure-search#bkmk_sql_search
However I get this error:
Cannot convert the literal '0' to the expected type 'Edm.String'.
Am I misunderstanding the docs? Is the python int different than the SQL Server int through the Azure Search SDK?
I'm using pyodbc to connect to an Azure Synapse db. Retrieving the rows with cursor loop. This is basically what I'm doing...
search_client = SearchIndexClient(env.search_endpoint,
env.search_index,
SearchApiKeyCredential(env.search_api_key),
logging_enable=True)
conn = pyodbc.connect(env.sqlconnstr_synapse_connstr, autocommit=True)
query = f"SELECT * FROM [{env.source_schema}].[{source_table}]"
cursor = conn.cursor()
cursor.execute(query)
source_table_columns = [source_table_column[0] for source_table_column in cursor.description]
rows = []
for source_table_values in cursor.fetchmany(MAX_ROWS_TO_FETCH):
source_table_row = dict(zip(source_table_columns,
source_table_values))
rows.append(source_table_row)
upload = search_client.upload_documents(documents=rows)
If the row contains a row with an int value and the search index table field is Edm.String, we get the error.
Cannot convert the literal '0' to the expected type 'Edm.String'.
Thank you for providing the code snippet. The data type mapping link is applicable when using an Indexer to populate an Index.
Indexers provide a convenient mechanism to load documents into an Index from a source datasource. They perform the mapping outlined here by default or can take in an optional fieldMappings.
In the case of the code snippet where an index is being updated manually, when there is a type mismatch between source & target, that would be handled by casting/converting etc. by the user. In the code snippet after you have the dictionary, you can convert the int into a string using str() before uploading the batch in to the Index
source_table_row[column_name] = str(source_table_row[column_name])
This is a python sample that creates an indexer to update an index
Working on Azure Cognitive Search with backend as MS SQL table, have some scenarios where need help to define a query.
Sample table structure and data :
Scenarios 1 : Need to define a query which will return data based on category.
I have tied query using search.ismatch but its uses prefix search and matches other categories as well with similar kind of values i.e. "Embedded" and "Embedded Vision"
$filter=Region eq 'AA' and search.ismatch('Embedded*','Category')
https://{AZ_RESOURCE_NAME}.search.windows.net/indexes/{INDEX_NAME}/docs?api-version=2020-06-30-Preview&$count=true&$filter=Region eq 'AA' and search.ismatch('Embedded*','Category')
And it will response with below result, where it include "Embedded" and "Embedded Vision" both categories.
But my expectation is to fetch data only if it match "Embedded" category, as highlighted below
Scenario 2: For the above Scenario 1, Need little enhancement to find records with multiple category
For example if I pass multiple categories (i.e. "Embedded" , "Automation") need below highlighted output
you'll need to use a different analyzer which will break the tokens on every ';' just for the category field rather than 'whitespaces'.
You should first ensure your Category data is populated as a Collection(Edm.String) in the index. See Supported Data Types in the official documentation. Each of your semicolon-separated values should be separate values in the collection, in a property called Category (or similar).
You can then filter by string values in the collection. See rules for filtering string collections. Assuming that your index contains a string collection field called Category, you can filter by categories containing Embedded like this:
Category/any(c: c eq 'Embedded')
You can filter by multiple values like this:
Category/any(c: search.in(c, 'Embedded, Automation'))
Start with clean data in your index using proper types for the data you have. This allows you to implement proper facets and you can utilize the syntax made specifically for this. Trying to work around this with wildcards is a hack that should be avoided.
To solve above mention problem used a below SQL function which will convert category to a json string array supported by Collection(Edm.String) data type in Azure Search.
Sql Function
CREATE FUNCTION dbo.GetCategoryAsArray
(
#ID VARCHAR(20)
)
RETURNS NVARCHAR(MAX)
AS
BEGIN
DECLARE #result NVARCHAR(MAX) = ''
SET #result = REPLACE(
STUFF(
(SELECT
','''+ TRIM(Value) + ''''
FROM dbo.TABLEA p
CROSS APPLY STRING_SPLIT (Category, ';')
WHERE p.ID = #ID
FOR XML PATH('')
),1,1,''),'&','&')
RETURN '[' + #result + ']'
END
GO
View to use function and return desired data
CREATE View dbo.TABLEA_VIEW AS
select
id
,dbo. GetCategoryAsArray(id) as CategoryArr
,type
,region
,Category
from dbo.TABLEA
Defined a new Azure Search Index using above SQL View as data source and during Index column mapping defined CategoryArr column as Collection(Edm.String) data type
Query to use to achieve expected output from Azure Search
$filter=Region eq 'AA' and CategoryArr/any(c: search.in(c, 'Embedded, Automation'))
I'm attempting to export data from an older Orchard db and am having problems finding which table the content of a blog post is stored. I've tried using a number of different 'Search all columns' spocs to search all tables and columns but am not finding text from the post itself.
If I have a blog post where the opening sentence is:
This sentence contains a unique word.
I would have expected at least one of the various 'Search all columns' examples to have turned up a table/column. But so far, none have.
thx
Orchard store data based on two tables, ContentItemRecord and ContentItemVersionRecord, which store meta data for content items like BlogPost, and these content items built from multiple parts, each part has it's table and the relation between the item and it's parts is based on Id (if not draftable) or ContentItemRecord_Id (if draftable) columns
if we take BlogPost type as example, which built from TitlePart, BodyPart, AutoroutePart and CommonPart, and you want to select all the data of post (id = 90), then you can find it's title in TitlePartRecord table (ContentItemRecord_Id = 90), and the body text of it in BodyPartRecord table with same relation as title part record, and the route part in AutorouteRecord table with same relation, and the common meta data in CommonPartRecord (Id = 90).
This is the way to extract data from Orchard database, hope this will help you.
Tnx to #mdameer...
and the related query of madmeer's answer is this:
SELECT * FROM dbo.default_Title_TitlePartRecord
inner join dbo.default_Orchard_Framework_ContentItemRecord on
dbo.default_Title_TitlePartRecord.ContentItemRecord_id=dbo.default_Orchard_Framework_ContentItemRecord.Id
inner join dbo.default_Common_BodyPartRecord on
dbo.default_Common_BodyPartRecord.ContentItemRecord_id=dbo.default_Orchard_Framework_ContentItemRecord.Id
where dbo.default_Title_TitlePartRecord.ContentItemRecord_id=90
and this is the rightsolution
Just in case it may be useful for others, the following is the actual SQL query used to migrate an Orchard instance to Umbraco. It is derived from the excellent answers by mdameerand and Iman Salehi:
SELECT t.Title, f.Data, b.Text FROM dbo.Title_TitlePartRecord t
inner join dbo.Orchard_Framework_ContentItemRecord f on
t.ContentItemRecord_id=f.Id
inner join dbo.Common_BodyPartRecord b on
b.ContentItemRecord_id=f.Id
AND b.Id = (
SELECT MAX(m2.Id)
FROM dbo.Common_BodyPartRecord m2
WHERE m2.ContentItemRecord_id = f.Id
)
AND t.Id = (
SELECT MAX(m2.Id)
FROM dbo.Title_TitlePartRecord m2
WHERE m2.ContentItemRecord_id = f.Id
)
Objective:
I have a table called Publication that contains Id, RecordId, EntityType and a couple other columns. I select all the records that need to be published to another database from that table. I then loop that collection to process the records and move the records to the other db.
Background:
The EntityType column is used to Identify the Set that the context needs to retrieve. I also use reflection to create a object of that type to see if it implements a certain type of interface. If the record being processed does implement that interface then I know that the RecordId for that record in the Publication table is not a PK in the Set() but rather a FK.
this code works fine when I am going after the PK values for EntityTypes that do not inherit the specific interface.
object authoringRecordVersion = PublishingFactory.AuthoringContext.Set(recordType.Entity.GetType()).Find(record.RecordId);
Problem:
DbContext.Set(EntityType).Find(PK) goes after the PrimaryKey value. How can I tell Set() to search like this sudo code example since 'Where' is not allowed
object authoringRecordVersion = PublishingFactory.AuthoringContext.Set(recordType.Entity.GetType()).Where(c => c.HeaderRecordId == record.RecordId)
Update:
I am working on Implementing the following. Will advise results tomorrow
var sql = "SELECT * from " + record.Entity + " WHERE HeaderRecordId = '" + record.RecordId + "'";
authoringRecordVersion = PublishingFactory.AuthoringContext.Set(recordType.Entity.GetType()).SqlQuery(sql).AsNoTracking();
.SqlQuery(sql).AsNoTracking();
does work effectively. Don't know why I didn't see this earlier.
i've a trouble with linq, i'll explain on example :
i have a database table called Employee which got FirstName and LastName columns,
and a method to search employees which gets a nameList list as argument, the elements in this list are names formatted like this one "Fred Burn", or this1 "Grim Reaper",
already tryed these approaches with no luck =[
//just all employees
var allData = from emp in Context.Employee select emp;
var test1 = from emp in allData
where(emp.FirstName + " " + emp.LastName).Contains
("" + ((from n in nameList select n).FirstOrDefault()))
select emp;
var test2 = (from emp in allData
where (emp.FirstName + " " + emp.LastName)
== ((from n in nameList select n).FirstOrDefault())
select emp);
var test3 = from emp in allData
where (from n in nameList select n).Contains
(emp.FirstName + " " + emp.LastName)
select emp;
first and second queries give : {"Unable to create a constant value of type 'Closure type'. Only primitive types ('such as Int32, String, and Guid') are supported in this context."} exceptionand third : {"LINQ to Entities does not recognize the method 'Boolean Contains[String](System.Collections.Generic.IEnumerable`1[System.String], System.String)' method, and this method cannot be translated into a store expression."}
would be glad to hear your suggestions :)
Thank You!
p.s.
yea i know it's possible to split names in list and compare them separately, but still curious why wont these queries work.
I assume nameList in this case is an in memory collection and that you are trying to use the LINQ to SQL trick creating a SQL "IN" clause inside of the Entity Framework. Unfortunately, EF doesn't support this syntax (yet). Depending on the number of records you are trying to match, you may be able to run separate queries for each child you are desiring. Alternatively, you could build an entitysql query using concatenation to append the multiple items from the nameList as separate OR clauses in the WHERE operation.