I would like to write query to remote Azure SQL database.
I followed the tutorial via Query Data Source - Method 1
I was successful to run the query from tutorial:
#results1 =
SELECT *
FROM EXTERNAL MyAzureSQLDBDataSource EXECUTE #"SELECT ##SERVERNAME AS serverName, GETDATE() AS dayTime, DB_NAME() AS databaseName";
But...
I would like to update this query to following form:
DECLARE #queryA string = #"SELECT ##SERVERNAME AS serverName, GETDATE() AS dayTime, DB_NAME() AS databaseName";
#results2 =
SELECT *
FROM EXTERNAL MyAzureSQLDBDataSource EXECUTE #queryA;
I got an error
E_CSC_USER_SYNTAXERROR: syntax error. Expected one of: string-literal
Any idea why I cannot use query stored in string value?
In real query I need to dynamically create query based on parameters in where statement.
Thank you in advance
According to this article https://msdn.microsoft.com/en-us/library/azure/mt621291.aspx you can provide only a literal, not a variable:
EXECUTE csharp_string_literal
The string literal contains a query
expression in the language supported by the remote data source. E.g.,
if the data source is an Azure SQL Database, then the query string
would be T-SQL.
Related
I'm trying to do a simple incremental update from an on-prem database as source to Azure SQL database based on a varchar column called "RP" in On-Prem database that contains "date+staticdescription" for example: "20210314MetroFactory"
1- I've created a Lookup activity called Lookup1 using a table created in Azure SQL Database and uses this Query
"Select RP from SubsetwatermarkTable"
2- I've created a Copy data activity where the source settings have this Query
"Select * from SourceDevSubsetTable WHERE RP NOT IN '#{activity('Lookup1').output.value}'"
When debugging -- I'm getting the error:
Failure type: User configuration issue
Details: Failure happened on 'Source' side.
'Type=System.Data.SqlClient.SqlException,Message=Incorrect syntax near
'[{"RP":"20210307_1Plant
1KAO"},{"RP":"20210314MetroFactory"},{"RP":"20210312MetroFactory"},{"RP":"20210312MetroFactory"},{"RP":"2'.,Source=.Net
SqlClient Data
Provider,SqlErrorNumber=102,Class=15,ErrorCode=-2146232060,State=1,Errors=[{Class=15,Number=102,State=1,Message=Incorrect
syntax near
'[{"RP":"20210311MetroFactory"},{"RP":"20210311MetroFactory"},{"RP":"202103140MetroFactory"},{"RP":"20210308MetroFactory"},{"RP":"2'.,},],'
Can anyone tell me what I am doing wrong and how to fix it even if it requires creating more activities.
Note: There is no LastModifiedDate column in the table. Also I haven't yet created the StoredProcedure that will update the Lookup table when it is done with the incremental copy.
Steve is right as to why it is failling and the query you need in the Copy Data.
As he says, you want a comma-separated list of quoted values to use in your IN clause.
You can get this more easily though - from your Lookup directly using this query:-
select stuff(
(
select ','''+rp+''''
from subsetwatermarktable
for xml path('')
)
, 1, 1, ''
) as in_clause
The sub-query gets the comma separated list with quotes around each rp-value, but has a spurious comma at the start - the outer query with stuff removes this.
Now tick the First Row Only box on the Lookup and change your Copy Data source query to:
select *
from SourceDevSubsetTable
where rp not in (#{activity('lookup').output.firstRow.in_clause})
The result of #activity('Lookup1').output.value is an array like your error shows
[{"RP":"20210307_1Plant
1KAO"},{"RP":"20210314MetroFactory"},{"RP":"20210312MetroFactory"},{"RP":"20210312MetroFactory"},{"RP":"2'.,Source=.Net
SqlClient Data
Provider,SqlErrorNumber=102,Class=15,ErrorCode=-2146232060,State=1,Errors=[{Class=15,Number=102,State=1,Message=Incorrect
syntax near
'[{"RP":"20210311MetroFactory"},{"RP":"20210311MetroFactory"},{"RP":"202103140MetroFactory"},{"RP":"20210308MetroFactory"},{"RP":"2'.,},]
However, your SQL should be like this:Select * from SourceDevSubsetTable WHERE RP NOT IN ('20210307_1Plant 1KAO','20210314MetroFactory',...).
To achieve this in ADF, you need to do something like this:
create three variables like the following screenshot:
loop your result of #activity('Lookup1').output.value and append 'item().RP' to arrayvalues:
expression:#activity('Lookup1').output.value
expression:#concat(variables('apostrophe'),item().RP,variables('apostrophe'))
3.cast arrayvalues to string and add parentheses by Set variable activity
expression:#concat('(',join(variables('arrayvalues'),','),')')
4.copy to your Azure SQL database
expression:Select * from SourceDevSubsetTable WHERE RP NOT IN #{variables('stringvalues')}
We have a bug in Final Testing. I am trying to make the following Query in a LookUp Activity:
SELECT ShortName, EEOXrefCode FROM [MercerStagingDev].[MILKY-WAYTEST\AppSQLVST4DotNetDev-R].[Copy of Battelle Work Assignment Fields Jobs]
WHERE XRefCode = #variables('JobXRefCode')
I have the Set Variable showing the correct String 'JobXrefCode' but am getting a crazy error. Please help; I must figure out how to do this "syntax" this weekend.
Thanks!!
Mike Kiser
Error code
2100
Troubleshooting guide
Failure type
User configuration issue
Details
Failure happened on 'Source' side. ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: 'Must declare the scalar variable "#variables".',Source=,''Type=System.Data.SqlClient.SqlException,Message=Must declare the scalar variable "#variables".,Source=.Net SqlClient Data Provider,SqlErrorNumber=137,Class=15,ErrorCode=-2146232060,State=2,Errors=[{Class=15,Number=137,State=2,Message=Must declare the scalar variable "#variables".,},],'
Please try this expression:
SELECT ShortName, EEOXrefCode
FROM [MercerStagingDev].[MILKY-WAYTEST\AppSQLVST4DotNetDev-R].[Copy of Battelle Work Assignment Fields Jobs]
WHERE XRefCode = '#{variables('JobXRefCode')}'
Update:
Expressions can also appear inside strings, using a feature called
string interpolation where expressions are wrapped in #{ ... }. For
example: "name" : "First Name: #{pipeline().parameters.firstName} Last
Name: #{pipeline().parameters.lastName}"
Using string interpolation, the result is always a string.
So when you do something like this SQL: SELECT * FROM TableName WHERE ColumnName = 'value', you can use this expression.
Here are some samples in ADF Tutorials:
Ref:
Expressions and functions in Azure Data Factory
Incrementally load data from Azure SQL Database to Azure Blob storage using the Azure portal
Incrementally load data from multiple tables in SQL Server to a database in Azure SQL Database using the Azure portal
I am using Boto3 package in python3 to execute an Athena query. From the documentation of Boto3, I understand that I can specify a query execution context, i.e. a database name under which the query has to be executed. With a properly specified query execution context, we can omit the fully qualified table name(db_name.table_name) from the query and instead use just the table name.
So the query SELECT * FROM db1.tab1 can be converted to SELECT * FROM tab1 with QueryExecutionContext : {'database':'db1'}
The problem: I need to run a query on Athena from python which looks something like this
SELECT *
FROM ((SELECT *
FROM db1.tab1 AS Temp1)
INNER JOIN (SELECT *
FROM db2.tab2 AS Temp2)
ON temp1.id = temp2.id)
As we can see, the query joins tables from two different databases. If I want to omit the database names from this query, how do I specify the QueryExecutionContext ?
QueryExecutionContext accepts only one database as an argument.So if you want to run a query across multiple databases then you have to pass fully qualified table name along with database.
I created a new table in the Bluemix SQL Database service by uploading a csv (baseball.csv) and took the default table name of "baseball".
I created a simple app in Node.js which is just trying to select data from the table with select * from baseball, but I keep getting the following error:
[IBM][CLI Driver][DB2/NT] SQL0204N "USERxxxx.BASEBALL" in an undefined name
Why can't it find my database table?
This issue seems independent of bluemix, rather it is usage error.
This error is possibly caused by following:
The object identified by name is not defined in the database.
User response
Ensure that the object name (including any required qualifiers) is correctly specified in the SQL statement and it exists.
try running "list tables" from command prompt to check if your table spelling is correct or not.
http://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.messages.sql.doc/doc/msql00204n.html?cp=SSEPGG_9.7.0%2F2-6-27-0-130
I created the table from SQL Database web UI in bluemix and took the default name of baseball. It looks like this creates a case-sensitive table name.
Unfortunately for me, the sql_db libary (and all db2 clients I believe) auto-capitalizes the SQL query into "SELECT * FROM BASEBALL"
The solution was to either
A. Explicitly name my table BASEBALL in the web UI; or
B. Modify my sql query by quoting the table name:
select * from "baseball"
More info at http://www.ibm.com/developerworks/data/library/techarticle/0203adamache/0203adamache.html#N10121
I have written MDX query and assigned it to Adomd Connection object. When i Run Code Analysis, it gives me CA2100 Review SQL queries for security vulnerabilities error we can't directly supply query to connection objects. It Says either we should embed it in Stored Procedure or use Parametrized query. But in my case, there are no parameters for this query. So kindly help, how can I remove this CA 2100 Security error. PFB the code. Thanks in advance.
conn.Open();
// Adomd Connection Object
var adomdCommand = new AdomdCommand()
{
Connection = conn,
CommandType = CommandType.Text,
CommandText = mdxQuery
};
//Execute command to return cell set..
CellSet csResult = adomdCommand.ExecuteCellSet();
conn.Close();
You can create stored procedure in SQL Server that performs execution of AS query. Assuming sp's are pre-compiled, that's secure. You should do the following:
Add your AS server as linked server to SQL Database server (via Server Objects in SSMS)
Create stored procedure. The general scheme is (sp body):
declare #tsqlquery varchar(1000)
declare #mdxquery varchar(2000)
set #tsqlquery = 'SELECT
"[DimA].[A].[A name].[MEMBER_CAPTION]" as dimensionName,
convert(float, "[Measures].[X]") AS measureValue
FROM OPENQUERY(<**YOUR LINKED SERVER**>,'
set #mdxquery = '''**YOUR MDX QUERY**' + '''' + ')'
EXEC(#tsqlquery + #mdxquery)
You can also create parametized store procedure that modifies MDX query text based on parameters. As long as you are using Stored Procedures, you are safe. For example, our report server queries only use SP's, not direct AS queries.
PS. If your AS server has role security defined, you should enable ImpersonateCurrentUser for the database, then your role will work.