I am trying to do a COPY INTO statement for about 10 files of size 100mb each that are stored in the data lake, and it keeps throwing this error;
Msg 110806, Level 11, State 0, Line 7
110806;A distributed query failed: A severe error occurred on the current command. The results, if any, should be discarded.
Operation cancelled by user.
The command i used is;
COPY INTO AAA.AAAA FROM 'https://AAAA.blob.core.windows.net/data_*.csv' WITH (
CREDENTIAL = (IDENTITY = 'MANAGED IDENTITY'), FIELDQUOTE = N'"', FIELDTERMINATOR = N',',FIRSTROW = 2 );
where did i go wrong? please advise.
Related
I'm getting an error when attempting to insert from a temp table into a table that exists in Synapse, here is the relevant code:
def load_adls_data(self, schema: str, table: str, environment: str, filepath: str, columns: list) -> str:
if self.exists_schema(schema):
if self.exists_table(schema, table):
if environment.lower() == 'prod':
schema = "lvl0"
else:
schema = f"{environment.lower()}_lvl0"
temp_table = self.generate_temp_create_table(schema, table, columns)
sql0 = """
IF OBJECT_ID('tempdb..#CopyDataFromADLS') IS NOT NULL
BEGIN
DROP TABLE #CopyDataFromADLS;
END
"""
sql1 = """
{}
COPY INTO #CopyDataFromADLS FROM
'{}'
WITH
(
FILE_TYPE = 'CSV',
FIRSTROW = 1
)
INSERT INTO {}.{}
SELECT *, GETDATE(), '{}' from #CopyDataFromADLS
""".format(temp_table, filepath, schema, table, Path(filepath).name)
print(sql1)
conn = pyodbc.connect(self._synapse_cnx_str)
conn.autocommit = True
with conn.cursor() as db:
db.execute(sql0)
db.execute(sql1)
If I get rid of the insert statement and just do a select from the temp table in the script:
SELECT * FROM #CopyDataFromADLS
I get the same error in either case:
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Not able to validate external location because The remote server returned an error: (409) Conflict. (105215) (SQLExecDirectW)')
I've run the generated code for both the insert and the select in Synapse and they ran perfectly. Google has no real info on this so could someone assist with this? Thanks
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Not able to validate external location because The remote server returned an error: (409) Conflict. (105215) (SQLExecDirectW)')
This error occurs mostly because of authentication or access.
Make sure you have blob storage contributor access.
In the copy into script, add the authentication key for blob storage, unless it is a public blob storage.
I tried to repro this using copy into statement without authentication and got the same error.
After adding authentication using SAS key data is copied successfully.
Refer the Microsoft document for permissions required for bulk load using copy into statements.
I am trying to connect DB2/IDAA using ADFV2 - while executing simple query "select * from table" - I am getting below error:
Operation on target Copy data from IDAA failed: An error has occurred on the Source side. 'Type = Microsoft.HostIntegration.DrdaClient.DrdaException, Message = Exception or type' Microsoft.HostIntegration.Drda.Common.DrdaException 'was thrown. SQLSTATE = HY000 SQLCODE = -343, Source = Microsoft.HostIntegration.Drda.Requester, '
I checked a lot and tried various options but still it's an issue.
I tried query "select * from table with ur" - query to call with read-only but still get above result.
If I use query like select * from table; commit; - then activity succeeded but no record fetch.
Is anyone have solution ?
I have my linked service setup like this. additional connection properties value is : SET CURRENT QUERY ACCELERATION = ALL
I'm trying to call a Teradata Stored procedure in python. But it is giving the following error.
cursor = session.cursor()
cursor.callproc("usr.RESULTSET", (teradata.InOutParam("today"),
teradata.OutParam("p2")))
output = cursor.fetchone()
print(output)
Error
raise DatabaseError(i[2], u"[{}] {}".format(i[0], msg), i[0])
teradata.api.DatabaseError: (6, '[HY000] [Teradata][ODBC Teradata Driver] (6) Internal Error (Exception).')
INFO:teradata.udaexec: UdaExec exiting. (2019-05-17 10:02:13,350; udaexec.py:68)
I managed to execute the query via bteq file, and now it's giving me the correct result set. Not sure why my previous method failed. bteq file contains the stored procedure and it's parameters.
ex: stored_procedure.bteq
CALL dbname.UsrFail(arg1,arg2)
rows = []
with td.TeraSession(td_sys_name, td_cred) as session:
results = session.execute(file="/tperf/stored_procedure.bteq")
for result in results.fetchall():
rows.append(result.values)
I am trying to read a particular data from database. And executing the query to get the result. I am running the code in a while (true) loop so that I am doing the operation again and again.
In some of the iterations, I am getting result but in some cases it shows com.datastax.driver.core.exceptions.TraceRetrievalException and couldn't get the execution time for the query.
Cassandra version-3.11.4 , Datastax version-3.7
Exception thrown :
com.datastax.driver.core.exceptions.TraceRetrievalException: Unable to retrieve complete query trace for id bf7902b0-5d26-11e9-befd-d97cd54dc732 after 5 tries
at com.datastax.driver.core.QueryTrace.doFetchTrace(QueryTrace.java:242)
at com.datastax.driver.core.QueryTrace.maybeFetchTrace(QueryTrace.java:176)
at com.datastax.driver.core.QueryTrace.getDurationMicros(QueryTrace.java:105)
at com.cassandra.datastax.DatastaxTestBootstrapper.lambda$0(DatastaxTestBootstrapper.java:227)
at java.util.Collections$SingletonList.forEach(Collections.java:4822)
at com.cassandra.datastax.DatastaxTestBootstrapper.readData(DatastaxTestBootstrapper.java:227)
at com.cassandra.datastax.MultipleThread.run(MultipleThread.java:13)
at java.lang.Thread.run(Thread.java:748)
Is there any way I can overcome this exception or can I increase the number of retries to retrieve complete query trace?
Statement statement1 = new SimpleStatement(
"SELECT * FROM keyspace.table where key='405861500500033'").enableTracing();
ResultSet resultSet = session.execute(statement1);
resultSet.getAllExecutionInfo()
.forEach(e -> System.out.println("time : " + e.getQueryTrace().getDurationMicros()));
I am trying to fetch some data from azure data lake to azure datawarehouse, but I am unable to do it I have followed the documentation link
https://learn.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-load-from-azure-data-lake-store
But I am getting this error when I am trying to create an external table, I have created another web/api app but still was not able to access thE application here is the error which I am facing
EXTERNAL TABLE access failed due to internal error: 'Java exception raised on call to HdfsBridge_IsDirExist. Java exception message:
GETFILESTATUS failed with error 0x83090aa2 (Forbidden. ACL verification failed. Either the resource does not exist or the user is not authorized to perform the requested operation.). [0ec4b8e0-b16d-470e-9c98-37818176a188][2017-08-14T02:30:58.9795172-07:00]: Error [GETFILESTATUS failed with error 0x83090aa2 (Forbidden. ACL verification failed. Either the resource does not exist or the user is not authorized to perform the requested operation.). [0ec4b8e0-b16d-470e-9c98-37818176a188][2017-08-14T02:30:58.9795172-07:00]] occurred while accessing external file.'
Here is the script which I am trying to get it to work with
CREATE DATABASE SCOPED CREDENTIAL ADLCredential2
WITH
IDENTITY = '2ec11315-5a30-4bea-9428-e511bf3fa8a1#https://login.microsoftonline.com/24708086-c2ce-4b77-8d61-7e6fe8303971/oauth2/token',
SECRET = '3Htr2au0b0wvmb3bwzv1FekK88YQYZCUrJy7OB3NzYs='
;
CREATE EXTERNAL DATA SOURCE AzureDataLakeStore11
WITH (
TYPE = HADOOP,
LOCATION = 'adl://test.azuredatalakestore.net/',
CREDENTIAL = ADLCredential2
);
CREATE EXTERNAL FILE FORMAT TextFileFormat
WITH
( FORMAT_TYPE = DELIMITEDTEXT
, FORMAT_OPTIONS ( FIELD_TERMINATOR = '|'
, DATE_FORMAT = 'yyyy-MM-dd HH:mm:ss.fff'
, USE_TYPE_DEFAULT = FALSE
)
);
CREATE EXTERNAL TABLE [extccsm].[external_medication]
(
person_id varchar(4000),
encounter_id varchar(4000),
fin varchar(4000),
mrn varchar(4000),
icd_code varchar(4000),
icd_description varchar(300),
priority integer,
optional1 varchar(4000),
optional2 varchar(4000),
optional3 varchar(4000),
load_identifier varchar(4000),
upload_time datetime2,
xx_person_id varchar(4000),--Person ID is the ID that we will use to represent the person through out the process uniquely, This requires initial analysis to determine how to set it
xx_encounter_id varchar(4000),--Encounter ID is the ID that will represent the encounter uniquely through out the process, This requires initial analysis to determine hos to set it based on client data
mod_optional1 varchar(4000),
mod_optional2 varchar(4000),
mod_optional3 varchar(4000),
mod_optional4 varchar(4000),
mod_optional5 varchar(4000),
mod_loadidentifier datetime2
)
WITH
(
LOCATION='\testfiles\procedure_azure.txt000\',
DATA_SOURCE = AzureDataLakeStore11, --DATA SOURCE THE BLOB STORAGE
FILE_FORMAT = TextFileFormat, --TYPE OF FILE FORMAT
REJECT_TYPE = percentage,
REJECT_VALUE = 1,
REJECT_SAMPLE_VALUE = 0
);
Please tell me whats wrong here?
I can reproduce this but it's hard to narrow down exactly. I think it's to do with permissions. From the Azure portal:
Data Lake Store > yourDataLakeAccount > your folder > Access
From there, make sure your AD Application has Read, Write and Execute permission on the relevant files / folders. Start with one file initially. I can reproduce the error by assigning / unassigning the Execute permissions but need to repeat the steps to confirm. I'll retrace my steps but for now concentrate your search here. In my example below, my Azure Active Directory Application is called adwAndPolybase; you can see I've given it Read, Write and Execute. I also experimented with the Advanced and 'Apply to children' options: