I have a test account for snowflake. I can able to a fetch data from Python but unable to fetch it from Pyspark. Error is showing like unable to create a stage for shared DB. How the stage is creating in Python-snowflake connector?
i think the python connector creates a temp stage for the results in S3
you can see that when you run the connector in debug mode
Related
Good Day
I configured a Pipeline Copy Data job in Azure Data Factory to extract data from Jira with an API call using the rest API connector in Azure.
When i configure and test the connection it is successful.
Now when i try to preview the data in the Copy container i get the following error.
Does anyone know what this error means and how do i bypass it?
I believe i am not the first one trying to extract data from Jira via Rest API.
Thank you and Regards
Rayno
Error occurred when deserializing source JSON file ".Check if the data
is in valid JSON object format.Unexpected character encountered while
parsing value:<.Path".....
I think the error already indicates the root cause.You data format is invalid JSON format,you could try to simulate rest api invoke to make sure if the situation exists.ADF can't help you handle the illegal deserialization.
In addition,according to the connector doc,ADF supports JIRA connector.Maybe you could try to have a try on that.
I have a project in gitlab which has a script called dockerRun.sh that runs in a docker container. This script interacts with dynamodb and fetches data.
Below is the snippet from dockerRun.sh which fetches data from dynamodb table
cycleId=$(aws dynamodb scan --table-name cycle_info --filter-expression "isLatest = :a" --projection-expression 'cycleId' --expression-attribute-values '{":a":{"S":"true"}}')
Results is :
In getCycleId function
retrieving latest AIXM CycleId from dynamodb
None 1 2
CYCLEID 1911.191024
Now, I have another project in gitlab and in this project's cicd pipeline I have added a stage where I am calling the same dockerRun.sh from the previous project mentioned. While I run this pipeline and when the dockerRun.sh gets executed this time dynamodb fetch gives an error which is :
An error occurred (ResourceNotFoundException) when calling the Scan operation: Requested resource not found
I have cross checked that the data is present in the database.
So the summary is both time same script is running but while accessing it from another project it does not fetches result. Can somebody help with this.
Check that the other project has the right AWS credentials to access the account that has the DynamoDB table
I have setup a custom SSIS IR, however I'm having problems reading files from the current working directory or temp folder on the IR node
https://learn.microsoft.com/en-us/sql/integration-services/lift-shift/ssis-azure-files-file-shares?view=sql-server-2017
The work flow of my test package is
Load compressed file to Azure file share
Unzip file
Modify file, saving it the current working group folder on the IR node (this path .\testfile.json)
Load file to Azure SQL DB
The last step is where I'm having issues, I receive the below error message. Maybe looks to be related to security, but no idea how to access the SSIS IR node to check this.
Execute SQL Task:Error: Executing the query "DECLARE #request
VARCHAR(MAX) SELECT #request =..." failed with the following error:
"Cannot bulk load because the file ".\testfile.json" could not be
opened. Operating system error code (null).". Possible failure
reasons: Problems with the query, "ResultSet" property not set
correctly, parameters not set correctly, or connection not established
correctly.
How can I fix this issue?
From just the error message, looks like you're using BULK INSERT in Execute SQL Task to load data into Azure SQL DB. BULK INSERT into Azure SQL DB can only work from Azure Storage Blob, but not from file systems/SSIS IR nodes. To load data from the current working directory of SSIS IR nodes into Azure SQL DB, you can use a Data Flow with Flat File Source and ADO.NET Destination.
I'm trying to play with AWS Data Pipelines (and then Glue later) and am following Copy MySQL Data Using the AWS Data Pipeline Console. However, when I execute the pipeline, I get
DriverClass not found for database:mariadb
I would expect this to "just work," but why is it not providing it's own driver? Or is driver for MySQL not equal to driver for MariaDB?
Right, after fighting with this all day, I found the following link which solves it: https://forums.aws.amazon.com/thread.jspa?messageID=834603&tstart=0
Basically:
You are getting the error because you are using the RdsDatabase, it needs to be the JdbcDatabase when using mariadb.
"type": "JdbcDatabase",
"connectionString": "jdbc:mysql://thing-master.cpbygfysczsq.eu-west-1.rds.amazonaws.com:3306/db_name",
"jdbcDriverClass" : "com.mysql.jdbc.Driver"
FULL credit goes to Webstar34 (https://forums.aws.amazon.com/profile.jspa?userID=452398)
I am trying to load custom properties in my spark application using :-
command-runner.jar,spark-submit,--deploy-mode,cluster,--properties-file,s3://spark-config-test/myprops.conf,--num-executors,5,--executor-cores,2,--class,com.amazon.Main,#{input.directoryPath}/SWALiveOrderModelSpark-1.0-super.jar
However, I am getting the following exception:-
Exception in thread "main" java.lang.IllegalArgumentException: Invalid
properties file 's3://spark-config-test/myprops.conf''. at
org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
at
org.apache.spark.launcher.AbstractCommandBuilder.loadPropertiesFile(AbstractCommandBuilder.java:284)
at
org.apache.spark.launcher.AbstractCommandBuilder.getEffectiveConfig(AbstractCommandBuilder.java:264)
at
org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:233)
at org
Is this the correct way to load file from S3?
You can't load a properties file directly from S3. Instead you will need to download the properties file to your master node somewhere, then submit the spark job referencing the local path on that node. You can do the download by using command-runner.jar to run the aws cli utility.