I'm trying to install Giraph on HDInsight cluster with hadoop, using script actions.
After 30+- minutes when deploying the cluster, an error shows up.
Deployment failed
Deployment to resource group 'graphs' failed. Additional details from
the underlying API that might be helpful: At least one resource
deployment operation failed. Please list deployment operations for
details. Please see https://aka.ms/arm-debug for usage details.
Thanks in advance.
Thanks a lot for reporting this issue. We found what the issue is and fixed it.
Issue: There’s a deadlock when the Giraph script is provided during cluster creation. The Giraph script waits for /example/jars to be created in DFS (Wasb/ADLS), but /example/jars can only be created after the Giraph script completes. This issue doesn’t repro for runtime scripts since at the point the script is run, /example/jars has already existed.
Note: We have created and deployed the fix for the scripts. And I have also tested creating a cluster with the updated version, which works fine. Please test on your side and let me know.
Related
Instead of the expected output from a display(my_dataframe), I get Failed to fetch the result. Retry when looking at the completed run (also marked as success).
The notebook runs fine, including the expected outputs, when run as an on-demand notebook (same cluster config etc.). It seems to be a UI issue? I honestly don't even know where to look for possible causes.
I had the same problem while running a job on Azure Databricks and restarting my computer (or maybe the explorer..) helped.
The script install_hdisamples.py is executed at HDInsight cluster creation. It creates some folders with file samples and some tables in the referenced storage.
Is it possible to avoid the deployment of such script at cluster creation? Can it be done in the ARM template?
I don't think it's currently possible to avoid running the script. That said, you could create a script action that undoes what the script does. This is a flaky solution, and I acknowledge it's not perfect.
I am more interested in understanding the motivation for wanting to stop the script from running. Would you mind elaborating?
I have created a Spark HDInsight Cluster on Azure. The cluster was used to run different jobs (either Spark or Hive).
Until a month ago, the history of the jobs could be seen in the Spark History Server dashboard. It seems that following the update that introduced Spark 1.6.0, this dashboard is no longer showing any applications.
I have also tried to bypass this issue by executing the PowerShell cmdlet for get-azurehdinsightjob as sugested here. The output is again an empty list of applications.
I would appreciate any help as this dashboard used to work and now all my experiments are stalled.
I managed to solve the issue by deleting everything inside wasb:///hdp/spark-events. Maybe the issue was related to the size of the folder, as no other log files could be appended.
All the following jobs are now appearing successfully in the Spark History Server dashboard.
I'm deploying an Azure data factory by deploying an ARM template using Visual Studio, basically following this Azure tutorial exactly, step by step.
The template defines a data factory, with an Azure Storage linked service (for reading and writing source and output data), an input dataset and an output data set, an HDInsight on-demand linked service, and a pipeline which runs an HDInsight HIVE activity to run a HIVE script which processes the input datasets into an output data set.
Everything deploys successfully and the pipeine activity starts. However I get the following error from the activity:
Exception in thread "main" java.lang.RuntimeException:
java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:445)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) at
org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:619) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
org.apache.hadoop.util.RunJar.main(RunJar.java:136)
I have found various posts such as this one and this one suggesting the porblem is a known bug caused by a dash or hyphen in the HIVE metastore database name.
My problem is that using an ARM template to deploy an HDInsigh cluster on demand, I have no access to the cluster itself, so I can't make any manual config changes (the idea of on-demand is that it is transient, only created to serve a set of demands and then deletes itself).
The issue can be reproduced easily simply by following the tutorial step by step.
The only possible glimmer of hope I have found is by setting the hcatalogLinkedServiceName as documented here, which is designed to allow you to use your own Azure SQL database as the hive metastore. However, this doesn't work either - if I use that property, I get:
‘JamesTestTutorialARMDataFactory/HDInsightOnDemandLinkedService’
failed with message ‘HCatalog integration is not enabled for this
subscription.’
My subscription is unrestricted, and should have all the features of Azure available. So now I'm completely stuck. It seems that currently, using Hive with on-demand HDInsight is basically impossible?
If anyone can think of anything to try, I'm all ears!
Thanks
I recently studied the tutorial, and revised the tutorial. Here is my version, https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-create-linux-clusters-adf/. I didn't see the error. The Hive table name doesn't have hyphen. I think mine is a bit easier to follow. I made a few minor changes on the template itself.
I managed to get in contact with the author of the tutorial on GIT - who contacted the Azure product team, and this was their response:
...this is a known issue with Linux based HDI clusters when you use
them with ADF. The HDI team has fixed the issue and will be deploying
the fix in the coming weeks. In the meantime, you have to use Window
based HDI clusters with ADF.
Please use windows as osType for now. I have fixed the article in GIT
and it will go live some time tomorrow.
The tutorial I link to has indeed been changed to use windows instead of linux. I've tried this and it works.
I am trying out the Kubernetes setup from here The problem is that it hangs with the following output :
Waiting for cluster initialization.
This will continually check to see if the API for kubernetes is reachable.
This might loop forever if there was some uncaught error during start
up.
...................................................................................................................................................................
And it just hangs there?
According to https://github.com/chanezon/azure-linux/tree/master/coreos/kubernetes Kubernetes for GIT is currently broken, I also wasnt able to get it running unfortunately :(