I have error when running a pipeline in azure synapse. If I execute the synapse notebook manually it works good (readind and writing). But when I call the name notebook from the ForEach activity inside the pipeline it fails to run. I have the following error. Previously, I had no porblem.
Error
{
"errorCode": "6000",
"message": "{\n \"code\": 400,\n \"message\": \"Failed to run notebook due to invalid request. [Error: Not supported language in Synapse: ]\",\n \"result\": {\n \"errorMessage\": null,\n \"details\": null\n }\n}",
"failureType": "UserError",
"target": "1 - Full load Parquet",
"details": []
}
Maybe worth to mention that, If I run first cmd i notebook manually and then trigger the pipeline, it run perfectly. So I dont know why but I requires the manuall start of engine?
Related
I'm running a rather hefty build in my Azure pipeline, which involves processing a large amount of data, and hence requires too much memory for my buildagent to handle. My approach is therefore to start up an linux VM, run the build there, and push up the resulting docker image to my container registry.
To achieve this, I'm using the Azure CLI task to issue commands to the VM (e.g. az vm start, az vm run-command ... etc).
The problem I am facing is that az vm run-command "succeeds" even if the script that you run on the VM returns a nonzero status code. For example, this "bad" vm script:
az vm run-command invoke -g <group> -n <vmName> --command-id RunShellScript --scripts "cd /nonexistent/path"
returns the following response:
{
"value": [
{
"code": "ProvisioningState/succeeded",
"displayStatus": "Provisioning succeeded",
"level": "Info",
"message": "Enable succeeded: \n[stdout]\n\n[stderr]\n/var/lib/waagent/run-command/download/87/script.sh: 1: cd: can't cd to /nonexistent/path\n",
"time": null
}
]
}
So, the command succeeds, presumably because it succeeded in executing the script on the VM. The fact that the script actually failed on the VM is buried in the response "message"
I would like my Azure pipeline task to fail if the script on the VM returns a nonzero status code. How would I achieve that?
One idea would be to parse the response (somehow) and search the text under stderr - but that sounds like a real hassle, and I'm not sure even how to "access" the response within the task.
Have you enabled the option "Fail on Standard Error" on the Azure CLI task? If not, you can try to enable it and run the pipeline again to see if the error "cd: can't cd to /nonexistent/path" can make the task run failed.
If the task still is passed, the error "cd: can't cd to /nonexistent/path" should not be a Standard Error. In this situation, you may need to add more command lines in your script to monitor the output logs of the az command. Once there is any output message shows error, execute "exit 1" to exit the script and return a Standard Error to make the task be failed.
I solved this by using the SSH pipeline task - this allowed me to connect to the VM via SSH, and run the given script on the machine "directly" via SSH.
This means from the context of the task, you get the status code from the script itself running on the VM. You also see any console output inside the task logs, which was obscured when using az vm run-command.
Here's an example:
- task: SSH#0
displayName: My VM script
timeoutInMinutes: 10
inputs:
sshEndpoint: <sshConnectionName>
runOptions: inline
inline: |
echo "Write your script here"
Not that the SSH connection needs to be set up as a service connection using the Azure pipelines UI. You reference the name of the service connection you set up in yaml.
I am learning about K8s and did setup a release pipeline with a kubectl apply. I've setup the AKS cluster via Terraform and on the first run all seemed fine. Once I destroyed the cluster I reran the pipeline, I get issues which I believe are related to the kubeconfig file mentioned in the exception. I tried the cloud shell etc. to get to the file or reset it but I wasn't succesful. How can I get back to a clean state?
2020-12-09T09:08:51.7047177Z ##[section]Starting: kubectl apply
2020-12-09T09:08:51.7482440Z ==============================================================================
2020-12-09T09:08:51.7483217Z Task : Kubectl
2020-12-09T09:08:51.7483729Z Description : Deploy, configure, update a Kubernetes cluster in Azure Container Service by running kubectl commands
2020-12-09T09:08:51.7484058Z Version : 0.177.0
2020-12-09T09:08:51.7484996Z Author : Microsoft Corporation
2020-12-09T09:08:51.7485587Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/deploy/kubernetes
2020-12-09T09:08:51.7485955Z ==============================================================================
2020-12-09T09:08:52.7640528Z [command]C:\ProgramData\Chocolatey\bin\kubectl.exe --kubeconfig D:\a\_temp\kubectlTask\1607504932712\config apply -f D:\a\r1\a/medquality-cordapp/k8s
2020-12-09T09:08:54.1555570Z Unable to connect to the server: dial tcp: lookup mq-k8s-dfee38f6.hcp.switzerlandnorth.azmk8s.io: no such host
2020-12-09T09:08:54.1798118Z ##[error]The process 'C:\ProgramData\Chocolatey\bin\kubectl.exe' failed with exit code 1
2020-12-09T09:08:54.1853710Z ##[section]Finishing: kubectl apply
Update, workflow tasks of the release pipeline:
Initially I get the artifact, clone of the repo containing the k8s yamls, then the stage does a kubectl apply.
"workflowTasks": [
{
"environment": {},
"taskId": "cbc316a2-586f-4def-be79-488a1f503564",
"version": "0.*",
"name": "kubectl apply",
"refName": "",
"enabled": true,
"alwaysRun": false,
"continueOnError": false,
"timeoutInMinutes": 0,
"definitionType": null,
"overrideInputs": {},
"condition": "succeeded()",
"inputs": {
"kubernetesServiceEndpoint": "82e5971b-9ac6-42c6-ac43-211d2f6b60e4",
"namespace": "",
"command": "apply",
"useConfigurationFile": "false",
"configuration": "",
"arguments": "-f $(System.DefaultWorkingDirectory)/medquality-cordapp/k8s",
"secretType": "dockerRegistry",
"secretArguments": "",
"containerRegistryType": "Azure Container Registry",
"dockerRegistryEndpoint": "",
"azureSubscriptionEndpoint": "",
"azureContainerRegistry": "",
"secretName": "",
"forceUpdate": "true",
"configMapName": "",
"forceUpdateConfigMap": "false",
"useConfigMapFile": "false",
"configMapFile": "",
"configMapArguments": "",
"versionOrLocation": "version",
"versionSpec": "1.7.0",
"checkLatest": "false",
"specifyLocation": "",
"cwd": "$(System.DefaultWorkingDirectory)",
"outputFormat": "json",
"kubectlOutput": ""
}
}
]
```
I can see you are using kubernetesServiceEndpoint as the Service connection type in Kubectl task.
Once I destroyed the cluster I reran the pipeline, I get issues....
If the cluster was destroyed. The kubernetesServiceEndpoint in azure devops is still connected to the origin cluster. Kubectl task which using the origin kubernetesServiceEndpoint is still looking for the old cluster. And it will fail with above error, since the old cluster was destroyed.
You can fix this issue by updating the kubernetesServiceEndpoint in azure devops with the newly created cluster:
Go to Azure devops Project settings-->Service connections--> Find your Kubernetes Service connection-->Click Edit to update the configuration.
But if your kubernete cluster gets destroyed and recreated frequently. I would suggest using Azure Resource Manager as the Service connection type to connect to the cluster in Kubectl task. See below screenshot.
By using azureSubscriptionEndpoint and specifying azureResourceGroup, if only the cluster's name doesnot change, It doesnot matter how many times the cluster is recreated.
See document to create an Azure Resource Manager service connection
When you destroy and reprovision AKS cluster the kube API URL and some other things change, but as you found out, nothing updates this automatically on your configured clients.
What I do to get access new and reprovisioned AKS clusters is :
az aks get-credentials --subscription <sub> -g <rg> -n <aksname> -a --overwrite
I'm trying to list steps in a pipeline using AzureML CLI extension, but get an error:
>az ml run list -g <group> -w <workspace> --pipeline-run-id 00886abe-3f4e-4412-aec3-584e8c991665
UserErrorException:
Message: Cannot specify ['--last'] for pipeline runs
InnerException None
ErrorResponse
{
"error": {
"code": "UserError",
"message": "Cannot specify ['--last'] for pipeline runs"
}
}
From help it looks like --last option takes the default value 10 despite the fact that it is not supported for the --pipeline-run-id. How the latter is supposed to work?
I'm very new to Azure IoT Edge and I'm trying to deploy to my Raspberry PI : Image Recognition with Azure IoT Edge and Cognitive Services
but after Build & Push IoT Edge Solution and Deploy it to Single Device ID I see none of those 2 modules listed in Docker PS -a & Iotedge list
And when try to check it on EdgeAgent Logs there's error message and it seems EdgeAgent get error while creating those Modules (camera-capture and image-classifier-service)
I've tried :
1. Re-build it from fresh folder package
2. Pull the image manually from Azure Portal and run the image manually by script
I'm stuck on this for days.
in deployment.arm32v7.json for those modules I define the Image with registered registry url :
"modules": {
"camera-capture": {
"version": "1.0",
"type": "docker",
"status": "running",
"restartPolicy": "always",
"settings": {
"image": "zzzz.azurecr.io/camera-capture-opencv:1.1.12-arm32v7",
"createOptions": "{\"Env\":[\"Video=0\",\"azureSpeechServicesKey=2f57f2d9f1074faaa0e9484e1f1c08c1\",\"AiEndpoint=http://image-classifier-service:80/image\"],\"HostConfig\":{\"PortBindings\":{\"5678/tcp\":[{\"HostPort\":\"5678\"}]},\"Devices\":[{\"PathOnHost\":\"/dev/video0\",\"PathInContainer\":\"/dev/video0\",\"CgroupPermissions\":\"mrw\"},{\"PathOnHost\":\"/dev/snd\",\"PathInContainer\":\"/dev/snd\",\"CgroupPermissions\":\"mrw\"}]}}"
}
},
"image-classifier-service": {
"version": "1.0",
"type": "docker",
"status": "running",
"restartPolicy": "always",
"settings": {
"image": "zzzz.azurecr.io/image-classifier-service:1.1.5-arm32v7",
"createOptions": "{\"HostConfig\":{\"Binds\":[\"/home/pi/images:/images\"],\"PortBindings\":{\"8000/tcp\":[{\"HostPort\":\"80\"}],\"5679/tcp\":[{\"HostPort\":\"5679\"}]}}}"
}
Error message from EdgeAgent Logs :
(Inner Exception #0) Microsoft.Azure.Devices.Edge.Agent.Edgelet.EdgeletCommunicationException- Message:Error calling Create module
image-classifier-service: Could not create module image-classifier-service
caused by: Could not pull image zzzzz.azurecr.io/image-classifier-service:1.1.5-arm32v7
caused by: Get https://zzzzz.azurecr.io/v2/image-classifier-service/manifests/1.1.5-arm32v7: unauthorized: authentication required
When trying to run the pulled image by script :
sudo docker run --rm --name testName -it zzzz.azurecr.io/camera-capture-opencv:1.1.12-arm32v7
None
I get this error :
Camera Capture Azure IoT Edge Module. Press Ctrl-C to exit.
Error: Time:Fri May 24 10:01:09 2019 File:/usr/sdk/src/c/iothub_client/src/iothub_client_core_ll.c Func:retrieve_edge_environment_variabes Line:191 Environment IOTEDGE_AUTHSCHEME not set
Error: Time:Fri May 24 10:01:09 2019 File:/usr/sdk/src/c/iothub_client/src/iothub_client_core_ll.c Func:IoTHubClientCore_LL_CreateFromEnvironment Line:1572 retrieve_edge_environment_variabes failed
Error: Time:Fri May 24 10:01:09 2019 File:/usr/sdk/src/c/iothub_client/src/iothub_client_core.c Func:create_iothub_instance Line:941 Failure creating iothub handle
Unexpected error IoTHubClient.create_from_environment, IoTHubClientResult.ERROR from IoTHub
When you pulled the image directly with docker run, it pulled but then failed to run outside of the edge runtime, which is expected. But when the edge agent tried to pull it, it failed because it was not authorized. No credentials were supplied to the runtime, so it attempted to access the registry anonymously.
Make sure that you add your container registry credentials to the deployment so that edge runtime can pull images. The deployment should contain something like the following in the runtime settings:
"MyRegistry" :{
"username": "<username>",
"password": "<password>",
"address": "<registry-name>.azurecr.io"
}
As #silent pointed out in the comments, the documentation is here, including an example deployment that includes container registry credentials.
I'm presently hosting an Azure Scale set running Windows Server 2012 R2 that is setup with the Chef extension (Chef.Bootstrap.WindowsAzure.ChefClient). When the VM is provisioned, the extension reports back that it succeeded via the Azure portal however the registered node on the Chef server is not updated to retain the provided run list and the first run isn't fully completed. This is causing subsequent chef-client runs to be performed with an empty run list. When I observe the reports on chef server, I see a run with a status of aborted with no error.
Upon review of the WindowsAzure Plugins chef-client.log file, I can see that it tries to execute the run list but seems to be interrupted with the following FATAL
FATAL: Errno::EINVAL: Invalid argument # io_writev - <STDOUT>
There is no chef-stacktrace.out file created as well. The ARM extension definition looks like:
{
"type": "extensions",
"name": "ChefClient",
"properties": {
"publisher": "Chef.Bootstrap.WindowsAzure",
"type": "ChefClient",
"typeHandlerVersion": "1210.12",
"autoUpgradeMinorVersion": true,
"settings": {
"client_rb": "ssl_verify_mode :verify_none\nnode_name ENV[\"COMPUTERNAME\"]",
"runlist": "recipe[example]",
"autoUpdateClient": "false",
"deleteChefConfig": "false",
"bootstrap_options": {
"chef_server_url": "https://mychefserver.com/organizations/myorg",
"validation_client_name": "myorg-validator",
"environment": "dev"
}
},
"protectedSettings": {
"validation_key": "-----BEGIN RSA PRIVATE KEY----- ... -----END RSA PRIVATE KEY----"
}
}
}
In order to troubleshoot, I've tried to reduce my example cookbook down to a single DSC script which installs IIS. Even this step, I've executed it multiple ways such as using windows_feature, powershell_script, and dsc_script. All result end up with the same error. Here is the current script
powershell_script 'Install IIS' do
code 'Add-WindowsFeature Web-Server'
guard_interpreter :powershell_script
not_if "(Get-WindowsFeature -Name Web-Server).Installed"
end
If I override the run list and call chef-client manually, everything succeeds. I'm having trouble honing in on whether this is the Azure Chef Extension, the Chef client, or the cookbook.
As far as I can tell, communication with the Chef server looks good as the necessary pem files are exchanged, chef-client is installed, and the cookbook is downloaded and cached from the server. The cache gets removed on the subsequent run however with the empty run list. Here are the contents of first-boot.json:
{"run_list":["recipe[example]"]}
Here are the versions in play:
chef-client version: 14.1.12
Azure Chef Extension version: 1210.12.110.1001
Server version: Windows Server 2012 R2
Any ideas what could be going on?
It turns out my analysis was incorrect about which resource was causing the problem. It appears that the first boot run was failing when using dsc_script as the resource to install the web server. When using the following powershell_script resource, it succeeded and the run list attached for future runs.
powershell_script 'Install IIS' do
code 'Add-WindowsFeature Web-Server'
guard_interpreter :powershell_script
not_if "(Get-WindowsFeature -Name Web-Server).Installed"
end