The deployment with vscode run 100% fine,
in the log I see it uses oryx.
header:
import datetime
import logging
import adal
import requests
import json
I want to upload the code using Azure Pipelines though, for the sake of automation.
Here is my code
steps:
- bash: |
if [ -f extensions.csproj ]
then
dotnet build extensions.csproj --output ./bin
fi
displayName: 'Build extensions'
- task: UsePythonVersion#0
displayName: 'Use Python 3.9'
inputs:
versionSpec: '3.9'
- bash: |
python3.9 -m venv worker_venv
source worker_venv/bin/activate
pip3.9 install setuptools
pip3.9 install -r requirements.txt
displayName: 'Install application dependencies'
- task: ArchiveFiles#2
displayName: "Archive files"
inputs:
rootFolderOrFile: "$(System.DefaultWorkingDirectory)/functions"
includeRootFolder: false
archiveType: zip
archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
replaceExistingArchive: true
- publish: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
artifact: drop
- task: AzureFunctionApp#1
displayName: 'Deploy functions to Function App'
inputs:
azureSubscription: Service-Conn
appType: functionAppLinux
appName: 'pythontest'
package: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
runtimeStack: 'Python|3.9'
deploymentMethod: 'zipDeploy'
resourceGroupName: $(resourcegroup_name_app)
But I end up with module not found error (in the monitor of function in azure portal).
Result: Failure Exception: ModuleNotFoundError: No module named 'adal'.
the uploaded zip have site packages
there is no error in pipeline
What am I missing? Any ideas guys?
pip install --target="./.python_packages/lib/site-packages" -r ./requirements.txt is the command you want to run if you need to ship your libraries in the deployment zip file instead of running pip on function app service.
Your pipeline code installed libraries into .venv and function app runtime will not use that folder.
refs:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-how-to-azure-devops?tabs=dotnet-core%2Cyaml%2Cpython#build-your-app
https://learn.microsoft.com/en-us/azure/azure-functions/deployment-zip-push#deployment-customization
I believe you have to run a task to install pip in pipeline console. See link below for better clarity: https://learn.microsoft.com/en-us/azure/devops/pipelines/ecosystems/python?view=azure-devops
Related
My azure function app(python) is throwing an exception: module typing has no attribute '_classVar'. A fix for this would be to uninstall the dataclasses package. How do I uninstall this package on a python azure function using pip?
If I run pip uninstall dataclasses, will this reflect on deployment?
If you are using python version 3.7 or greater you need to uninstall the dataclass library using the same pip uninstall dataclasses.
As The dataclasses package is a backport of the Python 3.7 dataclass functionality.
Or, if still you want to exist dataclasses you can downgrade your python version to 3.6.
For more information please refer the below links:
Blog|AttributeError: module ‘typing’ has no attribute ‘_ClassVar’ with Tune
Similar GitHub Issue
I was also having a lot of trouble trying to deploy azure functions from an Azure Devops pipeline with a Python 3.7 environment, so I decided to place this here as it might help someone else with the same problem.
You need to prepare the following yaml file with your respective variables.
trigger:
- {{ branch }}
variables:
# Azure Resource Manager connection created during pipeline creation
azureSubscription: '{{ azureRmConnection.Id }}'
# Function app name
functionAppName: '{{ functionAppName }}'
# Agent VM image name
vmImageName: 'ubuntu-latest'
# Working Directory
workingDirectory: '{{ workingDirectory }}'
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- bash: |
if [ -f extensions.csproj ]
then
dotnet build extensions.csproj --runtime ubuntu.16.04-x64 --output ./bin
fi
workingDirectory: $(workingDirectory)
displayName: 'Build extensions'
- task: UsePythonVersion#0
displayName: 'Use Python 3.6'
inputs:
versionSpec: 3.6 # Functions V2 supports Python 3.6 as of today
- bash: |
pip install --target="./.python_packages/lib/site-packages" -r ./requirements.txt
rm -rf ./.python_packages/lib/site-packages/dataclasses-0.6*
rm ./.python_packages/lib/site-packages/dataclasses.py
workingDirectory: $(workingDirectory)
displayName: 'Install application dependencies'
- task: ArchiveFiles#2
displayName: 'Archive files'
inputs:
rootFolderOrFile: '$(workingDirectory)'
includeRootFolder: false
archiveType: zip
archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
replaceExistingArchive: true
- publish: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
artifact: drop
- stage: Deploy
displayName: Deploy stage
dependsOn: Build
condition: succeeded()
jobs:
- deployment: Deploy
displayName: Deploy
environment: 'development'
pool:
vmImage: $(vmImageName)
strategy:
runOnce:
deploy:
steps:
- task: AzureFunctionApp#1
displayName: 'Azure functions app deploy'
inputs:
azureSubscription: '$(azureSubscription)'
appType: functionAppLinux
appName: $(functionAppName)
package: '$(Pipeline.Workspace)/drop/$(Build.BuildId).zip'
These are the key lines after installing the requirements.txt. These will remove the package from the site-packages folder.
rm -rf ./.python_packages/lib/site-packages/dataclasses-0.6*
rm ./.python_packages/lib/site-packages/dataclasses.py
pip uninstall dataclasses will not work because you are not in the right folder.
Hope this helps!
so I´m trying to create a python environment in az pipelines and publish as an artifact in order to use it across jobs. I can publish the artifact, also it is downloaded in next job, problem I have is that seems that "environment/bin/activate" doesn´t work, what am I doing wrong? is there another way to achieve that?
build and publish artifact
jobs:
- job: BuildPythonArtifact
workspace:
clean: all
pool:
vmImage: $(UBUNTU_DEFAULT_VERSION)
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: "3.9"
displayName: "Use Python 3.9"
- script: |
python -m venv env
source env/bin/activate
python -m pip install --upgrade pip
pip install -r requirements.txt
workingDirectory: $(Build.SourcesDirectory)
displayName: "Install requirements"
- task: PublishPipelineArtifact#1
inputs:
targetPath: $(Build.SourcesDirectory)/env
artifactName: pythonenv
download artifact
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: "3.9"
displayName: "Use Python 3.9"
- task: DownloadPipelineArtifact#2
inputs:
artifact: pythonenv
- script: |
source $(Pipeline.Workspace)/bin/activate
files are there, bit seems that it ignores the env activate
What I expect is to can activate or reuse python env to avoid installing requirements in every job
I deploy Azure functions using yaml scripts.
For some reasons, I do not put all my package requirements in the file requirements.txt, as this is used by some other processes.
Yet, when deploying my web app through a YAML pipeline, I want to install additional python packages. However, the resulting app that is deployed crashes with errors saying that it does not have those packages installed.
my pipeline:
# Python to Linux Web App on Azure
# Build your Python project and deploy it to Azure as a Linux Web App.
# Change python version to one thats appropriate for your application.
# https://learn.microsoft.com/azure/devops/pipelines/languages/python
trigger:
- master
variables:
# Azure Resource Manager connection created during pipeline creation
azureServiceConnectionId: .........................
# Web app name
webAppName: .........................
# Agent VM image name
vmImageName: 'ubuntu-latest'
# Environment name
environmentName: .........................
# Project root folder. Point to the folder containing manage.py file.
projectRoot: $(System.DefaultWorkingDirectory)
# Python version: 3.8
pythonVersion: '3.8'
stages:
- stage: Build
displayName: Build stage
jobs:
- job: BuildJob
pool:
vmImage: $(vmImageName)
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '$(pythonVersion)'
displayName: 'Use Python $(pythonVersion)'
- script: |
python -m venv antenv
source antenv/bin/activate
python -m pip install --upgrade pip
pip install setup
pip install -r requirements.txt
pip install pyodbc
workingDirectory: $(projectRoot)
displayName: "Install requirements"
- task: ArchiveFiles#2
displayName: 'Archive files'
inputs:
rootFolderOrFile: '$(projectRoot)'
includeRootFolder: false
archiveType: zip
archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
replaceExistingArchive: true
- upload: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
displayName: 'Upload package'
artifact: drop
- stage: Deploy
displayName: 'Deploy Web App'
dependsOn: Build
condition: succeeded()
jobs:
- deployment: DeploymentJob
pool:
vmImage: $(vmImageName)
environment: $(environmentName)
strategy:
runOnce:
deploy:
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '$(pythonVersion)'
displayName: 'Use Python version'
- task: AzureWebApp#1
displayName: 'Deploy Azure Web App : .........................
inputs:
azureSubscription: $(azureServiceConnectionId)
appName: $(webAppName)
package: $(Pipeline.Workspace)/drop/$(Build.BuildId).zip
startUpCommand: 'startup-nocontainer.sh'
Here, pyodbc is installed separately in the pipeline, and not present in the file requirements.txt.
Checking the logs of the pipeline, I can see that the installation was performed successfully.
Yet when the app starts, it crashes at first import of pyodbc, as if the app that is eventually deployed only relyed on requirements.txt.
Any idea?
I think pyodbc is not being installed in the venv you create. The venv contains only the packages you specify in requirements.txt
Check the solutions proposed in this old post
I am trying to copy a file to Azure Databricks DBFS through Azure Devops pipeline. The following is a snippet from the yml file I am using:
stages:
- stage: MYBuild
displayName: "My Build"
jobs:
- job: BuildwhlAndRunPytest
pool:
vmImage: 'ubuntu-16.04'
steps:
- task: UsePythonVersion#0
displayName: 'Use Python 3.7'
inputs:
versionSpec: '3.7'
addToPath: true
architecture: 'x64'
- script: |
pip install pytest requests setuptools wheel pytest-cov
pip install -U databricks-connect==7.3.*
displayName: 'Load Python Dependencies'
- checkout: self
persistCredentials: true
clean: true
- script: |
echo "y
$(databricks-host)
$(databricks-token)
$(databricks-cluster)
$(databricks-org-id)
8787" | databricks-connect configure
databricks-connect test
env:
databricks-token: $(databricks-token)
displayName: 'Configure DBConnect'
- script: |
databricks fs cp test-proj/pyspark-lib/configs/config.ini dbfs:/configs/test-proj/config.ini
I get the following error at the stage where I am invoking the databricks fs cp command:
/home/vsts/work/_temp/2278f7d5-1d96-4c4e-a501-77c07419773b.sh: line 7: databricks: command not found
However, when I run databricks-connect test, it is able to execute the command successfully. Kindly help if I am missing some steps somewhere.
The databricks command is located in the databricks-cli package, not in the databricks-connect, so you need to change your pip install command.
Also, for databricks command you can just set the environment variables DATABRICKS_HOST and DATABRICKS_TOKEN and it will work, like this:
- script: |
pip install pytest requests setuptools wheel
pip install -U databricks-cli
displayName: 'Load Python Dependencies'
- script: |
databricks fs cp ... dbfs:/...
env:
DATABRICKS_HOST: $(DATABRICKS_HOST)
DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)
displayName: 'Copy artifacts'
P.S. Here is an example on how to do CI/CD on Databricks + notebooks. You could be also interested in cicd-templates project.
I currently have a pipeline in Azure Dev Ops configured to build, copy, archive, and then deploy a Vue.js app to an Azure App Service running on Linux. When I SSH into the server I do see the outputs of the build appearing successfully in the wwwroot folder but when I pull up the URL of the App Service I'm just shown the default landing page. Below are screenshots of my wwwroot folder, the page I'm seeing when I pull up the URL and a copy of the yml file for the pipeline. Is there something else missing from the yml file that I need to add to tell the app service to start the vue.js app by chance, I'm fairly new to this stuff.
# Build a Node.js project that uses Vue.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
trigger:
- master
pool:
vmImage: ubuntu-latest
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
npm run build
displayName: 'npm install and build'
workingDirectory: $(Build.SourcesDirectory)/client-app
- task: CopyFiles#2
inputs:
Contents: '$(Build.SourcesDirectory)/client-app/dist/**'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
flattenFolders: true
- task: ArchiveFiles#2
inputs:
rootFolderOrFile: '$(Build.ArtifactStagingDirectory)'
includeRootFolder: false
archiveType: 'zip'
archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
replaceExistingArchive: true
verbose: true
- task: AzureRmWebAppDeployment#4
inputs:
ConnectionType: 'AzureRM'
azureSubscription: 'Azure subscription 1 (f719f76c-c23e-4160-aa93-914eb7851fdb)'
appType: 'webAppLinux'
WebAppName: 'trashbot'
packageForLinux: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
You need to add startup command on portal.
pm2 serve /home/site/wwwroot --no-daemon --spa
Related Posts:
1. Application is not loading when deployed via azure app service
2. Default Documents (custom 404) on Azure AppService Linux website