Databricks recently added support for "files in repos" which is a neat feature. It gives a lot more flexibility to the projects, since we can now add .json config files and even write custom python modules that exists solely in our closed environment.
However, I just noticed that the standard way of deploying from an Azure git repo to a workspace does not support arbitrary files. First off, all .py files are converted to notebooks, breaking the custom modules that we wrote for our project. Secondly, it intentionally skips files ending in one of the following: .scala, .py, .sql, .SQL, .r, .R, .ipynb, .html, .dbc, which means our .json config files are missing when the deployment is finished.
Is there any way to get around these issues or will we have to revert everything to use notebooks like we used to?
You need to stop doing deployment the old way as it depends on the Workspace REST API that doesn't support arbitrary files. Instead you need to have a Git checkout in your destination workspace, and update that checkout to a given branch/tag when doing release. This is could be done via Repos API, or databricks cli. Here is an example of how to do that with cli from DevOps pipeline.
- script: |
echo "Checking out the releases branch"
databricks repos update --path $(STAGING_DIRECTORY) --branch "$(Build.SourceBranchName)"
env:
DATABRICKS_HOST: $(DATABRICKS_HOST)
DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)
displayName: 'Update Staging repository'
Related
I am using the 'az repos create' command to create a git repo.
But at this point, there is no master branch created. I understand I can do a git add, commit and push to create the master branch. But this requires a clone to be made first. I am wondering if there is a way to automatically create the README.md using this CLI extension like while creating a repo from the GUI, it gives you an option to initialize the master branch with this file.
I checked some commands for Azure Repos, such as 'az repos create', 'az repos update' and 'az repos ref', did not find any available option to initialize the new repository with a README.md.
I also checked the related REST API, also did not find any available option to do this.
Currently, after creating the new repository via the az repos create command, you need to open the repository page in your web browser (you can use the --open parameter in the command), then manually initialize the repository with a README.md.
If you really need the option, I recommend that you can report a feature request on the repository Azure/azure-cli to ask adding an option to allow initializing the repository with a README.md when creating it via the az repos create command.
I've a working build pipeline in Azure Devops that essentially installs Python3.6, sets up a virtual environment (.env) and then executes all unit tests. It then uses as its final step, a copy operation to move all files, including the virtual environment to a drop folder.
My problem arises from creating a release pipe. I am running a bash script for the release pipeline that essentially installs the azure functions command tools, and then I activate the python virtual environment before I call the func azure publish instruction.
The error I get states that settings are encrypted and that I need to call func setting add to add settings, however, when run locally, the script executes without any error whatsoever.
Does anyone have a working release pipeline in Azure Devops for a python-based Azure Function that they'd be able to share with me, so I can perhaps see what I am doing wrong?
Here is the relevant bit of script that executes:
#!/usr/bin/env bash
FUNCTION_APP_NAME="secret"
FUNCTION_APP_FOLDER="evenMoreSecret"
# Install Azure Functions Core Tools
echo "--> Install Azure Functions Core Tools"
wget -q https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo apt-get update
sudo apt-get install azure-functions-core-tools -y
echo ">>>>>>>> Initialize Python Virtual Environment"
source .env/bin/activate
echo "--> Publish the function app to Azure Functions"
cd $FUNCTION_APP_FOLDER
func azure functionapp publish $FUNCTION_APP_NAME --build-native-deps
The script is executed using an Azure CLI, using a security principal which is tied to the azure account that it is targeting.
Usually with Azure DevOps you create several build steps that result in some build artifacts - these are defined in the azure-pipelines.yml file. You then do a release step to release the artifacts that you have created - this is created within the UI. This can involve deploying to a test server and then to production or however you want to configure it. What you are describing is doing the build and release step all in the one yaml file as the func publish is essentially doing a release and it seems to all be in the one script.
In the next release of the az cli there is a new command called az functionapp devops-build that will set up the DevOps pipeline with the seperate build and release steps. However, in the mean time, we have created a series of beta yaml files that we hope you can just drag and drop to do the build and release steps just within the build part (as you are doing).
The beta yaml files are here:
https://github.com/Azure/azure-functions-devops-build/wiki/Yaml-Samples
I must disclaim that they are not fully tested, nor are they supported yet.
I will answer myself as I've solved the problem.
To #Oliver Dolk: We do NOT want to publish as part of a build pipeline. The only thing I'm interested in is to set up a virtual environment and then run the unit tests.
The RELEASE stage is where we want to deploy the scripts copied over from the build step. These artifacts are then the basis for releasing into dev, test and prodution environment.
I was missing a very important step in my script; To create a local.settings.json file which contains encrypted settings for the functionapp.
In order to solve the problem, I only had to call the following:
func azure functionapp fetch-app-settings $FUNCTION_APP_NAME
This calls the azure functionApp, and retrieves it's settings into an encrypted local.settings.json which is then used during publishing.
For a complete script reference of both the build YAML script and the bash script that does the deployment, I've put both in an anonimized github repo:
https://github.com/digitaldias/Python-Examples
I'm trying to use Azure Web Apps (Linux) to host a basic static site. I configured everything so a new deployment happens with every Git push. I put my pre-built pages in my repo to confirm everything works fine with this setup.
Now I've removed the pre-built pages and kept only the templates and the build script (which is basically just an npm install and a mustatic 'compile') and I'd like to run this build script in my web app. I've scoured the internet but can't find anything.
How can I run a script upon first deployment and after each Git-push-triggered deployment?
How can I run a script upon first deployment and after each Git-push-triggered deployment?
First, you need to generate custom deployment script by using azure-cli tool.
1) Set the cli working mode to asm.
azure config mode asm
2) Run the custom deployment script generator command.
azure site deploymentscript --node -t bash
This will generate the files required to deploy your site.
.deployment - Contains the command to run for deploying your site.
deploy.sh - Contains the deployment script.
Now you can edit the deploy.cmd file and add your custom steps.
After that done, add the generated files to your repository (.deployment and deploy.sh) and push your repository to your Azure Web App and see your custom deployment running.
For more details, please refer to this blog post.
I setup a git server on Debian and created a folder called git in /srv/git/.
Inside the folder, I'd like to divide my git projects by cloud hosted cloud and on-prem hosted on-prem. Inside of these two folders contain the hostname and it's specific files needed for deployment, redeployment, etc.
So a server that does apache on the cloud with a hostname like webapp will have a folder structure like /srv/git/repos/cloud/webapp.
Should the command git init be ran from the cloud folder or webapp folder?
I presume your setting up the bare repo.
As you mentioned in the question, you want to have separate git projects plural here so what I understood is you want to have more than one project under cloud directory.
Method 1:
cloud
|------->webapp
|------->something1
|------->something2
Very straight way is doing /srv/git/repos/cloud/webapp.git here webapp is the git repo.
Setting up the webapp as git repo.
git init --bare /srv/git/repos/cloud/webapp.git
--bare: for the bare repo you can skip this using on exisiting repo.
Though init is not harmful to give on existing repository is safe. It will not overwrite things that are already there
Method 2:
cloud (git repo)
|------->webapp (submodule)
|------->something1 (submodule)
|------->something2 (submodule)
In another option where you can keep cloud as git repo and make webapp as submodule. That is repo inside the git repo.
You can choose what fits to you best.
I suggest to use two different repo, if your have plenty of submodules in one git repo it might be cumbersome at the start to keep track of things.
The answer to your question is that you should run git init from the cloud folder, giving webapp as the first argument, i.e. git init webapp.
That said, you could also create the webapp directory, change into it, and then run git: mkdir webapp; cd webapp; git init; The end result will be exactly the same, but why bother?.
I'm considering moving my php Google App Engine project from Codenvy to the Google Source Repository and edit it there with the Source Editor, but I don't see how to cause it to deploy my project. How do I do that?
Here's what works for me, found through guesswork, trial, error, Billy and only a little docs.
I had set up GC Repositories to have a repository which is a mirror to bitbucket , auto-named default. Note: gcloud for default below can fail to recognise a repository that got that name by Rename. And can mistake a non-existent repository for an empty one.
Recipe 1
UPDATE: Now, after updating the bitbucket source, the deployed app does not show the update, despite "Deployment successful"]4. I don't know why - perhaps due to version number. Workaround: Use Recipe 2.
1 Ensure project's app.yaml file contains application: and version: e.g. this
2 Go to Google Cloud Patform and select the project
3 Click Activate Google Cloud Shell http://i.imgur.com/Axjy17q.png
4 In Google Cloud Shell, enter:
gcloud source repos clone default
appcfg.py update default
rm -rf default
This took ~20s to deploy and ~30s to complete.
Recipe 2
1 Ensure project's app.yaml file does not contain application: or version: (else you'll get an error like this) e.g. this
2 Go to Google Cloud Patform and select the project
3 Click Activate Google Cloud Shell http://i.imgur.com/Axjy17q.png
4 In Google Cloud Shell enter:
gcloud source repos clone default
gcloud --quiet app deploy default/app.yaml
rm -rf default
Warning: This can leave a previous version accessible.
This took ~65s to complete.
Re timing, compare this, taking ~20s . Timings are for a Hello World project.