my ideal case is to build a secure CI/CD pipeline using cloud build applying SAST and other security test in order to prevent the developer to deploy code that has a smell or have any vulnerabilities
I currently using Github as my version control platform and hosting my app on google cloud
and with the help of google cloud build i did build a CI/CD pipeline to deploy my app each time we have an update , the current flow is whenever create a pull request in one of the main branches that reflects our environments prod/staging/dev a check status is triggered which is cloud build CI/CD.
the pipeline currently just build, scan and then push the image also starting the managed insatnce group update but its now my focus for this question, i am trying to follow the secure methodologies for DevSecOps by applying some security tests in the early stages like SAST,SCA and image scanning
so the issue is that google does not have the full ecosystem currently and no clear path for DevSecOps using cloud build as the Ci tool for such as what third party tools to integrate with in order to fill the gaps? this was the only clear path i did found until now https://cloud.google.com/blog/products/devops-sre/devsecops-and-cicd-using-google-cloud-built-in-services a blog to follow but not answering all the questions
what i have achieved until now is applying an IDE SAST tool like sonarlint also i found snyk
Although i mainly focused on the free tools as a start in order to test because as i said no clear integrations with cloud build is out there except for snyk.io i did found an integration with cloud build but its even not a native support yet
the reason for using cloud build not github actions that i don't want to store any service account key outside our project because it will be a general behavior in my company and its a more safer to keep our keys inside , so i need some ideas what tools i can integrate with cloud build in order to achieve that and how to apply it and is there a way to use github actions and cloud build at the same time while orchestrating the pull request check status order ? for example check the source code before start building the docker image
Related
Some context: Our Cloud Build process relies on manual triggers and about 8 substitutions to customize deploys to various firebase projects, hosting sites, and preview channels. Previously we used a bash script and gcloud to automate the selection of these substitution options, the "updating" of the trigger (via gcloud beta builds triggers import: our needs require us to use a single trigger, it's a long story), and the "running" of the trigger.
This bash script was hard to work with and improve, and through the import-run shenanigans actually led to some faulty deploys that caused all kinds of chaos: not great.
However, recently I found a way to pass substitution variables as part of a manual trigger operation using the Node.js library for Cloud Build (runTrigger with subs passed as part of the request)!
Problem: So I'm converting our build utility to Node, which is great, but as far as I can tell there isn't a native way to steam build logs from a running build in the console (except maybe with exec, but that feels hacky).
Am I missing something? Or should I be looking at one of the logging libraries?
I've tried my best scanning Google's docs and APIs (Cloud Build REST, the Node client library, etc.) but to no avail.
I couldn't find any documentation about build steps on the flexible environment. Only thing I found is that App Engine will run the start script from your package.json file after deployment, but is it possible to make it run the build script first? This is what Heroku does and I want to replicate it.
What you're looking for is the script called gcp-build as this one can perform a custom build step at deployment, just before starting the application. While this is only documented for Standard Environment as of now (I've let the engineers know), there are multiple public resources that can confirm this works on both environments. See the following links as reference:
Why does Google App Engine flex build step fail while standard works for the same code?
https://github.com/GoogleCloudPlatform/nodejs-docs-samples/tree/master/appengine/typescript
I'm trying to deploy a Laravel + Vue app over an Azure App Service - Web App. It is however very unclear and I cannot find any proper solution inside Microsoft's documentation to get it into working.
'Traditional' deployment workflow
What I typically do to deploy my code (outside CI/CD):
sync Git repository
run composer install
run npm run prod (which is a shorthand for compiling webpack in my case)
Done
There is a really easy approach with a Docker container, where in my Dockerfile I just configure php-apache image with additionally installed Nodejs (w. NPM).
However I would like to find a solution to use Azure's built-in features to configure this deployment. Is it possible?
I can use Windows or Linux Web Apps. No difference for me.
I recommend that you use continuous deployment. For specific operations, you can check the official documentation.
Recommended reason:
As long as it runs successfully locally and continuously deploys through git, the project can be released, and later updates only need to submit code through git.
You can easily view the deployment log in Action in git.
Simple operation and convenient update
Steps:
First, ensure that the project is running normally locally, and create web app services on the portal. (Linux is recommended for the nodejs program, which can avoid many problems caused by dependencies)
According to the official document, in the Deployment Center, select github for release
Check the release information of Action on the official github website and wait for the release to be completed
Note:
If it is a nodejs program or other language program, if the Linux operating system is used, the Startup Command may need to be configured in the Configuration. If the program cannot be accessed normally after release, then try to set npx serve -s (nodejs program, other Language program), and then proceed to restart the webapp.
I've requested to my Team Lead that we start integrating a CI/CD pipeline into most, if not all, of our projects. Our newest project relies heavily on our own, external class library that is referenced in the solution ; it is under "Dependencies" as a project reference.
The project runs fine when I build it in my machine using Visual Studio 2019, and before we needed to integrate an external library, it would build and release fine using our Azure DevOps pipelines.
However, with the addition of an external class library, when I try to run a build through Azure DevOps, I get the following error:
The project file ....csproj was not found.
I fully understand why it can't find it - because I need to pull in the external class library and build that first! There doesn't seem to be a lot of online material (not that I could find anyway!) that describes solutions to this other than "use nuget" ; unfortunately, it is a requirement from my Team Lead that this is not a route we go down - which has lead to a long couple of days!
With this in mind, I can't find another way to do this in Azure DevOps. I have looked into some sort of PowerShell command but to no avail thus far.
Has anyone run into this issue before with external class libraries in DevOps and can give me advice on the best way to approach it?
Generally speaking in 99,99% cases keeping a direct reference to the project is not a good idea. You can end up with really unmaintainable CI/CD logic and/or with dll versions mismatches during deployments. Actually I am an Architect in the project where I tried to fix that issue by migrating all dependencies to the NuGet server.
Azure Artifacts
You mentioned, that you are using Azure DevOps as main CI/CD tool, so this is a great opportunity to introduce Azure Artifacts as internal nuget server which is a part of Azure DevOps. For the first 2 GB it is free, here you have pricing details.
Alternatives
If for some reason you cant use Azure Artifacts, I recommend some alernatives:
MyGet
ProGet
Own nuget server
More information about alternatives you can find in this article.
I want to build web ui for StrongLoop. It would let a user build and deploy process with that UI like StrongLoop Arc.
There are simple node applications(Web Services) without created with StrongLoop tools. Need to deploy these applications via web ui. Solution in my mind is some server-side processes, listed steps below:
Upload zip folder(node application) to server
Extract zip and build to tar.gz by shell command (slc build) through node.js child_process API
Deploy tar.gz file to relevant StrongLoop host by shell command(slc deploy..) through API which is mentioned on previous step.
I wonder is there any alternative way to deploy node application(without created with StrongLoop tools) to StrongLoop host via web ui using some StrongLoop API?
I have looked API could not find specific solution.
What you require is a CDP (Continuous delivery pipeline) setup, there seem to be many ways in which you can achieve this (easiest way is using Codeship or similar platforms), but if you want to know how it works it requires a bit of orchestration tools to help you. To describe the steps I'll be using the following tools:
Docker (what is docker?)
Ansible (Use Cases and How it works?)
Jenkins (What is it and Why to use it?)
"There are many other combination of tools that you can look at, but this should give you an idea"
Now that we have the tools, I'll try to describe the deployment pipeline with a very basic use-case.
Step I "Ideally" - Creating a docker image for your nodejs application.
What generally everyone suggests is that you create a docker image of your application. Then save this image on docker-hub. How this will help you is that, now your nodejs application is contained inside a docker image which makes it independent of the Host and can be deployed anywhere you want.
To create this image all you need to do is create a Dockerfile, which is described in the in the link I've shared.
Step II "Ideally" - Creating an Ansible playbook to mimic the setup steps of your application.
Ansible playbooks are basically used to automate every manual process that you would need to do in order to setup-deploy-run your application. This decreases the need to run even trivial tasks like "slc build".
Step III "Ideally" - This is where we get to the UI stuff
By using Jenkins, you are given a UI which will help you configure tasks that can be combined with Github hooks and trigger the deployment as soon as you make a commit. This is explained in more details in the link shared.
So to summarize, This is what goes on at back to some extent, in order to automate the build and deployment of your application using UI. I hope this serves as a good starting point to achieve your requirements, and also in case you want skip these steps in the start, you could always go with Codeship or similar other tools to help you with the steps that you've mentioned.