When you deploy a Google Cloud Function, using the Node.js runtime, you can supply your code in a .zip file in Google Cloud Storage (--source=gs://...).
I make the zip file using all the compiled source, and the node_modules/ folder, so that GCF has everything it needs. I do this since I'm using private NPM modules, so npm i will not work inside the GCF build routine (that runs on cloud deploy inside GCP).
Problem:
Even if I include 100% of the needed modules in node_modules/, GCF still has deployment errors, since it tries to access the private NPM package repo.
If I remove the package.json's dependencies and devDependencies section, GCF will fail to start, saying it cannot find the modules.
So:
Why can't GCF just use bundled dependencies like other tools do (e.g. GitHub Actions)
What is the best practice for deploying GCFs using private NPM modules?
the best way is to include in your code a file called .npmrc, in your npm account can generate an access token with some permissions and that token is configured in the file that I mentioned above, in that way when the deploy is made, the command npm install can download the private packages.
Cloud Functions just released support for private dependencies and the Node.js runtime.
This allows accessing private packages without passing .nomrc credentials to Cloud Functions.
Related
How do you deploy an Azure function app with pnpm?
I'm using a rush monorepo and the rush deploy task which works great locally. The problem is that I'm unable to get ALL of my pnpm node_modules dependencies AND get my Azure function started in the cloud runtime environment
Min repro project here: https://github.com/luisnaranjo733/pnpm-rush-azure-functions
After running rush deploy, I get a nice ready to deploy project at the common/deploy path. When I try to deploy the commmon/apps/functions directory which contains the package.json for my functions project, Azure Functions mostly works except for when I import a dependency that itself has a dependency on something else.
This makes sense because of the way pnpm node_modules works, but in this case it's causing a huge headache with Azure Functions. See screenshot below for more details
The green highlighted node_modules is the one that only contains the dependencies declared by my project. As expected, only #azure/service-bus is there because that is the only dependency my functions project is using.
The orange highlighted node_modules is the pnpm global content adressable module store. This is where the dependencies of #azure/service-bus end up such a #azure/core-amqp.
When running locally via npm run start, my function is able to resolve the import to #azure/service-bus from the green node_modules and its internal dependencies from the orange node_modules. When running on the cloud runtime environment, this doesn't work :(
I can deploy the common/deploy/apps/functions project which works fine on the cloud runtime until I try to import #azure/service-bus which fails because it can't find its internal dependencies such as #azure/core-amqp.
Example cloud runtime file system when deploying the common/deploy/apps/functions folder
I see a few possible paths forward, but no clear winner
Revert back to npm, which sucks because I really wanted to use pnpm+rush.
Configure pnpm to copy and flatten my common/deploy/apps/functions/node_modules indistinguishable from a npm or yarn one (no symlinks to global content addressable module store). But this kind of negates the whole point of pnpm
Configure Azure Functions cloud runtime environment to use some kind of pnpm mode and figure out what's happening and solve it for me
Has anyone trodden this path or have any words of wisdom?
Thanks,
Luis
I am using GitHub packages to publish my private NPM packages. I need to download the published package of specific version for carrying out automation work. How can I download the package as a zip bundle using GitHub REST API or equivalent? Additionally, since it is a private package, it needs to be authenticated.
I know that equivalent functionality exists but it works for GitHub releases and assets. I could not find anything yet for GitHub packages!
After a few days of intermittent research regarding this question, here is what I have found. There seem to be two methods of retrieving and/or consuming privately published NPM packages from the GitHub Package Registry. And neither of them is an exact match to your desired method, so, here goes...
OPTION 1.
You can consume the npm package directly within your application provided you have a locally configured .nprmc file on your machine in your user directory (check first #~/.npmrc),
AND
you have created a GitHub personal access token with the following scopes.
repo: full (this is how you will be authenticated.)
workflow
write: packages
adminOrg: read
user: email
NOTE: You may not require all of these, but these are the permissions I used and had no issues. Once you generate the token, create or add it to your .npmrc file like so, replacing TOKEN with the actual token value.
//npm.pkg.github.com/:_authToken=TOKEN
Be sure you additionally add the following snippet into the project or container itself within another .npmrc file in the root directory of the codebase.
#YOUR_GITHUB_USERNAME:registry=https://npm.pkg.github.com
OPTION 2:
You can connect your package to a private GitHub repository, which will allow you to access the tar.gz artifact for each version of your package and directly download it to your local machine from the web.
github.com > Your profile > Packages > Connect repository > Select and link.
REFERENCE: How to publish packages to the GitHub Package Registry
I'm trying to deploy a microservice to GCF, which has a dependency of a private GitHub-hosted package. To gain access to the package, I added a .npmrc file to the function as described here, the file looks like this:
registry=https://npm.pkg.github.com/OWNER
//npm.pkg.github.com/:_authToken=PERSONAL-ACCESS-TOKEN
Also tried using a NPM_TOKEN env variable in the Cloud Function
NPM_TOKEN=PERSONAL-ACCESS-TOKEN
Both result in the following error:
OperationError: code=3, message=Build failed: { error: { canonicalCode: "INVALID_ARGUMENT" }}
npm ERR! 404 Not Found: #packagescope/packagename
Installing locally works fine, so does deploying on Zeit Now.
I just ran into this problem so I'm sharing the fix that works with node v8 and v10 Cloud Functions.
The following is required to use the .npmrc to install packages from a private Github packages registry:
The .npmrc needs to be located in the functions folder at the same level as the package.json file
A registry entry for the your account/org scope is required, including the url like so, assuming Anchorman uses Github: #ronburgundy:registry=https://npm.pkg.github.com/ronburgundy
A Github personal access token is required for authentication, like so //npm.pkg.github.com/:_authToken=ronburgundypersonalaccesstoken
Assuming you follow best practice and do not commit the .npmrc with secrets you will need to ensure that the file is transformed to include the personal access token prior to deploying via firebase cli, as there's no other way to inject the value at runtime.
So the original example would work if it looked like this:
#OWNER:registry=https://npm.pkg.github.com/OWNER
//npm.pkg.github.com/:_authToken=PERSONAL-ACCESS-TOKEN
While Github's docs seem to suggest that you should redirect ALL scoped and unscoped package installs to their registry it appears that Google Cloud Functions does not allow us to redirect all package installs to a private registry, only those that we configure based on scope.
I have created a private typings npm module that I am using for my firebase functions and app projects. When I went to deploy firebase functions, I get a big error for every function that basically says ERR! remote: Invalid username or password.
For what I have read, it looks like I need to create a .npmrc file and put it in the /functions directory. (https://cloud.google.com/functions/docs/writing/specifying-dependencies-nodejs#using_private_modules)
I cannot however find proper instructions on how to do this anywhere. From what I found, I have done the following:
ran npm login
ran npm token create --read-only
This then gave me a token that looks like this: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX.
I then create a file called .npmrc in my functions directory, and placed //registry.npmjs.org/:_authToken=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX in it.
Additionally, I saw that the error message was trying to use ssh to install my private repo package, I have not setup ssh and am using https instead, because of this I changed my package file to git+https://github.com/accounts-name/repo#commit-num so that it uses HTTPS.
However, I still get the same error message. What am I missing? The above is what I have strung together from lots of google searching.
It seems that you have made too many different changes while trying to make it work, so let's just go through the whole process step by step.
Check the package.json of your npm module and publish it:
Remove "private" property or set it to false because private modules in the npm are meant to be never published. That is not obvious but that is true.
Next step is to apply restricted access to the package. In order to do that, add such property in the package.json file:
"publishConfig": {
"access": "restricted"
},
Make sure that npm account you use for publishing supports private packages.
Now open the terminal in the root directory of your package, type npm login then sign in to npm. Check if you put the proper version in the package.json.
Run npm publish. The package should be published in few seconds. No worries, thanks to publishConfig property nobody can access it.
Now it is time to allow package installation in your project
Go to the directory of the project and open package.json file
Check that you have the name and version of your package in the dependencies list
Open browser, navigate to https://npmjs.com, login to your account, navigate to settings page of your account and open the tokens tab
Create new token and copy it
Now again go to the directory of your project, on same level where package.json file is situated (that is important!) and create .npmrc file there.
Put such string in the .npmrc file:
//registry.npmjs.org/:_authToken=TOKEN_HERE
You are done!
Deployment with remote CI/CD services
The easiest approach is not add .npmrc into .gitignore. In such case the file will be always in repository, so npm install will run smoothly on any machine where project was cloned
If you don't want to have token string in the repository, you can move it to the environment variable of your CI/CD service and then link .npmrc file to that variable. For example, you can put generated token into the NPM_TOKEN env variable (Just token from npmjs, not the whole string from .npmrc!)
And then change the .npmrc file in the next way:
//registry.npmjs.org/:_authToken=${NPM_TOKEN}.
So, with those steps you should be able to install your restricted packages without any issues. Good luck!
If you are trying to deploy you functions with firebase deploy from a CI and your .npmrc file looks like this.
#acmecorp:registry=https://npm.pkg.github.com/
//npm.pkg.github.com/:_authToken=${NPM_REGISTRY_TOKEN}
You will run into the problem even if you have the env var set.
Build failed: Error: Failed to replace env in config: ${NPM_REGISTRY_TOKEN}
Firebase for some reason needs access to that private repo. But the env var is not sent over to firebase.
Solution I've implemented was to replace ${NPM_REGISTRY_TOKEN} in the .npmrc file on every run of the CI pipeline.
sed -i.bak "s/\${NPM_REGISTRY_TOKEN}/${NPM_REGISTRY_TOKEN}/g" .npmrc
This breaks if you use Yarn. Took me a while to find a thread pointing to npm install in the firebase cli predeploy step. If there's no package-lock.json and you only use yarn, this will break. Remove yarn.lock and install using npm to resolve the issue.
I would like to include a javascript file as module dependency. It's generated by Kotlin multiplatform, and it's called common-js.js. This is part of my package.json:
"dependencies": {
"common-js": "file:common-js"
}
I put this file into node_modules/common-js.js and it works with firebase serve with no problems.
However when I run firebase deploy, the deploy fails with:
Build failed: exit status 254
npm ERR! addLocal Could not install /workspace/common-js
How to achieve this with Cloud Functions? I tried putting it to other directories than node_modules but it doesn't work even locally.
Cloud Functions currently doesn't support the deployment of a fully-materialized node_modules folder. What it will do, during deployment, is install all the modules in your package.json with the assumption that npm can find them in the npm registry.
If you want to upload code from your machine to Cloud Functions, you'll have to include that along with all your other source code.
If you want deploy code that can be obtained through some other public source (like a GitHub repo), you should install that using npm like this:
npm install https://github.com/your-name/your-repo.git
If your code can't be downloaded publicly, it needs to be included with your source.
As Doug mentioned here, Firebase Functions doesn't support upload of node_modules. I need to upload it with other sources, but I didn't find a way how to create a node module outside node_modules directory. I solved it by modifying the generated index.js file with a gradle task which just loads a JS file, not a node module. Here is my gradle task:
task modifyCommonJsPath(type: Copy) {
from "functions/generated"
into "functions"
include "index.js"
filter {
it.replaceAll( "require\\('common-js'\\)", "require('./generated/common-js')" )
}
}
compileKotlin2Js.finalizedBy modifyCommonJsPath