If we deploy a node.js 12 project to App Engine (standard environment) and it has a dependency in package.json that looks like this:
"some-lib": "git+https://github.com/a/b.git",
Then this will lead to a deployment error when we execute gcloud app deploy. This is from the App Engine logs:
Step #6 - "builder": npm ERR! Error while executing:
Step #6 - "builder": npm ERR! /usr/bin/git ls-remote -h -t ssh://git#github.com/a/b.git
Step #6 - "builder": npm ERR!
Step #6 - "builder": npm ERR! Host key verification failed.
Step #6 - "builder": npm ERR! fatal: Could not read from remote repository.
So even though the dependency is provided as a HTTPS URL, it will try to fetch it with SSH and this doesn't seem to work on App Engine.
It works if I go to package-lock.json and manually change the version string to start with https, but that's not really a viable solution, because it will be overwritten again.
Any ideas how it can be accomplished to provide a dependency with a GitHub URL (SSH or HTTPS) to App Engine?
Check first your git configuration in the deployment environment (Google App Engine, in your Google Cloud Console), assuming Git 2.26 or more:
git config --global --show-origin --show-scope-l
git config --system --show-origin --show-scope -l
# if you are in a local repository:
git config --local --show-origin --show-scope -l
Look for any configuration like url."ssh://git#github.com".insteadOf "git+https://github.com"
Such a rule would rewrite the URL, explaining why your initial dependency shows up as an SSH URL.
As we can see in this issue the problem relies in NPM that caused to convert any git+https reference into git+ssh.
From what I understood, the issue has been patched but is not available on NPM 6, that is the version used in the nodejs12 and nodejs14 enviroment of GAE.
What I suggest is to use nodejs16 as runtime of your App Engine application
Related
I was trying to deploy a react website to git hub but i face some issues, this is my first time dealing with Git, so expect dumb mistakes from me, I will elaborate exactly what I have done :
1 - created a repository on GitHub.
2 - opened the directory of my
website on my local pc from the terminal
3 - git init
4 - git add
--all
5 - git commit -m "whatever"
6 - git remote add origin https://github.com/MahmoudFarid296/star-game.git
7 - git push
--set-upstream origin master
8 - npm install gh-pages --save-dev
9 - npm run deploy -> faced an authentication issue which I looked for
and found the solution to be : git remote set-url origin
https://MahmoudFarid296:{my
token}#github.com/MahmoudFarid296/star-game.git
10 - git remote
set-url origin https://MahmoudFarid296:{my
token}#github.com/MahmoudFarid296/star-game.git
11 - npm run deploy
-> fatal: could not create work tree dir 'C:\Users\hp\Star-match-game\node_modules.cache\gh-pages\https!MahmoudFarid296!{my
token}#github.com!MahmoudFarid296!star-game.': Invalid argument
note: before pushing the files to the repository I did already set the homepage attribute in the package file to "homepage": "https://MahmoudFarid296.github.io/star-game",
and I added "predeploy": "npm run build",
"deploy": "gh-pages -d build" to the scripts.
now the 2 issues are :
1 - the problem in step 11 I didn't find solution for.
2 - I tried the same directory with same steps but without the readme file a few days ago and it did run the deploy and it gave me the message of published but when I opened the website on GitHub it just showed a blank page ,I tried adding /public to the URL as my index.html file was in public directory as you can see in the repository but still gave me a blank page.
I have a react based application that pulls in code from another private repository.
In my package.json this appears as
"mylib": "github:orgname/mylib"
And it all works locally.
When I deploy to Azure App Service I get the following error
npm ERR! Error while executing:
7npm ERR! C:\Program Files\Git\bin\git.EXE ls-remote -h -t ssh://git#github.com/orgname/mylib.git
8npm ERR!
9npm ERR! Host key verification failed.
10npm ERR! fatal: Could not read from remote repository.
11npm ERR!
12npm ERR! Please make sure you have the correct access rights
13npm ERR! and the repository exists.
14npm ERR!
15npm ERR! exited with error code: 128
I grabbed the public key from
https://[appname].scm.azurewebsites.net/api/sshkey?ensurePublicKey=1
and tried adding it to the account, organisation and repo on github, but I’m still getting the same error.
Is there something obvious I’m missing here?
To deploy an application to Azure app service using Private GitHub repository.
Below is the one of workaround you can follow:
I have created an demo application in VS Studio (.net) and push the project to my GitHub with private repository.
created an app service in Azure Portal . And Go to Deployment center>Select your organization as GitHub, repository(your private repository) and save it.
After sometime your application will reflect on Azure portal
For more information please refer this below links:
SO THREAD How Can I deploy Apps in app service with private GitHub repository
BLOG: Deploy Create React App to Azure App Services
MS Q&A: Cannot deploy NodeJS app from private GitHub repo.
I'm working on a Azure DevOps build pipeline for a project. I can't make any changes to the code itself besides the azure-pipeline.yaml file. (And to be honest, I know very little about the project itself)
I'm stuck on the NPM install dependencies step. I'm currently working with the YAML pipeline, but if there's a solution in the classic mode I'll go with that.
The issue is the following:
I've created the pipeline with and I check out a private Bitbucket repository according to the documentation:
resources:
repositories:
- repository: MyBitBucketRepo1
type: bitbucket
endpoint: MyBitBucketServiceConnection
name: MyBitBucketOrgOrUser/MyBitBucketRepo
Next I set the correct version of node, and execute a npm install task
- task: Npm#1
displayName: 'NPM install'
inputs:
command: 'install'
workingDir: 'the working directory'
So far so good. But, there is a dependency to another Bitbucket repository. In the package.json there is a dependecy like this:
another-dependency: git:https://bitbucket.org/organisation/repo.git#v1.1.3
I do have access to this repository, but if I run NPM install it can't re-use the credentials from the first repository.
I've tried adding both repositories to the resources in the hope that would work. But still the same error:
error fatal: Authentication failed for 'https://bitbucket.org/organisation/repo.git/'
I've tried to set up some caching mechanism, run npm install on the 2nd repo, store the dependencies, run npm install on the first one. But that didn't work unfortunately.
Is there a way in Azure Devops pipelines -without making changes to the project set-up- to make this work?
Thanks!
Normally I have the .npmrc on the Repo so I dont have to add any other task. Something like in this guide:
https://learn.microsoft.com/en-us/azure/devops/artifacts/get-started-npm?view=azure-devops&tabs=windows
And I never do something like that, but I think that you can authenticate with the external feed adding this task:
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/package/npm-authenticate?view=azure-devops
Reading a bit more, I dont know if you can do this without adding a .npmrc on your Repo. You have to create a ServiceConnection to store your login credentials, but on that you will need the .npmrc on your Repo.
Try it and tell my if this help!!
Npm will prompt for passwords when you run npm install command for your package.json locally. Since we can't enter the password during pipeline run in CI/CD pipeline, it causes the Authentication failed error.
An alternative workaround is to add credentials directly in url, like this:
"dependencies": {
"another-dependency": "git+https://<username>:<password>#bitbucket.org/xxx/repo.git"
}
See app-password:
username: your normal Bitbucket username
password: the app password
It has disadvantage since we store the app-password directly as plain-text in package.json file, which lacks security if someone else can access your package.json file. So it depends on you whether to use this workaround.
As a workaround for Azure Devops pipeline:
You can add a File Transform task to replace the old url with new Username+Password url before your npm install steps.
1.I have a package.json in root directory with content like git:https://bitbucket.org/organisation/repo.git#v1.1.3.
2.Define a dependencies.another-dependency variable with value git+https://<username>:<password>#bitbucket.org/..., set it as secret!
3.Then add the File Transform task like this:
4.Finally you'll get a new package.json file with content below:
It won't actually affect your package.json file under version control, it just add credentials temporarily during your pipeline.
I would like to fetch private npm from private repository using scopes.
I have set the .npmrc as such.
registry=https://registry.npmjs.org/
#myco:registry=https://nexus.myco.com/nexus/repository/
I have set the .yarnrc as such.
registry "https://registry.npmjs.org/"
"#myco:registry" "https://nexus.myco.com/nexus/repository/"
But when I do:
yarn --verbose add #myco/some-private-npm
It throws this error:
verbose 0.708 Performing "GET" request to "https://nexus.myco.com/nexus/repository/#myco%2fsome-private-npm".
verbose 0.792 Request "https://nexus.myco.com/nexus/repository/#myco%2fsome-private-npm" finished with status code 404.
When I do:
yarn --verbose add #myco:some-private-npm
It goes to this 400 url (Nexus: Invalid repository path):
verbose 0.957 Request "https://nexus.myco.com/nexus/repository/#myco:some-private-npm" finished with status code 400.
The actual npm is located at:
https://nexus.myco.com/nexus/repository/myco/some-private-npm
How do I make sure the url fetched does not have "#" and "%2f" is a "/"?
Thanks!
Per this github issue for yarn, try adding similar config to your .yarnrc:
registry "https://registry.npmjs.org/"
"#myco:registry" "https://nexus.myco.com/nexus/repository/"
Failing that, I recommend perusing the linked issue and trying the solutions provided.
I run into exact the same issue some time ago.
check at NPM if your repository team access level is read/write.
On my side the team i was in had only read access level. Switching solved the issue.
Another possible scenario is that the slash encoding isn't really the problem. You simply do not have access to that specific repository.
That would explain the 404, since if you don't have access, for security reasons, it's as if it doesn't exist, you don't get a 401.
I found this page when also trying to run npm publish of a .tgz file in Github Actions, and getting the following (replaced scope and package):
npm ERR! 404 Not Found - PUT https://registry.npmjs.org/#SCOPE%2fPACKAGE - Not found
npm ERR! 404
npm ERR! 404 '#SCOPE/PACKAGE#0.0.5' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
This package IS published so I thought the %2f was causing the issue. I was using NPM_TOKEN as an environment variable as well, as per npmjs documentation.
Solution:
- name: Setup Node.js environment
uses: actions/setup-node#v2.1.2
with:
node-version: '12'
registry-url: 'https://registry.npmjs.org'
- name: Publish to npmjs
run: |
npm install
npm run pack
npm publish <package tgz> --access public
env:
NPM_TOKEN: ${{ secrets.NPM_APIKEY }}
NODE_AUTH_TOKEN: ${{ secrets.NPM_APIKEY }}
via Github Documentation.
NOTE: I'm not sure NPM_TOKEN is still required here.
Make sure .yarnrc actually contains
registry "https://registry.npmjs.org/"
and not
registry "registry.npmjs.org"
That'll give the same error.
TL;DR: Working app, cloned it, clone doesn't start correctly from push hook (but works fine manually if I ssh in.) PATH has correct Node version added to it, but somewhere right in the last step, the incorrect Node version is prepended to the PATH again.
Path is correct here:
remote: PATH = /var/lib/openshift/.../app-root/data//node-v4.x.x-linux-x64/bin:/var/lib/openshift/.../app-root/runtime/repo/node_modules/.bin:/var/lib/openshift/...//.node_modules/.bin:/opt/rh/nodejs010/root/usr/bin:/bin:/usr/bin:/usr/sbin
Then incorrect immediately after, somewhere in here:
remote: Starting NodeJS cartridge
remote: Tue Aug 22 2017 15:39:10 GMT-0400 (EDT): Starting application 'staging' ...
So what scripts and hooks are represented in or before these last two lines? PATH isn't just adding lines to itself...
I have a working OpenShift v2 app running a NodeJS version new enough to support fat arrow notation.
It appears that it was set up per Custom node.js version on Openshift because the scripts from that repo (for using a marker file) are present in .openshift.
I set up a second one using rhc create --from-app based on the working one, reset the repo, then re-deployed onto it. The second one worked great, except for the final step of starting node:
remote: npm info ok
remote: NOTE: The .openshift/action_hooks/build hook is not executable, to make it executable:
remote: On Windows run: git update-index --chmod=+x .openshift/action_hooks/build
remote: On Linux/OSX run: chmod +x .openshift/action_hooks/build
remote: Preparing build for deployment
remote: Deployment id is cedf7f51
remote: Activating deployment
remote: NOTE: The .openshift/action_hooks/deploy hook is not executable, to make it executable:
remote: On Windows run: git update-index --chmod=+x .openshift/action_hooks/deploy
remote: On Linux/OSX run: chmod +x .openshift/action_hooks/deploy
remote:
remote: - pre_start_nodejs: Adding Node.js version 4.x.x binaries to path
remote: - PATH set to include custom node version (4.x.x) from
remote: /var/lib/openshift/.../app-root/data//node-v4.x.x-linux-x64/bin
remote: PATH = /var/lib/openshift/.../app-root/data//node-v4.x.x-linux-x64/bin:/var/lib/openshift/.../app-root/runtime/repo/node_modules/.bin:/var/lib/openshift/...//.node_modules/.bin:/opt/rh/nodejs010/root/usr/bin:/bin:/usr/bin:/usr/sbin
remote: Starting NodeJS cartridge
remote: Tue Aug 22 2017 15:39:10 GMT-0400 (EDT): Starting application 'staging' ...
remote: Waiting for application port (8080) become available ...
(Everything up to this point is exactly as it is on the working app, except for names.)
remote: Application 'staging' failed to start (port 8080 not available)
remote: -------------------------
remote: Git Post-Receive Result: failure
remote: Activation status: failure
remote: Activation failed for the following gears:
remote: ... (Error activating gear: CLIENT_ERROR: Failed to execute: 'control start' for /var/lib/openshift/.../nodejs
remote: #<IO:0x00000001cd42d0>
remote: #<IO:0x00000001cd4258>
remote: )
remote: Deployment completed with status: failure
remote: postreceive failed
rhc env and rhc app show show the settings to be identical in all the ways that they can be.
I know from other questions that the "port 8080" part above is a red herring. Plus, if I rhc ssh in and manually node www.js, it uses that port just fine and I can reach the app via browser.
So I investigated using rhc tail. I can see it failing to start, repeatedly, due to fat arrow notation:
pg.on('error', (err) => {
^
SyntaxError: Unexpected token >
at Module._compile (module.js:439:25)
at Object.Module._extensions..js (module.js:474:10)
...
DEBUG: Program node ./www.js exited with code 8
DEBUG: Starting child process with 'node ./www.js'
Yet, if I rhc ssh into this same server and run node --version, I get the newer version (the same version as the other server, which both pull it from the marker file in the .openshift directory I deployed.)
I'm guessing somehow the final step in the push hook is using Node 0.10, since that's what the cartridge is named on both apps. However, if I check the path being added to PATH, a newer Node version does in fact live there.
However, this is where things get interesting. The PATH reported above (where 4.x.x is prepended) is no longer the path by the time Node is launched. I changed www.js to just spit out process.env.PATH and compared the two. The latter has these two paths added to the beginning!
/opt/rh/nodejs010/root/usr/bin
/opt/rh/v8314/root/usr/bin
What is doing that and how can I stop it? What even has a chance during these lines of output?
remote: PATH = /var/lib/openshift/.../app-root/data//node-v4.x.x-linux-x64/bin:/var/lib/openshift/.../app-root/runtime/repo/node_modules/.bin:/var/lib/openshift/...//.node_modules/.bin:/opt/rh/nodejs010/root/usr/bin:/bin:/usr/bin:/usr/sbin
remote: Starting NodeJS cartridge
remote: Tue Aug 22 2017 15:39:10 GMT-0400 (EDT): Starting application 'staging' ...
(And why just on my second app, when I used --from-app to create it and all other settings seem to be equal between the two?)
Among one of the 129 forks of the repo used originally in the app to customize it's NodeJS version, there was a comment on an issue that solved it:
Add a line to .openshift\lib\utils:
# Add the node binary path to the PATH.
export OPENSHIFT_NODEJS_VERSION=0.6 ### this is the new line
export PATH="$node_bin_path:${PATH}"
"It's important, like you mentioned, that it's set to 0.6, EVEN IF YOU'RE NOT RUNNING 0.6."
It's also important that it's set to 0.6, even if the original Node version you're getting reverted back to isn't 0.6, either. (I tried with 0.10, with no luck. 0.6 is the magic answer, here.)
Node.js (Latest) Cartridge
#hub.openshift : Node.js Latest
By default, the Node.js version is determined by querying
semver.io/node/stable.
A different URL can be specified either via NODE_VERSION_URL
environment variable or by setting .openshift/NODE_VERSION_URL marker
in your application repository.
For instance, you'd get the latest
0.10.x by putting this in NODE_VERSION_URL variable or .openshift/NODE_VERSION_URL marker:
https://semver.io/node/resolve/0.10
If you're using a non-default Node.js version and you're planning to
scale the application across multiple gears, you must use the
environment variable (learn here why).
Other Possible Methods
pre_build action hook workaround :
#!/bin/bash
#NODE_VERSION_URL
TARGET_NODE_VERSION=$(node -e "var p = require('$OPENSHIFT_REPO_DIR/package.json')||{engines:{}}; console.log(encodeURI(p.engines.node||''));")
echo "https://semver.io/node/resolve/$TARGET_NODE_VERSION" > ${HOME}.env/user_vars/NODE_VERSION_URL
#NPM_VERSION_URL
TARGET_VERSION=$(node -e "var p = require('$OPENSHIFT_REPO_DIR/package.json')||{engines:{}}; console.log(encodeURI(p.engines.npm||''));")
echo "https://semver.io/npm/resolve/$TARGET_VERSION" > ${HOME}.env/user_vars/NPM_VERSION_URL
package.json engines workaround :
Adding the .openshift folder from this repo will let you run any version of nodejs that you need using the engines section of your package.json file (see the README for usage):
"engines": {
"node": ">= 0.12.0"
},