Scoped NPM for private repository converts slash "/" to "%2F" - node.js

I would like to fetch private npm from private repository using scopes.
I have set the .npmrc as such.
registry=https://registry.npmjs.org/
#myco:registry=https://nexus.myco.com/nexus/repository/
I have set the .yarnrc as such.
registry "https://registry.npmjs.org/"
"#myco:registry" "https://nexus.myco.com/nexus/repository/"
But when I do:
yarn --verbose add #myco/some-private-npm
It throws this error:
verbose 0.708 Performing "GET" request to "https://nexus.myco.com/nexus/repository/#myco%2fsome-private-npm".
verbose 0.792 Request "https://nexus.myco.com/nexus/repository/#myco%2fsome-private-npm" finished with status code 404.
When I do:
yarn --verbose add #myco:some-private-npm
It goes to this 400 url (Nexus: Invalid repository path):
verbose 0.957 Request "https://nexus.myco.com/nexus/repository/#myco:some-private-npm" finished with status code 400.
The actual npm is located at:
https://nexus.myco.com/nexus/repository/myco/some-private-npm
How do I make sure the url fetched does not have "#" and "%2f" is a "/"?
Thanks!

Per this github issue for yarn, try adding similar config to your .yarnrc:
registry "https://registry.npmjs.org/"
"#myco:registry" "https://nexus.myco.com/nexus/repository/"
Failing that, I recommend perusing the linked issue and trying the solutions provided.

I run into exact the same issue some time ago.
check at NPM if your repository team access level is read/write.
On my side the team i was in had only read access level. Switching solved the issue.

Another possible scenario is that the slash encoding isn't really the problem. You simply do not have access to that specific repository.
That would explain the 404, since if you don't have access, for security reasons, it's as if it doesn't exist, you don't get a 401.

I found this page when also trying to run npm publish of a .tgz file in Github Actions, and getting the following (replaced scope and package):
npm ERR! 404 Not Found - PUT https://registry.npmjs.org/#SCOPE%2fPACKAGE - Not found
npm ERR! 404
npm ERR! 404 '#SCOPE/PACKAGE#0.0.5' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
This package IS published so I thought the %2f was causing the issue. I was using NPM_TOKEN as an environment variable as well, as per npmjs documentation.
Solution:
- name: Setup Node.js environment
uses: actions/setup-node#v2.1.2
with:
node-version: '12'
registry-url: 'https://registry.npmjs.org'
- name: Publish to npmjs
run: |
npm install
npm run pack
npm publish <package tgz> --access public
env:
NPM_TOKEN: ${{ secrets.NPM_APIKEY }}
NODE_AUTH_TOKEN: ${{ secrets.NPM_APIKEY }}
via Github Documentation.
NOTE: I'm not sure NPM_TOKEN is still required here.

Make sure .yarnrc actually contains
registry "https://registry.npmjs.org/"
and not
registry "registry.npmjs.org"
That'll give the same error.

Related

Getting 403 Forbidden from Artifact Registry While Deploying a Cloud Function

What I Want to Do
I have published my private npm package onto Artifact Registry, and the diagram below is what I'm trying to do.
Basically I want to use the private package from both project-staging and project-production. So my package.json looks something like:
"dependencies": {
"#<scope>/private_package": "^1.0.0",
},
What I Did
Giving Permission to Cloud Build Service Account
I gave artifactregistry.reader permission to both <project-staging_id>#cloudbuild.gserviceaccount.com and <project-production_id>#cloudbuild.gserviceaccount.com under IAM menu in project-common.
.npmrc Preparation
cd path/to/project_directory
gcloud artifacts print-settings npm \
--project=project-common \
--repository=private_package \
--location=<location> \
--scope=#<scope>
vim .npmrc
# => add the output of above gcloud command
npx google-artifactregistry-auth .npmrc
deploy
gcloud config configurations activate project-staging
gcloud functions deploy <function_name> --gen2 --region=<region> --trigger-http --runtime=nodejs16 --entry-point=<entry_point>
This gives me the following error.
ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Build failed with status: FAILURE and message: npm ERR! code E403
npm ERR! 403 403 Forbidden - GET https://<URL>.tgz - Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "<resource_name>" (or it may not exist)
I don't know why I'm getting this even though
<project-staging_id>#cloudbuild.gserviceaccount.com has artifactregistry.reader permission.
Is there anything else I should do?
Normally the role roles/artifactregistry.reader is sufficient to download packages from the repo on project_common.
If project_staging and project_production have the roles/artifactregistry.reader on project_common it should work.
The roles/artifactregistry.reader role contains the following permissions :
artifactregistry.repositories.list
artifactregistry.repositories.get
artifactregistry.repositories.downloadArtifacts
artifactregistry.files.list
artifactregistry.files.get
artifactregistry.packages.list
artifactregistry.packages.listTagBindings
artifactregistry.repositories.listEffectiveTags
artifactregistry.packages.list
artifactregistry.tags.list
artifactregistry.tags.get
artifactregistry.versions.list
artifactregistry.versions.get
artifactregistry.locations.list
artifactregistry.locations.get
Please check again your configuration and the identity that performs the failed action.
This was because I had added .npmrc to .gcloudignore. Removing .npmrc from .gcloudignore made it work. Silly mistake.
I'm going to try Automate and protect your Cloud Function as a next step, but for now my problem has been solved :)

Why am i getting 503 error when 'npm access resctricted'

I have a problem with setting my currently-publicly-published npm package. What I want is to set its public status to restricted or private.
it should be npm access restricted <npm-package>, but I'll get this:
npm ERR! code E503
npm ERR! 503 Service Unavailable - POST https://registry.npmjs.com/-/package//access - Service Unavailable
Does anyone notice, what's wrong here?
Appreciate it!
Thanks to my sharp sight and AKX for pointing out, that a paid subscription is needed in order to make packages private.

NPM dependencies to another private Bitbucket repo Azure DevOps pipeline authentication fails

I'm working on a Azure DevOps build pipeline for a project. I can't make any changes to the code itself besides the azure-pipeline.yaml file. (And to be honest, I know very little about the project itself)
I'm stuck on the NPM install dependencies step. I'm currently working with the YAML pipeline, but if there's a solution in the classic mode I'll go with that.
The issue is the following:
I've created the pipeline with and I check out a private Bitbucket repository according to the documentation:
resources:
repositories:
- repository: MyBitBucketRepo1
type: bitbucket
endpoint: MyBitBucketServiceConnection
name: MyBitBucketOrgOrUser/MyBitBucketRepo
Next I set the correct version of node, and execute a npm install task
- task: Npm#1
displayName: 'NPM install'
inputs:
command: 'install'
workingDir: 'the working directory'
So far so good. But, there is a dependency to another Bitbucket repository. In the package.json there is a dependecy like this:
another-dependency: git:https://bitbucket.org/organisation/repo.git#v1.1.3
I do have access to this repository, but if I run NPM install it can't re-use the credentials from the first repository.
I've tried adding both repositories to the resources in the hope that would work. But still the same error:
error fatal: Authentication failed for 'https://bitbucket.org/organisation/repo.git/'
I've tried to set up some caching mechanism, run npm install on the 2nd repo, store the dependencies, run npm install on the first one. But that didn't work unfortunately.
Is there a way in Azure Devops pipelines -without making changes to the project set-up- to make this work?
Thanks!
Normally I have the .npmrc on the Repo so I dont have to add any other task. Something like in this guide:
https://learn.microsoft.com/en-us/azure/devops/artifacts/get-started-npm?view=azure-devops&tabs=windows
And I never do something like that, but I think that you can authenticate with the external feed adding this task:
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/package/npm-authenticate?view=azure-devops
Reading a bit more, I dont know if you can do this without adding a .npmrc on your Repo. You have to create a ServiceConnection to store your login credentials, but on that you will need the .npmrc on your Repo.
Try it and tell my if this help!!
Npm will prompt for passwords when you run npm install command for your package.json locally. Since we can't enter the password during pipeline run in CI/CD pipeline, it causes the Authentication failed error.
An alternative workaround is to add credentials directly in url, like this:
"dependencies": {
"another-dependency": "git+https://<username>:<password>#bitbucket.org/xxx/repo.git"
}
See app-password:
username: your normal Bitbucket username
password: the app password
It has disadvantage since we store the app-password directly as plain-text in package.json file, which lacks security if someone else can access your package.json file. So it depends on you whether to use this workaround.
As a workaround for Azure Devops pipeline:
You can add a File Transform task to replace the old url with new Username+Password url before your npm install steps.
1.I have a package.json in root directory with content like git:https://bitbucket.org/organisation/repo.git#v1.1.3.
2.Define a dependencies.another-dependency variable with value git+https://<username>:<password>#bitbucket.org/..., set it as secret!
3.Then add the File Transform task like this:
4.Finally you'll get a new package.json file with content below:
It won't actually affect your package.json file under version control, it just add credentials temporarily during your pipeline.

Define private registry in package.json

We have a private npm repository based on Sinopia
What should I define in package.json that some packages will be installed from Synopia rather then from global npm repository?
If I install it from command line I can run: npm install <package_name> --registry <http://<server:port>
P.S. tried to google and looked in official NPM documentation but have found nothing.
One of the method i know that is by .npmrc
You can also use .npmrc also inside the project
set configuration like this
registry = http://10.197.142.28:8081/repository/npm-internal/
init.author.name = Himanshu sharma
init.author.email = rmail#email.com
init.author.url = http://blog.example.com
# an email is required to publish npm packages
email=youremail#email.com
always-auth=true
_auth=YWRtaW46YWRtaW4xMjM=
auth can be generate by
username:password
echo -n 'admin:admin123' | openssl base64
output YWRtaW46YWRtaW4xMjM=
The whole point of sinopia is a private registry and a proxy at the same time. You can use uplinks install all your packages from one registry entry point. Sinopia is able to route to any registry if the local storage is not able to resolve the dependency. By default, he points to npmjs .
So, if you set your configuration like
# a list of other known repositories we can talk to
uplinks:
npmjs:
url: https://registry.npmjs.org/
packages:
'#*/*':
# scoped packages
access: $all
publish: $authenticated
proxy: npmjs
'**':
# allow all users (including non-authenticated users) to read and
# publish all packages
#
# you can specify usernames/groupnames (depending on your auth plugin)
# and three keywords: "$all", "$anonymous", "$authenticated"
access: $all
# allow all known users to publish packages
# (anyone can register by default, remember?)
publish: $authenticated
# if package is not available locally, proxy requests to 'npmjs' registry
proxy: npmjs
You should be able to resolve all your dependencies independently of the source of each of them
btw: sinopia has no longer maintained.

NPM Install not working with artifactory behind proxy

I have configured artifactory and npm to work behind a proxy:
npm config set registry http://myserver:8081/artifactory/api/npm/npmjs
When I call npm info bower it works fine but the tarball URL already looks suspicous: { tarball: 'http://myserver:8081/artifactory/api/npm/npmjs/registry.npmjs.org/bower/-/bower-1.3.12.tgz', shasum: '37de0edb3904baf90aee13384a1a379a05ee214c' }
npm install bower fails because of the wrong tarball URL. This is the relavant output of my npm-debug.log:
34 silly mapToRegistry registry http://myserver:8081/artifactory/api/npm/npmjs
35 silly mapToRegistry uri http://myserver:8081/artifactory/api/npm/npmjs/bower
36 verbose addRemoteTarball http://myserver:8081/artifactory/api/npm/npmjs/registry.npmjs.org/bower/-/bower-1.3.12.tgz not in flight; adding
37 verbose addRemoteTarball [ 'http://myserver:8081/artifactory/api/npm/npmjs/registry.npmjs.org/bower/-/bower-1.3.12.tgz',
37 verbose addRemoteTarball '37de0edb3904baf90aee13384a1a379a05ee214c' ]
38 info retry fetch attempt 1 at 11:39:11
39 info attempt registry request try #1 at 11:39:11
40 http fetch GET http://myserver:8081/artifactory/api/npm/npmjs/registry.npmjs.org/bower/-/bower-1.3.12.tgz
41 http fetch 404 http://myserver:8081/artifactory/api/npm/npmjs/registry.npmjs.org/bower/-/bower-1.3.12.tgz
42 error fetch failed http://myserver:8081/artifactory/api/npm/npmjs/registry.npmjs.org/bower/-/bower-1.3.12.tgz
The part registry.npmjs.org in the URL is wrong and the correct tarball URL should be this one: http://myserver:8081/artifactory/api/npm/npmjs/bower/-/bower-1.3.12.tgz. From this URL I can downlaod the tarball manually using curl.
Any ideas if I have configured something wrong or if there is a bug somewhere?
Artifactory v3.4.2, npm v2.5.1
It can happen when the remote repository that should proxy the npmjs registry is pointing on the wrong URL (e.g. http://registry.npmjs.com instead of .org).
Your Artifactory front end displays the correct string for the npm config
Does this match what your running ?
Mine looks like this
#Add this registry URL to your ~/.npmrc or use:
npm config set registry http://serverName:8081/artifactory/api/npm/npmjs
I'm just wondering if the config of the remote repo is correct in Artifactory
Another option is to try zapping the cache on the npm remote repo
Is there any info in the Artifactory system logs showing what its trying to do?
I have this issue when the package has a package-lock file with reference to npmjs.org. For example a file like this: https://github.com/mckn/gulp-nuget/blob/master/npm-shrinkwrap.json
I guess the only solution is using relative URLs in resolved field of the package-lock file.

Resources