We are trying to Install node in our asp.net MVC project how ever when we checked our code in it would fail the builds in team city. this is due the well known issue of the long module path names that NPM uses.
here is the log:
[08:07:46]Checking for changes
[08:07:49]Publishing internal artifacts (5s)
[08:07:54][Publishing internal artifacts] Sending build.start.properties.gz file
[08:07:49]Clearing temporary directory: C:\TeamCity\buildagent3\temp\buildTmp
[08:07:54]Clean build enabled: removing old files from C:\TeamCity\buildagent3\work\57c6a27fa330ee2f
[08:07:54]Checkout directory: C:\TeamCity\buildagent3\work\57c6a27fa330ee2f
[08:07:54]Updating sources: agent side checkout (15s)
[08:07:54][Updating sources] Will perform clean checkout. Reason: Checkout directory is empty or doesn't exist
[08:07:54][Updating sources] Cleaning C:\TeamCity\buildagent3\work\57c6a27fa330ee2f
[08:07:54][Updating sources] VCS Root: git - tempsearch (15s)
[08:07:54][VCS Root: git - tempsearch] revision: cf23c64dd32077edeb1b96a85d1be104bd127044
[08:07:54][VCS Root: git - tempsearch] Cleaning C:\TeamCity\buildagent3\work\57c6a27fa330ee2f
[08:07:54][VCS Root: git - tempsearch] The .git directory is missing in 'C:\TeamCity\buildagent3\work\57c6a27fa330ee2f'. Running 'git init'...
[08:08:05][VCS Root: git - tempsearch] Checking out branch refs/heads/develop in git - tempsearch in C:\TeamCity\buildagent3\work\57c6a27fa330ee2f with revision cf23c64dd32077edeb1b96a85d1be104bd127044
[08:08:10]
[Updating sources] Failed to perform checkout on agent: '"C:\Program Files (x86)\Git\bin\git.exe" checkout -q -f develop' command failed.
stderr: fatal: cannot create directory at 'node_modules/grunt-contrib-jasmine/node_modules/grunt-lib-phantomjs/node_modules/phantomjs/node_modules/fs-extra/node_modules/rimraf/node_modules/glob/node_modules/minimatch/node_modules/brace-expansion': No such file or directory
[08:08:10]Publishing internal artifacts
[08:08:10][Publishing internal artifacts] Sending build.finish.properties.gz file
[08:08:10]Build failed to start. Artifacts will not be published for this build
[08:08:10]Build finished
the error:
Error while applying patch: Failed to perform checkout on agent: '"C:\Program Files (x86)\Git\bin\git.exe" checkout -q -f develop' command failed.
stderr: fatal: cannot create directory at 'node_modules/grunt-contrib-jasmine/node_modules/grunt-lib-phantomjs/node_modules/phantomjs/node_modules/fs-extra/node_modules/rimraf/node_modules/glob/node_modules/minimatch/node_modules/brace-expansion': No such file or director
are there any long term solutions to this problem?
To the respond of the comment. I have a bit more space here ;).
It depends a bit on your exact situation how I would solve this. This is how I have set things up at the company here:
Setup for developing
Some node modules have to be installed globally, so each developer would have to install those globally (grunt-cli, karma, bower for example). The other node_modules that can/have to be installed locally you would add to a package.json file like:
"devDependencies": {
"grunt": "~0.4.2",
"grunt-karma": "~0.7.3",
"karma-jasmine": "~0.2.0",
"karma-firefox-launcher": "~0.1",
"karma-chrome-launcher": "~0.1",
"karma-phantomjs-launcher": "~0.1",
"grunt-contrib-clean": "~0.4.1",
"grunt-contrib-copy": "~0.4.1",
"grunt-contrib-jshint": "~0.10.0",
"grunt-contrib-concat": "~0.3.0",
"grunt-contrib-watch": "~0.5.3",
"grunt-contrib-uglify": "~0.4.0",
"grunt-contrib-less": "~0.9.0"
}
More information about package.json you can find here http://browsenpm.org/package.json. Note that bower has to be installed globally and locally (if you are using it).
This means that every develop has to do a global install of the node_modules you use once. You might want to choose a specific version of each node_module so that everyone is using the same version. You can install global modules like this:
~ npm install grunt-cli karma bower -g
You add the package.json file with the local modules to the base structure of your project, and you add this to the git repository. Each developer that will do a checkout will receive this file. They will have to do an install of the local modules so that everyone is using the same tools:
~ npm install
This means that the local modules don't have to be part of your git repo, which I find good practice since it isn't source code. But it also solves your problem of git not supporting the long node_modules paths.
If you have a build process with Node setup that doesn't change anymore, the npm_modules installed both globally and locally won't have to change. If you change the tools, each develop will have to redo the npm install. But for me this hasn't happened yet.
Build server
You will also have to install node and these global modules on the build server so that you have access to them during the build. What I have done is create a .bat file which gets executed during the build which includes a 'npm install' so that it will resolve the node_modules during the build as well. This way you also have access to the latest tools during a build.
Multiple projects
If you have multiple projects which use the same local node_modules for the build, you should take a look at grunt-collections. You can create one central location for the node_modules this way, so the developers only have to install them once for all the projects There is a really good post about this here:
Centralise node_modules in project with subproject
Hope this clears things ups
I had this error too but in my case the cause was using an outdated version of npm, v1.4.28.
Updating to npm v3 followed by rm -rf node_modules ; npm -i worked for me. npm issue 2697 has details of the 'maximally flat" folder structure included in npm v3 (released 2015-06-25).
Related
I am using "firebase-functions": "^3.6.0", "firebase-tools": "^8.0.0" to deploy my Angular: 10.0.14 app. All was working fine until I decided to move some components into a library created using the command ng generate library and made the package a submodule in my project under ./projects/my-lib
I build the lib using ng build my-lib
Then in my package.json installed the lib from file
"my-lib": "file:dist/my-lib"
I installed the package locally and it works fine when I test it locally but deploying using ng deploy raised the error
npm ERR! Could not install from "dist/mce-lib" as it does not contain a package.json file.
I have tried a couple of solutions I found online, but no luck yet
Try this
First, remove node_modules and package-lock.json file
rm -rf node_modules package-lock.json
Then, clean cache
npm cache clean --force
And last re-install modules
npm i
You need to create the library by using
ng generate library <your lib name>
This will setup your lib with the necessary files so that you can add it as a dependency.
Otherwise, if you want to do this manually it'll require more work.
UPDATE:
To be able to use the lib above locally without publishing you will need to do the following
From withing your app directory for which you generated the lib, go into ./projects/ as it was generated by the lib command above. Then run the following build command
ng build <your lib name> --prod
The prod tag is optional but it helps to run your code in that flag so that it is the closest to what it will look like once you publish
Then go back to your app root folder and install the application by referencing the dist folder. The build command above should have added your lib as its own folder withing dist
npm install ./dist/<your lib name>
NOTE: All of this is for dev testing purposes. Ideally, you will want to publish your lib to npmjs or github packages first and then install it properly in your app before you deploy your app to production
Another way of locally: though the use of npm link
cd ./dist/<your lib name>
npm link
Then going back to root folder of app, you should be able to run the installation as such
npm link <your lib name>
This creates a local link to your library in your app for testing purposes that doesn't modify your package.json file as installing the directory directly does. Once you are done, always make sure to run unlink
npm unlink <your lib name>
And withing your ./dist/
npm unlink
So after hours of trying to figure out how to solve this, I came across these resources that helped me to resolve the issue:
https://firebase.google.com/docs/functions/handle-dependencies#including_local_nodejs_modules_as_part_of_your_deployment_package
Note: The Firebase CLI ignores the local node_modules folder when deploying your function.
Firebase Functions local "file:" dependencies
So even if the app was working locally, the node_modules isn't deployed
Since I had installed python packages from git before, I thought why not make use of this? I am using Gitlab and this answer really helped me https://stackoverflow.com/a/50314604/3247914
I got it working by installing my package from git
On my local machine, npm run build works just fine. On my Docker image launched via Jenkins, I get issues like
Cannot find module: 'file-saver'. Make sure this package is installed.
You can install this package by running: npm install file-saver.
and
FAIL src/core/App.test.js
● Test suite failed to run
Cannot find module 'surface-nets' from 'vtext.js'
On my local machine, I have cleared the npm cache (npm cache clean -f), removed node_modules/, and reinstalled (npm i). I have even used npm update and the npm-check-updates package to update everything. I installed all peer dependencies. My local copy should be as wiped-clean as the copy within Docker is. In the Jenkinsfile, I put npm list and it shows surface-nets and file-saver and all my other packages. I also put ls node_modules/ and I can see the package folders are there. I have reduced my Dockerfile down to just 1 line: FROM node:current.
Why is it saying "Cannot find module" when the modules are installed?
The root cause of this issue is still unknown, but I did find out that the repository was corrupted. I ended up wiping the whole remote repo and git init-ing a new one, then pushing to that. I assume that corruption contributed to the issue at hand here.
I created an new folder my-app by running npx create-react-app my-app in node.js command. Once the folder was created, I wanted to upload the whole thing to my repo in my github account, so I opened git bash and run the commands to push all the files and folders to the new repo I created. After updating the repo, I found that all the files have been copies expect for the node_modules folder. I actually tried to run the npx create-react-app command in git bash command window, but failed. So I have to do it in node.js first then run the commands in git bash. Additionally, the repo was created without read me file. Did I do anything wrong here? why the folder was left behind. Can anyone help.
This is because create-react-app is bootstraped and comes with file .gitignore.
In which
node_modules are ignore
Why are they ignored because they are library which you can install any time by using npm install based on package.json file.
In package.json, there is section of dependencies, which are your node_modules. and npm install will make them available.
"dependencies": {
"bootstrap": "^4.1.3",
"fullcalendar-reactwrapper": "^1.0.7",
"moment": "^2.22.2",
"react": "^16.5.2",
"react-date-picker": "^7.0.0",
}
Git - Ignore node_modules folder everywhere
Nothing wrong with that, create-react-app has a lot of features and one of them is the automatic creation of a .gitignore file. If you check in your working directory, at the root there is that file.
Basically what this file does is telling to git what to track and what to ignore, it is common to totally ignore the node_modules folder because when you clone your repo, you'll just need your package.json to npm install your dependencies, so there is absolutely no need to upload a lot of data (node_modules could weight a lot).
gitignore docs
the /node_modules folder is in the .gitignore file created by create-react-app.
If you remove that entry from the .gitignore folder youdshould be able to commit the node_modules.
The create-react-app comes with a file called .gitignore. This file ensures unnecessary files are not committed to git. This is the reason your node_modules aren't being committed and pushed to github.
The main reason node_modules is included in the .gitignore is it's not necessary to push the file to the github.
you can get the node_modules folder whenever from package.json file of your project.
The second reason is the node_modules folder is large. Pushing it to github is a negative point.
If you want to get node_modules folder from a package.json file. You
just need to run the following commands from root of the project
directory when this package.json is present
if using yarn -
$ yarn install
if using npm -
$ npm install
I am building a simple node app and using Bootstrap to style my frontend. I want to deploy the app to Heroku by loading it from the GitHub repository. I don't want to put any compiled CSS/JS files in the repository, which means that they need to be compiled on Heroku after the source is pulled from GitHub.
The problem: Bootstrap's default package.json has its build dependencies in devDependencies, so the dependencies will not be installed on Heroku, which runs npm install in production mode.
Specifically, the problem comes up in my postinstall script, which consists of cd node_modules/bootstrap && npm install && ../.bin/grunt dist. My own npm install command does not end up installing anything because the overarching npm install --production is ignoring Bootstrap's devDependencies.
What is the best workaround to get Bootstrap's dependencies to be installed locally? I'm open to changing my workflow as long as (1) I don't have to put compiled files in my source repository and (2) I can still run the main npm install in production mode.
EDIT: Unless someone can think of a better solution, I think my options at this point are as follows:
Build a custom Github fork of Bootstrap that has its dependencies in dependencies instead of devDependencies
Build a custom Heroku buildpack that somehow leaves Bootstrap's dependencies intact
Abandon principle and compile Bootstrap locally, and put the compiled files in my source
Just use Heroku in development mode (obviously not a good idea)
I think I'm going to try for option 1, and go for option 3 if that fails. I'd still be interested in hearing any other ideas that people come up with.
I would look into using postinstall. The official documentation is on Heroku here.
Since you're doing some build steps at deploy time, you need some devDependencies (bootstrap sources in your case), but heroku will only install in production mode, like you said.
When connected with the heroku toolbelt client, run :
heroku config:set NPM_CONFIG_PRODUCTION=false
You'll have your devDependencies and you'll be able to build.
You only have to do that once (no need to do that each time you deploy).
PS: I don't know about your project, but maybe you need to put the NODE_ENV to production, here is the command:
heroku config:set NODE_ENV=production
You don't need yo put back the NPM_CONFIG_PRODUCTION flag back to true, this is the purpose of making the build on heroku.
But if you prefer, here is the workflow you could use (you said you were opened to change workflow):
ignore dist folder from your source repo (it should already be ignored)
make dist folder a git repo & add the git remote of heroku in repo of the dist folder
cd dist
git init
git remote add origin git#heroku.com:project-name.git
git pull origin master
make sure your build routine doesn't remove this /dist/.git folder - you can use the following glob (if you're using rimraf or something like it with grunt/gulp or whatever) : ['/dist/**/*','/dist/!.git/**/*']
This way, your workflow will be:
build (with your tool)
cd dist
git add .
git commit -m "new version bundled"
git push origin master there, you are in dist folder where origin is the remote for your heroku
You might do the same kind of things with github pages ...
I setup a development environment with Windows 8 and Ubuntu as a virtual machine. For that I use VirtualBox.
I also manage to create a shared folder in VirtualBox.
In this shared folder I try to start a project with ember-generator of Yeoman.
yo ember --skip-install --karma
npm install --no-bin-links
For installing modules NPM I use the option "--no-bin-links" not to create symbolic links. Unfortunately, I still have errors creations symbolic links ... Is what I use although this option ? There he has a bug ?
The NPM docs about parameter "--no-bin-links" say:
will prevent npm from creating symlinks for any binaries the package
might contain.
Which will just cause NPM to not create links in the node_modules/.bin folder. I also searched for a way to prevent NPM from creating symlinks when using npm install ../myPackage, but can't find any solution...
Update: The npm support team said this will reproduce the old behaviour (no symbolic links):
npm install $(npm pack <folder> | tail -1)
Works for me in git-bash on Windows 10.
This Stack Overflow page comes up in Google search results when trying to solve the issue of installing local modules (ie. npm install ../myPackage) and not wanting symbolic links. So I'm adding this answer below to help others who end up here.
Solution #1 - For development environment.
Using the solution proposed by the NPM support team as mentioned in the other answer works...
# Reproduces the old behavior of hard copies and not symlinks
npm install $(npm pack <folder> | tail -1)
This is fine in the development environment for manual installs.
Solution #2 - For build environment.
However, in our case, the development environment doesn't quite matter as much though because when committing our changes to Git, the ./node_modules/ folder is ignored anyway.
The files ./package.json and ./package-lock.json is what is important and is carried into our build environment.
In our build environment (part of our automated CI/CD pipeline), the automation just runs the npm install command and builds from the dependencies listed in the package.json file.
So, here is where the problem affects us. The locally referenced files in the dependencies list of the package.json causes symlinks to appear. Now we are back to the old problem. These symlinks then get carried into the build's output which move onto the Stage and Production environments.
What we did instead is use rsync in archive mode with the --copy-links option that turns symbolic links into copies of the original.
Here is what the command looks like in the automated build:
# Install dependencies based on ./package.json
npm install
# Make a copy that changes symlinks to hard copies
rsync --archive --verbose --copy-links ./node_modules/ ./node_modules_cp/
# Remove and replace
rm -r ./node_modules/
mv ./node_modules_cp/ ./node_modules/
I have a similar environment. Apparently the Virtualbox (vagrant) synchronisation has problems when renaming or moving files, which happens when updating modules.
If you do a file listing (ls -alhp) on the command line and see ??? for the file permissions, then it is time to reboot your virtualbox. This will set the permissions to valid values. Then use the --no-bin-links option when installing a module.