I'm using a node package to run a web server (among other benefits) for my project. The catch is, my project is only loaded on the server if it's within a directory of the node package. In other words, my directory structure looks like this:
<npm_pkg>/
<npm_pkg_src>/
clients/
<my_project_name>/
<my_project_src>
I would like to be able to use standard deployment processes for my project (e.g. gcloud app deploy, Travis continuous deployment, etc.), but I need to run my project from within a subdirectory of the larger package. Is there an easy way to force a git clone <pkg> during a build step and deploy my project in the target subdirectory?
I'm pretty new to CI/CD, but I tried to search around for similar examples and couldn't find any. Note: the parent project is not owned by me and thus I can't just use submodules without forking it (and I have no intention to alter it in any way). I also strictly just want to be able to trigger deploys based on my actual project's repository, if possible, whereas submodules would involve maintaining two and committing features twice (from what I understand).
Any help is much appreciated.
Edit: I forgot to mention that as part of this configuration I also need to run my server script from the root of the parent package. IOW, my package.json's start script will look like "start": "cd ../.. && npm start". Just in case it's relevant.
This might be what you’re looking for: CI/CD with App Engine
Clone from the repo and deploy from the subdirectory it is located, and Cloud Source Repositories can automate the whole process for you
I would also suggest you keep services separate, this will make things clearer for you and others that will/might be working on the project with you
Related
We've got an application (Angular/node in GitLab) that has a corresponding freestyle build in Jenkins. In the package.json, we need to reference a separately-built library of our own writing, also in GitLab. Like this:
"dependencies": {
"our-library": "git+ssh://git#ourcompany.com:2222/libraries/our-library.git#master"
}
When we build the application in Jenkins, it pulls the source for our-library, which does not contain the generated lib files. The issue is that ideally we would not like to push the lib files generated by a local our-library build just so that the Jenkins build will be able to find what it needs for the application build. It's almost like we need to be able to reference the /lib folder in the Jenkins our-library workspace instead of referencing the source from GitLab, or something.
The fact that I need to ask this question gives me the feeling that we're going about this all wrong. Any insight on how to build an app in Jenkins while referencing other node projects that we build as well?
Thanks a lot for any feedback!
That looks like a source dependency, which means your Jenkins build needs to also build said dependencies (from its sources)
The other approach is binary dependency, where your package.json references a local dependency that you have copied (from a binary referential like Nexus, or from any other source you want, like a shared folder where you have put the generated libs).
See for instance "Local dependency in package.json".
I am from the NodeJS/JavaScript world where I have npm and dependencies written down in the package.json. When I deploy it, I know that I just need to run npm install and all the dependencies consumed by the app will be installed.
How is it supposed to be done for a Go project? Suppose I have a source code of the app which I deploy remotely by, say, running git pull. Now, how do I make sure the dependencies are present? What I see is I need to install a package manager manually then install dependencies using it?
What's a standard way of deploying a Go app on a server?
First of all, you're indeed thinking like a JS developer. Go is compiled, and thus the proper way to deploy a Go app is not to use the source code at all - you build it on your build server, and deploy a binary. So on the server level you simply don't care anymore, the only place where you need dependencies is the build system.
Now, the standard way to do this in go is to vendor dependencies with your source, that is make sure they are included in the git repo. Another approach is the express them in a manifest file and fetch them with an external tool. These are both more reliable than the naive approach, of simply using go get in build time, fetching the current version of your dependencies (this requires no manifest file).
There are many tools for vendoring management, to name two: Godep and gb
I have a repo which consists of several "micro-services" which I upload to AWS's Lambda. In addition I have a few shared libraries that I'd like to package up when sending to AWS.
Therefore my directory structure looks like:
/micro-service-1
/dist
package.json
index.js
/micro-service-2
/dist
package.json
index.js
/shared-component-1
/dist
package.json
component-name-1.js
/shared-component-2
/dist
package.json
component-name-2.js
The basic deployment leverages the handy node-lambda npm module but when I reference a local shared component with a statement like:
var sharedService = require('../../shared-component-1/dist/index');
This works just fine with the node-lambda run command but node-lambda deploy drops this local dependency. Probably makes sense because I'm going below the "root" directory in my dependency so I thought maybe I'd leverage gulp to make this work but I'm pretty darn new to it so I may be doing something dumb. My strategy was to:
Have gulp deploy depend on a local-deps task
the local-deps task would:
npm build --production to a directory
then pipe this directory over to the micro-service under the /local directory
clean up the install in the shared
I would then refer to all shared components like so:
var sharedService = require('local/component-name-1');
Hopefully this makes what I'm trying to achieve. Does this strategy make sense? Is there a simpler way I should be considering? Does anyone have any examples of anything like this in "gulp speak"?
I have an answer to this! :D
TL;DR - Use npm link to link create a symbolic link between your common component and the dependent component.
So, I have a a project with only two modules:
- main-module
- referenced-module
Each of these is a node module. If I cd into referenced-module and run npm link, then cd into main-module and npm link referenced-module, npm will 'install' my referenced-module into my main-module and store it in my node_modules folder. NOTE: When running the second npm link, the name of the project is the one you find in your package.json, not the name of the directory (see npm link documentation, previously linked).
Now, in my main-module all I need to do is var test = require('referenced-module') and I can use that to my hearts content. Be sure to module.exports your code from your referenced-module!
Now, when you zip up main-module to deploy it to AWS Lambda, the links are resolved and the real modules are put in their place! I've tested this and it works, though not with node-lambda yet, though I don't see why this should be a problem (unless it does something different with the package restores).
What's nice about this approach as well is that any changes I make to my referenced-module are automatically picked up by my main-module during development, so I don't have to run any gulp tasks or anything to sync them.
I find this is quite a nice, clean solution and I was able to get it working within a few minutes. If anything I've described above doesn't make any sense (as I've only just discovered this solution myself!), please leave a comment and I'll try and clarify for you.
UPDATE FEB 2016
Depending on your requirements and how large your application is, there may be an interesting alternative that solves this problem even more elegantly than using symlinking. Take a look at Serverless. It's quite a neat way of structuring serverless applications and includes useful features like being able to assign API Gateway endpoints that trigger the Lambda function you are writing. It even allows you to script CloudFormation configurations, so if you have other resources to deploy then you could do so here. Need a 'beta' or 'prod' stage? This can do it for you too. I've been using it for just over a week and while there is a bit of setup to do and things aren't always as clear as you'd like, it is quite flexible and the support community is good!
While using serverless we faced a similar issue, when having the need to share code between AWS Lambdas. Initially we used to duplication the code, across each microservice, but later as always it became difficult to manage.
Since the development done in Windows Environment, using symbolic links was not an option for us.
Then we came up with a solution to use a shared folder to keep the local dependencies and use a custom written gulp task to copy these dependencies across each of the microservice endpoints so that the dependency can be required similar to npm package.
One of the decisions we made is not to keep two places to define the dependencies for microservices, so we used the same package.json to define the local shared dependencies, where gulp task passes this file and copy the shared dependencies accordingly also installing the npm dependencies with a single command.
Later we made the code open source as npm modules serverless-dependency-install and gulp-dependency-install.
I have a project that consists of several nod.js backend applications. The apps are using the same modules (which a placed outside of each ap folder in shared location). The aps they are to be deployed on differnt environments (servers), some code is for test, some for debug as usual.
If I choosed a platform (for example PaaS nodejitsu) for one of my apps, how I'm supposed to send there only production code for one of my apps? I deployed on nodejitsu and it just sends the app folder and uses package.json to configure the app. But there are a bunch of code that is not need (tests) for example and some code is external. And what If I want to obstruct server code too? How this issues are supposed to be soleved?
For front-end applications has a tons of methods to be build for production. I understand that the requirements are different, but didn't find any infromation on best practices fo how correctly to prepare node.js back end application for deploy.
Read section "Keeping files out of your package" in the NPM Developer page. It states following
Use a .npmignorefile to keep stuff out of your package. If there's no .npmignore file, but there is a .gitignore file, then npm will ignore the stuff matched by the .gitignore file. If you want to include something that is excluded by your .gitignore file, you can create an empty .npmignore file to override it.
Add those test files in .gitignore
or make another branch for production in git and push the production branch.
I have a Node.js Express app that I'm hosting on Nodejitsu. I'm already using env switches in Express for dev and prod, but obviously I'm going to need to push to different dev and prod servers.
For Nodejitsu, pushing to these servers requires a simple jitsu deploy, which looks to my package.json for app and domain/subdomain information.
I'm trying to think of the best way to handle the various pushes to dev and prod (and/r perhaps testing, staging, etc. in the future) given the package.json restriction. Clearly I would like to avoid maintaining several git repos, each with its own slightly different package.json.
What kind of workflow would be ideal for handling this?
We are't using Jitsu, but what we do is keep a couple of different version of the confs around in a confs folder. When we go to deploy our scripts copy the appropriate conf/package.json file in to the root dir.
You may be interested in this: https://github.com/nodejitsu/jitsu/issues/134
As Małecki points out in that issue, this can be done with npm scripts. You can have a few script setup in the package.json file(s) that do the appropriate pre-deploy work.