Is there an option to pick specific files to be only included in a gitlab release instead of having to have the whole repository packaged as zip/tar files.
.
├── docker-compose.yml
├── Dockerfile
├── flow-definitions/ -- folder of flow definition
│ ├── main_flow_definition.json -- root flow definition for nifi instance
│ ├── bin/ -- folder for flow definitions impacted by dev cycle
│ │ ├── impacted_flow_definition.json
│ │ ├── impacted_flow_definition_2.json
│ │ └── ...
│ └── scripts/ -- scripts relevant to data pipelines
│ ├── databases/ -- DDL for target databases used in ETL
│ │ ├── db_name/
│ │ │ ├── <DDL scripts>
│ │ │ └── ...
│ │ └── ...
│ └── groovy/ -- Groovy scripts used in NiFi processors
│ ├── df_folder/
│ │ ├── <groovy scripts used in processors>
│ │ └── ...
│ └── ...
├── README.md
└── .gitignore
Currently I do the releases in gitlab through the UI and it packages the whole repository when a release is done. As you can see from the above folder structure majority of the time the changes are only within the bin/ folder. As the docker file and other components don't usually change frequently.
So instead of packaging all the files, it would be good to release versions that just have the specific changed files in this case within the bin/ folder.
I couldn't find a way to actually achieve this through gitlab if that's an option available
I have monorepo setup and upgraded from yarn 2 to npm 7.
Was working perfectly fine till I tried to deploy my application to the standard app engine environment.
gcloud app deploy kicks off a build inside Cloud Build, which builds my project via buildpacks (nodejs14 buildpack).
My problem is, that this buildpack includes nodejs14 and npm#6 and without using the flex environment it doesn't seem that there is a way to use npm#7 in this standard app engine cloud build.
What I've tried by now:
Use same OS and install node_modules, tar them, upload them, untar them. (could not really do it)
Link packages locally before uploading (don't know if I did that right. App was not finding local packages afterwards)
Trying to use a cloudbuild.yaml, but it seems that this is not possible with standard environment.
in the custom build step "gcp-build": "npm i -g npm#7" (does not work, permission error, see full build log here)
I would really appreciate every pointer in any direction that could help me out.
This is my project setup:
[project]/
├── packages/
│ ├── app
│ │ ├── node_modules/
│ │ ├── src/
│ │ ├── app.yaml
│ │ ├── ...
│ │ └── package.json
│ ├── shared
│ │ ├── node_modules/
│ │ ├── src/
│ │ ├── ...
│ │ └── package.json
│ ├── shared-admin
│ │ ├── node_modules/
│ │ ├── src/
│ │ ├── ...
│ │ └── package.json
│ ├── base-server
│ │ ├── node_modules/
│ │ ├── src/
│ │ ├── ...
│ │ └── package.json
├── package.json
│
└── node_modules/
Many people are using lerna and/or yarn workspace.
I guess either migrating from them to Bazel, or just using them with Bazel together is good to be guided with an example project.
For example, currently, I have a directory structure like this, where foo is an express server and bar is a library consumed by foo, both based on typescript.
<project root>
├── jest.config.js
├── lerna.json
├── package.json
├── packages
│ ├── bar
│ │ ├── jest.config.js
│ │ ├── package.json
│ │ ├── src
│ │ │ └── index.ts
│ │ ├── test
│ │ │ └── unit
│ │ │ └── index.test.ts
│ │ ├── tsconfig.build.json
│ │ └── tsconfig.json
│ └── foo
│ ├── jest.config.js
│ ├── package.json
│ ├── src
│ │ ├── hello.ts
│ │ └── index.ts
│ ├── test
│ │ ├── integration
│ │ │ └── index.test.ts
│ │ └── unit
│ │ └── index.test.ts
│ ├── tsconfig.build.json
│ └── tsconfig.json
├── tsconfig.build.json
├── tsconfig.json
└── yarn.lock
How should I align it with Bazel, like you know, WORKSPACE, BUILD, and their contents?
Any tips or examples?
Thanks!
There are some examples of repo structures somewhat similar to this in the rules_nodejs examples directory. This shows (in this case an Angular app) having shared libs and consuming them, but the principle is the same here.
Generally, you'd only have one WORKSPACE file in the root of your project. While it's possible to have multiple package.json files for different apps and libs, it adds some extra complexity to the ts_library rules which, for getting started, may be best avoided. This example repo shows multiple package.json files, but without Typescript.
For BUILD (or BUILD.bazel) files, the minimum you'll need here is one in foo and one in bar (and one at the root). The more BUILD files you have, the more you split up the compilation units for your source, therefore increasing incrementality.
Then add ts_library rules to those BUILD files, docs for which can be found here, they also show the differences between using tsc directly and ts_library. You can then define source dependencies between foo and bar, a quick example shown below:
packages/foo/BUILD:
ts_libaray(
name = "foo",
srcs = glob(["src/**/*.ts"]),
deps = [
"//packages/bar", <-- this is the source dep for bar
"#npm//some-package",
],
)
packages/bar/BUILD:
ts_libaray(
name = "bar",
srcs = glob(["src/**/*.ts"]),
deps = [
"#npm//some-other-package",
],
)
I want to use the global node_modules (C:\Users\<user_name>\AppData\Roaming\npm\node_modules) or common node_modules (D:\node_modules) instead of local node_modules (D:\angular\projects\project1\node_modules).
I want to use the node_modules for various projects.
This is my project structure.
D:\
├── node_modules (common for all projects)
│ ├── Module-N
├── projects
│ ├── project1
│ │ ├── src
│ │ ├── app
│ │ | ├── Module-N (importing from common)
|
│ ├── project2
│ │ ├── src
│ │ ├── app
│ │ | ├── Module-N (importing from common)
|
├── ....
try working with relative path like when you require you own modules for the global models
for example
assuming we want to get a module from the global node modules and the file we are working on is seats on projects/project1/src
we will do require('../../../node_modules/my_global_model')
try looking into tsconfigs paths: https://www.typescriptlang.org/docs/handbook/module-resolution.html
In some cases, that comes into conflict with the Windows 260 chr path limitation. I'm having a lot of problems with npm install and this limitation.
.
├── app
│ └── node_modules
│ └── submodule
│ └── node_modules
│ └── submodule
│ └── node_modules
│ └── submodule
│ └── node_modules
│ └── submodule
│ └── to_infinity_and_beyond...
│ └── It's a madness!
...
That structure produces paths like:
c:/path_to_my_app/
node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/.....
What i'm doing wrong? It's there a way to avoid it?
Perhaps renaming "node_module" to "nm", or something like that, may helps to save some characters...
That's in advance!
As Brandon Tilley said:
npm dedupe
works fine for me!