I am quite new working with High-Performance Computing (HPC). In the one I am working I cannot install any packages from the internet, I don't even have access to the internet.
The support I have is very limited when I contact them. I am developing a small application and I need the last version of pandas. They have told me that to get the last version I need to do this:
module load lang/Anaconda3
sources/resources/conda/miniconda3/bin/activate
conda activate SigProfilerExtractor
This work but I want to create my own environment. I know how to do this on my own computer but in the HPC I do not know how to do this? I know that the packages must be somewhere but how can I install them in my environment from where they live?
And second question: They have tools and packages located in many different environments so it is very difficult to find out when the future tool I will use is. Not all environments have useful names and the manual they provided is not updated. If I need the tool mytool, how can I find it?
Is there a more or less easy way to be able to tell (1) which OS (Windows or Linux), and (2) which version of nodejs npm install was run on by looking at the node_modules folder? I tried looking in node_modules/.bin but I see both bash and .cmd files.
Quick back story for those interested: I maintain multiple applications, one of them being a legacy angular app we inherited. This app is extremely "fragile" and sensitive; last time we deployed it, it took the team a couple of days to get the environment "just right". We have multiple chroot environments and VMs, as well as user profiles with different nodejs setups, on both Linux and Windows, which are all fine-tuned for some of these legacy applications we manage.
The problem with this specific app is that I need to make a small change to it and recompile it, and the person managing it is not currently accessible. I am extremely worried about "disrupting" anything by touching the node_modules folder, since I know it took the team a long time and much trial & error to get everything just right (including package versions, a lot of which were tried and installed manually, etc). So after I make the change I need to ensure I'm calling everything from the same exact environment.
Use OS module
var os = require('os');
os.platform(); // 'darwin'
os.release(); //'10.8.0'
For node version,
console.log('Version: ' + process.version);
Guys I'm about to install "scrapy" and I was wondering if it would be a good idea to create a virtual environment? I'm also not the expert in doing these types of things, so I would have to research on it before doing anything... but my question still stands should I create one or should I just install it directly "pip3 install scrapy", I ask this because I read somewhere it can conflict with other frameworks, correct me if I'm wrong please.
Yes, you should try to create virtual environments if you have multiple frameworks.
PEP0405 proposes to add to Python a mechanism for lightweight "virtual environments" with their own site directories, optionally isolated from system site directories. Each virtual environment has its own Python binary (allowing creation of environments with various Python versions) and can have its own independent set of installed Python packages in its site directories, but shares the standard library with the base installed Python.
for more information check https://docs.python.org/3/library/venv.html and
https://www.python.org/dev/peps/pep-0405/
My Node application needs to be deployed on Windows and Linux. The main deployment package is built on a Linux CI server.
When this package is deployed to Windows, it crashes immediately due to missing native bindings, such as those for sqlite. Only the bindings for the build platform (Linux) are restored.
With a deadline approaching, we just set up a Windows build configuration which outputs a Windows specific package that contains the appropriate bindings, and we choose the appropriate artifact to bundle in the installer.
This works but feels fragile, as we would need to keep the Node versions in sync between the two otherwise unrelated environments. I would like to be able to do this with a single build configuration.
I couldn't find any guidance on how this is done. I'm imagining a command-line option like --platform=windows to npm ci, or a modification to package.json but I couldn't find any information about this. Presumably this is a reasonably rare requirement, and perhaps there is no tooling around this, which would be a shame.
Another requirement is that the application must be installed without an internet connection. We cannot run npm ci or npm install when we install it as some of our clients do not permit their servers to access the public internet.
Based on your requirements it sounds like building a package on each required platform would be the safest bet, with the least number moving parts to go wrong.
As the comments have suggested most projects rely on an npm install on the required platform so you are stepping into not that common territory.
This works but feels fragile, as we would need to keep the Node versions in sync between the two otherwise unrelated environments. I would like to be able to do this with a single build configuration.
Node uses NODE_MODULE_VERSION (displayed on the releases page) to track ABI compatibility for native modules. This only changes with a new major Node release number.
The CI builds would need to create app packages for each major version of Node you run on each platform. Keeping the Node.js major versions in sync for the application a good thing in any case. Running Node N and N-1 builds until that can be achieved is good cover and probably the best option with the air gap requirements.
NPM Cache
If the air gapped clients are largely on common networks, an NPM cache/proxy (nexus/verdaccio) may be of use. The NPM cache will need a process to snapshot the repo after a production npm install on all required platforms, to be pushed out to your endpoints. Unfortunately binary modules are often distributed out of band from NPM so won't be stored in regular NPM caches. Each client instance will need a complete build environment to build any native modules from source which can sometime present it's own difficulties on Windows platforms.
Alternatives
Node.js is not a great platform for distributing packaged applications to many diverse clients, especially if you need to distribute Node itself. Any language with an external VM requirement presents difficulties. Nodes package management choices and reliance on native modules exacerbate this.
I've given up in the past and converted clients (albeit thin) to Go, as it lends itself to cross platform distribution a lot better by removing the external runtime requirement and having less variables.
I recently started exploring npm and installed a github repo yoonic/nicistore.
But when I try to build it fails.
My question is if I start building things on top of node, which I see has tooo many dependencies from different vendors, Am I completely on the mercy of the respective package developers?
I have seen that most node based github repos fail to build in the first try. If I update one of the modules by running a console command, Is it likely to break all the application?
And if It is does, doesn't it prove node.js an unreliable and unstable development platform?
Think of it as the opposite of most other languages.
You are writing an app in Java.
You want to use LibA, LibB and LibC.
So you try to use LibC 2.4, and as soon as you do, your manager throws all kinds of errors at you.
Why?
Because LibB is using LibC 1.9
So now what are your options?
Strip out all of the calls to all of the new API for LibC that you wanted to use...
...or hope that LibB is open-source, and you can contribute an update for a new version of it, so that you can use the latest version of LibC (and hope it doesn't update).
So now you've done that... but now you've broken your LibA, because it wants the old LibB.
You didn't even want LibA, you just had to have it for your app to be happy with your framework, and the libs that you did actually want to use (B and C). LibA is closed-source, and isn't maintained, anymore. Tough luck. Go back to your old ways, and forget about how much better life could be, if you could only use your framework with the new version of LibC. Or start praying that your framework does a major rewrite, to get rid of the LibA dependency... but then figure out what new hell you have to deal with, just to get LibC working.
Is this really better than Node?
What node allows you to do is install dependencies which are at different versions than the same library that your dependencies are using.
Not that you can't do that with Java, too... but the entire community has decided that it's just not ever going to try to do that, and thus outlaw it at a tool level.
Next, you see too many things which leave you at the mercy of too many vendors...
Going back to Java (or C++, or nearly any mainstream language), looking at Java, itself, how many libraries are made by Sun Microsystems, or by James Gosling?
Moreover, if you want to boil it down, to suggest using only, say, one huge, overarching framework (like Spring MVC) and using no other libraries of any kind (like JodaTime), then how many libraries does Spring itself lean on, and why are they of no concern to you, even if you're just using the compiled VM bytecode?
In fact, a strong argument could be made to be more wary of compiled binaries, in languages where it was traditional to see strong, copy-left licensing like that of the GNU GPL... in that realm, you open yourself up to craziness.
Most of the Node stuff, by comparison, is dirt-simple freeware. And even if it's not, it's quickly replaceable as most are micro-libraries.
I would suggest that updating a Node package your server depends on, via CLI is less hazardous than doing the same to a full-fledged Java project, if your goal is to see your project compile again, some time in the next week, but with the newer fixes/features...
...but if you're talking about a full-scale, production application, you also want to be cognizant of what it is you're doing, with regards to your codebase, regardless.
As to why things don't build for you on the first try, assuming that you're on a non-Windows platform, and your environment is up to date, I don't know.
Most C/C++ projects I clone don't build for me, first try, either. I usually forget something, or there was something poorly documented, or the actual project was set up to make unfair assumptions about the system it would operate in.
Does that mean that C++ is an unreliable/unstable development platform?
Or the hours/days spent on getting Eclipse set up in an enterprise environment, with all kinds of crazy, company-specific projects and project settings?
It sounds like a case of bad design, more than anything.
Then again, most of my projects these days are wrapped in Docker containers. They all run in the same environment, whether they're running in Windows, on a Mac, or on the server. That tends to take the sting out of building projects, regardless of what language the code is in, or what VM / processor they're running on.
You should also be using NPM shrinkwrap files, or Yarn Lockfiles to preserve the build configuration, with the known-working versions of libraries. And you should have unit and integration tests to ensure that changing library versions has no discernible impact on your system.