Install pre release python packages via databricks API - databricks

How can I install pre release python packages via databricks api?
I'm using databricks API to install packages, like this:
{
"cluster_id":"10201-my-cluster",
"libraries":[
{
"pypi":{
"package":"simplejson==2.1.*",
"repo":"https://my-pypi-mirror.com"
}
}
]
}
Only stable versions are installed on the cluster.
I want something like this:
{
"cluster_id":"10201-my-cluster",
"libraries":[
{
"pypi":{
"package":"simplejson==2.1.*",
"repo":"https://my-pypi-mirror.com".
"pre-release": true
}
}
]
}
I'm using --pre flag to install pre release packages in my machine, but I don't know how can I do it in databricks.
I wanted to avoid having to install packages through init scripts.

Related

Cannot create new Nest project using yarn as a package manager

I couldn't create a new NestJS project when doing nest new project-name using yarn as a package manager. However, whenever I chooses npm, it successfully create the directories.
My nest version is 8.1.1.
My yarn version is 1.22.11.
My npm version is 6.14.1.
My node version is 14.0.0.
I am using Windows 10 git bash mintty

What is the fastest way to programatically install or update to the latest version a global package?

I have a npx script that will install/update globally a certain package to the latest version, if not already.
Running npm i -g myPackage to ensure the latest version takes too long. Looks like it will go over all the package dependencies and check their versions, even if the desired package is already at the latest version.
I want to do something like this, but programatically. I already use execa package for similar stuff and probably would suit it.:
npm list -g myPackage to check for package instalation
if installed, check if it is updated
npm outdated -g myPackage
if not installed or outdated,
npm i -g myPackage to install/update it.
Is there any ready solution for it or a built-in way to do it via npm? Else, how can I check programatically the results of the two verifications above?
Here is how I done it in TS, it's working. Not sure if it's the best way, but takes way less time than doing a npm i -g myPackage, if it's already updated:
import execa from 'execa';
async function checkGlobalPackageUpdate(packageName: string): Promise<'notInstalled' | 'outdated' | 'updated'> {
console.log(`Checking if ${packageName} is installed and updated...`);
// Check if it's installed
try {
await execa('npm', ['list', '-g', packageName]);
} catch {
return 'notInstalled';
}
// Check if it's updated
const result = await execa('npm', ['outdated', '-g', packageName]);
const updated = result.stdout === '';
return updated ? 'updated' : 'outdated';
}
If no better alternatives show up, I may create a simple npm/npx package to use it, with more configurations.

The engine "node" is incompatible with this module

I am getting below yarn error when deploying to AWS
error fs-extra#7.0.1: The engine "node" is incompatible with this module. Expected version ">=6 <7 || >=8". Got "7.0.0"
Any idea how will this be resolved?
Will this work out if I specify engine in package.json
{
"engines" : {
"node" : ">=8.0.0"
}
}
You can try to ignore the engines :
$ yarn install --ignore-engines
OR
$ yarn global add <your app> --ignore-engines
You can see all what you can ignore by running:
$ yarn help | grep -- --ignore
--ignore-scripts don't run lifecycle scripts
--ignore-platform ignore platform checks
--ignore-engines ignore engines check
--ignore-optional ignore optional dependencies
You need to upgrade your version of node.
I ran into this same issue.
If you used Homebrew run:
brew update # This updates Homebrew to latest version
brew upgrade node
If you use nvm run:
nvm current node -v # Checks your current version
nvm install <version> # Example: nvm install 12.14.1
For the above step go to https://nodejs.org/en/download/
Grab a version which satisfies the conditionals in your error, the latest version should work.
More Detailed Walkthrough: https://flaviocopes.com/how-to-update-node/
A fix that is a hack can be
yarn config set ignore-engines true
However if you want a permanent solution is to :
delete node_modules/, package-lock.json & yarn.lock
run yarn install or npm i again.
Add --ignore-engines to the suffix while installing the package like this:
yarn add <package_name> --ignore-engines
My problem was solved with yarn --ignore-engines, but I'm not sure why and how.
I had a similar issue on Ubuntu even after installing Nodejs many times with the latest version, it was showing always the same old Nodejs version; I discovered it was installing the similar old Debian package each time, even after executing the apt-get update command
Finally, I got it to work by purging the old nodeJs then adding different repository source, and installing nodeJs normally with the new distribution as follows:
sudo apt-get purge --auto-remove nodejs
curl -fsSL https://deb.nodesource.com/setup_14.x | sudo -E bash -
sudo apt-get install -y nodejs
Please find the list of all NodeJs distribution below
https://github.com/nodesource/distributions/blob/master/README.md
You may find other ways of doing the update, but this one worked for me.
I do NOT recommend using this:
% yarn install --ignore-engines
It avoids the issue instead of solving it.
A possible solution would be to update your node to version > 8.0.
% brew upgrade node
Or you could try installing multiple versions of node by using nodenv, in case you need them for other projects.
% brew install nodenv
% nodenv init
# Load nodenv automatically by appending
# the following to ~/.zshrc:
eval "$(nodenv init -)"
% nodenv install 6.0.0 //or some other version
Lots of answers say to set flags to ignore the version error.
A better option is to use this as a reminder to update your node version to something recent and supported by the package you want to install.
nvm install 16.16.0 # download & install locally on your system
nvm use 16.16.0 # update current project's .nvmrc file
Note that the 2nd command will update your local .nvmrc which specifies a per-project node version.
The node ecosystem turns over quickly, even "Long Term Support" (LTS) releases stop getting support after about 3 years. Use this page to see the latest LTS release version, and also make sure it matches the node version expected by the package you're installing, from the error message.
You can try:
Open you package.json
find
"engines": { "node": "14.x" }
change 14.x -> >=14.x
I recommend doing what the error message says and checking your Node.js version (node -v). The easiest way to upgrade Node.js is with the n version manager:
$ npm install -g n
Then install the latest (n latest) or LTS (n lts) version of Node.
What worked for me was to update Node to the latest version. Following any tutorial depending on your OS.
Upgrading Node.js to latest version
sudo npm cache clean -f
sudo npm install -g n
sudo n 10.22.1
node -v => Should be on 10.22.1
type what version of node you require as I have just put 10.22.1 as an example
Update your Node.js to the latest version.
https://nodejs.org/en/download/
you need to run the below command and your problem will be solved
yarn install --ignore-engines
or
npm install --ignore-engines
This is a lot more problematic than it seems on the surface.
If you include a module that requires node 6, but you have other modules that use node 11, you will get this error!
It's problematic when it's 3rd party modules you've used nom/yarn/etc. to install, as you don't have access to those package repos without doing git fork.
In my case, I am using yarn workspaces and some of the modules in the package.json files in the workspaces might require foo 1.0 while others require foo 2.0 and the 1.0 version might require node 6 and the 2.0 version might require node 14.
The only solution I found is to use --ignore-engines, though it clearly is what other(s) have posted - that this is not fixing the problem, just ignoring it in spite of any issues that might be caused (node 6 code might not run on node 14!).
Sometimes you cannot upgrade the Node engine (legacy projects, client requirements etc). The solution I found in this case was to downgrade the problematic versions by using "selective dependency resolution", as documented on Yarn:
https://classic.yarnpkg.com/lang/en/docs/selective-version-resolutions/
{
"name": "project",
"version": "1.0.0",
"dependencies": {
"left-pad": "1.0.0",
"c": "file:../c-1",
"d2": "file:../d2-1"
},
"resolutions": {
"d2/left-pad": "1.1.1",
"c/**/left-pad": "^1.1.2"
}
}
Notice the "resolutions" part. You can force those packages to downgraded versions (compatible with your older Node engine).
I found this problem now, with an old code, however, I solved it with:
yarn upgrade
Just found that not only I need to upgrade the node, but also need to install it.
This upgrades node to latest version:
brew upgrade node
This install the specific version of node:
nvm install 17.0.0
Upgrade your version of node , this issue will be resolved

How can I upgrade Libsass with npm?

I'm currently running NPM's node-sass tool, but the version of libsass it is running is 3.2.2, and the version I need to be running is 3.2.4, as this fixes a crucial bug in one of the frameworks I am using.
I can find no information on how to build and/or update either node-sass or libsass to meet my requirements. I am already running the latest version of node-sass, 3.1.2.
Yet, my node-sass package.json seems to have a key:value pair which indicates libsass is at 3.2.4, yet this clearly isn't correct.
What is the easiest way to upgrade my version of libsass?
Updates
6 June
I have done some additional searching, and still cannot get libsass to be at a version of 3.2.4. I have tried upgrading an older package of node-sass, and checking my environment variables for overrides. No solution yet.
7 June
It does appear that the Libsass version that is sourced by node-sass is 3.2.4, yet it is not being picked up, and is defaulting to a Libass binarypath:
path.join(__dirname, '..', 'vendor', sass.binaryName.replace(/_/, '/'));
which on my machine yields:
H:\myproj\node_modules\gulp-sass\node_modules\node-sass\vendor\win32-x64-14\binding.node
I have no idea what this means. Take a look at node-sass\lib\extensions.js on line 134:
sass.getBinaryPath = function(throwIfNotExists) {
var binaryPath;
if (flags['--sass-binary-path']) {
binaryPath = flags['--sass-binary-path'];
} else if (process.env.SASS_BINARY_PATH) {
binaryPath = process.env.SASS_BINARY_PATH;
} else if (pkg.nodeSassConfig && pkg.nodeSassConfig.binaryPath) {
binaryPath = pkg.nodeSassConfig.binaryPath;
// This is the only statement that executes successfully, my libsass binary path is coming from this location. Why?
} else {
binaryPath = path.join(__dirname, '..', 'vendor', sass.binaryName.replace(/_/, '/'));
}
if (!fs.existsSync(binaryPath) && throwIfNotExists) {
throw new Error(['`libsass` bindings not found in ', binaryPath, '. Try reinstalling `node-sass`?'].join(''));
}
return binaryPath;
};
sass.binaryPath = sass.getBinaryPath();
There is no special command for that. Take a look at the lib/extensions.js file. It has several intresting lines:
/**
* The default URL can be overriden using
* the environment variable SASS_BINARY_SITE
* or a command line option --sass-binary-site:
*
* node scripts/install.js --sass-binary-site http://example.com/
*
* The URL should to the mirror of the repository
* laid out as follows:
* SASS_BINARY_SITE/
* v3.0.0
* v3.0.0/freebsd-x64-14_binding.node
* ... etc. for all supported versions and platforms
*/
Libsass in this case is only a source folder. You can try to make a clean build. Remove node-sass and install it again.
npm install node-sass#3.0.0
...
node ./node_modules/.bin/node-sass --version
node-sass 3.0.0 (Wrapper) [JavaScript]
libsass 3.2.2 (Sass Compiler) [C/C++]
When updating:
npm update node-sass
node ./node_modules/.bin/node-sass --version
node-sass 3.1.2 (Wrapper) [JavaScript]
libsass 3.2.4 (Sass Compiler) [C/C++]
P.S. Be careful with #at-root in 3.2.4. It is bugged.
Update
If it will not solve your problem, try to remove all npm cache with
npm cache clean
Second update
Try to manually install the binding:
cd node-sass
rm -r vendor
node scripts/install.js --sass-binary-site https://github.com/sass/node-sass/releases/download/
It will output something like:
Binary downloaded and installed at /Users/sobolev/Documents/github/modernizr-mixin/node_modules/node-sass/vendor/darwin-x64-14/binding.node
Can you try below steps:
git clone https://github.com/sass/node-sass.git
cd node-sass
git checkout tags/v3.1.2
npm install . -g
node-sass -v
This should fix your problem.
The latest release of node-sass 3.2.0 says
This release bumps Libsass to 3.2.5 which brings with it a bunch of
fixes.
npm install node-sass will install now install a node-sass with a libsass >= 3.2.5.

install latest github release using npm

I would like to install a node package directly from github.
I am able to use a command like this:
npm install git://github.com/codeine-cd/codeineNodePeer.git#V0.0.1
However, I would like to use the latest release, I have a tag for that but I cant write it in the git url (like #latest)
Is it possible to do that any other way?

Resources