Guys I'm about to install "scrapy" and I was wondering if it would be a good idea to create a virtual environment? I'm also not the expert in doing these types of things, so I would have to research on it before doing anything... but my question still stands should I create one or should I just install it directly "pip3 install scrapy", I ask this because I read somewhere it can conflict with other frameworks, correct me if I'm wrong please.
Yes, you should try to create virtual environments if you have multiple frameworks.
PEP0405 proposes to add to Python a mechanism for lightweight "virtual environments" with their own site directories, optionally isolated from system site directories. Each virtual environment has its own Python binary (allowing creation of environments with various Python versions) and can have its own independent set of installed Python packages in its site directories, but shares the standard library with the base installed Python.
for more information check https://docs.python.org/3/library/venv.html and
https://www.python.org/dev/peps/pep-0405/
Related
This question already has answers here:
What is a virtualenv, and why should I use one?
(4 answers)
Why is virtualenv necessary?
(5 answers)
Closed 6 months ago.
I'm new to django and i want to know why i need to use a virtual enviroment for django.
You don't, but it's nice to have. If you plan on working on other python projects or having other people work on it, it'll probably be a good idea to have it..
then to set up the project in any pc all you need is:
virtualenv django
workon django
pip install -r requirements.txt
# ^ remember to make a req! it's super nice
# init the db
python manage.py runserver
Super easy!
Note: if you use windows you need the virtualenvwrapper package
The python installed on your system has its set of packages. When you create any python project, as you grow using Python and Django, you will see that it's not necessarily all the packages you wish to use for that project.
A virtual environment would allow you to have a somewhat isolated environment where you can use the Python version you want, the list of packages you want, and install new packages for that environment only.
A time will come when you would be required to list all the packages necessary to run a particular Django project and put these in a "requirements.txt" file using the desired package manager of your choice. A virtual environment would make it easy to track both the packages and their versions.
You can read the Whys here:
https://www.geeksforgeeks.org/python-virtual-environment/
https://stackoverflow.com/questions/39055728/importance-of-virtual-environment-setup-for-django-with-python#:~:text=In%20most%20simple%20way%2C%20a,installed%20in%20the%20host%20machine.
Virtual Environment is generally about control of versions of different libraries used in your project.
When the only computer is used and the only project is coded, you probably won't deal with versions. But imagine that you have two projects on two computers - one on Django 2.3 and another one - on Django 4. Some of third-party packages will require versions of Django not higher or not lower than the exact version. So if you wanted to switch projects between computers, you would have to reinstall all the libraries on your computers according to needed versions and struggle with some conflicts.
installing libraries inside of a virtual environment solves the problem.
I am quite new working with High-Performance Computing (HPC). In the one I am working I cannot install any packages from the internet, I don't even have access to the internet.
The support I have is very limited when I contact them. I am developing a small application and I need the last version of pandas. They have told me that to get the last version I need to do this:
module load lang/Anaconda3
sources/resources/conda/miniconda3/bin/activate
conda activate SigProfilerExtractor
This work but I want to create my own environment. I know how to do this on my own computer but in the HPC I do not know how to do this? I know that the packages must be somewhere but how can I install them in my environment from where they live?
And second question: They have tools and packages located in many different environments so it is very difficult to find out when the future tool I will use is. Not all environments have useful names and the manual they provided is not updated. If I need the tool mytool, how can I find it?
Although I am relatively new to using Linux, I would like to know more on how deploying packages works. I have tried searching for this but have had no luck. I have seen countless packages and install scripts that use the same looking 'graphical' command line install for the user to select options for the package. Take the Debian net install for example. [1]
As I have a lot to learn, I would only like a summary of how this is possible, and any resources that anyone has on how developers do this.
Thanks in advance.
[1] http://doudoulinux.org/blog/public/screenshots/install/install-selected-tasks.png
OK, now, i believe, i understand what you're after. I'll still insist that interactive configuration of packages is distro specific and shall not be the main form of package configuration.
It is preferred to ship in a default working configuration and then document how the user can change configuration files (normally in /etc) to reconfigure the package. It is often useful to ship several default configurations (example package: wpa_supplicant which ships with several examples of network configuration, all disabled by default) and allow the user to choose by uncommenting lines.
Debian
The debian specific way to get packages configured is debconf, its configuration is a simple shell script (or a perl script, or whatever else that can talk over STDIN/STDOUT) and a template file. The template file is what will provide options in the aptitude/apt-get interface as in the screenshot in your question.
It is worth reading the debian guidelines on package configuration to have an idea of which kind of configuration is too much.
And they also have a thorough tutorial. Since you said you do not have experience packaging i also recommend reading the introduction to packaging, which will tell you where the files shall be placed. Also, dbhelper is a great tool to place files in the correct place in the package directory.
On other distros
Each distro has its own way of adding configuration to packages. Debian is notable for its debconf as it is one of the most feature rich tools for configuring packages (together with dpkg-reconfigure).
Developers of the software that is sipped in a distro package are often different people from the ones that do the packaging. Configuration options left by developers are often much more thorough than in the package (e.g. inclusion or exclusion of certain libraries).
The fact that you're most familiar with the debian distro (that is an assumption based on your question tags), might give you a misleading idea of package configuration. Only debian based distros have so many configurable packages, other distros often use package dependencies to install differently configured packages. For example:
RedHat based distros (.rpm packages) have no tool such as debconf (as far as i am aware), they use distinct (conflicting) package names to install differently configured packages.
Arch Linux based distros enforce the user configuration. In essence, arch forces the user to configure their packages by going into their configuration files and changing the configuration (that is a very good thing if you want to learn).
Funny enough, both RPM and Arch based distros often ship po-debconf, an adapted debconf for that distro. Yet, i cannot tell much about it since i never tried it.
I have packaged my application into an RPM package, say, myapp.rpm. While installing this application, I would like to receive some inputs from the user (an example for input could be - environment where the app is getting installed - "dev", "qa", "uat", "prod"). Based on the input, the application will install the appropriate files. Is there a way to pass parameters while installing the application?
P.S.: A possible solution could be to create an RPM package for each environment. However, in our scenario, this is not a viable option since we have around 20 environments and we do not wish to have 20 different packages for the same application.
In general, RPM packages should not require user interaction. Time and time again, the RPM folks have stated that it is an explicit design goal of RPM to not have interactive installs. For packages that need some sort of input before first use, you typically ask for this information on first use, our you put it all in config files with macros or something and tell your users that they will have to configure the application before it is usable.
Even passing a parameter of some sort counts as end-user interaction. I think what you want is to have your pre or install scripts auto detect the environment somehow, maybe by having a file somewhere they can examine. I'll also point out that from an RPM user's perspective, having a package named *-qa.rpm is a lot more intuitive than passing some random parameter.
For your exact problem, if you are installing different content, you should create different packages. If you try to do things differently, you're going to end up fighting the RPM system more and more.
It isn't hard to create a build system that can spit out 20+ packages that are all mostly similar. I've done it with a template-ish spec file and some scripts run by make that will create the various spec files and build the RPMs. Without knowing the specifics, it sounds like you might even have a core package that all 20+ environment packages depend on, then the environment specific packages install whatever is specific to their target environment.
You could use the relocate option, e.g.
rpm -i --relocate /env=/uat somepkg.rpm
and have your script look up the variable data from a file located in the "env" directory
I think this is a very valid question, specially as soon as you are moving into the application development realm. There he configuration of the application for different target systems is your daily bread: you need to configure for Development, Integration Test, Acceptance Test, Production etc. I sure don't think building a seperate package for each enviroment is the solution. Basically it should be the same code running in different enviroments.
I know that this requirement is not supported by rpm. But what you can do as a work around is to use a simple config file, that the %pre script knows
to look for. The config file could be a simple shell script that for example sets environment variables, and then the different und pre and post scripts can use those.
have you struggled with Linux deployment before?
I need to deploy an application into a dedicated Linux box with no outside network access.
The configuration should be as simple as possible, robust for different configurations (missing libraries and build tools) and preferably automatic. Another difficulty I need to consider is that I need to connect to an Oracle database.
What would you recommend as the best way for deployment? I have some ideas, but not sure which is the best.
I can use Java
I will need to install JDK, and that solves mostly everything
Another big problem is that the code we presently have in Java is poorly written and slow.
I'm not sure if I need to install Instantclient to connect to Oracle under linux
I can use C (I do have the source code for a very well-written LGPL program)
And use dpkg to deploy
The Linux box is most likely a Ubuntu server, but I'm not sure which version is installed
I can't use apt-get, but I can copy all the packages I need
I know I can use dpkg -s to check which packages they are, but I'm really not sure if I might miss dependencies.
I guess I will need build-essentials and pcap or such
And use static linking
I configured it with ./configure LDFLAGS=-static with no errors and it works on my computer now
I have chroot into this directory and run it without problems, does this mean this is okay?
I really need to test this on a new Linux box to make sure
And use Statifier
I browsed stackoverflow and found this app, haven't tried it out yet.
Seems like people have used it with mixed success.
And create a build environment and make
I have no confidence that this is going to work
Using C leaves some problems
But the program is incomplete, I have to process this data, preferably not in C.
I have to install Instantclient, which is difficult to deploy
I can use Perl
I can't use CPAN
I have already downloaded the libraries, so maybe I could just copy them into the deployed machine, I am not sure how or whether this works
Perl is slow
I have to install Instantclient anyways
Please share your similar experience.
C with static linking solves a lot of the portability problems at the expense of a larger executable. To make sure that everything is truly getting statically linked and not secretly depending on any outside libraries, run ldd on your executable and make sure it isn't dynamically loading everything. Note that this won't be 100% portable amoung various linux machines because Oracle instantclient has some dependencies on kernel versions, but it should work on any reasonably new kernel.
Edit: If the box has LAN access and and just no internet access, why not run your own apt repository on the local network. You could even create a .deb for your application and put it on the same server, than on that machine you just need to execute apt-get myApplication and it will pull down your app and any noninstalled dependencies as well. Setting up an apt mirror is actually pretty easy and this would be pretty slick. If network access is missing alltogether, you can still create an install dvd that has all the debs including your app and set up apt-get to pull from there.