Is there a way to have custom behaviour for import statements in Python? How? E.g.:
import "https://github.com/kennethreitz/requests"
import requests#2.11.1
import requests#7322a09379565bbeba9bb40000b41eab8856352e
Alternatively, in case this isn't possible... Can this be achieved in a standard way with function calls? How? E.g.:
import gitloader
repo = gitloader.repo("https://github.com/kennethreitz/requests")
requests = repo.from_commit("7322a09379565bbeba9bb40000b41eab8856352e")
There are two reasons for why I would like to do this. The first reason is convenience (Golang style imports). The second reason is that I need cryptographic verification of plugin modules for a project. I'm using Python 3.x
What you're essentially asking is customization of the import steps. This was made possible with PEP 302 which brought about certain hooks for for customization.
That PEP is currently not the source from which you should learn how importing works; rather, look at the documentation for the import statement for reference. There it states:
Python includes a number of default finders and importers. The first one knows how to locate built-in modules, and the second knows how to locate frozen modules. A third default finder searches an import path for modules. The import path is a list of locations that may name file system paths or zip files. It can also be extended to search for any locatable resource, such as those identified by URLs.
The import machinery is extensible, so new finders can be added to extend the range and scope of module searching.
In short, you'll need to define the appropriate finder (to find the modules you're looking for) and an appropriate loader for loading these.
Related
Sometimes I see slightly different imports
import styled, { withTheme } from "styled-components/macro";
import styled, { withTheme } from "styled-components";
Since they have the same functionality I cannot understand the difference between them neither can google anything that could help.
In production styled-components generates unique hashes for css classes like .eeZmbc or .ZzNLl. These are used for saving space but are not useful for developers in development.
For semantic class names in development a babel plugin exists. It generates names like .Navbar__Item-sc-14eztoj-1 as in .FileName__StyledComponent-generatedHash to help us trace an element/style back to its source.
So if you use create-react-app you can use this plugin without ejecting and without adding it to babel config. You just have change your import from styled-components to styled-components/macro. A quick find-and-replace in you IDE will do the trick.
All the compile-time code transformation is handled by babel-plugin-macros
babel-plugin-macros defines a standard interface for libraries that want to use compile-time code transformation without requiring the user to add a babel plugin to their build system (other than babel-plugin-macros, which is ideally already in place).
Personally I add the babel plugins to the config file manually and use standard imports like styled-components.
I'm coming from a C# background and have recently started to pick up Python 3. One thing that has lead to quite a bit of confusion so far has been how to structure modules in a package.
In C#, I would create a namespace MyLibrary.Model, in which all my different commonly used classes would reside, such as User, Employee, Customer, etc.. Each of these classes would be in their own file (e.g. User.cs, Employee.cs, etc.), since C# really doesn't care about files at all and only cares about which namespace a class belongs to. I would then import this in the various other bits of the library with using MyLibrary.Model, and all classes in that namespace would be available.
I have noticed that this is not how Python 3 likes to do things. Specifically, statements like "just import everything" seem to go against the design philosophy of Python, and instead, I am supposed to only import what I really need.
My question now is, how should I structure such common class definitions so it "makes sense" in a Python package?
What I have tried so far
One class per file
This most closely mimics the "one class per file" convention in C#, leading to a directory tree as follows:
main.py
mylib/
__init__.py
common/
User.py
Employee.py
Customer.py
controller/
...
...
Back in the main.py file, I would write something like this:
from mylib.common.User import User;
from mylib.common.Employee import Employee;
from mylib.common.Customer import Customer;
This works, but it seems like it has a lot of verbosity that it really doesn't need.
All classes in one file
This seems to work better with the import system as I have understood it. Instead of putting every class into its own file, I would instead put every class into one file. The file structure would then look like this:
main.py
mylib/
__init__.py
common.py
controller/
...
...
Back in the main.py file, I would write something like this:
from mylib.common import User, Employee, Customer;
This seems much more succinct, but I feel like with a bigger project, this could mean a lot of class definitions in one file, and possibly lead to common.py becoming a bloated mess, containing a lot of classes that don't really have anything to do with each other.
Most larger projects I've seen split code into packages (directories) either by domain (customer) or function (models). These packages then had multiple modules with one or more classes in them (e.g. customer/model.py, customer/view.py... or models/customer.py, models/employee.py, etc.). Classes that were supposed to be exposed as package interface were then imported into package's __init__.py file which then allows user to write something like from mylib.customer import CustomerModel, CustomerView or from mylib import customer and then m = customer.CustomerModel(...)
The Hitchhiker's Guide to Python has a chapter covering general project layout from Python packaging perspective but also has bit on Modules and Packages - and basically gives similar set of recommendations there. That can be good starting point.
Good guidance in this can be also looking into existing popular projects that are larger and through some refactoring in past - like Django, Sphinx or SQLAlchemy.
Notwithstanding your having accepted an answer quite early in the game, which might discourage further answers from being offered, I will offer my two cents:
Since I generally utilize the MVC design pattern, I therefore divide my packages into models, views (which usually utilize Jinja2 templates) and controllers. Then there are, of course, packages containing various utility classes etc.
I do not stick to a one-class-per-file rule. If classes are meant to work together cooperatively, they will be in the same file. Then again I might have a file containing multiple utility classes that have nothing in common other than their being utility classes or functions.
But what I really wanted to discuss is the great potential that lies is the use of the __init__.py file that often goes untapped when it is just an empty file. The answer by #blami made reference to this file but did not fully explain its usage. This file is meant to initialize the entire package. You can place in this file import statements to automatically load other modules and packages and/or any other initialization code. In your example (to which I added a missing __init__.py file -- there should be one in each directory that defines a package), we have
main.py
mylib/
__init__.py
common/
__init__.py
User.py
Employee.py
Customer.py
controller/
...
...
Possible Contents of mylib/init.py
import requests # From the standard library, for example
from .common import * # This will use __all__ defined in the mylib.common package
VERSION = "1.1.1"
Possible Contents of mylib/common/init.py
__all__ = ["Employee", "Customer"] # from mylib.common import Employee, Customer
Note that in an __init__.py file for package some_package that the special variable __all__ can be coded to specify a list of submodules to be imported from package some_package when the user writes from some_package import *. In the absence of variable __all__ within the __init__.py file, the same import statement would just import the one package and whatever names are defined in that package and initialized by __init__.py.
Thus the next effect of the single statement import mylib is:
import requests, mylib.common.Employee, mylib.common.Customer, VERSION
So a judicious use of these __init__.py files can remove a lot of the bloat you were concerned about in using these classes.
I've come across some code recently that uses a trick that makes me rather nervous. The system I'm looking at has a Python extension Moo.so file stored outside the path and the developer wants to import it with just import Moo. For various reasons neither the file location nor sys.path can be changed, and the extension must be loaded with ExtensionFileLoader anyway.
So what has been done is to have a Moo.py in the path that loads the extension module and then replaces itself in sys.modules with the extension module, along the following lines:
' Moo.py '
from importlib.machinery import ExtensionFileLoader
loader = ExtensionFileLoader('AnotherNameForMoo', '/path/to/Moo.so')
module = loader.load_module()
sys.modules['Moo'] = module
Now this does actually work. (I have some tests of it in rather gory detail in this repo if you want to have a look.) It appears that, at least in CPython 3.4 through 3.7, import Moo does not bind to Moo the module that it loaded and put into sys.modules['Moo'], but instead binds the current value of sys.modules['Moo'] after the module's top-level script returns, regardless of whether or not that's what it originally put in there.
I can't find anything in any Python documentation that indicates that this is required behaviour rather than just an accident of implementation.
How safe is this? What are other ways that one might try to achieve a similar "bootstrap" effect?
I Have got some models (mapped on database tables), in a python project. I want to make them preload when the project bootup so i don't need to do like this in every file
from db.models.user import User
I tried to load them in init.py file, but they are not available.
TL;DR: You need to import your models in every file.
In Python, every module (*.py file) has its own namespace (set of global variables). There is no way to have a truly global variable (or class, function, etc.), because every name is local to the module in which it is defined. This is a deliberate feature of Python; it would be very bad if (for example) you could redefine the range() function in some random module and break everyone else's code.
You can place the import from db.models.user import User in db/models/__init__.py, but that won't make User globally accessible. Instead, it will create an alias db.models.User for the original class (db.models.user.User), which you still have to import as normal. This may save you a small amount of typing when importing the model in other places, and is a great way to flatten deep hierarchies of packages, but it does not make anything truly global, because you cannot make things global.
I need to find out why some module gets included into compilation.
There is some class that should not be included and I think there are some unused imports or bad architecture that requires unnecessary imports. Is there a way to find which modules import some module, which modules import these modules that include this module, and so on, tracking that down to main class of application?
You could use -D dump-dependencies for this, in which case the compiler will generate two files that can be used to follow the dependency graph in both directions:
dump/<target>/.dependants.dump
dump/<target>/.dependencies.dump
There is also a handy online tool created by Mark Knol that helps a lot with analyzing these files. To answer the question "what does Array depend on?", you can just upload the two files and enter "array" into the search field:
Conveniently, the results are also clickable.
I've just came up with very simple idea: just delete this file and there will be compilation errors in places where this module is imported.