I am creating pypi python package and for that i tired many solution avilable in different websites.
First of all I followed this Approach1, in this approach author has created class and functions but when tried this one my package gave me an error that no module and class name is not define
I tired many other Approaches like this one Approach2 in this approach author created only functions without class and __init__.py
and then tried this one also which is very common Approach3
and here is important files structure provided by pypi officials
├── LICENSE
├── README.md
├── example_pkg
│ └── __init__.py
├── setup.py
└── tests
I am ready with my package but after spending whole night finding which one is Best coding practice? and now got confused
so my questions are as follows
what is the use of file __init__.py?
best practice for giving directory, class and functions names so that it will help during importing.because according to these answers some people said import directory name, some said filename followed by class etc..
Can anyone provide me proper steps so that i can follow in upcoming packages also
I'd recommend following the Python Packaging Authorities guides and tutorials at https://packaging.python.org.
In addition, the PyPA publishes an example project skeleton that you can use as a starting point: https://github.com/pypa/sampleproject/
Related
So basically my problem is very similar to the problem presented in this question:
Correct place for private module to be used by multiple executables
My project structure is as follows:
crate
├── Cargo.toml
└── src
├── bin
│ ├── my-bin-1
│ │ ├── cli.rs
│ │ ├── main.rs
│ └── my-bin-2
│ ├── cli.rs
│ ├── main.rs
├── common.rs
└── lib.rs
lib.rs contains public code, and common.rs contains private code.
I want to have both of my binaries re-use code from common.rs, but I need this code to remain private.
That's why the aformentioned question's solution which is to define a public function in common.rs is not good for my case.
// common.rs
pub fn cant_leak_this_function() {} // uh-oh.. `pub` identifier should NOT be used here
I tried using the pub(crate) identifier, but that made the functions in common.rs private to the binaries.
What I need is some sort of a pub(crate-and-also-the-binaries-in-this-crate-but-not-for-external-crates) specifier (with a shorter name obviously..) which would allow me to use shared private code in my-bin-1/main.rs and in my-bin-2/main.rs, but will also keep that code hidden from external crates.
So far, the only solution that worked for me was creating
a mod.rs file inside a common directory within src/bin, (resulting in an awkward src/bin/common/mod.rs path) and then explicitly using the #[path = ..] feature to consume the shared code from the binaries. However, this method caused rust-analyzer to mark the functions from common/mod.rs as unused even though they clearly were used, and even worse, from some reading online I've come to understand that using the #[path = ..] feature is considered bad practice.
Is there a better way to solve this problem? What am I missing here?
Thanks in advance!
All items are either public or private outside of the crate. There is no mechanism to give other crates special permission. The best you can do is make it clear that while the function is technically public, it should not be used as such.
Take a trick out of the serde crate, which has some exports that must be visible to macros, but shouldn't be used directly (source):
// Used by generated code and doc tests. Not public API.
#[doc(hidden)]
#[path = "private/mod.rs"]
pub mod __private;
The #[doc(hidden)] means it won't sure up in https://docs.rs or your typical cargo doc invocations. And the __private module should hopefully make it obvious to anyone who does stumble across it that it shouldn't be used like other public items.
If this is still unsavory to you, then the only way to truly make it private is indeed to use #[path = ...] to import the file as modules in your separate binaries as the linked Q&A mentions. Then you don't need a library crate at all. The reason that #[path = ...] is bad practice, is because its easy to get wrong and will deceive those unaware of how it actually works.
The reason you would get an "unused" warning from your common functions is because it is essential duplicating the file. Within the specific binary that it is imported, the function may truly not be used. Even if it were used in the other binary, that wouldn't be considered when emitting that warning.
I'm coming from a C# background and have recently started to pick up Python 3. One thing that has lead to quite a bit of confusion so far has been how to structure modules in a package.
In C#, I would create a namespace MyLibrary.Model, in which all my different commonly used classes would reside, such as User, Employee, Customer, etc.. Each of these classes would be in their own file (e.g. User.cs, Employee.cs, etc.), since C# really doesn't care about files at all and only cares about which namespace a class belongs to. I would then import this in the various other bits of the library with using MyLibrary.Model, and all classes in that namespace would be available.
I have noticed that this is not how Python 3 likes to do things. Specifically, statements like "just import everything" seem to go against the design philosophy of Python, and instead, I am supposed to only import what I really need.
My question now is, how should I structure such common class definitions so it "makes sense" in a Python package?
What I have tried so far
One class per file
This most closely mimics the "one class per file" convention in C#, leading to a directory tree as follows:
main.py
mylib/
__init__.py
common/
User.py
Employee.py
Customer.py
controller/
...
...
Back in the main.py file, I would write something like this:
from mylib.common.User import User;
from mylib.common.Employee import Employee;
from mylib.common.Customer import Customer;
This works, but it seems like it has a lot of verbosity that it really doesn't need.
All classes in one file
This seems to work better with the import system as I have understood it. Instead of putting every class into its own file, I would instead put every class into one file. The file structure would then look like this:
main.py
mylib/
__init__.py
common.py
controller/
...
...
Back in the main.py file, I would write something like this:
from mylib.common import User, Employee, Customer;
This seems much more succinct, but I feel like with a bigger project, this could mean a lot of class definitions in one file, and possibly lead to common.py becoming a bloated mess, containing a lot of classes that don't really have anything to do with each other.
Most larger projects I've seen split code into packages (directories) either by domain (customer) or function (models). These packages then had multiple modules with one or more classes in them (e.g. customer/model.py, customer/view.py... or models/customer.py, models/employee.py, etc.). Classes that were supposed to be exposed as package interface were then imported into package's __init__.py file which then allows user to write something like from mylib.customer import CustomerModel, CustomerView or from mylib import customer and then m = customer.CustomerModel(...)
The Hitchhiker's Guide to Python has a chapter covering general project layout from Python packaging perspective but also has bit on Modules and Packages - and basically gives similar set of recommendations there. That can be good starting point.
Good guidance in this can be also looking into existing popular projects that are larger and through some refactoring in past - like Django, Sphinx or SQLAlchemy.
Notwithstanding your having accepted an answer quite early in the game, which might discourage further answers from being offered, I will offer my two cents:
Since I generally utilize the MVC design pattern, I therefore divide my packages into models, views (which usually utilize Jinja2 templates) and controllers. Then there are, of course, packages containing various utility classes etc.
I do not stick to a one-class-per-file rule. If classes are meant to work together cooperatively, they will be in the same file. Then again I might have a file containing multiple utility classes that have nothing in common other than their being utility classes or functions.
But what I really wanted to discuss is the great potential that lies is the use of the __init__.py file that often goes untapped when it is just an empty file. The answer by #blami made reference to this file but did not fully explain its usage. This file is meant to initialize the entire package. You can place in this file import statements to automatically load other modules and packages and/or any other initialization code. In your example (to which I added a missing __init__.py file -- there should be one in each directory that defines a package), we have
main.py
mylib/
__init__.py
common/
__init__.py
User.py
Employee.py
Customer.py
controller/
...
...
Possible Contents of mylib/init.py
import requests # From the standard library, for example
from .common import * # This will use __all__ defined in the mylib.common package
VERSION = "1.1.1"
Possible Contents of mylib/common/init.py
__all__ = ["Employee", "Customer"] # from mylib.common import Employee, Customer
Note that in an __init__.py file for package some_package that the special variable __all__ can be coded to specify a list of submodules to be imported from package some_package when the user writes from some_package import *. In the absence of variable __all__ within the __init__.py file, the same import statement would just import the one package and whatever names are defined in that package and initialized by __init__.py.
Thus the next effect of the single statement import mylib is:
import requests, mylib.common.Employee, mylib.common.Customer, VERSION
So a judicious use of these __init__.py files can remove a lot of the bloat you were concerned about in using these classes.
This question already has answers here:
How do I do a basic import/include of a function from one module to another in Rust 2015?
(3 answers)
Split a module across several files
(7 answers)
How can I include a module from another file from the same project?
(6 answers)
Closed 5 years ago.
How do I include a file with the full path my_project/src/include_me.rs in main.rs?
I've checked the dependencies guide, and all of them appear to be including a binary. I've also checked "The Book", but none of the examples there end in ".rs" either.
How do I make include_me.rs compile with the rest of the project?
There are basically two (main) ways in Rust to include code from somewhere else:
1. "Including" internal code
If your include_me.rs belongs to your project, you should move it to the same folder main.rs lies in:
└── src
├── include_me.rs
└── main.rs
Then you can write this in your main.rs:
mod include_me;
fn main() {
// Call a function defined in the other file (module)
include_me::some_function();
}
A mod declaration makes the Rust compiler look for the corresponding .rs files automatically!
So everything that belongs to your project, belongs in the same folder (or a subfolder thereof) as the folder where main.rs (or lib.rs) is lying. The files are then "included" via the module system. To read a good introduction into modules, please read the chapter on modules in the Rust book. You could also check out Rust by Example on that topic. The module system is pretty central and thus important to learning Rust.
2. "Including" external code
If your include_me.rs is something that doesn't belong to your actual project, but is rather a collection of useful things you are using in multiple projects, it should be seen as a library. To include code of such external libraries, you have to include it as an extern crate. And to make your life easier, you really want to use Cargo!
So let's prepare your include_me.rs as Cargo library project. You need the following file structure (which is automatically generated by cargo new my_library --lib):
. my_library
├── Cargo.toml
└── src
└── lib.rs
Now copy all the contents from include_me.rs into lib.rs (it is just convention to call the root file of a library project lib.rs). Let's say that the my_library folder's full path is ~/code/my_library.
Now let's prepare your main project's Cargo project. It has a similar file
structure (but a main.rs instead of lib.rs, because it's a executable-project, not a library-project):
. my_project
├── Cargo.toml
└── src
└── main.rs
To declare your dependency on my_library, you have to put this into your Cargo.toml:
[package]
name = "my_project"
version = "0.1.0"
authors = ["you"]
edition = "2018"
[dependencies]
my_library = { path = "~/code/my_library" }
You can also use relative paths ("../my_library"), but it only makes sense if you know that the two projects always stay where they are, relative to one another (like if they are both managed in the same repository).
Now you can, in your main.rs, write:
use my_library::some_function;
fn main() {
// Call a function defined in the other file (extern crate)
some_function();
}
If you want to upload any of those two projects, you have to interact with crates.io (or another crates registry, if your company has one), but this is another topic.
(Note: some time ago, it was necessary to write extern crate my_library; inside main.rs. This is not necessary anymore, but you might find old code with extern crate declarations.)
Any other ways?
Yes, but you shouldn't use those. There is the include!() macro which allows you to verbatim include other files, just like the #include from C-land. However, it is strongly discouraged to use this in situations where the module system would be able to solve your problem. include!() is only useful in very special situations, often linked to a more complex build system which generates code.
This is my project structure:
git_dir/
root/
__init__.py
tests/
__init__.piy
test1.py
foo_to_test/
foo.py
I'm using pytest to test foo.py, and test1.py is as follows:
from foo_to_test.foo import func_to_test
def test1():
assert something about func_to_test
From the terminal, i want to run
pytest tests
Now for the problem.
When using --import-mode append or --import-mode prepend it adds 'git_dir' to PYTHONPATH.
The problem is that 'git_dir' is not the project root.
I can add sys.path.append('git_dir/root') but the name 'git_dir' is different for other programmers working on other computers.
I've seen in the pytest documentation that using --import-mode importlib might solve my problem, but it doesn't seem to have any effect on my PYTHONPATH and i can't understand what it is doing.
So my questions are:
What --import-mode importlib is doing?
How can i automatically add my root to the path when testing so it will be the same for every programmer pulling this project?
This looks hard because it's not how it's supposed to work. It may be time to learn about packaging.
The Python module system is designed to look for modules on the sys.path directories, as you describe. This variable is to be populated by the user, somehow, and should preferably not be meddled with.
When a developer wants to use your package, they must make sure it is installed on the system they run the tests on.
If you want to make that easier for them, provide e.g. a project.toml file; then they can build your package, and install it with pip install /path/to/your/distribution-file.tgz.
More info about that here: https://setuptools.readthedocs.io/en/latest/userguide/quickstart.html#python-packaging-at-a-glance.
Your problem is that you have __init__.py in the root directory. When pytest finds a test module, it goes to the parent folders until it finds the one that does not have __init__.py. That's going to be you a directory that's added to sys.path, see Choosing a test layout / import rules. So your root directory should NOT be a Python module.
Now about importlib and why you probably don't need it
--import-mode=importlib circumvents the standard Python way of using modules and sys.path. The exact reasons are unclear from the docs:
With --import-mode=importlib things are less convoluted because pytest doesn’t need to change sys.path or sys.modules, making things much less surprising.
And they also mention that this allows test modules to have non-unique names. But why would we want this?! It seems like with proper file structure everything works fine even if we mess with sys.path.
Anyway, overall the usage of this option doesn't sound convincing, especially given this in the official docs:
Initially we intended to make importlib the default in future releases, however it is clear now that it has its own set of drawbacks so the default will remain prepend for the foreseeable future.
Suppose we have the following project structure:
E:\demo_proj\
utilities_package\
__init__.py
util_one.py
util_two.py
demo_package\
__init__.py
demo_sub_package\
__init__.py
demo_sub_sub_package\
demo_file.py
What is a sensible way for demo_file.py to import from utilities_package?
The utilities_package is in a much higher level directory than demo_file.py.
Although it is not shown in the little diagram I gave above, suppose that utilities_package is used everywhere throughout the project. It is not sensible to make utilities_package a sub-directory of demo_sub_sub_package
Some similar questions have been asked here and here.
The asker of the first question neglected to include a __init__.py in their packages. That is not an issue in my case.
The answers given to the the second question linked-to above are problematic:
Most of the answers are quite old. For example, Loader.load_module was deprecated after Python 3.4
Most of the answers are specific to the question in that they only go up one directory, whereas, I would like to go up several levels.
Some of the answers are high-level descriptions without code. Other answers have the opposite problem: they contain ugly-looking code, but don't bother to explain it.
I am open to solutions which require restructuring the project. I'm also interested in solutions which leave the project structure intact, but which insert the relevant code into demo_file.py
It already has the very epic solution for your problems. Please refer there: Relative imports for the billionth time. Just as a notation, the python3 and python2 have the totally different definitions for the relative and absolute import. For version 3+, any modules import directly using import module_name are treated as the absolute importation, which means this module must be included in the sys.path. Otherwise, for any modules import like from one_module import something or from . import something are treated like the relative importation. The dot-notation represent the current directory. However, any module is executed directly by itself, it will lose its namespace, instead it will always be treated as __main__, and all the files in the same directory will automatically add to sys.path by default. Hope it will help you.