I'm trying to add a couple of miscellaneous development helpers to my python project in such a way that I don't need to either import them or declare them global at the call site, just to save myself some typing.
Example usage would be something like:
# Somewhere, maybe src/__init__.py?
from pprint import pprint
superduperglobaleasyusenamespace.p = pprint
# A different file somewhere in my project
def whatever():
p('hello')
I looked at builtin but wasn't able to get it to work. If that's the correct solution, provide example code that works in python3.
Add this as very first line in your program:
__builtins__.p = pprint
In general, it is not recommend to modify the builtin name space. An import is just one line after all.
Example
# mod1.py
from pprint import pprint
__builtins__.p = pprint
import mod2
and
# mod2.p
print(p)
Now:
python mod1.py
<function pprint at 0x10df28e18>
Related
I am using Spyder as Editor/IDE.
I play with the options to include files/function/modules form other olders just to get a better understanding of how it works.
But Spyder rearranges my import statements in alphabetical order and thus breaks my functionality:
This is what I need:
import sys
sys.path.insert(1,'d:/pathtohelloworld/')
import helloworld
But when I do "Save File" in Spyder, it rearranges to
import helloworld
import sys
sys.path.insert(1,'d:/pathtohelloworld/')
and of course it will fail, since it cannot import "helloworld" before the path is defined via sys.path.insert.
This alphabetical order might be a good python style, but how can I fix this issue ?
I'm bit confused about how the global variables work. I have a large project, with around 50 files, and I need to define global variables for all those files.
What I did was define them in my projects main.py file, as following:
# ../myproject/main.py
# Define global myList
global myList
myList = []
# Imports
import subfile
# Do something
subfile.stuff()
print(myList[0])
I'm trying to use myList in subfile.py, as following
# ../myproject/subfile.py
# Save "hey" into myList
def stuff():
globals()["myList"].append("hey")
An other way I tried, but didn't work either
# ../myproject/main.py
# Import globfile
import globfile
# Save myList into globfile
globfile.myList = []
# Import subfile
import subfile
# Do something
subfile.stuff()
print(globfile.myList[0])
And inside subfile.py I had this:
# ../myproject/subfile.py
# Import globfile
import globfile
# Save "hey" into myList
def stuff():
globfile.myList.append("hey")
But again, it didn't work. How should I implement this? I understand that it cannot work like that, when the two files don't really know each other (well subfile doesn't know main), but I can't think of how to do it, without using io writing or pickle, which I don't want to do.
The problem is you defined myList from main.py, but subfile.py needs to use it. Here is a clean way to solve this problem: move all globals to a file, I call this file settings.py. This file is responsible for defining globals and initializing them:
# settings.py
def init():
global myList
myList = []
Next, your subfile can import globals:
# subfile.py
import settings
def stuff():
settings.myList.append('hey')
Note that subfile does not call init()— that task belongs to main.py:
# main.py
import settings
import subfile
settings.init() # Call only once
subfile.stuff() # Do stuff with global var
print settings.myList[0] # Check the result
This way, you achieve your objective while avoid initializing global variables more than once.
See Python's document on sharing global variables across modules:
The canonical way to share information across modules within a single program is to create a special module (often called config or cfg).
config.py:
x = 0 # Default value of the 'x' configuration setting
Import the config module in all modules of your application; the module then becomes available as a global name.
main.py:
import config
print (config.x)
In general, don’t use from modulename import *. Doing so clutters the importer’s namespace, and makes it much harder for linters to detect undefined names.
You can think of Python global variables as "module" variables - and as such they are much more useful than the traditional "global variables" from C.
A global variable is actually defined in a module's __dict__ and can be accessed from outside that module as a module attribute.
So, in your example:
# ../myproject/main.py
# Define global myList
# global myList - there is no "global" declaration at module level. Just inside
# function and methods
myList = []
# Imports
import subfile
# Do something
subfile.stuff()
print(myList[0])
And:
# ../myproject/subfile.py
# Save "hey" into myList
def stuff():
# You have to make the module main available for the
# code here.
# Placing the import inside the function body will
# usually avoid import cycles -
# unless you happen to call this function from
# either main or subfile's body (i.e. not from inside a function or method)
import main
main.mylist.append("hey")
Using from your_file import * should fix your problems. It defines everything so that it is globally available (with the exception of local variables in the imports of course).
for example:
##test.py:
from pytest import *
print hello_world
and:
##pytest.py
hello_world="hello world!"
Hai Vu answer works great, just one comment:
In case you are using the global in other module and you want to set the global dynamically, pay attention to import the other modules after you set the global variables, for example:
# settings.py
def init(arg):
global myList
myList = []
mylist.append(arg)
# subfile.py
import settings
def print():
settings.myList[0]
# main.py
import settings
settings.init("1st") # global init before used in other imported modules
# Or else they will be undefined
import subfile
subfile.print() # global usage
Your 2nd attempt will work perfectly, and is actually a really good way to handle variable names that you want to have available globally. But you have a name error in the last line. Here is how it should be:
# ../myproject/main.py
# Import globfile
import globfile
# Save myList into globfile
globfile.myList = []
# Import subfile
import subfile
# Do something
subfile.stuff()
print(globfile.myList[0])
See the last line? myList is an attr of globfile, not subfile. This will work as you want.
Mike
I just came across this post and thought of posting my solution, just in case of anyone being in the same situation as me, where there are quite some files in the developed program, and you don't have the time to think through the whole import sequence of your modules (if you didn't think of that properly right from the start, such as I did).
In such cases, in the script where you initiate your global(s), simply code a class which says like:
class My_Globals:
def __init__(self):
self.global1 = "initial_value_1"
self.global2 = "initial_value_2"
...
and then use, instead of the line in the script where you initiated your globals, instead of
global1 = "initial_value_1"
use
globals = My_Globals()
I was then able to retrieve / change the values of any of these globals via
globals.desired_global
in any script, and these changes were automatically also applied to all the other scripts using them. All worked now, by using the exact same import statements which previously failed, due to the problems mentioned in this post / discussion here. I simply thought of global object's properties being changing dynamically without the need of considering / changing any import logic, in comparison to simple importing of global variables, and that definitely was the quickest and easiest (for later access) approach to solve this kind of problem for me.
Based on above answers and links within I created a new module called global_variables.py:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# ==============================================================================
#
# global_variables.py - Global variables shared by all modules.
#
# ==============================================================================
USER = None # User ID, Name, GUID varies by platform
def init():
""" This should only be called once by the main module
Child modules will inherit values. For example if they contain
import global_variables as g
Later on they can reference 'g.USER' to get the user ID.
"""
global USER
import getpass
USER = getpass.getuser()
# End of global_variables.py
Then in my main module I use this:
import global_variables as g
g.init()
In another child imported module I can use:
import global_variables as g
# hundreds of lines later....
print(g.USER)
I've only spent a few minutes testing in two different python multiple-module programs but so far it's working perfectly.
Namespace nightmares arise when you do from config import mySharedThing. That can't be stressed enough.
It's OK to use from in other places.
You can even have a config module that's totally empty.
# my_config.py
pass
# my_other_module.py
import my_config
def doSomething():
print(my_config.mySharedThing.message)
# main.py
from dataclasses import dataclass
from my_other_module import doSomething
import my_config
#dataclass
class Thing:
message: str
my_config.mySharedThing = Thing('Hey everybody!')
doSomething()
result:
$ python3 main.py
Hey everybody!
But using objects you pulled in with from will take you down a path of frustration.
# my_other_module.py
from my_config import mySharedThing
def doSomething():
print(mySharedThing.message)
result:
$ python3 main.py
ImportError: cannot import name 'mySharedThing' from 'my_config' (my_config.py)
And maybe you'll try to fix it like this:
# my_config.py
mySharedThing = None
result:
$ python3 main.py
AttributeError: 'NoneType' object has no attribute 'message'
And then maybe you'll find this page and try to solve it by adding an init() method.
But the whole problem is the from.
This is how the code looks like
#!/usr/bin/env python3
#encoding:utf-8
import requests, numpy, fasttext, os, sys
from itertools import product
from math import sqrt
en_model=fasttext.load_model(path='crawl-300d-2M-subword.bin')
The script is intended to train a classification model using some NLP techniques. Here is the problem.
The last line of the snippet, for some unknown reason, outputs an empty line to the stderr, even though it runs without error. Is there any way to suppress it from the calling module, or do I have to hack into the fasttext module to know which line is causing this? In general, is there any way to suppress any stdout or stderr echo within a code snippet, specifically when I know that they are caused by modules being imported rather than what I wrote?
You can check contextlib.redirect_stdout and contextlib.redirect_stderr
Sample:
from contextlib import redirect_stdout
from contextlib import redirect_stderr
import os
def cache_stdouterr(func):
def wrapper():
with open(os.devnull,"w") as f:
with redirect_stdout(f):
with redirect_stderr(f):
func()
return wrapper
#cache_stdouterr
def noprint():
print("notprinted")
def toprint():
print("shouldbeprinted")
toprint()
noprint()
# python3 test_redirect.py
shouldbeprinted
This is the generic solution. For your specific problem , you can try this :
#!/usr/bin/env python3
#encoding:utf-8
import requests, numpy, fasttext, os, sys
from contextlib import redirect_stdout, redirect_stderr
from itertools import product
from math import sqrt
with redirect_stderr(open(os.devnull,"w")):
en_model=fasttext.load_model(path='crawl-300d-2M-subword.bin')
I meet a similar problem which can be simplified as following:
For example I have a file structure as following:
----folder
---- main.py
---- math.py
I define a function in math.py and I want to import this math.py in main.py .
The codes in math.py is following
# math.py
def f(x) :
return x**3
If I write codes in main.py as following
# main.py
import math
def main() :
print(math.f(3))
if __name__ == "__main__":
main()
then it returns AttributeError: module 'math' has no attribute 'f'
If I write codes in main.py as following
# main.py
from . import math
def main() :
print(math.f(3))
if __name__ == "__main__":
main()
Then it returns ImportError: cannot import name 'math' from '__main__' (main.py)
My question:
If I only want to import the module math.py in path folder which has the same name as build-in module, what should I do?
If in the main.py I want to use both math.f(x) defined in my math.py and built-in math.acos(x), what should I do?
PS: I meet similar problem since I have a long codes written by someone ten years ago. At that time there is no built-in module with such name (In fact it's not math module. I just simplify the problem by above question). And the functions of this module have been used at many places. Therefore it's almost impossible to change module's name since if so I need to carefully change all sites module.function().
It's pretty bad practice to name your modules after built-in modules. I'd recommend naming your math.py something else.
That being said, you could import it using the path with imp:
import imp
math = imp.load_source('math', './math.py')
Dove into a bit of a rabbit hole but here we go. As a disclaimer, like Jack said naming modules after builtins is very bad practice, and this can more easily be accomplished using imp as he suggested.
The reason you're having problems come from the interaction of a few things. When you type
import math
What your python does is look at sys.path. It will check in all the locations in sys.path for a module named math, and import the first one it finds. In your case, it finds your local math module first. After the import is completed, it adds it to sys.modules, but we'll get back to that.
Since you want to use both, first you can import your local math as you have. I would suggest importing it as a different name to keep it separate.
from . import math as local_math
After that, we need to mess with our sys.path to find the builtin math
sys.path = sys.path[::-1]
This reverses the order of sys.path, meaning it will look in your local directory first instead of last.
Now you might think this was enough, but if you try to import math here, python will see it in sys.modules and not bother trying to import again, so first
del sys.modules['math']
Then we can import the default math module
import math
and finally clean up our sys.path
sys.path = sys.path[::-1]
now we have access to everything we need
>>>math.cos(10)
-0.8390715290764524
>>>local_math.f(10)
1000
According to official docs: https://docs.python.org/3/library/imp.html
imp will be removed in version 3.12
Deprecated since version 3.4, will be removed in version 3.12: The imp module is deprecated in favor of importlib.
From imp approach (Deprecated):
import imp
m = imp.load_source('math', './math.py')
m.foo()
To importlib approach with the minimum code 'impact' would be:
def load_module(name, path):
from importlib import util as importlib_util
module_spec = importlib_util.spec_from_file_location(name, path)
module = importlib_util.module_from_spec(module_spec)
module_spec.loader.exec_module(module)
return module
m = load_module('math', './math.py')
m.foo()
i've seen plenty of importing questions but didn't find any that explained importing very "easily". there are are 3 types of importing that i know of, and none of them seem to do what i'm looking for or would like to do. eg. main.py has def a() def b() class c() and def d() and lets say i have a start.py file.
main:
def a():
print("1")
def b():
print("2")
class c():
def__init__(self,name = "Rick")
self.name = name
def d():
print("4")
so now im my start.py file i want to import everything from them. what is the best way? i have tried using import main and i run into issues after creating an instance of class c [ ricky = c() ]that ricky isn't defined or accessing ricky.name it will say module ricky has no attribute name. so that doesn't seem to work. what is this even used for if you aren't importing the entire main.py file?
then there is from main import a, b, c, d that seems to work just fine, but there really has to be another way than having to import every single function, definition, variable, and everything.
third there is from main import * i'm not sure how this one works, i have read some on it mainly stating there should be an __ all __ = everything i want imported. but where do i put this. at the very top of the page inside my main.py? but there still should be a better way?
is my import main just not working correctly? or do i have to list everything i want to import either in a from main import statement or in an __ all __ list?
does importing carry over to another py file? eg. 1.py 2.py 3.py if inside 2.py i import 3.py correctly and everything works. in 1.py can i just import 2.py and it will import 3.py into 1.py from the import statement inside of 2.py? or do i have to import 2.py and 3.py again into 1.py?
the 3 main imports:
import (pythonfile) aka "module" using this will import all classes and functions to be used. does not import other imports. to call something in the imported module eg. modlue: MAIN, function: FUNC ... to call: MAIN.FUNC()
from module import FUNC, CLASS, .... when using this import you don't need to call it with the module. it is almost as if it is right infront of you eg.
module: MAIN, function: FUNC ..... to call: FUNC()
from module import * a combination of the previous two imports. will import everything from the module to be accessed without calling with the module extention. this form imports other imports as well. so if you have two modules that need to talk to eachother. using this will cause an error since you will be trying to import a module into another module then back into it's self again. A imported into B, A and B imported back into A. it doesn't work. so watch when using. May cause other importing errors of trying to import multiple modules that share imports. eg. importing A and B into C if A and B share D (pending for testing)
from MAIN import * call function: FUNC()
hope this helps other people out who are having issues understanding exactly how importing works and calling your functions/classes and what not.