I have to get continuous data from a python function and one of its libraries(Overlay) depends on root privilege. I have to run that python function by using Flask.
First of all, I have searched how to give root access to a python script but unfortunately, I haven't found any which mimic my case.
The approach I have taken --
1/ Python file containing Overlay library suppose the name is child.py which contains a function name status_data
from pynq import PL
from pynq import Overlay
def main_path():
ol = Overlay("/home/xilinx/pynq/overlays/design_4/design_1_wrapper.bit")
return ol
def status_data():
ol = main_path()
data = '''my_code'''
return data
2/ Flask file name is app.py where I need the continuous data via an endpoint
from create_json_data import status_data
from flask import Flask, render_template
from flask import jsonify
'''mayn others lib, doesn't include'''
'''my_code'''
#app.route("/auto_update_table")
def parse_auto_update_table(name=None):
data_json = status_data()
return jsonify(data_json)
3/ Both files are in the same folder. I have run that child.py with sudo python3 child.py and it works. Executed the Flask code by sudo python3 app.py. My guess was as app.py runs with sudo so it will take all other dependents with root privilege. But I am failed. It executes but after some moments it crashes.
Is there any workaround to call child.py file's status_data function with root privilege from app.py without crash? Security isn't an issue so I can reveal my credentials to any script file and if also require I can give password manually but only once as it is a continuous process so over and over password typing is not a tangible solution.
As I know when you give admin permission (root) to an executable (your python script).
Whatever the executable does is with admin privileges. Try with calling it with the root account itself or sudo -u root --login "python3 app.py" to see if it works or not
Also saying it doesn't work is not helping. If you have a trace back. Add it to help us answer you or any more information that helps. I can't help you like this
EDITED AFTER user10634362 COMMENT BELOW
I have found a solution. First of all, I am changing the question title as after getting the solution I have understood the present title is not fit with the situation. Though giving root privilege is quite easy. sudo python3 app.py does the work. The major flaw was in
from pynq import Overlay
and
def main_path():
ol = Overlay("/home/xilinx/pynq/overlays/design_4/design_1_wrapper.bit")
return ol
I have called this main_path function repeatedly but it needs a handsome amount of delay and it was absent there. So I got an unwanted crash. After declaring
ol = Overlay("/home/xilinx/pynq/overlays/design_4/design_1_wrapper.bit")
I have passed ol as a global variable to status_data function and it solves the issue.
Related
So I tried many things (from SO and more) getting my tests running but nothing worked this is my current code:
test.py which I call to run the tests: python3 ./src/preprocess/python/test.py
import unittest
if __name__ == '__main__':
testsuite = unittest.TestLoader().discover('.')
unittest.TextTestRunner(verbosity=2).run(testsuite)
the test file looks like this:
import unittest
from scrapes.pdf import full_path_to_destination_txt_file
print(full_path_to_destination_txt_file)
class PreprocessingTest(unittest.TestCase):
def path_txt_appending(self):
self.assertEqual(full_path_to_destination_txt_file(
"test", "/usr/test"), "/usr/test/test.txt")
if __name__ == '__main__':
unittest.main(verbosity=2)
But the output is always like this:
python3 ./src/preprocess/python/test.py
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
Additional Information:
As you can see I call this not from my root directory. The test folder is in ./src/preprocess/python/test/ and has a __init__.pyfile included (there is also a init file on the level of test.py)
it would be okay for me if I have to code down all the calls for all the tests I just want to finish this
automatic search with -t does not work either so I thought the more robust method here with test.py would work...
using this framework is a requirement I have to follow
test_preprocessing.py is in the test folder and from scrapes.pdf import full_path_to_destination_txt_filescrapes is a module folder on the same level as test
When I call the single unit test directly in the command line it fails because of the relative import. But using the test.py (obviously) finds the modules.
What is wrong?
By default, unittest will only execute methods whose name starts with test:
testMethodPrefix
String giving the prefix of method names which will be interpreted as test methods. The default value is 'test'.
This affects getTestCaseNames() and all the loadTestsFrom*() methods.
from the docs.
Either change that attribute or (preferably) prefix your method name with test_.
I have a Python app that works locally, but on the server something goes wrong. So I have to do some debugging in an SSH session. The server logs tell me that something goes wrong in utils/do_something.py. I have created utils/log (rights 777), where the debugging values are supposed to go.
do_something.py looks like this:
import os
parent_folder = os.path.dirname(__file__)
log_file = os.path.join(parent_folder, 'log')
def do_something(arg1, arg2):
print('XXX')
f = open(log_file, 'a')
f.write('XXX\n')
stuff = goes.wrong
Loading the page causes do_something to run (as confirmed by the error logs). Locally the expected XXX appears in the console and in the log file. But nothing happens on the server.
I created a second file utils/blah.py:
print('BLAH')
import os
parent_folder = os.path.dirname(__file__)
log_file = os.path.join(parent_folder, 'log')
f = open(log_file, 'a')
f.write('BLAH\n')
When I run it with python blah.py, the expected BLAH appears in the console and the logfile.
I don't care that much about the difference between local and production server.
But I would like to understand the difference between do_something.py and blah.py.
Is there a better way to debug in an SSH session?
I work in a virtualenv in a mod_wsgi 4.6.5/Python3.7 environment on a Webfaction server. Some details about it can be seen in this question on the Webfaction forum.
Edit 1: On the server print seems to be discouraged anyway.
(See Where do things go when I ‘print’ them from my Django app?)
But what matters to me is f.write(). I just added print for comparison.
Edit 2: It is the same when I use the logging module. It works when I run blah.py, but nothing happens when loading the page runs do_something.
Edit 3: I tried the same with a simpler app, and the result is the same.I added the logging in views.py:
from django.http import HttpResponse
import os
parent_folder = os.path.dirname(__file__)
log_file = os.path.join(parent_folder, 'log')
def home_view(request):
f = open(log_file, 'a')
f.write('XXX\n')
return HttpResponse("Hello from home view.")
Locally this writes XXX to the log file every time the page is loaded. But not on the server.There are no errors in the server log.
Using the logging module: I am not sure why that did not work, but now it does.
import os
import logging
parent_folder = os.path.dirname(__file__)
log_file = os.path.join(parent_folder, 'log')
logging.basicConfig(filename=log_file, level=logging.DEBUG)
logging.debug('This works.')
Possibly I have used it with filename='log' instead of filename=log_file.
Locally the former creates the log file in the root folder. But on the server it must already exist.
Writing rights: It seems worth mentioning that touch log gave me a file I could not wright to, and lacking sudo I could not use chmod. So I used the trick install -b -m 777 /dev/null log.
It appears that you never explicitly close the file after writing, so the output is probably buffered. Your server and your local machine may have different settings with respect to file buffering, which would explain the differences you're experiencing.
If you want to open, write, and close a file, python's context managers are the best way to do it:
def home_view(request):
with open(log_file, 'a') as f:
f.write('XXX\n')
return HttpResponse("Hello from home view.")
You might check the privileges on the utils folder, and on any parent folders. Many times an http server will be running as user 'nobody', which has practically zip privileges. So unless the utils folder itself also has 0777 privileges, that could create a problem. You also might want to put the f.write in a try ... except block so that you can catch the error specifically and create a more useful / informative message about the error. Best.
I've searched a lot for this and have not yet found a definitive solution. The closest thing I've found is this:
import shutil
from os.path import join
import os
import time
import sys
minecraft_dir = ('server diectory')
world_dir = ('server world driectory')
def server_command(cmd):
os.system('screen -S -X stuff "{}\015"'.format(cmd))
on = "1"
while True:
command=input()
command=command.lower()
if on == "1":
if command==("start"):
os.chdir(minecraft_dir)
os.system('"C:\Program Files\Java\jre1.8.0_111\bin\java.exe" -Xms4G -Xmx4G -jar craftbukkit-1.10.2.jar nogui java')
print("Server started.")
on = "0"
else:
server_command(command)
When I launch this program and type 'start' the CMD flashes up and closes instantly. Instead I want the CMD to stay open with the minecraft sever running from it. I'm not sure why this happens or what the problem is, any help would be greatly appreciated.
p.s. I have edited this to my needs (such as removing a backup script that was unnecessary) but it didn't work before. The original link is: https://github.com/tschuy/minecraft-server-control
os.system will simply run the command then return to your python script with no way to further communicate with it.
On the other hand using subprocess.Popen gives you access to the process while it runs, including writing to it's .stdin which is how you send data to the server:
def server_command(cmd):
process.stdin.write(cmd+"\n") #just write the command to the input stream
process = None
executable = '"C:\Program Files\Java\jre1.8.0_111\bin\java.exe" -Xms4G -Xmx4G -jar craftbukkit-1.10.2.jar nogui java'
while True:
command=input()
command=command.lower()
if process is not None:
if command==("start"):
os.chdir(minecraft_dir)
process = subprocess.Popen(executable, stdin=subprocess.PIPE)
print("Server started.")
else:
server_command(command)
you can also pass stdout=subprocess.PIPE so you can also read it's output and stderr=subprocess.PIPE to read from it's error stream (if any)
As well instead of process.stdin.write(cmd+"\n") you could also use the file optional parameter of the print function, so this:
print(cmd, file=process.stdin)
Will write the data to process.stdin formatted in the same way that print normally does, like ending with newline for you unless passing end= to override it etc.
Both of the above answers do not work in the environment I tried them in.
I think the best way is to use RCON, not sending keys to a window.
RCON is the protocol used by games to run commands.
Many python libraries support Minecraft RCON, and the default server.properties file has an option for RCON.
We will use the python module: MCRON.
Install it. It works for windows, mac, linux.
Type:
pip install mcrcon
Lets configure your server to allow RCON.
In server.properties, find the line 'enable-rcon' and make it look like this:
enable-rcon=true
Restart and stop your server.
Find the line 'rcon.password' and set it to any password you will remember.
You can leave the port default at 25575.
Now, open your terminal and type:
mcron localhost
Or your server ip.
You will be prompted to enter the password you set.
Then you can run commands and will get the result.
But we are doing this with python, not the PYPI MCRON scripts - so do this.
from mcrcon import MCRcon as r
with r('localhost', 'insertyourpasswordhere') as mcr:
resp = mcr.command('/list')
print(resp) #there are 0/20 players online: - This will be different for you.
How do I load a python module, that is not built in. I'm trying to create a plugin system for a small project im working on. How do I load those "plugins" into python? And, instaed of calling "import module", use a string to reference the module.
Have a look at importlib
Option 1: Import an arbitrary file in an arbiatrary path
Assume there's a module at /path/to/my/custom/module.py containing the following contents:
# /path/to/my/custom/module.py
test_var = 'hello'
def test_func():
print(test_var)
We can import this module using the following code:
import importlib.machinery
myfile = '/path/to/my/custom/module.py'
sfl = importlib.machinery.SourceFileLoader('mymod', myfile)
mymod = sfl.load_module()
The module is imported and assigned to the variable mymod. We can then access the module's contents as:
mymod.test_var
# prints 'hello' to the console
mymod.test_func()
# also prints 'hello' to the console
Option 2: Import a module from a package
Use importlib.import_module
For example, if you want to import settings from a settings.py file in your application root folder, you could use
_settings = importlib.import_module('settings')
The popular task queue package Celery uses this a lot, rather than giving you code examples here, please check out their git repository
I have been using "ipython --script" to automatically save a .py file for each ipython notebook so I can use it to import classes into other notebooks. But this recenty stopped working, and I get the following error message:
`--script` is deprecated. You can trigger nbconvert via pre- or post-save hooks:
ContentsManager.pre_save_hook
FileContentsManager.post_save_hook
A post-save hook has been registered that calls:
ipython nbconvert --to script [notebook]
which behaves similarly to `--script`.
As I understand this I need to set up a post-save hook, but I do not understand how to do this. Can someone explain?
[UPDATED per comment by #mobius dumpling]
Find your config files:
Jupyter / ipython >= 4.0
jupyter --config-dir
ipython <4.0
ipython locate profile default
If you need a new config:
Jupyter / ipython >= 4.0
jupyter notebook --generate-config
ipython <4.0
ipython profile create
Within this directory, there will be a file called [jupyter | ipython]_notebook_config.py, put the following code from ipython's GitHub issues page in that file:
import os
from subprocess import check_call
c = get_config()
def post_save(model, os_path, contents_manager):
"""post-save hook for converting notebooks to .py scripts"""
if model['type'] != 'notebook':
return # only do this for notebooks
d, fname = os.path.split(os_path)
check_call(['ipython', 'nbconvert', '--to', 'script', fname], cwd=d)
c.FileContentsManager.post_save_hook = post_save
For Jupyter, replace ipython with jupyter in check_call.
Note that there's a corresponding 'pre-save' hook, and also that you can call any subprocess or run any arbitrary code there...if you want to do any thing fancy like checking some condition first, notifying API consumers, or adding a git commit for the saved script.
Cheers,
-t.
Here is another approach that doesn't invoke a new thread (with check_call). Add the following to jupyter_notebook_config.py as in Tristan's answer:
import io
import os
from notebook.utils import to_api_path
_script_exporter = None
def script_post_save(model, os_path, contents_manager, **kwargs):
"""convert notebooks to Python script after save with nbconvert
replaces `ipython notebook --script`
"""
from nbconvert.exporters.script import ScriptExporter
if model['type'] != 'notebook':
return
global _script_exporter
if _script_exporter is None:
_script_exporter = ScriptExporter(parent=contents_manager)
log = contents_manager.log
base, ext = os.path.splitext(os_path)
py_fname = base + '.py'
script, resources = _script_exporter.from_filename(os_path)
script_fname = base + resources.get('output_extension', '.txt')
log.info("Saving script /%s", to_api_path(script_fname, contents_manager.root_dir))
with io.open(script_fname, 'w', encoding='utf-8') as f:
f.write(script)
c.FileContentsManager.post_save_hook = script_post_save
Disclaimer: I'm pretty sure I got this from SO somwhere, but can't find it now. Putting it here so it's easier to find in future (:
I just encountered a problem where I didn't have rights to restart my Jupyter instance, and so the post-save hook I wanted couldn't be applied.
So, I extracted the key parts and could run this with python manual_post_save_hook.py:
from io import open
from re import sub
from os.path import splitext
from nbconvert.exporters.script import ScriptExporter
for nb_path in ['notebook1.ipynb', 'notebook2.ipynb']:
base, ext = splitext(nb_path)
script, resources = ScriptExporter().from_filename(nb_path)
# mine happen to all be in Python so I needn't bother with the full flexibility
script_fname = base + '.py'
with open(script_fname, 'w', encoding='utf-8') as f:
# remove 'In [ ]' commented lines peppered about
f.write(sub(r'[\n]{2}# In\[[0-9 ]+\]:\s+[\n]{2}', '\n', script))
You can add your own bells and whistles as you would with the standard post save hook, and the config is the correct way to proceed; sharing this for others who might end up in a similar pinch where they can't get the config edits to go into action.