I have a script that monitors a platforms chat, and logs it to a file. Here is the code to setup my logger:
import logging
import logging.handlers as handlers
import time
from datetime import datetime
chatlogger = logging.getLogger("chatlog")
chatlogger.setLevel(logging.INFO)
logHandler = handlers.TimedRotatingFileHandler('chatlog_', when='midnight', interval =1, encoding='utf-8')
logHandler.setLevel(logging.INFO)
logHandler.suffix="%Y%m%d.log"
chatlogger.addHandler(logHandler)
logHandler.doRollover() #this line is needed if when=midnight, otherwise it does not crate the proper file
This works, the chatlog_.yyyymmddd.log gets created and it rolls over when it should. However, there are two small issues I'd like to address/address differently.
The first is that the very first log file the script creates does not have the suffix; it is just 'chatlog_' and nothing else. I added in the doRollover() to correct this, is a different or better way to handle initiating the logfile? The script will be run 24/7(*or as close to that as possible), being restarted with the machine.
The second issue is more of an aesthetic thing. The logHandler.suffix() adds in a '.' between the filename and suffix. Is there something I can do to so stop that from happening?
Related
I'm developing a Python code that can run two applications and exchange information between them during their run time.
The basic scheme is something like:
start a subprocess with the 1st application
start a subprocess with the 2nd application
1st application performs some calculation, writes a file A, and waits for input
2nd application reads file A, performs some calculation, writes a file B, and waits for input
1st application reads file B, performs some calculation, writes a file C, and waits for input
...and so on until some condition is met
I know how to start one Python subprocess, and now I'm learning how to pass/receive information during run time.
I'm testing my Python code using a super-simple application that just reads a file, makes a plot, closes the plot, and returns 0.
I was able to pass an input to a subprocess using subprocess.communicate() and I could tell that the subprocess used that information (plot opens and closes), but here the problems started.
I can only send an input string once. After the first subprocess.communicate() in my code below, the subprocess hangs there. I suspect I might have to use subprocess.stdin.write() instead, since I read subprocess.communicate() will wait for the end of the file and I wish to send multiple times different inputs during the application run instead. But I also read that the use of stdin.write() and stdout.read() is discouraged. I tried this second alteranative (see #alternative in the code below), but in this case the application doesn't seem to receive the inputs, i.e. it doesn't do anything and the code ends.
Debugging is complicated because I haven't found a neat way to output what the subprocess is receiving as input and giving as output. (I tried to implement the solutions described here, but I must have done something wrong: Python: How to read stdout of subprocess in a nonblocking way, A non-blocking read on a subprocess.PIPE in Python)
Here is my working example. Any help is appreciated!
import os
import subprocess
from subprocess import PIPE
# Set application name
app_folder = 'my_folder_path'
full_name_app = os.path.join(app_folder, 'test_subprocess.exe')
# Start process
out_app = subprocess.Popen([full_name_app], stdin=PIPE, stdout=PIPE)
# Pass argument to process
N = 5
for n in range(N):
str_to_communicate = f'{{\'test_{n+1}.mat\', {{\'t\', \'y\'}}}}' # funny looking string - but this how it needs to be passed
bytes_to_communicate = str_to_communicate.encode()
output_communication = out_app.communicate(bytes_to_communicate)
# output_communication = out_app.stdin.write(bytes_to_communicate) # alternative
print(f'Communication command #{n+1} sent')
# Terminate process
out_app.terminate()
Is there anything in the Python API that lets you alter the artifact subdirectories? For example, I have a .json file stored here:
s3://mlflow/3/1353808bf7324824b7343658882b1e45/artifacts/feature_importance_split.json
MlFlow creates a 3/ key in s3. Is there a way to change to modify this key to something else (a date or the name of the experiment)?
As I commented above, yes, mlflow.create_experiment() does allow you set the artifact location using the artifact_location parameter.
However, sort of related, the problem with setting the artifact_location using the create_experiment() function is that once you create a experiment, MLflow will throw an error if you run the create_experiment() function again.
I didn't see this in the docs but it's confirmed that if an experiment already exists in the backend-store, MlFlow will not allow you to run the same create_experiment() function again. And as of this post, MLfLow does not have check_if_exists flag or a create_experiments_if_not_exists() function.
To make things more frustrating, you cannot set the artifcact_location in the set_experiment() function either.
So here is a pretty easy work around, it also avoids the "ERROR mlflow.utils.rest_utils..." stdout logging as well.
:
import os
from random import random, randint
from mlflow import mlflow,log_metric, log_param, log_artifacts
from mlflow.exceptions import MlflowException
try:
experiment = mlflow.get_experiment_by_name('oof')
experiment_id = experiment.experiment_id
except AttributeError:
experiment_id = mlflow.create_experiment('oof', artifact_location='s3://mlflow-minio/sample/')
with mlflow.start_run(experiment_id=experiment_id) as run:
mlflow.set_tracking_uri('http://localhost:5000')
print("Running mlflow_tracking.py")
log_param("param1", randint(0, 100))
log_metric("foo", random())
log_metric("foo", random() + 1)
log_metric("foo", random() + 2)
if not os.path.exists("outputs"):
os.makedirs("outputs")
with open("outputs/test.txt", "w") as f:
f.write("hello world!")
log_artifacts("outputs")
If it is the user's first time creating the experiment, the code will run into an AttributeError since experiment_id does not exist and the except code block gets executed creating the experiment.
If it is the second, third, etc the code is run, it will only execute the code under the try statement since the experiment now exists. Mlflow will now create a 'sample' key in your s3 bucket. Not fully tested but it works for me at least.
So I have this GUI that I made with tkinter and everything works well. What it does is connects to servers and sends commands for both Linux or Windows. I went ahead and used pyinstaller to create a windowed GUI without console and when I try to uses a specific function for sending Windows commands it will fail. If I create the GUI with a console that pops up before the GUI, it works like a charm. What I'm trying to figure out is how to get my GUI to work with the console being invisible to the user.
The part of my code that has the issue revolves around subprocess. To spare you all from the 400+ lines of code I wrote, I'm providing the specific code that has issues. Here is the snippet:
def rcmd_in(server):
import subprocess as sp
for i in command_list:
result = sp.run(['C:/"Path to executable"/rcmd.exe', '\\\\' + server, i],
universal_newlines=True, stdout=sp.PIPE, stderr=sp.STDOUT)
print(result.stdout)
The argument 'server' is passed from another function that calls to 'rcmd_in' and 'command_list' is a mutable list created in the root of the code, accessible for all functions.
Now, I have done my due diligence. I scoured multiple searches and came up with an edit to my code that makes an attempt to run my code with that console invisible, found using info from this link: recipe-subprocess. Here is what the edit looks like:
def rcmd_in(server):
import subprocess as sp
import os, os.path
si = sp.STARTUPINFO()
si.dwFlags |= sp.STARTF_USESHOWWINDOW
for i in command_list:
result = sp.run(['C:/"Path to executable"/rcmd.exe', '\\\\' + server, i],
universal_newlines=True, stdin=sp.PIPE, stdout=sp.PIPE,
stderr=sp.STDOUT, startupinfo=si, env=os.environ)
print(result.stdout)
The the problem I have now is when it runs an error of "Error:8 - Internal error -109" pops up. Let me add I tried using functions 'call()', 'Popen()', and others but only 'run()' seems to work.
I've reached a point where my brain hurts and I can use some help. Any suggestions? As always I am forever great full for anyone's help. Thanks in advance!
I figured it out and it only took me 5 days! :D
Looks like the reason the function would fail falls on how Windows handles stdin. I found a post that helped me edit my code to work with pyinstaller -w (--noconsole). Here is the updated code:
def rcmd_in(server):
import subprocess as sp
si = sp.STARTUPINFO()
si.dwFlags |= sp.STARTF_USESHOWWINDOW
for i in command_list:
result = sp.Popen(['C:/"Path to executable"/rcmd.exe', '\\\\' + server, i],
universal_newlines=True, stdin=sp.PIPE, stdout=sp.PIPE,
stderr=sp.PIPE, startupinfo=si)
print(result.stdout.read())
Note the change of functions 'run()' to 'Popen()'. The 'run()' function will not work with the print statement at the end. Also, for those of you who are curious the 'si' variable I created is preventing 'subprocess' from opening a console when being ran while using a GUI. I hope this will become useful to someone struggling with this. Cheers
Hi everyone i am trying to build a file watcher in python 3.5 using watchgod. I want to continuously watch a directory and if any file is added then i want to send a list of added files to another program which will perform a series of task. Following is my code in python :-
print("execution of main file begins !!!!")
import os
from watchgod import watch
#changes gives a set object when watch finds any kind of changes in directory
for changes in watch(r'C:\Users\Rajat.Malik\Desktop\Requests'):
fileStatus = [obj[0] for obj in list(changes) ] #converting set to list which gives file status as added, changed or modified
fileLocation = [obj[1] for obj in list(changes) ] #similarly getting list of location of files added
var2 = 0
for var1 in fileLocation:
if fileStatus[var2] == 1: #if file is added then passing all files to another code which will work on the list of files added
os.system('python split_thread_module.py '+var1) #now this code will start executing
var2 = var2 + 1
So the problem i am having is that while split_thread_module.py is executing the watcher is not watching the directory. Any file which is coming at time when split_thread_module.py is executing is not reflecting in changes. How can i watch the changes in directory and pass it to the other program on the fly even when the other program is executing. I am not a python programmer. Can anyone help me in this regard ?
Thanks in advance !!!!
Sorry for delayed, I'm the developer of watchgod. I've added a python-watchgod tag to your question which I'll watch (no pun intended) in future so I can answer such questions more quickly.
To answer your question, watchgod will not miss changes which occur in the filesystem while other code is running. They'll just be reported as changes next time watch iterates.
More generally the best approach would be to run the other code asynchronously so the main process can get back to watching the directory.
a few other hints for neater python
no need to call list(changes) in the comprehension
os.system is deprecated, better to use subprocess.run
since split_thread_module.py is also python, do you really need to run it in a separate process? Even if you do you might have more luck with python multiprocessing than starting a new process with the system's process initiation.
Overall you might try something like:
from concurrent.futures import ProcessPoolExecutor
from time import sleep
from watchgod import watch
def slow_job(status, location):
print(f'status: {status}, location: {location}, starting...')
sleep(10)
print(f'status: {status}, location: {location}, done')
with ProcessPoolExecutor() as executor:
for changes in watch('./tests'):
for status, location in changes:
executor.submit(slow_job, status, location)
I want to see what's happening with a specific operation in a python3 package I've been working on. I use pycallgraph and it looks great. But I can't figure out how to remove an entire tree of calls from the output.
I made a quick script make_call_graphs.py:
import doms.client.schedule as sched
from pycallgraph import PyCallGraph
from pycallgraph.output import GraphvizOutput
from pycallgraph import Config
from pycallgraph import GlobbingFilter
config = Config()
config.trace_filter = GlobbingFilter(exclude=[
'_find_and_load',
'_find_and_load.*', # Tried a few similar variations
'_handle_fromlist',
'_handle_fromlist.*',
])
with PyCallGraph(output=GraphvizOutput(output_file='schedule_hourly_call_graph.png'), config=config):
sched.hourly()
Before I started using the GlobbingFilter, _find_and_load was at the top of the tree outside of my doms library call stack. It seems that the filter only removes the top level block, but every subsequent call remains in the output. (See BEFORE and AFTER below)
Obviously I can read the result and copy every single call I don't want to see into the filter, but that is silly. What can I do to remove that whole chunk of stuff outside my doms box? Is there a RecursiveFilter or something I could use?
BEFORE:
AFTER:
The solution was much easier than I originally thought and right in front of me: the include kwarg given to the GlobbingFilter.
config.trace_filter = GlobbingFilter(include=['__main__', 'doms.*'])