I first noticed this problem a bit ago when I posted this thread. Essentially, I'm getting a very strange issue where Python "sees" files that don't actually exist in my file browser when working with files.
As in, if I open a file with an absolute path, modify and print the contents of it through Python, it will print exactly what it's supposed to. But when I try to open the same file through the same absolute path on my Windows file browser, the file is not updated with my modifications.
I experienced a similar issue recently as well - renaming files. Here's a simple script I wrote to replace all spaces in filenames with underscores.
rename.py:
import os
i = 0
path = "E:/sample_path/"
for filename in os.listdir(path):
src = path + filename
dst = path + filename.replace(" ", "_")
os.rename(src, dst)
i += 1
print(str(i) + " files processed.")
Upon doing some prints in Python, I can see that all the files in the directory are being renamed correctly but it just wasn't correctly updating when I actually viewed the directory. Both in the file browser and in using the dir command. Same with creating new files in Python, they exist in the eyes of Python, but they are nowhere to be found in Windows, even with hidden files being visible and all.
Now for the interesting part: This script works if I open the python editor in cmd and import it. So I know it's all correct, no syntax errors or anything - it's just a strange error occurring with Python.
For example, if I go into command prompt and type python rename.py it won't return any errors and will even output the correct results - x files processed. but it will not actually modify any "real" files.
But if I go into command prompt and type python to bring up the cmd editor, then type import rename it gives the correct output and updates all the files correctly. So for the time being this workaround helps, but it's a very strange issue and I have yet to see anyone else encounter it. It's almost like Python creates a temporary duplicate of the filesystem and is not completing it's sync back to Windows.
I've tried reinstalling Python several times, both x64 and x86 verisons and nothing has fixed it so far.
EDIT: Here's a simplified example illustrating my issue.
write.py:
f = open("newfile.txt", "w+")
f.write("hello new file")
f.close()
read.py:
f = open("newfile.txt", "r")
for l in f.readlines():
print(l)
If I run write.py, no errors are returned. But also no file named newfile.txt is visible in my current working directory. However, if I run read.py it prints hello new file. So clearly both of these are accessing the same invisible file somewhere.
I'm wondering if it's possible to populate sys.argv (or some other structure) with command line arguments in a jupyter/ipython notebook, similar to how it's done through a python script.
For instance, if I were to run a python script as follows:
python test.py False
Then sys.argv would contain the argument False. But if I run a jupyter notebook in a similar manner:
jupyter notebook test.ipynb False
Then the command line argument gets lost. Is there any way to access this argument from within the notebook itself?
After a lot of looking around I found very cumbersome, custom libraries, but solved it with a few lines of code which I thought was pretty slick. I used nbconvert to end up with an html report as output that contains all graphics and markdown from the notebook, but accepts command line parameters just as always through a minimal python wrapper:
The python file test_args.py (which takes command line params as normal):
import sys,os
IPYNB_FILENAME = 'test_argv.ipynb'
CONFIG_FILENAME = '.config_ipynb'
def main(argv):
with open(CONFIG_FILENAME,'w') as f:
f.write(' '.join(argv))
os.system('jupyter nbconvert --execute {:s} --to html'.format(IPYNB_FILENAME))
return None
if __name__ == '__main__':
main(sys.argv)
The notebook contains:
import sys,os,argparse
from IPython.display import HTML
CONFIG_FILE = '.config_ipynb'
if os.path.isfile(CONFIG_FILE):
with open(CONFIG_FILE) as f:
sys.argv = f.read().split()
else:
sys.argv = ['test_args.py', 'input_file', '--int_param', '12']
parser = argparse.ArgumentParser()
parser.add_argument("input_file",help="Input image, directory, or npy.")
parser.add_argument("--int_param", type=int, default=4, help="an optional integer parameter.")
args = parser.parse_args()
p = args.int_param
print(args.input_file,p)
and I can run the python notebook with arguments parsed as usual:
python test_args.py my_input_file --int_param 12
I tend to paste the block with argparse calls into the python wrapper so that command line errors are caught by the python script and -h works properly.
There are two projects I've found that do what you ask for
Papermill, will add a cell to your notebook with arguments that you pass to it on the command line. So this is quite straightforward, you define your defaults in the first cell (the should have parameters tag)
nbparameterise it is a similar concept but you don't tag your cell with defaults, it has to be first.
Here is a good resource discussing the issue: https://github.com/jupyter/help/issues/218
If the goal is to run a notebook with configurable arguments passed from commandline, I think the easiest way is to use environment variables, like this:
NB_ARGS=some_args jupyter nbconvert --execute --to html --template full some_notebook.ipynb
Then in the notebook, you can import os and use os.environ['NB_ARGS']. The variable value can be some text that contains key-value pairs or json for example.
At the top of the Jupyter cell, put a line like:
%%python - --option1 value1 --option2 value2 --etc
In your example:
%%python - True
This will run your script like in a command line with the args provided.
Example:
%%python - --option1 value1 --option2 value2 --etc
import sys
if __name__ == '__main__':
print(sys.argv)
will output:
['-', '--option1', 'value1', '--option2', 'value2', '--etc']
Hope it helps.
I think this Gist may help you : https://gist.github.com/gbishop/acf40b86a9bca2d571fa
This is an attempt at a simple argument parser for mostly key=value pairs that can be used both on the command line and in IPython notebooks. It support query parameters in notebook URLs and a Run command for notebooks.
Using args = parser.parse_args(args=[]) would work.
or for testing, you can declare it as class format.
class Args:
data = './data/penn'
model = 'LSTM'
emsize = 200
nhid = 200
args=Args()
sys.argv yields a list, so I used
sys.argv.append('hello')
in a jupyter notebook, which allowed me to append extra members and pretend like I'm passing in arguments from the command line.
I tried out the answers listed above, and came up with a different solution.
My original code was
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required=True, help="path to input image")
ap.add_argument("-y", "--yolo", required=True, help="base path to YOLO directory")
ap.add_argument("-c", "--confidence", type=float, default=0.5, help="minimum probability to filter weak detections")
ap.add_argument("-t", "--threshold", type=float, default=0.3, help="threshold when applying non-maxima suppression")
args = vars(ap.parse_args())
I tried to make a Class as
Class Args():
image='photo.jpg'
yolo='yolo-coco'
confidence=0.5
threshold=0.3
args=Args()
but futher code snippets were producing an error.
So I printed args after vars(ap.parse_args()) and found that it was a dictionary.
So just create a dictionary for the original args:
args={"image": 'photo.jpg', "yolo": 'yolo-coco', "confidence": 0.5,"threshold": 0.3}
A workaround is to make the jupyter notebook read the arguments from a file.
From the command line, modify the file and run the notebook.
I assume that you just want to parse some arguments to the notebooks, but it's not necessary to use the command line.
If you want to parse commands like.
python script.py --a A --b B
You can use the following code in the notebook:
cmd = '--a A --b B'
args = args = parser.parse_args(cmd)
For parse_args, you can find more information here.
A simple and naïve solution is to put the following snippet at the first line of your program:
import sys
sys.argv = "your expected command line arguments here".split()
After executing this command, packages like argparse will work well.
So, you can just run your scripts in the jupyter lab server, without opening a terminal and typing your arguments.
I am developing a all in one image converter tool using bash commands. So I want to know how to export a krita, ".kra" document with many layers, to a "png" image using command line ?
There is a command line converter for Krita. However from version 2.9 it was removed in favor of calling krita binary with arguments.
If krita version is 2.8 or less you should use calligraconverter
calligraconverter --batch -- input output
In 2.9 you would call krita like this
krita input --export --export-filename output
By default both commands output format is based on the output extension name. For more information run calligraconverter (2.8) OR krita(2.9) commands "--help".
Not a ready command-line utility solution as you wanted but still.
Found following python script from David Revoy's blog
Try to adapt it to your needs.
import sys, zipfile
import Image
import StringIO
if len(sys.argv) != 4:
sys.exit('Usage: '+sys.argv[0]+' <Input> <Output> <Size>')
thumbnail = zipfile.ZipFile(sys.argv[1]).read('preview.png')
im = Image.open(StringIO.StringIO(thumbnail))
im.thumbnail( (int(sys.argv[3]), int(sys.argv[3])) )
im.save(sys.argv[2],'png')
I've been trying to run these scripts https://github.com/balsamiq/mockups-strings-extractor within XP. I'm getting errors per this screenshot https://www.dropbox.com/s/rlbqp1iytkwvq3m/Screenshot%202014-05-30%2011.57.48.png
Also I tried CD into my test directory and although a text output file is generated it is empty and I still get these errors https://www.dropbox.com/s/odjfbr97e5i4gnn/Screenshot%202014-05-30%2012.09.31.png
Is anybody running Balsamiq on windows able to make this work ?
1) From the looks of the first error pic you included, you were trying to execute a Windows Shell command inside of a Python Interpreter. If you've still got the window open, type quit() before attempting your command again.
2) Your script was written for Python 2.x. If you're using Python 3.x, you'll need to add parentheses to the print lines in the script file and change urllib to urllib.parse. I made the changes below:
import os
import glob
import re
import urllib.parse
for infile in glob.glob( os.path.join(".", '*.bmml') ):
print("STRINGS FOUND IN " + infile)
print("===========================================================")
f = open (infile,"r")
data = f.read()
p = re.compile("<text>(.*?)</text>")
textStrings = p.findall(data)
print(urllib.parse.unquote('\n'.join(textStrings))+"\n")
f.close()
Hope this helps.
I would like to easily test my python programs without constantly using the python shell since each time the program is modified you have to quit, re-enter the python shell and import the program again. I am using a 2012 Macbook pro with OSX. I have the following code:
import sys
def read_strings(filename):
with open(filename) as file:
return file.read().split('>')[1:0]
file1 = sys.argv[1]
filename = read_strings(file1)
Essentially I would like to read into and split a txt file containing:
id1>id2>id3>id4
I am entering this into my command line:
pal-nat184-102-127:python_stuff ceb$ python3 program.py string.txt
However when I try the sys.argv approach on the command line my program returns nothing. Is this a good approach to testing code, could anyone point me in the correct direction?
This is what I would like to happen:
pal-nat184-102-127:python_stuff ceb$ python3 program.py string.txt
['id1', 'id2', 'id3', 'id4']
Let's take this a piece at a time:
However when I try the sys.argv approach on the command line my
program returns nothing
The final result of your program is that it writes a string into the variable filename. It's a little strange to have a program "return" a value. Generally, you want a program to print it's something out or save something to a file. I'm guessing it would ease your debugging if you modified your program by adding,
print (filename)
at the end: you'd be able to see the result of your program.
could anyone point me in the correct direction?
One other debugging note: It can be useful to write your .py files so that they can be run both independently at the command line or in a python shell. How you've currently structured your code, this will work semi-poorly. (Starting a shell and then importing your file will cause an error because sys.argv[1] isn't defined.)
A solution to this is to change your the bottom section of your code as follows:
if __name__ == '__main__':
file1 = sys.argv[1]
filename = read_strings(file1)
The if guard at the top says, "If running as a standalone script, then run what's below me. If you imported me from some place else, then do not execute what's below me."
Feel free to follow up below if I misinterpreted your question.
You never do anything with the result of read_strings. Try:
print(read_strings(file1))