I have low-Ram laptop but I need to work on whole genome data which is more than 1Gb. For this end, I connect to a supercomputer.
In a windows machine, I run the codes in IDLE or Pyscripter and when there is errors, Its easily identifiable because all the variables up to the error point is available and accessible. for example if you have code like this:
genome_dict={}
with open ('genome.fa') as file:
chromosome= parse(file)
sequence= parse(file)
genome_dict[chromosome[z]]=sequence[n][m]
if there is error in parsing chromosome and sequence variable, their values are accessible in IDLE.But in a supercomputer linux machine, when an error occur I could not obtain the variables to find out what is the problem, I can not use print variable because its simply too big to be printable.
my question is, is there any way to run a python script in a linux command line in a way that you can obtain the variables generated in the process of running the script after its has been finished processing with or without errors?
Related
I have a small issue with my code, running in Python 3. I'm trying to fool Raspbian, in order to make it believe a tty is an external device.
However, I can't read a single word I wrote previously with os.write(slave, text.encode()), using something like os.read(slave, 512).
I open the tty as follow master, slave = os.openpty()
I think I'm missing a parameter or something, but I can't find out what.
I tried accessing the tty in another terminal, with a cat <, with a subprocess, but the program still block when it has to read.
Please explain what is the problem.
Regards.
I think your mistake here is that you are trying to read the slave. If you read the master instead you should get your output.
Quote from: http://www.rkoucha.fr/tech_corner/pty_pdip.html
A pseudo-terminal is a pair of character mode devices also called pty. One is master and the other is slave and they are connected with a bidirectional channel. Any data written on the slave side is forwarded to the output of the master side. Conversely, any data written on the master side is forwarded to the output of the slave side as depicted in figure 2.
RPI
I've been running a python script using:
python
Anacondaon
linux server (4.15.0-36-generic #39~16.04.1-Ubuntu)
For two weeks now and I think it will take more than a month to finish. This is already too long and so I would like to kill the process but before that I would like to save some lists that have been constructed during the last two weeks.
So far I tried:
1) using pyrasite-shell with one the PID associated with my python script (I've more than 20 PIDs associated with my script according to htop) but it does not work (no shell opens).
2) I also tried to find some kernel address using ps -aux | grep python3 and then connect to this kernel via something like ipython console --existing d3d55f0d-52f1-4660-a43b-1fedb698452f.json and then look for local variables with locals()in the prompt shell but this did not work (none of my lists appear in the list of local variables).
Ideally, I would like to connect the python kernel running my code and save all the current lists via pickle. Or any other idea to retrieve the lists would be great.
This question already has answers here:
Print to the same line and not a new line? [duplicate]
(19 answers)
Closed 4 years ago.
I need to clear a printed line, but so far I have found no good answers for using python 3.7, IDLE on windows 10. I am trying to make a simple code that prints a changing variable. But I don't want tons of new lines being printed. I want to try and get it all on one line.
Is it possible to print a variable that has been updated later on in the code?
Do remember I am doing this in IDLE, not kali or something like that.
Thanks for all your help in advance.
The Python language definition defines when bytes will be sent to a file, such as sys.stdout, the default file for print. It does not define what the connected device does with the bytes.
When running code from IDLE, sys.stdout is initially connected to IDLE's Shell window. Shell is not a terminal and does not interpret terminal control codes other than '\n'. The reasons are a) IDLE is aimed at program development, by programmers, rather than program running by users, and developers sometimes need to see all the output from a program; and b) IDLE is cross-platform, while terminal behaviors are various, depending on the system, settings, and current modes (such as insert versus overwrite).
However, I am planning to add an option to run code in an IDLE editor with sys.stdout directed to the local system terminal/console.
Let's take e.g. "top" application which displays system information and periodically updates it.
I want to run it using node.js and display that information (and updates!).
Code I've come up with:
#!/usr/bin/env node
var spawn = require('child_process').spawn;
var top = spawn('top', []);
top.stdout.on('readable', function () {
console.log("readable");
console.log('stdout: '+top.stdout.read());
});
It doesn't behave the way I expected. In fact it produces nothing:
readable
stdout: null
readable
stdout:
readable
stdout: null
And then exits (that is also unexpected).
top application is taken just as an example. Goal is to proxy those updates through the node and display them on the screen (so same way as running top directly from command line).
My initial goal was to write script to send file using scp. Done that and then noticed that I am missing progress information which scp itself displays. Looked around at scp node modules and they also do not proxy it. So backtracked to common application like top.
top is an interactive console program designed to be run against a live pseudo-terminal.
As to your stdout reads, top is seeing that its stdin is not a tty and exiting with an error, thus no output on stdout. You can see this happen in the shell if you do echo | top it will exit because stdin will not be a tty.
Even if it was actually running though, it's output data is going to contain control characters for manipulating a fixed-dimension console. (like "move the cursor to the beginning of line 2"). It is an interactive user interface and a poor choice as a programmatic data source. "Screen scraping" and interpreting this data and extracting meaningful information is going to be quite difficult and fragile. Have you considered a cleaner approach such as getting the data you need out of the /proc/meminfo file and other special files the kernel exposes for this purpose? Ultimately top is getting all this data from readily-available special files and system calls, so you should be able to tap into data sources that are convenient for programmatic access instead of trying to screen scrape top.
Now of course, top has analytics code to do averages and so forth that you may have to re-implement, so both screen-scraping and going through clean data sources have pros and cons, and aspects that are easy and difficult. But my $0.02 would be focus on good data sources instead of trying to screen scrape a console UI.
Other options/resources to consider:
The free command such as free -m
vmstat
and other commands described in this article
the expect program is designed to help automate console programs that expect a terminal
And just to be clear, yes it is certainly possible to run top as a child process, trick it into thinking there's a tty and all the associated environment settings, and get at the data it is writing. It's just extremely complicated and is analogous to trying to get the weather by taking a photo of the weather channel on a TV screen and running optical character recognition on it. Points for style, but there are easier ways. Look into the expect command if you need to research more about tricking console programs into running as subprocesses.
Currently, I am using shell command line calls from my fortran program using non-standard SYSTEM intrinsic routine (similar to Fortran 2008 EXECUTE_COMMAND_LINE intrinsic):
CALL SYSTEM(commandStr)
where commandStr is a character string containing the shell command I want to execute. At the moment, I am not aware of a direct way to return the output of commandStr, but only its return status. So, what I am doing now is writing output into a file, and then reading the file from within the Fortran program. Example:
CALL SYSTEM('sed ''s/,//g'' myFile > dummyFile')
if I want to remove commas from myFile. I then use OPEN and READ to get contents of dummyFile.
This works just fine, however I am concerned about writing/reading files from disk, especially if I was doing this within a long loop, and if commandStr output was big. Is there a way to re-direct commandStr output into a memory buffer (not hard disk) which I could access from my Fortran program directly (maybe through a specific UNIT number)?
If this is in a POSIX environment, the library function popen() might also be available.
iunit = popen ('sed ''s/,//g'' myFile', 'r')
Look at the documentation for your Fortran environment since I'm not sure of the semantics for connecting the i/o to Fortran. If it is like the C runtime library, the file connection also needs a special function to close it, pclose().