I am writing unit tests (using Python 3.7, Pytest 3.6 and tox 3.0) for a function that compiles a series of shell commands as a list of strings, and then executes them using the subprocess module. The shell commands do the following:
creates a filename from the function arguments.
cd into a given directory.
checks for a file with a certain name and removes it if it exist.
creates a new file with the same name.
executes a program and pipes the output into the new file.
My test right now is mocking the subprocess module, and then asserting that it was called with a list of strings that contains all the expected commands given some test arguments.
Is there a way to test that the commands do what they are supposed to? Right now my test is only checking if the list of commands I feed to the subprocess module is the same as the one I have asserted it to be. This does not tell me whether the commands are the right ones for what I am trying to achieve. Rather it only serves as a test on whether I can write down the same string in two different source files.
Can I simulate the side effects that I expect the shell commands to have?
Your question combines two interesting topics: a) Testing code generated from a code generator and b) testing shell code.
For testing code from a generator, you have in principle to do the following: i) test that the generator creates the code that is expected - which you have already done, ii) test that the code snippets / pieces which the generator glues together actually behave (independently and in combination) as it is intended (which in your case are the shell code pieces that in the end together will form a valid shell program) - this is the part about testing shell code that will be adressed below, and iii) test that the inputs that control the generator are correct.
It is comparable with a compiler: i) is the compiler code, ii) are assembly code snippets that the compiler combines to get the resulting assembly program, and iii) is the source code that is given to the compiler to get it compiled. Once i), ii) and iii) are tested, there is only seldom the need to also test the assembly code (that is, on assembly code level). In particular, the source code iii) is ideally tested by test frameworks in the same programming language.
In your case it is not so clear how part iii) looks and how it can be tested, though.
Regarding the testing of shell code / shell code snippets: Shell code is dominated by interactions with other executables or the operating system. The type of problems that lies in interactions in shell code goes in the direction of, am I calling the right executables in the right order with the arguments in the right order with properly formatted argument values, and are the outputs in the form I expect them to be etc. To test all this, you should not apply unit-testing, but integration testing instead.
In your case this means that the shell code snippets should be integration-tested on the different target operating systems (that is, not isolated from the operating system). And, the various ways in which the generator puts these snippets together should also be integration tested to see if they together operate nicely on the operating system.
However, there can be shell code that is suitable for unit-testing. This is, for example, code performing computations within the shell, or string manipulations. I would even consider shell code with calls to certain fundamental tools like basename as suitable for unit-testing (interpreting such tools as being part of the 'standard library' if you like). In your case as you describe the generated shell code
creates a filename from the function arguments.
This sounds like one example of a good candidate for 'unit-testing' shell code: This filename creation functionality could be put into a shell function and then be tested in isolation.
With pytest-mock you can request the mocker fixture and then spy on subprocess functions:
def test_xxx(mocker):
mocker.spy(subprocess, 'call')
subprocess.call(...)
assert subprocess.call.call_count == 1
P.S. Tests with side effects are generally bad practice, so I would recommend running all the shell commands in a tmpdir (pytest fixture which creates a temporary directory).
Related
I am running a computer programming course that uses Sage and Python, however, I am not an expert programmer myself in Python. (FYI Sage is an alternative kernel to Python 3, which has a number of efficient packages for abstract algebra compiled in). I often find myself wanting to assign the students to program some task which Sage can already do built-in to the kernel, and I need to make sure they are not simply calling the built-in functionality. I am using a program called nbgrader which runs their code and asserts various facts about it to test that it is computing correctly, and can generally run any python command imaginable on the function, it then tests a series of assert commands and if they all pass then they get full credit for the problem.
I have tried to follow tutorials which have me delete the built-in function, or use unit test and mocking to otherwise subvert it. However, I get errors like:
can't set attributes of built-in/extension type 'sage.structure.element.Matrix when I try just to delete the built-in matrix multiplication functionality. Similar errors when I try to mock it. Is there some clever way for me to either delete or otherwise subvert a built-in function so that it throws an error if they attempt to call it? Or failing that, can I get python to print to a string a log of all function calls, e.g. so that sage.structure.element.Matrix.__mult__ would show up in this string if their function when run on some example uses the built-in matrix multiplication functionality?
I'm using Antlr4 to create an interpreter, lexer and parser. The GUI it will be used in contains QScintilla2.
As QScintilla does not need a parser and has a CustomLexer module will the (Antlr4 built, Python3 target) interpreter be enough?
I'm not asking for opinions but factual guidance. Thanks.
What does an Interpreter contain
An interpreter must have some way to parse the code and then some way to run it. Usually the "way to parse the code" would be handled by a lexer+parser, but lexerless parsing is also possible. Either way, the parser will create some intermediate representation of the code such as a tree or bytecode. The "way to run it" will then be a phase that iterates over the generated tree or bytecode and executes it. JIT-compilation (i.e. generating machine code from the tree or bytecode and then executing that) is also possible, but more advanced. You can also run various analyses between parsing and execution (for example you can check whether any undefined variables or used anywhere or you could do static type checking - though the latter is uncommon in interpreted languages).
When using ANTLR, ANTLR will generate a lexer and parser for you, the latter of which will produce a parse tree as a result, which you can iterate over using the generated listener or visitor. At that point you proceed as you see fit with your own code. For example, you could generate bytecode from the parse tree and execute that, translate the parse tree to a simplified tree and execute that or execute the parse tree directly in a visitor.
QScintilla is about displaying the language and is not linked to the interpreter. In an IDE the console is where the interpreter comes into play along with running the script (from a 'Run' button for example). The only thing which is common to QScintilla and the interpreter is the script file - the interpreter is not connected or linked to QScintilla. Does this make basic sense?
Yes, that makes sense, but it doesn't have to be entirely like that. That is, it can make sense to reuse certain parts of your interpreter to implement certain features in your editor/IDE, but you don't have to.
You've specifically mentioned the "Run" button and as far as that is concerned, the implementation of the interpreter (and whether or not it uses ANTLR) is of absolutely no concern. In fact it doesn't even matter which language the interpreter is written in. If your interpreter is named mylangi and you're currently editing a file named foo.mylang, then hitting the "Run" button should simply execute subprocess.run(["mylangi", "foo.mylang"]) and display the result in some kind of tab or window.
Same if you want to have a "console" or "REPL" window where you can interact with the interpreter: You simply invoke the interpreter as a subprocess and connect it to the tab or subwindow that displays the console. Again the implementation of the interpreter is irrelevant for this - you treat it like any other command line application.
Now other features that IDEs and code editors have are syntax highlighting, auto-completion and error highlighting.
For syntax highlighting you need some code that goes through the source and tells the editor which parts of the code should have which color (or boldness etc.). Using QScintilla, you accomplish this by giving a lexer class that does this. You can define such a class, by simply writing the necessary code to detect the types of tokens by hand, but you can also re-use the lexer generated by ANTLR. So that's one way in which the implementation of your interpreter could be re-used in the editor/IDE. However since a syntax highlighter is usually fairly straight forward to write by hand, you don't have to do it this way.
For code completion you need to understand which variables and functions are defined in the file, what their scope is, and which other files are included in the current file. These days it's becoming common to implement this logic in a so-called language-server that is separate tool that can be re-used from different editors and IDEs. Regardless of whether you implement this logic in such a language server or directly in your editor, you'll need a parser (and, if applicable, a type checker) to be able to answer these types of question. Again that's something that you can re-use from your interpreter and this time that's definitely a good idea because writing a second parser would be significant additional work (and easy to get out of sync with the interpreter's parser).
For error highlighting you can simply invoke the interpreter in "verify only" mode (i.e. only print out syntax errors and other errors that can be detected statically, but don't actually run the file -- many interpreters have such an option) and then parse the output to find out where to draw the squiggly lines. But you can also re-use the parser (and analyses if you have any) from your interpreter instead. If you go the route of having a language server, errors and warnings would also be handled by the language server.
I use FORTRAN gnu compiler to compile a piece of code written using fortran(.f90). Unlike in other compilers the output of write statement are not displayed in the screen rather written in the output file.
For example I have placed "write(*,*) 'Check it here'" in the middle of the source code so that this message is displayed in the screen when someone runs the compiled version of the code.
I dont understand why this message is not displayed in the terminal window while running the code, but it is written in the output file.
I would appreciate your help to resolve this !!
>
I am compiling these source codes:
https://github.com/firemodels/fds/tree/master/Source
makefile that I am using to compile the code is located here:
https://github.com/firemodels/fds/tree/master/Build/mpi_intel_linux_64
I run the program using a executable that makefile creates
The version of the compiler that I am using is
GNU Fortran (Ubuntu 5.4.0-6ubuntu1~16.04.5) 5.4.0 20160609
>
Thank you.
Way bigger picture: Is there a reason you're building FDS from source rather than downloading binaries directly from NIST i.e. from https://pages.nist.gov/fds-smv/downloads.html ?
Granted, if you're qualifying the code for safety-related use, you may need to compile from source rather than use someone else's binaries. You may need to add specific info to a header page such as code version, date of run, etc. to satisfy QA requirements.
If you're just learning about FDS (practicing fire analysis, learning about CFD, evaluating the code), I'd strongly suggest using NIST's binaries. If you need/want to compile it from source, we'll need more info to diagnose the problem.
That said, operating on the assumption that you have a use case that requires that you build the code, your specific problem seems to be that writing to the default output unit * isn't putting the output where you expect.
Modern Fortran provides the iso_fortran_env module which formalizes a lot of the obscure trivia of Fortran, in this case, default input and output units.
In the module you're editing, look for something like:
use iso_fortran_env
or
use iso_fortran_env, only: output_unit
or
use, intrinsic:: iso_fortran_env, only: STDOUT => output_unit
If you see an import of output_unit or (as in the last case) an alias to it, write to that unit instead of to *.
If you don't an import from iso_fortran_env, add the last line above to the routine or module you're printing from and write to STDOUT instead of *.
That may or may not fix things, depending on if the FDS authors do something strange to redirect IO. They might; I'm not sure how writing to screen works in an MPI environment where the code may run in parallel on a number of networked machines (I'd write to a networked logging system in that case, but that's just me). But in a simple case of a single instance of the code running, writing to output_unit is more precise than writing to * and more portable and legible than writing to 6.
Good luck with FDS; I tried using it briefly to model layer formation from a plume of hydrogen gas in air. FDS brought my poor 8 CPU machine to its knees so I went back to estimating it by hand instead of trying to make CFD work...
When you pass "--help" as an argument to a program in command line, generally the program provides standard output designed to help the user. Where is the source of this standard output? Is it within its own file or is it embedded within the object code in a series of print statements?
It's usually within the program itself -- a series of print statements as you mentioned.
In my VTS scripts almost all the SEND, EXPECT commands have similar parameters(Like DEST ADDRESS, DEST NETWORK etc). So is there a way to avoid duplication using a function or macro. I had not seen any functions/macros in the example scripts of VTS.
Unfortunately, the VTS script is limited in its possibilities. Something like function macros do not exist. But you can use rather than hard-coded values, parameters such as IUT_ADDR for an address.
Alternatively, you generate the VTS scripts through an additional tool. A Python script is conceivable that functions as a kind of preprocessor and create the VTS script in prebuild step.