Note: I know how to cythonize my own Python modules.
So, I have managed to cythonize some of my Python modules, and I've gained some speed. The question is, since cython:
The goal of cython is to be compatible with python, i.e. that you can cythonize any python code and it will work as before. Currently, a large fraction of python code already works. Cython furthermore allows you to optimize parts of your code and compile it into more efficient C code.
Source: What does Cython do with imports?
Now, the biggest issue in my code evolve around re.py, the built-in Python Regular Expression module, despite the fact, that I am per-compiling my regexes.
Related
Today I knew that pypy3 is faster than python3 for input() time through any algorithm problem. Performance differences were almost as much as 12 times.
Why is there such a difference?
Kindly check this, when we speak of Python programming language we often mean not just the language but also the implementation. Python is a specification for a language that can be implemented in many different ways.
The default implementation of the Python programming language is Cpython(assuming python3 you mean Cpython). As the name suggests Cpython is written in C language. Cpython compiles the python source code into intermediate bytecode, which is executed by the Cpython virtual machine.
Jython is an implementation of the Python programming language that can run on the Java platform. Jython programs use Java classes instead of Python modules. Jython compiles into Java byte code, which can then be run by Java virtual machine.
PyPy
If you want your code to run faster, you should probably just use PyPy. — Guido van Rossum (creator of Python)
Python is a dynamic programming language. Python is said to be slow as the default CPython implementation compiles the python source code in bytecode which is slow as compared to machine code(native code). Here PyPy comes in.
PyPy is an implementation of the Python programming language written in Python. The Interpreter is written in RPython (a subset of Python).
PyPy uses Just In Time (JIT) compilation. In simple terms, JIT uses compilation methods to make the interpreter system more efficient and fast. So basically JIT makes it possible to compile the source code into native machine code which makes it very fast.
PyPy also comes with default support for stackless mode, providing micro-threads for massive concurrency. It is said to be approximately 7.5 times faster than Cpython.
Hope this will help you.
CPython
It is interpreter for python language written in C and C++. Interpreter convert python code (which is written by humans and can be read by humans) to machine code (which can be read/understood by machines). This process involve various steps.
CPython is the reference implementation of Python, written in C. It compiles Python code to intermediate bytecode which is then interpreted by a virtual machine. CPython provides the highest level of compatibility with Python packages and C extension modules.
If you are writing open source Python code and want to reach the widest possible audience, targeting CPython is best. To use packages which rely on C extensions to function, CPython is your only implementation option.
All versions of the Python language are implemented in C because CPython is the reference implementation.
PyPy
It is JIT compiler for python language written in RPython. JIT compiler execute code which require compilation, i.e. JIT compile code at runtime, just before executing it.
PyPy is a Python interpreter implemented in a restricted statically-typed subset of the Python language called RPython. The interpreter features a just-in-time compiler and supports multiple back-ends (C, CLI, JVM).
PyPy aims for maximum compatibility with the reference CPython implementation while improving performance.
If you are looking to increase performance of your Python code, it’s worth giving PyPy a try. On a suite of benchmarks, it’s currently over 5 times faster than CPython.
PyPy supports Python 2.7. PyPy3, released in beta, targets Python 3.
I assume that when you say python3, you mean CPython which default and widely used implementation of python language.
CPython
It is interpreter for python language written in C and C++. Interpreter convert python code(which is written by human and can read by human) to machine code(which can read/understand by machine/computer). This process involve various steps.
PyPy
It is JIT compiler for python language written in RPython. JIT compiler execute code which require compilation, i.e. JIT compile code at runtime, just before executing it.
This different approach of handling python code of these two implementation is reason behind different int speed. Below link will give you more detail on this.
There are few more implementation of Python language which aims to achieve different goal.
CPython
PyPy
JIT
Design of CPython’s Compiler
PyPy
Alternatives
In a python-based project, I would like to use features data classes and libraries such as the current RxPY version.
However, we are bound to an environment (a Yocto-system) that only has Python 3.5.5
From TypeScript/JavaScript I know that some languages allow code to be transpiled to a lower language version.
Can Python 3.7 code be transpiled to a lower version?
Many thanks
You can't just use a tool to convert Python code to earlier versions, because those versions would sometime lack the libraries that you need. Using Python 3.7's dataclass is a good example.
A workaround for you would be to just install a port of dataclasses back in your 3.5 environment. This would be tricky. If you had python 3.6 you could just install this through
pip install dataclasses
But you can see in this issue that they never ported it back for python 3.5. If you check out their GitHub, maybe there are manual changes you could do to the source code to make it work. Maybe.
The easier way would be to find an alternative to dataclasses
I'm doing an extraction from Coq to Haskell that requires importing a couple of modules on the Haskell end. Is there any Coq extraction feature that allows you to do this automatically? I know I could just write a script to do this but I'd prefer to not reinvent the wheel if necessary.
There isn't, and it's very sad.
We've solved this problem with a Python script that adds several imports (fiximports.py), but this requires using the Haskell preprocessor (you use it by passing -F -pgmF fiximports.py to ghc) and results in unused imports warnings, and isn't terribly elegant.
I think the issue is that these imports are unnecessary for OCaml extraction, and the feature hasn't been designed and implemented for Haskell extraction.
I have several apps I'm developing that are for end users that have no idea how to use Python. I have already discovered how to setup a package that allows them to run any script without Python knowledge but I don't know how to minimize the distribution size by only including subsets (I.e. the actual function calls in large libs like NumPy) of each imported library that are required. Is there a way to output the actual subcomponents of each imported library that are actually accessed during the function? All my internet searches end up with cyclical imports which is not what I need. There must be some Python dependency walker equivalent I have yet to discover. Much appreciated any libs that can outline this.
[UPDATE]
I converted Snakefood 1.4 over to Python 3x (3.5 tested to build) with python setup.py install and saved it here: https://github.com/mrslezak/snakefood per the accepted answer.
Use Snakefood
Here's a command
sfood -i -r myscript.py | sfood-cluster > dependencies.txt
I really admire the functionality of Stackless Python, and I've been looking around for a way to emulate its syntax while still using the standard Python 3 interpreter. An article by Alex J. Champandard in a gamedev blog made it look as though the greenlet library could provide this functionality. I slightly modified his code, but the best makeshift tasklet wrapper I could come up with was a class holding a greenlet inside a variable, as such:
class tasklet():
def __init__(self,function=None,*variables):
global _scheduled
self.greenlet = greenlet.greenlet(function,None)
self.functioncall = function # Redundant backup
self.variables = variables
_scheduled.append(self)
self.blocked = False
The function then emulates Stackless' scheduling by passing the variables to the greenlet when calling its switch() method.
So far this appears to work, but I'd like to be able to call the tasklets in original Stackless syntax, e.g. tasklet(function)(*args), as opposed to the current syntax of tasklet(function,*args). I'm not sure where to look in the documentation to find out how to accomplish this. Is this even possible, or is it part of Stackless' changes to the interpreter?
According to this article from 2010-01-08 (with fixed links):
Stackless Python is an extended version of the Python language (and
its CPython reference implementation). New features include
lightweight coroutines (called tasklets), communication primitives
using message passing (called channels), manual and/or automatic
coroutine scheduling, not using the C stack Python function calls, and
serialization of coroutines (for reloading in another process).
Stackless Python could not be implemented as a Python extension module
– the core of the CPython compiler and interpreter had to be patched.
greenlet is an extension module to CPython providing coroutines and
low-level (explicit) scheduling. The most important advantage of
greenlet over Stackless Python is that greenlet could be implemented
as a Python extension module, so the whole Python interpreter doesn't
have to be recompiled in order to use greenlet. Disadvantages of
greenlet include speed (Stackless Python can be 10%, 35% or 900%
faster, depending on the workflow); possible memory leaks if
coroutines have references to each other; and that the provided
functionality is low-level (i.e. only manual coroutine scheduling, no
message passing provided).
greenstackless, the Python module I've recently developed, provides
most of the (high-level) Stackless Python API using greenlet, so it
eliminates the disadvantage of greenlet that it is low-level. See the
source code and some tests (the latter with tricky corner cases).
Please note that although greenstackless is optimized a bit, it can be
much slower than Stackless Python, and it also doesn't fix the memory
leaks. Using greenstackless is thus not recommended in production
environments; but it can be used as a temporary, drop-in replacement
for Stackless Python if replacing the Python interpreter is not
feasible.
Some other software that emulates Stackless using greenlet:
Concurrence: doesn't support stackless.main, tasklet.next,
tasklet.prev, tasklet.insert, tasklet.remove,
stackless.schedule_remove, doesn't send exceptions properly. (Because
of these features missing, it doesn't pass the unit test above.)
PyPy: doesn't support stackless.main, tasklet.next, tasklet.prev,
doesn't pass the unit test above.