Future Edit
As of twisted 15.5.0 and kivy 1.9.1, the kivy implementation works on python3. I also use AMP, which is not yet in the release but was merged 2 months ago and will be in the next twisted release.
Introduction
I'm developing an app using Kivy and plan to extend it using multi-processing and, more importantly Twisted. The app has a master component and a slave which should be controllable from various masters, including over a network (hence twisted).
Kivy has it's own twisted support code (in kivy/support.py called instal_twisted_reactor) which works in python2 but doesn't in python3 due to the unported status of the _threadedselect reactor.
I've hacked a port of twisted/internet/_threadedselect.py to python3 which is sufficient to run the kivy example code (one server GUI and one client GUI sending text). Normally I'd then go for a pull request upstream, but there's a few more factors in play:-
_threadedselect.py is, as best as I can tell, unmaintained in twisted, and in fact was last even mentioned on their mailing list back in 2006 (as a candidate for removal)
The twisted requirements for python3 porting include appropriate unit tests, and I don't have to knowledge (networking wise especially) to write appropriate unit tests for this reactor
I do not know whether any other project (besides kivy) even uses _threadedselect.py
Main Question
Given the above, is it advisable to put in the effort to get _treadedselect.py patched upstream (and henced ported to python3)? Or would it be more efficient to maintain a separate implementation in my own project (or optionally contributed to kivy's project).
Disclaimer: I'm (obviously) not a maintainer for either project
Non-essential details:
Kivy isn't the most popular UI base in python, hence it doesn't have a twisted maintained reactor (a la GTK, QT, TK etc.), but its the best fit for what I want to do. I've considered tornado etc. but that's really a matter for another question.
Also, the modifications needed are (so far) really minor. queue to Queue, zope interface decorators, except using as rather than comma, and using the function next rather than using a generator's next method.
Thank you for your interest in contributing to the Twisted ecosystem. I hope I can clear up your confusion so you can move forward.
First of all, Kivy is a very important Python library, and one we at the Twisted project definitely care about (I'm super happy to hear you're using them together!) so you are mistaken if you believe that Kivy doesn't have a reactor simply because it "isn't the most popular UI base in python"; the only reason it doesn't have a reactor in Twisted is that nobody contributed one.
There are a couple of possible ways forward, but they depend on some technical details you haven't covered in your question, and one SO answer is probably not enough to resolve the conflict; you will probably want to join the Twisted mailing list and discuss this there.
Regardless of the Python 3 port, _threadedselect is a private API, with no compatibility contract, and Kivy should not be using it. So in order to move forward, this will need to change. However, it is unlikely to be removed, as it is internally the basis of the wxpython reactor, which is a public API distributed with Twisted, and will eventually need to be ported to python 3 itself to complete the port.
The first way forward would be to improve _threadedselect's test coverage and make it a public API again. This strikes me as not necessarily the best move, because getting the thread interaction correct is very tricky and I would not want to encourage people - even people dealing with low-level enough implementation details to want to implement a reactor - to do it that way if any other options were available. However, if _threadedselect had a valid use-case within Kivy, and an advocate willing to do some maintenance work, this is definitely plausible.
The second way forward would be to create a reactor based on an internal socket-monitoring API within Kivy. Is there one? Most of the UIs which Kivy is a wrapper over (CoreFoundation, X11, Windows) do have socket-monitoring of some kind built in. Can Kivy tap into that, possibly? This is less error-prone than threading, if you can manage it, but sometimes it's impossible; I don't see any directly exposed APIs in Kivy for this. If you could develop such a thing it could be within Twisted or within Kivy depending on your preference.
I hope to hear from you in a more discussion-oriented forum soon :)
Related
How can I integrate a Python-Pyglet game with either Twisted or Asyncio?
I was going to ask a more general question, like "How can I add networking to a Python-Pyglet game?", but when doing any kind of web-search, "all roads lead to Twisted", anyway. However, both Pyglet and Twisted have their own event-loop, and it's not clear how they can be integrated. None of the other answers I've found, have more detail than "use Twisted", either.
Twisted does have a list of libraries it integrates with, including GUI ones like WxPython, but Pyglet is not on the list. I read one article that said Pyglet was going to be integrated with Twisted - from 2008, so I'm not holding my breath.
There is a pyglet-twisted library, but it's five years old on Github. There's no version information, but the sample-code has Python-2 print statements, so I'm guessing it's out-of-date. Using Pyglet and Twisted together
Same for Asyncio, which I also looked at - it has its own event-loop. Although Pyglet docs do have some information about doing a custom event-loop, I'm not sure how well it plays with others. I'd actually prefer using Asyncio, if it's feasible to get them to play nicely together.
You could get a quick win by basing code off of pigtwist and convert Py2 to Py3 syntax. Which doesn't look like a whole lot. There's also the crochet package that runs twisted in it's own thread, which you could run the network stack on. I also found this example, which uses a coiterator approach.
Hopefully pyglet will incorperate asyncio into itself given that it's a pure Python library. Maybe file a ticket with the team?
I'm writing a GUI application with PyQt4 (Python3). One my friend pointed out that using pyuic4 is a bad practice and referred me to uic module and Connecting Slots By Name features. He didn't have time to explain more and the references I have are rather short, I couldn't grasp the idea from them (uic module, LoadingUIFilesAtRuntime, connecting slots by name).
On StackOverflow there is at least one related question but the links to the literature are broken there.
I could follow standard tutorials and did simple GUI using pyuic, but now feel a little bit confused... Any good examples and/or references are welcome.
Firstly, using pyuic4 is certainly not "bad practice".
There are three main ways to get PyQt4 UI's into your code:
Write it all by hand yourself
Use pyuic4 to auto-generate a python module that can be imported
Use the uic package to load ui files directly at runtime
Of these, the first two are by far the most common, and most documentation, tutorials, advice, etc that you will come across will use those methods.
A good source for PyQt4 tutorials can be found in this section of the PyQt4 Wiki. However, I should probably point out that, although still relevant, many of them are quite old and so still use the old-style signals and slots.
However, the difference between the old- and new- styles is not that difficult to understand, so maybe a simple example is all that's needed.
Here's the old-style way to connect a button-click signal to a handler method (aka slot):
self.connect(self.button, QtCore.SIGNAL('clicked()'), self.handleButtonClick)
and here's the new-style way:
self.button.clicked(self.handleButtonClick)
As you can see, the new-style is much simpler and more pythonic. On the other hand, the old-style is quite similar to how signals are connected using C++ (and for this reason can still be useful in certain circumstances).
If you have problems with connecting signals when writing your GUIs, you can always ask a question here - but it's much easier to get good answers if you ask specific questions that include example code.
At http://nodejs.org/#about it says the following:
"Node is similar in design to and
influenced by systems like Ruby's
Event Machine or Python's Twisted.
Node takes the event model a bit
further—it presents the event loop as
a language construct instead of as a
library."
Are there other frameworks or languages that likewise expose the event loop?
Tcl has had this capability all along (last 2 decades). See http://www.tcl.tk/about/netserver.html
This is not to say Tcl does -- or should do -- everything Javascript and/or node.js does. But it is a valid answer to the question as to what "other frameworks or languages ... likewise expose the event loop?"
To be honest, I don't think there is something that's comparable to Node.js at the moment.
Why? Well basically because of the fact that JavaScript was single threaded from the start off, this made the language evolve in what it is today, a language that's perfectly fit for asynchronous programming, like you do it in Node.js.
Functions being first class objects, as well as having closures is a must if you want a Node like experience. For example you could just as well wrap plain C around the event lib and use that. But how much fun would that be? Even with twisted, you don't get even near the Node.js experience.
Second point is that Node.js has - except for the sync functions of the fs modules - no blocking functions, so while you can certainly do this style of programming in Python, you never know for sure if that library call isn't going to block your whole program. And kill the throughput of your server.
Also Node is fast, like in REALLY fast. V8 is definitely way ahead of Python and Ruby, yes you can write C-Extensions for both, but you can just as well do that for Node.js. Another plus point of using V8, Google is investing a ton of time/money into that Engine, another up to 2x improvement is already on the way with Crankshaft.
Node.js has more plus points, it's a complete framework (while Twisted is mainly async networking) and it's JavaScript.
The latter one may sound stupid, but the ability to re-use code and not having to do context switching , as well as being able to use mature frameworks for DOM manipulation (well, that is as soon as jsom gets into a more stable state) is another killer feature.
If you haven't done yet, I recommend you watch a couple of the talks listed on our Tag Wiki.
Especially the YUI one shows what possibilities await us in the near future.
So to sum it all up:
Although there are quite some frameworks out there that have an event loop, just having the loop itself won't give you the same experience as Node.js, so you should not expect a comparable experience when doing stuff in C or Java for example.
For the java platform I guess you could compare netty to node.js
I have been looking into MeeGo, maemo, Android architecture.
They all have Linux Kernel, build some libraries on it, then build middle layer libraries [e.g telephony, media etc...].
Suppose i wana build my own system, say Linux Kernel, with some binariers like glibc, Dbus,.... UI toolkit like GTK+ and its binaries.
I want to compile every project from source to customize my own linux system for desktop, netbook and handheld devices. [starting from netbook first :)]
How can i build my own customize system from kernel to UI.
I apologize in advance for a very long winded answer to what you thought would be a very simple question. Unfortunately, piecing together an entire operating system from many different bits in a coherent and unified manner is not exactly a trivial task. I'm currently working on my own Xen based distribution, I'll share my experience thus far (beyond Linux From Scratch):
1 - Decide on a scope and stick to it
If you have any hope of actually completing this project, you need write an explanation of what your new OS will be and do once its completed in a single paragraph. Print that out and tape it to your wall, directly in front of you. Read it, chant it, practice saying it backwards and whatever else may help you to keep it directly in front of any urge to succumb to feature creep.
2 - Decide on a package manager
This may be the single most important decision that you will make. You need to decide how you will maintain your operating system in regards to updates and new releases, even if you are the only subscriber. Anyone, including you who uses the new OS will surely find a need to install something that was not included in the base distribution. Even if you are pushing out an OS to power a kiosk, its critical for all deployments to keep themselves up to date in a sane and consistent manner.
I ended up going with apt-rpm because it offered the flexibility of the popular .rpm package format while leveraging apt's known sanity when it comes to dependencies. You may prefer using yum, apt with .deb packages, slackware style .tgz packages or your own format.
Decide on this quickly, because its going to dictate how you structure your build. Keep track of dependencies in each component so that its easy to roll packages later.
3 - Re-read your scope then configure your kernel
Avoid the kitchen sink syndrome when making a kernel. Look at what you want to accomplish and then decide what the kernel has to support. You will probably want full gadget support, compatibility with file systems from other popular operating systems, security hooks appropriate for people who do a lot of browsing, etc. You don't need to support crazy RAID configurations, advanced netfilter targets and minixfs, but wifi better work. You don't need 10GBE or infiniband support. Go through the kernel configuration carefully. If you can't justify including a module by its potential use, don't check it.
Avoid pulling in out of tree patches unless you absolutely need them. From time to time, people come up with new scheduling algorithms, experimental file systems, etc. It is very, very difficult to maintain a kernel that consumes from anything else but mainline.
There are exceptions, of course. If going out of tree is the only way to meet one of your goals stated in your scope. Just remain conscious of how much additional work you'll be making for yourself in the future.
4 - Re-read your scope then select your base userland
At the very minimum, you'll need a shell, the core utilities and an editor that works without an window manager. Paying attention to dependencies will tell you that you also need a C library and whatever else is needed to make the base commands work. As Eli answered, Linux From Scratch is a good resource to check. I also strongly suggest looking at the LSB (Linux standard base), this is a specification that lists common packages and components that are 'expected' to be included with any distribution. Don't follow the LSB as a standard, compare its suggestions against your scope. If the purpose of your OS does not necessitate inclusion of something and nothing you install will depend on it, don't include it.
5 - Re-read your scope and decide on a window system
Again, referring to the everything including the kitchen sink syndrome, try and resist the urge to just slap a stock install of KDE or GNOME on top of your base OS and call it done. Another common pitfall is to install a full blown version of either and work backwards by removing things that aren't needed. For the sake of sane dependencies, its really better to work on this from bottom up rather than top down.
Decide quickly on the UI toolkit that your distribution is going to favor and get it (with supporting libraries) in place. Define consistency in UIs quickly and stick to it. Nothing is more annoying than having 10 windows open that behave completely differently as far as controls go. When I see this, I diagnose the OS with multiple personality disorder and want to medicate its developer. There was just an uproar regarding Ubuntu moving window controls around, and they were doing it consistently .. the inconsistency was the behavior changing between versions. People get very upset if they can't immediately find a button or have to increase their mouse mileage.
6 - Re-read your scope and pick your applications
Avoid kitchen sink syndrome here as well. Choose your applications not only based on your scope and their popularity, but how easy they will be for you to maintain. Its very likely that you will be applying your own patches to them (even simple ones like messengers updating a blinking light on the toolbar).
Its important to keep every architecture that you want to support in mind as you select what you want to include. For instance, if Valgrind is your best friend, be aware that you won't be able to use it to debug issues on certain ARM platforms.
Pretend you are a company and will be an employee there. Does your company pass the Joel test? Consider a continuous integration system like Hudson, as well. It will save you lots of hair pulling as you progress.
As you begin unifying all of these components, you'll naturally be establishing your own SDK. Document it as you go, avoid breaking it on a whim (refer to your scope, always). Its perfectly acceptable to just let linux be linux, which turns your SDK more into formal guidelines than anything else.
In my case, I'm rather fortunate to be working on something that is designed strictly as a server OS. I don't have to deal with desktop caveats and I don't envy anyone who does.
7 - Additional suggestions
These are in random order, but noting them might save you some time:
Maintain patch sets to every line of upstream code that you modify, in numbered sequence. An example might be 00-make-bash-clairvoyant.patch, this allows you to maintain patches instead of entire forked repositories of upstream code. You'll thank yourself for this later.
If a component has a testing suite, make sure you add tests for anything that you introduce. Its easy to just say "great, it works!" and leave it at that, keep in mind that you'll likely be adding even more later, which may break what you added previously.
Use whatever version control system is in use by the authors when pulling in upstream code. This makes merging of new code much, much simpler and shaves hours off of re-basing your patches.
Even if you think upstream authors won't be interested in your changes, at least alert them to the fact that they exist. Coordination is essential, even if you simply learn that a feature you just put in is already in planning and will be implemented differently in the future.
You may be convinced that you will be the only person to ever use your OS. Design it as though millions will use it, you never know. This kind of thinking helps avoid kludges.
Don't pull upstream alpha code, no matter what the temptation may be. Red Hat tried that, it did not work out well. Stick to stable releases unless you are pulling in bug fixes. Major bug fixes usually result in upstream releases, so make sure you watch and coordinate.
Remember that it's supposed to be fun.
Finally, realize that rolling an entire from-scratch distribution is exponentially more complex than forking an existing distribution and simply adding whatever you feel that it lacks. You need to reward yourself often by booting your OS and actually using it productively. If you get too frustrated, consistently confused or find yourself putting off work on it, consider making a lightweight fork of Debian or Ubuntu. You can then go back and duplicate it entirely from scratch. Its no different than prototyping an application in a simpler / rapid language first before writing it for real in something more difficult. If you want to go this route (first), gNewSense offers utilities to fork your own OS directly from Ubuntu. Note, by default, their utilities will strip any non free bits (including binary kernel blobs) from the resulting distro.
I strongly suggest going the completely from scratch route (first) because the experience that you will gain is far greater than making yet another fork. However, its also important that you actually complete your project. Best is subjective, do what works for you.
Good luck on your project, see you on distrowatch.
Check out Linux From Scratch:
Linux From Scratch (LFS) is a project
that provides you with step-by-step
instructions for building your own
customized Linux system entirely from
source.
Use Gentoo Linux. It is a compile from source distribution, very customizable. I like it a lot.
So, I got this idea that I'd try to prototype an experimental user interface using OpenGL and some physics. I know little about either of the topics, but am pretty experienced with programming languages such as C++, Java and C#. After some initial research, I decided on using Python (with Eclipse/PyDev) and Qt, both new to me, and now have four different topics to learn more or less simultaneously.
I've gotten quite far with both OpenGL and Python, but while Python and its ecosystem initially seemed perfect for the task, I've now discovered some serious drawbacks. Bad API documentation and lacking code completion (due to dynamic typing), having to import every module I use in every other module gets tedious when having one class per module, having to select the correct module to run the program, and having to wait 30 seconds for the program to start and obscure the IDE before being notified of many obvious typos and other mistakes. It gets really annoying really fast. Quite frankly, i don't get what all the fuzz is about. Lambda functions, list comprehensions etc. are nice and all, but there's certainly more important things.
So, unless anyone can resolve at least some of these annoyances, Python is out. C++ is out as well, for obvious reasons, and C# is out, mainly for lack of portability. This leaves Java and JOGL as an attractive option, but I'm also curious about Ruby and Groovy. I'd like your opinion on these and others though, to keep my from making the same mistake again.
The requirements are:
Keeping the hell out of my way.
Good code completion. Complete method signatures, including data types and parameter names.
Good OpenGL support.
Qt support is preferable.
Object Oriented
Suitable for RAD, prototyping
Cross-platform
Preferably Open-Source, but at least free.
It seems you aren't mainly having a problem with Python itself, but instead with the IDE.
"Bad API documentation"
To what API? Python itself, Qt or some other library you are using?
"lacking code completion (due to dynamic typing)"
As long as you are not doing anything magic, I find that PyDev is pretty darn good at figuring these things out. If it gets lost, you can always typehint by doing:
assert isinstance(myObj, MyClass)
Then, PyDev will provide you with code completion even if myObj comes from a dynamic context.
"having to import every module I use in every other module gets tedious when having one class per module"
Install PyDev Extensions, it has auto-import on the fly. Or collect all your imports in a separate module and do:
from mymodulewithallimports import *
"having to select the correct module to run the program"
In Eclipse, you can set up a default startup file, or just check "use last run configuration". Then you never have to select it again.
"before being notified of many obvious typos and other mistakes"
Install PyDev Extensions, it has more advanced syntax checking and will happily notify you about unused imports/variables, uninitialized variables etc.
Looking just at your list I'd recommend C++; especially because Code Completion is so important to you.
About Python: Although I have few experience with OpenGL programming with Python (used C++ for that), the Python community offers a number of interesting modules for OpenGL development: pyopengl, pyglew, pygpu; just to name a few.
BTW, your import issue can be resolved easily by importing the modules in the __init__.py files of the directory the modules are contained in and then just importing the "parent" module. This is not recommended but nonetheless possible.
I don't understand why nobody has heard of the D programing language?
THIS IS THE PERFECT SOLUTION!!!!
The only real alternative if you desire all those things is to use Java, but honestly you're being a bit picky about features. Is code completion really that important a trait? Everything else you've listed is traditionally very well regarded with Python, so I don't see the issue.
The text editor (not even an IDE) which I use lets you import API function definitions. Code completion is not a language feature, especially with OpenGL. Just type gl[Ctrl+I] and you'd get the options.
I tried using Java3D and java once. I realized Java3D is a typical Java API... lots of objects to do simple things, and because it's Java, that translates to a lot of code. I then moved to Jython in Eclipse to which cleaned up the code, leaving me with only the complexity of Java3D.
So in the end, I went in the opposite direction. One advantage this has over pure python is I can use Java with all of Eclipse's benefits like autocomplete and move it over to python when parts get unwieldy in Java.
It seems like Pydev can offer code completion for you in Eclipse.
I started off doing OpenGL programming with GL4Java, which got migrated to JOGL and you should definately give it (JOGL) a try. Java offers most of the features you require (plus Eclipse gives you the code completion) and especially for JOGL there are a lot of tutorials out there to get you started.
Consider Boo -- it has many of Python's advantages while adopting features from elsewhere as well, and its compile-time type inference (when variables are neither explicitly given a specific type or explicitly duck typed) allows the kind of autocompletion support you're asking about.
The Tao.OpenGL library exposes OpenGL to .NET apps (such as those Boo compiles), with explicit support for Mono.
(Personally, I'm mostly a Python developer when not doing C or Java, but couldn't care less about autocompletion... but hey, it's your question; also, the one-class-per-module convention seems like a ridiculous amount of pain you're putting yourself through needlessly).