Automate Linux gui apps - linux

I'm looking to figure out if it's possible to automate linux gui apps, either on wayland or x.org.
I think that triggering mouse or keyboard inputs should be possible, but I'm not sure how to figure out where the button or text box I want to click on is located on the screen. In Windows we have the api which allows you to trigger events based on those xml selectors. Does anyone know if there is any similar way to find gui elements in Linux? I'm also not sure if wayland and x should offer these apis, or the desktop evironment, such as Gnome.
Thank you.

Related

Record global keyboard and mouse event in electron.js desktop app

I need some help regarding my project,
I am creating a Desktop app using electron.js, I want to capture keyboard and mouse events (just need a count of keypress events), can you please suggest to me the best solution liberties/package.
I have used iohook, but they have some issue in Linux and mac prebuild system, much appreciate if have any other solutions.
Thanks in advance.

Capture invisible (i.e.locked) virtual desktop

For test automation I'd like to capture a virtual desktop which is not visible. It is not even accessible, as a secure desktop is shown.
I know it is possible to hook into the composite manager ("dwm") to capture each and every window on that desktop. And I kmow it is possible to send events to windows on that desktop. (I know that because otherwise the test tools wouldn't work)
Before I start to re-implement the composite manager: Is it possible to get the DesktopWindow from dwm, and if so how Do I force dwm to do its job even if a secure desktop is shown?
If I have to bite the bullet and need to implemrnt compositing myself, what is the fastest way to order all windows bottom to tom and to render them to some image?
Does the win10 capture api work for invisible desktops?
To answer the last question: No, the new win10 capture API doesnt't help. For example the program
https://github.com/robmikh/SimpleRecorder/tree/master/SimpleRecorder
cannot capture a locked desktop nor can it capture sub windows.
The above is the elaborate version of:
GDI32Util.getScreenshot(handle)
with handle being the desktop window (doent work when locked) or some other window handle (works when locked, but misses the subwindows).
So the only option is to traverse all windows in z order from bottom to top.

Demystifying the Virtual Keyboard and Touchpad in Windows 10

I'm new to Windows development, and am looking for assistance on where to get started for a particular project.
In short, I want to create a windowed application that allows a user to send keyboard and mouse inputs to another application, by interacting with various UI controls via touch. Essentially a custom on-screen keyboard/touchpad that can be used for sending keyboard-shortcuts to other applications.
There are two applications in Windows 10 that perform exactly the way I would want my new app to - the On-Screen Keyboard and Touchpad:
https://support.microsoft.com/en-us/help/4337906/windows-10-open-the-on-screen-touchpad
https://support.microsoft.com/en-us/help/10762/windows-use-on-screen-keyboard
At the most basic level, I want to define my own interface (or allow the end user to define their own), and use the same code that the onscreen keyboard/touchpad are using for handling touch events and injecting inputs into the system.
I'm uncertain at what level I would need to start to get the functionality I need - UWP? WPF? C++?
If anyone has any insight into how the on-screen utilities were built, I think that would give me an excellent head start.

How to create a simple desktop environment?

I want to know how to create a simple desktop environment, such as KDE, Gnome and LXDE, for Linux, or specifically, Ubuntu/Kubuntu, but I do not know how to begin and how complex it is.
I want to create a simple, efficient and lightweight desktop and to reproduce some good features from KDE, Gnome and Windows XP. I do not want to reinvent the wheel, I want to design the graphical interface and main dialogs.
I could use C++ with Qt, or maybe with X11 calls. A simple desktop like TWM would be the first step, then I could add taskbar, start menu, new features and play with new ideas.
Thanks.
Edit
I have installed icewm and I think it is what I need to begin. On the login screen, I can choose if I use KDE (desktop environment) or icewm (window manager). I do not understand what a window manager actually is.
I have downloaded icewm source code and I am confused. There are some concepts that I have to understand, such as, what happens after the login screen when the desktop is loaded, how a window works, etc.
Edit 2
I think I need a good tutorial. I have to understand how a Window Manager or Desktop Environment work. There are some concepts that I have to know.
This is no simple feat but by no means impossible.
Other people have done it, there are plenty of DE or WM out there so there is no reason that you, with patient, skill and lots coffee couldn't do something great.
Learn the difference between DE and WM.
Test different DE and WM, maybe one does exactly what you want it to do?
Make your own custom DE
Here is a guide to get you started creating your own custom DE
After doing all this you should be better equipped to figure out what you actually want to do.
Fork a project and get started.
These are all open source projects, so I suggest you grab one of the simpler window managers and rummage around in its source code. I should warn you, though: this kind of project is not for the faint-hearted, and the likelihood of your little toy project becoming a mainstream desktop is low, so don't go into this imagining it's going to be anything more than a learning exercise (and if it does become something more, you'll be pleasantly surprised).
Strategically, your best bet is to fork an existing WM, rather than trying to build a brand new one from scratch.
One final point: Qt is a windowing framework for C++ that sits on top of a window manager. It isn't a window manager, per se, like Gnome/TWM/etc. are. And, as #ptomato rightly points out, Gnome is actually a desktop environment that itself sits on top of a window manager, which is selectable, but defaults to Mutter.
I would use X calls and learn how to boot to a CLI (command line interface). Booting to a CLI allows you to switch between window managers / desktop environments. Then I would design a window manager. And finally finish off with a desktop environment.
It starts with frame buffers. These are how 80's graphics worked.
Then you have the X window system. This however doesn't show you multiple windows at once. Only one at a time.
A window manager is the program that controls where your windows go. This allows for multiple X windows to take up one screen.
A desktop environment then builds on that to add things like a task bar, start menu and desktop icons.

notification/alert options in symbian OS

I'm trying to figure out whether or not I can register/call notifications in the symbian OS that's fit for the S60 series, using an app, that interrupt the OS and appear on the dashboard screen, the same way incoming calls and text messages do.
If so, is it just a basic alert, or can it contain a context menu (on either side) with options relating to the app?
I appreciate any help on this issue, as it'll determine whether the app is even feasible at all.
Thank you
If you need to pop up something on top of the home screen, that is perfectly feasible - a Window in a WindowGroup with the right priority will do (I've done this before). As such, this could have a menu, or whatever you need.
You don't say what events you need to hook on to when displaying this screen?

Resources