I'm looking for a way to get resize events on an xterm as an alternative to the winch signal. I need to get a signal for the xterm resize that is remote compatible, that is, could be used over a serial line/telnet/ssh/whatever. The winch signal is only for local machine tasks.
I know that vi/curses can do this because I have tried ssh and use vi to edit a file, and it responds to resizing of the window.
So after a bit of research, it does seem there is no in-band method for this. I don't think it would be difficult to add. You have two sides to a connection, ssh, serial or whatever, let's say A and B where A is on the host and B is the remote. To get behavior equivalent to a local windowed task, I would say at least two things are essential:
Mouse movement/button messaging.
Window resize messaging.
The second, can be a two step process. If the B task gets an alert that the window size has changed, it can use in band queries to determine what the new size is, using the (perhaps infamous) method of setting the cursor to the end of an impossibly large screen and then reading the actual location of the cursor that results, then restoring the location, all done with standard in-band controls.
Why care about in-band vs. out of band? Well, nobody cares now :-), but serial lines used to be quite popular, and in-band, that is, actual escape sequences sent down the line are going to work, but telnet or ssh "out of band" communication is not.
It would not be difficult to add, a local program that shims the ssh or telnet, or even xterm itself, can get the winch message and change that to an escape alert to the B side. Here the mouse messaging is instructive. The messages consist of two actions:
The B side must enable such messages, so that the A side won't send unknown escapes to it.
The A side must have an escape to signify winch.
Thus a new escape from B to A, and one from A to B.
I'm actually fairly satisfied with winch. What was wanted is to have the same capabilities as vi/curses (based on my observation). This kind of support is already far better than Windows, which (as far as I know) implements none of this remote side support.
I am trying to create a special military RADIO transmitter.
Basically, the flow is:
A solider will receive a message to transmit (about 10 times a day). Each message is of length 1024 bits exactly.
He will insert this message into the radio and validate it is inserted correctly.
The RADIO will repetitively transmit this message.
This is very important that the transmitter will not be hacked, because its very important in times of emergencies.
So, the assistance I ask from you is, how to preform stage 2 without risking getting infected.
If I will transfer the data using a DOK, it may be hacked.
If I will make the user type in 1024 bits, it will be safe but exhausting.
Any Ideas? (unlimited budget)
(It’s important to say that the data being transmitted is not a secret)
Thanks for any help you can supply!
Danny
Edit:
Basically, I want to create the most secure way to transfer a fixed number of bits (in this case 1024), from one (may be infected computer) to the other (air gaped computer).
without having any risk of a virus being transferred as well.
I don't mind if an hacker will change the data that is transferred from the infected computer, I just want that the length of the data will be exactly 1024, and avoiding virus to be inserted to the other computer.
Punch card (https://en.wikipedia.org/wiki/Punched_card) sounds like a good option, but an old one.
Any alternatives?
The transmitter is in the field, and is one dead soldier away from falling into enemy hands at any time. The enemy will take it apart, dissect it, learn how it works and use the protocol to send fraudulent messages that may contain exploit code to you, with or without the original equipment. You simply cannot prevent a trasmitter or otherwise mocked up "enemy" version of a transmitter from potentially transmitting bad stuff, because those are outside of your control. This is the old security adage "Never trust the client" taken to its most extreme level. Even punch cards could be tampered with.
Focus on what you can control: The receiving (or host) computer (which, contrary to your description, is not airgapped as it is receiving outside communication in your model) will need to validate the messages that come in from the client source; this validation will need to check for malicious messages and handle them safely (don't do anything functional with them, just log it, alert somebody and move on with life).
Your protocol should only be treating inbound messages like text or identifiers for message types. Under no circumstances should you be trying to interpret them as machine language instructions, and any SQL queries or strings that this message is appended to should be properly sanitized. This will prevent the host from executing any nasties that do come in.
I'm trying to debug a laggy machine vision camera by writing text timestamps to a terminal window and then observing how long it takes for the camera to 'detect' the screen change. My monitor has a 60hz refresh rate, so the screen is updated every ~17ms. Is there a way to determine at what point within that 17ms window the refresh timer currently is for an X11 application.
EDIT: After wrestling with the problem for nearly a day, I think the real question I should have asked was how to generate a visual signal that was sufficiently fast to test the camera images. My working hypothesis was that the camera was buffering frames before transmitting them, as the video stream seemed to lag behind other synchronised digital events (in this case, output signals to a robotic controller)
'xrefresh' is a tool which can trigger a refresh event on an X server. It does this by painting a global window of a specified color and then removing it, causing all subsequent windows to repaint. Even with this, I was still getting very inconsistent results when trying to correlate the captured frames against the monitor output, no matter what I tried to do, the video stream seemed to lag behind what I expected the monitor state to be. This could mean that either the camera was slow to capture or the monitor was slow to update. Fortunately, I eventually hit upon the idea of using the keyboard leds to verify the synchronicity of the camera frames. ('xset led' and 'xset -led'). This showed me immediately that in fact my computer monitor was slow to update, instead of the camera lagging behind.
Sorry for the wall of text - TL;DR:
What is the framerate of VNC connection (in frames/sec) - or rather, who determines it: client or server?
Any other suggestions for desktop screen capture - but "correctly timecoded"/ with unjittered framerate (with a stable period); and with possibility to obtain it as uncompressed (or lossless) image sequence?
Briefly - I have a typical problem that I am faced with: I sometimes develop hardware, and want to record a video that shows both commands entered on the PC ('desktop capture'), and responses of the hardware ('live video'). A chunk of an intro follows, before I get to the specific detail(s).
Intro/Context
My strategy, for now, is to use a video camera to record the process of hardware testing (as 'live' video) - and do a desktop capture at the same time. The video camera produces a 29.97 (30) FPS MPEG-2 .AVI video; and I want to get the desktop capture as an image sequence of PNGs at the same frame rate as the video. The idea, then, would be: if the frame rate of the two videos is the same; then I could simply
align the time of start of the desktop capture, with the matching point in the 'live' video
Set up a picture-in-picture, where a scaled down version of the desktop capture is put - as overlay - on top of the 'live' video
(where a portion of the screen on the 'live' video, serves as a visual sync source with the 'desktop capture' overlay)
Export a 'final' combined video, compressed appropriately for the Internet
In principle, I guess one could use a command line tool like ffmpeg for this process; however I would prefer to use a GUI for finding the alignment start point for the two videos.
Eventually, what I also want to achieve, is to preserve maximum quality when exporting the 'final' video: the 'live' video is already compressed when out of the camera, which means additional degradation when it passes through the Theora .ogv codec - which is why I'd like to keep the original videos, and use something like a command line to generate a 'final' video anew, if a different compression/resolution is required. This is also why I like to have the 'desktop capture' video as a PNG sequence (although I guess any uncompressed format would do): I take measures to 'adjust' the desktop, so there aren't many gradients, and lossless encoding (i.e. PNG) would be appropriate.
Desktop capture options
Well, there are many troubles in this process under Ubuntu Lucid, which I currently use (and you can read about some of my ordeals in 10.04: Video overlay/composite editing with Theora ogv - Ubuntu Forums). However, one of the crucial problems is the assumption, that the frame rate of the two incoming videos is equal - in reality, usually the desktop capture is of a lower framerate; and even worse, very often frames are out of sync.
This, then, requires the hassle of sitting in front of a video editor, and manually cutting and editing less-than-a-second clips on frame level - requiring hours of work for what will be in the end a 5 minute video. On the other hand, if the two videos ('live' and 'capture') did have the same framerate and sync: in principle, you wouldn't need more than a couple of minutes for finding the start sync point in a video editor - and the rest of the 'merged' video processing could be handled by a single command line. Which is why, in this post, I would like to focus on the desktop capture part.
As far as I can see, there are only few viable (as opposed to 5 Ways to Screencast Your Linux Desktop) alternatives for desktop capture in Linux / Ubuntu (note, I typically use a laptop as target for desktop capturing):
Have your target PC (laptop) clone the desktop on its VGA output; use a VGA-to-composite or VGA-to-S-video hardware to obtain a video signal from VGA; use video capture card on a different PC to grab video
Use recordMyDesktop on the target PC
Set up a VNC server (vino on Ubuntu; or vncserver) on the target PC to be captured; use VNC capture software (such as vncrec) on a different PC to grab/record the VNC stream (which can, subsequently, be converted to video).
Use ffmpeg with x11grab option
*(use some tool on the target PC, that would do a DMA transfer of a desktop image frame directly - from the graphics card frame buffer memory, to the network adapter memory)
Please note that the usefulness of the above approaches are limited by my context of use: the target PC that I want to capture, typically runs software (utilizing the tested hardware) that moves around massive ammounts of data; best you could say about describing such a system is "barely stable" :) I'd guess this is similar to problems gamers face, when wanting to obtain a video capture of a demanding game. And as soon as I start using something like recordMyDesktop, which also uses quite a bit of resources and wants to capture on the local hard disk - I immediately get severe kernel crashes (often with no vmcore generated).
So, in my context, I typically do assume involvement of a second computer - to run the capture and recording of the 'target' PC desktop. Other than that, the pros and cons I can see so far with the above options, are included below.
(Desktop preparation)
For all of the methods discussed below, I tend to "prepare" the desktop beforehand:
Remove desktop backgrounds and icons
Set the resolution down to 800x600 via System/Preferences/Monitors (gnome-desktop-properties)
Change color depth down to 16 bpp (using xdpyinfo | grep "of root" to check)
... in order to minimize the load on desktop capture software. Note that changing color depth on Ubuntu requires changes to xorg.conf; however, "No xorg.conf (is) found in /etc/X11 (Ubuntu 10.04)" - so you may need to run sudo Xorg -configure first.
In order to keep graphics resource use low, also I usually had compiz disabled - or rather, I'd have 'System/Preferences/Appearance/Visual Effects' set to "None". However, after I tried enabling compiz by setting 'Visual Effects' to "Normal" (which doesn't get saved), I can notice windows on the LCD screen are redrawn much faster; so I keep it like this, also for desktop capture. I find this a bit strange: how could more effects cause a faster screen refresh? It doesn't look like it's due to a proprietary driver (the card is "Intel Corporation N10 Family Integrated Graphics Controller", and no proprietary driver option is given by Ubuntu upon switch to compiz) - although, it could be that all the blurring and effects just cheat my eyes :) ).
Cloning VGA
Well, this is the most expencive option (as it requires additional purchase of not just one, but two pieces of hardware: VGA converter, and video capture card); and applicable mostly to laptops (which have both a screen + additional VGA output - for desktops one may also have to invest in an additional graphics card, or a VGA cloning hardware).
However, it is also the only option that requires no additional software of the target PC whatsoever (and thus uses 0% processing power of the target CPU) - AND also the only one that will give a video with a true, unjittered framerate of 30 fps (as it is performed by separate hardware - although, with the assumption that clock domains misalignment, present between individual hardware pieces, is negligible).
Actually, as I already own something like a capture card, I have already invested in a VGA converter - in expectation that it will eventually allow me to produce final "merged" videos with only 5 mins of looking for alignment point, and a single command line; but I am yet to see whether this process will work as intended. I'm also wandering how possible it will be to capture desktop as uncompressed video # 800x600, 30 fps.
recordMyDesktop
Well, if you run recordMyDesktop without any arguments - it starts first with capturing (what looks like) raw image data, in a folder like /tmp/rMD-session-7247; and after you press Ctrl-C to interrupt it, it will encode this raw image data into an .ogv. Obviously, grabbing large image data on the same hard disk as my test software (which also moves large ammounts of data), is usually a cause for an instacrash :)
Hence, what I tried doing is to setup Samba to share a drive on the network; then on the target PC, I'd connect to this drive - and instruct recordMyDesktop to use this network drive (via gvfs) as its temporary files location:
recordmydesktop --workdir /home/user/.gvfs/test\ on\ 192.168.1.100/capture/ --no-sound --quick-subsampling --fps 30 --overwrite -o capture.ogv
Note that, while this command will use the network location for temporary files (and thus makes it possible for recordMyDesktop to run in parallel with my software) - as soon as you hit Ctrl-C, it will start encoding and saving capture.ogv directly on the local hard drive of the target (though, at that point, I don't really care :) )
First of my nags with recordMyDesktop is that you cannot instruct it to keep the temporary files, and avoid encoding them, on end: you can use Ctrl+Alt+p for pause - or you can hit Ctrl-C quickly after the first one, to cause it to crash; which will then leave the temporary files (if you don't hit Ctrl-C quickly enough the second time, the program will "Cleanning up cache..."). You can then run, say:
recordmydesktop --rescue /home/user/.gvfs/test\ on\ 192.168.1.100/capture/rMD-session-7247/
... in order to convert the raw temporary data. However, more often than not, recordMyDesktop will itself segfault in the midst of performing this "rescue". Although, the reason why I want to keep the temp files, is to have the uncompressed source for the picture-in-picture montage. Note that the "--on-the-fly-encoding" will avoid using temp files altogether - at the expence of using more CPU processing power (which, for me, again is cause for crashes.)
Then, there is the framerate - obviously, you can set requested framerate using the '--fps N' option; however, that is no guarantee that you will actually obtain that framerate; for instance, I'd get:
recordmydesktop --fps 25
...
Saved 2983 frames in a total of 6023 requests
...
... for a capture with my test software running; which means that the actually achieved rate is more like 25*2983/6032 = 12.3632 fps!
Obviously, frames are dropped - and mostly that shows as video playback is too fast. However, if I lower the requested fps to 12 - then according to saved/total reports, I achieve something like 11 fps; and in this case, video playback doesn't look 'sped up'. And I still haven't tried aligning such a capture with a live video - so I have no idea if those frames that actually have been saved, also have an accurate timestamp.
VNC capture
The VNC capture, for me, consists of running a VNC server on the 'target' PC, and running vncrec (twibright edition) on the 'recorder' PC. As VNC server, I use vino, which is "System/Preferences/Remote Desktop (Preferences)". And apparently, even if vino configuration may not be the easiest thing to manage, vino as a server seems not too taxing to the 'target' PC; as I haven't experienced crashes when it runs in parallel with my test software.
On the other hand, when vncrec is capturing on the 'recorder' PC, it also raises a window showing you the 'target' desktop as it is seen in 'realtime'; when there are large updates (i.e. whole windows moving) on the 'target' - one can, quite visibly, see problems with the update/refresh rate on the 'recorder'. But, for only small updates (i.e. just a cursor moving on a static background), things seem OK.
This makes me wonder about one of my primary questions with this post - what is it, that sets the framerate in a VNC connection?
I haven't found a clear answer to this, but from bits and pieces of info (see refs below), I gather that:
The VNC server simply sends changes (screen changes + clicks etc) as fast as it can, when it receives them ; limited by the max network bandwidth that is available to the server
The VNC client receives those change events delayed and jittered by the network connection, and attempts to reconstruct the desktop "video" stream, again as fast as it can
... which means, one cannot state anything in terms of a stable, periodic frame rate (as in video).
As far as vncrec as a client goes, the end videos I get usually are declared as 10 fps, although frames can be rather displaced/jittered (which then requires the cutting in video editors). Note that the vncrec-twibright/README states: "The sample rate of the movie is 10 by default or overriden by VNCREC_MOVIE_FRAMERATE environment variable, or 10 if not specified."; however, the manpage also states "VNCREC_MOVIE_FRAMERATE - Specifies frame rate of the output movie. Has an effect only in -movie mode. Defaults to 10. Try 24 when your transcoder vomits from 10.". And if one looks into "vncrec/sockets.c" source, one can see:
void print_movie_frames_up_to_time(struct timeval tv)
{
static double framerate;
....
memcpy(out, bufoutptr, buffered);
if (appData.record)
{
writeLogHeader (); /* Writes the timestamp */
fwrite (bufoutptr, 1, buffered, vncLog);
}
... which shows that some timestamps are written - but whether those timestamps originate from the "original" 'target' PC, or the 'recorder' one, I cannot tell.
EDIT: thanks to the answer by #kanaka, I checked through vncrec/sockets.c again, and can see that it is the writeLogHeader function itself calling gettimeofday; so the timestamps it writes are local - that is, they originate from the 'recorder' PC (and hence, these timestamps do not accurately describe when the frames originated on the 'target' PC).
In any case, it still seems to me, that the server sends - and vncrec as client receives - whenever; and it is only in the process of encoding a video file from the raw capture afterwards, that some form of a frame rate is set/interpolated.
I'd also like to state that on my 'target' laptop, the wired network connection is broken; so the wireless is my only option to get access to the router and the local network - at far lower speed than the 100MB/s that the router could handle from wired connections. However, if the jitter in captured frames is caused by wrong timestamps due to load on the 'target' PC, I don't think good network bandwidth will help too much.
Finally, as far as VNC goes, there could be other alternatives to try - such as VNCast server (promising, but requires some time to build from source, and is in "early experimental version"); or MultiVNC (although, it just seems like a client/viewer, without options for recording).
ffmpeg with x11grab
Haven't played with this much, but, I've tried it in connection with netcat; this:
# 'target'
ffmpeg -f x11grab -b 8000k -r 30 -s 800x600 -i :0.0 -f rawvideo - | nc 192.168.1.100 5678
# 'recorder'
nc -l 0.0.0.0 5678 > raw.video #
... does capture a file, but ffplay cannot read the captured file properly; while:
# 'target'
ffmpeg -f x11grab -b 500k -r 30 -s 800x600 -i :0.0 -f yuv4mpegpipe -pix_fmt yuv444p - | nc 192.168.1.100 5678
# 'recorder'
nc -l 0.0.0.0 5678 | ffmpeg -i - /path/to/samplimg%03d.png
does produce .png images - but with compression artifacts (result of the compression involved with yuv4mpegpipe, I guess).
Thus, I'm not liking ffmpeg+x11grab too much currently - but maybe I simply don't know how to set it up for my needs.
*( graphics card -> DMA -> network )
I am, admittedly, not sure something like this exists - in fact, I would wager it doesn't :) And I'm no expert here, but I speculate:
if DMA memory transfer can be initiated from the graphics card (or its buffer that keeps the current desktop bitmap) as source, and the network adapter as destination - then in principle, it should be possible to obtain an uncompressed desktop capture with a correct (and decent) framerate. The point in using DMA transfer would be, of course, to relieve the processor from the task of copying the desktop image to the network interface (and thus, reduce the influence the capturing software can have on the processes running on the 'target' PC - especially those dealing with RAM or hard-disk).
A suggestion like this, of course, assumes that: there are massive ammounts of network bandwidth (for 800x600, 30 fps at least 800*600*3*30 = 43200000 bps = 42 MiB/s, which should be OK for local 100 MB/s networks); plenty of hard disk on the other PC that does the 'recording' - and finally, software that can afterwards read that raw data, and generate image sequences or videos based on it :)
The bandwidth and hard disk demands I could live with - as long as there is guarantee both for a stable framerate and uncompressed data; which is why I'd love to hear if something like this already exists.
-- -- -- -- --
Well, I guess that was it - as brief as I could put it :) Any suggestions for tools - or process(es), that can result with a desktop capture
in uncompressed format (ultimately convertible to uncompressed/lossless PNG image sequence), and
with a "correctly timecoded", stable framerate
..., that will ultimately lend itself to 'easy', single command-line processing for generating 'picture-in-picture' overlay videos - will be greatly appreciated!
Thanks in advance for any comments,
Cheers!
References
Experiences Producing a Screencast on Linux for CryptoTE - idlebox.net
The VideoLAN Forums • View topic - VNC Client input support (like screen://)
VNCServer throttles user inpt for slow client - Kyprianou, Mark - com.realvnc.vnc-list - MarkMail
Linux FAQ - X Windows: How do I Display and Control a Remote Desktop using VNC
How much bandwidth does VNC require? RealVNC - Frequently asked questions
x11vnc: a VNC server for real X displays
HowtoRecordVNC (an X11 session) - Debian Wiki
Alternative To gtk-RecordMyDesktop in Ubuntu
(Ffmpeg-user) How do I use pipes in ffmpeg
(ffmpeg-devel) (PATCH) Fix segfault in x11grab when drawing Cursor on Xservers that don't support the XFixes extension
You should get a badge for such a long well though out question. ;-)
In answer to your primary question, VNC uses the RFB protocol which is a remote frame buffer protocol (thus the acronym) not a streaming video protocol. The VNC client sends a FrameBufferUpdateRequest message to the server which contains a viewport region that the client is interested in and an incremental flag. If the incremental flag is not set then the server will respond with a FrameBufferUpdate message that contains the content of the region requested. If the incremental flag is set then the server may respond with a FrameBufferUpdate message that contains whatever parts of the region requested that have changed since the last time the client was sent that region.
The definition of how requests and updates interact is not crisply defined. The server won't necessarily respond to every request with an update if nothing has changed. If the server has multiple requests queued from the client it is also allowed to send a single update in response. In addition, the client really needs to be able to respond to an asynchronous update message from the server (not in response to a request) otherwise the client will fall out of sync (because RFB is not a framed protocol).
Often clients are simply implemented to send incremental update requests for the entire frame buffer viewport at a periodic interval and handle any server update messages as they arrive (i.e. no attempt is made to tie requests and updates together).
Here is a description of FrameBufferUpdateRequest messages.