I've got a quick question, but I can't find an answer.
Is it possible in linux (or in python) to see if an external usb pen drive is idling?
I need to know this for a python script I'm writing.
I need to rename a folder on an external usb pen drive as soon as nothing is writing to it.
edit: I know there is lsof command to list open files. 'lsof /theDir' only works half. It works OK when the process copying to the USB is still running. But when the process stops, lsof shows nothing. But the OS is still writing to the USB from its buffer.
You can check if all I/O has been processed by having a look at /sys/block/<dev>/stat.
The ninth column contains the number of I/Os currently in flight. Check https://www.kernel.org/doc/Documentation/block/stat.txt
Once this numner is zero the device should be idle.
To force all buffers to be written immediately you could execute sync and wait until it returns.
Nevertheless be aware that you have a race condition here if you are not controlling the writing - after you decided that the device is idle some other process could start writing to it.
Related
By now I have now spent at least 10 hours trying to get my head around the famous blog post by Linus Akesson, and Im still struggling. So let me ask my doubts about tty/ptty as a series of short questions.
1) Is the tty/ptty in user space or kernel space?
2) What is tty/ptty's connection to devices or drivers or some numbering or something?
3) The tty seems to be linked to something called the controlling terminal of a process, What is the relation and is every process related to a terminal?
4) On the whole I still dont understand where the heck this terminal concept fits in. A process wants to read something from the stdio, cant it simply do it from the required device file. What exactly is the problem that the tty intends to take care of?
5) I read somewhere that there are attempts to move the tty from the userspace to the kernel space. Is the tty simply a historical residue than a strong design feature.??
A clarification (which might answer some of your questions):
I think you meant pty (and not ptty) which is pseudo-tty/pseudo-terminal.
A tty (/dev/ttyx) - stands for teletype - is the original terminals (used a line printer for output and a keyboard for input!). A terminal is basically just a user interface device that uses text for input and output.
A pty (/dev/pty/n) is a pseudo-terminal - it's a software implementation that appears to the attached program like a terminal, but instead of communicating directly with a "real" terminal, it transfers the input and output to another program. It's the end point of telnet/SSH or even the GNOME terminal.
For example, when you ssh into a remote machine and run ls, the ls output is sent to a pseudo-terminal, the other side of which is attached to the SSH daemon.
EDIT:
As far as I know, the tty and so pty, are usermode. BUT they represent terminal-driver. What I mean is: the device file /dev/tty1 is the first virtual console. Most code lives in drivers/char, in the files tty_io.c and n_tty.c and vt.c (kernel source). In contrast to character devices in order to open those files tty_open routine is called, and trust me, it's way messier than opening a character device...
Tty/pty stands for terminal drivers mentioned above but they stands for serial ports (the"numbering" you said). I know very little about it so I don't want to say incorrect data... but you can search the net about it (or someone else can continue from here)
EDIT2:
You have changed the question so now it seems like I spoke out of context...
Anyway, tty has many different roles even nowday. Terminal driver is the way user-kernel can "communicate". There are some techniques such as terminal drivers, character device etc.
If you still have a question please comment and don't change the whole post....
I need to write a program in linux (debian, to be exact) that disables a USB device if it doesn't pass specific filters. For example, the program might be set to disallow webcams, usb sticks and keyboards, enable mice and printer (through usb). The filters may change in runtime. For example, the program might receive a message to enable usb sticks and it should respond without rebooting the system. The program is written in python but embedding c code (or others) is also acceptable.
What I have tried
I have tried many methods but some of them aren't about programming.
First, I tried to mess with udev. I can monitor the activities when a device is plugged and write filters. There used to be an option "ignore_device" that ignored the filtered devices. For example, to ignore all the devices that are a member of usb subsystem, I would write this as a udev rule:
ACTION=="add", SUBSYSTEM=="usb", OPTIONS+="ignore_device"
But this option is released in this release of udev. What I get so far is that udev can currently be used mainly for monitoring. Sure, I can write additional rules for the rule above that runs a script, but I have to do the disabling elsewhere.
Second, I tried ioctl to send a DISCONNECT signal to device handle. I'm testing this with a USB mouse. This is the python code for that: (I have also tried this in C, nothing changed)
import fcntl
import sys
USBDEVFS_RESET = ord('U') << (4*2) | 20
USBDEVFS_DISCONNECT = ord('U') << (4*2) | 22
raw_name = "/dev/bus/usb/{:03d}/{:03d}"
filename = raw_name.format(1,2)
fd = open(filename, "wb")
fcntl.ioctl(fd, USBDEVFS_DISCONNECT, 0)
Here, if I would send USBDEVFS_RESET, it works, the mouse input is ignored for a second or two. But disconnect signal raises an error:
IOError: [Errno 25] Inappropriate ioctl for device
What I get from here is, I cannot send disconnect signal to a mouse. Maybe a usb stick or printer or some other devices would work, I haven't tried. I want to develop the program as generic as possible so as to prevent writing additional device-specific code, so this approach seems useless for me. And another point here is that when I manually disconnect/connect my mouse, I see events in udev monitor. But when I send reset signal, no event is sent.
The udev monitor says that the mouse was mounted to this path: /sys/bus/usb/devices/1-3 (which is a symlink for /sys/devices/pci0000:00/0000:00:14.0/usb1/1-3). Some documents told me that this folder contains the device's settings and setting /power/level to "off" or "suspended" would turn the device power off. But I cannot manipulate any files in /power. Come to think about it, it might not be a good idea after all.
The Question
So, the question is, what is the best way to achieve such task? I have an idea but I'm not sure whether it will work and even if it does, it might be overkill. My idea is writing a "wrapper driver" that identifies itself to linux kernel as driver for all usb devices. The "wrapper driver" reads the device information and if the device is good to go, it acts as a wrapper for real drivers in the kernel, calling their functions. If not, the "wrapper driver" just ignores the device.
I'm not sure it can even be done, I'm not experienced in kernel or driver programming.
Another way is, -somehow- getting the handle of the device programatically and telling it to power off (or making it busy forever, whatever works). I have also done some little research but couldn't find a proper-easy way. They say all devices are considered as "files" but I cannot reach those files at all.
Note:
The question is mostly about linux internals but it also involves kernel programming. I read a lot about usb manipulation/monitoring programs, I read manual pages of udevadm. But these approaches do not help me at all. I think I need to alter either kernel or device internals programatically.
I have also tried manipulating authorized file that resides in /sys/devices/pci0000:00/0000:00:14.0/usb1/1-4 (for a keyboard). It's default value is 1. Changing it to 0 successfully disabled the bus (NOT device, but the physical usb port. The same device can still be used when plugged in another port). But making it 0 also stopped udev events from this usb port. So, I can disable the port if the user plugs in a forbidden device but I cannot decide when to enable it since I cannot listen to remove events in udev. Would it make sense if I delve deeper to lower levels of code (possibly kernel) and listen to usb events in some other way?
I think the simplest way to solve your problem is balcklisting all usb device drivers excepting mouse, keyboard etc.
The cleanest way is whitlisting mouse etc. with udev using usb device id's
Writing a wrapper driver is an absolute overkill never chose that way.
One possible solution could be tying usb access to user permissions. here is a related link you may find handy
http://robots.mobilerobots.com/wiki/Linux_udev_USB_Device_Permissions_Configuration
Assume the following situation under Linux:
A process is continuously reading from an USB-serial converter device (/dev/ttyUSB0). That device is suddenly unplugged and plugged in again (or is resetting itself for some reason). The process continues to have a valid file handle for /dev/ttyUSB0 but won't receive any data from the device unless the process re-opens the device (because udev has deleted and re-created the device node).
Is there a direct way to detect such a situation (ie. not indirectly by detecting a timeout in data flow) so that the process knows it has to re-open the device? Would it be reliable to monitor the modification time of /dev/ttyUSB0 using stat()?
Additional details:
The process opens the device file by using the standard open() function.
/dev is a tmpfs controlled by udev.
Note: I do not want to use any udev rules for this and prefer a solution implemented directly in the process.
If a USB device is hot-unplugged, operations on the device will begin failing with -EIO; you can detect this and take appropriate action.
If the device node is actually being removed and re-created (which I believe it is if you have udev), then you should be able to use the inode number to tell when this happens.
Just call fstat() on your open file descriptor, stat() on /dev/ttyUSB0, and compare the st_ino fields of the two struct stats.
And let me know if it actually works. :-)
I think FAM or gamin will detect these events.
Somewhere I read that opening the same file twice has an undefined semantics and should be avoided. In my situation I would like to open my own device multiple times associating multiple file descriptors to it. The file operations of my device are all safe. Is there some part of Linux between the sys call open() and the point it calls the registered file operation .open() that is unsafe?
It is perfectly fine to open the same device file twice, as long as your driver is ok with that.
There is no hidden part that would make it unsafe if it is safe in the kernel.
For example, some video application use one process to do the display or capture, while another opens the device file to handle the controls.
If the driver does not support multiple open, then it should return an error when a second open happens.
You may open a device twice in the same process, if the driver will let you do so. Synchronization is the responsibility of the driver.
However, if you are opening, say, a raw disk device as a privileged user, you will want to make sure you don't clobber your own data in your process.
Opening the same file twice has well-defined semantics in cases which make sense. Processes still need some form of synchronisation if they're all doing read/write, otherwise the file is likely to end up full of rubbish.
For a device driver, the semantics of multiple opens is entirely up to the driver - some drivers prohibit it, in others it works fine (think /dev/null for instance). In some drivers it has a very special meaning (e.g. sound cards may mix the sound output between multiple apps)
I've written a script to backup my server's HD every night. At the end of the script, I sync, wait a couple of minutes, sync and then I issue sg_start --stop to stop the device. The idea is to extend the lifetime of the device by switching the HD off after ten minutes of incremental backup (desktop disks will survive several thousand on/off cycles but only a few hundred hours of continuous running).
This doesn't always work; I often find the drive still spinning the next morning. Is there a shell command which I can use to check that the drive has stopped (so I can issue the stop command again [EDIT2]or write a script to create a process list when the drive is running so I can debug this[/EDIT2])?
[EDIT] I've tried sg_inq (as suggested by the sg_start man page) but this command always returns 0.
I've tried hdparm but it always returns "drive state is: unknown" for USB drives (attached via /dev/sdX) and when trying to spin down the drive, I get "HDIO_DRIVE_CMD(setidle1) failed: Input/output error".
sdparm seems to support to set the idle timer on the drive (see "Power mode condition page") but the IDLE option has "Changeable: n" and I haven't found an option which tells me the drive power state.
[EDIT2] Note: I can stop the drive with sg_start --stop from the console. That always works; it just doesn't always stay down until midnight. The sever is in the basement (where it's nice and cool) and I'd rather have a way to check whether the drive is up or not from the warm living room :) If I had a command which told me the status of the drive, I could write a script to alert me when it spins up (check every minute) and then I could try to figure out what might be causing this.
If that matters: I'm using openSUSE 11.1.
When you say you've tried hdparm, you haven't said what you have tried. I have some usb hard drives in an enclosure, and some of the commands work for it and others don't, but I guess it all depends on all facets of the transport mechanism.
hdparm -S 120 /dev/sda
Should tell the drive to sleep itself after ~10 minutes of inactivity.
I'm guessing you have already tried this, but its not obvious, and writing this as an answer may help a future reader.
Nothing accesses the drive but the backup script.
This is nice in theory, but I have found on odd occasions this is not enough. There are lots of processes and tasks that if they even look at the drive a spin up may occur if the lookup is for some reason out of the disk-cache.
Common culprits are tools like updatedb scanning all mounts for files, and fam or gamin doing funky stuff to monitor disks for changes.
If in doubt, add a layer of certainty by mounting the device before executing the script, and unmounting it when you are done.
Seeing things that could cause a wakeup
lsof +D /mountpoint
You should probably parse the output of this as well before attempting to unmount to be sure that nothing is still trying to use it.
You should probably also be doing a lazy umount,
umount -l /mountpoint
so if anything accesses it between you doing lsof| grep and calling umount, it will still unmount the drive and stop things being able to read it.
HAL and friends
Its also possible that HAL and friends are doing wakeups to the drive probing for connect/removal state. I really hope it isn't, but it does that on some device types. It seems an unlikely cause, but I'll consider anything possible.
Try fun stuff like
lsof /dev/devicehere
and
lsof /dev/devicehere1
Which appears to get a comprehensive list of all things that would be accessing the handle either directly or indirectly.
You also need to check the mount options and use "noatime" as a mount option, otherwise the kernel still updates the access times periodically. So this may be the cause of your problem too.
If hdparm returns this:
drive state is: unknown
And smartctl returns this:
CHECK POWER MODE: incomplete response, ATA output registers missing
CHECK POWER MODE not implemented, ignoring -n option
Then it is not possible to obtain the usb drive sleep status. I tried multiple creative things like the USB power consumption or monitoring file access, but I found nothing to verify the drive's state.
The only solution is to replace the USB SATA adapter (or enclosure) against one with a different USB bridge chip. The one that did not work for me, used the JMicron JMS567 (152d:0578 / 0x152d 0x0578) which could be perhaps updated, but I was not able to test it as the update tool needs an ARM CPU. Now I'm using two different adapters which both use the Asmedia ASM1051E / ASM1053E / ASM1153 / ASM1153E and they passthrough all commands.
You can check the bridge chip of your USB SATA adapter / enclosure as follows:
lsusb -v | grep -E ': ID |idVendor|idProduct'
Which returns something like:
Bus 002 Device 003: ID 174c:55aa ASMedia Technology Inc. ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge
idVendor 0x174c ASMedia Technology Inc.
idProduct 0x55aa ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge
I often find the drive still spinning the next morning.
Couldn't this just be because it was spun up again when the server eg. wrote to a log file or something during the night? You might try sdparm, which can return status information on a drive. But I think it's better to just set the option in your BIOS that lets your HD automatically spin down after an amount of inactivity, it's easier.
Have you tried stopping the drive with the following command:
eject -t /dev/yourHD
This works quite good for my USB hard drives.
If hdparm -C /dev/sda said drive state is: unknown than it's not supported for your drive and there is no way to tell
Your additional questions (answered by others already)
what's using a drive?
lsof
fuser
triggers
how to force a drive to stay sleeping?
hdparm
umount
your device can still be woken when unmounted but likely only by something you do (smartctl, blkid, etc)
Related: you can also automatically pause backup or whatever for an hour if your drives get to hot.