I am curious if it is possible artificially modify the server load in Ubuntu or more generally linux. I am working on an application that reacts to the server load, and in order to test it it would be nice if I could change the server load easily.
I am currently running an over-active program that will literally generate load, but I'd prefer to not continue overheating my laptop (it's getting hot!).
One of the most important things to know about Linux (or Unix) systems is, everything is just a file. Since you are just reading from /proc/loadavg, the easiest was for you to accomplish what you are after is simply make a text file that contains a line of text that you would see when running cat /proc/loadavg. Then have your program read from that file you created instead of /proc/loadavg and it will be none the wiser. If you want to test under different "artificial" situations, just change the text in this file and save. When your testing is done, simply change your program back to reading from /proc/loadavg and you can be sure it will work as expected.
Note, you can make this text file anywhere you want...in your home directory, in the program directory, wherever. However, you shouldn't make it in /proc. That directory is reserved for system objects.
You can use the stress command, see http://weather.ou.edu/~apw/projects/stress/
A tool to impose load on and stress test a computer system
sudo apt-get install stress
To avoid CPU warm, you can install a virtual machine with small cpu capacity. virtualbox and qemu-kvm are free.
Use chroot to run the various pieces of software you're testing with a specified directory as the root directory. Set up a manufactured/modified /proc/loadavg relative to that new root directory, too.
chroot will let you create a dummy file that appears to have /proc/loadavg as its path, so the software will observe your manufactured values even if you can't change your code to look for load data in a different location.
Since you don't want to actually/literally stress the machine, something like stress is not what you are after.
As stated, /proc/loadavg would be the place to set system load averages (faux loads).
But if that's also not the meat of what you're after, I would absolutely suggest
getloadavg
watchdog
and even possible Munin plugins
There're two methods.
Hacking /proc/loadavg
The machine is not overstressed
Your program reads load valus from a file
Todo: hack Linux to report fake load value
Modify your prg
The machine is not overstressed
Your program reads load valus from a file
Todo: change 4 characters in your prg: replace /proc/loadavg with /tmp/loadavg
You can decide now. Calculate costs ;)
Related
I'm currently using a piece of software (let's call it ThirdPartyApp) that reads files from a certain directory on my PC. I want to make my own software (call it MyApp) that generates files for ThirdPartyApp. When ThirdPartyApp tries to load /path/to/somefile, instead of somefile getting read from the hard drive, I want MyApp to get called and generate bytes in real time. This is similar to how reading from, say, /dev/urandom doesn't actually load a file called urandom, but instead loads the output of a random generator.
So, my question is, is this even possible to do in userspace? If so, what is this called? I'm not asking for a recommendation of a specific library or anything like that; I just need to know what to google to find info about doing something like this. Oh, and I only care about making this work on Linux, if that's a limiting factor. Thanks!
check out fuse file system : en.wikipedia.org/wiki/Filesystem_in_Userspace – Matt Joyce
Also check out named pipes. Btw, if you control starting this ThirdPartyApp then you can simply run MyApp just before that. – Kenney
The thing is, I want to track if a user tries to open a file on a shared account. I'm looking for any record/technique that helps me know if the concerned file is opened, at run time.
I want to create a script which monitors if the file is open, and if it is, I want it to send an alert to a particular email address. The file I'm thinking of is a regular file.
I tried using lsof | grep filename for checking if a file is open in gedit, but the command doesn't return anything.
Actually, I'm trying this for a pet project, and thus the question.
The command lsof -t filename shows the IDs of all processes that have the particular file opened. lsof -t filename | wc -w gives you the number of processes currently accessing the file.
The fact that a file has been read into an editor like gedit does not mean that the file is still open. The editor most likely opens the file, reads its contents and then closes the file. After you have edited the file you have the choice to overwrite the existing file or save as another file.
You could (in addition of other answers) use the Linux-specific inotify(7) facilities.
I am understanding that you want to track one (or a few) particular given file, with a fixed file path (actually a given i-node). E.g. you would want to track when /var/run/foobar is accessed or modified, and do something when that happens
In particular, you might want to install and use incrond(8) and configure it thru incrontab(5)
If you want to run a script when some given file (on a native local, e.g. Ext4, BTRS, ... but not NFS file system) is accessed or modified, use inotify incrond is exactly done for that purpose.
PS. AFAIK, inotify don't work well for remote network files, e.g. NFS filesystems (in particular when another NFS client machine is modifying a file).
If the files you are fond of are somehow source files, you might be interested by revision control systems (like git) or builder systems (like GNU make); in a certain way these tools are related to file modification.
You could also have the particular file system sits in some FUSE filesystem, and write your own FUSE daemon.
If you can restrict and modify the programs accessing the file, you might want to use advisory locking, e.g. flock(2), lockf(3).
Perhaps the data sitting in the file should be in some database (e.g. sqlite or a real DBMS like PostGreSQL ou MongoDB). ACID properties are important ....
Notice that the filesystem and the mount options may matter a lot.
You might want to use the stat(1) command.
It is difficult to help more without understanding the real use case and the motivation. You should avoid some XY problem
Probably, the workflow is wrong (having a shared file between several users able to write it), and you should approach the overall issue in some other way. For a pet project I would at least recommend using some advisory lock, and access & modify the information only thru your own programs (perhaps setuid) using flock (this excludes ordinary editors like gedit or commands like cat ...). However, your implicit use case seems to be well suited for a DBMS approach (a database does not have to contain a lot of data, it might be tiny), or some index locked file like GDBM library is handling.
Remember that on POSIX systems and Linux, several processes can access (and even modify) the same file simultaneously (unless you use some locking or synchronization).
Reading the Advanced Linux Programming book (freely available) would give you a broader picture (but it does not mention inotify which appeared aften the book was written).
You can use ls -lrt, it displays the last RW operations in the shell. Then you can conclude whether the file is opened or not. Make sure that you are in the exact directory.
I want to run a program when my embedded Linux's desktop has started up, in the same way as Windows runs programs in the "Startup" folder. How can I do this?
Specifically, my target hardware is Beaglebone Black, the Debian variant (rev C board). The Window Manager is the default one.
In Linux these are called init scripts and usually sit in /etc/init.d. How they should be defined varies between different distros but today many use the Linux Standard Base (LSB) Init Script format.
Good readings on this:
https://wiki.debian.org/LSBInitScripts
https://www.debian-administration.org/article/28/Making_scripts_run_at_boot_time_with_Debian
There are multiple ways to start a program, it turns out. LXDE - the window manager - supports auto-start of .desktop files places in either ~/.config/autostart or /etc/xdg/autostart - hooray!
http://wiki.lxde.org/en/Autostart
Except... though I can run a simple program as proof-of-concept in this way, when I try to run mine, it fails. I can't figure out why. The file
.xsession-errors.old
contains X server errors ("resource temporarily unavailable").
I am now using another mechanism - running the code from a shell script (this is necessary because I need to specify a working directory for the program). This uses the "autostart" file in /etc/xdg/lxsession/, and at least it works. Well kind of. I either have to "sleep 5" before running, or prefixing the run with an # symbol which forces a retry if it fails. It looks a little like something my code is dependent on is not in place at the precise time the autostart mechanism finds it. I can find no way of ensuring startup order. This is plainly a crock of stinky stuff.
I want to create a file from kernel and this file must be accessed from user space. Other ways of communication (for example ioctl) is not suitable, because the user space application needs only files, and I don't have the source code of it.
I need to do this on MAC. If I were using Linux, I would use sysfs for it, but on MacOs they dont have sysfs, so I decided to end up with devfs
I created the sample soultion and everything works great, but the problem is that the device file (devfs file) does not have size. The user-space code checks for file size and skips this file. I know how big the size will be, but I dont know how to set it to devfs file.
I dont want to create the file in real filesystem, because it can be quite big. All I want is to redirect reads and writes to my internal functions.
FUSE (http://en.wikipedia.org/wiki/Filesystem_in_Userspace) would be ideal for be, but this involves user-space daemon.
Any suggestions?
I was wondering if there exists a way to run an untrusted C program under a sandbox in Linux. Something that would prevent the program from opening files, or network connections, or forking, exec, etc?
It would be a small program, a homework assignment, that gets uploaded to a server and has unit tests executed on it. So the program would be short lived.
I have used Systrace to sandbox untrusted programs both interactively and in automatic mode. It has a ptrace()-based backend which allows its use on a Linux system without special privileges, as well as a far faster and more poweful backend which requires patching the kernel.
It is also possible to create a sandbox on Unix-like systems using chroot(1), although that is not quite as easy or secure. Linux Containers and FreeBSD jails are a better alternative to chroot. Another alternative on Linux is to use a security framework like SELinux or AppArmor, which is what I would propose for production systems.
We would be able to help you more if you told as what exactly it is that you want to do.
EDIT:
Systrace would work for your case, but I think that something based on the Linux Security Model like AppArmor or SELinux is a more standard, and thus preferred, alternative, depending on your distribution.
EDIT 2:
While chroot(1) is available on most (all?) Unix-like systems, it has quite a few issues:
It can be broken out of. If you are going to actually compile or run untrusted C programs on your system, you are especially vulnerable to this issue. And if your students are anything like mine, someone WILL try to break out of the jail.
You have to create a full independent filesystem hierarchy with everything that is necessary for your task. You do not have to have a compiler in the chroot, but anything that is required to run the compiled programs should be included. While there are utilities that help with this, it's still not trivial.
You have to maintain the chroot. Since it is independent, the chroot files will not be updated along with your distribution. You will have to either recreate the chroot regularly, or include the necessary update tools in it, which would essentially require that it be a full-blown Linux distribution. You will also have to keep system and user data (passwords, input files e.t.c.) synchronized with the host system.
chroot() only protects the filesystem. It does not prevent a malicious program from opening network sockets or a badly-written one from sucking up every available resource.
The resource usage problem is common among all alternatives. Filesystem quotas will prevent programs from filling the disk. Proper ulimit (setrlimit() in C) settings can protect against memory overuse and any fork bombs, as well as put a stop to CPU hogs. nice(1) can lower the priority of those programs so that the computer can be used for any tasks that are deemed more important with no problem.
I wrote an overview of sandboxing techniques in Linux recently. I think your easiest approach would be to use Linux containers (lxc) if you dont mind about forking and so on, which don't really matter in this environment. You can give the process a read only root file system, an isolated loopback network connection, and you can still kill it easily and set memory limits etc.
Seccomp is going to be a bit difficult, as the code cannot even allocate memory.
Selinux is the other option, but I think it might be more work than a container.
Firejail is one of the most comprehensive tools to do that - it support seccomp, filesystem containers, capabilities and more:
https://firejail.wordpress.com/features-3/
You can use Qemu to test assignments quickly. This procedure below takes less than 5 seconds on my 5 year old laptop.
Let's assume the student has to develop a program that takes unsigned ints, each on their own line, until a line with "-1" arrives. The program should then average all the ints and output "Average: %f". Here's how you could test program completely isolated:
First, get root.bin from Jslinux, we'll use that as the userland (it has the tcc C-compiler):
wget https://github.com/levskaya/jslinux-deobfuscated/raw/master/root.bin
We want to put the student's submission in root.bin, so set up the loop device:
sudo losetup /dev/loop0 root.bin
(you could use fuseext2 for this too, but it's not very stable. If it stabilizes, you won't need root for any of this)
Make an empty directory:
mkdir mountpoint
Mount root.bin:
sudo mount /dev/loop0 mountpoint
Enter the mounted filesystem:
cd mountpoint.
Fix rights:
sudo chown -R `whoami` .
mkdir -p etc/init.d
vi etc/init.d:
#!/bin/sh
cd /root
echo READY 2>&1 > /dev/ttyS0
tcc assignment.c 2>&1 > /dev/ttyS0
./a.out 2>&1 > /dev/ttyS0
chmod +x etc/init.d/rcS
Copy the submission to the VM:
cp ~/student_assignment.c root/assignment.c
Exit the VM's root FS:
cd ..
sudo umount mountpoint
Now the image is ready, we just need to run it. It will compile and run the submission after booting.
mkfifo /tmp/guest_output
Open a seperate terminal and start listening for guest output:
dd if=/tmp/guest_output bs=1
In another terminal:
qemu-system-i386 -kernel vmlinuz-3.5.0-27-generic -initrd root.bin -monitor stdio -nographic -serial pipe:/tmp/guestoutput
(I just used the Ubuntu kernel here, but many kernels will work)
When the guest output shows "READY", you can send keys to the VM from the qemu prompt.
For example, to test this assignment, you could do
(qemu) sendkey 1
(qemu) sendkey 4
(qemu) sendkey ret
(qemu) sendkey 1
(qemu) sendkey 0
(qemu) sendkey ret
(qemu) sendkey minus
(qemu) sendkey 1
(qemu) sendkey ret
Now Average = 12.000000 should appear on the guest output pipe. If it doesn't, the student failed.
Quit qemu: quit
A program passing the test is here: https://stackoverflow.com/a/14424295/309483. Just use tcclib.h instead of stdio.h.
Try User-mode Linux. It has about 1% performance overhead for CPU-intensive jobs, but it may be 6 times slower for I/O-intensive jobs.
Running it inside a virtual machine should offer you all the security and restrictions you want.
QEMU would be a good fit for that and all the work (downloading the application, updating the disk image, starting QEMU, running the application inside it, and saving the output for later retrieval) could be scripted for automated tests runs.
When it goes about sanboxing based on ptrace (strace) check-out:
"sydbox" sandbox and "pinktrace" programming library ( it's C99 but there are bindings to python and ruby as far as I know).
Collected links related to topic:
http://www.diigo.com/user/wierzowiecki/sydbox
(sorry that not direct links, but no enough reputation points yet)
seccomp and seccomp-bpf accomplish this with the least effort: https://www.kernel.org/doc/Documentation/prctl/seccomp_filter.txt
ok thanks to all the answers they helped ME a lot. But i would suggest none of them as an solution for the person who asked the original question. All mentioned tools require to much work for the purpose to test students code as a teacher,tutor,prof. The best way in this case would be in my opinion virtualbox. Ok, its emulates an complete x68-system and has nothing to do with the meaning of sandboxing in this way but if i imagine my programming teacher it would be the best for him. So "apt-get install virtualbox" on debian based systems, all others head over to http://virtualbox.org/ , create a vm, add an iso, click install, wait some time and be lucky. It will be much easier to use as to set up user-mode-linux or doing some heavy strace stuff...
And if you have fears about your students hacking you i guess you have an authority problem and a solution for that would be threaten them that you will sue the living daylights out of them if you can prove just one bite of maleware in the work they give you...
Also if there is a class and 1% of it is as good as he could do such things, dont bore them with such simple tasks and give them some big ones where they have to code some more. Integrative learning is best for everyone so dont relay on old deadlocked structures...
And of cause, never use the same computer for important things (like writing attestations and exams), that you are using for things like browsing the web and testing software.
Use an off line computer for important things and an on line computer for all other things.
However to everyone else who isnt a paranoid teacher (dont want to offend anybody, i am just the opinion that you should learn the basics about security and our society before you start being a programmers teacher...)
... where was i ... for everyone else:
happy hacking !!