Some software (for ex. the NetBeans IDE) ship the Linux installers in .sh files. Curious about how exactly they 'package' a whole IDE into a 'shell script', I opened the file in an editor. I saw some plain text shell scripting code and then some random gibberish, which I reckon is 'binary' or non-plain text.
I am wondering how they mix plain shell scripts and then probably call the 'non-readable' stuff, which would be the binaries.
Any insight on this?
Basically, it's a shell script prepended to a compressed archive of some sort, such as a tar archive. You use the tail or sed command on yourself (the $0 variable in Bourne shell) to strip off the shell script at the front and pass the rest to your unarchiver.
For example, create the following script as self-extracting:
#!/bin/sh -e
sed -e '1,/^exit$/d' "$0" | tar xzf - && ./project/Setup
exit
The sed command above deletes all lines from the first line of the file to the first one that starts with "exit", and then passes the rest on through. If what starts immediately after the "exit" line is a tar file, the tar command will extract it. If that's successful, the ./project/Setup file (presumably extracted from the tarball) will be executed.
Then:
mkdir project
echo "#!/bin/sh" > project/Setup
echo "echo This is the setup script!" >> project/Setup
chmod +x project/Setup
tar czf - project >> self-extracting
Now, if you get rid of your old project directory, you can run self-extracting and it will extract that tar file and run the setup script.
You might want to check out makeself.sh
From the authors' notes.
makeself.sh is a small shell script that generates a self-extractable tar.gz archive from a directory. The resulting file appears as a shell script (many of those have a .run suffix), and can be launched as is. The archive will then uncompress itself to a temporary directory and an optional arbitrary command will be executed (for example an installation script).
Makeself archives also include checksums for integrity self-validation (CRC and/or MD5 checksums).
The makeself.sh script itself is used only to create the archives from a directory of files. The resultant archive is actually a compressed (using gzip, bzip2, or compress) TAR archive, with a small shell script stub at the beginning. This small stub performs all the steps of extracting the files, running the embedded command, and removing the temporary files when it's all over. All what the user has to do to install the software contained in such an archive is to "run" the archive [that is execute the script]
I am trying to keep the code of this script as portable as possible, i.e it's not relying on any bash-specific features and only calls commands that are installed on any functioning UNIX-compatible system. This script as well as the archives it generates should run on any Unix flavor, with any compatible Bourne shell, provided of course that the compression programs are available.
Finally, the makeself package itself comes as a self-extracting script called makeself.run.
Add a Binary Payload to your Shell Scripts
GNU sharutils:
http://www.gnu.org/software/sharutils/
is a toolset for creating shell archives, and provides some additional features that may be helpful (such as checksums to ensuring that the payload is not damaged in transit).
Protecting against malicious modifications is not really feasible when the final product has to be interpretable by the shell - anyone who understood the generation technique could modify the checksum as well.
There are also other/commercial software installer builder (like InstallAnywhere) they basically have their own version of shar/makeself.
Netbeans has their own installer engine, and part of it, which does the unpacking and launching is done in the NBI native launcher component: http://wiki.netbeans.org/NBINativeLaunchers
Creates a shell(script) archive for Linux/Unix/MacOS and native executable for Windows. You can use that tool for your own projects, also.
Related
I'm trying to execute an ansible playbook that has a task utilizing the unarchive module. Because I'm doing this on OSX, I need it to use gnu-tar, instead of the bsd tar that typically comes with OSX, since BSD tar is not officially supported.
I've installed gnu-tar using brew, and placed this package's gnubin folder ahead of other paths in the $PATH variable. However, my playbook still looks for tar in the /usr/bin folder (the location of the original tar), instead of the newly installed gnu-tar.
My question is, how does the unarchive module look for the tar binary, and what would be the best way for me to override this behavior so that it utilizes gnu-tar, instead?
I've found the solution.
Before we proceed, this is my code for making sure that the brew-installed gnu-tar takes precedence over the bsd-tar that comes with OSX:
# gnu-tar
export PATH="/usr/local/opt/gnu-tar/libexec/gnubin:$PATH"
export MANPATH="/usr/local/opt/gnu-tar/libexec/gnuman:$MANPATH"
This code was originally located in my ~/.bash_profile file.
The way that ansible looks up the tar binary is as you'd expect: the $PATH variable, like any other process. However, running a play creates an interactive non-login shell. As explained here, the ~/.bashrc file gets loaded for interactive non-login shells, while ~/.bash_profile gets loaded for login shells.
Since my code was located in the ~/.bash_profile file, it never executes whenever I run a play. Again, this is because running a play creates an interactive non-login shell that does not load ~/.bash_profile. Hence, the code should actually be in ~/.bashrc.
True enough, the moment I moved my code to the ~/.bashrc was when my task started using the installed gnu-tar. I was able to confirm this by creating a task to register which tar into a variable and printing it in another debug task.
From the source code it looks like tar is looking for a gtar binary and if not a tar
# Prefer gtar (GNU tar) as it supports the compression options -zjJ
self.cmd_path = self.module.get_bin_path('gtar', None)
if not self.cmd_path:
# Fallback to tar
self.cmd_path = self.module.get_bin_path('tar')
Thus, aliasing your brew installed gnu tar as "gtar" or "tar" should fix the problem
I've tried multiple ways of creating a zip or a tar.gz on the mac using GUI or command lines, and I have tried decompressing on the Linux side and gotten various errors, from things like "File.XML" and "File.xml" both appearing in a directory, to all sorts of others about something being truncated, etc.
Without listing all my experiments on the command line on the Mac and Linux (using tcsh), what should 2 bullet proof commands be to:
1) make a zip file of a directory (with no __MACOSX folders)
2) unzip / untar (whatever) the Mac zip on Linux with no errors (and no __MACOSX folders)
IT staff on the Linux side said they "usually use .gz and use gzip and gunzip commands".
Thanks!
After much research and experimentation, I found this works every time:
1) Create a zipped tar file with this command on the Mac in Terminal:
tar -cvzf your_archive_name.tar.gz your_folder_name/
2) When you FTP the file from one server to another, make sure you do so with binary mode turned on
3) Unzip and untar in two steps in your shell on the Linux box (in this case, tcsh):
gunzip your_archive_name.tar.gz
tar -xvf your_archive_name.tar
On my Mac and in ssh bash I use the following simple commands:
Create Zip File (-czf)
tar -czf NAME.tgz FOLDER
Extract Zip File (-xzf)
tar -xzf NAME.tgz
Best, Mike
First off, the File.XML and File.xml cannot both appear in an HFS+ file system. It is possible, but very unusual, for someone to format a case-sensitive HFSX file system that would permit that. Can you really create two such files and see them listed separately?
You can use the -X option with zip to prevent resource forks and extended attributes from being saved. You can also throw in a -x .DS_Store to get rid of those files as well.
For tar, precede it with COPYFILE_DISABLE=true or setenv COPYFILE_DISABLE true, depending on your shell. You can also throw in an --exclude=.DS_Store.
Your "IT Staff" gave you a pretty useless answer, since gzip can only compress one file. gzip has to be used in combination with tar to archive a directory.
I have never used applescript before and I'm trying to find out how to zip a folder on the desktop, that's all and it's giving me a hard time
If you're happy to use Applescript to just invoke sh, you can use
do shell script "zip /Users/you/Desktop/out.zip /Users/you/Desktop/in.file"
do shell script "unzip -f /Users/you/out.zip"
(The -f option is "freshen", which will stop unzip from asking if you want to overwrite files. To always overwrite, use -o.)
I am trying to do this:
I want to call a make (Makefile exists in some other directory, abc path can be used) from a shell script located in a different directory. How do I do this?
Since shell scripting does not allow me to cd into the Makefile directory and execute make, how can I write the shell command (by giving path to the Makefile to be executed) to execute make?
GNU make accepts many options, notably -C to change directory before running, and -f for giving the Makefile to follow.
Combine them appropriately.
Consider using remake to ease debugging (notably with -x) of Makefile related issues. With GNU make version 4 or better, also use make --trace...
You could have your own executable shell script (e.g. in your $HOME/bin/ which would be in your $PATH) which uses both cd and make).
You could consider other build automation tools (ninja perhaps)
Read also P.Miller's paper Recursive Make considered harmful
I'm using a linux software solution that uses the tar command to backup huge amounts of data.
The command which is hardcoded into the binary which calls the tar is:
/bin/tar --exclude "/backup" --exclude / --ignore-failed-read -cvjf - /pbackup 2>>'/tar_err.log' | split -b 1000m - '/backup/temp/backup.tar.bz2'
There is no chance to change the command, as it is harcoded. It uses bzip2 to compress the data. I experienced a strong performance improvement (up to 60%) when using the parameter --use-compress-prog=pbzip2 which utilizes all CPU cores.
By symlinking the bzip2 from /bin/bzip2 to the pbzip2 binary I tried to trick the software, however when monitoring the process it still uses bzip2 as I tink this is built into tar.
I know it is a tricky question but is there any way to utilize pbzip2 without changing this command that is externally called?
My system is Debian Sequeeze.
Thanks very much!
Danger: ugly solution ahead; backup the binary before proceeding
First of all, check if the hardcoded string is easily accessible: use strings on your binary, and see if it displays the string you said (probably it will be in several pieces, e.g. /bin/tar, --exclude, --ignore-failed-read, ...).
If this succeeds, grab your hex editor of choice, open the binary and look for the hardcoded string; if it's split in several pieces, the one you need is the one containing /bin/tar; overwrite tar with some arbitrary three-letter name, e.g. fkt (fake tar; a quick Google search didn't turn up any result for /usr/bin/fkt, so we should be safe).
The program should now call your /usr/bin/fkt instead of the regular tar.
Now, put in your /bin a script like this:
#!/bin/sh
/bin/tar --use-compress-prog=pbzip2 $*
call it with the name you chose before (fkt) and set the permissions correctly (they should be 755 and owned by root). This script just takes all the parameters it gets and call the real tar, adding in front of them the parameter you need.
Another solution, that I suggested in the comments, may be creating a chroot just for the application, renaming tar to some other name (realtar, maybe?) and calling the script above tar (obviously now you should change the /bin/tar inside the script to /bin/realtar).
If the program is not updated very often and the trick worked at the first try I would probably go with the first solution, setting up and maintaining chroots is not fun.
Why not move /bin/tar to (say) /bin/tar-original
Then create a script /bin/tar to do whatever you want it to do.