generate image (e.g. jpg) of a web page? - linux

I want to create an image what a web page looks like,
e.g. create a small thumbnail of the html + images.
it does not have to be perfect (e.g. flash /javascript rendering).
I will call use the code on linux, ideally would be some java library, but a command line tool would be cool as well.
any ideas?

Try CutyCapt, a command-line utility. It uses Webkit for rendering and outputs in various formats (SVG, PNG, etc.).

you can get it nearly perfect, and cross platform too, by using a browser plugin.
FireShot or ScreenGrab for Firefox.
6 Google Chrome Screenshot Webpage Capture Extensions
BrowserShots is an open source project that may have some code you can use.
also see:
Command line program to create website screenshots (on Linux)
Convert web page to image
How to take screenshot of whole web page, rather than what shows on the screen
What is the best way to create a web page thumbnail?
Convert HTML to an image

To take a screenshot in the terminal with ImageMagick, type the following line into a terminal and then click-and-drag the mouse over a section of the screen:
import MyScreenshot.png
To capture the entire screen and after some delay and resize it, use the following command:
import -window root -resize 400×300 -delay 200 screenshot.png
You may use a mixture of xwininfo and import to retrieve the window id of the browser and make a screenshot of that window. A bash script to automate this process would be something like this:
#!/bin/bash
window_id=`xwininfo -tree -root | grep Mozilla | awk '{print $1}'`
import -window $window_id -resize 100x100 tumb.png
This script will create a 100x100 screenshot of Firefox on the current directory under the name tumb.png
Several sources show how to run a bash script from inside a Java application, google can help you on that. If you are in a hurry, check this and this.

After reading this page, I was thinking, let me fire up midori browser: http://midori-browser.org/ and when I tried the -h option, I have seen:
-s, --snapshot Take a snapshot of the specified URI
QutyCapt is difficult to compile, and has many dependencies. Midori has it less. It outputs the PNG of the website into TMP folder. One can get the file with:
midori -s http://www.rcdwealth.com new.png 2>/dev/null | awk '{ print $4}'
After that, the file can be converted to thumbnail size by using ImageMagick's convert program.

If you're interested in Java, maybe you could look at browser automation using Selenium-RC http://seleniumhq.com
It's a little java server that you can install on the box and the program itself will execute remote commands in a web browser.
Steps like (this is pseudo code by the way, I code my Selenium in php and I can't recall 100% of the specifics off the top of my head)
selenium.location("http://foo.com")
selenium.open("/folder/sub/bar.html")
selenium.captureScreenshot("/tmp/" + this.getClass().getName() + "."
+ testMethodName + ".png");
Actually, I just did a quick websearch for the exact syntax on that last one ... and this guy has a blog with what might actually be working code in java :)
https://dev.youdevise.com/YDBlog/index.php?title=capture_screenshots_of_selenium_browser_&more=1&c=1&tb=1&pb=1
There's also a number of websites that provide this service "cross browser and OS" I just can't recall what they are. Basically they've got a cloud of every single operating system and browser combination, and they log on with each machine, take a screen and store it on their site for you to come back to in a few hours when they're done.
Ahh... another websearch and it's yours :) http://browsershots.org/

Related

Name screen session log using session name

I'm using this very simple .screenrc:
logtstamp on
logfile /tmp/screenlog-%S.log
I tried launching screens with these two methods:
screen -L -S testing
screen -S tester -L
but the filename used is /tmp/screenlog.0S.log. What am I doing wrong? Using Screen version 4.00.03jw4 (FAU) 2-May-06, and according to the manual I should be able to name the log file using the session name
If you look at the man page (man screen) for your (8-year-old?) version of screen, you'll see it's missing the %S specifier. They must have added it since your version. I'm not sure why Ubuntu 12.04 shipped screen from 2006..
P.S. I'd advocate looking into tmux. It's a little bit harder to learn, but a lot more flexible: You can move windows between sessions, You can see multiple windows at once, You can nest sessions inside of other sessions, etc.
Also, if you are just looking to log the output of long-running processes, take a look at nohup.

Trying to extract field from browser page

I'm trying to extract one field from Firefox to my local Ubuntu 12.04 PC and Mac OS 19.7.4 from an online form
I can manually save the page locally as a text document and then search for the text using Unix script but this seems rather cumbersome & I require it to be automated. Is there another more efficient method?
My background is on Macs but the company is trialling Linux PC's, so please be tolerant of my relevant Ubuntu ignorance.
If you mean to program something try
WWW:Mechanize library, it have python and perl bindings,
several mousescripting engines in lunux, (actionaz)
test automation tool which works with firefox (Selenium)
You can do it by simple BASH script.
Take a look at some useful stuff like:
wget
sed
grep
and then nothing will by cumbersome and everything can go automatic.
If you want to go with the method that you mentioned, you can use curl to automate the saving of the form. Your BASH script would then look something like this:
curl http://locationofonlineform.com -o tempfile
valueOfField=$(grep patternToFindField tempfile)
// Do stuff
echo $valueOfField
If you want to get rid of the temporary file, you can directly feed the result of curl into the grep command.

Chrome on Linux - query the browser to see what tabs are open?

I am running Chromium (the open source chrome version) on Ubuntu Linux. Can I write a programme to see what tabs I have open? I would like to write a programme to monitor how much time I'm spending on things. Is there a command line programme, some way to invoke the chromium-browser command, or some dbus incantation that will tell me what tabs I have open and what URL each tab is at?
Chrome on Linux - query the browser to see what tabs are open?
For chromium :
strings ~/'.config/chromium/Default/Current Session' | 'grep' -E '^https?://'
Indeed there is a command line option which can open the door to a running chrome (chromium) process --remote-shell-port. Through this "debugging back-door" you may be able the get the list of open tabs.
Look at chromedevtools for further inspiration.
UPDATE:
Chrome DevTools is deprecated and not supported anymore since Version >17.0.950.*
See WebKit-Protocol manual if the new Debug-Framework provides similar manners to accomplish the task.
Here is a more general solution (works with other applications as well) by querying the X window under focus using xdotool
while true; do
xdotool getwindowfocus getwindowname;
sleep 10;
done
This outputs the following for instance:
Tilix: Defaultpeter-ThinkPad-T5801: peter#peter-ThinkPad-T580: ~
Chrome on Linux - query the browser to see what tabs are open? - Stack Overflow - Google Chrome
Local KVM
untitled — Atom
untitled — Atom
Open File
iostat_xtmz_3.out — ~/Work/KappAhl/Test1 — Atom
Tilix: Defaultpeter-ThinkPad-T5801: peter#peter-ThinkPad-T580: ~*
An expansion on the unix command above (I don't have enough reputation to comment). I was trying to just get a count of tabs. This still isn't perfect because I think the file has the entire history of all tabs in it. I guess they are in order, but not obvious how to separate them.
strings ~/Library/Application\ Support/Google/Chrome/Default/Sessions/Tabs_* | sed -nE 's/^([^:]+):\/\/(.*)\/$/\2/p' | grep -v "newtab" | grep -v "new-tab-page" | sort | uniq | wc -l
This is on mac, so your paths and sed options may vary.
The basic idea is to get rid of trailing slashes (lots of redirects just add a slash) and newtabs so we can get an accurate count. For my current tabs file this went from 181 tabs open down to a count of 35. That actually looks like an undercount right now, but it is a lot closer.
I have written a tool to extract data from chrome session files for precisely this purpose. https://github.com/lemnos/chrome-session-dump. Running it like so chrome-session-dump will produce a list of tabs (in order) which can subsequently be passed to firefox. E.G chrome-session-dump|xargs firefox. You can also obtain the currently open tab via -active for processing by external scripts.

How to download images from "wikimedia search result" using wget?

I need to mirror every images which appear on this page:
http://commons.wikimedia.org/w/index.php?title=Special:Search&ns0=1&ns6=1&ns12=1&ns14=1&ns100=1&ns106=1&redirs=0&search=buitenzorg&limit=900&offset=0
The mirror result should give us the full size images, not the thumbnails.
What is the best way to do this with wget?
UPDATE:
I update the solution below.
Regex is your friend my friend!
Using cat, egrep and wget youll get this task done pretty fast
Download the search results URI wget, then run
cat DownloadedSearchResults.html | egrep (?<=class="searchResultImage".+href=").+?\.jpg/
That should give you http://commons.wikimedia.org/ based links to each of the image's web page. Now, for each one of those results, download it and run:
cat DownloadedSearchResult.jpg | egrep (?<=class="fullImageLink".*href=").+?\.jpg
That should give you a direct link to the highest resolution available for that image.
Im hoping your bash knowledge will do the rest. Good Luck.
Came here with the same problem .. found this >> http://meta.wikimedia.org/wiki/Wikix
I don't have access to a linux machine now, so I didn't try it yet.
It is quite difficult to write all the script in stackoverflow editor, you can find the script at the address below. The script only downloads all images at the first page, you can modify it to automate download process in another page.
http://pastebin.com/xuPaqxKW

Command line program to create website screenshots (on Linux) [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
What is a good command line tool to create screenshots of websites on Linux? I need to automatically generate screenshots of websites without human interaction. The only tool that I found was khtml2png, but I wonder if there are others that aren't based on khtml (i.e. have good JavaScript support, ...).
A little more detail might be useful...
Start a firefox (or other browser) in an X session, either on your console or using a vncserver. You can use the --height and --width options to set the size of the window to full screen. Another firefox command can be used to set the URL being displayed in the first firefox window. Now you can grab the screen image with one of several commands, such as the "import" command from the Imagemagick package, or using gimp, or fbgrab, or xv.
#!/bin/sh
# start a server with a specific DISPLAY
vncserver :11 -geometry 1024x768
# start firefox in this vnc session
firefox --display :11
# read URLs from a data file in a loop
count=1
while read url
do
# send URL to the firefox session
firefox --display :11 $url
# take a picture after waiting a bit for the load to finish
sleep 5
import -window root image$count.jpg
count=`expr $count + 1`
done < url_list.txt
# clean up when done
vncserver -kill :11
Try nice small tool CutyCapt, which depends only on Qt and QtWebkit. ;)
Have a look at PhantomJS, which seems to be a free scritable Webkit engine that runs on Linux, OSX and Windows. I've not used it since we currently use Browshot (commercial solution), but when all our credits run out, we will seriously have a loot at it (since it's free and can run on our servers)
scrot is a command line tool for taking screenshots. See the man page and this tutorial.
You might also want to look at scripting the browser. There are firefox add-ons that take screenshots such as screengrab (which can capture the entire page if you want, not just the visible bit) and you could then script the browser with greasemonkey to take the screenshots.
See Webkit2png.
I think this is what I used in the past.
Edit I discover I haven't used the above, but found this page with reviews of many different programs and techniques.
I know its not a command line tool but you could easily script up something to use http://browsershots.org/ Not that useful for applications not hosted on external IPs.
A great tool none the less.
I don't know of anything custom built, I'm sure there could be something done with the gecko engine to render to a png file instead of the screen ...
Or, you could fire up firefox in full screen mode in a dedicated VNC server instance and use a screenshot grabber to take the screenshot. Fullscreen = minimal chrome, VNC server instance = no visible UI + you can choose your resolution.
Use xinit with Xvnc as the X server to do this - you'll need to read all the manpages.
Downsides are that the screenshot is always the same size, doesn't resize according to the web page ...
There is the import command, but you'll need X, and a little bash script that open the browser window, then take the screenshot and close the browser.
You can find more information here, or just typing import --help in a shell ;)
http://khtml2png.sourceforge.net/
The deb file
http://sourceforge.net/projects/khtml2png/files/khtml2png2/2.7.6/khtml2png_2.7.6_i386.deb/download
worked on my Ubuntu after installing libkonq4 ... but you may have to cover other dependencies.
I think javascript support may be better now!
Stephan
Not for the command line but at least for usage in batch operation for a larger set of urls you may use firefox with its addon fireshot (licensed version?).
Open tabs for all urls in your set (e.g. "open tabs for all bookmarks in this folder...").
Then in fireshot launch "Capture all tabs"
In the edit window then call "select all shots -> save all shots"
Having set the screenshot properties (size, fileformat, etc.) before you end with a nice set of shotfiles.
Steffen

Resources