How to dump the html file from nutch - nutch

Question

I'm new in nutch. i've crawled a lot of websites from internet. i want to get html content of segments. hence, i've dumped by follow command:
./nutch mergesegs crawl/merged crawl/segments/*
and then :
./nutch readseg -dump crawl/merged/* dumpedContent
now. i have 2 files at dumpedContent : dump and .dump.crc
the size of dump is too big(82GB).
how to dump each of original web pages in one file? or how to dump in small files?

Answer 1

You're getting a big file because you're merging the segments first with (mergesegs) you could try to dump each individual segment into it's own file.
At the moment the SegmentReader class doesn't support splitting each individual URL into a separated file, and not sure if that's something that we would like to support. For really big crawls this would definitively be a problem. Anyhow keep in mind that the -dump option always attach some metadata to the crawled URL, so you're not getting only the HTML content, but also some metadata. For example:
Recno:: 0
URL:: http://example.org
CrawlDatum::
Version: 7
Status: 33 (fetch_success)
Fetch time: Wed Oct 25 16:32:14 CEST 2017
Modified time: Thu Jan 01 01:00:00 CET 1970
Retries since fetch: 0
Retry interval: 2592000 seconds (30 days)
Score: 0.0
Signature: null
Metadata:
_ngt_=1508941926882
_repr_=http://example.org
_pst_=success(1), lastModified=0
_rs_=478
Content-Type=text/html
nutch.protocol.code=200
Content::
Version: -1
url: http://example.org
base: http://example.org
contentType: text/html
metadata: X-Cache=HIT Connection=close Date=Wed, 25 Oct 2017 14:30:53 GMT nutch.crawl.score=0.0 nutch.fetch.time=1508941934366 Accept-Ranges=bytes nutch.segment.name=20171025163209 Cache-Control=max-age=600 Content-Encoding=gzip Vary=Accept-Encoding,Cookie Expires=Wed, 25 Oct 2017 14:40:53 GMT Content-Length=20133 X-Cache-Hits=1 _fst_=33 Age=78 Content-Type=text/html; charset=UTF-8
Content:
...
So you'll need to process this files to get the raw HTML.
Other option is indexing the content into Solr/ES with the -addBinaryContent flag and you'll have the raw content of the page stored in Solr/ES. The advantage here is that you can query for specific URLs. So you could extract the data from Solr/ES into whatever format/files that you want.
Yet another option is using the bin/nutch commoncrawldump feature, that will allow you to output the content into a different format, not sure now if is possible to do the 1 URL to 1 file relation.

Answer 2

Nutch SegmentReader is a good way to dump all your HTML content into one file. This generally leads to storing the HTML content from your starting URL(and their inlinks and outlinks as well).
However, if you need them to be parsed and stored separately, you may want to look into writing plugins. You may define where and what to store based on what is parsed. I recently tried this and it is efficient when it comes to storing separate HTML pages in a directory folder. Hope this helps.