How to make nutch index only pages with certain text? - nutch

Question

I have 2 requirements.
First one is that I want Nutch to index only pages that contain certain words in the html. For example I only want nutch to index pages that contain "wounderful" word in the html.
Second one is that I want nutch to index certain URLs from the site. For example I want nutch to index URLs that are similar to "mywebsite.com/XXXX/ABC/XXXX" or "mywebsite.com/grow.php/ABC/XXXX", where "XXXX" can be any word of any length.
This is the content of my seed.txt file
http://mysite.org/
this is the content of my regex-urlfilter.txt
+^http://mysite.org/work/.*?/text/
I have commented
#+.
By I am still getting below error
crawl started in: crawl
rootUrlDir = bin/urls
threads = 10
depth = 3
solrUrl=http://localhost:8983/solr/
topN = 5
Injector: starting at 2013-07-09 11:05:51
Injector: crawlDb: crawl/crawldb
Injector: urlDir: bin/urls
Injector: Converting injected urls to crawl db entries.
Injector: total number of urls rejected by filters: 1
Injector: total number of urls injected after normalization and filtering: 0
Injector: Merging injected urls into crawl db.
Injector: finished at 2013-07-09 11:06:08, elapsed: 00:00:17
Generator: starting at 2013-07-09 11:06:08
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 5
Generator: jobtracker is 'local', generating exactly one partition.
Generator: 0 records selected for fetching, exiting ...
Stopping at depth=0 - no more URLs to fetch.
No URLs to fetch - check your seed list and URL filters.
crawl finished: crawl

Answer 1

Start here to setup your desired URL pattern. Then look into plugins to parse your content and decide what should be indexed.

Answer 2

It shows Injector rejects your url in seed file
Injector: total number of urls rejected by filters: 1
Your regex is not working or there will be any other patterns which rejects your url like -.*(/[^/]+)/[^/]+\1/[^/]+\1/ or -[?*!#=]

Answer 3

Know this is pretty old, but just wanted to add my two cents to the topic related the crawling vs. indexing filter, for nutch-1.13
regex-urlfilter testing
If you want to test your regex-urlfilter.txt expressions you can use the plugin testing like this
$ bin/nutch plugin urlfilter-regex org.apache.nutch.urlfilter.regex.RegexURLFilter
This will give no feedback, but if you type urls and press enter, you'll see an echo of it, with a '-' or '+' prefix, telling you if the url passes the configuration filter.
such as
http://aaa.com
-http://aaa.com
http://bbb.com
+http://bbb.com
if config is something like
+^http://bbb.com\.*
-.*
crawling filter vs. index filter
This is not well documented, and took me a while to found a clue.
If we want to make different filtering precision (broad on crawling, but more detailed on indexing) we can do the following.
First, if we're using the bin/crawl script, just add
a -filter option at the end of the filtering command
the param that specifies the regex file to use; -Durlfilter.regex.file)
like this
< __bin_nutch index $JAVA_PROPERTIES "$CRAWL_PATH"/crawldb -linkdb "$CRAWL_PATH"/linkdb "$CRAWL_PATH"/segments/$SEGMENT
> __bin_nutch index $JAVA_PROPERTIES -Durlfilter.regex.file=regex-urlfilter-index.txt "$CRAWL_PATH"/crawldb -linkdb "$CRAWL_PATH"/linkdb "$CRAWL_PATH"/segments/$SEGMENT -filter
Otherwise, just append both parameters to the bin/nutch index command, if you're using them without the crawl script
And now, enter the desired configuration in the 'regex-urlfilter-index.txt' file.
Thanks to Arthurs question in grokbase for the insight:
http://grokbase.com/t/nutch/user/1579evs40h/filtering-at-index-time-with-a-different-regex-urlfilter-txt-from-crawl