How to fix issue with nutch readseg not dumping any content? - nutch

Question

I am using Nutch 1.4 local on iOS, to crawl a website and nutch
readseg dump does not return any relevant information.
What am I missing?
I am trying to extract 'category' as new metadata from url.
I am using replace to extract substring from the url. I am able to
run the code and index the documents in Google Cloud Search. But
it is not capturing the category.
To debug this end to end I like to verify that the correct value
is extracted by nutch in category metadata. I verified that regex
is correct with a regex tester. I want to log metadata
values - url, category in the log or stdout. I so not see any
pertinent information in hadoop.log even in DEBUG.
$bin/nutch readseg -dump TestCrawl/segments/* segmentAllContent
SegmentReader: dump segment: TestCrawl/segments/20190128171825
SegmentReader: done
logs/hadoop.log -
2019-01-29 11:40:02,275 INFO segment.SegmentReader -
SegmentReader:
dump segment: TestCrawl/segments/20190128171825 .
2019-01-29 11:40:02,463 WARN util.NativeCodeLoader - Unable to
load
native-hadoop library for your platform... using builtin-java
classes where applicable.
log4j.properties
log4j.logger.org.apache.nutch=DEBUG
nutch-site.xml
<property>
<name>index.replace.regexp</name>
<value>
urlmatch=.*mycompany\.com\/([a-zA-Z0-9-]+)
url:category=$1
</value>
</property>
<property>
<name>urlmeta.tags</name>
<value>title,category</value>
<description>
test
</description>
</property>
<property>
<name>index.parse.md</name>
<value>*</value>
<description> test </description
</property>

Answer 1

The readseg -dump command only writes everything contained in the segment as plain text to the output directory segmentAllContent. It does not run the indexer and consequently does not call the plugin index-replace. You may use the command bin/nutch indexchecker to check whether the plugin is configured properly.
Please note that the plugin index-replace is not available in Nutch 1.4, it has been added with Nutch 1.11.
Example how to use the indexchecker to check the index-replace plugin:
% bin/nutch indexchecker \
-Dplugin.includes='protocol-okhttp|parse-html|index-(basic|replace|static)' \
-Dindexingfilter.order='org.apache.nutch.indexer.basic.BasicIndexingFilter org.apache.nutch.indexer.staticfield.StaticFieldIndexer org.apache.nutch.indexer.replace.ReplaceIndexer' \
-Dindex.static='category:unknown' \
-Dindex.replace.regexp=$'hostmatch=localhost\ncategory=/.+/intranet/' \
http://localhost/
...
host : localhost
id : http://localhost/
title : Apache2 Ubuntu Default Page: It works
category : intranet
url : http://localhost/
...
the plugin index-static is configured to add a field "category" with value "unknown"
the plugin index-replace changes the value to "intranet" if the hostname is "localhost" (the $'...' notation is expands \n)