How do i fix the error in replace regex flag component? - nutch

Question

I am using nutch 1.14 since I am using GCS indexer. Here is what I have in nutch-site.xml
<property>
<name>index.replace.regexp</name>
<value>
urlmatch=.*example.com\/[a-zA-Z0-9-]+
url:category=/https:\/\/www.example.com\/([a-zA-Z0-9-]+)/$1/
</value>
</property>
I am getting the error:
$ grep 'replace' logs/hadoop.log
ERROR replace.ReplaceIndexer - Pattern
url:category=/https:\/\/www.example.com\/([a-zA-Z0-9-]+)/$1/, has invalid flags component
I get the same when i change the line in nutch-site.xml to:
-url:category=/https:\/\/www.mydomain.com\/([a-zA-Z0-9-]+)/$1/2
I want to get part of url in category. Ex: If url is https://www.example.com/testcategory , i like category to be testcategory
Thanks.

Answer 1

If I understand correctly you want to get what is after the domain (example.com) and place it in a category field, right?
In that case, you have an error in your regex. You want to capture everything after example.com/<category> then you need to configure your urlmatch like:
urlmatch=.*example\.com\/([a-zA-Z0-9-]+)
In this case ([a-zA-Z0-9-]+) will create a capture group accessible through $1. And then you can set the field like:
url:category=$1
This would get what was captured with the capture group and place it a category field.
You can test the regex in: https://regex101.com/r/bMLqOq/1.