Release Notes - ASF JIRA

Release Notes - Nutch - Version 1.13 - HTML format

Configure Release Notes

Sub-task

[NUTCH-2246] - Refactor /seed endpoint for backward compatibility

Bug

[NUTCH-1553] - Property 'indexer.delete.robots.noindex' not working when using parser-html.
[NUTCH-2242] - lastModified not always set
[NUTCH-2291] - Fix mrunit dependencies
[NUTCH-2337] - urlnormalizer-basic to strip empty port
[NUTCH-2345] - FetchItemQueue logs are logged with wrong class name
[NUTCH-2349] - urlnormalizer-basic NPE for ill-formed URL "http:/"
[NUTCH-2357] - Index metadata throw Exception because writable object cannot be cast to Text
[NUTCH-2359] - Parsefilter-regex raises IndexOutOfBoundsException when rules are ill-formed
[NUTCH-2364] - http.agent.rotate: IllegalArgumentException / last element of agent names ignored
[NUTCH-2366] - Deprecated Job constructor in hostdb/ReadHostDb.java

New Feature

[NUTCH-2132] - Publisher/Subscriber model for Nutch to emit events

Improvement

[NUTCH-1308] - Add main() to ZipParser
[NUTCH-2164] - Inconsistent 'Modified Time' in crawl db
[NUTCH-2234] - Upgrade to elasticsearch 2.3.3
[NUTCH-2236] - Upgrade to Hadoop 2.7.2
[NUTCH-2262] - Utilize parameterized logging notation across Fetcher
[NUTCH-2272] - Index checker server to optionally keep client connection open
[NUTCH-2286] - CrawlDbReader -stats to show fetch time and interval
[NUTCH-2287] - Indexer-elastic plugin should use Elasticsearch BulkProcessor and BackoffPolicy
[NUTCH-2299] - Remove obsolete properties protocol.plugin.check.*
[NUTCH-2300] - Fetcher to optionally save robots.txt
[NUTCH-2327] - Seeds injected in REST workflow must be ingested into HDFS
[NUTCH-2329] - Update Slf4j logging for Java 8 and upgrade miredot plugin version
[NUTCH-2336] - SegmentReader to implement Tool
[NUTCH-2352] - Log with Generic Class Name at Nutch 1.x
[NUTCH-2355] - Protocol plugins to set cookie if Cookie metadata field is present
[NUTCH-2367] - Get single record from HostDB

Task

[NUTCH-2171] - Upgrade Nutch Trunk to Java 1.8

Edit/Copy Release Notes

The text area below allows the project release notes to be edited and copied to another document.

Release Notes - Nutch - Version 1.13
    
<h2>        Sub-task
</h2>
<ul>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2246'>NUTCH-2246</a>] -         Refactor /seed endpoint for backward compatibility
</li>
</ul>
            
<h2>        Bug
</h2>
<ul>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-1553'>NUTCH-1553</a>] -         Property &#39;indexer.delete.robots.noindex&#39; not working when using parser-html.
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2242'>NUTCH-2242</a>] -         lastModified not always set
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2291'>NUTCH-2291</a>] -         Fix mrunit dependencies
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2337'>NUTCH-2337</a>] -         urlnormalizer-basic to strip empty port
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2345'>NUTCH-2345</a>] -         FetchItemQueue logs are logged with wrong class name
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2349'>NUTCH-2349</a>] -         urlnormalizer-basic NPE for ill-formed URL &quot;http:/&quot;
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2357'>NUTCH-2357</a>] -         Index metadata throw Exception because writable object cannot be cast to Text
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2359'>NUTCH-2359</a>] -         Parsefilter-regex raises IndexOutOfBoundsException when rules are ill-formed
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2364'>NUTCH-2364</a>] -         http.agent.rotate: IllegalArgumentException / last element of agent names ignored
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2366'>NUTCH-2366</a>] -         Deprecated Job constructor in hostdb/ReadHostDb.java
</li>
</ul>
            
<h2>        New Feature
</h2>
<ul>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2132'>NUTCH-2132</a>] -         Publisher/Subscriber model for Nutch to emit events 
</li>
</ul>
    
<h2>        Improvement
</h2>
<ul>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-1308'>NUTCH-1308</a>] -         Add main() to ZipParser
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2164'>NUTCH-2164</a>] -         Inconsistent &#39;Modified Time&#39; in crawl db
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2234'>NUTCH-2234</a>] -         Upgrade to elasticsearch 2.3.3
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2236'>NUTCH-2236</a>] -         Upgrade to Hadoop 2.7.2
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2262'>NUTCH-2262</a>] -         Utilize parameterized logging notation across Fetcher
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2272'>NUTCH-2272</a>] -         Index checker server to optionally keep client connection open
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2286'>NUTCH-2286</a>] -         CrawlDbReader -stats to show fetch time and interval
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2287'>NUTCH-2287</a>] -         Indexer-elastic plugin should use Elasticsearch BulkProcessor and BackoffPolicy
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2299'>NUTCH-2299</a>] -         Remove obsolete properties protocol.plugin.check.*
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2300'>NUTCH-2300</a>] -         Fetcher to optionally save robots.txt
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2327'>NUTCH-2327</a>] -         Seeds injected in REST workflow must be ingested into HDFS
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2329'>NUTCH-2329</a>] -         Update Slf4j logging for Java 8 and upgrade miredot plugin version
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2336'>NUTCH-2336</a>] -         SegmentReader to implement Tool
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2352'>NUTCH-2352</a>] -         Log with Generic Class Name at Nutch 1.x
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2355'>NUTCH-2355</a>] -         Protocol plugins to set cookie if Cookie metadata field is present
</li>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2367'>NUTCH-2367</a>] -         Get single record from HostDB
</li>
</ul>
            
<h2>        Task
</h2>
<ul>
<li>[<a href='https://issues.apache.org/jira/browse/NUTCH-2171'>NUTCH-2171</a>] -         Upgrade Nutch Trunk to Java 1.8
</li>
</ul>