All Classes and Interfaces
Class
Description
Scaffolding class for the various Checker implementations.
Abstract class that implements { @see org.apache.nutch.tools.CommonCrawlFormat } interface.
This class provides common methods for implementations of
FetchSchedule.This class implements an adaptive re-fetch algorithm.
URLNormalizer capable of dealing with AJAX URL's.
Indexing filter that offers an option to either index all inbound anchor text
for a document or deduplicate anchors.
Adds arbitrary searchable fields to a document from the class and method
the user identifies in the config.
A input format the reads arc files.
The
ArchRecordReader class provides a record reader which reads
records from arc files.
The
ArcSegmentCreator is a replacement for fetcher that will
take arc files as input and produce a nutch segment as output.RegexURLFilterBase implementation based on the dk.brics.automaton Finite-State
Automata for JavaTM.
Adds basic searchable fields to a document.
Converts URLs to a normal form:
remove dot segments in path:
/./ or /../
remove default ports, e.g.A decorator to Metadata that adds for case-insensitive lookup of keys.
Adds basic searchable fields to a document.
Adds metadata identifying the Creative Commons license used, if any.
Walks DOM tree, looking for RDF in comments and licenses in anchors.
Parse a CIDR block
notation and test whether an IP address is contained in the subnet range
defined by the CIDR.
CircularDependencyException will be thrown if a circular
dependency is detected.The class scans CrawlDB looking for entries with status DB_GONE (404) or
DB_DUPLICATE and sends delete requests to indexers for those documents.
Client.java encapsulates functionalities necessary for nutch to get dir list
and retrieve file from an FTP server.
Writes documents to CloudSearch.
The Common Crawl Data Dumper tool enables one to reverse generate the raw
content from Nutch segment data directories into a common crawling data
format, consumed by many applications.
Interface for all CommonCrawl formatter.
Factory class that creates new
CommonCrawlFormat objects (a.k.a.This class provides methods to map crawled data on JSON using Jackson Streaming APIs.
This class provides methods to map crawled data on JSON using Jettinson APIs.
This class provides methods to map crawled data on JSON using a StringBuilder object.
An input format that takes Nutch Content objects and converts them to text
while converting newline endings to spaces.
Extracts some simple crawl completion stats from the crawldb
Stats will be sorted by host/domain and will be of the form:
1 www.spitzer.caltech.edu FETCHED
50 www.spitzer.caltech.edu UNFETCHED
A Comparator optimized for CrawlDatum.
These are instantiated once for each host.
This class takes the output of the fetcher and updates the crawldb
accordingly.
This class provides a way to separate the URL normalization and filtering
steps from the rest of CrawlDb manipulation code.
This tool merges several CrawlDb-s into one, optionally filtering URLs
through the current URLFilters, to skip prohibited pages.
Read utility for the CrawlDB.
Merge new page entries with existing entries.
A collection of Creative Commons properties names.
Write Nutch documents to a CSV file (comma separated values), i.e., dump
index as CSV or tab-separated plain text table.
Adds serialized DOM to parse data, useful for debugging, to understand how
the parser implementation interprets a document (not only HTML).
Generic deduplicator which groups fetched URLs with the same digest and marks
all of them as duplicate except the one with the highest score (based on the
score in the crawldb, which is not necessarily the same as the score
indexed).
Combine multiple new entries for a url.
This is a placeholder/example of a technique or use case where we do multiple
interaction with the web driver and need data from each such interaction in the end.
This class implements the default re-fetch schedule.
A collection of utility methods for working on deflated data.
This scoring filter limits the number of hops from the initial seed urls.
Utility that converts DMOZ
RDF into a flat file of URLs to be injected.
Filters URLs based on a file containing domain suffixes, domain names, and
hostnames.
Extracts some very basic statistics about domains from the crawldb
Filters URLs based on a file containing domain suffixes, domain names, and
hostnames.
This class takes SAX events (in addition to some extra events that SAX
doesn't handle yet) and adds the result to a document or document fragment.
A collection of methods for extracting content from DOM trees.
A collection of methods for extracting content from DOM trees.
A collection of Dublin Core metadata names.
DummyIndexWriter.
Sends NutchDocuments to a configured Elasticsearch index.
A simple class for detecting character encodings.
A utility class for tracking errors by category with automatic classification.
Error type categories for classification.
This implementation of
URLExemptionFilter
uses regex configuration to check if URL is eligible for exemption from
the db.ignore.external.links configuration property.An
Extension is a kind of listener descriptor that will be
installed on a concrete ExtensionPoint that acts as kind of
Publisher.The
ExtensionPoint provide meta information of a extension
point.A wrapper that invokes external command to do real parsing job.
Filters URLs based on a file of regular expressions using host/domains
matching first.
Rule for
DenyPath .* or DenyPath .?A collection of Feed property names extracted by the ROME library.
A RSS/ATOM Feed
Parser that rapidly parses all referenced links and
content present in the feed.A queue-based fetcher.
Reducer that passes through (url, datum) records and merges TDigests from
map tasks to set job-level latency percentile counters.
Splits FetcherOutput entries into multiple map files.
This class picks items from queues and fetches the pages.
This class is used to capture the various events occurring
at fetch time.
Type of event to specify start, end or reporting of a fetch item.
This class handles the publishing of the events to the queue implementation.
This class describes the item to be fetched.
This class handles FetchItems which come from the same host ID (be it a
proto/hostname or proto/IP pair).
A collection of queues that keeps track of the total number of items, and
provides items eligible for fetching from any queue.
Simple custom crawl datum processor that counts the number of records that
are overdue for fetching, e.g.
This interface defines the contract for implementations that manipulate fetch
times and re-fetch intervals.
Creates and caches a
FetchSchedule implementation.POJO to store a filename, its match pattern and its replacement string.
This class is a protocol plugin used for file: scheme.
The file dumper tool enables one to reverse generate the raw content from
Nutch segment data directories.
Thrown for File error codes.
FileResponse.java mimics file replies as http response.
This tool generates fetchlists (segments to be fetched) from plain text files
containing one URL per line.
Utility methods for common filesystem operations.
This class is a protocol plugin used for ftp: scheme.
Thrown for Ftp error codes.
Superclass for important exceptions thrown during FTP talk, that must be
handled with care.
Exception indicating bad reply of SYST command.
Exception indicating failure of opening data connection.
Exception indicating control channel is closed by server end, due to forced
closure of data channel at client (our) end.
Exception indicating unrecognizable reply from server after forced closure of
data channel by client (our) side.
FtpResponse.java mimics ftp replies as http response.
This class is used for parsing robots for urls belonging to FTP protocol.
Generates a subset of a CrawlDb to fetch.
Update the CrawlDB so that the next generate won't include the same URLs.
Sort fetch lists by hash of URL.
Selects entries due for fetch.
Select and invert subset due for fetch.
Collect until limit is reached.
A generic Writable wrapper that can inject Configuration to
Configurables
Simple utility class which builds a
NutchDocument based on input from
GeoIPIndexingFilter, where configuration is also read.This plugin implements an indexing filter which takes advantage of the GeoIP2-java API.
Supported GeoIP database types.
A collection of utility methods for working on GZIPed data.
HtmlParseFilter to retrieve h1 and h2 values from the DOM.
URL normalizer for mapping hosts to their desired form.
Class for parsing META Directives from DOM trees.
Class for parsing META Directives from DOM trees.
This class holds the information about HTML "meta" tags extracted from a
page.
Extension point for DOM-based HTML parsers.
Creates and caches
HtmlParseFilter implementing plugins.
This class is a protocol plugin that configures an HTTP client for Basic,
Digest and NTLM authentication schemes for web server as well as proxy
server.
The base level of services required for Http Authentication
Can be used to identify problems during creation of Authentication objects.
Provides the Http protocol implementation with the ability to authenticate
when prompted.
Implementation of RFC 2617 Basic Authentication.
Parse and format HTTP dates in HTTP headers, e.g., used to fill the
"If-Modified-Since" request header field.
A collection of HTTP header names.
An HTTP response.
An HTTP response.
An HTTP response.
This class is used for parsing robots for urls belonging to HTTP protocol.
This class is typically invoked from within
IndexingJob and handles all MapReduce
functionality required when undertaking indexing.Extension point for indexing.
Creates and caches
IndexingFilter implementing plugins.Reads and parses a URL and run the indexers on it.
Generic indexer which relies on the plugins implementing IndexWriter
Creates and caches
IndexWriter implementing plugins.Injector takes a flat text file of URLs (or a folder containing text files)
and merges ("injects") these URLs into the CrawlDb.
InjectMapper reads
the CrawlDb seeds are injected into
the plain-text seed files and parses each line into the URL and
metadata.
Combine multiple new entries for a url.
An incoming link to a page.
A list of
Inlinks.Optionally limit or block connections to IP address ranges
(localhost/loopback or site-local addresses, subnet ranges given in CIDR
notation, or single IP addresses).
An
IndexingFilter that allows filtering of
documents based on a JEXL expression.Utility methods for handling JEXL expressions used in crawl and index
pipelines.
This class is a heuristic link extractor for JavaScript files and code
snippets.
Sends Nutch documents to a configured Kafka Cluster
An
IndexingFilter that add a
lang (language) field to the document.A utility class for tracking latency metrics using TDigest for percentile
calculation.
A class for holding link information including the url, anchor text, a score,
the timestamp of the link and a link type.
Maintains an inverted link map, listing incoming links for each url.
This class provides a way to separate the URL normalization and filtering
steps from the rest of LinkDb manipulation code.
This tool merges several LinkDb-s into one, optionally filtering URLs through
the current URLFilters, to skip prohibited URLs and links.
Read utility for the LinkDb.
The LinkDumper tool creates a database of node to inlink information that can
be read using the nested Reader class.
Inverts outlinks from the WebGraph to inlinks and attaches node
information.
Wraps all values in ObjectWritables.
Inverts outlinks to inlinks while attaching node information to the
outlink.
Bean class which holds url to node information.
Writable class which holds an array of LinkNode objects.
Merges LinkNode objects into a single array value per url.
Reader class which will print out the url and all of its inlinks to system
out.
Utility methods for handling application-level locking.
Creates a custom analyzer based on user provided inputs
Default implementation of a page signature.
A multi-valued metadata container.
Indexer which can be configured to extract metadata from the crawldb, parse
metadata or content metadata.
For documentation:
org.apache.nutch.scoring.metadataParse HTML meta tags (keywords, description) and store them in the parse
metadata so that they can be indexed with the index-metadata plugin with the
prefix 'metatag.'.
This is a simple decorator that adds metadata to any Writable-s that can be
serialized by
NutchWritable.Extension of @see AdaptiveFetchSchedule that allows for more flexible
configuration of DEC and INC factors for various MIME-types.
An
IndexingFilter that allows filtering
of documents based on the MIME Type detected by TikaThis is a facade class to insulate Nutch from its underlying Mime Type
substrate library, Apache Tika.
MissingDependencyException will be thrown if a plugin dependency
cannot be found.This class creates a model used to store Document vector representation of the corpus.
Add (or reset) a few metaData properties as respective fields (if they are
available), so that they can be accurately used within the search index.
Html Parse filter that classifies the outlinks from the parseresult as
relevant or irrelevant based on the parseText's relevancy (using a training
file where you can give positive and negative example texts see the
description of parsefilter.naivebayes.trainfile) and if found irrelevant it
gives the link a second chance if it contains any of the words from the list
given in parsefilter.naivebayes.wordlist.
A class which holds the number of inlinks and outlinks for a given url along
with an inlink score from a link analysis program and any metadata.
A tools that dumps out the top urls by number of inlinks, number of outlinks,
or by score, to a text file.
Outputs the hosts or domains with an associated value.
Outputs the host or domain as key for this record and numInlinks,
numOutlinks or score as the value.
Outputs either the sum or the top value for this record.
Outputs the top urls sorted in descending order.
Outputs the url with the appropriate number of inlinks, outlinks, or for
score.
Flips and collects the url and numeric sort value.
Reads and prints to system out information for a single node from the NodeDb
in the WebGraph.
A utility class that allows the walking of any DOM tree using a stack instead
of recursion.
A collection of Nutch internal metadata constants.
Utility to create Hadoop
Configurations that include Nutch-specific
resources.A
NutchDocument is the unit of indexing.This class represents a multi-valued field with a weight.
A
NutchIndexAction is the new unit of indexing holding the document
and action information.A
Job for Nutch jobs.Centralized constants for Hadoop metrics counter groups and names.
All publisher subscriber model implementations should implement this interface.
Container to store whether and why content has been truncated
Sends NutchDocuments to a configured OpenSearch index.
This plugin implements a variant of an Online Page Importance Computation
(OPIC) score, described in this paper:
Abiteboul, Serge and Preda, Mihai and Cobena, Gregory (2003), Adaptive
On-Line Page Importance Computation.
Orphan scoring filter that determines whether a page has become orphaned,
e.g.
An outgoing link from a page.
Extractor to extract
Outlinks / URLs from
plain text using Regular Expressions.The result of parsing a page's raw content.
Data extracted from a page's content.
The result of parsing a page's raw content.
A parser for content generated by a
Protocol implementation.Parser checker, useful for testing parser.
A utility class that stores result of a parse.
Creates and caches
Parser plugins.Sends LATENCY_KEY to partition 0 so one reducer merges all TDigests.
This URLNormalizer doesn't change urls.
Defines the capability of a class to be plugged into Nutch.
A nutch-plugin is an container for a set of custom logic that provide
extensions to the nutch core functionality or another plugin that provides an
API for extending.
The
PluginClassLoader is a child-first classloader that only
contains classes of the runtime libraries setuped in the plugin manifest file
and exported libraries of plugins that are required plugins.The
PluginDescriptor provide access to all meta information of a
nutch-plugin, as well to the internationalizable resources and the plugin own
classloader.The
PluginManifestParser provides a mechanism for
parsing Nutch plugin manifest files (plugin.xml) contained
in a String of plugin directories.The plugin repository is a registry of all plugins.
PluginRuntimeException will be thrown until a exception in the
plugin managemnt occurs.A class for efficiently matching
Strings against a set of
prefixes.Filters URLs based on a file of URL prefixes.
This is a support class for logging all ftp command/reply traffic.
A retriever of url content.
Deprecated.
Creates and caches
Protocol plugins.Simple aggregate to pass from protocol plugins both content and protocol
status.
Extracts protocol status code information from the crawl database.
URL normalizer to normalize the protocol for all URLs of a given host or
domain, e.g.
URL normalizer plugin for normalizing query strings but sorting query string
parameters.
This class feeds the queues with input items, and re-fills them as items
are consumed by FetcherThread-s.
Client for RabbitMQ
RegexParseFilter.
A generic regular expression rule.
Filters URLs based on a file of regular expressions using the
Java Regex implementation.Generic
URLFilter based on regular
expressions.Allows users to do regex substitutions on all/any URLs that are encountered,
which is useful for stripping session IDs from URLs.
An
IndexingFilter that add tag
field(s) to the document.Adds microformat rel-tags of document if found.
Do pattern replacements on selected field contents prior to indexing.
Simple runnable that performs DNS lookup for a single host.
A simple tool that will spin up multiple threads to resolve urls to ip
addresses.
A response interface.
This class uses crawler-commons for handling the parsing of
robots.txt files.Updates the score from the WebGraph node database into the crawl database.
Changes input into ObjectWritables.
Creates new CrawlDatum objects with the updated score from the NodeDb or
with a cleared score.
A contract defining behavior of scoring plugins.
Specialized exception for errors during scoring.
Creates and caches
ScoringFilter implementing plugins.Checks whether a segment is valid, or has a certain status (generated,
fetched, parsed), or can be used safely for a certain processing step
(e.g., indexing).
Interface used to filter segments during segment merge.
This class wraps all
SegmentMergeFilter extensions in a single object
so it is easier to operate on them.This tool takes several segments and merges their data together.
Wraps inputs in an
MetaWrapper, to permit merging different types
in reduce and use additional metadata.NOTE: in selecting the latest version we rely exclusively on the segment
name (not all segment data contain time information).
Utility class for handling information about segment parts.
Dump the content of a segment.
Implements a text output format
Tool to list properties and their values set by the current Nutch
configuration
Factory class, which instantiates a Signature implementation according to the
current Configuration configuration.
Performs sitemap processing by fetching
sitemap links, parsing the content and merging the URLs from sitemaps (with
the metadata) into the CrawlDb.
Configurable normalization of trailing slashes in URLs.
A decorator to Metadata that adds spellchecking capabilities to property
names.
A simple plugin called at indexing that adds fields with static data.
A collection of String processing utility methods.
SubCollection represents a subset of index, you can define url patterns that
will indicate that particular page (url) is part of SubCollection.
A class for efficiently matching
Strings against a set of
suffixes.Filters URLs based on a file of URL suffixes.
Implementation of a page signature.
An implementation of a page signature.
Wrapper for Tika parsers.
Adds the public suffix (aka.
TrieStringMatcher is a base class for simple tree-based string matching.
Tool to create a HostDB from the CrawlDB.
Mapper ingesting HostDB and CrawlDB entries.
Interface used to allow exemptions to external domain resources by overriding
db.ignore.external.links.Creates and caches
URLExemptionFilter implementing plugins.Interface used to limit which URLs enter Nutch.
Checks one given filter or all filters.
Creates and caches plugins implementing
URLFilter.This is part of the URL Meta plugin.
For documentation:
org.apache.nutch.scoring.urlmetaInterface used to convert URLs to normal form and optionally perform
substitutions
Checks one given normalizer or all normalizers.
This class uses a "chained filter" pattern to run defined normalizers.
Partition urls by host, domain name or IP depending on the value of the
parameter 'partition.url.mode' which can be 'byHost', 'byDomain' or 'byIP'
This URLStreamHandlerFactory knows about all the plugins
in use and thus can create the correct URLStreamHandler
even if it comes from a plugin classpath.
Utility class for URL analysis
Validates URLs.
MapReduce job to exports Nutch segments as WARC files.
Creates three databases, one for inlinks, one for outlinks, and a node
database that holds the number of in and outlinks to a url and the current
score for the url.
The OutlinkDb creates a database of all outlinks.
Passes through existing LinkDatum objects from an existing OutlinkDb and
maps out new LinkDatum objects from new crawls ParseData.
Class used to verify whether the specified ch conforms to the XML
1.0 definition of whitespace.
ZipParser class based on MSPowerPointParser class by Stephan Strittmatter.
ProtocolExceptioninstead.