Uses of Package
org.apache.nutch.net
Packages that use org.apache.nutch.net
Package
Description
Subcollection is a subset of an index.
Index content, configure and run indexing and cleaning jobs to
add, update, and delete documents from an index.
Web-related interfaces: URL
filters
and normalizers.URL normalizer performing basic normalizations:
remove default ports, e.g., port 80 for
http:// URLs
remove needless slashes and dot segments in the path component
remove anchors
use percent-encoding (only) where needed
E.g.,
https://www.example.org/a/../b//./select%2Dlang.php?lang=español#anchor
is normalized to https://www.example.org/b/select-lang.php?lang=espa%C3%B1ol
Optional and configurable normalizations are:
convert Internationalized Domain Names (IDNs) uniquely either to the
ASCII (Punycode) or Unicode representation, see property
urlnormalizer.basic.host.idn
remove a trailing dot from host names, see property
urlnormalizer.basic.host.trim-trailing-dot
URL normalizer renaming hosts to a canonical form listed in the
configuration file.
URL normalizer dummy which does not change URLs.
URL normalizer to normalize the protocol for all URLs of a given host or
domain.
URL normalizer which sort the elements in the query part to avoid duplicates
by permutations.
URL normalizer with configurable rules based on regular expressions
(
Pattern).The
Parse interface and related classes.Generic
URL filter library,
abstracting away from regular expression implementations.URL filter plugin based on
dk.brics.automaton Finite-State
Automata for JavaTM.
URL filter plugin to include only URLs which match an element in a given list of
domain suffixes, domain names, and/or host names.
URL filter plugin to exclude URLs by domain suffixes, domain names, and/or host names.
URL filter plugin that first does fast exact suffix matches on host/domain
names before applying regular expressions to the path component of a URL.
URL filter plugin which identifies exemptions to external urls when
when external urls are set to ignore.
URL filter plugin to include only URLs which match one of a given list of URL prefixes.
URL filter plugin to include and/or exclude URLs matching Java regular expressions.
URL filter plugin to either exclude or include only URLs which match
one of the given (path) suffixes.
URL filter plugin that validates given urls.
-
Classes in org.apache.nutch.net used by org.apache.nutch.collection
-
Classes in org.apache.nutch.net used by org.apache.nutch.hostdbClassDescriptionCreates and caches plugins implementing
URLFilter.This class uses a "chained filter" pattern to run defined normalizers. -
Classes in org.apache.nutch.net used by org.apache.nutch.indexerClassDescriptionThis class uses a "chained filter" pattern to run defined normalizers.
-
Classes in org.apache.nutch.net used by org.apache.nutch.net
-
Classes in org.apache.nutch.net used by org.apache.nutch.net.urlnormalizer.ajaxClassDescriptionInterface used to convert URLs to normal form and optionally perform substitutions
-
Classes in org.apache.nutch.net used by org.apache.nutch.net.urlnormalizer.basicClassDescriptionInterface used to convert URLs to normal form and optionally perform substitutions
-
Classes in org.apache.nutch.net used by org.apache.nutch.net.urlnormalizer.hostClassDescriptionInterface used to convert URLs to normal form and optionally perform substitutions
-
Classes in org.apache.nutch.net used by org.apache.nutch.net.urlnormalizer.passClassDescriptionInterface used to convert URLs to normal form and optionally perform substitutions
-
Classes in org.apache.nutch.net used by org.apache.nutch.net.urlnormalizer.protocolClassDescriptionInterface used to convert URLs to normal form and optionally perform substitutions
-
Classes in org.apache.nutch.net used by org.apache.nutch.net.urlnormalizer.querystringClassDescriptionInterface used to convert URLs to normal form and optionally perform substitutions
-
Classes in org.apache.nutch.net used by org.apache.nutch.net.urlnormalizer.regexClassDescriptionInterface used to convert URLs to normal form and optionally perform substitutions
-
Classes in org.apache.nutch.net used by org.apache.nutch.net.urlnormalizer.slashClassDescriptionInterface used to convert URLs to normal form and optionally perform substitutions
-
Classes in org.apache.nutch.net used by org.apache.nutch.parseClassDescriptionCreates and caches
URLExemptionFilterimplementing plugins.Creates and caches plugins implementingURLFilter.This class uses a "chained filter" pattern to run defined normalizers. -
Classes in org.apache.nutch.net used by org.apache.nutch.urlfilter.api
-
Classes in org.apache.nutch.net used by org.apache.nutch.urlfilter.automaton
-
Classes in org.apache.nutch.net used by org.apache.nutch.urlfilter.domain
-
Classes in org.apache.nutch.net used by org.apache.nutch.urlfilter.domaindenylist
-
Classes in org.apache.nutch.net used by org.apache.nutch.urlfilter.fast
-
Classes in org.apache.nutch.net used by org.apache.nutch.urlfilter.ignoreexemptClassDescriptionInterface used to allow exemptions to external domain resources by overriding
db.ignore.external.links.Interface used to limit which URLs enter Nutch. -
Classes in org.apache.nutch.net used by org.apache.nutch.urlfilter.prefix
-
Classes in org.apache.nutch.net used by org.apache.nutch.urlfilter.regex
-
Classes in org.apache.nutch.net used by org.apache.nutch.urlfilter.suffix
-
Classes in org.apache.nutch.net used by org.apache.nutch.urlfilter.validator