Package org.apache.nutch.collection

Class Subcollection

java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.nutch.collection.Subcollection
All Implemented Interfaces:
Configurable, URLFilter, Pluggable
public class Subcollection extends Configured implements URLFilter
SubCollection represents a subset of index, you can define url patterns that will indicate that particular page (url) is part of SubCollection.

  • Field Details

  • Constructor Details

    • Subcollection

      public Subcollection(String id, String name, Configuration conf)
      public Constructor
      Parameters:
      id - Id of SubCollection
      name - Name of SubCollection
      conf - A populated Configuration

    • Subcollection

      public Subcollection(String id, String name, String key, Configuration conf)
      public Constructor
      Parameters:
      id - Id of SubCollection
      name - Name of SubCollection
      key - SubCollection key
      conf - A populated Configuration

    • Subcollection

      public Subcollection(Configuration conf)

  • Method Details

    • getName

      public String getName()
      Returns:
      Returns the name

    • getKey

      public String getKey()
      Returns:
      Returns the key

    • getId

      public String getId()
      Returns:
      Returns the id

    • getWhiteList

      public List<String> getWhiteList()
      Returns whitelist
      Returns:
      Whitelist entries

    • getWhiteListString

      public String getWhiteListString()
      Returns whitelist String
      Returns:
      Whitelist String

    • getBlackListString

      public String getBlackListString()
      Returns blacklist String
      Returns:
      Blacklist String

    • setWhiteList

      public void setWhiteList(ArrayList<String> whiteList)
      Parameters:
      whiteList - The whiteList to set.

    • filter

      public String filter(String urlString)
      Simple "indexOf" currentFilter for matching patterns. 
        rules for evaluation are as follows:
  1. if pattern matches in blacklist then url is rejected
  2. if pattern matches in whitelist then url is allowed
  3. url is rejected
      Specified by:
      filter in interface URLFilter
      Parameters:
      urlString - the URL string the filter is applied on
      Returns:
      the original URL string if the URL is accepted by the filter or null in case the URL is rejected
      See Also:

    • initialize

      public void initialize(Element collection)
      Initialize Subcollection from dom element
      Parameters:
      collection - A DOM Element for use in creating the Subcollection

    • parseList

      protected void parseList(List<String> list, String text)
      Create a list of patterns from a chunk of text, patterns are separated with a newline
      Parameters:
      list - An initialized List to insert String patterns.
      text - A chunkl fo text (hopefully) containing patterns.

    • setBlackList

      public void setBlackList(String list)
      Set contents of blacklist from String
      Parameters:
      list - the blacklist contents

    • setWhiteList

      public void setWhiteList(String list)
      Set contents of whitelist from String
      Parameters:
      list - the whitelist contents