Package org.apache.nutch.protocol.selenium

Class HttpWebClient

java.lang.Object
org.apache.nutch.protocol.selenium.HttpWebClient
public class HttpWebClient extends Object

  • Constructor Details

    • HttpWebClient

      public HttpWebClient()

  • Method Details

    • getDriverForPage

      public static org.openqa.selenium.WebDriver getDriverForPage(String url, Configuration conf)

    • createFirefoxWebDriver

      public static org.openqa.selenium.WebDriver createFirefoxWebDriver(String firefoxDriverPath, boolean enableHeadlessMode)

    • createChromeWebDriver

      public static org.openqa.selenium.WebDriver createChromeWebDriver(String chromeDriverPath, boolean enableHeadlessMode)

    • createFirefoxRemoteWebDriver

      public static org.openqa.selenium.remote.RemoteWebDriver createFirefoxRemoteWebDriver(URL seleniumHubUrl, boolean enableHeadlessMode)

    • createChromeRemoteWebDriver

      public static org.openqa.selenium.remote.RemoteWebDriver createChromeRemoteWebDriver(URL seleniumHubUrl, boolean enableHeadlessMode)

    • createRandomRemoteWebDriver

      public static org.openqa.selenium.remote.RemoteWebDriver createRandomRemoteWebDriver(URL seleniumHubUrl, boolean enableHeadlessMode)

    • createDefaultRemoteWebDriver

      public static org.openqa.selenium.remote.RemoteWebDriver createDefaultRemoteWebDriver(URL seleniumHubUrl, boolean enableHeadlessMode)

    • cleanUpDriver

      public static void cleanUpDriver(org.openqa.selenium.WebDriver driver)

    • getHtmlPage

      public static String getHtmlPage(String url, Configuration conf)
      Function for obtaining the HTML using the selected selenium webdriver There are a number of configuration properties within nutch-site.xml which determine whether to take screenshots of the rendered pages and persist them as timestamped .png's into HDFS.
      Parameters:
      url - the URL to fetch and render
      conf - the Configuration
      Returns:
      the html page

    • getHtmlPage

      public static String getHtmlPage(String url)