Package org.apache.nutch.util

Class EncodingDetector

java.lang.Object

org.apache.nutch.util.EncodingDetector

public class EncodingDetector
extends Object

A simple class for detecting character encodings.

Broadly this encompasses two functions, which are distinctly separate:

1. Auto detecting a set of "clues" from input text.
2. Taking a set of clues and making a "best guess" as to the "real" encoding.

A caller will often have some extra information about what the encoding might be (e.g. from the HTTP header or HTML meta-tags, often wrong but still potentially useful clues). The types of clues may differ from caller to caller. Thus a typical calling sequence is:

• Run step (1) to generate a set of auto-detected clues;
• Combine these clues with the caller-dependent "extra clues" available;
• Run step (2) to guess what the most probable answer is.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	MIN_CONFIDENCE_KEY
static final int	NO_THRESHOLD

Constructor Summary

Constructors

Constructor	Description
EncodingDetector(Configuration conf)

Method Summary

Modifier and Type	Method	Description
void	addClue(String value, String source)
void	addClue(String value, String source, int confidence)
void	autoDetectClues(Content content, boolean filter)
void	clearClues()	Clears all clues.
String	guessEncoding(Content content, String defaultValue)	Guess the encoding with the previously specified list of clues.
static void	main(String[] args)
static String	parseCharacterEncoding(String contentType)	Parse the character encoding from the specified content type header.
static String	resolveEncodingAlias(String encoding)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

NO_THRESHOLD

public static final int NO_THRESHOLD

See Also:

• Constant Field Values

MIN_CONFIDENCE_KEY

public static final String MIN_CONFIDENCE_KEY

See Also:

• Constant Field Values

Constructor Details

EncodingDetector

public EncodingDetector(Configuration conf)

Method Details

autoDetectClues

public void autoDetectClues(Content content,
 boolean filter)

addClue

public void addClue(String value,
 String source,
 int confidence)

addClue

public void addClue(String value,
 String source)

guessEncoding

public String guessEncoding(Content content,
 String defaultValue)

Guess the encoding with the previously specified list of clues.

Parameters:

content - Content instance

defaultValue - Default encoding to return if no encoding can be detected with enough confidence. Note that this will not be normalized with resolveEncodingAlias(java.lang.String)

Returns:

Guessed encoding or defaultValue

clearClues

public void clearClues()

Clears all clues.

resolveEncodingAlias

public static String resolveEncodingAlias(String encoding)

parseCharacterEncoding

public static String parseCharacterEncoding(String contentType)

Parse the character encoding from the specified content type header. If the content type is null, or there is no explicit character encoding, null is returned.
This method was copied from org.apache.catalina.util.RequestUtil, which is licensed under the Apache License, Version 2.0 (the "License").

Parameters:

contentType - a content type header

Returns:

a trimmed string representation of the 'charset=' value, null if this is not available

main

public static void main(String[] args)
                 throws IOException

Throws:

IOException