程序包 com.agentsflex.core.file2text.extractor.impl

类 HtmlExtractor

java.lang.Object

com.agentsflex.core.file2text.extractor.impl.HtmlExtractor

所有已实现的接口:

FileExtractor

public class HtmlExtractor
extends Object
implements FileExtractor

增强版 HTML 文档提取器 支持可配置的噪音过滤规则（含中文网站常见广告）

字段概要

字段

修饰符和类型	字段	说明
private static final Set<String>	CLASS_KEYWORDS
private static final Set<String>	CUSTOM_CLASS_KEYWORDS
private static final Set<String>	CUSTOM_SELECTORS
private static final Set<String>	DEFAULT_SELECTORS
private static final Pattern	ID_CLASS_PATTERN
private static final Set<String>	SUPPORTED_EXTENSIONS
private static final Set<String>	SUPPORTED_MIME_TYPES

从接口继承的字段 com.agentsflex.core.file2text.extractor.FileExtractor

ORDER_COMPARATOR

构造器概要

构造器

构造器	说明
HtmlExtractor()

方法概要

修饰符和类型	方法	说明
static void	addCustomKeyword(String keyword)	添加自定义 class/id 关键词
static void	addCustomSelector(String selector)	添加自定义噪音选择器（CSS 选择器）
private void	appendNodeText(org.jsoup.nodes.Node node, StringBuilder text, int level)
private void	extractBodyContent(org.jsoup.nodes.Document doc, StringBuilder text)
private void	extractTable(org.jsoup.nodes.Element table, StringBuilder text)
String	extractText(DocumentSource source)
private void	extractTitle(org.jsoup.nodes.Document doc, StringBuilder text)
private String	getExtension(String fileName)
int	getOrder()
private boolean	isBlockLevel(String tagName)
private void	removeElementsBySelectors(org.jsoup.nodes.Element body)	使用 CSS 选择器移除噪音
private void	removeElementsWithKeywords(org.jsoup.nodes.Element body)	移除 class 或 id 包含关键词的元素
private String	repeat(String string, int times)
boolean	supports(DocumentSource source)	判断该 Extractor 是否支持处理此文档

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

SUPPORTED_MIME_TYPES

private static final Set<String> SUPPORTED_MIME_TYPES

SUPPORTED_EXTENSIONS

private static final Set<String> SUPPORTED_EXTENSIONS

DEFAULT_SELECTORS

private static final Set<String> DEFAULT_SELECTORS

CLASS_KEYWORDS

private static final Set<String> CLASS_KEYWORDS

ID_CLASS_PATTERN

private static final Pattern ID_CLASS_PATTERN

CUSTOM_SELECTORS

private static final Set<String> CUSTOM_SELECTORS

CUSTOM_CLASS_KEYWORDS

private static final Set<String> CUSTOM_CLASS_KEYWORDS

构造器详细资料

HtmlExtractor

public HtmlExtractor()

方法详细资料

addCustomSelector

public static void addCustomSelector(String selector)

添加自定义噪音选择器（CSS 选择器）

addCustomKeyword

public static void addCustomKeyword(String keyword)

添加自定义 class/id 关键词

supports

public boolean supports(DocumentSource source)

从接口复制的说明: FileExtractor

判断该 Extractor 是否支持处理此文档

指定者:

supports 在接口中 FileExtractor

extractText

public String extractText(DocumentSource source)
                   throws IOException

指定者:

extractText 在接口中 FileExtractor

抛出:

IOException

extractTitle

private void extractTitle(org.jsoup.nodes.Document doc,
 StringBuilder text)

extractBodyContent

private void extractBodyContent(org.jsoup.nodes.Document doc,
 StringBuilder text)

removeElementsBySelectors

private void removeElementsBySelectors(org.jsoup.nodes.Element body)

使用 CSS 选择器移除噪音

removeElementsWithKeywords

private void removeElementsWithKeywords(org.jsoup.nodes.Element body)

移除 class 或 id 包含关键词的元素

repeat

private String repeat(String string,
 int times)

appendNodeText

private void appendNodeText(org.jsoup.nodes.Node node,
 StringBuilder text,
 int level)

isBlockLevel

private boolean isBlockLevel(String tagName)

extractTable

private void extractTable(org.jsoup.nodes.Element table,
 StringBuilder text)

getOrder

public int getOrder()

指定者:

getOrder 在接口中 FileExtractor

getExtension

private String getExtension(String fileName)