org.apache.flink.orc

类 OrcColumnarRowInputFormat<BatchT,SplitT extends org.apache.flink.connector.file.src.FileSourceSplit>

java.lang.Object

org.apache.flink.orc.AbstractOrcFileInputFormat<org.apache.flink.table.data.RowData,BatchT,SplitT>

org.apache.flink.orc.OrcColumnarRowInputFormat<BatchT,SplitT>

所有已实现的接口:

Serializable, org.apache.flink.api.java.typeutils.ResultTypeQueryable<org.apache.flink.table.data.RowData>, org.apache.flink.connector.file.src.reader.BulkFormat<org.apache.flink.table.data.RowData,SplitT>, org.apache.flink.table.connector.format.FileBasedStatisticsReportableInputFormat

直接已知子类:

OrcColumnarRowFileInputFormat

public class OrcColumnarRowInputFormat<BatchT,SplitT extends org.apache.flink.connector.file.src.FileSourceSplit>
extends AbstractOrcFileInputFormat<org.apache.flink.table.data.RowData,BatchT,SplitT>
implements org.apache.flink.table.connector.format.FileBasedStatisticsReportableInputFormat

An ORC reader that produces a stream of ColumnarRowData records.

This class can add extra fields through ColumnBatchFactory, for example, add partition fields, which can be extracted from path. Therefore, the getProducedType() may be different and types of extra fields need to be added.

另请参阅:

序列化表格

嵌套类概要

从类继承的嵌套类/接口 org.apache.flink.orc.AbstractOrcFileInputFormat

AbstractOrcFileInputFormat.OrcReaderBatch<T,BatchT>, AbstractOrcFileInputFormat.OrcVectorizedReader<T,BatchT>

从接口继承的嵌套类/接口 org.apache.flink.connector.file.src.reader.BulkFormat

org.apache.flink.connector.file.src.reader.BulkFormat.Reader<T>, org.apache.flink.connector.file.src.reader.BulkFormat.RecordIterator<T>

字段概要

从类继承的字段 org.apache.flink.orc.AbstractOrcFileInputFormat

batchSize, conjunctPredicates, hadoopConfigWrapper, schema, selectedFields, shim

构造器概要

构造器

构造器和说明
OrcColumnarRowInputFormat(OrcShim<BatchT> shim, org.apache.hadoop.conf.Configuration hadoopConfig, org.apache.orc.TypeDescription schema, int[] selectedFields, List<OrcFilters.Predicate> conjunctPredicates, int batchSize, ColumnBatchFactory<BatchT,SplitT> batchFactory, org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData> producedTypeInfo)

方法概要

限定符和类型	方法和说明
static <SplitT extends org.apache.flink.connector.file.src.FileSourceSplit> OrcColumnarRowInputFormat<org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch,SplitT>	createPartitionedFormat(OrcShim<org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch> shim, org.apache.hadoop.conf.Configuration hadoopConfig, org.apache.flink.table.types.logical.RowType tableType, List<String> partitionKeys, org.apache.flink.connector.file.table.PartitionFieldExtractor<SplitT> extractor, int[] selectedFields, List<OrcFilters.Predicate> conjunctPredicates, int batchSize, java.util.function.Function<org.apache.flink.table.types.logical.RowType,org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData>> rowTypeInfoFactory) Create a partitioned OrcColumnarRowInputFormat, the partition columns can be generated by split.
AbstractOrcFileInputFormat.OrcReaderBatch<org.apache.flink.table.data.RowData,BatchT>	createReaderBatch(SplitT split, OrcVectorizedBatchWrapper<BatchT> orcBatch, org.apache.flink.connector.file.src.util.Pool.Recycler<AbstractOrcFileInputFormat.OrcReaderBatch<org.apache.flink.table.data.RowData,BatchT>> recycler, int batchSize) Creates the AbstractOrcFileInputFormat.OrcReaderBatch structure, which is responsible for holding the data structures that hold the batch data (column vectors, row arrays, ...) and the batch conversion from the ORC representation to the result format.
org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData>	getProducedType() Gets the type produced by this format.
org.apache.flink.table.plan.stats.TableStats	reportStatistics(List<org.apache.flink.core.fs.Path> files, org.apache.flink.table.types.DataType producedDataType)

从类继承的方法 org.apache.flink.orc.AbstractOrcFileInputFormat

createReader, isSplittable, restoreReader

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

OrcColumnarRowInputFormat

public OrcColumnarRowInputFormat(OrcShim<BatchT> shim,
                                 org.apache.hadoop.conf.Configuration hadoopConfig,
                                 org.apache.orc.TypeDescription schema,
                                 int[] selectedFields,
                                 List<OrcFilters.Predicate> conjunctPredicates,
                                 int batchSize,
                                 ColumnBatchFactory<BatchT,SplitT> batchFactory,
                                 org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData> producedTypeInfo)

方法详细资料

createReaderBatch

public AbstractOrcFileInputFormat.OrcReaderBatch<org.apache.flink.table.data.RowData,BatchT> createReaderBatch(SplitT split,
                                                                                                               OrcVectorizedBatchWrapper<BatchT> orcBatch,
                                                                                                               org.apache.flink.connector.file.src.util.Pool.Recycler<AbstractOrcFileInputFormat.OrcReaderBatch<org.apache.flink.table.data.RowData,BatchT>> recycler,
                                                                                                               int batchSize)

从类复制的说明: AbstractOrcFileInputFormat

Creates the AbstractOrcFileInputFormat.OrcReaderBatch structure, which is responsible for holding the data structures that hold the batch data (column vectors, row arrays, ...) and the batch conversion from the ORC representation to the result format.

指定者:

createReaderBatch 在类中 AbstractOrcFileInputFormat<org.apache.flink.table.data.RowData,BatchT,SplitT extends org.apache.flink.connector.file.src.FileSourceSplit>

getProducedType

public org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData> getProducedType()

从类复制的说明: AbstractOrcFileInputFormat

Gets the type produced by this format.

指定者:

getProducedType 在接口中 org.apache.flink.api.java.typeutils.ResultTypeQueryable<org.apache.flink.table.data.RowData>

指定者:

getProducedType 在接口中 org.apache.flink.connector.file.src.reader.BulkFormat<org.apache.flink.table.data.RowData,SplitT extends org.apache.flink.connector.file.src.FileSourceSplit>

指定者:

getProducedType 在类中 AbstractOrcFileInputFormat<org.apache.flink.table.data.RowData,BatchT,SplitT extends org.apache.flink.connector.file.src.FileSourceSplit>

reportStatistics

public org.apache.flink.table.plan.stats.TableStats reportStatistics(List<org.apache.flink.core.fs.Path> files,
                                                                     org.apache.flink.table.types.DataType producedDataType)

指定者:

reportStatistics 在接口中 org.apache.flink.table.connector.format.FileBasedStatisticsReportableInputFormat

createPartitionedFormat

public static <SplitT extends org.apache.flink.connector.file.src.FileSourceSplit> OrcColumnarRowInputFormat<org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch,SplitT> createPartitionedFormat(OrcShim<org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch> shim,
                                                                                                                                                                                                      org.apache.hadoop.conf.Configuration hadoopConfig,
                                                                                                                                                                                                      org.apache.flink.table.types.logical.RowType tableType,
                                                                                                                                                                                                      List<String> partitionKeys,
                                                                                                                                                                                                      org.apache.flink.connector.file.table.PartitionFieldExtractor<SplitT> extractor,
                                                                                                                                                                                                      int[] selectedFields,
                                                                                                                                                                                                      List<OrcFilters.Predicate> conjunctPredicates,
                                                                                                                                                                                                      int batchSize,
                                                                                                                                                                                                      java.util.function.Function<org.apache.flink.table.types.logical.RowType,org.apache.flink.api.common.typeinfo.TypeInformation<org.apache.flink.table.data.RowData>> rowTypeInfoFactory)

Create a partitioned OrcColumnarRowInputFormat, the partition columns can be generated by split.