Type Members

1.  class PythonBatch extends Batch
2.  class PythonBatchWrite extends BatchWrite

   A BatchWrite for python data source writing.

   A BatchWrite for python data source writing. Responsible for generating the writer factory.

3.  case class PythonBatchWriterFactory(source: UserDefinedPythonDataSource, pickledWriteFunc: Array[Byte], inputSchema: StructType, jobArtifactUUID: Option[String]) extends DataWriterFactory with Product with Serializable
4.  class PythonCustomMetric extends CustomMetric
5.  class PythonCustomTaskMetric extends CustomTaskMetric
6.  case class PythonDataSourceCreationResult(dataSource: Array[Byte], schema: StructType) extends Product with Serializable

   Used to store the result of creating a Python data source in the Python process.

7.  case class PythonDataSourceReadInfo(func: Array[Byte], partitions: Seq[Array[Byte]]) extends Product with Serializable
8.  class PythonDataSourceV2 extends TableProvider

   Data Source V2 wrapper for Python Data Source.

9.  case class PythonDataSourceWriteInfo(func: Array[Byte], writer: Array[Byte]) extends Product with Serializable

   Hold the results of running UserDefinedPythonDataSourceWriteRunner.

10.  case class PythonInputPartition(index: Int, pickedPartition: Array[Byte]) extends InputPartition with Product with Serializable
11.  case class PythonLookupAllDataSourcesResult(names: Array[String], dataSources: Array[Array[Byte]]) extends Product with Serializable

    All Data Sources in Python

12.  class PythonMicroBatchStream extends MicroBatchStream with Logging with AcceptsLatestSeenOffset
13.  class PythonPartitionReaderFactory extends PartitionReaderFactory
14.  class PythonScan extends Scan
15.  class PythonScanBuilder extends ScanBuilder
16.  case class PythonStreamingInputPartition(index: Int, pickedPartition: Array[Byte], blockId: Option[PythonStreamBlockId]) extends InputPartition with Product with Serializable
17.  class PythonStreamingPartitionReaderFactory extends PartitionReaderFactory with Logging
18.  class PythonStreamingSinkCommitRunner extends PythonPlannerRunner[Unit]

    This class is a proxy to invoke commit or abort methods in Python DataSourceStreamWriter.

    This class is a proxy to invoke commit or abort methods in Python DataSourceStreamWriter. A runner spawns a python worker process. In the main function, set up communication between JVM and python process through socket and create a DataSourceStreamWriter instance. In an infinite loop, the python worker process receive write commit messages from the socket, then commit or abort a microbatch.

19.  case class PythonStreamingSourceOffset(json: String) extends Offset with Product with Serializable
20.  class PythonStreamingWrite extends StreamingWrite

    A streamingWrite for python data source writing.

    A streamingWrite for python data source writing. Responsible for generating the writer factory, committing or aborting a microbatch.

21.  class PythonStreamingWriterFactory extends PythonBatchWriterFactory with StreamingDataWriterFactory
22.  class PythonTable extends Table with SupportsRead with SupportsWrite
23.  class PythonWrite extends Write
24.  class PythonWriteBuilder extends WriteBuilder with SupportsTruncate
25.  case class PythonWriterCommitMessage(pickledMessage: Array[Byte]) extends WriterCommitMessage with Product with Serializable
26.  case class UserDefinedPythonDataSource(dataSourceCls: PythonFunction) extends Product with Serializable

    A user-defined Python data source.

    A user-defined Python data source. This is used by the Python API. Defines the interation between Python and JVM.

    dataSourceCls

    The Python data source class.