c
org.apache.spark.sql.execution.datasources.v2.python
UserDefinedPythonDataSource
Companion object UserDefinedPythonDataSource
case class UserDefinedPythonDataSource(dataSourceCls: PythonFunction) extends Product with Serializable
A user-defined Python data source. This is used by the Python API. Defines the interation between Python and JVM.
- dataSourceCls
The Python data source class.
Linear Supertypes
Ordering
- Alphabetic
- By Inheritance
Inherited
- UserDefinedPythonDataSource
- Serializable
- Product
- Equals
- AnyRef
- Any
- Hide All
- Show All
Visibility
- Public
- Protected
Instance Constructors
- new UserDefinedPythonDataSource(dataSourceCls: PythonFunction)
- dataSourceCls
The Python data source class.
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def commitWriteInPython(writer: Array[Byte], messages: Array[WriterCommitMessage], abort: Boolean = false): Unit
(Driver-side) Run Python process to either commit or abort a write operation.
- def createDataSourceInPython(shortName: String, options: CaseInsensitiveStringMap, userSpecifiedSchema: Option[StructType]): PythonDataSourceCreationResult
(Driver-side) Run Python process, and get the pickled Python Data Source instance and its schema.
- def createMapInBatchEvaluatorFactory(pickledFunc: Array[Byte], funcName: String, inputSchema: StructType, outputSchema: StructType, metrics: Map[String, SQLMetric], jobArtifactUUID: Option[String]): MapInBatchEvaluatorFactory
(Executor-side) Create an iterator that execute the Python function.
- def createPythonFunction(pickledFunc: Array[Byte]): PythonFunction
- def createPythonMetrics(): Array[CustomMetric]
- def createPythonTaskMetrics(taskMetrics: Map[String, Long]): Array[CustomTaskMetric]
- def createReadInfoInPython(pythonResult: PythonDataSourceCreationResult, outputSchema: StructType, isStreaming: Boolean): PythonDataSourceReadInfo
(Driver-side) Run Python process, and get the partition read functions, and partition information.
- def createWriteInfoInPython(provider: String, inputSchema: StructType, options: CaseInsensitiveStringMap, overwrite: Boolean, isStreaming: Boolean): PythonDataSourceWriteInfo
(Driver-side) Run Python process and get pickled write function.
- val dataSourceCls: PythonFunction
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def productElementNames: Iterator[String]
- Definition Classes
- Product
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)