Packages

case class MergedParameters(parameters: Map[String, String]) extends Product with Serializable

Adds validators and accessors to string map

Linear Supertypes
Serializable, Product, Equals, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MergedParameters
  2. Serializable
  3. Product
  4. Equals
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new MergedParameters(parameters: Map[String, String])

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def abortDetachedQuery: Boolean
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def awsAccessKey: Option[String]
  7. def awsSecretKey: Option[String]
  8. def azureSAS: Option[String]
  9. def bindVariableEnabled: Boolean
  10. def checkBucketConfiguration: Boolean

    Returns true if bucket lifecycle configuration should be checked

  11. def checkTableExistenceInCurrentSchemaOnly: Boolean
  12. def checkTableExistenceWithFullyQualifiedName: Boolean
  13. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native() @HotSpotIntrinsicCandidate()
  14. def columnMap: Option[Map[String, String]]

    Retrieve Column mapping data.

    Retrieve Column mapping data. None if empty

  15. def columnMapping: String
  16. def columnMismatchBehavior: String
  17. def continueOnError: Boolean

    Set on_error parameter to continue in COPY command todo: create data validation function in spark side instead of using COPY COMMAND

  18. def createPerQueryTempDir(): String

    Creates a per-query subdirectory in the rootTempDir, with a random UUID.

  19. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. def expectedPartitionCount: Int
  21. def expectedPartitionSize: Long
  22. def extraCopyOptions: String

    Extra options to append to the Snowflake COPY command (e.g.

    Extra options to append to the Snowflake COPY command (e.g. "MAXERROR 100").

  23. def force(): Boolean

    Whether or not to have FORCE in the COPY statement generated by the Spark connector

  24. def forceSkipPrePostActionsCheck: Boolean
  25. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  26. def getQueryIDUrl(queryID: String): String
  27. def getQueryResultFormat: Option[String]

    Snowflake query result format

  28. def getS3StageVpceDnsName: Option[String]
  29. def getSnowflakeTableSchema: StructType
  30. def getStringTimestampFormat: Option[String]
  31. def getTimeOutputFormat: Option[String]

    Snowflake time output format

  32. def isExecuteQueryWithSyncMode: Boolean
  33. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  34. def isSslON: Boolean
  35. def isTimestampSnowflake(timestampFormat: String): Boolean
  36. def isTimezoneSnowflake: Boolean
  37. def isTimezoneSnowflakeDefault: Boolean
  38. def isTimezoneSpark: Boolean
  39. def keepOriginalColumnNameCase: Boolean
  40. def maxRetryCount: Int
  41. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  42. def nonProxyHosts: Option[String]
  43. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  44. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  45. lazy val parallelism: Option[Int]

    Number of threads used for PUT/GET.

  46. val parameters: Map[String, String]
  47. def postActions: Array[String]

    List of semi-colon separated SQL statements to run after successful write operations.

    List of semi-colon separated SQL statements to run after successful write operations. This can be useful for running GRANT operations to make your new tables readable to other users and groups.

    If the action string contains %s, the table name will be substituted in, in case a staging table is being used.

    Defaults to empty.

  48. def preActions: Array[String]

    List of semi-colon separated SQL statements to run before write operations.

    List of semi-colon separated SQL statements to run before write operations. This can be useful for running DELETE operations to clean up data

    If the action string contains %s, the table name will be substituted in, in case a staging table is being used.

    Defaults to empty.

  49. def privateKey: Option[PrivateKey]

    Generate private key form pem key value

    Generate private key form pem key value

    returns

    private key object

  50. def productElementNames: Iterator[String]
    Definition Classes
    Product
  51. def proxyHost: Option[String]
  52. lazy val proxyInfo: Option[ProxyInfo]
  53. def proxyPassword: Option[String]
  54. def proxyPort: Option[String]
  55. def proxyProtocol: Option[String]
  56. def proxyUser: Option[String]
  57. def purge(): Boolean

    Whether or not to have PURGE in the COPY statement generated by the Spark connector

  58. def query: Option[String]

    The Snowflake query to be used as the target when loading data.

  59. def quoteJsonFieldName: Boolean
  60. def replaceSpecialCharacter(name: String, toUpperCase: Boolean): String
  61. lazy val rootTempDir: String

    A root directory to be used for intermediate data exchange, expected to be on cloud storage (S3 or Azure storage), or somewhere that can be written to and read from by Snowflake.

    A root directory to be used for intermediate data exchange, expected to be on cloud storage (S3 or Azure storage), or somewhere that can be written to and read from by Snowflake. Make sure that credentials are available for this cloud provider.

  62. lazy val rootTempDirStorageType: FSType
  63. def s3maxfilesize: String

    Max file size used to move data out from Snowflake

  64. def setColumnMap(fromSchema: Option[StructType], toSchema: Option[StructType]): Unit

    set column map

  65. def setSnowflakeTableSchema(schema: StructType): Unit
  66. def sfAccount: Option[String]

    Snowflake account - optional

  67. def sfAuthenticator: Option[String]

    Mapping OAuth and authenticator values

  68. def sfCompress: Boolean

    Snowflake use compression on/off - "on" by default

  69. def sfDatabase: String

    Snowflake database name

  70. def sfExtraOptions: Map[String, AnyRef]

    Returns a map of options that are not known to the connector, and are passed verbosely to the JDBC driver

  71. def sfFullURL: String

    URL pointing to the snowflake database including protocol.

    URL pointing to the snowflake database including protocol. for example, https://host:port

  72. def sfPassword: String

    Snowflake password

  73. def sfRole: Option[String]

    Snowflake role - optional

  74. def sfSSL: String

    Snowflake SSL on/off - "on" by default

  75. def sfSchema: String

    Snowflake schema

  76. def sfTimestampLTZOutputFormat: Option[String]
  77. def sfTimestampNTZOutputFormat: Option[String]
  78. def sfTimestampTZOutputFormat: Option[String]
  79. def sfTimezone: Option[String]

    Snowflake timezone- optional

  80. def sfToken: Option[String]
  81. def sfURL: String

    URL pointing to the snowflake database, simply host:port

  82. def sfUser: String

    Snowflake user

  83. def sfWarehouse: Option[String]

    Snowflake warehouse

  84. def sfWorkloadIdentityProvider: Option[String]
  85. def skipWriteWhenWritingEmptyDataFrame: Boolean
  86. def snowflakeStage: Option[String]

    User-specified Snowflake stage name for data transfer operations.

    User-specified Snowflake stage name for data transfer operations. When specified, this stage will be used instead of creating a temporary stage.

  87. def stagingTableNameRemoveQuotesOnly: Boolean
  88. def storagePath: Option[String]
  89. def streamingStage: Option[String]
  90. def supportAWSStageEndPoint: Boolean
  91. def supportShareConnection: Boolean
  92. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  93. def table: Option[TableName]

    The Snowflake table to be used as the target when loading or writing data.

  94. def temporaryAWSCredentials: Option[AWSCredentials]

    Temporary AWS credentials which are passed to Snowflake.

    Temporary AWS credentials which are passed to Snowflake. These only need to be supplied by the user when Hadoop is configured to authenticate to S3 via IAM roles assigned to EC2 instances.

  95. def temporaryAzureStorageCredentials: Option[StorageCredentialsSharedAccessSignature]

    SAS Token to be passed to Snowflake to access data in Azure storage.

    SAS Token to be passed to Snowflake to access data in Azure storage. We currently don't support full storage account key so this has to be provided if customer would like to load data through their storage account directly.

  96. def toSnowflakeSchema(stagingSchema: StructType): StructType
  97. def toStagingSchema(snowflakeSchema: StructType): StructType
  98. def toString(): String
    Definition Classes
    MergedParameters → AnyRef → Any
  99. def treatDecimalAsLong: Boolean
  100. def trimSpace: Boolean

    Boolean that specifies whether to remove white space from String fields Defaults to false

  101. def truncateColumns(): Boolean

    Whether or not to have TRUNCATE_COLUMNS in the COPY statement generated by the Spark connector.

  102. def truncateTable: Boolean

    Truncate table when overwriting.

    Truncate table when overwriting. Keep the table schema

  103. def uploadChunkSize: Int
  104. def useAWSRegionURL: Boolean
  105. def useAwsMultiplePartsUpload: Boolean
  106. def useCopyUnload: Boolean
  107. def useExponentialBackoff: Boolean
  108. def useJsonInWrite(): Boolean

    Use JSON format when loading structured data from Spark to Snowflake

  109. def useParquetInWrite(): Boolean

    Use parquet format when loading data from Spark to Snowflake

  110. def useParseJsonForWrite: Boolean
  111. def useProxy: Boolean

    Proxy related parameters.

  112. def useStagingTable: Boolean

    When true, data is always loaded into a new temporary table when performing an overwrite.

    When true, data is always loaded into a new temporary table when performing an overwrite. This is to ensure that the whole load process succeeds before dropping any data from Snowflake, which can be useful if, in the event of failures, stale data is better than no data for your systems.

    Defaults to true.

  113. lazy val usingExternalStage: Boolean
  114. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  115. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  116. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped