Package org.apache.flink.connector.dynamodb.sink

Class DynamoDbSink<InputT>

  • Type Parameters:
    InputT - Type of the elements handled by this sink
    All Implemented Interfaces:
    Serializable, org.apache.flink.api.connector.sink2.Sink<InputT>, org.apache.flink.api.connector.sink2.StatefulSink<InputT,​org.apache.flink.connector.base.sink.writer.BufferedRequestState<DynamoDbWriteRequest>>
    @PublicEvolving
public class DynamoDbSink<InputT>
extends org.apache.flink.connector.base.sink.AsyncSinkBase<InputT,​DynamoDbWriteRequest>
    A DynamoDB Sink that performs async requests against a destination table using the buffering protocol specified in AsyncSinkBase.

    The sink internally uses a DynamoDbAsyncClient to communicate with the AWS endpoint.

    The behaviour of the buffering may be specified by providing configuration during the sink build time.

    • maxBatchSize: the maximum size of a batch of entries that may be written to DynamoDb. DynamoDB client supports only up to 25 elements in the batch.
    • maxInFlightRequests: the maximum number of in flight requests that may exist, if any more in flight requests need to be initiated once the maximum has been reached, then it will be blocked until some have completed
    • maxBufferedRequests: the maximum number of elements held in the buffer, requests to sink will backpressure while the number of elements in the buffer is at the maximum
    • maxBatchSizeInBytes: this setting will not have any effect on DynamoDBSink batch implementation
    • maxTimeInBufferMS: the maximum amount of time an entry is allowed to live in the buffer, if any element reaches this age, the entire buffer will be flushed immediately
    • maxRecordSizeInBytes: this setting will not have any effect on DynamoDBSink batch implementation
    • failOnError: when an exception is encountered while persisting to DynamoDb, the job will fail immediately if failOnError is set
    • overwriteByPartitionKeys: list of attribute key names for the sink to deduplicate on if you want to bypass the no duplication limitation of a single batch write request. Batching DynamoDB sink will drop request items in the buffer if their primary keys(composite) values are the same as the newly added ones. The newer request item in a single batch takes precedence.

    Please see the writer implementation in DynamoDbSinkWriter

    See Also:
    Serialized Form

    • Nested Class Summary

      • Nested classes/interfaces inherited from interface org.apache.flink.api.connector.sink2.Sink

        org.apache.flink.api.connector.sink2.Sink.InitContext

      • Nested classes/interfaces inherited from interface org.apache.flink.api.connector.sink2.StatefulSink

        org.apache.flink.api.connector.sink2.StatefulSink.StatefulSinkWriter<InputT extends Object,​WriterStateT extends Object>, org.apache.flink.api.connector.sink2.StatefulSink.WithCompatibleState

    • Constructor Summary

      Constructors 
      Modifier Constructor Description
      protected DynamoDbSink​(org.apache.flink.connector.base.sink.writer.ElementConverter<InputT,​DynamoDbWriteRequest> elementConverter, int maxBatchSize, int maxInFlightRequests, int maxBufferedRequests, long maxBatchSizeInBytes, long maxTimeInBufferMS, long maxRecordSizeInBytes, boolean failOnError, String tableName, List<String> overwriteByPartitionKeys, Properties dynamoDbClientProperties)  

    • Constructor Detail

      • DynamoDbSink

        protected DynamoDbSink​(org.apache.flink.connector.base.sink.writer.ElementConverter<InputT,​DynamoDbWriteRequest> elementConverter,
                       int maxBatchSize,
                       int maxInFlightRequests,
                       int maxBufferedRequests,
                       long maxBatchSizeInBytes,
                       long maxTimeInBufferMS,
                       long maxRecordSizeInBytes,
                       boolean failOnError,
                       String tableName,
                       List<String> overwriteByPartitionKeys,
                       Properties dynamoDbClientProperties)

    • Method Detail

      • createWriter

        @Internal
public org.apache.flink.api.connector.sink2.StatefulSink.StatefulSinkWriter<InputT,​org.apache.flink.connector.base.sink.writer.BufferedRequestState<DynamoDbWriteRequest>> createWriter​(org.apache.flink.api.connector.sink2.Sink.InitContext context)
                                                                                                                                                                                       throws IOException
        Throws:
        IOException

      • restoreWriter

        @Internal
public org.apache.flink.api.connector.sink2.StatefulSink.StatefulSinkWriter<InputT,​org.apache.flink.connector.base.sink.writer.BufferedRequestState<DynamoDbWriteRequest>> restoreWriter​(org.apache.flink.api.connector.sink2.Sink.InitContext context,
                                                                                                                                                                                               Collection<org.apache.flink.connector.base.sink.writer.BufferedRequestState<DynamoDbWriteRequest>> recoveredState)
                                                                                                                                                                                        throws IOException
        Throws:
        IOException

      • getWriterStateSerializer

        @Internal
public org.apache.flink.core.io.SimpleVersionedSerializer<org.apache.flink.connector.base.sink.writer.BufferedRequestState<DynamoDbWriteRequest>> getWriterStateSerializer()