Package org.apache.beam.sdk.io.solr

Class SolrIO.Write

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData
    Enclosing class:
    SolrIO
    public abstract static class SolrIO.Write
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.solr.common.SolrInputDocument>,​org.apache.beam.sdk.values.PDone>
    A PTransform writing data to Solr.
    See Also:
    Serialized Form

    • Field Summary

      • Fields inherited from class org.apache.beam.sdk.transforms.PTransform

        annotations, displayData, name, resourceHints

    • Constructor Summary

      Constructors 
      Constructor Description
      Write()  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      org.apache.beam.sdk.values.PDone expand​(org.apache.beam.sdk.values.PCollection<org.apache.solr.common.SolrInputDocument> input)  
      SolrIO.Write to​(java.lang.String collection)
      Provide name of collection while reading from Solr.
      SolrIO.Write withConnectionConfiguration​(SolrIO.ConnectionConfiguration connectionConfiguration)
      Provide the Solr connection configuration object.
      SolrIO.Write withMaxBatchSize​(int batchSize)
      Provide a maximum size in number of documents for the batch.
      SolrIO.Write withRetryConfiguration​(SolrIO.RetryConfiguration retryConfiguration)
      Provides configuration to retry a failed batch call to Solr.

      • Methods inherited from class org.apache.beam.sdk.transforms.PTransform

        addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

    • Constructor Detail

      • Write

        public Write()

    • Method Detail

      • to

        public SolrIO.Write to​(java.lang.String collection)
        Provide name of collection while reading from Solr.
        Parameters:
        collection - the collection toward which the requests will be issued

      • withMaxBatchSize

        public SolrIO.Write withMaxBatchSize​(int batchSize)
        Provide a maximum size in number of documents for the batch. Depending on the execution engine, size of bundles may vary, this sets the maximum size. Change this if you need to have smaller batch. Default max batch size is 1000.
        Parameters:
        batchSize - maximum batch size in number of documents

      • withRetryConfiguration

        public SolrIO.Write withRetryConfiguration​(SolrIO.RetryConfiguration retryConfiguration)
        Provides configuration to retry a failed batch call to Solr. A batch is considered as failed if the underlying CloudSolrClient surfaces HttpSolrClient.RemoteSolrException, SolrServerException or IOException. Users should consider that retrying might compound the underlying problem which caused the initial failure. Users should also be aware that once retrying is exhausted the error is surfaced to the runner which may then opt to retry the current partition in entirety or abort if the max number of retries of the runner is completed. Retrying uses an exponential backoff algorithm, with minimum backoff of 5 seconds and then surfacing the error once the maximum number of retries or maximum configuration duration is exceeded.

        Example use: 

        
 SolrIO.write()
   .withRetryConfiguration(SolrIO.RetryConfiguration.create(10, Duration.standardMinutes(3))
   ...
        Parameters:
        retryConfiguration - the rules which govern the retry behavior
        Returns:
        the SolrIO.Write with retrying configured

      • expand

        public org.apache.beam.sdk.values.PDone expand​(org.apache.beam.sdk.values.PCollection<org.apache.solr.common.SolrInputDocument> input)
        Specified by:
        expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.solr.common.SolrInputDocument>,​org.apache.beam.sdk.values.PDone>