Class SharePartition

java.lang.Object
kafka.server.share.SharePartition
public class SharePartition extends Object
The SharePartition is used to track the state of a partition that is shared between multiple consumers. The class maintains the state of the records that have been fetched from the leader and are in-flight.

  • Method Summary

    Modifier and Type
    Method
    Description
    CompletableFuture<Void>
    acknowledge(String memberId, List<org.apache.kafka.server.share.acknowledge.ShareAcknowledgementBatch> acknowledgementBatches)
    Acknowledge the fetched records for the share partition.
    org.apache.kafka.server.share.fetch.ShareAcquiredRecords
    acquire(String memberId, org.apache.kafka.clients.consumer.internals.ShareAcquireMode shareAcquireMode, int batchSize, int maxFetchRecords, long fetchOffset, org.apache.kafka.server.storage.log.FetchPartitionData fetchPartitionData, org.apache.kafka.server.storage.log.FetchIsolation isolationLevel)
    Acquire the fetched records for the share partition.
    boolean
    maybeAcquireFetchLock(org.apache.kafka.common.Uuid fetchId)
    Prior to fetching records from the leader, the fetch lock is acquired to ensure that the same share partition is not fetched concurrently by multiple clients.
    CompletableFuture<Void>
    maybeInitialize()
    May initialize the share partition by reading the state from the persister.
    long
    nextFetchOffset()
    The next fetch offset is used to determine the next offset that should be fetched from the leader.
    CompletableFuture<Void>
    releaseAcquiredRecords(String memberId)
    Release the acquired records for the share partition.

    Methods inherited from class Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • maybeInitialize

      public CompletableFuture<Void> maybeInitialize()
      May initialize the share partition by reading the state from the persister. The share partition is initialized only if the state is in the EMPTY state. If the share partition is in ACTIVE state, the method completes the future successfully. For other states, the method completes the future with exception, which might be re-triable.
      Returns:
      The method returns a future which is completed when the share partition is initialized or completes with an exception if the share partition is in non-initializable state.

    • nextFetchOffset

      public long nextFetchOffset()
      The next fetch offset is used to determine the next offset that should be fetched from the leader. The offset should be the next offset after the last fetched batch but there could be batches/ offsets that are either released by acknowledge API or lock timed out hence the next fetch offset might be different from the last batch next offset. Hence, method checks if the next fetch offset should be recomputed else returns the last computed next fetch offset.
      Returns:
      The next fetch offset that should be fetched from the leader.

    • acquire

      public org.apache.kafka.server.share.fetch.ShareAcquiredRecords acquire(String memberId, org.apache.kafka.clients.consumer.internals.ShareAcquireMode shareAcquireMode, int batchSize, int maxFetchRecords, long fetchOffset, org.apache.kafka.server.storage.log.FetchPartitionData fetchPartitionData, org.apache.kafka.server.storage.log.FetchIsolation isolationLevel)
      Acquire the fetched records for the share partition. The acquired records are added to the in-flight records and the next fetch offset is updated to the next offset that should be fetched from the leader.

      The method always acquire the full batch records. The cache state can consist of multiple full batches as a single batch. This behavior is driven by client configurations (batch size and max fetch records) and allows for efficient client acknowledgements. However, partial batches can exist in the cache only after a leader change and partial acknowledgements have been persisted prior leader change. In such case, when a share partition loses track of a batch's start and end offsets (e.g., after a leader change and partial acknowledgements), the cache stores the batch based on the offset range provided by the persister. This method handles these special batches by maintaining this range up to the last offset returned by the persister. No special handling is required afterward; the cache will eventually return to managing full batches.

      For compacted topics, batches may be non-contiguous, and records within cached batches may contain gaps. Because this method operates at the batch level, it acquires entire batches and relies on the client to report any gaps in the data. Whether non-contiguous batches are acquired depends on the first and last offsets of the fetched batches. Batches outside of this boundary will never be acquired. For instance, if fetched batches cover offsets [0-9 and 20-29], and the configured batch size and maximum fetch records are large enough (greater than 30 in this example), the intervening batch [10-19] will be acquired. Since full fetched batch is acquired, the client is responsible for reporting any data gaps. However, if the [0-9] and [20-29] ranges are fetched in separate calls to this method, the [10-19] batch will not be acquired and cannot exist in the cache.

      However, for compacted topics, previously acquired batches (e.g., due to acquisition lock timeout or explicit client release) might become available for acquisition again. But subsequent fetches may reveal that these batches, or parts of them, have been removed by compaction. Because this method works with whole batches, the disappearance of individual offsets within a batch requires no special handling; the batch will be re-acquired, and the client will report the gaps. But if an entire batch has been compacted away, this method must archive it in the cache to allow the Share Partition Start Offset (SPSO) to progress. This is accomplished by comparing the fetchOffset (the offset from which the log was read) with the first base offset of the fetch response. Any batches from fetchOffset to first base offset of the fetch response are archived.

      Parameters:
      memberId - The member id of the client that is fetching the record.
      shareAcquireMode - The share acquire mode to acquire the records.
      batchSize - The number of records per acquired records batch.
      maxFetchRecords - The maximum number of records that should be acquired, this is a soft limit and the method might acquire more records than the maxFetchRecords, if the records are already part of the same fetch batch.
      fetchOffset - The fetch offset for which the records are fetched.
      fetchPartitionData - The fetched records for the share partition.
      isolationLevel - The isolation level for the share fetch request.
      Returns:
      The acquired records for the share partition.

    • acknowledge

      public CompletableFuture<Void> acknowledge(String memberId, List<org.apache.kafka.server.share.acknowledge.ShareAcknowledgementBatch> acknowledgementBatches)
      Acknowledge the fetched records for the share partition. The accepted batches are removed from the in-flight records once persisted. The next fetch offset is updated to the next offset that should be fetched from the leader, if required.
      Parameters:
      memberId - The member id of the client that is fetching the record.
      acknowledgementBatches - The acknowledgement batch list for the share partition.
      Returns:
      A future which is completed when the records are acknowledged.

    • releaseAcquiredRecords

      public CompletableFuture<Void> releaseAcquiredRecords(String memberId)
      Release the acquired records for the share partition. The next fetch offset is updated to the next offset that should be fetched from the leader.
      Parameters:
      memberId - The member id of the client whose records shall be released.
      Returns:
      A future which is completed when the records are released.

    • maybeAcquireFetchLock

      public boolean maybeAcquireFetchLock(org.apache.kafka.common.Uuid fetchId)
      Prior to fetching records from the leader, the fetch lock is acquired to ensure that the same share partition is not fetched concurrently by multiple clients. The fetch lock is released once the records are fetched and acquired.
      Parameters:
      fetchId - - the caller's id that is trying to acquire the fetch lock.
      Returns:
      A boolean which indicates whether the fetch lock is acquired.