Consumer

zio.kafka.consumer.Consumer
See theConsumer companion trait
object Consumer

Attributes

Companion
trait
Graph
Supertypes
class Object
trait Matchable
class Any
Self type
Consumer.type

Members list

Type members

Classlikes

See ConsumerSettings.withOffsetRetrieval.

See ConsumerSettings.withOffsetRetrieval.

Attributes

Companion
trait
Supertypes
trait Sum
trait Mirror
class Object
trait Matchable
class Any
Self type
sealed trait AutoOffsetStrategy

Attributes

Companion
object
Supertypes
class Object
trait Matchable
class Any
Known subtypes
object Earliest
object Latest
object None
Self type
case object CommitTimeout extends RuntimeException, NoStackTrace

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
trait NoStackTrace
class RuntimeException
class Exception
class Throwable
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type

Attributes

Companion
trait
Supertypes
trait Sum
trait Mirror
class Object
trait Matchable
class Any
Self type
sealed trait OffsetRetrieval

See ConsumerSettings.withOffsetRetrieval.

See ConsumerSettings.withOffsetRetrieval.

Attributes

Companion
object
Supertypes
class Object
trait Matchable
class Any
Known subtypes
class Auto
class Manual

Types

A callback for consumer diagnostic events.

A callback for consumer diagnostic events.

Attributes

Value members

Concrete methods

def consumeWith[R : Tag, R1 : Tag, K, V](settings: ConsumerSettings, subscription: Subscription, keyDeserializer: Deserializer[R, K], valueDeserializer: Deserializer[R, V], commitRetryPolicy: Schedule[Any, Any, Any])(f: (ConsumerRecord[K, V]) => URIO[R1, Unit]): RIO[R & R1, Unit]

Execute an effect for each record and commit the offset after processing

Execute an effect for each record and commit the offset after processing

This method is the easiest way of processing messages on a Kafka topic.

Messages on a single partition are processed sequentially, while the processing of multiple partitions happens in parallel.

Offsets are committed after execution of the effect. They are batched when a commit action is in progress to avoid backpressuring the stream. When commits fail due to a org.apache.kafka.clients.consumer.RetriableCommitFailedException they are retried according to commitRetryPolicy

The effect should absorb any failures. Failures should be handled by retries or ignoring the error, which will result in the Kafka message being skipped.

Messages are processed with 'at least once' consistency: it is not guaranteed that every message that is processed by the effect has a corresponding offset commit before stream termination.

Usage example:

val settings: ConsumerSettings = ???
val subscription = Subscription.Topics(Set("my-kafka-topic"))

val consumerIO = Consumer.consumeWith(settings, subscription, Serdes.string, Serdes.string) { record =>
 // Process the received record here
 putStrLn(s"Received record: ${record.key()}: ${record.value()}")
}

Type parameters

K

Type of keys (an implicit Deserializer should be in scope)

R

Environment for the consuming effect

R1

Environment for the deserializers

V

Type of values (an implicit Deserializer should be in scope)

Value parameters

commitRetryPolicy

Retry commits that failed due to a RetriableCommitFailedException according to this schedule

f

Function that returns the effect to execute for each message. It is passed the org.apache.kafka.clients.consumer.ConsumerRecord.

keyDeserializer

Deserializer for the key of the messages

settings

Settings for creating a Consumer

subscription

Topic subscription parameters

valueDeserializer

Deserializer for the value of the messages

Attributes

Returns

Effect that completes with a unit value only when interrupted. May fail when the Consumer fails.

Create a zio-kafka Consumer from an org.apache.kafka KafkaConsumer.

Create a zio-kafka Consumer from an org.apache.kafka KafkaConsumer.

You are responsible for all of the following:

  • creating and closing the KafkaConsumer,
  • making sure auto.commit is disabled,
  • creating access as a fair semaphore with a single permit,
  • acquire a permit from access before using the consumer, and release it afterward,
  • not using the following consumer methods: subscribe, unsubscribe, assign, poll, commit*, seek, pause, resume, and enforceRebalance,
  • keeping the consumer config given to the java consumer in sync with the properties in settings (for example by constructing settings with ConsumerSettings(bootstrapServers).withProperties(config)).

Any deviation of these rules is likely to cause hard to track errors.

Semaphore access is shared between you and the zio-kafka consumer. Use it as short as possible; while you hold a permit the zio-kafka consumer is blocked.

Value parameters

access

A Semaphore with 1 permit.

javaConsumer

Consumer

settings

Settings

Attributes

A new consumer.

A new consumer.

Attributes

def runWithGracefulShutdown[R1, R2, E, A](control: StreamControl[R1, E, A], shutdownTimeout: Duration)(withStream: (ZStream[R1, E, A]) => ZIO[R2 & Scope, E, Any]): ZIO[R1 & R2, E, Any]

Takes a StreamControl for some stream and runs the given ZIO workflow on that stream such that, when interrupted, stops fetching records and gracefully waits for the ZIO workflow to complete.

Takes a StreamControl for some stream and runs the given ZIO workflow on that stream such that, when interrupted, stops fetching records and gracefully waits for the ZIO workflow to complete.

This is useful for running streams from within your application's Main class, such that streams are cleanly stopped when the application is shutdown (for example by your container runtime).

WARNING: this is an EXPERIMENTAL API and may disappear or change in an incompatible way without notice in any zio-kafka version.

Value parameters

shutdownTimeout

Timeout for the workflow to complete after initiating the graceful shutdown

streamControl

Result of one of the Consumer's methods returning a StreamControl

withStream

Takes the stream as input and returns a ZIO workflow that processes the stream. As in most programs the given workflow runs until an external interruption, the result value (Any type) is meaningless. withStream is typically something like:

   stream => stream.mapZIO(record => ZIO.debug(record))
                   .mapZIO(record => record.offset.commit)

Attributes

Concrete fields

A diagnostics implementation that does nothing.

A diagnostics implementation that does nothing.

Attributes

val offsetBatches: ZSink[Any, Nothing, Offset, Nothing, OffsetBatch]