Type Members

1.  case class EventProcessingConfig(windowing: Windowing) extends Product with Serializable

   Configures how events are fed into the EventProcessor

   Configures how events are fed into the EventProcessor

   windowing

   Whether to open a new EventProcessor to handle a timed window of events (e.g. for the transformer) or whether to feed events to a single EventProcessor in a continuous endless stream (e.g. Enrich)

   Note

   This class is not for user-facing configuration. The application itself should have an opinion on these parameters, e.g. Enrich always wants NoWindowing, but Transformer always wants Windowing

2.  type EventProcessor[F[_]] = (Stream[F, TokenedEvents]) => Stream[F, Token]

   An application that processes a source of events

   An application that processes a source of events

   The EventProcessor is implemented by the specific application (e.g. Enrich or Transformer). Once implemented, we can create a runnable program by pairing it with a SourceAndAck.

   The SourceAndAck provides the EventProcessor with events and tokens. The EventProcessor must emit the tokens after it has fully processed the events.

3.  trait SourceAndAck[F[_]] extends AnyRef

   The machinery for sourcing events from and external stream and then acking/checkpointing them.

   The machinery for sourcing events from and external stream and then acking/checkpointing them.

   Implementations of this trait are provided by the sources library (e.g. kinesis, kafka, pubsub) whereas implementations of EventProcessor are provided by the specific application (e.g. enrich, transformer, loaders)

4.  case class TokenedEvents(events: Chunk[ByteBuffer], ack: Token, earliestSourceTstamp: Option[Instant]) extends Product with Serializable

   The events as they are fed into a EventProcessor

   The events as they are fed into a EventProcessor

   events

   Each item in the Chunk is an event read from the external stream, before parsing

   ack

   The EventProcessor must emit this token after it has fully processed the batch of events. When the EventProcessor emits the token, it is an instruction to the SourceAndAck to ack/checkpoint the events.

   earliestSourceTstamp

   The timestamp that an event was originally written to the source stream. Used for calculating the latency metric.