ConsumeAzureEventHub 2.3.0¶
BUNDLE¶
org.apache.nifi | nifi-azure-nar
DESCRIPTION¶
Receives messages from Microsoft Azure Event Hubs with checkpointing to ensure consistent event processing. Checkpoint tracking avoids consuming a message multiple times and enables reliable resumption of processing in the event of intermittent network failures. Checkpoint tracking requires external storage and provides the preferred approach to consuming messages from Azure Event Hubs. In clustered environment, ConsumeAzureEventHub processor instances form a consumer group and the messages are distributed among the cluster nodes (each message is processed on one cluster node only).
INPUT REQUIREMENT¶
FORBIDDEN
Supports Sensitive Dynamic Properties¶
false
PROPERTIES¶
Property |
Description |
|---|---|
Service Bus Endpoint |
To support namespaces not in the default windows.net domain. |
Transport Type |
Advanced Message Queuing Protocol Transport Type for communication with Azure Event Hubs |
checkpoint-strategy |
Specifies which strategy to use for storing and retrieving partition ownership and checkpoint information for each partition. |
event-hub-batch-size |
The number of messages to process within a NiFi session. This parameter affects throughput and consistency. NiFi commits its session and Event Hubs checkpoints after processing this number of messages. If NiFi session is committed, but fails to create an Event Hubs checkpoint, then it is possible that the same messages will be received again. The higher number, the higher throughput, but possibly less consistent. |
event-hub-consumer-group |
The name of the consumer group to use. |
event-hub-initial-offset |
Specify where to start receiving messages if offset is not yet stored in the checkpoint store. |
event-hub-message-receive-timeout |
The amount of time this consumer should wait to receive the Batch Size before returning. |
event-hub-name |
The name of the event hub to pull messages from. |
event-hub-namespace |
The namespace that the Azure Event Hubs is assigned to. This is generally equal to <Event Hub Names>-ns. |
event-hub-prefetch-count |
|
event-hub-shared-access-policy-name |
The name of the shared access policy. This policy must have Listen claims. |
event-hub-shared-access-policy-primary-key |
The key of the shared access policy. Either the primary or the secondary key can be used. |
proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. Supported proxies: HTTP + AuthN |
record-reader |
The Record Reader to use for reading received messages. The event hub name can be referred by Expression Language ‘${eventhub.name}’ to access a schema. |
record-writer |
The Record Writer to use for serializing Records to an output FlowFile. The event hub name can be referred by Expression Language ‘${eventhub.name}’ to access a schema. If not specified, each message will create a FlowFile. |
storage-account-key |
The Azure Storage account key to store event hub consumer group state. |
storage-account-name |
Name of the Azure Storage account to store event hub consumer group state. |
storage-container-name |
Name of the Azure Storage container to store the event hub consumer group state. If not specified, event hub name is used. |
storage-sas-token |
The Azure Storage SAS token to store Event Hub consumer group state. Always starts with a ? character. |
use-managed-identity |
Choose whether or not to use the managed identity of Azure VM/VMSS |
STATE MANAGEMENT¶
SCOPES |
DESCRIPTION |
|---|---|
CLUSTER |
Local state is used to store the client id. Cluster state is used to store partition ownership and checkpoint information when component state is configured as the checkpointing strategy. |
LOCAL |
Local state is used to store the client id. Cluster state is used to store partition ownership and checkpoint information when component state is configured as the checkpointing strategy. |
RELATIONSHIPS¶
NAME |
DESCRIPTION |
|---|---|
success |
FlowFiles received from Event Hub. |
WRITES ATTRIBUTES¶
NAME |
DESCRIPTION |
|---|---|
eventhub.enqueued.timestamp |
The time (in milliseconds since epoch, UTC) at which the message was enqueued in the event hub |
eventhub.offset |
The offset into the partition at which the message was stored |
eventhub.sequence |
The sequence number associated with the message |
eventhub.name |
The name of the event hub from which the message was pulled |
eventhub.partition |
The name of the partition from which the message was pulled |
eventhub.property.* |
The application properties of this message. IE: ‘application’ would be ‘eventhub.property.application’ |