` and update `wd.authn.accessTokenEndpoint` to point to the proxy's token endpoint.
## Step 5: Connect and run queries
### Connect to the Workday LDQ service
Open a connection using the `config` object. A successful connection confirms that authentication, networking, and configuration are all correct.
```python
from workday_ldq import create_connection
connection = create_connection(config)
print("Connected successfully!")
```
### Run a query
Use a cursor to send SQL to the Workday data service. Queries run against Workday's Unified Data Catalog, not Snowflake tables.
```python
cursor = connection.cursor()
cursor.execute("SELECT COUNT(*) FROM workday_core.public.worker")
results = cursor.fetchall()
print(results)
cursor.close()
```
### Load results into a DataFrame
```python
cursor = connection.cursor()
cursor.execute("SELECT * FROM workday_core.public.worker LIMIT 100")
columns = [desc[0] for desc in cursor.description]
rows = cursor.fetchall()
df = pd.DataFrame(rows, columns=columns)
cursor.close()
df.head()
```
### Close the connection
Always close the connection when you're finished to release resources on both the Snowflake and Workday sides.
```python
connection.close()
```
## Sample queries
These queries target Workday data, not Snowflake tables, and must be run through the LDQ connector. Paste each query string into a `cursor.execute()` call as shown above, not directly in a Snowflake Worksheet.
The examples below use `workday_core.public` as the catalog and schema. Your environment may use different names. Always run the discovery queries first to confirm what's available in your tenant.
### Discover available catalogs and schemas
```sql
SHOW CATALOGS
```
```sql
SHOW SCHEMAS IN workday_core
```
```sql
SHOW TABLES IN workday_core.public
```
### Count all workers
```sql
SELECT COUNT(*) AS total_workers
FROM workday_core.public.worker
```
### List active workers with job titles
```sql
SELECT
w.worker_id,
w.full_name,
w.employee_type,
jp.job_title
FROM workday_core.public.worker w
JOIN workday_core.public.job_profile jp
ON w.job_profile_id = jp.job_profile_id
WHERE w.active = TRUE
LIMIT 50
```
### Headcount by management level
```sql
SELECT
management_level,
COUNT(*) AS headcount
FROM workday_core.public.worker
WHERE active = TRUE
GROUP BY management_level
ORDER BY headcount DESC
```
### Workers hired in the last 90 days
```sql
SELECT
worker_id,
full_name,
hire_date,
business_title
FROM workday_core.public.worker
WHERE hire_date >= CURRENT_DATE - INTERVAL '90' DAY
ORDER BY hire_date DESC
```
In EA, available objects are limited to Workforce and Talent. Row-level security controls are planned for GA. Table and column-level access is controlled by the ISU's security group in Workday.
## Next steps
With data in a DataFrame, you can use [Cortex Code](/user-guide/data-integration/zero-copy/workday/cortex-code) to write queries, build visualizations, and get AI-assisted analysis of your Workday data.
---
title: ConnectWebSocket 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/connectwebsocket.md
section: Loading & Unloading Data
---
# ConnectWebSocket 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-websocket-processors-nar
## Description
Acts as a WebSocket client endpoint to interact with a remote WebSocket server. FlowFiles are transferred to downstream relationships according to received message types as WebSocket client configured with this processor receives messages from remote WebSocket server. If a new flowfile is passed to the processor, the previous sessions will be closed and any data being sent will be aborted.
## Tags
WebSocket, consume, listen, subscribe
## Input Requirement
ALLOWED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| websocket-client-controller-service |
A WebSocket CLIENT Controller Service which can connect to a WebSocket server. |
| websocket-client-id |
The client ID to identify WebSocket session. It should be unique within the WebSocket Client Controller Service. Otherwise, it throws WebSocketConfigurationException when it gets started. |
## Relationships
| Name |
Description |
| binary message |
The WebSocket binary message output |
| connected |
The WebSocket session is established |
| disconnected |
The WebSocket session is disconnected |
| failure |
FlowFile holding connection configuration attributes (like URL or HTTP headers) in case of connection failure |
| success |
FlowFile holding connection configuration attributes (like URL or HTTP headers) in case of successful connection |
| text message |
The WebSocket text message output |
## Writes attributes
| Name |
Description |
| websocket.controller.service.id |
WebSocket Controller Service id. |
| websocket.session.id |
Established WebSocket session id. |
| websocket.endpoint.id |
WebSocket endpoint id. |
| websocket.local.address |
WebSocket client address. |
| websocket.remote.address |
WebSocket server address. |
| websocket.message.type |
TEXT or BINARY. |
---
title: ConsumeAMQP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumeamqp.md
section: Loading & Unloading Data
---
# ConsumeAMQP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-amqp-nar
## Description
Consumes AMQP Messages from an AMQP Broker using the AMQP 0.9.1 protocol. Each message that is received from the AMQP Broker will be emitted as its own FlowFile to the 'success' relationship.
## Tags
amqp, consume, get, message, rabbit, receive
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| AMQP Version |
AMQP Version. Currently only supports AMQP v0.9.1. |
| Auto-Acknowledge Messages |
If false (Non-Auto-Acknowledge), the messages will be acknowledged by the processor after transferring the FlowFiles to success and committing the NiFi session. Non-Auto-Acknowledge mode provides 'at-least-once' delivery semantics. If true (Auto-Acknowledge), messages that are delivered to the AMQP Client will be auto-acknowledged by the AMQP Broker just after sending them out. This generally will provide better throughput but will also result in messages being lost upon restart/crash of the AMQP Broker, NiFi or the processor. Auto-Acknowledge mode provides 'at-most-once' delivery semantics and it is recommended only if losing messages is acceptable. |
| Batch Size |
The maximum number of messages that should be processed in a single session. Once this many messages have been received (or once no more messages are readily available), the messages received will be transferred to the 'success' relationship and the messages will be acknowledged to the AMQP Broker. Setting this value to a larger number could result in better performance, particularly for very small messages, but can also result in more messages being duplicated upon sudden restart of NiFi. |
| Brokers |
A comma-separated list of known AMQP Brokers in the format <host>:<port> (e.g., localhost:5672). If this is set, Host Name and Port are ignored. Only include hosts from the same AMQP cluster. |
| Client Certificate Authentication Enabled |
Authenticate using the SSL certificate rather than user name/password. |
| Header Key Prefix |
Text to be prefixed to header keys as the are added to the FlowFile attributes. Processor will append '.' to the value of this property |
| Header Output Format |
Defines how to output headers from the received message |
| Header Separator |
The character that is used to separate key-value for header in String. The value must be only one character. |
| Host Name |
Network address of AMQP broker (e.g., localhost). If Brokers is set, then this property is ignored. |
| Max Inbound Message Body Size |
Maximum body size of inbound (received) messages. |
| Password |
Password used for authentication and authorization. |
| Port |
Numeric value identifying Port of AMQP broker (e.g., 5671). If Brokers is set, then this property is ignored. |
| Prefetch Count |
The maximum number of unacknowledged messages for the consumer. If consumer has this number of unacknowledged messages, AMQP broker will no longer send new messages until consumer acknowledges some of the messages already delivered to it. Allowed values: from 0 to 65535.0 means no limit |
| Queue |
The name of the existing AMQP Queue from which messages will be consumed. Usually pre-defined by AMQP administrator. |
| Remove Curly Braces |
If true Remove Curly Braces, Curly Braces in the header will be automatically remove. |
| SSL Context Service |
The SSL Context Service used to provide client certificate information for TLS/SSL connections. |
| Username |
Username used for authentication and authorization. |
| Virtual Host |
Virtual Host name which segregates AMQP system for enhanced security. |
## Relationships
| Name |
Description |
| success |
All FlowFiles that are received from the AMQP queue are routed to this relationship |
## Writes attributes
| Name |
Description |
| amqp$appId |
The App ID field from the AMQP Message |
| amqp$contentEncoding |
The Content Encoding reported by the AMQP Message |
| amqp$contentType |
The Content Type reported by the AMQP Message |
| amqp$headers |
The headers present on the AMQP Message. Added only if processor is configured to output this attribute. |
| <Header Key Prefix>.<attribute> |
Each message header will be inserted with this attribute name, if processor is configured to output headers as attribute |
| amqp$deliveryMode |
The numeric indicator for the Message's Delivery Mode |
| amqp$priority |
The Message priority |
| amqp$correlationId |
The Message's Correlation ID |
| amqp$replyTo |
The value of the Message's Reply-To field |
| amqp$expiration |
The Message Expiration |
| amqp$messageId |
The unique ID of the Message |
| amqp$timestamp |
The timestamp of the Message, as the number of milliseconds since epoch |
| amqp$type |
The type of message |
| amqp$userId |
The ID of the user |
| amqp$clusterId |
The ID of the AMQP Cluster |
| amqp$routingKey |
The routingKey of the AMQP Message |
| amqp$exchange |
The exchange from which AMQP Message was received |
---
title: ConsumeAzureEventHub 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumeazureeventhub.md
section: Loading & Unloading Data
---
# ConsumeAzureEventHub 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-azure-nar
## Description
Receives messages from Microsoft Azure Event Hubs with checkpointing to ensure consistent event processing. Checkpoint tracking avoids consuming a message multiple times and enables reliable resumption of processing in the event of intermittent network failures. Checkpoint tracking requires external storage and provides the preferred approach to consuming messages from Azure Event Hubs. In clustered environment, ConsumeAzureEventHub processor instances form a consumer group and the messages are distributed among the cluster nodes (each message is processed on one cluster node only).
## Tags
azure, cloud, eventhub, events, microsoft, streaming, streams
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Batch Size |
The number of messages to process within a NiFi session. This parameter affects throughput and consistency. NiFi commits its session and Event Hubs checkpoints after processing this number of messages. If NiFi session is committed, but fails to create an Event Hubs checkpoint, then it is possible that the same messages will be received again. The higher number, the higher throughput, but possibly less consistent. |
| Checkpoint Strategy |
Specifies which strategy to use for storing and retrieving partition ownership and checkpoint information for each partition. |
| Consumer Group |
The name of the consumer group to use. |
| Event Hub Name |
The name of the event hub to pull messages from. |
| Event Hub Namespace |
The namespace that the Azure Event Hubs is assigned to. This is generally equal to <Event Hub Names>-ns. |
| Initial Offset |
Specify where to start receiving messages if offset is not yet stored in the checkpoint store. |
| Message Receive Timeout |
The amount of time this consumer should wait to receive the Batch Size before returning. |
| Prefetch Count |
|
| Record Reader |
The Record Reader to use for reading received messages. The event hub name can be referred by Expression Language '$\{eventhub.name\}' to access a schema. |
| Record Writer |
The Record Writer to use for serializing Records to an output FlowFile. The event hub name can be referred by Expression Language '$\{eventhub.name\}' to access a schema. If not specified, each message will create a FlowFile. |
| Service Bus Endpoint |
To support namespaces not in the default windows.net domain. |
| Shared Access Policy Key |
The key of the shared access policy. Either the primary or the secondary key can be used. |
| Shared Access Policy Name |
The name of the shared access policy. This policy must have Listen claims. |
| Storage Account Key |
The Azure Storage account key to store event hub consumer group state. |
| Storage Account Name |
Name of the Azure Storage account to store event hub consumer group state. |
| Storage Container Name |
Name of the Azure Storage container to store the event hub consumer group state. If not specified, event hub name is used. |
| Storage SAS Token |
The Azure Storage SAS token to store Event Hub consumer group state. Always starts with a ? character. |
| Transport Type |
Advanced Message Queuing Protocol Transport Type for communication with Azure Event Hubs |
| Use Azure Managed Identity |
Choose whether or not to use the managed identity of Azure VM/VMSS |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## State management
| Scopes |
Description |
| LOCAL |
Local state is used to store the client id. Cluster state is used to store partition ownership and checkpoint information when component state is configured as the checkpointing strategy. |
| CLUSTER |
Local state is used to store the client id. Cluster state is used to store partition ownership and checkpoint information when component state is configured as the checkpointing strategy. |
## Relationships
| Name |
Description |
| success |
FlowFiles received from Event Hub. |
## Writes attributes
| Name |
Description |
| eventhub.enqueued.timestamp |
The time (in milliseconds since epoch, UTC) at which the message was enqueued in the event hub |
| eventhub.offset |
The offset into the partition at which the message was stored |
| eventhub.sequence |
The sequence number associated with the message |
| eventhub.name |
The name of the event hub from which the message was pulled |
| eventhub.partition |
The name of the partition from which the message was pulled |
| eventhub.property.* |
The application properties of this message. IE: 'application' would be 'eventhub.property.application' |
---
title: ConsumeBoxEnterpriseEvents 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumeboxenterpriseevents.md
section: Loading & Unloading Data
---
# ConsumeBoxEnterpriseEvents 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-box-nar
## Description
Consumes Enterprise Events from Box admin_logs_streaming Stream Type. The content of the events is sent to the 'success' relationship as a JSON array. The last known position of the Box stream is stored in the processor state and is used to resume the stream from the last known position when the processor is restarted.
## Tags
box, storage
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Box Client Service |
Controller Service used to obtain a Box API connection. |
| Event Types |
A comma separated list of Enterprise Events to consume. If not set, all Events are consumed. See Additional Details for more information. |
| Start Event Position |
What position to consume the Events from. |
| Start Offset |
The offset to start consuming the Events from. |
## State management
| Scopes |
Description |
| CLUSTER |
The last known position of the Box Event stream is stored in the processor state and is used to resume the stream from the last known position when the processor is restarted. |
## Relationships
| Name |
Description |
| success |
Events received successfully will be sent out this relationship. |
## See also
- [org.apache.nifi.processors.box.ConsumeBoxEvents](/user-guide/data-integration/openflow/processors/consumeboxevents)
- [org.apache.nifi.processors.box.FetchBoxFile](/user-guide/data-integration/openflow/processors/fetchboxfile)
- [org.apache.nifi.processors.box.ListBoxFile](/user-guide/data-integration/openflow/processors/listboxfile)
---
title: ConsumeBoxEvents 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumeboxevents.md
section: Loading & Unloading Data
---
# ConsumeBoxEvents 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-box-nar
## Description
Consumes all events from Box. This processor can be used to capture events such as uploads, modifications, deletions, etc. The content of the events is sent to the 'success' relationship as a JSON array. Events can be dropped in case of NiFi restart or if the queue capacity is exceeded. The last known position of the Box stream is stored in the processor state and is used to resume the stream from the last known position when the processor is restarted.
## Tags
box, storage
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Box Client Service |
Controller Service used to obtain a Box API connection. |
| Queue Capacity |
The maximum size of the internal queue used to buffer events being transferred from the underlying stream to the processor. Setting this value higher allows more messages to be buffered in memory during surges of incoming messages, but increases the total memory used by the processor during these surges. |
## State management
| Scopes |
Description |
| CLUSTER |
The last known position of the Box stream is stored in the processor state and is used to resume the stream from the last known position when the processor is restarted. |
## Relationships
| Name |
Description |
| success |
Events received successfully will be sent out this relationship. |
## See also
- [org.apache.nifi.processors.box.FetchBoxFile](/user-guide/data-integration/openflow/processors/fetchboxfile)
- [org.apache.nifi.processors.box.ListBoxFile](/user-guide/data-integration/openflow/processors/listboxfile)
- [org.apache.nifi.processors.box.PutBoxFile](/user-guide/data-integration/openflow/processors/putboxfile)
---
title: ConsumeElasticsearch 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumeelasticsearch.md
section: Loading & Unloading Data
---
# ConsumeElasticsearch 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-elasticsearch-restapi-nar
## Description
A processor that repeatedly runs a paginated query against a field using a Range query to consume new Documents from an Elasticsearch index/query. The processor will retrieve multiple pages of results until either no more results are available or the Pagination Keep Alive expiration is reached, after which the Range query will automatically update the field constraint based on the last retrieved Document value.
## Tags
elasticsearch, elasticsearch7, elasticsearch8, elasticsearch9, json, page, query, scroll, search
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Additional Filters |
One or more query filters in JSON syntax, not Lucene syntax. Ex: [\{"match":\{"somefield":"somevalue"\}\}, \{"match":\{"anotherfield":"anothervalue"\}\}]. These filters wil be used as part of a Bool query's filter. |
| Aggregation Results Format |
Format of Aggregation output. |
| Aggregation Results Split |
Output a flowfile containing all aggregations or one flowfile for each individual aggregation. |
| Aggregations |
One or more query aggregations (or "aggs"), in JSON syntax. Ex: \{"items": \{"terms": \{"field": "product", "size": 10\}\}\} |
| Client Service |
An Elasticsearch client service to use for running queries. |
| Fields |
Fields of indexed documents to be retrieved, in JSON syntax. Ex: ["user.id", "http.response.*", \{"field": "@timestamp", "format": "epoch_millis"\}] |
| Index |
The name of the index to use. |
| Initial Value |
The initial value to use for the query if the processor has not run previously. If the processor has run previously and stored a value in its state, this property will be ignored. If no value is provided, and the processor has not previously run, no Range query bounds will be used, i.e. all documents will be retrieved in the specified "Sort Order". |
| Initial Value Date Format |
If the "Range Query Field" is a Date field, convert the "Initial Value" to a date with this format. If not specified, Elasticsearch will use the date format provided by the "Range Query Field"'s mapping. For valid syntax, see https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html (https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html) |
| Initial Value Date Time Zone |
If the "Range Query Field" is a Date field, convert the "Initial Value" to UTC with this time zone. Valid values are ISO 8601 UTC offsets, such as "+01:00" or "-08:00", and IANA time zone IDs, such as "Europe/London". |
| Max JSON Field String Length |
The maximum allowed length of a string value when parsing a JSON document or attribute. |
| Output No Hits |
Output a "hits" flowfile even if no hits found for query. If true, an empty "hits" flowfile will be output even if "aggregations" are output. |
| Pagination Keep Alive |
Pagination "keep_alive" period. Period Elasticsearch will keep the scroll/pit cursor alive in between requests (this is not the time expected for all pages to be returned, but the maximum allowed time for requests between page retrievals). |
| Pagination Type |
Pagination method to use. Not all types are available for all Elasticsearch versions, check the Elasticsearch docs to confirm which are applicable and recommended for your service. |
| Query Attribute |
If set, the executed query will be set on each result flowfile in the specified attribute. |
| Range Query Field |
Field to be tracked as part of an Elasticsearch Range query using a "gt" bound match. This field must exist within the Elasticsearch document for it to be retrieved. |
| Script Fields |
Fields to created using script evaluation at query runtime, in JSON syntax. Ex: \{"test1": \{"script": \{"lang": "painless", "source": "doc[ 'price'].value * 2"\}\}, "test2": \{"script": \{"lang": "painless", "source": "doc[ 'price'].value * params.factor", "params": \{"factor": 2.0\}\}\}\} |
| Search Results Format |
Format of Hits output. |
| Search Results Split |
Output a flowfile containing all hits or one flowfile for each individual hit or one flowfile containing all hits from all paged responses. |
| Size |
The maximum number of documents to retrieve in the query. If the query is paginated, this "size" applies to each page of the query, not the "size" of the entire result set. |
| Sort |
Sort results by one or more fields, in JSON syntax. Ex: [\{"price" : \{"order" : "asc", "mode" : "avg"\}\}, \{"post_date" : \{"format": "strict_date_optional_time_nanos"\}\}] |
| Sort Order |
The order in which to sort the "Range Query Field". A "sort" clause for the "Range Query Field" field will be prepended to any provided "Sort" clauses. If a "sort" clause already exists for the "Range Query Field" field, it will not be updated. |
| Type |
The type of this document (used by Elasticsearch for indexing and searching). |
## State management
| Scopes |
Description |
| CLUSTER |
The pagination state (scrollId, searchAfter, pitId, hitCount, pageCount, pageExpirationTimestamp, trackingRangeValue) is retained in between invocations of this processor until the Scroll/PiT has expired (when the current time is later than the last query execution plus the Pagination Keep Alive interval). |
## Relationships
| Name |
Description |
| aggregations |
Aggregations are routed to this relationship. |
| failure |
All flowfiles that fail for reasons unrelated to server availability go to this relationship. |
| hits |
Search hits are routed to this relationship. |
| retry |
All flowfiles that fail due to server/cluster availability go to this relationship. |
## Writes attributes
| Name |
Description |
| mime.type |
application/json |
| page.number |
The number of the page (request), starting from 1, in which the results were returned that are in the output flowfile |
| hit.count |
The number of hits that are in the output flowfile |
| elasticsearch.query.error |
The error message provided by Elasticsearch if there is an error querying the index. |
## See also
- [org.apache.nifi.processors.elasticsearch.PaginatedJsonQueryElasticsearch](/user-guide/data-integration/openflow/processors/paginatedjsonqueryelasticsearch)
- [org.apache.nifi.processors.elasticsearch.SearchElasticsearch](/user-guide/data-integration/openflow/processors/searchelasticsearch)
---
title: ConsumeGCPubSub 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumegcpubsub.md
section: Loading & Unloading Data
---
# ConsumeGCPubSub 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-gcp-nar
## Description
Consumes messages from the configured Google Cloud PubSub subscription. The 'Batch Size' property specified the maximum number of messages that will be pulled from the subscription in a single request. The 'Processing Strategy' property specifies if each message should be its own FlowFile or if messages should be grouped into a single FlowFile. Using the Demarcator strategy will provide best throughput when the format allows it. Using Record lets you convert data format as well as doing schema enforcement. Using the FlowFile strategy will generate one FlowFile per message and will have the message's attributes as FlowFile attributes.
## Tags
consume, gcp, google, google-cloud, message, pubsub
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| GCP Credentials Provider Service |
The Controller Service used to obtain Google Cloud Platform credentials. |
| Message Demarcator |
Since the PubSub client receives messages in batches, this Processor has an option to output FlowFiles which contains all the messages in a single batch. This property allows you to provide a string (interpreted as UTF-8) to use for demarcating apart multiple messages. To enter special character such as 'new line' use CTRL+Enter or Shift+Enter depending on the OS. |
| Output Strategy |
The format used to output the Kafka Record into a FlowFile Record. |
| Processing Strategy |
Strategy for processing PubSub Records and writing serialized output to FlowFiles |
| Record Reader |
The Record Reader to use for incoming messages |
| Record Writer |
The Record Writer to use in order to serialize the outgoing FlowFiles |
| api-endpoint |
Override the gRPC endpoint in the form of [host:port] |
| gcp-project-id |
Google Cloud Project ID |
| gcp-pubsub-publish-batch-size |
Indicates the number of messages the cloud service should bundle together in a batch. If not set and left empty, only one message will be used in a batch |
| gcp-pubsub-subscription |
Name of the Google Cloud Pub/Sub Subscription |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| success |
FlowFiles are routed to this relationship after a successful Google Cloud Pub/Sub operation. |
## Writes attributes
| Name |
Description |
| gcp.pubsub.ackId |
Acknowledgement Id of the consumed Google Cloud PubSub message |
| gcp.pubsub.messageSize |
Serialized size of the consumed Google Cloud PubSub message |
| gcp.pubsub.attributesCount |
Number of attributes the consumed PubSub message has, if any |
| gcp.pubsub.publishTime |
Timestamp value when the message was published |
| gcp.pubsub.subscription |
Name of the PubSub subscription |
| Dynamic Attributes |
Other than the listed attributes, this processor may write zero or more attributes, if the original Google Cloud Publisher client added any attributes to the message while sending |
## See also
- [org.apache.nifi.processors.gcp.pubsub.PublishGCPubSub](/user-guide/data-integration/openflow/processors/publishgcpubsub)
---
title: ConsumeIMAP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumeimap.md
section: Loading & Unloading Data
---
# ConsumeIMAP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-email-nar
## Description
Consumes messages from Email Server using IMAP protocol. The raw-bytes of each received email message are written as contents of the FlowFile
## Tags
Consume, Email, Get, Imap, Ingest, Ingress, Message
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Authorization Mode |
How to authorize sending email on the user's behalf. |
| Connection Timeout |
The amount of time to wait to connect to Email server |
| Delete Messages |
Specify whether mail messages should be deleted after retrieval. |
| Fetch Size |
Specify the maximum number of Messages to fetch per call to Email Server. |
| Folder |
Email folder to retrieve messages from (e.g., INBOX) |
| Host Name |
Network address of Email server (e.g., pop.gmail.com, imap.gmail.com . .) |
| Mark Messages as Read |
Specify if messages should be marked as read after retrieval. |
| OAuth2 Access Token Provider |
OAuth2 service that can provide access tokens. |
| Password |
Password used for authentication and authorization with Email server. |
| Port |
Numeric value identifying Port of Email server (e.g., 993) |
| Use SSL |
Specifies if IMAP connection must be obtained via SSL encrypted connection (i.e., IMAPS) |
| User Name |
User Name used for authentication and authorization with Email server. |
## Relationships
| Name |
Description |
| success |
All messages that are the are successfully received from Email server and converted to FlowFiles are routed to this relationship |
---
title: ConsumeJMS 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumejms.md
section: Loading & Unloading Data
---
# ConsumeJMS 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-jms-processors-nar
## Description
Consumes JMS Message of type BytesMessage, TextMessage, ObjectMessage, MapMessage or StreamMessage transforming its content to a FlowFile and transitioning it to 'success' relationship. JMS attributes such as headers and properties will be copied as FlowFile attributes. MapMessages will be transformed into JSONs and then into byte arrays. The other types will have their raw contents as byte array transferred into the flowfile.
## Tags
consume, get, jms, message, receive
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Acknowledgement Mode |
The JMS Acknowledgement Mode. Using Auto Acknowledge can cause messages to be lost on restart of NiFi but may provide better performance than Client Acknowledge. |
| Connection Client ID |
The client id to be set on the connection, if set. For durable non shared consumer this is mandatory, for all others it is optional, typically with shared consumers it is undesirable to be set. Please see JMS spec for further details |
| Connection Factory Service |
The Controller Service that is used to obtain Connection Factory. Alternatively, the 'JNDI *' or the 'JMS *' properties can also be used to configure the Connection Factory. |
| Destination Name |
The name of the JMS Destination. Usually provided by the administrator (e.g., 'topic://myTopic' or 'myTopic'). |
| Destination Type |
The type of the JMS Destination. Could be one of 'QUEUE' or 'TOPIC'. Usually provided by the administrator. Defaults to 'QUEUE' |
| Durable subscription |
If destination is Topic if present then make it the consumer durable. @see https://jakarta.ee/specifications/platform/9/apidocs/jakarta/jms/session#createDurableConsumer-jakarta.jms (https://jakarta.ee/specifications/platform/9/apidocs/jakarta/jms/session#createDurableConsumer-jakarta.jms). Topic-java.lang. String- |
| Error Queue Name |
The name of a JMS Queue where - if set - unprocessed messages will be routed. Usually provided by the administrator (e.g., 'queue://myErrorQueue' or 'myErrorQueue').Only applicable if 'Destination Type' is set to 'QUEUE' |
| Maximum Batch Size |
The maximum number of messages to publish or consume in each invocation of the processor. |
| Message Selector |
The JMS Message Selector to filter the messages that the processor will receive |
| Password |
Password used for authentication and authorization. |
| SSL Context Service |
The SSL Context Service used to provide client certificate information for TLS/SSL connections. |
| Shared subscription |
If destination is Topic if present then make it the consumer shared. @see https://jakarta.ee/specifications/platform/9/apidocs/jakarta/jms/session#createSharedConsumer-jakarta.jms (https://jakarta.ee/specifications/platform/9/apidocs/jakarta/jms/session#createSharedConsumer-jakarta.jms). Topic-java.lang. String- |
| Subscription Name |
The name of the subscription to use if destination is Topic and is shared or durable. |
| Timeout |
How long to wait to consume a message from the remote broker before giving up. |
| User Name |
User Name used for authentication and authorization. |
| broker |
URI pointing to the network location of the JMS Message broker. Example for ActiveMQ: '[tcp://myhost:61616](tcp://myhost:61616)'. Examples for IBM MQ: 'myhost(1414)' and 'myhost01(1414),myhost02(1414)'. |
| cf |
The fully qualified name of the JMS ConnectionFactory implementation class (eg. org.apache.activemq. ActiveMQConnectionFactory). |
| cflib |
Path to the directory with additional resources (eg. JARs, configuration files etc.) to be added to the classpath (defined as a comma separated list of values). Such resources typically represent target JMS client libraries for the ConnectionFactory implementation. |
| character-set |
The name of the character set to use to construct or interpret TextMessages |
| connection.factory.name |
The name of the JNDI Object to lookup for the Connection Factory. |
| java.naming.factory.initial |
The fully qualified class name of the JNDI Initial Context Factory Class (java.naming.factory.initial). |
| java.naming.provider.url |
The URL of the JNDI Provider to use as the value for java.naming.provider.url. See additional details documentation for allowed URL schemes. |
| java.naming.security.credentials |
The Credentials to use when authenticating with JNDI (java.naming.security.credentials). |
| java.naming.security.principal |
The Principal to use when authenticating with JNDI (java.naming.security.principal). |
| naming.factory.libraries |
Specifies jar files and/or directories to add to the ClassPath in order to load the JNDI / JMS client libraries. This should be a comma-separated list of files, directories, and/or URLs. If a directory is given, any files in that directory will be included, but subdirectories will not be included (i.e., it is not recursive). |
| output-strategy |
The format used to output the JMS message into a FlowFile record. |
| record-reader |
The Record Reader to use for parsing received JMS Messages into Records. |
| record-writer |
The Record Writer to use for serializing Records before writing them to a FlowFile. |
## Restrictions
| Required Permission |
Explanation |
| reference remote resources |
Client Library Location can reference resources over HTTP |
## Relationships
| Name |
Description |
| parse.failure |
If a message cannot be parsed using the configured Record Reader, the contents of the message will be routed to this Relationship as its own individual FlowFile. |
| success |
All FlowFiles that are received from the JMS Destination are routed to this relationship |
## Writes attributes
| Name |
Description |
| jms_deliveryMode |
The JMSDeliveryMode from the message header. |
| jms_expiration |
The JMSExpiration from the message header. |
| jms_priority |
The JMSPriority from the message header. |
| jms_redelivered |
The JMSRedelivered from the message header. |
| jms_timestamp |
The JMSTimestamp from the message header. |
| jms_correlationId |
The JMSCorrelationID from the message header. |
| jms_messageId |
The JMSMessageID from the message header. |
| jms_type |
The JMSType from the message header. |
| jms_replyTo |
The JMSReplyTo from the message header. |
| jms_destination |
The JMSDestination from the message header. |
| jms.messagetype |
The JMS message type, can be TextMessage, BytesMessage, ObjectMessage, MapMessage or StreamMessage). |
| other attributes |
Each message property is written to an attribute. |
## See also
- [org.apache.nifi.jms.processors.PublishJMS](/user-guide/data-integration/openflow/processors/publishjms)
---
title: ConsumeKafka 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumekafka.md
section: Loading & Unloading Data
---
# ConsumeKafka 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-kafka-nar
## Description
Consumes messages from Apache Kafka Consumer API. The complementary NiFi processor for sending messages is PublishKafka. The Processor supports consumption of Kafka messages, optionally interpreted as NiFi records. Please note that, at this time (in read record mode), the Processor assumes that all records that are retrieved from a given partition have the same schema. For this mode, if any of the Kafka messages are pulled but cannot be parsed or written with the configured Record Reader or Record Writer, the contents of the message will be written to a separate FlowFile, and that FlowFile will be transferred to the 'parse.failure' relationship. Otherwise, each FlowFile is sent to the 'success' relationship and may contain many individual messages within the single FlowFile. A 'record.count' attribute is added to indicate how many messages are contained in the FlowFile. No two Kafka messages will be placed into the same FlowFile if they have different schemas, or if they have different values for a message header that is included by the <Headers to Add as Attributes> property.
## Tags
avro, consume, csv, get, ingest, ingress, json, kafka, openflow, pubsub, record, topic
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Commit Offsets |
Specifies whether this Processor should commit the offsets to Kafka after receiving messages. Typically, this value should be set to true so that messages that are received are not duplicated. However, in certain scenarios, we may want to avoid committing the offsets, that the data can be processed and later acknowledged by PublishKafka in order to provide Exactly Once semantics. |
| Content Field |
Specifies under what field of the record the content will be added. If not set, the content will be at the root of the record |
| Group ID |
Kafka Consumer Group Identifier corresponding to Kafka group.id property |
| Header Encoding |
Character encoding applied when reading Kafka Record Header values and writing FlowFile attributes |
| Header Name Pattern |
Regular Expression Pattern applied to Kafka Record Header Names for selecting Header Values to be written as FlowFile attributes |
| Headers Field Parent |
Specifies under what field of the record the headers field will be added. If not set, the headers field will be at the root of the record |
| Kafka Connection Service |
Provides connections to Kafka Broker for publishing Kafka Records |
| Key Attribute Encoding |
Encoding for value of configured FlowFile attribute containing Kafka Record Key. |
| Key Field Parent |
Specifies under what field of the record the key field will be added. If not set, the key field will be at the root of the record |
| Key Format |
Specifies how to represent the Kafka Record Key in the output FlowFile |
| Key Record Reader |
The Record Reader to use for parsing the Kafka Record Key into a Record |
| Max Uncommitted Time |
Specifies the maximum amount of time that the Processor can consume from Kafka before it must transfer FlowFiles on through the flow and commit the offsets to Kafka (if appropriate). A larger time period can result in longer latency |
| Message Demarcator |
Since KafkaConsumer receives messages in batches, this Processor has an option to output FlowFiles which contains all Kafka messages in a single batch for a given topic and partition and this property allows you to provide a string (interpreted as UTF-8) to use for demarcating apart multiple Kafka messages. This is an optional property and if not provided each Kafka message received will result in a single FlowFile which time it is triggered. To enter special character such as 'new line' use CTRL+Enter or Shift+Enter depending on the OS |
| Metadata Field |
Specifies under what field of the record the metadata will be added. If not set, the metadata will be at the root of the record |
| Metadata Received Timestamp Field |
If specified a timestamp will be placed under the specified field in the metadata of record in the output FlowFile |
| Output Strategy |
The format used to output the Kafka Record into a FlowFile Record. |
| Processing Strategy |
Strategy for processing Kafka Records and writing serialized output to FlowFiles |
| Record Reader |
The Record Reader to use for incoming Kafka messages |
| Record Writer |
The Record Writer to use in order to serialize the outgoing FlowFiles |
| Separate By Key |
When this property is enabled, two messages will only be added to the same FlowFile if both of the Kafka Messages have identical keys. |
| Topic Format |
Specifies whether the Topics provided are a comma separated list of names or a single regular expression |
| Topics |
The name or pattern of the Kafka Topics from which the Processor consumes Kafka Records. More than one can be supplied if comma separated. |
| auto.offset.reset |
Automatic offset configuration applied when no previous consumer offset found corresponding to Kafka auto.offset.reset property |
## Relationships
| Name |
Description |
| success |
FlowFiles containing one or more serialized Kafka Records |
## Writes attributes
| Name |
Description |
| record.count |
The number of records received |
| mime.type |
The MIME Type that is provided by the configured Record Writer |
| kafka.count |
The number of messages written if more than one |
| kafka.key |
The key of message if present and if single message. How the key is encoded depends on the value of the 'Key Attribute Encoding' property. |
| kafka.offset |
The offset of the message in the partition of the topic. |
| kafka.timestamp |
The timestamp of the message in the partition of the topic. |
| kafka.partition |
The partition of the topic the message or message bundle is from |
| kafka.topic |
The topic the message or message bundle is from |
| kafka.tombstone |
Set to true if the consumed message is a tombstone message |
## See also
- [com.snowflake.openflow.runtime.processors.kafka.PublishKafka](/user-guide/data-integration/openflow/processors/publishkafka)
---
title: ConsumeKinesisStream 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumekinesisstream.md
section: Loading & Unloading Data
---
# ConsumeKinesisStream 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-aws-nar
## Description
Reads data from the specified AWS Kinesis stream and outputs a FlowFile for every processed Record (raw) or a FlowFile for a batch of processed records if a Record Reader and Record Writer are configured. At-least-once delivery of all Kinesis Records within the Stream while the processor is running. AWS Kinesis Client Library can take several seconds to initialise before starting to fetch data. Uses DynamoDB for check pointing and CloudWatch (optional) for metrics. Ensure that the credentials provided have access to DynamoDB and CloudWatch (optional) along with Kinesis.
## Tags
amazon, aws, consume, kinesis, stream
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| AWS Credentials Provider service |
The Controller Service that is used to obtain AWS credentials provider |
| Amazon Kinesis Stream Name |
The name of Kinesis Stream |
| Application Name |
The Kinesis stream reader application name. |
| Checkpoint Interval |
Interval between Kinesis checkpoints |
| Communications Timeout |
|
| DynamoDB Override |
DynamoDB override to use non-AWS deployments |
| Endpoint Override URL |
Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. |
| Failover Timeout |
Kinesis Client Library failover timeout |
| FlowFile Handling On Schema Difference |
The strategy used when records in a Kinesis Stream change their schema in a single batch. |
| Graceful Shutdown Timeout |
Kinesis Client Library graceful shutdown timeout |
| Initial Stream Position |
Initial position to read Kinesis streams. |
| Output Strategy |
The format used to output the Kinesis Record into a FlowFile Record. |
| Record Reader |
The Record Reader to use for reading received messages. The Kinesis Stream name can be referred to by Expression Language '$\{kinesis.name\}' to access a schema. If Record Reader/Writer are not specified, each Kinesis Record will create a FlowFile. |
| Record Writer |
The Record Writer to use for serializing Records to an output FlowFile. The Kinesis Stream name can be referred to by Expression Language '$\{kinesis.name\}' to access a schema. If Record Reader/Writer are not specified, each Kinesis Record will create a FlowFile. |
| Region |
|
| Report Metrics to CloudWatch |
Whether to report Kinesis usage metrics to CloudWatch. |
| Retry Count |
Number of times to retry a Kinesis operation (process record, checkpoint, shutdown) |
| Retry Wait |
Interval between Kinesis operation retries (process record, checkpoint, shutdown) |
| Stream Position Timestamp |
Timestamp position in stream from which to start reading Kinesis Records. Required if Initial position to read Kinesis streams. is AT_TIMESTAMP. Uses the Timestamp Format to parse value into a Date. |
| Timestamp Format |
Format to use for parsing the Stream Position Timestamp into a Date and converting the Kinesis Record's Approximate Arrival Timestamp into a FlowFile attribute. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| success |
FlowFiles are routed to success relationship |
## Writes attributes
| Name |
Description |
| aws.kinesis.partition.key |
Partition key of the (last) Kinesis Record read from the Shard |
| aws.kinesis.shard.id |
Shard ID from which the Kinesis Record was read |
| aws.kinesis.sequence.number |
The unique identifier of the (last) Kinesis Record within its Shard |
| aws.kinesis.approximate.arrival.timestamp |
Approximate arrival timestamp of the (last) Kinesis Record read from the stream |
| mime.type |
Sets the mime.type attribute to the MIME Type specified by the Record Writer (if configured) |
| record.count |
Number of records written to the FlowFiles by the Record Writer (if configured) |
| record.error.message |
This attribute provides on failure the error message encountered by the Record Reader or Record Writer (if configured) |
## See also
- [org.apache.nifi.processors.aws.kinesis.stream.PutKinesisStream](/user-guide/data-integration/openflow/processors/putkinesisstream)
---
title: ConsumeMQTT 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumemqtt.md
section: Loading & Unloading Data
---
# ConsumeMQTT 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-mqtt-nar
## Description
Subscribes to a topic and receives messages from an MQTT broker
## Tags
IOT, MQTT, consume, listen, subscribe
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Broker URI |
The URI(s) to use to connect to the MQTT broker (e.g., [tcp://localhost:1883](tcp://localhost:1883)). The 'tcp', 'ssl', 'ws' and 'wss'schemes are supported. In order to use 'ssl', the SSL Context Service property must be set. When a comma-separated URI list is set (e.g., [tcp://localhost:1883,tcp://localhost:1884](tcp://localhost:1883,tcp://localhost:1884)), the processor will use a round-robin algorithm to connect to the brokers on connection failure. |
| Client ID |
MQTT client ID to use. If not set, a UUID will be generated. |
| Connection Timeout (seconds) |
Maximum time interval the client will wait for the network connection to the MQTT server to be established. The default timeout is 30 seconds. A value of 0 disables timeout processing meaning the client will wait until the network connection is made successfully or fails. |
| Group ID |
MQTT consumer group ID to use. If group ID not set, client will connect as individual consumer. |
| Keep Alive Interval (seconds) |
Defines the maximum time interval between messages sent or received. It enables the client to detect if the server is no longer available, without having to wait for the TCP/IP timeout. The client will ensure that at least one message travels across the network within each keep alive period. In the absence of a data-related message during the time period, the client sends a very small "ping" message, which the server will acknowledge. A value of 0 disables keepalive processing in the client. |
| Last Will Message |
The message to send as the client's Last Will. |
| Last Will QoS Level |
QoS level to be used when publishing the Last Will Message. |
| Last Will Retain |
Whether to retain the client's Last Will. |
| Last Will Topic |
The topic to send the client's Last Will to. |
| MQTT Specification Version |
The MQTT specification version when connecting with the broker. See the allowable value descriptions for more details. |
| Max Queue Size |
The MQTT messages are always being sent to subscribers on a topic regardless of how frequently the processor is scheduled to run. If the 'Run Schedule' is significantly behind the rate at which the messages are arriving to this processor, then a back up can occur in the internal queue of this processor. This property specifies the maximum number of messages this processor will hold in memory at one time in the internal queue. This data would be lost in case of a NiFi restart. |
| Password |
Password to use when connecting to the broker |
| Quality of Service(QoS) |
The Quality of Service (QoS) to receive the message with. Accepts values '0', '1' or '2'; '0' for 'at most once', '1' for 'at least once', '2' for 'exactly once'. |
| SSL Context Service |
The SSL Context Service used to provide client certificate information for TLS/SSL connections. |
| Session Expiry Interval |
After this interval the broker will expire the client and clear the session state. |
| Session state |
Whether to start a fresh or resume previous flows. See the allowable value descriptions for more details. |
| Topic Filter |
The MQTT topic filter to designate the topics to subscribe to. |
| Username |
Username to use when connecting to the broker |
| add-attributes-as-fields |
If setting this property to true, default fields are going to be added in each record: _topic, _qos, _isDuplicate, _isRetained. |
| message-demarcator |
With this property, you have an option to output FlowFiles which contains multiple messages. This property allows you to provide a string (interpreted as UTF-8) to use for demarcating apart multiple messages. This is an optional property ; if not provided, and if not defining a Record Reader/Writer, each message received will result in a single FlowFile. To enter special character such as 'new line' use CTRL+Enter or Shift+Enter depending on the OS. |
| record-reader |
The Record Reader to use for parsing received MQTT Messages into Records. |
| record-writer |
The Record Writer to use for serializing Records before writing them to a FlowFile. |
## Relationships
| Name |
Description |
| Message |
The MQTT message output |
| parse.failure |
If a message cannot be parsed using the configured Record Reader, the contents of the message will be routed to this Relationship as its own individual FlowFile. |
## Writes attributes
| Name |
Description |
| record.count |
The number of records received |
| mqtt.broker |
MQTT broker that was the message source |
| mqtt.topic |
MQTT topic on which message was received |
| mqtt.qos |
The quality of service for this message. |
| mqtt.isDuplicate |
Whether or not this message might be a duplicate of one which has already been received. |
| mqtt.isRetained |
Whether or not this message was from a current publisher, or was "retained" by the server as the last message published on the topic. |
## See also
- [org.apache.nifi.processors.mqtt.PublishMQTT](/user-guide/data-integration/openflow/processors/publishmqtt)
---
title: ConsumePOP3 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumepop3.md
section: Loading & Unloading Data
---
# ConsumePOP3 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-email-nar
## Description
Consumes messages from Email Server using POP3 protocol. The raw-bytes of each received email message are written as contents of the FlowFile
## Tags
Consume, Email, Get, Ingest, Ingress, Message, POP3
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Authorization Mode |
How to authorize sending email on the user's behalf. |
| Connection Timeout |
The amount of time to wait to connect to Email server |
| Delete Messages |
Specify whether mail messages should be deleted after retrieval. |
| Fetch Size |
Specify the maximum number of Messages to fetch per call to Email Server. |
| Folder |
Email folder to retrieve messages from (e.g., INBOX) |
| Host Name |
Network address of Email server (e.g., pop.gmail.com, imap.gmail.com . .) |
| OAuth2 Access Token Provider |
OAuth2 service that can provide access tokens. |
| Password |
Password used for authentication and authorization with Email server. |
| Port |
Numeric value identifying Port of Email server (e.g., 993) |
| User Name |
User Name used for authentication and authorization with Email server. |
## Relationships
| Name |
Description |
| success |
All messages that are the are successfully received from Email server and converted to FlowFiles are routed to this relationship |
---
title: ConsumeSlack 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumeslack.md
section: Loading & Unloading Data
---
# ConsumeSlack 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-slack-nar
## Description
Retrieves messages from one or more configured Slack channels. The messages are written out in JSON format. See Usage / Additional Details for more information about how to configure this Processor and enable it to retrieve messages from Slack.
## Tags
conversation, conversation.history, slack, social media, team, text, unstructured
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Access Token |
OAuth Access Token used for authenticating/authorizing the Slack request sent by NiFi. This may be either a User Token or a Bot Token. It must be granted the channels:history, groups:history, im:history, or mpim:history scope, depending on the type of conversation being used. |
| Batch Size |
The maximum number of messages to retrieve in a single request to Slack. The entire response will be parsed into memory, so it is important that this be kept in mind when setting this value. |
| Channels |
A comma-separated list of Slack Channels to Retrieve Messages From. Each element in the list may be either a Channel ID, such as C0L9VCD47, or (for public channels only) the name of a channel, prefixed with a # sign, such as #general. If any channel name is provided instead,instead of an ID, the Access Token provided must be granted the channels:read scope in order to resolve the Channel ID. See the Processor's Additional Details for information on how to find a Channel ID. |
| Include Message Blocks |
Specifies whether or not the output JSON should include the value of the 'blocks' field for each Slack Message. This field includes information such as individual parts of a message that are formatted using rich text. This may be useful, for instance, for parsing. However, it often accounts for a significant portion of the data and as such may be set to null when it is not useful to you. |
| Include Null Fields |
Specifies whether or not fields that have null values should be included in the output JSON. If true, any field in a Slack Message that has a null value will be included in the JSON with a value of null. If false, the key omitted from the output JSON entirely. Omitting null values results in smaller messages that are generally more efficient to process, but including the values may provide a better understanding of the format, especially for schema inference. |
| Reply Monitor Frequency |
After consuming all messages in a given channel, this Processor will periodically poll all "threaded messages", aka Replies, whose timestamp falls between now and the amount of time specified by the <Reply Monitor Window> property. This property determines how frequently those messages are polled. Setting the value to a shorter duration may result in replies to messages being captured more quickly, providing a lower latency. However, it will also result in additional resource use and could trigger Rate Limiting to occur. |
| Reply Monitor Window |
After consuming all messages in a given channel, this Processor will periodically poll all "threaded messages", aka Replies, whose timestamp is between now and this amount of time in the past in order to check for any new replies. Setting this value to a larger value may result in additional resource use and may result in Rate Limiting. However, if a user replies to an old thread that was started outside of this window, the reply may not be captured. |
| Resolve Usernames |
Specifies whether or not User IDs should be resolved to usernames. By default, Slack Messages provide the ID of the user that sends a message, such as U0123456789, but not the username, such as NiFiUser. The username may be resolved, but it may require additional calls to the Slack API and requires that the Token used be granted the users:read scope. If set to true, usernames will be resolved with a best-effort policy: if a username cannot be obtained, it will be skipped over. Also, note that when a username is obtained, the Message's <username> field is populated, and the <text> field is updated such that any mention will be output such as "Hi @user" instead of "Hi <@U1234567>". |
## State management
| Scopes |
Description |
| CLUSTER |
Maintains a mapping of Slack Channel IDs to the timestamp of the last message that was retrieved for that channel. This allows the processor to only retrieve messages that have been posted since the last time the processor was run. This state is stored in the cluster so that if the Primary Node changes, the new node will pick up where the previous node left off. |
## Relationships
| Name |
Description |
| success |
Slack messages that are successfully received will be routed to this relationship |
## Writes attributes
| Name |
Description |
| slack.channel.id |
The ID of the Slack Channel from which the messages were retrieved |
| slack.message.count |
The number of slack messages that are included in the FlowFile |
| mime.type |
Set to application/json, as the output will always be in JSON format |
## See also
- [org.apache.nifi.processors.slack.ListenSlack](/user-guide/data-integration/openflow/processors/listenslack)
---
title: ConsumeSlackConversation 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumeslackconversation.md
section: Loading & Unloading Data
---
# ConsumeSlackConversation 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-slack-processors-nar
## Description
Retrieves messages from Slack conversations available to the App. New conversations are fetched based on the 'Reply Monitor Frequency'. Ingested messages are written out in JSON format. See Usage / Additional Details for more information about how to configure this Processor and enable it to retrieve messages from Slack.
## Tags
conversation, conversation.history, slack, social media, team, text, unstructured
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Access Token |
OAuth Access Token used for authenticating/authorizing the Slack request sent by NiFi. This may be either a User Token or a Bot Token. It must be granted the channels:history, groups:history, im:history, or mpim:history scope, depending on the type of conversation being used. |
| Batch Size |
The maximum number of messages to retrieve in a single request to Slack. The entire response will be parsed into memory, so it is important that this be kept in mind when setting this value. |
| Rate Limiter Service |
Slack Rate Limiter Service to coordinate rate limiting across processors |
| Reply Monitor Frequency |
After consuming all messages in a given channel, this Processor will periodically poll all "threaded messages", aka Replies, whose timestamp falls between now and the amount of time specified by the <Reply Monitor Window> property. This property determines how frequently those messages are polled. Setting the value to a shorter duration may result in replies to messages being captured more quickly, providing a lower latency. However, it will also result in additional resource use and could trigger Rate Limiting to occur. This also determines how frequently newly added channels are checked. |
| Reply Monitor Window |
After consuming all messages in a given channel, this Processor will periodically poll all "threaded messages", aka Replies, whose timestamp is between now and this amount of time in the past in order to check for any new replies. Setting this value to a larger value may result in additional resource use and may result in Rate Limiting. However, if a user replies to an old thread that was started outside of this window, the reply may not be captured. |
| Resolve Usernames |
Specifies whether or not User IDs should be resolved to usernames. By default, Slack Messages provide the ID of the user that sends a message, such as U0123456789, but not the username, such as NiFiUser. The username may be resolved, but it may require additional calls to the Slack API and requires that the Token used be granted the users:read scope. If set to true, usernames will be resolved with a best-effort policy: if a username cannot be obtained, it will be skipped over. Also, note that when a username is obtained, the Message's <username> field is populated, and the <text> field is updated such that any mention will be output such as "Hi @user" instead of "Hi <@U1234567>". |
## State management
| Scopes |
Description |
| CLUSTER |
Maintains a mapping of Slack Channel IDs to the timestamp of the last message that was retrieved for that channel. This allows the processor to only retrieve messages that have been posted since the last time the processor was run. This state is stored in the cluster so that if the Primary Node changes, the new node will pick up where the previous node left off. |
## Relationships
| Name |
Description |
| success |
Slack messages that are successfully received will be routed to this relationship |
## Writes attributes
| Name |
Description |
| slack.channel.id |
The ID of the Slack Channel from which the messages were retrieved |
| slack.message.count |
The number of slack messages that are included in the FlowFile |
| mime.type |
Set to application/json, as the output will always be in JSON format |
---
title: ConsumeSlackHistory 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumeslackhistory.md
section: Loading & Unloading Data
---
# ConsumeSlackHistory 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-slack-processors-nar
## Description
Fetches historical messages from all Slack channels available to the App. This processor queries Slack's conversations.history and conversations.replies to retrieve older messages and outputs the result as records. The processor tracks the earliest retrieved message timestamp in the cluster state to allow it to continue the historical load on subsequent executions. Channels are discovered automatically, no channel ID or name needs to be configured.
## Tags
consume, conversation, history, slack
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Access Token |
OAuth Access Token used for authenticating the Slack request. It must be granted the channels:history (and, if resolving usernames, users:read) scope. |
| Batch Size |
The maximum number of messages to retrieve in a single request to Slack. |
| Channel Refresh Frequency |
The frequency at which the processor refreshes the list of Slack channels accessible to the App. This helps detect newly available channels or remove channels that are no longer available. |
| Include Message Blocks |
Specifies whether the output JSON should include the value of the 'blocks' field for each Slack Message. |
| Include Null Fields |
Specifies whether fields that have null values should be included in the output JSON. If true, any field with a null value will be output as null; if false, it will be omitted. |
| Rate Limiter Service |
Slack Rate Limiter Service to coordinate rate limiting across processors |
| Resolve Usernames |
Specifies whether User IDs should be resolved to usernames. If true, usernames will be resolved with a best-effort policy; if a username cannot be obtained, it will be skipped. |
## State management
| Scopes |
Description |
| CLUSTER |
Maintains a mapping of Slack Channel IDs to the earliest message timestamp that has been retrieved. When no more messages are available, a flag is set indicating that the historical load is complete for that channel. This state is stored in the cluster so that if the Primary Node changes, the new node will pick up where the previous node left off. |
## Relationships
| Name |
Description |
| success |
FlowFiles containing the JSON-encoded Slack conversation history are routed to this relationship |
## Writes attributes
| Name |
Description |
| slack.channel.id |
The ID of the Slack Channel from which the messages were retrieved |
| slack.channel.name |
The name of the Slack Channel from which the messages were retrieved |
| slack.message.count |
The number of Slack messages that are included in the FlowFile |
| mime.type |
Set to application/json, the output will always be in JSON format |
---
title: ConsumeSnowflakeStream 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumesnowflakestream.md
section: Loading & Unloading Data
---
# ConsumeSnowflakeStream 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-snowflake-processors-nar
## Description
Fetches data from a Snowflake stream and writes it to a FlowFile. The stream must be created in the database before using this processor. The processor will consume the stream and write the records to the FlowFile using the specified Record Writer. The processor will also add an attribute to the FlowFile with the name of the stream. The processor will not work if the stream is stale. Instead it will log an error message and stop processing. Stale stream has to be recreated in the database. After the stream is recreated in the database the processor will continue to read and process CDC records. For more information on Snowflake streams, see the <a href="[https://docs.snowflake.cn/en/user-guide/streams-intro](https://docs.snowflake.cn/en/user-guide/streams-intro)">snowflake documentation</a>.
## Tags
connection, database, jdbc, openflow, snowflake, stream, table, view
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Max Chunk Size |
Number of records to write into a single FlowFile. This value might be slightly exceeded. |
| Record Writer |
The Record Writer to use for CDC record serialization |
| Snowflake Connection Service |
Database Connection Service for accessing Snowflake |
| Stream Name |
The name of the stream in the database |
## Relationships
| Name |
Description |
| success |
For FlowFiles with stream CDC records |
## Writes attributes
| Name |
Description |
| snowflake.stream.name |
Name of the Snowflake Stream |
---
title: ConsumeTwitter 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/consumetwitter.md
section: Loading & Unloading Data
---
# ConsumeTwitter 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-social-media-nar
## Description
Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the tweet to be returned as part of the response. See https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the Tweet object model.
## Tags
json, social media, status, tweets, twitter
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| backfill-minutes |
The number of minutes (up to 5 minutes) of streaming data to be requested after a disconnect. Only available for project with academic research access. See https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/recovery-and-redundancy-features |
| backoff-attempts |
The number of reconnection tries the processor will attempt in the event of a disconnection of the stream for any reason, before throwing an exception. To start a stream after this exception occur and the connection is fixed, please stop and restart the processor. If the valueof this property is 0, then backoff will never occur and the processor will always need to be restartedif the stream fails. |
| backoff-time |
The duration to backoff before requesting a new stream ifthe current one fails for any reason. Will increase by factor of 2 every time a restart fails |
| base-path |
The base path that the processor will use for making HTTP requests. The default value should be sufficient for most use cases. |
| batch-size |
The maximum size of the number of Tweets to be written to a single FlowFile. Will write fewer Tweets based on the number available in the queue at the time of processor invocation. |
| bearer-token |
The Bearer Token provided by Twitter. |
| connect-timeout |
The maximum time in which client should establish a connection with the Twitter API before a time out. Setting the value to 0 disables connection timeouts. |
| expansions |
A comma-separated list of expansions for objects in the returned tweet. See https://developer.twitter.com/en/docs/twitter-api/expansions for proper usage. Possible field values include: author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id |
| maximum-backoff-time |
The maximum duration to backoff to start attempting a new stream. It is recommended that this number be much higher than the 'Backoff Time' property |
| media-fields |
A comma-separated list of media fields to be returned as part of the tweet. Refer to https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media for proper usage. Possible field values include: alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, promoted_metrics, public_metrics, type, url, width |
| place-fields |
A comma-separated list of place fields to be returned as part of the tweet. Refer to https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place for proper usage. Possible field values include: contained_within, country, country_code, full_name, geo, id, name, place_type |
| poll-fields |
A comma-separated list of poll fields to be returned as part of the tweet. Refer to https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll for proper usage. Possible field values include: duration_minutes, end_datetime, id, options, voting_status |
| queue-size |
Maximum size of internal queue for streamed messages |
| read-timeout |
The maximum time of inactivity between receiving tweets from Twitter through the API before a timeout. Setting the value to 0 disables read timeouts. |
| stream-endpoint |
The source from which the processor will consume Tweets. |
| tweet-fields |
A comma-separated list of tweet fields to be returned as part of the tweet. Refer to https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet for proper usage. Possible field values include: attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, public_metrics, referenced_tweets, reply_settings, source, text, withheld |
| user-fields |
A comma-separated list of user fields to be returned as part of the tweet. Refer to https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user for proper usage. Possible field values include: created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, protected, public_metrics, url, username, verified, withheld |
## Relationships
| Name |
Description |
| success |
FlowFiles containing an array of one or more Tweets |
## Writes attributes
| Name |
Description |
| mime.type |
The MIME Type set to application/json |
| tweets |
The number of Tweets in the FlowFile |
---
title: ControlRate 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/controlrate.md
section: Loading & Unloading Data
---
# ControlRate 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Controls the rate at which data is transferred to follow-on processors. If you configure a very small Time Duration, then the accuracy of the throttle gets worse. You can improve this accuracy by decreasing the Yield Duration, at the expense of more Tasks given to the processor.
## Tags
rate, rate control, throttle, throughput
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Grouping Attribute |
By default, a single "throttle" is used for all FlowFiles. If this value is specified, a separate throttle is used for each value specified by the attribute with this name. Changing this value resets the rate counters. |
| Maximum Data Rate |
The maximum rate at which data should pass through this processor. The format of this property is expected to be a Data Size (such as '1 MB') representing bytes per Time Duration. |
| Maximum FlowFile Rate |
The maximum rate at which FlowFiles should pass through this processor. The format of this property is expected to be a positive integer representing FlowFiles count per Time Duration |
| Maximum Rate |
The maximum rate at which data should pass through this processor. The format of this property is expected to be a positive integer, or a Data Size (such as '1 MB') if Rate Control Criteria is set to 'data rate'. |
| Rate Control Criteria |
Indicates the criteria that is used to control the throughput rate. Changing this value resets the rate counters. |
| Rate Controlled Attribute |
The name of an attribute whose values build toward the rate limit if Rate Control Criteria is set to 'attribute value'. The value of the attribute referenced by this property must be a positive long, or the FlowFile will be routed to failure. This value is ignored if Rate Control Criteria is not set to 'attribute value'. Changing this value resets the rate counters. |
| Rate Exceeded Strategy |
Specifies how to handle an incoming FlowFile when the maximum data rate has been exceeded. |
| Time Duration |
The amount of time to which the Maximum Rate pertains. Changing this value resets the rate counters. |
## Relationships
| Name |
Description |
| failure |
FlowFiles will be routed to this relationship if they are missing a necessary Rate Controlled Attribute or the attribute is not in the expected format |
| success |
FlowFiles are transferred to this relationship under normal conditions |
## Use cases
| Limit the rate at which data is sent to a downstream system with little to no bursts |
| ------------------------------------------------------------------------------------------ |
| Limit the rate at which FlowFiles are sent to a downstream system with little to no bursts |
| Reject requests that exceed a specific rate with little to no bursts |
| Reject requests that exceed a specific rate, allowing for bursts |
---
title: ConvertCharacterSet 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/convertcharacterset.md
section: Loading & Unloading Data
---
# ConvertCharacterSet 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Converts a FlowFile's content from one character set to another
## Tags
character set, characterset, convert, text
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Input Character Set |
The name of the CharacterSet to expect for Input |
| Output Character Set |
The name of the CharacterSet to convert to |
## Relationships
| Name |
Description |
| success |
|
---
title: ConvertRecord 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/convertrecord.md
section: Loading & Unloading Data
---
# ConvertRecord 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Converts records from one data format to another using configured Record Reader and Record Write Controller Services. The Reader and Writer must be configured with "matching" schemas. By this, we mean the schemas must have the same field names. The types of the fields do not have to be the same if a field value can be coerced from one type to another. For instance, if the input schema has a field named "balance" of type double, the output schema can have a field named "balance" with a type of string, double, or float. If any field is present in the input that is not present in the output, the field will be left out of the output. If any field is specified in the output schema but is not present in the input data/schema, then the field will not be present in the output or will have a null value, depending on the writer.
## Tags
avro, convert, csv, freeform, generic, json, log, logs, record, schema, text
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Include Zero Record FlowFiles |
When converting an incoming FlowFile, if the conversion results in no data, this property specifies whether or not a FlowFile will be sent to the corresponding relationship |
| Record Reader |
Specifies the Controller Service to use for reading incoming data |
| Record Writer |
Specifies the Controller Service to use for writing out the records |
## Relationships
| Name |
Description |
| failure |
If a FlowFile cannot be transformed from the configured input format to the configured output format, the unchanged FlowFile will be routed to this relationship |
| success |
FlowFiles that are successfully transformed will be routed to this relationship |
## Writes attributes
| Name |
Description |
| mime.type |
Sets the mime.type attribute to the MIME Type specified by the Record Writer |
| record.count |
The number of records in the FlowFile |
| record.error.message |
This attribute provides on failure the error message encountered by the Reader or Writer. |
## Use cases
| Convert data from one record-oriented format to another |
| ------------------------------------------------------- |
---
title: ConvertToJournalSchema 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/converttojournalschema.md
section: Loading & Unloading Data
---
# ConvertToJournalSchema 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-database-cdc-processors-nar
## Description
Converts the incoming database schema into the appropriate schema for a Snowflake CDC Journal table.
## Tags
Snowflake, cdc, journal
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Relationships
| Name |
Description |
| failure |
FlowFiles are routed to this relationship if the schema cannot be translated. |
| original |
The original FlowFile is routed to this relationship when the schema is successfully converted. |
| success |
FlowFiles are routed to this relationship after the schema has been converted. |
---
title: CopyAzureBlobStorage_v12 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/copyazureblobstorage_v12.md
section: Loading & Unloading Data
---
# CopyAzureBlobStorage_v12 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-azure-nar
## Description
Copies a blob in Azure Blob Storage from one account/container to another. The processor uses Azure Blob Storage client library v12.
## Tags
azure, blob, cloud, microsoft, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Conflict Resolution Strategy |
Specifies whether an existing blob will have its contents replaced upon conflict. |
| Create Container |
Specifies whether to check if the container exists and to automatically create it if it does not. Permission to list containers is required. If false, this check is not made, but the Put operation will fail if the container does not exist. |
| Destination Blob Name |
The full name of the destination blob defaults to the Source Blob Name when not specified |
| Destination Container Name |
Name of the Azure storage container destination defaults to the Source Container Name when not specified |
| Destination Storage Credentials |
Controller Service used to obtain Azure Blob Storage Credentials. |
| Source Blob Name |
The full name of the source blob |
| Source Container Name |
Name of the Azure storage container that will be copied |
| Source Storage Credentials |
Credentials Service used to obtain Azure Blob Storage Credentials to read Source Blob information |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. In case of SOCKS, it is not guaranteed that the selected SOCKS Version will be used by the processor. |
## Relationships
| Name |
Description |
| failure |
Unsuccessful operations will be transferred to the failure relationship. |
| success |
All successfully processed FlowFiles are routed to this relationship |
## Writes attributes
| Name |
Description |
| azure.container |
The name of the Azure Blob Storage container |
| azure.blobname |
The name of the blob on Azure Blob Storage |
| azure.primaryUri |
Primary location of the blob |
| azure.etag |
ETag of the blob |
| azure.blobtype |
Type of the blob (either BlockBlob, PageBlob or AppendBlob) |
| mime.type |
MIME Type of the content |
| lang |
Language code for the content |
| azure.timestamp |
Timestamp of the blob |
| azure.length |
Length of the blob |
| azure.error.code |
Error code reported during blob operation |
| azure.ignored |
When Conflict Resolution Strategy is 'ignore', this property will be true/false depending on whether the blob was ignored. |
## See also
- [org.apache.nifi.processors.azure.storage.DeleteAzureBlobStorage_v12](/user-guide/data-integration/openflow/processors/deleteazureblobstorage_v12)
- [org.apache.nifi.processors.azure.storage.FetchAzureBlobStorage_v12](/user-guide/data-integration/openflow/processors/fetchazureblobstorage_v12)
- [org.apache.nifi.processors.azure.storage.ListAzureBlobStorage_v12](/user-guide/data-integration/openflow/processors/listazureblobstorage_v12)
- [org.apache.nifi.processors.azure.storage.PutAzureBlobStorage_v12](/user-guide/data-integration/openflow/processors/putazureblobstorage_v12)
---
title: CopyS3Object 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/copys3object.md
section: Loading & Unloading Data
---
# CopyS3Object 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-aws-nar
## Description
Copies a file from one bucket and key to another in AWS S3
## Tags
AWS, Amazon, Archive, Copy, S3
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| AWS Credentials Provider service |
The Controller Service that is used to obtain AWS credentials provider |
| Canned ACL |
Amazon Canned ACL for an object, one of: BucketOwnerFullControl, BucketOwnerRead, LogDeliveryWrite, AuthenticatedRead, PublicReadWrite, PublicRead, Private; will be ignored if any other ACL/permission/owner property is specified |
| Communications Timeout |
The amount of time to wait in order to establish a connection to AWS or receive data from AWS before timing out. |
| Custom Signer Class Name |
Fully qualified class name of the custom signer class. The signer must implement com.amazonaws.auth. Signer interface. |
| Custom Signer Module Location |
Comma-separated list of paths to files and/or directories which contain the custom signer's JAR file and its dependencies (if any). |
| Destination Bucket |
The bucket that will receive the copy. |
| Destination Key |
The target key in the target bucket |
| Endpoint Override URL |
Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. |
| FullControl User List |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Full Control for an object |
| Owner |
The Amazon ID to use for the object's owner |
| Read ACL User List |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have permissions to read the Access Control List for an object |
| Read Permission User List |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Read Access for an object |
| Region |
The AWS Region to connect to. |
| SSL Context Service |
Specifies an optional SSL Context Service that, if provided, will be used to create connections |
| Signer Override |
The AWS S3 library uses Signature Version 4 by default but this property allows you to specify the Version 2 signer to support older S3-compatible services or even to plug in your own custom signer implementation. |
| Source Bucket |
The bucket that contains the file to be copied. |
| Source Key |
The source key in the source bucket |
| Write ACL User List |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have permissions to change the Access Control List for an object |
| Write Permission User List |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Write Access for an object |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| failure |
If the Processor is unable to process a given FlowFile, it will be routed to this Relationship. |
| success |
FlowFiles are routed to this Relationship after they have been successfully processed. |
## See also
- [org.apache.nifi.processors.aws.s3.DeleteS3Object](/user-guide/data-integration/openflow/processors/deletes3object)
- [org.apache.nifi.processors.aws.s3.FetchS3Object](/user-guide/data-integration/openflow/processors/fetchs3object)
- [org.apache.nifi.processors.aws.s3.GetS3ObjectMetadata](/user-guide/data-integration/openflow/processors/gets3objectmetadata)
- [org.apache.nifi.processors.aws.s3.GetS3ObjectTags](/user-guide/data-integration/openflow/processors/gets3objecttags)
- [org.apache.nifi.processors.aws.s3.ListS3](/user-guide/data-integration/openflow/processors/lists3)
- [org.apache.nifi.processors.aws.s3.PutS3Object](/user-guide/data-integration/openflow/processors/puts3object)
- [org.apache.nifi.processors.aws.s3.TagS3Object](/user-guide/data-integration/openflow/processors/tags3object)
---
title: CountText 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/counttext.md
section: Loading & Unloading Data
---
# CountText 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Counts various metrics on incoming text. The requested results will be recorded as attributes. The resulting flowfile will not have its content modified.
## Tags
character, count, line, text, word
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| ajust-immediately |
If true, the counter will be updated immediately, without regard to whether the ProcessSession is commit or rolled back;otherwise, the counter will be incremented only if and when the ProcessSession is committed. |
| character-encoding |
Specifies a character encoding to use. |
| split-words-on-symbols |
If enabled, the word count will identify strings separated by common logical delimiters [ _ - . ] as independent words (ex. split-words-on-symbols = 4 words). |
| text-character-count |
If enabled, will count the number of characters (including whitespace and symbols, but not including newlines and carriage returns) present in the incoming text. |
| text-line-count |
If enabled, will count the number of lines present in the incoming text. |
| text-line-nonempty-count |
If enabled, will count the number of lines that contain a non-whitespace character present in the incoming text. |
| text-word-count |
If enabled, will count the number of words (alphanumeric character groups bounded by whitespace) present in the incoming text. Common logical delimiters [_-.] do not bound a word unless 'Split Words on Symbols' is true. |
## Relationships
| Name |
Description |
| failure |
If the flowfile text cannot be counted for some reason, the original file will be routed to this destination and nothing will be routed elsewhere |
| success |
The flowfile contains the original content with one or more attributes added containing the respective counts |
## Writes attributes
| Name |
Description |
| text.line.count |
The number of lines of text present in the FlowFile content |
| text.line.nonempty.count |
The number of lines of text (with at least one non-whitespace character) present in the original FlowFile |
| text.word.count |
The number of words present in the original FlowFile |
| text.character.count |
The number of characters (given the specified character encoding) present in the original FlowFile |
## See also
- [org.apache.nifi.processors.standard.SplitText](/user-guide/data-integration/openflow/processors/splittext)
---
title: CreateAmazonAdsReport 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/createamazonadsreport.md
section: Loading & Unloading Data
---
# CreateAmazonAdsReport 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-amazon-ads-processors-nar
## Description
Processor which creates report configuration for Amazon Ads connector. By default it runs once a day.
## Tags
Amazon, Amazon Ads, report
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Access Token Provider |
Service providing OAuth access token. |
| Amazon Advertising Client ID |
Client ID of the Amazon Advertising user. |
| Region |
Environment from which advertising data will be downloaded. |
| Report Ad Product |
Type of advertising product being reported. |
| Report Columns |
List of columns fetched from Reporting API. |
| Report Filters |
Set of filters used to trim returned data. |
| Report Group By |
Level of granularity of the report. |
| Report Ingestion Strategy |
Configuration of the report ingestion. |
| Report Ingestion Window |
How many days from the past should be downloaded during incremental ingestion. |
| Report Name |
Unique name of the report. |
| Report Profile ID |
The profile ID associated with an advertising account in a specific marketplace. |
| Report Start Date |
Start date from which the ingestion should happen. |
| Report Time Unit |
Date aggregation. |
| Report Type |
Data type contained in the report. |
| Web Client Service Provider |
Service providing client for REST request execution. |
## State management
| Scopes |
Description |
| CLUSTER |
Stores information about last report definition in form of hash to detect schema changes. Incrementally loaded reports persist last ingestion date to define ingestion date ranges after initial load. Additionally start date is saved. |
## Relationships
| Name |
Description |
| success |
Response FlowFiles transferred when receiving success response from Amazon Ads Reporting API. |
## Writes attributes
| Name |
Description |
| amazon.ads.report.id |
Unique identifier of the currently prepared job. |
| amazon.ads.report.name |
Unique name of the report. |
| amazon.ads.ingestion.strategy |
Strategy which defines if the report will be downloaded as a SNAPSHOT or INCREMENTALLY. |
| amazon.ads.run.id |
Unique identifier of the current ingestion process. |
| amazon.ads.ingestion.start.date |
Date from which data is downloaded from Amazon Ads (including given date). |
| amazon.ads.ingestion.end.date |
Date to which data is downloaded from Amazon Ads (including given date). |
| amazon.ads.report.schema.changed |
Flag meaning if the report schema has changed between processor executions. |
| avro.schema |
Avro schema containing set of all configured fields. |
| fragment.identifier |
A unique ID of each ingestion run. Lets you identify all flow files generated during a single run. |
| fragment.index |
Number representing unique identifier in batch of flowfiles generated during one ingestion run. |
| fragment.count |
Amount of flowfiles generated during processor execution. |
---
title: CreateAzureOpenAiEmbeddings 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/createazureopenaiembeddings.md
section: Loading & Unloading Data
---
# CreateAzureOpenAiEmbeddings 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-openai-nar
## Description
Uses Azure OpenAI to create embeddings for text. The input text can be provided as a single FlowFile or as a record-oriented FlowFile.
## Tags
azure, chatbot, embeddings, gen ai, generative ai, llm, nlp, openai, openflow, text
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| API Key |
The API Key for authenticating to Azure OpenAI |
| Deployment Name |
The name of the OpenAI model deployment to use for creating embeddings |
| Dimensions |
The number of dimensions to request the resulting output embeddings have. This is only supported in text-embedding-3 and later models. |
| Embeddings Record Path |
The path to the field in the record where the embeddings are to be written. |
| Max Batch Size |
The maximum number of records to include in each batch sent to OpenAI |
| OpenAI Service Name |
The name of the OpenAI service to use |
| Record Reader |
The record reader to use for reading record-oriented data. If the incoming data is to be treated as plaintext, this property should be left unset. |
| Record Writer |
The Record Writer to use for writing the output |
| Text Record Path |
The path to the field in the record that contains the text to be embedded. If the incoming data is to be treated as plaintext, this property should be left unset. |
| User |
An identifier for the remote user on whose behalf the request is being made; OpenAI uses this to detect and prevent abuse. |
| Web Client Service |
The Web Client Service to use for communicating with OpenAI |
## Relationships
| Name |
Description |
| failure |
The original FlowFile will be routed to this relationship if the embeddings could not be created |
| success |
The embeddings will be routed to this relationship |
## Writes attributes
| Name |
Description |
| record.count |
The number of records written to the output |
| mime.type |
The MIME type of the output data, based on the chosen Record Writer |
## Use cases
| Create embeddings for text using Azure OpenAI's Embeddings |
| ---------------------------------------------------------- |
## See also
- [com.snowflake.openflow.runtime.processors.openai.CreateOpenAiEmbeddings](/user-guide/data-integration/openflow/processors/createopenaiembeddings)
- [com.snowflake.openflow.runtime.processors.openai.PromptAzureOpenAI](/user-guide/data-integration/openflow/processors/promptazureopenai)
---
title: CreateBoxFileMetadataInstance 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/createboxfilemetadatainstance.md
section: Loading & Unloading Data
---
# CreateBoxFileMetadataInstance 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-box-nar
## Description
Creates a metadata instance for a Box file using a specified template with values from the flowFile content. The Box API requires newly created templates to be created with the scope set as enterprise so no scope is required. The input record should be a flat key-value object where each field name is used as the metadata key.
## Tags
box, create, metadata, storage, templates
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Box Client Service |
Controller Service used to obtain a Box API connection. |
| File ID |
The ID of the file for which to create metadata. |
| Record Reader |
The Record Reader to use for parsing the incoming data |
| Template Key |
The key of the metadata template to use for creation. |
## Relationships
| Name |
Description |
| failure |
A FlowFile is routed to this relationship if an error occurs during metadata creation. |
| file not found |
FlowFiles for which the specified Box file was not found will be routed to this relationship. |
| success |
A FlowFile is routed to this relationship after metadata has been successfully created. |
| template not found |
FlowFiles for which the specified metadata template was not found will be routed to this relationship. |
## Writes attributes
| Name |
Description |
| box.id |
The ID of the file for which metadata was created |
| box.template.key |
The template key used for metadata creation |
| error.code |
The error code returned by Box |
| error.message |
The error message returned by Box |
## See also
- [org.apache.nifi.processors.box.FetchBoxFile](/user-guide/data-integration/openflow/processors/fetchboxfile)
- [org.apache.nifi.processors.box.ListBoxFile](/user-guide/data-integration/openflow/processors/listboxfile)
- [org.apache.nifi.processors.box.ListBoxFileMetadataTemplates](/user-guide/data-integration/openflow/processors/listboxfilemetadatatemplates)
- [org.apache.nifi.processors.box.UpdateBoxFileMetadataInstance](/user-guide/data-integration/openflow/processors/updateboxfilemetadatainstance)
---
title: CreateBoxMetadataTemplate 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/createboxmetadatatemplate.md
section: Loading & Unloading Data
---
# CreateBoxMetadataTemplate 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-box-nar
## Description
Creates a Box metadata template using field specifications from the flowFile content. Expects a schema with fields: " 'type' (required), 'key' (required), 'displayName' (optional), 'description' (optional), 'hidden' (optional, boolean).
## Tags
box, create, metadata, storage, templates
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Box Client Service |
Controller Service used to obtain a Box API connection. |
| Hidden |
Whether the template should be hidden in the Box UI. |
| Record Reader |
The Record Reader to use for parsing the incoming data |
| Template Key |
The key of the metadata template to create (used for API calls). |
| Template Name |
The display name of the metadata template to create. |
## Relationships
| Name |
Description |
| failure |
A FlowFile is routed to this relationship if an error occurs during template creation. |
| success |
A FlowFile is routed to this relationship after a template has been successfully created. |
## Writes attributes
| Name |
Description |
| box.template.name |
The template name that was created |
| box.template.key |
The template key that was created |
| box.template.scope |
The template scope. |
| box.template.fields.count |
Number of fields created for the template |
| error.code |
The error code returned by Box |
| error.message |
The error message returned by Box |
## See also
- [org.apache.nifi.processors.box.ListBoxFileMetadataTemplates](/user-guide/data-integration/openflow/processors/listboxfilemetadatatemplates)
- [org.apache.nifi.processors.box.UpdateBoxFileMetadataInstance](/user-guide/data-integration/openflow/processors/updateboxfilemetadatainstance)
---
title: CreateCohereEmbeddings 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/createcohereembeddings.md
section: Loading & Unloading Data
---
# CreateCohereEmbeddings 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-cohere-nar
## Description
Uses Cohere to create embeddings for text. The input text can be provided as a single FlowFile or as a record-oriented FlowFile.
## Tags
chatbot, cohere, embeddings, gen ai, generative ai, llm, nlp, openflow, text
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Cohere API Key |
The API Key for authenticating to Cohere |
| Embedding Type |
Specifies the types of embeddings you want to get back. |
| Embeddings Model |
The model to use for embeddings, available models are listed at https://docs.cohere.com/reference/embed (https://docs.cohere.com/reference/embed) |
| Embeddings Record Path |
The path to the field in the record where the embeddings are to be written. |
| Input Type |
Specifies the type of input passed to the model. Required for embedding models v3 and higher. |
| Max Batch Size |
The maximum number of records to include in each batch sent to Cohere |
| Record Reader |
The record reader to use for reading record-oriented data. If the incoming data is to be treated as plaintext, this property should be left unset. |
| Record Writer |
The Record Writer to use for writing the output |
| Text Record Path |
The path to the field in the record that contains the text to be embedded. If the incoming data is to be treated as plaintext, this property should be left unset. |
| Truncate Policy |
One of NONE%start%END to specify how the API will handle inputs longer than the maximum token length. |
| User |
An identifier for the remote user on whose behalf the request is being made. |
## Relationships
| Name |
Description |
| failure |
The original FlowFile will be routed to this relationship if the embeddings could not be created |
| success |
The embeddings will be routed to this relationship |
## Writes attributes
| Name |
Description |
| record.count |
The number of records written to the output |
| mime.type |
The MIME type of the output data, based on the chosen Record Writer |
## Use cases
| Create embeddings for text using Cohere's Embedding model |
| --------------------------------------------------------- |
---
title: CreateMetaAdsReport 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/createmetaadsreport.md
section: Loading & Unloading Data
---
# CreateMetaAdsReport 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-meta-ads-processors-nar
## Description
Processor which creates report configuration for Meta Ads connector. By default it runs once a day.
## Tags
Facebook, Meta, Meta Ads, report
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Access Token |
Token required to request Meta Ads Marketing API. It must match pattern 'Bearer <Access Token Value>'. |
| Action Report Time |
Determine the report time of action stats. |
| Click Attribution Window |
Attribution window for the click action. |
| Meta Ads API Version |
Version of Meta Ads API which is used for report generation. |
| Report Breakdowns |
List of values which determine how to break down the result. Multiple breakdowns can be picked, but only some combinations work. |
| Report Fields |
List of fields fetched from Marketing API. If non are selected most used fields will be downloaded. |
| Report Ingestion Strategy |
Configuration of the report ingestion. |
| Report Level |
Granularity of the report. |
| Report Name |
Unique name of the report. |
| Report Object ID |
ID of the object from which data will be fetched. It can be Account, Campaign, Ad or Ad Set ID. |
| Report Start Date |
Start date from which the ingestion should happen. |
| Report Time Increment |
Value of aggregation in days. |
| View Attribution Window |
Attribution window for the view action. |
| Web Client Service Provider |
Service providing client for REST request execution. |
## State management
| Scopes |
Description |
| CLUSTER |
Stores information about last report definition in form of hash to detect schema changes. Incrementally loaded reports persist last ingestion date to define ingestion date ranges after initial load. Additionally start date is saved. |
## Relationships
| Name |
Description |
| success |
Response FlowFiles transferred when receiving success response from Meta Ads Marketing API. |
## Writes attributes
| Name |
Description |
| meta.ads.report.id |
Unique identifier of the currently prepared job. |
| meta.ads.report.name |
Unique name of the report. |
| meta.ads.report.ingestion.strategy |
Strategy which defines if the report will be downloaded as a SNAPSHOT or INCREMENTALLY. |
| meta.ads.run.id |
Unique identifier of the current ingestion process. |
| meta.ads.ingestion.start.date |
Date from which data is downloaded from Meta Ads (including given date). |
| meta.ads.ingestion.end.date |
Date to which data is downloaded from Meta Ads (including given date). |
| meta.ads.report.schema.changed |
Flag meaning if the report schema has changed between processor executions. |
| avro.schema |
Avro schema containing set of all configured fields. |
---
title: CreateOpenAiEmbeddings 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/createopenaiembeddings.md
section: Loading & Unloading Data
---
# CreateOpenAiEmbeddings 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-openai-nar
## Description
Uses OpenAI to create embeddings for text. The input text can be provided as a single FlowFile or as a record-oriented FlowFile.
## Tags
chatbot, embeddings, gen ai, generative ai, llm, nlp, openai, openflow, text
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Dimensions |
The number of dimensions to request the resulting output embeddings have. This is only supported in text-embedding-3 and later models. |
| Embeddings Model |
The model to use for embeddings |
| Embeddings Record Path |
The path to the field in the record where the embeddings are to be written. |
| Max Batch Size |
The maximum number of records to include in each batch sent to OpenAI |
| OpenAI API Key |
The API Key for authenticating to OpenAI |
| OpenAI Organization |
The organization to use for OpenAI |
| Record Reader |
The record reader to use for reading record-oriented data. If the incoming data is to be treated as plaintext, this property should be left unset. |
| Record Writer |
The Record Writer to use for writing the output |
| Text Record Path |
The path to the field in the record that contains the text to be embedded. If the incoming data is to be treated as plaintext, this property should be left unset. |
| User |
An identifier for the remote user on whose behalf the request is being made; OpenAI uses this to detect and prevent abuse. |
| Web Client Service |
The Web Client Service to use for communicating with OpenAI |
## Relationships
| Name |
Description |
| failure |
The original FlowFile will be routed to this relationship if the embeddings could not be created |
| success |
The embeddings will be routed to this relationship |
## Writes attributes
| Name |
Description |
| record.count |
The number of records written to the output |
| mime.type |
The MIME type of the output data, based on the chosen Record Writer |
## Use cases
| Create embeddings for text using OpenAI's Embeddings |
| ---------------------------------------------------- |
## See also
- [com.snowflake.openflow.runtime.processors.openai.CreateAzureOpenAiEmbeddings](/user-guide/data-integration/openflow/processors/createazureopenaiembeddings)
- [com.snowflake.openflow.runtime.processors.openai.PromptOpenAI](/user-guide/data-integration/openflow/processors/promptopenai)
---
title: CreateSnowflakeEmbeddings 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/createsnowflakeembeddings.md
section: Loading & Unloading Data
---
# CreateSnowflakeEmbeddings 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-snowflake-processors-nar
## Description
Create vector embeddings using Snowflake Cortex Large Language Model functions
## Tags
chatbot, embeddings, gen ai, generative ai, llm, nlp, openflow, snowflake, text
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Dimensions |
The number of dimensions to request the resulting output embeddings have. |
| Embeddings Model |
The model to use for embeddings |
| Record Writer |
The Record Writer to use for writing the output |
| Snowflake Connection Service |
Database Connection Service for accessing Snowflake |
## Relationships
| Name |
Description |
| failure |
The original FlowFile will be routed to this relationship if the embeddings could not be created |
| success |
The embeddings will be routed to this relationship |
## Writes attributes
| Name |
Description |
| record.count |
The number of records written to the output |
| mime.type |
The MIME type of the output data, based on the chosen Record Writer |
---
title: CreateVertexAIEmbeddings 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/createvertexaiembeddings.md
section: Loading & Unloading Data
---
# CreateVertexAIEmbeddings 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-vertexai-nar
## Description
Uses VertexAI to create embeddings for text. The input text can be provided as a single FlowFile or as a record-oriented FlowFile.
## Tags
chatbot, cloud, embeddings, gcp, gen ai, generative ai, google, llm, nlp, openflow, text, vertex
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Auto Truncate |
If set to false, text that exceeds the token limit causes the request to fail. |
| Embeddings Model |
The model to use for embeddings, available models are listed at https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#models (https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#models) |
| Embeddings Record Path |
The path to the field in the record where the embeddings are to be written. |
| GCP Credentials Service |
The Controller Service used to obtain Google Cloud Platform credentials. |
| GCP Location |
The location to configure the Vertex client with |
| GCP Project ID |
The project ID to configure the Vertex client with |
| Max Batch Size |
The maximum number of records to include in each batch sent to VertexAI |
| Model Publisher |
The publisher of the model |
| Output Dimensionality |
Used to specify output embedding size. If set, output embeddings will be truncated to the size specified. |
| Record Reader |
The record reader to use for reading record-oriented data. If the incoming data is to be treated as plaintext, this property should be left unset. |
| Record Writer |
The Record Writer to use for writing the output |
| Task Type |
Used to convey intended downstream application of embeddings to help the model tune embeddings for a specific purpose. |
| Text Record Path |
The path to the field in the record that contains the text to be embedded. If the incoming data is to be treated as plaintext, this property should be left unset. |
| User |
An identifier for the remote user on whose behalf the request is being made. |
## Relationships
| Name |
Description |
| failure |
The original FlowFile will be routed to this relationship if the embeddings could not be created |
| success |
The embeddings will be routed to this relationship |
## Writes attributes
| Name |
Description |
| record.count |
The number of records written to the output |
| mime.type |
The MIME type of the output data, based on the chosen Record Writer |
## Use cases
| Create embeddings for text using VertexAI's Embedding model |
| ----------------------------------------------------------- |
---
title: CryptographicHashContent 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/cryptographichashcontent.md
section: Loading & Unloading Data
---
# CryptographicHashContent 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Calculates a cryptographic hash value for the flowfile content using the given algorithm and writes it to an output attribute. Please refer to https://csrc.nist.gov/Projects/Hash-Functions/NIST-Policy-on-Hash-Functions (https://csrc.nist.gov/Projects/Hash-Functions/NIST-Policy-on-Hash-Functions) for help to decide which algorithm to use.
## Tags
blake2, content, cryptography, hash, md5, sha
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| fail_when_empty |
Route to failure if the content is empty. While hashing an empty value is valid, some flows may want to detect empty input. |
| hash_algorithm |
The hash algorithm to use. Note that not all of the algorithms available are recommended for use (some are provided for legacy compatibility). There are many things to consider when picking an algorithm; it is recommended to use the most secure algorithm possible. |
## Relationships
| Name |
Description |
| failure |
Used for flowfiles that have no content if the 'fail on empty' setting is enabled |
| success |
Used for flowfiles that have a hash value added |
## Writes attributes
| Name |
Description |
| content_<algorithm> |
This processor adds an attribute whose value is the result of hashing the flowfile content. The name of this attribute is specified by the value of the algorithm, e.g. 'content_SHA-256'. |
---
title: CSVReader
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/csvreader.md
section: Loading & Unloading Data
---
# CSVReader
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Parses CSV-formatted data, returning each row in the CSV file as a separate record. This reader allows for inferring a schema based on the first line of the CSV, if a 'header line' is present, or providing an explicit schema for interpreting the values. See Controller Service's Usage for further documentation.
## Tags
comma, csv, delimited, parse, reader, record, row, separated, values
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Allow Duplicate Header Names |
Allow Duplicate Header Names |
true |
- true
- false
|
Whether duplicate header names are allowed. Header names are case-sensitive, for example "name" and "Name" are treated as separate fields.Handling of duplicate header names is CSV Parser specific (where applicable):* Apache Commons CSV - duplicate headers will result in column data "shifting" right with new fields created for "unknown_field_index_X" where "X" is the CSV column index number* Jackson CSV - duplicate headers will be de-duplicated with the field value being that of the right-most duplicate CSV column* FastCSV - duplicate headers will be de-duplicated with the field value being that of the left-most duplicate CSV column |
| CSV Format * |
CSV Format |
custom |
- Custom Format
- RFC 4180
- Microsoft Excel
- Tab-Delimited
- MySQL Format
- Informix Unload
- Informix Unload Escape Disabled
|
Specifies which "format" the CSV data is in, or specifies if custom formatting should be used. |
| Character Set * |
Character Set |
UTF-8 |
|
The Character Encoding that is used to encode/decode the CSV file |
| Comment Marker |
Comment Marker |
|
|
The character that is used to denote the start of a comment. Any line that begins with this comment will be ignored. |
| Date Format |
Date Format |
|
|
Specifies the format to use when reading/writing Date fields. If not specified, Date fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, MM/dd/yyyy for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters, as in 01/01/2017). |
| Escape Character * |
Escape Character |
|
|
The character that is used to escape characters that would otherwise have a specific meaning to the CSV Parser. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Escape Character at runtime, then it will be skipped and the default Escape Character will be used. Setting it to an empty string means no escape character should be used. |
| Ignore CSV Header Column Names |
Ignore CSV Header Column Names |
false |
- true
- false
|
If the first line of a CSV is a header, and the configured schema does not match the fields named in the header line, this controls how the Reader will interpret the fields. If this property is true, then the field names mapped to each column are driven only by the configured schema and any fields not in the schema will be ignored. If this property is false, then the field names found in the CSV Header will be used as the names of the fields. |
| Null String |
Null String |
|
|
Specifies a String that, if present as a value in the CSV, should be considered a null field instead of using the literal value. |
| Quote Character * |
Quote Character |
" |
|
The character that is used to quote values so that escape characters do not have to be used. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Quote Character at runtime, then it will be skipped and the default Quote Character will be used. |
| Record Separator * |
Record Separator |
n |
|
Specifies the characters to use in order to separate CSV Records |
| Schema Access Strategy * |
Schema Access Strategy |
infer-schema |
- Use 'Schema Name' Property
- Use 'Schema Text' Property
- Schema Reference Reader
- Use String Fields From Header
- Infer Schema
|
Specifies how to obtain the schema that is to be used for interpreting the data. |
| Schema Branch |
Schema Branch |
|
|
Specifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored. |
| Schema Name |
Schema Name |
$\{schema.name\} |
|
Specifies the name of the schema to lookup in the Schema Registry property |
| Schema Reference Reader * |
Schema Reference Reader |
|
|
Service implementation responsible for reading FlowFile attributes or content to determine the Schema Reference Identifier |
| Schema Registry |
Schema Registry |
|
|
Specifies the Controller Service to use for the Schema Registry |
| Schema Text |
Schema Text |
$\{avro.schema\} |
|
The text of an Avro-formatted Schema |
| Schema Version |
Schema Version |
|
|
Specifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved. |
| Time Format |
Time Format |
|
|
Specifies the format to use when reading/writing Time fields. If not specified, Time fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, HH:mm:ss for a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 18:04:15). |
| Timestamp Format |
Timestamp Format |
|
|
Specifies the format to use when reading/writing Timestamp fields. If not specified, Timestamp fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, MM/dd/yyyy HH:mm:ss for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters; and then followed by a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 01/01/2017 18:04:15). |
| Treat First Line as Header * |
Treat First Line as Header |
false |
- true
- false
|
Specifies whether or not the first line of CSV should be considered a Header or should be considered a record. If the Schema Access Strategy indicates that the columns must be defined in the header, then this property will be ignored, since the header must always be present and won't be processed as a Record. Otherwise, if 'true', then the first line of CSV data will not be processed as a record and if 'false',then the first line will be interpreted as a record. |
| Trim Fields * |
Trim Fields |
true |
- true
- false
|
Whether or not white space should be removed from the beginning and end of fields |
| Trim double quote * |
Trim double quote |
true |
- true
- false
|
Whether or not to trim starting and ending double quotes. For example: with trim string '"test"' would be parsed to 'test', without trim would be parsed to '"test"'.If set to 'false' it means full compliance with RFC-4180. Default value is true, with trim. |
| Value Separator * |
Value Separator |
, |
|
The character that is used to separate values/fields in a CSV Record. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Value Separator at runtime, then it will be skipped and the default Value Separator will be used. |
| CSV Parser * |
csv-reader-csv-parser |
commons-csv |
- Apache Commons CSV
- Jackson CSV
- FastCSV
|
Specifies which parser to use to read CSV records. NOTE: Different parsers may support different subsets of functionality and may also exhibit different levels of performance. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: CSVRecordLookupService
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/csvrecordlookupservice.md
section: Loading & Unloading Data
---
# CSVRecordLookupService
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
A reloadable CSV file-based lookup service. When the lookup key is found in the CSV file, the columns are returned as a Record. All returned fields will be strings. The first line of the csv file is considered as header.
## Tags
cache, csv, enrich, join, key, lookup, record, reloadable, value
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| CSV Format * |
CSV Format |
default |
- Custom Format
- RFC 4180
- Microsoft Excel
- Tab-Delimited
- MySQL Format
- Informix Unload
- Informix Unload Escape Disabled
- Default Format
- RFC4180
|
Specifies which "format" the CSV data is in, or specifies if custom formatting should be used. |
| Character Set * |
Character Set |
UTF-8 |
|
The Character Encoding that is used to decode the CSV file. |
| Comment Marker |
Comment Marker |
|
|
The character that is used to denote the start of a comment. Any line that begins with this comment will be ignored. |
| Escape Character * |
Escape Character |
|
|
The character that is used to escape characters that would otherwise have a specific meaning to the CSV Parser. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Escape Character at runtime, then it will be skipped and the default Escape Character will be used. Setting it to an empty string means no escape character should be used. |
| Quote Character * |
Quote Character |
" |
|
The character that is used to quote values so that escape characters do not have to be used. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Quote Character at runtime, then it will be skipped and the default Quote Character will be used. |
| Quote Mode * |
Quote Mode |
MINIMAL |
- Quote All Values
- Quote Minimal
- Quote Non-Numeric Values
- Do Not Quote Values
|
Specifies how fields should be quoted when they are written |
| Trim Fields * |
Trim Fields |
true |
- true
- false
|
Whether or not white space should be removed from the beginning and end of fields |
| Value Separator * |
Value Separator |
, |
|
The character that is used to separate values/fields in a CSV Record. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Value Separator at runtime, then it will be skipped and the default Value Separator will be used. |
| CSV File * |
csv-file |
|
|
Path to a CSV File in which the key value pairs can be looked up. |
| Ignore Duplicates * |
ignore-duplicates |
true |
- true
- false
|
Ignore duplicate keys for records in the CSV file. |
| Lookup Key Column * |
lookup-key-column |
|
|
The field in the CSV file that will serve as the lookup key. This is the field that will be matched against the property specified in the lookup processor. |
## State management
This component does not store state.
## Restricted
## Restrictions
| Required Permission |
Explanation |
| read filesystem |
Provides operator the ability to read from any file that NiFi has access to. |
## System Resource Considerations
This component does not specify system resource considerations.
---
title: CSVRecordSetWriter
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/csvrecordsetwriter.md
section: Loading & Unloading Data
---
# CSVRecordSetWriter
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Writes the contents of a RecordSet as CSV data. The first line written will be the column names (unless the 'Include Header Line' property is false). All subsequent lines will be the values corresponding to the record fields.
## Tags
csv, delimited, record, recordset, result, row, separated, serializer, set, tab, tsv, writer
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| CSV Format * |
CSV Format |
custom |
- Custom Format
- RFC 4180
- Microsoft Excel
- Tab-Delimited
- MySQL Format
- Informix Unload
- Informix Unload Escape Disabled
|
Specifies which "format" the CSV data is in, or specifies if custom formatting should be used. |
| Character Set * |
Character Set |
UTF-8 |
|
The Character Encoding that is used to encode/decode the CSV file |
| Comment Marker |
Comment Marker |
|
|
The character that is used to denote the start of a comment. Any line that begins with this comment will be ignored. |
| Date Format |
Date Format |
|
|
Specifies the format to use when reading/writing Date fields. If not specified, Date fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, MM/dd/yyyy for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters, as in 01/01/2017). |
| Escape Character * |
Escape Character |
|
|
The character that is used to escape characters that would otherwise have a specific meaning to the CSV Parser. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Escape Character at runtime, then it will be skipped and the default Escape Character will be used. Setting it to an empty string means no escape character should be used. |
| Include Header Line * |
Include Header Line |
true |
- true
- false
|
Specifies whether or not the CSV column names should be written out as the first line. |
| Include Trailing Delimiter * |
Include Trailing Delimiter |
false |
- true
- false
|
If true, a trailing delimiter will be added to each CSV Record that is written. If false, the trailing delimiter will be omitted. |
| Null String |
Null String |
|
|
Specifies a String that, if present as a value in the CSV, should be considered a null field instead of using the literal value. |
| Quote Character * |
Quote Character |
" |
|
The character that is used to quote values so that escape characters do not have to be used. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Quote Character at runtime, then it will be skipped and the default Quote Character will be used. |
| Quote Mode * |
Quote Mode |
MINIMAL |
- Quote All Values
- Quote Minimal
- Quote Non-Numeric Values
- Do Not Quote Values
|
Specifies how fields should be quoted when they are written |
| Record Separator * |
Record Separator |
n |
|
Specifies the characters to use in order to separate CSV Records |
| Schema Access Strategy * |
Schema Access Strategy |
inherit-record-schema |
- Inherit Record Schema
- Use 'Schema Name' Property
- Use 'Schema Text' Property
|
Specifies how to obtain the schema that is to be used for interpreting the data. |
| Schema Branch |
Schema Branch |
|
|
Specifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored. |
| Schema Cache |
Schema Cache |
|
|
Specifies a Schema Cache to add the Record Schema to so that Record Readers can quickly lookup the schema. |
| Schema Name |
Schema Name |
$\{schema.name\} |
|
Specifies the name of the schema to lookup in the Schema Registry property |
| Schema Reference Reader * |
Schema Reference Reader |
|
|
Service implementation responsible for reading FlowFile attributes or content to determine the Schema Reference Identifier |
| Schema Reference Writer * |
Schema Reference Writer |
|
|
Service implementation responsible for writing FlowFile attributes or content header with Schema reference information |
| Schema Registry |
Schema Registry |
|
|
Specifies the Controller Service to use for the Schema Registry |
| Schema Text |
Schema Text |
$\{avro.schema\} |
|
The text of an Avro-formatted Schema |
| Schema Version |
Schema Version |
|
|
Specifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved. |
| Schema Write Strategy * |
Schema Write Strategy |
no-schema |
- Do Not Write Schema
- Set 'schema.name' Attribute
- Set 'avro.schema' Attribute
- Schema Reference Writer
|
Specifies how the schema for a Record should be added to the data. |
| Time Format |
Time Format |
|
|
Specifies the format to use when reading/writing Time fields. If not specified, Time fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, HH:mm:ss for a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 18:04:15). |
| Timestamp Format |
Timestamp Format |
|
|
Specifies the format to use when reading/writing Timestamp fields. If not specified, Timestamp fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, MM/dd/yyyy HH:mm:ss for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters; and then followed by a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 01/01/2017 18:04:15). |
| Trim Fields * |
Trim Fields |
true |
- true
- false
|
Whether or not white space should be removed from the beginning and end of fields |
| Value Separator * |
Value Separator |
, |
|
The character that is used to separate values/fields in a CSV Record. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Value Separator at runtime, then it will be skipped and the default Value Separator will be used. |
| CSV Writer * |
csv-writer |
commons-csv |
- Apache Commons CSV
- FastCSV
|
Specifies which writer implementation to use to write CSV records. NOTE: Different writers may support different subsets of functionality and may also exhibit different levels of performance. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: DatabaseLookup
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/databaselookup.md
section: Loading & Unloading Data
---
# DatabaseLookup
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
A Lookup Service that allows for enrichment with a database using a user-specified SQL statement. The SQL statement may reference any value from the FlowFile's Record that is provided by the calling Processor.
## Tags
database, enrich, join, lookup, openflow, rdbms, record, sql
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Connection Pooling Service * |
Connection Pooling Service |
|
|
The Connection Pooling Service that is used to obtain a connection to the database |
| Max Array Size * |
Max Array Size |
1000 |
|
The maximum number of records to include in the array. This is a mechanism to ensure that the returned results due not cause memory issues. If the result set contains more records than this value, the lookup will fail. If the desire is instead to limit the number of rows returned, a LIMIT clause should be added to the SQL. |
| Multiple Result Field Name * |
Multiple Result Field Name |
results |
|
If multiple results are returned, they will be combined into an array. This property dictates the name of the field in the returned record. |
| Multiple Result Strategy * |
Multiple Result Strategy |
Fail |
- Use Array
- Use First Only
- Fail
|
Specifies how to handle the situation where the lookup results in multiple records. |
| SQL * |
SQL |
|
|
The SQL statement to execute against the database in order to lookup the value. The statement may reference any attributes or values from the incoming Record that are provided by the calling Processor via Expression Language. The processor is will extract any Expression Language expressions and replace them with parameterized values so that the SQL can be safely executed, avoiding SQL Injection attacks. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: DatabaseRecordLookupService
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/databaserecordlookupservice.md
section: Loading & Unloading Data
---
# DatabaseRecordLookupService
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
A relational-database-based lookup service. When the lookup key is found in the database, the specified columns (or all if Lookup Value Columns are not specified) are returned as a Record. Only one row will be returned for each lookup, duplicate database entries are ignored.
## Tags
cache, database, enrich, join, key, lookup, rdbms, record, reloadable, value
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Cache Expiration |
Cache Expiration |
|
|
Time interval to clear all cache entries. If the Cache Size is zero then this property is ignored. |
| Default Decimal Precision * |
Default Decimal Precision |
10 |
|
When a DECIMAL/NUMBER value is written as a 'decimal' Avro logical type, a specific 'precision' denoting number of available digits is required. Generally, precision is defined by column data type definition or database engines default. However undefined precision (0) can be returned from some database engines. 'Default Decimal Precision' is used when writing those undefined precision numbers. |
| Default Decimal Scale * |
Default Decimal Scale |
0 |
|
When a DECIMAL/NUMBER value is written as a 'decimal' Avro logical type, a specific 'scale' denoting number of available decimal digits is required. Generally, scale is defined by column data type definition or database engines default. However when undefined precision (0) is returned, scale can also be uncertain with some database engines. 'Default Decimal Scale' is used when writing those undefined numbers. If a value has more decimals than specified scale, then the value will be rounded-up, e.g. 1.53 becomes 2 with scale 0, and 1.5 with scale 1. |
| Cache Size * |
dbrecord-lookup-cache-size |
0 |
|
Specifies how many lookup values/records should be cached. The cache is shared for all tables and keeps a map of lookup values to records. Setting this property to zero means no caching will be done and the table will be queried for each lookup value in each record. If the lookup table changes often or the most recent data must be retrieved, do not use the cache. |
| Clear Cache on Enabled * |
dbrecord-lookup-clear-cache-on-enabled |
true |
- true
- false
|
Whether to clear the cache when this service is enabled. If the Cache Size is zero then this property is ignored. Clearing the cache when the service is enabled ensures that the service will first go to the database to get the most recent data. |
| Database Connection Pooling Service * |
dbrecord-lookup-dbcp-service |
|
|
The Controller Service that is used to obtain connection to database |
| Lookup Key Column * |
dbrecord-lookup-key-column |
|
|
The column in the table that will serve as the lookup key. This is the column that will be matched against the property specified in the lookup processor. Note that this may be case-sensitive depending on the database. |
| Table Name * |
dbrecord-lookup-table-name |
|
|
The name of the database table to be queried. Note that this may be case-sensitive depending on the database. |
| Lookup Value Columns |
dbrecord-lookup-value-columns |
|
|
A comma-delimited list of columns in the table that will be returned when the lookup key matches. Note that this may be case-sensitive depending on the database. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: DatabaseRecordSink
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/databaserecordsink.md
section: Loading & Unloading Data
---
# DatabaseRecordSink
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides a service to write records using a configured database connection.
## Tags
connection, database, db, jdbc, record
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Catalog Name |
db-record-sink-catalog-name |
|
|
The name of the catalog that the statement should update. This may not apply for the database that you are updating. In this case, leave the field empty |
| Database Connection Pooling Service * |
db-record-sink-dcbp-service |
|
|
The Controller Service that is used to obtain a connection to the database for sending records. |
| Max Wait Time * |
db-record-sink-query-timeout |
0 seconds |
|
The maximum amount of time allowed for a running SQL statement , zero means there is no limit. Max time less than 1 second will be equal to zero. |
| Quote Column Identifiers |
db-record-sink-quoted-identifiers |
false |
- true
- false
|
Enabling this option will cause all column names to be quoted, allowing you to use reserved words as column names in your tables. |
| Quote Table Identifiers |
db-record-sink-quoted-table-identifiers |
false |
- true
- false
|
Enabling this option will cause the table name to be quoted to support the use of special characters in the table name. |
| Schema Name |
db-record-sink-schema-name |
|
|
The name of the schema that the table belongs to. This may not apply for the database that you are updating. In this case, leave the field empty |
| Table Name * |
db-record-sink-table-name |
|
|
The name of the table that the statement should affect. |
| Translate Field Names |
db-record-sink-translate-field-names |
true |
- true
- false
|
If true, the Processor will attempt to translate field names into the appropriate column names for the table specified. If false, the field names must match the column names exactly, or the column will not be updated |
| Unmatched Column Behavior |
db-record-sink-unmatched-column-behavior |
Fail on Unmatched Columns |
- Ignore Unmatched Columns
- Warn on Unmatched Columns
- Fail on Unmatched Columns
|
If an incoming record does not have a field mapping for all of the database table's columns, this property specifies how to handle the situation |
| Unmatched Field Behavior |
db-record-sink-unmatched-field-behavior |
Ignore Unmatched Fields |
- Ignore Unmatched Fields
- Fail on Unmatched Fields
|
If an incoming record has a field that does not map to any of the database table's columns, this property specifies how to handle the situation |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: DBCPConnectionPool
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/dbcpconnectionpool.md
section: Loading & Unloading Data
---
# DBCPConnectionPool
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides Database Connection Pooling Service. Connections can be asked from pool and returned after usage.
## Tags
connection, database, dbcp, jdbc, pooling, store
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Database Connection URL * |
Database Connection URL |
|
|
A database connection URL used to connect to a database. May contain database system name, host, port, database name and some parameters. The exact syntax of a database connection URL is specified by your DBMS. |
| Database Driver Class Name * |
Database Driver Class Name |
|
|
Database driver class name |
| Database Driver Location(s) |
Database Driver Location(s) |
|
|
Comma-separated list of files/folders and/or URLs containing the driver JAR and its dependencies (if any). For example '/var/tmp/mariadb-java-client-1.1.7.jar' |
| Database User |
Database User |
|
|
Database user name |
| Kerberos User Service |
Kerberos User Service |
|
|
Specifies the Kerberos User Controller Service that should be used for authenticating with Kerberos |
| Max Total Connections * |
Max Total Connections |
8 |
|
The maximum number of active connections that can be allocated from this pool at the same time, or negative for no limit. |
| Max Wait Time * |
Max Wait Time |
500 millis |
|
The maximum amount of time that the pool will wait (when there are no available connections) for a connection to be returned before failing, or -1 to wait indefinitely. |
| Maximum Connection Lifetime |
Maximum Connection Lifetime |
-1 |
|
The maximum lifetime of a connection. After this time is exceeded the connection will fail the next activation, passivation or validation test. A value of zero or less means the connection has an infinite lifetime. |
| Maximum Idle Connections |
Maximum Idle Connections |
8 |
|
The maximum number of connections that can remain idle in the pool without extra ones being released. Set to any negative value to allow unlimited idle connections. |
| Minimum Evictable Idle Time |
Minimum Evictable Idle Time |
30 mins |
|
The minimum amount of time a connection may sit idle in the pool before it is eligible for eviction. |
| Minimum Idle Connections |
Minimum Idle Connections |
0 |
|
The minimum number of connections that can remain idle in the pool without extra ones being created. Set to or zero to allow no idle connections. |
| Password |
Password |
|
|
The password for the database user |
| Soft Minimum Evictable Idle Time |
Soft Minimum Evictable Idle Time |
-1 |
|
The minimum amount of time a connection may sit idle in the pool before it is eligible for eviction by the idle connection evictor, with the extra condition that at least a minimum number of idle connections remain in the pool. When the not-soft version of this option is set to a positive value, it is examined first by the idle connection evictor: when idle connections are visited by the evictor, idle time is first compared against it (without considering the number of idle connections in the pool) and then against this soft option, including the minimum idle connections constraint. |
| Time Between Eviction Runs |
Time Between Eviction Runs |
-1 |
|
The time period to sleep between runs of the idle connection evictor thread. When non-positive, no idle connection evictor thread will be run. |
| Validation Query |
Validation Query |
|
|
Validation query used to validate connections before returning them. When connection is invalid, it gets dropped and new valid connection will be returned. Note!! Using validation might have some performance penalty. |
## State management
This component does not store state.
## Restricted
## Restrictions
| Required Permission |
Explanation |
| reference remote resources |
Database Driver Location can reference resources over HTTP |
## System Resource Considerations
This component does not specify system resource considerations.
---
title: DBCPConnectionPoolLookup
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/dbcpconnectionpoollookup.md
section: Loading & Unloading Data
---
# DBCPConnectionPoolLookup
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides a DBCPService that can be used to dynamically select another DBCPService. This service requires an attribute named 'database.name' to be passed in when asking for a connection, and will throw an exception if the attribute is missing. The value of 'database.name' will be used to select the DBCPService that has been registered with that name. This will allow multiple DBCPServices to be defined and registered, and then selected dynamically at runtime by tagging flow files with the appropriate 'database.name' attribute.
## Tags
connection, database, dbcp, jdbc, pooling, store
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: DebugFlow 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/debugflow.md
section: Loading & Unloading Data
---
# DebugFlow 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
The DebugFlow processor aids testing and debugging the FlowFile framework by allowing various responses to be explicitly triggered in response to the receipt of a FlowFile or a timer event without a FlowFile if using timer or cron based scheduling. It can force responses needed to exercise or test various failure modes that can occur when a processor runs.
## Tags
FlowFile, debug, flow, processor, test, utility
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| @OnScheduled Pause Time |
Specifies how long the processor should sleep in the @OnScheduled method, so that the processor can be forced to take a long time to start up |
| @OnStopped Pause Time |
Specifies how long the processor should sleep in the @OnStopped method, so that the processor can be forced to take a long time to shutdown |
| @OnUnscheduled Pause Time |
Specifies how long the processor should sleep in the @OnUnscheduled method, so that the processor can be forced to take a long time to respond when user clicks stop |
| Content Size |
The number of bytes to write each time that the FlowFile is written to |
| CustomValidate Pause Time |
Specifies how long the processor should sleep in the customValidate() method |
| Fail When @OnScheduled called |
Specifies whether or not the Processor should throw an Exception when the methods annotated with @OnScheduled are called |
| Fail When @OnStopped called |
Specifies whether or not the Processor should throw an Exception when the methods annotated with @OnStopped are called |
| Fail When @OnUnscheduled called |
Specifies whether or not the Processor should throw an Exception when the methods annotated with @OnUnscheduled are called |
| FlowFile Exception Class |
Exception class to be thrown (must extend java.lang. RuntimeException). |
| FlowFile Exception Iterations |
Number of FlowFiles to throw exception. |
| FlowFile Failure Iterations |
Number of FlowFiles to forward to failure relationship. |
| FlowFile Rollback Iterations |
Number of FlowFiles to roll back (without penalty). |
| FlowFile Rollback Penalty Iterations |
Number of FlowFiles to roll back with penalty. |
| FlowFile Rollback Yield Iterations |
Number of FlowFiles to roll back and yield. |
| FlowFile Success Iterations |
Number of FlowFiles to forward to success relationship. |
| Ignore Interrupts When Paused |
If the Processor's thread(s) are sleeping (due to one of the "Pause Time" properties above), and the thread is interrupted, this indicates whether the Processor should ignore the interrupt and continue sleeping or if it should allow itself to be interrupted. |
| No FlowFile Exception Class |
Exception class to be thrown if no FlowFile (must extend java.lang. RuntimeException). |
| No FlowFile Exception Iterations |
Number of times to throw NPE exception if no FlowFile. |
| No FlowFile Skip Iterations |
Number of times to skip onTrigger if no FlowFile. |
| No FlowFile Yield Iterations |
Number of times to yield if no FlowFile. |
| OnTrigger Pause Time |
Specifies how long the processor should sleep in the onTrigger() method, so that the processor can be forced to take a long time to perform its task |
| Write Iterations |
Number of times to write to the FlowFile |
## Relationships
| Name |
Description |
| failure |
FlowFiles that failed to process. |
| success |
FlowFiles processed successfully. |
---
title: DecryptContentAge 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/decryptcontentage.md
section: Loading & Unloading Data
---
# DecryptContentAge 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-cipher-nar
## Description
Decrypt content using the age-encryption.org/v1 specification. Detects binary or ASCII armored content encoding using the initial file header bytes. The age standard uses ChaCha20-Poly1305 for authenticated encryption of the payload. The age-keygen command supports generating X25519 key pairs for encryption and decryption operations.
## Tags
ChaCha20-Poly1305, X25519, age, age-encryption.org, encryption
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Private Key Identities |
One or more X25519 Private Key Identities, separated with newlines, encoded according to the age specification, starting with AGE-SECRET-KEY-1 |
| Private Key Identity Resources |
One or more files or URLs containing X25519 Private Key Identities, separated with newlines, encoded according to the age specification, starting with AGE-SECRET-KEY-1 |
| Private Key Source |
Source of information determines the loading strategy for X25519 Private Key Identities |
## Relationships
| Name |
Description |
| failure |
Decryption Failed |
| success |
Decryption Completed |
## See also
- [org.apache.nifi.processors.cipher.EncryptContentAge](/user-guide/data-integration/openflow/processors/encryptcontentage)
---
title: DecryptContentPGP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/decryptcontentpgp.md
section: Loading & Unloading Data
---
# DecryptContentPGP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-pgp-nar
## Description
Decrypt contents of OpenPGP messages. Using the Packaged Decryption Strategy preserves OpenPGP encoding to support subsequent signature verification.
## Tags
Encryption, GPG, OpenPGP, PGP, RFC 4880
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| decryption-strategy |
Strategy for writing files to success after decryption |
| passphrase |
Passphrase used for decrypting data encrypted with Password-Based Encryption |
| private-key-service |
PGP Private Key Service for decrypting data encrypted with Public Key Encryption |
## Relationships
| Name |
Description |
| failure |
Decryption Failed |
| success |
Decryption Succeeded |
## Writes attributes
| Name |
Description |
| pgp.literal.data.filename |
Filename from decrypted Literal Data |
| pgp.literal.data.modified |
Modified Date from decrypted Literal Data |
| pgp.symmetric.key.algorithm.block.cipher |
Symmetric-Key Algorithm Block Cipher |
| pgp.symmetric.key.algorithm.id |
Symmetric-Key Algorithm Identifier |
## See also
- [org.apache.nifi.processors.pgp.EncryptContentPGP](/user-guide/data-integration/openflow/processors/encryptcontentpgp)
- [org.apache.nifi.processors.pgp.SignContentPGP](/user-guide/data-integration/openflow/processors/signcontentpgp)
- [org.apache.nifi.processors.pgp.VerifyContentPGP](/user-guide/data-integration/openflow/processors/verifycontentpgp)
---
title: DeduplicateRecord 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deduplicaterecord.md
section: Loading & Unloading Data
---
# DeduplicateRecord 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
This processor de-duplicates individual records within a record set. It can operate on a per-file basis using an in-memory hashset or bloom filter. When configured with a distributed map cache, it de-duplicates records across multiple files.
## Tags
change, dedupe, distinct, dupe, duplicate, filter, hash, modify, record, replace, text, unique, update
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| bloom-filter-certainty |
The desired false positive probability when using the BloomFilter type. Using a value of .05 for example, guarantees a five-percent probability that the result is a false positive. The closer to 1 this value is set, the more precise the result at the expense of more storage space utilization. |
| cache-identifier |
An optional expression language field that overrides the record's computed cache key. This field has an additional attribute available: $\{record.hash.value\}, which contains the cache key derived from dynamic properties (if set) or record fields. |
| deduplication-strategy |
The strategy to use for detecting and routing duplicate records. The option for detecting duplicates across a single FlowFile operates in-memory, whereas detection spanning multiple FlowFiles utilises a distributed map cache. |
| distributed-map-cache |
This property is required when the deduplication strategy is set to 'multiple files.' The map cache will for each record, atomically check whether the cache key exists and if not, set it. |
| filter-capacity-hint |
An estimation of the total number of unique records to be processed. The more accurate this number is will lead to fewer false negatives on a BloomFilter. |
| filter-type |
The filter used to determine whether a record has been seen before based on the matching RecordPath criteria. If hash set is selected, a Java HashSet object will be used to deduplicate all encountered records. If the bloom filter option is selected, a bloom filter will be used. The bloom filter option is less memory intensive, but has a chance of having false positives. |
| include-zero-record-flowfiles |
If a FlowFile sent to either the duplicate or non-duplicate relationships contains no records, a value of _false_ in this property causes the FlowFile to be dropped. Otherwise, the empty FlowFile is emitted. |
| put-cache-identifier |
For each record, check whether the cache identifier exists in the distributed map cache. If it doesn't exist and this property is true, put the identifier to the cache. |
| record-hashing-algorithm |
The algorithm used to hash the cache key. |
| record-reader |
Specifies the Controller Service to use for reading incoming data |
| record-writer |
Specifies the Controller Service to use for writing out the records |
## Relationships
| Name |
Description |
| duplicate |
Records detected as duplicates are routed to this relationship. |
| failure |
If unable to communicate with the cache, the FlowFile will be penalized and routed to this relationship |
| non-duplicate |
Records not found in the cache are routed to this relationship. |
| original |
The original input FlowFile is sent to this relationship unless a fatal error occurs. |
## Writes attributes
| Name |
Description |
| record.count |
Number of records written to the destination FlowFile. |
## See also
- [org.apache.nifi.processors.standard.DetectDuplicate](/user-guide/data-integration/openflow/processors/detectduplicate)
---
title: DeleteAzureBlobStorage_v12 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deleteazureblobstorage_v12.md
section: Loading & Unloading Data
---
# DeleteAzureBlobStorage_v12 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-azure-nar
## Description
Deletes the specified blob from Azure Blob Storage. The processor uses Azure Blob Storage client library v12.
## Tags
azure, blob, cloud, microsoft, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Blob Name |
The full name of the blob |
| Container Name |
Name of the Azure storage container. In case of PutAzureBlobStorage processor, container can be created if it does not exist. |
| Delete Snapshots Option |
Specifies the snapshot deletion options to be used when deleting a blob. |
| Storage Credentials |
Controller Service used to obtain Azure Blob Storage Credentials. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. In case of SOCKS, it is not guaranteed that the selected SOCKS Version will be used by the processor. |
## Relationships
| Name |
Description |
| failure |
Unsuccessful operations will be transferred to the failure relationship. |
| success |
All successfully processed FlowFiles are routed to this relationship |
## See also
- [org.apache.nifi.processors.azure.storage.CopyAzureBlobStorage_v12](/user-guide/data-integration/openflow/processors/copyazureblobstorage_v12)
- [org.apache.nifi.processors.azure.storage.FetchAzureBlobStorage_v12](/user-guide/data-integration/openflow/processors/fetchazureblobstorage_v12)
- [org.apache.nifi.processors.azure.storage.ListAzureBlobStorage_v12](/user-guide/data-integration/openflow/processors/listazureblobstorage_v12)
- [org.apache.nifi.processors.azure.storage.PutAzureBlobStorage_v12](/user-guide/data-integration/openflow/processors/putazureblobstorage_v12)
---
title: DeleteAzureDataLakeStorage 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deleteazuredatalakestorage.md
section: Loading & Unloading Data
---
# DeleteAzureDataLakeStorage 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-azure-nar
## Description
Deletes the provided file from Azure Data Lake Storage
## Tags
adlsgen2, azure, cloud, datalake, microsoft, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| ADLS Credentials |
Controller Service used to obtain Azure Credentials. |
| Directory Name |
Name of the Azure Storage Directory. The Directory Name cannot contain a leading '/'. The root directory can be designated by the empty string value. In case of the PutAzureDataLakeStorage processor, the directory will be created if not already existing. |
| File Name |
The filename |
| Filesystem Name |
Name of the Azure Storage File System (also called Container). It is assumed to be already existing. |
| Filesystem Object Type |
They type of the file system object to be deleted. It can be either folder or file. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. In case of SOCKS, it is not guaranteed that the selected SOCKS Version will be used by the processor. |
## Relationships
| Name |
Description |
| failure |
Files that could not be written to Azure storage for some reason are transferred to this relationship |
| success |
Files that have been successfully written to Azure storage are transferred to this relationship |
## See also
- [org.apache.nifi.processors.azure.storage.FetchAzureDataLakeStorage](/user-guide/data-integration/openflow/processors/fetchazuredatalakestorage)
- [org.apache.nifi.processors.azure.storage.ListAzureDataLakeStorage](/user-guide/data-integration/openflow/processors/listazuredatalakestorage)
- [org.apache.nifi.processors.azure.storage.PutAzureDataLakeStorage](/user-guide/data-integration/openflow/processors/putazuredatalakestorage)
---
title: DeleteBoxFileMetadataInstance 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deleteboxfilemetadatainstance.md
section: Loading & Unloading Data
---
# DeleteBoxFileMetadataInstance 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-box-nar
## Description
Deletes a metadata instance from a Box file using the specified template key
## Tags
box, delete, metadata, storage, templates
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Box Client Service |
Controller Service used to obtain a Box API connection. |
| File ID |
The ID of the file from which to delete metadata. |
| Template Key |
The key of the metadata template instance to delete. |
## Relationships
| Name |
Description |
| failure |
A FlowFile is routed to this relationship if an error occurs during metadata deletion. |
| file not found |
FlowFiles for which the specified Box file was not found will be routed to this relationship. |
| success |
A FlowFile is routed to this relationship after metadata has been successfully deleted. |
| template not found |
FlowFiles for which the specified metadata template was not found will be routed to this relationship. |
## Writes attributes
| Name |
Description |
| box.id |
The ID of the file from which metadata was deleted |
| box.template.key |
The template key used for metadata deletion |
| error.code |
The error code returned by Box |
| error.message |
The error message returned by Box |
## See also
- [org.apache.nifi.processors.box.CreateBoxFileMetadataInstance](/user-guide/data-integration/openflow/processors/createboxfilemetadatainstance)
- [org.apache.nifi.processors.box.FetchBoxFileMetadataInstance](/user-guide/data-integration/openflow/processors/fetchboxfilemetadatainstance)
- [org.apache.nifi.processors.box.UpdateBoxFileMetadataInstance](/user-guide/data-integration/openflow/processors/updateboxfilemetadatainstance)
---
title: DeleteByQueryElasticsearch 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deletebyqueryelasticsearch.md
section: Loading & Unloading Data
---
# DeleteByQueryElasticsearch 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-elasticsearch-restapi-nar
## Description
Delete from an Elasticsearch index using a query. The query can be loaded from a flowfile body or from the Query parameter.
## Tags
delete, elastic, elasticsearch, elasticsearch7, elasticsearch8, elasticsearch9, query
## Input Requirement
ALLOWED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Client Service |
An Elasticsearch client service to use for running queries. |
| Index |
The name of the index to use. |
| Max JSON Field String Length |
The maximum allowed length of a string value when parsing a JSON document or attribute. |
| Query |
A query in JSON syntax, not Lucene syntax. Ex: \{"query":\{"match":\{"somefield":"somevalue"\}\}\}. If this parameter is not set, the query will be read from the flowfile content. If the query (property and flowfile content) is empty, a default empty JSON Object will be used, which will result in a "match_all" query in Elasticsearch. |
| Query Attribute |
If set, the executed query will be set on each result flowfile in the specified attribute. |
| Query Clause |
A "query" clause in JSON syntax, not Lucene syntax. Ex: \{"match":\{"somefield":"somevalue"\}\}. If the query is empty, a default JSON Object will be used, which will result in a "match_all" query in Elasticsearch. |
| Query Definition Style |
How the JSON Query will be defined for use by the processor. |
| Type |
The type of this document (used by Elasticsearch for indexing and searching). |
## Relationships
| Name |
Description |
| failure |
If the "by query" operation fails, and a flowfile was read, it will be sent to this relationship. |
| retry |
All flowfiles that fail due to server/cluster availability go to this relationship. |
| success |
If the "by query" operation succeeds, and a flowfile was read, it will be sent to this relationship. |
## Writes attributes
| Name |
Description |
| elasticsearch.delete.took |
The amount of time that it took to complete the delete operation in ms. |
| elasticsearch.delete.error |
The error message provided by Elasticsearch if there is an error running the delete. |
---
title: DeleteDBFSResource 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deletedbfsresource.md
section: Loading & Unloading Data
---
# DeleteDBFSResource 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-databricks-processors-nar
## Description
Delete a DBFS files and directories.
## Tags
databricks, dbfs, openflow
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| DBFS File Path |
DBFS file path e.g. /directory/file.txt |
| Databricks Client |
Databricks Client Service. |
## Relationships
| Name |
Description |
| failure |
Databricks failure relationship |
| success |
Databricks success relationship |
## Writes attributes
| Name |
Description |
| error.code |
The error code for the SQL statement if an error occurred. |
| error.message |
The error message for the SQL statement if an error occurred. |
---
title: DeleteDynamoDB 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deletedynamodb.md
section: Loading & Unloading Data
---
# DeleteDynamoDB 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-aws-nar
## Description
Deletes a document from DynamoDB based on hash and range key. The key can be string or number. The request requires all the primary keys for the operation (hash or hash and range key)
## Tags
AWS, Amazon, Delete, DynamoDB, Remove
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| AWS Credentials Provider service |
The Controller Service that is used to obtain AWS credentials provider |
| Batch items for each request (between 1 and 50) |
The items to be retrieved in one batch |
| Communications Timeout |
|
| Endpoint Override URL |
Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. |
| Hash Key Name |
The hash key name of the item |
| Hash Key Value |
The hash key value of the item |
| Hash Key Value Type |
The hash key value type of the item |
| Range Key Name |
The range key name of the item |
| Range Key Value |
|
| Range Key Value Type |
The range key value type of the item |
| Region |
|
| SSL Context Service |
Specifies an optional SSL Context Service that, if provided, will be used to create connections |
| Table Name |
The DynamoDB table name |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| failure |
FlowFiles are routed to failure relationship |
| success |
FlowFiles are routed to success relationship |
| unprocessed |
FlowFiles are routed to unprocessed relationship when DynamoDB is not able to process all the items in the request. Typical reasons are insufficient table throughput capacity and exceeding the maximum bytes per request. Unprocessed FlowFiles can be retried with a new request. |
## Writes attributes
| Name |
Description |
| dynamodb.key.error.unprocessed |
DynamoDB unprocessed keys |
| dynmodb.range.key.value.error |
DynamoDB range key error |
| dynamodb.key.error.not.found |
DynamoDB key not found |
| dynamodb.error.exception.message |
DynamoDB exception message |
| dynamodb.error.code |
DynamoDB error code |
| dynamodb.error.message |
DynamoDB error message |
| dynamodb.error.service |
DynamoDB error service |
| dynamodb.error.retryable |
DynamoDB error is retryable |
| dynamodb.error.request.id |
DynamoDB error request id |
| dynamodb.error.status.code |
DynamoDB status code |
## See also
- [org.apache.nifi.processors.aws.dynamodb.GetDynamoDB](/user-guide/data-integration/openflow/processors/getdynamodb)
- [org.apache.nifi.processors.aws.dynamodb.PutDynamoDB](/user-guide/data-integration/openflow/processors/putdynamodb)
- [org.apache.nifi.processors.aws.dynamodb.PutDynamoDBRecord](/user-guide/data-integration/openflow/processors/putdynamodbrecord)
---
title: DeleteFile 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deletefile.md
section: Loading & Unloading Data
---
# DeleteFile 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Deletes a file from the filesystem.
## Tags
delete, file, files, filesystem, local, remove
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Directory Path |
The path to the directory the file to delete is located in. |
| Filename |
The name of the file to delete. |
## Restrictions
| Required Permission |
Explanation |
| read filesystem |
Provides operator the ability to read from any file that NiFi has access to. |
| write filesystem |
Provides operator the ability to delete any file that NiFi has access to. |
## Relationships
| Name |
Description |
| failure |
All FlowFiles, for which an existing file could not be deleted, are routed to this relationship |
| not found |
All FlowFiles, for which the file to delete did not exist, are routed to this relationship |
| success |
All FlowFiles, for which an existing file has been deleted, are routed to this relationship |
## Use cases
| Delete source file only after its processing completed |
| ------------------------------------------------------ |
---
title: DeleteGCSObject 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deletegcsobject.md
section: Loading & Unloading Data
---
# DeleteGCSObject 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-gcp-nar
## Description
Deletes objects from a Google Cloud Bucket. If attempting to delete a file that does not exist, FlowFile is routed to success.
## Tags
delete, gcs, google, google cloud, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| GCP Credentials Provider Service |
The Controller Service used to obtain Google Cloud Platform credentials. |
| gcp-project-id |
Google Cloud Project ID |
| gcp-retry-count |
How many retry attempts should be made before routing to the failure relationship. |
| gcs-bucket |
Bucket of the object. |
| gcs-generation |
The generation of the object to be deleted. If null, will use latest version of the object. |
| gcs-key |
Name of the object. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
| storage-api-url |
Overrides the default storage URL. Configuring an alternative Storage API URL also overrides the HTTP Host header on requests as described in the Google documentation for Private Service Connections. |
## Relationships
| Name |
Description |
| failure |
FlowFiles are routed to this relationship if the Google Cloud Storage operation fails. |
| success |
FlowFiles are routed to this relationship after a successful Google Cloud Storage operation. |
## See also
- [org.apache.nifi.processors.gcp.storage.FetchGCSObject](/user-guide/data-integration/openflow/processors/fetchgcsobject)
- [org.apache.nifi.processors.gcp.storage.ListGCSBucket](/user-guide/data-integration/openflow/processors/listgcsbucket)
- [org.apache.nifi.processors.gcp.storage.PutGCSObject](/user-guide/data-integration/openflow/processors/putgcsobject)
---
title: DeleteGridFS 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deletegridfs.md
section: Loading & Unloading Data
---
# DeleteGridFS 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-mongodb-nar
## Description
Deletes a file from GridFS using a file name or a query.
## Tags
delete, gridfs, mongodb
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| delete-gridfs-query |
A valid MongoDB query to use to find and delete one or more files from GridFS. |
| gridfs-bucket-name |
The GridFS bucket where the files will be stored. If left blank, it will use the default value 'fs' that the MongoDB client driver uses. |
| gridfs-client-service |
The MongoDB client service to use for database connections. |
| gridfs-database-name |
The name of the database to use |
| gridfs-file-name |
The name of the file in the bucket that is the target of this processor. GridFS file names do not include path information because GridFS does not sort files into folders within a bucket. |
| mongo-query-attribute |
If set, the query will be written to a specified attribute on the output flowfiles. |
## Relationships
| Name |
Description |
| failure |
When there is a failure processing the flowfile, it goes to this relationship. |
| success |
When the operation succeeds, the flowfile is sent to this relationship. |
---
title: DeleteMilvus 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deletemilvus.md
section: Loading & Unloading Data
---
# DeleteMilvus 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-milvus-processors-nar
## Description
Deletes vectors from Milvus database from a collection by ID. Unmatched IDs are ignored by Milvus and not deleted.
## Tags
chatbot, delete, embeddings, gen ai, genai, generative ai, llm, metadata, milvus, openflow, text, vector
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Collection Name |
The name of the Milvus collection name to use |
| Delete Filter |
The filter to use in the delete request. Example: id like "prefix%" |
| Delete Strategy |
The strategy to use for deleting vectors in Milvus |
| ID Record Path |
The path to the ID field in the record |
| Milvus Connection Service |
Connection Service for accessing Milvus Database |
| Partition |
Partition of the vector database that you want to perform operations in. If the database has only one partition leave empty. |
| Record Reader |
The Record Reader to use for reading the FlowFile |
## Relationships
| Name |
Description |
| failure |
FlowFiles that cannot be sent to Milvus, and for which a retry is not expected to be successful, are routed to this relationship |
| retry |
FlowFiles that fail to be sent to Milvus, but for which a retry may help, are routed to this relationship |
| success |
FlowFiles that are successfully sent to Milvus are routed to this relationship |
## See also
- [com.snowflake.openflow.runtime.processors.milvus.UpsertMilvus](/user-guide/data-integration/openflow/processors/upsertmilvus)
---
title: DeleteMongo 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deletemongo.md
section: Loading & Unloading Data
---
# DeleteMongo 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-mongodb-nar
## Description
Executes a delete query against a MongoDB collection. The query is provided in the body of the flowfile and the user can select whether it will delete one or many documents that match it.
## Tags
delete, mongo, mongodb
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Mongo Collection Name |
The name of the collection to use |
| Mongo Database Name |
The name of the database to use |
| delete-mongo-delete-mode |
Choose between deleting one document by query or many documents by query. |
| delete-mongo-fail-on-no-delete |
Determines whether to send the flowfile to the success or failure relationship if nothing is successfully deleted. |
| mongo-client-service |
If configured, this property will use the assigned client service for connection pooling. |
## Relationships
| Name |
Description |
| failure |
All FlowFiles that cannot be written to MongoDB are routed to this relationship |
| success |
All FlowFiles that are written to MongoDB are routed to this relationship |
---
title: DeletePinecone 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deletepinecone.md
section: Loading & Unloading Data
---
# DeletePinecone 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-pinecone-nar
## Description
Deletes vectors from a Pinecone index.
## Tags
delete, embeddings, genai, generative ai, openflow, pinecone, rag, retrieval augmented generation, vector store
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| ID Prefix |
The Pinecone vector ID prefix. If specified, only the vectors whose IDs start with the given value will be deleted. |
| Pinecone API Key |
The API key for the Pinecone service |
| Pinecone Index |
The name of the Pinecone index to use |
| Pinecone Namespace |
The name of the Pinecone namespace to use |
| Web Client Service |
The Web Client Service to use for communicating with Pinecone |
## Relationships
| Name |
Description |
| failure |
FlowFiles that cannot be sent to Pinecone, and for which a retry is not expected to be successful, are routed to this relationship |
| retry |
FlowFiles that fail to be sent to Pinecone, but for which a retry may help, are routed to this relationship |
| success |
FlowFiles that are successfully sent to Pinecone are routed to this relationship |
## Use cases
| Delete all vectors from a Pinecone index. |
| ------------------------------------------------------------------------- |
| Delete a namespace, along with all of its vectors, from a Pinecone index. |
| Delete all vectors for a particular document from a Pinecone index. |
---
title: DeleteQueryJob 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deletequeryjob.md
section: Loading & Unloading Data
---
# DeleteQueryJob 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-salesforce-processors-nar
## Description
Deletes a Query Job in Salesforce using the Bulk API 2.0.
## Tags
bulk, delete, job, preview, query, salesforce
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Job ID |
The ID of the job for which the status is checked. |
| Salesforce Client |
Salesforce Client to interact with the APIs |
## Relationships
| Name |
Description |
| comms.failure |
A FlowFile is routed to this relationship if the Query Job status could not be retrieved but the operation might be retried |
| failure |
A FlowFile is routed to this relationship if the Query Job status could not be retrieved |
| success |
If the Query Job has been successfully deleted, the FlowFile is routed to this relationship |
## See also
- [com.snowflake.openflow.runtime.processors.salesforce.AbortQueryJob](/user-guide/data-integration/openflow/processors/abortqueryjob)
- [com.snowflake.openflow.runtime.processors.salesforce.GetQueryJobResult](/user-guide/data-integration/openflow/processors/getqueryjobresult)
- [com.snowflake.openflow.runtime.processors.salesforce.GetQueryJobStatus](/user-guide/data-integration/openflow/processors/getqueryjobstatus)
- [com.snowflake.openflow.runtime.processors.salesforce.SubmitQueryJob](/user-guide/data-integration/openflow/processors/submitqueryjob)
---
title: DeleteS3Object 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deletes3object.md
section: Loading & Unloading Data
---
# DeleteS3Object 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-aws-nar
## Description
Deletes a file from an Amazon S3 Bucket. If attempting to delete a file that does not exist, FlowFile is routed to success.
## Tags
AWS, Amazon, Archive, Delete, S3
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| AWS Credentials Provider service |
The Controller Service that is used to obtain AWS credentials provider |
| Bucket |
The S3 Bucket to interact with |
| Communications Timeout |
The amount of time to wait in order to establish a connection to AWS or receive data from AWS before timing out. |
| Custom Signer Class Name |
Fully qualified class name of the custom signer class. The signer must implement com.amazonaws.auth. Signer interface. |
| Custom Signer Module Location |
Comma-separated list of paths to files and/or directories which contain the custom signer's JAR file and its dependencies (if any). |
| Endpoint Override URL |
Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. |
| FullControl User List |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Full Control for an object |
| Object Key |
The S3 Object Key to use. This is analogous to a filename for traditional file systems. |
| Owner |
The Amazon ID to use for the object's owner |
| Read ACL User List |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have permissions to read the Access Control List for an object |
| Read Permission User List |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Read Access for an object |
| Region |
The AWS Region to connect to. |
| SSL Context Service |
Specifies an optional SSL Context Service that, if provided, will be used to create connections |
| Signer Override |
The AWS S3 library uses Signature Version 4 by default but this property allows you to specify the Version 2 signer to support older S3-compatible services or even to plug in your own custom signer implementation. |
| Version |
The Version of the Object to delete |
| Write ACL User List |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have permissions to change the Access Control List for an object |
| Write Permission User List |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Write Access for an object |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| failure |
If the Processor is unable to process a given FlowFile, it will be routed to this Relationship. |
| success |
FlowFiles are routed to this Relationship after they have been successfully processed. |
## Writes attributes
| Name |
Description |
| s3.exception |
The class name of the exception thrown during processor execution |
| s3.additionalDetails |
The S3 supplied detail from the failed operation |
| s3.statusCode |
The HTTP error code (if available) from the failed operation |
| s3.errorCode |
The S3 moniker of the failed operation |
| s3.errorMessage |
The S3 exception message from the failed operation |
## See also
- [org.apache.nifi.processors.aws.s3.CopyS3Object](/user-guide/data-integration/openflow/processors/copys3object)
- [org.apache.nifi.processors.aws.s3.FetchS3Object](/user-guide/data-integration/openflow/processors/fetchs3object)
- [org.apache.nifi.processors.aws.s3.GetS3ObjectMetadata](/user-guide/data-integration/openflow/processors/gets3objectmetadata)
- [org.apache.nifi.processors.aws.s3.GetS3ObjectTags](/user-guide/data-integration/openflow/processors/gets3objecttags)
- [org.apache.nifi.processors.aws.s3.ListS3](/user-guide/data-integration/openflow/processors/lists3)
- [org.apache.nifi.processors.aws.s3.PutS3Object](/user-guide/data-integration/openflow/processors/puts3object)
- [org.apache.nifi.processors.aws.s3.TagS3Object](/user-guide/data-integration/openflow/processors/tags3object)
---
title: DeleteSFTP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deletesftp.md
section: Loading & Unloading Data
---
# DeleteSFTP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Deletes a file residing on an SFTP server.
## Tags
delete, remote, remove, sftp
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Algorithm Negotiation |
Configuration strategy for SSH algorithm negotiation |
| Batch Size |
The maximum number of FlowFiles to send in a single connection |
| Ciphers Allowed |
A comma-separated list of Ciphers allowed for SFTP connections. Leave unset to allow all. Available options are: 3des-cbc, aes128-cbc, aes128-ctr, [aes128-gcm@openssh.com](mailto:aes128-gcm@openssh.com), aes192-cbc, aes192-ctr, aes256-cbc, aes256-ctr, [aes256-gcm@openssh.com](mailto:aes256-gcm@openssh.com), arcfour128, arcfour256, blowfish-cbc, [chacha20-poly1305@openssh.com](mailto:chacha20-poly1305@openssh.com), none |
| Connection Timeout |
Amount of time to wait before timing out while creating a connection |
| Data Timeout |
When transferring a file between the local and remote system, this value specifies how long is allowed to elapse without any data being transferred between systems |
| Directory Path |
The path to the directory the file to delete is located in. |
| Filename |
The name of the file to delete. |
| Host Key File |
If supplied, the given file will be used as the Host Key; otherwise, if 'Strict Host Key Checking' property is applied (set to true) then uses the 'known_hosts' and 'known_hosts2' files from ~/.ssh directory else no host key file will be used |
| Hostname |
The fully qualified hostname or IP address of the remote system |
| Key Algorithms Allowed |
A comma-separated list of Key Algorithms allowed for SFTP connections. Leave unset to allow all. Available options are: ecdsa-sha2-nistp256, [ecdsa-sha2-nistp256-cert-v01@openssh.com](mailto:ecdsa-sha2-nistp256-cert-v01@openssh.com), ecdsa-sha2-nistp384, [ecdsa-sha2-nistp384-cert-v01@openssh.com](mailto:ecdsa-sha2-nistp384-cert-v01@openssh.com), ecdsa-sha2-nistp521, [ecdsa-sha2-nistp521-cert-v01@openssh.com](mailto:ecdsa-sha2-nistp521-cert-v01@openssh.com), rsa-sha2-256, [rsa-sha2-256-cert-v01@openssh.com](mailto:rsa-sha2-256-cert-v01@openssh.com), rsa-sha2-512, [rsa-sha2-512-cert-v01@openssh.com](mailto:rsa-sha2-512-cert-v01@openssh.com), [sk-ecdsa-sha2-nistp256@openssh.com](mailto:sk-ecdsa-sha2-nistp256@openssh.com), [sk-ssh-ed25519@openssh.com](mailto:sk-ssh-ed25519@openssh.com), ssh-dss, [ssh-dss-cert-v01@openssh.com](mailto:ssh-dss-cert-v01@openssh.com), ssh-ed25519, [ssh-ed25519-cert-v01@openssh.com](mailto:ssh-ed25519-cert-v01@openssh.com), ssh-rsa, [ssh-rsa-cert-v01@openssh.com](mailto:ssh-rsa-cert-v01@openssh.com) |
| Key Exchange Algorithms Allowed |
A comma-separated list of Key Exchange Algorithms allowed for SFTP connections. Leave unset to allow all. Available options are: curve25519-sha256, [curve25519-sha256@libssh.org](mailto:curve25519-sha256@libssh.org), curve448-sha512, diffie-hellman-group-exchange-sha1, diffie-hellman-group-exchange-sha256, diffie-hellman-group1-sha1, diffie-hellman-group14-sha1, diffie-hellman-group14-sha256, diffie-hellman-group15-sha512, diffie-hellman-group16-sha512, diffie-hellman-group17-sha512, diffie-hellman-group18-sha512, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, mlkem1024nistp384-sha384, mlkem768nistp256-sha256, mlkem768x25519-sha256, sntrup761x25519-sha512, [sntrup761x25519-sha512@openssh.com](mailto:sntrup761x25519-sha512@openssh.com) |
| Message Authentication Codes Allowed |
A comma-separated list of Message Authentication Codes allowed for SFTP connections. Leave unset to allow all. Available options are: hmac-md5, hmac-md5-96, hmac-sha1, hmac-sha1-96, [hmac-sha1-etm@openssh.com](mailto:hmac-sha1-etm@openssh.com), hmac-sha2-256, [hmac-sha2-256-etm@openssh.com](mailto:hmac-sha2-256-etm@openssh.com), hmac-sha2-512, [hmac-sha2-512-etm@openssh.com](mailto:hmac-sha2-512-etm@openssh.com) |
| Password |
Password for the user account |
| Port |
The port that the remote system is listening on for file transfers |
| Private Key Passphrase |
Password for the private key |
| Private Key Path |
The fully qualified path to the Private Key file |
| Send Keep Alive On Timeout |
Send a Keep Alive message every 5 seconds up to 5 times for an overall timeout of 25 seconds. |
| Strict Host Key Checking |
Indicates whether or not strict enforcement of hosts keys should be applied |
| Use Compression |
Indicates whether or not ZLIB compression should be used when transferring files |
| Username |
Username |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| failure |
All FlowFiles, for which an existing file could not be deleted, are routed to this relationship |
| not found |
All FlowFiles, for which the file to delete did not exist, are routed to this relationship |
| success |
All FlowFiles, for which an existing file has been deleted, are routed to this relationship |
## Use cases
| Delete source file only after its processing completed |
| ------------------------------------------------------ |
---
title: DeleteSQS 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deletesqs.md
section: Loading & Unloading Data
---
# DeleteSQS 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-aws-nar
## Description
Deletes a message from an Amazon Simple Queuing Service Queue
## Tags
AWS, Amazon, Delete, Queue, SQS
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| AWS Credentials Provider service |
The Controller Service that is used to obtain AWS credentials provider |
| Communications Timeout |
|
| Endpoint Override URL |
Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. |
| Queue URL |
The URL of the queue delete from |
| Receipt Handle |
The identifier that specifies the receipt of the message |
| Region |
|
| SSL Context Service |
Specifies an optional SSL Context Service that, if provided, will be used to create connections |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| failure |
FlowFiles are routed to failure relationship |
| success |
FlowFiles are routed to success relationship |
## See also
- [org.apache.nifi.processors.aws.sqs.GetSQS](/user-guide/data-integration/openflow/processors/getsqs)
- [org.apache.nifi.processors.aws.sqs.PutSQS](/user-guide/data-integration/openflow/processors/putsqs)
---
title: DeleteUnityCatalogResource 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/deleteunitycatalogresource.md
section: Loading & Unloading Data
---
# DeleteUnityCatalogResource 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-databricks-processors-nar
## Description
Delete a Unity Catalog file or directory.
## Tags
databricks, openflow, unity catalog
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Databricks Client |
Databricks Client Service. |
| Missing Resource Policy |
What to action to take if the resource is not found. |
| Unity Catalog Resource Path |
Unity Catalog resource path e.g. /Volumes/catalog/schema/volume_name/path |
## Relationships
| Name |
Description |
| failure |
Databricks failure relationship |
| success |
Databricks success relationship |
## Writes attributes
| Name |
Description |
| error.code |
The error code for the SQL statement if an error occurred. |
| error.message |
The error message for the SQL statement if an error occurred. |
---
title: DescribeDataShare 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/describedatashare.md
section: Loading & Unloading Data
---
# DescribeDataShare 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-salesforce-processors-nar
## Description
Describe the specified data share metadata in Salesforce Data Cloud.
## Tags
daas, data cloud, describe, object, preview, salesforce, sfdc
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Data Share Name |
The name of the Data Share to describe. |
| Salesforce Data Cloud Client |
Salesforce Data Cloud Client to interact with the APIs |
## Relationships
| Name |
Description |
| comms.failure |
A FlowFile is routed to this relationship if the data share metadata could not be retrieved but the operation might be retried |
| failure |
A FlowFile is routed to this relationship if the data share metadata could not be retrieved |
| success |
FlowFile containing the data share metadata will be routed to this relationship |
## Writes attributes
| Name |
Description |
| explicitDataLakeObjects |
Comma-separated list of the names of the explicit data lake objects. |
| implicitDataLakeObjects |
Comma-separated list of the names of the implicit data lake objects. |
| dataModelObjects |
Comma-separated list of the names of the data model objects. |
| calculatedInsightObjects |
Comma-separated list of the names of the calculated insights objects. |
## See also
- [com.snowflake.openflow.runtime.processors.salesforce.ListSFDCDataShares](/user-guide/data-integration/openflow/processors/listsfdcdatashares)
---
title: DescribeSFDCObject 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/describesfdcobject.md
section: Loading & Unloading Data
---
# DescribeSFDCObject 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-salesforce-processors-nar
## Description
Describe the specified object metadata in Salesforce.
## Tags
describe, object, preview, salesforce, sfdc
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Object Fields Filter JSON |
JSON representation describing which fields to include or exclude for Salesforce objects. |
| Object Name |
The name of the object to describe. |
| Salesforce Client |
Salesforce Client to interact with the APIs |
## Relationships
| Name |
Description |
| comms.failure |
A FlowFile is routed to this relationship if the object metadata could not be retrieved but the operation might be retried |
| failure |
A FlowFile is routed to this relationship if the object metadata could not be retrieved |
| success |
FlowFile containing the object metadata will be routed to this relationship |
## Writes attributes
| Name |
Description |
| sObjectFields |
Comma-separated list of the fields of the object (without non-queryable fields). |
| sObjectExcludedFields |
Comma-separated list of the non-queryable fields of the object. |
| sObjectSchema |
The schema associated to the object based on its fields (without non-queryable fields). |
## See also
- [com.snowflake.openflow.runtime.processors.salesforce.AbortQueryJob](/user-guide/data-integration/openflow/processors/abortqueryjob)
- [com.snowflake.openflow.runtime.processors.salesforce.DeleteQueryJob](/user-guide/data-integration/openflow/processors/deletequeryjob)
- [com.snowflake.openflow.runtime.processors.salesforce.GetQueryJobResult](/user-guide/data-integration/openflow/processors/getqueryjobresult)
- [com.snowflake.openflow.runtime.processors.salesforce.ListSFDCObjects](/user-guide/data-integration/openflow/processors/listsfdcobjects)
---
title: DetectDuplicate 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/detectduplicate.md
section: Loading & Unloading Data
---
# DetectDuplicate 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Caches a value, computed from FlowFile attributes, for each incoming FlowFile and determines if the cached value has already been seen. If so, routes the FlowFile to 'duplicate' with an attribute named 'original.identifier' that specifies the original FlowFile 's "description", which is specified in the <FlowFile Description> property. If the FlowFile is not determined to be a duplicate, the Processor routes the FlowFile to' non-duplicate'
## Tags
dedupe, dupe, duplicate, hash
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Age Off Duration |
Time interval to age off cached FlowFiles |
| Cache Entry Identifier |
A FlowFile attribute, or the results of an Attribute Expression Language statement, which will be evaluated against a FlowFile in order to determine the value used to identify duplicates; it is this value that is cached |
| Cache The Entry Identifier |
When true this cause the processor to check for duplicates and cache the Entry Identifier. When false, the processor would only check for duplicates and not cache the Entry Identifier, requiring another processor to add identifiers to the distributed cache. |
| Distributed Cache Service |
The Controller Service that is used to cache unique identifiers, used to determine duplicates |
| FlowFile Description |
When a FlowFile is added to the cache, this value is stored along with it so that if a duplicate is found, this description of the original FlowFile will be added to the duplicate's "original.flowfile.description" attribute |
## Relationships
| Name |
Description |
| duplicate |
If a FlowFile has been detected to be a duplicate, it will be routed to this relationship |
| failure |
If unable to communicate with the cache, the FlowFile will be penalized and routed to this relationship |
| non-duplicate |
If a FlowFile's Cache Entry Identifier was not found in the cache, it will be routed to this relationship |
## Writes attributes
| Name |
Description |
| original.flowfile.description |
All FlowFiles routed to the duplicate relationship will have an attribute added named original.flowfile.description. The value of this attribute is determined by the attributes of the original copy of the data and by the FlowFile Description property. |
## See also
---
title: DeveloperBoxClientService
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/developerboxclientservice.md
section: Loading & Unloading Data
---
# DeveloperBoxClientService
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides Box client objects through which Box API calls can be used. This using a developer token and is for testing only.
## Tags
box, client, provider
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Developer Token * |
Developer Token |
|
|
The Developer Token to use to interact with the Box API. This is for testing only and should not be used in production. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: DistributedMapCacheLookupService
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/distributedmapcachelookupservice.md
section: Loading & Unloading Data
---
# DistributedMapCacheLookupService
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Lets you choose a distributed map cache client to retrieve the value associated to a key. The coordinates that are passed to the lookup must contain the key 'key'.
## Tags
cache, distributed, enrich, key, lookup, map, value
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Character Encoding * |
character-encoding |
UTF-8 |
- ISO-8859-1
- UTF-8
- UTF-16
- UTF-16LE
- UTF-16BE
- US-ASCII
|
Specifies a character encoding to use. |
| Distributed Cache Service * |
distributed-map-cache-service |
|
|
The Controller Service that is used to get the cached values. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: DistributeLoad 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/distributeload.md
section: Loading & Unloading Data
---
# DistributeLoad 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Distributes FlowFiles to downstream processors based on a Distribution Strategy. If using the Round Robin strategy, the default is to assign each destination a weighting of 1 (evenly distributed). However, optional properties can be added to the change this; adding a property with the name '5' and value '10' means that the relationship with name '5' will be receive 10 FlowFiles in each iteration instead of 1.
## Tags
distribute, load balance, round robin, route, weighted
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Distribution Strategy |
Determines how the load will be distributed. Relationship weight is in numeric order where '1' has the greatest weight. |
| Number of Relationships |
Determines the number of Relationships to which the load should be distributed |
## Relationships
| Name |
Description |
| 1 |
Where to route flowfiles for this relationship index |
## Writes attributes
| Name |
Description |
| distribute.load.relationship |
The name of the specific relationship the FlowFile has been routed through |
---
title: DuplicateFlowFile 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/duplicateflowfile.md
section: Loading & Unloading Data
---
# DuplicateFlowFile 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Intended for load testing, this processor will create the configured number of copies of each incoming FlowFile. The original FlowFile as well as all generated copies are sent to the 'success' relationship. In addition, each FlowFile gets an attribute 'copy.index'set to the copy number, where the original FlowFile gets a value of zero, and all copies receive incremented integer values.
## Tags
duplicate, load, test
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Number of Copies |
Specifies how many copies of each incoming FlowFile will be made |
## Relationships
| Name |
Description |
| success |
The original FlowFile and all copies will be sent to this relationship |
## Writes attributes
| Name |
Description |
| copy.index |
A zero-based incrementing integer value based on which copy the FlowFile is. |
---
title: ElasticSearchClientServiceImpl
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/elasticsearchclientserviceimpl.md
section: Loading & Unloading Data
---
# ElasticSearchClientServiceImpl
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
A controller service for accessing an Elasticsearch client, using the Elasticsearch (low-level) REST Client.
## Tags
client, elasticsearch, elasticsearch6, elasticsearch7, elasticsearch8
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| API Key * |
API Key |
|
|
Encoded API key. |
| API Key ID * |
API Key ID |
|
|
Unique identifier of the API key. |
| Authorization Scheme * |
Authorization Scheme |
BASIC |
- None
- PKI
- Basic
- API Key
- JWT
|
Authorization Scheme used for optional authentication to Elasticsearch. |
| Character Set * |
Character Set |
UTF-8 |
|
The charset to use for interpreting the response from Elasticsearch. |
| Connect timeout * |
Connect timeout |
5000 |
|
Controls the amount of time, in milliseconds, before a timeout occurs when trying to connect. |
| Enable Compression * |
Enable Compression |
false |
- true
- false
|
Whether the REST client should compress requests using gzip content encoding and add the "Accept-Encoding: gzip" header to receive compressed responses |
| HTTP Hosts * |
HTTP Hosts |
|
|
A comma-separated list of HTTP hosts that host Elasticsearch query nodes.The HTTP Hosts should be valid URIs including protocol, domain and port for each entry.For example "https://elasticsearch1:9200 (https://elasticsearch1:9200), https://elasticsearch2:9200 (https://elasticsearch2:9200)".Note that the Host is included in requests as a header (typically including domain and port, e.g. elasticsearch:9200). |
| JWT Shared Secret * |
JWT Shared Secret |
|
|
JWT realm Shared Secret. |
| Node Selector * |
Node Selector |
ANY |
- Any
- Skip Dedicated Masters
|
Selects Elasticsearch nodes that can receive requests. Used to keep requests away from dedicated Elasticsearch master nodes |
| OAuth2 Access Token Provider * |
OAuth2 Access Token Provider |
|
|
The OAuth2 Access Token Provider used to provide JWTs for Bearer Token Authorization with Elasticsearch. |
| Password * |
Password |
|
|
The password to use with XPack security. |
| Path Prefix |
Path Prefix |
|
|
Sets the path's prefix for every request used by the http client. For example, if this is set to "/my/path", then any client request will become "/my/path/" + endpoint. In essence, every request's endpoint is prefixed by this pathPrefix. The path prefix is useful for when Elasticsearch is behind a proxy that provides a base path or a proxy that requires all paths to start with '/'; it is not intended for other purposes and it should not be supplied in other scenarios |
| Read Timeout * |
Read Timeout |
60000 |
|
Controls the amount of time, in milliseconds, before a timeout occurs when waiting for a response. |
| Run As User |
Run As User |
|
|
The username to impersonate within Elasticsearch. |
| SSL Context Service |
SSL Context Service |
|
|
The SSL Context Service used to provide client certificate information for TLS/SSL connections. This service only applies if the Elasticsearch endpoint(s) have been secured with TLS/SSL. |
| Send Meta Header * |
Send Meta Header |
true |
- true
- false
|
Whether to send a "X-Elastic-Client-Meta" header that describes the runtime environment. It contains information that is similar to what could be found in User-Agent. Using a separate header allows applications to use User-Agent for their own needs, e.g. to identify application version or other environment information |
| Sniff Cluster Nodes * |
Sniff Cluster Nodes |
false |
- true
- false
|
Periodically sniff for nodes within the Elasticsearch cluster via the Elasticsearch Node Info API. If Elasticsearch security features are enabled (default to "true" for 8.x+), the Elasticsearch user must have the "monitor" or "manage" cluster privilege to use this API.Note that all HTTP Hosts (and those that may be discovered within the cluster using the Sniffer) must use the same protocol, e.g. http or https, and be contactable using the same client settings. Finally the Elasticsearch "network.publish_host" must match one of the "network.bind_host" list entries see https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html (https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html) for more information |
| Sniff on Failure * |
Sniff on Failure |
false |
- true
- false
|
Enable sniffing on failure, meaning that after each failure the Elasticsearch nodes list gets updated straight away rather than at the following ordinary sniffing round |
| Sniffer Failure Delay * |
Sniffer Failure Delay |
1 min |
|
Delay between an Elasticsearch request failure and updating available Cluster nodes using the Sniffer |
| Sniffer Interval * |
Sniffer Interval |
5 mins |
|
Interval between Cluster sniffer operations |
| Sniffer Request Timeout * |
Sniffer Request Timeout |
1 sec |
|
Cluster sniffer timeout for node info requests |
| Strict Deprecation * |
Strict Deprecation |
false |
- true
- false
|
Whether the REST client should return any response containing at least one warning header as a failure |
| Suppress Null and Empty Values * |
Suppress Null and Empty Values |
always-suppress |
- Never Suppress
- Always Suppress
|
Specifies how the writer should handle null and empty fields (including objects and arrays) |
| Username * |
Username |
|
|
The username to use with XPack security. |
| Proxy Configuration Service |
proxy-configuration-service |
|
|
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: ElasticSearchLookupService
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/elasticsearchlookupservice.md
section: Loading & Unloading Data
---
# ElasticSearchLookupService
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Lookup a record from Elasticsearch Server associated with the specified document ID. The coordinates that are passed to the lookup must contain the key 'id'.
## Tags
elasticsearch, enrich, lookup, record
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Client Service * |
Client Service |
|
|
An ElasticSearch client service to use for running queries. |
| Index * |
Index |
|
|
The name of the index to read from |
| Schema Access Strategy * |
Schema Access Strategy |
infer |
- Use 'Schema Name' Property
- Use 'Schema Text' Property
- Infer from Result
|
Specifies how to obtain the schema that is to be used for interpreting the data. |
| Schema Branch |
Schema Branch |
|
|
Specifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored. |
| Schema Name |
Schema Name |
$\{schema.name\} |
|
Specifies the name of the schema to lookup in the Schema Registry property |
| Schema Registry |
Schema Registry |
|
|
Specifies the Controller Service to use for the Schema Registry |
| Schema Text |
Schema Text |
$\{avro.schema\} |
|
The text of an Avro-formatted Schema |
| Schema Version |
Schema Version |
|
|
Specifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved. |
| Type |
Type |
|
|
The type of this document (used by Elasticsearch for indexing and searching) |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: ElasticSearchStringLookupService
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/elasticsearchstringlookupservice.md
section: Loading & Unloading Data
---
# ElasticSearchStringLookupService
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Lookup a string value from Elasticsearch Server associated with the specified document ID. The coordinates that are passed to the lookup must contain the key 'id'.
## Tags
elasticsearch, enrich, key, lookup, value
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Client Service * |
Client Service |
|
|
An ElasticSearch client service to use for running queries. |
| Index * |
Index |
|
|
The name of the index to read from |
| Type |
Type |
|
|
The type of this document (used by Elasticsearch for indexing and searching) |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: EmailRecordSink
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/emailrecordsink.md
section: Loading & Unloading Data
---
# EmailRecordSink
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides a RecordSinkService that can be used to send records in email using the specified writer for formatting.
## Tags
email, record, send, sink, smtp, write
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| BCC |
bcc |
|
|
The recipients to include in the BCC-Line of the email. Comma separated sequence of addresses following RFC822 syntax. |
| CC |
cc |
|
|
The recipients to include in the CC-Line of the email. Comma separated sequence of addresses following RFC822 syntax. |
| From * |
from |
|
|
Specifies the Email address to use as the sender. Comma separated sequence of addresses following RFC822 syntax. |
| Record Writer * |
record-sink-record-writer |
|
|
Specifies the Controller Service to use for writing out the records. |
| SMTP Auth * |
smtp-auth |
true |
|
Flag indicating whether authentication should be used |
| SMTP Hostname * |
smtp-hostname |
|
|
The hostname of the SMTP Server that is used to send Email Notifications |
| SMTP Password |
smtp-password |
|
|
Password for the SMTP account |
| SMTP Port * |
smtp-port |
25 |
|
The Port used for SMTP communications |
| SMTP SSL * |
smtp-ssl |
false |
|
Flag indicating whether SSL should be enabled |
| SMTP STARTTLS * |
smtp-starttls |
false |
|
Flag indicating whether STARTTLS should be enabled. If the server does not support STARTTLS, the connection continues without the use of TLS |
| SMTP Username |
smtp-username |
|
|
Username for the SMTP account |
| SMTP X-Mailer Header * |
smtp-xmailer-header |
NiFi |
|
X-Mailer used in the header of the outgoing email |
| Subject * |
subject |
Message from NiFi |
|
The email subject |
| To |
to |
|
|
The recipients to include in the To-Line of the email. Comma separated sequence of addresses following RFC822 syntax. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: EmbeddedHazelcastCacheManager
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/embeddedhazelcastcachemanager.md
section: Loading & Unloading Data
---
# EmbeddedHazelcastCacheManager
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
A service that runs embedded Hazelcast and provides cache instances backed by that. The server does not ask for authentication, it is recommended to run it within secured network.
## Tags
cache, hazelcast
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Hazelcast Cluster Name * |
hazelcast-cluster-name |
nifi |
|
Name of the Hazelcast cluster. |
| Hazelcast Clustering Strategy * |
hazelcast-clustering-strategy |
none |
- None
- All Nodes
- Explicit
|
Specifies with what strategy the Hazelcast cluster should be created. |
| Hazelcast Instances |
hazelcast-instances |
|
|
Only used with "Explicit" Clustering Strategy! List of NiFi instance host names which should be part of the Hazelcast cluster. Host names are separated by comma. The port specified in the "Hazelcast Port" property will be used as server port. The list must contain every instance that will be part of the cluster. Other instances will join the Hazelcast cluster as clients. |
| Hazelcast Port * |
hazelcast-port |
5701 |
|
Port for the Hazelcast instance to use. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: EncodeContent 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/encodecontent.md
section: Loading & Unloading Data
---
# EncodeContent 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Encode or decode the contents of a FlowFile using Base64, Base32, or hex encoding schemes
## Tags
base32, base64, decode, encode, hex
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Encoded Line Length |
Each line of encoded data will contain up to the configured number of characters, rounded down to the nearest multiple of 4. |
| Encoding |
Specifies the type of encoding used. |
| Line Output Mode |
Controls the line formatting for encoded content based on selected property values. |
| Mode |
Specifies whether the content should be encoded or decoded. |
## Relationships
| Name |
Description |
| failure |
Any FlowFile that cannot be encoded or decoded will be routed to failure |
| success |
Any FlowFile that is successfully encoded or decoded will be routed to success |
---
title: EncryptContentAge 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/encryptcontentage.md
section: Loading & Unloading Data
---
# EncryptContentAge 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-cipher-nar
## Description
Encrypt content using the age-encryption.org/v1 specification. Supports binary or ASCII armored content encoding using configurable properties. The age standard uses ChaCha20-Poly1305 for authenticated encryption of the payload. The age-keygen command supports generating X25519 key pairs for encryption and decryption operations.
## Tags
ChaCha20-Poly1305, X25519, age, age-encryption.org, encryption
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| File Encoding |
Output encoding for encrypted files. Binary encoding provides optimal processing performance. |
| Public Key Recipient Resources |
One or more files or URLs containing X25519 Public Key Recipients, separated with newlines, encoded according to the age specification, starting with age1 |
| Public Key Recipients |
One or more X25519 Public Key Recipients, separated with newlines, encoded according to the age specification, starting with age1 |
| Public Key Source |
Source of information determines the loading strategy for X25519 Public Key Recipients |
## Relationships
| Name |
Description |
| failure |
Encryption Failed |
| success |
Encryption Completed |
## See also
- [org.apache.nifi.processors.cipher.DecryptContentAge](/user-guide/data-integration/openflow/processors/decryptcontentage)
---
title: EncryptContentPGP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/encryptcontentpgp.md
section: Loading & Unloading Data
---
# EncryptContentPGP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-pgp-nar
## Description
Encrypt contents using OpenPGP. The processor reads input and detects OpenPGP messages to avoid unnecessary additional wrapping in Literal Data packets.
## Tags
Encryption, GPG, OpenPGP, PGP, RFC 4880
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| file-encoding |
File Encoding for encryption |
| passphrase |
Passphrase used for encrypting data with Password-Based Encryption |
| public-key-search |
PGP Public Key Search will be used to match against the User ID or Key ID when formatted as uppercase hexadecimal string of 16 characters |
| public-key-service |
PGP Public Key Service for encrypting data with Public Key Encryption |
| symmetric-key-algorithm |
Symmetric-Key Algorithm for encryption |
## Relationships
| Name |
Description |
| failure |
Encryption Failed |
| success |
Encryption Succeeded |
## Writes attributes
| Name |
Description |
| pgp.symmetric.key.algorithm |
Symmetric-Key Algorithm |
| pgp.symmetric.key.algorithm.block.cipher |
Symmetric-Key Algorithm Block Cipher |
| pgp.symmetric.key.algorithm.key.size |
Symmetric-Key Algorithm Key Size |
| pgp.symmetric.key.algorithm.id |
Symmetric-Key Algorithm Identifier |
| pgp.file.encoding |
File Encoding |
| pgp.compression.algorithm |
Compression Algorithm |
| pgp.compression.algorithm.id |
Compression Algorithm Identifier |
## See also
- [org.apache.nifi.processors.pgp.DecryptContentPGP](/user-guide/data-integration/openflow/processors/decryptcontentpgp)
- [org.apache.nifi.processors.pgp.SignContentPGP](/user-guide/data-integration/openflow/processors/signcontentpgp)
- [org.apache.nifi.processors.pgp.VerifyContentPGP](/user-guide/data-integration/openflow/processors/verifycontentpgp)
---
title: EnforceOrder 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/enforceorder.md
section: Loading & Unloading Data
---
# EnforceOrder 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Enforces expected ordering of FlowFiles that belong to the same data group within a single node. Although PriorityAttributePrioritizer can be used on a connection to ensure that flow files going through that connection are in priority order, depending on error-handling, branching, and other flow designs, it is possible for FlowFiles to get out-of-order. EnforceOrder can be used to enforce original ordering for those FlowFiles. [IMPORTANT] In order to take effect of EnforceOrder, FirstInFirstOutPrioritizer should be used at EVERY downstream relationship UNTIL the order of FlowFiles physically get FIXED by operation such as MergeContent or being stored to the final destination.
## Tags
order, sort
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| batch-count |
The maximum number of FlowFiles that EnforceOrder can process at an execution. |
| group-id |
EnforceOrder is capable of multiple ordering groups. 'Group Identifier' is used to determine which group a FlowFile belongs to. This property will be evaluated with each incoming FlowFile. If evaluated result is empty, the FlowFile will be routed to failure. |
| inactive-timeout |
Indicates the duration after which state for an inactive group will be cleared from managed state. Group is determined as inactive if any new incoming FlowFile has not seen for a group for specified duration. Inactive Timeout must be longer than Wait Timeout. If a FlowFile arrives late after its group is already cleared, it will be treated as a brand new group, but will never match the order since expected preceding FlowFiles are already gone. The FlowFile will eventually timeout for waiting and routed to 'overtook'. To avoid this, group states should be kept long enough, however, shorter duration would be helpful for reusing the same group identifier again. |
| initial-order |
When the first FlowFile of a group arrives, initial target order will be computed and stored in the managed state. After that, target order will start being tracked by EnforceOrder and stored in the state management store. If Expression Language is used but evaluated result was not an integer, then the FlowFile will be routed to failure, and initial order will be left unknown until consecutive FlowFiles provide a valid initial order. |
| maximum-order |
If specified, any FlowFiles that have larger order will be routed to failure. This property is computed only once for a given group. After a maximum order is computed, it will be persisted in the state management store and used for other FlowFiles belonging to the same group. If Expression Language is used but evaluated result was not an integer, then the FlowFile will be routed to failure, and maximum order will be left unknown until consecutive FlowFiles provide a valid maximum order. |
| order-attribute |
A name of FlowFile attribute whose value will be used to enforce order of FlowFiles within a group. If a FlowFile does not have this attribute, or its value is not an integer, the FlowFile will be routed to failure. |
| wait-timeout |
Indicates the duration after which waiting FlowFiles will be routed to the 'overtook' relationship. |
## State management
| Scopes |
Description |
| LOCAL |
EnforceOrder uses following states per ordering group: '<groupId>.target' is a order number which is being waited to arrive next. When a FlowFile with a matching order arrives, or a FlowFile overtakes the FlowFile being waited for because of wait timeout, target order will be updated to (FlowFile.order + 1). '<groupId>.max is the maximum order number for a group. '<groupId>.updatedAt' is a timestamp when the order of a group was updated last time. These managed states will be removed automatically once a group is determined as inactive, see 'Inactive Timeout' for detail. |
## Relationships
| Name |
Description |
| failure |
A FlowFiles which does not have required attributes, or fails to compute those will be routed to this relationship |
| overtook |
A FlowFile that waited for preceding FlowFiles longer than Wait Timeout and overtook those FlowFiles, will be routed to this relationship. |
| skipped |
A FlowFile that has an order younger than current, which means arrived too late and skipped, will be routed to this relationship. |
| success |
A FlowFile with a matching order number will be routed to this relationship. |
| wait |
A FlowFile with non matching order will be routed to this relationship |
## Writes attributes
| Name |
Description |
| EnforceOrder.startedAt |
All FlowFiles going through this processor will have this attribute. This value is used to determine wait timeout. |
| EnforceOrder.result |
All FlowFiles going through this processor will have this attribute denoting which relationship it was routed to. |
| EnforceOrder.detail |
FlowFiles routed to 'failure' or 'skipped' relationship will have this attribute describing details. |
| EnforceOrder.expectedOrder |
FlowFiles routed to 'wait' or 'skipped' relationship will have this attribute denoting expected order when the FlowFile was processed. |
---
title: EnrichAttributes 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/enrichattributes.md
section: Loading & Unloading Data
---
# EnrichAttributes 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-enrichment-nar
## Description
Looks up a value using the configured Lookup Service and adds the results to the FlowFile as one or more attributes. Frequently, this is used in conjunction with the DatabaseLookup Service in order to enrich a FlowFile by querying a database and adding the results as attributes.
## Tags
attributes, database, enrichment, json, lookup, openflow
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Attribute Name |
The name of the attribute to add, whose contents will be the JSON representation of the Record returned from the Lookup Service. |
| Attribute Prefix |
A prefix to apply to all attribute names that are added. |
| Flattening Strategy |
When a Record is returned from the Lookup Service, this property specifies how the Record should be flattened into the FlowFile's attributes |
| Lookup Service |
The Lookup Service to use for enrichment |
## Relationships
| Name |
Description |
| failure |
If unable to enrich a given FlowFile for any reason, the FlowFile will be routed to this relationship. |
| matched |
FlowFiles that are successfully enriched with the Record from the Lookup Service are routed to this relationship. |
| unmatched |
FlowFiles for which the Lookup Service did not find a match are routed to this relationship. |
## Use cases
| Query a database to retrieve information based on the attributes of a FlowFile |
| ------------------------------------------------------------------------------ |
## See also
---
title: EnrichCdcStream 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/enrichcdcstream.md
section: Loading & Unloading Data
---
# EnrichCdcStream 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-database-cdc-processors-nar
## Description
Enriches incoming FlowFiles that come from CaptureChangePostgreSQL, etc. with information pertaining to which Journal Table to write to and relevant schema information. This Processor manages the schema versions for each table being processed in order to ensure that the correct Journal Table is used for each FlowFile.
## Tags
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| CDC Schema Registry |
Specifies the CDC Schema Registry to use for managing the schemas of the CDC data |
| Record Reader |
Specifies the Record Reader to use for reading the incoming data |
| Record Writer |
Specifies the Record Writer to use for writing the outgoing data |
| Table State Service |
Holds the state of replicated tables |
## State management
| Scopes |
Description |
| CLUSTER |
Tracks the current journal table version for each table being processed. |
## Relationships
| Name |
Description |
| failure |
If any FlowFile is unable to be read, it will be routed to this Relationship. |
| schema update |
If any schema update is required in order to handle incoming Records, a FlowFile is routed to this relationship. The FlowFile will include the schema information to indicate what changes are required. |
| skipped ddl event |
This Relationship will be used for any DDL / Schema Change events that do not result in a change to the destination table's schema. |
| success |
Rows to be inserted into the Snowflake table will be routed to this Relationship. |
| table not in state |
Used when a FlowFile references a table that does not exist in the state of replicated tables, probably after it was removed from replication. |
## Writes attributes
| Name |
Description |
| table.schema.generation |
The index of the journal table for incremental processing. |
| table.schema.initial |
Marks the initial generation of a journal table. |
| destination.table.schema |
The updated schema for the destination table. This attribute is only written for DDL events. |
## See also
- [com.snowflake.openflow.runtime.processors.database.CaptureChangePostgreSQL](/user-guide/data-integration/openflow/processors/capturechangepostgresql)
---
title: EvaluateJsonPath 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/evaluatejsonpath.md
section: Loading & Unloading Data
---
# EvaluateJsonPath 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Evaluates one or more JsonPath expressions against the content of a FlowFile. The results of those expressions are assigned to FlowFile Attributes or are written to the content of the FlowFile itself, depending on configuration of the Processor. JsonPaths are entered by adding user-defined properties; the name of the property maps to the Attribute Name into which the result will be placed (if the Destination is flowfile-attribute; otherwise, the property name is ignored). The value of the property must be a valid JsonPath expression. A Return Type of 'auto-detect' will make a determination based off the configured destination. When 'Destination' is set to 'flowfile-attribute,' a return type of 'scalar' will be used. When 'Destination' is set to 'flowfile-content,' a return type of 'JSON' will be used. If the JsonPath evaluates to a JSON array or JSON object and the Return Type is set to 'scalar' the FlowFile will be unmodified and will be routed to failure. A Return Type of JSON can return scalar values if the provided JsonPath evaluates to the specified value and will be routed as a match. If Destination is 'flowfile-content' and the JsonPath does not evaluate to a defined path, the FlowFile will be routed to 'unmatched' without having its contents modified. If Destination is 'flowfile-attribute' and the expression matches nothing, attributes will be created with empty strings as the value unless 'Path Not Found Behaviour' is set to 'skip', and the FlowFile will always be routed to 'matched.'
## Tags
JSON, JsonPath, evaluate
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Destination |
Indicates whether the results of the JsonPath evaluation are written to the FlowFile content or a FlowFile attribute; if using attribute, must specify the Attribute Name property. If set to flowfile-content, only one JsonPath may be specified, and the property name is ignored. |
| Max String Length |
The maximum allowed length of a string value when parsing the JSON document |
| Null Value Representation |
Indicates the desired representation of JSON Path expressions resulting in a null value. |
| Path Not Found Behavior |
Indicates how to handle missing JSON path expressions when destination is set to 'flowfile-attribute'. Selecting 'warn' will generate a warning when a JSON path expression is not found. Selecting 'skip' will omit attributes for any unmatched JSON path expressions. |
| Return Type |
Indicates the desired return type of the JSON Path expressions. Selecting 'auto-detect' will set the return type to 'json' for a Destination of 'flowfile-content', and 'scalar' for a Destination of 'flowfile-attribute'. |
## Relationships
| Name |
Description |
| failure |
FlowFiles are routed to this relationship when the JsonPath cannot be evaluated against the content of the FlowFile; for instance, if the FlowFile is not valid JSON |
| matched |
FlowFiles are routed to this relationship when the JsonPath is successfully evaluated and the FlowFile is modified as a result |
| unmatched |
FlowFiles are routed to this relationship when the JsonPath does not match the content of the FlowFile and the Destination is set to flowfile-content |
---
title: EvaluateRagAnswerCorrectness 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/evaluateraganswercorrectness.md
section: Loading & Unloading Data
---
# EvaluateRagAnswerCorrectness 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-rag-evaluation-processors-nar
## Description
Evaluates the correctness of generated answers in a Retrieval-Augmented Generation (RAG) context by computing metrics such as F1 score, cosine similarity, and answer correctness. The processor uses an LLM (e.g., OpenAI's GPT) to assess the generated answer against the ground truth.
## Tags
ai, answer correctness, evaluation, llm, nlp, openai, openflow, rag
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Cosine Similarity Weight |
The weight to apply to the cosine similarity when calculating answer correctness (between 0.0 and 1.0) |
| Evaluation Results Record Path |
The RecordPath to write the results of the evaluation to. |
| F1 Score Weight |
The weight to apply to the F1 score when calculating answer correctness (between 0.0 and 1.0) |
| Generated Answer Record Path |
The path to the answer field in the record |
| Generated Answer Vector Record Path |
The path to the answer vector field in the record. |
| Ground Truth Record Path |
The RecordPath to the ground truth field in the record. |
| Ground Truth Vector Record Path |
The path to the ground truth vector field in the record. |
| LLM Provider Service |
The provider service for sending evaluation prompts to LLM |
| Question Record Path |
The RecordPath to the question field in the record. |
| Record Reader |
The Record Reader to use for reading the FlowFile. |
| Record Writer |
The Record Writer to use for writing the results. |
## Relationships
| Name |
Description |
| failure |
FlowFiles that cannot be processed are routed to this relationship |
| success |
FlowFiles that are successfully processed are routed to this relationship |
## Writes attributes
| Name |
Description |
| average.f1Score |
The average F1 score computed over all records. |
| average.cosineSim |
The average cosine similarity between the ground truth and answer embeddings. |
| average.answerCorrectness |
The average answer correctness score computed over all records. |
| json.parse.failures |
Number of JSON parse failures encountered. |
## Use cases
| Use this processor to assess the quality of answers generated by an LLM in comparison to ground truth answers, providing metrics that can be used for monitoring and improving the performance of RAG systems. |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
---
title: EvaluateRagFaithfulness 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/evaluateragfaithfulness.md
section: Loading & Unloading Data
---
# EvaluateRagFaithfulness 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-rag-evaluation-processors-nar
## Description
Evaluates the faithfulness of generated answers in a Retrieval-Augmented Generation (RAG) system by analyzing responses using an LLM (e.g., OpenAI's GPT). The processor enriches each FlowFile record with faithfulness metrics and detailed analysis.
## Tags
ai, evaluation, faithfulness, llm, nlp, openai, openflow, rag
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Context Identifier Record Path |
The RecordPath to the array of contexts IDs in the record. |
| Context Record Path |
The RecordPath to the array of contexts in the record. |
| Evaluation Results Record Path |
The RecordPath to write the results of the evaluation to. |
| Generated Answer Record Path |
The path to the answer field in the record |
| LLM Provider Service |
The provider service for sending evaluation prompts to LLM |
| Question Record Path |
The RecordPath to the question field in the record. |
| Record Reader |
The Record Reader to use for reading the FlowFile. |
| Record Writer |
The Record Writer to use for writing the results. |
## Relationships
| Name |
Description |
| failure |
FlowFiles that cannot be processed are routed to this relationship |
| success |
FlowFiles that are successfully processed are routed to this relationship |
## Writes attributes
| Name |
Description |
| average.answer.faithfulness |
The average faithfulness score computed over all records. |
| json.parse.failures |
Number of JSON parse failures encountered. |
## Use cases
| Use this processor to assess the faithfulness of answers generated by an LLM compared to the provided context. It provides metrics that can be used for monitoring and improving the performance of RAG systems. |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
---
title: EvaluateRagRetrieval 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/evaluateragretrieval.md
section: Loading & Unloading Data
---
# EvaluateRagRetrieval 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-rag-evaluation-processors-nar
## Description
Calculates retrieval metrics (Precision@N, Recall@N, FScore@N, MAP@N, MRR) for a RAG system using an LLM as a judge. For each record, it uses both Precision and Recall prompts to evaluate the response, and adds the metrics as attributes to the FlowFile.
## Tags
evaluation, fscore, llm, metrics, mrr, openai, openflow, precision, rag, recall, retrieval
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Context Identifier Record Path |
The RecordPath to the array of contexts IDs in the record. |
| Context Record Path |
The RecordPath to the array of contexts in the record. |
| Evaluation Results Record Path |
The RecordPath to write the results of the evaluation to. |
| Ground Truth Record Path |
The RecordPath to the ground truth field in the record. |
| LLM Provider Service |
The provider service for sending evaluation prompts to LLM |
| Question Record Path |
The RecordPath to the question field in the record. |
| Record Reader |
The Record Reader to use for reading the FlowFile. |
| Record Writer |
The Record Writer to use for writing the results. |
## Relationships
| Name |
Description |
| failure |
FlowFiles that cannot be processed are routed to this relationship |
| success |
FlowFiles that are successfully processed are routed to this relationship |
## Writes attributes
| Name |
Description |
| n |
The average number of retrieved documents per query. |
| precision.at.n |
The average precision at N over all queries. |
| recall.at.n |
The average recall at N over all queries. |
| fscore.at.n |
The average F-Score at N over all queries. |
| mrr |
The Mean Reciprocal Rank. |
| retrieval.eval.failures |
Number of records where the eval could not be calculated. |
| json.parse.failures |
Number of JSON parse failures encountered. |
---
title: EvaluateXPath 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/evaluatexpath.md
section: Loading & Unloading Data
---
# EvaluateXPath 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Evaluates one or more XPaths against the content of a FlowFile. The results of those XPaths are assigned to FlowFile Attributes or are written to the content of the FlowFile itself, depending on configuration of the Processor. XPaths are entered by adding user-defined properties; the name of the property maps to the Attribute Name into which the result will be placed (if the Destination is flowfile-attribute; otherwise, the property name is ignored). The value of the property must be a valid XPath expression. If the XPath evaluates to more than one node and the Return Type is set to 'nodeset' (either directly, or via 'auto-detect' with a Destination of 'flowfile-content'), the FlowFile will be unmodified and will be routed to failure. If the XPath does not evaluate to a Node, the FlowFile will be routed to 'unmatched' without having its contents modified. If Destination is flowfile-attribute and the expression matches nothing, attributes will be created with empty strings as the value, and the FlowFile will always be routed to 'matched'
## Tags
XML, XPath, evaluate
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Destination |
Indicates whether the results of the XPath evaluation are written to the FlowFile content or a FlowFile attribute; if using attribute, must specify the Attribute Name property. If set to flowfile-content, only one XPath may be specified, and the property name is ignored. |
| Return Type |
Indicates the desired return type of the Xpath expressions. Selecting 'auto-detect' will set the return type to 'nodeset' for a Destination of 'flowfile-content', and 'string' for a Destination of 'flowfile-attribute'. |
| Validate DTD |
Allow embedded Document Type Declaration in XML. This feature should be disabled to avoid XML entity expansion vulnerabilities. |
## Relationships
| Name |
Description |
| failure |
FlowFiles are routed to this relationship when the XPath cannot be evaluated against the content of the FlowFile; for instance, if the FlowFile is not valid XML, or if the Return Type is 'nodeset' and the XPath evaluates to multiple nodes |
| matched |
FlowFiles are routed to this relationship when the XPath is successfully evaluated and the FlowFile is modified as a result |
| unmatched |
FlowFiles are routed to this relationship when the XPath does not match the content of the FlowFile and the Destination is set to flowfile-content |
## Writes attributes
| Name |
Description |
| user-defined |
This processor adds user-defined attributes if the <Destination> property is set to flowfile-attribute. |
---
title: EvaluateXQuery 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/evaluatexquery.md
section: Loading & Unloading Data
---
# EvaluateXQuery 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Evaluates one or more XQueries against the content of a FlowFile. The results of those XQueries are assigned to FlowFile Attributes or are written to the content of the FlowFile itself, depending on configuration of the Processor. XQueries are entered by adding user-defined properties; the name of the property maps to the Attribute Name into which the result will be placed (if the Destination is 'flowfile-attribute'; otherwise, the property name is ignored). The value of the property must be a valid XQuery. If the XQuery returns more than one result, new attributes or FlowFiles (for Destinations of 'flowfile-attribute' or 'flowfile-content' respectively) will be created for each result (attributes will have a '.n' one-up number appended to the specified attribute name). If any provided XQuery returns a result, the FlowFile(s) will be routed to 'matched'. If no provided XQuery returns a result, the FlowFile will be routed to 'unmatched'. If the Destination is 'flowfile-attribute' and the XQueries matche nothing, no attributes will be applied to the FlowFile.
## Tags
XML, XPath, XQuery, evaluate
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Destination |
Indicates whether the results of the XQuery evaluation are written to the FlowFile content or a FlowFile attribute. If set to <flowfile-content>, only one XQuery may be specified and the property name is ignored. If set to <flowfile-attribute> and the XQuery returns more than one result, multiple attributes will be added to theFlowFile, each named with a '.n' one-up number appended to the specified attribute name |
| Output: Indent |
Specifies whether the processor may add additional whitespace when outputting a result tree. |
| Output: Method |
Identifies the overall method that should be used for outputting a result tree. |
| Output: Omit XML Declaration |
Specifies whether the processor should output an XML declaration when transforming a result tree. |
| Validate DTD |
Allow embedded Document Type Declaration in XML. This feature should be disabled to avoid XML entity expansion vulnerabilities. |
## Relationships
| Name |
Description |
| failure |
FlowFiles are routed to this relationship when the XQuery cannot be evaluated against the content of the FlowFile. |
| matched |
FlowFiles are routed to this relationship when the XQuery is successfully evaluated and the FlowFile is modified as a result |
| unmatched |
FlowFiles are routed to this relationship when the XQuery does not match the content of the FlowFile and the Destination is set to flowfile-content |
## Writes attributes
| Name |
Description |
| user-defined |
This processor adds user-defined attributes if the <Destination> property is set to flowfile-attribute . |
---
title: ExcelReader
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/excelreader.md
section: Loading & Unloading Data
---
# ExcelReader
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Parses a Microsoft Excel document returning each row in each sheet as a separate record. This reader allows for inferring a schema from all the required sheets or providing an explicit schema for interpreting the values. See Controller Service 's Usage for further documentation. This reader is capable of processing both password and non password protected .xlsx (XSSF 2007 OOXML file format) and older .xls (HSSF'97(-2007) file format) Excel documents.
## Tags
cell, excel, parse, reader, record, row, spreadsheet, values, xls, xlsx
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Date Format |
Date Format |
|
|
Specifies the format to use when reading/writing Date fields. If not specified, Date fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, MM/dd/yyyy for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters, as in 01/01/2017). |
| Input File Type * |
Input File Type |
XLSX |
- XLS
- XLSX
|
Specifies type of Excel input file. |
| Password * |
Password |
|
|
The password for a password protected Excel spreadsheet |
| Protection Type * |
Protection Type |
UNPROTECTED |
- Unprotected
- Password Protected
|
Specifies whether an Excel spreadsheet is protected by a password or not. |
| Required Sheets |
Required Sheets |
|
|
Comma-separated list of Excel document sheet names whose rows should be extracted from the excel document. If this property is left blank then all the rows from all the sheets will be extracted from the Excel document. The list of names is case sensitive. Any sheets not specified in this value will be ignored. An exception will be thrown if a specified sheet(s) are not found. |
| Row Evaluation Strategy * |
Row Evaluation Strategy |
STANDARD |
- Standard
- All Rows
|
A strategy to select how many rows after the starting row to use for determining the schema. |
| Schema Access Strategy * |
Schema Access Strategy |
Use Starting Row |
- Use 'Schema Name' Property
- Use 'Schema Text' Property
- Schema Reference Reader
- Use Starting Row
- Infer Schema
|
Specifies how to obtain the schema that is to be used for interpreting the data. |
| Schema Branch |
Schema Branch |
|
|
Specifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored. |
| Schema Name |
Schema Name |
$\{schema.name\} |
|
Specifies the name of the schema to lookup in the Schema Registry property |
| Schema Reference Reader * |
Schema Reference Reader |
|
|
Service implementation responsible for reading FlowFile attributes or content to determine the Schema Reference Identifier |
| Schema Registry |
Schema Registry |
|
|
Specifies the Controller Service to use for the Schema Registry |
| Schema Text |
Schema Text |
$\{avro.schema\} |
|
The text of an Avro-formatted Schema |
| Schema Version |
Schema Version |
|
|
Specifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved. |
| Starting Row * |
Starting Row |
1 |
|
The row number of the first row to start processing (One based). Use this to skip over rows of data at the top of a worksheet that are not part of the dataset. When using the 'Use Starting Row' strategy this should be the column header row. |
| Time Format |
Time Format |
|
|
Specifies the format to use when reading/writing Time fields. If not specified, Time fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, HH:mm:ss for a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 18:04:15). |
| Timestamp Format |
Timestamp Format |
|
|
Specifies the format to use when reading/writing Timestamp fields. If not specified, Timestamp fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, MM/dd/yyyy HH:mm:ss for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters; and then followed by a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 01/01/2017 18:04:15). |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: ExecuteGroovyScript 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/executegroovyscript.md
section: Loading & Unloading Data
---
# ExecuteGroovyScript 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-groovyx-nar
## Description
Experimental Extended Groovy script processor. The script is responsible for handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by the script. If the handling is incomplete or incorrect, the session will be rolled back.
## Tags
groovy, groovyx, script
## Input Requirement
ALLOWED
## Supports Sensitive Dynamic Properties
true
## Properties
| Property |
Description |
| groovyx-additional-classpath |
Classpath list separated by semicolon or comma. You can use masks like _*_, _*.jar_ in file name. |
| groovyx-failure-strategy |
What to do with unhandled exceptions. If you want to manage exception by code then keep the default value _rollback_. If _transfer to failure_ selected and unhandled exception occurred then all flowFiles received from incoming queues in this session will be transferred to _failure_ relationship with additional attributes set: ERROR_MESSAGE and ERROR_STACKTRACE. If _rollback_ selected and unhandled exception occurred then all flowFiles received from incoming queues will be penalized and returned. If the processor has no incoming connections then this parameter has no effect. |
| groovyx-script-body |
Body of script to execute. Only one of Script File or Script Body may be used |
| groovyx-script-file |
Path to script file to execute. Only one of Script File or Script Body may be used |
## State management
| Scopes |
Description |
| LOCAL |
Scripts can store and retrieve state using the State Management APIs. Consult the State Manager section of the Developer's Guide for more details. |
| CLUSTER |
Scripts can store and retrieve state using the State Management APIs. Consult the State Manager section of the Developer's Guide for more details. |
## Restrictions
| Required Permission |
Explanation |
| execute code |
Provides operator the ability to execute arbitrary code assuming all permissions that NiFi has. |
## Relationships
| Name |
Description |
| failure |
FlowFiles that failed to be processed |
| success |
FlowFiles that were successfully processed |
## See also
- [org.apache.nifi.processors.script.ExecuteScript](/user-guide/data-integration/openflow/processors/executescript)
---
title: ExecuteProcess 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/executeprocess.md
section: Loading & Unloading Data
---
# ExecuteProcess 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Runs an operating system command specified by the user and writes the output of that command to a FlowFile. If the command is expected to be long-running, the Processor can output the partial data on a specified interval. When this option is used, the output is expected to be in textual format, as it typically does not make sense to split binary data on arbitrary time-based intervals.
## Tags
command, external, invoke, process, script, source
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Argument Delimiter |
Delimiter to use to separate arguments for a command [default: space]. Must be a single character. |
| Batch Duration |
If the process is expected to be long-running and produce textual output, a batch duration can be specified so that the output will be captured for this amount of time and a FlowFile will then be sent out with the results and a new FlowFile will be started, rather than waiting for the process to finish before sending out the results |
| Command |
Specifies the command to be executed; if just the name of an executable is provided, it must be in the user's environment PATH. |
| Command Arguments |
The arguments to supply to the executable delimited by white space. White space can be escaped by enclosing it in double-quotes. |
| Output MIME type |
Specifies the value to set for the "mime.type" attribute. This property is ignored if 'Batch Duration' is set. |
| Redirect Error Stream |
If true will redirect any error stream output of the process to the output stream. This is particularly helpful for processes which write extensively to the error stream or for troubleshooting. |
| Working Directory |
The directory to use as the current working directory when executing the command |
## Restrictions
| Required Permission |
Explanation |
| execute code |
Provides operator the ability to execute arbitrary code assuming all permissions that NiFi has. |
## Relationships
| Name |
Description |
| success |
All created FlowFiles are routed to this relationship |
## Writes attributes
| Name |
Description |
| command |
Executed command |
| command.arguments |
Arguments of the command |
| mime.type |
Sets the MIME type of the output if the 'Output MIME Type' property is set and 'Batch Duration' is not set |
---
title: ExecuteScript 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/executescript.md
section: Loading & Unloading Data
---
# ExecuteScript 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-scripting-nar
## Description
Experimental - Executes a script given the flow file and a process session. The script is responsible for handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by the script. If the handling is incomplete or incorrect, the session will be rolled back. Experimental: Impact of sustained usage not yet verified.
## Tags
clojure, execute, groovy, script
## Input Requirement
ALLOWED
## Supports Sensitive Dynamic Properties
true
## Properties
| Property |
Description |
| Module Directory |
Comma-separated list of paths to files and/or directories which contain modules required by the script. |
| Script Body |
Body of script to execute. Only one of Script File or Script Body may be used |
| Script Engine |
Language Engine for executing scripts |
| Script File |
Path to script file to execute. Only one of Script File or Script Body may be used |
## State management
| Scopes |
Description |
| LOCAL |
Scripts can store and retrieve state using the State Management APIs. Consult the State Manager section of the Developer's Guide for more details. |
| CLUSTER |
Scripts can store and retrieve state using the State Management APIs. Consult the State Manager section of the Developer's Guide for more details. |
## Restrictions
| Required Permission |
Explanation |
| execute code |
Provides operator the ability to execute arbitrary code assuming all permissions that NiFi has. |
## Relationships
| Name |
Description |
| failure |
FlowFiles that failed to be processed |
| success |
FlowFiles that were successfully processed |
## See also
- [org.apache.nifi.processors.script.InvokeScriptedProcessor](/user-guide/data-integration/openflow/processors/invokescriptedprocessor)
---
title: ExecuteSQL 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/executesql.md
section: Loading & Unloading Data
---
# ExecuteSQL 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Executes provided SQL select query. Query result will be converted to Avro format. Streaming is used so arbitrarily large result sets are supported. This processor can be scheduled to run on a timer, or cron expression, using the standard scheduling methods, or it can be triggered by an incoming FlowFile. If it is triggered by an incoming FlowFile, then attributes of that FlowFile will be available when evaluating the select query, and the query may use the ? to escape parameters. In this case, the parameters to use must exist as FlowFile attributes with the naming convention sql.args. N.type and sql.args. N.value, where N is a positive integer. The sql.args. N.type is expected to be a number indicating the JDBC Type. The content of the FlowFile is expected to be in UTF-8 format. FlowFile attribute 'executesql.row.count' indicates how many rows were selected.
## Tags
database, jdbc, query, select, sql
## Input Requirement
ALLOWED
## Supports Sensitive Dynamic Properties
true
## Properties
| Property |
Description |
| Content Output Strategy |
Specifies the strategy for writing FlowFile content when processing input FlowFiles. The strategy applies when handling queries that do not produce results. |
| Database Connection Pooling Service |
The Controller Service that is used to obtain connection to database |
| Default Decimal Precision |
When a DECIMAL/NUMBER value is written as a 'decimal' Avro logical type, a specific 'precision' denoting number of available digits is required. Generally, precision is defined by column data type definition or database engines default. However undefined precision (0) can be returned from some database engines. 'Default Decimal Precision' is used when writing those undefined precision numbers. |
| Default Decimal Scale |
When a DECIMAL/NUMBER value is written as a 'decimal' Avro logical type, a specific 'scale' denoting number of available decimal digits is required. Generally, scale is defined by column data type definition or database engines default. However when undefined precision (0) is returned, scale can also be uncertain with some database engines. 'Default Decimal Scale' is used when writing those undefined numbers. If a value has more decimals than specified scale, then the value will be rounded-up, e.g. 1.53 becomes 2 with scale 0, and 1.5 with scale 1. |
| Max Wait Time |
The maximum amount of time allowed for a running SQL select query , zero means there is no limit. Max time less than 1 second will be equal to zero. |
| Normalize Table and Column Names |
Whether to change non-Avro-compatible characters in column names to Avro-compatible characters. For example, colons and periods will be changed to underscores in order to build a valid Avro record. |
| SQL Query |
The SQL query to execute. The query can be empty, a constant value, or built from attributes using Expression Language. If this property is specified, it will be used regardless of the content of incoming flowfiles. If this property is empty, the content of the incoming flow file is expected to contain a valid SQL select query, to be issued by the processor to the database. Note that Expression Language is not evaluated for flow file contents. |
| Use Avro Logical Types |
Whether to use Avro Logical Types for DECIMAL/NUMBER, DATE, TIME and TIMESTAMP columns. If disabled, written as string. If enabled, Logical types are used and written as its underlying type, specifically, DECIMAL/NUMBER as logical 'decimal': written as bytes with additional precision and scale meta data, DATE as logical 'date-millis': written as int denoting days since Unix epoch (1970-01-01), TIME as logical 'time-millis': written as int denoting milliseconds since Unix epoch, and TIMESTAMP as logical 'timestamp-millis': written as long denoting milliseconds since Unix epoch. If a reader of written Avro records also knows these logical types, then these values can be deserialized with more context depending on reader implementation. |
| compression-format |
Compression type to use when writing Avro files. Default is None. |
| esql-auto-commit |
Enables or disables the auto commit functionality of the DB connection. Default value is 'true'. The default value can be used with most of the JDBC drivers and this functionality doesn't have any impact in most of the cases since this processor is used to read data. However, for some JDBC drivers such as PostgreSQL driver, it is required to disable the auto committing functionality to limit the number of result rows fetching at a time. When auto commit is enabled, postgreSQL driver loads whole result set to memory at once. This could lead for a large amount of memory usage when executing queries which fetch large data sets. More Details of this behaviour in PostgreSQL driver can be found in https://jdbc.postgresql.org//documentation/head/query.html (https://jdbc.postgresql.org//documentation/head/query.html). |
| esql-fetch-size |
The number of result rows to be fetched from the result set at a time. This is a hint to the database driver and may not be honored and/or exact. If the value specified is zero, then the hint is ignored. |
| esql-max-rows |
The maximum number of result rows that will be included in a single FlowFile. This will allow you to break up very large result sets into multiple FlowFiles. If the value specified is zero, then all rows are returned in a single FlowFile. |
| esql-output-batch-size |
The number of output FlowFiles to queue before committing the process session. When set to zero, the session will be committed when all result set rows have been processed and the output FlowFiles are ready for transfer to the downstream relationship. For large result sets, this can cause a large burst of FlowFiles to be transferred at the end of processor execution. If this property is set, then when the specified number of FlowFiles are ready for transfer, then the session will be committed, thus releasing the FlowFiles to the downstream relationship. NOTE: The fragment.count attribute will not be set on FlowFiles when this property is set. |
| sql-post-query |
A semicolon-delimited list of queries executed after the main SQL query is executed. Example like setting session properties after main query. It 's possible to include semicolons in the statements themselves by escaping them with a backslash (';'). Results/outputs from these queries will be suppressed if there are no errors. |
| sql-pre-query |
A semicolon-delimited list of queries executed before the main SQL query is executed. For example, set session properties before main query. It 's possible to include semicolons in the statements themselves by escaping them with a backslash (';'). Results/outputs from these queries will be suppressed if there are no errors. |
## Relationships
| Name |
Description |
| failure |
SQL query execution failed. Incoming FlowFile will be penalized and routed to this relationship |
| success |
Successfully created FlowFile from SQL query result set. |
## Writes attributes
| Name |
Description |
| executesql.row.count |
Contains the number of rows returned by the query. If 'Max Rows Per Flow File' is set, then this number will reflect the number of rows in the Flow File instead of the entire result set. |
| executesql.query.duration |
Combined duration of the query execution time and fetch time in milliseconds. If 'Max Rows Per Flow File' is set, then this number will reflect only the fetch time for the rows in the Flow File instead of the entire result set. |
| executesql.query.executiontime |
Duration of the query execution time in milliseconds. This number will reflect the query execution time regardless of the 'Max Rows Per Flow File' setting. |
| executesql.query.fetchtime |
Duration of the result set fetch time in milliseconds. If 'Max Rows Per Flow File' is set, then this number will reflect only the fetch time for the rows in the Flow File instead of the entire result set. |
| executesql.resultset.index |
Assuming multiple result sets are returned, the zero based index of this result set. |
| executesql.error.message |
If processing an incoming flow file causes an Exception, the Flow File is routed to failure and this attribute is set to the exception message. |
| fragment.identifier |
If 'Max Rows Per Flow File' is set then all FlowFiles from the same query result set will have the same value for the fragment.identifier attribute. This can then be used to correlate the results. |
| fragment.count |
If 'Max Rows Per Flow File' is set then this is the total number of FlowFiles produced by a single ResultSet. This can be used in conjunction with the fragment.identifier attribute in order to know how many FlowFiles belonged to the same incoming ResultSet. If Output Batch Size is set, then this attribute will not be populated. |
| fragment.index |
If 'Max Rows Per Flow File' is set then the position of this FlowFile in the list of outgoing FlowFiles that were all derived from the same result set FlowFile. This can be used in conjunction with the fragment.identifier attribute to know which FlowFiles originated from the same query result set and in what order FlowFiles were produced |
| input.flowfile.uuid |
If the processor has an incoming connection, outgoing FlowFiles will have this attribute set to the value of the input FlowFile's UUID. If there is no incoming connection, the attribute will not be added. |
---
title: ExecuteSQLRecord 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/executesqlrecord.md
section: Loading & Unloading Data
---
# ExecuteSQLRecord 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Executes provided SQL select query. Query result will be converted to the format specified by a Record Writer. Streaming is used so arbitrarily large result sets are supported. This processor can be scheduled to run on a timer, or cron expression, using the standard scheduling methods, or it can be triggered by an incoming FlowFile. If it is triggered by an incoming FlowFile, then attributes of that FlowFile will be available when evaluating the select query, and the query may use the ? to escape parameters. In this case, the parameters to use must exist as FlowFile attributes with the naming convention sql.args. N.type and sql.args. N.value, where N is a positive integer. The sql.args. N.type is expected to be a number indicating the JDBC Type. The content of the FlowFile is expected to be in UTF-8 format. FlowFile attribute 'executesql.row.count' indicates how many rows were selected.
## Tags
database, jdbc, query, record, select, sql
## Input Requirement
ALLOWED
## Supports Sensitive Dynamic Properties
true
## Properties
| Property |
Description |
| Database Connection Pooling Service |
The Controller Service that is used to obtain connection to database |
| Default Decimal Precision |
When a DECIMAL/NUMBER value is written as a 'decimal' Avro logical type, a specific 'precision' denoting number of available digits is required. Generally, precision is defined by column data type definition or database engines default. However undefined precision (0) can be returned from some database engines. 'Default Decimal Precision' is used when writing those undefined precision numbers. |
| Default Decimal Scale |
When a DECIMAL/NUMBER value is written as a 'decimal' Avro logical type, a specific 'scale' denoting number of available decimal digits is required. Generally, scale is defined by column data type definition or database engines default. However when undefined precision (0) is returned, scale can also be uncertain with some database engines. 'Default Decimal Scale' is used when writing those undefined numbers. If a value has more decimals than specified scale, then the value will be rounded-up, e.g. 1.53 becomes 2 with scale 0, and 1.5 with scale 1. |
| Max Wait Time |
The maximum amount of time allowed for a running SQL select query , zero means there is no limit. Max time less than 1 second will be equal to zero. |
| SQL Query |
The SQL query to execute. The query can be empty, a constant value, or built from attributes using Expression Language. If this property is specified, it will be used regardless of the content of incoming flowfiles. If this property is empty, the content of the incoming flow file is expected to contain a valid SQL select query, to be issued by the processor to the database. Note that Expression Language is not evaluated for flow file contents. |
| Use Avro Logical Types |
Whether to use Avro Logical Types for DECIMAL/NUMBER, DATE, TIME and TIMESTAMP columns. If disabled, written as string. If enabled, Logical types are used and written as its underlying type, specifically, DECIMAL/NUMBER as logical 'decimal': written as bytes with additional precision and scale meta data, DATE as logical 'date-millis': written as int denoting days since Unix epoch (1970-01-01), TIME as logical 'time-millis': written as int denoting milliseconds since Unix epoch, and TIMESTAMP as logical 'timestamp-millis': written as long denoting milliseconds since Unix epoch. If a reader of written Avro records also knows these logical types, then these values can be deserialized with more context depending on reader implementation. |
| esql-auto-commit |
Enables or disables the auto commit functionality of the DB connection. Default value is 'true'. The default value can be used with most of the JDBC drivers and this functionality doesn't have any impact in most of the cases since this processor is used to read data. However, for some JDBC drivers such as PostgreSQL driver, it is required to disable the auto committing functionality to limit the number of result rows fetching at a time. When auto commit is enabled, postgreSQL driver loads whole result set to memory at once. This could lead for a large amount of memory usage when executing queries which fetch large data sets. More Details of this behaviour in PostgreSQL driver can be found in https://jdbc.postgresql.org//documentation/head/query.html (https://jdbc.postgresql.org//documentation/head/query.html). |
| esql-fetch-size |
The number of result rows to be fetched from the result set at a time. This is a hint to the database driver and may not be honored and/or exact. If the value specified is zero, then the hint is ignored. |
| esql-max-rows |
The maximum number of result rows that will be included in a single FlowFile. This will allow you to break up very large result sets into multiple FlowFiles. If the value specified is zero, then all rows are returned in a single FlowFile. |
| esql-output-batch-size |
The number of output FlowFiles to queue before committing the process session. When set to zero, the session will be committed when all result set rows have been processed and the output FlowFiles are ready for transfer to the downstream relationship. For large result sets, this can cause a large burst of FlowFiles to be transferred at the end of processor execution. If this property is set, then when the specified number of FlowFiles are ready for transfer, then the session will be committed, thus releasing the FlowFiles to the downstream relationship. NOTE: The fragment.count attribute will not be set on FlowFiles when this property is set. |
| esqlrecord-normalize |
Whether to change characters in column names. For example, colons and periods will be changed to underscores. |
| esqlrecord-record-writer |
Specifies the Controller Service to use for writing results to a FlowFile. The Record Writer may use Inherit Schema to emulate the inferred schema behavior, i.e. an explicit schema need not be defined in the writer, and will be supplied by the same logic used to infer the schema from the column types. |
| sql-post-query |
A semicolon-delimited list of queries executed after the main SQL query is executed. Example like setting session properties after main query. It 's possible to include semicolons in the statements themselves by escaping them with a backslash (';'). Results/outputs from these queries will be suppressed if there are no errors. |
| sql-pre-query |
A semicolon-delimited list of queries executed before the main SQL query is executed. For example, set session properties before main query. It 's possible to include semicolons in the statements themselves by escaping them with a backslash (';'). Results/outputs from these queries will be suppressed if there are no errors. |
## Relationships
| Name |
Description |
| failure |
SQL query execution failed. Incoming FlowFile will be penalized and routed to this relationship |
| success |
Successfully created FlowFile from SQL query result set. |
## Writes attributes
| Name |
Description |
| executesql.row.count |
Contains the number of rows returned in the select query |
| executesql.query.duration |
Combined duration of the query execution time and fetch time in milliseconds |
| executesql.query.executiontime |
Duration of the query execution time in milliseconds |
| executesql.query.fetchtime |
Duration of the result set fetch time in milliseconds |
| executesql.resultset.index |
Assuming multiple result sets are returned, the zero based index of this result set. |
| executesql.error.message |
If processing an incoming flow file causes an Exception, the Flow File is routed to failure and this attribute is set to the exception message. |
| fragment.identifier |
If 'Max Rows Per Flow File' is set then all FlowFiles from the same query result set will have the same value for the fragment.identifier attribute. This can then be used to correlate the results. |
| fragment.count |
If 'Max Rows Per Flow File' is set then this is the total number of FlowFiles produced by a single ResultSet. This can be used in conjunction with the fragment.identifier attribute in order to know how many FlowFiles belonged to the same incoming ResultSet. If Output Batch Size is set, then this attribute will not be populated. |
| fragment.index |
If 'Max Rows Per Flow File' is set then the position of this FlowFile in the list of outgoing FlowFiles that were all derived from the same result set FlowFile. This can be used in conjunction with the fragment.identifier attribute to know which FlowFiles originated from the same query result set and in what order FlowFiles were produced |
| input.flowfile.uuid |
If the processor has an incoming connection, outgoing FlowFiles will have this attribute set to the value of the input FlowFile's UUID. If there is no incoming connection, the attribute will not be added. |
| mime.type |
Sets the mime.type attribute to the MIME Type specified by the Record Writer. |
| record.count |
The number of records output by the Record Writer. |
---
title: ExecuteSQLStatement 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/executesqlstatement.md
section: Loading & Unloading Data
---
# ExecuteSQLStatement 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-database-processors-nar
## Description
Executes a SQL DDL or DML Statement against a database. This Processor allows Expression Language to be evaluated against FlowFile attributes in order to parameterize the SQL for each FlowFile.
## Tags
database, delete, insert, jdbc, openflow, sql, update
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Connection Pooling Service |
The Connection Pooling Service that is used to obtain a connection to the database |
| Max Batch Size |
The maximum number of FlowFiles to process in a single batch |
| Max Content Reference Size |
If the SQL property references $\{flowfile_content\}, this property specifies the maximum size of the FlowFile that is allowed to be read into memory. If the FlowFile is larger than this value, the FlowFile will be routed to failure. If the SQL property does not reference $\{flowfile_content\}, this value has no effect. |
| SQL |
The SQL statement to execute. The SQL may make use of Expression Language to reference attributes. In this case, the Processor will rewrite the query using parameters in order to avoid SQL Injection attacks. When referencing Expression Language, the entire value must be a single Expression. For example, _INSERT INTO TABLE X (name) VALUES ( '$\{name\}')_ is valid, but _INSERT INTO TABLE X (name) VALUES ( 'Mr. $\{name\}')_ is not because Expression Language is used within a String value. The SQL may also reference _$\{flowfile_content\}_ in order to reference the content of the FlowFile as UTF-8 encoded text. |
## Relationships
| Name |
Description |
| failure |
The SQL statement could not be executed |
| success |
The SQL statement was successfully executed |
---
title: ExecuteStreamCommand 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/executestreamcommand.md
section: Loading & Unloading Data
---
# ExecuteStreamCommand 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
The ExecuteStreamCommand processor provides a flexible way to integrate external commands and scripts into NiFi data flows. ExecuteStreamCommand can pass the incoming FlowFile's content to the command that it executes similarly how piping works.
## Tags
command, command execution, execute, stream
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
true
## Properties
| Property |
Description |
| Argument Delimiter |
Delimiter to use to separate arguments for a command [default: ;]. Must be a single character |
| Command Arguments |
The arguments to supply to the executable delimited by the ';' character. |
| Command Path |
Specifies the command to be executed; if just the name of an executable is provided, it must be in the user's environment PATH. |
| Ignore STDIN |
If true, the contents of the incoming flowfile will not be passed to the executing command |
| Max Attribute Length |
If routing the output of the stream command to an attribute, the number of characters put to the attribute value will be at most this amount. This is important because attributes are held in memory and large attributes will quickly cause out of memory issues. If the output goes longer than this value, it will truncated to fit. Consider making this smaller if able. |
| Output Destination Attribute |
If set, the output of the stream command will be put into an attribute of the original FlowFile instead of a separate FlowFile. There will no longer be a relationship for 'output stream' or 'nonzero status'. The value of this property will be the key for the output attribute. |
| Output MIME Type |
Specifies the value to set for the "mime.type" attribute. This property is ignored if 'Output Destination Attribute' is set. |
| Working Directory |
The directory to use as the current working directory when executing the command |
| argumentsStrategy |
Strategy for configuring arguments to be supplied to the command. |
## Restrictions
| Required Permission |
Explanation |
| execute code |
Provides operator the ability to execute arbitrary code assuming all permissions that NiFi has. |
## Relationships
| Name |
Description |
| nonzero status |
The destination path for the flow file created from the command's output, if the returned status code is non-zero. All flow files routed to this relationship will be penalized. |
| original |
The original FlowFile will be routed. It will have new attributes detailing the result of the script execution. |
| output stream |
The destination path for the flow file created from the command's output, if the returned status code is zero. |
## Writes attributes
| Name |
Description |
| execution.command |
The name of the command executed |
| execution.command.args |
The semi-colon delimited list of arguments. Sensitive properties will be masked |
| execution.status |
The exit status code returned from executing the command |
| execution.error |
Any error messages returned from executing the command |
| mime.type |
Sets the MIME type of the output if the 'Output MIME Type' property is set and 'Output Destination Attribute' is not set |
---
title: Explore data products from Salesforce Data Cloud
source: https://docs.snowflake.cn/en/user-guide/data-integration/zero-copy/salesforce/explore-data-products.md
section: Loading & Unloading Data
---
# Explore data products from Salesforce Data Cloud
- [About Salesforce Data Cloud and Snowflake](/user-guide/data-integration/zero-copy/about-salesforce-datacloud)
- [Set up the Salesforce Data Cloud Zerocopy Connector](/user-guide/data-integration/zero-copy/salesforce/setup)
- [Set up Salesforce Data Cloud for Zero-Copy](/user-guide/data-integration/zero-copy/salesforce/setup-salesforce)
- [Salesforce Data Cloud Zerocopy Connector: Security and privileges](/user-guide/data-integration/zero-copy/salesforce/security)
This topic describes how to list available Salesforce data products, mount them as catalog-linked databases, and query the shared data in Snowflake.
Before performing the steps in this topic:
- The Zerocopy Connector must be in `CONNECTED` state. See [Set up the Salesforce Data Cloud Zerocopy Connector](/user-guide/data-integration/zero-copy/salesforce/setup).
- Your Salesforce administrator must have created and linked at least one Data Share to the connector. See [Set up Salesforce Data Cloud for Zero-Copy](/user-guide/data-integration/zero-copy/salesforce/setup-salesforce).
## List shared data products
After your Salesforce administrator links a Data Share to the Snowflake V2 Data Share Target, call `SYSTEM$ZEROCOPY_CONNECTOR_LIST_SHARES` to see what's available:
```sql
SELECT SYSTEM$ZEROCOPY_CONNECTOR_LIST_SHARES('my_db.my_schema.my_sfdc_connector');
```
The function returns a JSON array. Each element represents one shared data product. The `name` field is the value you pass as `SHARE_NAME` or `SHARE_NAME_FILTER` when creating a catalog-linked database.
```text
[
{
"name": "contact_v1",
"status": "UNMOUNTED",
"catalog_linked_databases": []
},
{
"name": "opportunity_v1",
"status": "MOUNTED",
"catalog_linked_databases": [ { "name": "MARKETINGSHARE" } ]
}
]
```
The `status` field indicates whether the data share is available to mount or already mounted:
| Status |
Description |
| `UNMOUNTED` |
Shared by Salesforce Data Cloud. No catalog-linked database has been created yet. |
| `MOUNTED` |
A catalog-linked database exists for this share. |
To parse the output into a tabular format:
```sql
WITH raw AS (
SELECT PARSE_JSON(
SYSTEM$ZEROCOPY_CONNECTOR_LIST_SHARES('my_db.my_schema.my_sfdc_connector')
) AS json_data
)
SELECT
f.value:name::STRING AS share_name,
f.value:status::STRING AS status,
CASE
WHEN ARRAY_SIZE(f.value:catalog_linked_databases) > 0
THEN f.value:catalog_linked_databases[0]:name::STRING
ELSE NULL
END AS mounted_database
FROM raw,
LATERAL FLATTEN(INPUT => json_data) f;
```
## Create a catalog-linked database
Mounting a share creates a catalog-linked database that contains the shared data as queryable schemas. Snowflake automatically creates views on top of them.
You can mount using the Snowsight UI or SQL.
### Using Snowsight
1. In Snowsight, navigate to **Ingestion** %raa% **Zero-Copy**.
2. Select the **Available connectors** tab and click your Salesforce connector.
3. On the **Catalog linked databases** tab, click **Mount all data shares**.
The catalog-linked database is created immediately.
### Using SQL
```sql
-- (Recommended) Mount all shares — each share becomes a schema
CREATE DATABASE my_sfdc_db
LINKED_ZEROCOPY_CONNECTOR = (
CONNECTOR_NAME = 'my_db.my_schema.my_sfdc_connector',
ALL_SHARES = TRUE,
SYNC_INTERVAL_SECONDS = 30 -- optional; controls how often new shares are detected
);
-- Mount a filtered set of shares
CREATE DATABASE my_sfdc_db
LINKED_ZEROCOPY_CONNECTOR = (
CONNECTOR_NAME = 'my_db.my_schema.my_sfdc_connector',
SHARE_NAME_FILTER = ('share1', 'share2')
);
-- Mount a single share
CREATE DATABASE my_sfdc_db
LINKED_ZEROCOPY_CONNECTOR = (
CONNECTOR_NAME = 'my_db.my_schema.my_sfdc_connector',
SHARE_NAME = 'my_share'
);
```
To confirm the database was created:
```sql
SHOW DATABASES LIKE 'MY_SFDC_DB%';
```
## Explore the data
### Data model overview
When you mount a share, each data share appears as a **schema** within the catalog-linked database. Within each schema, Salesforce data objects are exposed as **views**.
### Discover schemas and views
```sql
-- Data shares are mounted as schemas in the catalog-linked database
SHOW SCHEMAS IN DATABASE my_sfdc_db;
-- Views are the queryable layer — use these for all queries
SHOW VIEWS IN SCHEMA my_sfdc_db.my_share_schema;
-- Inspect columns before querying
SHOW COLUMNS IN VIEW my_sfdc_db.my_share_schema.ssot__Account__dlm;
```
Views are created shortly after the catalog-linked database is mounted. If `SHOW VIEWS` returns no results immediately, wait for 1 minute and try again.
### Query the data
Query Salesforce data via the views in each schema. The view names are determined by what your Salesforce administrator included in the Data Share.
```sql
-- Query a Data Lake Object (DLO)
SELECT * FROM my_sfdc_db.my_share_schema.Case_Home__dll LIMIT 10;
-- Query a Data Model Object (DMO)
SELECT * FROM my_sfdc_db.my_share_schema.ssot__PriceBook__dlm LIMIT 10;
-- Query a Calculated Insights Object (CIO)
SELECT * FROM my_sfdc_db.my_share_schema.Product_Sku_Aggregation__cio LIMIT 10;
```
Replace `my_sfdc_db`, `my_share_schema`, and the view names with the actual values returned by `SHOW SCHEMAS` and `SHOW VIEWS` in your environment.
## Create table as select (CTAS)
To persist query results as a native Snowflake table for use in dashboards, ML models, or data sharing:
```sql
CREATE DATABASE IF NOT EXISTS my_ctas_db;
USE DATABASE my_ctas_db;
-- Snapshot a Salesforce data model object into a native Snowflake table
CREATE OR REPLACE TABLE account_snapshot AS
SELECT *
FROM my_sfdc_db.my_share_schema.ssot__Account__dlm;
SELECT * FROM account_snapshot LIMIT 10;
```
## Drop a catalog-linked database
All catalog-linked databases must be dropped before you can disconnect or drop the connector.
Catalog-linked databases do not support `UNDROP`.
```sql
DROP DATABASE my_sfdc_db;
```
---
title: Explore Data Products from SAP® BDC Connect for Snowflake
source: https://docs.snowflake.cn/en/user-guide/data-integration/zero-copy/sap-sql/explore-data-products.md
section: Loading & Unloading Data
---
# Explore Data Products from %sapbdc%
- [About Snowflake and SAP® Zero-Copy Integration](/user-guide/data-integration/zero-copy/about-sap-snowflake)
- [Set Up SAP® BDC Connect for Snowflake Zerocopy Connector](/user-guide/data-integration/zero-copy/sap-sql/setup)
- [SAP® BDC Connect for Snowflake Zerocopy Connector — Security and Privileges](/user-guide/data-integration/zero-copy/sap-sql/security)
This topic describes how to use a Zerocopy Connector to list available SAP®
data products, create catalog-linked databases, and query the shared data in
Snowflake.
The connector must be in `CONNECTED` state before performing any of the
steps in this topic.
## In SAP® BDC, choose data products to share with Snowflake
To search for and share data products with Snowflake, users must use the central SAP Business Data Cloud catalog and have a global role that grants them the following privileges:
- BDC Data Packages (read) - To access SAP Business Data Cloud.
- Catalog Asset (read) - To access the catalog and view objects in the Assets and Data Products collections.
- Cloud Data Product (share) - To share data products to target systems.
Users with these privileges can share data products from the SAP Business Data Cloud catalog with the desired SAP Snowflake account to make them available for consumption to specific roles in that account.
To share data products with Snowflake:
1. In the central SAP Business Data Cloud catalog, select data products to share with an SAP Snowflake account
2. From **Catalog & Marketplace**, search for (or use filters) to find the data products to be shared
3. From the search results, select **Share** in the data product to be shared (for example, customer)
to open the **Manage Share Access** dialog
4. In the **Overview** section, learn more about the data product by reviewing its details and available objects.
5. Under **Target System**:
1. Choose the Snowflake account with the enrolled Zerocopy Connector to share with (if there is more than one).
2. Select **Update**.
A message confirms the share process has started. After it finishes, a notification shows the result.
## In Snowflake, list shared data products
To list the data products that SAP® BDC has shared with your Snowflake account,
call the `SYSTEM$ZEROCOPY_CONNECTOR_LIST_SHARES` function:
```sql
SELECT SYSTEM$ZEROCOPY_CONNECTOR_LIST_SHARES('my_db.my_schema.my_sap_connector');
```
The function returns a JSON array. Each element represents one shared data
product:
```text
[
{
"name": "usid:b077d21c-b7a2-479a-a20e-bba1dbe91034:ns:sap.s4pce:r:SalesOrder:v:1",
"id": "25c0de58-6e61-4bcc-ba68-c2c15b7a2d4b",
"display_name": "Sales Order (BDF730, sap.s4pce:apiResource:SalesOrder:v1)",
"comment": "An agreement between a vendor and a customer to provide products on a specific date.",
"status": "MOUNTED",
"catalog_linked_databases": [ { "name": "SALES_ORDER_CLD" } ],
"properties": {
"sap.ord.apiResource.ordId": "sap.s4pce:apiResource:SalesOrder:v1",
"sap.ord.systemInstance.name": "BDF730",
"sap.ord.systemInstance.id": "30f962e7-791c-41d7-9e72-1534823e8b21"
}
}
]
```
To filter and search more easily, parse the JSON output into a tabular format
using `PARSE_JSON` and `LATERAL FLATTEN`:
```sql
WITH raw AS (
SELECT PARSE_JSON(
SYSTEM$ZEROCOPY_CONNECTOR_LIST_SHARES('my_db.my_schema.my_sap_connector')
) AS json_data
)
SELECT
f.value:name::STRING AS name,
f.value:id::STRING AS id,
f.value:display_name::STRING AS display_name,
f.value:comment::STRING AS comment,
f.value:properties['sap.ord.apiResource.ordId']::STRING AS api_resource_ord_id,
f.value:properties['sap.ord.systemInstance.name']::STRING AS system_instance_name,
f.value:properties['sap.ord.systemInstance.id']::STRING AS system_instance_id
FROM raw,
LATERAL FLATTEN(INPUT => json_data) f;
```
## Create a Catalog-Linked Database
To mount a shared SAP® data product in Snowflake, create a catalog-linked database using the
`LINKED_ZEROCOPY_CONNECTOR` clause. The role requires `CREATE DATABASE` on
the account and `USAGE` on the connector. The owner of the catalog-linked database can be
different from the owner of the connector.
```sql
CREATE DATABASE my_sales_order
LINKED_ZEROCOPY_CONNECTOR = (
CONNECTOR_NAME = 'my_db.my_schema.my_sap_connector',
SHARE_NAME = 'usid:b077d21c-b7a2-479a-a20e-bba1dbe91034:ns:sap.s4pce:r:SalesOrder:v:1',
SYNC_INTERVAL_SECONDS = 86400
);
```
When a catalog-linked database is created, a read-only schema named
`snowflake$` is automatically created within it. This schema contains
[Semantic Views](/user-guide/views-semantic/overview) generated
from the SAP® Core Schema Notation (CSN). Semantic Views add business
meaning to the incoming shared data by defining metrics, entities, and
relationships — enabling consistent business definitions and powering
AI capabilities such as
[Cortex Analyst](/user-guide/snowflake-cortex/cortex-analyst)
directly on top of the SAP® data in Snowflake.
Use `SYNC_INTERVAL_SECONDS` to control how frequently Snowflake
automatically discovers schema and table changes from the shared data
product. The value can range from 30 to 86400 seconds (1 day). The
default value for SAP® BDC is 86400 seconds.
You can create multiple catalog-linked databases from the same connector, one per data product
shared from SAP® BDC.
To confirm the database was created, use [SHOW DATABASES](/sql-reference/sql/show-databases):
```sql
SHOW DATABASES LIKE 'MY_SALES_ORDER%';
```
## Explore the Data
List the schemas and tables available in the catalog-linked database:
```sql
SHOW SCHEMAS IN DATABASE my_sales_order;
SHOW TABLES IN DATABASE my_sales_order;
```
Query the data:
```sql
SELECT * FROM my_sales_order.salesorder.salesorder LIMIT 100;
```
You can join tables across multiple catalog-linked databases. For example, to find the top
customers by revenue using data from two shared data products:
```sql
SELECT
s.salesorder,
s.soldtoparty,
c.customername,
c.country,
s.totalnetamount
FROM my_sales_order.salesorder.salesorder s
JOIN my_customers.customer.customer c
ON s.soldtoparty = c.customer
WHERE s.overallsdprocessingstatus != 'C'
ORDER BY s.totalnetamount DESC
LIMIT 10;
```
## Create Table As Select (CTAS)
To persist query results as a native Snowflake table, use CREATE TABLE AS
SELECT (CTAS). Create a new database to hold the results:
```sql
CREATE DATABASE IF NOT EXISTS my_ctas_db;
USE DATABASE my_ctas_db;
CREATE OR REPLACE TABLE top_customers_by_revenue AS
SELECT
c.customer,
c.customername,
c.country,
c.region,
c.businesstype,
COUNT(DISTINCT s.salesorder) AS num_orders,
SUM(s.totalnetamount) AS total_revenue,
AVG(s.totalnetamount) AS avg_order_amount
FROM my_customers.customer.customer c
JOIN my_sales_order.salesorder.salesorder s
ON c.customer = s.soldtoparty
WHERE c.deletionindicator = FALSE
GROUP BY 1, 2, 3, 4, 5;
-- Query the result table
SELECT * FROM top_customers_by_revenue LIMIT 10;
```
## Drop a Catalog-Linked Database
All catalog-linked databases must be dropped before you can disconnect or drop the connector.
Catalog-linked databases do not support `UNDROP`.
```sql
DROP DATABASE my_sales_order;
```
---
title: ExternalHazelcastCacheManager
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/externalhazelcastcachemanager.md
section: Loading & Unloading Data
---
# ExternalHazelcastCacheManager
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
A service that provides cache instances backed by Hazelcast running outside of NiFi.
## Tags
cache, hazelcast
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Hazelcast Cluster Name * |
hazelcast-cluster-name |
nifi |
|
Name of the Hazelcast cluster. |
| Hazelcast Connection Timeout * |
hazelcast-connection-timeout |
20 secs |
|
The maximum amount of time the client tries to connect or reconnect before giving up. |
| Hazelcast Initial Backoff * |
hazelcast-retry-backoff-initial |
1 secs |
|
The amount of time the client waits before it tries to reestablish connection for the first time. |
| Hazelcast Maximum Backoff * |
hazelcast-retry-backoff-maximum |
5 secs |
|
The maximum amount of time the client waits before it tries to reestablish connection. |
| Hazelcast Backoff Multiplier * |
hazelcast-retry-backoff-multiplier |
1.5 |
|
A multiplier by which the wait time is increased before each attempt to reestablish connection. |
| Hazelcast Server Address * |
hazelcast-server-address |
|
|
Addresses of one or more the Hazelcast instances, using \{host:port\} format, separated by comma. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: ExtractAvroMetadata 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/extractavrometadata.md
section: Loading & Unloading Data
---
# ExtractAvroMetadata 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-avro-nar
## Description
Extracts metadata from the header of an Avro datafile.
## Tags
avro, metadata, schema
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Count Items |
If true the number of items in the datafile will be counted and stored in a FlowFile attribute 'item.count'. The counting is done by reading blocks and getting the number of items for each block, thus avoiding de-serializing. The items being counted will be the top-level items in the datafile. For example, with a schema of type record the items will be the records, and for a schema of type Array the items will be the arrays (not the number of entries in each array). |
| Fingerprint Algorithm |
The algorithm used to generate the schema fingerprint. Available choices are based on the Avro recommended practices for fingerprint generation. |
| Metadata Keys |
A comma-separated list of keys indicating key/value pairs to extract from the Avro file header. The key 'avro.schema' can be used to extract the full schema in JSON format, and 'avro.codec' can be used to extract the codec name if one exists. |
## Relationships
| Name |
Description |
| failure |
A FlowFile is routed to this relationship if it cannot be parsed as Avro or metadata cannot be extracted for any reason |
| success |
A FlowFile is routed to this relationship after metadata has been extracted. |
## Writes attributes
| Name |
Description |
| schema.type |
The type of the schema (i.e. record, enum, etc.). |
| schema.name |
Contains the name when the type is a record, enum or fixed, otherwise contains the name of the primitive type. |
| schema.fingerprint |
The result of the Fingerprint Algorithm as a Hex string. |
| item.count |
The total number of items in the datafile, only written if Count Items is set to true. |
---
title: ExtractEmailAttachments 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/extractemailattachments.md
section: Loading & Unloading Data
---
# ExtractEmailAttachments 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-email-nar
## Description
Extract attachments from a mime formatted email file, splitting them into individual flowfiles.
## Tags
email, split
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Relationships
| Name |
Description |
| attachments |
Each individual attachment will be routed to the attachments relationship |
| failure |
FlowFiles that could not be parsed |
| original |
The original file |
## Writes attributes
| Name |
Description |
| filename |
The filename of the attachment |
| email.attachment.parent.filename |
The filename of the parent FlowFile |
| email.attachment.parent.uuid |
The UUID of the original FlowFile. |
| mime.type |
The mime type of the attachment. |
---
title: ExtractEmailHeaders 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/extractemailheaders.md
section: Loading & Unloading Data
---
# ExtractEmailHeaders 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-email-nar
## Description
Using the flowfile content as source of data, extract header from an RFC compliant email file adding the relevant attributes to the flowfile. This processor does not perform extensive RFC validation but still requires a bare minimum compliance with RFC 2822
## Tags
email, split
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Additional Header List |
COLON separated list of additional headers to be extracted from the flowfile content. NOTE the header key is case insensitive and will be matched as lower-case. Values will respect email contents. |
| Email Address Parsing |
If "strict", strict address format parsing rules are applied to mailbox and mailbox list fields, such as "to" and "from" headers, and FlowFiles with poorly formed addresses will be routed to the failure relationship, similar to messages that fail RFC compliant format validation. If "non-strict", the processor will extract the contents of mailbox list headers as comma-separated values without attempting to parse each value as well-formed Internet mailbox addresses. This is optional and defaults to Strict Address Parsing |
## Relationships
| Name |
Description |
| failure |
Flowfiles that could not be parsed as a RFC-2822 compliant message |
| success |
Extraction was successful |
## Writes attributes
| Name |
Description |
| email.headers.bcc.* |
Each individual BCC recipient (if available) |
| email.headers.cc.* |
Each individual CC recipient (if available) |
| email.headers.from.* |
Each individual mailbox contained in the From of the Email (array as per RFC-2822) |
| email.headers.message-id |
The value of the Message-ID header (if available) |
| email.headers.received_date |
The Received-Date of the message (if available) |
| email.headers.sent_date |
Date the message was sent |
| email.headers.subject |
Subject of the message (if available) |
| email.headers.to.* |
Each individual TO recipient (if available) |
| email.attachment_count |
Number of attachments of the message |
---
title: ExtractGrok 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/extractgrok.md
section: Loading & Unloading Data
---
# ExtractGrok 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Evaluates one or more Grok Expressions against the content of a FlowFile, adding the results as attributes or replacing the content of the FlowFile with a JSON notation of the matched content
## Tags
delimit, extract, grok, log, parse, text
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Character Set |
The Character Set in which the file is encoded |
| Destination |
Control if Grok output value is written as a new flowfile attributes, in this case each of the Grok identifier that is matched in the flowfile will be added as an attribute, prefixed with "grok." or written in the flowfile content. Writing to flowfile content will overwrite any existing flowfile content. |
| Grok Expression |
Grok expression. If other Grok expressions are referenced in this expression, they must be provided in the Grok Pattern File if set or exist in the default Grok patterns |
| Grok Pattern file |
Custom Grok pattern definitions. These definitions will be loaded after the default Grok patterns. The Grok Parser will use the default Grok patterns when this property is not configured. |
| Keep Empty Captures |
If true, then empty capture values will be included in the returned capture map. |
| Maximum Buffer Size |
Specifies the maximum amount of data to buffer (per file) in order to apply the Grok expressions. Files larger than the specified maximum will not be fully evaluated. |
| Named captures only |
Only store named captures from grok |
## Restrictions
| Required Permission |
Explanation |
| reference remote resources |
Patterns can reference resources over HTTP |
## Relationships
| Name |
Description |
| matched |
FlowFiles are routed to this relationship when the Grok Expression is successfully evaluated and the FlowFile is modified as a result |
| unmatched |
FlowFiles are routed to this relationship when no provided Grok Expression matches the content of the FlowFile |
## Writes attributes
| Name |
Description |
| grok.XXX |
When operating in flowfile-attribute mode, each of the Grok identifier that is matched in the flowfile will be added as an attribute, prefixed with "grok." For example,if the grok identifier "timestamp" is matched, then the value will be added to an attribute named "grok.timestamp" |
---
title: ExtractRecordSchema 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/extractrecordschema.md
section: Loading & Unloading Data
---
# ExtractRecordSchema 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Extracts the record schema from the FlowFile using the supplied Record Reader and writes it to the *avro.schema* attribute.
## Tags
avro, csv, freeform, generic, json, record, schema, text, xml
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| cache-size |
Specifies the number of schemas to cache. This value should reflect the expected number of different schemas that may be in the incoming FlowFiles. This ensures more efficient retrieval of the schemas and thus the processor performance. |
| record-reader |
Specifies the Controller Service to use for reading incoming data |
## Relationships
| Name |
Description |
| failure |
If a FlowFile's record schema cannot be extracted from the configured input format, the FlowFile will be routed to this relationship |
| success |
FlowFiles whose record schemas are successfully extracted will be routed to this relationship |
## Writes attributes
| Name |
Description |
| record.error.message |
This attribute provides on failure the error message encountered by the Reader. |
| avro.schema |
This attribute provides the schema extracted from the input FlowFile using the provided RecordReader. |
---
title: ExtractSchemaColumns 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/extractschemacolumns.md
section: Loading & Unloading Data
---
# ExtractSchemaColumns 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-record-schema-nar
## Description
Extracts the record schema columns from the FlowFile using the supplied Record Reader and writes it to the *schema.columns* attribute.
## Tags
avro, csv, freeform, generic, json, record, schema, text, xml
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| End Column Index |
Specifies index of the column in schema to which columns should be taken. |
| Record Reader |
Specifies the Controller Service to use for reading incoming data |
| Start Column Index |
Specifies index of the column (numbered from 1) in schema from which columns should be taken. |
## Relationships
| Name |
Description |
| failure |
If a FlowFile's record schema cannot be extracted from the configured input format, the FlowFile will be routed to this relationship |
| success |
FlowFiles whose record schemas are successfully extracted will be routed to this relationship |
## Writes attributes
| Name |
Description |
| record.error.message |
This attribute provides on failure the error message encountered by the Reader. |
| schema.columns |
This attribute provides columns extracted from the input FlowFile using the provided RecordReader. |
---
title: ExtractStructuredBoxFileMetadata 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/extractstructuredboxfilemetadata.md
section: Loading & Unloading Data
---
# ExtractStructuredBoxFileMetadata 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-box-nar
## Description
Extracts metadata from a Box file using Box AI. The extraction can use either a template or a list of fields. The extracted metadata is written to the FlowFile content as JSON.
## Tags
ai, box, extract, metadata, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Box Client Service |
Controller Service used to obtain a Box API connection. |
| Extraction Method |
The method to use for extracting metadata. TEMPLATE uses a Box metadata template for extraction. FIELDS uses a JSON schema of fields (read from FlowFile content) for extraction. |
| File ID |
The ID of the file from which to extract metadata. |
| Record Reader |
The Record Reader to use for parsing the incoming data. Required when Extraction Method is FIELDS. |
| Template Key |
The key of the metadata template to use for extraction. Required when Extraction Method is TEMPLATE. |
## Relationships
| Name |
Description |
| failure |
A FlowFile is routed to this relationship if an error occurs during metadata extraction. |
| file not found |
FlowFiles for which the specified Box file was not found will be routed to this relationship. |
| success |
A FlowFile is routed to this relationship after metadata has been successfully extracted. |
| template not found |
FlowFiles for which the specified metadata template was not found will be routed to this relationship. |
## Writes attributes
| Name |
Description |
| box.id |
The ID of the file from which metadata was extracted |
| box.ai.template.key |
The template key used for extraction (when using TEMPLATE extraction method) |
| box.ai.extraction.method |
The extraction method used (TEMPLATE or FIELDS) |
| box.ai.completion.reason |
The completion reason from the AI extraction |
| mime.type |
Set to 'application/json' for the JSON content |
| error.code |
The error code returned by Box |
| error.message |
The error message returned by Box |
## See also
- [org.apache.nifi.processors.box.FetchBoxFile](/user-guide/data-integration/openflow/processors/fetchboxfile)
- [org.apache.nifi.processors.box.ListBoxFile](/user-guide/data-integration/openflow/processors/listboxfile)
- [org.apache.nifi.processors.box.ListBoxFileMetadataTemplates](/user-guide/data-integration/openflow/processors/listboxfilemetadatatemplates)
- [org.apache.nifi.processors.box.UpdateBoxFileMetadataInstance](/user-guide/data-integration/openflow/processors/updateboxfilemetadatainstance)
---
title: ExtractText 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/extracttext.md
section: Loading & Unloading Data
---
# ExtractText 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Evaluates one or more Regular Expressions against the content of a FlowFile. The results of those Regular Expressions are assigned to FlowFile Attributes. Regular Expressions are entered by adding user-defined properties; the name of the property maps to the Attribute Name into which the result will be placed. The attributes are generated differently based on the enabling of named capture groups. If named capture groups are not enabled: The first capture group, if any found, will be placed into that attribute name. But all capture groups, including the matching string sequence itself will also be provided at that attribute name with an index value provided, with the exception of a capturing group that is optional and does not match - for example, given the attribute name "regex" and expression "abc(def)?(g)" we would add an attribute "regex.1" with a value of "def" if the "def" matched. If the "def" did not match, no attribute named "regex.1" would be added but an attribute named "regex.2" with a value of "g" will be added regardless. If named capture groups are enabled: Each named capture group, if found will be placed into the attributes name with the name provided. If enabled the matching string sequence itself will be placed into the attribute name. If multiple matches are enabled, and index will be applied after the first set of matches. The exception is a capturing group that is optional and does not match For example, given the attribute name "regex" and expression "abc(?<NAMED>def)?(?<NAMED-TWO>g)" we would add an attribute "regex. NAMED" with the value of "def" if the "def" matched. We would add an attribute "regex. NAMED-TWO" with the value of "g" if the "g" matched regardless. The value of the property must be a valid Regular Expressions with one or more capturing groups. If named capture groups are enabled, all capture groups must be named. If they are not, then the processor configuration will fail validation. If the Regular Expression matches more than once, only the first match will be used unless the property enabling repeating capture group is set to true. If any provided Regular Expression matches, the FlowFile(s) will be routed to 'matched'. If no provided Regular Expression matches, the FlowFile will be routed to 'unmatched' and no attributes will be applied to the FlowFile.
## Tags
Regular Expression, Text, evaluate, extract, regex
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Character Set |
The Character Set in which the file is encoded |
| Enable Canonical Equivalence |
Indicates that two characters match only when their full canonical decompositions match. |
| Enable Case-insensitive Matching |
Indicates that two characters match even if they are in a different case. Can also be specified via the embedded flag (?i). |
| Enable DOTALL Mode |
Indicates that the expression '.' should match any character, including a line terminator. Can also be specified via the embedded flag (?s). |
| Enable Literal Parsing of the Pattern |
Indicates that Metacharacters and escape characters should be given no special meaning. |
| Enable Multiline Mode |
Indicates that '^' and '$' should match just after and just before a line terminator or end of sequence, instead of only the beginning or end of the entire input. Can also be specified via the embedded flag (?m). |
| Enable Unicode Predefined Character Classes |
Specifies conformance with the Unicode Technical Standard #18: Unicode Regular Expression Annex C: Compatibility Properties. Can also be specified via the embedded flag (?U). |
| Enable Unicode-aware Case Folding |
When used with 'Enable Case-insensitive Matching', matches in a manner consistent with the Unicode Standard. Can also be specified via the embedded flag (?u). |
| Enable Unix Lines Mode |
Indicates that only the 'line terminator is recognized in the behavior of'. ','^ ', and'$'. Can also be specified via the embedded flag (?d). |
| Enable named group support |
If set to true, when named groups are present in the regular expression, the name of the group will be used in the attribute name as opposed to the group index. All capturing groups must be named, if the number of groups (not including capture group 0) does not equal the number of named groups validation will fail. |
| Enable repeating capture group |
If set to true, every string matching the capture groups will be extracted. Otherwise, if the Regular Expression matches more than once, only the first match will be extracted. |
| Include Capture Group 0 |
Indicates that Capture Group 0 should be included as an attribute. Capture Group 0 represents the entirety of the regular expression match, is typically not used, and could have considerable length. |
| Maximum Buffer Size |
Specifies the maximum amount of data to buffer (per FlowFile) in order to apply the regular expressions. FlowFiles larger than the specified maximum will not be fully evaluated. |
| Maximum Capture Group Length |
Specifies the maximum number of characters a given capture group value can have. Any characters beyond the max will be truncated. |
| Permit Whitespace and Comments in Pattern |
In this mode, whitespace is ignored, and embedded comments starting with # are ignored until the end of a line. Can also be specified via the embedded flag (?x). |
## Relationships
| Name |
Description |
| matched |
FlowFiles are routed to this relationship when the Regular Expression is successfully evaluated and the FlowFile is modified as a result |
| unmatched |
FlowFiles are routed to this relationship when no provided Regular Expression matches the content of the FlowFile |
---
title: FetchAzureBlobStorage_v12 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchazureblobstorage_v12.md
section: Loading & Unloading Data
---
# FetchAzureBlobStorage_v12 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-azure-nar
## Description
Retrieves the specified blob from Azure Blob Storage and writes its content to the content of the FlowFile. The processor uses Azure Blob Storage client library v12.
## Tags
azure, blob, cloud, microsoft, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Blob Name |
The full name of the blob |
| Client-Side Encryption Key ID |
Specifies the ID of the key to use for client-side encryption. |
| Client-Side Encryption Key Type |
Specifies the key type to use for client-side encryption. |
| Client-Side Encryption Local Key |
When using local client-side encryption, this is the raw key, encoded in hexadecimal |
| Container Name |
Name of the Azure storage container. In case of PutAzureBlobStorage processor, container can be created if it does not exist. |
| Range Length |
The number of bytes to download from the blob, starting from the Range Start. An empty value or a value that extends beyond the end of the blob will read to the end of the blob. |
| Range Start |
The byte position at which to start reading from the blob. An empty value or a value of zero will start reading at the beginning of the blob. |
| Storage Credentials |
Controller Service used to obtain Azure Blob Storage Credentials. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. In case of SOCKS, it is not guaranteed that the selected SOCKS Version will be used by the processor. |
## Relationships
| Name |
Description |
| failure |
Unsuccessful operations will be transferred to the failure relationship. |
| success |
All successfully processed FlowFiles are routed to this relationship |
## Writes attributes
| Name |
Description |
| azure.container |
The name of the Azure Blob Storage container |
| azure.blobname |
The name of the blob on Azure Blob Storage |
| azure.primaryUri |
Primary location of the blob |
| azure.etag |
ETag of the blob |
| azure.blobtype |
Type of the blob (either BlockBlob, PageBlob or AppendBlob) |
| mime.type |
MIME Type of the content |
| lang |
Language code for the content |
| azure.timestamp |
Timestamp of the blob |
| azure.length |
Length of the blob |
## Use Cases Involving Other Components
| Retrieve all files in an Azure Blob Storage container |
| ----------------------------------------------------- |
## See also
- [org.apache.nifi.processors.azure.storage.DeleteAzureBlobStorage_v12](/user-guide/data-integration/openflow/processors/deleteazureblobstorage_v12)
- [org.apache.nifi.processors.azure.storage.ListAzureBlobStorage_v12](/user-guide/data-integration/openflow/processors/listazureblobstorage_v12)
- [org.apache.nifi.processors.azure.storage.PutAzureBlobStorage_v12](/user-guide/data-integration/openflow/processors/putazureblobstorage_v12)
---
title: FetchAzureDataLakeStorage 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchazuredatalakestorage.md
section: Loading & Unloading Data
---
# FetchAzureDataLakeStorage 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-azure-nar
## Description
Fetch the specified file from Azure Data Lake Storage
## Tags
adlsgen2, azure, cloud, datalake, microsoft, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| ADLS Credentials |
Controller Service used to obtain Azure Credentials. |
| Directory Name |
Name of the Azure Storage Directory. The Directory Name cannot contain a leading '/'. The root directory can be designated by the empty string value. In case of the PutAzureDataLakeStorage processor, the directory will be created if not already existing. |
| File Name |
The filename |
| Filesystem Name |
Name of the Azure Storage File System (also called Container). It is assumed to be already existing. |
| Number of Retries |
The number of automatic retries to perform if the download fails. |
| Range Length |
The number of bytes to download from the object, starting from the Range Start. An empty value or a value that extends beyond the end of the object will read to the end of the object. |
| Range Start |
The byte position at which to start reading from the object. An empty value or a value of zero will start reading at the beginning of the object. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. In case of SOCKS, it is not guaranteed that the selected SOCKS Version will be used by the processor. |
## Relationships
| Name |
Description |
| failure |
Files that could not be written to Azure storage for some reason are transferred to this relationship |
| success |
Files that have been successfully written to Azure storage are transferred to this relationship |
## Writes attributes
| Name |
Description |
| azure.datalake.storage.statusCode |
The HTTP error code (if available) from the failed operation |
| azure.datalake.storage.errorCode |
The Azure Data Lake Storage moniker of the failed operation |
| azure.datalake.storage.errorMessage |
The Azure Data Lake Storage error message from the failed operation |
## Use Cases Involving Other Components
| Retrieve all files in an Azure DataLake Storage directory |
| --------------------------------------------------------- |
## See also
- [org.apache.nifi.processors.azure.storage.DeleteAzureDataLakeStorage](/user-guide/data-integration/openflow/processors/deleteazuredatalakestorage)
- [org.apache.nifi.processors.azure.storage.ListAzureDataLakeStorage](/user-guide/data-integration/openflow/processors/listazuredatalakestorage)
- [org.apache.nifi.processors.azure.storage.PutAzureDataLakeStorage](/user-guide/data-integration/openflow/processors/putazuredatalakestorage)
---
title: FetchBoxFile 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchboxfile.md
section: Loading & Unloading Data
---
# FetchBoxFile 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-box-nar
## Description
Fetches files from a Box Folder. Designed to be used in tandem with ListBoxFile.
## Tags
box, fetch, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Box Client Service |
Controller Service used to obtain a Box API connection. |
| File ID |
The ID of the File to fetch |
## Relationships
| Name |
Description |
| failure |
A FlowFile will be routed here for each File for which fetch was attempted but failed. |
| success |
A FlowFile will be routed here for each successfully fetched File. |
## Writes attributes
| Name |
Description |
| box.id |
The id of the file |
| filename |
The name of the file |
| path |
The folder path where the file is located |
| box.size |
The size of the file |
| box.timestamp |
The last modified time of the file |
| error.code |
The error code returned by Box |
| error.message |
The error message returned by Box |
## See also
- [org.apache.nifi.processors.box.ListBoxFile](/user-guide/data-integration/openflow/processors/listboxfile)
- [org.apache.nifi.processors.box.PutBoxFile](/user-guide/data-integration/openflow/processors/putboxfile)
---
title: FetchBoxFileInfo 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchboxfileinfo.md
section: Loading & Unloading Data
---
# FetchBoxFileInfo 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-box-nar
## Description
Fetches metadata for files from Box and adds it to the FlowFile's attributes.
## Tags
box, fetch, metadata, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Box Client Service |
Controller Service used to obtain a Box API connection. |
| File ID |
The ID of the File to fetch metadata for |
## Relationships
| Name |
Description |
| failure |
A FlowFile will be routed here if fetching the file metadata fails. |
| not.found |
FlowFiles for which the specified Box file was not found. |
| success |
A FlowFile will be routed here after successfully fetching the file metadata. |
## Writes attributes
| Name |
Description |
| box.id |
The id of the file |
| filename |
The name of the file |
| path |
The folder path where the file is located |
| box.path.folder.ids |
A comma separated list of file path_collection IDs |
| box.size |
The size of the file |
| box.timestamp |
The last modified time of the file |
| box.created.at |
The creation date of the file |
| box.owner |
The name of the file owner |
| box.owner.id |
The ID of the file owner |
| box.owner.login |
The login of the file owner |
| box.description |
The description of the file |
| box.etag |
The etag of the file |
| box.sha1 |
The SHA-1 hash of the file |
| box.content.created.at |
The date the content was created |
| box.content.modified.at |
The date the content was modified |
| box.item.status |
The status of the file (active, trashed, etc.) |
| box.sequence_id |
The sequence ID of the file |
| box.parent.folder.id |
The ID of the parent folder |
| box.trashed.at |
The date the file was trashed, if applicable |
| box.purged.at |
The date the file was purged, if applicable |
| box.shared.link |
The shared link of the file, if any |
| error.code |
The error code returned by Box |
| error.message |
The error message returned by Box |
## See also
- [org.apache.nifi.processors.box.FetchBoxFile](/user-guide/data-integration/openflow/processors/fetchboxfile)
- [org.apache.nifi.processors.box.ListBoxFile](/user-guide/data-integration/openflow/processors/listboxfile)
- [org.apache.nifi.processors.box.PutBoxFile](/user-guide/data-integration/openflow/processors/putboxfile)
---
title: FetchBoxFileMetadataInstance 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchboxfilemetadatainstance.md
section: Loading & Unloading Data
---
# FetchBoxFileMetadataInstance 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-box-nar
## Description
Retrieves specific metadata instance associated with a Box file using template key and scope.
## Tags
box, instance, metadata, storage, template
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Box Client Service |
Controller Service used to obtain a Box API connection. |
| File ID |
The ID of the file for which to fetch metadata. |
| Template Key |
The metadata template key to retrieve. |
| Template Scope |
The metadata template scope (e.g., 'enterprise', 'global'). |
## Relationships
| Name |
Description |
| failure |
A FlowFile will be routed here if there is an error fetching metadata instance from the file. |
| file not found |
FlowFiles for which the specified Box file was not found will be routed to this relationship. |
| success |
A FlowFile containing the metadata instance will be routed to this relationship upon successful processing. |
| template not found |
FlowFiles for which the specified metadata template was not found will be routed to this relationship. |
## Writes attributes
| Name |
Description |
| box.id |
The ID of the file from which metadata was fetched |
| box.metadata.template.key |
The metadata template key |
| box.metadata.template.scope |
The metadata template scope |
| mime.type |
The MIME Type of the FlowFile content |
| error.code |
The error code returned by Box |
| error.message |
The error message returned by Box |
## See also
- [org.apache.nifi.processors.box.FetchBoxFile](/user-guide/data-integration/openflow/processors/fetchboxfile)
- [org.apache.nifi.processors.box.FetchBoxFileInfo](/user-guide/data-integration/openflow/processors/fetchboxfileinfo)
- [org.apache.nifi.processors.box.ListBoxFile](/user-guide/data-integration/openflow/processors/listboxfile)
- [org.apache.nifi.processors.box.ListBoxFileMetadataInstances](/user-guide/data-integration/openflow/processors/listboxfilemetadatainstances)
---
title: FetchBoxFileRepresentation 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchboxfilerepresentation.md
section: Loading & Unloading Data
---
# FetchBoxFileRepresentation 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-box-nar
## Description
Fetches a Box file representation using a representation hint and writes it to the FlowFile content.
## Tags
box, cloud, content, download, file, representation, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Box Client Service |
Controller Service used to obtain a Box API connection. |
| File ID |
The ID of the Box file to retrieve. |
| Representation Type |
The type of representation to fetch. Common values include 'pdf', 'text', 'jpg', 'png', etc. |
## Relationships
| Name |
Description |
| failure |
FlowFiles that encounter errors during processing will be routed to this relationship. |
| file.not.found |
FlowFiles for which the specified Box file was not found. |
| representation.not.found |
FlowFiles for which the specified Box file's requested representation was not found. |
| success |
FlowFiles that are successfully processed will be routed to this relationship. |
## Writes attributes
| Name |
Description |
| box.id |
The ID of the Box file. |
| box.file.name |
The name of the Box file. |
| box.file.size |
The size of the Box file in bytes. |
| box.file.created.time |
The timestamp when the file was created. |
| box.file.modified.time |
The timestamp when the file was last modified. |
| box.file.mime.type |
The MIME type of the file. |
| box.file.representation.type |
The representation type that was fetched. |
| box.error.message |
The error message returned by Box if the operation fails. |
| box.error.code |
The error code returned by Box if the operation fails. |
## See also
- [org.apache.nifi.processors.box.FetchBoxFile](/user-guide/data-integration/openflow/processors/fetchboxfile)
- [org.apache.nifi.processors.box.ListBoxFile](/user-guide/data-integration/openflow/processors/listboxfile)
---
title: FetchDistributedMapCache 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchdistributedmapcache.md
section: Loading & Unloading Data
---
# FetchDistributedMapCache 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Computes cache key(s) from FlowFile attributes, for each incoming FlowFile, and fetches the value(s) from the Distributed Map Cache associated with each key. If configured without a destination attribute, the incoming FlowFile 's content is replaced with the binary data received by the Distributed Map Cache. If there is no value stored under that key then the flow file will be routed to' not-found '. Note that the processor will always attempt to read the entire cached value into memory before placing it in it's destination. This could be potentially problematic if the cached value is very large.
## Tags
cache, distributed, fetch, map
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Cache Entry Identifier |
A comma-delimited list of FlowFile attributes, or the results of Attribute Expression Language statements, which will be evaluated against a FlowFile in order to determine the value(s) used to identify duplicates; it is these values that are cached. NOTE: Only a single Cache Entry Identifier is allowed unless Put Cache Value In Attribute is specified. Multiple cache lookups are only supported when the destination is a set of attributes (see the documentation for 'Put Cache Value In Attribute' for more details including naming convention. |
| Character Set |
The Character Set in which the cached value is encoded. This will only be used when routing to an attribute. |
| Distributed Cache Service |
The Controller Service that is used to get the cached values. |
| Max Length To Put In Attribute |
If routing the cache value to an attribute of the FlowFile (by setting the "Put Cache Value in attribute" property), the number of characters put to the attribute value will be at most this amount. This is important because attributes are held in memory and large attributes will quickly cause out of memory issues. If the output goes longer than this value, it will be truncated to fit. Consider making this smaller if able. |
| Put Cache Value In Attribute |
If set, the cache value received will be put into an attribute of the FlowFile instead of a the content of theFlowFile. The attribute key to put to is determined by evaluating value of this property. If multiple Cache Entry Identifiers are selected, multiple attributes will be written, using the evaluated value of this property, appended by a period (.) and the name of the cache entry identifier. |
## Relationships
| Name |
Description |
| failure |
If unable to communicate with the cache or if the cache entry is evaluated to be blank, the FlowFile will be penalized and routed to this relationship |
| not-found |
If a FlowFile's Cache Entry Identifier was not found in the cache, it will be routed to this relationship |
| success |
If the cache was successfully communicated with it will be routed to this relationship |
## Writes attributes
| Name |
Description |
| user-defined |
If the 'Put Cache Value In Attribute' property is set then whatever it is set to will become the attribute key and the value would be whatever the response was from the Distributed Map Cache. If multiple cache entry identifiers are selected, multiple attributes will be written, using the evaluated value of this property, appended by a period (.) and the name of the cache entry identifier. For example, if the Cache Entry Identifier property is set to 'id,name', and the user-defined property is named 'fetched', then two attributes will be written, fetched.id and fetched.name, containing their respective values. |
## See also
- [org.apache.nifi.processors.standard.PutDistributedMapCache](/user-guide/data-integration/openflow/processors/putdistributedmapcache)
---
title: FetchDropbox 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchdropbox.md
section: Loading & Unloading Data
---
# FetchDropbox 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-dropbox-processors-nar
## Description
Fetches files from Dropbox. Designed to be used in tandem with ListDropbox.
## Tags
dropbox, fetch, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Dropbox Credential Service |
Controller Service used to obtain Dropbox credentials (App Key, App Secret, Access Token, Refresh Token). See controller service's Additional Details for more information. |
| File |
The Dropbox identifier or path of the Dropbox file to fetch. The 'File'should match the following regular expression pattern: /.*|id:.* . When ListDropbox is used for input, either '$\{dropbox.id\}' (identifying files by Dropbox id) or '$\{path\}/$\{filename\}' (identifying files by path) can be used as 'File' value. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| failure |
A FlowFile will be routed here for each File for which fetch was attempted but failed. |
| success |
A FlowFile will be routed here for each successfully fetched File. |
## Writes attributes
| Name |
Description |
| error.message |
The error message returned by Dropbox |
| dropbox.id |
The Dropbox identifier of the file |
| path |
The folder path where the file is located |
| filename |
The name of the file |
| dropbox.size |
The size of the file |
| dropbox.timestamp |
The server modified time of the file |
| dropbox.revision |
Revision of the file |
## See also
- [org.apache.nifi.processors.dropbox.ListDropbox](/user-guide/data-integration/openflow/processors/listdropbox)
- [org.apache.nifi.processors.dropbox.PutDropbox](/user-guide/data-integration/openflow/processors/putdropbox)
---
title: FetchFile 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchfile.md
section: Loading & Unloading Data
---
# FetchFile 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Reads the contents of a file from disk and streams it into the contents of an incoming FlowFile. Once this is done, the file is optionally moved elsewhere or deleted to help keep the file system organized.
## Tags
fetch, files, filesystem, get, ingest, ingress, input, local, source
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Completion Strategy |
Specifies what to do with the original file on the file system once it has been pulled into NiFi |
| File to Fetch |
The fully-qualified filename of the file to fetch from the file system |
| Log level when file not found |
Log level to use in case the file does not exist when the processor is triggered |
| Log level when permission denied |
Log level to use if the current application user does not have sufficient permissions to read the file |
| Move Conflict Strategy |
If Completion Strategy is set to Move File and a file already exists in the destination directory with the same name, this property specifies how that naming conflict should be resolved |
| Move Destination Directory |
The directory to the move the original file to once it has been fetched from the file system. This property is ignored unless the Completion Strategy is set to "Move File". If the directory does not exist, it will be created. |
## Restrictions
| Required Permission |
Explanation |
| read filesystem |
Provides operator the ability to read from any file that NiFi has access to. |
| write filesystem |
Provides operator the ability to delete any file that NiFi has access to. |
## Relationships
| Name |
Description |
| failure |
Any FlowFile that could not be fetched from the file system for any reason other than insufficient permissions or the file not existing will be transferred to this Relationship. |
| not.found |
Any FlowFile that could not be fetched from the file system because the file could not be found will be transferred to this Relationship. |
| permission.denied |
Any FlowFile that could not be fetched from the file system due to the user running NiFi not having sufficient permissions will be transferred to this Relationship. |
| success |
Any FlowFile that is successfully fetched from the file system will be transferred to this Relationship. |
## Use Cases Involving Other Components
| Ingest all files from a directory into NiFi |
| ----------------------------------------------------------------------- |
| Ingest specific files from a directory into NiFi, filtering on filename |
## See also
- [org.apache.nifi.processors.standard.GetFile](/user-guide/data-integration/openflow/processors/getfile)
- [org.apache.nifi.processors.standard.ListFile](/user-guide/data-integration/openflow/processors/listfile)
- [org.apache.nifi.processors.standard.PutFile](/user-guide/data-integration/openflow/processors/putfile)
---
title: FetchFTP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchftp.md
section: Loading & Unloading Data
---
# FetchFTP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Fetches the content of a file from a remote FTP server and overwrites the contents of an incoming FlowFile with the content of the remote file.
## Tags
fetch, files, ftp, get, ingest, input, remote, retrieve, source
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Completion Strategy |
Specifies what to do with the original file on the server once it has been pulled into NiFi. If the Completion Strategy fails, a warning will be logged but the data will still be transferred. |
| Connection Mode |
The FTP Connection Mode |
| Connection Timeout |
Amount of time to wait before timing out while creating a connection |
| Create Directory |
Used when 'Completion Strategy' is 'Move File'. Specifies whether or not the remote directory should be created if it does not exist. |
| Data Timeout |
When transferring a file between the local and remote system, this value specifies how long is allowed to elapse without any data being transferred between systems |
| Hostname |
The fully-qualified hostname or IP address of the host to fetch the data from |
| Internal Buffer Size |
Set the internal buffer size for buffered data streams |
| Log Level When File Not Found |
Log level to use in case the file does not exist when the processor is triggered |
| Move Destination Directory |
The directory on the remote server to move the original file to once it has been ingested into NiFi. This property is ignored unless the Completion Strategy is set to 'Move File'. The specified directory must already exist on the remote system if 'Create Directory' is disabled, or the rename will fail. |
| Password |
Password for the user account |
| Port |
The port to connect to on the remote host to fetch the data from |
| Remote File |
The fully qualified filename on the remote system |
| Transfer Mode |
The FTP Transfer Mode |
| Use Compression |
Indicates whether or not ZLIB compression should be used when transferring files |
| Username |
Username |
| ftp-use-utf8 |
Tells the client to use UTF-8 encoding when processing files and filenames. If set to true, the server must also support UTF-8 encoding. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| comms.failure |
Any FlowFile that could not be fetched from the remote server due to a communications failure will be transferred to this Relationship. |
| not.found |
Any FlowFile for which we receive a 'Not Found' message from the remote server will be transferred to this Relationship. |
| permission.denied |
Any FlowFile that could not be fetched from the remote server due to insufficient permissions will be transferred to this Relationship. |
| success |
All FlowFiles that are received are routed to success |
## Writes attributes
| Name |
Description |
| ftp.remote.host |
The hostname or IP address from which the file was pulled |
| ftp.remote.port |
The port that was used to communicate with the remote FTP server |
| ftp.remote.filename |
The name of the remote file that was pulled |
| filename |
The filename is updated to point to the filename fo the remote file |
| path |
If the Remote File contains a directory name, that directory name will be added to the FlowFile using the 'path' attribute |
| fetch.failure.reason |
The name of the failure relationship applied when routing to any failure relationship |
## Use Cases Involving Other Components
| Retrieve all files in a directory of an FTP Server |
| -------------------------------------------------- |
## See also
- [org.apache.nifi.processors.standard.GetFTP](/user-guide/data-integration/openflow/processors/getftp)
- [org.apache.nifi.processors.standard.GetSFTP](/user-guide/data-integration/openflow/processors/getsftp)
- [org.apache.nifi.processors.standard.PutFTP](/user-guide/data-integration/openflow/processors/putftp)
- [org.apache.nifi.processors.standard.PutSFTP](/user-guide/data-integration/openflow/processors/putsftp)
---
title: FetchGCSObject 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchgcsobject.md
section: Loading & Unloading Data
---
# FetchGCSObject 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-gcp-nar
## Description
Fetches a file from a Google Cloud Bucket. Designed to be used in tandem with ListGCSBucket.
## Tags
fetch, gcs, google, google cloud, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| GCP Credentials Provider Service |
The Controller Service used to obtain Google Cloud Platform credentials. |
| gcp-project-id |
Google Cloud Project ID |
| gcp-retry-count |
How many retry attempts should be made before routing to the failure relationship. |
| gcs-bucket |
Bucket of the object. |
| gcs-generation |
The generation of the Object to download. If not set, the latest generation will be downloaded. |
| gcs-key |
Name of the object. |
| gcs-object-range-length |
The number of bytes to download from the object, starting from the Range Start. An empty value or a value that extends beyond the end of the object will read to the end of the object. |
| gcs-object-range-start |
The byte position at which to start reading from the object. An empty value or a value of zero will start reading at the beginning of the object. |
| gcs-server-side-encryption-key |
An AES256 Key (encoded in base64) which the object has been encrypted in. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
| storage-api-url |
Overrides the default storage URL. Configuring an alternative Storage API URL also overrides the HTTP Host header on requests as described in the Google documentation for Private Service Connections. |
## Relationships
| Name |
Description |
| failure |
FlowFiles are routed to this relationship if the Google Cloud Storage operation fails. |
| success |
FlowFiles are routed to this relationship after a successful Google Cloud Storage operation. |
## Writes attributes
| Name |
Description |
| filename |
The name of the file, parsed if possible from the Content-Disposition response header |
| gcs.bucket |
Bucket of the object. |
| gcs.key |
Name of the object. |
| gcs.size |
Size of the object. |
| gcs.cache.control |
Data cache control of the object. |
| gcs.component.count |
The number of components which make up the object. |
| gcs.content.disposition |
The data content disposition of the object. |
| gcs.content.encoding |
The content encoding of the object. |
| gcs.content.language |
The content language of the object. |
| mime.type |
The MIME/Content-Type of the object |
| gcs.crc32c |
The CRC32C checksum of object's data, encoded in base64 in big-endian order. |
| gcs.create.time |
The creation time of the object (milliseconds) |
| gcs.update.time |
The last modification time of the object (milliseconds) |
| gcs.encryption.algorithm |
The algorithm used to encrypt the object. |
| gcs.encryption.sha256 |
The SHA256 hash of the key used to encrypt the object |
| gcs.etag |
The HTTP 1.1 Entity tag for the object. |
| gcs.generated.id |
The service-generated for the object |
| gcs.generation |
The data generation of the object. |
| gcs.md5 |
The MD5 hash of the object's data encoded in base64. |
| gcs.media.link |
The media download link to the object. |
| gcs.metageneration |
The metageneration of the object. |
| gcs.owner |
The owner (uploader) of the object. |
| gcs.owner.type |
The ACL entity type of the uploader of the object. |
| gcs.acl.owner |
A comma-delimited list of ACL entities that have owner access to the object. Entities will be either email addresses, domains, or project IDs. |
| gcs.acl.writer |
A comma-delimited list of ACL entities that have write access to the object. Entities will be either email addresses, domains, or project IDs. |
| gcs.acl.reader |
A comma-delimited list of ACL entities that have read access to the object. Entities will be either email addresses, domains, or project IDs. |
| gcs.uri |
The URI of the object as a string. |
## Use Cases Involving Other Components
| Retrieve all files in a Google Compute Storage (GCS) bucket |
| ----------------------------------------------------------- |
## See also
- [org.apache.nifi.processors.gcp.storage.DeleteGCSObject](/user-guide/data-integration/openflow/processors/deletegcsobject)
- [org.apache.nifi.processors.gcp.storage.ListGCSBucket](/user-guide/data-integration/openflow/processors/listgcsbucket)
- [org.apache.nifi.processors.gcp.storage.PutGCSObject](/user-guide/data-integration/openflow/processors/putgcsobject)
---
title: FetchGoogleDrive 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchgoogledrive.md
section: Loading & Unloading Data
---
# FetchGoogleDrive 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-gcp-nar
## Description
Fetches files from a Google Drive Folder. Designed to be used in tandem with ListGoogleDrive. Please see Additional Details to set up access to Google Drive.
## Tags
drive, fetch, google, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Google Doc Export Type |
Google Documents cannot be downloaded directly from Google Drive but instead must be exported to a specified MIME Type. In the event that the incoming FlowFile's MIME Type indicates that the file is a Google Document, this property specifies the MIME Type to export the document to. |
| Google Drawing Export Type |
Google Drawings cannot be downloaded directly from Google Drive but instead must be exported to a specified MIME Type. In the event that the incoming FlowFile's MIME Type indicates that the file is a Google Drawing, this property specifies the MIME Type to export the drawing to. |
| Google Presentation Export Type |
Google Presentations cannot be downloaded directly from Google Drive but instead must be exported to a specified MIME Type. In the event that the incoming FlowFile's MIME Type indicates that the file is a Google Presentation, this property specifies the MIME Type to export the presentation to. |
| Google Spreadsheet Export Type |
Google Spreadsheets cannot be downloaded directly from Google Drive but instead must be exported to a specified MIME Type. In the event that the incoming FlowFile's MIME Type indicates that the file is a Google Spreadsheet, this property specifies the MIME Type to export the spreadsheet to. |
| connect-timeout |
Maximum wait time for connection to Google Drive service. |
| drive-file-id |
The Drive ID of the File to fetch. Please see Additional Details for information on how to obtain the Drive ID. |
| gcp-credentials-provider-service |
The Controller Service used to obtain Google Cloud Platform credentials. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
| read-timeout |
Maximum wait time for response from Google Drive service. |
## Relationships
| Name |
Description |
| failure |
A FlowFile will be routed here for each File for which fetch was attempted but failed. |
| success |
A FlowFile will be routed here for each successfully fetched File. |
## Writes attributes
| Name |
Description |
| drive.id |
The id of the file |
| filename |
The name of the file |
| mime.type |
The MIME type of the file |
| drive.size |
The size of the file. Set to 0 when the file size is not available (e.g. externally stored files). |
| drive.size.available |
Indicates if the file size is known / available |
| drive.timestamp |
The last modified time or created time (whichever is greater) of the file. The reason for this is that the original modified date of a file is preserved when uploaded to Google Drive. 'Created time' takes the time when the upload occurs. However uploaded files can still be modified later. |
| drive.created.time |
The file's creation time |
| drive.modified.time |
The file's last modification time |
| drive.owner |
The owner of the file |
| drive.last.modifying.user |
The last modifying user of the file |
| drive.web.view.link |
Web view link to the file |
| drive.web.content.link |
Web content link to the file |
| drive.parent.folder.id |
The id of the file's parent folder |
| drive.parent.folder.name |
The name of the file's parent folder |
| drive.shared.drive.id |
The id of the shared drive (if the file is located on a shared drive) |
| drive.shared.drive.name |
The name of the shared drive (if the file is located on a shared drive) |
| error.code |
The error code returned by Google Drive |
| error.message |
The error message returned by Google Drive |
## Use Cases Involving Other Components
| Retrieve all files in a Google Drive folder |
| ------------------------------------------- |
## See also
- [org.apache.nifi.processors.gcp.drive.ListGoogleDrive](/user-guide/data-integration/openflow/processors/listgoogledrive)
- [org.apache.nifi.processors.gcp.drive.PutGoogleDrive](/user-guide/data-integration/openflow/processors/putgoogledrive)
---
title: FetchGoogleDriveFileComments 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchgoogledrivefilecomments.md
section: Loading & Unloading Data
---
# FetchGoogleDriveFileComments 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-google-drive-nar
## Description
Fetches comments and their replies for a Google Drive file. The file ID can be set by a FlowFile attribute. Records include comment metadata such as deleted status, resolved status, anchors, and a nested array of replies.
## Tags
comments, drive, gcp, google, openflow, replies
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| File ID |
Google Drive file ID. |
| GCP Credentials Service |
Controller Service used to obtain Google Cloud Platform credentials. |
| Record Writer |
Specifies the Record Writer to use when writing the comments. |
## Relationships
| Name |
Description |
| failure |
FlowFiles are routed here if the processor fails to retrieve comments. |
| not.found |
A FlowFile is routed here if the file was not found. |
| retry |
FlowFiles are routed here if a connection or rate-limit issue occurs. |
| success |
All FlowFiles that are successfully processed are routed here. |
## Writes attributes
| Name |
Description |
| record.count |
Number of comment records returned (not including replies). |
| google.drive.file.id |
The file ID from which comments were fetched. |
---
title: FetchGoogleDriveMetadata 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchgoogledrivemetadata.md
section: Loading & Unloading Data
---
# FetchGoogleDriveMetadata 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-google-drive-nar
## Description
Fetches Google Drive file metadata. This includes the file's name, size, MIME type, and permissions. The file ID must be provided as a FlowFile attribute.
## Tags
authorization, cloud, drive, gcp, google, openflow, permissions, storage, unstructured
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| File ID |
An id of an file to retrieve the metadata for |
| GCP Credentials Service |
The Controller Service used to obtain Google Cloud Platform credentials. |
## Relationships
| Name |
Description |
| failure |
A FlowFile is routed here if the processor fails to retrieve Google Drive file metadata. |
| not.found |
A FlowFile is routed here if the file metadata was not found |
| retry |
A FlowFile is routed here if the processor should retry the request (e.g., after rate limiting). |
| success |
A FlowFile is routed here after successfully retrieving Google Drive file metadata. |
## Writes attributes
| Name |
Description |
| google.drive.drive.id |
The ID of the Shared Google Drive. |
| google.drive.file.name |
The name of the file. |
| google.drive.created.time |
The timestamp when the file was created, in milliseconds since the Unix epoch. |
| google.drive.modified.time |
The timestamp when the file was modified, in milliseconds since the Unix epoch. |
| google.drive.size |
The size of the file in bytes. |
| google.drive.md5 |
The MD5 checksum of the file. |
| google.drive.mime.type |
The MIME type of the file. |
| google.drive.version |
The version of the file. This changes based on user and system based updates to the file. |
| google.drive.webUrl |
A link for opening the file in a relevant Google editor or viewer in a browser. |
| google.drive.lastModifiedBy.displayName |
A display name of the user that modified the file. |
| google.drive.lastModifiedBy.email |
An email of the user that modified the file. |
| google.drive.permissions.<role>.users |
A comma-separated list of email addresses for users with the specified role. Valid roles are 'owner', 'organizer', 'fileOrganizer', 'writer', 'commenter', 'reader'. For example, if the owner is [john.doe@gmail.com](mailto:john.doe@gmail.com) and users [jane.doe@gmail.com](mailto:jane.doe@gmail.com) and [jake.doe@gmail.com](mailto:jake.doe@gmail.com) are readers, there would be an attribute named _google.drive.permissions.owner.users_ with the value _john.doe@gmail.com_, and an attribute named _google.drive.permissions.reader.users_ with the value _jane.doe@gmail.com, jake.doe@gmail.com_ |
| google.drive.permissions.<role>.groups |
A comma-separated list of email addresses for groups with the specified role. Valid roles are 'owner', 'organizer', 'fileOrganizer', 'writer', 'commenter', 'reader'. For example, if the owner is _employees@openflow-all-dev.iam.gserviceaccount.com_ and the group _contractors@openflow-all-dev.iam.gserviceaccount.com_ is a reader, there would be an attribute named _google.drive.permissions.owner.groups_ with the value _employees@openflow-all-dev.iam.gserviceaccount.com_, and an attribute named _google.drive.permissions.reader.groups_ with the value _contractors@openflow-all-dev.iam.gserviceaccount.com_ |
| google.drive.permissions.<role>.domains |
A comma-separated list of domain names for which all users have the given role. Valid roles are 'owner', 'organizer', 'fileOrganizer', 'writer', 'commenter', 'reader'. For example, if all users in the domain _snowflake.com_ have the role of reader, there would be an attribute named _google.drive.permissions.reader.domains_ with the value _snowflake.com_ |
| google.drive.permissions.<role>.public |
If a file is shared publicly, this attribute will be added with a value of 'true' for any role that applies to the public. |
| google.drive.file.path |
The hierarchical path of the file in Google Drive, e.g. 'parent_folder/child_folder/file.txt'. |
## See also
- [com.snowflake.openflow.runtime.processors.google.CaptureGoogleDriveChanges](/user-guide/data-integration/openflow/processors/capturegoogledrivechanges)
---
title: FetchGridFS 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchgridfs.md
section: Loading & Unloading Data
---
# FetchGridFS 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-mongodb-nar
## Description
Retrieves one or more files from a GridFS bucket by file name or by a user-defined query.
## Tags
fetch, gridfs, mongo
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| gridfs-bucket-name |
The GridFS bucket where the files will be stored. If left blank, it will use the default value 'fs' that the MongoDB client driver uses. |
| gridfs-client-service |
The MongoDB client service to use for database connections. |
| gridfs-database-name |
The name of the database to use |
| gridfs-file-name |
The name of the file in the bucket that is the target of this processor. |
| gridfs-query |
A valid MongoDB query to use to fetch one or more files from GridFS. |
| mongo-operation-mode |
This option controls when results are made available to downstream processors. If Stream Query Results is enabled, provenance will not be tracked relative to the input flowfile if an input flowfile is received and starts the query. In Stream Query Results mode errors will be handled by sending a new flowfile with the original content and attributes of the input flowfile to the failure relationship. Streaming should only be used if there is reliable connectivity between MongoDB and NiFi. |
| mongo-query-attribute |
If set, the query will be written to a specified attribute on the output flowfiles. |
## Relationships
| Name |
Description |
| failure |
When there is a failure processing the flowfile, it goes to this relationship. |
| original |
The original input flowfile goes to this relationship if the query does not cause an error |
| success |
When the operation succeeds, the flowfile is sent to this relationship. |
## Writes attributes
| Name |
Description |
| gridfs.file.metadata |
The custom metadata stored with a file is attached to this property if it exists. |
---
title: FetchJiraFields 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchjirafields.md
section: Loading & Unloading Data
---
# FetchJiraFields 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-atlassian-processors-nar
## Description
Retrieves comprehensive metadata for all fields available in the Jira Cloud instance using the REST API v3 /field endpoint. For each field, returns detailed information including field ID/key, display name, field properties, JQL clause names for queries, and schema details with data types.
## Tags
api, atlassian, fetch, jira, rest
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| API Token |
Jira API token for authorization |
| Authorization Method |
Authorization method for Jira Cloud API |
| Environment URL |
URL to the Atlassian Jira Environment |
| Issue Fields |
A list of fields to return for each issue. This property accepts a comma-separated list. |
| Jira Email |
Email address associated with Jira account |
| Request Rate Manager |
Controller service for keeping track of rate limits for Atlassian APIs |
| Web Client Service |
Controller service for managing HTTP connections to Jira |
## Relationships
| Name |
Description |
| failure |
Failed to fetch Jira fields, e.g., due to connection issues or invalid credentials |
| retry |
Retryable failure occurred, e.g. rate limiting |
| success |
Successfully fetched Jira fields |
## Writes attributes
| Name |
Description |
| mime.type |
The MIME type of the returned response, always set to 'application/json' |
## See also
- [com.snowflake.openflow.runtime.atlassian.jira.processors.FetchJiraIssues](/user-guide/data-integration/openflow/processors/fetchjiraissues)
---
title: FetchJiraIssues 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchjiraissues.md
section: Loading & Unloading Data
---
# FetchJiraIssues 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-atlassian-processors-nar
## Description
Fetches issues from Jira Cloud using REST API v3 with configurable search options. Provides two search modes: 1. Simple Search - Filter by project name, status category, created/updated dates 2. Advanced Search - Use custom JQL (Jira Query Language) expressions Key features: - Smart pagination handling with automatic state management - Incremental sync capability using timestamps between processor runs - Timezone-aware date handling using Jira user's timezone - Configurable issue fields retrieval - Adds metadata to FlowFiles: source URL (jira.source.url), query (jira.query.jql), statement type (statement.type) - Adds insert,upsert attributes for downstream processing The processor maintains cluster state to resume operations after restarts Authentication is handled via basic auth using Jira email/API token credentials. Currently that is the only supported method. LIMITATIONS: - Jira issue deletes are not detected.
## Tags
api, atlassian, fetch, jira, rest
## Input Requirement
ALLOWED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| API Token |
Jira API token for authorization |
| Authorization Method |
Authorization method for Jira Cloud API |
| Created After |
Filter issues created after specified date/time (optional, format: yyyy-MM-dd) |
| Environment URL |
URL to the Atlassian Jira Environment |
| Issue Fields |
A list of fields to return for each issue. This property accepts a comma-separated list. |
| JQL Query |
JQL query string (required when using JQL query type) |
| Jira Email |
Email address associated with Jira account |
| Maximum Page Size |
The Maximum Page Size value must be between 50 and 1000 |
| Project Names |
Comma-separated list of project names for simple search |
| Request Rate Manager |
Controller service for keeping track of rate limits for Atlassian APIs |
| Search Type |
Type of search to perform |
| Status Category |
Status category filter for simple search (optional) |
| Updated After |
Filter issues updated after specified date/time (optional, format: yyyy-MM-dd) |
| Web Client Service |
Controller service for managing HTTP connections to Jira |
## State management
| Scopes |
Description |
| CLUSTER |
Stores pagination state to maintain position between restarts. Resets when ingestion configuration changes. |
## Relationships
| Name |
Description |
| retry |
Retryable failure occurred, e.g. rate limiting |
| success |
Successfully fetched Jira issues |
## Writes attributes
| Name |
Description |
| mime.type |
application/json |
| jira.query.jql |
The JQL query used for this fetch |
| jira.source.url |
URL of the Jira source |
| statement.type |
Statement type INSERT, UPSERT |
## See also
- [com.snowflake.openflow.runtime.atlassian.jira.processors.FetchJiraFields](/user-guide/data-integration/openflow/processors/fetchjirafields)
---
title: FetchMicrosoftDataverseTable 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchmicrosoftdataversetable.md
section: Loading & Unloading Data
---
# FetchMicrosoftDataverseTable 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-dataverse-processors-nar
## Description
Fetch records from Microsoft Dataverse Tables
## Tags
dataverse
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Delete Schema |
|
| Environment URL |
URL to Microsoft Dataverse Environment |
| Logical Name |
Logical Name of Dataverse Table |
| Max Page Size |
Defines how many records will be fetched from Dataverse at once |
| OAuth2 Access Token Provider |
Enables managed retrieval of OAuth2 Bearer Token. |
| Record Writer |
Specifies the Controller Service to use for writing out the records |
| Rows Number Limit |
Defines maximum number of rows returned in a single flow file. Multiple request will be made to API to reach the limit. When not set, a page size value will be used effectively. |
| Table Name |
Dataverse Table Name |
| Upsert Schema |
|
| Web Client Service Provider |
Creates instance of web client. |
## State management
| Scopes |
Description |
| CLUSTER |
status |
## Relationships
| Name |
Description |
| failure |
FlowFile with errors occurred while fetching from Dataverse. |
| retry |
FlowFile with maintainable errors occurred while fetching from Dataverse. |
| success |
FlowFile with fetched data stored as records. |
---
title: FetchS3Object 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchs3object.md
section: Loading & Unloading Data
---
# FetchS3Object 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-aws-nar
## Description
Retrieves the contents of an S3 Object and writes it to the content of a FlowFile
## Tags
AWS, Amazon, Fetch, Get, S3
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| AWS Credentials Provider service |
The Controller Service that is used to obtain AWS credentials provider |
| Bucket |
The S3 Bucket to interact with |
| Communications Timeout |
The amount of time to wait in order to establish a connection to AWS or receive data from AWS before timing out. |
| Custom Signer Class Name |
Fully qualified class name of the custom signer class. The signer must implement com.amazonaws.auth. Signer interface. |
| Custom Signer Module Location |
Comma-separated list of paths to files and/or directories which contain the custom signer's JAR file and its dependencies (if any). |
| Encryption Service |
Specifies the Encryption Service Controller used to configure requests. PutS3Object: For backward compatibility, this value is ignored when 'Server Side Encryption' is set. FetchS3Object: Only needs to be configured in case of Server-side Customer Key, Client-side KMS and Client-side Customer Key encryptions. |
| Endpoint Override URL |
Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. |
| Object Key |
The S3 Object Key to use. This is analogous to a filename for traditional file systems. |
| Range Length |
The number of bytes to download from the object, starting from the Range Start. An empty value or a value that extends beyond the end of the object will read to the end of the object. |
| Range Start |
The byte position at which to start reading from the object. An empty value or a value of zero will start reading at the beginning of the object. |
| Region |
The AWS Region to connect to. |
| Requester Pays |
If true, indicates that the requester consents to pay any charges associated with retrieving objects from the S3 bucket. This sets the 'x-amz-request-payer' header to 'requester'. |
| SSL Context Service |
Specifies an optional SSL Context Service that, if provided, will be used to create connections |
| Signer Override |
The AWS S3 library uses Signature Version 4 by default but this property allows you to specify the Version 2 signer to support older S3-compatible services or even to plug in your own custom signer implementation. |
| Version |
The Version of the Object to download |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| failure |
If the Processor is unable to process a given FlowFile, it will be routed to this Relationship. |
| success |
FlowFiles are routed to this Relationship after they have been successfully processed. |
## Writes attributes
| Name |
Description |
| s3.url |
The URL that can be used to access the S3 object |
| s3.bucket |
The name of the S3 bucket |
| path |
The path of the file |
| absolute.path |
The path of the file |
| filename |
The name of the file |
| hash.value |
The MD5 sum of the file |
| hash.algorithm |
MD5 |
| mime.type |
If S3 provides the content type/MIME type, this attribute will hold that file |
| s3.etag |
The ETag that can be used to see if the file has changed |
| s3.exception |
The class name of the exception thrown during processor execution |
| s3.additionalDetails |
The S3 supplied detail from the failed operation |
| s3.statusCode |
The HTTP error code (if available) from the failed operation |
| s3.errorCode |
The S3 moniker of the failed operation |
| s3.errorMessage |
The S3 exception message from the failed operation |
| s3.expirationTime |
If the file has an expiration date, this attribute will be set, containing the milliseconds since epoch in UTC time |
| s3.expirationTimeRuleId |
The ID of the rule that dictates this object's expiration time |
| s3.sseAlgorithm |
The server side encryption algorithm of the object |
| s3.version |
The version of the S3 object |
| s3.encryptionStrategy |
The name of the encryption strategy that was used to store the S3 object (if it is encrypted) |
## Use cases
| Fetch a specific file from S3 |
| ----------------------------- |
## Use Cases Involving Other Components
| Retrieve all files in an S3 bucket |
| ------------------------------------------------------------- |
| Retrieve only files from S3 that meet some specified criteria |
| Retrieve new files as they arrive in an S3 bucket |
## See also
- [org.apache.nifi.processors.aws.s3.CopyS3Object](/user-guide/data-integration/openflow/processors/copys3object)
- [org.apache.nifi.processors.aws.s3.DeleteS3Object](/user-guide/data-integration/openflow/processors/deletes3object)
- [org.apache.nifi.processors.aws.s3.GetS3ObjectMetadata](/user-guide/data-integration/openflow/processors/gets3objectmetadata)
- [org.apache.nifi.processors.aws.s3.GetS3ObjectTags](/user-guide/data-integration/openflow/processors/gets3objecttags)
- [org.apache.nifi.processors.aws.s3.ListS3](/user-guide/data-integration/openflow/processors/lists3)
- [org.apache.nifi.processors.aws.s3.PutS3Object](/user-guide/data-integration/openflow/processors/puts3object)
- [org.apache.nifi.processors.aws.s3.TagS3Object](/user-guide/data-integration/openflow/processors/tags3object)
---
title: FetchSFTP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchsftp.md
section: Loading & Unloading Data
---
# FetchSFTP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Fetches the content of a file from a remote SFTP server and overwrites the contents of an incoming FlowFile with the content of the remote file.
## Tags
fetch, files, get, ingest, input, remote, retrieve, sftp, source
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Algorithm Negotiation |
Configuration strategy for SSH algorithm negotiation |
| Ciphers Allowed |
A comma-separated list of Ciphers allowed for SFTP connections. Leave unset to allow all. Available options are: 3des-cbc, aes128-cbc, aes128-ctr, [aes128-gcm@openssh.com](mailto:aes128-gcm@openssh.com), aes192-cbc, aes192-ctr, aes256-cbc, aes256-ctr, [aes256-gcm@openssh.com](mailto:aes256-gcm@openssh.com), arcfour128, arcfour256, blowfish-cbc, [chacha20-poly1305@openssh.com](mailto:chacha20-poly1305@openssh.com), none |
| Completion Strategy |
Specifies what to do with the original file on the server once it has been pulled into NiFi. If the Completion Strategy fails, a warning will be logged but the data will still be transferred. |
| Connection Timeout |
Amount of time to wait before timing out while creating a connection |
| Create Directory |
Used when 'Completion Strategy' is 'Move File'. Specifies whether or not the remote directory should be created if it does not exist. |
| Data Timeout |
When transferring a file between the local and remote system, this value specifies how long is allowed to elapse without any data being transferred between systems |
| Disable Directory Listing |
Control how 'Move Destination Directory' is created when 'Completion Strategy' is 'Move File' and 'Create Directory' is enabled. If set to 'true', directory listing is not performed prior to create missing directories. By default, this processor executes a directory listing command to see target directory existence before creating missing directories. However, there are situations that you might need to disable the directory listing such as the following. Directory listing might fail with some permission setups (e.g. chmod 100) on a directory. Also, if any other SFTP client created the directory after this processor performed a listing and before a directory creation request by this processor is finished, then an error is returned because the directory already exists. |
| Host Key File |
If supplied, the given file will be used as the Host Key; otherwise, if 'Strict Host Key Checking' property is applied (set to true) then uses the 'known_hosts' and 'known_hosts2' files from ~/.ssh directory else no host key file will be used |
| Hostname |
The fully-qualified hostname or IP address of the host to fetch the data from |
| Key Algorithms Allowed |
A comma-separated list of Key Algorithms allowed for SFTP connections. Leave unset to allow all. Available options are: ecdsa-sha2-nistp256, [ecdsa-sha2-nistp256-cert-v01@openssh.com](mailto:ecdsa-sha2-nistp256-cert-v01@openssh.com), ecdsa-sha2-nistp384, [ecdsa-sha2-nistp384-cert-v01@openssh.com](mailto:ecdsa-sha2-nistp384-cert-v01@openssh.com), ecdsa-sha2-nistp521, [ecdsa-sha2-nistp521-cert-v01@openssh.com](mailto:ecdsa-sha2-nistp521-cert-v01@openssh.com), rsa-sha2-256, [rsa-sha2-256-cert-v01@openssh.com](mailto:rsa-sha2-256-cert-v01@openssh.com), rsa-sha2-512, [rsa-sha2-512-cert-v01@openssh.com](mailto:rsa-sha2-512-cert-v01@openssh.com), [sk-ecdsa-sha2-nistp256@openssh.com](mailto:sk-ecdsa-sha2-nistp256@openssh.com), [sk-ssh-ed25519@openssh.com](mailto:sk-ssh-ed25519@openssh.com), ssh-dss, [ssh-dss-cert-v01@openssh.com](mailto:ssh-dss-cert-v01@openssh.com), ssh-ed25519, [ssh-ed25519-cert-v01@openssh.com](mailto:ssh-ed25519-cert-v01@openssh.com), ssh-rsa, [ssh-rsa-cert-v01@openssh.com](mailto:ssh-rsa-cert-v01@openssh.com) |
| Key Exchange Algorithms Allowed |
A comma-separated list of Key Exchange Algorithms allowed for SFTP connections. Leave unset to allow all. Available options are: curve25519-sha256, [curve25519-sha256@libssh.org](mailto:curve25519-sha256@libssh.org), curve448-sha512, diffie-hellman-group-exchange-sha1, diffie-hellman-group-exchange-sha256, diffie-hellman-group1-sha1, diffie-hellman-group14-sha1, diffie-hellman-group14-sha256, diffie-hellman-group15-sha512, diffie-hellman-group16-sha512, diffie-hellman-group17-sha512, diffie-hellman-group18-sha512, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, mlkem1024nistp384-sha384, mlkem768nistp256-sha256, mlkem768x25519-sha256, sntrup761x25519-sha512, [sntrup761x25519-sha512@openssh.com](mailto:sntrup761x25519-sha512@openssh.com) |
| Log Level When File Not Found |
Log level to use in case the file does not exist when the processor is triggered |
| Message Authentication Codes Allowed |
A comma-separated list of Message Authentication Codes allowed for SFTP connections. Leave unset to allow all. Available options are: hmac-md5, hmac-md5-96, hmac-sha1, hmac-sha1-96, [hmac-sha1-etm@openssh.com](mailto:hmac-sha1-etm@openssh.com), hmac-sha2-256, [hmac-sha2-256-etm@openssh.com](mailto:hmac-sha2-256-etm@openssh.com), hmac-sha2-512, [hmac-sha2-512-etm@openssh.com](mailto:hmac-sha2-512-etm@openssh.com) |
| Move Destination Directory |
The directory on the remote server to move the original file to once it has been ingested into NiFi. This property is ignored unless the Completion Strategy is set to 'Move File'. The specified directory must already exist on the remote system if 'Create Directory' is disabled, or the rename will fail. |
| Password |
Password for the user account |
| Port |
The port to connect to on the remote host to fetch the data from |
| Private Key Passphrase |
Password for the private key |
| Private Key Path |
The fully qualified path to the Private Key file |
| Remote File |
The fully qualified filename on the remote system |
| Send Keep Alive On Timeout |
Send a Keep Alive message every 5 seconds up to 5 times for an overall timeout of 25 seconds. |
| Strict Host Key Checking |
Indicates whether or not strict enforcement of hosts keys should be applied |
| Use Compression |
Indicates whether or not ZLIB compression should be used when transferring files |
| Username |
Username |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| comms.failure |
Any FlowFile that could not be fetched from the remote server due to a communications failure will be transferred to this Relationship. |
| not.found |
Any FlowFile for which we receive a 'Not Found' message from the remote server will be transferred to this Relationship. |
| permission.denied |
Any FlowFile that could not be fetched from the remote server due to insufficient permissions will be transferred to this Relationship. |
| success |
All FlowFiles that are received are routed to success |
## Writes attributes
| Name |
Description |
| sftp.remote.host |
The hostname or IP address from which the file was pulled |
| sftp.remote.port |
The port that was used to communicate with the remote SFTP server |
| sftp.remote.filename |
The name of the remote file that was pulled |
| filename |
The filename is updated to point to the filename fo the remote file |
| path |
If the Remote File contains a directory name, that directory name will be added to the FlowFile using the 'path' attribute |
| fetch.failure.reason |
The name of the failure relationship applied when routing to any failure relationship |
## Use Cases Involving Other Components
| Retrieve all files in a directory of an SFTP Server |
| --------------------------------------------------- |
## See also
- [org.apache.nifi.processors.standard.GetFTP](/user-guide/data-integration/openflow/processors/getftp)
- [org.apache.nifi.processors.standard.GetSFTP](/user-guide/data-integration/openflow/processors/getsftp)
- [org.apache.nifi.processors.standard.PutFTP](/user-guide/data-integration/openflow/processors/putftp)
- [org.apache.nifi.processors.standard.PutSFTP](/user-guide/data-integration/openflow/processors/putsftp)
---
title: FetchSharepointFile 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchsharepointfile.md
section: Loading & Unloading Data
---
# FetchSharepointFile 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-msgraph-nar
## Description
Fetches the contents of a file from a Sharepoint Drive, optionally downloading a PDF or HTML version of the file when applicable. Any FlowFile that represents a Sharepoint folder will be routed to success without fetching contents.
## Tags
cdc, document, graph, microsoft, openflow, sharepoint, unstructured
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Authentication Service |
The service that provides authentication for the SharePoint API |
| Download PDF/HTML Version |
Sharepoint supports automatically converting certain file formats to PDF or HTML. If this property is set to _true_, the Processor will inspect the FlowFile's filename extension to determine if the file can be converted to PDF or HTML. If the file can be converted, the Processor will download the converted version. If the file cannot be converted, the Processor will download the original file. If this property is set to _false_, the Processor will always download the original file. |
| Drive ID |
The ID of the drive that contains the file to fetch |
| Fallback Retry Duration |
The time to wait before retrying the operation after a communication failure. This value is used when the response doesn't contain a Retry-After header. |
| Item ID |
The ID of the item to fetch |
| Update Extension |
If true, the Processor will update the filename extension to match the format of the downloaded file |
## Relationships
| Name |
Description |
| comms.failure |
A FlowFile is routed here if the processor failed to communicate with the Graph API. Can be retried |
| failure |
An incoming FlowFile is routed to this relationship if the contents of the item could not be fetched |
| not.found |
A FlowFile is routed here if the item was not found |
| success |
An incoming FlowFile is routed to this relationship after the contents of the item have been fetched and written to the FlowFile |
## Use Cases Involving Other Components
| Fetch a file from Sharepoint by the Site URL, Drive Name and file path. |
| ----------------------------------------------------------------------- |
## See also
- [com.snowflake.openflow.runtime.processors.sharepoint.CaptureSharepointChanges](/user-guide/data-integration/openflow/processors/capturesharepointchanges)
---
title: FetchSharepointMetadata 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchsharepointmetadata.md
section: Loading & Unloading Data
---
# FetchSharepointMetadata 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-msgraph-nar
## Description
For each drive item retrieves its metadata and permissions and writes them as FlowFile attributes.
## Tags
cdc, document, graph, library, microsoft, openflow, sharepoint, unstructured
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Authentication Service |
The service that provides authentication for the SharePoint API |
| Drive ID |
A drive id where the Sharepoint file resides |
| Fallback Retry Duration |
The time to wait before retrying the operation after a communication failure. This value is used when the response doesn't contain a Retry-After header. |
| Fetch Item Permissions |
If true, the Processor will fetch user and group permission information for the captured Sharepoint item. |
| Item ID |
An id of an item to retrieve the metadata for |
| Item Permissions To Fetch |
A comma-separated list of permission types to fetch for the captured Sharepoint item. Available permission types: USER, GROUP, SITE_USER, SITE_GROUP. |
| Site ID |
A site id where the Sharepoint file resides |
## Relationships
| Name |
Description |
| comms.failure |
A FlowFile is routed here if the processor failed to communicate with the Graph API. Can be retried |
| failure |
An incoming FlowFile is routed to this relationship if the metadata and permissions of the item could not be fetched |
| not.found |
A FlowFile is routed here if the item was not found |
| success |
An incoming FlowFile is routed to this relationship after the metadata and permissions of the item have been fetched and written to the FlowFile attributes |
## Writes attributes
| Name |
Description |
| sharepoint.item.id |
The ID of the Sharepoint item. |
| sharepoint.item.type |
The type of the Sharepoint item. Possible values are 'File' and 'Folder'. |
| sharepoint.path |
The path of the Sharepoint item. This is the path relative to the root of the Document Library. |
| sharepoint.filename |
The name of the Sharepoint item. This attribute is not available for 'Deleted' changes. |
| sharepoint.size |
The size of the Sharepoint item. |
| sharepoint.createdAt |
The creation timestamp of the Sharepoint item. |
| sharepoint.lastModified |
The last modified timestamp of the Sharepoint item. |
| sharepoint.createdBy.<identity>.id |
An id of the identity that created the Sharepoint item. This attribute is not always available. |
| sharepoint.createdBy.<identity>.displayName |
A display name of the identity that created the Sharepoint item. This attribute is not always available. |
| sharepoint.createdBy.<identity>.email |
An email of the identity that created the Sharepoint item. This attribute is not always available. |
| sharepoint.lastModifiedBy.<identity>.id |
An id of the identity that modified the Sharepoint item last. This attribute is not always available. |
| sharepoint.lastModifiedBy.<identity>.displayName |
A display name of the identity that modified the Sharepoint item last. This attribute is not always available. |
| sharepoint.lastModifiedBy.<identity>.email |
An email of the identity that modified the Sharepoint item last. This attribute is not always available. |
| sharepoint.drive.id |
The ID of the Sharepoint Drive that contains the item. |
| sharepoint.site.id |
The ID of the Sharepoint Site that contains the item. |
| sharepoint.ctag |
The CTag of the Sharepoint item. |
| sharepoint.etag |
The ETag of the Sharepoint item. |
| sharepoint.webUrl |
The browser view url of the Sharepoint item. |
| sharepoint.permissions.read.groups |
A comma-separated list of groups that have read permissions on the Sharepoint item. For each group, if an e-mail address is available in Sharepoint, it will be included. Additionally, the group principal, such as _mygroup@mytenant.onmicrosoft.com_, is included. |
| sharepoint.permissions.read.groups.ids |
A comma-separated list of group IDs that have read permissions on the Sharepoint item. |
| sharepoint.permissions.read.users |
A comma-separated list of users that have read permissions on the Sharepoint item. For each user, if an e-mail address is available in Sharepoint, it will be included. Additionally, the user principal, such as _johndoe@mytenant.onmicrosoft.com_, is included. |
| sharepoint.permissions.read.users.ids |
A comma-separated list of Microsoft365 user IDs that have read permissions on the Sharepoint item. |
| sharepoint.permissions.read.siteusers |
A comma-separated list of Sharepoint site user emails that have read permissions on the Sharepoint item. |
| sharepoint.permissions.read.siteusers.ids |
A comma-separated list of Sharepoint site user IDs that have read permissions on the Sharepoint item. |
| sharepoint.permissions.read.sitegroups.ids |
A comma-separated list of Sharepoint site group IDs that have read permissions on the Sharepoint item. |
| filename |
The name of the Sharepoint item. |
| path |
The path of the Sharepoint item. This is the path relative to the root of the Document Library. |
| mime.type |
The MIME type of the Sharepoint item. This attribute is only available for 'File' items. |
| hash.quickxor |
The QuickXor hash of the Sharepoint item. This attribute is not always available. |
| hash.sha256 |
The SHA-256 hash of the Sharepoint item. This attribute is not always available. |
| hash.sha1 |
The SHA-1 hash of the Sharepoint item. This attribute is not always available. |
| hash.crc32 |
The CRC32 hash of the Sharepoint item. This attribute is not always available. |
---
title: FetchSlackConversationInfo 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchslackconversationinfo.md
section: Loading & Unloading Data
---
# FetchSlackConversationInfo 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-slack-processors-nar
## Description
Fetches Slack conversation info and member emails
## Tags
conversation, conversation.members, slack, social media, team
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Access Token |
OAuth Access Token used for authenticating/authorizing the Slack request sent by NiFi. This may be either a User Token or a Bot Token. It must be granted the channels:history, groups:history, im:history, or mpim:history scope, depending on the type of conversation being used. |
| Cache Expiration |
User emails are cached to reduce network lookups. A longer expiration reduces network overhead but can cause data to be out of sync. |
| Cache Size |
User emails are cached to reduce network lookups. A larger cache consumes memory but reduces network overhead. |
| Channel |
The Slack Channel ID to retrieve info from. Leave blank to iterate over every available Conversation. |
| Rate Limiter Service |
Slack Rate Limiter Service to coordinate rate limiting across processors |
## Relationships
| Name |
Description |
| conversations |
Each configured Slack Conversation info and members will be routed to this relationship in separate FlowFiles |
| failure |
If Slack Conversation metadata is unable to be received the input FlowFile will be routed to this relationship |
| original |
Original input FlowFile that has been successfully processed. |
## Writes attributes
| Name |
Description |
| conversation.members.count |
Set to the number of members of the conversation |
| conversation.id |
Set to the number of members of the conversation |
| channel.name |
Set to the name of the channel if the conversation is a channel |
| mime.type |
Set to application/json, as the output will always be in JSON format |
---
title: FetchSlackFile 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchslackfile.md
section: Loading & Unloading Data
---
# FetchSlackFile 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-slack-processors-nar
## Description
Downloads a file shared on Slack. Writes the file content to the FlowFile content and FlowFile attributes from the file.
## Tags
download, file, slack
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Bot Token |
The Bot Token that is registered to your Slack application |
| Channel ID |
The Slack Channel ID where the file was shared. |
| File ID |
The Slack File ID to download. |
| Rate Limiter Service |
Slack Rate Limiter Service to coordinate rate limiting across processors |
| Web Client Service |
The Web Client Service to use for downloading files from Slack |
## Relationships
| Name |
Description |
| failure |
FlowFiles that could not be processed are routed to this relationship |
| success |
FlowFiles containing successfully downloaded Slack files are routed to this relationship |
## Writes attributes
| Name |
Description |
| mime.type |
The MIME type of the downloaded file |
| filename |
The name of the downloaded file |
| slack.file.name |
The Slack File name |
| slack.file.mimetype |
The Slack File MIME type |
| slack.file.size |
The Slack File size in bytes |
| slack.conversation.id |
The Slack Channel ID |
| slack.event.ts |
The Slack event timestamp |
## See also
- [com.snowflake.openflow.runtime.processors.slack.FetchSlackConversationInfo](/user-guide/data-integration/openflow/processors/fetchslackconversationinfo)
- [com.snowflake.openflow.runtime.processors.slack.FetchSlackMessage](/user-guide/data-integration/openflow/processors/fetchslackmessage)
---
title: FetchSlackMessage 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchslackmessage.md
section: Loading & Unloading Data
---
# FetchSlackMessage 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-slack-processors-nar
## Description
Fetches data about a single Slack message
## Tags
conversation, conversation.history, slack, social media, team, text, unstructured
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Access Token |
OAuth Access Token used for authenticating/authorizing the Slack request sent by NiFi. This may be either a User Token or a Bot Token. It must be granted the channels:history, groups:history, im:history, or mpim:history scope, depending on the type of conversation being used. |
| Channel |
The Slack Channel ID to Retrieve a message from. |
| Include Message Blocks |
Specifies whether or not the output JSON should include the value of the 'blocks' field for each Slack Message. This field includes information such as individual parts of a message that are formatted using rich text. This may be useful, for instance, for parsing. However, it often accounts for a significant portion of the data and as such may be set to null when it is not useful to you. |
| Include Null Fields |
Specifies whether or not fields that have null values should be included in the output JSON. If true, any field in a Slack Message that has a null value will be included in the JSON with a value of null. If false, the key omitted from the output JSON entirely. Omitting null values results in smaller messages that are generally more efficient to process, but including the values may provide a better understanding of the format, especially for schema inference. |
| Message Timestamp |
The timestamp of the message which is also its ID within a channel. |
| Rate Limiter Service |
Slack Rate Limiter Service to coordinate rate limiting across processors |
| Resolve Usernames |
Specifies whether or not User IDs should be resolved to usernames. By default, Slack Messages provide the ID of the user that sends a message, such as U0123456789, but not the username, such as NiFiUser. The username may be resolved, but it may require additional calls to the Slack API and requires that the Token used be granted the users:read scope. If set to true, usernames will be resolved with a best-effort policy: if a username cannot be obtained, it will be skipped over. Also, note that when a username is obtained, the Message's <username> field is populated, and the <text> field is updated such that any mention will be output such as "Hi @user" instead of "Hi <@U1234567>". |
| Thread Timestamp |
The timestamp of the thread the message belongs to. This can be null or empty unless the message is a reply to another message. |
## Relationships
| Name |
Description |
| failure |
Slack messages that fail to be received will be routed to this relationship |
| not found |
Slack messages that were not found on the Slack server will be routed to this relationship |
| success |
Slack messages that are successfully received will be routed to this relationship |
## Writes attributes
| Name |
Description |
| mime.type |
Set to application/json, as the output will always be in JSON format |
---
title: FetchSmb 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchsmb.md
section: Loading & Unloading Data
---
# FetchSmb 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-smb-nar
## Description
Fetches files from a SMB Share. Designed to be used in tandem with ListSmb.
## Tags
cifs, fetch, files, samba, smb
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Completion Strategy |
Specifies what to do with the original file on the server once it has been processed. If the Completion Strategy fails, a warning will be logged but the data will still be transferred. |
| Create Destination Directory |
Specifies whether or not the remote directory should be created if it does not exist. |
| Destination Directory |
The directory on the remote server to move the original file to once it has been processed. |
| remote-file |
The full path of the file to be retrieved from the remote server. Expression language is supported. |
| smb-client-provider-service |
Specifies the SMB client provider to use for creating SMB connections. |
## Relationships
| Name |
Description |
| failure |
A FlowFile will be routed here when failed to fetch its content. |
| success |
A FlowFile will be routed here for each successfully fetched file. |
## Writes attributes
| Name |
Description |
| error.code |
The error code returned by SMB when the fetch of a file fails. |
| error.message |
The error message returned by SMB when the fetch of a file fails. |
## See also
- [org.apache.nifi.processors.smb.GetSmbFile](/user-guide/data-integration/openflow/processors/getsmbfile)
- [org.apache.nifi.processors.smb.ListSmb](/user-guide/data-integration/openflow/processors/listsmb)
- [org.apache.nifi.processors.smb.PutSmbFile](/user-guide/data-integration/openflow/processors/putsmbfile)
---
title: FetchSnowflakeTableProperties 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchsnowflaketableproperties.md
section: Loading & Unloading Data
---
# FetchSnowflakeTableProperties 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-snowflake-processors-nar
## Description
Reads properties from a table and stores them as flow file attributes.
## Tags
database, jdbc, openflow, snowflake
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Connection Pool |
The connection pool to use to connect to Snowflake |
| Schema Name |
The name of the schema |
| Table Metadata Cache Expiration Time |
The time in seconds after which the cache entry will be removed |
| Table Name |
The name of the table |
| Use Table Metadata Cache |
Whether to cache table's metadata instead of reading it directly from Snowflake. |
## Relationships
| Name |
Description |
| failure |
The incoming FlowFile is routed to this relationship if the properties cannot be read |
| success |
The incoming FlowFile is routed to this relationship after the table properties has been successfully read |
| table not found |
The incoming FlowFile is routed to this relationship if the specified table does not exist. |
---
title: FetchSourceTableSchema 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchsourcetableschema.md
section: Loading & Unloading Data
---
# FetchSourceTableSchema 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-database-cdc-processors-nar
## Description
Fetches the table schema (i.e., column names, data types, etc.) for a given table in a database, converting the data types to Snowflake-compatible types. The schema is written to the FlowFile content as a JSON object, in a form such as: \{ "columns": [ \{ "name": "<columnName>", "type": "<snowflakeType>", "nullable": <true|false>, "scale": <scale>, "precision": <precision> \}, ... ], "primaryKeys": ["<primaryKey1>", "<primaryKey2>", ...] \}
## Tags
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Column Filter Service |
Specifies the Column Filter Service to be used for filtering out unwanted columns |
| Connection Pool |
The connection pool to use to fetch the source table schema |
| Schema Name |
The name of the schema that the source table is stored in |
| Table Name |
The name of the source table |
## Relationships
| Name |
Description |
| failure |
FlowFiles are routed to this relationship in the event that the source table's schema cannot be fetched |
| success |
FlowFiles are routed to this relationship when the source table's schema is successfully fetched |
| table not found |
FlowFiles are routed to this relationship when the source table does not exist |
## Writes attributes
| Name |
Description |
| mime.type |
application/json |
| dbms.type |
The type of database management system (DBMS) that the source table is stored in. E.g. _POSTGRESQL_ |
| primary.key.count |
The number of primary keys in the source table |
| column.count |
The number of columns in the source table |
---
title: FetchTableSnapshot 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/fetchtablesnapshot.md
section: Loading & Unloading Data
---
# FetchTableSnapshot 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-database-cdc-processors-nar
## Description
Fetches a snapshot of a table from a database. The snapshot is fetched incrementally, using the primary key columns of the table to fetch rows in batches. Replicating a table without primary key is not supported. The snapshot is written to a FlowFile in the specified Record Writer format. The input FlowFile is expected to consist of a JSON representation of the table schema in the following format: \{ "columns": [\{ "name": "<column name>", "type": "<column type>" \}, \{ "name": "<column name>", "type": "<column type>" \}, ... ], "primaryKeys": ["<name of first primary key column>", "<name of second primary key column>", ...] \} Only those columns that are specified in the schema will be fetched from the table.
## Tags
database, fetch, rdbms, snapshot, snowflake, table
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Connection Pool |
The connection pool to use to fetch the database snapshot |
| Fetch Size |
The maximum number of rows loaded into memory at once |
| JDBC Driver Location |
Comma-separated list of files/folders and/or URLs containing the driver JAR and its dependencies (if any). For example '/var/tmp/postgresql-java-client-42.7.5.jar' |
| Max Batch Size |
The maximum number of rows to fetch in a single batch |
| Record Writer |
The record writer to use to write the fetched snapshot |
| Schema Name |
The name of the schema to fetch the snapshot from |
| Table Name |
The name of the table to fetch the snapshot from |
## Relationships
| Name |
Description |
| complete |
When the snapshot is complete, the original FlowFile will be routed to this relationship |
| failure |
If the data cannot be retrieved from the table represented by the FlowFile, the FlowFile will be routed to this relationship. |
| retryable failure |
If the data cannot be retrieved from the table represented by the FlowFile but we expect it to be possible in future, the FlowFile will be routed to this relationship. |
| rows |
When the snapshot is successfully retrieved from the table represented by the FlowFile, the rows will be routed to this relationship. |
## Writes attributes
| Name |
Description |
| snapshot.complete |
Indicates whether the snapshot is complete |
| rows.total.fetched |
The total number of rows fetched for the table |
| rows.delta.fetched |
The number of rows fetched for the table in the last iteration |
| start.row.index |
The index of the first row within the snapshot for a given iteration, starting from 0 |
| last.row.index |
The index of the last row within the snapshot for a given iteration, starting from 0 |
| fetch.delta.time.in.millis |
The time in milliseconds taken to fetch the rows in the last iteration |
| fetch.total.time.in.millis |
The time in milliseconds taken so far to fetch the rows |
---
title: FilterAttribute 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/filterattribute.md
section: Loading & Unloading Data
---
# FilterAttribute 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Filters the attributes of a FlowFile by retaining specified attributes and removing the rest or by removing specified attributes and retaining the rest.
## Tags
Attribute Expression Language, attributes, delete, filter, modification, regex, regular expression, remove, retain
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Attribute Matching Strategy |
Specifies the strategy to filter attributes by. |
| Filter Mode |
Specifies the strategy to apply on filtered attributes. Either 'Remove' or 'Retain' only the matching attributes. |
| Filtered Attributes |
A set of attribute names to filter from FlowFiles. Each attribute name is separated by the comma delimiter ','. |
| Filtered Attributes Pattern |
A regular expression to match names of attributes to filter from FlowFiles. |
## Relationships
| Name |
Description |
| success |
All successful FlowFiles are routed to this relationship |
## Use cases
| Retain all FlowFile attributes matching a regular expression |
| ------------------------------------------------------------ |
| Remove only a specified set of FlowFile attributes |
---
title: FindConfluencePages 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/findconfluencepages.md
section: Loading & Unloading Data
---
# FindConfluencePages 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-atlassian-processors-nar
## Description
Processor for finding Confluence pages using space name and page name.
## Tags
Preview, atlassian, confluence, fetch, pages
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Confluence Client Service |
Controller service for managing connections to Confluence |
| Confluence Page Name |
Name of the Confluence Page. If not provided, all pages in the space will be retrieved. |
| Confluence Space Name |
Name of the Confluence Space |
## Relationships
| Name |
Description |
| failure |
Failed to find Confluence pages |
| not found |
Pages for given space name and page name not found |
| retry |
Retryable failure occurred, e.g. rate limiting |
| success |
Successfully found Confluence pages |
## Writes attributes
| Name |
Description |
| confluence.page.name |
Unique identifier of the Confluence page. |
| confluence.page.change.type |
Informs about status change for the searched page. |
| confluence.page.url |
Confluence page url. |
| confluence.page.title |
Confluence page title. |
| confluence.page.last.modification.date |
Last modification date of the Confluence page. |
| confluence.space.name |
Name of the Confluence space. |
---
title: FindSharepointDriveItem 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/findsharepointdriveitem.md
section: Loading & Unloading Data
---
# FindSharepointDriveItem 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-msgraph-nar
## Description
Finds a Sharepoint Drive Item by its Drive ID and Item path.
## Tags
document, graph, microsoft, openflow, sharepoint, unstructured
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Authentication Service |
The service that provides authentication for the SharePoint API. |
| Drive ID |
The ID of the Sharepoint Drive. |
| Fallback Retry Duration |
The time to wait before retrying the operation after a communication failure. This value is used when the response doesn't contain a Retry-After header. |
| Item Path |
The path of the Drive Item to find in a Drive. |
## Relationships
| Name |
Description |
| comms.failure |
A FlowFile is routed here if the processor failed to communicate with the Graph API. Can be retried |
| failure |
An incoming FlowFile is routed to this relationship if an unexpected error has occurred |
| found |
An incoming FlowFile is routed to this relationship, with attributes about the Item added, if the specified item was found in Sharepoint |
| not.found |
An incoming FlowFile is routed to this relationship if the specified item was not found in Sharepoint |
## Writes attributes
| Name |
Description |
| sharepoint.item.id |
The ID of the Sharepoint Drive Item. |
| sharepoint.item.type |
The type of the Sharepoint Drive Item, possible values are 'File' and 'Folder'. |
## See also
- [com.snowflake.openflow.runtime.processors.sharepoint.FetchSharepointFile](/user-guide/data-integration/openflow/processors/fetchsharepointfile)
- [com.snowflake.openflow.runtime.processors.sharepoint.ListSharepointDrives](/user-guide/data-integration/openflow/processors/listsharepointdrives)
---
title: FlattenJson 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/flattenjson.md
section: Loading & Unloading Data
---
# FlattenJson 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Provides the user with the ability to take a nested JSON document and flatten it into a simple key/value pair document. The keys are combined at each level with a user-defined separator that defaults to '.'. This Processor also lets you unflatten the flattened JSON. It supports four kinds of flatten mode such as normal, keep-arrays, dot notation for MongoDB query and keep-primitive-arrays. Default flatten mode is 'keep-arrays'.
## Tags
flatten, json, unflatten
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| flatten-json-character-set |
The Character Set in which file is encoded |
| flatten-json-pretty-print-json |
Specifies whether or not resulted json should be pretty printed |
| flatten-json-return-type |
Specifies the desired return type of json such as flatten/unflatten |
| flatten-json-separator |
The separator character used for joining keys. Must be a JSON-legal character. |
| flatten-mode |
Specifies how json should be flattened/unflattened |
| ignore-reserved-characters |
If true, reserved characters in keys will be ignored |
## Relationships
| Name |
Description |
| failure |
Files that cannot be flattened/unflattened go to this relationship. |
| success |
Successfully flattened/unflattened files go to this relationship. |
---
title: ForkEnrichment 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/forkenrichment.md
section: Loading & Unloading Data
---
# ForkEnrichment 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Used in conjunction with the JoinEnrichment processor, this processor is responsible for adding the attributes that are necessary for the JoinEnrichment processor to perform its function. Each incoming FlowFile will be cloned. The original FlowFile will have appropriate attributes added and then be transferred to the 'original' relationship. The clone will have appropriate attributes added and then be routed to the 'enrichment' relationship. See the documentation for the JoinEnrichment processor (and especially its Additional Details) for more information on how these Processors work together and how to perform enrichment tasks in NiFi by using these Processors.
## Tags
enrich, fork, join, record
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Relationships
| Name |
Description |
| enrichment |
A clone of the incoming FlowFile will be routed to this relationship, after adding appropriate attributes. |
| original |
The incoming FlowFile will be routed to this relationship, after adding appropriate attributes. |
## Writes attributes
| Name |
Description |
| enrichment.group.id |
The Group ID to use in order to correlate the 'original' FlowFile with the 'enrichment' FlowFile. |
| enrichment.role |
The role to use for enrichment. This will either be ORIGINAL or ENRICHMENT. |
## See also
- [org.apache.nifi.processors.standard.JoinEnrichment](/user-guide/data-integration/openflow/processors/joinenrichment)
---
title: ForkRecord 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/forkrecord.md
section: Loading & Unloading Data
---
# ForkRecord 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
This processor allows the user to fork a record into multiple records. The user must specify at least one Record Path, as a dynamic property, pointing to a field of type ARRAY containing RECORD objects. The processor accepts two modes: 'split' and 'extract'. In both modes, there is one record generated per element contained in the designated array. In the 'split' mode, each generated record will preserve the same schema as given in the input but the array will contain only one element. In the 'extract' mode, the element of the array must be of record type and will be the generated record. Additionally, in the 'extract' mode, it is possible to specify if each generated record should contain all the fields of the parent records from the root level to the extracted record. This assumes that the fields to add in the record are defined in the schema of the Record Writer controller service. See examples in the additional details documentation of this processor.
## Tags
array, content, event, fork, record, stream
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| fork-mode |
Specifies the forking mode of the processor |
| include-parent-fields |
This parameter is only valid with the 'extract' mode. If set to true, all the fields from the root level to the given array will be added as fields of each element of the array to fork. |
| record-reader |
Specifies the Controller Service to use for reading incoming data |
| record-writer |
Specifies the Controller Service to use for writing out the records |
## Relationships
| Name |
Description |
| failure |
In case a FlowFile generates an error during the fork operation, it will be routed to this relationship |
| fork |
The FlowFiles containing the forked records will be routed to this relationship |
| original |
The original FlowFiles will be routed to this relationship |
## Writes attributes
| Name |
Description |
| record.count |
The generated FlowFile will have a 'record.count' attribute indicating the number of records that were written to the FlowFile. |
| mime.type |
The MIME Type indicated by the Record Writer |
| <Attributes from Record Writer> |
Any Attribute that the configured Record Writer returns will be added to the FlowFile. |
---
title: FreeFormTextRecordSetWriter
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/freeformtextrecordsetwriter.md
section: Loading & Unloading Data
---
# FreeFormTextRecordSetWriter
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Writes the contents of a RecordSet as free-form text. The configured text is able to make use of the Expression Language to reference each of the fields that are available in a Record, as well as the attributes in the FlowFile and variables. If there is a name collision, the field name/value is used before attributes or variables. Each record in the RecordSet will be separated by a single newline character.
## Tags
el, expression, freeform, language, record, recordset, resultset, serialize, text, writer
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Character Set * |
Character Set |
UTF-8 |
|
The Character set to use when writing the data to the FlowFile |
| Text * |
Text |
|
|
The text to use when writing the results. This property will evaluate the Expression Language using any of the fields available in a Record. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: GCPCredentialsControllerService
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/gcpcredentialscontrollerservice.md
section: Loading & Unloading Data
---
# GCPCredentialsControllerService
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Defines credentials for Google Cloud Platform processors. Uses Application Default credentials without configuration. Application Default credentials support environmental variable (GOOGLE_APPLICATION_CREDENTIALS) pointing to a credential file, the config generated by *gcloud auth application-default login*, AppEngine/Compute Engine service accounts, etc.
## Tags
credentials, gcp, provider
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Delegation Strategy * |
Delegation Strategy |
Service Account |
- Service Account
- Delegated Account
|
The Delegation Strategy determines which account is used when calls are made with the GCP Credential. |
| Delegation User * |
Delegation User |
|
|
This user will be impersonated by the service account for api calls. API calls made using this credential will appear as if they are coming from delegate user with the delegate user's access. Any scopes supplied from processors to this credential must have domain-wide delegation setup with the service account. |
| Use Application Default Credentials |
application-default-credentials |
false |
- true
- false
|
If true, uses Google Application Default Credentials, which checks the GOOGLE_APPLICATION_CREDENTIALS environment variable for a filepath to a service account JSON key, the config generated by the gcloud sdk, the App Engine service account, and the Compute Engine service account. |
| Use Compute Engine Credentials |
compute-engine-credentials |
false |
- true
- false
|
If true, uses Google Compute Engine Credentials of the Compute Engine VM Instance which NiFi is running on. |
| Proxy Configuration Service |
proxy-configuration-service |
|
|
Specifies the Proxy Configuration Controller Service to proxy network requests. |
| Service Account JSON |
service-account-json |
|
|
The raw JSON containing a Service Account keyfile. |
| Service Account JSON File |
service-account-json-file |
|
|
Path to a file containing a Service Account key file in JSON format. |
## State management
This component does not store state.
## Restricted
## Restrictions
| Required Permission |
Explanation |
| access environment credentials |
The default configuration can read environment variables and system properties for credentials |
## System Resource Considerations
This component does not specify system resource considerations.
---
title: GCSFileResourceService
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/gcsfileresourceservice.md
section: Loading & Unloading Data
---
# GCSFileResourceService
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides a Google Compute Storage (GCS) file resource for other components.
## Tags
file, gcs, resource
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Bucket * |
Bucket |
$\{gcs.bucket\} |
|
Bucket of the object. |
| Name * |
Name |
$\{filename\} |
|
Name of the object. |
| GCP Credentials Provider Service * |
gcp-credentials-provider-service |
|
|
The Controller Service used to obtain Google Cloud Platform credentials. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: gen 2 connector configuration and versioning
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/gen2/connector-versioning.md
section: Loading & Unloading Data
---
# Gen 2 connector configuration and versioning
Available to all accounts.
- [Second generation Openflow objects and interfaces](/user-guide/data-integration/openflow/gen2/index)
- [Configure a connector with the setup wizard](/user-guide/data-integration/openflow/gen2/setup-connector-wizard)
- [Configure a gen 2 connector with SQL](/user-guide/data-integration/openflow/gen2/configure-connector-sql)
- [Manage the gen 2 Openflow connector lifecycle](/user-guide/data-integration/openflow/gen2/manage-connector-lifecycle)
- [Using a Git repository in Snowflake](/developer-guide/git/git-overview)
**Using the setup wizard or** **Installed Connectors** **only?** You do not need this topic for
day-to-day work. The Openflow UI creates and manages configuration versions automatically as you
install, edit, and save connector settings. Labels such as **Draft** and **Edits not applied**
reflect the same version states described below—you can follow the wizard and UI prompts without
reading about stages or `COMMIT`.
Read this topic if you configure connectors with SQL, `GET`/`PUT` on stages, Git, or other
automation—or if you want to understand what the UI is doing under the hood.
Gen 2 Openflow connectors are **File Based Entities (FBEs)**: Snowflake objects whose
configuration is stored as files on an internal **versioned stage** that Snowflake creates and
manages on the connector object. You do not create this stage with `CREATE STAGE`—it is attached
automatically when the connector is created. Configuration files (for example, `config.json`)
and assets (for example, JDBC driver JARs) live under versioned paths on that stage.
This topic explains the version states, UI labels, SQL workflow for editing connector configuration,
and how to list, download, and upload files on the connector stage. Other Snowflake object types
(for example, dbt projects and Cortex agents) use a similar versioned-stage model with their own
`snow://` URI schemes; here we cover Openflow connectors only.
## Access the connector's versioned stage
Each connector version is an immutable snapshot in the connector's version history. Versions behave
similarly to Git commits: if a new version does not change a file, both versions can reference the
same underlying file. Use the `snow://openflow_connector/` URI scheme to reference files:
```
snow://openflow_connector/../versions//[]
```
version is one of:
- `live` — writable working copy (see [](#label-openflow-fbe-version-states) below)
- `LAST` — alias for the most recently committed (default) version
- `VERSION$N` — a specific committed version (for example, `VERSION$1`)
- A user-assigned name — set when you commit; preserved for later reference (for example,
`production-config`)
You can use a user-assigned name or `VERSION$N` in `snow://` paths and SQL when referencing a
committed version.
List, download, and upload files with the standard stage commands documented in
[File staging commands](/sql-reference/commands-file) (`LIST`, `GET`, and `PUT`). Pass the URI as a quoted string (for
example, `GET 'snow://openflow_connector/...'`).
Snowsight does not support `GET` or `PUT` on connector stages. Use %sf-cli% or another client
that supports stage file operations.
You can also inspect version metadata with `SHOW VERSIONS IN OPENFLOW CONNECTOR` and connector
properties (including the active version) with `DESCRIBE OPENFLOW CONNECTOR`. See
[SHOW VERSIONS IN OPENFLOW CONNECTOR](/sql-reference/sql/show-versions-in-openflow-connector) and [DESCRIBE OPENFLOW CONNECTOR](/sql-reference/sql/desc-openflow-connector).
## Version states
| Term |
Description |
SQL path |
| **live** |
In-progress edits (writable stage). Created automatically when a connector is created via UI or SQL `FROM DEFINITION`. |
`versions/live` |
| **default** |
The committed version the runtime runs when the connector starts. |
`versions/LAST` (alias) |
| **LAST** |
Alias for the most recently committed (default) version. |
`versions/LAST` |
The **live** version is your working copy. The **default** version is what runs when the connector
starts in the target runtime. In stage paths, `versions/LAST` points to the **default** (committed)
version. Committing the live version creates a new immutable default and removes the live version.
While a connector is **RUNNING**, you can still edit the live version. Changes are not applied to
the running connector until you **COMMIT** (UI **Apply** or SQL `COMMIT`).
## UI labels
| UI label |
Meaning |
| **Draft** |
Live version exists; no default version yet. The connector was created but never committed. Commit before starting. |
| **Edits not applied** |
Both live and default versions exist. Changes are in progress but not committed. |
| *(no label)* |
Only a default version exists. Stable state. |
## Configuration workflow
New connectors created with `CREATE OPENFLOW CONNECTOR ... FROM DEFINITION` or the setup wizard
start with a live version and no default (**Draft**).
1. Upload or edit files in the live version (skip `ADD LIVE VERSION` on a new connector—it already
has a live version).
2. Commit to promote live to default:
```sql
ALTER OPENFLOW CONNECTOR my_db.my_schema.my_connector COMMIT;
```
3. To discard uncommitted changes:
```sql
ALTER OPENFLOW CONNECTOR my_db.my_schema.my_connector ABORT;
```
4. To edit a committed connector, create a new live version seeded from the current default:
```sql
ALTER OPENFLOW CONNECTOR my_db.my_schema.my_connector ADD LIVE VERSION FROM LAST;
```
## Stage file operations
The following examples use the [snow://openflow_connector/ URI](#label-openflow-fbe-versioned-stage).
Replace `my_db.my_schema.my_connector` with your connector's fully qualified name.
List files in the live version:
```sql
LS 'snow://openflow_connector/my_db.my_schema.my_connector/versions/live';
```
Download the last committed `config.json` (use `snow sql` or another client that supports
`GET` on stages; Snowsight does not support `GET`/`PUT` on stages):
```sql
GET 'snow://openflow_connector/my_db.my_schema.my_connector/versions/LAST/config.json'
file:///path/to/local/;
```
Upload to the live version:
```sql
PUT 'file:///path/to/config.json'
'snow://openflow_connector/my_db.my_schema.my_connector/versions/live/config.json'
AUTO_COMPRESS = FALSE
OVERWRITE = TRUE;
```
After uploading, commit:
```sql
ALTER OPENFLOW CONNECTOR my_db.my_schema.my_connector COMMIT;
SELECT SYSTEM$WAIT_FOR_STABLE_OPENFLOW_CONNECTORS(600, 'my_db.my_schema.my_connector');
```
## Create from a known configuration
Use this workflow when you already have a **validated** connector configuration and want to create
another connector with the same settings—for example, standing up a matching connector in a second
runtime, or automating repeat deployments with CI/CD.
This is **not** the path for your first connector. Create and configure a connector once with the
setup wizard or `CREATE OPENFLOW CONNECTOR ... FROM DEFINITION`, then export the configuration.
### Definition vs configuration
A catalog **definition** (for example, `OPENFLOW_POSTGRES_CDC`) and instance **configuration**
(`config.json`) are separate:
- `FROM DEFINITION` in `CREATE OPENFLOW CONNECTOR` selects the catalog connector type. You then
edit `config.json` on the connector's versioned stage and **COMMIT**.
- `FROM` a stage path supplies a complete configuration bundle from any stage reference—a Git
repository stage (`@my_git_repo/...`), another connector's stage in the same account
(`snow://openflow_connector/...`), or another internal stage. The catalog definition is named
inside `config.json` as `connectorDefinitionId`—there is no `FROM DEFINITION` clause in the
`CREATE` command.
When you use `FROM` with a stage path, Openflow reads `connectorDefinitionId` from `config.json`
to determine the connector type—you do not use `FROM DEFINITION` in `CREATE`. The stage holds only
instance configuration: `config.json`, asset files (such as JDBC drivers), and metadata. Snowflake
supplies the catalog connector package at create time; you do not copy it onto the stage.
Before you run `CREATE`, update connection URLs, secret references, and destination settings for
the target runtime.
### Initial state difference
| Created via |
Initial state |
| UI or SQL `FROM DEFINITION` |
Live version exists; no default (**Draft**). Commit before starting. |
| Stage `FROM '@[/path/]'` or `snow://...` |
Default version exists; no live version. Ready to start immediately (no **COMMIT** step). |
### Workflow
1. Create and configure a connector with the wizard or `FROM DEFINITION`. Commit the configuration
when it works in your source environment.
2. Register a Git repository in Snowflake if you do not already have one. See
[Using a Git repository in Snowflake](/developer-guide/git/git-overview).
3. Export the connector configuration:
```sql
ALTER OPENFLOW CONNECTOR my_db.my_schema.my_connector PUSH TO
'@my_git_repo/branches/main/connectors/my_connector'
USERNAME = 'my-git-username'
PASSWORD = 'my-git-token'
NAME = 'My Name'
EMAIL = 'my.email@example.com'
COMMENT = 'Export connector config';
```
4. Review and update `config.json` in the repository for the target environment (secrets, URLs,
destination settings).
5. Create the new connector from the stage path:
```sql
CREATE OPENFLOW CONNECTOR my_db.my_schema.my_connector_prod
IN RUNTIME my_db.my_schema.my_prod_runtime
FROM '@my_git_repo/branches/main/connectors/my_connector/'
COMMENT = 'Created from validated stage config';
```
You can also use a `snow://openflow_connector/.../versions/LAST/` URI to clone a connector in
the same account without Git—for example,
`'snow://openflow_connector/my_db.my_schema.my_connector/versions/LAST/'`.
For `CREATE`, `ADD VERSION FROM`, `PUSH`, and `PULL` syntax, see [CREATE OPENFLOW CONNECTOR](/sql-reference/sql/create-openflow-connector) and [ALTER OPENFLOW CONNECTOR](/sql-reference/sql/alter-openflow-connector).
## Secrets in configuration
Sensitive values (for example, database passwords) should reference
[Snowflake secrets](/sql-reference/sql/create-secret) rather than plain text in
`config.json`. See [Secrets in configuration](#label-openflow-configure-connector-sql-secrets) in [Configure a gen 2 connector with SQL](/user-guide/data-integration/openflow/gen2/configure-connector-sql).
---
title: GenerateAnswersFromContext 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/generateanswersfromcontext.md
section: Loading & Unloading Data
---
# GenerateAnswersFromContext 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-rag-evaluation-processors-nar
## Description
Generates synthetic answers for each question present in the incoming records using a Large Language Model (LLM). For every record, the processor extracts the question and its associated context based on the specified RecordPaths, constructs a prompt, and sends it to an LLM provider to obtain a synthetic answer. The generated answer is then inserted into the record at the designated RecordPath.
## Tags
ai, answers, contextual, generation, llm, nlp, openai, openflow, rag, synthetic
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Answer Record Path |
The RecordPath to the synthetically generated answers |
| Context Record Path |
The RecordPath to the array of contexts in the record. |
| LLM Provider Service |
The provider service for sending evaluation prompts to LLM |
| Max Character Context Length |
Maximum character length of context window. |
| Question Record Path |
The RecordPath to the question field in the record. |
| Record Reader |
The Record Reader to use for reading the FlowFile. |
| Record Writer |
The Record Writer to use for writing the results. |
## Relationships
| Name |
Description |
| failure |
FlowFiles that cannot be processed are routed to this relationship |
| success |
FlowFiles that are successfully processed are routed to this relationship |
## Writes attributes
| Name |
Description |
| answers.successfully.generated |
The total number of successfully generated synthetic answers for the FlowFile. |
| answers.failed.generated |
The total number of synthetic answer generation attempts that failed for the FlowFile. |
| json.parse.failures |
Number of JSON parse failures encountered. |
---
title: GenerateAnswersFromGroundTruth 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/generateanswersfromgroundtruth.md
section: Loading & Unloading Data
---
# GenerateAnswersFromGroundTruth 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-rag-evaluation-processors-nar
## Description
Generates synthetic answers for each question in the incoming records using an LLM. The synthetic answers are added to the specified RecordPath within each record. Additionally, the processor tracks the number of answers generated and updates the FlowFile attributes accordingly.
## Tags
ai, answers, generation, llm, nlp, openai, openflow, rag, synthetic
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Answer Record Path |
The RecordPath to the synthetically generated answers. |
| Ground Truth Record Path |
The RecordPath to the ground truth field in the record. |
| LLM Provider Service |
The provider service for sending evaluation prompts to LLM |
| Question Record Path |
The RecordPath to the question field in the record. |
| Record Reader |
The Record Reader to use for reading the FlowFile. |
| Record Writer |
The Record Writer to use for writing the results. |
## Relationships
| Name |
Description |
| failure |
FlowFiles that cannot be processed are routed to this relationship |
| success |
FlowFiles that are successfully processed are routed to this relationship |
## Writes attributes
| Name |
Description |
| answers.successfully.generated |
The total number of successfully synthetic answers generated for the FlowFile. |
| answers.failed.generated |
The total number of failed answer generation for the FlowFile. |
| json.parse.failures |
Number of JSON parse failures encountered. |
---
title: GenerateFlowFile 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/generateflowfile.md
section: Loading & Unloading Data
---
# GenerateFlowFile 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
This processor creates FlowFiles with random data or custom content. GenerateFlowFile is useful for load testing, configuration, and simulation. Also see DuplicateFlowFile for additional load testing.
## Tags
generate, load, random, test
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Batch Size |
The number of FlowFiles to be transferred in each invocation |
| Data Format |
Specifies whether the data should be Text or Binary |
| File Size |
The size of the file that will be used |
| Unique FlowFiles |
If true, each FlowFile that is generated will be unique. If false, a random value will be generated and all FlowFiles will get the same content but this offers much higher throughput |
| character-set |
Specifies the character set to use when writing the bytes of Custom Text to a flow file. |
| generate-ff-custom-text |
If Data Format is text and if Unique FlowFiles is false, then this custom text will be used as content of the generated FlowFiles and the File Size will be ignored. Finally, if Expression Language is used, evaluation will be performed only once per batch of generated FlowFiles |
| mime-type |
Specifies the value to set for the "mime.type" attribute. |
## Relationships
| Name |
Description |
| success |
|
## Writes attributes
| Name |
Description |
| mime.type |
Sets the MIME type of the output if the 'Mime Type' property is set |
---
title: GenerateJSON 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/generatejson.md
section: Loading & Unloading Data
---
# GenerateJSON 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-record-generation-nar
## Description
Produces a batch of JSON Objects with random field values based on a configurable JSON Schema.
## Tags
JSON, JSON Schema, generate, random
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Batch Size |
Number of records generated per FlowFile produced |
| JSON Schema |
JSON Schema version 2020-12 describing an object with properties indicating type and format for each field |
| Output Structure |
Structure for writing batches of records to each FlowFile |
## Relationships
| Name |
Description |
| success |
FlowFiles with generated JSON records |
---
title: GenerateRecord 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/generaterecord.md
section: Loading & Unloading Data
---
# GenerateRecord 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
This processor creates FlowFiles with records having random value for the specified fields. GenerateRecord is useful for testing, configuration, and simulation. It uses either user-defined properties to define a record schema or a provided schema and generates the specified number of records using random data for the fields in the schema.
## Tags
fake, generate, random, test
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| null-percentage |
The percent probability (0-100%) that a generated value for any nullable field will be null. Set this property to zero to have no null values, or 100 to have all null values. |
| nullable-fields |
Whether the generated fields will be nullable. Note that this property is ignored if Schema Text is set. Also it only affects the schema of the generated data, not whether any values will be null. If this property is true, see 'Null Value Percentage' to set the probability that any generated field will be null. |
| number-of-records |
Specifies how many records will be generated for each outgoing FlowFile. |
| record-writer |
Specifies the Controller Service to use for writing out the records |
| schema-text |
The text of an Avro-formatted Schema used to generate record data. If this property is set, any user-defined properties are ignored. |
## Relationships
| Name |
Description |
| success |
FlowFiles that are successfully created will be routed to this relationship |
## Writes attributes
| Name |
Description |
| mime.type |
Sets the mime.type attribute to the MIME Type specified by the Record Writer |
| record.count |
The number of records in the FlowFile |
---
title: GenerateTableFetch 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/generatetablefetch.md
section: Loading & Unloading Data
---
# GenerateTableFetch 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Generates SQL select queries that fetch "pages" of rows from a table. The partition size property, along with the table 's row count, determine the size and number of pages and generated FlowFiles. In addition, incremental fetching can be achieved by setting Maximum-Value Columns, which causes the processor to track the columns' maximum values, thus only fetching rows whose columns 'values exceed the observed maximums. This processor is intended to be run on the Primary Node only. This processor can accept incoming connections; the behavior of the processor is different whether incoming connections are provided: - If no incoming connection(s) are specified, the processor will generate SQL queries on the specified processor schedule. Expression Language is supported for many fields, but no FlowFile attributes are available. However the properties will be evaluated using the Environment/System properties. - If incoming connection(s) are specified and no FlowFile is available to a processor task, no work will be performed. - If incoming connection(s) are specified and a FlowFile is available to a processor task, the FlowFile's attributes may be used in Expression Language for such fields as Table Name and others. However, the Max-Value Columns and Columns to Return fields must be empty or refer to columns that are available in each specified table.
## Tags
database, fetch, generate, jdbc, query, select, sql
## Input Requirement
ALLOWED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Columns to Return |
A comma-separated list of column names to be used in the query. If your database requires special treatment of the names (quoting, e.g.), each name should include such treatment. If no column names are supplied, all columns in the specified table will be returned. NOTE: It is important to use consistent column names for a given table for incremental fetch to work properly. |
| Database Connection Pooling Service |
The Controller Service that is used to obtain a connection to the database. |
| Database Dialect Service |
Database Dialect Service for generating statements specific to a particular service or vendor. |
| Max Wait Time |
The maximum amount of time allowed for a running SQL select query , zero means there is no limit. Max time less than 1 second will be equal to zero. |
| Maximum-value Columns |
A comma-separated list of column names. The processor will keep track of the maximum value for each column that has been returned since the processor started running. Using multiple columns implies an order to the column list, and each column 's values are expected to increase more slowly than the previous columns' values. Thus, using multiple columns implies a hierarchical structure of columns, which is usually used for partitioning tables. This processor can be used to retrieve only those rows that have been added/updated since the last retrieval. Note that some JDBC types such as bit/boolean are not conducive to maintaining maximum value, so columns of these types should not be listed in this property, and will result in error(s) during processing. If no columns are provided, all rows from the table will be considered, which could have a performance impact. NOTE: It is important to use consistent max-value column names for a given table for incremental fetch to work properly. |
| Table Name |
The name of the database table to be queried. |
| db-fetch-db-type |
Database Type for generating statements specific to a particular service or vendor. The Generic Type supports most cases but selecting a specific type enables optimal processing or additional features. |
| db-fetch-where-clause |
A custom clause to be added in the WHERE condition when building SQL queries. |
| gen-table-column-for-val-partitioning |
The name of a column whose values will be used for partitioning. The default behavior is to use row numbers on the result set for partitioning into 'pages' to be fetched from the database, using an offset/limit strategy. However for certain databases, it can be more efficient under the right circumstances to use the column values themselves to define the 'pages'. This property should only be used when the default queries are not performing well, when there is no maximum-value column or a single maximum-value column whose type can be coerced to a long integer (i.e. not date or timestamp), and the column values are evenly distributed and not sparse, for best performance. |
| gen-table-custom-orderby-column |
The name of a column to be used for ordering the results if Max-Value Columns are not provided and partitioning is enabled. This property is ignored if either Max-Value Columns is set or Partition Size = 0. NOTE: If neither Max-Value Columns nor Custom ORDER BY Column is set, then depending on the database/driver, the processor may report an error and/or the generated SQL may result in missing and/or duplicate rows. This is because without an explicit ordering, fetching each partition is done using an arbitrary ordering. |
| gen-table-fetch-partition-size |
The number of result rows to be fetched by each generated SQL statement. The total number of rows in the table divided by the partition size gives the number of SQL statements (i.e. FlowFiles) generated. A value of zero indicates that a single FlowFile is to be generated whose SQL statement will fetch all rows in the table. |
| gen-table-output-flowfile-on-zero-results |
Depending on the specified properties, an execution of this processor may not result in any SQL statements generated. When this property is true, an empty FlowFile will be generated (having the parent of the incoming FlowFile if present) and transferred to the 'success' relationship. When this property is false, no output FlowFiles will be generated. |
## State management
| Scopes |
Description |
| CLUSTER |
After performing a query on the specified table, the maximum values for the specified column(s) will be retained for use in future executions of the query. This allows the Processor to fetch only those records that have max values greater than the retained values. This can be used for incremental fetching, fetching of newly added rows, etc. To clear the maximum values, clear the state of the processor per the State Management documentation |
## Relationships
| Name |
Description |
| failure |
This relationship is only used when SQL query execution (using an incoming FlowFile) failed. The incoming FlowFile will be penalized and routed to this relationship. If no incoming connection(s) are specified, this relationship is unused. |
| success |
Successfully created FlowFile from SQL query result set. |
## Writes attributes
| Name |
Description |
| generatetablefetch.sql.error |
If the processor has incoming connections, and processing an incoming FlowFile causes a SQL Exception, the FlowFile is routed to failure and this attribute is set to the exception message. |
| generatetablefetch.tableName |
The name of the database table to be queried. |
| generatetablefetch.columnNames |
The comma-separated list of column names used in the query. |
| generatetablefetch.whereClause |
Where clause used in the query to get the expected rows. |
| generatetablefetch.maxColumnNames |
The comma-separated list of column names used to keep track of data that has been returned since the processor started running. |
| generatetablefetch.limit |
The number of result rows to be fetched by the SQL statement. |
| generatetablefetch.offset |
Offset to be used to retrieve the corresponding partition. |
| fragment.identifier |
All FlowFiles generated from the same query result set will have the same value for the fragment.identifier attribute. This can then be used to correlate the results. |
| fragment.count |
This is the total number of FlowFiles produced by a single ResultSet. This can be used in conjunction with the fragment.identifier attribute in order to know how many FlowFiles belonged to the same incoming ResultSet. |
| fragment.index |
This is the position of this FlowFile in the list of outgoing FlowFiles that were all generated from the same execution. This can be used in conjunction with the fragment.identifier attribute to know which FlowFiles originated from the same execution and in what order FlowFiles were produced |
## See also
- [org.apache.nifi.processors.standard.ExecuteSQL](/user-guide/data-integration/openflow/processors/executesql)
- [org.apache.nifi.processors.standard.ListDatabaseTables](/user-guide/data-integration/openflow/processors/listdatabasetables)
- [org.apache.nifi.processors.standard.QueryDatabaseTable](/user-guide/data-integration/openflow/processors/querydatabasetable)
---
title: GeoEnrichIP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/geoenrichip.md
section: Loading & Unloading Data
---
# GeoEnrichIP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-enrich-nar
## Description
Looks up geolocation information for an IP address and adds the geo information to FlowFile attributes. The geo data is provided as a MaxMind database. The attribute that contains the IP address to lookup is provided by the 'IP Address Attribute' property. If the name of the attribute provided is 'X', then the attributes added by enrichment will take the form X.geo.<fieldName>
## Tags
enrich, geo, ip, maxmind
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| IP Address Attribute |
The name of an attribute whose value is a dotted decimal IP address for which enrichment should occur |
| Log Level |
The Log Level to use when an IP is not found in the database. Accepted values: INFO, DEBUG, WARN, ERROR. |
| MaxMind Database File |
Path to Maxmind IP Enrichment Database File |
## Relationships
| Name |
Description |
| found |
Where to route flow files after successfully enriching attributes with data provided by database |
| not found |
Where to route flow files after unsuccessfully enriching attributes because no data was found |
## Writes attributes
| Name |
Description |
| X.geo.lookup.micros |
The number of microseconds that the geo lookup took |
| X.geo.city |
The city identified for the IP address |
| X.geo.accuracy |
The accuracy radius if provided by the database (in Kilometers) |
| X.geo.latitude |
The latitude identified for this IP address |
| X.geo.longitude |
The longitude identified for this IP address |
| X.geo.subdivision.N |
Each subdivision that is identified for this IP address is added with a one-up number appended to the attribute name, starting with 0 |
| X.geo.subdivision.isocode.N |
The ISO code for the subdivision that is identified by X.geo.subdivision.N |
| X.geo.country |
The country identified for this IP address |
| X.geo.country.isocode |
The ISO Code for the country identified |
| X.geo.postalcode |
The postal code for the country identified |
---
title: GeoEnrichIPRecord 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/geoenrichiprecord.md
section: Loading & Unloading Data
---
# GeoEnrichIPRecord 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-enrich-nar
## Description
Looks up geolocation information for an IP address and adds the geo information to FlowFile attributes. The geo data is provided as a MaxMind database. This version uses the NiFi Record API to allow large scale enrichment of record-oriented data sets. Each field provided by the MaxMind database can be directed to a field of the user's choosing by providing a record path for that field configuration.
## Tags
enrich, geo, ip, maxmind, record
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| City Record Path |
Record path for putting the city identified for the IP address |
| Country ISO Code Record Path |
Record path for putting the ISO Code for the country identified |
| Country Postal Code Record Path |
Record path for putting the postal code for the country identified |
| Country Record Path |
Record path for putting the country identified for this IP address |
| IP Address Record Path |
The record path to retrieve the IP address for doing the lookup. |
| Latitude Record Path |
Record path for putting the latitude identified for this IP address |
| Log Level |
The Log Level to use when an IP is not found in the database. Accepted values: INFO, DEBUG, WARN, ERROR. |
| Longitude Record Path |
Record path for putting the longitude identified for this IP address |
| MaxMind Database File |
Path to Maxmind IP Enrichment Database File |
| Record Reader |
Record reader service to use for reading the flowfile contents. |
| Record Writer |
Record writer service to use for enriching the flowfile contents. |
| Separate Enriched From Not Enriched |
Separate records that have been enriched from ones that have not. Default behavior is to send everything to the found relationship if even one record is enriched. |
## Relationships
| Name |
Description |
| found |
Where to route flow files after successfully enriching attributes with data provided by database |
| not found |
Where to route flow files after unsuccessfully enriching attributes because no data was found |
| original |
The original input flowfile goes to this relationship regardless of whether the content was enriched or not. |
---
title: GetAmazonAdsReport 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getamazonadsreport.md
section: Loading & Unloading Data
---
# GetAmazonAdsReport 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-amazon-ads-processors-nar
## Description
Processor downloading report from Amazon Ads if ready.
## Tags
Amazon, Amazon Ads, report
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Access Token Provider |
Service providing OAuth access token. |
| Amazon Advertising Client ID |
Client ID of the Amazon Advertising user. |
| Region |
Environment from which advertising data will be downloaded. |
| Report ID |
ID of the generated report. |
| Report Profile ID |
The profile ID associated with an advertising account in a specific marketplace. |
| Web Client Service Provider |
Service providing client for REST request execution. |
## Relationships
| Name |
Description |
| failure |
Error FlowFiles transferred when receiving error response from Amazon Ads Reporting API or when an error occurred during response processing. |
| retry |
Response FlowFiles transferred when report prepared by Amazon Ads Reporting API is not yet ready to be downloaded. |
| success |
Response FlowFiles transferred when receiving COMPLETED response from Amazon Ads Reporting API. |
## Writes attributes
| Name |
Description |
| mime.type |
Mime type of the returned report. |
---
title: GetAwsPollyJobStatus 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getawspollyjobstatus.md
section: Loading & Unloading Data
---
# GetAwsPollyJobStatus 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-aws-nar
## Description
Retrieves the current status of an AWS Polly job.
## Tags
AWS, Amazon, ML, Machine Learning, Polly
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| AWS Credentials Provider service |
The Controller Service that is used to obtain AWS credentials provider |
| AWS Task ID |
|
| Communications Timeout |
|
| Endpoint Override URL |
Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. |
| Region |
|
| SSL Context Service |
Specifies an optional SSL Context Service that, if provided, will be used to create connections |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| failure |
The job failed, the original FlowFile will be routed to this relationship. |
| original |
Upon successful completion, the original FlowFile will be routed to this relationship. |
| running |
The job is currently still being processed |
| success |
Job successfully finished. FlowFile will be routed to this relation. |
## Writes attributes
| Name |
Description |
| PollyS3OutputBucket |
The bucket name where polly output will be located. |
| filename |
Object key of polly output. |
| outputLocation |
S3 path-style output location of the result. |
## See also
- [org.apache.nifi.processors.aws.ml.polly.StartAwsPollyJob](/user-guide/data-integration/openflow/processors/startawspollyjob)
---
title: GetAwsTextractJobStatus 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getawstextractjobstatus.md
section: Loading & Unloading Data
---
# GetAwsTextractJobStatus 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-aws-nar
## Description
Retrieves the current status of an AWS Textract job.
## Tags
AWS, Amazon, ML, Machine Learning, Textract
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| AWS Credentials Provider service |
The Controller Service that is used to obtain AWS credentials provider |
| AWS Task ID |
|
| Communications Timeout |
|
| Endpoint Override URL |
Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. |
| Region |
|
| SSL Context Service |
Specifies an optional SSL Context Service that, if provided, will be used to create connections |
| Textract Type |
Supported values: "Document Analysis", "Document Text Detection", "Expense Analysis" |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| failure |
The job failed, the original FlowFile will be routed to this relationship. |
| original |
Upon successful completion, the original FlowFile will be routed to this relationship. |
| running |
The job is currently still being processed |
| success |
Job successfully finished. FlowFile will be routed to this relation. |
| throttled |
Retrieving results failed for some reason, but the issue is likely to resolve on its own, such as Provisioned Throughput Exceeded or a Throttling failure. It is generally expected to retry this relationship. |
## See also
- [org.apache.nifi.processors.aws.ml.textract.StartAwsTextractJob](/user-guide/data-integration/openflow/processors/startawstextractjob)
---
title: GetAwsTranscribeJobStatus 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getawstranscribejobstatus.md
section: Loading & Unloading Data
---
# GetAwsTranscribeJobStatus 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-aws-nar
## Description
Retrieves the current status of an AWS Transcribe job.
## Tags
AWS, Amazon, ML, Machine Learning, Transcribe
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| AWS Credentials Provider service |
The Controller Service that is used to obtain AWS credentials provider |
| AWS Task ID |
|
| Communications Timeout |
|
| Endpoint Override URL |
Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. |
| Region |
|
| SSL Context Service |
Specifies an optional SSL Context Service that, if provided, will be used to create connections |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| failure |
The job failed, the original FlowFile will be routed to this relationship. |
| original |
Upon successful completion, the original FlowFile will be routed to this relationship. |
| running |
The job is currently still being processed |
| success |
Job successfully finished. FlowFile will be routed to this relation. |
| throttled |
Retrieving results failed for some reason, but the issue is likely to resolve on its own, such as Provisioned Throughput Exceeded or a Throttling failure. It is generally expected to retry this relationship. |
## Writes attributes
| Name |
Description |
| outputLocation |
S3 path-style output location of the result. |
## See also
- [org.apache.nifi.processors.aws.ml.transcribe.StartAwsTranscribeJob](/user-guide/data-integration/openflow/processors/startawstranscribejob)
---
title: GetAwsTranslateJobStatus 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getawstranslatejobstatus.md
section: Loading & Unloading Data
---
# GetAwsTranslateJobStatus 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-aws-nar
## Description
Retrieves the current status of an AWS Translate job.
## Tags
AWS, Amazon, ML, Machine Learning, Translate
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| AWS Credentials Provider service |
The Controller Service that is used to obtain AWS credentials provider |
| AWS Task ID |
|
| Communications Timeout |
|
| Endpoint Override URL |
Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. |
| Region |
|
| SSL Context Service |
Specifies an optional SSL Context Service that, if provided, will be used to create connections |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| failure |
The job failed, the original FlowFile will be routed to this relationship. |
| original |
Upon successful completion, the original FlowFile will be routed to this relationship. |
| running |
The job is currently still being processed |
| success |
Job successfully finished. FlowFile will be routed to this relation. |
| throttled |
Retrieving results failed for some reason, but the issue is likely to resolve on its own, such as Provisioned Throughput Exceeded or a Throttling failure. It is generally expected to retry this relationship. |
## Writes attributes
| Name |
Description |
| outputLocation |
S3 path-style output location of the result. |
## See also
- [org.apache.nifi.processors.aws.ml.translate.StartAwsTranslateJob](/user-guide/data-integration/openflow/processors/startawstranslatejob)
---
title: GetAzureEventHub 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getazureeventhub.md
section: Loading & Unloading Data
---
# GetAzureEventHub 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-azure-nar
## Description
Receives messages from Microsoft Azure Event Hubs without reliable checkpoint tracking. In clustered environment, GetAzureEventHub processor instances work independently and all cluster nodes process all messages (unless running the processor in Primary Only mode). ConsumeAzureEventHub offers the recommended approach to receiving messages from Azure Event Hubs. This processor creates a thread pool for connections to Azure Event Hubs.
## Tags
azure, cloud, eventhub, events, microsoft, streaming, streams
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Consumer Group |
The name of the consumer group to use when pulling events |
| Event Hub Name |
Name of Azure Event Hubs source |
| Event Hub Namespace |
Namespace of Azure Event Hubs prefixed to Service Bus Endpoint domain |
| Message Enqueue Time |
A timestamp (ISO-8601 Instant) formatted as YYYY-MM-DDThhmmss.sssZ (2016-01-01T01:01:01.000Z) from which messages should have been enqueued in the Event Hub to start reading from |
| Partition Receiver Fetch Size |
The number of events that a receiver should fetch from an Event Hubs partition before returning. The default is 100 |
| Partition Receiver Timeout |
The amount of time in milliseconds a Partition Receiver should wait to receive the Fetch Size before returning. The default is 60000 |
| Service Bus Endpoint |
To support namespaces not in the default windows.net domain. |
| Shared Access Policy Key |
The key of the shared access policy. Either the primary or the secondary key can be used. |
| Shared Access Policy Name |
The name of the shared access policy. This policy must have Listen claims. |
| Transport Type |
Advanced Message Queuing Protocol Transport Type for communication with Azure Event Hubs |
| Use Azure Managed Identity |
Choose whether or not to use the managed identity of Azure VM/VMSS |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| success |
Any FlowFile that is successfully received from the event hub will be transferred to this Relationship. |
## Writes attributes
| Name |
Description |
| eventhub.enqueued.timestamp |
The time (in milliseconds since epoch, UTC) at which the message was enqueued in the event hub |
| eventhub.offset |
The offset into the partition at which the message was stored |
| eventhub.sequence |
The Azure sequence number associated with the message |
| eventhub.name |
The name of the event hub from which the message was pulled |
| eventhub.partition |
The name of the event hub partition from which the message was pulled |
| eventhub.property.* |
The application properties of this message. IE: 'application' would be 'eventhub.property.application' |
## See also
- [org.apache.nifi.processors.azure.eventhub.ConsumeAzureEventHub](/user-guide/data-integration/openflow/processors/consumeazureeventhub)
---
title: GetAzureQueueStorage_v12 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getazurequeuestorage_v12.md
section: Loading & Unloading Data
---
# GetAzureQueueStorage_v12 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-azure-nar
## Description
Retrieves the messages from an Azure Queue Storage. The retrieved messages will be deleted from the queue by default. If the requirement is to consume messages without deleting them, set 'Auto Delete Messages' to 'false'. Note: There might be chances of receiving duplicates in situations like when a message is received but was unable to be deleted from the queue due to some unexpected situations.
## Tags
azure, cloud, dequeue, microsoft, queue, storage
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Auto Delete Messages |
Specifies whether the received message is to be automatically deleted from the queue. |
| Credentials Service |
Controller Service used to obtain Azure Storage Credentials. |
| Endpoint Suffix |
Storage accounts in public Azure always use a common FQDN suffix. Override this endpoint suffix with a different suffix in certain circumstances (like Azure Stack or non-public Azure regions). |
| Message Batch Size |
The number of messages to be retrieved from the queue. |
| Queue Name |
Name of the Azure Storage Queue |
| Request Timeout |
The timeout for read or write requests to Azure Queue Storage. Defaults to 1 second. |
| Visibility Timeout |
The duration during which the retrieved message should be invisible to other consumers. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. In case of SOCKS, it is not guaranteed that the selected SOCKS Version will be used by the processor. |
## Relationships
| Name |
Description |
| success |
All successfully processed FlowFiles are routed to this relationship |
## Writes attributes
| Name |
Description |
| azure.queue.uri |
The absolute URI of the configured Azure Queue Storage |
| azure.queue.insertionTime |
The time when the message was inserted into the queue storage |
| azure.queue.expirationTime |
The time when the message will expire from the queue storage |
| azure.queue.messageId |
The ID of the retrieved message |
| azure.queue.popReceipt |
The pop receipt of the retrieved message |
## See also
- [org.apache.nifi.processors.azure.storage.queue.PutAzureQueueStorage_v12](/user-guide/data-integration/openflow/processors/putazurequeuestorage_v12)
---
title: GetBoxFileCollaborators 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getboxfilecollaborators.md
section: Loading & Unloading Data
---
# GetBoxFileCollaborators 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-box-nar
## Description
Retrieves all collaborators on a Box file and adds the collaboration information to the FlowFile's attributes.
## Tags
box, collaboration, permissions, sharing, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Box Client Service |
Controller Service used to obtain a Box API connection. |
| File ID |
The ID of the Box file to retrieve collaborators for |
| Roles |
A comma-separated list of collaboration roles to retrieve. Available roles: editor, viewer, previewer, uploader, previewer uploader, viewer uploader, co-owner, owner. If not specified, no filtering by role will be applied. |
| Statuses |
A comma-separated list of collaboration statuses to retrieve. Available statuses: accepted, pending, rejected. If not specified, no filtering by status will be applied. |
## Relationships
| Name |
Description |
| failure |
FlowFiles that encounter errors during processing will be routed to this relationship |
| not.found |
FlowFiles for which the specified Box file was not found |
| success |
FlowFiles that have been successfully processed will be routed to this relationship |
## Writes attributes
| Name |
Description |
| box.id |
The id of the file |
| box.collaborations.<status>.users.ids |
Comma-separated list of user collaborator IDs by status |
| box.collaborations.<status>.groups.ids |
Comma-separated list of group collaborator IDs by status |
| box.collaborations.<status>.users.emails |
Comma-separated list of user collaborator emails by status |
| box.collaborations.<status>.groups.emails |
Comma-separated list of group collaborator emails by status |
| box.collaborations.<status>.<role>.users.ids |
Comma-separated list of user collaborator IDs by status and role. Only present when both Roles and Statuses properties are set. |
| box.collaborations.<status>.<role>.users.logins |
Comma-separated list of user collaborator logins by status and role. Only present when both Roles and Statuses properties are set. |
| box.collaborations.<status>.<role>.groups.ids |
Comma-separated list of group collaborator IDs by status and role. Only present when both Roles and Statuses properties are set. |
| box.collaborations.<status>.<role>.groups.emails |
Comma-separated list of group collaborator emails by status and role. Only present when both Roles and Statuses properties are set. |
| box.collaborations.count |
Total number of collaborations on the file |
| error.code |
The error code returned by Box |
| error.message |
The error message returned by Box |
## See also
- [org.apache.nifi.processors.box.FetchBoxFile](/user-guide/data-integration/openflow/processors/fetchboxfile)
- [org.apache.nifi.processors.box.ListBoxFile](/user-guide/data-integration/openflow/processors/listboxfile)
---
title: GetBoxGroupMembers 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getboxgroupmembers.md
section: Loading & Unloading Data
---
# GetBoxGroupMembers 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-box-nar
## Description
Retrieves members for a Box Group and writes their details in FlowFile attributes.
## Tags
box, metadata, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Box Client Service |
Controller Service used to obtain a Box API connection. |
| Group ID |
The ID of the Group to retrieve members for |
## Relationships
| Name |
Description |
| failure |
The FlowFile will be routed here when Group memberships retrieval was attempted but failed. |
| not.found |
The FlowFile will be routed here when the Group was not found. |
| success |
The FlowFile will be routed here after successfully retrieving Group members. |
## Writes attributes
| Name |
Description |
| box.group.user.ids |
A comma-separated list of user IDs in the group. |
| box.group.user.logins |
A comma-separated list of user Logins (emails) in the group. |
| error.code |
An http error code returned by Box. |
| error.message |
An error message returned by Box. |
---
title: GetConfluenceAuditRecords 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getconfluenceauditrecords.md
section: Loading & Unloading Data
---
# GetConfluenceAuditRecords 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-atlassian-processors-nar
## Description
Processor listing Confluence audit records.
## Tags
Preview, atlassian, audit log, confluence
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Audit Log Fetch Limit |
How many audit logs will be fetched from Confluence API in one request |
| Confluence Client Service |
Controller service for managing connections to Confluence |
## State management
| Scopes |
Description |
| CLUSTER |
Stores last synchronization timestamp. |
## Relationships
| Name |
Description |
| failure |
Failed to fetch Confluence audit records |
| original |
The input Flow File is routed to the original relationship. |
| retry |
Retryable failure occurred, e.g. rate limiting |
| success |
Successfully fetched Confluence audit records |
## Writes attributes
| Name |
Description |
| confluence.group.ids |
List of identifiers of the Confluence groups. |
| confluence.page.names |
List of the names of the Confluence page. |
| confluence.space.names |
List of the Confluence spaces. |
| confluence.continue.fetching |
Indicates whether there are more pages to fetch (true/false). |
---
title: GetConfluenceGroupUsers 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getconfluencegroupusers.md
section: Loading & Unloading Data
---
# GetConfluenceGroupUsers 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-atlassian-processors-nar
## Description
Processor that downloads information about users belonging to a given Confluence group
## Tags
Preview, atlassian, confluence, groups, users
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Confluence Client Service |
Controller service for managing connections to Confluence |
| Confluence Group ID |
Identifier of the Confluence Group |
## Relationships
| Name |
Description |
| failure |
Failed to fetch Confluence group users |
| retry |
Retryable failure occurred, e.g. rate limiting |
| success |
Successfully fetched Confluence group users |
## Writes attributes
| Name |
Description |
| confluence.group.user.ids |
Identifiers of the Confluence group users. |
| confluence.group.user.emails |
Emails of the Confluence group users. |
---
title: GetConfluencePageContent 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getconfluencepagecontent.md
section: Loading & Unloading Data
---
# GetConfluencePageContent 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-atlassian-processors-nar
## Description
Processor downloading Confluence pages.
## Tags
Preview, atlassian, confluence, content, fetch, page
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Body Format |
Format in which body of the Confluence Page will be fetched |
| Confluence Client Service |
Controller service for managing connections to Confluence |
| Confluence Page ID |
Identifier of the Confluence Page |
## Relationships
| Name |
Description |
| failure |
Failed to fetch Confluence page |
| not found |
Confluence page not found |
| removed |
Confluence page was removed |
| retry |
Retryable failure occurred, e.g. rate limiting |
| success |
Successfully fetched Confluence page |
## Writes attributes
| Name |
Description |
| mime.type |
text/html |
| confluence.page.version |
Version of the Confluence page. |
| confluence.page.last.modification.date |
Last modification date of the Confluence page. |
| confluence.page.change.type |
Informs about status change for the searched page. |
---
title: GetConfluencePageIds 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getconfluencepageids.md
section: Loading & Unloading Data
---
# GetConfluencePageIds 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-atlassian-processors-nar
## Description
Downloads changed Confluence pages since the last sync and emits each as a FlowFile with metadata.
## Tags
Preview, atlassian, changes, confluence, fetch, pages
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Confluence Client Service |
Controller service for managing connections to Confluence |
| Page IDs |
Comma separated list of page IDs to filter page by; only pages with these IDs are returned |
| Space IDs |
Comma separated list of space IDs to filter pages by; only pages from these spaces are returned |
| Start Date |
Start date from which the ingestion should happen (format: yyyy-MM-dd, inclusive) |
## State management
| Scopes |
Description |
| CLUSTER |
Stores pagination state to maintain position between restarts. |
## Relationships
| Name |
Description |
| failure |
Failed to fetch changed Confluence pages |
| original |
The input Flow File is routed to the original relationship. |
| retry |
Retryable failure occurred, e.g. rate limiting |
| success |
Successfully fetched changed Confluence pages |
## Writes attributes
| Name |
Description |
| confluence.page.id |
Unique identifier of the Confluence page. |
| confluence.page.change.type |
Informs about status change for the searched page. |
| confluence.page.url |
Confluence page url. |
| confluence.page.title |
Confluence page title. |
| confluence.page.last.modification.date |
Last modification date of the Confluence page. |
| confluence.space.id |
Unique identifier of the Confluence space. |
| confluence.continue.fetching |
Indicates whether there are more pages to fetch (true/false). |
---
title: GetConfluencePagePermissions 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getconfluencepagepermissions.md
section: Loading & Unloading Data
---
# GetConfluencePagePermissions 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-atlassian-processors-nar
## Description
Processor downloading Confluence page permissions.
## Tags
Preview, atlassian, confluence, page, permissions
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Confluence Client Service |
Controller service for managing connections to Confluence |
| Confluence Page ID |
Identifier of the Confluence Page |
## Relationships
| Name |
Description |
| failure |
Failed to fetch and parse Confluence page permissions. |
| page not found |
Confluence page not found |
| restrictions changed |
Confluence page restrictions changed since last fetch |
| retry |
Retryable failure occurred, e.g. rate limiting |
| success |
Successfully fetched Confluence page permissions. |
## Writes attributes
| Name |
Description |
| confluence.permissions.users |
IDs of users with permissions to the Confluence page |
| confluence.permissions.emails |
Emails of users with permissions to the Confluence page |
| confluence.permissions.groups |
Groups with permissions to the Confluence page |
---
title: GetConfluenceSpaceIds 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getconfluencespaceids.md
section: Loading & Unloading Data
---
# GetConfluenceSpaceIds 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-atlassian-processors-nar
## Description
Processor for retrieving Confluence space ids.
## Tags
atlassian, confluence, preview, spaces
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Confluence Client Service |
Controller service for managing connections to Confluence |
| Space Keys |
Comma-separated list of space keys to filter. If not specified, all spaces will be retrieved. |
## Relationships
| Name |
Description |
| retry |
Retryable failure occurred, e.g. rate limiting |
| success |
Successfully fetched Confluence spaces |
## Writes attributes
| Name |
Description |
| confluence.space.ids |
List of identifiers of the Confluence spaces. |
---
title: GetConfluenceSpacePermissions 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getconfluencespacepermissions.md
section: Loading & Unloading Data
---
# GetConfluenceSpacePermissions 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-atlassian-processors-nar
## Description
Processor downloading Confluence space permissions.
## Tags
Preview, atlassian, confluence, permissions, space
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Confluence Client Service |
Controller service for managing connections to Confluence |
| Confluence Space ID |
Identifier of the Confluence Space. |
## Relationships
| Name |
Description |
| failure |
Failed to fetch and parse Confluence space permissions. |
| retry |
Retryable failure occurred, e.g. rate limiting |
| space not found |
Confluence space not found |
| success |
Successfully fetched Confluence space permissions. |
## Writes attributes
| Name |
Description |
| confluence.permissions.users |
IDs of users with permissions to the Confluence space |
| confluence.permissions.emails |
Emails of users with permissions to the Confluence space |
| confluence.permissions.groups |
Groups with permissions to the Confluence space |
---
title: GetDataShareCredentials 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getdatasharecredentials.md
section: Loading & Unloading Data
---
# GetDataShareCredentials 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-salesforce-processors-nar
## Description
Describe the specified data share metadata in Salesforce Data Cloud.
## Tags
daas, data cloud, describe, object, preview, salesforce, sfdc
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Calculated Insights Objects |
Comma separated list of Calculated Insight Object names to describe. |
| Connection Pooling Service |
The Connection Pooling Service that is used to create the Snowflake volumes holding the credentials. |
| Data Lake Objects |
Comma separated list of Data Lake Object names to describe. |
| Data Model Objects |
Comma separated list of Data Model Object names to describe. |
| Data Share Name |
The name of the Data Share to describe. |
| Salesforce Data Cloud Client |
Salesforce Data Cloud Client to interact with the APIs |
## State management
| Scopes |
Description |
| CLUSTER |
Provides information about the last time an external volume has been created/updated for credentials. |
## Relationships
| Name |
Description |
| comms.failure |
A FlowFile is routed to this relationship if the data share credentials metadata could not be retrieved but the operation might be retried |
| failure |
A FlowFile is routed to this relationship if the data share credentials cannot be retrieved or volumes cannot be created |
| success |
FlowFile containing the data share metadata after successful creation of the volumes will be routed to this relationship |
## See also
- [com.snowflake.openflow.runtime.processors.salesforce.ListSFDCDataShares](/user-guide/data-integration/openflow/processors/listsfdcdatashares)
---
title: GetDataShareTables 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getdatasharetables.md
section: Loading & Unloading Data
---
# GetDataShareTables 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-salesforce-processors-nar
## Description
Describe the specified data share metadata in Salesforce Data Cloud.
## Tags
daas, data cloud, describe, object, preview, salesforce, sfdc
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Calculated Insights Objects |
Comma separated list of Calculated Insight Object names to describe. |
| Data Lake Objects |
Comma separated list of Data Lake Object names to describe. |
| Data Model Objects |
Comma separated list of Data Model Object names to describe. |
| Data Share Name |
The name of the Data Share to describe. |
| Salesforce Data Cloud Client |
Salesforce Data Cloud Client to interact with the APIs |
## Relationships
| Name |
Description |
| comms.failure |
A FlowFile is routed to this relationship if the data share tables metadata could not be retrieved but the operation might be retried |
| failure |
A FlowFile is routed to this relationship if the data share tables metadata could not be retrieved |
| success |
FlowFile containing the data share tables metadata will be routed to this relationship |
## See also
- [com.snowflake.openflow.runtime.processors.salesforce.ListSFDCDataShares](/user-guide/data-integration/openflow/processors/listsfdcdatashares)
---
title: GetDBFSFile 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getdbfsfile.md
section: Loading & Unloading Data
---
# GetDBFSFile 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-databricks-processors-nar
## Description
Read a DBFS file.
## Tags
databricks, dbfs, openflow
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| DBFS File Path |
DBFS file path e.g. /directory/file.txt |
| Databricks Client |
Databricks Client Service. |
## Relationships
| Name |
Description |
| failure |
Databricks failure relationship |
| success |
Databricks success relationship |
## Writes attributes
| Name |
Description |
| error.code |
The error code for the SQL statement if an error occurred. |
| error.message |
The error message for the SQL statement if an error occurred. |
---
title: GetDynamoDB 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getdynamodb.md
section: Loading & Unloading Data
---
# GetDynamoDB 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-aws-nar
## Description
Retrieves a document from DynamoDB based on hash and range key. The key can be string or number. For any get request all the primary keys are required (hash or hash and range based on the table keys).A Json Document ( 'Map') attribute of the DynamoDB item is read into the content of the FlowFile.
## Tags
AWS, Amazon, DynamoDB, Fetch, Get
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| AWS Credentials Provider service |
The Controller Service that is used to obtain AWS credentials provider |
| Batch items for each request (between 1 and 50) |
The items to be retrieved in one batch |
| Communications Timeout |
|
| Endpoint Override URL |
Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. |
| Hash Key Name |
The hash key name of the item |
| Hash Key Value |
The hash key value of the item |
| Hash Key Value Type |
The hash key value type of the item |
| Json Document attribute |
The Json document to be retrieved from the dynamodb item ( 's' type in the schema) |
| Range Key Name |
The range key name of the item |
| Range Key Value |
|
| Range Key Value Type |
The range key value type of the item |
| Region |
|
| SSL Context Service |
Specifies an optional SSL Context Service that, if provided, will be used to create connections |
| Table Name |
The DynamoDB table name |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| failure |
FlowFiles are routed to failure relationship |
| not found |
FlowFiles are routed to not found relationship if key not found in the table |
| success |
FlowFiles are routed to success relationship |
| unprocessed |
FlowFiles are routed to unprocessed relationship when DynamoDB is not able to process all the items in the request. Typical reasons are insufficient table throughput capacity and exceeding the maximum bytes per request. Unprocessed FlowFiles can be retried with a new request. |
## Writes attributes
| Name |
Description |
| dynamodb.key.error.unprocessed |
DynamoDB unprocessed keys |
| dynmodb.range.key.value.error |
DynamoDB range key error |
| dynamodb.key.error.not.found |
DynamoDB key not found |
| dynamodb.error.exception.message |
DynamoDB exception message |
| dynamodb.error.code |
DynamoDB error code |
| dynamodb.error.message |
DynamoDB error message |
| dynamodb.error.service |
DynamoDB error service |
| dynamodb.error.retryable |
DynamoDB error is retryable |
| dynamodb.error.request.id |
DynamoDB error request id |
| dynamodb.error.status.code |
DynamoDB status code |
## See also
- [org.apache.nifi.processors.aws.dynamodb.DeleteDynamoDB](/user-guide/data-integration/openflow/processors/deletedynamodb)
- [org.apache.nifi.processors.aws.dynamodb.PutDynamoDB](/user-guide/data-integration/openflow/processors/putdynamodb)
- [org.apache.nifi.processors.aws.dynamodb.PutDynamoDBRecord](/user-guide/data-integration/openflow/processors/putdynamodbrecord)
---
title: GetElasticsearch 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getelasticsearch.md
section: Loading & Unloading Data
---
# GetElasticsearch 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-elasticsearch-restapi-nar
## Description
Elasticsearch get processor that uses the official Elastic REST client libraries to fetch a single document from Elasticsearch by _id. Note that the full body of the document will be read into memory before being written to a FlowFile for transfer.
## Tags
elasticsearch, elasticsearch7, elasticsearch8, elasticsearch9, index, json, put, record
## Input Requirement
ALLOWED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Attribute Name |
The name of the FlowFile attribute to use for the retrieved document output. |
| Client Service |
An Elasticsearch client service to use for running queries. |
| Destination |
Indicates whether the retrieved document is written to the FlowFile content or a FlowFile attribute. |
| Document Id |
The _id of the document to retrieve. |
| Index |
The name of the index to use. |
| Type |
The type of this document (used by Elasticsearch for indexing and searching). |
## Relationships
| Name |
Description |
| document |
Fetched documents are routed to this relationship. |
| failure |
All flowfiles that fail for reasons unrelated to server availability go to this relationship. |
| not_found |
A FlowFile is routed to this relationship if the specified document does not exist in the Elasticsearch cluster. |
| retry |
All flowfiles that fail due to server/cluster availability go to this relationship. |
## Writes attributes
| Name |
Description |
| filename |
The filename attribute is set to the document identifier |
| elasticsearch.index |
The Elasticsearch index containing the document |
| elasticsearch.type |
The Elasticsearch document type |
| elasticsearch.get.error |
The error message provided by Elasticsearch if there is an error fetching the document. |
## See also
- [org.apache.nifi.processors.elasticsearch.JsonQueryElasticsearch](/user-guide/data-integration/openflow/processors/jsonqueryelasticsearch)
---
title: GetFile 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getfile.md
section: Loading & Unloading Data
---
# GetFile 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Creates FlowFiles from files in a directory. NiFi will ignore files it doesn't have at least read permissions for.
## Tags
files, filesystem, get, ingest, ingress, input, local, source
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Batch Size |
The maximum number of files to pull in each invocation of the processor |
| File Filter |
Only files whose names match the given regular expression will be picked up |
| Ignore Hidden Files |
Indicates whether or not hidden files should be ignored |
| Input Directory |
The input directory from which to pull files |
| Keep Source File |
If true, the file is not deleted after it has been copied to the Content Repository; this causes the file to be picked up continually and is useful for testing purposes. If not keeping original NiFi will need write permissions on the directory it is pulling from otherwise it will ignore the file. |
| Maximum File Age |
The maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored |
| Maximum File Size |
The maximum size that a file can be in order to be pulled |
| Minimum File Age |
The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored |
| Minimum File Size |
The minimum size that a file must be in order to be pulled |
| Path Filter |
When Recurse Subdirectories is true, then only subdirectories whose path matches the given regular expression will be scanned |
| Polling Interval |
Indicates how long to wait before performing a directory listing |
| Recurse Subdirectories |
Indicates whether or not to pull files from subdirectories |
## Restrictions
| Required Permission |
Explanation |
| read filesystem |
Provides operator the ability to read from any file that NiFi has access to. |
| write filesystem |
Provides operator the ability to delete any file that NiFi has access to. |
## Relationships
| Name |
Description |
| success |
All files are routed to success |
## Writes attributes
| Name |
Description |
| filename |
The filename is set to the name of the file on disk |
| path |
The path is set to the relative path of the file's directory on disk. For example, if the <Input Directory> property is set to /tmp, files picked up from /tmp will have the path attribute set to ./. If the <Recurse Subdirectories> property is set to true and a file is picked up from /tmp/abc/1/2/3, then the path attribute will be set to abc/1/2/3 |
| file.creationTime |
The date and time that the file was created. May not work on all file systems |
| file.lastModifiedTime |
The date and time that the file was last modified. May not work on all file systems |
| file.lastAccessTime |
The date and time that the file was last accessed. May not work on all file systems |
| file.owner |
The owner of the file. May not work on all file systems |
| file.group |
The group owner of the file. May not work on all file systems |
| file.permissions |
The read/write/execute permissions of the file. May not work on all file systems |
| absolute.path |
The full/absolute path from where a file was picked up. The current 'path' attribute is still populated, but may be a relative path |
## See also
- [org.apache.nifi.processors.standard.FetchFile](/user-guide/data-integration/openflow/processors/fetchfile)
- [org.apache.nifi.processors.standard.PutFile](/user-guide/data-integration/openflow/processors/putfile)
---
title: GetFileResource 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getfileresource.md
section: Loading & Unloading Data
---
# GetFileResource 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
This processor creates FlowFiles with the content of the configured File Resource. GetFileResource is useful for load testing, configuration, and simulation.
## Tags
file, generate, load, test
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| File Resource |
Location of the File Resource (Local File or URL). This file will be used as content of the generated FlowFiles. |
| MIME Type |
Specifies the value to set for the [mime.type] attribute. |
## Restrictions
| Required Permission |
Explanation |
| read filesystem |
Provides operator the ability to read from any file that NiFi has access to. |
| reference remote resources |
File Resource can reference resources over HTTP/HTTPS |
## Relationships
| Name |
Description |
| success |
|
## Writes attributes
| Name |
Description |
| mime.type |
Sets the MIME type of the output if the 'MIME Type' property is set |
| Dynamic property key |
Value for the corresponding dynamic property, if any is set |
---
title: GetFTP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getftp.md
section: Loading & Unloading Data
---
# GetFTP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Fetches files from an FTP Server and creates FlowFiles from them
## Tags
FTP, fetch, files, get, ingest, input, remote, retrieve, source
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Connection Mode |
The FTP Connection Mode |
| Connection Timeout |
Amount of time to wait before timing out while creating a connection |
| Data Timeout |
When transferring a file between the local and remote system, this value specifies how long is allowed to elapse without any data being transferred between systems |
| Delete Original |
Determines whether or not the file is deleted from the remote system after it has been successfully transferred |
| File Filter Regex |
Provides a Java Regular Expression for filtering Filenames; if a filter is supplied, only files whose names match that Regular Expression will be fetched |
| Follow Symbolic Links |
If true, will pull even symbolic files and also nested symbolic subdirectories; otherwise, will not read symbolic files and will not traverse symbolic link subdirectories |
| Hostname |
The fully qualified hostname or IP address of the remote system |
| Ignore Dotted Files |
If true, files whose names begin with a dot (".") will be ignored |
| Internal Buffer Size |
Set the internal buffer size for buffered data streams |
| Max Selects |
The maximum number of files to pull in a single connection |
| Password |
Password for the user account |
| Path Filter Regex |
When Search Recursively is true, then only subdirectories whose path matches the given Regular Expression will be scanned |
| Polling Interval |
Determines how long to wait between fetching the listing for new files |
| Port |
The port that the remote system is listening on for file transfers |
| Remote Path |
The path on the remote system from which to pull or push files |
| Remote Poll Batch Size |
The value specifies how many file paths to find in a given directory on the remote system when doing a file listing. This value in general should not need to be modified but when polling against a remote system with a tremendous number of files this value can be critical. Setting this value too high can result very poor performance and setting it too low can cause the flow to be slower than normal. |
| Search Recursively |
If true, will pull files from arbitrarily nested subdirectories; otherwise, will not traverse subdirectories |
| Transfer Mode |
The FTP Transfer Mode |
| Use Natural Ordering |
If true, will pull files in the order in which they are naturally listed; otherwise, the order in which the files will be pulled is not defined |
| Username |
Username |
| ftp-use-utf8 |
Tells the client to use UTF-8 encoding when processing files and filenames. If set to true, the server must also support UTF-8 encoding. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| success |
All FlowFiles that are received are routed to success |
## Writes attributes
| Name |
Description |
| filename |
The filename is set to the name of the file on the remote server |
| path |
The path is set to the path of the file's directory on the remote server. For example, if the <Remote Path> property is set to /tmp, files picked up from /tmp will have the path attribute set to /tmp. If the <Search Recursively> property is set to true and a file is picked up from /tmp/abc/1/2/3, then the path attribute will be set to /tmp/abc/1/2/3 |
| file.lastModifiedTime |
The date and time that the source file was last modified |
| file.lastAccessTime |
The date and time that the file was last accessed. May not work on all file systems |
| file.owner |
The numeric owner id of the source file |
| file.group |
The numeric group id of the source file |
| file.permissions |
The read/write/execute permissions of the source file |
| absolute.path |
The full/absolute path from where a file was picked up. The current 'path' attribute is still populated, but may be a relative path |
## See also
- [org.apache.nifi.processors.standard.PutFTP](/user-guide/data-integration/openflow/processors/putftp)
---
title: GetGcpVisionAnnotateFilesOperationStatus 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getgcpvisionannotatefilesoperationstatus.md
section: Loading & Unloading Data
---
# GetGcpVisionAnnotateFilesOperationStatus 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-gcp-nar
## Description
Retrieves the current status of an Google Vision operation.
## Tags
Cloud, Google, Machine Learning, Vision
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| gcp-credentials-provider-service |
The Controller Service used to obtain Google Cloud Platform credentials. |
| operationKey |
The unique identifier of the Vision operation. |
## Relationships
| Name |
Description |
| failure |
FlowFiles are routed to failure relationship |
| original |
Upon successful completion, the original FlowFile will be routed to this relationship. |
| running |
The job is currently still being processed |
| success |
FlowFiles are routed to success relationship |
## See also
- [org.apache.nifi.processors.gcp.vision.StartGcpVisionAnnotateFilesOperation](/user-guide/data-integration/openflow/processors/startgcpvisionannotatefilesoperation)
---
title: GetGcpVisionAnnotateImagesOperationStatus 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getgcpvisionannotateimagesoperationstatus.md
section: Loading & Unloading Data
---
# GetGcpVisionAnnotateImagesOperationStatus 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-gcp-nar
## Description
Retrieves the current status of an Google Vision operation.
## Tags
Cloud, Google, Machine Learning, Vision
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| gcp-credentials-provider-service |
The Controller Service used to obtain Google Cloud Platform credentials. |
| operationKey |
The unique identifier of the Vision operation. |
## Relationships
| Name |
Description |
| failure |
FlowFiles are routed to failure relationship |
| original |
Upon successful completion, the original FlowFile will be routed to this relationship. |
| running |
The job is currently still being processed |
| success |
FlowFiles are routed to success relationship |
## See also
- [org.apache.nifi.processors.gcp.vision.StartGcpVisionAnnotateImagesOperation](/user-guide/data-integration/openflow/processors/startgcpvisionannotateimagesoperation)
---
title: GetGoogleAdsReport 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getgoogleadsreport.md
section: Loading & Unloading Data
---
# GetGoogleAdsReport 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-google-ads-nar
## Description
A processor which can interact with Google Ads Reporting API. By default it fetches data once a day
## Tags
Google, Google Ads, report
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Client Account ID |
ID of the Google Ads account for which the report should be fetched |
| GCP Credentials Service |
Controller Service used to obtain Google Cloud Platform credentials. |
| Google Ads Resource Name |
Name of the resource that should be used in 'FROM' clause of the query |
| Google Developer Token |
Developer token required to access Google APIs |
| Report Attributes |
List of comma-separated report attributes |
| Report Metrics |
List of comma-separated report metrics |
| Report Segments |
List of comma-separated report segments |
| Report Start Date |
Start date from which the ingestion should happen. |
## State management
| Scopes |
Description |
| CLUSTER |
Stores information about last report definition in form of hash to detect schema changes. In incremental ingestion (when the 'segments.date' segment is selected) it keeps track of latest ingested date to download only new data chunks. Additionally start date is saved. |
## Relationships
| Name |
Description |
| failure |
Error FlowFiles transferred when receiving error response from Google Ads Reporting API or when an error occurred during response processing. |
| success |
Response FlowFiles transferred when receiving success response from Google Ads Reporting API. |
## Writes attributes
| Name |
Description |
| google.ads.client.account.id |
ID of the account in Google Ads for which given report should be ingested |
| google.ads.resource.name |
Name of the resource in Google Ads that is a source for the report |
| google.ads.query |
Query used to fetch data from Google Ads StreamSearch API |
| google.ads.attributes |
Attributes of the selected resource |
| google.ads.metrics |
Metrics collected in the context of a given resource |
| google.ads.segments |
Buckets in which metrics should be grouped |
| google.ads.ingestion.strategy |
The strategy used for ingestion. Can be 'SNAPSHOT' or 'INCREMENTAL' |
| google.ads.start.date |
Date from which data is downloaded from Google Ads (including given date) |
| google.ads.end.date |
Date to which data is downloaded from Google Ads (including given date) |
| google.ads.report.schema.changed |
Flag meaning if the report schema has changed between processor executions |
| google.ads.report.conversion.window |
Number of days which are fetched from Google Ads during incremental load. Based on Conversion Window values |
| fragment.identifier |
A unique ID of each ingestion run. Lets you identify all flow files generated during a single run. |
| fragment.index |
Number representing unique identifier in batch of flowfiles generated during one ingestion run |
| fragment.count |
Amount of flowfiles generated during processor execution |
| avro.schema |
Avro schema representing fetched data |
| mime.type |
Mime type of the returned report. |
---
title: GetGoogleGroupMembers 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getgooglegroupmembers.md
section: Loading & Unloading Data
---
# GetGoogleGroupMembers 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-google-drive-nar
## Description
Retrieves the members of one or more Google Groups, specified as a comma-separated list of group IDs that is given as a FlowFile attribute. Supports both immediate (top-level) and nested group member retrieval. Outputs four FlowFile attributes: 'google.group.member.user.ids', 'google.group.member.user.emails', 'google.group.member.group.ids', and 'google.group.member.group.emails'. When nested fetching is enabled, it recursively expands sub-groups up to the specified depth. If an attribute already exists on the FlowFile, the new values are concatenated to the existing value (separated by a comma).
## Tags
cloud, directory, gcp, google, groups, membership
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Fetch Nested Groups |
When enabled, recursively fetches members from nested groups within the specified groups. When disabled, only top-level members are retrieved. |
| GCP Credentials Service |
Specifies the Controller Service used to obtain Google Cloud Platform credentials. |
| Google Group IDs |
Specifies the comma-separated list of Google Group IDs (email addresses for the groups). Supports Expression Language. |
| Nested Depth Limit |
Maximum depth to traverse when fetching nested group members. |
## Relationships
| Name |
Description |
| failure |
A FlowFile is routed here if the processor fails to retrieve Google group members. |
| not.found |
A FlowFile is routed here if for each Google group that was not found. |
| retry |
A FlowFile is routed here if the processor should retry the request (e.g., after rate limiting). |
| success |
A FlowFile is routed here after successfully retrieving Google group members. |
## Writes attributes
| Name |
Description |
| google.group.ids |
A comma-separated list of Google Group IDs that were found. |
| google.group.member.user.ids |
A comma-separated list of user IDs found in the specified groups. When nested fetching is enabled, includes users from nested groups up to the specified depth. |
| google.group.member.user.emails |
A comma-separated list of user email addresses found in the specified groups. When nested fetching is enabled, includes users from nested groups up to the specified depth. |
| google.group.member.group.ids |
A comma-separated list of nested group IDs found in the specified groups. When nested fetching is enabled, includes all groups discovered during recursive traversal. |
| google.group.member.group.emails |
A comma-separated list of nested group email addresses found in the specified groups. When nested fetching is enabled, includes all groups discovered during recursive traversal. |
## See also
- [com.snowflake.openflow.runtime.processors.google.CaptureGoogleDriveChanges](/user-guide/data-integration/openflow/processors/capturegoogledrivechanges)
---
title: GetGoogleSheets 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getgooglesheets.md
section: Loading & Unloading Data
---
# GetGoogleSheets 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-google-sheets-processors-nar
## Description
Processor responsible for fetching data from Google Sheets. By default it fetches data once a day.
## Tags
Google, Google Sheets, spreadsheet
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Date Time Render Option |
Determines how dates should be rendered in the output. |
| GCP Credentials Service |
Controller Service used to obtain Google Cloud Platform credentials. |
| Ranges |
The A1 notation or R1C1 notation of the comma-separated ranges to retrieve values from. For example: Sheet1!A1:B2,Sheet2!D4:E5,Sheet3. The first row in a sheet must represent column names. If not specified, all sheets will be downloaded. |
| Spreadsheet ID |
ID of the Google Sheets Spreadsheet. Can be found in the URL of the spreadsheet. |
| Value Render Option |
Determines how values should be rendered in the output. |
## Relationships
| Name |
Description |
| failure |
FlowFile with errors occurred while fetching from Google Sheets. |
| success |
FlowFile containing a JSON array where each object represents a row from the source sheet. Keys correspond to column headers from the first row, and values to the respective row entries. |
## Writes attributes
| Name |
Description |
| google.sheets.spreadsheet.id |
ID of the Google Sheets Spreadsheet. |
| google.sheets.range |
Range in Google Sheets Spreadsheet that was fetched. |
| run.id |
A unique ID of each ingestion run. Lets you identify all flow files generated during a single run. |
| destination.table.schema |
A Snowflake schema of the destination table in the following format: \{ "columns": [ \{ "name": "<column name>", "type": "<column type>", "nullable": <true/false>, "precision": <precision, only for numeric type>, "scale": <scale, only for numeric type> \}, ... ], "primaryKeys": ["<name of first primary key column>", "<name of second primary key column>", ...] \} |
---
title: GetHubSpot 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/gethubspot.md
section: Loading & Unloading Data
---
# GetHubSpot 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-hubspot-nar
## Description
Retrieves JSON data from a private HubSpot application. This processor is intended to be run on the Primary Node only.
## Tags
hubspot
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| access-token |
Access Token to authenticate requests |
| incremental-delay |
The ending timestamp of the time window will be adjusted earlier by the amount configured in this property. For example, with a property value of 10 seconds, an ending timestamp of 12:30:45 would be changed to 12:30:35. Set this property to avoid missing objects when the clock of your local machines and HubSpot servers 'clock are not in sync and to protect against HubSpot's mechanism that changes last updated timestamps after object creation. |
| incremental-initial-start-time |
This property specifies the start time that the processor applies when running the first request. The expected format is a UTC date-time such as '2011-12-03T10:15:30Z' |
| is-incremental |
The processor can incrementally load the queried objects so that each object is queried exactly once. For each query, the processor queries objects within a time window where the objects were modified between the previous run time and the current time (optionally adjusted by the Incremental Delay property). |
| object-type |
The HubSpot Object Type requested |
| result-limit |
The maximum number of results to request for each invocation of the Processor |
| web-client-service-provider |
Controller service for HTTP client operations |
## State management
| Scopes |
Description |
| CLUSTER |
In case of incremental loading, the start and end timestamps of the last query time window are stored in the state. When the 'Result Limit' property is set, the paging cursor is saved after executing a request. Only the objects after the paging cursor will be retrieved. The maximum number of retrieved objects can be set in the 'Result Limit' property. |
## Relationships
| Name |
Description |
| success |
For FlowFiles created as a result of a successful HTTP request. |
## Writes attributes
| Name |
Description |
| mime.type |
Sets the MIME type to application/json |
---
title: GetHubSpotObject 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/gethubspotobject.md
section: Loading & Unloading Data
---
# GetHubSpotObject 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-hubspot-processors-nar
## Description
Get a HubSpot object and its associations by ID or unique value.
## Tags
Preview, hubspot
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| HubSpot Service |
HubSpot Client Service. |
| Object ID Property |
HubSpot property used to uniquely identify the object. |
| Object ID Value |
Matching HubSpot property value to search for. |
| Object Type |
HubSpot object type |
## Relationships
| Name |
Description |
| failure |
HubSpot fail relationship |
| missing |
HubSpot object does not exist. |
| retry |
HubSpot retry relationship. FlowFiles that failed to process due to a server timeout or rate limit related error. FlowFiles routed here should be routed back into the processor. |
| success |
HubSpot success relationship |
## See also
- [com.snowflake.openflow.runtime.processors.hubspot.GetHubSpotSchema](/user-guide/data-integration/openflow/processors/gethubspotschema)
- [com.snowflake.openflow.runtime.processors.hubspot.ListArchivedHubSpotData](/user-guide/data-integration/openflow/processors/listarchivedhubspotdata)
- [com.snowflake.openflow.runtime.processors.hubspot.ListHubSpotObjects](/user-guide/data-integration/openflow/processors/listhubspotobjects)
- [com.snowflake.openflow.runtime.processors.hubspot.PutHubSpot](/user-guide/data-integration/openflow/processors/puthubspot)
---
title: GetHubSpotSchema 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/gethubspotschema.md
section: Loading & Unloading Data
---
# GetHubSpotSchema 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-hubspot-processors-nar
## Description
Retrieves schema information for HubSpot object types including field names, types, and labels. Outputs detailed field metadata as JSON for schema discovery and mapping purposes.
## Tags
Preview, crm, hubspot, metadata, schema
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| HubSpot Service |
HubSpot Client Service. |
| Object Type |
HubSpot object type |
## Relationships
| Name |
Description |
| failure |
HubSpot fail relationship |
| retry |
HubSpot retry relationship. FlowFiles that failed to process due to a server timeout or rate limit related error. FlowFiles routed here should be routed back into the processor. |
| success |
HubSpot success relationship |
## Writes attributes
| Name |
Description |
| hubspot.object.type |
The HubSpot object type |
| hubspot.field.count |
Number of fields retrieved |
| mime.type |
MIME type of the output (application/json) |
## See also
- [com.snowflake.openflow.runtime.processors.hubspot.GetHubSpotObject](/user-guide/data-integration/openflow/processors/gethubspotobject)
- [com.snowflake.openflow.runtime.processors.hubspot.ListArchivedHubSpotData](/user-guide/data-integration/openflow/processors/listarchivedhubspotdata)
- [com.snowflake.openflow.runtime.processors.hubspot.ListHubSpotObjects](/user-guide/data-integration/openflow/processors/listhubspotobjects)
- [com.snowflake.openflow.runtime.processors.hubspot.PutHubSpot](/user-guide/data-integration/openflow/processors/puthubspot)
---
title: GetLinkedInAdsReport 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getlinkedinadsreport.md
section: Loading & Unloading Data
---
# GetLinkedInAdsReport 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-linkedin-ads-processors-nar
## Description
Processor downloading metrics from the LinkedIn Reporting APIs.
## Tags
LinkedIn, LinkedIn Ads, ads, report
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Accounts |
List of comma-separated accounts. |
| Campaign Groups |
List of comma-separated campaign groups. |
| Campaigns |
List of comma-separated campaigns. |
| Companies |
List of comma-separated companies. |
| Conversion Window |
Timeframe for which data is refreshed during incremental load. |
| Metrics |
List of comma-separated metrics. |
| OAuth Token Provider |
Service providing OAuth access token. |
| Pivots |
List of comma-separated pivots. |
| Report Name |
Unique name of the report. |
| Shares |
List of comma-separated shares. |
| Start Date |
Start date from which ingestion should begin. It must be in the yyyy-MM-dd format. |
| Time Granularity |
Time granularity of results. |
| Web Client Service Provider |
Service providing client for REST request execution. |
## State management
| Scopes |
Description |
| CLUSTER |
Stores information about last report definition in form of hash to detect schema changes. Incrementally loaded reports persist last ingestion date to define ingestion date ranges after initial load. Additionally start date is saved. |
## Relationships
| Name |
Description |
| success |
Response FlowFiles transferred when successfully processed a response from the LinkedIn Ads Reporting API. |
## Writes attributes
| Name |
Description |
| linkedin.ads.report.name |
Unique name of the report. |
| linkedin.ads.run.id |
Unique identifier of the run. |
| avro.schema |
Avro schema that contains a set of all configured metrics and pivots. |
| linkedin.ads.ingestion.strategy |
Strategy that defines whether the report will be downloaded as SNAPSHOT or INCREMENTAL. |
| linkedin.ads.report.schema.changed |
Flag that indicates whether the report schema has changed between processor executions. |
| linkedin.ads.ingestion.start.date |
Date from which data is downloaded from LinkedIn Ads (including a given date). |
| linkedin.ads.ingestion.end.date |
Date to which data is downloaded from LinkedIn Ads (including a given date). |
---
title: GetMicrosoft365GroupMembers 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getmicrosoft365groupmembers.md
section: Loading & Unloading Data
---
# GetMicrosoft365GroupMembers 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-msgraph-nar
## Description
Retrieves Microsoft365 group members and emits a FlowFile for each change that occurs. This includes membership changes.
## Tags
cdc, document, graph, library, microsoft, sharepoint, unstructured
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Authentication Service |
The service that provides authentication for the SharePoint API |
| Fallback Retry Duration |
The time to wait before retrying the operation after a communication failure. This value is used when the response doesn't contain a Retry-After header. |
| Microsoft365 Group id |
Specifies a Microsoft365 group id to retrieve the members for. Supports Expression Language. |
## Relationships
| Name |
Description |
| comms.failure |
A FlowFile is routed here if the processor failed to communicate with the Graph API. Can be retried |
| failure |
An incoming FlowFile is routed to this relationship if the group members could not be fetched |
| not.found |
A FlowFile is routed here if the group was not found |
| success |
A FlowFile is routed here if the group members were successfully retrieved |
## Writes attributes
| Name |
Description |
| microsoft365.group.user.ids |
A comma-separated list of Microsoft365 user ids that are members of the Microsoft365 group. |
| microsoft365.group.user.emails |
A comma-separated list of user emails that are members of the Microsoft365 group. |
---
title: GetMongo 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getmongo.md
section: Loading & Unloading Data
---
# GetMongo 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-mongodb-nar
## Description
Creates FlowFiles from documents in MongoDB loaded by a user-specified query.
## Tags
get, mongodb, read
## Input Requirement
ALLOWED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Batch Size |
The number of elements to be returned from the server in one batch |
| Limit |
The maximum number of elements to return |
| Mongo Collection Name |
The name of the collection to use |
| Mongo Database Name |
The name of the database to use |
| Projection |
The fields to be returned from the documents in the result set; must be a valid BSON document |
| Query |
The selection criteria to do the lookup. If the field is left blank, it will look for input from an incoming connection from another processor to provide the query as a valid JSON document inside of the FlowFile's body. If this field is left blank and a timer is enabled instead of an incoming connection, that will result in a full collection fetch using a "\{\}" query. |
| Sort |
The fields by which to sort; must be a valid BSON document |
| get-mongo-send-empty |
If a query executes successfully, but returns no results, send an empty JSON document signifying no result. |
| json-type |
By default, MongoDB's Java driver returns "extended JSON". Some of the features of this variant of JSON may cause problems for other JSON parsers that expect only standard JSON types and conventions. This configuration setting controls whether to use extended JSON or provide a clean view that conforms to standard JSON. |
| mongo-charset |
Specifies the character set of the document data. |
| mongo-client-service |
If configured, this property will use the assigned client service for connection pooling. |
| mongo-date-format |
The date format string to use for formatting Date fields that are returned from Mongo. It is only applied when the JSON output format is set to Standard JSON. |
| mongo-query-attribute |
If set, the query will be written to a specified attribute on the output flowfiles. |
| results-per-flowfile |
How many results to put into a FlowFile at once. The whole body will be treated as a JSON array of results. |
| use-pretty-printing |
Choose whether or not to pretty print the JSON from the results of the query. Choosing 'True' can greatly increase the space requirements on disk depending on the complexity of the JSON document |
## Relationships
| Name |
Description |
| failure |
All input FlowFiles that are part of a failed query execution go here. |
| original |
All input FlowFiles that are part of a successful query execution go here. |
| success |
All FlowFiles that have the results of a successful query execution go here. |
## Writes attributes
| Name |
Description |
| mongo.database.name |
The database where the results came from. |
| mongo.collection.name |
The collection where the results came from. |
---
title: GetMongoRecord 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getmongorecord.md
section: Loading & Unloading Data
---
# GetMongoRecord 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-mongodb-nar
## Description
A record-based version of GetMongo that uses the Record writers to write the MongoDB result set.
## Tags
fetch, get, json, mongo, mongodb, record
## Input Requirement
ALLOWED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Batch Size |
The number of elements to be returned from the server in one batch |
| Limit |
The maximum number of elements to return |
| Mongo Collection Name |
The name of the collection to use |
| Mongo Database Name |
The name of the database to use |
| Projection |
The fields to be returned from the documents in the result set; must be a valid BSON document |
| Query |
The selection criteria to do the lookup. If the field is left blank, it will look for input from an incoming connection from another processor to provide the query as a valid JSON document inside of the FlowFile's body. If this field is left blank and a timer is enabled instead of an incoming connection, that will result in a full collection fetch using a "\{\}" query. |
| Sort |
The fields by which to sort; must be a valid BSON document |
| get-mongo-record-writer-factory |
The record writer to use to write the result sets. |
| mongo-client-service |
If configured, this property will use the assigned client service for connection pooling. |
| mongo-query-attribute |
If set, the query will be written to a specified attribute on the output flowfiles. |
| mongodb-schema-name |
The name of the schema in the configured schema registry to use for the query results. |
## Relationships
| Name |
Description |
| failure |
All input FlowFiles that are part of a failed query execution go here. |
| original |
All input FlowFiles that are part of a successful query execution go here. |
| success |
All FlowFiles that have the results of a successful query execution go here. |
## Writes attributes
| Name |
Description |
| mongo.database.name |
The database where the results came from. |
| mongo.collection.name |
The collection where the results came from. |
---
title: GetQueryJobResult 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getqueryjobresult.md
section: Loading & Unloading Data
---
# GetQueryJobResult 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-salesforce-processors-nar
## Description
Gets the results of a Query Job in Salesforce using the Bulk API 2.0. The output is CSV and GZIP compression is used.
## Tags
bulk, job, preview, query, salesforce
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Job ID |
The ID of the job for which the status is checked. |
| Salesforce Client |
Salesforce Client to interact with the APIs |
## Relationships
| Name |
Description |
| comms.failure |
A FlowFile is routed to this relationship if the Query Job result could not be retrieved but the operation might be retried |
| failure |
A FlowFile is routed to this relationship if the Query Job Results could not be retrieved |
| success |
If Query Job Results have been successfully retrieved, the FlowFile is routed to this relationship |
## See also
- [com.snowflake.openflow.runtime.processors.salesforce.AbortQueryJob](/user-guide/data-integration/openflow/processors/abortqueryjob)
- [com.snowflake.openflow.runtime.processors.salesforce.DeleteQueryJob](/user-guide/data-integration/openflow/processors/deletequeryjob)
- [com.snowflake.openflow.runtime.processors.salesforce.GetQueryJobStatus](/user-guide/data-integration/openflow/processors/getqueryjobstatus)
- [com.snowflake.openflow.runtime.processors.salesforce.SubmitQueryJob](/user-guide/data-integration/openflow/processors/submitqueryjob)
---
title: GetQueryJobStatus 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getqueryjobstatus.md
section: Loading & Unloading Data
---
# GetQueryJobStatus 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-salesforce-processors-nar
## Description
Gets the status of a Query Job in Salesforce using the Bulk API 2.0.
## Tags
bulk, job, preview, query, salesforce, status
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Job ID |
The ID of the job for which the status is checked. |
| Salesforce Client |
Salesforce Client to interact with the APIs |
## Relationships
| Name |
Description |
| comms.failure |
A FlowFile is routed to this relationship if the Query Job status could not be retrieved but the operation might be retried |
| failure |
A FlowFile is routed to this relationship if the Query Job status could not be retrieved |
| job.aborted |
If the Query Job has been aborted, the FlowFile is routed to this relationship |
| job.completed |
If the Query Job completed, the FlowFile is routed to this relationship |
| job.failed |
If the Query Job failed, the FlowFile is routed to this relationship |
| wait |
If the Query Job is in the processing queue or in progress, the FlowFile is routed to this relationship |
## Writes attributes
| Name |
Description |
| jobState |
The current state of processing for the job. |
| systemModstamp |
The UTC date and time when the API last updated the job information. |
| numberRecordsProcessed |
The number of records processed in this job. |
| retries |
The number of times that Salesforce attempted to save the results of an operation. Repeated attempts indicate a problem such as a lock contention. |
| totalProcessingTime |
The number of milliseconds taken to process the job. |
| isPkChunkingSupported |
Whether PK chunking is supported for the queried object (true), or isn't supported (false). |
## See also
- [com.snowflake.openflow.runtime.processors.salesforce.AbortQueryJob](/user-guide/data-integration/openflow/processors/abortqueryjob)
- [com.snowflake.openflow.runtime.processors.salesforce.DeleteQueryJob](/user-guide/data-integration/openflow/processors/deletequeryjob)
- [com.snowflake.openflow.runtime.processors.salesforce.GetQueryJobResult](/user-guide/data-integration/openflow/processors/getqueryjobresult)
- [com.snowflake.openflow.runtime.processors.salesforce.SubmitQueryJob](/user-guide/data-integration/openflow/processors/submitqueryjob)
---
title: GetS3ObjectMetadata 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/gets3objectmetadata.md
section: Loading & Unloading Data
---
# GetS3ObjectMetadata 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-aws-nar
## Description
Check for the existence of an Object in S3 and fetch its Metadata without attempting to download it. This processor can be used as a router for workflows that need to check on an Object in S3 before proceeding with data processing
## Tags
AWS, Amazon, Archive, Exists, S3
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| AWS Credentials Provider service |
The Controller Service that is used to obtain AWS credentials provider |
| Bucket |
The S3 Bucket to interact with |
| Communications Timeout |
The amount of time to wait in order to establish a connection to AWS or receive data from AWS before timing out. |
| Custom Signer Class Name |
Fully qualified class name of the custom signer class. The signer must implement com.amazonaws.auth. Signer interface. |
| Custom Signer Module Location |
Comma-separated list of paths to files and/or directories which contain the custom signer's JAR file and its dependencies (if any). |
| Endpoint Override URL |
Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. |
| FullControl User List |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Full Control for an object |
| Metadata Attribute Include Pattern |
A regular expression pattern to use for determining which object metadata entries are included as FlowFile attributes. This pattern is only applied to the 'found' relationship and will not be used to filter the error attributes in the 'failure' relationship. |
| Metadata Target |
This determines where the metadata will be written when found. |
| Object Key |
The S3 Object Key to use. This is analogous to a filename for traditional file systems. |
| Owner |
The Amazon ID to use for the object's owner |
| Read ACL User List |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have permissions to read the Access Control List for an object |
| Read Permission User List |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Read Access for an object |
| Region |
The AWS Region to connect to. |
| SSL Context Service |
Specifies an optional SSL Context Service that, if provided, will be used to create connections |
| Signer Override |
The AWS S3 library uses Signature Version 4 by default but this property allows you to specify the Version 2 signer to support older S3-compatible services or even to plug in your own custom signer implementation. |
| Version |
The Version of the Object for which to retrieve Metadata |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| failure |
If the Processor is unable to process a given FlowFile, it will be routed to this Relationship. |
| found |
An object was found in the bucket at the supplied key |
| not found |
No object was found in the bucket the supplied key |
## See also
- [org.apache.nifi.processors.aws.s3.DeleteS3Object](/user-guide/data-integration/openflow/processors/deletes3object)
- [org.apache.nifi.processors.aws.s3.FetchS3Object](/user-guide/data-integration/openflow/processors/fetchs3object)
- [org.apache.nifi.processors.aws.s3.GetS3ObjectTags](/user-guide/data-integration/openflow/processors/gets3objecttags)
- [org.apache.nifi.processors.aws.s3.ListS3](/user-guide/data-integration/openflow/processors/lists3)
- [org.apache.nifi.processors.aws.s3.PutS3Object](/user-guide/data-integration/openflow/processors/puts3object)
- [org.apache.nifi.processors.aws.s3.TagS3Object](/user-guide/data-integration/openflow/processors/tags3object)
---
title: GetS3ObjectTags 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/gets3objecttags.md
section: Loading & Unloading Data
---
# GetS3ObjectTags 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-aws-nar
## Description
Check for the existence of an Object in S3 and fetch its Tags without attempting to download it. This processor can be used as a router for workflows that need to check on an Object in S3 before proceeding with data processing
## Tags
AWS, Amazon, Archive, Exists, S3
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| AWS Credentials Provider service |
The Controller Service that is used to obtain AWS credentials provider |
| Bucket |
The S3 Bucket to interact with |
| Communications Timeout |
The amount of time to wait in order to establish a connection to AWS or receive data from AWS before timing out. |
| Custom Signer Class Name |
Fully qualified class name of the custom signer class. The signer must implement com.amazonaws.auth. Signer interface. |
| Custom Signer Module Location |
Comma-separated list of paths to files and/or directories which contain the custom signer's JAR file and its dependencies (if any). |
| Endpoint Override URL |
Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. |
| FullControl User List |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Full Control for an object |
| Object Key |
The S3 Object Key to use. This is analogous to a filename for traditional file systems. |
| Owner |
The Amazon ID to use for the object's owner |
| Read ACL User List |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have permissions to read the Access Control List for an object |
| Read Permission User List |
A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Read Access for an object |
| Region |
The AWS Region to connect to. |
| SSL Context Service |
Specifies an optional SSL Context Service that, if provided, will be used to create connections |
| Signer Override |
The AWS S3 library uses Signature Version 4 by default but this property allows you to specify the Version 2 signer to support older S3-compatible services or even to plug in your own custom signer implementation. |
| Tag Attribute Include Pattern |
A regular expression pattern to use for determining which object tags are included as FlowFile attributes. This pattern is only applied to the 'found' relationship and will not be used to filter the error attributes in the 'failure' relationship. |
| Tags Target |
This determines where the tags will be written when found. |
| Version |
The Version of the Object for which to retrieve Tags |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| failure |
If the Processor is unable to process a given FlowFile, it will be routed to this Relationship. |
| found |
An object was found in the bucket at the supplied key |
| not found |
No object was found in the bucket the supplied key |
## See also
- [org.apache.nifi.processors.aws.s3.DeleteS3Object](/user-guide/data-integration/openflow/processors/deletes3object)
- [org.apache.nifi.processors.aws.s3.FetchS3Object](/user-guide/data-integration/openflow/processors/fetchs3object)
- [org.apache.nifi.processors.aws.s3.GetS3ObjectMetadata](/user-guide/data-integration/openflow/processors/gets3objectmetadata)
- [org.apache.nifi.processors.aws.s3.ListS3](/user-guide/data-integration/openflow/processors/lists3)
- [org.apache.nifi.processors.aws.s3.PutS3Object](/user-guide/data-integration/openflow/processors/puts3object)
- [org.apache.nifi.processors.aws.s3.TagS3Object](/user-guide/data-integration/openflow/processors/tags3object)
---
title: GetSFTP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getsftp.md
section: Loading & Unloading Data
---
# GetSFTP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Fetches files from an SFTP Server and creates FlowFiles from them
## Tags
fetch, files, get, ingest, input, remote, retrieve, sftp, source
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Algorithm Negotiation |
Configuration strategy for SSH algorithm negotiation |
| Ciphers Allowed |
A comma-separated list of Ciphers allowed for SFTP connections. Leave unset to allow all. Available options are: 3des-cbc, aes128-cbc, aes128-ctr, [aes128-gcm@openssh.com](mailto:aes128-gcm@openssh.com), aes192-cbc, aes192-ctr, aes256-cbc, aes256-ctr, [aes256-gcm@openssh.com](mailto:aes256-gcm@openssh.com), arcfour128, arcfour256, blowfish-cbc, [chacha20-poly1305@openssh.com](mailto:chacha20-poly1305@openssh.com), none |
| Connection Timeout |
Amount of time to wait before timing out while creating a connection |
| Data Timeout |
When transferring a file between the local and remote system, this value specifies how long is allowed to elapse without any data being transferred between systems |
| Delete Original |
Determines whether or not the file is deleted from the remote system after it has been successfully transferred |
| File Filter Regex |
Provides a Java Regular Expression for filtering Filenames; if a filter is supplied, only files whose names match that Regular Expression will be fetched |
| Follow Symbolic Links |
If true, will pull even symbolic files and also nested symbolic subdirectories; otherwise, will not read symbolic files and will not traverse symbolic link subdirectories |
| Host Key File |
If supplied, the given file will be used as the Host Key; otherwise, if 'Strict Host Key Checking' property is applied (set to true) then uses the 'known_hosts' and 'known_hosts2' files from ~/.ssh directory else no host key file will be used |
| Hostname |
The fully qualified hostname or IP address of the remote system |
| Ignore Dotted Files |
If true, files whose names begin with a dot (".") will be ignored |
| Key Algorithms Allowed |
A comma-separated list of Key Algorithms allowed for SFTP connections. Leave unset to allow all. Available options are: ecdsa-sha2-nistp256, [ecdsa-sha2-nistp256-cert-v01@openssh.com](mailto:ecdsa-sha2-nistp256-cert-v01@openssh.com), ecdsa-sha2-nistp384, [ecdsa-sha2-nistp384-cert-v01@openssh.com](mailto:ecdsa-sha2-nistp384-cert-v01@openssh.com), ecdsa-sha2-nistp521, [ecdsa-sha2-nistp521-cert-v01@openssh.com](mailto:ecdsa-sha2-nistp521-cert-v01@openssh.com), rsa-sha2-256, [rsa-sha2-256-cert-v01@openssh.com](mailto:rsa-sha2-256-cert-v01@openssh.com), rsa-sha2-512, [rsa-sha2-512-cert-v01@openssh.com](mailto:rsa-sha2-512-cert-v01@openssh.com), [sk-ecdsa-sha2-nistp256@openssh.com](mailto:sk-ecdsa-sha2-nistp256@openssh.com), [sk-ssh-ed25519@openssh.com](mailto:sk-ssh-ed25519@openssh.com), ssh-dss, [ssh-dss-cert-v01@openssh.com](mailto:ssh-dss-cert-v01@openssh.com), ssh-ed25519, [ssh-ed25519-cert-v01@openssh.com](mailto:ssh-ed25519-cert-v01@openssh.com), ssh-rsa, [ssh-rsa-cert-v01@openssh.com](mailto:ssh-rsa-cert-v01@openssh.com) |
| Key Exchange Algorithms Allowed |
A comma-separated list of Key Exchange Algorithms allowed for SFTP connections. Leave unset to allow all. Available options are: curve25519-sha256, [curve25519-sha256@libssh.org](mailto:curve25519-sha256@libssh.org), curve448-sha512, diffie-hellman-group-exchange-sha1, diffie-hellman-group-exchange-sha256, diffie-hellman-group1-sha1, diffie-hellman-group14-sha1, diffie-hellman-group14-sha256, diffie-hellman-group15-sha512, diffie-hellman-group16-sha512, diffie-hellman-group17-sha512, diffie-hellman-group18-sha512, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, mlkem1024nistp384-sha384, mlkem768nistp256-sha256, mlkem768x25519-sha256, sntrup761x25519-sha512, [sntrup761x25519-sha512@openssh.com](mailto:sntrup761x25519-sha512@openssh.com) |
| Max Selects |
The maximum number of files to pull in a single connection |
| Message Authentication Codes Allowed |
A comma-separated list of Message Authentication Codes allowed for SFTP connections. Leave unset to allow all. Available options are: hmac-md5, hmac-md5-96, hmac-sha1, hmac-sha1-96, [hmac-sha1-etm@openssh.com](mailto:hmac-sha1-etm@openssh.com), hmac-sha2-256, [hmac-sha2-256-etm@openssh.com](mailto:hmac-sha2-256-etm@openssh.com), hmac-sha2-512, [hmac-sha2-512-etm@openssh.com](mailto:hmac-sha2-512-etm@openssh.com) |
| Password |
Password for the user account |
| Path Filter Regex |
When Search Recursively is true, then only subdirectories whose path matches the given Regular Expression will be scanned |
| Polling Interval |
Determines how long to wait between fetching the listing for new files |
| Port |
The port that the remote system is listening on for file transfers |
| Private Key Passphrase |
Password for the private key |
| Private Key Path |
The fully qualified path to the Private Key file |
| Remote Path |
The path on the remote system from which to pull or push files |
| Remote Poll Batch Size |
The value specifies how many file paths to find in a given directory on the remote system when doing a file listing. This value in general should not need to be modified but when polling against a remote system with a tremendous number of files this value can be critical. Setting this value too high can result very poor performance and setting it too low can cause the flow to be slower than normal. |
| Search Recursively |
If true, will pull files from arbitrarily nested subdirectories; otherwise, will not traverse subdirectories |
| Send Keep Alive On Timeout |
Send a Keep Alive message every 5 seconds up to 5 times for an overall timeout of 25 seconds. |
| Strict Host Key Checking |
Indicates whether or not strict enforcement of hosts keys should be applied |
| Use Compression |
Indicates whether or not ZLIB compression should be used when transferring files |
| Use Natural Ordering |
If true, will pull files in the order in which they are naturally listed; otherwise, the order in which the files will be pulled is not defined |
| Username |
Username |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| success |
All FlowFiles that are received are routed to success |
## Writes attributes
| Name |
Description |
| filename |
The filename is set to the name of the file on the remote server |
| path |
The path is set to the path of the file's directory on the remote server. For example, if the <Remote Path> property is set to /tmp, files picked up from /tmp will have the path attribute set to /tmp. If the <Search Recursively> property is set to true and a file is picked up from /tmp/abc/1/2/3, then the path attribute will be set to /tmp/abc/1/2/3 |
| file.lastModifiedTime |
The date and time that the source file was last modified |
| file.owner |
The numeric owner id of the source file |
| file.group |
The numeric group id of the source file |
| file.permissions |
The read/write/execute permissions of the source file |
| absolute.path |
The full/absolute path from where a file was picked up. The current 'path' attribute is still populated, but may be a relative path |
## See also
- [org.apache.nifi.processors.standard.PutSFTP](/user-guide/data-integration/openflow/processors/putsftp)
---
title: GetSharepointSiteGroupMembers 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getsharepointsitegroupmembers.md
section: Loading & Unloading Data
---
# GetSharepointSiteGroupMembers 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-sharepoint-rest-nar
## Description
Retrieves all members of a SharePoint site group.
## Tags
groups, membership, microsoft, openflow, sharepoint
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Group ID |
The ID of the SharePoint group. |
| OAuth2 Access Token Provider |
Enables managed retrieval of OAuth2 Bearer Token. |
| Site URL |
The URL of the SharePoint site. |
| Web Client Service |
The Web Client Service to use for communicating with SharePoint. |
## Relationships
| Name |
Description |
| comms.failure |
A FlowFile is routed here if the processor failed to communicate with SharePoint. Can be retried |
| failure |
A FlowFile is routed here if the group members could not be fetched |
| success |
A FlowFile is routed here if the group members were successfully retrieved |
## Writes attributes
| Name |
Description |
| sharepoint.group.user.ids |
The IDs of the users in the SharePoint site group. |
| sharepoint.group.user.emails |
The emails of the users in the SharePoint site group. |
## See also
- [com.snowflake.openflow.runtime.processors.sharepoint.rest.ListSharepointSiteGroups](/user-guide/data-integration/openflow/processors/listsharepointsitegroups)
---
title: GetShopify 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getshopify.md
section: Loading & Unloading Data
---
# GetShopify 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-shopify-nar
## Description
Retrieves objects from a custom Shopify store. The processor yield time must be set to the account's rate limit accordingly.
## Tags
shopify
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| CUSTOMERS |
Customer resource to query |
| DISCOUNTS |
Discount resource to query |
| INVENTORY |
Inventory resource to query |
| ONLINE_STORE |
Online Store resource to query |
| ORDERS |
Order resource to query |
| PRODUCT |
Product resource to query |
| SALES_CHANNELS |
Sales Channel resource to query |
| STORE_PROPERTIES |
Store Property resource to query |
| access-token |
Access Token to authenticate requests |
| api-version |
The Shopify REST API version |
| incremental-delay |
The ending timestamp of the time window will be adjusted earlier by the amount configured in this property. For example, with a property value of 10 seconds, an ending timestamp of 12:30:45 would be changed to 12:30:35. Set this property to avoid missing objects when the clock of your local machines and Shopify servers' clock are not in sync. |
| incremental-initial-start-time |
This property specifies the start time when running the first request. Represents an ISO 8601-encoded date and time string. For example, 3:50 pm on September 7, 2019 in the time zone of UTC (Coordinated Universal Time) is represented as "2019-09-07T15:50:00Z". |
| is-incremental |
The processor can incrementally load the queried objects so that each object is queried exactly once. For each query, the processor queries objects which were created or modified after the previous run time but before the current time. |
| object-category |
Shopify object category |
| result-limit |
The maximum number of results to request for each invocation of the Processor |
| store-domain |
The domain of the Shopify store, e.g. nifistore.myshopify.com |
| web-client-service-provider |
Controller service for HTTP client operations |
## State management
| Scopes |
Description |
| CLUSTER |
For a few resources the processor supports incremental loading. The list of the resources with the supported parameters can be found in the additional details. |
## Relationships
| Name |
Description |
| success |
For FlowFiles created as a result of a successful query. |
## Writes attributes
| Name |
Description |
| mime.type |
Sets the MIME type to application/json |
---
title: GetSmbFile 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getsmbfile.md
section: Loading & Unloading Data
---
# GetSmbFile 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-smb-nar
## Description
Reads file from a samba network location to FlowFiles. Use this processor instead of a cifs mounts if share access control is important. Configure the Hostname, Share and Directory accordingly: \[Hostname][Share][pathtoDirectory]
## Tags
samba, smb, cifs, files, get
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Batch Size |
The maximum number of files to pull in each iteration |
| Directory |
The network folder to which files should be written. This is the remaining relative path after the share: \hostnameshare[dir1dir2]. |
| Domain |
The domain used for authentication. Optional, in most cases username and password is sufficient. |
| File Filter |
Only files whose names match the given regular expression will be picked up |
| Hostname |
The network host to which files should be written. |
| Ignore Hidden Files |
Indicates whether or not hidden files should be ignored |
| Keep Source File |
If true, the file is not deleted after it has been copied to the Content Repository; this causes the file to be picked up continually and is useful for testing purposes. If not keeping original NiFi will need write permissions on the directory it is pulling from otherwise it will ignore the file. |
| Password |
The password used for authentication. Required if Username is set. |
| Path Filter |
When Recurse Subdirectories is true, then only subdirectories whose path matches the given regular expression will be scanned |
| Polling Interval |
Indicates how long to wait before performing a directory listing |
| Recurse Subdirectories |
Indicates whether or not to pull files from subdirectories |
| Share |
The network share to which files should be written. This is the "first folder"after the hostname: \hostname[share]dir1dir2 |
| Share Access Strategy |
Indicates which shared access are granted on the file during the read. None is the most restrictive, but the safest setting to prevent corruption. |
| Username |
The username used for authentication. If no username is set then anonymous authentication is attempted. |
| enable-dfs |
Enables accessing Distributed File System (DFS) and following DFS links during SMB operations. |
| smb-dialect |
The SMB dialect is negotiated between the client and the server by default to the highest common version supported by both end. In some rare cases, the client-server communication may fail with the automatically negotiated dialect. This property can be used to set the dialect explicitly (e.g. to downgrade to a lower version), when those situations would occur. |
| timeout |
Timeout for read and write operations. |
| use-encryption |
Turns on/off encrypted communication between the client and the server. The property's behavior is SMB dialect dependent: SMB 2.x does not support encryption and the property has no effect. In case of SMB 3.x, it is a hint/request to the server to turn encryption on if the server also supports it. |
## Relationships
| Name |
Description |
| success |
All files are routed to success |
## Writes attributes
| Name |
Description |
| filename |
The filename is set to the name of the file on the network share |
| path |
The path is set to the relative path of the file's network share name. For example, if the input is set to \hostnamesharetmp, files picked up from tmp will have the path attribute set to tmp |
| file.creationTime |
The date and time that the file was created. May not work on all file systems |
| file.lastModifiedTime |
The date and time that the file was last modified. May not work on all file systems |
| file.lastAccessTime |
The date and time that the file was last accessed. May not work on all file systems |
| absolute.path |
The full path from where a file was picked up. This includes the hostname and the share name |
## See also
- [org.apache.nifi.processors.smb.FetchSmb](/user-guide/data-integration/openflow/processors/fetchsmb)
- [org.apache.nifi.processors.smb.ListSmb](/user-guide/data-integration/openflow/processors/listsmb)
- [org.apache.nifi.processors.smb.PutSmbFile](/user-guide/data-integration/openflow/processors/putsmbfile)
---
title: GetSplunk 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getsplunk.md
section: Loading & Unloading Data
---
# GetSplunk 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-splunk-nar
## Description
Retrieves data from Splunk Enterprise.
## Tags
get, logs, splunk
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| API Version |
Select which version of the Splunk Search API to use for search operations. Version 2 is recommended for newer Splunk instances. |
| Application |
The Splunk Application to query. |
| Connection Timeout |
Max wait time for connection to the Splunk server. |
| Earliest Time |
The value to use for the earliest time when querying. Only used with a Time Range Strategy of Provided. See Splunk's documentation on Search Time Modifiers for guidance in populating this field. |
| Hostname |
The ip address or hostname of the Splunk server. |
| Latest Time |
The value to use for the latest time when querying. Only used with a Time Range Strategy of Provided. See Splunk's documentation on Search Time Modifiers for guidance in populating this field. |
| Output Mode |
The output mode for the results. |
| Owner |
The owner to pass to Splunk. |
| Password |
The password to authenticate to Splunk. |
| Port |
The port of the Splunk server. |
| Query |
The query to execute. Typically beginning with a <search> command followed by a search clause, such as <search source="[tcp:7689](tcp:7689)"> to search for messages received on TCP port 7689. |
| Read Timeout |
Max wait time for response from the Splunk server. |
| SSL Context Service |
The SSL Context Service used to provide client certificate information for TLS/SSL connections. |
| Scheme |
The scheme for connecting to Splunk. |
| Security Protocol |
The security protocol to use for communicating with Splunk. |
| Time Field Strategy |
Indicates whether to search by the time attached to the event, or by the time the event was indexed in Splunk. |
| Time Range Strategy |
Indicates how to apply time ranges to each execution of the query. Selecting a managed option allows the processor to apply a time range from the last execution time to the current execution time. When using <Managed from Beginning>, an earliest time will not be applied on the first execution, and thus all records searched. When using <Managed from Current> the earliest time of the first execution will be the initial execution time. When using <Provided>, the time range will come from the Earliest Time and Latest Time properties, or no time range will be applied if these properties are left blank. |
| Time Zone |
The Time Zone to use for formatting dates when performing a search. Only used with Managed time strategies. |
| Token |
The token to pass to Splunk. |
| Username |
The username to authenticate to Splunk. |
## State management
| Scopes |
Description |
| CLUSTER |
If using one of the managed Time Range Strategies, this processor will store the values of the latest and earliest times from the previous execution so that the next execution of the can pick up where the last execution left off. The state will be cleared and start over if the query is changed. |
## Relationships
| Name |
Description |
| success |
Results retrieved from Splunk are sent out this relationship. |
## Writes attributes
| Name |
Description |
| splunk.query |
The query that performed to produce the FlowFile. |
| splunk.earliest.time |
The value of the earliest time that was used when performing the query. |
| splunk.latest.time |
The value of the latest time that was used when performing the query. |
---
title: GetSQS 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getsqs.md
section: Loading & Unloading Data
---
# GetSQS 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-aws-nar
## Description
Fetches messages from an Amazon Simple Queuing Service Queue
## Tags
AWS, Amazon, Fetch, Get, Poll, Queue, SQS
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| AWS Credentials Provider service |
The Controller Service that is used to obtain AWS credentials provider |
| Auto Delete Messages |
Specifies whether the messages should be automatically deleted by the processors once they have been received. |
| Batch Size |
The maximum number of messages to send in a single network request |
| Character Set |
The Character Set that should be used to encode the textual content of the SQS message |
| Communications Timeout |
|
| Endpoint Override URL |
Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. |
| Queue URL |
The URL of the queue to get messages from |
| Receive Message Wait Time |
The maximum amount of time to wait on a long polling receive call. Setting this to a value of 1 second or greater will reduce the number of SQS requests and decrease fetch latency at the cost of a constantly active thread. |
| Region |
|
| SSL Context Service |
Specifies an optional SSL Context Service that, if provided, will be used to create connections |
| Visibility Timeout |
The amount of time after a message is received but not deleted that the message is hidden from other consumers |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## Relationships
| Name |
Description |
| success |
FlowFiles are routed to success relationship |
## Writes attributes
| Name |
Description |
| hash.value |
The MD5 sum of the message |
| hash.algorithm |
MD5 |
| sqs.message.id |
The unique identifier of the SQS message |
| sqs.receipt.handle |
The SQS Receipt Handle that is to be used to delete the message from the queue |
## See also
- [org.apache.nifi.processors.aws.sqs.DeleteSQS](/user-guide/data-integration/openflow/processors/deletesqs)
- [org.apache.nifi.processors.aws.sqs.PutSQS](/user-guide/data-integration/openflow/processors/putsqs)
---
title: GetUnityCatalogFile 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getunitycatalogfile.md
section: Loading & Unloading Data
---
# GetUnityCatalogFile 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-databricks-processors-nar
## Description
Read a Unity Catalog file up to 5 GiB.
## Tags
databricks, openflow, unity catalog
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Databricks Client |
Databricks Client Service. |
| Unity Catalog File Path |
Unity Catalog file path e.g. /Volumes/catalog/schema/volume_name/file.txt |
## Relationships
| Name |
Description |
| failure |
Databricks failure relationship |
| success |
Databricks success relationship |
## Writes attributes
| Name |
Description |
| error.code |
The error code for the SQL statement if an error occurred. |
| error.message |
The error message for the SQL statement if an error occurred. |
---
title: GetUnityCatalogFileMetadata 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getunitycatalogfilemetadata.md
section: Loading & Unloading Data
---
# GetUnityCatalogFileMetadata 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-databricks-processors-nar
## Description
Checks for Unity Catalog file metadata.
## Tags
databricks, openflow, unity catalog
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Databricks Client |
Databricks Client Service. |
| Unity Catalog File Path |
Unity Catalog file path e.g. /Volumes/catalog/schema/volume_name/file.txt |
## Relationships
| Name |
Description |
| failure |
Databricks failure relationship |
| not.found |
The original FlowFile is transferred to this relationship if no Unity Catalog can be found at the specified path |
| success |
Databricks success relationship |
## Writes attributes
| Name |
Description |
| mime.type |
The content type of the checked file. |
| uc.size |
The size of the Unity Catalog file. |
| uc.lastModifiedTime |
The last modified time of the Unity Catalog file in milliseconds since epoch in UTC time. |
| error.code |
The error code for the SQL statement if an error occurred. |
| error.message |
The error message for the SQL statement if an error occurred. |
---
title: GetWorkdayReport 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getworkdayreport.md
section: Loading & Unloading Data
---
# GetWorkdayReport 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-workday-processors-nar
## Description
A processor which can interact with a configurable Workday Report. The processor can forward the content without modification, or you can transform it by providing the specific Record Reader and Record Writer services based on your needs. You can also remove fields by defining schema in the Record Writer. Supported Workday report formats are: csv, simplexml, json
## Tags
Workday, report
## Input Requirement
ALLOWED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Access Token Provider |
Enables managed retrieval of OAuth2 Bearer Token. |
| Authorization Type |
The type of authorization for retrieving data from Workday resources. |
| Web Client Service Provider |
Web client which is used to communicate with the Workday API. |
| Workday Password |
The password provided for authentication of Workday requests. Encoded using Base64 for HTTP Basic Authentication as described in RFC 7617. |
| Workday Report URL |
HTTP remote URL of Workday report including a scheme of http or https, as well as a hostname or IP address with optional port and path elements. |
| Workday Username |
The username provided for authentication of Workday requests. Encoded using Base64 for HTTP Basic Authentication as described in RFC 7617. |
| record-reader |
Specifies the Controller Service to use for parsing incoming data and determining the data's schema. |
| record-writer |
The Record Writer to use for serializing Records to an output FlowFile. |
## Relationships
| Name |
Description |
| failure |
Request FlowFiles transferred when receiving socket communication errors. |
| original |
Request FlowFiles transferred when receiving HTTP responses with a status code between 200 and 299. |
| success |
Response FlowFiles transferred when receiving HTTP responses with a status code between 200 and 299. |
## Writes attributes
| Name |
Description |
| getworkdayreport.java.exception.class |
The Java exception class raised when the processor fails |
| getworkdayreport.java.exception.message |
The Java exception message raised when the processor fails |
| mime.type |
Sets the mime.type attribute to the MIME Type specified by the Source / Record Writer |
| record.count |
The number of records in an outgoing FlowFile. This is only populated on the 'success' relationship when Record Reader and Writer is set. |
---
title: GetZendesk 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/getzendesk.md
section: Loading & Unloading Data
---
# GetZendesk 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-zendesk-nar
## Description
Incrementally fetches data from Zendesk API.
## Tags
zendesk
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| web-client-service-provider |
Controller service for HTTP client operations. |
| zendesk-authentication-type-name |
Type of authentication to Zendesk API. |
| zendesk-authentication-value-name |
Password or authentication token for Zendesk login user. |
| zendesk-export-method |
Method for incremental export. |
| zendesk-query-start-timestamp |
Initial timestamp to query Zendesk API from in Unix timestamp seconds format. |
| zendesk-resource |
The particular Zendesk resource which is meant to be exported. |
| zendesk-subdomain |
Name of the Zendesk subdomain. |
| zendesk-user |
Login user to Zendesk subdomain. |
## State management
| Scopes |
Description |
| CLUSTER |
Paging cursor for Zendesk API is stored. Cursor is updated after each successful request. |
## Relationships
| Name |
Description |
| success |
For FlowFiles created as a result of a successful HTTP request. |
## Writes attributes
| Name |
Description |
| record.count |
The number of records fetched by the processor. |
---
title: GrokReader
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/grokreader.md
section: Loading & Unloading Data
---
# GrokReader
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides a mechanism for reading unstructured text data, such as log files, and structuring the data so that it can be processed. The service is configured using Grok patterns. The service reads from a stream of data and splits each message that it finds into a separate Record, each containing the fields that are configured. If a line in the input does not match the expected message pattern, the line of text is either considered to be part of the previous message or is skipped, depending on the configuration, with the exception of stack traces. A stack trace that is found at the end of a log message is considered to be part of the previous message but is added to the 'stackTrace' field of the Record. If a record has no stack trace, it will have a NULL value for the stackTrace field (assuming that the schema does in fact include a stackTrace field of type String). Assuming that the schema includes a '_raw' field of type String, the raw message will be included in the Record.
## Tags
grok, logfiles, logs, logstash, parse, pattern, reader, record, regex, text, unstructured
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Grok Expressions * |
Grok Expression |
|
|
Specifies the format of a log line in Grok format. This allows the Record Reader to understand how to parse each log line. The property supports one or more Grok expressions. The Reader attempts to parse input lines according to the configured order of the expressions.If a line in the log file does not match any expressions, the line will be assumed to belong to the previous log message.If other Grok patterns are referenced by this expression, they need to be supplied in the Grok Pattern File property. |
| Grok Patterns |
Grok Pattern File |
|
|
Grok Patterns to use for parsing logs. If not specified, a built-in default Pattern file will be used. If specified, all patterns specified will override the default patterns. See the Controller Service's Additional Details for a list of pre-defined patterns. |
| Schema Access Strategy * |
Schema Access Strategy |
string-fields-from-grok-expression |
- Use String Fields From Grok Expression
- Use 'Schema Name' Property
- Use 'Schema Text' Property
- Schema Reference Reader
|
Specifies how to obtain the schema that is to be used for interpreting the data. |
| Schema Branch |
Schema Branch |
|
|
Specifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored. |
| Schema Name |
Schema Name |
$\{schema.name\} |
|
Specifies the name of the schema to lookup in the Schema Registry property |
| Schema Reference Reader * |
Schema Reference Reader |
|
|
Service implementation responsible for reading FlowFile attributes or content to determine the Schema Reference Identifier |
| Schema Registry |
Schema Registry |
|
|
Specifies the Controller Service to use for the Schema Registry |
| Schema Text |
Schema Text |
$\{avro.schema\} |
|
The text of an Avro-formatted Schema |
| Schema Version |
Schema Version |
|
|
Specifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved. |
| No Match Behavior * |
no-match-behavior |
append-to-previous-message |
- Append to Previous Message
- Skip Line
- Raw Line
|
If a line of text is encountered and it does not match the given Grok Expression, and it is not part of a stack trace, this property specifies how the text should be processed. |
## State management
This component does not store state.
## Restricted
## Restrictions
| Required Permission |
Explanation |
| reference remote resources |
Patterns and Expressions can reference resources over HTTP |
## System Resource Considerations
This component does not specify system resource considerations.
---
title: Guidelines for using Python extensions in Openflow
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors-python-ext-guide.md
section: Loading & Unloading Data
---
# Guidelines for using Python extensions in Openflow
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
This topic describes the limitations, supported configurations, and best practices when
using Python extensions in Openflow.
Python processors in Openflow use NiFi's Py4J bridge architecture, which has fundamentally
different resource characteristics than native Java processors. Because Python processors
run as external OS processes outside the JVM, they consume additional system memory, are not
governed by NiFi's internal resource management, and have limited observability. These
differences affect runtime sizing, capacity planning, and monitoring.
## Architecture differences
Python processors run as external OS processes rather than within the JVM. This
architecture affects how resources are allocated, monitored, and managed:
| Processor type |
Java processor |
Python processor |
| Runtime environment |
JVM internal threads |
External OS process |
| Memory management |
Managed within JVM heap |
Separate process memory |
| Lifecycle |
NiFi-controlled |
External process lifecycle |
| Monitoring |
Full NiFi observability |
Limited visibility |
## Runtime size constraints
Python extensions are only available on Medium and Large runtimes. Small runtimes
do not support Python processors due to CPU and memory constraints. Snowflake Openflow
blocks Python extensions on Small runtimes:
| Runtime size |
Python support |
Notes |
| Small |
Not supported |
Python processors are blocked on Small runtimes due to CPU and memory constraints. |
| Medium |
Limited (up to 2 Python processors) |
The limit is for the entire runtime, not per connector or process group. This limit is currently a recommendation that will be an enforced maximum value for Openflow runtimes in the future. |
| Large |
Limited (up to 4 Python processors) |
The limit is for the entire runtime, not per connector or process group. This limit is currently a recommendation that will be an enforced maximum value for Openflow runtimes in the future. |
## Best practices
Follow these guidelines for working with Python processors in Openflow:
- Use Java for CPU-heavy operations. Java provides more efficient thread management
within the JVM. Groovy scripting is a Java-based alternative.
- Use Medium or Large runtimes. Python is not available on Small runtimes.
- Limit the number of Python processors. Stay within the documented limits per runtime size.
- Monitor resource usage. Watch for memory pressure and CPU contention.
- Plan for upgrades. Custom Python processors might require a virtual environment (venv) reset
after runtime upgrades. For more information, see
[Restore Python processors following runtime upgrades](#label-openflow-python-ext-restore).
- Use single-threaded Python processors. Openflow does not support Python processors spawning
subprocesses or using multithreading.
## Limitations on using Python processors
The following limitations apply when using Python processors in Openflow.
- Runtime constraints
-
Python extensions can only be used with Medium or Large runtimes. Python extensions
cannot be used with Small runtimes. This is disabled by the platform.
- Memory overhead
-
Each Python processor spawns an external OS process with its own memory footprint.
Python processes can collectively compete with the JVM for resources.
- No NiFi resource management
-
Python processors are not observed or limited by NiFi's internal resource management.
CPU-heavy Python operations can consume approximately 50% of total server CPU time.
- Monitoring gaps
-
The platform lacks visibility into external Python process health and resource consumption.
- Upgrade handling
-
After runtime upgrades, custom Python processors might fail to load or exhibit unexpected
behavior until virtual environments are recreated.
## Restore Python processors following runtime upgrades
If Python processors fail after upgrading the runtime, do the following:
1. Increment the processor version in the `ProcessorDetails.version` field.
2. Rebuild and re-upload the NiFi Archive (NAR) binary. This triggers the Python virtual
environment cache to reset.
3. Remove and re-add the processor on the canvas. This triggers reinitialization of the
Py4J bridge.
---
title: HandleHttpRequest 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/handlehttprequest.md
section: Loading & Unloading Data
---
# HandleHttpRequest 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Starts an HTTP Server and listens for HTTP Requests. For each request, creates a FlowFile and transfers to 'success'. This Processor is designed to be used in conjunction with the HandleHttpResponse Processor in order to create a Web Service. In case of a multipart request, one FlowFile is generated for each part.
## Tags
http, https, ingress, listen, request, web service
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Additional HTTP Methods |
A comma-separated list of non-standard HTTP Methods that should be allowed |
| Allow DELETE |
Allow HTTP DELETE Method |
| Allow GET |
Allow HTTP GET Method |
| Allow HEAD |
Allow HTTP HEAD Method |
| Allow OPTIONS |
Allow HTTP OPTIONS Method |
| Allow POST |
Allow HTTP POST Method |
| Allow PUT |
Allow HTTP PUT Method |
| Allowed Paths |
A Regular Expression that specifies the valid HTTP Paths that are allowed in the incoming URL Requests. If this value is specified and the path of the HTTP Requests does not match this Regular Expression, the Processor will respond with a 404: NotFound |
| Client Authentication |
Specifies whether or not the Processor should authenticate clients. This value is ignored if the <SSL Context Service> Property is not specified or the SSL Context provided uses only a KeyStore and not a TrustStore. |
| Default URL Character Set |
The character set to use for decoding URL parameters if the HTTP Request does not supply one |
| HTTP Context Map |
The HTTP Context Map Controller Service to use for caching the HTTP Request Information |
| HTTP Protocols |
HTTP Protocols supported for Application Layer Protocol Negotiation with TLS |
| Hostname |
The Hostname to bind to. If not specified, will bind to all hosts |
| Listening Port |
The Port to listen on for incoming HTTP requests |
| Maximum Threads |
The maximum number of threads that the embedded HTTP server will use for handling requests. |
| Request Header Maximum Size |
The maximum supported size of HTTP headers in requests sent to this processor |
| SSL Context Service |
The SSL Context Service to use in order to secure the server. If specified, the server will accept only HTTPS requests; otherwise, the server will accept only HTTP requests |
| container-queue-size |
The size of the queue for Http Request Containers |
| multipart-read-buffer-size |
The threshold size, at which the contents of an incoming file would be written to disk. Only applies for requests with Content-Type: multipart/form-data. It is used to prevent denial of service type of attacks, to prevent filling up the heap or disk space. |
| multipart-request-max-size |
The max size of the request. Only applies for requests with Content-Type: multipart/form-data, and is used to prevent denial of service type of attacks, to prevent filling up the heap or disk space |
| parameters-to-attributes |
A comma-separated list of HTTP parameters or form data to output as attributes |
## Relationships
| Name |
Description |
| success |
All content that is received is routed to the 'success' relationship |
## Writes attributes
| Name |
Description |
| http.context.identifier |
An identifier that allows the HandleHttpRequest and HandleHttpResponse to coordinate which FlowFile belongs to which HTTP Request/Response. |
| mime.type |
The MIME Type of the data, according to the HTTP Header "Content-Type" |
| http.servlet.path |
The part of the request URL that is considered the Servlet Path |
| http.context.path |
The part of the request URL that is considered to be the Context Path |
| http.method |
The HTTP Method that was used for the request, such as GET or POST |
| http.local.name |
IP address/hostname of the server |
| http.server.port |
Listening port of the server |
| http.query.string |
The query string portion of the Request URL |
| http.remote.host |
The hostname of the requestor |
| http.remote.addr |
The hostname:port combination of the requestor |
| http.remote.user |
The username of the requestor |
| http.protocol |
The protocol used to communicate |
| http.request.uri |
The full Request URL |
| http.auth.type |
The type of HTTP Authorization used |
| http.principal.name |
The name of the authenticated user making the request |
| http.query.param.XXX |
Each of query parameters in the request will be added as an attribute, prefixed with "http.query.param." |
| http.param.XXX |
Form parameters in the request that are configured by "Parameters to Attributes List" will be added as an attribute, prefixed with "http.param.". Putting form parameters of large size is not recommended. |
| http.subject.dn |
The Distinguished Name of the requestor. This value will not be populated unless the Processor is configured to use an SSLContext Service |
| http.issuer.dn |
The Distinguished Name of the entity that issued the Subject's certificate. This value will not be populated unless the Processor is configured to use an SSLContext Service |
| http.certificate.sans.N.name |
X.509 Client Certificate Subject Alternative Name value from mutual TLS authentication. The attribute name has a zero-based index ordered according to the content of Client Certificate |
| http.certificate.sans.N.nameType |
X.509 Client Certificate Subject Alternative Name type from mutual TLS authentication. The attribute name has a zero-based index ordered according to the content of Client Certificate. The attribute value is one of the General Names from RFC 3280 Section 4.1.2.7 |
| http.headers.XXX |
Each of the HTTP Headers that is received in the request will be added as an attribute, prefixed with "http.headers." For example, if the request contains an HTTP Header named "x-my-header", then the value will be added to an attribute named "http.headers.x-my-header" |
| http.headers.multipart.XXX |
Each of the HTTP Headers that is received in the multipart request will be added as an attribute, prefixed with "http.headers.multipart." For example, if the multipart request contains an HTTP Header named "content-disposition", then the value will be added to an attribute named "http.headers.multipart.content-disposition" |
| http.multipart.size |
For requests with Content-Type "multipart/form-data", the part's content size is recorded into this attribute |
| http.multipart.content.type |
For requests with Content-Type "multipart/form-data", the part's content type is recorded into this attribute |
| http.multipart.name |
For requests with Content-Type "multipart/form-data", the part's name is recorded into this attribute |
| http.multipart.filename |
For requests with Content-Type "multipart/form-data", when the part contains an uploaded file, the name of the file is recorded into this attribute. Files are stored temporarily at the default temporary-file directory specified in "java.io.File" Java Docs) |
| http.multipart.fragments.sequence.number |
For requests with Content-Type "multipart/form-data", the part's index is recorded into this attribute. The index starts with 1. |
| http.multipart.fragments.total.number |
For requests with Content-Type "multipart/form-data", the count of all parts is recorded into this attribute. |
## See also
- [org.apache.nifi.processors.standard.HandleHttpResponse](/user-guide/data-integration/openflow/processors/handlehttpresponse)
---
title: HandleHttpResponse 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/handlehttpresponse.md
section: Loading & Unloading Data
---
# HandleHttpResponse 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Sends an HTTP Response to the Requestor that generated a FlowFile. This Processor is designed to be used in conjunction with the HandleHttpRequest in order to create a web service.
## Tags
egress, http, https, response, web service
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Attributes to add to the HTTP Response (Regex) |
Specifies the Regular Expression that determines the names of FlowFile attributes that should be added to the HTTP response |
| HTTP Context Map |
The HTTP Context Map Controller Service to use for caching the HTTP Request Information |
| HTTP Status Code |
The HTTP Status Code to use when responding to the HTTP Request. See Section 10 of RFC 2616 for more information. |
## Relationships
| Name |
Description |
| failure |
FlowFiles will be routed to this Relationship if the Processor is unable to respond to the requestor. This may happen, for instance, if the connection times out or if NiFi is restarted before responding to the HTTP Request. |
| success |
FlowFiles will be routed to this Relationship after the response has been successfully sent to the requestor |
## See also
- [org.apache.nifi.processors.standard.HandleHttpRequest](/user-guide/data-integration/openflow/processors/handlehttprequest)
---
title: HazelcastMapCacheClient
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/hazelcastmapcacheclient.md
section: Loading & Unloading Data
---
# HazelcastMapCacheClient
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
An implementation of DistributedMapCacheClient that uses Hazelcast as the backing cache. This service relies on another controller service that manages the actual Hazelcast calls, set in the Hazelcast Cache Manager property.
## Tags
cache, hazelcast, map
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Hazelcast Cache Manager * |
hazelcast-cache-manager |
|
|
A Hazelcast Cache Manager which manages connections to Hazelcast and provides cache instances. |
| Hazelcast Cache Name * |
hazelcast-cache-name |
|
|
The name of a given cache. A Hazelcast cluster may handle multiple independent caches, each identified by a name. Clients using caches with the same name are working on the same data structure within Hazelcast. |
| Hazelcast Entry Lifetime * |
hazelcast-entry-ttl |
5 min |
|
Indicates how long the written entries should exist in Hazelcast. Setting it to '0 secs' means that the data will exist until its deletion or until the Hazelcast server is shut down. Using _EmbeddedHazelcastCacheManager_ as cache manager will not provide policies to limit the size of the cache. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: HikariCPConnectionPool
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/hikaricpconnectionpool.md
section: Loading & Unloading Data
---
# HikariCPConnectionPool
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides Database Connection Pooling Service based on HikariCP. Connections can be asked from pool and returned after usage.
## Tags
connection, database, dbcp, hikari, jdbc, pooling, store
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Database Connection URL * |
hikaricp-connection-url |
|
|
A database connection URL used to connect to a database. May contain database system name, host, port, database name and some parameters. The exact syntax of a database connection URL is specified by your DBMS. |
| Database Driver Class Name * |
hikaricp-driver-classname |
|
|
The fully-qualified class name of the JDBC driver. Example: com.mysql.jdbc.Driver |
| Database Driver Location(s) |
hikaricp-driver-locations |
|
|
Comma-separated list of files/folders and/or URLs containing the driver JAR and its dependencies (if any). For example '/var/tmp/mariadb-java-client-1.1.7.jar' |
| Kerberos User Service |
hikaricp-kerberos-user-service |
|
|
Specifies the Kerberos User Controller Service that should be used for authenticating with Kerberos |
| Max Connection Lifetime |
hikaricp-max-conn-lifetime |
-1 |
|
The maximum lifetime of a connection. After this time is exceeded the connection will fail the next activation, passivation or validation test. A value of zero or less means the connection has an infinite lifetime. |
| Max Total Connections * |
hikaricp-max-total-conns |
10 |
|
This property controls the maximum size that the pool is allowed to reach, including both idle and in-use connections. Basically this value will determine the maximum number of actual connections to the database backend. A reasonable value for this is best determined by your execution environment. When the pool reaches this size, and no idle connections are available, the service will block for up to connectionTimeout milliseconds before timing out. |
| Max Wait Time * |
hikaricp-max-wait-time |
500 millis |
|
The maximum amount of time that the pool will wait (when there are no available connections) for a connection to be returned before failing, or 0 <time units> to wait indefinitely. |
| Minimum Idle Connections * |
hikaricp-min-idle-conns |
10 |
|
This property controls the minimum number of idle connections that HikariCP tries to maintain in the pool. If the idle connections dip below this value and total connections in the pool are less than 'Max Total Connections', HikariCP will make a best effort to add additional connections quickly and efficiently. It is recommended that this property to be set equal to 'Max Total Connections'. |
| Password |
hikaricp-password |
|
|
The password for the database user |
| Database User |
hikaricp-username |
|
|
Database user name |
| Validation Query |
hikaricp-validation-query |
|
|
Validation Query used to validate connections before returning them. When connection is invalid, it gets dropped and new valid connection will be returned. NOTE: Using validation might have some performance penalty. |
## State management
This component does not store state.
## Restricted
## Restrictions
| Required Permission |
Explanation |
| reference remote resources |
Database Driver Location can reference resources over HTTP |
## System Resource Considerations
This component does not specify system resource considerations.
---
title: HttpRecordSink
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/httprecordsink.md
section: Loading & Unloading Data
---
# HttpRecordSink
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Format and send Records to a configured uri using HTTP post. The Record Writer formats the records which are sent as the body of the HTTP post request. JsonRecordSetWriter is often used with this processor because many HTTP posts require a JSON body.
## Tags
http, post, record, sink
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| API URL * |
API URL |
|
|
The URL which receives the HTTP requests. |
| Maximum Batch Size * |
Maximum Batch Size |
0 |
|
Specifies the maximum number of records to send in the body of each HTTP request. Zero means the batch size is not limited, and all records are sent together in a single HTTP request. |
| OAuth2 Access Token Provider |
OAuth2 Access Token Provider |
|
|
OAuth2 service that provides the access tokens for the HTTP requests. |
| Web Service Client Provider * |
Web Service Client Provider |
|
|
Controller service to provide the HTTP client for sending the HTTP requests. |
| Record Writer * |
record-sink-record-writer |
|
|
Specifies the Controller Service to use for writing out the records. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: IdentifyMimeType 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/identifymimetype.md
section: Loading & Unloading Data
---
# IdentifyMimeType 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Attempts to identify the MIME Type used for a FlowFile. If the MIME Type can be identified, an attribute with the name 'mime.type' is added with the value being the MIME Type. If the MIME Type cannot be determined, the value will be set to 'application/octet-stream'. In addition, the attribute 'mime.extension' will be set if a common file extension for the MIME Type is known. If the MIME Type detected is of type text/*, attempts to identify the charset used and an attribute with the name 'mime.charset' is added with the value being the charset.
## Tags
MIME, bzip2, compression, file, gzip, identify, mime.type, zip
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Custom MIME Configuration |
A URL or file path to a custom Tika Mime type configuration or the actual content of a custom Tika Mime type configuration. |
| config-strategy |
Select the loading strategy for MIME Type configuration to be used. |
| use-filename-in-detection |
If true will pass the filename to Tika to aid in detection. |
## Relationships
| Name |
Description |
| success |
All FlowFiles are routed to success |
## Writes attributes
| Name |
Description |
| mime.type |
This Processor sets the FlowFile's mime.type attribute to the detected MIME Type. If unable to detect the MIME Type, the attribute's value will be set to application/octet-stream |
| mime.extension |
This Processor sets the FlowFile's mime.extension attribute to the file extension associated with the detected MIME Type. If there is no correlated extension, the attribute's value will be empty |
| mime.charset |
This Processor sets the FlowFile's mime.charset attribute to the detected charset. If unable to detect the charset or the detected MIME type is not of type text/*, the attribute will not be set |
---
title: Install and configure the Openflow Connector for Oracle
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/oracle/setup-connector.md
section: Loading & Unloading Data
---
# Install and configure the %oracleofc%
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
The %oracleofc% is also subject to additional terms of service beyond the standard
connector terms of service. For more information, see the
[Openflow Connector for Oracle Addendum](https://www.snowflake.cn/en/legal/optional-offerings/offering-specific-terms/openflow-oracle-terms/).
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
- [About Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/about)
- [Openflow Connector for Oracle: Enable and manage commercial terms](/user-guide/data-integration/openflow/connectors/oracle/manage-commercial-terms)
- [Openflow Connector for Oracle: Set up Snowflake](/user-guide/data-integration/openflow/connectors/oracle/setup-snowflake)
- [Openflow Connector for Oracle: Set up incremental replication without snapshots](/user-guide/data-integration/openflow/connectors/oracle/incremental-replication)
This topic describes the steps to install and configure the %oracleofc% connector.
As a data engineer, perform the following tasks to install and configure the connector:
## Install the connector
To install the connector, do the following as a data engineer:
1. Navigate to the Openflow overview page. In the **Featured connectors** section, select **View more connectors**.
2. On the Openflow connectors page, find the connector and select **Install**.
3. In the **Select runtime** dialog, select your runtime from the **Available runtimes** drop-down list and click **Install**.
Before you install the connector, ensure that you have created a database and schema in Snowflake for the connector to store ingested data.
4. Authenticate to the deployment with your Snowflake account credentials and select **Allow** when prompted to allow the runtime application to access your Snowflake account. The connector installation process takes a few minutes to complete.
5. Authenticate to the runtime with your Snowflake account credentials.
The Openflow canvas appears with the connector process group added to it.
## Runtime sizing
The runtime size determines the CPU, memory, and disk available to the
connector. The available sizes are Small, Medium, and Large. Choose the size
when you create the runtime: you can't change the size of an existing runtime
in place.
Size the runtime based on the sustained workload it needs to handle across all
connectors running on it. Sustained means typical steady-state throughput, not
peak. Peak load can temporarily increase connector queues and end-to-end
replication latency; the workload catches up when the load drops back to the
steady-state level.
The following ranges are starting points based on internal benchmarks and
production customer data. They aren't service guarantees. Your fit depends on
row size, event distribution, schema width, and source burstiness. Start at
the lower bound, measure runtime CPU, memory, queue depth, and end-to-end
replication latency in production, then increase from there.
- Light workload (aggregate sustained throughput below approximately 1,000
events per second, fewer than approximately 100 actively changing tables): a
Small runtime can host a single low-volume connector. Pack additional
connectors on Small only when each source is genuinely light.
- Moderate workload (approximately 1,000 to 5,000 events per second, hundreds
of actively changing tables): a Medium runtime can typically host 5 to 8
connectors.
- Heavy workload (approximately 5,000 to 15,000 events per second, hundreds to
low thousands of actively changing tables): a Large runtime can typically host
15 or more connectors. If you want a smaller blast radius, split
across two Medium runtimes instead.
## Running multiple connectors on one runtime
You can run multiple CDC connector instances on a single runtime. This is
useful for replicating many small databases, for example a multi-tenant SaaS
with one database per tenant, or a fleet of operational databases per business
unit or region.
When you run multiple CDC connector instances of the same type on one runtime,
keep their shared Source and Destination parameter contexts intact and
override only the per-connector values in each Ingestion context. For the
recommended process, see [](#run-multiple-connectors-on-one-runtime).
Run a connector on a dedicated runtime, not packed with others, when any of the
following applies:
- A single source sustains more than approximately 15,000 events per second.
- You need sub-1-minute end-to-end replication latency under load.
- You can't tolerate noisy-neighbor effects from other sources sharing the
runtime.
Each replicated table can consume two Snowpipe Streaming pipes: one for snapshot
replication and one for incremental replication. As you pack more tables onto a
runtime, check your account's [Snowpipe Streaming pipe limit](/user-guide/snowpipe-streaming/snowpipe-streaming-high-performance-limitations#pipe-limits)
and raise it before you approach the cap.
## Resize a runtime
Runtime size is fixed at creation, so to change size you run the connector on
a different runtime. You have two options depending on whether you want to
preserve the current replication progress.
If you don't need to keep the progress of the current connector, the simplest
path is to create a new runtime at the size you need and install a new connector
instance on it. The new connector starts from scratch: it snapshots all
configured tables and then captures ongoing changes from that point. The
replication progress of the existing connector is discarded.
To keep the progress of the current connector, for example to avoid
re-snapshotting tables that took a long time to snapshot initially, migrate the
connector to the new runtime. This reuses the existing destination tables and
resumes incremental replication from where it left off.
For migration instructions, see [Reinstall the connector](#label-oracle-reinstall-connector).
## Configure the connector
To configure the connector, do the following as a data engineer:
1. Right-click on the added runtime and select **Parameters**.
2. Populate the required parameter values.
For more information on the required parameter values, see the following sections:
- [](#label-oracle-snowflake-destination-parameters): Used to establish connection with Snowflake.
- [](#label-oracle-ingestion-parameters): Used to specify the tables to replicate.
- [](#label-oracle-source-parameters): Used to define the configuration of data downloaded from Oracle.
To run multiple CDC connector instances on one runtime, see [](/user-guide/data-integration/openflow/connectors/cdc-runtime-sizing#run-multiple-connectors-on-one-runtime).
### Snowflake Destination Parameters
| Parameter |
Description |
Required |
| Destination Database |
The database where data is persisted. It must already exist in Snowflake and the connector's
role must have `USAGE` and `CREATE SCHEMA` on it. The name is case-sensitive. For unquoted
identifiers, provide the name in uppercase.
|
Yes |
| Destination Schema Pattern |
A pattern for the names of destination schemas where data is persisted. The connector creates the schemas if
they don't exist.
You can customize the pattern per ingested table using these optional variables:
- `${source.database.name}`: a source table's database.
- `${source.schema.name}`: a source table's schema.
- `${source.table.name}`: a source table's name.
For example, for a table with the qualified name `source_db.tenant_a.data`,
the pattern `prefix_${source.database.name}_${source.schema.name}` evaluates to
`prefix_source_db_tenant_a`.
To ingest all tables into a single schema, provide a schema name without any variables,
like `destination_schema`.
Don't change this setting after the connector has begun ingesting data. Changing
this setting after ingestion has begun breaks the existing ingestion. If you must
change this setting, create a new connector instance.
|
Yes |
| Snowflake Authentication Strategy |
When using:
- **Snowflake Openflow Deployment** or **BYOC**: Use SNOWFLAKE_MANAGED.
This token is managed automatically by Snowflake.
BYOC deployments must have previously configured
[execute-as roles](#label-deployment-byoc-setup-runtime-role) to use SNOWFLAKE_MANAGED.
- **BYOC**: Alternatively, BYOC can use KEY_PAIR as the value for the authentication strategy.
|
Yes |
| Snowflake Account Identifier |
When using:
- **SNOWFLAKE_MANAGED** Authentication Strategy: Must be blank.
- **KEY_PAIR**: Snowflake account name formatted as [organization-name]-[account-name].
|
Yes |
| Snowflake Connection Strategy |
When using KEY_PAIR, specify the strategy for connecting to Snowflake:
- **STANDARD** (default): Connect using standard public routing to Snowflake services.
- **PRIVATE_CONNECTIVITY**: Connect using private addresses associated with the supporting cloud platform such as AWS PrivateLink.
|
Required for BYOC with KEY_PAIR only, otherwise ignored. |
| Snowflake Private Key |
When using:
- **SNOWFLAKE_MANAGED** Authentication Strategy: Must be blank.
- **KEY_PAIR**: Must be the RSA private key used for authentication.
-
The RSA key must be formatted according to PKCS8 standards and have standard PEM headers and footers.
Note that either a Snowflake Private Key File or a Snowflake Private Key must be defined.
|
No |
| Snowflake Private Key File |
When using:
- **SNOWFLAKE_MANAGED** Authentication Strategy: The private key file must be blank.
- **KEY_PAIR**: Upload the file that contains the RSA private key used for authentication to Snowflake,
formatted according to PKCS8 standards and including standard PEM headers and footers.
The header line begins with `-----BEGIN PRIVATE`.
To upload the private key file, select the **Reference asset** checkbox.
|
No |
| Snowflake Private Key Password |
When using:
- **SNOWFLAKE_MANAGED** Authentication Strategy: Must be blank.
- **KEY_PAIR**: Provide the password associated with the Snowflake Private Key File.
|
No |
| Snowflake Role |
When using:
- **SNOWFLAKE_MANAGED** Authentication Strategy: Use the runtime's execute-as role (or a child role granted to it).
You can find your execute-as role in the Openflow UI by navigating to **View Details** for your runtime.
- **KEY_PAIR**: Use a valid role configured for your service user.
|
Yes |
| Snowflake Username |
When using:
- **SNOWFLAKE_MANAGED** Authentication Strategy: Must be blank.
- **KEY_PAIR**: Provide the username used to connect to the Snowflake instance.
|
Yes |
| Oversized Value Strategy |
Determines how the connector handles values that exceed its internal size limits (16 MB) during replication.
Possible values are:
- **Fail Table** (default): The table is marked as permanently failed, and replication stops for that table.
- **Set Null**: The value is replaced with `NULL` in the destination table.
Use this to prevent table failures when it is acceptable to lose data in tables beyond the oversized value.
|
No |
| Error Handling Strategy |
Determines how the connector handles invalid rows that Snowflake rejects during ingestion.
Possible values are:
- **Fail Table** (default): The table is marked as failed on the first invalid row, and replication stops for that table.
- **Log Errors and Continue** : The connector keeps replicating the valid rows and records each rejected row in the table's error table.
|
No |
| Table Storage Format |
Standard Snowflake tables or Iceberg tables. Defaults to **STANDARD**. Don't change after the connector starts. |
Yes |
| Iceberg Version |
The Iceberg table version, 2 or 3 (default 3). Ignored unless Table Storage Format is **ICEBERG**. Don't change this value after ingestion begins. |
No |
| Snowflake Warehouse |
Snowflake warehouse used to run merge queries. Start with `XSMALL`; for many tables, a
multi-cluster warehouse scales better than a larger size.
|
Yes |
### Oracle Ingestion Parameters
| Parameter |
Description |
| Included Table Names |
Comma-separated list of fully-qualified table paths.
Tables must be specified using fully-qualified database, schema, and table name format: DATABASE_NAME.SCHEMA_NAME.TABLE_NAME.
For example: `MYPDB.SALES.CUSTOMERS, MYPDB.SALES.ORDERS`
|
| Included Table Regex |
A regular expression to match table paths for automatic inclusion of existing and new tables.
The regex pattern must match the three-part naming convention: DATABASE_NAME.SCHEMA_NAME.TABLE_NAME.
For example: `MYPDB\.SALES\..*` to match all tables in the SALES schema within the MYPDB database.
|
| Column Filter JSON |
Optional. A JSON array of filter objects specifying which columns to include or exclude per table.
For syntax details and examples, see [Replicate a subset of columns in a table](#replicate-a-subset-of-columns-in-a-table).
|
| Table Key Configuration Service |
Optional. A `MultiDatabaseJsonTableKeyConfigService` controller service that
supplies a user-declared logical key for one or more tables. The service exposes a
**Table Key Configuration JSON** property where you define the key mappings.
When configured, the logical key takes the highest priority and overrides any
primary key, unique constraint, or unique index that the connector would otherwise
auto-detect.
For more information on when to use this and how to configure it, see
[](#label-oracle-logical-key).
|
| Merge Task Schedule CRON |
CRON expression defining periods when merge operations from Journal to Destination Table will be
triggered. Set it to `* * * * * ?` if you want continuous merges, or configure a time schedule to
limit warehouse run time. The connector evaluates the schedule in the UTC time zone.
For example:
- The string `* 0 * * * ?` indicates that you want to schedule merges at the full hour for one minute.
- The string `* 20 14 ? * MON-FRI` indicates that you want to schedule merges at 2:20 PM every
Monday through Friday.
For additional information and examples, see the cron triggers tutorial in the Quartz Documentation (https://www.quartz-scheduler.org/documentation/quartz-2.5.x/tutorials/crontrigger.html).
|
| Object Identifier Resolution |
Specifies how source object identifiers such as schemas, tables, and
column names are stored and queried in Snowflake. This setting
determines if you must use double quotes in SQL queries.
Option 1: Default, case-insensitive (recommended).
- **Transformation**: All identifiers are converted to uppercase. For
example, `My_Table` becomes `MY_TABLE`.
- **Queries**: SQL queries are case-insensitive and don't require SQL
double quotes.
For example `SELECT * FROM my_table;` returns the same results as `SELECT * FROM MY_TABLE;`.
Snowflake recommends using this option if database objects aren't expected to have mixed case names.
Option 2: Case-sensitive.
- **Transformation**: Case is preserved.
For example, `My_Table` remains `My_Table`.
- **Queries**: SQL queries must use double quotes to match the exact
case for database objects.
For example, `SELECT * FROM "My_Table";`.
Do not change this setting after connector ingestion has begun.
Changing this setting after ingestion has begun breaks the existing ingestion.
If you must change this setting, create a new connector instance.
|
| Snapshot Fetching Strategy |
Determines the snapshot load fetching strategy:
- **CONCURRENT_BY_ROWID** (default): Splits tables into chunks bound by ranges of physical row ids, and retrieves each chunk in parallel.
This strategy isn't currently supported when the connector reads from a read-only database like Active Data Guard physical standby.
- **SEQUENTIAL_BY_PRIMARY_KEY**: Uses fixed-size batches retrieved sequentially by the table's replication key (primary key, unique constraint,
unique index, or logical key). Despite the name, this strategy uses whatever key the connector resolved for the table, not specifically the primary key.
|
| Concurrent Snapshot Queries |
Maximum number of concurrent queries to the source database to run in the Snapshot flow. Increasing this can speed up snapshotting large numbers of tables, but will also increase the load on the source database. |
### Oracle Source Parameters
| Parameter |
Description |
Required |
| Oracle Connection URL |
JDBC URL of the database connection to the DB.
The URL must specify the target container (PDB or CDB) that contains the data to be replicated.
For example `jdbc:oracle:thin:@:/YOUR_DB_NAME` where YOUR_DB_NAME is the name of your PDB or CDB.
When SSL is enabled, use the TCPS protocol, for example
`jdbc:oracle:thin:@tcps://:/YOUR_DB_NAME`.
The connector works within a single database/container. Ensure the JDBC URL points directly to the container that holds the tables to be replicated.
|
Yes |
| Oracle Username |
Username of the connect user that has access to the XStream Server. |
Yes |
| Oracle Password |
Password of the connect user that has access to the XStream Server. |
Yes |
| Oracle SSL Mode |
Controls SSL encryption for connections to the Oracle database.
- **DISABLED**, which is the default: Connect without SSL.
- **VERIFY_CA**: Connect with SSL. Verifies that a trusted Certificate Authority
issued the server certificate.
- **VERIFY_IDENTITY**: Connect with SSL. Verifies the CA certificate and that the
server hostname matches the certificate's subject.
When set to VERIFY_CA or VERIFY_IDENTITY, you must also provide the Oracle Wallet Filename parameter.
|
Yes |
| Oracle Wallet Filename |
Upload the file that contains the Oracle auto-login wallet file (`cwallet.sso`).
The wallet must contain the trusted server certificate for SSL connections.
For information about creating the wallet, see [](#label-configure-ssl-connections).
|
Required when SSL Mode is not DISABLED |
| Oracle Database Processor Multiplier |
Core Processor Licensing Factor as described in Oracle Processor Core Factor Table (https://www.oracle.com/contracts/docs/processor-core-factor-table-070634.pdf). |
Required for Embedded License only |
| Oracle Database Processor Cores |
The number of processor cores in your Oracle database. |
Required for Embedded License only |
| XStream Billing Acknowledgement |
A confirmation of the licensing agreement. |
Required for Embedded License only |
| XStream Out Server Name |
The name of the XStream Server that must already exist in Oracle. |
Yes |
| XStream Out Server URL |
JDBC URL of the database connection for XStream, which must use the OCI driver.
For example `jdbc:oracle:oci:@:/SID`.
When SSL is enabled, use the TCPS protocol, for example
`jdbc:oracle:oci:@tcps://:/SID`.
When SSL Mode is enabled, the connector automatically adds `SSL_SERVER_DN_MATCH`
and `MY_WALLET_DIRECTORY` to the XStream URL. You don't need to include these manually.
|
Yes |
## Restart table replication
A table in FAILED state — for example, due to a missing primary key or unsupported schema change — does not restart automatically. If a table enters a FAILED state or you need to restart replication from scratch, use the following procedure to remove and re-add the table to replication.
If the failure was caused by an issue in the source table such as a missing primary key, resolve that issue in the source database before continuing.
1. Remove the table from replication, using one of the following methods:
- Add the table to the **Re-snapshot Table Exclusions** parameter to temporarily exclude it from replication. This is convenient when the table is matched by an **Included Table Regex** that you don't want to change.
- In the Ingestion Parameters context, either remove the table from **Included Table Names** or modify the **Included Table Regex** so the table is no longer matched.
2. Verify the table has been removed:
1. In the Openflow runtime canvas, right-click a processor group and choose **Controller Services**.
2. In the table listing controller services, locate the **Table State Store** row, click the three vertical dots on the right side of the row, then choose **View State**.
You must wait until the table's state is fully removed from this list before proceeding. Do not continue until this configuration change has completed.
3. Clean up the destination: Once the table's state shows as fully removed, manually [DROP](/sql-reference/sql/drop-table) the destination table in Snowflake. Note that the connector will not overwrite an existing destination table during the snapshot phase; if the table still exists, replication will fail again. Optionally, the journal table and stream can also be removed if they are no longer needed.
4. Re-add the table by reversing the change you made in the first step: either remove the table from **Re-snapshot Table Exclusions**, or add it back to **Included Table Names** or **Included Table Regex**. The connector then re-snapshots the table.
5. Verify the restart: Check the **Table State Store** using the instructions given previously. The state of the table should appear with the status NEW, then transition to SNAPSHOT_REPLICATION, and finally INCREMENTAL_REPLICATION.
## Replicate a subset of columns in a table
The connector can filter the data replicated per table to a subset of configured columns.
Primary key columns are always included regardless of exclusions.
To apply column filters, set the **Column Filter JSON** parameter in the Ingestion Parameters context
to a JSON array of filter objects, one per table you want to filter.
Columns can be included or excluded by name or by regular expression pattern. You can apply a single condition per table,
or combine multiple conditions, with exclusions always taking precedence over inclusions.
## Syntax
Each object in the array identifies a table and specifies which columns to include or exclude.
Because this connector uses three-part fully qualified names (database, schema, and table), each object
can include a `database` or `databasePattern` field in addition to the schema and table fields.
```javascript
[
{
"database": "" | "databasePattern": "",
"schema": "" | "schemaPattern": "",
"table": "" | "tablePattern": "",
"included": ["", ""],
"excluded": ["", ""],
"includedPattern": "",
"excludedPattern": ""
}
]
```
The following rules apply:
- Use `database`, `schema`, and `table` for exact name matching, or `databasePattern`,
`schemaPattern`, and `tablePattern` for regex matching. You can't use both a field and its
pattern variant in the same object (for example, `schema` and `schemaPattern` can't both appear).
- At least one of `included`, `excluded`, `includedPattern`, or `excludedPattern` must be provided.
- When both included and excluded filters are specified, exclusions take precedence.
- When multiple filters match the same table, the last matching filter is used, with exact matches
taking precedence over pattern-based filters.
- The value can be an array of objects to apply different filters to different tables.
## Examples
Include specific columns by name:
```javascript
[
{
"database": "my_db",
"schema": "dbo",
"table": "orders",
"included": ["account_id", "status", "created_at"]
}
]
```
Exclude specific columns by name:
```javascript
[
{
"database": "my_db",
"schema": "dbo",
"table": "orders",
"excluded": ["internal_note", "debug_flag"]
}
]
```
Combine an include pattern with a specific exclusion (for example, include all email columns except `admin_email`):
```javascript
[
{
"database": "my_db",
"schema": "dbo",
"table": "contacts",
"includedPattern": ".*_email",
"excluded": ["admin_email"]
}
]
```
Mix a database pattern with an exact schema and table name to apply a filter across databases:
```javascript
[
{
"databasePattern": "prod_.*",
"schema": "dbo",
"table": "customers",
"excluded": ["internal_note"]
}
]
```
Pass multiple filter objects to apply different rules to different tables:
```javascript
[
{"database": "my_db", "schema": "dbo", "table": "orders", "included": ["account_id", "status"]},
{"database": "my_db", "schema": "dbo", "table": "customers", "excludedPattern": ".*_internal"}
]
```
### Including and excluding the same column
Removing a column from a table's replicated set (by excluding it or by removing
it from the included list) has the same effect on the destination as dropping
the column at the source: the connector soft-deletes the column on the
destination by renaming it with a suffix (by default, `__SNOWFLAKE_DELETED`).
If you then add the column back to the replicated set and later remove it a
second time, replication for the affected table fails because the soft-deleted
column name is already taken. To recover, restart replication for the affected
table.
## Specify a logical key for a table
The connector requires a replication key for every table it replicates. By default, the
connector picks the replication key automatically, in this order: a primary key, then a
qualifying unique constraint, then a qualifying unique index. For the full selection
rules, see [](#label-oracle-replication-key-selection).
A *logical key* is a user-declared replacement for the auto-detected key. Configure a
logical key when:
- A table has no primary key and no qualifying unique constraint or unique index, but
one or more columns are unique in the data.
- A specific column or set of columns should be used as the replication key, regardless
of what the connector would auto-detect (for example, to override a synthetic primary
key).
A logical key takes the highest priority. When the connector finds a logical key for a
table, it uses that key and ignores any primary key, unique constraint, or unique index
on the table.
### JSON syntax
The **Table Key Configuration JSON** value is a JSON array. Each entry maps one
table to its logical key columns:
```json
[
{
"database": "",
"schema": "",
"table": "",
"logicalKey": ["", ""]
}
]
```
The fields are:
| Field |
Description |
| `database` |
Required. The exact source database (PDB or CDB) name, matching the database in the table's three-part fully qualified name. |
| `schema` |
Required. The exact source schema name. |
| `table` |
Required. The exact source table name. |
| `logicalKey` |
Required. A non-empty array of source column names that uniquely identify rows in the table. |
The following rules apply:
- `database`, `schema`, and `table` matching is **case-sensitive**. Oracle stores
identifiers in uppercase by default, so use uppercase names unless the identifiers
were created with double-quoted mixed-case or lowercase names.
- `logicalKey` column matching is case-insensitive. The connector matches column
names against the source table schema regardless of case.
- An entry whose `database`, `schema`, and `table` don't match any replicated
table is silently ignored.
### Logical key configuration examples
A single-column logical key on a table without a primary key:
```json
[
{
"database": "MYPDB",
"schema": "SALES",
"table": "AUDIT_LOG",
"logicalKey": ["EVENT_ID"]
}
]
```
A composite logical key:
```json
[
{
"database": "MYPDB",
"schema": "SALES",
"table": "ORDER_LINES",
"logicalKey": ["ORDER_ID", "LINE_ITEM_ID"]
}
]
```
Logical keys for several tables in one JSON value:
```json
[
{
"database": "MYPDB",
"schema": "SALES",
"table": "AUDIT_LOG",
"logicalKey": ["EVENT_ID"]
},
{
"database": "MYPDB",
"schema": "SALES",
"table": "ORDER_LINES",
"logicalKey": ["ORDER_ID", "LINE_ITEM_ID"]
}
]
```
### Restrictions
The connector rejects the configuration when any of the following is true:
- `logicalKey` is missing, empty, or not an array.
- `logicalKey` contains duplicate column names (compared case-insensitively).
- `logicalKey` contains the pseudo-column `ROWID`. `ROWID` isn't a reliable
replication key because it can change when a row is moved (for example, after a table
rebuild or partition operation).
- `logicalKey` contains a column name that doesn't exist in the source table.
When the configuration is rejected, the connector either fails to enable the
controller service (for structural issues detected at enablement time) or holds the
table in the `NEW` state (for issues detected when the table is initialized). After
you fix the configuration, replication for the table resumes without resetting state.
### Warnings logged for risky configurations
The connector accepts the following configurations but logs a warning at table
initialization. Verify the data carefully or arrange a periodic full reload to correct
drift.
When choosing logical-key columns, prefer columns with high cardinality and, where
possible, monotonically increasing values. Low-cardinality or non-monotonic keys can
degrade snapshot performance if you use the `SEQUENTIAL_BY_PRIMARY_KEY` strategy, which
orders rows by the replication key.
- A logical-key column is a large-object type (`BLOB`, `CLOB`, `NCLOB`, `LONG`,
`LONG RAW`). Using large objects as keys severely degrades MERGE performance.
- A logical-key column is a floating-point type (`FLOAT`, `DOUBLE`, `REAL`,
`BINARY_FLOAT`, `BINARY_DOUBLE`). Floating-point comparisons can produce
inconsistent results because of precision differences.
- The composite logical key has more than five columns. Long composite keys often
indicate a design issue and might degrade MERGE performance.
- The logical key replaces an existing primary key on the table.
### Limitation: Changes to a logical-key value
When a source `UPDATE` changes the value of a logical-key column, the connector
does **not** soft-delete the row keyed by the old value before inserting the row
keyed by the new value. The destination table ends up with two active rows for
what's a single row in the source: the original row, still active under its old
key value, and a new row under the new key value.
This differs from how the connector handles a primary-key value change on tables
that don't use a logical key. For more information on that behavior, see
[](#label-oracle-replication-key-value-change).
To avoid this limitation, choose logical-key columns whose values don't change in
the source. If logical-key values do change, periodically run a full reload for
the affected tables to reconcile the destination with the source.
### Schema changes that affect a logical key
Logical keys reference column names. The connector doesn't follow renames or drops of
those columns:
- If a logical-key column is dropped on the source, replication for the affected table
fails. The table is marked `FAILED`. For recovery steps, see
[](#label-oracle-logical-key-invalidated) in the troubleshooting topic.
- If a logical-key column is renamed on the source, the configuration still references
the old name and replication fails. Update the JSON to use the new name and restart
table replication.
## Configure scheduling of merge tasks
The connector uses a warehouse to merge change data capture (CDC) data into destination tables.
The processor named Merge Journal to Destination triggers this operation. When there are no new
changes, or when no new FlowFiles are waiting in the Merge Journal to Destination queue, no merge
is triggered and the warehouse is available for auto-suspension.
To limit warehouse cost and restrict merges to scheduled times, use the CRON expression in the
Merge Task Schedule CRON parameter. It throttles the FlowFiles that reach the Merge Journal to
Destination processor, so merges are triggered only during the specified period. The connector
evaluates the schedule in the UTC time zone.
For additional information and examples, see the cron triggers tutorial in the Quartz Documentation (https://www.quartz-scheduler.org/documentation/quartz-2.5.x/tutorials/crontrigger.html).
## Run the flow
1. Right-click on the plane and select **Enable all Controller Services**.
2. Right-click on the imported process group and select **Start**. The connector starts the data ingestion.
## Next steps
- (Optional) [Set up incremental replication without snapshots](/user-guide/data-integration/openflow/connectors/oracle/incremental-replication).
- [Monitor the flow](/user-guide/data-integration/openflow/monitor).
---
title: InvokeHTTP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/invokehttp.md
section: Loading & Unloading Data
---
# InvokeHTTP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
An HTTP client processor which can interact with a configurable HTTP Endpoint. The destination URL and HTTP Method are configurable. When the HTTP Method is PUT, POST or PATCH, the FlowFile contents are included as the body of the request and FlowFile attributes are converted to HTTP headers, optionally, based on configuration properties.
## Tags
client, http, https, rest
## Input Requirement
ALLOWED
## Supports Sensitive Dynamic Properties
true
## Properties
| Property |
Description |
| Connection Timeout |
Maximum time to wait for initial socket connection to the HTTP URL. |
| HTTP Method |
HTTP request method (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS). Arbitrary methods are also supported. Methods other than POST, PUT and PATCH will be sent without a message body. |
| HTTP URL |
HTTP remote URL including a scheme of http or https, as well as a hostname or IP address with optional port and path elements. Any encoding of the URL must be done by the user. |
| HTTP/2 Disabled |
Disable negotiation of HTTP/2 protocol. HTTP/2 requires TLS. HTTP/1.1 protocol supported is required when HTTP/2 is disabled. |
| OAuth2 Access Token Refresh Strategy |
Specifies which strategy should be used to refresh the OAuth2 Access Token. |
| Request Body Enabled |
Enable sending HTTP request body for PATCH, POST, or PUT methods. |
| Request Chunked Transfer-Encoding Enabled |
Enable sending HTTP requests with the Transfer-Encoding Header set to chunked, and disable sending the Content-Length Header. Transfer-Encoding applies to the body in HTTP/1.1 requests as described in RFC 7230 Section 3.3.1 |
| Request Content-Encoding |
HTTP Content-Encoding applied to request body during transmission. The receiving server must support the selected encoding to avoid request failures. |
| Request Content-Type |
HTTP Content-Type Header applied to when sending an HTTP request body for PATCH, POST, or PUT methods. The Content-Type defaults to application/octet-stream when not configured. |
| Request Date Header Enabled |
Enable sending HTTP Date Header on HTTP requests as described in RFC 7231 Section 7.1.1.2. |
| Request Digest Authentication Enabled |
Enable Digest Authentication on HTTP requests with Username and Password credentials as described in RFC 7616. |
| Request Failure Penalization Enabled |
Enable penalization of request FlowFiles when receiving HTTP response with a status code between 400 and 499. |
| Request Header Attributes Pattern |
Regular expression that defines which FlowFile attributes to send as HTTP headers in the request. If not defined, no attributes are sent as headers. Dynamic properties will be always be sent as headers. The dynamic property name will be the header key and the dynamic property value, interpreted as Expression Language, will be the header value. Attributes and their values are limited to ASCII characters due to the requirement of the HTTP protocol. |
| Request Multipart Form-Data Filename Enabled |
Enable sending the FlowFile filename attribute as the filename parameter in the Content-Disposition Header for multipart/form-data HTTP requests. |
| Request Multipart Form-Data Name |
Enable sending HTTP request body formatted using multipart/form-data and using the form name configured. |
| Request OAuth2 Access Token Provider |
Enables managed retrieval of OAuth2 Bearer Token applied to HTTP requests using the Authorization Header. |
| Request Password |
The password provided for authentication of HTTP requests. Encoded using Base64 for HTTP Basic Authentication as described in RFC 7617. |
| Request User-Agent |
HTTP User-Agent Header applied to requests. RFC 7231 Section 5.5.3 describes recommend formatting. |
| Request Username |
The username provided for authentication of HTTP requests. Encoded using Base64 for HTTP Basic Authentication as described in RFC 7617. |
| Response Body Attribute Name |
FlowFile attribute name used to write an HTTP response body for FlowFiles transferred to the Original relationship. |
| Response Body Attribute Size |
Maximum size in bytes applied when writing an HTTP response body to a FlowFile attribute. Attributes exceeding the maximum will be truncated. |
| Response Body Ignored |
Disable writing HTTP response FlowFiles to Response relationship |
| Response Cache Enabled |
Enable HTTP response caching described in RFC 7234. Caching responses considers ETag and other headers. |
| Response Cache Size |
Maximum size of HTTP response cache in bytes. Caching responses considers ETag and other headers. |
| Response Cookie Strategy |
Strategy for accepting and persisting HTTP cookies. Accepting cookies enables persistence across multiple requests. |
| Response FlowFile Naming Strategy |
Determines the strategy used for setting the filename attribute of FlowFiles transferred to the Response relationship. |
| Response Generation Required |
Enable generation and transfer of a FlowFile to the Response relationship regardless of HTTP response status code received. |
| Response Header Request Attributes Enabled |
Enable adding HTTP response headers as attributes to FlowFiles transferred to the Original, Retry or No Retry relationships. |
| Response Header Request Attributes Prefix |
Prefix to HTTP response headers when included as attributes to FlowFiles transferred to the Original, Retry or No Retry relationships. It is recommended to end with a separator character like '.' or '-'. |
| Response Redirects Enabled |
Enable following HTTP redirects sent with HTTP 300 series responses as described in RFC 7231 Section 6.4. |
| SSL Context Service |
SSL Context Service provides trusted certificates and client certificates for TLS communication. |
| Socket Idle Connections |
Maximum number of idle connections to the HTTP URL. |
| Socket Idle Timeout |
Maximum time to wait before closing idle connections to the HTTP URL. |
| Socket Read Timeout |
Maximum time to wait for receiving responses from a socket connection to the HTTP URL. |
| Socket Write Timeout |
Maximum time to wait for write operations while sending requests from a socket connection to the HTTP URL. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. In case of SOCKS, it is not guaranteed that the selected SOCKS Version will be used by the processor. |
## Relationships
| Name |
Description |
| Failure |
Request FlowFiles transferred when receiving socket communication errors. |
| No Retry |
Request FlowFiles transferred when receiving HTTP responses with a status code between 400 an 499. |
| Original |
Request FlowFiles transferred when receiving HTTP responses with a status code between 200 and 299. |
| Response |
Response FlowFiles transferred when receiving HTTP responses with a status code between 200 and 299. Enabling [Response Generation Required] changes routing behavior, sending unsuccessful responses to their corresponding relationships and also sending FlowFiles to the Response relationship as well, regardless of status code received. |
| Retry |
Request FlowFiles transferred when receiving HTTP responses with a status code between 500 and 599. |
## Writes attributes
| Name |
Description |
| invokehttp.status.code |
The status code that is returned |
| invokehttp.status.message |
The status message that is returned |
| invokehttp.response.body |
In the instance where the status code received is not a success (2xx) then the response body will be put to the 'invokehttp.response.body' attribute of the request FlowFile. |
| invokehttp.request.url |
The original request URL |
| invokehttp.request.duration |
Duration (in milliseconds) of the HTTP call to the external endpoint |
| invokehttp.response.url |
The URL that was ultimately requested after any redirects were followed |
| invokehttp.tx.id |
The transaction ID that is returned after reading the response |
| invokehttp.remote.dn |
The DN of the remote server |
| invokehttp.java.exception.class |
The Java exception class raised when the processor fails |
| invokehttp.java.exception.message |
The Java exception message raised when the processor fails |
| user-defined |
If the 'Put Response Body In Attribute' property is set then whatever it is set to will become the attribute key and the value would be the body of the HTTP response. |
---
title: InvokeScriptedProcessor 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/invokescriptedprocessor.md
section: Loading & Unloading Data
---
# InvokeScriptedProcessor 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-scripting-nar
## Description
Experimental - Invokes a script engine for a Processor defined in the given script. The script must define a valid class that implements the Processor interface, and it must set a variable 'processor' to an instance of the class. Processor methods such as onTrigger() will be delegated to the scripted Processor instance. Also any Relationships or PropertyDescriptors defined by the scripted processor will be added to the configuration dialog. The scripted processor can implement public void setLogger(ComponentLog logger) to get access to the parent logger, as well as public void onScheduled(ProcessContext context) and public void onStopped(ProcessContext context) methods to be invoked when the parent InvokeScriptedProcessor is scheduled or stopped, respectively. NOTE: The script will be loaded when the processor is populated with property values, see the Restrictions section for more security implications. Experimental: Impact of sustained usage not yet verified.
## Tags
groovy, invoke, script
## Input Requirement
## Supports Sensitive Dynamic Properties
true
## Properties
| Property |
Description |
| Module Directory |
Comma-separated list of paths to files and/or directories which contain modules required by the script. |
| Script Body |
Body of script to execute. Only one of Script File or Script Body may be used |
| Script Engine |
Language Engine for executing scripts |
| Script File |
Path to script file to execute. Only one of Script File or Script Body may be used |
## State management
| Scopes |
Description |
| LOCAL |
Scripts can store and retrieve state using the State Management APIs. Consult the State Manager section of the Developer's Guide for more details. |
| CLUSTER |
Scripts can store and retrieve state using the State Management APIs. Consult the State Manager section of the Developer's Guide for more details. |
## Restrictions
| Required Permission |
Explanation |
| execute code |
Provides operator the ability to execute arbitrary code assuming all permissions that NiFi has. |
## See also
- [org.apache.nifi.processors.script.ExecuteScript](/user-guide/data-integration/openflow/processors/executescript)
---
title: IPLookupService
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/iplookupservice.md
section: Loading & Unloading Data
---
# IPLookupService
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
A lookup service that provides several types of enrichment information for IP addresses. The service is configured by providing a MaxMind Database file and specifying which types of enrichment should be provided for an IP Address or Hostname. Each type of enrichment is a separate lookup, so configuring the service to provide all of the available enrichment data may be slower than returning only a portion of the available enrichments. In order to use this service, a lookup must be performed using key of 'ip' and a value that is a valid IP address or hostname. View the Usage of this component and choose to view Additional Details for more information, such as the Schema that pertains to the information that is returned.
## Tags
anonymous, cellular, domain, enrich, geo, ip, ipgeo, isp, lookup, maxmind, tor
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| MaxMind Database File * |
database-file |
|
|
Path to Maxmind IP Enrichment Database File |
| Lookup Anonymous IP Information * |
lookup-anonymous-ip |
false |
- true
- false
|
Specifies whether or not information about whether or not the IP address belongs to an anonymous network should be returned. |
| Lookup Geo Enrichment * |
lookup-city |
true |
- true
- false
|
Specifies whether or not information about the geographic information, such as cities, corresponding to the IP address should be returned |
| Lookup Connection Type * |
lookup-connection-type |
false |
- true
- false
|
Specifies whether or not information about the Connection Type corresponding to the IP address should be returned. If true, the lookup will contain a 'connectionType' field that (if populated) will contain a value of 'Dialup', 'Cable/DSL', 'Corporate', or 'Cellular' |
| Lookup Domain Name * |
lookup-domain |
false |
- true
- false
|
Specifies whether or not information about the Domain Name corresponding to the IP address should be returned. If true, the lookup will contain second-level domain information, such as foo.com but will not contain bar.foo.com |
| Lookup ISP * |
lookup-isp |
false |
- true
- false
|
Specifies whether or not information about the Information Service Provider corresponding to the IP address should be returned |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: ISPEnrichIP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/ispenrichip.md
section: Loading & Unloading Data
---
# ISPEnrichIP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-enrich-nar
## Description
Looks up ISP information for an IP address and adds the information to FlowFile attributes. The ISP data is provided as a MaxMind ISP database. (Note that this is NOT the same as the GeoLite database utilized by some geo enrichment tools). The attribute that contains the IP address to lookup is provided by the 'IP Address Attribute' property. If the name of the attribute provided is 'X', then the attributes added by enrichment will take the form X.isp.<fieldName>
## Tags
ISP, enrich, ip, maxmind
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| IP Address Attribute |
The name of an attribute whose value is a dotted decimal IP address for which enrichment should occur |
| Log Level |
The Log Level to use when an IP is not found in the database. Accepted values: INFO, DEBUG, WARN, ERROR. |
| MaxMind Database File |
Path to Maxmind IP Enrichment Database File |
## Relationships
| Name |
Description |
| found |
Where to route flow files after successfully enriching attributes with data provided by database |
| not found |
Where to route flow files after unsuccessfully enriching attributes because no data was found |
## Writes attributes
| Name |
Description |
| X.isp.lookup.micros |
The number of microseconds that the geo lookup took |
| X.isp.asn |
The Autonomous System Number (ASN) identified for the IP address |
| X.isp.asn.organization |
The Organization Associated with the ASN identified |
| X.isp.name |
The name of the ISP associated with the IP address provided |
| X.isp.organization |
The Organization associated with the IP address provided |
---
title: JettyWebSocketClient
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/jettywebsocketclient.md
section: Loading & Unloading Data
---
# JettyWebSocketClient
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Implementation of WebSocketClientService. This service uses Jetty WebSocket client module to provide WebSocket session management throughout the application.
## Tags
Jetty, WebSocket, client
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Authentication Header Charset * |
Authentication Header Charset |
US-ASCII |
|
The charset for Basic Authentication header base64 string. |
| Connection Attempt Count * |
Connection Attempt Count |
3 |
|
The number of times to try and establish a connection. |
| Connection Timeout * |
Connection Timeout |
3 sec |
|
The timeout to connect the WebSocket URI. |
| Custom Authorization |
Custom Authorization |
|
|
Configures a custom HTTP Authorization Header as described in RFC 7235 Section 4.2. Setting a custom Authorization Header excludes configuring the User Name and User Password properties for Basic Authentication. |
| HTTP Proxy Host |
HTTP Proxy Host |
|
|
The host name of the HTTP Proxy. |
| HTTP Proxy Port |
HTTP Proxy Port |
|
|
The port number of the HTTP Proxy. |
| Idle Timeout * |
Idle Timeout |
0 sec |
|
The maximum amount of time that a WebSocket connection may remain idle before it is closed. A value of 0 sec disables the timeout. |
| Input Buffer Size * |
Input Buffer Size |
4 kb |
|
The size of the input (read from network layer) buffer size. |
| Max Binary Message Size * |
Max Binary Message Size |
64 kb |
|
The maximum size of a binary message during parsing/generating. |
| Max Text Message Size * |
Max Text Message Size |
64 kb |
|
The maximum size of a text message during parsing/generating. |
| Password |
Password |
|
|
The user password for Basic Authentication. |
| SSL Context Service |
SSL Context Service |
|
|
The SSL Context Service to use in order to secure the server. If specified, the server will accept only WSS requests; otherwise, the server will accept only WS requests |
| Session Maintenance Interval * |
Session Maintenance Interval |
10 sec |
|
The interval between session maintenance activities. A WebSocket session established with a WebSocket server can be terminated due to different reasons including restarting the WebSocket server or timing out inactive sessions. This session maintenance activity is periodically executed in order to reconnect those lost sessions, so that a WebSocket client can reuse the same session id transparently after it reconnects successfully. The maintenance activity is executed until corresponding processors or this controller service is stopped. |
| Username |
Username |
|
|
The user name for Basic Authentication. |
| WebSocket URI * |
WebSocket URI |
|
|
The WebSocket URI this client connects to. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: JettyWebSocketServer
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/jettywebsocketserver.md
section: Loading & Unloading Data
---
# JettyWebSocketServer
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Implementation of WebSocketServerService. This service uses Jetty WebSocket server module to provide WebSocket session management throughout the application.
## Tags
Jetty, WebSocket, server
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Basic Authentication Enabled * |
Basic Authentication Enabled |
false |
- true
- false
|
If enabled, client connection requests are authenticated with Basic authentication using the specified Login Provider. |
| Basic Authentication Path Spec |
Basic Authentication Path Spec |
/* |
|
Specify a Path Spec to apply Basic Authentication. |
| Basic Authentication Roles |
Basic Authentication Roles |
`**` |
|
The authenticated user must have one of specified role. Multiple roles can be set as comma separated string. '*' represents any role and so does '**' any role including no role. |
| Client Authentication * |
Client Authentication |
no |
- No Authentication
- Want Authentication
- Need Authentication
|
Specifies whether or not the Processor should authenticate client by its certificate. This value is ignored if the <SSL Context Service> Property is not specified or the SSL Context provided uses only a KeyStore and not a TrustStore. |
| Idle Timeout * |
Idle Timeout |
0 sec |
|
The maximum amount of time that a WebSocket connection may remain idle before it is closed. A value of 0 sec disables the timeout. |
| Input Buffer Size * |
Input Buffer Size |
4 kb |
|
The size of the input (read from network layer) buffer size. |
| Login Service |
Login Service |
hash |
- HashLoginService |
Specify which Login Service to use for Basic Authentication. |
| Max Binary Message Size * |
Max Binary Message Size |
64 kb |
|
The maximum size of a binary message during parsing/generating. |
| Max Text Message Size * |
Max Text Message Size |
64 kb |
|
The maximum size of a text message during parsing/generating. |
| Port * |
Port |
|
|
The port number on which this WebSocketServer listens to. |
| SSL Context Service |
SSL Context Service |
|
|
The SSL Context Service to use in order to secure the server. If specified, the server will accept only WSS requests; otherwise, the server will accept only WS requests |
| Users Properties File |
users-properties-file |
|
|
Specify a property file containing users for Basic Authentication using HashLoginService. See http://www.eclipse.org/jetty/documentation/current/configuring-security.html (http://www.eclipse.org/jetty/documentation/current/configuring-security.html) for detail. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: JMSConnectionFactoryProvider
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/jmsconnectionfactoryprovider.md
section: Loading & Unloading Data
---
# JMSConnectionFactoryProvider
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides a generic service to create vendor specific javax.jms. ConnectionFactory implementations. The Connection Factory can be served once this service is configured successfully.
## Tags
integration, jms, messaging, publish, queue, subscribe, topic
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| JMS SSL Context Service |
SSL Context Service |
|
|
The SSL Context Service used to provide client certificate information for TLS/SSL connections. |
| JMS Broker URI |
broker |
|
|
URI pointing to the network location of the JMS Message broker. Example for ActiveMQ: '[tcp://myhost:61616](tcp://myhost:61616)'. Examples for IBM MQ: 'myhost(1414)' and 'myhost01(1414),myhost02(1414)'. |
| JMS Connection Factory Implementation Class * |
cf |
|
|
The fully qualified name of the JMS ConnectionFactory implementation class (eg. org.apache.activemq.ActiveMQConnectionFactory). |
| JMS Client Libraries |
cflib |
|
|
Path to the directory with additional resources (eg. JARs, configuration files etc.) to be added to the classpath (defined as a comma separated list of values). Such resources typically represent target JMS client libraries for the ConnectionFactory implementation. |
## State management
This component does not store state.
## Restricted
## Restrictions
| Required Permission |
Explanation |
| reference remote resources |
Client Library Location can reference resources over HTTP |
## System Resource Considerations
This component does not specify system resource considerations.
---
title: JndiJmsConnectionFactoryProvider
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/jndijmsconnectionfactoryprovider.md
section: Loading & Unloading Data
---
# JndiJmsConnectionFactoryProvider
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides a service to lookup an existing JMS ConnectionFactory using the Java Naming and Directory Interface (JNDI).
## Tags
integration, jms, jndi, messaging, publish, queue, subscribe, topic
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| JNDI Name of the Connection Factory * |
connection.factory.name |
|
|
The name of the JNDI Object to lookup for the Connection Factory. |
| JNDI Initial Context Factory Class * |
java.naming.factory.initial |
|
|
The fully qualified class name of the JNDI Initial Context Factory Class (java.naming.factory.initial). |
| JNDI Provider URL * |
java.naming.provider.url |
|
|
The URL of the JNDI Provider to use as the value for java.naming.provider.url. See additional details documentation for allowed URL schemes. |
| JNDI Credentials |
java.naming.security.credentials |
|
|
The Credentials to use when authenticating with JNDI (java.naming.security.credentials). |
| JNDI Principal |
java.naming.security.principal |
|
|
The Principal to use when authenticating with JNDI (java.naming.security.principal). |
| JNDI / JMS Client Libraries |
naming.factory.libraries |
|
|
Specifies jar files and/or directories to add to the ClassPath in order to load the JNDI / JMS client libraries. This should be a comma-separated list of files, directories, and/or URLs. If a directory is given, any files in that directory will be included, but subdirectories will not be included (i.e., it is not recursive). |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: JoinEnrichment 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/joinenrichment.md
section: Loading & Unloading Data
---
# JoinEnrichment 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Joins together Records from two different FlowFiles where one FlowFile, the 'original' contains arbitrary records and the second FlowFile, the 'enrichment' contains additional data that should be used to enrich the first. See Additional Details for more information on how to configure this processor and the different use cases that it aims to accomplish.
## Tags
combine, enrichment, fork, join, merge, record, recordpath, sql, streams, wrap
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Default Decimal Precision |
When a DECIMAL/NUMBER value is written as a 'decimal' Avro logical type, a specific 'precision' denoting number of available digits is required. Generally, precision is defined by column data type definition or database engines default. However undefined precision (0) can be returned from some database engines. 'Default Decimal Precision' is used when writing those undefined precision numbers. |
| Default Decimal Scale |
When a DECIMAL/NUMBER value is written as a 'decimal' Avro logical type, a specific 'scale' denoting number of available decimal digits is required. Generally, scale is defined by column data type definition or database engines default. However when undefined precision (0) is returned, scale can also be uncertain with some database engines. 'Default Decimal Scale' is used when writing those undefined numbers. If a value has more decimals than specified scale, then the value will be rounded-up, e.g. 1.53 becomes 2 with scale 0, and 1.5 with scale 1. |
| Enrichment Record Reader |
The Record Reader for reading the 'enrichment' FlowFile |
| Insertion Record Path |
Specifies where in the 'original' Record the 'enrichment' Record's fields should be inserted. Note that if the RecordPath does not point to any existing field in the original Record, the enrichment will not be inserted. |
| Join Strategy |
Specifies how to join the two FlowFiles into a single FlowFile |
| Maximum number of Bins |
Specifies the maximum number of bins that can be held in memory at any one time |
| Original Record Reader |
The Record Reader for reading the 'original' FlowFile |
| Record Writer |
The Record Writer to use for writing the results. If the Record Writer is configured to inherit the schema from the Record, the schema that it will inherit will be the result of merging both the 'original' record schema and the 'enrichment' record schema. |
| SQL |
The SQL SELECT statement to evaluate. Expression Language may be provided, but doing so may result in poorer performance. Because this Processor is dealing with two FlowFiles at a time, it 's also important to understand how attributes will be referenced. If both FlowFiles have an attribute with the same name but different values, the Expression Language will resolve to the value provided by the' enrichment' FlowFile. |
| Timeout |
Specifies the maximum amount of time to wait for the second FlowFile once the first arrives at the processor, after which point the first FlowFile will be routed to the 'timeout' relationship. |
## Relationships
| Name |
Description |
| failure |
If both the 'original' and 'enrichment' FlowFiles arrive at the processor but there was a failure in joining the records, both of those FlowFiles will be routed to this relationship. |
| joined |
The resultant FlowFile with Records joined together from both the original and enrichment FlowFiles will be routed to this relationship |
| original |
Both of the incoming FlowFiles ('original' and 'enrichment') will be routed to this Relationship. I.e., this is the 'original' version of both of these FlowFiles. |
| timeout |
If one of the incoming FlowFiles (i.e., the 'original' FlowFile or the 'enrichment' FlowFile) arrives to this Processor but the other does not arrive within the configured Timeout period, the FlowFile that did arrive is routed to this relationship. |
## Writes attributes
| Name |
Description |
| mime.type |
Sets the mime.type attribute to the MIME Type specified by the Record Writer |
| record.count |
The number of records in the FlowFile |
## See also
- [org.apache.nifi.processors.standard.ForkEnrichment](/user-guide/data-integration/openflow/processors/forkenrichment)
---
title: JoltTransformJSON 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/jolttransformjson.md
section: Loading & Unloading Data
---
# JoltTransformJSON 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-jolt-nar
## Description
Applies a list of Jolt specifications to either the FlowFile JSON content or a specified FlowFile JSON attribute. If the JSON transform fails, the original FlowFile is routed to the 'failure' relationship.
## Tags
cardinality, chainr, default, jolt, json, remove, shift, sort, transform
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Custom Module Directory |
Comma-separated list of paths to files and/or directories which contain modules containing custom transformations (that are not included on NiFi's classpath). |
| Custom Transformation Class Name |
Fully Qualified Class Name for Custom Transformation |
| JSON Source |
Specifies whether the Jolt transformation is applied to FlowFile JSON content or to specified FlowFile JSON attribute. |
| JSON Source Attribute |
The FlowFile attribute containing JSON to be transformed. |
| Jolt Specification |
Jolt Specification for transformation of JSON data. The value for this property may be the text of a Jolt specification or the path to a file containing a Jolt specification. 'Jolt Specification' must be set, or the value is ignored if the Jolt Sort Transformation is selected. |
| Jolt Transform |
Specifies the Jolt Transformation that should be used with the provided specification. |
| Max String Length |
The maximum allowed length of a string value when parsing the JSON document |
| Pretty Print |
Apply pretty print formatting to the output of the Jolt transform |
| Transform Cache Size |
Compiling a Jolt Transform can be fairly expensive. Ideally, this will be done only once. However, if the Expression Language is used in the transform, we may need a new Transform for each FlowFile. This value controls how many of those Transforms we cache in memory in order to avoid having to compile the Transform each time. |
## Relationships
| Name |
Description |
| failure |
If the JSON transformation fails (e.g., due to invalid JSON in the content or attribute), the original FlowFile is routed to this relationship. |
| success |
The FlowFile with successfully transformed content or updated attribute will be routed to this relationship |
## Writes attributes
| Name |
Description |
| mime.type |
Always set to application/json |
---
title: JoltTransformRecord 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/jolttransformrecord.md
section: Loading & Unloading Data
---
# JoltTransformRecord 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-jolt-nar
## Description
Applies a JOLT specification to each record in the FlowFile payload. A new FlowFile is created with transformed content and is routed to the 'success' relationship. If the transform fails, the original FlowFile is routed to the 'failure' relationship.
## Tags
cardinality, chainr, defaultr, jolt, record, removr, shiftr, sort, transform
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Custom Module Directory |
Comma-separated list of paths to files and/or directories which contain modules containing custom transformations (that are not included on NiFi's classpath). |
| Custom Transformation Class Name |
Fully Qualified Class Name for Custom Transformation |
| Jolt Specification |
Jolt Specification for transformation of JSON data. The value for this property may be the text of a Jolt specification or the path to a file containing a Jolt specification. 'Jolt Specification' must be set, or the value is ignored if the Jolt Sort Transformation is selected. |
| Jolt Transform |
Specifies the Jolt Transformation that should be used with the provided specification. |
| Transform Cache Size |
Compiling a Jolt Transform can be fairly expensive. Ideally, this will be done only once. However, if the Expression Language is used in the transform, we may need a new Transform for each FlowFile. This value controls how many of those Transforms we cache in memory in order to avoid having to compile the Transform each time. |
| jolt-record-record-reader |
Specifies the Controller Service to use for parsing incoming data and determining the data's schema. |
| jolt-record-record-writer |
Specifies the Controller Service to use for writing out the records |
## Relationships
| Name |
Description |
| failure |
If a FlowFile fails processing for any reason (for example, the FlowFile records cannot be parsed), it will be routed to this relationship |
| original |
The original FlowFile that was transformed. If the FlowFile fails processing, nothing will be sent to this relationship |
| success |
The FlowFile with transformed content will be routed to this relationship |
## Writes attributes
| Name |
Description |
| record.count |
The number of records in an outgoing FlowFile |
| mime.type |
The MIME Type that the configured Record Writer indicates is appropriate |
---
title: JSLTTransformJSON 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/jslttransformjson.md
section: Loading & Unloading Data
---
# JSLTTransformJSON 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-jslt-nar
## Description
Applies a JSLT transformation to the FlowFile JSON payload. A new FlowFile is created with transformed content and is routed to the 'success' relationship. If the JSLT transform fails, the original FlowFile is routed to the 'failure' relationship.
## Tags
jslt, json, transform
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| jslt-transform-cache-size |
Compiling a JSLT Transform can be fairly expensive. Ideally, this will be done only once. However, if the Expression Language is used in the transform, we may need a new Transform for each FlowFile. This value controls how many of those Transforms we cache in memory in order to avoid having to compile the Transform each time. |
| jslt-transform-pretty_print |
Apply pretty-print formatting to the output of the JSLT transform |
| jslt-transform-result-filter |
A filter for output JSON results using a JSLT expression. This property supports changing the default filter, which removes JSON objects with null values, empty objects and empty arrays from the output JSON. This JSLT must return true for each JSON object to be included and false for each object to be removed. Using a filter value of "true" to disables filtering. |
| jslt-transform-transformation |
JSLT Transformation for transform of JSON data. Any NiFi Expression Language present will be evaluated first to get the final transform to be applied. The JSLT Tutorial provides an overview of supported expressions: https://github.com/schibsted/jslt/blob/master/tutorial.md (https://github.com/schibsted/jslt/blob/master/tutorial.md) |
| jslt-transform-transformation-strategy |
Whether to apply the JSLT transformation to the entire FlowFile contents or each JSON object in the root-level array |
## Relationships
| Name |
Description |
| failure |
If a FlowFile fails processing for any reason (for example, the FlowFile is not valid JSON), it will be routed to this relationship |
| success |
The FlowFile with transformed content will be routed to this relationship |
## Writes attributes
| Name |
Description |
| mime.type |
Always set to application/json |
---
title: JsonConfigBasedBoxClientService
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/jsonconfigbasedboxclientservice.md
section: Loading & Unloading Data
---
# JsonConfigBasedBoxClientService
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides Box client objects through which Box API calls can be used.
## Tags
box, client, provider
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Account ID * |
Account ID |
|
|
The ID of the Box account which the app will act on behalf of. |
| App Actor * |
App Actor |
impersonated-user |
- Service Account
- Impersonated User
|
Specifies on behalf of whom Box API calls will be made. |
| App Config File |
App Config File |
|
|
Full path of an App config JSON file. See Additional Details for more information. |
| App Config JSON |
App Config JSON |
|
|
The raw JSON containing an App config. See Additional Details for more information. |
| Connect Timeout * |
Connect Timeout |
10 secs |
|
Maximum amount of time to wait before failing during initial socket connection. |
| Read Timeout * |
Read Timeout |
30 secs |
|
Maximum amount of time to wait before failing while reading socket responses. |
| Proxy Configuration Service |
proxy-configuration-service |
|
|
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: JsonPathReader
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/jsonpathreader.md
section: Loading & Unloading Data
---
# JsonPathReader
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Parses JSON records and evaluates user-defined JSON Path 's against each JSON object. While the reader expects each record to be well-formed JSON, the content of a FlowFile may consist of many records, each as a well-formed JSON array or JSON object with optional whitespace between them, such as the common'JSON-per-line' format. If an array is encountered, each element in that array will be treated as a separate record. User-defined properties define the fields that should be extracted from the JSON in order to form the fields of a Record. Any JSON field that is not extracted via a JSONPath will not be returned in the JSON Records.
## Tags
json, jsonpath, parser, reader, record
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Allow Comments * |
Allow Comments |
false |
- true
- false
|
Whether to allow comments when parsing the JSON document |
| Date Format |
Date Format |
|
|
Specifies the format to use when reading/writing Date fields. If not specified, Date fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, MM/dd/yyyy for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters, as in 01/01/2017). |
| Max String Length * |
Max String Length |
20 MB |
|
The maximum allowed length of a string value when parsing the JSON document |
| Schema Access Strategy * |
Schema Access Strategy |
infer-schema |
- Use 'Schema Name' Property
- Use 'Schema Text' Property
- Schema Reference Reader
- Infer Schema
|
Specifies how to obtain the schema that is to be used for interpreting the data. |
| Schema Branch |
Schema Branch |
|
|
Specifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored. |
| Schema Name |
Schema Name |
$\{schema.name\} |
|
Specifies the name of the schema to lookup in the Schema Registry property |
| Schema Reference Reader * |
Schema Reference Reader |
|
|
Service implementation responsible for reading FlowFile attributes or content to determine the Schema Reference Identifier |
| Schema Registry |
Schema Registry |
|
|
Specifies the Controller Service to use for the Schema Registry |
| Schema Text |
Schema Text |
$\{avro.schema\} |
|
The text of an Avro-formatted Schema |
| Schema Version |
Schema Version |
|
|
Specifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved. |
| Time Format |
Time Format |
|
|
Specifies the format to use when reading/writing Time fields. If not specified, Time fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, HH:mm:ss for a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 18:04:15). |
| Timestamp Format |
Timestamp Format |
|
|
Specifies the format to use when reading/writing Timestamp fields. If not specified, Timestamp fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, MM/dd/yyyy HH:mm:ss for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters; and then followed by a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 01/01/2017 18:04:15). |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: JsonQueryElasticsearch 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/jsonqueryelasticsearch.md
section: Loading & Unloading Data
---
# JsonQueryElasticsearch 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-elasticsearch-restapi-nar
## Description
A processor that allows the user to run a query (with aggregations) written with the Elasticsearch JSON DSL. It does not automatically paginate queries for the user. If an incoming relationship is added to this processor, it will use the flowfile's content for the query. Care should be taken on the size of the query because the entire response from Elasticsearch will be loaded into memory all at once and converted into the resulting flowfiles.
## Tags
elasticsearch, elasticsearch7, elasticsearch8, elasticsearch9, get, json, query, read
## Input Requirement
ALLOWED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Aggregation Results Format |
Format of Aggregation output. |
| Aggregation Results Split |
Output a flowfile containing all aggregations or one flowfile for each individual aggregation. |
| Aggregations |
One or more query aggregations (or "aggs"), in JSON syntax. Ex: \{"items": \{"terms": \{"field": "product", "size": 10\}\}\} |
| Client Service |
An Elasticsearch client service to use for running queries. |
| Fields |
Fields of indexed documents to be retrieved, in JSON syntax. Ex: ["user.id", "http.response.*", \{"field": "@timestamp", "format": "epoch_millis"\}] |
| Index |
The name of the index to use. |
| Max JSON Field String Length |
The maximum allowed length of a string value when parsing a JSON document or attribute. |
| Output No Hits |
Output a "hits" flowfile even if no hits found for query. If true, an empty "hits" flowfile will be output even if "aggregations" are output. |
| Query |
A query in JSON syntax, not Lucene syntax. Ex: \{"query":\{"match":\{"somefield":"somevalue"\}\}\}. If this parameter is not set, the query will be read from the flowfile content. If the query (property and flowfile content) is empty, a default empty JSON Object will be used, which will result in a "match_all" query in Elasticsearch. |
| Query Attribute |
If set, the executed query will be set on each result flowfile in the specified attribute. |
| Query Clause |
A "query" clause in JSON syntax, not Lucene syntax. Ex: \{"match":\{"somefield":"somevalue"\}\}. If the query is empty, a default JSON Object will be used, which will result in a "match_all" query in Elasticsearch. |
| Query Definition Style |
How the JSON Query will be defined for use by the processor. |
| Script Fields |
Fields to created using script evaluation at query runtime, in JSON syntax. Ex: \{"test1": \{"script": \{"lang": "painless", "source": "doc[ 'price'].value * 2"\}\}, "test2": \{"script": \{"lang": "painless", "source": "doc[ 'price'].value * params.factor", "params": \{"factor": 2.0\}\}\}\} |
| Search Results Format |
Format of Hits output. |
| Search Results Split |
Output a flowfile containing all hits or one flowfile for each individual hit. |
| Size |
The maximum number of documents to retrieve in the query. If the query is paginated, this "size" applies to each page of the query, not the "size" of the entire result set. |
| Sort |
Sort results by one or more fields, in JSON syntax. Ex: [\{"price" : \{"order" : "asc", "mode" : "avg"\}\}, \{"post_date" : \{"format": "strict_date_optional_time_nanos"\}\}] |
| Type |
The type of this document (used by Elasticsearch for indexing and searching). |
## Relationships
| Name |
Description |
| aggregations |
Aggregations are routed to this relationship. |
| failure |
All flowfiles that fail for reasons unrelated to server availability go to this relationship. |
| hits |
Search hits are routed to this relationship. |
| original |
All original flowfiles that don't cause an error to occur go to this relationship. |
## Writes attributes
| Name |
Description |
| mime.type |
application/json |
| aggregation.name |
The name of the aggregation whose results are in the output flowfile |
| aggregation.number |
The number of the aggregation whose results are in the output flowfile |
| hit.count |
The number of hits that are in the output flowfile |
| elasticsearch.query.error |
The error message provided by Elasticsearch if there is an error querying the index. |
## See also
- [org.apache.nifi.processors.elasticsearch.PaginatedJsonQueryElasticsearch](/user-guide/data-integration/openflow/processors/paginatedjsonqueryelasticsearch)
---
title: JsonRecordSetWriter
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/jsonrecordsetwriter.md
section: Loading & Unloading Data
---
# JsonRecordSetWriter
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Writes the results of a RecordSet as either a JSON Array or one JSON object per line. If using Array output, then even if the RecordSet consists of a single row, it will be written as an array with a single element. If using One Line Per Object output, the JSON objects cannot be pretty-printed.
## Tags
json, record, recordset, resultset, row, serialize, writer
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Allow Scientific Notation * |
Allow Scientific Notation |
false |
- true
- false
|
Specifies whether or not scientific notation should be used when writing numbers |
| Date Format |
Date Format |
|
|
Specifies the format to use when reading/writing Date fields. If not specified, Date fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, MM/dd/yyyy for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters, as in 01/01/2017). |
| Pretty Print JSON * |
Pretty Print JSON |
false |
- true
- false
|
Specifies whether or not the JSON should be pretty printed |
| Schema Access Strategy * |
Schema Access Strategy |
inherit-record-schema |
- Inherit Record Schema
- Use 'Schema Name' Property
- Use 'Schema Text' Property
|
Specifies how to obtain the schema that is to be used for interpreting the data. |
| Schema Branch |
Schema Branch |
|
|
Specifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored. |
| Schema Cache |
Schema Cache |
|
|
Specifies a Schema Cache to add the Record Schema to so that Record Readers can quickly lookup the schema. |
| Schema Name |
Schema Name |
$\{schema.name\} |
|
Specifies the name of the schema to lookup in the Schema Registry property |
| Schema Reference Reader * |
Schema Reference Reader |
|
|
Service implementation responsible for reading FlowFile attributes or content to determine the Schema Reference Identifier |
| Schema Reference Writer * |
Schema Reference Writer |
|
|
Service implementation responsible for writing FlowFile attributes or content header with Schema reference information |
| Schema Registry |
Schema Registry |
|
|
Specifies the Controller Service to use for the Schema Registry |
| Schema Text |
Schema Text |
$\{avro.schema\} |
|
The text of an Avro-formatted Schema |
| Schema Version |
Schema Version |
|
|
Specifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved. |
| Schema Write Strategy * |
Schema Write Strategy |
no-schema |
- Do Not Write Schema
- Set 'schema.name' Attribute
- Set 'avro.schema' Attribute
- Schema Reference Writer
|
Specifies how the schema for a Record should be added to the data. |
| Time Format |
Time Format |
|
|
Specifies the format to use when reading/writing Time fields. If not specified, Time fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, HH:mm:ss for a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 18:04:15). |
| Timestamp Format |
Timestamp Format |
|
|
Specifies the format to use when reading/writing Timestamp fields. If not specified, Timestamp fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, MM/dd/yyyy HH:mm:ss for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters; and then followed by a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 01/01/2017 18:04:15). |
| Compression Format * |
compression-format |
none |
- none
- gzip
- bzip2
- xz-lzma2
- snappy
- snappy framed
- zstd
|
The compression format to use. Valid values are: GZIP, BZIP2, ZSTD, XZ-LZMA2, LZMA, Snappy, and Snappy Framed |
| Compression Level * |
compression-level |
1 |
- 0
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
|
The compression level to use; this is valid only when using GZIP compression. A lower value results in faster processing but less compression; a value of 0 indicates no compression but simply archiving |
| Output Grouping * |
output-grouping |
output-array |
- Array
- One Line Per Object
|
Specifies how the writer should output the JSON records (as an array or one object per line, e.g.) Note that if 'One Line Per Object' is selected, then Pretty Print JSON must be false. |
| Suppress Null Values * |
suppress-nulls |
never-suppress |
- Never Suppress
- Always Suppress
- Suppress Missing Values
|
Specifies how the writer should handle a null field |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: JsonTableColumnFilter
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/jsontablecolumnfilter.md
section: Loading & Unloading Data
---
# JsonTableColumnFilter
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides a table column filter based on a JSON configuration. The JSON configuration should be an array of objects, where each object represents a table and its column filter. The object should have the following properties: - schema: the schema name of the table - table: the table name - included: an array of column names to include - excluded: an array of column names to exclude - includedPattern: a regular expression pattern to include columns - excludedPattern: a regular expression pattern to exclude columns The schema and table must be provided for each object, and one or more of the *included*, *excluded*, *includedPattern*, or *excludedPattern* properties must be provided. If any column is included as both included and excluded, the column will be excluded. If only a single filter is provided, the JSON configuration may be a single JSON object, rather than an array.
## Tags
column, database, filter, snowflake, table
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Filter JSON |
Filter JSON |
|
|
JSON representation of the column filter |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: JsonTreeReader
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/jsontreereader.md
section: Loading & Unloading Data
---
# JsonTreeReader
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Parses JSON into individual Record objects. While the reader expects each record to be well-formed JSON, the content of a FlowFile may consist of many records, each as a well-formed JSON array or JSON object with optional whitespace between them, such as the common 'JSON-per-line' format. If an array is encountered, each element in that array will be treated as a separate record. If the schema that is configured contains a field that is not present in the JSON, a null value will be used. If the JSON contains a field that is not present in the schema, that field will be skipped. See the Usage of the Controller Service for more information and examples.
## Tags
json, parser, reader, record, tree
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Allow Comments * |
Allow Comments |
false |
- true
- false
|
Whether to allow comments when parsing the JSON document |
| Date Format |
Date Format |
|
|
Specifies the format to use when reading/writing Date fields. If not specified, Date fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, MM/dd/yyyy for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters, as in 01/01/2017). |
| Max String Length * |
Max String Length |
20 MB |
|
The maximum allowed length of a string value when parsing the JSON document |
| Schema Access Strategy * |
Schema Access Strategy |
infer-schema |
- Infer Schema
- Use 'Schema Name' Property
- Use 'Schema Text' Property
- Schema Reference Reader
|
Specifies how to obtain the schema that is to be used for interpreting the data. |
| Schema Branch |
Schema Branch |
|
|
Specifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored. |
| Schema Name |
Schema Name |
$\{schema.name\} |
|
Specifies the name of the schema to lookup in the Schema Registry property |
| Schema Reference Reader * |
Schema Reference Reader |
|
|
Service implementation responsible for reading FlowFile attributes or content to determine the Schema Reference Identifier |
| Schema Registry |
Schema Registry |
|
|
Specifies the Controller Service to use for the Schema Registry |
| Schema Text |
Schema Text |
$\{avro.schema\} |
|
The text of an Avro-formatted Schema |
| Schema Version |
Schema Version |
|
|
Specifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved. |
| Time Format |
Time Format |
|
|
Specifies the format to use when reading/writing Time fields. If not specified, Time fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, HH:mm:ss for a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 18:04:15). |
| Timestamp Format |
Timestamp Format |
|
|
Specifies the format to use when reading/writing Timestamp fields. If not specified, Timestamp fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, MM/dd/yyyy HH:mm:ss for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters; and then followed by a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 01/01/2017 18:04:15). |
| Schema Application Strategy * |
schema-application-strategy |
SELECTED_PART |
- Whole JSON
- Selected Part
|
Specifies whether the schema is defined for the whole JSON or for the selected part starting from "Starting Field Name". |
| Schema Inference Cache |
schema-inference-cache |
|
|
Specifies a Schema Cache to use when inferring the schema. If not populated, the schema will be inferred each time. However, if a cache is specified, the cache will first be consulted and if the applicable schema can be found, it will be used instead of inferring the schema. |
| Starting Field Name |
starting-field-name |
|
|
Skips forward to the given nested JSON field (array or object) to begin processing. |
| Starting Field Strategy * |
starting-field-strategy |
ROOT_NODE |
- Root Node
- Nested Field
|
Start processing from the root node or from a specified nested node. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: JWTBearerOAuth2AccessTokenProvider
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/jwtbeareroauth2accesstokenprovider.md
section: Loading & Unloading Data
---
# JWTBearerOAuth2AccessTokenProvider
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides OAuth 2.0 access tokens that can be used as Bearer authorization header in HTTP requests. This controller service is for implementing the OAuth 2.0 JWT Bearer Flow.
## Tags
access token, authorization, hjwt, oauth2, provider
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Assertion Parameter Name * |
Assertion Parameter Name |
assertion |
|
Name of the parameter to use for the JWT assertion in the request to the token endpoint. |
| Audience |
Audience |
|
|
The audience claim (aud) for the JWT. Space-separated list of audiences if multiple are expected. |
| Grant Type * |
Grant Type |
[urn:ietf:params:oauth:grant-type:jwt-bearer](urn:ietf:params:oauth:grant-type:jwt-bearer) |
|
Value to set for the grant_type parameter in the request to the token endpoint. |
| Issuer |
Issuer |
|
|
The issuer claim (iss) for the JWT. |
| JWT Expiration Time * |
JWT Expiration Time |
1 hour |
|
Expiration time used to set the corresponding claim of the JWT. In case the returned access token does not includean expiration time, this will be used with the refresh window to re-acquire a new access token. |
| JWT ID |
JWT ID |
|
|
The "jti" (JWT ID) claim provides a unique identifier for the JWT. The identifier value must be assigned in amanner that ensures that there's a negligible probability that the same value will be accidentally assigned to adifferent data object; if the application uses multiple issuers, collisions MUST be prevented among values producedby different issuers as well. The "jti" value is a case-sensitive string. If set, it is recommended to set thisvalue to $\{UUID()\}. |
| Key ID |
Key ID |
|
|
The ID of the public key used to sign the JWT. It'll be used as the kid header in the JWT. |
| Private Key Service * |
Private Key Service |
|
|
The private key service to use for signing JWTs. |
| Refresh Window * |
Refresh Window |
5 minutes |
|
The service will attempt to refresh tokens expiring within the refresh window, subtracting the configured duration from the token expiration. |
| SSL Context Service * |
SSL Context Service |
|
|
An instance of SSLContextProvider configured with a certificate that will be used to set the x5t header. Must be using RSA algorithm. |
| Scope |
Scope |
|
|
The scope claim (scope) for the JWT. |
| Set JWT Header X.509 Cert Thumbprint * |
Set JWT Header X.509 Cert Thumbprint |
false |
- true
- false
|
If true, will set the JWT header x5t field with the base64url-encoded SHA-256 thumbprint of the X.509 certificate's DER encoding.If set to true, an instance of SSLContextProvider must be configured with a certificate using RSA algorithm. |
| Signing Algorithm * |
Signing Algorithm |
PS256 |
- RS256
- RS384
- RS512
- PS256
- PS384
- PS512
- ES256
- ES384
- ES512
- Ed25519
|
The algorithm to use for signing the JWT. |
| Subject |
Subject |
|
|
The subject claim (sub) for the JWT. |
| Token Endpoint URL * |
Token Endpoint URL |
|
|
The URL of the OAuth2 token endpoint. |
| Web Client Service * |
Web Client Service |
|
|
The Web Client Service to use for calling the token endpoint. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: Kafka3ConnectionService
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/kafka3connectionservice.md
section: Loading & Unloading Data
---
# Kafka3ConnectionService
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides and manages connections to Kafka Brokers for producer or consumer operations.
## Tags
kafka, openflow
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| SSL Context Service |
SSL Context Service |
|
|
Service supporting SSL communication with Kafka brokers |
| Acknowledgment Wait Time * |
ack.wait.time |
5 sec |
|
After sending a message to Kafka, this indicates the amount of time that the service will wait for a response from Kafka.If Kafka does not acknowledge the message within this time period, the service will throw an exception. |
| Bootstrap Servers * |
bootstrap.servers |
|
|
Comma-separated list of Kafka Bootstrap Servers in the format host:port. Corresponds to Kafka bootstrap.servers property |
| Client Timeout * |
default.api.timeout.ms |
60 sec |
|
Default timeout for Kafka client operations. Mapped to Kafka default.api.timeout.ms. The Kafka request.timeout.ms property is derived from half of the configured timeout |
| Transaction Isolation Level * |
isolation.level |
read_committed |
- Read Committed
- Read Uncommitted
|
Specifies how the service should handle transaction isolation levels when communicating with Kafka.The uncommitted option means that messages will be received as soon as they are written to Kafka but will be pulled, even if the producer cancels the transactions.The committed option configures the service to not receive any messages for which the producer's transaction was canceled, but this can result in some latency since theconsumer must wait for the producer to finish its entire transaction instead of pulling as the messages become available.Corresponds to Kafka isolation.level property. |
| Max Metadata Wait Time * |
max.block.ms |
5 sec |
|
The amount of time publisher will wait to obtain metadata or wait for the buffer to flush during the 'send' call before failing theentire 'send' call. Corresponds to Kafka max.block.ms property |
| Max Poll Records * |
max.poll.records |
10000 |
|
Maximum number of records Kafka should return in a single poll. |
| SASL Mechanism * |
sasl.mechanism |
GSSAPI |
- GSSAPI
- PLAIN
- SCRAM-SHA-256
- SCRAM-SHA-512
|
SASL mechanism used for authentication. Corresponds to Kafka Client sasl.mechanism property |
| SASL Password * |
sasl.password |
|
|
Password provided with configured username when using PLAIN or SCRAM SASL Mechanisms |
| SASL Username * |
sasl.username |
|
|
Username provided with configured password when using PLAIN or SCRAM SASL Mechanisms |
| Security Protocol * |
security.protocol |
PLAINTEXT |
- PLAINTEXT
- SSL
- SASL_PLAINTEXT
- SASL_SSL
|
Security protocol used to communicate with brokers. Corresponds to Kafka Client security.protocol property |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: ListArchivedHubSpotData 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listarchivedhubspotdata.md
section: Loading & Unloading Data
---
# ListArchivedHubSpotData 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-hubspot-processors-nar
## Description
Lists archived data from HubSpot for the chosen object type and generates one FlowFile per listed object with the corresponding metadata as FlowFile attributes. The object type must be searchable, which means it supports access to the /search endpoint. For more information about searchable object types, see: https://developers.hubspot.com/docs/reference/api/crm/objects/objects#search (https://developers.hubspot.com/docs/reference/api/crm/objects/objects#search)")
## Tags
Preview, hubspot
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| HubSpot Service |
HubSpot Client Service. |
| Object Type |
HubSpot object type |
| Updated After |
Filter objects updated after specified date (format: yyyy-MM-dd) |
## State management
| Scopes |
Description |
| CLUSTER |
Maintains pagination state and last sync timestamp to continue data retrieval from the last known position after restarts and to fetch only changed data. |
## Relationships
| Name |
Description |
| failure |
HubSpot fail relationship |
| original |
The input Flow File is routed to the original relationship. |
| retry |
HubSpot retry relationship. FlowFiles that failed to process due to a server timeout or rate limit related error. FlowFiles routed here should be routed back into the processor. |
| success |
HubSpot success relationship |
## Writes attributes
| Name |
Description |
| mime.type |
application/json |
| statement.type |
DELETE |
| hubspot.object.type |
HubSpot Object Type for this fetch |
| hubspot.object.id |
HubSpot Object ID for this fetch |
| hubspot.run.id |
Timestamp of the start of this run. Obtained from the incoming FlowFile or current time if not available |
| hubspot.is_last |
Whether this is the last paged object of the ingestion |
## Use cases
| This processor is typically used in conjunction with a GenerateFlowFile processor |
| --------------------------------------------------------------------------------- |
## See also
- [com.snowflake.openflow.runtime.processors.hubspot.GetHubSpotObject](/user-guide/data-integration/openflow/processors/gethubspotobject)
- [com.snowflake.openflow.runtime.processors.hubspot.GetHubSpotSchema](/user-guide/data-integration/openflow/processors/gethubspotschema)
- [com.snowflake.openflow.runtime.processors.hubspot.ListHubSpotObjects](/user-guide/data-integration/openflow/processors/listhubspotobjects)
- [com.snowflake.openflow.runtime.processors.hubspot.PutHubSpot](/user-guide/data-integration/openflow/processors/puthubspot)
---
title: ListAzureBlobStorage_v12 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listazureblobstorage_v12.md
section: Loading & Unloading Data
---
# ListAzureBlobStorage_v12 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-azure-nar
## Description
Lists blobs in an Azure Blob Storage container. Listing details are attached to an empty FlowFile for use with FetchAzureBlobStorage. This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left off without duplicating all of the data. The processor uses Azure Blob Storage client library v12.
## Tags
azure, blob, cloud, microsoft, storage
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Blob Name Prefix |
Search prefix for listing |
| Container Name |
Name of the Azure storage container. In case of PutAzureBlobStorage processor, container can be created if it does not exist. |
| Entity Tracking Initial Listing Target |
Specify how initial listing should be handled. Used by 'Tracking Entities'strategy. |
| Entity Tracking State Cache |
Listed entities are stored in the specified cache storage so that this processor can resume listing across NiFi restart or in case of primary node change. 'Tracking Entities'strategy require tracking information of all listed entities within the last 'Tracking Time Window'. To support large number of entities, the strategy uses DistributedMapCache instead of managed state. Cache key format is 'ListedEntities::\{processorId\}(::\{nodeId\})'. If it tracks per node listed entities, then the optional '::\{nodeId\}' part is added to manage state separately. E.g. cluster wide cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b', per node cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3' The stored cache content is Gzipped JSON string. The cache key will be deleted when target listing configuration is changed. Used by 'Tracking Entities'strategy. |
| Entity Tracking Time Window |
Specify how long this processor should track already-listed entities. 'Tracking Entities'strategy can pick any entity whose timestamp is inside the specified time window. For example, if set to '30 minutes', any entity having timestamp in recent 30 minutes will be the listing target when this processor runs. A listed entity is considered 'new/updated' and a FlowFile is emitted if one of following condition meets: 1. does not exist in the already-listed entities, 2. has newer timestamp than the cached entity, 3. has different size than the cached entity. If a cached entity 's timestamp becomes older than specified time window, that entity will be removed from the cached already-listed entities. Used by'Tracking Entities'strategy. |
| Listing Strategy |
Specify how to determine new/updated entities. See each strategy descriptions for detail. |
| Maximum File Age |
The maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored |
| Maximum File Size |
The maximum size that a file can be in order to be pulled |
| Minimum File Age |
The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored |
| Minimum File Size |
The minimum size that a file must be in order to be pulled |
| Record Writer |
Specifies the Record Writer to use for creating the listing. If not specified, one FlowFile will be created for each entity that is listed. If the Record Writer is specified, all entities will be written to a single FlowFile instead of adding attributes to individual FlowFiles. |
| Storage Credentials |
Controller Service used to obtain Azure Blob Storage Credentials. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. In case of SOCKS, it is not guaranteed that the selected SOCKS Version will be used by the processor. |
## State management
| Scopes |
Description |
| CLUSTER |
After performing a listing of blobs, the timestamp of the newest blob is stored if 'Tracking Timestamps' Listing Strategy is in use (by default). This allows the Processor to list only blobs that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data. |
## Relationships
| Name |
Description |
| success |
All FlowFiles that are received are routed to success |
## Writes attributes
| Name |
Description |
| azure.container |
The name of the Azure Blob Storage container |
| azure.blobname |
The name of the blob on Azure Blob Storage |
| azure.primaryUri |
Primary location of the blob |
| azure.etag |
ETag of the blob |
| azure.blobtype |
Type of the blob (either BlockBlob, PageBlob or AppendBlob) |
| mime.type |
MIME Type of the content |
| lang |
Language code for the content |
| azure.timestamp |
Timestamp of the blob |
| azure.length |
Length of the blob |
## See also
- [org.apache.nifi.processors.azure.storage.CopyAzureBlobStorage_v12](/user-guide/data-integration/openflow/processors/copyazureblobstorage_v12)
- [org.apache.nifi.processors.azure.storage.DeleteAzureBlobStorage_v12](/user-guide/data-integration/openflow/processors/deleteazureblobstorage_v12)
- [org.apache.nifi.processors.azure.storage.FetchAzureBlobStorage_v12](/user-guide/data-integration/openflow/processors/fetchazureblobstorage_v12)
- [org.apache.nifi.processors.azure.storage.PutAzureBlobStorage_v12](/user-guide/data-integration/openflow/processors/putazureblobstorage_v12)
---
title: ListAzureDataLakeStorage 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listazuredatalakestorage.md
section: Loading & Unloading Data
---
# ListAzureDataLakeStorage 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-azure-nar
## Description
Lists directory in an Azure Data Lake Storage Gen 2 filesystem
## Tags
adlsgen2, azure, cloud, datalake, microsoft, storage
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| ADLS Credentials |
Controller Service used to obtain Azure Credentials. |
| Directory Name |
Name of the Azure Storage Directory. The Directory Name cannot contain a leading '/'. The root directory can be designated by the empty string value. In case of the PutAzureDataLakeStorage processor, the directory will be created if not already existing. |
| Entity Tracking Initial Listing Target |
Specify how initial listing should be handled. Used by 'Tracking Entities'strategy. |
| Entity Tracking State Cache |
Listed entities are stored in the specified cache storage so that this processor can resume listing across NiFi restart or in case of primary node change. 'Tracking Entities'strategy require tracking information of all listed entities within the last 'Tracking Time Window'. To support large number of entities, the strategy uses DistributedMapCache instead of managed state. Cache key format is 'ListedEntities::\{processorId\}(::\{nodeId\})'. If it tracks per node listed entities, then the optional '::\{nodeId\}' part is added to manage state separately. E.g. cluster wide cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b', per node cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3' The stored cache content is Gzipped JSON string. The cache key will be deleted when target listing configuration is changed. Used by 'Tracking Entities'strategy. |
| Entity Tracking Time Window |
Specify how long this processor should track already-listed entities. 'Tracking Entities'strategy can pick any entity whose timestamp is inside the specified time window. For example, if set to '30 minutes', any entity having timestamp in recent 30 minutes will be the listing target when this processor runs. A listed entity is considered 'new/updated' and a FlowFile is emitted if one of following condition meets: 1. does not exist in the already-listed entities, 2. has newer timestamp than the cached entity, 3. has different size than the cached entity. If a cached entity 's timestamp becomes older than specified time window, that entity will be removed from the cached already-listed entities. Used by'Tracking Entities'strategy. |
| File Filter |
Only files whose names match the given regular expression will be listed |
| Filesystem Name |
Name of the Azure Storage File System (also called Container). It is assumed to be already existing. |
| Include Temporary Files |
Whether to include temporary files when listing the contents of configured directory paths. |
| Listing Strategy |
Specify how to determine new/updated entities. See each strategy descriptions for detail. |
| Maximum File Age |
The maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored |
| Maximum File Size |
The maximum size that a file can be in order to be pulled |
| Minimum File Age |
The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored |
| Minimum File Size |
The minimum size that a file must be in order to be pulled |
| Path Filter |
When 'Recurse Subdirectories' is true, then only subdirectories whose paths match the given regular expression will be scanned |
| Record Writer |
Specifies the Record Writer to use for creating the listing. If not specified, one FlowFile will be created for each entity that is listed. If the Record Writer is specified, all entities will be written to a single FlowFile instead of adding attributes to individual FlowFiles. |
| Recurse Subdirectories |
Indicates whether to list files from subdirectories of the directory |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. In case of SOCKS, it is not guaranteed that the selected SOCKS Version will be used by the processor. |
## State management
| Scopes |
Description |
| CLUSTER |
After performing a listing of files, the timestamp of the newest file is stored. This allows the Processor to list only files that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data. |
## Relationships
| Name |
Description |
| success |
All FlowFiles that are received are routed to success |
## Writes attributes
| Name |
Description |
| azure.filesystem |
The name of the Azure File System |
| azure.filePath |
The full path of the Azure File |
| azure.directory |
The name of the Azure Directory |
| azure.filename |
The name of the Azure File |
| azure.length |
The length of the Azure File |
| azure.lastModified |
The last modification time of the Azure File |
| azure.etag |
The ETag of the Azure File |
## See also
- [org.apache.nifi.processors.azure.storage.DeleteAzureDataLakeStorage](/user-guide/data-integration/openflow/processors/deleteazuredatalakestorage)
- [org.apache.nifi.processors.azure.storage.FetchAzureDataLakeStorage](/user-guide/data-integration/openflow/processors/fetchazuredatalakestorage)
- [org.apache.nifi.processors.azure.storage.PutAzureDataLakeStorage](/user-guide/data-integration/openflow/processors/putazuredatalakestorage)
---
title: ListBoxFile 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listboxfile.md
section: Loading & Unloading Data
---
# ListBoxFile 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-box-nar
## Description
Lists files in a Box folder. Each listed file may result in one FlowFile, the metadata being written as FlowFile attributes. Or - in case the 'Record Writer' property is set - the entire result is written as records to a single FlowFile. This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left off without duplicating all of the data.
## Tags
box, storage
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Box Client Service |
Controller Service used to obtain a Box API connection. |
| Entity Tracking Initial Listing Target |
Specify how initial listing should be handled. Used by 'Tracking Entities'strategy. |
| Entity Tracking State Cache |
Listed entities are stored in the specified cache storage so that this processor can resume listing across NiFi restart or in case of primary node change. 'Tracking Entities'strategy require tracking information of all listed entities within the last 'Tracking Time Window'. To support large number of entities, the strategy uses DistributedMapCache instead of managed state. Cache key format is 'ListedEntities::\{processorId\}(::\{nodeId\})'. If it tracks per node listed entities, then the optional '::\{nodeId\}' part is added to manage state separately. E.g. cluster wide cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b', per node cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3' The stored cache content is Gzipped JSON string. The cache key will be deleted when target listing configuration is changed. Used by 'Tracking Entities'strategy. |
| Entity Tracking Time Window |
Specify how long this processor should track already-listed entities. 'Tracking Entities'strategy can pick any entity whose timestamp is inside the specified time window. For example, if set to '30 minutes', any entity having timestamp in recent 30 minutes will be the listing target when this processor runs. A listed entity is considered 'new/updated' and a FlowFile is emitted if one of following condition meets: 1. does not exist in the already-listed entities, 2. has newer timestamp than the cached entity, 3. has different size than the cached entity. If a cached entity 's timestamp becomes older than specified time window, that entity will be removed from the cached already-listed entities. Used by'Tracking Entities'strategy. |
| Folder ID |
The ID of the folder from which to pull list of files. |
| Listing Strategy |
Specify how to determine new/updated entities. See each strategy descriptions for detail. |
| Minimum File Age |
The minimum age a file must be in order to be considered; any files younger than this will be ignored. |
| Record Writer |
Specifies the Record Writer to use for creating the listing. If not specified, one FlowFile will be created for each entity that is listed. If the Record Writer is specified, all entities will be written to a single FlowFile instead of adding attributes to individual FlowFiles. |
| Search Recursively |
When 'true', will include list of files from sub-folders. Otherwise, will return only files that are within the folder defined by the 'Folder ID' property. |
## State management
| Scopes |
Description |
| CLUSTER |
The processor stores necessary data to be able to keep track what files have been listed already. What exactly needs to be stored depends on the 'Listing Strategy'. |
## Relationships
| Name |
Description |
| success |
All FlowFiles that are received are routed to success |
## Writes attributes
| Name |
Description |
| box.id |
The id of the file |
| filename |
The name of the file |
| path |
The folder path where the file is located |
| box.size |
The size of the file |
| box.timestamp |
The last modified time of the file |
## See also
- [org.apache.nifi.processors.box.FetchBoxFile](/user-guide/data-integration/openflow/processors/fetchboxfile)
- [org.apache.nifi.processors.box.PutBoxFile](/user-guide/data-integration/openflow/processors/putboxfile)
---
title: ListBoxFileInfo 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listboxfileinfo.md
section: Loading & Unloading Data
---
# ListBoxFileInfo 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-box-nar
## Description
Fetches file metadata for each file in a Box Folder. Takes a flowFile with a folder ID attribute and outputs flowFiles with records containing all file metadata.
## Tags
box, fetch, files, folder, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Box Client Service |
Controller Service used to obtain a Box API connection. |
| Folder ID |
The ID of the folder from which to fetch files. |
| Minimum File Age |
The minimum age a file must be in order to be considered; any files younger than this will be ignored. |
| Record Writer |
Specifies the Controller Service to use for writing the metadata records. Must be set. |
| Search Recursively |
When 'true', will include files from sub-folders. Otherwise, will return only files that are within the folder defined by the 'Folder ID' property. |
## Relationships
| Name |
Description |
| failure |
A FlowFile will be routed here if there is an error fetching file metadata from the folder. |
| not.found |
FlowFiles for which the specified Box folder was not found will be routed to this relationship. |
| success |
A FlowFile containing the file metadata records will be routed to this relationship upon successful processing. |
## Writes attributes
| Name |
Description |
| box.folder.id |
The ID of the folder from which files were fetched |
| record.count |
The number of records in the FlowFile |
| mime.type |
The MIME Type specified by the Record Writer |
| error.code |
The error code returned by Box |
| error.message |
The error message returned by Box |
## See also
- [org.apache.nifi.processors.box.FetchBoxFile](/user-guide/data-integration/openflow/processors/fetchboxfile)
- [org.apache.nifi.processors.box.ListBoxFile](/user-guide/data-integration/openflow/processors/listboxfile)
- [org.apache.nifi.processors.box.PutBoxFile](/user-guide/data-integration/openflow/processors/putboxfile)
---
title: ListBoxFileMetadataInstances 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listboxfilemetadatainstances.md
section: Loading & Unloading Data
---
# ListBoxFileMetadataInstances 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-box-nar
## Description
Retrieves all metadata instances associated with a Box file.
## Tags
box, instances, metadata, storage, templates
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Box Client Service |
Controller Service used to obtain a Box API connection. |
| File ID |
The ID of the file for which to fetch metadata. |
## Relationships
| Name |
Description |
| failure |
A FlowFile will be routed here if there is an error fetching metadata instances from the file. |
| not found |
FlowFiles for which the specified Box file was not found will be routed to this relationship. |
| success |
A FlowFile containing the metadata instances records will be routed to this relationship upon successful processing. |
## Writes attributes
| Name |
Description |
| box.id |
The ID of the file from which metadata was fetched |
| record.count |
The number of records in the FlowFile |
| mime.type |
The MIME Type specified by the Record Writer |
| box.metadata.instances.names |
Comma-separated list of instances names |
| box.metadata.instances.count |
Number of metadata instances found |
| error.code |
The error code returned by Box |
| error.message |
The error message returned by Box |
## See also
- [org.apache.nifi.processors.box.FetchBoxFile](/user-guide/data-integration/openflow/processors/fetchboxfile)
- [org.apache.nifi.processors.box.FetchBoxFileInfo](/user-guide/data-integration/openflow/processors/fetchboxfileinfo)
- [org.apache.nifi.processors.box.ListBoxFile](/user-guide/data-integration/openflow/processors/listboxfile)
---
title: ListBoxFileMetadataTemplates 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listboxfilemetadatatemplates.md
section: Loading & Unloading Data
---
# ListBoxFileMetadataTemplates 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-box-nar
## Description
Retrieves all metadata templates associated with a Box file.
## Tags
box, metadata, storage, templates
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Box Client Service |
Controller Service used to obtain a Box API connection. |
| File ID |
The ID of the file for which to fetch metadata. |
## Relationships
| Name |
Description |
| failure |
A FlowFile will be routed here if there is an error fetching metadata templates from the file. |
| not found |
FlowFiles for which the specified Box file was not found will be routed to this relationship. |
| success |
A FlowFile containing the metadata template records will be routed to this relationship upon successful processing. |
## Writes attributes
| Name |
Description |
| box.file.id |
The ID of the file from which metadata was fetched |
| record.count |
The number of records in the FlowFile |
| mime.type |
The MIME Type specified by the Record Writer |
| box.metadata.templates.names |
Comma-separated list of template names |
| box.metadata.templates.count |
Number of metadata templates found |
| error.code |
The error code returned by Box |
| error.message |
The error message returned by Box |
## See also
- [org.apache.nifi.processors.box.FetchBoxFile](/user-guide/data-integration/openflow/processors/fetchboxfile)
- [org.apache.nifi.processors.box.FetchBoxFileInfo](/user-guide/data-integration/openflow/processors/fetchboxfileinfo)
- [org.apache.nifi.processors.box.ListBoxFile](/user-guide/data-integration/openflow/processors/listboxfile)
---
title: ListConfluenceGroups 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listconfluencegroups.md
section: Loading & Unloading Data
---
# ListConfluenceGroups 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-atlassian-processors-nar
## Description
Processor listing Confluence groups.
## Tags
Preview, atlassian, confluence, groups
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Confluence Client Service |
Controller service for managing connections to Confluence |
## Relationships
| Name |
Description |
| retry |
Retryable failure occurred, e.g. rate limiting |
| success |
Successfully fetched Confluence group page |
## Writes attributes
| Name |
Description |
| confluence.group.ids |
List of identifiers of the Confluence groups. |
---
title: ListDatabaseTables 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listdatabasetables.md
section: Loading & Unloading Data
---
# ListDatabaseTables 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Generates a set of flow files, each containing attributes corresponding to metadata about a table from a database connection. Once metadata about a table has been fetched, it will not be fetched again until the Refresh Interval (if set) has elapsed, or until state has been manually cleared.
## Tags
database, jdbc, list, sql, table
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| list-db-include-count |
Whether to include the table's row count as a flow file attribute. This affects performance as a database query will be generated for each table in the retrieved list. |
| list-db-refresh-interval |
The amount of time to elapse before resetting the processor state, thereby causing all current tables to be listed. During this interval, the processor may continue to run, but tables that have already been listed will not be re-listed. However new/added tables will be listed as the processor runs. A value of zero means the state will never be automatically reset, the user must Clear State manually. |
| list-db-tables-catalog |
The name of a catalog from which to list database tables. The name must match the catalog name as it is stored in the database. If the property is not set, the catalog name will not be used to narrow the search for tables. If the property is set to an empty string, tables without a catalog will be listed. |
| list-db-tables-db-connection |
The Controller Service that is used to obtain connection to database |
| list-db-tables-name-pattern |
A pattern for matching tables in the database. Within a pattern, "%" means match any substring of 0 or more characters, and "_" means match any one character. The pattern must match the table name as it is stored in the database. If the property is not set, all tables will be retrieved. |
| list-db-tables-schema-pattern |
A pattern for matching schemas in the database. Within a pattern, "%" means match any substring of 0 or more characters, and "_" means match any one character. The pattern must match the schema name as it is stored in the database. If the property is not set, the schema name will not be used to narrow the search for tables. If the property is set to an empty string, tables without a schema will be listed. |
| list-db-tables-types |
A comma-separated list of table types to include. For example, some databases support TABLE and VIEW types. If the property is not set, tables of all types will be returned. |
| record-writer |
Specifies the Record Writer to use for creating the listing. If not specified, one FlowFile will be created for each entity that is listed. If the Record Writer is specified, all entities will be written to a single FlowFile instead of adding attributes to individual FlowFiles. |
## State management
| Scopes |
Description |
| CLUSTER |
After performing a listing of tables, the timestamp of the query is stored. This allows the Processor to not re-list tables the next time that the Processor is run. Specifying the refresh interval in the processor properties will indicate that when the processor detects the interval has elapsed, the state will be reset and tables will be re-listed as a result. This processor is meant to be run on the primary node only. |
## Relationships
| Name |
Description |
| success |
All FlowFiles that are received are routed to success |
## Writes attributes
| Name |
Description |
| db.table.name |
Contains the name of a database table from the connection |
| db.table.catalog |
Contains the name of the catalog to which the table belongs (may be null) |
| db.table.schema |
Contains the name of the schema to which the table belongs (may be null) |
| db.table.fullname |
Contains the fully-qualified table name (possibly including catalog, schema, etc.) |
| db.table.type |
Contains the type of the database table from the connection. Typical types are "TABLE", "VIEW", "SYSTEM TABLE", "GLOBAL TEMPORARY", "LOCAL TEMPORARY", "ALIAS", "SYNONYM" |
| db.table.remarks |
Contains the name of a database table from the connection |
| db.table.count |
Contains the number of rows in the table |
## Use Cases Involving Other Components
| Perform a full load of a database, retrieving all rows from all tables, or a specific set of tables. |
| ---------------------------------------------------------------------------------------------------- |
---
title: ListDBFSDirectory 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listdbfsdirectory.md
section: Loading & Unloading Data
---
# ListDBFSDirectory 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-databricks-processors-nar
## Description
List file names in a DBFS directory and output a new FlowFile with the filename.
## Tags
databricks, dbfs, openflow
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| DBFS File Path |
DBFS file path e.g. /directory/file.txt |
| Databricks Client |
Databricks Client Service. |
| Include Directories |
Include directories in FlowFiles produced. |
| Recursive Directory Listing |
Recursively list files in sub directories. |
## Relationships
| Name |
Description |
| failure |
Databricks failure relationship |
| original |
The original FlowFile is routed to this relationship when processing is successful. |
| success |
Databricks success relationship |
## Writes attributes
| Name |
Description |
| filename |
Base filename of the DBFS file or directory. |
| path |
Path to parent directory containing the DBFS file or directory. |
| absolute.path |
Full path to the DBFS file or directory. |
| dbfs.resourceType |
The type of resource, 'file' or 'directory' of the DBFS resource. |
| dbfs.size |
The size of the DBFS file. |
| dbfs.lastModifiedTime |
The last modified time of the DBFS file, in milliseconds since epoch in UTC time. |
| error.code |
The error code for the SQL statement if an error occurred. |
| error.message |
The error message for the SQL statement if an error occurred. |
---
title: ListDropbox 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listdropbox.md
section: Loading & Unloading Data
---
# ListDropbox 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-dropbox-processors-nar
## Description
Retrieves a listing of files from Dropbox (shortcuts are ignored). Each listed file may result in one FlowFile, the metadata being written as FlowFile attributes. When the 'Record Writer' property is set, the entire result is written as records to a single FlowFile. This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left off without duplicating all of the data.
## Tags
dropbox, storage
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Dropbox Credential Service |
Controller Service used to obtain Dropbox credentials (App Key, App Secret, Access Token, Refresh Token). See controller service's Additional Details for more information. |
| Entity Tracking Initial Listing Target |
Specify how initial listing should be handled. Used by 'Tracking Entities'strategy. |
| Entity Tracking State Cache |
Listed entities are stored in the specified cache storage so that this processor can resume listing across NiFi restart or in case of primary node change. 'Tracking Entities'strategy require tracking information of all listed entities within the last 'Tracking Time Window'. To support large number of entities, the strategy uses DistributedMapCache instead of managed state. Cache key format is 'ListedEntities::\{processorId\}(::\{nodeId\})'. If it tracks per node listed entities, then the optional '::\{nodeId\}' part is added to manage state separately. E.g. cluster wide cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b', per node cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3' The stored cache content is Gzipped JSON string. The cache key will be deleted when target listing configuration is changed. Used by 'Tracking Entities'strategy. |
| Entity Tracking Time Window |
Specify how long this processor should track already-listed entities. 'Tracking Entities'strategy can pick any entity whose timestamp is inside the specified time window. For example, if set to '30 minutes', any entity having timestamp in recent 30 minutes will be the listing target when this processor runs. A listed entity is considered 'new/updated' and a FlowFile is emitted if one of following condition meets: 1. does not exist in the already-listed entities, 2. has newer timestamp than the cached entity, 3. has different size than the cached entity. If a cached entity 's timestamp becomes older than specified time window, that entity will be removed from the cached already-listed entities. Used by'Tracking Entities'strategy. |
| Folder |
The Dropbox identifier or path of the folder from which to pull list of files. 'Folder'should match the following regular expression pattern: /.*|id:.* . Example for folder identifier: id:odTlUvbpIEAAAAAAAAAGGQ. Example for folder path: /Team1/Task1. |
| Listing Strategy |
Specify how to determine new/updated entities. See each strategy descriptions for detail. |
| Minimum File Age |
The minimum age a file must be in order to be considered; any files newer than this will be ignored. |
| Record Writer |
Specifies the Record Writer to use for creating the listing. If not specified, one FlowFile will be created for each entity that is listed. If the Record Writer is specified, all entities will be written to a single FlowFile instead of adding attributes to individual FlowFiles. |
| Search Recursively |
Indicates whether to list files from subfolders of the Dropbox folder. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## State management
| Scopes |
Description |
| CLUSTER |
The processor stores necessary data to be able to keep track what files have been listed already. What exactly needs to be stored depends on the 'Listing Strategy'. |
## Relationships
| Name |
Description |
| success |
All FlowFiles that are received are routed to success |
## Writes attributes
| Name |
Description |
| dropbox.id |
The Dropbox identifier of the file |
| path |
The folder path where the file is located |
| filename |
The name of the file |
| dropbox.size |
The size of the file |
| dropbox.timestamp |
The server modified time of the file |
| dropbox.revision |
Revision of the file |
## See also
- [org.apache.nifi.processors.dropbox.FetchDropbox](/user-guide/data-integration/openflow/processors/fetchdropbox)
- [org.apache.nifi.processors.dropbox.PutDropbox](/user-guide/data-integration/openflow/processors/putdropbox)
---
title: ListenFTP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listenftp.md
section: Loading & Unloading Data
---
# ListenFTP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Starts an FTP server that listens on the specified port and transforms incoming files into FlowFiles. The URI of the service will be ftp://\{hostname\}:\{port\}. The default port is 2221.
## Tags
FTP, FTPS, ingest, listen
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Address |
The address the FTP server should be bound to. If not set (or set to 0.0.0.0), the server binds to all available addresses (i.e. all network interfaces of the host machine). |
| Password |
If the Username is set, then a password must also be specified. The password provided by the client trying to log in to the FTP server will be checked against this password. |
| Port |
The Port to listen on for incoming connections. On Linux, root privileges are required to use port numbers below 1024. |
| SSL Context Service |
Specifies the SSL Context Service that can be used to create secure connections. If an SSL Context Service is selected, then a keystore file must also be specified in the SSL Context Service. Without a keystore file, the processor cannot be started successfully. Specifying a truststore file is optional. If a truststore file is specified, client authentication is required (the client needs to send a certificate to the server).Regardless of the selected TLS protocol, the highest available protocol is used for the connection. For example if NiFi is running on Java 11 and TLSv1.2 is selected in the controller service as the preferred TLS Protocol, TLSv1.3 will be used (regardless of TLSv1.2 being selected) because Java 11 supports TLSv1.3. |
| Username |
The name of the user that is allowed to log in to the FTP server. If a username is provided, a password must also be provided. If no username is specified, anonymous connections will be permitted. |
## Relationships
| Name |
Description |
| success |
Relationship for successfully received files. |
## Writes attributes
| Name |
Description |
| filename |
The name of the file received via the FTP/FTPS connection. |
| path |
The path pointing to the file's target directory. E.g.: file.txt is uploaded to /Folder1/SubFolder, then the value of the path attribute will be "/Folder1/SubFolder/" (note that it ends with a separator character). |
---
title: ListenHTTP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listenhttp.md
section: Loading & Unloading Data
---
# ListenHTTP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Starts an HTTP Server and listens on a given base path to transform incoming requests into FlowFiles. The default URI of the Service will be http://\{hostname\}:\{port\}/contentListener. Only HEAD and POST requests are supported. GET, PUT, DELETE, OPTIONS and TRACE will result in an error and the HTTP response status code 405; CONNECT will also result in an error and the HTTP response status code 400. GET is supported on <service_URI>/healthcheck. If the service is available, it returns "200 OK" with the content "OK". The health check functionality can be configured to be accessible via a different port. For details, see the documentation of the "Listening Port for health check requests" property. A Record Reader and Record Writer property can be enabled on the processor to process incoming requests as records. Record processing is not allowed for multipart requests and request in FlowFileV3 format (minifi). If the incoming request contains a FlowFileV3 package format, the data will be unpacked automatically into individual FlowFile(s) contained within the package; the original FlowFile names are restored.
## Tags
http, https, ingest, listen, rest
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Authorized DN Pattern |
A Regular Expression to apply against the Subject's Distinguished Name of incoming connections. If the Pattern does not match the Subject DN, the processor will respond with a status of HTTP 403 Forbidden. |
| Base Path |
Base path for incoming connections |
| HTTP Headers to receive as Attributes (Regex) |
Specifies the Regular Expression that determines the names of HTTP Headers that should be passed along as FlowFile attributes |
| HTTP Protocols |
HTTP Protocols supported for Application Layer Protocol Negotiation with TLS |
| Listening Port |
The Port to listen on for incoming connections |
| Max Unconfirmed Flowfile Time |
The maximum amount of time to wait for a FlowFile to be confirmed before it is removed from the cache |
| Request Header Maximum Size |
The maximum supported size of HTTP headers in requests sent to this processor |
| Return Code |
The HTTP return code returned after every HTTP call |
| SSL Context Service |
SSL Context Service enables support for HTTPS |
| authorized-issuer-dn-pattern |
A Regular Expression to apply against the Issuer's Distinguished Name of incoming connections. If the Pattern does not match the Issuer DN, the processor will respond with a status of HTTP 403 Forbidden. |
| client-authentication |
Client Authentication policy for TLS connections. Required when SSL Context Service configured. |
| health-check-port |
The port to listen on for incoming health check requests. If set, it must be different from the Listening Port. Configure this port if the processor is set to use two-way SSL and a load balancer that does not support client authentication for health check requests is used. Only /<base_path>/healthcheck service is available via this port and only GET and HEAD requests are supported. If the processor is set not to use SSL, SSL will not be used on this port, either. If the processor is set to use one-way SSL, one-way SSL will be used on this port. If the processor is set to use two-way SSL, one-way SSL will be used on this port (client authentication not required). |
| max-thread-pool-size |
The maximum number of threads to be used by the embedded Jetty server. The value can be set between 8 and 1000. The value of this property affects the performance of the flows and the operating system, therefore the default value should only be changed in justified cases. A value that is less than the default value may be suitable if only a small number of HTTP clients connect to the server. A greater value may be suitable if a large number of HTTP clients are expected to make requests to the server simultaneously. |
| multipart-read-buffer-size |
The threshold size, at which the contents of an incoming file would be written to disk. Only applies for requests with Content-Type: multipart/form-data. It is used to prevent denial of service type of attacks, to prevent filling up the heap or disk space. |
| multipart-request-max-size |
The max size of the request. Only applies for requests with Content-Type: multipart/form-data, and is used to prevent denial of service type of attacks, to prevent filling up the heap or disk space |
| record-reader |
The Record Reader to use parsing the incoming FlowFile into Records |
| record-writer |
The Record Writer to use for serializing Records after they have been transformed |
## Relationships
| Name |
Description |
| success |
Relationship for successfully received FlowFiles |
## Use cases
| Unpack FlowFileV3 content received in a POST |
| -------------------------------------------- |
## Use Cases Involving Other Components
| Limit the date flow rate that is accepted |
| ----------------------------------------- |
---
title: ListenOTLP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listenotlp.md
section: Loading & Unloading Data
---
# ListenOTLP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-opentelemetry-nar
## Description
Collect OpenTelemetry messages over HTTP or gRPC. Supports standard Export Service Request messages for logs, metrics, and traces. Implements OpenTelemetry OTLP Specification 1.0.0 with OTLP/gRPC and OTLP/HTTP. Provides protocol detection using the HTTP Content-Type header.
## Tags
OTLP, OTel, OpenTelemetry, logs, metrics, telemetry, traces
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Address |
Internet Protocol Address on which to listen for OTLP Export Service Requests. The default value enables listening on all addresses. |
| Batch Size |
Maximum number of OTLP request resource elements included in each FlowFile produced |
| Client Authentication |
Client authentication policy for TLS communication with HTTPS |
| Port |
TCP port number on which to listen for OTLP Export Service Requests over HTTP and gRPC |
| Queue Capacity |
Maximum number of OTLP request resource elements that can be received and queued |
| SSL Context Service |
SSL Context Service enables TLS communication for HTTPS |
| Worker Threads |
Number of threads responsible for decoding and queuing incoming OTLP Export Service Requests |
## Relationships
| Name |
Description |
| success |
Export Service Requests containing OTLP Telemetry |
## Writes attributes
| Name |
Description |
| mime.type |
Content-Type set to application/json |
| resource.type |
OpenTelemetry Resource Type: LOGS, METRICS, or TRACES |
| resource.count |
Count of resource elements included in messages |
---
title: ListenSlack 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listenslack.md
section: Loading & Unloading Data
---
# ListenSlack 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-slack-nar
## Description
Retrieves real-time messages or Slack commands from one or more Slack conversations. The messages are written out in JSON format. Note that this Processor should be used to obtain real-time messages and commands from Slack and does not provide a mechanism for obtaining historical messages. The ConsumeSlack Processor should be used for an initial load of messages from a channel. See Usage / Additional Details for more information about how to configure this Processor and enable it to retrieve messages and commands from Slack.
## Tags
command, event, listen, message, real-time, receive, slack, social media, team, text, unstructured
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| App Token |
The Application Token that is registered to your Slack application |
| Bot Token |
The Bot Token that is registered to your Slack application |
| Event Type to Receive |
Specifies the type of Event that the Processor should respond to |
| Resolve User Details |
Specifies whether the Processor should lookup details about the Slack User who sent the received message. If true, the output JSON will contain an additional field named 'userDetails'. The 'user' field will still contain the ID of the user. In order to enable this capability, the Bot Token must be granted the 'users:read' and optionally the 'users.profile:read' Bot Token Scope. If the rate limit is exceeded when retrieving this information, the received message will be rejected and must be re-delivered. |
## Relationships
| Name |
Description |
| success |
All FlowFiles that are created will be sent to this Relationship. |
## Writes attributes
| Name |
Description |
| mime.type |
Set to application/json, as the output will always be in JSON format |
| slack.event.type |
Set to the type of Slack event that occurred |
## See also
- [org.apache.nifi.processors.slack.ConsumeSlack](/user-guide/data-integration/openflow/processors/consumeslack)
---
title: ListenSyslog 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listensyslog.md
section: Loading & Unloading Data
---
# ListenSyslog 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Listens for Syslog messages being sent to a given port over TCP or UDP. Incoming messages are checked against regular expressions for RFC5424 and RFC3164 formatted messages. The format of each message is: (<PRIORITY>)(VERSION )(TIMESTAMP) (HOSTNAME) (BODY) where version is optional. The timestamp can be an RFC5424 timestamp with a format of "yyyy-MM-dd 'T'HH:mm:ss. SZ" or "yyyy-MM-dd 'T'HH:mm:ss. S+hh:mm", or it can be an RFC3164 timestamp with a format of "MMM d HH:mm:ss". If an incoming messages matches one of these patterns, the message will be parsed and the individual pieces will be placed in FlowFile attributes, with the original message in the content of the FlowFile. If an incoming message does not match one of these patterns it will not be parsed and the syslog.valid attribute will be set to false with the original message in the content of the FlowFile. Valid messages will be transferred on the success relationship, and invalid messages will be transferred on the invalid relationship.
## Tags
listen, logs, syslog, tcp, udp
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Character Set |
Specifies the character set of the Syslog messages. Note that Expression language is not evaluated per FlowFile. |
| Client Auth |
The client authentication policy to use for the SSL Context. Only used if an SSL Context Service is provided. |
| Local Network Interface |
The name of a local network interface to be used to restrict listening to a specific LAN. |
| Max Batch Size |
The maximum number of Syslog events to add to a single FlowFile. If multiple events are available, they will be concatenated along with the <Message Delimiter> up to this configured maximum number of messages |
| Max Size of Message Queue |
The maximum size of the internal queue used to buffer messages being transferred from the underlying channel to the processor. Setting this value higher allows more messages to be buffered in memory during surges of incoming messages, but increases the total memory used by the processor. |
| Max Size of Socket Buffer |
The maximum size of the socket buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped. |
| Message Delimiter |
Specifies the delimiter to place between Syslog messages when multiple messages are bundled together (see <Max Batch Size> property). |
| Parse Messages |
Indicates if the processor should parse the Syslog messages. If set to false, each outgoing FlowFile will only contain the sender, protocol, and port, and no additional attributes. |
| Port |
The port for Syslog communication. Note that Expression language is not evaluated per FlowFile. |
| Protocol |
The protocol for Syslog communication. |
| Receive Buffer Size |
The size of each buffer used to receive Syslog messages. Adjust this value appropriately based on the expected size of the incoming Syslog messages. When UDP is selected each buffer will hold one Syslog message. When TCP is selected messages are read from an incoming connection until the buffer is full, or the connection is closed. |
| SSL Context Service |
The Controller Service to use in order to obtain an SSL Context. If this property is set, syslog messages will be received over a secure connection. |
| Socket Keep Alive |
Whether or not to have TCP socket keep alive turned on. Timing details depend on operating system properties. |
| Worker Threads |
Number of threads responsible for decoding and queuing incoming syslog messages |
## Relationships
| Name |
Description |
| invalid |
Syslog messages that do not match one of the expected formats will be sent out this relationship as a FlowFile per message. |
| success |
Syslog messages that match one of the expected formats will be sent out this relationship as a FlowFile per message. |
## Writes attributes
| Name |
Description |
| syslog.priority |
The priority of the Syslog message. |
| syslog.severity |
The severity of the Syslog message derived from the priority. |
| syslog.facility |
The facility of the Syslog message derived from the priority. |
| syslog.version |
The optional version from the Syslog message. |
| syslog.timestamp |
The timestamp of the Syslog message. |
| syslog.hostname |
The hostname or IP address of the Syslog message. |
| syslog.sender |
The hostname of the Syslog server that sent the message. |
| syslog.body |
The body of the Syslog message, everything after the hostname. |
| syslog.valid |
An indicator of whether this message matched the expected formats. If this value is false, the other attributes will be empty and only the original message will be available in the content. |
| syslog.protocol |
The protocol over which the Syslog message was received. |
| syslog.port |
The port over which the Syslog message was received. |
| mime.type |
The mime.type of the FlowFile which will be text/plain for Syslog messages. |
## See also
- [org.apache.nifi.processors.standard.ParseSyslog](/user-guide/data-integration/openflow/processors/parsesyslog)
- [org.apache.nifi.processors.standard.PutSyslog](/user-guide/data-integration/openflow/processors/putsyslog)
---
title: ListenTCP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listentcp.md
section: Loading & Unloading Data
---
# ListenTCP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Listens for incoming TCP connections and reads data from each connection using a line separator as the message demarcator. The default behavior is for each message to produce a single FlowFile, however this can be controlled by increasing the Batch Size to a larger value for higher throughput. The Receive Buffer Size must be set as large as the largest messages expected to be received, meaning if every 100kb there is a line separator, then the Receive Buffer Size must be greater than 100kb. The processor can be configured to use an SSL Context Service to only allow secure connections. When connected clients present certificates for mutual TLS authentication, the Distinguished Names of the client certificate's issuer and subject are added to the outgoing FlowFiles as attributes. The processor does not perform authorization based on Distinguished Name values, but since these values are attached to the outgoing FlowFiles, authorization can be implemented based on these attributes.
## Tags
listen, ssl, tcp, tls
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Batching Message Delimiter |
Specifies the delimiter to place between messages when multiple messages are bundled together (see <Max Batch Size> property). |
| Character Set |
Specifies the character set of the received data. |
| Client Auth |
The client authentication policy to use for the SSL Context. Only used if an SSL Context Service is provided. |
| Local Network Interface |
The name of a local network interface to be used to restrict listening to a specific LAN. |
| Max Batch Size |
The maximum number of messages to add to a single FlowFile. If multiple messages are available, they will be concatenated along with the <Message Delimiter> up to this configured maximum number of messages |
| Max Size of Message Queue |
The maximum size of the internal queue used to buffer messages being transferred from the underlying channel to the processor. Setting this value higher allows more messages to be buffered in memory during surges of incoming messages, but increases the total memory used by the processor during these surges. |
| Max Size of Socket Buffer |
The maximum size of the socket buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped. |
| Port |
The port to listen on for communication. |
| Receive Buffer Size |
The size of each buffer used to receive messages. Adjust this value appropriately based on the expected size of the incoming messages. |
| SSL Context Service |
The Controller Service to use in order to obtain an SSL Context. If this property is set, messages will be received over a secure connection. |
| Worker Threads |
The maximum number of worker threads available for servicing TCP connections. |
| idle-timeout |
The amount of time a client's connection will remain open if no data is received. The default of 0 seconds will leave connections open until they are closed by the client. |
| pool-receive-buffers |
Enable or disable pooling of buffers that the processor uses for handling bytes received on socket connections. The framework allocates buffers as needed during processing. |
## Relationships
| Name |
Description |
| success |
Messages received successfully will be sent out this relationship. |
## Writes attributes
| Name |
Description |
| tcp.sender |
The sending host of the messages. |
| tcp.port |
The sending port the messages were received. |
| client.certificate.issuer.dn |
For connections using mutual TLS, the Distinguished Name of the Certificate Authority that issued the client's certificate is attached to the FlowFile. |
| client.certificate.subject.dn |
For connections using mutual TLS, the Distinguished Name of the client certificate's owner (subject) is attached to the FlowFile. |
---
title: ListenUDP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listenudp.md
section: Loading & Unloading Data
---
# ListenUDP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Listens for Datagram Packets on a given port. The default behavior produces a FlowFile per datagram, however for higher throughput the Max Batch Size property may be increased to specify the number of datagrams to batch together in a single FlowFile. This processor can be restricted to listening for datagrams from a specific remote host and port by specifying the Sending Host and Sending Host Port properties, otherwise it will listen for datagrams from all hosts and ports.
## Tags
ingest, listen, source, udp
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Batching Message Delimiter |
Specifies the delimiter to place between messages when multiple messages are bundled together (see <Max Batch Size> property). |
| Character Set |
Specifies the character set of the received data. |
| Local Network Interface |
The name of a local network interface to be used to restrict listening to a specific LAN. |
| Max Batch Size |
The maximum number of messages to add to a single FlowFile. If multiple messages are available, they will be concatenated along with the <Message Delimiter> up to this configured maximum number of messages |
| Max Size of Message Queue |
The maximum size of the internal queue used to buffer messages being transferred from the underlying channel to the processor. Setting this value higher allows more messages to be buffered in memory during surges of incoming messages, but increases the total memory used by the processor. |
| Max Size of Socket Buffer |
The maximum size of the socket buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped. |
| Port |
The port to listen on for communication. |
| Receive Buffer Size |
The size of each buffer used to receive messages. Adjust this value appropriately based on the expected size of the incoming messages. |
| Sending Host |
IP, or name, of a remote host. Only Datagrams from the specified Sending Host Port and this host will be accepted. Improves Performance. May be a system property or an environment variable. |
| Sending Host Port |
Port being used by remote host to send Datagrams. Only Datagrams from the specified Sending Host and this port will be accepted. Improves Performance. May be a system property or an environment variable. |
## Relationships
| Name |
Description |
| success |
Messages received successfully will be sent out this relationship. |
## Writes attributes
| Name |
Description |
| udp.sender |
The sending host of the messages. |
| udp.port |
The sending port the messages were received. |
---
title: ListenUDPRecord 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listenudprecord.md
section: Loading & Unloading Data
---
# ListenUDPRecord 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Listens for Datagram Packets on a given port and reads the content of each datagram using the configured Record Reader. Each record will then be written to a flow file using the configured Record Writer. This processor can be restricted to listening for datagrams from a specific remote host and port by specifying the Sending Host and Sending Host Port properties, otherwise it will listen for datagrams from all hosts and ports.
## Tags
ingest, listen, record, source, udp
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Character Set |
Specifies the character set of the received data. |
| Local Network Interface |
The name of a local network interface to be used to restrict listening to a specific LAN. |
| Max Size of Message Queue |
The maximum size of the internal queue used to buffer messages being transferred from the underlying channel to the processor. Setting this value higher allows more messages to be buffered in memory during surges of incoming messages, but increases the total memory used by the processor. |
| Max Size of Socket Buffer |
The maximum size of the socket buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped. |
| Port |
The port to listen on for communication. |
| Receive Buffer Size |
The size of each buffer used to receive messages. Adjust this value appropriately based on the expected size of the incoming messages. |
| batch-size |
The maximum number of datagrams to write as records to a single FlowFile. The Batch Size will only be reached when data is coming in more frequently than the Poll Timeout. |
| poll-timeout |
The amount of time to wait when polling the internal queue for more datagrams. If no datagrams are found after waiting for the configured timeout, then the processor will emit whatever records have been obtained up to that point. |
| record-reader |
The Record Reader to use for reading the content of incoming datagrams. |
| record-writer |
The Record Writer to use in order to serialize the data before writing to a flow file. |
| sending-host |
IP, or name, of a remote host. Only Datagrams from the specified Sending Host Port and this host will be accepted. Improves Performance. May be a system property or an environment variable. |
| sending-host-port |
Port being used by remote host to send Datagrams. Only Datagrams from the specified Sending Host and this port will be accepted. Improves Performance. May be a system property or an environment variable. |
## Relationships
| Name |
Description |
| parse.failure |
If a datagram cannot be parsed using the configured Record Reader, the contents of the message will be routed to this Relationship as its own individual FlowFile. |
| success |
Messages received successfully will be sent out this relationship. |
## Writes attributes
| Name |
Description |
| udp.sender |
The sending host of the messages. |
| udp.port |
The sending port the messages were received. |
| record.count |
The number of records written to the flow file. |
| mime.type |
The mime-type of the writer used to write the records to the flow file. |
---
title: ListenWebSocket 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listenwebsocket.md
section: Loading & Unloading Data
---
# ListenWebSocket 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-websocket-processors-nar
## Description
Acts as a WebSocket server endpoint to accept client connections. FlowFiles are transferred to downstream relationships according to received message types as the WebSocket server configured with this processor receives client requests
## Tags
WebSocket, consume, listen, subscribe
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| server-url-path |
The WebSocket URL Path on which this processor listens to. Must starts with '/', e.g. '/example'. |
| websocket-server-controller-service |
A WebSocket SERVER Controller Service which can accept WebSocket requests. |
## Relationships
| Name |
Description |
| binary message |
The WebSocket binary message output |
| connected |
The WebSocket session is established |
| disconnected |
The WebSocket session is disconnected |
| text message |
The WebSocket text message output |
## Writes attributes
| Name |
Description |
| websocket.controller.service.id |
WebSocket Controller Service id. |
| websocket.session.id |
Established WebSocket session id. |
| websocket.endpoint.id |
WebSocket endpoint id. |
| websocket.local.address |
WebSocket server address. |
| websocket.remote.address |
WebSocket client address. |
| websocket.message.type |
TEXT or BINARY. |
---
title: ListFile 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listfile.md
section: Loading & Unloading Data
---
# ListFile 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Retrieves a listing of files from the input directory. For each file listed, creates a FlowFile that represents the file so that it can be fetched in conjunction with FetchFile. This Processor is designed to run on Primary Node only in a cluster when 'Input Directory Location' is set to 'Remote'. If the primary node changes, the new Primary Node will pick up where the previous node left off without duplicating all the data. When 'Input Directory Location' is 'Local', the 'Execution' mode can be anything, and synchronization won't happen. Unlike GetFile, this Processor does not delete any data from the local filesystem.
## Tags
file, filesystem, get, ingest, list, source
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Entity Tracking Initial Listing Target |
Specify how initial listing should be handled. Used by 'Tracking Entities'strategy. |
| Entity Tracking Node Identifier |
The configured value will be appended to the cache key so that listing state can be tracked per NiFi node rather than cluster wide when tracking state is scoped to LOCAL. Used by 'Tracking Entities'strategy. |
| Entity Tracking State Cache |
Listed entities are stored in the specified cache storage so that this processor can resume listing across NiFi restart or in case of primary node change. 'Tracking Entities'strategy require tracking information of all listed entities within the last 'Tracking Time Window'. To support large number of entities, the strategy uses DistributedMapCache instead of managed state. Cache key format is 'ListedEntities::\{processorId\}(::\{nodeId\})'. If it tracks per node listed entities, then the optional '::\{nodeId\}' part is added to manage state separately. E.g. cluster wide cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b', per node cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3' The stored cache content is Gzipped JSON string. The cache key will be deleted when target listing configuration is changed. Used by 'Tracking Entities'strategy. |
| Entity Tracking Time Window |
Specify how long this processor should track already-listed entities. 'Tracking Entities'strategy can pick any entity whose timestamp is inside the specified time window. For example, if set to '30 minutes', any entity having timestamp in recent 30 minutes will be the listing target when this processor runs. A listed entity is considered 'new/updated' and a FlowFile is emitted if one of following condition meets: 1. does not exist in the already-listed entities, 2. has newer timestamp than the cached entity, 3. has different size than the cached entity. If a cached entity 's timestamp becomes older than specified time window, that entity will be removed from the cached already-listed entities. Used by'Tracking Entities'strategy. |
| File Filter |
Only files whose names match the given regular expression will be picked up |
| Ignore Hidden Files |
Indicates whether or not hidden files should be ignored |
| Include File Attributes |
Whether or not to include information such as the file's Last Modified Time and Owner as FlowFile Attributes. Depending on the File System being used, gathering this information can be expensive and as a result should be disabled. This is especially true of remote file shares. |
| Input Directory |
The input directory from which files to pull files |
| Input Directory Location |
Specifies where the Input Directory is located. This is used to determine whether state should be stored locally or across the cluster. |
| Listing Strategy |
Specify how to determine new/updated entities. See each strategy descriptions for detail. |
| Maximum File Age |
The maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored |
| Maximum File Size |
The maximum size that a file can be in order to be pulled |
| Minimum File Age |
The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored |
| Minimum File Size |
The minimum size that a file must be in order to be pulled |
| Path Filter |
When Recurse Subdirectories is true, then only subdirectories whose path matches the given regular expression will be scanned |
| Record Writer |
Specifies the Record Writer to use for creating the listing. If not specified, one FlowFile will be created for each entity that is listed. If the Record Writer is specified, all entities will be written to a single FlowFile instead of adding attributes to individual FlowFiles. |
| Recurse Subdirectories |
Indicates whether to list files from subdirectories of the directory |
| Target System Timestamp Precision |
Specify timestamp precision at the target system. Since this processor uses timestamp of entities to decide which should be listed, it is crucial to use the right timestamp precision. |
| max-listing-time |
The maximum amount of time that listing any single directory is expected to take. If the listing for the directory specified by the 'Input Directory' property, or the listing of any subdirectory (if 'Recurse' is set to true) takes longer than this amount of time, a warning bulletin will be generated for each directory listing that exceeds this amount of time. |
| max-operation-time |
The maximum amount of time that any single disk operation is expected to take. If any disk operation takes longer than this amount of time, a warning bulletin will be generated for each operation that exceeds this amount of time. |
| max-performance-metrics |
If the 'Track Performance' property is set to 'true', this property indicates the maximum number of files whose performance metrics should be held onto. A smaller value for this property will result in less heap utilization, while a larger value may provide more accurate insights into how the disk access operations are performing |
| track-performance |
Whether or not the Processor should track the performance of disk access operations. If true, all accesses to disk will be recorded, including the file being accessed, the information being obtained, and how long it takes. This is then logged periodically at a DEBUG level. While the amount of data will be capped, this option may still consume a significant amount of heap (controlled by the 'Maximum Number of Files to Track' property), but it can be very useful for troubleshooting purposes if performance is poor is degraded. |
## State management
| Scopes |
Description |
| LOCAL |
After performing a listing of files, the timestamp of the newest file is stored. This allows the Processor to list only files that have been added or modified after this date the next time that the Processor is run. Whether the state is stored with a Local or Cluster scope depends on the value of the <Input Directory Location> property. |
| CLUSTER |
After performing a listing of files, the timestamp of the newest file is stored. This allows the Processor to list only files that have been added or modified after this date the next time that the Processor is run. Whether the state is stored with a Local or Cluster scope depends on the value of the <Input Directory Location> property. |
## Relationships
| Name |
Description |
| success |
All FlowFiles that are received are routed to success |
## Writes attributes
| Name |
Description |
| filename |
The name of the file that was read from filesystem. |
| path |
The path is set to the relative path of the file's directory on filesystem compared to the Input Directory property. For example, if Input Directory is set to /tmp, then files picked up from /tmp will have the path attribute set to "/". If the Recurse Subdirectories property is set to true and a file is picked up from /tmp/abc/1/2/3, then the path attribute will be set to "abc/1/2/3/". |
| absolute.path |
The absolute.path is set to the absolute path of the file's directory on filesystem. For example, if the Input Directory property is set to /tmp, then files picked up from /tmp will have the path attribute set to "/tmp/". If the Recurse Subdirectories property is set to true and a file is picked up from /tmp/abc/1/2/3, then the path attribute will be set to "/tmp/abc/1/2/3/". |
| file.owner |
The user that owns the file in filesystem |
| file.group |
The group that owns the file in filesystem |
| file.size |
The number of bytes in the file in filesystem |
| file.permissions |
The permissions for the file in filesystem. This is formatted as 3 characters for the owner, 3 for the group, and 3 for other users. For example rw-rw-r– |
| file.lastModifiedTime |
The timestamp of when the file in filesystem was last modified as 'yyyy-MM-dd'T'HH:mm:ssZ' |
| file.lastAccessTime |
The timestamp of when the file in filesystem was last accessed as 'yyyy-MM-dd'T'HH:mm:ssZ' |
| file.creationTime |
The timestamp of when the file in filesystem was created as 'yyyy-MM-dd'T'HH:mm:ssZ' |
## See also
- [org.apache.nifi.processors.standard.FetchFile](/user-guide/data-integration/openflow/processors/fetchfile)
- [org.apache.nifi.processors.standard.GetFile](/user-guide/data-integration/openflow/processors/getfile)
- [org.apache.nifi.processors.standard.PutFile](/user-guide/data-integration/openflow/processors/putfile)
---
title: ListFTP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listftp.md
section: Loading & Unloading Data
---
# ListFTP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Performs a listing of the files residing on an FTP server. For each file that is found on the remote server, a new FlowFile will be created with the filename attribute set to the name of the file on the remote server. This can then be used in conjunction with FetchFTP in order to fetch those files.
## Tags
files, ftp, ingest, input, list, remote, source
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Connection Mode |
The FTP Connection Mode |
| Connection Timeout |
Amount of time to wait before timing out while creating a connection |
| Data Timeout |
When transferring a file between the local and remote system, this value specifies how long is allowed to elapse without any data being transferred between systems |
| Entity Tracking Initial Listing Target |
Specify how initial listing should be handled. Used by 'Tracking Entities'strategy. |
| Entity Tracking State Cache |
Listed entities are stored in the specified cache storage so that this processor can resume listing across NiFi restart or in case of primary node change. 'Tracking Entities'strategy require tracking information of all listed entities within the last 'Tracking Time Window'. To support large number of entities, the strategy uses DistributedMapCache instead of managed state. Cache key format is 'ListedEntities::\{processorId\}(::\{nodeId\})'. If it tracks per node listed entities, then the optional '::\{nodeId\}' part is added to manage state separately. E.g. cluster wide cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b', per node cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3' The stored cache content is Gzipped JSON string. The cache key will be deleted when target listing configuration is changed. Used by 'Tracking Entities'strategy. |
| Entity Tracking Time Window |
Specify how long this processor should track already-listed entities. 'Tracking Entities'strategy can pick any entity whose timestamp is inside the specified time window. For example, if set to '30 minutes', any entity having timestamp in recent 30 minutes will be the listing target when this processor runs. A listed entity is considered 'new/updated' and a FlowFile is emitted if one of following condition meets: 1. does not exist in the already-listed entities, 2. has newer timestamp than the cached entity, 3. has different size than the cached entity. If a cached entity 's timestamp becomes older than specified time window, that entity will be removed from the cached already-listed entities. Used by'Tracking Entities'strategy. |
| File Filter Regex |
Provides a Java Regular Expression for filtering Filenames; if a filter is supplied, only files whose names match that Regular Expression will be fetched |
| Follow Symbolic Links |
If true, will pull even symbolic files and also nested symbolic subdirectories; otherwise, will not read symbolic files and will not traverse symbolic link subdirectories |
| Hostname |
The fully qualified hostname or IP address of the remote system |
| Ignore Dotted Files |
If true, files whose names begin with a dot (".") will be ignored |
| Internal Buffer Size |
Set the internal buffer size for buffered data streams |
| Listing Strategy |
Specify how to determine new/updated entities. See each strategy descriptions for detail. |
| Password |
Password for the user account |
| Path Filter Regex |
When Search Recursively is true, then only subdirectories whose path matches the given Regular Expression will be scanned |
| Port |
The port to connect to on the remote host to fetch the data from |
| Record Writer |
Specifies the Record Writer to use for creating the listing. If not specified, one FlowFile will be created for each entity that is listed. If the Record Writer is specified, all entities will be written to a single FlowFile instead of adding attributes to individual FlowFiles. |
| Remote Path |
The path on the remote system from which to pull or push files |
| Remote Poll Batch Size |
The value specifies how many file paths to find in a given directory on the remote system when doing a file listing. This value in general should not need to be modified but when polling against a remote system with a tremendous number of files this value can be critical. Setting this value too high can result very poor performance and setting it too low can cause the flow to be slower than normal. |
| Search Recursively |
If true, will pull files from arbitrarily nested subdirectories; otherwise, will not traverse subdirectories |
| Target System Timestamp Precision |
Specify timestamp precision at the target system. Since this processor uses timestamp of entities to decide which should be listed, it is crucial to use the right timestamp precision. |
| Transfer Mode |
The FTP Transfer Mode |
| Username |
Username |
| ftp-use-utf8 |
Tells the client to use UTF-8 encoding when processing files and filenames. If set to true, the server must also support UTF-8 encoding. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## State management
| Scopes |
Description |
| CLUSTER |
After performing a listing of files, the timestamp of the newest file is stored. This allows the Processor to list only files that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node will not duplicate the data that was listed by the previous Primary Node. |
## Relationships
| Name |
Description |
| success |
All FlowFiles that are received are routed to success |
## Writes attributes
| Name |
Description |
| ftp.remote.host |
The hostname of the FTP Server |
| ftp.remote.port |
The port that was connected to on the FTP Server |
| ftp.listing.user |
The username of the user that performed the FTP Listing |
| file.owner |
The numeric owner id of the source file |
| file.group |
The numeric group id of the source file |
| file.permissions |
The read/write/execute permissions of the source file |
| file.size |
The number of bytes in the source file |
| file.lastModifiedTime |
The timestamp of when the file in the filesystem waslast modified as 'yyyy-MM-dd'T'HH:mm:ssZ' |
| filename |
The name of the file on the FTP Server |
| path |
The fully qualified name of the directory on the FTP Server from which the file was pulled |
## See also
- [org.apache.nifi.processors.standard.FetchFTP](/user-guide/data-integration/openflow/processors/fetchftp)
- [org.apache.nifi.processors.standard.GetFTP](/user-guide/data-integration/openflow/processors/getftp)
- [org.apache.nifi.processors.standard.PutFTP](/user-guide/data-integration/openflow/processors/putftp)
---
title: ListGCSBucket 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listgcsbucket.md
section: Loading & Unloading Data
---
# ListGCSBucket 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-gcp-nar
## Description
Retrieves a listing of objects from a GCS bucket. For each object that is listed, creates a FlowFile that represents the object so that it can be fetched in conjunction with FetchGCSObject. This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left off without duplicating all of the data.
## Tags
gcs, google, google cloud, list, storage
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Entity Tracking Initial Listing Target |
Specify how initial listing should be handled. Used by 'Tracking Entities'strategy. |
| Entity Tracking State Cache |
Listed entities are stored in the specified cache storage so that this processor can resume listing across NiFi restart or in case of primary node change. 'Tracking Entities'strategy require tracking information of all listed entities within the last 'Tracking Time Window'. To support large number of entities, the strategy uses DistributedMapCache instead of managed state. Cache key format is 'ListedEntities::\{processorId\}(::\{nodeId\})'. If it tracks per node listed entities, then the optional '::\{nodeId\}' part is added to manage state separately. E.g. cluster wide cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b', per node cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3' The stored cache content is Gzipped JSON string. The cache key will be deleted when target listing configuration is changed. Used by 'Tracking Entities'strategy. |
| Entity Tracking Time Window |
Specify how long this processor should track already-listed entities. 'Tracking Entities'strategy can pick any entity whose timestamp is inside the specified time window. For example, if set to '30 minutes', any entity having timestamp in recent 30 minutes will be the listing target when this processor runs. A listed entity is considered 'new/updated' and a FlowFile is emitted if one of following condition meets: 1. does not exist in the already-listed entities, 2. has newer timestamp than the cached entity, 3. has different size than the cached entity. If a cached entity 's timestamp becomes older than specified time window, that entity will be removed from the cached already-listed entities. Used by'Tracking Entities'strategy. |
| GCP Credentials Provider Service |
The Controller Service used to obtain Google Cloud Platform credentials. |
| gcp-project-id |
Google Cloud Project ID |
| gcp-retry-count |
How many retry attempts should be made before routing to the failure relationship. |
| gcs-bucket |
Bucket of the object. |
| gcs-prefix |
The prefix used to filter the object list. In most cases, it should end with a forward slash ( '/'). |
| gcs-use-generations |
Specifies whether to use GCS Generations, if applicable. If false, only the latest version of each object will be returned. |
| listing-strategy |
Specify how to determine new/updated entities. See each strategy descriptions for detail. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
| record-writer |
Specifies the Record Writer to use for creating the listing. If not specified, one FlowFile will be created for each entity that is listed. If the Record Writer is specified, all entities will be written to a single FlowFile instead of adding attributes to individual FlowFiles. |
| storage-api-url |
Overrides the default storage URL. Configuring an alternative Storage API URL also overrides the HTTP Host header on requests as described in the Google documentation for Private Service Connections. |
## State management
| Scopes |
Description |
| CLUSTER |
After performing a listing of keys, the timestamp of the newest key is stored, along with the keys that share that same timestamp. This allows the Processor to list only keys that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data. |
## Relationships
| Name |
Description |
| success |
FlowFiles are routed to this relationship after a successful Google Cloud Storage operation. |
## Writes attributes
| Name |
Description |
| filename |
The name of the file |
| gcs.bucket |
Bucket of the object. |
| gcs.key |
Name of the object. |
| gcs.size |
Size of the object. |
| gcs.cache.control |
Data cache control of the object. |
| gcs.component.count |
The number of components which make up the object. |
| gcs.content.disposition |
The data content disposition of the object. |
| gcs.content.encoding |
The content encoding of the object. |
| gcs.content.language |
The content language of the object. |
| mime.type |
The MIME/Content-Type of the object |
| gcs.crc32c |
The CRC32C checksum of object's data, encoded in base64 in big-endian order. |
| gcs.create.time |
The creation time of the object (milliseconds) |
| gcs.update.time |
The last modification time of the object (milliseconds) |
| gcs.encryption.algorithm |
The algorithm used to encrypt the object. |
| gcs.encryption.sha256 |
The SHA256 hash of the key used to encrypt the object |
| gcs.etag |
The HTTP 1.1 Entity tag for the object. |
| gcs.generated.id |
The service-generated for the object |
| gcs.generation |
The data generation of the object. |
| gcs.md5 |
The MD5 hash of the object's data encoded in base64. |
| gcs.media.link |
The media download link to the object. |
| gcs.metageneration |
The metageneration of the object. |
| gcs.owner |
The owner (uploader) of the object. |
| gcs.owner.type |
The ACL entity type of the uploader of the object. |
| gcs.acl.owner |
A comma-delimited list of ACL entities that have owner access to the object. Entities will be either email addresses, domains, or project IDs. |
| gcs.acl.writer |
A comma-delimited list of ACL entities that have write access to the object. Entities will be either email addresses, domains, or project IDs. |
| gcs.acl.reader |
A comma-delimited list of ACL entities that have read access to the object. Entities will be either email addresses, domains, or project IDs. |
| gcs.uri |
The URI of the object as a string. |
## See also
- [org.apache.nifi.processors.gcp.storage.DeleteGCSObject](/user-guide/data-integration/openflow/processors/deletegcsobject)
- [org.apache.nifi.processors.gcp.storage.FetchGCSObject](/user-guide/data-integration/openflow/processors/fetchgcsobject)
- [org.apache.nifi.processors.gcp.storage.PutGCSObject](/user-guide/data-integration/openflow/processors/putgcsobject)
---
title: ListGoogleDrive 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listgoogledrive.md
section: Loading & Unloading Data
---
# ListGoogleDrive 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-gcp-nar
## Description
Performs a listing of concrete files (shortcuts are ignored) in a Google Drive folder. If the 'Record Writer' property is set, a single Output FlowFile is created, and each file in the listing is written as a single record to the output file. Otherwise, for each file in the listing, an individual FlowFile is created, the metadata being written as FlowFile attributes. This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left off without duplicating all of the data. Please see Additional Details to set up access to Google Drive.
## Tags
drive, google, storage
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Entity Tracking Initial Listing Target |
Specify how initial listing should be handled. Used by 'Tracking Entities'strategy. |
| Entity Tracking State Cache |
Listed entities are stored in the specified cache storage so that this processor can resume listing across NiFi restart or in case of primary node change. 'Tracking Entities'strategy require tracking information of all listed entities within the last 'Tracking Time Window'. To support large number of entities, the strategy uses DistributedMapCache instead of managed state. Cache key format is 'ListedEntities::\{processorId\}(::\{nodeId\})'. If it tracks per node listed entities, then the optional '::\{nodeId\}' part is added to manage state separately. E.g. cluster wide cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b', per node cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3' The stored cache content is Gzipped JSON string. The cache key will be deleted when target listing configuration is changed. Used by 'Tracking Entities'strategy. |
| Entity Tracking Time Window |
Specify how long this processor should track already-listed entities. 'Tracking Entities'strategy can pick any entity whose timestamp is inside the specified time window. For example, if set to '30 minutes', any entity having timestamp in recent 30 minutes will be the listing target when this processor runs. A listed entity is considered 'new/updated' and a FlowFile is emitted if one of following condition meets: 1. does not exist in the already-listed entities, 2. has newer timestamp than the cached entity, 3. has different size than the cached entity. If a cached entity 's timestamp becomes older than specified time window, that entity will be removed from the cached already-listed entities. Used by'Tracking Entities'strategy. |
| Listing Strategy |
Specify how to determine new/updated entities. See each strategy descriptions for detail. |
| Record Writer |
Specifies the Record Writer to use for creating the listing. If not specified, one FlowFile will be created for each entity that is listed. If the Record Writer is specified, all entities will be written to a single FlowFile instead of adding attributes to individual FlowFiles. |
| connect-timeout |
Maximum wait time for connection to Google Drive service. |
| folder-id |
The ID of the folder from which to pull list of files. Please see Additional Details to set up access to Google Drive and obtain Folder ID. WARNING: Unauthorized access to the folder is treated as if the folder was empty. This results in the processor not creating outgoing FlowFiles. No additional error message is provided. |
| gcp-credentials-provider-service |
The Controller Service used to obtain Google Cloud Platform credentials. |
| min-age |
The minimum age a file must be in order to be considered; any files younger than this will be ignored. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
| read-timeout |
Maximum wait time for response from Google Drive service. |
| recursive-search |
When 'true', will include list of files from concrete sub-folders (ignores shortcuts). Otherwise, will return only files that have the defined 'Folder ID' as their parent directly. WARNING: The listing may fail if there are too many sub-folders (500+). |
## State management
| Scopes |
Description |
| CLUSTER |
The processor stores necessary data to be able to keep track what files have been listed already. What exactly needs to be stored depends on the 'Listing Strategy'. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data. |
## Relationships
| Name |
Description |
| success |
All FlowFiles that are received are routed to success |
## Writes attributes
| Name |
Description |
| drive.id |
The id of the file |
| filename |
The name of the file |
| mime.type |
The MIME type of the file |
| drive.size |
The size of the file. Set to 0 when the file size is not available (e.g. externally stored files). |
| drive.size.available |
Indicates if the file size is known / available |
| drive.timestamp |
The last modified time or created time (whichever is greater) of the file. The reason for this is that the original modified date of a file is preserved when uploaded to Google Drive. 'Created time' takes the time when the upload occurs. However uploaded files can still be modified later. |
| drive.created.time |
The file's creation time |
| drive.modified.time |
The file's last modification time |
| drive.path |
The path of the file's directory from the base directory. The path contains the folder names in URL encoded form because Google Drive allows special characters in file names, including '/' (slash) and '' (backslash). The URL encoded folder names are separated by '/' in the path. |
| drive.owner |
The owner of the file |
| drive.last.modifying.user |
The last modifying user of the file |
| drive.web.view.link |
Web view link to the file |
| drive.web.content.link |
Web content link to the file |
| drive.parent.folder.id |
The id of the file's parent folder |
| drive.parent.folder.name |
The name of the file's parent folder |
| drive.listed.folder.id |
The id of the base folder that was listed |
| drive.listed.folder.name |
The name of the base folder that was listed |
| drive.shared.drive.id |
The id of the shared drive (if the file is located on a shared drive) |
| drive.shared.drive.name |
The name of the shared drive (if the file is located on a shared drive) |
## See also
- [org.apache.nifi.processors.gcp.drive.FetchGoogleDrive](/user-guide/data-integration/openflow/processors/fetchgoogledrive)
- [org.apache.nifi.processors.gcp.drive.PutGoogleDrive](/user-guide/data-integration/openflow/processors/putgoogledrive)
---
title: ListGoogleDriveFileInfo 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listgoogledrivefileinfo.md
section: Loading & Unloading Data
---
# ListGoogleDriveFileInfo 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-google-drive-nar
## Description
Lists all files and folders in a specified Google Drive. The processor requires a Drive ID and can optionally list files recursively through all folders within the drive.
## Tags
cloud, drive, files, gcp, google, list, openflow, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Drive ID |
The ID of the drive to list files from. This can be a shared drive ID. |
| GCP Credentials Service |
The Controller Service used to obtain Google Cloud Platform credentials. |
| Include Folders |
When 'true', both files and folders will be included in the results. When 'false', only files (not folders) will be included. |
| Minimum File Age |
The minimum age a file must be in order to be considered; any files younger than this will be ignored. |
| Record Writer |
Specifies the Controller Service to use for writing the metadata records. Must be set. |
| Search Recursively |
When 'true', will recursively list files in all folders within the drive. When 'false', will only list files at the root level of the drive. |
## Relationships
| Name |
Description |
| failure |
A FlowFile will be routed here if there is an error fetching file metadata. |
| retry |
A FlowFile is routed here if the processor should retry the request (e.g., after rate limiting). |
| success |
A FlowFile containing the file metadata records will be routed to this relationship upon successful processing. |
## Writes attributes
| Name |
Description |
| google.drive.drive.id |
The ID of the drive from which files were listed |
| record.count |
The number of records in the FlowFile |
| mime.type |
The MIME Type specified by the Record Writer |
| google.drive.error.code |
The error code if the request to Google Drive API fails |
| google.drive.error.message |
The error message if the request to Google Drive API fails |
## See also
- [com.snowflake.openflow.runtime.processors.google.CaptureGoogleDriveChanges](/user-guide/data-integration/openflow/processors/capturegoogledrivechanges)
- [com.snowflake.openflow.runtime.processors.google.FetchGoogleDriveMetadata](/user-guide/data-integration/openflow/processors/fetchgoogledrivemetadata)
---
title: ListGoogleGroups 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listgooglegroups.md
section: Loading & Unloading Data
---
# ListGoogleGroups 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-google-drive-nar
## Description
Lists all of the groups for a given domain in Google Workspace. It supports an optional 'Query' to filter the groups. The retrieved group metadata (id, etag, email, name, directMembersCount, description) are output to a Record Writer.
## Tags
cloud, directory, domain, gcp, google, groups, list
## Input Requirement
ALLOWED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Custom Query |
Custom query to filter the returned groups. For example, 'email=test-*'. See Google's Admin SDK Directory API documentation for supported syntax. |
| GCP Credentials Service |
Controller Service used to obtain Google Cloud Platform credentials. |
| Google Domain |
Domain name to list Google Groups (e.g., 'example.com'). |
| Record Writer |
Record writer used for writing out the records of retrieved Google Groups. |
## Relationships
| Name |
Description |
| failure |
FlowFiles are routed here if the processor fails to retrieve Google Groups. |
| retry |
FlowFiles are routed here if a transient failure occurs (e.g. rate-limited, socket timeouts) and should be retried. |
| success |
A FlowFile containing a record set of the groups is routed here upon success. |
## Writes attributes
| Name |
Description |
| record.count |
The number of records (groups) returned. |
| mime.type |
The MIME type for the resulting FlowFile. |
## See also
- [com.snowflake.openflow.runtime.processors.google.GetGoogleGroupMembers](/user-guide/data-integration/openflow/processors/getgooglegroupmembers)
---
title: ListHubSpotObjects 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listhubspotobjects.md
section: Loading & Unloading Data
---
# ListHubSpotObjects 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-hubspot-processors-nar
## Description
Fetches data from HubSpot for specified object types, and generates one FlowFile per listed object with the corresponding metadata as FlowFile attributes. The object type must be searchable, which means it supports access to the /search endpoint. For more information about searchable object types, see: https://developers.hubspot.com/docs/reference/api/crm/objects/objects#search (https://developers.hubspot.com/docs/reference/api/crm/objects/objects#search)")
## Tags
Preview, hubspot
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| HubSpot Service |
HubSpot Client Service. |
| Object Type |
HubSpot object type |
| Updated After |
Filter objects updated after specified date (format: yyyy-MM-dd) |
## State management
| Scopes |
Description |
| CLUSTER |
Maintains pagination state and last sync timestamp to continue data retrieval from the last known position after restarts and to fetch only changed data. |
## Relationships
| Name |
Description |
| failure |
HubSpot fail relationship |
| original |
The input Flow File is routed to the original relationship. |
| retry |
HubSpot retry relationship. FlowFiles that failed to process due to a server timeout or rate limit related error. FlowFiles routed here should be routed back into the processor. |
| success |
HubSpot success relationship |
## Writes attributes
| Name |
Description |
| mime.type |
application/json |
| statement.type |
Always 'UPSERT' for this processor |
| hubspot.object.type |
HubSpot Object Type for this fetch |
| hubspot.object.id |
HubSpot Object ID for this fetch |
| hubspot.run.id |
Timestamp of the start of this run. Obtained from the incoming FlowFile or current time if not available |
| hubspot.is_last |
Whether this is the last paged object of the ingestion |
## Use cases
| This processor is typically used in conjunction with a GenerateFlowFile processor |
| --------------------------------------------------------------------------------- |
## See also
- [com.snowflake.openflow.runtime.processors.hubspot.GetHubSpotObject](/user-guide/data-integration/openflow/processors/gethubspotobject)
- [com.snowflake.openflow.runtime.processors.hubspot.GetHubSpotSchema](/user-guide/data-integration/openflow/processors/gethubspotschema)
- [com.snowflake.openflow.runtime.processors.hubspot.ListArchivedHubSpotData](/user-guide/data-integration/openflow/processors/listarchivedhubspotdata)
- [com.snowflake.openflow.runtime.processors.hubspot.PutHubSpot](/user-guide/data-integration/openflow/processors/puthubspot)
---
title: ListMicrosoftDataverseTables 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listmicrosoftdataversetables.md
section: Loading & Unloading Data
---
# ListMicrosoftDataverseTables 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-dataverse-processors-nar
## Description
List Tables from Microsoft Dataverse environments
## Tags
dataverse
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Environment URL |
URL to Microsoft Dataverse Environment |
| OAuth2 Access Token Provider |
Enables managed retrieval of OAuth2 Bearer Token. |
| Tables Filter Strategy |
List of table names. Output will be limited to those names if defined. |
| Tables Filter Value |
Value of Table Names filter. It is regexp or separated list, depending on selected filtering strategy. |
| Web Client Service Provider |
Creates instance of web client. |
## Relationships
| Name |
Description |
| failure |
FlowFile with errors occurred while fetching from Dataverse. |
| success |
FlowFile with listed tables from Dataverse. |
---
title: ListS3 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/lists3.md
section: Loading & Unloading Data
---
# ListS3 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-aws-nar
## Description
Retrieves a listing of objects from an S3 bucket. For each object that is listed, creates a FlowFile that represents the object so that it can be fetched in conjunction with FetchS3Object. This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left off without duplicating all of the data.
## Tags
AWS, Amazon, S3, list
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| AWS Credentials Provider service |
The Controller Service that is used to obtain AWS credentials provider |
| Bucket |
The S3 Bucket to interact with |
| Communications Timeout |
The amount of time to wait in order to establish a connection to AWS or receive data from AWS before timing out. |
| Custom Signer Class Name |
Fully qualified class name of the custom signer class. The signer must implement com.amazonaws.auth. Signer interface. |
| Custom Signer Module Location |
Comma-separated list of paths to files and/or directories which contain the custom signer's JAR file and its dependencies (if any). |
| Delimiter |
The string used to delimit directories within the bucket. Please consult the AWS documentation for the correct use of this field. |
| Endpoint Override URL |
Endpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints. |
| Entity Tracking Initial Listing Target |
Specify how initial listing should be handled. Used by 'Tracking Entities'strategy. |
| Entity Tracking State Cache |
Listed entities are stored in the specified cache storage so that this processor can resume listing across NiFi restart or in case of primary node change. 'Tracking Entities'strategy require tracking information of all listed entities within the last 'Tracking Time Window'. To support large number of entities, the strategy uses DistributedMapCache instead of managed state. Cache key format is 'ListedEntities::\{processorId\}(::\{nodeId\})'. If it tracks per node listed entities, then the optional '::\{nodeId\}' part is added to manage state separately. E.g. cluster wide cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b', per node cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3' The stored cache content is Gzipped JSON string. The cache key will be deleted when target listing configuration is changed. Used by 'Tracking Entities'strategy. |
| Entity Tracking Time Window |
Specify how long this processor should track already-listed entities. 'Tracking Entities'strategy can pick any entity whose timestamp is inside the specified time window. For example, if set to '30 minutes', any entity having timestamp in recent 30 minutes will be the listing target when this processor runs. A listed entity is considered 'new/updated' and a FlowFile is emitted if one of following condition meets: 1. does not exist in the already-listed entities, 2. has newer timestamp than the cached entity, 3. has different size than the cached entity. If a cached entity 's timestamp becomes older than specified time window, that entity will be removed from the cached already-listed entities. Used by'Tracking Entities'strategy. |
| List Type |
Specifies whether to use the original List Objects or the newer List Objects Version 2 endpoint. |
| Listing Batch Size |
If not using a Record Writer, this property dictates how many S3 objects should be listed in a single batch. Once this number is reached, the FlowFiles that have been created will be transferred out of the Processor. Setting this value lower may result in lower latency by sending out the FlowFiles before the complete listing has finished. However, it can significantly reduce performance. Larger values may take more memory to store all of the information before sending the FlowFiles out. This property is ignored if using a Record Writer, as one of the main benefits of the Record Writer is being able to emit the entire listing as a single FlowFile. |
| Listing Strategy |
Specify how to determine new/updated entities. See each strategy descriptions for detail. |
| Maximum Object Age |
The maximum age that an S3 object can be in order to be considered; any object older than this amount of time (according to last modification date) will be ignored |
| Minimum Object Age |
The minimum age that an S3 object must be in order to be considered; any object younger than this amount of time (according to last modification date) will be ignored |
| Prefix |
The prefix used to filter the object list. Do not begin with a forward slash '/'. In most cases, it should end with a forward slash '/'. |
| Record Writer |
Specifies the Record Writer to use for creating the listing. If not specified, one FlowFile will be created for each entity that is listed. If the Record Writer is specified, all entities will be written to a single FlowFile instead of adding attributes to individual FlowFiles. |
| Region |
The AWS Region to connect to. |
| Requester Pays |
If true, indicates that the requester consents to pay any charges associated with listing the S3 bucket. This sets the 'x-amz-request-payer' header to 'requester'. Note that this setting is not applicable when 'Use Versions' is 'true'. |
| SSL Context Service |
Specifies an optional SSL Context Service that, if provided, will be used to create connections |
| Signer Override |
The AWS S3 library uses Signature Version 4 by default but this property allows you to specify the Version 2 signer to support older S3-compatible services or even to plug in your own custom signer implementation. |
| Use Versions |
Specifies whether to use S3 versions, if applicable. If false, only the latest version of each object will be returned. |
| Write Object Tags |
If set to 'True', the tags associated with the S3 object will be written as FlowFile attributes |
| Write User Metadata |
If set to 'True', the user defined metadata associated with the S3 object will be added to FlowFile attributes/records |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## State management
| Scopes |
Description |
| CLUSTER |
After performing a listing of keys, the timestamp of the newest key is stored, along with the keys that share that same timestamp. This allows the Processor to list only keys that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data. |
## Relationships
| Name |
Description |
| success |
FlowFiles are routed to this Relationship after they have been successfully processed. |
## Writes attributes
| Name |
Description |
| s3.bucket |
The name of the S3 bucket |
| s3.region |
The region of the S3 bucket |
| filename |
The name of the file |
| s3.etag |
The ETag that can be used to see if the file has changed |
| s3.isLatest |
A boolean indicating if this is the latest version of the object |
| s3.lastModified |
The last modified time in milliseconds since epoch in UTC time |
| s3.length |
The size of the object in bytes |
| s3.storeClass |
The storage class of the object |
| s3.version |
The version of the object, if applicable |
| s3.tag.___ |
If 'Write Object Tags' is set to 'True', the tags associated to the S3 object that is being listed will be written as part of the flowfile attributes |
| s3.user.metadata.___ |
If 'Write User Metadata' is set to 'True', the user defined metadata associated to the S3 object that is being listed will be written as part of the flowfile attributes |
## See also
- [org.apache.nifi.processors.aws.s3.CopyS3Object](/user-guide/data-integration/openflow/processors/copys3object)
- [org.apache.nifi.processors.aws.s3.DeleteS3Object](/user-guide/data-integration/openflow/processors/deletes3object)
- [org.apache.nifi.processors.aws.s3.FetchS3Object](/user-guide/data-integration/openflow/processors/fetchs3object)
- [org.apache.nifi.processors.aws.s3.GetS3ObjectMetadata](/user-guide/data-integration/openflow/processors/gets3objectmetadata)
- [org.apache.nifi.processors.aws.s3.GetS3ObjectTags](/user-guide/data-integration/openflow/processors/gets3objecttags)
- [org.apache.nifi.processors.aws.s3.PutS3Object](/user-guide/data-integration/openflow/processors/puts3object)
- [org.apache.nifi.processors.aws.s3.TagS3Object](/user-guide/data-integration/openflow/processors/tags3object)
---
title: ListSFDCDataShares 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listsfdcdatashares.md
section: Loading & Unloading Data
---
# ListSFDCDataShares 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-salesforce-processors-nar
## Description
List the available data shares in the organization that are available to the identified user.
## Tags
list, objects, preview, salesforce, sfdc
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Salesforce Data Cloud Client |
Salesforce Data Cloud Client to interact with the APIs |
## Relationships
| Name |
Description |
| success |
FlowFile containing the list of available objects will be routed to this relationship |
## Writes attributes
| Name |
Description |
| nbObjects |
The number of data shares listed in the organization that are available to the identified user. |
## See also
- [com.snowflake.openflow.runtime.processors.salesforce.DeleteQueryJob](/user-guide/data-integration/openflow/processors/deletequeryjob)
- [com.snowflake.openflow.runtime.processors.salesforce.DescribeSFDCObject](/user-guide/data-integration/openflow/processors/describesfdcobject)
- [com.snowflake.openflow.runtime.processors.salesforce.GetQueryJobResult](/user-guide/data-integration/openflow/processors/getqueryjobresult)
- [com.snowflake.openflow.runtime.processors.salesforce.SubmitQueryJob](/user-guide/data-integration/openflow/processors/submitqueryjob)
---
title: ListSFDCObjects 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listsfdcobjects.md
section: Loading & Unloading Data
---
# ListSFDCObjects 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-salesforce-processors-nar
## Description
List the available objects in the organization that are available to the identified user.
## Tags
list, objects, preview, salesforce, sfdc
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Salesforce Client |
Salesforce Client to interact with the APIs |
## Relationships
| Name |
Description |
| success |
FlowFile containing the list of available objects will be routed to this relationship |
## Writes attributes
| Name |
Description |
| nbObjects |
The number of objects listed in the organization that are available to the identified user. |
## See also
- [com.snowflake.openflow.runtime.processors.salesforce.DeleteQueryJob](/user-guide/data-integration/openflow/processors/deletequeryjob)
- [com.snowflake.openflow.runtime.processors.salesforce.DescribeSFDCObject](/user-guide/data-integration/openflow/processors/describesfdcobject)
- [com.snowflake.openflow.runtime.processors.salesforce.GetQueryJobResult](/user-guide/data-integration/openflow/processors/getqueryjobresult)
- [com.snowflake.openflow.runtime.processors.salesforce.SubmitQueryJob](/user-guide/data-integration/openflow/processors/submitqueryjob)
---
title: ListSFTP 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listsftp.md
section: Loading & Unloading Data
---
# ListSFTP 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Performs a listing of the files residing on an SFTP server. For each file that is found on the remote server, a new FlowFile will be created with the filename attribute set to the name of the file on the remote server. This can then be used in conjunction with FetchSFTP in order to fetch those files.
## Tags
files, ingest, input, list, remote, sftp, source
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Algorithm Negotiation |
Configuration strategy for SSH algorithm negotiation |
| Ciphers Allowed |
A comma-separated list of Ciphers allowed for SFTP connections. Leave unset to allow all. Available options are: 3des-cbc, aes128-cbc, aes128-ctr, [aes128-gcm@openssh.com](mailto:aes128-gcm@openssh.com), aes192-cbc, aes192-ctr, aes256-cbc, aes256-ctr, [aes256-gcm@openssh.com](mailto:aes256-gcm@openssh.com), arcfour128, arcfour256, blowfish-cbc, [chacha20-poly1305@openssh.com](mailto:chacha20-poly1305@openssh.com), none |
| Connection Timeout |
Amount of time to wait before timing out while creating a connection |
| Data Timeout |
When transferring a file between the local and remote system, this value specifies how long is allowed to elapse without any data being transferred between systems |
| Entity Tracking Initial Listing Target |
Specify how initial listing should be handled. Used by 'Tracking Entities'strategy. |
| Entity Tracking State Cache |
Listed entities are stored in the specified cache storage so that this processor can resume listing across NiFi restart or in case of primary node change. 'Tracking Entities'strategy require tracking information of all listed entities within the last 'Tracking Time Window'. To support large number of entities, the strategy uses DistributedMapCache instead of managed state. Cache key format is 'ListedEntities::\{processorId\}(::\{nodeId\})'. If it tracks per node listed entities, then the optional '::\{nodeId\}' part is added to manage state separately. E.g. cluster wide cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b', per node cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3' The stored cache content is Gzipped JSON string. The cache key will be deleted when target listing configuration is changed. Used by 'Tracking Entities'strategy. |
| Entity Tracking Time Window |
Specify how long this processor should track already-listed entities. 'Tracking Entities'strategy can pick any entity whose timestamp is inside the specified time window. For example, if set to '30 minutes', any entity having timestamp in recent 30 minutes will be the listing target when this processor runs. A listed entity is considered 'new/updated' and a FlowFile is emitted if one of following condition meets: 1. does not exist in the already-listed entities, 2. has newer timestamp than the cached entity, 3. has different size than the cached entity. If a cached entity 's timestamp becomes older than specified time window, that entity will be removed from the cached already-listed entities. Used by'Tracking Entities'strategy. |
| File Filter Regex |
Provides a Java Regular Expression for filtering Filenames; if a filter is supplied, only files whose names match that Regular Expression will be fetched |
| Follow Symbolic Links |
If true, will pull even symbolic files and also nested symbolic subdirectories; otherwise, will not read symbolic files and will not traverse symbolic link subdirectories |
| Host Key File |
If supplied, the given file will be used as the Host Key; otherwise, if 'Strict Host Key Checking' property is applied (set to true) then uses the 'known_hosts' and 'known_hosts2' files from ~/.ssh directory else no host key file will be used |
| Hostname |
The fully qualified hostname or IP address of the remote system |
| Ignore Dotted Files |
If true, files whose names begin with a dot (".") will be ignored |
| Key Algorithms Allowed |
A comma-separated list of Key Algorithms allowed for SFTP connections. Leave unset to allow all. Available options are: ecdsa-sha2-nistp256, [ecdsa-sha2-nistp256-cert-v01@openssh.com](mailto:ecdsa-sha2-nistp256-cert-v01@openssh.com), ecdsa-sha2-nistp384, [ecdsa-sha2-nistp384-cert-v01@openssh.com](mailto:ecdsa-sha2-nistp384-cert-v01@openssh.com), ecdsa-sha2-nistp521, [ecdsa-sha2-nistp521-cert-v01@openssh.com](mailto:ecdsa-sha2-nistp521-cert-v01@openssh.com), rsa-sha2-256, [rsa-sha2-256-cert-v01@openssh.com](mailto:rsa-sha2-256-cert-v01@openssh.com), rsa-sha2-512, [rsa-sha2-512-cert-v01@openssh.com](mailto:rsa-sha2-512-cert-v01@openssh.com), [sk-ecdsa-sha2-nistp256@openssh.com](mailto:sk-ecdsa-sha2-nistp256@openssh.com), [sk-ssh-ed25519@openssh.com](mailto:sk-ssh-ed25519@openssh.com), ssh-dss, [ssh-dss-cert-v01@openssh.com](mailto:ssh-dss-cert-v01@openssh.com), ssh-ed25519, [ssh-ed25519-cert-v01@openssh.com](mailto:ssh-ed25519-cert-v01@openssh.com), ssh-rsa, [ssh-rsa-cert-v01@openssh.com](mailto:ssh-rsa-cert-v01@openssh.com) |
| Key Exchange Algorithms Allowed |
A comma-separated list of Key Exchange Algorithms allowed for SFTP connections. Leave unset to allow all. Available options are: curve25519-sha256, [curve25519-sha256@libssh.org](mailto:curve25519-sha256@libssh.org), curve448-sha512, diffie-hellman-group-exchange-sha1, diffie-hellman-group-exchange-sha256, diffie-hellman-group1-sha1, diffie-hellman-group14-sha1, diffie-hellman-group14-sha256, diffie-hellman-group15-sha512, diffie-hellman-group16-sha512, diffie-hellman-group17-sha512, diffie-hellman-group18-sha512, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, mlkem1024nistp384-sha384, mlkem768nistp256-sha256, mlkem768x25519-sha256, sntrup761x25519-sha512, [sntrup761x25519-sha512@openssh.com](mailto:sntrup761x25519-sha512@openssh.com) |
| Listing Strategy |
Specify how to determine new/updated entities. See each strategy descriptions for detail. |
| Maximum File Age |
The maximum age that a file must be in order to be pulled; any file older than this amount of time (according to last modification date) will be ignored |
| Maximum File Size |
The maximum size that a file can be in order to be pulled |
| Message Authentication Codes Allowed |
A comma-separated list of Message Authentication Codes allowed for SFTP connections. Leave unset to allow all. Available options are: hmac-md5, hmac-md5-96, hmac-sha1, hmac-sha1-96, [hmac-sha1-etm@openssh.com](mailto:hmac-sha1-etm@openssh.com), hmac-sha2-256, [hmac-sha2-256-etm@openssh.com](mailto:hmac-sha2-256-etm@openssh.com), hmac-sha2-512, [hmac-sha2-512-etm@openssh.com](mailto:hmac-sha2-512-etm@openssh.com) |
| Minimum File Age |
The minimum age that a file must be in order to be pulled; any file younger than this amount of time (according to last modification date) will be ignored |
| Minimum File Size |
The minimum size that a file must be in order to be pulled |
| Password |
Password for the user account |
| Path Filter Regex |
When Search Recursively is true, then only subdirectories whose path matches the given Regular Expression will be scanned |
| Port |
The port that the remote system is listening on for file transfers |
| Private Key Passphrase |
Password for the private key |
| Private Key Path |
The fully qualified path to the Private Key file |
| Record Writer |
Specifies the Record Writer to use for creating the listing. If not specified, one FlowFile will be created for each entity that is listed. If the Record Writer is specified, all entities will be written to a single FlowFile instead of adding attributes to individual FlowFiles. |
| Remote Path |
The path on the remote system from which to pull or push files |
| Search Recursively |
If true, will pull files from arbitrarily nested subdirectories; otherwise, will not traverse subdirectories |
| Send Keep Alive On Timeout |
Send a Keep Alive message every 5 seconds up to 5 times for an overall timeout of 25 seconds. |
| Strict Host Key Checking |
Indicates whether or not strict enforcement of hosts keys should be applied |
| Target System Timestamp Precision |
Specify timestamp precision at the target system. Since this processor uses timestamp of entities to decide which should be listed, it is crucial to use the right timestamp precision. |
| Use Compression |
Indicates whether or not ZLIB compression should be used when transferring files |
| Username |
Username |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. |
## State management
| Scopes |
Description |
| CLUSTER |
After performing a listing of files, the timestamp of the newest file is stored. This allows the Processor to list only files that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node will not duplicate the data that was listed by the previous Primary Node. |
## Relationships
| Name |
Description |
| success |
All FlowFiles that are received are routed to success |
## Writes attributes
| Name |
Description |
| sftp.remote.host |
The hostname of the SFTP Server |
| sftp.remote.port |
The port that was connected to on the SFTP Server |
| sftp.listing.user |
The username of the user that performed the SFTP Listing |
| file.owner |
The numeric owner id of the source file |
| file.group |
The numeric group id of the source file |
| file.permissions |
The read/write/execute permissions of the source file |
| file.size |
The number of bytes in the source file |
| file.lastModifiedTime |
The timestamp of when the file in the filesystem waslast modified as 'yyyy-MM-dd'T'HH:mm:ssZ' |
| filename |
The name of the file on the SFTP Server |
| path |
The fully qualified name of the directory on the SFTP Server from which the file was pulled |
| mime.type |
The MIME Type that is provided by the configured Record Writer |
## See also
- [org.apache.nifi.processors.standard.FetchSFTP](/user-guide/data-integration/openflow/processors/fetchsftp)
- [org.apache.nifi.processors.standard.GetSFTP](/user-guide/data-integration/openflow/processors/getsftp)
- [org.apache.nifi.processors.standard.PutSFTP](/user-guide/data-integration/openflow/processors/putsftp)
---
title: ListSharepointDrives 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listsharepointdrives.md
section: Loading & Unloading Data
---
# ListSharepointDrives 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-msgraph-nar
## Description
Emits a FlowFile for each Drive present in the specified Sharepoint Site.
## Tags
document, graph, microsoft, openflow, sharepoint, unstructured
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Authentication Service |
The service that provides authentication for the SharePoint API. |
| Site URL |
The URL of the Sharepoint Site. |
## Relationships
| Name |
Description |
| success |
FlowFiles for each Drive are routed to this relationship |
## Writes attributes
| Name |
Description |
| sharepoint.site.url |
The URL of the Sharepoint Site. |
| sharepoint.site.id |
The ID of the Sharepoint Site. |
| sharepoint.drive.name |
The name of the Sharepoint Drive. |
| sharepoint.drive.id |
The ID of the Sharepoint Drive. |
## See also
- [com.snowflake.openflow.runtime.processors.sharepoint.FetchSharepointFile](/user-guide/data-integration/openflow/processors/fetchsharepointfile)
- [com.snowflake.openflow.runtime.processors.sharepoint.FindSharepointDriveItem](/user-guide/data-integration/openflow/processors/findsharepointdriveitem)
---
title: ListSharepointSiteGroups 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listsharepointsitegroups.md
section: Loading & Unloading Data
---
# ListSharepointSiteGroups 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-sharepoint-rest-nar
## Description
Lists all SharePoint site groups available on a specified SharePoint site.
## Tags
groups, list, microsoft, openflow, sharepoint
## Input Requirement
ALLOWED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| OAuth2 Access Token Provider |
Enables managed retrieval of OAuth2 Bearer Token. |
| Record Writer |
Record writer used for writing out the records of retrieved Sharepoint Site Groups. |
| Site URL |
The URL of the SharePoint site. |
| Web Client Service |
The Web Client Service to use for communicating with Sharepoint. |
## Relationships
| Name |
Description |
| success |
Successfully listed all SharePoint site groups. Each group will be represented as a separate FlowFile. |
## Writes attributes
| Name |
Description |
| record.count |
The number of records (groups) returned. |
| mime.type |
The MIME type for the resulting FlowFile. |
## See also
- [com.snowflake.openflow.runtime.processors.sharepoint.rest.GetSharepointSiteGroupMembers](/user-guide/data-integration/openflow/processors/getsharepointsitegroupmembers)
---
title: ListSmb 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listsmb.md
section: Loading & Unloading Data
---
# ListSmb 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-smb-nar
## Description
Lists concrete files shared via SMB protocol. Each listed file may result in one FlowFile, the metadata being written as FlowFile attributes. Or - in case the 'Record Writer' property is set - the entire result is written as records to a single FlowFile. This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left off without duplicating all of the data.
## Tags
list, samba, smb, cifs, files
## Input Requirement
FORBIDDEN
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Entity Tracking Initial Listing Target |
Specify how initial listing should be handled. Used by 'Tracking Entities'strategy. |
| Entity Tracking State Cache |
Listed entities are stored in the specified cache storage so that this processor can resume listing across NiFi restart or in case of primary node change. 'Tracking Entities'strategy require tracking information of all listed entities within the last 'Tracking Time Window'. To support large number of entities, the strategy uses DistributedMapCache instead of managed state. Cache key format is 'ListedEntities::\{processorId\}(::\{nodeId\})'. If it tracks per node listed entities, then the optional '::\{nodeId\}' part is added to manage state separately. E.g. cluster wide cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b', per node cache key ='ListedEntities::8dda2321-0164-1000-50fa-3042fe7d6a7b::nifi-node3' The stored cache content is Gzipped JSON string. The cache key will be deleted when target listing configuration is changed. Used by 'Tracking Entities'strategy. |
| Entity Tracking Time Window |
Specify how long this processor should track already-listed entities. 'Tracking Entities'strategy can pick any entity whose timestamp is inside the specified time window. For example, if set to '30 minutes', any entity having timestamp in recent 30 minutes will be the listing target when this processor runs. A listed entity is considered 'new/updated' and a FlowFile is emitted if one of following condition meets: 1. does not exist in the already-listed entities, 2. has newer timestamp than the cached entity, 3. has different size than the cached entity. If a cached entity 's timestamp becomes older than specified time window, that entity will be removed from the cached already-listed entities. Used by'Tracking Entities'strategy. |
| Listing Strategy |
Specify how to determine new/updated entities. See each strategy descriptions for detail. |
| Record Writer |
Specifies the Record Writer to use for creating the listing. If not specified, one FlowFile will be created for each entity that is listed. If the Record Writer is specified, all entities will be written to a single FlowFile instead of adding attributes to individual FlowFiles. |
| Target System Timestamp Precision |
Specify timestamp precision at the target system. Since this processor uses timestamp of entities to decide which should be listed, it is crucial to use the right timestamp precision. |
| directory |
The network folder from which to list files. This is the remaining relative path after the share: [smb://HOSTNAME:PORT/SHARE/[DIRECTORY]/sub/directories](smb://HOSTNAME:PORT/SHARE/[DIRECTORY]/sub/directories). It is also possible to add subdirectories. The given path on the remote file share must exist. This can be checked using verification. You may mix Windows and Linux-style directory separators. |
| file-filter |
Only files whose names match the given regular expression will be listed. |
| file-name-suffix-filter |
Files ending with the given suffix will be omitted. Can be used to make sure that files that are still uploading are not listed multiple times, by having those files have a suffix and remove the suffix once the upload finishes. This is highly recommended when using 'Tracking Entities' or 'Tracking Timestamps' listing strategies. |
| initial-listing-strategy |
Specifies how to handle existing files on the SMB share when the processor is started for the first time (or its state has been cleared). |
| initial-listing-timestamp |
The timestamp from which the files will be listed when the processor is started for the first time (or its state has been cleared). The value can be specified as an epoch timestamp in milliseconds or as a UTC datetime in a format such as 2025-02-01T00:00:00Z |
| max-file-age |
Any file older than the given value will be omitted. |
| max-file-size |
Any file larger than the given value will be omitted. |
| min-file-age |
The minimum age that a file must be in order to be listed; any file younger than this amount of time will be ignored. |
| min-file-size |
Any file smaller than the given value will be omitted. |
| path-filter |
Only files whose paths (up to the file's parent directory) match the given regular expression will be listed. |
| smb-client-provider-service |
Specifies the SMB client provider to use for creating SMB connections. |
## State management
| Scopes |
Description |
| CLUSTER |
After performing a listing of files, the state of the previous listing can be stored in order to list files continuously without duplication. |
## Relationships
| Name |
Description |
| success |
All FlowFiles that are received are routed to success |
## Writes attributes
| Name |
Description |
| filename |
The name of the file that was read from filesystem. |
| shortName |
The short name of the file that was read from filesystem. |
| path |
The path is set to the relative path of the file's directory on the remote filesystem compared to the Share root directory. For example, for a given remote locationsmb://HOSTNAME:PORT/SHARE/DIRECTORY, and a file is being listed from smb://HOSTNAME:PORT/SHARE/DIRECTORY/sub/folder/file then the path attribute will be set to "DIRECTORY/sub/folder". |
| serviceLocation |
The SMB URL of the share. |
| lastModifiedTime |
The timestamp of when the file's content changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ss'. |
| creationTime |
The timestamp of when the file was created in the filesystem as 'yyyy-MM-dd'T'HH:mm:ss'. |
| lastAccessTime |
The timestamp of when the file was accessed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ss'. |
| changeTime |
The timestamp of when the file's attributes was changed in the filesystem as 'yyyy-MM-dd'T'HH:mm:ss'. |
| size |
The size of the file in bytes. |
| allocationSize |
The number of bytes allocated for the file on the server. |
## See also
- [org.apache.nifi.processors.smb.FetchSmb](/user-guide/data-integration/openflow/processors/fetchsmb)
- [org.apache.nifi.processors.smb.GetSmbFile](/user-guide/data-integration/openflow/processors/getsmbfile)
- [org.apache.nifi.processors.smb.PutSmbFile](/user-guide/data-integration/openflow/processors/putsmbfile)
---
title: ListTableNames 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listtablenames.md
section: Loading & Unloading Data
---
# ListTableNames 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-database-cdc-processors-nar
## Description
Fetches all source table names and matches them with one of the possible configurations: - regexp expression e.g. "(?i)customer.(orders|payments)" - it matches names in case insensitive way. It would match both "CUSTOMER.ORDERS" and "customer.orders" source table names. - comma separated list of source table names. e.g. "customer.orders, customer.payments". It matches source table names in case sensitive way i.e. "customer.orders" source table will be forwarded to MATCH relationship but "customer. ORDERS" won 't match. Matched source tables that cannot be replicated will be routed to FAILURE relationship, each table in a separate FlowFile, with a reason in attributes. Configuration is passed as a FlowFile attribute. Source table name is represented as <schema_name>.<table_name> so both inputs should take that into consideration. Matched source table names are forwarded to MATCHED relationship. Processor generates a single FlowFile with matching tables. Disclaimers - Postgresql lets you define database object names in case sensitive or case insensitive way. When user creates a table using following query'CREATE TABLE ORDERS(id int not null) 'then internally Postgresql stores it using lower case letters i.e. orders. To enforce case sensitivity user has to wrap the table name with double quotes i.e.'CREATE TABLE "ORDERS"(id int not null)'. This is important aspect when configuring table that we would like to replicate.
## Tags
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Connection Pool |
The Controller Service that is used to obtain a connection to the database. |
| Included Comma Separated Source Table Names |
The list of comma separated list of tables to replicate. A single table should be formatted as <schema_name>.<table_name> e.g. customer.orders, customer.payments. This is combined with the regular expression to include any matching table. |
| Included Source Table Pattern |
Regular Expression for specifying table names to replicate e.g. customer.(orders|payments). This is combined with the comma-separated list to include any matching table. |
## Relationships
| Name |
Description |
| failure |
If a FlowFile attribute cannot be read or is incorrect, it will be routed to this Relationship. |
| matched |
Successfully created FlowFile, with a list of matching tables found in the source database. |
## Writes attributes
| Name |
Description |
| source.schema.name |
Name of the schema of the table from which an event originated |
| source.table.name |
Name of the table from which an event originated |
| source.entry |
The original entry that was attempted to parse when processing table names |
| reason |
Reason why table cannot be replicated |
| source.database.version.major |
The major version of the source database. |
| mime.type |
The MIME type of the FlowFile content. |
---
title: ListUnityCatalogDirectory 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/listunitycatalogdirectory.md
section: Loading & Unloading Data
---
# ListUnityCatalogDirectory 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-databricks-processors-nar
## Description
List file names in a Unity Catalog directory and output a new FlowFile with the filename.
## Tags
databricks, openflow, unity catalog
## Input Requirement
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Databricks Client |
Databricks Client Service. |
| Include Directories |
Include directories in FlowFiles produced. |
| Recursive Directory Listing |
Recursively list files in sub directories. |
| Unity Catalog Directory Path |
Unity Catalog directory path e.g. /Volumes/catalog/schema/volume_name/directory |
## Relationships
| Name |
Description |
| failure |
Databricks failure relationship |
| original |
The original FlowFile is routed to this relationship when processing is successful. |
| success |
Databricks success relationship |
## Writes attributes
| Name |
Description |
| filename |
Base filename of the Unity Catalog file or directory. |
| path |
Path to parent directory containing the Unity Catalog file or directory. |
| absolute.path |
Full path to the Unity Catalog file or directory. |
| uc.resourceType |
The type of resource, 'file' or 'directory' of the Unity Catalog resource. |
| uc.size |
The size of the Unity Catalog file. |
| uc.lastModifiedTime |
The last modified time of the Unity Catalog file in milliseconds since epoch in UTC time. |
| error.code |
The error code for the SQL statement if an error occurred. |
| error.message |
The error message for the SQL statement if an error occurred. |
---
title: LogAttribute 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/logattribute.md
section: Loading & Unloading Data
---
# LogAttribute 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Emits attributes of the FlowFile at the specified log level
## Tags
attributes, logging
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Attributes to Ignore |
A comma-separated list of Attributes to ignore. If not specified, no attributes will be ignored unless _Attributes to Ignore by Regular Expression_ is modified. There's an OR relationship between the two properties. |
| Attributes to Log |
A comma-separated list of Attributes to Log. If not specified, all attributes will be logged unless _Attributes to Log by Regular Expression_ is modified. There's an AND relationship between the two properties. |
| Log FlowFile Properties |
Specifies whether or not to log FlowFile "properties", such as Entry Date, Lineage Start Date, and content size |
| Log Level |
The Log Level to use when logging the Attributes |
| Log Payload |
If true, the FlowFile's payload will be logged, in addition to its attributes; otherwise, just the Attributes will be logged. |
| Log prefix |
Log prefix appended to the log lines. It helps to distinguish the output of multiple LogAttribute processors. |
| Output Format |
Specifies the format to use for logging FlowFile attributes |
| attributes-to-ignore-regex |
A regular expression indicating the Attributes to Ignore. If not specified, no attributes will be ignored unless _Attributes to Ignore_ is modified. There's an OR relationship between the two properties. |
| attributes-to-log-regex |
A regular expression indicating the Attributes to Log. If not specified, all attributes will be logged unless _Attributes to Log_ is modified. There's an AND relationship between the two properties. |
| character-set |
The name of the CharacterSet to use |
## Relationships
| Name |
Description |
| success |
All FlowFiles are routed to this relationship |
---
title: LoggingRecordSink
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/loggingrecordsink.md
section: Loading & Unloading Data
---
# LoggingRecordSink
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides a RecordSinkService that can be used to log records to the application log (nifi-app.log, e.g.) using the specified writer for formatting.
## Tags
log, record, sink
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Log Level * |
logsink-log-level |
INFO |
- TRACE
- DEBUG
- INFO
- WARN
- ERROR
- FATAL
- NONE
|
The Log Level at which to log records (INFO, DEBUG, e.g.) |
| Record Writer * |
record-sink-record-writer |
|
|
Specifies the Controller Service to use for writing out the records. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: LogMessage 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/logmessage.md
section: Loading & Unloading Data
---
# LogMessage 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Emits a log message at the specified log level
## Tags
attributes, logging
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| log-level |
The Log Level to use when logging the message: [trace, debug, info, warn, error] |
| log-message |
The log message to emit |
| log-prefix |
Log prefix appended to the log lines. It helps to distinguish the output of multiple LogMessage processors. |
## Relationships
| Name |
Description |
| success |
All FlowFiles are routed to this relationship |
---
title: LookupAttribute 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/lookupattribute.md
section: Loading & Unloading Data
---
# LookupAttribute 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Lookup attributes from a lookup service
## Tags
Attribute Expression Language, attributes, cache, enrich, join, lookup
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| include-empty-values |
Include null or blank values for keys that are null or blank |
| lookup-service |
The lookup service to use for attribute lookups |
## Relationships
| Name |
Description |
| failure |
FlowFiles with failing lookups are routed to this relationship |
| matched |
FlowFiles with matching lookups are routed to this relationship |
| unmatched |
FlowFiles with missing lookups are routed to this relationship |
---
title: LookupRecord 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/lookuprecord.md
section: Loading & Unloading Data
---
# LookupRecord 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Extracts one or more fields from a Record and looks up a value for those fields in a LookupService. If a result is returned by the LookupService, that result is optionally added to the Record. In this case, the processor functions as an Enrichment processor. Regardless, the Record is then routed to either the 'matched' relationship or 'unmatched' relationship (if the 'Routing Strategy' property is configured to do so), indicating whether or not a result was returned by the LookupService, allowing the processor to also function as a Routing processor. The "coordinates" to use for looking up a value in the Lookup Service are defined by adding a user-defined property. Each property that is added will have an entry added to a Map, where the name of the property becomes the Map Key and the value returned by the RecordPath becomes the value for that key. If multiple values are returned by the RecordPath, then the Record will be routed to the 'unmatched' relationship (or 'success', depending on the 'Routing Strategy' property's configuration). If one or more fields match the Result RecordPath, all fields that match will be updated. If there is no match in the configured LookupService, then no fields will be updated. I.e., it will not overwrite an existing value in the Record with a null value. Please note, however, that if the results returned by the LookupService are not accounted for in your schema (specifically, the schema that is configured for your Record Writer) then the fields will not be written out to the FlowFile.
## Tags
avro, convert, csv, database, db, enrichment, filter, json, logs, lookup, record, route
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Root Record Path |
A RecordPath that points to a child Record within each of the top-level Records in the FlowFile. If specified, the additional RecordPath properties will be evaluated against this child Record instead of the top-level Record. This allows for performing enrichment against multiple child Records within a single top-level Record. |
| lookup-service |
The Lookup Service to use in order to lookup a value in each Record |
| record-path-lookup-miss-result-cache-size |
Specifies how many lookup values/records should be cached. Setting this property to zero means no caching will be done and the table will be queried for each lookup value in each record. If the lookup table changes often or the most recent data must be retrieved, do not use the cache. |
| record-reader |
Specifies the Controller Service to use for reading incoming data |
| record-update-strategy |
This property defines the strategy to use when updating the record with the value returned by the Lookup Service. |
| record-writer |
Specifies the Controller Service to use for writing out the records |
| result-contents |
When a result is obtained that contains a Record, this property determines whether the Record itself is inserted at the configured path or if the contents of the Record (i.e., the sub-fields) will be inserted at the configured path. |
| result-record-path |
A RecordPath that points to the field whose value should be updated with whatever value is returned from the Lookup Service. If not specified, the value that is returned from the Lookup Service will be ignored, except for determining whether the FlowFile should be routed to the 'matched' or 'unmatched' Relationship. |
| routing-strategy |
Specifies how to route records after a Lookup has completed |
## Relationships
| Name |
Description |
| failure |
If a FlowFile cannot be enriched, the unchanged FlowFile will be routed to this relationship |
| success |
All records will be sent to this Relationship if configured to do so, unless a failure occurs |
## Writes attributes
| Name |
Description |
| mime.type |
Sets the mime.type attribute to the MIME Type specified by the Record Writer |
| record.count |
The number of records in the FlowFile |
## See also
- [org.apache.nifi.processors.standard.ConvertRecord](/user-guide/data-integration/openflow/processors/convertrecord)
- [org.apache.nifi.processors.standard.SplitRecord](/user-guide/data-integration/openflow/processors/splitrecord)
---
title: Maintain Openflow Connector for Amazon Kinesis Data Streams
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/kinesis/maintenance.md
section: Loading & Unloading Data
---
# Maintain %kinesis%
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
- [Openflow Connector for Amazon Kinesis Data Streams](/user-guide/data-integration/openflow/connectors/kinesis/about)
- [Set up Openflow Connector for Amazon Kinesis Data Streams](/user-guide/data-integration/openflow/connectors/kinesis/setup)
- [Troubleshooting the Openflow Connector for Amazon Kinesis Data Streams](/user-guide/data-integration/openflow/connectors/kinesis/troubleshoot)
- [Performance tuning of the Openflow Connector for Amazon Kinesis Data Streams](/user-guide/data-integration/openflow/connectors/kinesis/performance-tuning)
This topic describes how to maintain the %kinesis% connector, including how to manage and reset the connector state.
## Manage connector state
The %kinesis% connector uses DynamoDB to store the consumer application state.
### DynamoDB table created by the connector
The connector creates a DynamoDB table with the name specified in `AWS Kinesis Application Name`.
The table stores the checkpointed sequence number for each shard in the stream. This tracks which records have been processed.
If multiple processors use the same application name, they cooperate to consume data from the stream
and share this table. If processors have different application names, each creates its own table
to independently track consumed records.
## Reset the connector state
If the connector state in DynamoDB becomes corrupted or inconsistent, you may need to reset it.
There are two approaches to reset the connector state.
### Reset by changing the application name
The simplest way to reset the connector state is to change the AWS Kinesis Application Name parameter:
1. Stop the connector.
2. Navigate to the connector's parameter context.
3. Change the `AWS Kinesis Application Name` parameter value to a new value.
4. Start the connector.
The connector creates a new DynamoDB table with the new application name and begins consuming
records from the position specified by the [AWS Kinesis Initial Stream Position](#label-kinesis-json-source-parameters) parameter.
- When you change the application name, the connector doesn't delete the old DynamoDB table.
You must manually delete it through the AWS Console or the AWS CLI.
- If your IAM policy restricts DynamoDB access to a specific table name, you must update the policy
to allow access to the new table name. For more information on configuring IAM permissions,
see [Set up Openflow Connector for Amazon Kinesis Data Streams](/user-guide/data-integration/openflow/connectors/kinesis/setup).
### Reset by deleting the DynamoDB table
Alternatively, you can delete the existing DynamoDB table to reset the state:
1. Stop the connector.
2. In the AWS Console or using the AWS CLI, delete the DynamoDB table associated with the application name.
3. Start the connector.
The connector recreates the table and begins consuming records from the position specified by the [AWS Kinesis Initial Stream Position](#label-kinesis-json-source-parameters) parameter.
Resetting the connector state causes the connector to reprocess records from the position specified by the
initial stream position. Depending on your [AWS Kinesis Initial Stream Position](#label-kinesis-json-source-parameters) setting,
this may result in duplicate data being ingested into Snowflake or data not being ingested at all.
---
title: Maintain the Openflow Connector for Shopify
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/shopify/maintain.md
section: Loading & Unloading Data
---
# Maintain the %shopifyof%
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
- [About the Openflow Connector for Shopify](/user-guide/data-integration/openflow/connectors/shopify/about)
- [Set up the Openflow Connector for Shopify](/user-guide/data-integration/openflow/connectors/shopify/setup)
- [Object definition overrides for the Openflow Connector for Shopify](/user-guide/data-integration/openflow/connectors/shopify/object-definitions)
- [Troubleshoot the Openflow Connector for Shopify](/user-guide/data-integration/openflow/connectors/shopify/troubleshoot)
This topic describes maintenance tasks for the %shopifyof%, including how to reset connector state to trigger a fresh bulk load.
## Reset connector state
The connector maintains an internal state to track bulk-load completion status and the
incremental watermark for each object type. In some situations, you might need to
reset the connector to perform a fresh bulk load, for example, after resolving a data issue
or after the connector has been stopped for an extended period.
### Reset all objects
To reset all objects and force a full bulk reload:
1. Stop all processors in the flow by right-clicking on the connector process group and
selecting **Stop**.
2. Ensure that no in-flight FlowFiles are being processed. You can verify this by checking
that all queues in the flow are empty.
3. Right-click on the canvas and select **Disable all controller services**.
4. Go to **Controller services** and locate the **Shopify State Service**.
5. Select the menu for **Shopify State Service**, then select **View state** and select
**Clear state**.
6. Right-click on the canvas and select **Enable all controller services**, then start all
processors to resume the connector.
The connector treats a cleared state as a fresh start and performs a bulk load for all
configured objects on the next execution.
### Reset a specific object
To reset a single object type and re-ingest it from scratch without affecting other objects:
1. Stop all processors in the flow.
2. Ensure all queues are empty.
3. Right-click on the canvas and select **Disable all controller services**.
4. Go to **Controller services** and locate the **Shopify State Service**.
5. Select the menu for **Shopify State Service**, then select **View state**.
6. Select the trash icon next to the specific object type (for example, `orders`) to delete
its state entry.
7. Re-enable all controller services and start the flow.
The connector performs a fresh bulk load for that object type and then resumes incremental
updates, while other objects continue from their existing watermark.
---
title: Manage Openflow
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/manage.md
section: Loading & Unloading Data
---
# Manage Openflow
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
- [About Openflow](/user-guide/data-integration/openflow/about)
- [Set up Openflow - BYOC](/user-guide/data-integration/openflow/setup-openflow-byoc)
- [Set up Openflow - Snowflake Deployment - Task overview](/user-guide/data-integration/openflow/setup-openflow-spcs)
- [Monitor Openflow using telemetry data](/user-guide/data-integration/openflow/monitor)
- [Snowflake Openflow version history](/user-guide/data-integration/openflow/version-history)
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
This topic covers the following management tasks:
- [Back up flow definitions and protect runtime state](#back-up-flow-definitions-and-protect-runtime-state)
- [Runtime availability and autoscaling behavior](#runtime-availability-and-autoscaling-behavior)
- [Delete a deployment](#delete-a-deployment)
- [Upgrade a deployment](#upgrade-a-deployment)
- [Upgrade a runtime](#upgrade-a-runtime)
- [Upgrade a connector](#upgrade-a-connector)
## Back up flow definitions and protect runtime state
Flow definitions and runtime-local state (including processor configuration and Apache NiFi flow state held on the runtime) live on **Openflow runtime storage**, not in Snowflake tables. If you remove, replace, or manually tear down that infrastructure **without** exporting your flows first, that data can be **lost permanently**. Snowflake does not provide Time Travel or Fail-safe for this storage.
Before you delete a deployment, delete or recreate a runtime, or manually remove underlying Snowpark Container Services resources or compute tied to Openflow, **export** your flows from the canvas. Right-click the **process group** %ra% **Version** %ra% **Export** (or use the equivalent command your canvas shows).
**Routine upgrades** through the supported Openflow UI ([Upgrade a deployment](#label-update-a-deployment) and [Upgrade a runtime](#label-openflow-upgrading-a-runtime)) are different from destructive removal. You should still export flows regularly as a best practice.
Do not run [DROP ROLE](/sql-reference/sql/drop-role) for a role that provisions or owns Openflow objects until you transfer ownership and privileges to another role you intend to keep (for example with `GRANT OWNERSHIP`). Dropping a role revokes grants and can leave deployments in a broken state.
## Runtime availability and autoscaling behavior
Openflow runtime nodes are not strictly always-on, single-host processes.
Each runtime is a Kubernetes workload that the cluster can reschedule onto
a different compute host. When that happens, the runtime briefly restarts
while a new pod becomes ready. Plan your flows to tolerate short
interruptions rather than assuming the runtime stays on the same host
indefinitely.
Snowflake doesn't automatically upgrade BYOC runtimes. Upgrades happen
only when a deployment owner initiates them through the Openflow UI or
the deployment agent. Restarts you observe outside of an upgrade window
are typically caused by cluster rebalancing or by host-level events on
the underlying compute.
For Openflow Snowflake deployments running on
[Snowpark Container Services](/developer-guide/snowpark-container-services/overview)
(SPCS), runtimes can also be affected briefly by the scheduled SPCS
[maintenance window](/developer-guide/snowpark-container-services/working-with-compute-pool#label-spcs-working-with-compute-pool-maintenance-window).
### Causes of runtime restarts
- Runtime or deployment upgrades
-
When the owner of a deployment runs an upgrade, the affected runtime
restarts to pick up the new version. See
[Upgrade a runtime](#label-openflow-upgrading-a-runtime) and
[Upgrade a deployment](#label-update-a-deployment).
- Cluster rebalancing and autoscaling
-
Openflow scales the underlying compute up and down based on demand.
See [Openflow BYOC cost and scaling considerations](/user-guide/data-integration/openflow/cost-byoc) for details on
how BYOC deployments scale the EC2 node group. During scale-in,
node-drain, or rebalancing events, the cluster can reschedule a runtime
pod from one node to another so that the cluster continues to run
efficiently.
- Cloud provider host events
-
The virtual machines that host BYOC runtimes are subject to events
outside Snowflake's control, including instance retirement, unexpected
reboots, and host-level maintenance performed by the cloud service
provider. When a host becomes unavailable, the cluster reschedules the
affected runtime onto a healthy node.
### What to expect during a restart
- Openflow runtimes and connectors maintain data integrity across
restarts. In-flight data held in the runtime's persistent storage is
preserved, and the flow resumes after the new pod is ready.
- Expect a short service interruption while the new pod starts and
reattaches its storage.
- Diagnostic output may report
`LAST_REQUESTED_RESTART_REASON: "nifi.properties changed"` after a
reschedule, even when no NiFi configuration was modified. The runtime
operator reconciles the underlying StatefulSet whenever the pod
identity or node assignment changes, so this message can reflect a
reschedule rather than an actual configuration change.
### Design flows for resilience
Because brief runtime interruptions are expected, design your flows to
recover automatically:
- Configure source and destination connectors to checkpoint progress so
that processing resumes from the last committed position after a
restart.
- For streaming sources such as
[Kafka](/user-guide/data-integration/openflow/connectors/kafka/about)
or [Kinesis](/user-guide/data-integration/openflow/connectors/kinesis/about),
rely on consumer-group offsets or sequence numbers rather than
in-memory state on the runtime.
- [Monitor your runtimes](/user-guide/data-integration/openflow/monitor)
so that you're notified if a restart doesn't recover on its own within
the expected window.
- Choose your caching strategy with restarts in mind. A local, in-memory
cache is cleared when a runtime node restarts, and Openflow's locally
persisted caches are managed per runtime node rather than shared across
the cluster. If your flow depends on cache state surviving restarts or
being shared across nodes, use an external cache service such as Redis.
## Delete a deployment
Deleting a deployment removes the management compute pool and all deployment-level
configuration. You must delete all runtimes first. Any data or objects
already integrated into Snowflake aren't affected.
Deleting a deployment can't be undone. Before you delete, make sure all runtimes
have been removed and you no longer need the deployment configuration.
**Gen 2 (SQL):** Drop the deployment directly:
```sql
DROP OPENFLOW DEPLOYMENT my_deployment;
```
**Gen 1 BYOC (AWS Console):**
1. Navigate to EC2 Instances.
2. Select the `openflow-agent-{deployment-key}` instance with your deployment key.
3. Click **Connect** at the top of the page.
4. Switch from **EC2 Instance Connect** to **Connect using EC2 Instance Connect Endpoint**. Leave the default EC2 Instance Connect Endpoint
in place.
5. Click **Connect**. A new browser tab or window will appear with a
command-line interface.
6. Run `./destroy.sh` from the shell.
- This may take 20-30 minutes. If your connection is interrupted, the process continues running in the background.
- You can log back in and view its status with the command: `journalctl -u docker -f -n 250`
- The `destroy` process is complete when you see output of `delete successful`.
7. Navigate to
CloudFormation (https://us-east-1.console.aws.amazon.com/cloudformation/home)
in the AWS Console for your region.
8. Delete the CloudFormation stack for your deployment.
From Snowsight:
1. In the navigation menu, select **Ingestion** %raa% **Openflow**.
2. Select **Launch Openflow**.
3. Select the **Deployments** tab.
4. In the row of the deployment you want to delete, select the More options icon.
5. Select **Delete**.
6. In the confirmation dialog, type `delete` to confirm deletion.
7. Click **Delete deployment**.
## Upgrade a deployment
A deployment includes several components: the agent, deployment service, deployment UI,
runtime gateway, and runtime operator. You can upgrade Snowflake deployments and eligible
BYOC deployments directly from the UI: on the **Deployments** tab, an eligible deployment
shows an **Upgrade** option in its More options (%sf-vertical-more-button%) menu. If a BYOC
deployment isn't eligible, that option doesn't appear, so upgrade it using the deployment
agent script instead. For details on what's included in each release, see
[Openflow version history](/user-guide/data-integration/openflow/version-history).
**Snowflake deployments** are upgraded automatically by Snowflake on a rolling basis. If your
deployment is on an older version, it will be upgraded to the latest version automatically;
you do not need to initiate the upgrade yourself. Once a deployment has been upgraded to a
recent version, it will continue to receive automatic upgrades going forward.
**BYOC deployments** are not upgraded automatically. You determine upgrade timing and
frequency using the [deployment agent script](#upgrade-via-the-deployment-agent-byoc) or the
[UI](#upgrade-from-the-ui).
### Upgrade from the UI
The UI upgrade path applies to BYOC deployments. Snowflake deployments are upgraded
automatically and do not require manual intervention.
1. Sign in to %sf-web-interface-link%.
2. In the navigation menu, select **Ingestion** %raa% **Openflow**.
3. Select **Launch Openflow**.
4. Select the **Deployments** tab.
5. Look for the upgrade arrow to the left of the deployment name. This indicates an upgrade is available.
6. Select %sf-vertical-more-button% next to the deployment %raa% **Upgrade**.
### Upgrade via the deployment agent (BYOC)
Use the deployment agent script for older BYOC deployments that cannot be upgraded via the UI, or when you prefer to upgrade manually. This upgrades the agent, deployment service, deployment UI, ingress controller, runtime operator, and all other component dependencies.
#### Connect to the deployment agent
1. Navigate to Openflow.
2. Select the **Deployments** tab.
3. View your deployment details and note the deployment key.
4. In your AWS account, view the EC2 instances and filter using the deployment key.
5. Locate the deployment agent EC2 instance named `openflow-agent-{deployment-key}`.
6. Connect using EC2 Instance Connect Endpoint and accepting all defaults.
7. Run the remaining commands from the new browser tab or window that appears with a command-line interface.
#### Check for available upgrades
```bash
cat ~/.upgrade
```
The script will display the latest available version of the various deployment components.
If no upgrades are available, you will see an output similar to this:
```text
AGENT_IMAGE_VERSION_UPGRADE=
OPERATOR_CHART_VERSION_UPGRADE=
GATEWAY_IMAGE_VERSION_UPGRADE=
DPS_CHART_VERSION_UPGRADE=
DPUI_CHART_VERSION_UPGRADE=
```
Otherwise, you will see the version that upgraded components will use, such as:
```text
AGENT_IMAGE_VERSION_UPGRADE=0.17.0
OPERATOR_CHART_VERSION_UPGRADE=0.31.0
GATEWAY_IMAGE_VERSION_UPGRADE=
DPS_CHART_VERSION_UPGRADE=
DPUI_CHART_VERSION_UPGRADE=
```
#### Upgrading the AMI for the Openflow BYOC deployment
When you upgrade your Openflow BYOC deployment, Openflow will find and upgrade to the latest AMI for Amazon Linux 2023 recommended by
AWS Systems Manager (https://aws.amazon.com/systems-manager/).
If a new AMI is found, it will restart all Openflow services in your deployment, and runtimes will be temporarily halted.
Openflow runtimes and connectors maintain data integrity across restarts automatically.
Snowflake does not automatically upgrade deployments. You determine upgrade timing and frequency.
#### Initiate the upgrade
If the output indicates that upgrades are available, run the following script to initiate the upgrade. Older Openflow deployments may use the script `upgrade-data-plane.sh` instead.
```bash
./upgrade.sh
```
You will see output similar to this:
```text
openflow-data-plane-agent-aws is set to version 0.16.0
Upgrade set to version 0.17.0
openflow-dataplane-service-chart is set to version 0.47.0
No upgrade is available
openflow-dataplane-ui-chart is set to version 0.5.0
No upgrade is available
openflow-runtime-gateway is set to version 2025.6.8.2
No upgrade is available
runtime-operator-chart is set to version 0.30.0
Upgrade set to version 0.31.0
```
Then, you have two options:
- Wait for an automatic upgrade: The system will automatically initiate the upgrade process within approximately 10 minutes.
- Manual upgrade: To start the upgrade immediately, run the following command:
```bash
./create.sh
```
#### Monitor the upgrade process
To track the progress of the upgrade, use the `journalctl` command:
```bash
journalctl -u openflow-apply-infrastructure -f -n 250
```
#### Verify a successful upgrade
A successful upgrade will typically show output similar to this:
```text
All resources applied successfully and log uploaded to s3
openflow-apply-infrastructure.service: Deactivated successfully
```
## Upgrade a runtime
Snowflake periodically releases runtime updates that introduce new Openflow processors, newer versions of
existing processors, or new runtime functionality. When updates are available, an indicator
appears next to the runtime name in the UI. For details on what's included in each release, see
[Openflow version history](/user-guide/data-integration/openflow/version-history).
Only the owner of a deployment can perform an upgrade.
1. Sign in to %sf-web-interface-link%.
2. In the navigation menu, select **Ingestion** %raa% **Openflow**.
3. Select **Launch Openflow**.
4. Select the **Runtimes** tab.
5. Look for the upgrade arrow to the left of the runtime name. This indicates an upgrade is available.
6. Select %sf-vertical-more-button% next to the runtime %raa% **Upgrade**.
## Upgrade a connector
Connector updates are made available by Snowflake when functionality is added,
processing logic is improved, or new processor versions are used–for example, to add support for a new source API version.
This section describes upgrading **gen 1** connectors on the runtime canvas. For gen 2 connectors,
see [Manage the gen 2 Openflow connector lifecycle](/user-guide/data-integration/openflow/gen2/manage-connector-lifecycle).
When connector updates are available, you will see an **Upgrade** icon in your process group on the canvas.
You can only upgrade connectors after you have [upgraded their runtime](#label-openflow-upgrading-a-runtime).
To upgrade a connector, do the following:
1. In the navigation menu, select **Ingestion** %raa% **Openflow**.
2. Select **Launch Openflow**.
3. Select the **Runtimes** tab.
4. Select the runtime name, or select **View Canvas** in the **More Options** menu to navigate to the canvas.
5. Find the processor groups with a red upgrade arrow next to their names. For each of these groups, change the version:
1. Recommended: Check to see whether the parameter uses a custom value for the Parameter context. If so, make a note of the custom value. You will need to reapply it after the upgrade.
1. Right-click the process group and select **Parameters**.
2. Select **Parameters** in the Parameter Contexts list.
3. Select the **Inheritance** tab, and check if it uses custom values. If so, make a note of the custom values.
2. Right-click the group and select **Version** %ra% **Change Version**.
3. Select the latest available version and select **Change**.
4. Confirm that the connector was upgraded to the latest version. The upgraded version should show a green check mark.
5. Confirm that all processors in the connector's process group are running. If not, start them.
You can also validate the version by hovering over the speech bubble at the bottom right of the process group.
6. If you noted a custom parameter value in step 4, reapply the custom value. For more information, see [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors).
### Configure Snowflake Connector Flow Registry
Early preview releases of Openflow did not configure a runtime for connector upgrades.
If you don't see the Version option when right clicking on a process group, you
have to configure the Snowflake Connector Flow Registry and manually enable version control for existing connectors.
To configure the Snowflake Connector Flow Registry, do the following:
1. Navigate to the canvas.
2. Click on the menu in the top right corner and select **Controller Settings**.
3. Switch to the **Registry Clients** tab.
4. Click the **+** icon to add a new Registry Client.
5. Select the **ConnectorFlowRegistryClient** and select **Add**.
6. Click **More Options** for the **ConnectorFlowRegistryClient** row and select **Edit**.
7. Enter `/nifi/configuration_resources/connector_flow_registry` as the value
for **Storage Location** and select **Apply**.
After configuring the Snowflake Connector Flow Registry you can now enable version control for your existing connectors.
To enable version control for existing connectors, do the following:
1. Navigate to the canvas and locate the process group where you want to add version control.
2. Right click on the process group and select **Version** %raa% **Set Version**.
3. In the **Set Version** dialog, choose the flow that matches your process group.
For example, choose **sqlserver** if you are using the SQL Server connector.
Note that flow names do not exactly match the connector name.
4. Select the latest version and then select **Set version** to enable version control.
5. From the canvas, right click on the process group again and select **Version** %raa% **Revert Local Changes**
to apply the latest connector version.
6. Review the list of changes and select **Revert**.
7. Confirm that your connector was upgraded to the latest version which should now show a green check mark.
You can also validate the version by hovering over the speech bubble at the bottom right of the process group.
---
title: Manage the gen 2 Openflow connector lifecycle
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/gen2/manage-connector-lifecycle.md
section: Loading & Unloading Data
---
# Manage the gen 2 Openflow connector lifecycle
Available to all accounts.
- [Second generation Openflow objects and interfaces](/user-guide/data-integration/openflow/gen2/index)
- [Openflow gen 1 and gen 2](/user-guide/data-integration/openflow/gen2/openflow-generations)
- [Configure a connector with the setup wizard](/user-guide/data-integration/openflow/gen2/setup-connector-wizard)
- [Monitor connectors using the Openflow Connectors Dashboard](/user-guide/data-integration/openflow/connectors-dashboard)
- [Manage Openflow](/user-guide/data-integration/openflow/manage)
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
- [ALTER OPENFLOW CONNECTOR](/sql-reference/sql/alter-openflow-connector)
- [SYSTEM$WAIT_FOR_STABLE_OPENFLOW_CONNECTORS](/sql-reference/functions/system_wait_for_stable_openflow_connectors)
This topic describes how to manage the **gen 2** Openflow connector lifecycle: **start and stop**
ingestion and **remove** a gen 2 connector after it is created. To monitor connector health,
throughput, and ingestion status, use the
[Openflow Connectors Dashboard](/user-guide/data-integration/openflow/connectors-dashboard).
For gen 1 connectors, follow the lifecycle guidance in each connector's public setup topic. See
[Openflow gen 1 and gen 2](/user-guide/data-integration/openflow/gen2/openflow-generations) for how to tell gen 1 resources from gen 2.
These tasks apply no matter how the gen 2 connector was created: using the
[Configure a connector with the setup wizard](/user-guide/data-integration/openflow/gen2/setup-connector-wizard), [Configure a gen 2 connector with SQL](/user-guide/data-integration/openflow/gen2/configure-connector-sql),
or other supported SQL/API automation.
## Start and stop data movement
After a gen 2 connector is installed on a runtime, start it to begin reading from the source and
writing to Snowflake. Use **Start** from the connector's menu on the **Installed Connectors** tab.
Use **Stop** when you need to pause ingestion—for example before maintenance, upgrades described in your
connector's documentation, or before removal. Stopping leaves the connector installed but idle.
Some connectors retain external resources while stopped (for example, database replication slots).
Do not leave connectors stopped for long periods on busy sources unless you understand the impact; see
your connector's setup or maintenance topic.
For **gen 2** connectors, use **Start** and **Stop** from the connector menu on
**Installed Connectors**, or run `ALTER OPENFLOW CONNECTOR ... START` or `STOP` with SQL. Do
not use the canvas for configuration or processor-level start/stop. See
[ALTER OPENFLOW CONNECTOR](/sql-reference/sql/alter-openflow-connector) for syntax and wait functions.
## Remove a connector
Gen 2 connector removal follows **stop** → **terminate** → **drop**. **`TERMINATE` drains**
in-flight data before removal. Complete each step before starting the next.
In UI-driven workflows, wait for each step to finish before starting the next. In scripts, call
`SYSTEM$WAIT_FOR_STABLE_OPENFLOW_CONNECTORS` after asynchronous `ALTER` commands. See
[SYSTEM$WAIT_FOR_STABLE_OPENFLOW_CONNECTORS](/sql-reference/functions/system_wait_for_stable_openflow_connectors).
### Remove a connector (UI)
1. From the **Installed Connectors** tab, open the connector **menu** and select **Stop**.
2. From the connector **menu** on **Installed Connectors**, select **Delete** (terminates the
connector and drains in-flight data).
3. From the same menu, select **Drop**.
### Remove a connector (SQL)
**Delete** in the UI corresponds to `ALTER OPENFLOW CONNECTOR ... TERMINATE`; **Drop** corresponds
to `DROP OPENFLOW CONNECTOR`. `DROP` requires `OWNERSHIP` on the connector.
```sql
ALTER OPENFLOW CONNECTOR my_db.my_schema.my_connector STOP;
SELECT SYSTEM$WAIT_FOR_STABLE_OPENFLOW_CONNECTORS(600, 'my_db.my_schema.my_connector');
ALTER OPENFLOW CONNECTOR my_db.my_schema.my_connector TERMINATE;
SELECT SYSTEM$WAIT_FOR_STABLE_OPENFLOW_CONNECTORS(600, 'my_db.my_schema.my_connector');
DROP OPENFLOW CONNECTOR my_db.my_schema.my_connector;
```
For full command syntax, privileges, and additional `ALTER` options, see
[ALTER OPENFLOW CONNECTOR](/sql-reference/sql/alter-openflow-connector).
**Delete**, **Drop**, `TERMINATE`, and `DROP OPENFLOW CONNECTOR` are irreversible for Openflow
entities. Snowflake does not support undrop for these objects. These steps do not remove destination
tables or external resources (such as PostgreSQL replication slots). Read confirmation dialogs
carefully.
For source-specific cleanup after removal (for example PostgreSQL replication slots), see **Stop or delete
the connector** in [Set up the Openflow Connector for PostgreSQL](/user-guide/data-integration/openflow/connectors/postgres/setup) and analogous
sections for other connectors.
---
title: MapCacheClientService
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/mapcacheclientservice.md
section: Loading & Unloading Data
---
# MapCacheClientService
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides the ability to communicate with a MapCacheServer. This can be used in order to share a Map between nodes in a NiFi cluster
## Tags
cache, cluster, distributed, map, state
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Communications Timeout * |
Communications Timeout |
30 secs |
|
Specifies how long to wait when communicating with the remote server before determining that there is a communications failure if data cannot be sent or received |
| SSL Context Service |
SSL Context Service |
|
|
If specified, indicates the SSL Context Service that is used to communicate with the remote server. If not specified, communications will not be encrypted |
| Server Hostname * |
Server Hostname |
|
|
The name of the server that is running the DistributedMapCacheServer service |
| Server Port * |
Server Port |
4557 |
|
The port on the remote server that is to be used when communicating with the DistributedMapCacheServer service |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: MapCacheServer
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/mapcacheserver.md
section: Loading & Unloading Data
---
# MapCacheServer
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides a map (key/value) cache that can be accessed over a socket. Interaction with this service is typically accomplished via a Map Cache Client Service.
## Tags
cache, cluster, distributed, key/value, map, server
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Eviction Strategy * |
Eviction Strategy |
Least Frequently Used |
- Least Frequently Used
- Least Recently Used
- First In, First Out
|
Determines which strategy should be used to evict values from the cache to make room for new entries |
| Maximum Cache Entries * |
Maximum Cache Entries |
10000 |
|
The maximum number of cache entries that the cache can hold |
| Persistence Directory |
Persistence Directory |
|
|
If specified, the cache will be persisted in the given directory; if not specified, the cache will be in-memory only |
| Port * |
Port |
4557 |
|
The port to listen on for incoming connections |
| SSL Context Service |
SSL Context Service |
|
|
If specified, this service will be used to create an SSL Context that will be used to secure communications; if not specified, communications will not be secure |
| Maximum Read Size |
maximum-read-size |
1 MB |
|
The maximum number of network bytes to read for a single cache item |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: MergeContent 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/mergecontent.md
section: Loading & Unloading Data
---
# MergeContent 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Merges a Group of FlowFiles together based on a user-defined strategy and packages them into a single FlowFile. It is recommended that the Processor be configured with only a single incoming connection, as Group of FlowFiles will not be created from FlowFiles in different connections. This processor updates the mime.type attribute as appropriate. NOTE: this processor should NOT be configured with Cron Driven for the Scheduling Strategy.
## Tags
archive, concatenation, content, correlation, flowfile-stream, flowfile-stream-v3, merge, stream, tar, zip
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Attribute Strategy |
Determines which FlowFile attributes should be added to the bundle. If 'Keep All Unique Attributes' is selected, any attribute on any FlowFile that gets bundled will be kept unless its value conflicts with the value from another FlowFile. If 'Keep Only Common Attributes' is selected, only the attributes that exist on all FlowFiles in the bundle, with the same value, will be preserved. |
| Bin Termination Check |
Specifies an Expression Language Expression that is to be evaluated against each FlowFile. If the result of the expression is 'true', the bin that the FlowFile corresponds to will be terminated, even if the bin has not met the minimum number of entries or minimum size. Note that if the FlowFile that triggers the termination of the bin is itself larger than the Maximum Bin Size, it will be placed into its own bin without triggering the termination of any other bin. When using this property, it is recommended to use Prioritizers in the flow's connections to ensure that the ordering is as desired. |
| Compression Level |
Specifies the compression level to use when using the Zip Merge Format; if not using the Zip Merge Format, this value is ignored |
| Correlation Attribute Name |
If specified, like FlowFiles will be binned together, where 'like FlowFiles' means FlowFiles that have the same value for this Attribute. If not specified, FlowFiles are bundled by the order in which they are pulled from the queue. |
| Delimiter Strategy |
Determines if Header, Footer, and Demarcator should point to files containing the respective content, or if the values of the properties should be used as the content. |
| Demarcator File |
Filename or text specifying the demarcator to use. If not specified, no demarcator is supplied. |
| FlowFile Insertion Strategy |
If a given FlowFile terminates the bin based on the <Bin Termination Check> property, specifies where the FlowFile should be included in the bin. |
| Footer File |
Filename or text specifying the footer to use. If not specified, no footer is supplied. |
| Header File |
Filename or text specifying the header to use. If not specified, no header is supplied. |
| Keep Path |
If using the Zip or Tar Merge Format, specifies whether or not the FlowFiles' paths should be included in their entry names. |
| Max Bin Age |
The maximum age of a Bin that will trigger a Bin to be complete. Expected format is <duration> <time unit> where <duration> is a positive integer and time unit is one of seconds, minutes, hours |
| Maximum Group Size |
The maximum size for the bundle. If not specified, there is no maximum. |
| Maximum Number of Entries |
The maximum number of files to include in a bundle |
| Maximum number of Bins |
Specifies the maximum number of bins that can be held in memory at any one time |
| Merge Format |
Determines the format that will be used to merge the content. |
| Merge Strategy |
Specifies the algorithm used to merge content. The 'Defragment' algorithm combines fragments that are associated by attributes back into a single cohesive FlowFile. The 'Bin-Packing Algorithm' generates a FlowFile populated by arbitrarily chosen FlowFiles |
| Minimum Group Size |
The minimum size for the bundle |
| Minimum Number of Entries |
The minimum number of files to include in a bundle |
| Tar Modified Time |
If using the Tar Merge Format, specifies if the Tar entry should store the modified timestamp either by expression (e.g. $\{file.lastModifiedTime\} or static value, both of which must match the ISO8601 format 'yyyy-MM-dd'T 'HH:mm:ssZ'. |
| mergecontent-metadata-strategy |
For FlowFiles whose input format supports metadata (Avro, e.g.), this property determines which metadata should be added to the bundle. If 'Use First Metadata' is selected, the metadata keys/values from the first FlowFile to be bundled will be used. If 'Keep Only Common Metadata' is selected, only the metadata that exists on all FlowFiles in the bundle, with the same value, will be preserved. If 'Ignore Metadata' is selected, no metadata is transferred to the outgoing bundled FlowFile. If 'Do Not Merge Uncommon Metadata' is selected, any FlowFile whose metadata values do not match those of the first bundled FlowFile will not be merged. |
## Relationships
| Name |
Description |
| failure |
If the bundle cannot be created, all FlowFiles that would have been used to created the bundle will be transferred to failure |
| merged |
The FlowFile containing the merged content |
| original |
The FlowFiles that were used to create the bundle |
## Writes attributes
| Name |
Description |
| filename |
When more than 1 file is merged, the filename comes from the segment.original.filename attribute. If that attribute does not exist in the source FlowFiles, then the filename is set to the number of nanoseconds matching system time. Then a filename extension may be applied:if Merge Format is TAR, then the filename will be appended with .tar, if Merge Format is ZIP, then the filename will be appended with .zip, if Merge Format is FlowFileStream, then the filename will be appended with .pkg |
| merge.count |
The number of FlowFiles that were merged into this bundle |
| merge.bin.age |
The age of the bin, in milliseconds, when it was merged and output. Effectively this is the greatest amount of time that any FlowFile in this bundle remained waiting in this processor before it was output |
| merge.uuid |
UUID of the merged flow file that will be added to the original flow files attributes. |
| merge.reason |
This processor allows for several thresholds to be configured for merging FlowFiles. This attribute indicates which of the Thresholds resulted in the FlowFiles being merged. For an explanation of each of the possible values and their meanings, see the Processor's Usage / documentation and see the 'Additional Details' page. |
## Use cases
| Concatenate FlowFiles with textual content together in order to create fewer, larger FlowFiles. |
| ----------------------------------------------------------------------------------------------------------------------------------------------- |
| Concatenate FlowFiles with binary content together in order to create fewer, larger FlowFiles. |
| Reassemble a FlowFile that was previously split apart into smaller FlowFiles by a processor such as SplitText, UnpackContext, SplitRecord, etc. |
## See also
- [org.apache.nifi.processors.standard.MergeRecord](/user-guide/data-integration/openflow/processors/mergerecord)
- [org.apache.nifi.processors.standard.SegmentContent](/user-guide/data-integration/openflow/processors/segmentcontent)
---
title: MergeRecord 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/mergerecord.md
section: Loading & Unloading Data
---
# MergeRecord 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
This Processor merges together multiple record-oriented FlowFiles into a single FlowFile that contains all of the Records of the input FlowFiles. This Processor works by creating 'bins' and then adding FlowFiles to these bins until they are full. Once a bin is full, all of the FlowFiles will be combined into a single output FlowFile, and that FlowFile will be routed to the 'merged' Relationship. A bin will consist of potentially many 'like FlowFiles'. In order for two FlowFiles to be considered 'like FlowFiles', they must have the same Schema (as identified by the Record Reader) and, if the <Correlation Attribute Name> property is set, the same value for the specified attribute. See Processor Usage and Additional Details for more information. NOTE: this processor should NOT be configured with Cron Driven for the Scheduling Strategy.
## Tags
content, correlation, event, merge, record, stream
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Attribute Strategy |
Determines which FlowFile attributes should be added to the bundle. If 'Keep All Unique Attributes' is selected, any attribute on any FlowFile that gets bundled will be kept unless its value conflicts with the value from another FlowFile. If 'Keep Only Common Attributes' is selected, only the attributes that exist on all FlowFiles in the bundle, with the same value, will be preserved. |
| correlation-attribute-name |
If specified, two FlowFiles will be binned together only if they have the same value for this Attribute. If not specified, FlowFiles are bundled by the order in which they are pulled from the queue. |
| max-bin-age |
The maximum age of a Bin that will trigger a Bin to be complete. Expected format is <duration> <time unit> where <duration> is a positive integer and time unit is one of seconds, minutes, hours |
| max-bin-size |
The maximum size for the bundle. If not specified, there is no maximum. This is a 'soft limit' in that if a FlowFile is added to a bin, all records in that FlowFile will be added, so this limit may be exceeded by up to the number of bytes in last input FlowFile. |
| max-records |
The maximum number of Records to include in a bin. This is a 'soft limit' in that if a FlowFIle is added to a bin, all records in that FlowFile will be added, so this limit may be exceeded by up to the number of records in the last input FlowFile. |
| max.bin.count |
Specifies the maximum number of bins that can be held in memory at any one time. This number should not be smaller than the maximum number of concurrent threads for this Processor, or the bins that are created will often consist only of a single incoming FlowFile. |
| merge-strategy |
Specifies the algorithm used to merge records. The 'Defragment' algorithm combines fragments that are associated by attributes back into a single cohesive FlowFile. The 'Bin-Packing Algorithm' generates a FlowFile populated by arbitrarily chosen FlowFiles |
| min-bin-size |
The minimum size of for the bin |
| min-records |
The minimum number of records to include in a bin |
| record-reader |
Specifies the Controller Service to use for reading incoming data |
| record-writer |
Specifies the Controller Service to use for writing out the records |
## Relationships
| Name |
Description |
| failure |
If the bundle cannot be created, all FlowFiles that would have been used to created the bundle will be transferred to failure |
| merged |
The FlowFile containing the merged records |
| original |
The FlowFiles that were used to create the bundle |
## Writes attributes
| Name |
Description |
| record.count |
The merged FlowFile will have a 'record.count' attribute indicating the number of records that were written to the FlowFile. |
| mime.type |
The MIME Type indicated by the Record Writer |
| merge.count |
The number of FlowFiles that were merged into this bundle |
| merge.bin.age |
The age of the bin, in milliseconds, when it was merged and output. Effectively this is the greatest amount of time that any FlowFile in this bundle remained waiting in this processor before it was output |
| merge.uuid |
UUID of the merged FlowFile that will be added to the original FlowFiles attributes |
| merge.completion.reason |
This processor allows for several thresholds to be configured for merging FlowFiles. This attribute indicates which of the Thresholds resulted in the FlowFiles being merged. For an explanation of each of the possible values and their meanings, see the Processor's Usage / documentation and see the 'Additional Details' page. |
| <Attributes from Record Writer> |
Any Attribute that the configured Record Writer returns will be added to the FlowFile. |
## Use cases
| Combine together many arbitrary Records in order to create a single, larger file |
| -------------------------------------------------------------------------------- |
## Use Cases Involving Other Components
| Combine together many Records that have the same value for a particular field in the data, in order to create a single, larger file |
| ----------------------------------------------------------------------------------------------------------------------------------- |
## See also
- [org.apache.nifi.processors.standard.MergeContent](/user-guide/data-integration/openflow/processors/mergecontent)
- [org.apache.nifi.processors.standard.PartitionRecord](/user-guide/data-integration/openflow/processors/partitionrecord)
- [org.apache.nifi.processors.standard.SplitRecord](/user-guide/data-integration/openflow/processors/splitrecord)
---
title: MergeSnowflakeJournalTable 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/mergesnowflakejournaltable.md
section: Loading & Unloading Data
---
# MergeSnowflakeJournalTable 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-database-cdc-processors-nar
## Description
Triggers a merge operation on changes from journal table to a destination table in Snowflake. The merge operation is performed asynchronously and the processor polls the result of the operation. If the query is still in progress the FlowFile will be penalized.
## Tags
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Destination Database Name |
The name of the Snowflake database where the data is being ingested to. |
| Merge Query Retry Count |
Indicates how many times the merge query should be retried if it fails. |
| Object Identifier Resolution |
Controls how source object identifiers (schemas, tables, columns) are stored and queried in Snowflake. This setting determines whether you will need to use double quotes in your SQL queries. The 'Case-Sensitive' option is the default, production behavior — 'Case-Insensitive' is considered preview for the time being. |
| Placeholder Value |
The value of the payload placeholder to look for in a MERGE. This will be converted to the destination column's data type. |
| Snowflake Connection Pool |
The Controller Service that is used to obtain a connection to the Snowflake database to perform merge operation. |
| Unchanged Value Strategy |
Determines how the MERGE query should handle unchanged values in journal columns. By default it expects full values. |
## Relationships
| Name |
Description |
| ddl |
DDL to execute. |
| deleted during compaction |
FlowFile deleted during compaction based on table name and generation. |
| failure |
Failure query execution. |
| failure retry |
Retry failure query execution. |
| poll query result |
Scheduled async query execution. |
| success |
Success query execution. |
| unknown file type |
Unknown file type. |
## Writes attributes
| Name |
Description |
| merge.query.id |
The ID of the query that is used to merge the journal table into the target table. |
---
title: MicrosoftClientCertificateOAuth2TokenProvider
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/microsoftclientcertificateoauth2tokenprovider.md
section: Loading & Unloading Data
---
# MicrosoftClientCertificateOAuth2TokenProvider
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides OAuth2 access tokens for the Microsoft Graph API using client_credentials with a client certificate.
## Tags
access token, authorization, graph, http, microsoft, oauth2, provider
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Client ID * |
Client ID |
|
|
The Client ID for the Microsoft Graph API |
| Refresh Window * |
Refresh Window |
5 s |
|
The service will attempt to refresh tokens expiring within the refresh window, subtracting the configured duration from the token expiration. |
| SSL Context Service * |
SSL Context Service |
|
|
An instance of SSLContextProvider configured with a certificate and a private key which will be used to sign the JWT assertion. The keys must use RSA algorithm. |
| Tenant ID * |
Tenant ID |
|
|
The Tenant ID for the Microsoft Graph API |
| Token Scope * |
Token Scope |
|
|
The scope of the requested token.For Graph API should be: https://graph.microsoft.com/.defaultFor (https://graph.microsoft.com/.defaultFor) Sharepoint should in the following format: https://organization.sharepoint.com/.default (https://organization.sharepoint.com/.default) |
| Web Client Service * |
Web Client Service |
|
|
The Web Client Service to retrieve access tokens. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: MicrosoftGraphAuthenticationProvider
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/microsoftgraphauthenticationprovider.md
section: Loading & Unloading Data
---
# MicrosoftGraphAuthenticationProvider
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides authentication for the Microsoft Graph API, which can be used for interacting with Microsoft 365 services.
## Tags
graph, microsoft, openflow
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Authentication Mechanism * |
Authentication Mechanism |
Client Secret |
- Client Secret
- Username / Password
|
The mechanism to use for authenticating with the Microsoft Graph API |
| Client ID * |
Client ID |
|
|
The Client ID for the Microsoft Graph API |
| Client Secret * |
Client Secret |
|
|
The Client Secret for the Microsoft Graph API |
| Password * |
Password |
|
|
The password to use for authentication |
| Tenant ID * |
Tenant ID |
|
|
The Tenant ID for the Microsoft Graph API |
| Username * |
Username |
|
|
The username to use for authentication |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: Migrate from the legacy Openflow Connector for Jira Cloud
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/jira-cloud/migrate-from-legacy.md
section: Loading & Unloading Data
---
# Migrate from the legacy %jira%
This feature is not available in the People's Republic of China.
- **Generally available:** The %jiracore% flow.
- **[Public Preview](https://www.snowflake.cn/en/legal/optional-offerings/offering-specific-terms/preview-terms-of-service/):** The %jiraagile% flow.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and Google Cloud Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
- [About Openflow Connector for Jira Cloud](/user-guide/data-integration/openflow/connectors/jira-cloud/about)
- [Set up the Atlassian Jira Cloud (Core) flow](/user-guide/data-integration/openflow/connectors/jira-cloud/setup-core)
- [Set up the Atlassian Jira Cloud (Agile) flow](/user-guide/data-integration/openflow/connectors/jira-cloud/setup-agile)
This topic describes how to migrate from the legacy %jira% to the new %jira%.
## Overview
The new connector is a complete rewrite that changes how data is stored in Snowflake.
It consists of two separate flows: a **core flow** (issues, projects, comments, changelogs,
worklogs, users, votes, watchers, remote links, issue security schemes, deleted issues, and lookup
tables for issue type, priority, resolution, and status) and an **agile flow** (boards, sprints,
board mappings).
The core flow and agile flow can write to the same Snowflake destination schema, since they
create tables with different names. The legacy connector and the new connector can run side by
side during migration — including on the same Openflow runtime — as long as they write to
**separate** destination schemas, so you can validate the new output before decommissioning
the legacy connector.
## Feature comparison
| Aspect |
Legacy connector |
New connector |
| Entities |
Issues only (with optional worklog enrichment). |
Core flow: issues, projects, users, comments, changelogs, worklogs, votes, watchers, remote links, security schemes, permissions, project components, project versions, user groups, deleted issues, and lookup tables for issue type, priority, resolution, and status. Agile flow: boards, sprints, board-sprint, board-project, board-issue mappings. |
| Concurrency |
Single-threaded. |
Parallel per-project issue fetching, with optional multi-node distribution. |
| Schema strategy |
Raw JSON in an `OBJECT` column with a dynamically generated flattened view. |
Explicit column schemas per entity, evolved additively from the API responses. |
| Deletion tracking |
Not supported. |
Tracks deleted issues via Jira audit log polling (optional). |
| Agile data |
Not supported. |
Available through a separate agile flow. |
## Key differences
### Schema changes
The most significant difference is how data is stored in Snowflake:
| Aspect |
Legacy connector |
New connector |
| Issues table |
Single table with an `ISSUE` column containing the full raw JSON as an `OBJECT` type. A flattened `_VIEW` is auto-generated. |
Explicit columns per field. Column names are derived from Jira field display names. No raw JSON fallback. |
| Other entities |
Not available. Comments and worklogs are embedded in the issue JSON. |
Separate tables: `BOARD`, `BOARD_ISSUE`, `BOARD_PROJECT`, `BOARD_SPRINT`, `CHANGELOG`, `COMMENT`, `DELETED_ISSUE`, `FIELD`, `ISSUE_REMOTE_LINK`, `ISSUE_SECURITY_SCHEME`, `ISSUE_TYPE`, `ISSUE_VOTE`, `ISSUE_WATCHER`, `PERMISSION`, `PRIORITY`, `PROJECT`, `PROJECT_COMPONENT`, `PROJECT_VERSION`, `RESOLUTION`, `SPRINT`, `STATUS`, `USER`, `USER_GROUP`, `WORKLOG`. See [](#label-jira-entities) for the full inventory. |
| Views |
Auto-generated `_VIEW` with all issue fields flattened.
| No views created. Data is directly queryable from the destination tables. |
Any queries that reference the legacy `ISSUE` column (for example, `SELECT issue:fields:summary`) or the
auto-generated `_VIEW` must be rewritten to use the new column names directly (for example, `SELECT SUMMARY`).
### Parameter changes
The following parameters from the legacy connector are not available in the new connector:
| Legacy parameter |
Current equivalent |
| Search Type |
Removed. The new connector always fetches all issues from discovered projects. Use `Project Keys Filter` to limit ingestion to specific projects. |
| JQL Query |
Removed. The new connector doesn't support arbitrary JQL for issue filtering. Use `Project Keys Filter` instead. |
| Project Names |
Replaced by `Project Keys Filter`, which accepts project keys (not names or IDs). |
| Status Category |
Removed. The new connector fetches all issues regardless of status. |
| Updated After |
Removed. The new connector manages incremental state automatically. |
| Created After |
Removed. The new connector manages incremental state automatically. |
| Destination Table |
Removed. The new connector creates fixed table names per entity (`ISSUE`, `PROJECT`, `COMMENT`, and others) in the configured destination schema. |
| Fetch All Worklogs |
Removed. The new connector fetches all worklogs into a separate `WORKLOG` table by default when `WORKLOG` is listed in `Enabled Tables`. |
| Connection Method |
Not exposed as a parameter. The new connector uses the `DIRECT` connection method. |
The following parameters are introduced in the new connector:
| Parameter |
Description |
| Deletes Fetch Strategy |
Enables tracking of deleted issues via the Jira audit log. Not available in the legacy connector. |
| Merge Interval |
Time interval between journal-to-destination merge operations. Available in both the core flow and the agile flow. |
Additionally, agile data (boards, sprints, and board mappings) is now available through a separate
agile flow rather than a parameter toggle. See [Set up the Atlassian Jira Cloud (Agile) flow](/user-guide/data-integration/openflow/connectors/jira-cloud/setup-agile) for details on installing and configuring
the agile flow.
### API token scopes
If you're using API tokens with scopes, the new connector may require additional scopes
depending on the features you enable. See [](#label-jira-core-api-scopes) for the core flow scopes and [](#label-jira-agile-api-scopes) for the agile flow scopes.
### Snowflake privileges
The new connector requires only `CREATE TABLE` on the destination schema. The legacy
connector additionally required `CREATE VIEW` to create flattened issue views. The new
connector doesn't create views, so the `CREATE VIEW` privilege is no longer needed. If you're
reusing an existing role, you can revoke `CREATE VIEW` after the legacy connector is
decommissioned.
## Migration steps
1. **Set up the new connector.** Install the core flow on the same
or a different Openflow runtime. If you need agile data, also install the agile flow.
Configure both flows to write to a **different destination schema** than the one used by the legacy
connector. This allows the legacy and new connectors to run simultaneously.
2. **Map your legacy configuration to the new parameters.**
- Copy the `Jira Email`, `Jira API Token`, and `Environment URL` values from the legacy connector
to the new core flow. If using the agile flow, configure these values separately for that flow as well.
- If the legacy connector uses `Project Names`, convert them to project keys for the
`Project Keys Filter` parameter.
- If the legacy connector uses a `JQL Query`, evaluate whether `Project Keys Filter` covers your
use case. If your JQL filters by criteria other than project (for example, status or custom fields),
those filters aren't available in the new connector. All matching issues from the configured
projects are ingested.
- Set `Issue Fields` to match your previous configuration. The default changed from `*all` (legacy)
to `*standard`.
- Configure the Snowflake destination parameters (database, schema, warehouse, credentials) for each flow.
3. **Start the new connector.** Run the core flow and allow the initial load to complete.
If using the agile flow, start it as well.
4. **Validate the data.** Compare the data in the new destination tables against the legacy destination
table to check for completeness. Expect some differences: the legacy connector didn't track deletes,
so issues that were deleted in Jira still appear in the legacy table but not in the new `ISSUE`
table (or they appear with `_SNOWFLAKE_DELETED = TRUE` if delete tracking is enabled). Row counts
will not match exactly when any issues have been deleted.
```sql
-- Compare issue counts (expect differences if issues were deleted in Jira)
SELECT COUNT(*) AS legacy_count FROM legacy_schema.JIRA_ISSUES;
SELECT COUNT(*) AS new_count FROM new_schema.ISSUE;
-- Spot-check specific issues. ISSUE_TYPE, PRIORITY, RESOLUTION, and STATUS
-- are Jira IDs; join the lookup tables to resolve names.
SELECT i.KEY, i.SUMMARY, s.NAME AS status_name
FROM new_schema.ISSUE i
LEFT JOIN new_schema.STATUS s ON i.STATUS = s.ID
WHERE i.KEY = 'PROJ-123';
```
5. **Update downstream queries.** Rewrite any queries, views, dashboards, or pipelines that reference
the legacy table structure. Key changes:
- Replace references to the legacy `ISSUE` `OBJECT` column or `_VIEW` with direct column references.
- Replace `FLATTEN`-based queries with standard `SELECT` statements.
- Add `JOIN` statements to combine data across the new entity tables (for example, join `ISSUE`
with `COMMENT` on `ISSUE_ID`, or join `ISSUE` to `STATUS` on `ISSUE.STATUS = STATUS.ID` to
resolve status names).
- If you want queries to ignore deleted issues, filter on the new `_SNOWFLAKE_DELETED` column
(`WHERE _SNOWFLAKE_DELETED = FALSE`). The legacy connector didn't track deletes at all, so
legacy queries against `JIRA_ISSUES` returned issues that had since been removed in Jira.
6. **Stop the legacy connector.** Once you've confirmed that the new data is complete and downstream
consumers have been updated, stop the legacy connector process group. Both new flows (core and agile)
can continue running independently.
7. **Clean up.** Optionally, drop the legacy destination table and view after confirming they're no
longer needed.
When the legacy connector and the new connector use the same Jira API token, they share the
same Jira API rate limits. Running both simultaneously roughly doubles the API call volume, which
may cause rate limiting on Jira instances with heavy API usage. Consider reducing the legacy
ingestion frequency during the migration period, or run the new connector with a separate
API token whose rate budget you can manage independently.
---
title: ModifyBytes 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/modifybytes.md
section: Loading & Unloading Data
---
# ModifyBytes 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Discard byte range at the start and end or all content of a binary file.
## Tags
binary, discard, keep
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| End Offset |
Number of bytes removed at the end of the file. |
| Remove All Content |
Remove all content from the FlowFile superseding Start Offset and End Offset properties. |
| Start Offset |
Number of bytes removed at the beginning of the file. |
## Relationships
| Name |
Description |
| success |
Processed flowfiles. |
---
title: ModifyCompression 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/modifycompression.md
section: Loading & Unloading Data
---
# ModifyCompression 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-compress-nar
## Description
Changes the compression algorithm used to compress the contents of a FlowFile by decompressing the contents of FlowFiles using a user-specified compression algorithm and recompressing the contents using the specified compression format properties. This processor operates in a very memory efficient way so very large objects well beyond the heap size are generally fine to process
## Tags
brotli, bzip2, compress, content, deflate, gzip, lz4-framed, lzma, recompress, snappy, snappy framed, snappy-hadoop, xz-lzma2, zstd
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Input Compression Strategy |
The strategy to use for decompressing input FlowFiles |
| Output Compression Level |
The compression level for output FlowFiles for supported formats. A lower value results in faster processing but less compression; a value of 0 indicates no (that is, simple archiving) for gzip or minimal for xz-lzma2 compression. Higher levels can mean much larger memory usage such as the case with levels 7-9 for xz-lzma/2 so be careful relative to heap size. |
| Output Compression Strategy |
The strategy to use for compressing output FlowFiles |
| Output Filename Strategy |
Processing strategy for filename attribute on output FlowFiles |
## Relationships
| Name |
Description |
| failure |
FlowFiles will be transferred to the failure relationship on compression modification errors |
| success |
FlowFiles will be transferred to the success relationship on compression modification success |
## Writes attributes
| Name |
Description |
| mime.type |
The appropriate MIME Type is set based on the value of the Compression Format property. If the Compression Format is 'no compression' this attribute is removed as the MIME Type is no longer known. |
---
title: MongoDBControllerService
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/mongodbcontrollerservice.md
section: Loading & Unloading Data
---
# MongoDBControllerService
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides a controller service that configures a connection to MongoDB and provides access to that connection to other Mongo-related components.
## Tags
mongo, mongodb, service
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Database User |
Database User |
|
|
Database user name |
| Mongo URI * |
Mongo URI |
|
|
MongoURI, typically of the form: mongodb://host1[:port1][,host2[:port2],...] |
| Password |
Password |
|
|
The password for the database user |
| SSL Context Service |
SSL Context Service |
|
|
The SSL Context Service used to provide client certificate information for TLS/SSL connections. |
| Write Concern * |
Write Concern |
ACKNOWLEDGED |
- ACKNOWLEDGED
- UNACKNOWLEDGED
- FSYNCED
- JOURNALED
- REPLICA_ACKNOWLEDGED
- MAJORITY
- W1
- W2
- W3
|
The write concern to use |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: MongoDBLookupService
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/controllers/mongodblookupservice.md
section: Loading & Unloading Data
---
# MongoDBLookupService
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
## Description
Provides a lookup service based around MongoDB. Each key that is specified will be added to a query as-is. For example, if you specify the two keys, user and email, the resulting query will be \{ "user": "tester", "email": "[tester@test.com](mailto:tester@test.com)" \}. The query is limited to the first result (findOne in the Mongo documentation). If no "Lookup Value Field" is specified then the entire MongoDB result document minus the _id field will be returned as a record.
## Tags
lookup, mongo, mongodb, record
## Properties
In the list below required Properties are shown with an asterisk (*).
Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name |
API Name |
Default Value |
Allowable Values |
Description |
| Schema Access Strategy * |
Schema Access Strategy |
infer |
- Use 'Schema Name' Property
- Use 'Schema Text' Property
- Infer from Result
|
Specifies how to obtain the schema that is to be used for interpreting the data. |
| Schema Branch |
Schema Branch |
|
|
Specifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored. |
| Schema Name |
Schema Name |
$\{schema.name\} |
|
Specifies the name of the schema to lookup in the Schema Registry property |
| Schema Registry |
Schema Registry |
|
|
Specifies the Controller Service to use for the Schema Registry |
| Schema Text |
Schema Text |
$\{avro.schema\} |
|
The text of an Avro-formatted Schema |
| Schema Version |
Schema Version |
|
|
Specifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved. |
| Mongo Collection Name * |
mongo-collection-name |
|
|
The name of the collection to use |
| Mongo Database Name * |
mongo-db-name |
|
|
The name of the database to use |
| Client Service * |
mongo-lookup-client-service |
|
|
A MongoDB controller service to use with this lookup service. |
| Projection |
mongo-lookup-projection |
|
|
Specifies a projection for limiting which fields will be returned. |
| Lookup Value Field |
mongo-lookup-value-field |
|
|
The field whose value will be returned when the lookup key(s) match a record. If not specified then the entire MongoDB result document minus the _id field will be returned as a record. |
## State management
This component does not store state.
## Restricted
This component is not restricted.
## System Resource Considerations
This component does not specify system resource considerations.
---
title: Monitor connectors using the Openflow Connectors Dashboard
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors-dashboard.md
section: Loading & Unloading Data
---
" />
# Monitor connectors using the Openflow Connectors Dashboard
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
- [Monitor Openflow using telemetry data](/user-guide/data-integration/openflow/monitor)
- [Manage Openflow](/user-guide/data-integration/openflow/manage)
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
The Openflow Connectors Dashboard provides a high-level view of all installed connectors, health snapshots,
and key performance indicators, such as the aggregated average throughput and total data ingested by all connectors matching
the filter criteria.
## Prerequisites
To use the Openflow Connectors Dashboard, the following prerequisites must be met:
- You need at least read-only permissions on the event table.
- You must have the following minimum Openflow versions:
- BYOC deployment: 1.36.0
- Snowflake deployment: 1.26.0
- Runtime: 2026.3.17.13
- You must have the following minimum connector versions. These versions apply to Database connectors only.
Other connector types don't have a minimum version requirement for dashboard support.
| Connector |
Minimum version |
| MySQL |
0.33.0 |
| PostgreSQL |
0.39.0 |
| MongoDB |
0.17.0 |
| SQL Server |
0.27.0 |
| Oracle Embedded License |
0.25.0 |
| Oracle Independent License |
0.24.0 |
See [Snowflake Openflow version history](/user-guide/data-integration/openflow/version-history) for more information.
## Access the Openflow Connectors Dashboard
1. Sign in to %sf-web-interface-link%.
2. In the navigation menu, select **Ingestion** %raa% **Openflow** and navigate to the **Connector Observability** tab.
The Openflow Connectors Dashboard appears.
## The Openflow Connectors Dashboard overview
The Openflow Connectors Dashboard displays the following information:
- **Status**
-
Shows the number of connectors with the following statuses:
- **Healthy**: Didn't encounter any errors during the selected time period.
- **Unhealthy**: Logged errors in the event table during the selected time period or has one or more tables in
**Failed** state (Database connectors only).
- **Upgrade required**: Openflow deployment, runtime, or connector aren't running the minimum required versions
to display health and performance metrics. Review the version prerequisites and upgrade as needed.
- **Average throughput**
-
Measures the rate at which data is read from source systems and sent to Snowflake across all connectors.
- The **Average throughput** %raa% **Ingested** metric measures how fast data is sent to Snowflake across all connectors that match
the primary filter criteria (time frame and event table).
- The **Average throughput** %raa% **Read** metric measures how fast Openflow reads data from source systems across all connectors that match
the primary filter criteria (time frame and event table).
- **Total data ingested**
-
Shows how much data all connectors that match the primary filter criteria for time frame and event table have sent to Snowflake during the selected time period.
Use this metric to quickly identify ingestion anomalies over a specific time period.
For custom telemetry queries beyond the dashboard, see [Monitor Openflow using telemetry data](/user-guide/data-integration/openflow/monitor).
- **Total data ingested** and **Average throughput** metrics include both raw payload and structural overhead such as JSON keys, braces,
and delimiters. Because these metrics track the total transmitted volume, these figures might be higher than the uncompressed data reported
by Snowpipe Streaming or the final storage volume in your destination table.
- The connectors appear in the list if they match the selected filter criteria and have recorded telemetry events during the selected time frame.
- If you examine longer time frames, the list might show connectors that were previously deleted.
For example, you deployed a connector six days ago, and then deleted that connector two days ago. If you set the time frame to **Last 7 days**,
the connector appears in the list because it recorded telemetry events in the last 7 days.
### Filtering connectors
The Openflow Connectors Dashboard supports the following filters:
- **Event table**
-
The Openflow connectors event table you want to monitor. This filter displays event tables that are associated with at least one Openflow deployment,
as well as the default event table and the account event table. You can select only one event table at a time. Event table views are also supported.
The event table is set when you set up Openflow.
To view the event table associated with an Openflow deployment, use the [DESCRIBE OPENFLOW DATA PLANE INTEGRATION](/sql-reference/sql/desc-oflow-data-plane-integration) command.
See [Set up Openflow - Snowflake Deployment](/user-guide/data-integration/openflow/setup-openflow-spcs-deployment) or
[Set up Openflow - BYOC](/user-guide/data-integration/openflow/setup-openflow-byoc) for more information on configuring event tables.
- Time frame
-
Use this filter to identify relevant connectors in a specific time frame.
To get the most up-to-date results about the connector health, select the **Last Hour** time period.
- **Status**
-
Enables filtering for **Healthy**, **Unhealthy**, or **All** connectors.
- **Source**
-
Enables filtering by the source system based on known deployed connectors. The filter only shows sources that are used by your connectors.
- **Deployment**
-
Enables filtering by Snowflake Openflow deployments.
This filter displays data plane integration names, which are composed of the prefix `OPENFLOW_DATAPLANE_` followed by the deployment ID.
To find the deployment ID, navigate to Openflow, select the **Deployments** tab, then select **View Details**.
- **Runtime**
-
Enables filtering by Snowflake Openflow runtimes.
This filter displays the runtime keys. To match runtime keys with Openflow runtime names in the UI, navigate to Openflow, select the **Runtimes** tab, then
select **View Details**, and find the corresponding key.
- **Type**
-
Enables filtering by connector type: Databases, SaaS, Streaming, Unstructured, Other.
- Primary filters (event table and time frame) are applied before secondary filters (status, source, deployment, runtime, or type).
- The secondary filters (status, source, deployment, runtime, type) don't apply to the throughput and data ingested visuals.
## Monitoring Openflow connectors
To monitor the connector details, select %vertical-more-icon% %raa% **View Details**.
### Database connectors
The details page shows the following information for each table that is part of the Database connector configuration:
- **Table replication status**
-
Tables can either be in **Active** or **Failed** replication status. The replication status is based on the most recent telemetry event
that is available for the table. Events that cause replication to fail for a table immediately result in a **Failed** replication
status in the dashboard. Use the **Failure Reason** message to identify the issue.
- **Error distribution**
-
Helps you understand when the connector experienced issues, so that you can identify any potential problems with source systems,
connector configuration, or the Snowflake destination.
- **Table name**
-
Shows the schema and table names for all tables that are configured to be replicated by the connector. The list matches the
**Included Table Names** or **Included Table Regex** configuration parameters of the connector.
- **Replication status**
-
Shows whether each table is in **Active** or **Failed** replication status.
- **Replication phase**
-
Shows the current table replication phase. After configuration in the connector, tables enter the **New** replication
phase, progress to the **Snapshot Load** phase, perform the initial load, and ultimately enter the **Incremental Replication** phase
when individual change data capture events are processed.
- **Last Ingested**
-
Shows the timestamp of the last inserted record into the destination table during the selected time frame. When looking at this
metric, consider a short delay between the records being ingested and events being logged and available to query.
The connector updates this timestamp after each merge query and continues to emit the last known value every minute, even when
no new data is being ingested. If no merge query has occurred for more than 3 days, the connector stops emitting the metric
entirely and the dashboard shows **More than 3 days ago** instead of a timestamp. If the last ingestion timestamp falls outside
the selected time frame, switch to a longer window such as **Last 24 hours** or **Last 7 days** to retrieve it.
You can use the **Replication status**, **Replication phase**, and time frame filters to narrow down the table list.
### All connectors
- **Connector status**
-
Shows the connector health status: **Healthy** if no error messages were encountered during the selected time frame,
or **Unhealthy** if any error messages were encountered.
- **Error distribution**
-
Shows a count of how many errors this connector encountered during the selected time period.
- **Average throughput**
-
Measures the rate at which data is read from source systems and ingested into Snowflake for the selected connector.
- The **Average throughput** %raa% **Ingested** metric measures how fast the selected connector ingests data into Snowflake.
- The **Average throughput** %raa% **Read** metric measures how fast the selected connector reads data from source systems.
- **Total data ingested**
-
Shows how much data the selected connector has ingested into Snowflake during the selected time period.
Use this metric to quickly identify ingestion anomalies over a specific time period.
### Custom flows
Custom flows built on the Openflow canvas can also be monitored on the dashboard, but only if they are represented
as process groups on the root canvas and are actively version-controlled in a customer Git repository using the
Openflow Git integration. Custom flows that don't meet these criteria don't appear in the dashboard.
For more information, see [Version control for custom flows](/user-guide/data-integration/openflow/version-control-custom-flows).
## Debugging Openflow connectors
The Openflow Connectors Dashboard serves as an entry point for debugging connector-specific issues and makes all connector logs easily accessible to users.
### Troubleshoot connectors with AI
Use AI-assisted troubleshooting to get root cause analysis and remediation steps for unhealthy connectors directly from the dashboard,
without writing a prompt or switching tools. The AI assistant combines event table logs, connector metrics, and built-in runbooks
to identify the issue and recommend next steps. It can also surface source system or Snowflake destination configuration problems
that contribute to the failure.
You can start AI-assisted troubleshooting at two scopes:
- **Troubleshoot a whole connector**
-
Use this option to investigate everything affecting a connector. The AI assistant looks at all errors and metrics for the connector
during the selected time frame and reports back with the most impactful issues and how to address them.
To troubleshoot a whole connector, do one of the following:
- In the connectors list, select the troubleshoot icon next to the status on the row of an **Unhealthy** connector. In the list, the
control is an icon only, without a text label.
- On the connector details page, select the **Troubleshoot** button in the page header.
- **Troubleshoot a specific issue**
-
Use this option to focus on a single error. The AI assistant scopes its analysis to that error and returns targeted root cause analysis
and remediation steps.
To troubleshoot a specific issue, navigate to the connector details page, select the **Issues** tab, locate the error
you want to investigate, and select the **Troubleshoot** button on that error.
The troubleshoot control, shown as an icon in the connectors list and as a **Troubleshoot** button elsewhere, only appears for
connectors or issues in an **Unhealthy** state.
### Viewing the connector errors
To view all errors that a connector encountered in the selected time frame, first navigate to the connector details page by
selecting %vertical-more-icon% %raa% **View Details**, and then select the **Issues** tab.
The error headline tells you what type of error the connector encountered, and the content provides the entire stacktrace of the error.
### Viewing the connector logs
You might also want to look at additional connector logs to understand the context around an error message. To view all logs for the selected connector,
select %vertical-more-icon% %raa% **View logs**.
After you open the log explorer, you can also change the filters to view logs for different connectors or for entire
runtimes or deployments. The log explorer supports Openflow-specific filters like the dataplane ID, the runtime key, and the process group ID.
### Accessing the Openflow canvas
When you identify a connector issue, you probably need to navigate to the Openflow canvas to fix it; for example, adjust some configuration parameters or
upgrade to a newer connector version.
To navigate to the selected connector in the Openflow canvas, select %vertical-more-icon% %raa% **Go to canvas**.
## Optimizing performance
### Select a larger warehouse
Use the warehouse selector in the top right section of the screen to choose a different warehouse to run the queries.
While larger warehouses run queries faster, they take longer to resume, which might increase the initial page load time.
### Set up clustering on the Openflow event table
By using clustering keys, you can avoid unnecessary scanning of micro-partitions during querying, significantly accelerating
the performance of queries that reference these columns. For more information, see
[](#label-data-clustering).
Run the following query, replacing the placeholders with your Openflow event table:
```sqlsyntax
ALTER TABLE ..
CLUSTER BY (
DATE_TRUNC('HOUR', timestamp),
RECORD_TYPE,
CAST(record_attributes:"metricNameHash" AS STRING)
);
```
- Automatic clustering consumes Snowflake credits using serverless compute resources. To learn how many credits
per compute-hour are consumed, refer to the "Serverless Feature Credit Table" in the
[Snowflake Service Consumption Table](https://www.snowflake.cn/legal-files/CreditConsumptionTable.pdf).
- After you enable clustering on your event table, a background process starts that takes some time to complete.
After the process is complete, you should see improved performance when using the dashboard.
### Reduce the queried time frame
Selecting a smaller time frame in the filter scans less data and leads to faster query performance.
Use the **Last Hour** filter for the best performance and the most up-to-date view of your connector health and performance.
## Limitations
- The Openflow Connectors Dashboard uses data stored in event tables to provide insight into Openflow connectors. Depending on the selected time period and event table,
information provided on the dashboard might not reflect the current status of a connector.
- Detailed health monitoring is currently only available for Database connectors.
- The connector details page for Database connectors displays up to 75,000 tables.
- The **Deployment** and **Runtime** filters use internal names that differ from the display names in the Openflow UI.
For details on matching these names, see [Filtering connectors](#label-openflow-dashboard-filtering).
- The **Last Ingested** column for Database connector tables shows **More than 3 days ago** when the connector hasn't performed
a merge query in more than 3 days, because the connector stops emitting the last-ingestion metric after that period.
If the last ingestion timestamp falls outside the selected time frame, switch to a longer window such as **Last 24 hours**
or **Last 7 days** to retrieve it.
## Known issues
- After upgrading the deployment, runtime, and connector to the versions mentioned in the prerequisites, the error count metric is only accurate
for errors encountered after the upgrade.
---
title: Monitor Openflow
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/monitor-overview.md
section: Loading & Unloading Data
---
# Monitor Openflow
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
- [Set up Openflow - BYOC](/user-guide/data-integration/openflow/setup-openflow-byoc)
- [Set up Openflow - Snowflake Deployment - Task overview](/user-guide/data-integration/openflow/setup-openflow-spcs)
- [Manage Openflow](/user-guide/data-integration/openflow/manage)
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
Openflow provides two approaches for monitoring your data integration pipelines:
- [Monitor connectors using the Openflow Connectors Dashboard](/user-guide/data-integration/openflow/connectors-dashboard)
-
Use the Openflow Connectors Dashboard in Snowsight to get a high-level view of connector health, throughput,
and data ingestion. The dashboard provides filtering, error distribution, and per-connector detail pages.
- [Monitor Openflow using telemetry data](/user-guide/data-integration/openflow/monitor)
-
Query the Openflow telemetry data stored in your event table to monitor logs, application metrics, JVM and system
metrics, and build custom queries tailored to your environment.
---
title: Monitor Openflow using telemetry data
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/monitor.md
section: Loading & Unloading Data
---
# Monitor Openflow using telemetry data
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
- [Set up Openflow - BYOC](/user-guide/data-integration/openflow/setup-openflow-byoc)
- [Set up Openflow - Snowflake Deployment - Task overview](/user-guide/data-integration/openflow/setup-openflow-spcs)
- [Manage Openflow](/user-guide/data-integration/openflow/manage)
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
- [All controller services (alphabetical)](/user-guide/data-integration/openflow/controllers/index)
This topic describes how to monitor the state of Openflow and troubleshoot problems.
## Accessing Openflow logs
Snowflake sends Openflow logs to the event table you configured when you set up Openflow
([BYOC](#label-openflow-event-table) | [Snowflake deployment](#label-openflow-spcs-event-table)).
Snowflake recommends that you include a timestamp in the WHERE clause of event table queries.
This is particularly important because of the potential volume of data generated by various Snowflake components.
By applying filters, you can retrieve a smaller subset of data, which improves query performance.
To get started quickly with Openflow's telemetry, see [Example Queries](#label-openflow-example-queries) below.
## Openflow Telemetry Schema
For information about the event table columns, see [Event table columns](/developer-guide/logging-tracing/event-table-columns).
The following sections describe how Openflow structures telemetry in an Event Table.
### Resource Attributes
Describes the event metadata set by Openflow. For general information on other
types of resource attributes see [](#label-event-table-resource-attributes-column) in the Event Table columns documentation.
| Name |
Type |
Description |
| application |
String |
The fixed value _openflow_ |
| cloud.service.provider |
String |
One of _aws_, _snowflake_ |
| container.id |
String |
Unique identifier of the container |
| container.image.name |
String |
Fully qualified name of the container image. All Openflow images are hosted by Snowflake repositories.
For example, *<account>-openflow-<env>.registry-internal.snowflakecomputing.cn/openflow/openflow/openflow_repo/runtime-server*
|
| container.image.tag |
String |
Version of the container image |
| k8s.container.name |
String |
The name of the K8s container. Openflow Runtime containers will start with the "Runtime Key" and end with *-gateway* or *-server*.
For example, an Openflow Runtime named "PostgreSQL CDC" with a Runtime Key of postgresql-cdc, so it would have container names of:
- postgresql-cdc-gateway
- postgresql-cdc-server
|
| k8s.container.restart_count |
Numeric String |
The number of times this container has restarted since it was created. |
| k8s.namespace.name |
String |
K8s namespace of the pod or container, starting with _runtime-_ for Openflow Runtimes. Values also include _kube-system_ and _openflow-runtime-infra_. |
| k8s.node.name |
String |
The internal domain name of the EKS node hosting the pod / container, or the EKS node itself.
For example, ip-10-12-13-144.us-west-2.compute.internal
|
| k8s.pod.name |
String |
The name of the K8s pod. Openflow Runtime pods will start with the "Runtime Key" and end with a numeric identifier for each pod replica. This number can grow up to the "Max Nodes" set for the Runtime, indexed at 0.
For example, an Openflow Runtime named "PostgreSQL CDC" with a Runtime Key of postgresql-cdc and 3 nodes would have pod names of:
- postgresql-cdc-0
- postgresql-cdc-1
- postgresql-cdc-2
|
| k8s.pod.start_time |
ISO 8601 Date String |
Timestamp that the pod was started |
| k8s.pod.uid |
UUID String |
Unique identifier of the pod within the cluster |
| deployment.version |
String |
The Openflow deployment version. |
| openflow.dataplane.id |
UUID String |
The unique identifier of the Openflow Deployment, matching the "ID" shown in the Snowflake Openflow UI through Deployment > View Details. |
- Resource Attributes Example:
-
```json
{
"application": "openflow",
"cloud.service.provider": "aws",
"container.id": "a1b2c3d4e5f6",
"container.image.name": "example-openflow-prod.registry-internal.snowflakecomputing.cn/openflow/openflow/openflow_repo/runtime-server",
"container.image.tag": "2026.3.17.13",
"deployment.version": "1.35.0",
"k8s.container.name": "pg-dev-server",
"k8s.container.restart_count": "0",
"k8s.namespace.name": "runtime-pg-dev",
"k8s.node.name": "ip-10-10-62-36.us-east-2.compute.internal",
"k8s.pod.name": "pg-dev-0",
"k8s.pod.start_time": "2025-04-25T22:14:29Z",
"k8s.pod.uid": "94610175-1685-4c8f-b0a1-42898d1058e6",
"openflow.dataplane.id": "abeddb4f-95ae-45aa-95b1-b4752f30c64a"
}
```
### Scope
| Name |
Type |
Description |
| name |
String |
Provider of the metric. One of:
- *runtime* for Openflow Connector metrics
- *github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver* for system-level metrics
|
- Scope Example:
-
```json
{
"name": "runtime"
}
```
### Record Type
Depending on the type of Openflow telemetry represented by this row, this will be one of:
- LOG
- METRIC
Openflow does not collect TRACE records, but that is also a valid type for this column in Snowflake Event Tables.
### Record
Optional. This JSON object describes the type of metric represented by this row.
| Name |
Type |
Description |
| metric |
Object |
Contains two fields:
- *name* for the unique metric produced, typically using dot-delimited namespaces
- *unit* for the value represented by the type, such as byte, nanosecond, and thread
The name and unit values vary widely. For the full list, see [Application Metrics](#label-openflow-application-metrics) below.
|
| metric_type |
String |
One of:
- *gauge* for most Openflow metrics, a snapshot value that can increase or decrease
- *sum* for cumulative metrics like pod CPU time and network IO
|
| value_type |
String |
The primitive type of the value produced by this metric. One of:
- INT
- DOUBLE
|
| aggregation_temporality |
String |
Optional. Set to cumulative for metrics that are strictly increasing and dependent on previous values, such as pod CPU time and network IO. |
| is_monotonic |
Boolean |
Optional. For cumulative metrics, this is true to show that it is strictly increasing within the time series. |
- Record Example:
-
```json
{
"metric": {
"name": "connection.queued.duration.max",
"unit": "millisecond"
},
"metric_type": "gauge",
"value_type": "INT"
}
```
### Record Attributes
#### Logs
Record attributes for Logs will typically indicate where this log was sourced. For example, logs from an Openflow Runtime named *testruntime* could have Record Attributes of:
```json
{
"log.file.path": "/var/log/pods/runtime-testruntime_testruntime-0_66d80cdb-9484-40a4-bdba-f92eb0af14c7/testruntime-server/0.log",
"log.iostream": "stdout",
"logtag": "F"
}
```
#### System Metrics
System metrics like CPU usage will typically not set Record Attributes, so this will be *null*.
#### Openflow Application Metrics
Record Attributes for Application or "Flow" metrics provide details about the component in the data pipeline that produced the metric. This will vary based on the type of component. See [Application Metrics](#label-openflow-application-metrics)
```json
{
"component": "PutSnowpipeStreaming",
"execution.node": "ALL",
"group.id": "c052f9d7-7f76-3013-a2c5-d3b064fa7326",
"id": "c69e2913-22a9-36bb-a159-6a5ed1fb9d63",
"name": "PutSnowpipeStreaming",
"type": "processor"
}
```
### Value
This column contains the raw value of the telemetry. For metrics, this will be a numeric value (integer or double). For logs, this will either be a semi-structured string value or a well-formatted JSON string.
#### Openflow Runtime Logs
Openflow Runtimes emit most logs as JSON, so applying Snowflake's [TRY_PARSE_JSON](/sql-reference/functions/try_parse_json) to the *VALUE* column allows you to further break this value into the following structured fields:
| Name |
Type |
Description |
| formattedMessage |
String |
The actual log message emitted from the Runtime logger. |
| level |
String |
One of:
- ERROR
- WARN
- INFO
- DEBUG
- TRACE
|
| loggerName |
String |
The fully qualified classname for the logger. Openflow processors will typically use logger names that start with *com.snowflake.openflow.runtime.processors*.
This is useful to view logs for a specific processor, controller service, or bundled library.
|
| nanoseconds |
Integer |
Nanosecond-level time that this log message was created, starting at milliseconds.
For example, a nanosecond value of 111222333 could correspond to a timestamp value of 1749180210111 with the leftmost 3 digits of nanosecond matching the right-most 3 digits of timestamp.
|
| threadName |
String |
Name of the thread handling this call. For example, _Timer-Driven Process Thread-7_ |
| throwable |
JSON Object |
*null* when there is no exception or stacktrace for this log message. Otherwise, it logs the stacktrace as a JSON string with fields:
- *className* - the exception thrown
- *message* - any message logged with the exception
- *stepArray* - array of method calls for the stack trace, including:
- *className*
- *fileName*
- *lineNumber*
- *methodName*
|
| timestamp |
Integer |
Time that this log message was created, represented as milliseconds since the UNIX epoch.
For example, 1749180210044 indicates that the log was created at 2025-06-05 03:23:30.044 UTC
|
| mdc |
JSON Object |
Mapped Diagnostic Context (MDC) providing additional flow-level context for the log entry. Contains the following fields:
- *processGroupId* - unique identifier of the process group
- *processGroupIdPath* - hierarchical path of process group IDs
- *processGroupName* - name of the process group
- *processGroupNamePath* - hierarchical path of process group names
- *registeredFlowIdentifier* - identifier of the registered flow (present for all versioned flows, including out-of-the-box Openflow connectors)
- *registeredFlowVersion* - version of the registered flow (present for all versioned flows, including out-of-the-box Openflow connectors)
For example:
```json
{
"processGroupId": "6dc1d98f-019d-1000-ffff-ffffa3ba8a09",
"processGroupIdPath": "/58385a8b-019d-1000-2a52-9ef1c34b0e5f/6dc1d98f-019d-1000-ffff-ffffa3ba8a09",
"processGroupName": "latency targets",
"processGroupNamePath": "/Openflow/latency targets",
"registeredFlowIdentifier": "sqlserver-multidatabase",
"registeredFlowVersion": "0.29.0-ebb7a257"
}
```
|
## Application Metrics
The following list covers all application metrics available for Openflow Runtimes. Runtimes only emit a subset of metrics relevant to Openflow Connectors to persist in a Snowflake Event Table.
Snowflake's OpenTelemetry Reporting Task can send some or all metrics to any OTLP destination.
### Connection Metrics
| Metric Name |
Unit |
Description |
| connection.input.bytes |
bytes |
Size of Items Input |
| connection.input.count |
items |
Count of Items Input |
| connection.output.bytes |
bytes |
Size of Items Output |
| connection.output.count |
items |
Count of Items Output |
| connection.queued.bytes |
bytes |
Size of Items Queued |
| connection.queued.bytes.max |
bytes |
Max Size of Items Queued |
| connection.queued.count |
items |
Count of Items Queued |
| connection.queued.count.max |
items |
Max Count of Items Queued |
| connection.queued.duration.total |
milliseconds |
Total Duration of Queued Items |
| connection.queued.duration.max |
milliseconds |
Max Duration of Queued Items |
| connection.backpressure.threshold.bytes |
bytes |
The maximum size of data in bytes that can be queued in this connection before it applies back pressure. |
| connection.backpressure.threshold.objects |
items |
The configured maximum number of FlowFiles that can be queued in this connection before it applies back pressure. |
| connection.loadbalance.status.load_balance_not_configured |
binary, 0 or 1 |
1 if the connection does not have a configured load balance setting. Otherwise, 0. |
| connection.loadbalance.status.load_balance_active |
binary, 0 or 1 |
1 if the connection is load balancing across the cluster. Otherwise, 0. |
| connection.loadbalance.status.load_balance_inactive |
binary, 0 or 1 |
1 if the connection is not load balancing across the cluster. Otherwise, 0. |
### Connection Record Attributes
Each Connection metric includes the following Record Attributes:
| Attribute |
Description |
| id |
The unique identifier of the connection |
| name |
The user-visible name of the connection |
| type |
The fixed value _connection_ |
| source.id |
The unique identifier of the component that is sending FlowFiles to this connection |
| source.name |
The user-visible name of the component that is sending FlowFiles to this connection |
| destination.id |
The unique identifier of the component that is receiving FlowFiles from this connection |
| destination.name |
The user-visible name of the component that is receiving FlowFiles from this connection |
| group.id |
The unique identifier of the Process Group that contains this Connection |
### Input and Output Port Metrics
Input Port and Output Ports are technically two separate types of components. For consistency, metrics and attributes for Input and Output Ports are the same, with the exception of the *type* attribute that indicates whether it is an input port or an output port.
| Metric Name |
Unit |
Description |
| port.thread.count.active |
threads |
Number of Active Threads |
| port.bytes.received |
bytes |
Number of Bytes Received |
| port.bytes.sent |
bytes |
Number of Bytes Sent |
| port.flowfiles.received |
flowfiles |
Number of FlowFiles Received |
| port.flowfiles.sent |
flowfiles |
Number of FlowFiles Sent |
| port.input.bytes |
bytes |
Size of Items Input |
| port.input.count |
items |
Count of Items Input |
| port.output.bytes |
bytes |
Size of Items Output |
| port.output.count |
items |
Count of Items Output |
### Input and Output Port Record Attributes
Each Port metric includes the following Record Attributes:
| Attribute |
Description |
| id |
The unique identifier of the port |
| name |
The user-visible name of the port |
| type |
One of _port-input_ or _port-output_ |
| group.id |
The unique identifier of the Process Group that contains this Port |
### Process Group Metrics
| Metric Name |
Unit |
Description |
| processgroup.thread.count.active |
threads |
Number of Active Threads |
| processgroup.thread.count.stateless |
threads |
Number of Stateless Threads |
| processgroup.thread.count.terminated |
threads |
Number of Terminated Threads |
| processgroup.bytes.read |
bytes |
Number of Bytes Read |
| processgroup.bytes.received |
bytes |
Number of Bytes Received |
| processgroup.bytes.transferred |
bytes |
Number of Bytes Transferred |
| processgroup.bytes.sent |
bytes |
Number of Bytes Sent |
| processgroup.bytes.written |
bytes |
Number of Bytes Written |
| processgroup.flowfiles.received |
flowfiles |
Number of FlowFiles Received |
| processgroup.flowfiles.sent |
flowfiles |
Number of FlowFiles Sent |
| processgroup.flowfiles.transferred |
flowfiles |
Number of FlowFiles Transferred |
| processgroup.input.count |
items |
Number of Items Input |
| processgroup.input.content.size |
bytes |
Size of Items Input |
| processgroup.output.count |
items |
Number of Items Output |
| processgroup.output.content.size |
bytes |
Size of Items Output |
| processgroup.queued.count |
items |
Number of Items Queued |
| processgroup.queued.content.size |
bytes |
Size of Items Queued |
| processgroup.time.processing |
nanoseconds |
Time Spent Processing |
### Process Group Record Attributes
Each Process Group metric includes the following Record Attributes:
| Attribute |
Description |
| id |
The unique identifier of the Process Group |
| name |
The user-visible name of the Process Group |
| type |
The fixed value _process-group_ |
| tree.level |
The depth of the Process Group, relative to the root process group of the flow. Process Groups at the highest level of the flow will have a tree.level of 1 |
### Processor Metrics
| Metric Name |
Unit |
Description |
| processor.thread.count.active |
thread |
Number of Active Threads |
| processor.thread.count.terminated |
thread |
Number of Terminated Threads |
| processor.time.lineage.average |
nanosecond |
Average Lineage Duration |
| processor.invocations |
invocations |
Number of Invocations |
| processor.bytes.read |
byte |
Number of Bytes Read |
| processor.bytes.received |
byte |
Number of Bytes Received |
| processor.bytes.sent |
byte |
Number of Bytes Sent |
| processor.bytes.written |
byte |
Number of Bytes Written |
| processor.flowfiles.received |
flowfiles |
Number of FlowFiles Received |
| processor.flowfiles.removed |
flowfiles |
Number of FlowFiles Removed |
| processor.flowfiles.sent |
flowfiles |
Number of FlowFiles Sent |
| processor.input.count |
item |
Number of Items Input |
| processor.input.content.size |
bytes |
Size of Items Input |
| processor.output.count |
item |
Number of Items Output |
| processor.output.content.size |
byte |
Size of Items Output |
| processor.time.processing |
nanosecond |
Time Spent Processing |
| processor.run.status.running |
binary, 0 or 1 |
1 if running; 0 otherwise |
| processor.run.status.stopped |
binary, 0 or 1 |
1 if stopped; 0 otherwise |
| processor.run.status.validating |
binary, 0 or 1 |
1 if validating; 0 otherwise |
| processor.run.status.invalid |
binary, 0 or 1 |
1 if invalid; 0 otherwise |
| processor.run.status.disabled |
binary, 0 or 1 |
1 if disabled; 0 otherwise |
| processor.counter |
count |
Value of the counter |
### Processor Record Attributes
Each Processor metric includes the following Record Attributes:
| Attribute |
Description |
| id |
The unique identifier of the processor |
| name |
The user-visible and user-editable name of the Processor |
| type |
The fixed value _processor_ |
| component |
The immutable class name of the processor. |
| execution.node |
Either _ALL_ or _PRIMARY_, depending on how this Processor is configured to run |
| group.id |
The unique identifier of the Process Group that contains this Processor |
### Additional Attributes for Counters
In addition to the standard Processor attributes above, *processor.counter* metrics include the following:
| Attribute |
Description |
| type |
The fixed value _counter_ |
| counter |
The user- or system-generated name of the counter |
### Remote Process Group Metrics
| Metric Name |
Unit |
Description |
| remoteprocessgroup.thread.count.active |
threads |
Number of Active Threads |
| remoteprocessgroup.remote.port.count.active |
ports |
Number of Active Remote Ports |
| remoteprocessgroup.remote.port.count.inactive |
ports |
Number of Inactive Remote Ports |
| remoteprocessgroup.duration.lineage.average |
nanoseconds |
Average Lineage Duration |
| remoteprocessgroup.refresh.age |
milliseconds |
Time since last refresh |
| remoteprocessgroup.received.count |
items |
Number of Received Items |
| remoteprocessgroup.received.content.size |
bytes |
Size of Received Items |
| remoteprocessgroup.sent.count |
items |
Number of Sent Items |
| remoteprocessgroup.sent.content.size |
bytes |
Size of Sent Items |
| remoteprocessgroup.transmission.status.transmitting |
binary, 0 or 1 |
1 if the Remote Process Group is transmitting. Otherwise, 0. |
| remoteprocessgroup.transmission.status.nottransmitting |
binary, 0 or 1 |
0 if the Remote Process Group is transmitting. Otherwise, 1. |
### Remote Process Group Record Attributes
Each Remote Process Group metric includes the following Record Attributes:
| Attribute |
Description |
| id |
The unique identifier of the remote process group |
| name |
The user-visible name of the Remote Process Group |
| group.id |
The unique identifier of the Process Group that contains this Remote Process Group |
| authorization.issue |
The Authorization used to access the Remote Process Group |
| target.uri |
The URI of the Remote Process Group |
| type |
The fixed value _remote-process-group_ |
### JVM Metrics
| Metric Name |
Unit |
Description |
| jvm.memory.heap.used |
bytes |
The amount of memory currently occupied by objects on the JVM Heap |
| jvm.memory.heap.committed |
bytes |
The amount of memory guaranteed to be available for use by the JVM Heap |
| jvm.memory.heap.max |
bytes |
Maximum amount of memory allocated for the JVM Heap |
| jvm.memory.heap.init |
bytes |
Initial amount of memory allocated for the JVM Heap |
| jvm.memory.heap.usage |
percentage |
JVM Heap Usage |
| jvm.memory.non-heap.usage |
percentage |
JVM Non-Heap Usage |
| jvm.memory.total.init |
bytes |
Initial amount of memory allocated for the JVM |
| jvm.memory.total.used |
bytes |
Current amount of memory used by the JVM |
| jvm.memory.total.max |
bytes |
Maximum amount of memory that can be used by the JVM |
| jvm.memory.total.committed |
bytes |
The amount of memory guaranteed to be available for use by the JVM |
| jvm.threads.count |
threads |
Number of live threads |
| jvm.threads.deadlocks |
threads |
JVM Thread Deadlocks |
| jvm.threads.daemon.count |
threads |
Number of live daemon threads |
| jvm.uptime |
seconds |
Number of seconds the JVM process has been running |
| jvm.file.descriptor.usage |
percentage |
Percentage of available file descriptors currently in use. |
| jvm.gc.G1-Concurrent-GC.runs |
runs |
Total number of times that the G1 Concurrent Garbage Collection has run |
| jvm.gc.G1-Concurrent-GC.time |
milliseconds |
Total amount of time that the G1 Concurrent Garbage Collection has been running |
| jvm.gc.G1-Young-Generation.runs |
runs |
Total number of times that the G1 Young Generation has run |
| jvm.gc.G1-Young-Generation.time |
milliseconds |
Total amount of time that the G1 Young Generation has been running |
| jvm.gc.G1-Old-Generation.runs |
runs |
Total number of times that the G1 Old Generation has run |
| jvm.gc.G1-Old-Generation.time |
milliseconds |
Total amount of time that the G1 Old Generation has been running |
### JVM Record Attributes
JVM metrics do not provide Record Attributes.
### CPU Metrics
| Metric Name |
Unit |
Description |
| cores.available |
cores |
The number of available cores for the Runtime |
| cores.load |
percentage |
Either the system load average or -1 if it is not available |
### CPU Record Attributes
| Attribute |
Description |
| id |
The fixed value _cpu_ |
| name |
The name of the operating system |
| architecture |
The architecture of the operating system |
| version |
The version of the operating system |
### Storage Metrics
| Metric Name |
Unit |
Description |
| storage.free |
bytes |
The amount of free storage for a given repository |
| storage.used |
bytes |
The amount of used storage for a given repository |
### Storage Record Attributes
| Attribute |
Description |
| id |
The unique identifier of the storage repository |
| name |
Same as id and provided for consistency |
| storage.type |
One of _flowfile_, _content_, or _provenance_ |
## Example Queries
The following queries are examples to get you started with Openflow Telemetry.
All queries assume that Openflow is configured to send telemetry to the default Event Table of *SNOWFLAKE.TELEMETRY.EVENTS*. If your Snowflake Account or Openflow Deployment is configured with a different Event Table, substitute that table name where you see *SNOWFLAKE.TELEMETRY.EVENTS*.
### Find Stuck FlowFiles
This query returns connections with FlowFiles that have been queued for more than some threshold, indicating that they may be stuck and require intervention. Adjust the 30 minute threshold as needed for your use case.
```sql
SELECT * FROM (
SELECT
resource_attributes:"openflow.dataplane.id" as Deployment_ID,
resource_attributes:"k8s.namespace.name" as Runtime_Key,
record_attributes:name as Connection_Name,
record_attributes:id as Connection_ID,
MAX(TO_NUMBER(value / 60 / 1000)) as Max_Queued_File_Minutes
FROM snowflake.telemetry.events
WHERE true
AND record_type = 'METRIC'
AND record:metric:name = 'connection.queued.duration.max'
AND timestamp > dateadd(minutes, -30, sysdate())
GROUP BY 1, 2, 3, 4
ORDER BY Max_Queued_File_Minutes DESC
) WHERE Max_Queued_File_Minutes > 30;
```
### Find Error Logs for Openflow Runtimes
```sql
SELECT
timestamp,
Deployment_ID,
Runtime_Key,
parsed_log:level as log_level,
parsed_log:loggerName as logger,
parsed_log:formattedMessage as message,
parsed_log
FROM (
SELECT
timestamp,
resource_attributes:"openflow.dataplane.id" as Deployment_ID,
resource_attributes:"k8s.namespace.name" as Runtime_Key,
TRY_PARSE_JSON(value) as parsed_log
FROM snowflake.telemetry.events
WHERE true
AND timestamp > dateadd('minutes', -30, sysdate())
AND record_type = 'LOG'
AND resource_attributes:"k8s.namespace.name" like 'runtime-%'
ORDER BY timestamp DESC
) WHERE log_level = 'ERROR';
```
### Find Running and Non-Running Processors
Some flows expect that all processors are in a "running" state, even if they are not actively processing data.
This query helps you find any processors that are running or in another state, such as:
- stopped
- invalid
- disabled
```sql
SELECT
timestamp,
resource_attributes:"openflow.dataplane.id" as Deployment_ID,
resource_attributes:"k8s.namespace.name" as Runtime_Key,
record_attributes:component as Processor,
record_attributes:id as Processor_ID,
TO_NUMBER(value) as Running
FROM snowflake.telemetry.events
WHERE true
AND record:metric:name = 'processor.run.status.running'
AND record_type = 'METRIC'
AND timestamp > dateadd(minutes, -30, sysdate());
```
### Find High CPU Usage for Openflow Runtimes
Slow data flows or reduced throughput may be the result of a bottleneck on the CPU. Openflow Runtimes scale up automatically, based on the number of minimum and maximum nodes you have configured.
If an Openflow Runtime is using its maximum number of nodes and still CPU usage remains high, consider:
1. Increasing the maximum number of nodes allocated to the Runtime
2. Troubleshoot the Connector or flow to identify the bottleneck
Snowsight Charts provide an easy way to visualize query results for CPU usage over time.
```sql
SELECT
timestamp,
resource_attributes:"openflow.dataplane.id" as Deployment_ID,
resource_attributes:"k8s.namespace.name" as Runtime_Key,
resource_attributes:"k8s.pod.name" as Runtime_Pod,
TO_NUMBER(value, 10, 3) * 100 as CPU_Usage_Percentage
FROM snowflake.telemetry.events
WHERE true
AND timestamp > dateadd(minute, -30, sysdate())
AND record_type = 'METRIC'
AND record:metric:name ilike 'container.cpu.usage'
AND resource_attributes:"k8s.namespace.name" ilike 'runtime-%'
AND resource_attributes:"k8s.container.name" ilike '%-server'
ORDER BY timestamp desc, CPU_Usage_Percentage desc;
```
---
title: Monitor the Openflow Connector for Salesforce Bulk API
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/monitor.md
section: Loading & Unloading Data
---
# Monitor the %salesforcebulkapiof%
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
- [About the Openflow Connector for Salesforce Bulk API](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/about)
- [Openflow Connector for Salesforce Bulk API: Configure the connector](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/configure-connector)
- [Troubleshooting the Openflow Connector for Salesforce Bulk API](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/troubleshoot)
- [Monitor Openflow using telemetry data](/user-guide/data-integration/openflow/monitor)
The connector writes information about completed Salesforce Bulk API jobs to logs in the event table. You can query these logs to track the objects and number of records replicated to Snowflake.
The examples on this page query `OPENFLOW.TELEMETRY.EVENTS`. If your Openflow deployment sends telemetry to a different event table, replace the table name in the examples. Adjust the 30-minute time range as needed.
## Query replication activity
By default, the connector logs the Salesforce object type, number of records processed, bulk job ID, and system modification timestamp. Use the following query when **Enable Merge Metrics** is set to `false`:
```sql
WITH connector_logs AS (
SELECT
timestamp,
resource_attributes:"openflow.dataplane.id"::VARCHAR AS deployment_id,
resource_attributes:"k8s.namespace.name"::VARCHAR AS runtime_key,
TRY_PARSE_JSON(value) AS parsed_log
FROM OPENFLOW.TELEMETRY.EVENTS
WHERE timestamp >= DATEADD('minutes', -30, CURRENT_TIMESTAMP())
AND record_type = 'LOG'
AND resource_attributes:"k8s.namespace.name"::VARCHAR LIKE 'runtime-%'
),
salesforce_logs AS (
SELECT
timestamp,
deployment_id,
runtime_key,
parsed_log:formattedMessage::VARCHAR AS message
FROM connector_logs
WHERE parsed_log:loggerName::VARCHAR = 'org.apache.nifi.processors.standard.LogMessage'
AND CONTAINS(parsed_log:formattedMessage::VARCHAR, 'SALESFORCE_BULK_API - ')
)
SELECT
timestamp,
deployment_id,
runtime_key,
TRIM(REGEXP_SUBSTR(message, 'ObjectType = ([^;]+)', 1, 1, 'e', 1)) AS object_type,
TRY_TO_NUMBER(TRIM(REGEXP_SUBSTR(message, 'Records = ([^;]+)', 1, 1, 'e', 1))) AS records,
TRIM(REGEXP_SUBSTR(message, 'BulkJobID = ([^;]*)', 1, 1, 'e', 1)) AS bulk_job_id,
TRIM(REGEXP_SUBSTR(message, 'SystemModstamp = ([^;]+)', 1, 1, 'e', 1)) AS system_modstamp
FROM salesforce_logs
ORDER BY timestamp DESC;
```
## Query merge metrics
Set **Enable Merge Metrics** to `true` to include detailed record counts in the logs. The connector runs an additional query before each incremental merge to calculate the counts. This query uses the warehouse configured in **Snowflake Warehouse**.
Merge metrics are available only for Salesforce objects that include the `IsDeleted` field. Use the following query to retrieve the replication activity and merge metrics:
```sql
WITH connector_logs AS (
SELECT
timestamp,
resource_attributes:"openflow.dataplane.id"::VARCHAR AS deployment_id,
resource_attributes:"k8s.namespace.name"::VARCHAR AS runtime_key,
TRY_PARSE_JSON(value) AS parsed_log
FROM OPENFLOW.TELEMETRY.EVENTS
WHERE timestamp >= DATEADD('minutes', -30, CURRENT_TIMESTAMP())
AND record_type = 'LOG'
AND resource_attributes:"k8s.namespace.name"::VARCHAR LIKE 'runtime-%'
),
salesforce_logs AS (
SELECT
timestamp,
deployment_id,
runtime_key,
parsed_log:formattedMessage::VARCHAR AS message
FROM connector_logs
WHERE parsed_log:loggerName::VARCHAR = 'org.apache.nifi.processors.standard.LogMessage'
AND CONTAINS(parsed_log:formattedMessage::VARCHAR, 'SALESFORCE_BULK_API - ')
)
SELECT
timestamp,
deployment_id,
runtime_key,
TRIM(REGEXP_SUBSTR(message, 'ObjectType = ([^;]+)', 1, 1, 'e', 1)) AS object_type,
TRY_TO_NUMBER(TRIM(REGEXP_SUBSTR(message, 'Records = ([^;]+)', 1, 1, 'e', 1))) AS records,
TRIM(REGEXP_SUBSTR(message, 'BulkJobID = ([^;]*)', 1, 1, 'e', 1)) AS bulk_job_id,
TRIM(REGEXP_SUBSTR(message, 'SystemModstamp = ([^;]+)', 1, 1, 'e', 1)) AS system_modstamp,
TRY_TO_NUMBER(TRIM(REGEXP_SUBSTR(message, 'ROWS_ADDED = ([^;]+)', 1, 1, 'e', 1))) AS rows_added,
TRY_TO_NUMBER(TRIM(REGEXP_SUBSTR(message, 'ROWS_ADDED_DELETED = ([^;]+)', 1, 1, 'e', 1))) AS rows_added_deleted,
TRY_TO_NUMBER(TRIM(REGEXP_SUBSTR(message, 'ROWS_UPDATED = ([^;]+)', 1, 1, 'e', 1))) AS rows_updated,
TRY_TO_NUMBER(TRIM(REGEXP_SUBSTR(message, 'ROWS_DELETED = ([^;]+)', 1, 1, 'e', 1))) AS rows_deleted,
TRY_TO_NUMBER(TRIM(REGEXP_SUBSTR(message, 'ROWS_RESTORED = ([^;]+)', 1, 1, 'e', 1))) AS rows_restored
FROM salesforce_logs
ORDER BY timestamp DESC;
```
The metrics have the following meanings:
| Metric |
Initial load |
Incremental load |
| `ROWS_ADDED` |
Records loaded that aren't marked as deleted. |
Active source records that don't exist in the destination table. |
| `ROWS_ADDED_DELETED` |
Records loaded that are already marked as deleted. |
Source records marked as deleted that don't exist in the destination table. |
| `ROWS_UPDATED` |
`0` |
Active source records that match active records in the destination table. |
| `ROWS_DELETED` |
`0` |
Source records marked as deleted that match active records in the destination table. |
| `ROWS_RESTORED` |
`0` |
Active source records that match records marked as deleted in the destination table. |
`ROWS_UPDATED` counts active source records that match active destination records. It doesn't compare individual field values and doesn't indicate whether a field value changed.
The merge metric columns in the query return `NULL` when the log doesn't contain merge metrics. This occurs when **Enable Merge Metrics** is set to `false`, the Salesforce object doesn't include the `IsDeleted` field, or the log was generated by an earlier connector version that didn't support merge metrics.
---
title: MonitorActivity 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/monitoractivity.md
section: Loading & Unloading Data
---
# MonitorActivity 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Monitors the flow for activity and sends out an indicator when the flow has not had any data for some specified amount of time and again when the flow's activity is restored
## Tags
active, activity, detection, flow, inactive, monitor
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Activity Restored Message |
The message that will be the content of FlowFiles that are sent to 'activity.restored' relationship |
| Continually Send Messages |
If true, will send inactivity indicator continually every Threshold Duration amount of time until activity is restored; if false, will send an indicator only when the flow first becomes inactive |
| Copy Attributes |
If true, will copy all flow file attributes from the flow file that resumed activity to the newly created indicator flow file |
| Inactivity Message |
The message that will be the content of FlowFiles that are sent to the 'inactive' relationship |
| Monitoring Scope |
Specify how to determine activeness of the flow. 'node' means that activeness is examined at individual node separately. It can be useful if DFM expects each node should receive flow files in a distributed manner. With 'cluster', it defines the flow is active while at least one node receives flow files actively. If NiFi is running as standalone mode, this should be set as 'node', if it 's' cluster ', NiFi logs a warning message and act as' node'scope. |
| Reporting Node |
Specify which node should send notification flow-files to inactive and activity.restored relationships. With 'all', every node in this cluster send notification flow-files. 'primary' means flow-files will be sent only from a primary node. If NiFi is running as standalone mode, this should be set as 'all', even if it 's' primary ', NiFi act as' all'. |
| Reset State on Restart |
When the processor gets started or restarted, if set to true, the initial state will always be active. Otherwise, the last reported flow state will be preserved. |
| Threshold Duration |
Determines how much time must elapse before considering the flow to be inactive |
| Wait for Activity |
When the processor gets started or restarted, if set to true, only send an inactive indicator if there had been activity beforehand. Otherwise send an inactive indicator even if there had not been activity beforehand. |
## State management
| Scopes |
Description |
| LOCAL |
MonitorActivity stores the last timestamp at each node as state, so that it can examine activity at cluster wide. If 'Copy Attribute' is set to true, then flow file attributes are also persisted. In local scope, it stores last known activity timestamp if the flow is inactive. |
| CLUSTER |
MonitorActivity stores the last timestamp at each node as state, so that it can examine activity at cluster wide. If 'Copy Attribute' is set to true, then flow file attributes are also persisted. In local scope, it stores last known activity timestamp if the flow is inactive. |
## Relationships
| Name |
Description |
| activity.restored |
This relationship is used to transfer an Activity Restored indicator when FlowFiles are routing to 'success' following a period of inactivity |
| inactive |
This relationship is used to transfer an Inactivity indicator when no FlowFiles are routed to 'success' for Threshold Duration amount of time |
| success |
All incoming FlowFiles are routed to success |
## Writes attributes
| Name |
Description |
| inactivityStartMillis |
The time at which Inactivity began, in the form of milliseconds since Epoch |
| inactivityDurationMillis |
The number of milliseconds that the inactivity has spanned |
---
title: MoveAzureDataLakeStorage 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/moveazuredatalakestorage.md
section: Loading & Unloading Data
---
# MoveAzureDataLakeStorage 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-azure-nar
## Description
Moves content within an Azure Data Lake Storage Gen 2. After the move, files will be no longer available on source location.
## Tags
adlsgen2, azure, cloud, datalake, microsoft, storage
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| ADLS Credentials |
Controller Service used to obtain Azure Credentials. |
| Conflict Resolution Strategy |
Indicates what should happen when a file with the same name already exists in the output directory |
| Destination Directory |
Name of the Azure Storage Directory where the files will be moved. The Directory Name cannot contain a leading '/'. The root directory can be designated by the empty string value. Non-existing directories will be created. If the original directory structure should be kept, the full directory path needs to be provided after the destination directory. e.g.: destdir/$\{azure.directory\} |
| Destination Filesystem |
Name of the Azure Storage File System where the files will be moved. |
| File Name |
The filename |
| Source Directory |
Name of the Azure Storage Directory from where the move should happen. The Directory Name cannot contain a leading '/'. The root directory can be designated by the empty string value. |
| Source Filesystem |
Name of the Azure Storage File System from where the move should happen. |
| proxy-configuration-service |
Specifies the Proxy Configuration Controller Service to proxy network requests. In case of SOCKS, it is not guaranteed that the selected SOCKS Version will be used by the processor. |
## Relationships
| Name |
Description |
| failure |
Files that could not be written to Azure storage for some reason are transferred to this relationship |
| success |
Files that have been successfully written to Azure storage are transferred to this relationship |
## Writes attributes
| Name |
Description |
| azure.source.filesystem |
The name of the source Azure File System |
| azure.source.directory |
The name of the source Azure Directory |
| azure.filesystem |
The name of the Azure File System |
| azure.directory |
The name of the Azure Directory |
| azure.filename |
The name of the Azure File |
| azure.primaryUri |
Primary location for file content |
| azure.length |
The length of the Azure File |
## See also
- [org.apache.nifi.processors.azure.storage.DeleteAzureDataLakeStorage](/user-guide/data-integration/openflow/processors/deleteazuredatalakestorage)
- [org.apache.nifi.processors.azure.storage.FetchAzureDataLakeStorage](/user-guide/data-integration/openflow/processors/fetchazuredatalakestorage)
- [org.apache.nifi.processors.azure.storage.ListAzureDataLakeStorage](/user-guide/data-integration/openflow/processors/listazuredatalakestorage)
---
title: Notify 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/notify.md
section: Loading & Unloading Data
---
# Notify 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
org.apache.nifi | nifi-standard-nar
## Description
Caches a release signal identifier in the distributed cache, optionally along with the FlowFile's attributes. Any flow files held at a corresponding Wait processor will be released once this signal in the cache is discovered.
## Tags
cache, distributed, map, notify, release, signal
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| attribute-cache-regex |
Any attributes whose names match this regex will be stored in the distributed cache to be copied to any FlowFiles released from a corresponding Wait processor. Note that the uuid attribute will not be cached regardless of this value. If blank, no attributes will be cached. |
| distributed-cache-service |
The Controller Service that is used to cache release signals in order to release files queued at a corresponding Wait processor |
| release-signal-id |
A value, or the results of an Attribute Expression Language statement, which will be evaluated against a FlowFile in order to determine the release signal cache key |
| signal-buffer-count |
Specify the maximum number of incoming flow files that can be buffered until signals are notified to cache service. The more buffer can provide the better performance, as it reduces the number of interactions with cache service by grouping signals by signal identifier when multiple incoming flow files share the same signal identifier. |
| signal-counter-delta |
A value, or the results of an Attribute Expression Language statement, which will be evaluated against a FlowFile in order to determine the signal counter delta. Specify how much the counter should increase. For example, if multiple signal events are processed at upstream flow in batch oriented way, the number of events processed can be notified with this property at once. Zero (0) has a special meaning, it clears target count back to 0, which is especially useful when used with Wait Releasable FlowFile Count = Zero (0) mode, to provide 'open-close-gate' type of flow control. One (1) can open a corresponding Wait processor, and Zero (0) can negate it as if closing a gate. |
| signal-counter-name |
A value, or the results of an Attribute Expression Language statement, which will be evaluated against a FlowFile in order to determine the signal counter name. Signal counter name is useful when a corresponding Wait processor needs to know the number of occurrences of different types of events, such as success or failure, or destination data source names, etc. |
## Relationships
| Name |
Description |
| failure |
When the cache cannot be reached, or if the Release Signal Identifier evaluates to null or empty, FlowFiles will be routed to this relationship |
| success |
All FlowFiles where the release signal has been successfully entered in the cache will be routed to this relationship |
## Writes attributes
| Name |
Description |
| notified |
All FlowFiles will have an attribute 'notified'. The value of this attribute is true, is the FlowFile is notified, otherwise false. |
## See also
- [org.apache.nifi.processors.standard.Wait](/user-guide/data-integration/openflow/processors/wait)
---
title: Object definition overrides for the Openflow Connector for Shopify
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/shopify/object-definitions.md
section: Loading & Unloading Data
---
# Object definition overrides for the %shopifyof%
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
- [About the Openflow Connector for Shopify](/user-guide/data-integration/openflow/connectors/shopify/about)
- [Set up the Openflow Connector for Shopify](/user-guide/data-integration/openflow/connectors/shopify/setup)
- [Maintain the Openflow Connector for Shopify](/user-guide/data-integration/openflow/connectors/shopify/maintain)
- [Troubleshoot the Openflow Connector for Shopify](/user-guide/data-integration/openflow/connectors/shopify/troubleshoot)
This topic describes the **Object Definitions Override** parameter in detail, including the
full schema, promoted column and child field definitions, and a complete example.
The **Object Definitions Override** parameter accepts a JSON array of object definitions.
Each definition can add a new object type or fully replace an existing catalog entry.
## Object definition schema
The **Object Definitions Override** value must be valid JSON. If the JSON is malformed,
the connector fails to start. Validate your JSON before applying the override.
The following fields are supported in each object definition:
| Field |
Description |
| `apiType` |
**(Required)** The query endpoint name in the Shopify Admin GraphQL API. This must match the root query field exactly (for example, `orders` for the orders query (https://shopify.dev/docs/api/admin-graphql/2026-04/queries/orders), `products` for the products query (https://shopify.dev/docs/api/admin-graphql/2026-04/queries/products)). Used as the key for lookup and override matching. |
| `tableName` |
**(Required)** The Snowflake destination table name. |
| `gidTypeName` |
The Shopify GID resource type (for example, `Order`, `Product`). Used for delete cascade and child record routing. |
| `additionalGidTypeNames` |
Array of additional GID type names that also map to this object. Use when Shopify returns the same resource under more than one GID type name, so that records are routed to the correct table regardless of which GID type appears in the response. |
| `graphqlFields` |
List of GraphQL selection fields. Each entry is a field name, a nested selection (for example, `"totalPriceSet { shopMoney { amount currencyCode } }"`), or an aliased field with arguments (for example, `"tier: metafield(key: \"custom.tier\") { value }"`). Aliases are useful for querying metafields by key. |
| `requiredQueryArgs` |
Map of fixed GraphQL argument key-value pairs appended to every query for this object. Use for endpoints that require non-standard arguments that aren't covered by the built-in query parameters (for example, `{"type": "SALES_CHANNEL"}`). |
| `supportsIncremental` |
Whether the object supports incremental sync. Default: `true`. |
| `incrementalField` |
The field used for watermark-based incremental queries (for example, `updatedAt`, `createdAt`). The field must both exist on the returned type **and** be accepted as a filter by the query root's `query:` argument. A field that exists on the type but isn't supported as a filter causes `GetShopifyIncremental` to fail with `Invalid search field: `. If this happens, set `supportsIncremental` to `false` and `refreshStrategy` to `FULL_PERIODIC` instead.
|
| `refreshStrategy` |
Controls the sync mode. `INCREMENTAL` (default) uses watermark-based incremental queries. `FULL_PERIODIC` performs a complete re-sync on each run instead. `PARENT_PIGGYBACKED` means this object is extracted from another object's query response and is not queried independently. |
| `supportsDeletes` |
Whether the connector should track deletion events for this object. Default: `false`. |
| `promotedColumns` |
Array of column definitions that extract values from the JSON payload into dedicated Snowflake columns. For more information, see [Promoted columns](#label-promoted-columns). |
| `childFields` |
Array of child connection definitions that are extracted into separate tables. For more information, see [Child fields](#label-child-fields). |
| `ignoredFields` |
List of field names to exclude from queries. Matches only on the leading name of each top-level entry in `graphqlFields`. For nested fields inside a sub-selection (for example, a field inside `defaultEmailAddress { ... }`), `ignoredFields` has no effect. Remove the field directly from the sub-selection in `graphqlFields` instead.
|
| `supportsBulk` |
Whether the object supports bulk queries through the Shopify Bulk Operations API. Default: `true`. |
| `sortKeys` |
List of sort key values (from the object's corresponding `SortKeys` enum) used to order results during bulk and incremental queries. For example, `["UPDATED_AT", "ID"]`. |
| `sortKeyStyle` |
How sort key values are formatted in queries. `ENUM` (default) uses bare enum values (for example, `UPDATED_AT`). `STRING` uses quoted lowercase strings (for example, `"updated_at"`). Use `STRING` for object types that accept string sort keys, such as metaobjects. |
## Promoted columns
Promoted columns extract specific values from the raw JSON payload into dedicated typed
columns in the destination table. This makes frequently queried fields available as
first-class Snowflake columns for efficient filtering and aggregation.
Each promoted column has the following fields:
| Field |
Description |
| `name` |
The Snowflake column name (uppercase recommended). |
| `path` |
A JSONPath expression pointing to the value in the raw record (for example, `$.email`, `$.totalPriceSet.shopMoney.amount`). |
| `type` |
The Snowflake column type. The following values are supported:
| Value |
Snowflake type |
Notes |
| `string` |
`VARCHAR` |
None |
| `integer` |
`NUMBER(38,0)` |
None |
| `boolean` |
`BOOLEAN` |
None |
| `float` |
`FLOAT` |
None |
| `money` |
`NUMBER(38,4)` |
Converts Shopify amount strings to numeric. |
| `timestamp` |
`TIMESTAMP_TZ` |
ISO-8601 strings. |
| `date` |
`DATE` |
None |
| `id` |
`NUMBER(38,0)` |
Strips the `gid://shopify/*/` prefix and returns the numeric ID. |
| `gid` |
`VARCHAR` |
Stores the full GID string. |
| `json` |
`VARIANT` |
Stores sub-objects as VARIANT. |
|
## Child fields
Child field definitions extract nested connections (such as order line items) into separate
Snowflake tables. Each child table includes a `__PARENT_ID` column linking records back
to the parent.
Each child field has the following fields:
| Field |
Description |
| `fieldName` |
The GraphQL connection field name in the parent object (for example, `lineItems`). |
| `tableName` |
The Snowflake table name for the child records. |
| `gidTypeName` |
The Shopify GID type for the child (for example, `LineItem`). |
| `connectionType` |
`edges` (paginated connection) or `array` (inline array). Default: `edges`. |
| `pageSize` |
The `first:` limit applied to this child connection in incremental queries. Default and maximum: `250`. |
| `graphqlFields` |
Explicit GraphQL selection set for the child table. If omitted, the connector parses the child's fields from the matching connection entry in the parent's `graphqlFields` list. |
| `promotedColumns` |
Array of promoted column definitions for the child table, using the same schema as top-level `promotedColumns`. |
Shopify rejects `pageSize` values above 250 with a `first cannot exceed 250` error. This limit doesn't apply to bulk loads: the Shopify Bulk Operations API ignores the `first:` argument and returns all child records. For more information, see [Limitations](/user-guide/data-integration/openflow/connectors/shopify/about#label-shopify-connector-limitations).
## Union types and GID routing
When a query root returns a union type, the records in the Shopify Bulk API response carry GID types that correspond to the **concrete wrapper node types**, not the query root name or the GraphQL union type name. The connector's `PartitionShopifyByObject` processor routes records by GID type, so if the GID type isn't registered in `gidTypeName` or `additionalGidTypeNames`, records are routed to failure.
For example, the `discountNodes` query root returns the `Discount` union. The records in the response carry GID types `DiscountCodeNode` and `DiscountAutomaticNode`, not `DiscountNode` or `Discount`. To route all records to the same table, set `gidTypeName` to one concrete type and list the others in `additionalGidTypeNames`.
The following example configures `discountNodes` correctly:
```json
[
{
"apiType": "discountNodes",
"tableName": "DISCOUNTS",
"gidTypeName": "DiscountCodeNode",
"additionalGidTypeNames": ["DiscountAutomaticNode"],
"graphqlFields": [
"id",
"discount { ... on DiscountCodeBasic { title status startsAt endsAt createdAt updatedAt } ... on DiscountAutomaticBasic { title status startsAt endsAt createdAt updatedAt } }"
]
}
]
```
When using `promotedColumns` on a union-type object, the JSON root after partitioning is the wrapper node (for example, `{ id, discount: { … } }`), not the inner union member. Promoted column `path` values must include the wrapper field, for example `$.discount.title`, not `$.title`.
For discount syncing with incremental support, use `discountNodes` rather than `codeDiscountNodes` or `automaticDiscountNodes`. The per-subtype query roots don't accept `updated_at` as a filter, so they require `supportsIncremental: false` and `refreshStrategy: "FULL_PERIODIC"`.
To verify which GID types appear in a real response before writing the definition, run a small test bulk load or check the Shopify documentation for the concrete types returned by the query root.
## Example: Register a custom object type with promoted columns
The following override customizes an existing catalog entry to add scalar fields, nested object
selections, a metafield alias, and promoted columns. One promoted column extracts a value
directly from the aliased metafield.
```json
[
{
"apiType": "draftOrders",
"tableName": "DRAFT_ORDERS",
"gidTypeName": "DraftOrder",
"supportsBulk": true,
"supportsIncremental": true,
"incrementalField": "updatedAt",
"ignoredFields": [],
"sortKeys": ["UPDATED_AT", "ID"],
"supportsDeletes": false,
"graphqlFields": [
"id",
"createdAt",
"updatedAt",
"name",
"status",
"email",
"currencyCode",
"totalQuantityOfLineItems",
"customer { id }",
"totalPriceSet { shopMoney { amount currencyCode } }",
"billingAddress { address1 city countryCode zip }",
"draft_po_number: metafield(key: \"custom.draft_po_number\") { key namespace compareDigest createdAt id jsonValue legacyResourceId updatedAt value definition { id description key pinnedPosition } }"
],
"promotedColumns": [
{ "name": "STATUS", "path": "$.status", "type": "string" },
{ "name": "NAME", "path": "$.name", "type": "string" },
{ "name": "CUSTOMER_ID", "path": "$.customer.id", "type": "gid" },
{ "name": "TOTAL_PRICE_AMOUNT", "path": "$.totalPriceSet.shopMoney.amount", "type": "money" },
{ "name": "DRAFT_PO_NUMBER", "path": "$.draft_po_number.value", "type": "string" }
],
"childFields": []
}
]
```
## Example: Override an object with child fields
The following override customizes the `orders` object to extract line items and fulfillments
into separate tables. Line items use a paginated connection (`edges`); fulfillments are an
inline array in the parent response (`array`).
```json
[
{
"apiType": "orders",
"tableName": "ORDERS",
"gidTypeName": "Order",
"graphqlFields": [
"id",
"createdAt",
"updatedAt",
"name",
"email",
"lineItems(first: 250) { edges { cursor node { id title quantity originalUnitPriceSet { shopMoney { amount currencyCode } } } } }",
"fulfillments { id status createdAt }"
],
"childFields": [
{
"fieldName": "lineItems",
"tableName": "ORDER_LINE_ITEMS",
"gidTypeName": "LineItem",
"connectionType": "edges"
},
{
"fieldName": "fulfillments",
"tableName": "ORDER_FULFILLMENTS",
"gidTypeName": "Fulfillment",
"connectionType": "array"
}
]
}
]
```
---
title: OpenAiTranscribeAudio 2025.10.9.21
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/processors/openaitranscribeaudio.md
section: Loading & Unloading Data
---
# OpenAiTranscribeAudio 2025.10.9.21
This feature is not available in the People's Republic of China.
This feature is not available in the People's Republic of China.
Openflow Snowflake Deployments are available to all accounts in AWS, Azure, and GCP [](#label-na-general-regions).
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
**Related Topics**
- [About Openflow](/user-guide/data-integration/openflow/about)
- [All processors (alphabetical)](/user-guide/data-integration/openflow/processors/index)
## Bundle
com.snowflake.openflow.runtime | runtime-openai-nar
## Description
Transcribes audio into English text. The audio data must be in one of these formats: flac, mp3, mp4, mpeg, mpga, m4a, ogg, wav, webm
## Tags
audio, flac, m4a, mp3, mp4, mpeg, mpga, ogg, openai, openflow, speech-to-text, text, transcribe, translate, wav, webm
## Input Requirement
REQUIRED
## Supports Sensitive Dynamic Properties
false
## Properties
| Property |
Description |
| Model Name |
The name of the OpenAI Model to use |
| OpenAI API Key |
The API Key for interacting with OpenAI |
| Prompt |
Text that can be used to guide the model's style or continue a previous audio segment. The text must be in English. |
| Response Format |
Specifies which format is desired for the output |
| Temperature |
The sampling temperature to use. The value must be a floating-point number between 0.0 and 1.0. A higher value, such as 0.8 will result in more of an interpreted translation, whereas a value of 0.0 will result in a more literal translation. |
## Relationships
| Name |
Description |
| failure |
FlowFiles that could not be transcribed are routed to this relationship. |
| success |
FlowFiles that have been successfully transcribed will be transferred to this relationship. |
## Use Cases Involving Other Components
| Create embeddings for audio data and insert them into Pinecone so that the audio can be made available to a large language model (LLM) such as OpenAI's GPT models. |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
---
title: Openflow BYOC - Set up custom ingress
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/setup-openflow-byoc-custom-ingress.md
section: Loading & Unloading Data
---
# Openflow BYOC - Set up custom ingress
This feature is not available in the People's Republic of China.
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
- [About Openflow: BYOC deployments](/user-guide/data-integration/openflow/about-byoc)
- [Manage Openflow](/user-guide/data-integration/openflow/manage)
- [Monitor Openflow using telemetry data](/user-guide/data-integration/openflow/monitor)
- [Troubleshoot Openflow](/user-guide/data-integration/openflow/troubleshoot)
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
This topic describes the considerations for and steps required to set up an Openflow BYOC deployment with a custom ingress solution managed within your own AWS account.
## Benefits
Custom ingress for Openflow BYOC deployments provides your organization with:
- Stronger security with network-level restrictions that can limit access to only your VPN or private network.
- Full control over the URL and TLS certificate used to access Openflow to meet your security and compliance requirements.
## Considerations
With Snowflake managed ingress, Openflow creates the necessary DNS records, public load balancer, and manages the TLS certificate for the Openflow runtimes in your BYOC deployment.
When you enable custom ingress, Openflow will no longer automatically manage external DNS records, will not create a public load balancer automatically, and will no longer manage certificates for the Openflow runtimes. You must manage these resources within your own AWS account.
## Configure custom ingress in Snowflake Openflow
1. Enable custom ingress during deployment creation.
- During deployment creation, enable **Custom ingress** and specify your preferred fully qualified domain name (FQDN) in the **Hostname** field.
- You must be able to manage this DNS record and create a TLS certificate for this FQDN. Do not use a subdomain of `snowflakecomputing.cn`.
- You must not include the protocol **https://** or a trailing slash **/** in the FQDN.
- For example, if you specify `openflow01.your-domain.org`, you will access a runtime named "My Runtime" at `https://openflow01.your-domain.org/my-runtime/nifi/`.
2. Download the CloudFormation template. This file has all of the settings required for Openflow to run as your custom ingress domain.
## Configure custom ingress in AWS
`{deployment-key}` represents the Openflow unique identifier applied to cloud resources created and managed by Openflow for a particular deployment.
This is in the `DataPlaneKey` parameter of the CloudFormation template, also available in Openflow through the **View Details** menu option for the deployment.
1. Add the following tag to the private subnets for your Openflow deployment:
- Key: **kubernetes.io/role/internal-elb**
- Value: `1`
2. If your private subnets are used by other EKS clusters, you must also tag them with the name of the Openflow cluster. This allows Openflow to create a load balancer alongside other load balancers.
- Key: **kubernetes.io/cluster/\{deployment-key\}**
- Value: `1`
3. Upload the CloudFormation template. Wait approximately 30 minutes for Openflow to create the internal network load balancer.
- You can find the internal network load balancer in the AWS Console under **EC2** %ra% **Load Balancers**.
- The load balancer will be named `runtime-ingress-{deployment-key}`.
4. Obtain the internal IP address of the Openflow-managed AWS internal network load balancer.
- Under **EC2** %ra% **Load Balancers**, navigate to the details page and copy the **DNS name** of the Load Balancer.
- Log into your agent EC2 instance (identified as **openflow-agent-\{deployment-key\}**) and run the command `nslookup {openflow-load-balancer-dns-name}`.
- Copy the IP addresses of the Openflow-managed AWS internal network load balancer. These are destinations for the target group of the load balancer you will create in a following step.
5. Provision a TLS certificate.
- Obtain a TLS certificate for the load balancer that will handle traffic to the Openflow runtime UIs. You can generate a certificate using AWS Certificate Manager (ACM) or import an existing certificate.
6. Create a network load balancer that will route traffic to the Openflow-managed AWS internal network load balancer.
1. In your AWS account, create a Network Load Balancer with the following configuration:
- Name: We recommend the naming convention `custom-ingress-external-{deployment-key}`, where `{deployment-key}` is the key of your Openflow deployment.
- Type: **Network Load Balancer**
- Scheme: **Internal** or **Internet-facing**, depending on your requirements.
- VPC: Select the VPC of your deployment
- Availability Zones: Select both Availability Zones where your Openflow deployment is running.
- Subnets: Select the private subnets of your VPC for an **Internal** Load Balancer, or the public subnets of your VPC for an **Internet-facing** Load Balancer.
- Security groups: Select or create a security group that allows traffic on port `443`
- Default SSL/TLS server certificate: Import your SSL/TLS certificate
- Target group: Create a new target group with the following settings:
- Target type: **IP addresses**
- Protocol: **TLS**
- Port: **443**
- VPC: Verify the VPC matches your deployment
- Type the IP address of the internal network load balancer created by Openflow (obtained in the previous step) as the target and select **Include as pending below**.
2. Once the load balancer is created, copy the DNS name for the load balancer to use in the next step.
3. For more information on how to create a network load balancer, see Create a Network Load Balancer (https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html).
7. Create a DNS CNAME record that maps your custom ingress FQDN to the AWS load balancer's DNS name.
- For detailed DNS configuration instructions in Route 53, see Create records in Route 53 (https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-creating.html).
## Verification
1. The Openflow deployment shows a status of **Active** in the **Deployments** page.
2. Create a runtime in the Openflow deployment.
3. Once the runtime is **Active**, click on the runtime name or use the **View canvas** menu option to access the runtime's UI.
4. Openflow directs you to the runtime with the hostname specified during deployment creation. For example, `https://openflow01.your-domain.org/my-runtime/nifi/`.
## Troubleshooting
The following sections provide troubleshooting steps for common issues with custom ingress. If you are still experiencing issues after performing these checks, file a [Snowflake Support](https://docs.snowflake.cn/user-guide/contacting-support) case.
### Load balancer target health check
The target group for your network load balancer should list the IP addresses of the Openflow-managed internal network load balancer as targets. All of these targets should show as **Healthy**. If targets are **Unhealthy**, use the following checks to narrow down where traffic is failing.
1. In the AWS console, open **EC2** %ra% **Load Balancers**.
2. Locate the Openflow-managed load balancer that manages ingress to the Kubernetes cluster. This load balancer is named `runtime-ingress-{deployment-key}`.
3. Review the target health for that load balancer under the **Resource map** tab.
4. If the Openflow-managed load balancer is not active or has **Unhealthy** targets:
- Traffic may be blocked between the Openflow-managed load balancer and the BYOC cluster, or a service inside the cluster may not be ready.
- Generate a diagnostic bundle by running `./diagnostics.sh` from the **openflow-agent-\{deployment-key\}** EC2 instance and attach it to a [Snowflake Support](https://docs.snowflake.cn/user-guide/contacting-support) case.
5. If the Openflow-managed load balancer is active and has healthy targets, check the target health for your load balancer.
6. If your load balancer's targets are **Unhealthy**, the path from your load balancer to the Openflow-managed load balancer is the most likely problem:
- **Incorrect or stale IP addresses in your target group.** The Openflow-managed load balancer exposes multiple IP addresses that can change over time. To get the latest values, run `nslookup` with the **DNS name** of the Openflow-managed load balancer. Update your load balancer's targets as necessary.
- **Security group rules.** Confirm that inbound rules on the Openflow-managed load balancer's security groups allow TCP `443` from your load balancer. Traffic can fail if your load balancer can't reach the Openflow load balancer on port `443`.
### Browser security blocking
Some problems with custom ingress are caused by corporate browser security, firewalls, or web proxies that block or inspect traffic to your custom hostname. Those policies are separate from AWS load balancer configuration. You may find that users can't open the Openflow UI even when AWS load balancers report healthy targets.
To verify connectivity through the load balancers to the Openflow services:
1. In the AWS console, open **EC2** %ra% **Load Balancers** to get the DNS name of the load balancer that is serving traffic and the TLS certificate for your custom ingress domain name.
- This is **not** the **runtime-ingress-\{deployment-key\}** load balancer.
2. From the **openflow-agent-\{deployment-key\}** EC2 instance, verify connectivity through the load balancers to the Openflow deployment. Run the command:
```bash
curl -kv https://{your-load-balancer-dns-name}
```
- If the command outputs the expected certificate information and a successful 404 status code response, you have successfully verified connectivity to your Openflow deployment.
- If the command times out or returns an error, create a [Snowflake Support](https://docs.snowflake.cn/user-guide/contacting-support) case and attach a diagnostic bundle generated by running `./diagnostics.sh` from the Openflow Agent instance.
3. From the Openflow Agent instance, you can also verify the DNS CNAME record for your custom ingress FQDN. Run the command:
```bash
source ~/.env && nslookup $DOMAIN
```
- If the command returns the IP addresses of the load balancer that is performing TLS termination for your custom ingress domain name, you have successfully verified the DNS CNAME record.
- If the command returns no results, the DNS CNAME record is not configured correctly. Check the DNS record for your custom ingress FQDN and ensure it points to your load balancer's DNS name.
If the Openflow Agent connected successfully through your load balancer's DNS and you have verified the DNS CNAME record, a security policy or firewall is likely blocking traffic from your browser to the Openflow BYOC deployment. Work with your security team to allowlist your custom ingress FQDN.
---
title: Openflow BYOC - Set up encrypted EBS volumes
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/setup-openflow-byoc-encrypted-volumes.md
section: Loading & Unloading Data
---
# Openflow BYOC - Set up encrypted EBS volumes
This feature is not available in the People's Republic of China.
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
- [About Openflow: BYOC deployments](/user-guide/data-integration/openflow/about-byoc)
- [Manage Openflow](/user-guide/data-integration/openflow/manage)
- [Monitor Openflow using telemetry data](/user-guide/data-integration/openflow/monitor)
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
This topic describes the steps to set up an Openflow BYOC deployment with encrypted Elastic Block Storage (EBS) volumes using one of the following methods:
- [](#label-openflow-byoc-encrypted-ebs-kms-key)
- [](#label-openflow-byoc-encrypted-ebs-default-encryption)
Both of these solutions provide encrypted EBS volumes that meet the following storage requirements of Openflow BYOC:
- Root volume for the Openflow Agent EC2 instance
- Root volumes for the EC2 instances in each EKS Cluster Node Group
- Persistent volumes for Openflow's runtimes and supporting components
- `$AWS_ACCOUNT_ID` represents the AWS Account ID of the account where Openflow is deployed.
- `$AWS_REGION` represents the AWS Region of the account, for example `us-west-2`.
- `$AWS_KMS_KEY_ARN` represents the Amazon Resource Name (ARN) of the Amazon Key Management Service (AWS KMS) key that Openflow will use for encrypted EBS volumes.
- `$DEPLOYMENT_KEY` represents the Openflow unique identifier applied to cloud resources created and managed by Openflow for a particular deployment.
This is in the `DataPlaneKey` parameter of the CloudFormation template, also available in Openflow through the **View Details** menu option for the deployment.
## Prerequisites
This topic assumes that you have completed the prerequisites for setting up Openflow BYOC. For more information, see [Set up Openflow - BYOC](/user-guide/data-integration/openflow/setup-openflow-byoc).
You must also have access to an AWS KMS key that Openflow will use for encrypted EBS volumes.
## Provide a specific AWS KMS Key for Encrypted EBS Volumes
When uploading the CloudFormation template for your Openflow BYOC Deployment, you can provide the ARN for the AWS KMS key that Openflow uses for encrypted EBS volumes.
Using this configuration, Openflow makes requests for encrypted EBS volumes, ensuring that all SCP policies are satisfied. Snowflake recommends this approach for most customers.
This allows you to use different KMS keys for different applications, reducing the risk of a single key being compromised.
To ensure that Openflow has the necessary permissions to use this key, perform the following tasks:
1. Ensure that the AWS KMS key grants permissions to the AWS Autoscaling Service Role. The Key Policy must include the following statement:
```json
{
"Sid": "Allow Autoscaling to use the key",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::$AWS_ACCOUNT_ID:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
},
"Action": [
"kms:CreateGrant",
"kms:Decrypt",
"kms:Encrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "*"
}
```
2. Enter the ARN of the AWS KMS key in the `EBSKMSKeyArn` parameter of the CloudFormation stack when uploading the template.
For example, `arn:aws:kms:$AWS_REGION:$AWS_ACCOUNT_ID:key/1a1a11aa-aa1a-aaa1a-a1a1-000000000000`.
Approximately 20 minutes after uploading the CloudFormation template, the Openflow BYOC Deployment creates a new IAM Role with the name `$DEPLOYMENT_KEY-eks-role`.
3. Add the following statement to the KMS key policy to grant permissions for Openflow to use the key:
```json
{
"Sid": "Allow Openflow Deployment to encrypt EBS volumes",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::$AWS_ACCOUNT_ID:role/$DEPLOYMENT_KEY-eks-role"
},
"Action": [
"kms:Decrypt",
"kms:Encrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:CreateGrant",
"kms:DescribeKey"
],
"Resource": "*"
}
```
Openflow automatically detects the new permissions for the KMS key and continues the installation process. The Openflow BYOC deployment will become `Active` after approximately 20 minutes.
## Enable Encrypted EBS Volumes by default for your AWS Account
AWS accounts can encrypt new EBS volumes by default by following the AWS EBS encryption by default documentation (https://docs.aws.amazon.com/ebs/latest/userguide/encryption-by-default.html).
With this configuration, Openflow makes requests for unencrypted EBS volumes, but the AWS API will return an encrypted EBS volume. The following steps ensure that Openflow has permissions to use the KMS key for these encrypted volumes.
Whether you choose to use the AWS managed key `aws/ebs` or your own KMS key, you must attach an IAM Policy to the Openflow IAM Role `$DEPLOYMENT_KEY-eks-role` that grants the necessary permissions to use the key.
1. Create an IAM Policy to allow Openflow to use the KMS key by replacing `$AWS_KMS_KEY_ARN` with the ARN of the KMS key.
```json
{
"Sid": "Allow Openflow EKS Role to encrypt EBS volumes",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:Encrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:CreateGrant",
"kms:DescribeKey"
],
"Resource": "$AWS_KMS_KEY_ARN"
}
```
2. Ensure that the AWS KMS key grants permissions to the AWS Autoscaling Service Role. The Key Policy must include the following statement:
```json
{
"Sid": "Allow Autoscaling to use the key",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::$AWS_ACCOUNT_ID:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
},
"Action": [
"kms:CreateGrant",
"kms:Decrypt",
"kms:Encrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "*"
}
```
3. When uploading the Openflow BYOC CloudFormation template:
- Leave the optional `EBSKMSKeyArn` parameter blank.
- Set the `AdditionalEksRolePolicyArns` parameter to the ARN of the new IAM Policy created previously. For example, `arn:aws:iam::$AWS_ACCOUNT_ID:policy/openflow-kms-key-access-policy`.
Approximately 20 minutes after uploading the CloudFormation template, the Openflow BYOC Deployment creates a new IAM Role with the name `$DEPLOYMENT_KEY-eks-role`.
4. Add the following statement to the KMS key policy to grant permissions for Openflow to use the key:
```json
{
"Sid": "Allow Openflow Deployment to encrypt EBS volumes",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::$AWS_ACCOUNT_ID:role/$DEPLOYMENT_KEY-eks-role"
},
"Action": [
"kms:Decrypt",
"kms:Encrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:CreateGrant",
"kms:DescribeKey"
],
"Resource": "*"
}
```
Openflow automatically detects the new permissions for the KMS key and continues the installation process. The Openflow BYOC deployment will become `Active` after approximately 20 minutes.
---
title: Openflow BYOC cost and scaling considerations
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/cost-byoc.md
section: Loading & Unloading Data
---
# Openflow BYOC cost and scaling considerations
This feature is not available in the People's Republic of China.
Openflow BYOC deployments are available to all accounts in AWS [](#label-na-general-regions).
- [About Openflow: BYOC deployments](/user-guide/data-integration/openflow/about-byoc)
- [Set up Openflow - BYOC](/user-guide/data-integration/openflow/setup-openflow-byoc)
- [Manage Openflow](/user-guide/data-integration/openflow/manage)
- [Monitor Openflow using telemetry data](/user-guide/data-integration/openflow/monitor)
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
Snowflake Openflow BYOC has cost considerations in multiple areas, including infrastructure, compute, data ingestion and others.
Scaling Openflow involves understanding these costs. The following sections describe Openflow BYOC costs in general,
and provide a number of examples of scaling Openflow BYOC runtimes and associated costs.
## Openflow BYOC costs
When using Openflow, you can incur the following types of costs:
| Cost category |
Description |
| Openflow (shown as **Openflow Compute BYOC** on your Snowflake bill) |
Cost based on the number of virtual CPU cores (vCPU) used by connector runtimes
within your "bring your own cloud (BYOC)" environment. You are charged for active runtimes only.
The compute used for Openflow management processes is excluded from this specific charge.
Credits are billed per-second with a 60-second minimum.
For an example of using vCPU and the impacts of scaling, see [](#label-openflow-byoc-scaling-overview).
For information on the rate per vCPU per hour, refer to Table 1(g) in the [Snowflake Service Consumption Table](https://www.snowflake.cn/legal-files/CreditConsumptionTable.pdf).
Additionally, the [METERING_DAILY_HISTORY](/sql-reference/account-usage/metering_daily_history)
and [METERING_HISTORY](/sql-reference/account-usage/metering_history) views in the
[Account Usage](/sql-reference/account-usage) schema can provide additional details on Openflow compute costs
using queries for *SERVICE_TYPE=OPENFLOW_COMPUTE_BYOC*.
See [Exploring compute cost](/user-guide/cost-exploring-compute) for more information on exploring compute costs in Snowflake.
|
| Infrastructure (only for BYOC configuration) |
For BYOC deployments, you directly pay your cloud provider, for example, AWS,
for the underlying infrastructure provisioned in your environment to run Openflow.
This primarily includes compute (for runtimes you provision to run the connectors and for managing the runtimes),
networking, and storage costs, and appears on your CSP bill.
The EC2 compute requirements are illustrated in the following image:
For information about monitoring costs with AWS resource tags for BYOC deployments, see
[BYOC deployment customization and tagging behavior](/user-guide/data-integration/openflow/setup-openflow-byoc#label-openflow-byoc-customization-tagging).
|
| Ingestion |
Cost for loading data into Snowflake using services such as Snowpipe or Snowpipe Streaming, based on data volume.
Appears on your Snowflake bill under respective ingestion services line items.
Certain connectors may require a standard Snowflake warehouse, incurring additional warehouse costs.
For example, database CDC connectors require a Snowflake warehouse for both initial snapshot and
incremental Change Data Capture (CDC).
You can schedule [MERGE](/sql-reference/sql/merge) operations to manage the compute cost.
|
| Telemetry Data Ingest |
Standard Snowflake charges for sending logs and metrics to Openflow deployments
and sending runtime logs to your event table within Snowflake.
The rate for credits per GB of telemetry data can be found in Table 5 in the [Snowflake Service Consumption Table](https://www.snowflake.cn/legal-files/CreditConsumptionTable.pdf).
|
## Openflow BYOC scaling
The runtimes and scaling behavior you choose are crucial for managing costs effectively.
Openflow supports different runtime types, each with its own scaling characteristics.
### Runtime types and the associated costs
The following table illustrates the scaling behavior of various runtimes and their associated costs:
| Runtimes | Activity | Snowflake costs | Cloud costs |
| ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- |
| No runtimes | None | No cost | Compute and storage of Dataplane |
| 1 small runtime (1 vCPU)
(min 1 max 2) | Active for 1 hour
Runtime does not scale to 2. | 1 runtime x 1 node x 1 vCPU x 1 hour = 1
Total = 1 vCPU-hour | Compute and storage of Dataplane |
| 2 small runtimes (1 vCPU) (min/max=2)
1 large runtime (8 vCPU) (min/max=10) | Small: 2 nodes active for 1 hour
Large: 10 nodes active for 1 hour | 2 runtimes x 2 nodes x 1 vCPU x 1 hour = 4 vCPU
1 runtime x 10 nodes x 8 vCPU x 1 hour = 80 vCPU
Total = 84 vCPU-hours | Compute and storage of Dataplane |
| 1 medium (4 vCPU)
(min =1 max=2) | First 20 minutes, 1 node is running
Scales to 2 nodes for the remaining 40 minutes of the hour
Total 1 hour
| 20 minutes = 1/3 hour
1 runtime x 1 node x 4 vCPU x 1/3 hour = 4/3
1 runtime x 2 nodes x 4 vCPU x 2/3 hour = 16/3
Total = 6 2/3 vCPU-hours | Compute and storage of Dataplane |
| 1 medium (4 vCPU)
(min/max=2) | First 30 minutes, 2 nodes running
Suspends after first 30 minutes. | 30 minutes = 1/2 hour
1 runtime x 2 nodes x 4 vCPU x 1/2 hour = 4
Total = 4 vCPU-hours | Compute and storage of Dataplane |
### Mapping runtimes to EC2 instance types
Choosing a runtime type (t-shirt size) results in the runtime pods being scheduled on the associated EC2
node group \{key\}-sm-group, \{key\}-md-group, or \{key\}-lg-group with resources described in the following table:
| Runtime type | vCPUs | Available memory (GB) | EC2 instance type | EC2 node group | EC2 node - CPUs | EC2 node - memory (GB) |
| ------------ | ----- | --------------------- | ----------------- | ---------------- | --------------- | ---------------------- |
| Small | 1 | 2 | m7i.xlarge | \{key\}-sm-group | 4 | 16 |
| Medium | 4 | 10 | m7i.4xlarge | \{key\}-md-group | 16 | 64 |
| Large | 8 | 20 | m7i.8xlarge | \{key\}-lg-group | 32 | 128 |
The type of runtime that you choose impacts the number of cores (vCPUs) consumed each second. Openflow scales the underlying EC2 node group
when additional pods need to be scheduled, based on CPU consumption, and up to the maximum node setting set during runtime creation.
EKS node groups are configured with a minimum size of 0 nodes and a maximum of 50 nodes.
The desired size is dynamically adjusted depending on the runtime required CPU and memory.
Customers are charged by their cloud service provider for the underlying nodes that host their runtime.
The underlying EC2 instances are created when the first runtime of a respective size is scheduled.
### Examples for calculating Openflow BYOC runtime consumption
- A user requests a BYOC deployment from Openflow and then installs the Openflow agent and deployment
-
- The user has not created any runtimes. 0 vCPUs are allocated, so there is no Openflow software cost.
- The user is charged by their cloud service provider for the provisioned compute and storage of the Openflow BYOC deployment.
- Total Openflow consumption = 0 vCPU-hours
- A user creates one small runtime with Min Nodes = 1 and Max Nodes = 2. Runtime stays at 1 node for 1 hour.
-
- 1 small runtime = 1 vCPU
- Total Openflow consumption = 1 vCPU-hour
- A user creates 2 small runtimes with min/max of 2 nodes each, and one large runtime with min/max of 10 nodes. These runtimes are active for 1 hour
-
- 2 small runtimes at 2 nodes = 2 runtimes x 2 nodes x 1 vCPU = 4 vCPUs
- 1 large runtime at 10 nodes = 1 runtime x 10 nodes x 8 vCPU = 80 vCPUs
- Total Openflow consumption = (4 vCPU + 80 vCPU) x 1 hour = 84 vCPU-hours
- A user creates 1 medium runtime with 1 node. After 20 minutes, it scales to 2 nodes and remains at 2 nodes for the rest of the hour.
-
- 1 medium runtime = 4 vCPUs
- 20 minutes = 1/3 hour; 40 minutes = 2/3 hour
- (1 node x 4 vCPU x 1/3 hour) + (2 nodes x 4 vCPU x 2/3 hour)
- 4/3 vCPU-hours + 16/3 vCPU-hours
- Total Openflow consumption = 20/3 vCPU-hours, so approximately 6.67 vCPU-hours
- A user creates 1 medium runtime with 2 nodes, then suspends it after 30 minutes
-
- 1 medium runtime = 4 vCPU
- 30 minutes = 1/2 hour
- Total Openflow consumption = (2 nodes x 4 vCPU x 1/2 hour) = 4 vCPU-hours
---
title: Openflow Connector for Amazon Kinesis Data Streams
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/kinesis/about.md
section: Loading & Unloading Data
---
# %kinesis%
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
- [Set up Openflow Connector for Amazon Kinesis Data Streams](/user-guide/data-integration/openflow/connectors/kinesis/setup)
- [Performance tuning of the Openflow Connector for Amazon Kinesis Data Streams](/user-guide/data-integration/openflow/connectors/kinesis/performance-tuning)
- [Maintain Openflow Connector for Amazon Kinesis Data Streams](/user-guide/data-integration/openflow/connectors/kinesis/maintenance)
- [Troubleshooting the Openflow Connector for Amazon Kinesis Data Streams](/user-guide/data-integration/openflow/connectors/kinesis/troubleshoot)
- [Openflow Connector for Kinesis Data Streams: Configuring DLQ handling](/user-guide/data-integration/openflow/connectors/kinesis/configuring-dead-letter-queue-handling)
## About
This topic describes the basic concepts of %kinesis%, including its workflow and limitations.
You can use Amazon Kinesis Data Streams (https://docs.aws.amazon.com/streams/latest/dev/introduction.html) to collect and process large streams of data records in real time. Producers continually push data to Kinesis Data Streams, and consumers process the data in real time.
A Kinesis data stream is a set of shards (https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html#shard). Each shard has a sequence of data records. A data record is the unit of data stored in a Kinesis data stream. Data records are composed of a sequence number, a partition key, and a data blob, which is an immutable sequence of bytes.
%kinesis% reads data from Kinesis streams and writes it into Snowflake tables using the
[Snowpipe Streaming](/user-guide/snowpipe-streaming/snowpipe-streaming-high-performance-overview) architecture.
Use this connector if you're looking to do the following:
- Ingest real-time events from Amazon Kinesis into Snowflake for near real-time analytics
- Ingest real-time events from Amazon Kinesis into Snowflake-managed Iceberg™ tables
- Accelerate your ingestion even more by combining Openflow speed with the Interactive Tables feature
- Use Single Message Transforms to enrich or filter data before it lands in Snowflake.
## Limitations
- One connector supports only ingestion from a single stream.
- Autoscaling is not supported. The number of Openflow runtime min and max nodes should be constant for the runtime where %kinesis% is deployed.
- The connector supports routing Kinesis traffic through Snowflake outbound AWS PrivateLink. DynamoDB traffic must use the public endpoint because Amazon DynamoDB doesn't support Private DNS. For more information, see [](#label-kinesis-configure-aws-privatelink).
### Limitations of fault tolerance with the connector
Kinesis Streams can be configured with a retention time. If for any reason the %kinesis% is not able to ingest data for more than the retention time, then expired records will not be loaded.
## Using different data types or data manipulation
The connector is configured to work with the JSON data type. It can be modified and extended in many ways. See the dedicated sub-pages in the setup section for guidance on making necessary changes, and the following shared streaming customization guides:
- [Configuring Avro data type ingestion](/user-guide/data-integration/openflow/connectors/streaming/configuring-avro-data-type-ingestion)
- [Configuring Protobuf data type ingestion](/user-guide/data-integration/openflow/connectors/streaming/configuring-protobuf-data-type-ingestion)
- [Configuring custom transformations](/user-guide/data-integration/openflow/connectors/streaming/configuring-custom-transformations)
- [Configuring Dead Letter Queue (DLQ) handling](/user-guide/data-integration/openflow/connectors/streaming/configuring-dead-letter-queue-handling) and the Kinesis-specific [Kinesis as destination for DLQ messages](configuring-dead-letter-queue-handling)
- [Configuring Private Key Authentication](/user-guide/data-integration/openflow/connectors/streaming/configuring-private-key-authentication)
### Supported data types
%kinesis% supports the following data types:
- **JSON (available by default in the connector)**
- [Avro](/user-guide/data-integration/openflow/connectors/streaming/configuring-avro-data-type-ingestion) (extra configuration required)
- [Protobuf](/user-guide/data-integration/openflow/connectors/streaming/configuring-protobuf-data-type-ingestion) (extra configuration required)
## Next steps
- [Set up Openflow Connector for Amazon Kinesis Data Streams](/user-guide/data-integration/openflow/connectors/kinesis/setup)
---
title: Openflow Connector for Kinesis Data Streams: Configuring DLQ handling
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/kinesis/configuring-dead-letter-queue-handling.md
section: Loading & Unloading Data
---
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
- [About the connector](about)
- [Set up the connector](setup)
- [Configuring Dead Letter Queue (DLQ) handling](/user-guide/data-integration/openflow/connectors/streaming/configuring-dead-letter-queue-handling)
# Openflow Connector for Kinesis Data Streams: Configuring DLQ handling
This topic explains how to configure a **Kinesis stream** as a destination for Dead Letter Queue (DLQ) messages on the **Kinesis high-performance connector**, plus the other Kinesis-specific parts of DLQ handling (the source processor, parse-failure relationship, and credential reuse).
DLQ handling is shared between the streaming connectors. Read the general guide first --- [Configuring Dead Letter Queue (DLQ) handling](/user-guide/data-integration/openflow/connectors/streaming/configuring-dead-letter-queue-handling) --- for the common concepts: the failure envelope, the Snowflake-table route, the raw/structured branches, funnels, and DLQ sink failure handling. This page covers only what is specific to Kinesis.
You don't have to apply this customization by hand. The **Openflow skill in Snowflake CoCo** can perform it for you --- describe the change you want and it edits the flow following the steps on this page. We recommend using the skill instead of configuring the components manually.
## Connector grounding
| Item |
Kinesis high-performance |
| Source processor |
`ConsumeKinesis` |
| Parse-failure relationship |
`parse.failure` |
| Connection / credentials to reuse |
`AWSCredentialsProviderControllerService` + **Region** + **Stream Name** |
| Stream-route publisher |
`PutKinesisStream` |
| Record reader / writer |
`JsonTreeReader` / `JsonRecordSetWriter` |
| Destination processor |
`PublishSnowpipeStreaming` |
## Route the parse failure into the DLQ
On a fresh connector the `ConsumeKinesis` `parse.failure` relationship is auto-terminated. Remove the auto-termination and connect `parse.failure` to the **RAW funnel** described in the [common guide](/user-guide/data-integration/openflow/connectors/streaming/configuring-dead-letter-queue-handling).
**Capture the error reason.** `ConsumeKinesis` writes a `record.error.message` FlowFile attribute on parse/serde failure. Use `${record.error.message}` for the `error_message` field in the raw/structured branch metadata (the Kafka connector has no equivalent attribute).
## Kinesis stream as destination for DLQ messages
Use this route to publish failed records back to a Kinesis stream. **Publish the original failed payload as-is --- there is no envelope and no record wrapping.** Connect the failure sources directly to a `PutKinesisStream` sink; the envelope (`raw_payload` / `structured_payload`) is only for the Snowflake-table route, because a stream consumer wants the original bytes.
**Credential reuse:** The DLQ publisher reuses the same `AWSCredentialsProviderControllerService` + **Region** as `ConsumeKinesis` --- that is, the **same AWS account/region**. If your DLQ stream lives in a **different** account or region, configure the publisher with the appropriate credentials/region (and for a fully separate environment, a separate connector).
### Step 1: Create the PutKinesisStream processor
1. Add a `PutKinesisStream` processor to the connector's process group.
2. Set the following properties:
| Property |
Value |
| Stream Name |
Your DLQ stream name. |
| AWS Credentials Provider Service |
The same `AWSCredentialsProviderControllerService` used by `ConsumeKinesis`. |
| Region |
The same region as `ConsumeKinesis`. |
`PutKinesisStream` publishes the entire FlowFile content as a single Kinesis message. If the FlowFile contains multiple records (for example, NDJSON with one record per line), use a `SplitText` processor before `PutKinesisStream` to split the FlowFile into individual FlowFiles, one per line.
### Step 2: Wire the failure sources to the publisher
- Connect the failure sources (the `parse.failure` relationship, and any transformation/error relationships) **directly** to this publisher --- no raw/structured branches are built for the stream route.
- Route the publisher's `failure`, `invalid` relationships to the [DLQ sink failure handling](/user-guide/data-integration/openflow/connectors/streaming/configuring-dead-letter-queue-handling#label-openflow-streaming-dlq-sink-failure). Use a **bounded** `failure` retry (for example, retry count 3) so transient stream issues recover but persistent failures still reach the parking-lot. Do **not** use an effectively-infinite retry (for example, 9999).
## Snowflake table as destination for DLQ messages
Identical to both connectors. See [Route B --- Snowflake table](/user-guide/data-integration/openflow/connectors/streaming/configuring-dead-letter-queue-handling#label-openflow-streaming-dlq-route-b) in the common guide.
## Troubleshooting
| Symptom |
Likely cause |
| DLQ publisher writes to the wrong stream/account |
`PutKinesisStream` reuses the `AWSCredentialsProviderControllerService` + **Region** of `ConsumeKinesis` --- a different account/region needs the appropriate credentials and region. |
For shared symptoms (raw branch, `structured_payload`, grants, parking-lot funnel), see the [common troubleshooting table](/user-guide/data-integration/openflow/connectors/streaming/configuring-dead-letter-queue-handling).
---
title: Openflow Connector for MySQL: Data mapping
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/mysql/data-mapping.md
section: Loading & Unloading Data
---
# %mysql%: Data mapping
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
- [About Openflow Connector for MySQL](/user-guide/data-integration/openflow/connectors/mysql/about)
- [Set up the Openflow Connector for MySQL](/user-guide/data-integration/openflow/connectors/mysql/setup)
This topic describes MySQL data types are mapped
to Snowflake data types.
## MySQL to Snowflake data type mapping
The following table shows how MySQL data types are mapped to Snowflake data types
when replicating data.
| MySQL type |
Snowflake type |
Notes |
| DECIMAL / NUMERIC |
NUMBER |
The maximum number of digits in DECIMAL format for MySQL is 65. For Snowflake, the maximum is 38. Precision is lost when exceeded. |
| INT / INTEGER |
INT |
|
| TINYINT / BOOL |
INT |
|
| SMALLINT |
INT |
|
| MEDIUMINT |
INT |
|
| BIGINT |
INT |
|
| YEAR |
INT |
|
| FLOAT |
FLOAT |
|
| DOUBLE |
FLOAT |
|
| VARCHAR |
TEXT |
|
| CHAR |
TEXT |
Trailing spaces aren't preserved. |
| TINYTEXT |
TEXT |
|
| TEXT |
TEXT |
|
| MEDIUMTEXT |
TEXT |
|
| LONGTEXT |
TEXT |
Supported by default up to 16 MB. |
| ENUM |
TEXT |
Stored as a string value. For example, for `ENUM('one', 'two')` the possible values are `'one'` and `'two'`. |
| SET |
TEXT |
Stored as a comma-separated string in column declaration order. For example, for `SET('one', 'two')` the possible values are `''`, `'one'`, `'two'`, and `'one,two'`. |
| BIT |
TEXT |
Represented as a hexadecimal string. For example: `'83060c183060c183'`. |
| DATE |
DATE |
|
| DATETIME |
TIMESTAMP_NTZ |
|
| TIMESTAMP |
TIMESTAMP_TZ |
Values are stored in UTC. |
| TIME |
TIME |
|
| BINARY |
BINARY |
|
| VARBINARY |
BINARY |
|
| TINYBLOB |
BINARY |
|
| BLOB |
BINARY |
|
| MEDIUMBLOB |
BINARY |
Supported by default up to 8 MB. |
| LONGBLOB |
BINARY |
Supported by default up to 8 MB. |
| JSON |
VARIANT |
Supported by default up to 16 MB. |
For types with default size limits (8 MB / 16 MB) in this table, it is possible to raise these limits. For details, see [Oversized values](/user-guide/data-integration/openflow/connectors/mysql/about#label-mysql-oversized-values).
Any MySQL data types not listed in this table are mapped to TEXT by default.
---
title: Openflow Connector for MySQL: Iceberg table destinations
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/mysql/iceberg.md
section: Loading & Unloading Data
---
# Openflow Connector for MySQL: Iceberg table destinations
Available to all accounts.
- [About Openflow Connector for MySQL](/user-guide/data-integration/openflow/connectors/mysql/about)
- [Set up the Openflow Connector for MySQL](/user-guide/data-integration/openflow/connectors/mysql/setup)
- [Openflow Connector for MySQL: Data mapping](/user-guide/data-integration/openflow/connectors/mysql/data-mapping)
- [Data types for Apache Iceberg™ tables](/user-guide/tables-iceberg-data-types)
- [Snowflake storage for Apache Iceberg™ tables](/user-guide/tables-iceberg-internal-storage)
- [CREATE EXTERNAL VOLUME](/sql-reference/sql/create-external-volume)
The Openflow Connector for MySQL supports writing to Snowflake-managed Apache %iceberg-tm% tables
as an opt-in destination format. Iceberg v2 and v3 are both supported. Setting **Table Storage Format** = `ICEBERG`
and choosing an **Iceberg Version** are the only connector-level changes required. The external volume,
catalog, and serialization policy are inherited from the Snowflake destination database defaults.
The Iceberg specification version is set via the **Iceberg Version** connector parameter, which
defaults to `3` for both Gen2 (Openflow UI wizard) and Gen1 (parameter context) connectors.
Storage can be either [Snowflake storage for Apache %iceberg-tm% tables](/user-guide/tables-iceberg-internal-storage)
(`EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'`) or an external volume in your cloud storage. When you use
Snowflake storage, no external cloud storage or IAM grants are required.
Existing connectors using standard tables aren't affected.
## Prerequisites
- **Openflow runtime**: An existing runtime to host the connector.
- **MySQL source configured for CDC**: Binary logging enabled (`log_bin = ON`,
`binlog_format = ROW`, `binlog_row_image = FULL`), a user with `REPLICATION SLAVE` and
`REPLICATION CLIENT` privileges, and a sufficiently long `binlog_expire_logs_seconds` for
snapshot reconciliation. For details, see
[Set up the Openflow Connector for MySQL](/user-guide/data-integration/openflow/connectors/mysql/setup).
- **External volume in your cloud storage**: An external volume configured for Iceberg storage,
with USAGE granted to the connector's Snowflake role. See
[CREATE EXTERNAL VOLUME](/sql-reference/sql/create-external-volume). Not required when using
Snowflake storage (`EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'`).
- **Snowflake destination database**: An existing database configured with Iceberg parameters
(next section).
## Step 1: Configure the Snowflake destination database
Set the Iceberg defaults on the destination database. The connector reads these defaults at runtime
for external volume and serialization policy. The Iceberg specification version is configured
per-connector via the **Iceberg Version** parameter (see Step 3), not solely via the database-level
`ICEBERG_VERSION_DEFAULT`.
### Option A: Snowflake storage
When you use Snowflake storage, Snowflake stores and manages the Iceberg table files for you.
No external cloud storage or IAM grants are required.
```sql
CREATE DATABASE
EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'
STORAGE_SERIALIZATION_POLICY = ;
```
To configure an existing database:
```sql
ALTER DATABASE SET
EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'
STORAGE_SERIALIZATION_POLICY = ;
```
### Option B: External volume in your cloud storage
If you need to keep table files in your own cloud storage, configure the database with your
external volume:
```sql
CREATE DATABASE
EXTERNAL_VOLUME = ''
STORAGE_SERIALIZATION_POLICY = ;
```
To configure an existing database:
```sql
ALTER DATABASE SET
EXTERNAL_VOLUME = ''
STORAGE_SERIALIZATION_POLICY = ;
```
| Parameter |
Required |
Notes |
| EXTERNAL_VOLUME |
Yes |
The external volume for Iceberg file storage. |
| ICEBERG_VERSION_DEFAULT |
No |
`2` or `3`. Legacy fallback for older connector flows where the **Iceberg Version** parameter is
unset. New connectors set the version via the connector parameter (Step 3) and do not require this
database setting.
|
| STORAGE_SERIALIZATION_POLICY |
Yes |
`COMPATIBLE` produces Parquet files readable by external engines. `OPTIMIZED` enables
Snowflake-specific query optimizations. Choose based on your data query needs. For more information,
see [STORAGE_SERIALIZATION_POLICY](/sql-reference/parameters#storage-serialization-policy).
|
`CATALOG = 'SNOWFLAKE'` is set automatically by the connector on each CREATE ICEBERG TABLE
statement. Don't set it at the database level.
The base location for each table is auto-derived using the
[flat layout](/user-guide/tables-iceberg-managing-external-volumes#label-tables-iceberg-snowflake-managed-flat-layout):
`STORAGE_BASE_URL/database/schema/table_name.randomId/[data | metadata]/`.
No user configuration is needed.
If using an external volume in your cloud storage (Option B), grant the connector's Snowflake role
USAGE on the external volume:
```sql
GRANT USAGE ON EXTERNAL VOLUME TO ROLE ;
```
This step is not required for Snowflake storage.
## Step 2: Set Table Storage Format in the connector's parameter context
Set the **Table Storage Format** parameter to `ICEBERG` in the connector's destination parameter context.
The default is `STANDARD`.
For the full connector creation and configuration workflow, see
[Set up the Openflow Connector for MySQL](/user-guide/data-integration/openflow/connectors/mysql/setup).
## Step 3: Set the Iceberg version
Set the **Iceberg Version** connector parameter to `2` or `3`. This controls the Iceberg specification
version used for type mapping (for example, JSON maps to `variant` on v3 vs `string` on v2)
and the `ICEBERG_VERSION=` clause in CREATE ICEBERG TABLE DDL.
- **Gen2 (Openflow UI wizard)**: **Iceberg Version** is a required field when **Table Storage Format** =
`ICEBERG`, defaulting to `3`. This setting is immutable after the connector configuration is first
applied.
- **Gen1 (parameter context)**: The **Iceberg Version** parameter defaults to `3`. Review and change
to `2` if needed before starting the connector. Do not change this value after ingestion begins.
## Step 4: Start and verify
Start the connector as usual. After the initial snapshot completes, verify the destination tables
are Iceberg:
```sql
-- Confirm the table is Iceberg
SELECT GET_DDL('TABLE', '..');
-- Confirm the Iceberg version on the database
SHOW PARAMETERS LIKE 'ICEBERG_VERSION_DEFAULT' IN DATABASE ;
```
## Known limitations
- **Tri-Secret Secure accounts and Snowflake storage**: Accounts with Tri-Secret Secure
(TSS) enabled may be unable to create new Snowflake-managed Iceberg tables that use
[Snowflake storage for Apache %iceberg-tm% tables](/user-guide/tables-iceberg-internal-storage).
For details, see [Encryption](/user-guide/tables-iceberg-internal-storage#encryption).
- **Incompatible type change.** When the source column type changes to a type that maps to a
different Iceberg type, the table is marked as failed and requires a resnapshot. See
[Type mapping reference](#type-mapping-reference) for the complete source-to-Iceberg type
mapping.
- **Parameter change within the same Iceberg type.** The connector doesn't recognize parameter
changes within the same Iceberg type (for example, changing `decimal(10,2)` to `decimal(20,2)`).
The column retains its current Iceberg type.
- **BIGINT to BIGINT UNSIGNED schema evolution not allowed**: Signed `BIGINT` maps to `long`
while `BIGINT UNSIGNED` maps to `decimal(20,0)`. Iceberg does not allow promotion from `long`
to `decimal`, so this schema change on the source will fail replication.
- **Do not change Table Storage Format or Iceberg Version after the connector starts**:
The connector's **Table Storage Format** and **Iceberg Version** parameter should not be modified
after ingestion begins. Gen2 connectors enforce this by making **Iceberg Version** immutable after
first apply. Mixing settings across destination tables is not supported. To switch, follow the
steps in [Switching table storage format or Iceberg version](#switching-table-storage-format-or-iceberg-version).
## Type mapping reference
The following table shows how MySQL types map to Snowflake standard and Iceberg destination types:
| MySQL type |
Snowflake (Standard) |
Iceberg v3 |
Iceberg v2 |
| TINYINT / SMALLINT / MEDIUMINT (signed or unsigned) |
INT |
`long` |
`long` |
| INT (signed) |
INT |
`long` |
`long` |
| INT UNSIGNED |
INT |
`long` |
`long` |
| BIGINT (signed) |
INT |
`long` |
`long` |
| BIGINT UNSIGNED |
INT |
`decimal(20,0)` |
`decimal(20,0)` |
| YEAR |
INT |
`long` |
`long` |
| FLOAT (signed/unsigned) |
FLOAT |
`double` |
`double` |
| DOUBLE (signed/unsigned) |
FLOAT |
`double` |
`double` |
| DECIMAL(P,S) (P ≤ 38) |
NUMBER(P,S) |
`decimal(P,S)` |
`decimal(P,S)` |
| DECIMAL(P,S) (P > 38) |
TEXT |
`string` |
`string` |
| BOOLEAN / BOOL |
INT |
`long` |
`long` |
| DATE |
DATE |
`date` |
`date` |
| TIME |
TIME |
`time` |
`time` |
| DATETIME |
TIMESTAMP_NTZ |
`timestamp` |
`timestamp` |
| TIMESTAMP |
TIMESTAMP_TZ |
`timestamptz` |
`timestamptz` |
| CHAR / VARCHAR / TEXT / TINYTEXT / MEDIUMTEXT / LONGTEXT |
TEXT |
`string` |
`string` |
| ENUM / SET |
TEXT |
`string` |
`string` |
| BIT |
TEXT |
`string` |
`string` |
| BINARY / VARBINARY / BLOB / TINYBLOB / MEDIUMBLOB / LONGBLOB |
BINARY |
`binary` |
`binary` |
| JSON |
VARIANT |
`variant` |
`string` |
| GEOMETRY family |
TEXT |
`string` |
`string` |
Source types not listed in the table are mapped to TEXT on standard tables and `string` on Iceberg
tables.
## Switching table storage format or Iceberg version
Switching between Standard and Iceberg, or between Iceberg v2 and v3, requires recreating the
connector. Follow these steps:
1. Stop the connector.
2. Delete the process group in Openflow.
3. Manually clean up the destination database (drop the replicated schemas/tables, or use a new
database).
4. Reimport the connector with the new **Table Storage Format** and select the target **Iceberg Version**
when configuring the connector.
This ensures all connector state is correctly cleaned up within Openflow. The new connector performs
a fresh snapshot into the destination.
## Upgrading an existing connector to use Iceberg Version pinning
Gen2 connector version `2026.7.21` and Gen1 connector version `0.53.0` introduce the
**Iceberg Version** parameter. If you are upgrading from an earlier connector version (for example,
Gen1 `0.50.0` to `0.53.0` or later), a new **Iceberg Version** field appears that you must configure
to match your existing destination tables.
1. Stop the connector.
2. [Upgrade the runtime](/user-guide/data-integration/openflow/manage#label-openflow-upgrading-a-runtime)
to version `2026.7.21` or later.
3. [Upgrade the connector](/user-guide/data-integration/openflow/manage#upgrade-a-connector)
in place (Gen2: to version `2026.7.21` or later; Gen1: to version `0.53.0` or later).
4. Set the **Iceberg Version** parameter to match your existing destination tables:
- **Gen2 (Openflow UI wizard)**: After upgrading, open the connector configuration wizard.
The **Destination details** step now includes a required **Iceberg Version** field, defaulting
to `3`. If your existing destination tables are Iceberg v2, change it to `2` before applying.
This choice is locked after first apply and cannot be changed later.
- **Gen1 (parameter context)**: The **Iceberg Version** parameter defaults to `3` after the flow
upgrade. If your existing destination tables are Iceberg v2, change it to `2` before starting
the connector.
5. Start the connector.
Selecting an **Iceberg Version** that doesn't match your existing destination tables can cause
type-mapping errors or DDL failures. Always verify the version of your existing tables before
choosing a value.
## References
- [CREATE EXTERNAL VOLUME](/sql-reference/sql/create-external-volume)
- [Data types for Apache Iceberg tables](/user-guide/tables-iceberg-data-types)
- [ALTER DATABASE](/sql-reference/sql/alter-database)
- [STORAGE_SERIALIZATION_POLICY](/sql-reference/parameters#storage-serialization-policy)
- [Set up the Openflow Connector for MySQL](/user-guide/data-integration/openflow/connectors/mysql/setup)
---
title: Openflow Connector for MySQL: Maintenance
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/mysql/maintenance.md
section: Loading & Unloading Data
---
# %mysql%: Maintenance
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
- [About Openflow](/user-guide/data-integration/openflow/about)
- [Manage Openflow](/user-guide/data-integration/openflow/manage)
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
- [Set up the Openflow Connector for MySQL](/user-guide/data-integration/openflow/connectors/mysql/setup)
- [Openflow Connector for MySQL: Data mapping](/user-guide/data-integration/openflow/connectors/mysql/data-mapping)
This topic describes important maintenance considerations and best practices for
maintaining the %mysql% such as reinstalling the connector or setting the starting binary log position for loading.
These operations are often used in conjunction with [Incremental replication without snapshots](/user-guide/data-integration/openflow/connectors/mysql/incremental-replication).
## Check the replication status of a table
Interim failures, such as connection errors or temporary source unavailability during a high-availability failover, do not prevent table replication. Replicated tables keep their current status and the connector retries on the next polling cycle. However, permanent failures, such as unsupported data types, prevent table replication.
To troubleshoot replication issues or verify that a table has been successfully removed from the replication flow, check the Table State Store:
1. In the Openflow runtime canvas, right-click a processor group and choose **Controller Services**. A table listing controller services displays.
2. Locate the row labeled **Table State Store**, click the **More** %sf-vertical-more-button% button on the right side of the row, and then choose **View State**.
A list of tables and their current states displays. Type in the search box to filter the list by table name. The possible states are:
- **NEW**: The table is scheduled for replication but replication hasn't started.
- **SNAPSHOT_REPLICATION**: The connector is copying existing data. This status displays until all records are stored in the destination table.
- **INCREMENTAL_REPLICATION**: The connector is actively replicating changes. This status displays after snapshot replication ends and continues to display indefinitely until a table is either removed from replication or replication fails.
- **FAILED**: Replication has permanently stopped due to an error.
The Openflow runtime canvas doesn't display table status changes — only the current table status. However, table status changes are recorded in logs when they occur. Look for the following log message:
```text
Replication state for table .. changed from to
```
If a permanent failure prevents table replication, remove the table from replication. After you address the problem that caused the failure, you can add the table back to replication. For more information, see [Restart table replication](#label-of-mysql-restart-table-replication).
## Restart table replication
This procedure re-snapshots the table in place. It requires Runtime Extensions version `2026.5.14.16` or later and connector version `0.49.0` or later. On earlier versions, re-snapshotting a table that already exists in Snowflake fails instead of reloading in place. Upgrade Runtime Extensions first, and then upgrade the connector flow before you use this procedure.
A table in a FAILED state (for example, due to a missing primary key or an unsupported schema change) does not restart automatically. If a table enters a FAILED state or you need to restart replication from scratch, use the following procedure to remove and re-add the table to replication.
If the failure was caused by an issue in the source table such as a missing primary key, resolve that issue in the source database before continuing.
1. Remove the table from replication, using one of the following methods:
- Add the table to the **Re-snapshot Table Exclusions** parameter to temporarily exclude it from replication. This approach is convenient when the table is matched by an **Included Table Regex** that you don't want to change.
- In the **Ingestion Parameters** context, either remove the table from **Included Table Names** or modify the **Included Table Regex** so the table is no longer matched.
2. Verify the table has been removed:
1. In the Openflow runtime canvas, right-click a processor group and choose **Controller Services**.
2. In the table listing controller services, locate the **Table State Store** row, click the three vertical dots on the right side of the row, then choose **View State**.
You must wait until the table's state is fully removed from this list before proceeding. Don't continue until this configuration change has completed.
3. Wait until all queues in the connector are empty before you re-add the table. When all FlowFiles have been processed, the **Queued** value on the connector's processor group becomes zero.
Don't re-add the table while change events that were captured before you removed it are still queued. When you re-add a table, the connector loads the new snapshot in append-only mode, so any leftover change event that merges into the table after the re-snapshot might create duplicate rows in the destination table.
4. Re-add the table by reversing the change you made in the first step: either remove the table from **Re-snapshot Table Exclusions**, or add it back to **Included Table Names** or **Included Table Regex**.
You do not need to drop the destination table first. The connector re-snapshots the table in place: it makes a zero-copy [clone](/sql-reference/sql/create-clone) of the current destination table to an archive table named `_ARCHIVE_`, clears the destination table, and then loads the fresh snapshot into the same destination table. Because the destination table object is preserved, dependent objects such as streams remain attached and continue to work.
The archive table retains a copy of the destination table's contents from immediately before the reload, as a safeguard. The connector does not read from or write to it again, so you can drop it at any time once the backup is no longer needed, typically after you confirm that the re-snapshot completed and the destination data is correct.
5. Verify the restart: Check the **Table State Store** using the instructions given previously. The state of the table should appear with the status NEW, then transition to SNAPSHOT_REPLICATION, and finally to INCREMENTAL_REPLICATION.
## Increase the oversized value limit
By default, the connector replicates individual values up to 16 MB and marks any table that contains a larger value as permanently failed. If your Snowflake account has the `ENABLE_OPENFLOW_CDC_MYSQL_SSV2` parameter set to `true`, the per-value limit can be raised from 16 MB to **128 MB**.
The 128 MB limit applies in two ways: it's both the maximum size of a single value and the maximum total size of a row. The connector adds metadata columns to every replicated row (`_SNOWFLAKE_UPDATED_AT`, `_SNOWFLAKE_INSERTED_AT`, `_SNOWFLAKE_DELETED`) that count toward the per-row limit, along with all other columns in the row. As a result, a single value can't reach the full 128 MB in practice when the row includes other data.
The increased limit doesn't apply equally to all column types.
In Snowflake, the maximum size for `BINARY` is **64 MB** (`BINARY(67108864)`), even when the increased size limits are enabled. Only `VARCHAR`, `VARIANT`, `ARRAY`, and `OBJECT` columns can hold up to 128 MB.
### Check whether the 128 MB limit is available
You may not be able to verify the `ENABLE_OPENFLOW_CDC_MYSQL_SSV2` parameter value by querying it. To check if it is enabled, see if the FlowFiles flow through the **Upload Rows via Snowpipe Streaming 2** processor (not through **Upload Rows via Snowpipe Streaming**).
### Configure the processors
Update the **Oversized Value Limit** property to `128 MB` on both of the following processors:
- **Fetch Table Rows** (in the **Snapshot Load** group)
- **Read MySQL CDC Stream** (in the **Incremental Load** group)
For each processor:
1. Locate the processor in the flow. On the connector canvas, you can use the search box in the top-right corner to find processors by name.
2. Right-click the processor and select **Configure**.
3. Open the **Properties** tab.
4. Set **Oversized Value Limit** to `128 MB`.
5. Apply the change.
For tables that are already being replicated and have destination columns narrower than `VARCHAR(134217728)` or `BINARY(67108864)`, see [](#label-of-mysql-migrate-oversized-value-tables).
### Migrate existing tables
The steps in [](#label-of-mysql-increase-oversized-value-limit) raise the limit for newly created destination tables. If a table is already being replicated and its destination column type is **not** `VARCHAR(134217728)` or `BINARY(67108864)`, but you now want to load values larger than the original 16 MB limit, you must manually widen the column type on **both** the journal and destination tables.
Before you migrate, check the current destination column type, because it can vary depending on when the snapshot replication was performed.
You must stop replication for the affected table before altering its journal or destination tables. Altering these tables while replication is active can corrupt in-flight data.
To migrate a table:
1. Stop replication for the affected table by stopping the topmost processors of the **Snapshot Load** and **Incremental Load** groups until all queues are empty. For the equivalent stop procedure, see the substeps of [](#label-mysql-reinstall-connector).
2. Widen the column on both the journal table and the destination table, according to the column type:
1. **VARCHAR columns**: a single `ALTER TABLE ... ALTER COLUMN ... SET DATA TYPE VARCHAR(134217728)` is sufficient on both the journal and destination tables.
2. **BINARY columns**: Snowflake doesn't allow widening `BINARY` in place. You must:
1. Add a new column of type `BINARY(67108864)` on both the journal and destination tables.
2. Copy data from the original column into the new column.
3. Drop the original column and rename the new column to the original name.
3. Restart replication by re-enabling the processors.
### Performance considerations
Raising the per-value limit increases the amount of data that the connector loads into memory and moves through the flow, which raises the load on both the runtime and the warehouse. Size the runtime and warehouse accordingly.
During both snapshot and incremental replication, the queue in front of the **Upload Rows via Snowpipe Streaming 2** processor can fill with FlowFiles and trigger back pressure, which consumes a large amount of runtime disk space. For larger tables, use a Large runtime to provide additional storage. For guidance on choosing a size, see [Runtime sizing](/user-guide/data-integration/openflow/connectors/cdc-runtime-sizing#label-openflow-cdc-runtime-sizing).
#### Snapshot replication
During snapshot replication, the product of `fetchSize * rowSize * concurrentQueries` can't exceed the heap size of the NiFi runtime, where:
- `fetchSize` is the number of rows fetched per query, set on the **Fetch Table Rows** processor (default: 100).
- `rowSize` is the size of a single row being fetched.
- `concurrentQueries` is the number of concurrent queries, set on the **Fetch Table Rows** processor (default: 2).
This memory requirement applies even when **Oversized Value Strategy** is set to **Set Null**, because the connector must load each oversized value into memory before it can replace the value with `NULL`.
If the source database contains many densely packed oversized values, consider excluding the affected column from replication before you start the snapshot. For example, if a column contains 1 GB values, loading even nine rows (~9 GB) can exhaust the heap and cause an out-of-memory error on a Medium runtime.
To speed up snapshot replication, you can increase the number of channels that the **Upload Rows via Snowpipe Streaming 2** processor uses. The number of channels is set by the processor's **Channel Group** property, which defaults to `${chunk.index:isEmpty():ifElse('1', ${chunk.index:mod(8)})}`.
To increase the number of channels:
1. Locate the **Upload Rows via Snowpipe Streaming 2** processor in the flow.
2. Stop the processor. You must stop the processor before you can change its properties.
3. Right-click the processor and select **Configure**.
4. Open the **Properties** tab.
5. In the **Channel Group** property, increase the value `8` in the expression. For example, change `8` to `16` to double the number of channels.
6. Apply the change.
7. Start the processor.
While a snapshot replication is in progress, only increase the number of channels. Decreasing the number of channels during an active snapshot can cause data loss.
#### Incremental replication
When the source produces frequent changes to rows that contain large values, you might need a Large warehouse. With smaller warehouses, replicating many 8 MB rows can cause an out-of-memory error. By contrast, replicating 128 MB rows with continuous merges completes without warehouse errors, because the connector streams the data file by file through the **Upload Rows via Snowpipe Streaming 2** processor and the merge processes it gradually.
Incremental replication is also subject to the MySQL transaction size limitation: a single transaction must fit into a binary log message of no more than 4 GB. For more information, see [Limitations](/user-guide/data-integration/openflow/connectors/mysql/about#limitations).
## Enable error logging on an existing schema
When you set the **Error Handling Strategy** parameter to **Log Errors and Continue**, the connector enables error logging automatically only on tables that it creates afterward. Tables that the connector created earlier don't capture rejected rows until you turn on error logging for them. For more information about the error-handling strategies, see [](/user-guide/data-integration/openflow/connectors/mysql/about#label-mysql-error-handling).
Because the connector stores journal tables in the same schema as the destination tables, you can turn on error logging for a whole destination schema at once. Run the following stored procedure once per destination schema. Replace `my_database` with your destination database and `my_schema` with the destination schema.
The schema name is passed as a quoted identifier (for example, `'"my_schema"'`) so it matches the exact, case-sensitive name that the connector created. For more information about how the connector names destination schemas, see [](/user-guide/data-integration/openflow/connectors/mysql/setup#label-of-mysql-destination-parameters).
```sql
USE DATABASE my_database;
WITH enable_error_logging AS PROCEDURE (schema_name STRING)
RETURNS STRING
LANGUAGE SQL
AS
$$
DECLARE
tables RESULTSET;
table_count NUMBER DEFAULT 0;
BEGIN
SHOW TABLES IN SCHEMA IDENTIFIER(:schema_name);
-- Assign AFTER SHOW TABLES so LAST_QUERY_ID() refers to that result
tables := (
SELECT "database_name", "schema_name", "name"
FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()))
WHERE "kind" = 'TABLE'
);
FOR t IN tables DO
-- Double-quote each identifier so names with special characters are handled safely
EXECUTE IMMEDIATE
'ALTER TABLE "' || REPLACE(t."database_name", '"', '""') || '".' ||
'"' || REPLACE(t."schema_name", '"', '""') || '".' ||
'"' || REPLACE(t."name", '"', '""') || '" ' ||
'SET ERROR_LOGGING = TRUE';
table_count := table_count + 1;
END FOR;
RETURN 'Enabled ERROR_LOGGING on ' || table_count || ' table(s) in schema ' || :schema_name;
END;
$$
CALL enable_error_logging('"my_schema"');
```
### Verify that error logging is enabled
To confirm that error logging is enabled on every table in a schema, run the following procedure. It reports how many tables have error logging enabled and how many don't.
```sql
USE DATABASE my_database;
WITH verify_error_logging AS PROCEDURE (schema_name STRING)
RETURNS STRING
LANGUAGE SQL
AS
$$
DECLARE
tables RESULTSET;
probe RESULTSET;
total_tables NUMBER DEFAULT 0;
logging_enabled NUMBER DEFAULT 0;
disabled_or_invisible NUMBER DEFAULT 0;
BEGIN
SHOW TABLES IN SCHEMA IDENTIFIER(:schema_name);
-- Assign AFTER SHOW TABLES so LAST_QUERY_ID() refers to that result
tables := (
SELECT "database_name", "schema_name", "name"
FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()))
WHERE "kind" = 'TABLE'
);
FOR t IN tables DO
total_tables := total_tables + 1;
-- Probe ERROR_TABLE(): it succeeds only when error logging is enabled and visible
BEGIN
probe := (
EXECUTE IMMEDIATE
'SELECT 1 FROM ERROR_TABLE(' ||
'"' || REPLACE(t."database_name", '"', '""') || '".' ||
'"' || REPLACE(t."schema_name", '"', '""') || '".' ||
'"' || REPLACE(t."name", '"', '""') || '"' ||
') LIMIT 1'
);
logging_enabled := logging_enabled + 1;
EXCEPTION
WHEN STATEMENT_ERROR THEN
disabled_or_invisible := disabled_or_invisible + 1;
END;
END FOR;
RETURN 'schema=' || :schema_name ||
', total_tables=' || total_tables ||
', error_logging_enabled=' || logging_enabled ||
', error_logging_disabled_or_not_visible=' || disabled_or_invisible;
END;
$$
CALL verify_error_logging('"my_schema"');
```
## Reclaim journal table storage
Journal tables hold every change to a replicated table. The connector never drops them, but it only
reads the latest journal for each replicated source table, using append-only streams on top of the
journals. To reclaim storage, you can:
- Truncate all journal tables at any time.
- Drop the journal tables related to source tables that were removed from replication.
- Drop all but the latest generation journal tables for actively replicated tables.
For example, if your connector is set to actively replicate source table `orders`, and you have
earlier removed table `customers` from replication, you may have the following journal tables. In
this case you can drop all of them *except* `orders_5678_2`.
```text
customers_1234_1
customers_1234_2
orders_5678_1
orders_5678_2
```
## Reinstall the connector
This section provides instructions on how to reinstall the connector, and continue replicating data for
the same tables without having to snapshot them again.
It covers situations where the new connector is installed in the same runtime, as well as those where it's moved to a new runtime.
For the connector to continue replicating from the same CDC stream position where it stopped before reinstallation,
the source database must retain the binary log long enough to cover the time since the prior connector was stopped
and the new connector is started.
Make sure the `binlog_expire_logs_seconds` parameter of the MySQL server is high enough, and keep the reinstallation time to a minimum.
The value of `binlog_expire_logs_seconds` needs to be longer than the expected time to reinstall the connector.
Typically 86400s, a day in seconds, is sufficient; however, longer times might be appropriate to ensure time to reinstall.
### Prerequisites
Review and note connector parameter context values.
If you're reinstalling the connector in the same runtime, you can reuse the existing context.
If the new instance is located in a different runtime, you must re-enter all parameters.
1. Finish processing all in-flight FlowFiles in the existing connector, then stop the connector.
1. Sign in to %sf-web-interface-link%.
2. In the navigation menu, select **Ingestion** %raa% **Openflow**.
3. Select **Launch Openflow**.
4. In the **Openflow** pane, select the **Runtimes** tab.
5. Select the runtime containing the connector.
6. Select the connector.
7. Stop the topmost processor **Set Tables for Replication** in the **Snapshot Load** group.
8. Stop the topmost processor **Read MySQL CDC Stream** in the **Incremental Load** group.
9. If you changed the value of the **Merge Task Schedule CRON** parameter, return it to `* * * * * ?`, otherwise queues won't be emptied until the next scheduled run.
Wait until all FlowFiles in the connector have been processed, and all queues are empty.
When all FlowFiles have been processed, the **Queued** value on the connector's processor group becomes zero.
If there are any items left in the original connector's queues, there may be data gaps when the new connector starts.
10. Stop all processors and controller services in the connector.
The existing connector can remain in the runtime and doesn't interfere with the new instance, as long as it remains stopped.
2. If you're moving the connector to a new runtime, download the flow definition from the existing connector so that you can recreate the connector with its current state instead of configuring it from scratch. Downloading a flow definition requires Openflow Runtime Server version 2026.6.4.18 or later.
1. Right-click the connector's process group, then select **Download flow definition**.
2. Select both of the following options, then download the flow definition:
- **Export with External Services**: includes the controller services that the connector references from parent process groups.
- **Export with Components State**: includes component state, such as binary log positions and incremental replication state, so that replication continues from where it left off.
3. Create the connector in the target runtime:
- If you downloaded the flow definition, import it into the new runtime. Importing the flow definition preserves the component state captured during the export, so the connector resumes incremental replication from its previous positions.
- Otherwise, create a new instance of the connector. If you're using the same runtime as the original connector, you can choose to keep the existing parameter contexts and reuse the settings.
4. If you're installing into a different runtime or you deleted the previous parameter contexts, enter the configuration settings into the new parameter contexts,
including the table names and patterns as described in [Set up the Openflow Connector for MySQL](/user-guide/data-integration/openflow/connectors/mysql/setup). A downloaded flow definition doesn't include sensitive values, such as passwords, so you must re-enter them.
5. Navigate to the `MySQL Ingestion Parameters` context, and set the following parameters:
- Set the `Ingestion Type` parameter to `incremental`. For more information on the concerns, see [](#label-mysql-incremental-replication).
- Set the `Starting Binlog Position` parameter to `Earliest`.
For more information and potential concerns, see [](#label-mysql-connector-start-restart-incremental-load-from-earliest-available-binary-log-position).
If you imported the flow definition with **Export with Components State** selected, the connector retains its previous binary log positions. In this case, leave `Starting Binlog Position` set to `Latest` to continue replication from where it stopped.
6. Start the new connector.
### Usage notes
The new connector uses the existing destination tables that were created by the original connector, but the connector creates new journal tables.
## Specify load from binary log position
The %mysql% connector allows you to select the starting position where MySQL binary logs are read.
By default, the connector reads from the latest available position. Alternatively, you can choose the earliest position available on the source instance.
Choosing to start from the earliest position is common when reinstalling the connector.
This allows the new instance to catch up and continue replicating existing tables without having to snapshot each again.
Note that switching a running connector from latest to earliest position causes the entire available binary log
to be re-read, re-processed, and re-applied to the destination table.
While the binary log is being re-read, the columns and data in affected destination tables
can become out of sync with their sources until all events have been re-processed and merged.
The following parameters that control snapshot loads are available in the `Ingestion Parameters` context:
| Parameter |
Description |
| Starting Binlog Position |
- `Latest` (default): CDC stream reading starts at the latest available position and continues from there.
- `Earliest`: Switches the incremental load to start, or restart reading from the earliest available
binary log position.
|
| Re-read Tables in State |
- `New` (default):
While re-reading the binary log, only those events will be processed
from new tables added to replication after the re-reading started.
Other events are discarded until the connector reaches the position just before re-reading started.
- `Any active`: Re-read and re-process events from any table currently in replication.
|
To determine whether the connector finished re-reading the binary log:
1. Navigate to the Openflow canvas.
2. Open the **Incremental Load** process group.
3. Right-click the topmost processor named **Read MySQL CDC Stream**, then select **View state**.
4. Compare the state entries:
- **binlog.position.rewind**: the latest position the processor read before re-reading of the binary log started.
- **binlog.position.dml**: the current latest position read by the processor. As long as this value is lower than the rewind value above, the processor is still re-reading the binary log.
### Usage notes
- After a running connector is switched to read from the earliest position, and starts running,
the process can't be reconfigured or canceled, and will continue until the currently-read position reaches the position from before it started.
- Switching to the earliest position on a running connector will, for any tables being re-processed,
finish their existing journals, and create new journal tables.
- If the binary log contains events from a previous table that was dropped
and re-created in the source database, re-reading the stream re-processes all events in the current destination.
The connector can't distinguish between a previous and current source table if they share the same name.
---
title: Openflow Connector for MySQL: Set up incremental replication without snapshots
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/mysql/incremental-replication.md
section: Loading & Unloading Data
---
# %mysql%: Set up incremental replication without snapshots
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
- [About Openflow](/user-guide/data-integration/openflow/about)
- [Manage Openflow](/user-guide/data-integration/openflow/manage)
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
- [Set up the Openflow Connector for MySQL](/user-guide/data-integration/openflow/connectors/mysql/setup)
- [Openflow Connector for MySQL: Data mapping](/user-guide/data-integration/openflow/connectors/mysql/data-mapping)
You can configure the %mysql% connector to immediately replicate incremental changes for newly added tables, bypassing snapshots. Use incremental load to continue replication without snapshotting every table again when you reinstall the connector over previously replicated data.
To enable incremental replication in a new connector instance:
1. Set up the connector as described in [Set up the Openflow Connector for MySQL](/user-guide/data-integration/openflow/connectors/mysql/setup).
2. In the `MySQL Ingestion Parameters` context, set the `Ingestion Type` parameter to `incremental`.
## Enable incremental replication without snapshots
To enable incremental replication on an existing connector:
1. sign in to %sf-web-interface-link%.
2. in the navigation menu, select **Ingestion** %raa% **Openflow**.
3. In the **Openflow** pane select the **Runtimes** tab.
4. Select the runtime containing the connector.
5. Select the connector.
6. In the `Ingestion Parameters` context, specify `Ingestion Type` = `incremental`.
7. Add new replication tables. These tables immediately switch to their incremental load.
To return to replicating tables with the snapshot load, change **Ingestion Type** from `incremental` to `full`.
# Usage notes
- Changing the value of **Ingestion Type** does not impact any tables that have begun replicating data.
Tables currently in the snapshot phase continue until the snapshot load is complete.
- While **Ingestion Type** is set to `incremental`, new tables added to the list of replicated tables bypass the snapshot phase.
This includes new tables added to the source database that match the `Included Table Regex` parameter.
Ensure that the ingestion type is set to `incremental` to bypass the snapshot phase.
Connectors should only remain in `incremental` mode as long as required as it bypasses snapshots.
Once customer needs for incremental updates have been satisfied the connector should be returned to `full` mode.
- For tables that bypass snapshot load, the connector creates a destination table in Snowflake,
by executing `CREATE TABLE IF NOT EXISTS`, only if no destination table already exists.
Tables going through the snapshot require that no destination table exist.
## Recover a table using incremental-only mode
If a table's snapshot completed successfully but incremental replication later failed, you don't need to remove the table and snapshot it again. Instead, you can recover the table by replaying the changes that are still available in the source binary logs (binlog) and merging them onto the existing destination table.
Incremental replication can fail for several reasons, for example:
- A record in the source database can't be read because it has an incorrect or unsupported format.
- A row exceeds the maximum supported size.
- A merge operation can't complete.
- A transient error persists through so many retries that the table enters the FAILED state.
To recover the table without a new snapshot, remove it from replication, switch the connector to incremental-only mode reading from the earliest available position, and add the table back. The connector reads all available changes from the oldest available binary log position, then replays and reapplies them to the destination table.
Before you recover the table, address the underlying cause of the failure. Otherwise, the connector encounters the same error again when it replays the changes. For example, raise the per-value limit (see [Increase the oversized value limit](/user-guide/data-integration/openflow/connectors/mysql/maintenance#label-of-mysql-increase-oversized-value-limit)) or fix the problematic record in the source database.
To recover the table:
1. Remove the table from replication. In the `Ingestion Parameters` context, remove the table from **Included Table Names**, or modify **Included Table Regex** so the table is no longer matched. Wait until the table's state is fully removed from the **Table State Store** controller service before you continue.
Don't drop the destination table. This procedure reuses the existing destination table and replays incremental changes onto it.
2. Stop the connector's process group so that you can change its configuration. On the connector canvas, right-click the connector's process group and select **Stop**.
3. In the `Ingestion Parameters` context, set the `Ingestion Type` parameter to `incremental`.
4. Set the `Starting Binlog Position` parameter to `Earliest`. The connector reads all available changes again from the oldest available binary log position, then replays and reapplies them to the destination table. For more information, see [Specify load from binary log position](/user-guide/data-integration/openflow/connectors/mysql/maintenance#label-mysql-connector-start-restart-incremental-load-from-earliest-available-binary-log-position).
Leave `Re-read Tables in State` at its default value, `New`, so that only the table you add back reads from the earliest position. Tables already in replication continue from their last positions.
5. Add the table back to replication by reversing the change you made in step 1.
6. Start the connector's process group. Right-click the connector's process group and select **Start**.
7. Wait until the table returns to incremental replication. In the **Table State Store** controller service state, the table transitions to INCREMENTAL_REPLICATION when recovery completes.
8. Revert the changes you made in steps 3 and 4: set `Ingestion Type` and `Starting Binlog Position` back to their previous values.
This procedure recovers only the changes still retained in the source binary logs. If the binary log retention period expired and some changes were purged, the recovered table can have gaps. In that case, you must take a new snapshot to fully resynchronize the table.
---
title: Openflow Connector for Oracle: Configure the Oracle database
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/oracle/setup-oracledb.md
section: Loading & Unloading Data
---
# %oracleofc%: Configure the Oracle database
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
The %oracleofc% is also subject to additional terms of service beyond the standard
connector terms of service. For more information, see the
[Openflow Connector for Oracle Addendum](https://www.snowflake.cn/en/legal/optional-offerings/offering-specific-terms/openflow-oracle-terms/).
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
- [About Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/about)
- [Openflow Connector for Oracle: Enable and manage commercial terms](/user-guide/data-integration/openflow/connectors/oracle/manage-commercial-terms)
- [Set up tasks for the Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/setup-tasks)
- [Openflow Connector for Oracle: Set up Snowflake](/user-guide/data-integration/openflow/connectors/oracle/setup-snowflake)
This topic describes how to set up the Oracle database for %oracleofc%.
Your Oracle database setup depends on your organization's security policies
and database architecture. For example, if tables reside in a Container
Database (CDB), a Pluggable Database (PDB), multiple PDBs, or a combination.
The steps provided in this topic are examples only. Modify them
as required for your environment.
As an Oracle database administrator, perform the following procedures on your source database:
1. [](#label-set-up-archived-redo-logs-retention-period)
2. [](#label-enable-xstream-and-supplemental-logging)
3. [](#label-create-xstream-administrator-user)
4. [](#label-granting-xstream-administrator-privileges)
5. [](#label-configure-xstream-server-connect-user)
6. [](#label-create-xstream-outbound-server)
7. [](#label-set-xstream-outbound-server-connect-user)
8. [](#label-set-xstream-outbound-server-capture-user)
9. (Optional) [](#label-oracle-standby-setup)
10. (Optional) [](#label-configure-ssl-connections)
The steps in this topic are written for a multi-tenant architecture with a Container
Database (CDB) and one or more Pluggable Databases (PDB). If your Oracle database uses a single-tenant
architecture, see [](#label-setup-xstream-single-tenant).
## Configure the retention period for archived redo logs
You must enable the `ARCHIVELOG` mode to ensure that change data is available for replication.
If you use AWS RDS for Oracle, you must also configure the retention period for archived redo logs.
Determine this period based on the volume of changes in the source database and your storage capacity.
To set the retention period, for example to 24 hours, follow the procedures in the following table:
| Database version |
Procedure |
| AWS RDS (Standard) |
Run the following:
```sql
begin
rdsadmin.rdsadmin_util.set_configuration(
name => 'archivelog retention hours',
value => '24');
end;
/
commit;
```
For more information, see
Retaining archived redo logs (https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.Oracle.CommonDBATasks.RetainRedoLogs.html).
|
| AWS RDS Custom |
1. Create a text file named `/opt/aws/rdscustomagent/config/redo_logs_custom_configuration.json`.
2. Add a JSON object to this file in the following format: `{"archivedLogRetentionHours" : "24"}`.
For more information, see
Restoring an RDS Custom for Oracle instance (https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-backup.pitr.html).
|
## Enable XStream and supplemental logging
XStream is included with Oracle Database and doesn't require any additional software.
To enable and configure XStream replication to capture and stream change data, run the following commands:
1. Enable XStream replication:
```sql
ALTER SYSTEM SET enable_goldengate_replication=TRUE SCOPE=BOTH;
ALTER SYSTEM SET STREAMS_POOL_SIZE = 2560M;
```
Snowflake recommends setting the streams pool size to 2.5 GB. This allocation covers the following:
- 1 GB for Capture
- 1 GB for Apply
- An additional 25% buffer
To enable supplemental logging to ensure that the redo logs capture the information required for logical
replication, run the following commands:
1. Confirm that the database is in ARCHIVELOG mode as shown in the following example:
```sql
SELECT LOG_MODE, FORCE_LOGGING FROM V$DATABASE;
```
Snowflake recommends forcing logging at the database or tablespace level.
2. Set the container to the root container and add supplemental logging to the database:
```sql
ALTER SESSION SET CONTAINER = CDB$ROOT;
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
```
Alternatively, you can enable logging only on specific tables as shown in the following example:
```sql
ALTER TABLE schema_name.table_name ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
```
## Create the XStream administrator user
An XStream administrator user is required to manage XStream components, including the
creation and alteration of outbound servers.
You can either create a dedicated user for this purpose or use an existing user,
provided that the necessary XStream administration privileges are granted (see the next section).
The following example details the setup of a dedicated XStream administrator user in the root container of a CDB.
The following example assumes that the database also has a PDB containing tables to be replicated.
Connect as SYSDBA or a user with appropriate privileges and run the following commands:
```sql
-- Switch to the root container.
ALTER SESSION SET CONTAINER = CDB$ROOT;
-- Create a tablespace for the XStream administrator user.
CREATE TABLESPACE xstream_adm_tbs DATAFILE '/path/to/your/cdb/xstream_adm_tbs.dbf'
SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
-- Switch to the Pluggable Database (PDB) and create a tablespace there.
ALTER SESSION SET CONTAINER = YOUR_PDB_NAME;
CREATE TABLESPACE xstream_adm_tbs DATAFILE '/path/to/your/pdb/xstream_adm_tbs.dbf'
SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
-- Switch back to the root container to create the common user.
ALTER SESSION SET CONTAINER = CDB$ROOT;
-- Create the XStream administrator user.
-- Note: the 'c##' prefix indicates a common user in a CDB environment, and CONTAINER=ALL grants privileges across all containers.
-- Replace "YOUR_XSTREAM_ADMIN_PASSWORD" with a strong, secure password.
CREATE USER c##xstreamadmin IDENTIFIED BY "YOUR_XSTREAM_ADMIN_PASSWORD"
DEFAULT TABLESPACE xstream_adm_tbs
QUOTA UNLIMITED ON xstream_adm_tbs
CONTAINER=ALL;
```
## Grant XStream administrator privileges
Connect as SYSDBA or a user with appropriate privileges and grant the required privileges
to the XStream administrator user.
1. Grant the CREATE SESSION privilege to the XStream administrator:
```sql
GRANT CREATE SESSION TO c##xstreamadmin CONTAINER=ALL;
```
2. Grant XStream capture privileges using one of the following commands, depending on your Oracle Database version:
| Database version |
Command |
| Oracle Database 21c and earlier |
Run the following:
```sql
BEGIN
DBMS_XSTREAM_AUTH.GRANT_ADMIN_PRIVILEGE(
grantee => 'c##xstreamadmin',
privilege_type => 'CAPTURE',
grant_select_privileges => TRUE,
container => 'ALL');
END;
/
```
|
| Oracle Database 23c and later |
Oracle Database 23c introduced a dedicated `XSTREAM_CAPTURE` system privilege. Run the following:
```sql
GRANT XSTREAM_CAPTURE TO c##xstreamadmin CONTAINER=ALL;
```
|
## Configure XStream server connect user
The Snowflake Openflow Connector uses a dedicated connect user to establish a connection to the XStream Outbound Server and receive change data.
This user requires specific privileges to facilitate replication:
- **Read from XStream Outbound Server**: The user must be able to access the change data stream from the configured XStream Outbound Server.
- **Select from Data Dictionary Views**: The connect user needs SELECT access to various data dictionary views.
This can be achieved by granting SELECT_CATALOG_ROLE or SELECT ANY DICTIONARY.
If granting SELECT ANY DICTIONARY isn't desired due to company policy, the user specifically needs SELECT access to the following views:
- ALL_USERS
- ALL_TABLES
- ALL_TAB_COLS
- ALL_CONS_COLUMNS
- ALL_CONSTRAINTS
- ALL_INDEXES
- ALL_IND_COLUMNS
- V$DATABASE
If you're replicating data from multiple PDBs, also grant SELECT access to:
- V$CONTAINERS
- V$PARAMETER
- CDB_TABLES
- CDB_USERS
`ALL_INDEXES` and `ALL_IND_COLUMNS` are required so the connector can detect
unique constraints and unique indexes as replication keys when a table has no
primary key. For more information on the selection algorithm, see
[](#label-oracle-replication-key-selection).
- **Select from Source Tables**: The user must have SELECT privileges on all tables that are intended for replication.
The following is an example of how to set up such a user in the root container of the CDB.
The example assumes that the database also has a PDB containing tables to be replicated.
```sql
-- Connect as SYSDBA or a user with appropriate privileges
-- Switch to the root container.
ALTER SESSION SET CONTAINER = CDB$ROOT;
-- Create the connect user.
-- Replace "YOUR_CAPTURE_USER_PASSWORD" with a strong, secure password.
CREATE USER c##connectuser IDENTIFIED BY "YOUR_CAPTURE_USER_PASSWORD"
CONTAINER=ALL;
-- Grant necessary privileges to the connect user.
-- You can choose to grant access to specific tables
-- instead of SELECT ANY TABLE for more granular control,
-- for example, GRANT SELECT ON schema.table TO c##connectuser;
GRANT CREATE SESSION, SELECT_CATALOG_ROLE, SELECT ANY TABLE TO c##connectuser CONTAINER=ALL;
```
If your database is multi-tenant and the connector is connected to a CDB to replicate
data from multiple PDBs, grant the connect user the additional privileges needed to
switch between containers and read data dictionary information across all of them:
```sql
ALTER USER c##connectuser SET CONTAINER_DATA = ALL CONTAINER = CURRENT;
GRANT SET CONTAINER TO c##connectuser CONTAINER=ALL;
```
If you granted SELECT on individual data dictionary views instead of
SELECT_CATALOG_ROLE or SELECT ANY DICTIONARY, also grant SELECT on
`V$CONTAINERS`, `V$PARAMETER`, `CDB_TABLES`, and `CDB_USERS`.
## Create XStream Outbound Server
The XStream Outbound Server captures changes from redo logs for consumption by the Openflow Connector. Define which schemas or tables to replicate.
For more information, see DBMS_XSTREAM_ADM.CREATE_OUTBOUND Documentation (https://docs.oracle.com/en/database/oracle/oracle-database/19/arpls/DBMS_XSTREAM_ADM.html#GUID-A602ED86-0F5A-4A27-92A0-55D5ADC0AF0D).
Create the outbound server on a writable database: the primary, a logical
standby, or a downstream capture database. You can't create it on an Active
Data Guard physical standby. For supported standby topologies, see
[](/user-guide/data-integration/openflow/connectors/oracle/about#label-oracle-standby-data-guard-support)
and [](#label-oracle-standby-setup).
Important considerations for replication scope:
- If a table is included in the XStream Outbound filtering rules command, it won't be replicated.
- A table or schema included here must also be defined in the connector parameters for it to be replicated.
You can include an entire schema in the server filtering rules and later, in the connector parameters,
specify only certain tables within that schema for replication.
The XStream Outbound Server can only be created from the root container. However,
starting with Oracle Database version 23ai, it can also be created on the PDB level.
To avoid a significant hit to your CPU and network, and to prevent your queues from being filled with irrelevant data, it's essential to use a granular approach. The best way to do this is with the DBMS_XSTREAM_ADM.ADD_TABLE_RULES procedure, which lets you choose only the specific tables
you need.
The following examples show how to set up the XStream Outbound Server based on different replication needs. In practice, when setting up your XStream Outbound Server on your production environment, you should be selective about what changes you capture. Capturing everything can have serious consequences for your database's performance and resource usage.
For information on how to configure XStream Outbound Server, see
Configuring XStream Out (https://docs.oracle.com/en/database/oracle/oracle-database/19/xstrm/configuring-xstream-out.html#GUID-A1C8430E-565B-4F66-8E00-495F283AAAFB).
**Example 1:** Capture all tables from all schemas in the root container and all PDBs
```sql
-- Connect as a user with XStream admin privileges to the root container.
-- Ensure serveroutput is enabled to see messages from the PL/SQL block.
SET SERVEROUTPUT ON;
DECLARE
tables DBMS_UTILITY.UNCL_ARRAY;
schemas DBMS_UTILITY.UNCL_ARRAY;
BEGIN
-- To replicate all tables in all schemas across all containers, set both to NULL.
tables(1) := NULL;
schemas(1) := NULL;
DBMS_XSTREAM_ADM.CREATE_OUTBOUND(
server_name => 'XOUT1',
table_names => tables,
schema_names => schemas,
include_ddl => TRUE
);
DBMS_OUTPUT.PUT_LINE('XStream Outbound Server created.');
EXCEPTION
WHEN OTHERS THEN
DBMS_OUTPUT.PUT_LINE('Error creating XStream Outbound Server: ' || SQLERRM);
RAISE;
END;
/
```
**Example 2:** Capture all tables from a single schema in a Pluggable Database (PDB)
```sql
-- Connect as a user with XStream admin privileges to the root container.
-- Ensure serveroutput is enabled to see messages from the PL/SQL block.
SET SERVEROUTPUT ON;
DECLARE
tables DBMS_UTILITY.UNCL_ARRAY;
schemas DBMS_UTILITY.UNCL_ARRAY;
BEGIN
-- To replicate all tables in a schema in the single PDB, set source_container_name.
tables(1) := NULL;
schemas(1) := 'schema_name';
DBMS_XSTREAM_ADM.CREATE_OUTBOUND(
server_name => 'XOUT1',
table_names => tables,
schema_names => schemas,
include_ddl => TRUE,
source_container_name => 'YOUR_PDB_NAME'
);
DBMS_OUTPUT.PUT_LINE('XStream Outbound Server created.');
EXCEPTION
WHEN OTHERS THEN
DBMS_OUTPUT.PUT_LINE('Error creating XStream Outbound Server: ' || SQLERRM);
RAISE;
END;
/
```
## Set up the XStream Outbound Server Connect User
Set the connect user on the XStream Outbound Server. This ensures that the previously created connect user is associated with the XStream Outbound Server (XOUT1), allowing it to receive change data.
The following example assumes that the connect user is c##connectuser.
```sql
BEGIN
DBMS_XSTREAM_ADM.ALTER_OUTBOUND(
server_name => 'XOUT1',
connect_user => 'c##connectuser');
END;
/
```
## Set up the XStream Outbound Server Capture User
If you want the data to be captured by the same user that created the server (the administrator), skip this section.
If you configured a separate capture user, configure the XStream Outbound Server to run
as this user. This ensures that the dedicated capture user is associated with the XStream Outbound Server (XOUT1), allowing that user to capture change data.
```sql
BEGIN
DBMS_XSTREAM_ADM.ALTER_OUTBOUND(
server_name => 'XOUT1',
capture_user => 'yourcaptureuser');
END;
/
```
## Set up XStream for single-tenant databases
The default architecture for Oracle 12c and later is a multi-tenant architecture with
a Container Database (CDB) and one or more Pluggable Databases (PDB). Oracle 11g uses
a single-tenant (non-CDB) architecture.
If your Oracle database uses a single-tenant architecture, note the following
differences in setting up XStream:
- Do not use `ALTER SESSION SET CONTAINER` commands. In a single-tenant
database, there is only one instance, so container switching doesn't apply.
- Create only one `xstream_adm_tbs` tablespace. Do not create a second
tablespace in a PDB.
- Do not use the `C##` prefix on user names. For example, create
`xstreamadmin` instead of `c##xstreamadmin` and `connectuser` instead
of `c##connectuser`. The `C##` prefix is required only in multi-tenant
environments.
- Do not include `CONTAINER=ALL` or `container => 'ALL'` in any commands.
These clauses grant privileges across multiple containers and don't apply
in a single-tenant database.
## Data Guard or standby capture (optional)
By default, the procedures in this topic create the XStream outbound server on
the primary (source) database. If you want to keep replication load off the primary, review
[](/user-guide/data-integration/openflow/connectors/oracle/about#label-oracle-standby-data-guard-support)
and choose a supported topology before you create the outbound server.
### Logical standby requirements
On a logical standby, create the XStream outbound server and connect the
connector to that standby the same way you would for a primary. Before you
create the outbound server or start the connector, set Database Guard to
`STANDBY`.
Logical standbys default to Database Guard `ALL`, which blocks the XStream
client from reading the outbound server and raises
`ORA-16224: Database Guard is enabled`.
1. Check the current guard status:
```sql
SELECT guard_status FROM v$database;
```
2. If the result is `ALL`, set Database Guard to `STANDBY`:
```sql
ALTER DATABASE GUARD STANDBY;
```
Then complete the XStream setup procedures in this topic on the logical
standby, and point the connector's Oracle connection URL and XStream Out Server
URL at that standby.
### Active Data Guard requirements for snapshot load
You can run the connector's snapshot load against an Active Data Guard physical
standby to avoid reading application tables on the primary. Because the standby
is read-only:
- Set **Snapshot Fetching Strategy** to `SEQUENTIAL_BY_PRIMARY_KEY` in the
connector parameters. `CONCURRENT_BY_ROWID` requires creating a parallel
task, which isn't allowed on a read-only standby.
- Don't create the XStream outbound server on the Active Data Guard standby.
For CDC, create it on the primary, a logical standby, or a downstream
capture database instead.
Point the connector's Oracle connection URL at the Active Data Guard standby
for snapshot reads. Point the XStream Out Server URL at the writable database
that hosts the outbound server (for example, the primary or a downstream
capture database).
### Downstream capture for CDC
To run CDC off the primary, configure Oracle downstream capture so that a
separate database receives redo from the primary, then create the XStream
outbound server on that downstream database by using the procedures in this
topic.
Downstream capture provides redo only. It can't serve snapshot queries. If you
also need an initial snapshot without reading the primary, combine Active Data
Guard for the snapshot with downstream capture for CDC, or use a logical
standby for both stages.
For Oracle's downstream capture concepts and setup, see
XStream Out Concepts (https://docs.oracle.com/en/database/oracle/oracle-database/19/xstrm/xstream-out-concepts.html)
in the Oracle documentation.
## Configure SSL connections (optional)
The %oracleofc% supports encrypted SSL connections to the Oracle database using the TCPS
(TCP with SSL) protocol. When SSL is enabled, both the database connection and the XStream connection use encrypted communication.
To use SSL, you must:
1. [](#label-enable-tcps-on-oracle-database)
2. [](#label-create-client-wallet)
### Enable TCPS on the Oracle database
You must configure the Oracle database to accept connections using the TCPS protocol.
Follow the procedure for your database environment.
#### On-premises / OCI
1. Create an SSL server wallet with the server certificate.
2. Configure the `listener.ora` to include a TCPS endpoint (default port 2484).
3. Configure the `sqlnet.ora` to reference the server wallet.
4. Restart the listener.
For more information, see
Configuring Transport Layer Security Encryption (https://docs.oracle.com/en/database/oracle/oracle-database/23/dbseg/configuring-transport-layer-security-encryption.html).
#### AWS RDS (Standard)
1. Add the Oracle SSL option to the option group associated with the DB instance.
2. Specify the SSL port (for example, 2484).
For more information, see
Oracle Secure Sockets Layer (https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.Oracle.Options.SSL.html).
### Create a client wallet
After TCPS is enabled on the database, create an Oracle auto-login wallet (`cwallet.sso`)
containing the server's trusted certificate. This wallet is provided to the connector so
that it can verify the server during the SSL handshake.
1. Export the server certificate from the Oracle database server as a PEM file.
2. Use the Oracle `orapki` utility to create a client wallet and import the server certificate:
```bash
orapki wallet create -wallet /path/to/client/wallet -pwd -auto_login
orapki wallet add -wallet /path/to/client/wallet -pwd \
-trusted_cert -cert /path/to/server-cert.pem
```
3. Copy the generated `cwallet.sso` file to a location accessible by the Openflow runtime.
For AWS RDS, download the root certificate from AWS instead of exporting it from the
database server. For more information, see
Connecting to an RDS for Oracle DB instance using SSL (https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.Oracle.Options.SSL.Connecting.html).
For more information, see
Using the orapki Utility to Manage PKI Elements (https://docs.oracle.com/en/database/oracle/oracle-database/23/dbseg/using-the-orapki-utility-to-manage-pki-elements.html).
## Next steps
[Configure the connector](/user-guide/data-integration/openflow/connectors/oracle/setup-connector).
---
title: Openflow Connector for Oracle: Data mapping
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/oracle/data-mapping.md
section: Loading & Unloading Data
---
# %oracleofc%: Data mapping
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
The %oracleofc% is also subject to additional terms of service beyond the standard
connector terms of service. For more information, see the
[Openflow Connector for Oracle Addendum](https://www.snowflake.cn/en/legal/optional-offerings/offering-specific-terms/openflow-oracle-terms/).
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
- [About Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/about)
- [Openflow Connector for Oracle: Enable and manage commercial terms](/user-guide/data-integration/openflow/connectors/oracle/manage-commercial-terms)
- [Set up tasks for the Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/setup-tasks)
This topic describes how Oracle data types are mapped to Snowflake data types when replicating data.
## Oracle to Snowflake data type mapping
The following table shows how Oracle data types are mapped to Snowflake data types
when replicating data.
| Oracle type |
Snowflake type |
Notes |
| NUMBER |
NUMBER |
If precision is undefined, mapped to NUMBER(38, 19). If precision or scale exceeds Snowflake limitations (precision > 38 or scale > 37), the value is stored as TEXT. |
| FLOAT |
FLOAT |
|
| BINARY_FLOAT |
FLOAT |
|
| BINARY_DOUBLE |
FLOAT |
|
| CHAR |
TEXT |
|
| VARCHAR2 |
TEXT |
|
| NCHAR |
TEXT |
|
| NVARCHAR2 |
TEXT |
|
| CLOB |
TEXT |
Supported by default up to 16 MB. |
| NCLOB |
TEXT |
Supported by default up to 16 MB. |
| LONG |
TEXT |
Supported by default up to 16 MB. |
| DATE |
TIMESTAMP_NTZ |
|
| TIMESTAMP |
TIMESTAMP_NTZ |
|
| TIMESTAMP WITH TIME ZONE |
TIMESTAMP_TZ |
|
| TIMESTAMP WITH LOCAL TIME ZONE |
TIMESTAMP_LTZ |
|
| INTERVAL |
TEXT |
|
| INTERVAL YEAR TO MONTH |
TEXT |
|
| INTERVAL DAY TO SECOND |
TEXT |
|
| RAW |
BINARY |
|
| LONG RAW |
BINARY |
Supported by default up to 8 MB. |
| BLOB |
BINARY |
Supported by default up to 8 MB. |
| BOOLEAN |
BOOLEAN |
|
| JSON |
VARIANT |
Supported by default up to 16 MB. |
| XMLTYPE |
TEXT |
Supported by default up to 16 MB. |
For types with default size limits (8 MB / 16 MB) in this table, it is possible to raise these limits. For details, see [Oversized values](/user-guide/data-integration/openflow/connectors/oracle/about#label-oracle-oversized-values).
Any Oracle data types not listed in this table are mapped to TEXT by default.
## Next steps
Review [Set up tasks for the Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/setup-tasks) to set up the connector.
---
title: Openflow Connector for Oracle: Enable and manage commercial terms
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/oracle/manage-commercial-terms.md
section: Loading & Unloading Data
---
# %oracleofc%: Enable and manage commercial terms
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
The %oracleofc% is also subject to additional terms of service beyond the standard
connector terms of service. For more information, see the
[Openflow Connector for Oracle Addendum](https://www.snowflake.cn/en/legal/optional-offerings/offering-specific-terms/openflow-oracle-terms/).
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
- [About Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/about)
- [Set up tasks for the Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/setup-tasks)
- [Install and configure the Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/setup-connector)
- [Openflow Connector for Oracle: Maintenance](/user-guide/data-integration/openflow/connectors/oracle/maintenance)
This topic describes how to enable the %oracleofc% in the list of available connectors and manage
the licensing lifecycle.
This task must be performed by the organization administrator (ORGADMIN).
Setting up the %oracleofc% is a two-stage process. First, enable Oracle XStream services to make
the connector available for installation. Then, finalize the license configuration after
the connector detects your source database inventory.
## Part 1: Enable service (pre-installation)
By default, the %oracleofc% isn't displayed in the list of available connectors. You must accept the
[Openflow Connector for Oracle Addendum](https://www.snowflake.cn/en/legal/optional-offerings/offering-specific-terms/openflow-oracle-terms/)
terms to make it available for installation. This is required for all license models.
1. Sign in to %sf-web-interface-link%.
2. In the navigation menu, select **Admin** %raa% **Terms**.
3. Locate the item **Oracle Connector Terms** in the list.
4. Select **Review & Enable**.
After you complete these steps, the following changes take effect:
- The %oracleofc% listing becomes visible in the list of available connectors.
- A new **Openflow for Oracle** tab appears in the **Admin** %raa% **Terms** page.
## Part 2: License setup and lifecycle
Complete the steps for the license model you selected during configuration:
- [Option A: Embedded license for 36-month commitment (Snowflake-provided)](#label-oracle-embedded-license-setup-36)
- [Option B: Embedded license for 12-month commitment (Snowflake-provided)](#label-oracle-embedded-license-setup-12)
- [Option C: Independent license / BYOL](#label-oracle-byol-license-setup)
### Option A: Embedded license for 36-month commitment (Snowflake-provided)
For this licensing model, you must activate the trial to enable the connector.
Even if you install the connector, data replication doesn't start until this step is complete.
#### Step 1: Start the trial (prerequisite)
To start the trial:
1. Sign in to %sf-web-interface-link%.
2. In the navigation menu, select **Admin** %raa% **Terms**.
3. Select the **Openflow for Oracle** tab.
4. Locate the **Trial Status** card (status: "Ready to Activate").
5. Select **Start Trial**.
6. Accept the terms to start the 60-day trial period.
This action enables the captureChangeOracle processor, allowing it to connect to
your database.
#### Step 2: Configure connector
After starting the trial, install and configure the connector. For more information,
see [Configure the connector](/user-guide/data-integration/openflow/connectors/oracle/setup-connector).
After the connector successfully connects to the source database, a subscription is
automatically created and displayed in the **Openflow for Oracle** dashboard.
#### Step 3: Verify inventory
1. Sign in to %sf-web-interface-link%.
2. In the navigation menu, select **Admin** %raa% **Terms**.
3. Select the **Openflow for Oracle** tab.
4. Review the **Subscription Inventory** section.
5. Verify that the CPU core count matches your physical source database hardware.
6. If the core count is incorrect, update the runtime configuration.
#### Step 4: Lifecycle management
For more information about the licensing models and terms, see
[Licensing models and critical constraints](#label-oracle-licensing-models).
The following table describes the actions available at each stage of the embedded
license lifecycle.
| Stage |
Action |
Result |
| Trial period (Day 1 to 60) |
Select **Cancel Trial** in the **Openflow for Oracle** dashboard before Day 60. |
Oracle XStream services stop. No charges are incurred. |
| 36-month commitment (Day 61+) |
No action required. If the trial isn't canceled, the non-cancelable 36-month term begins automatically on Day 61. |
The license can't be canceled during this period. If your Snowflake agreement is terminated, the full remaining balance is due immediately. |
| Post-term S&M renewal (after month 36) |
The license fee drops to $0. The Support & Maintenance (S&M) fee auto-renews in 12-month increments, billed monthly. You may opt out of S&M renewal in the **Openflow for Oracle** dashboard. |
If you opt out and S&M coverage expires, the connector is permanently locked. To resume, you must purchase a new Embedded License, which resets the 36-month commitment. |
### Option B: Embedded license for 12-month commitment (Snowflake-provided)
For this licensing model, you must activate the trial to enable the connector.
Even if you install the connector, data replication doesn't start until this step is complete.
#### Step 1: Start the trial (prerequisite)
To start the trial:
1. Sign in to %sf-web-interface-link%.
2. In the navigation menu, select **Admin** %raa% **Terms**.
3. Select the **Openflow for Oracle** tab.
4. Locate the **Trial Status** card (status: "Ready to Activate").
5. Select **Start Trial**.
6. Accept the terms to start the 60-day trial period.
This action enables the captureChangeOracle processor, allowing it to connect to
your database.
#### Step 2: Configure connector
After starting the trial, install and configure the connector. For more information,
see [Configure the connector](/user-guide/data-integration/openflow/connectors/oracle/setup-connector).
After the connector successfully connects to the source database, a subscription is
automatically created and displayed in the **Openflow for Oracle** dashboard.
#### Step 3: Verify inventory
1. Sign in to %sf-web-interface-link%.
2. In the navigation menu, select **Admin** %raa% **Terms**.
3. Select the **Openflow for Oracle** tab.
4. Review the **Subscription Inventory** section.
5. Verify that the CPU core count matches your physical source database hardware.
6. If the core count is incorrect, update the runtime configuration.
#### Step 4: Lifecycle management
For more information about the licensing models and terms, see
[Licensing models and critical constraints](#label-oracle-licensing-models).
The following table describes the actions available at each stage of the embedded
license lifecycle.
| Stage |
Action |
Result |
| Trial period (Day 1 to 60) |
Select **Cancel Trial** in the **Openflow for Oracle** dashboard before Day 60. |
Oracle XStream services stop. No charges are incurred. |
| 12-month commitment (Day 61+) |
No action required. If the trial isn't canceled, the non-cancelable 12-month term begins automatically on Day 61, with license fees paid upfront in full. |
The license can't be canceled during this period. If your Snowflake agreement is terminated, the full remaining balance is due immediately. |
| Post-term S&M renewal (after month 12) |
The license fee drops to $0. The Support & Maintenance (S&M) fee auto-renews in 12-month increments, billed annually. You may opt out of S&M renewal in the **Openflow for Oracle** dashboard. |
If you opt out and S&M coverage expires, the connector is permanently locked. To resume, you must purchase a new Embedded License, which resets the 12-month commitment. |
### Option C: Independent license / BYOL
If you are using the independent license (Bring Your Own License), no prior trial activation
is required.
#### Step 1: Configure the connector
To set up the connector with the independent/BYOL license, follow the steps in
[Configure the connector](/user-guide/data-integration/openflow/connectors/oracle/setup-connector).
#### Step 2: Verify inventory (recommended)
Verify that Snowflake has correctly identified your database inventory.
1. Sign in to %sf-web-interface-link%.
2. In the navigation menu, select **Admin** %raa% **Terms**.
3. Select the **Openflow for Oracle** tab.
4. Review the database inventory details.
The **Start Trial** button doesn't appear for this license model, and the
Embedded License commitment-period rules don't apply. You are responsible for
maintaining a valid Oracle license that includes XStream entitlements.
---
title: Openflow Connector for Oracle: Iceberg table destinations
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/oracle/iceberg.md
section: Loading & Unloading Data
---
# Openflow Connector for Oracle: Iceberg table destinations
Available to all accounts.
- [About Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/about)
- [Set up tasks for the Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/setup-tasks)
- [Openflow Connector for Oracle: Data mapping](/user-guide/data-integration/openflow/connectors/oracle/data-mapping)
- [Data types for Apache Iceberg™ tables](/user-guide/tables-iceberg-data-types)
- [Snowflake storage for Apache Iceberg™ tables](/user-guide/tables-iceberg-internal-storage)
- [CREATE EXTERNAL VOLUME](/sql-reference/sql/create-external-volume)
The Openflow Connector for Oracle (multi-database) supports writing to Snowflake-managed Apache
%iceberg-tm% tables as an opt-in destination format. Iceberg v2 and v3 are both supported. Setting
**Table Storage Format** = `ICEBERG` and choosing an **Iceberg Version** are the only connector-level
changes required. The external volume, catalog, and serialization policy are inherited from the
Snowflake destination database defaults. The Iceberg specification version is set via the
**Iceberg Version** connector parameter, which defaults to `3`.
Storage can be either [Snowflake storage for Apache %iceberg-tm% tables](/user-guide/tables-iceberg-internal-storage)
(`EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'`) or an external volume in your cloud storage. When you use
Snowflake storage, no external cloud storage or IAM grants are required.
Existing connectors using standard tables aren't affected.
## Prerequisites
- **Openflow runtime**: An existing runtime to host the connector.
- **Oracle source configured for CDC**: Archive logging enabled (`ARCHIVELOG` mode), supplemental
logging configured, and a LogMiner or XStream user with the required privileges. For details, see
[Set up the Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/setup-tasks).
- **Snowpipe Streaming v2**: The Oracle connector must have Snowpipe Streaming v2 based writes
enabled. This is required for Iceberg table destinations.
- **External volume in your cloud storage**: An external volume configured for Iceberg storage,
with USAGE granted to the connector's Snowflake role. See
[CREATE EXTERNAL VOLUME](/sql-reference/sql/create-external-volume). Not required when using
Snowflake storage (`EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'`).
- **Snowflake destination database**: An existing database configured with Iceberg parameters
(next section).
## Step 1: Configure the Snowflake destination database
Set the Iceberg defaults on the destination database. The connector reads these defaults at runtime
for external volume and serialization policy. The Iceberg specification version is configured
per-connector via the **Iceberg Version** parameter (see Step 3), not solely via the database-level
`ICEBERG_VERSION_DEFAULT`.
### Option A: Snowflake storage
When you use Snowflake storage, Snowflake stores and manages the Iceberg table files for you.
No external cloud storage or IAM grants are required.
```sql
CREATE DATABASE
EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'
STORAGE_SERIALIZATION_POLICY = ;
```
To configure an existing database:
```sql
ALTER DATABASE SET
EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'
STORAGE_SERIALIZATION_POLICY = ;
```
### Option B: External volume in your cloud storage
If you need to keep table files in your own cloud storage, configure the database with your
external volume:
```sql
CREATE DATABASE
EXTERNAL_VOLUME = ''
STORAGE_SERIALIZATION_POLICY = ;
```
To configure an existing database:
```sql
ALTER DATABASE SET
EXTERNAL_VOLUME = ''
STORAGE_SERIALIZATION_POLICY = ;
```
| Parameter |
Required |
Notes |
| EXTERNAL_VOLUME |
Yes |
The external volume for Iceberg file storage. |
| ICEBERG_VERSION_DEFAULT |
No |
`2` or `3`. Legacy fallback for older connector flows where the **Iceberg Version** parameter is
unset. New connectors set the version via the connector parameter (Step 3) and do not require this
database setting.
|
| STORAGE_SERIALIZATION_POLICY |
Yes |
`COMPATIBLE` produces Parquet files readable by external engines. `OPTIMIZED` enables
Snowflake-specific query optimizations. Choose based on your data query needs. For more information,
see [STORAGE_SERIALIZATION_POLICY](/sql-reference/parameters#storage-serialization-policy).
|
`CATALOG = 'SNOWFLAKE'` is set automatically by the connector on each CREATE ICEBERG TABLE
statement. Don't set it at the database level.
The base location for each table is auto-derived using the
[flat layout](/user-guide/tables-iceberg-managing-external-volumes#label-tables-iceberg-snowflake-managed-flat-layout):
`STORAGE_BASE_URL/database/schema/table_name.randomId/[data | metadata]/`.
No user configuration is needed.
If using an external volume in your cloud storage (Option B), grant the connector's Snowflake role
USAGE on the external volume:
```sql
GRANT USAGE ON EXTERNAL VOLUME TO ROLE ;
```
This step is not required for Snowflake storage.
## Step 2: Set Table Storage Format in the connector's parameter context
Set the **Table Storage Format** parameter to `ICEBERG` in the connector's destination parameter context.
The default is `STANDARD`.
For the full connector creation and configuration workflow, see
[Set up the Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/setup-tasks).
## Step 3: Set the Iceberg version
Set the **Iceberg Version** connector parameter to `2` or `3`. This controls the Iceberg specification
version used for type mapping (for example, JSON maps to `variant` on v3 versus `string` on v2)
and the `ICEBERG_VERSION=` clause in CREATE ICEBERG TABLE DDL.
The **Iceberg Version** parameter defaults to `3`. Review and change to `2` if needed before starting
the connector. Do not change this value after ingestion begins.
## Step 4: Start and verify
Start the connector as usual. After the initial snapshot completes, verify the destination tables
are Iceberg:
```sql
-- Confirm the table is Iceberg
SELECT GET_DDL('TABLE', '..');
-- Confirm the Iceberg version on the database
SHOW PARAMETERS LIKE 'ICEBERG_VERSION_DEFAULT' IN DATABASE ;
```
## Known limitations
- **Tri-Secret Secure accounts and Snowflake storage**: Accounts with Tri-Secret Secure
(TSS) enabled may be unable to create new Snowflake-managed Iceberg tables that use
[Snowflake storage for Apache %iceberg-tm% tables](/user-guide/tables-iceberg-internal-storage).
For details, see [Encryption](/user-guide/tables-iceberg-internal-storage#encryption).
- **Incompatible type change:** When the source column type changes to a type that maps to a
different Iceberg type, the table is marked as failed and requires a resnapshot. See
[Type mapping reference](#type-mapping-reference) for the complete source-to-Iceberg type
mapping.
- **Parameter change within the same Iceberg type:** The connector doesn't recognize parameter
changes within the same Iceberg type (for example, changing `decimal(10,2)` to `decimal(20,2)`).
The column retains its current Iceberg type.
- **Nanosecond timestamp range restriction on v3**: When a TIMESTAMP(7..9) column maps to
`timestamp_ns` or `timestamptz_ns` on v3, the representable date range narrows to
1677-09-21 through 2262-04-11. Values outside this range are rejected at insert time.
- **TIMESTAMP WITH TIME ZONE offset collapsed to UTC**: Iceberg has no offset-preserving
timestamp type. The original timezone offset or region name is lost; only the UTC instant is
stored.
- **Source TIMESTAMP precision widening not supported on v3**: If a source column's precision
increases (for example, TIMESTAMP(6) altered to TIMESTAMP(7)), the Iceberg column type cannot
be promoted from `timestamp` to `timestamp_ns`. The connector was created based on the original
precision.
- **Do not change Table Storage Format or Iceberg Version after the connector starts**:
The connector's **Table Storage Format** and **Iceberg Version** parameter should not be modified
after ingestion begins. Mixing settings across destination tables is not supported. To switch,
follow the steps in [Switching table storage format or Iceberg version](#switching-table-storage-format-or-iceberg-version).
## Type mapping reference
The following table shows how Oracle types map to Snowflake standard and Iceberg destination types:
| Oracle type |
Snowflake (Standard) |
Iceberg v3 |
Iceberg v2 |
| NUMBER(P,S) (P ≤ 38) |
NUMBER(P,S) |
`decimal(P,S)` |
`decimal(P,S)` |
| NUMBER (no precision) |
NUMBER(38,19) |
`decimal(38,19)` |
`decimal(38,19)` |
| INTEGER / SMALLINT / INT |
NUMBER(38,0) |
`decimal(38,0)` |
`decimal(38,0)` |
| FLOAT(P) |
FLOAT |
`double` |
`double` |
| BINARY_FLOAT |
FLOAT |
`double` |
`double` |
| BINARY_DOUBLE |
FLOAT |
`double` |
`double` |
| BOOLEAN (Oracle 23ai+) |
BOOLEAN |
`boolean` |
`boolean` |
| DATE |
TIMESTAMP_NTZ |
`timestamp` |
`timestamp` |
| TIMESTAMP(0..6) |
TIMESTAMP_NTZ |
`timestamp` |
`timestamp` |
| TIMESTAMP(7..9) |
TIMESTAMP_NTZ |
`timestamp_ns` |
`timestamp` (truncated) |
| TIMESTAMP(0..6) WITH TIME ZONE |
TIMESTAMP_TZ |
`timestamptz` |
`timestamptz` |
| TIMESTAMP(7..9) WITH TIME ZONE |
TIMESTAMP_TZ |
`timestamptz_ns` |
`timestamptz` (truncated) |
| TIMESTAMP WITH LOCAL TIME ZONE (0..6) |
TIMESTAMP_LTZ |
`timestamptz` |
`timestamptz` |
| TIMESTAMP WITH LOCAL TIME ZONE (7..9) |
TIMESTAMP_LTZ |
`timestamptz_ns` |
`timestamptz` (truncated) |
| INTERVAL YEAR TO MONTH |
TEXT |
`string` |
`string` |
| INTERVAL DAY TO SECOND |
TEXT |
`string` |
`string` |
| CHAR / NCHAR / VARCHAR2 / NVARCHAR2 |
TEXT |
`string` |
`string` |
| CLOB / NCLOB / LONG |
TEXT |
`string` |
`string` |
| RAW |
BINARY |
`binary` |
`binary` |
| BLOB / LONG RAW |
BINARY |
`binary` |
`binary` |
| JSON (Oracle 21c+) |
VARIANT |
`variant` |
`string` |
| XMLTYPE |
TEXT |
`string` |
`string` |
| ROWID / UROWID |
TEXT |
`string` |
`string` |
Source types not listed in the table are mapped to TEXT on standard tables and `string` on Iceberg
tables.
## Switching table storage format or Iceberg version
Switching between Standard and Iceberg, or between Iceberg v2 and v3, requires recreating the
connector. Follow these steps:
1. Stop the connector.
2. Delete the process group in Openflow.
3. Manually clean up the destination database (drop the replicated schemas/tables, or use a new
database).
4. Reimport the connector with the new **Table Storage Format** and select the target **Iceberg Version**
when configuring the connector.
This ensures all connector state is correctly cleaned up within Openflow. The new connector performs
a fresh snapshot into the destination.
## Upgrading an existing connector to use Iceberg Version pinning
Connector version `0.42.0` (embedded license) / `0.41.0` (independent license) introduces the
**Iceberg Version** parameter. If you are upgrading from an earlier connector version, a new
**Iceberg Version** field appears that you must configure to match your existing destination tables.
1. Stop the connector.
2. [Upgrade the runtime](/user-guide/data-integration/openflow/manage#label-openflow-upgrading-a-runtime)
to version `2026.7.21` or later.
3. [Upgrade the connector](/user-guide/data-integration/openflow/manage#upgrade-a-connector) in place
to the version listed above or later.
4. The **Iceberg Version** parameter defaults to `3` after the flow upgrade. Review and change to `2`
if your existing destination tables are Iceberg v2 before starting the connector.
5. Start the connector.
Selecting an **Iceberg Version** that doesn't match your existing destination tables can cause
type-mapping errors or DDL failures. Always verify the version of your existing tables before
choosing a value.
## References
- [CREATE EXTERNAL VOLUME](/sql-reference/sql/create-external-volume)
- [Data types for Apache Iceberg tables](/user-guide/tables-iceberg-data-types)
- [ALTER DATABASE](/sql-reference/sql/alter-database)
- [STORAGE_SERIALIZATION_POLICY](/sql-reference/parameters#storage-serialization-policy)
- [Set up the Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/setup-tasks)
---
title: Openflow Connector for Oracle: Maintenance
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/oracle/maintenance.md
section: Loading & Unloading Data
---
# %oracleofc%: Maintenance
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
The %oracleofc% is also subject to additional terms of service beyond the standard
connector terms of service. For more information, see the
[Openflow Connector for Oracle Addendum](https://www.snowflake.cn/en/legal/optional-offerings/offering-specific-terms/openflow-oracle-terms/).
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
- [About Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/about)
- [Openflow Connector for Oracle: Enable and manage commercial terms](/user-guide/data-integration/openflow/connectors/oracle/manage-commercial-terms)
- [Openflow Connector for Oracle: Set up incremental replication without snapshots](/user-guide/data-integration/openflow/connectors/oracle/incremental-replication)
This topic describes maintenance tasks for the %oracleofc%, such as reinstalling the
connector or setting the starting redo log position.
These operations are often used in conjunction with [Incremental replication without snapshots](/user-guide/data-integration/openflow/connectors/oracle/incremental-replication).
## Check the replication status of a table
Interim failures, such as connection errors or temporary source unavailability during a high-availability failover, do not prevent table replication. Replicated tables keep their current status and the connector retries on the next polling cycle. However, permanent failures, such as unsupported data types, prevent table replication.
To troubleshoot replication issues or verify that a table has been successfully removed from the replication flow, check the Table State Store:
1. In the Openflow runtime canvas, right-click a processor group and choose **Controller Services**. A table listing controller services displays.
2. Locate the row labeled **Table State Store**, click the **More** %sf-vertical-more-button% button on the right side of the row, and then choose **View State**.
A list of tables and their current states displays. Type in the search box to filter the list by table name. The possible states are:
- **NEW**: The table is scheduled for replication but replication hasn't started.
- **SNAPSHOT_REPLICATION**: The connector is copying existing data. This status displays until all records are stored in the destination table.
- **INCREMENTAL_REPLICATION**: The connector is actively replicating changes. This status displays after snapshot replication ends and continues to display indefinitely until a table is either removed from replication or replication fails.
- **FAILED**: Replication has permanently stopped due to an error.
The Openflow runtime canvas doesn't display table status changes — only the current table status. However, table status changes are recorded in logs when they occur. Look for the following log message:
```text
Replication state for table .. changed from to
```
If a permanent failure prevents table replication, remove the table from replication. After you address the problem that caused the failure, you can add the table back to replication. For more information, see [Restart table replication](#label-of-oracle-restart-table-replication).
## Increase the oversized value limit
By default, the connector replicates individual values up to 16 MB and marks any table that contains a larger value as permanently failed. If your Snowflake account has the `ENABLE_OPENFLOW_CDC_ORACLE_SSV2` parameter set to `true`, the per-value limit can be raised from 16 MB to **128 MB**.
The 128 MB limit applies in two ways: it's both the maximum size of a single value and the maximum total size of a row. The connector adds metadata columns to every replicated row (`_SNOWFLAKE_UPDATED_AT`, `_SNOWFLAKE_INSERTED_AT`, `_SNOWFLAKE_DELETED`) that count toward the per-row limit, along with all other columns in the row. As a result, a single value can't reach the full 128 MB in practice when the row includes other data.
The increased limit doesn't apply equally to all column types.
In Snowflake, the maximum size for `BINARY` is **64 MB** (`BINARY(67108864)`), even when the increased size limits are enabled. Only `VARCHAR`, `VARIANT`, `ARRAY` and `OBJECT` columns can hold up to 128 MB.
### Check whether the 128 MB limit is available
You may not be able to verify the `ENABLE_OPENFLOW_CDC_ORACLE_SSV2` parameter value by querying it. To check if it is enabled, see if the FlowFiles flow through **Upload Rows via Snowpipe Streaming 2** processor (not through **Upload Rows via Snowpipe Streaming**).
### Configure the processors
Update the **Oversized Value Limit** property to `128 MB` on the following processors:
- **Fetch Rows by ROWID Range** (in the **Snapshot Load** group) — used when **Snapshot Fetching Strategy** is `CONCURRENT_BY_ROWID`
- **Fetch Table Rows** (in the **Snapshot Load** group) — used when **Snapshot Fetching Strategy** is `SEQUENTIAL_BY_PRIMARY_KEY`
- **Read Oracle CDC Stream** (in the **Incremental Load** group)
For each processor:
1. Locate the processor in the flow. On the connector canvas, you can use the search box in the top-right corner to find processors by name.
2. Right-click the processor and select **Configure**.
3. Open the **Properties** tab.
4. Set **Oversized Value Limit** to `128 MB`.
5. Apply the change.
For tables that are already being replicated and have destination columns narrower than `VARCHAR(134217728)` or `BINARY(67108864)`, see [](#label-of-oracle-migrate-oversized-value-tables).
### Migrate existing tables
The steps in [](#label-of-oracle-increase-oversized-value-limit) raise the limit for newly created destination tables. If a table is already being replicated and its destination column type is **not** `VARCHAR(134217728)` or `BINARY(67108864)`, but you now want to load values larger than the original 16 MB limit, you must manually widen the column type on **both** the journal and destination tables.
Before you migrate, check the current destination column type, because it can vary depending on when the snapshot replication was performed.
You must stop replication for the affected table before altering its journal or destination tables. Altering these tables while replication is active can corrupt in-flight data.
To migrate a table:
1. Stop replication for the affected table by stopping the topmost processors of the **Snapshot Load** and **Incremental Load** groups until all queues are empty. For the equivalent stop procedure, see the substeps in [](#label-oracle-reinstall-connector).
2. Widen the column on both the journal table and the destination table, according to the column type:
1. For **VARCHAR** columns, run `ALTER TABLE ... ALTER COLUMN ... SET DATA TYPE VARCHAR(134217728)` on the journal table and on the destination table (one statement per table).
2. For **BINARY** columns, Snowflake doesn't allow widening `BINARY` in place, so do the following on both the journal and destination tables:
1. Add a new column of type `BINARY(67108864)`.
2. Copy data from the original column into the new column.
3. Drop the original column and rename the new column to the original name.
3. Restart replication by re-enabling the processors.
### Performance considerations
Raising the per-value limit increases the amount of data that the connector loads into memory and moves through the flow, which raises the load on both the runtime and the warehouse. Size the runtime and warehouse accordingly.
When **Oversized Value Strategy** is set to **Set Null**, the connector still loads each oversized value into memory before it can replace it with `NULL`. If your tables contain multi-gigabyte LOB columns, exclude those columns from replication.
During both snapshot and incremental replication, the queue in front of the **Upload Rows via Snowpipe Streaming 2** processor can fill with FlowFiles and trigger back pressure, which consumes a large amount of runtime disk space. For larger tables, use a Large runtime to provide additional storage. For guidance on choosing a size, see [Runtime sizing](/user-guide/data-integration/openflow/connectors/oracle/setup-connector#label-oracle-runtime-sizing).
#### Snapshot replication
Large LOB values can substantially increase snapshot runtime memory and disk use, because **Fetch Rows by ROWID Range** reads full row payloads (including LOBs) into the runtime before emitting FlowFiles.
The **Split Table into Chunks** processor sizes ROWID ranges from table heap blocks. Because LOBs are stored in separate segments, their size is not included in that calculation. When non-LOB column data is small compared with LOB payloads, a single chunk can still require **Fetch Rows by ROWID Range** to read multi-gigabytes of data. Output FlowFiles for that chunk are not passed downstream until the JDBC fetch for the entire ROWID range completes, so later processors may not receive data for a long time.
#### Incremental replication
When the source produces frequent changes to rows that contain large values, you might need a Large warehouse. High-frequency merges of many moderately large rows (for example, many 8 MB values) can require a large single merge operation, and smaller warehouses can run out of memory. By contrast, fewer very large rows (for example, 128 MB values) are streamed file by file through the **Upload Rows via Snowpipe Streaming 2** processor, and each file is merged incrementally, which typically completes without warehouse errors even on smaller warehouses.
During incremental CDC, **Read Oracle CDC Stream** materializes each changed row in memory before it is processed. This applies even when **Oversized Value Strategy** is set to **Set Null**: the connector must load the full row, including all LOB columns, before it can replace oversized values with `NULL`. If a single row contains multiple multi-gigabyte LOB columns, the total in-memory size can exceed the runtime heap and cause an out-of-memory error. Exclude such columns from replication before you rely on incremental CDC.
## Enable error logging on an existing schema
When you set the **Error Handling Strategy** parameter to **Log Errors and Continue**, the connector enables error logging automatically only on tables that it creates afterward. Tables that the connector created earlier don't capture rejected rows until you turn on error logging for them. For more information about the error-handling strategies, see [](/user-guide/data-integration/openflow/connectors/oracle/about#label-oracle-error-handling).
Because the connector stores journal tables in the same schema as the destination tables, you can turn on error logging for a whole destination schema at once. Run the following stored procedure once per destination schema. Replace `my_database` with your destination database and `my_schema` with the destination schema.
The schema name is passed as a quoted identifier (for example, `'"my_schema"'`) so it matches the exact, case-sensitive name that the connector created. For more information about how the connector names destination schemas, see [](/user-guide/data-integration/openflow/connectors/oracle/setup-connector#label-oracle-snowflake-destination-parameters).
```sql
USE DATABASE my_database;
WITH enable_error_logging AS PROCEDURE (schema_name STRING)
RETURNS STRING
LANGUAGE SQL
AS
$$
DECLARE
tables RESULTSET;
table_count NUMBER DEFAULT 0;
BEGIN
SHOW TABLES IN SCHEMA IDENTIFIER(:schema_name);
-- Assign AFTER SHOW TABLES so LAST_QUERY_ID() refers to that result
tables := (
SELECT "database_name", "schema_name", "name"
FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()))
WHERE "kind" = 'TABLE'
);
FOR t IN tables DO
-- Double-quote each identifier so names with special characters are handled safely
EXECUTE IMMEDIATE
'ALTER TABLE "' || REPLACE(t."database_name", '"', '""') || '".' ||
'"' || REPLACE(t."schema_name", '"', '""') || '".' ||
'"' || REPLACE(t."name", '"', '""') || '" ' ||
'SET ERROR_LOGGING = TRUE';
table_count := table_count + 1;
END FOR;
RETURN 'Enabled ERROR_LOGGING on ' || table_count || ' table(s) in schema ' || :schema_name;
END;
$$
CALL enable_error_logging('"my_schema"');
```
### Verify that error logging is enabled
To confirm that error logging is enabled on every table in a schema, run the following procedure. It reports how many tables have error logging enabled and how many don't.
```sql
USE DATABASE my_database;
WITH verify_error_logging AS PROCEDURE (schema_name STRING)
RETURNS STRING
LANGUAGE SQL
AS
$$
DECLARE
tables RESULTSET;
probe RESULTSET;
total_tables NUMBER DEFAULT 0;
logging_enabled NUMBER DEFAULT 0;
disabled_or_invisible NUMBER DEFAULT 0;
BEGIN
SHOW TABLES IN SCHEMA IDENTIFIER(:schema_name);
-- Assign AFTER SHOW TABLES so LAST_QUERY_ID() refers to that result
tables := (
SELECT "database_name", "schema_name", "name"
FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()))
WHERE "kind" = 'TABLE'
);
FOR t IN tables DO
total_tables := total_tables + 1;
-- Probe ERROR_TABLE(): it succeeds only when error logging is enabled and visible
BEGIN
probe := (
EXECUTE IMMEDIATE
'SELECT 1 FROM ERROR_TABLE(' ||
'"' || REPLACE(t."database_name", '"', '""') || '".' ||
'"' || REPLACE(t."schema_name", '"', '""') || '".' ||
'"' || REPLACE(t."name", '"', '""') || '"' ||
') LIMIT 1'
);
logging_enabled := logging_enabled + 1;
EXCEPTION
WHEN STATEMENT_ERROR THEN
disabled_or_invisible := disabled_or_invisible + 1;
END;
END FOR;
RETURN 'schema=' || :schema_name ||
', total_tables=' || total_tables ||
', error_logging_enabled=' || logging_enabled ||
', error_logging_disabled_or_not_visible=' || disabled_or_invisible;
END;
$$
CALL verify_error_logging('"my_schema"');
```
## Reinstall the connector
This section provides instructions on how to reinstall the connector, and continue replicating data for
the same tables without having to snapshot them again.
It covers situations where the new connector is installed in the same runtime, as well as moved to a new runtime.
For the connector to continue replicating from the same CDC stream position where it stopped before reinstallation,
the source database must retain the archived redo logs long enough to cover the time after the prior connector was stopped
and before the new connector is started.
Ensure the archived redo log retention period of the Oracle database is high enough, and keep the reinstallation time to a minimum.
Typically a retention period of 24 hours is sufficient; however, longer times might be appropriate to ensure time to reinstall.
For more information on configuring archived redo log retention, see [Openflow Connector for Oracle: Configure the Oracle database](/user-guide/data-integration/openflow/connectors/oracle/setup-oracledb).
### Prerequisites
Review and note connector parameter context values.
If you're reinstalling the connector in the same runtime, you can reuse the existing context.
If the new instance is located in a different runtime, you must re-enter all parameters.
1. Finish processing all in-flight FlowFiles in the existing connector, then stop the connector.
1. Sign in to %sf-web-interface-link%.
2. In the navigation menu, select **Ingestion** %raa% **Openflow**.
3. Select **Launch Openflow**.
4. In the **Openflow** pane select the **Runtimes** tab.
5. Select the runtime containing the connector.
6. Select the connector.
7. Stop the topmost processor **Set Tables for Replication** in the **Snapshot Load** group.
8. Stop the topmost processor **Read Oracle CDC Stream** in the **Incremental Load** group.
9. If you changed the value of the **Merge Task Schedule CRON** parameter, return it to `* * * * * ?`. Otherwise, queues won't be emptied until the next scheduled run.
Wait until all FlowFiles in the connector have been processed, and all queues are empty.
When all FlowFiles have been processed, the **Queued** value on the connector's processor group becomes zero.
If any items remain in the original connector's queues, data gaps might occur when the new connector starts.
10. Stop all processors and controller services in the connector.
The existing connector can remain in the runtime and doesn't interfere with the new instance, as long as it remains stopped.
2. If you're moving the connector to a new runtime, download the flow definition from the existing connector so that you can recreate the connector with its current state instead of configuring it from scratch. Downloading a flow definition requires Openflow Runtime Server version 2026.6.4.18 or later.
1. Right-click the connector's process group, then select **Download flow definition**.
2. Select both of the following options, then download the flow definition:
- **Export with External Services**: includes the controller services that the connector references from parent process groups.
- **Export with Components State**: includes component state, such as redo log positions and incremental replication state, so that replication continues from where it left off.
3. Create the connector in the target runtime:
- If you downloaded the flow definition, import it into the new runtime. Importing the flow definition preserves the component state captured during the export, so the connector resumes incremental replication from its previous positions.
- Otherwise, create a new instance of the connector. If you're using the same runtime as the original connector, you can choose to keep the existing parameter contexts and reuse the settings.
4. If you're installing into a different runtime or you deleted the previous parameter contexts, enter the configuration settings into the new parameter contexts,
including the table names and patterns as described in [Install and configure the Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/setup-connector). A downloaded flow definition doesn't include sensitive values (such as passwords) or uploaded files (such as the Oracle auto-login wallet file), so you must re-enter and re-upload them.
5. Navigate to the `Oracle Ingestion Parameters` context, and set the following parameters:
- Set the `Ingestion Type` parameter to `incremental`. For more information on the concerns, see [](#label-oracle-incremental-replication).
- Set the `Starting XStream Position` parameter to `Earliest`.
For more information and potential concerns, see [](#label-oracle-alter-xstream-outbound-server).
If you imported the flow definition with **Export with Components State** selected, the connector retains its previous redo log positions. In this case, leave `Starting XStream Position` set to `Latest` to continue replication from where it stopped.
6. Start the new connector.
### Usage notes
The new connector uses the existing destination tables that were created by the original connector, but the connector creates new journal tables.
## Alter XStream outbound server
The connector regularly updates the XStream server with the latest SCN position it processed. If the connector
is reinstalled and connects to the same XStream outbound server, it will resume reading from the SCN position where it left off.
This SCN number can be checked with:
```sql
SELECT PROCESSED_LOW_SCN
FROM DBA_XSTREAM_OUTBOUND_PROGRESS
WHERE SERVER_NAME = 'XOUT1';
```
If you want to re-read data from an earlier position, you must first change the start SCN of the XStream server:
```sql
BEGIN
DBMS_XSTREAM_ADM.ALTER_OUTBOUND(
server_name => 'XOUT1',
start_scn =>
);
END;
/
```
The value of `` must be a valid SCN within the range of available redo logs. The lowest SCN that the start position can be reset to can be checked with:
```sql
SELECT REQUIRED_CHECKPOINT_SCN
FROM DBA_CAPTURE
WHERE CLIENT_NAME = 'XOUT1';
```
This is the lowest SCN for which the capture process requires redo information.
## Specify load from XStream position
The %oracleofc% connector allows you to select the starting position where Oracle redo logs are read.
By default the connector reads from the latest available position. Alternatively, you can choose the earliest position available on the source instance.
Choosing to start from the earliest position is common when reinstalling the connector.
This allows the new instance to catch up and continue replicating existing tables without having to snapshot each again.
Switching a running connector from latest to earliest position causes all available redo logs
to be re-read, re-processed, and re-applied to the destination table.
While the redo logs are being re-read, the columns and data in affected destination tables
can become out of sync with their sources until all events have been re-processed and merged.
The following parameters are available in the `Ingestion Parameters` context:
| Parameter |
Description |
| Starting XStream Position |
- `Latest` (default): CDC stream reading starts at the latest available position and continues from there.
- `Earliest`: Switches the incremental load to start, or restart reading from the earliest available
XStream position.
|
| Re-read Tables in State |
- `New` (default):
While re-reading the redo logs, only those LCRs (Logical Change Records) will be processed
from new tables added to replication after the re-reading started.
Other LCRs are discarded until the connector reaches the position just before re-reading started.
- `Any active`: Re-read and re-process events from any table currently in replication.
|
To determine whether the connector finished re-reading the redo logs:
1. Navigate to the Openflow canvas.
2. Open the **Incremental Load** process group.
3. Right-click the topmost processor named **Read Oracle CDC Stream**, then select **View state**.
4. Compare the state entries:
- **lcr.position.rewind**: the latest position the processor read before re-reading of the redo logs started.
- **lcr.position.last**: the current latest position read by the processor. As long as this value is lower than the rewind value above, the processor is still re-reading the redo logs.
### Usage notes
- After a running connector is switched to read from the earliest position, and starts running,
the process can't be reconfigured or cancelled, and continues until the currently-read position reaches the position from before it started.
- Switching to the earliest position on a running connector will, for any tables being re-processed,
finish their existing journals, and create new journal tables.
- If the redo log contains events from a previous table that was dropped
and re-created in the source database, re-reading the stream re-processes all events in the current destination.
The connector can't distinguish between a previous and current source table if they share the same name.
Schema changes (such as ALTER TABLE statements that add or drop columns) aren't supported
while re-reading the redo logs from the earliest position. If any table's schema was
altered between the earliest available SCN and the current position, that table should
be removed from replication and re-added with a fresh snapshot instead.
---
title: Openflow Connector for Oracle: Set up incremental replication without snapshots
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/oracle/incremental-replication.md
section: Loading & Unloading Data
---
# %oracleofc%: Set up incremental replication without snapshots
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
The %oracleofc% is also subject to additional terms of service beyond the standard
connector terms of service. For more information, see the
[Openflow Connector for Oracle Addendum](https://www.snowflake.cn/en/legal/optional-offerings/offering-specific-terms/openflow-oracle-terms/).
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
- [About Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/about)
- [Openflow Connector for Oracle: Enable and manage commercial terms](/user-guide/data-integration/openflow/connectors/oracle/manage-commercial-terms)
- [Install and configure the Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/setup-connector)
- [Openflow Connector for Oracle: Maintenance](/user-guide/data-integration/openflow/connectors/oracle/maintenance)
This topic describes how to configure the %oracleofc% connector to start replicating incremental changes for newly added tables immediately, bypassing snapshots. This configuration is useful when you reinstall the connector over previously replicated data and want to continue replication without snapshotting every table again.
You can enable incremental replication on either a new or an existing connector instance.
## Enable incremental replication without snapshots on a new connector
To enable incremental replication on a new connector instance:
1. Set up the connector as described in [Install and configure the Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/setup-connector).
2. In the `Oracle Ingestion Parameters` context, set the `Ingestion Type` parameter to `incremental`.
## Enable incremental replication without snapshots on an existing connector
To enable incremental replication on an existing connector:
1. sign in to %sf-web-interface-link%.
2. in the navigation menu, select **Ingestion** %raa% **Openflow**.
3. In the **Openflow** pane select the **Runtimes** tab.
4. Select the runtime containing the connector.
5. Select the connector.
6. In the `Ingestion Parameters` context, specify `Ingestion Type` = `incremental`.
7. Add new replication tables. These tables immediately switch to their incremental load.
To return to replicating tables with the snapshot load, change **Ingestion Type** from `incremental` to `full`.
# Usage notes
- Changing the value of **Ingestion Type** does not impact any tables that have begun replicating data.
Tables currently in the snapshot phase continue until the snapshot load is complete.
- While **Ingestion Type** is set to `incremental`, new tables added to the list of replicated tables bypass the snapshot phase.
This includes new tables added to the source database that match the `Included Table Regex` parameter.
Ensure that the ingestion type is set to `incremental` to bypass the snapshot phase.
Connectors should only remain in `incremental` mode as long as required as it bypasses snapshots.
Once customer needs for incremental updates have been satisfied the connector should be returned to `full` mode.
- For tables that bypass snapshot load, the connector creates a destination table in Snowflake,
by executing `CREATE TABLE IF NOT EXISTS`, only if no destination table already exists.
Tables going through the snapshot require that no destination table exist.
## Recover a table using incremental-only mode
If a table's snapshot completed successfully but incremental replication later failed, you don't need to remove the table and snapshot it again. Instead, you can recover the table by replaying the changes that are still available in the source redo logs through XStream and merging them onto the existing destination table.
Incremental replication can fail for several reasons, for example:
- A record in the source database can't be read because it has an incorrect or unsupported format.
- A row exceeds the maximum supported size.
- A merge operation can't complete.
- A transient error persists through so many retries that the table enters the FAILED state.
To recover the table without a new snapshot, remove it from replication, switch the connector to incremental-only mode reading from the earliest available position, and add the table back. The connector reads all available changes from the oldest available XStream position, then replays and reapplies them to the destination table.
Before you recover the table, address the underlying cause of the failure. Otherwise, the connector encounters the same error again when it replays the changes. For example, raise the per-value limit (see [Increase the oversized value limit](/user-guide/data-integration/openflow/connectors/oracle/maintenance#label-of-oracle-increase-oversized-value-limit)) or fix the problematic record in the source database.
To recover the table:
1. Remove the table from replication. In the `Ingestion Parameters` context, remove the table from **Included Table Names**, or modify **Included Table Regex** so the table is no longer matched. Wait until the table's state is fully removed from the **Table State Store** controller service before you continue.
Don't drop the destination table. This procedure reuses the existing destination table and replays incremental changes onto it.
2. Stop the connector's process group so that you can change its configuration. On the connector canvas, right-click the connector's process group and select **Stop**.
3. In the `Ingestion Parameters` context, set the `Ingestion Type` parameter to `incremental`.
4. Set the `Starting XStream Position` parameter to `Earliest`. The connector reads all available changes again from the oldest available XStream position, then replays and reapplies them to the destination table. For more information, see the "Specify load from XStream position" section in [Openflow Connector for Oracle: Maintenance](/user-guide/data-integration/openflow/connectors/oracle/maintenance).
Leave `Re-read Tables in State` at its default value, `New`, so that only the table you add back reads from the earliest position. Tables already in replication continue from their last positions.
5. Add the table back to replication by reversing the change you made in step 1.
6. Start the connector's process group. Right-click the connector's process group and select **Start**.
7. Wait until the table returns to incremental replication. In the **Table State Store** controller service state, the table transitions to INCREMENTAL_REPLICATION when recovery completes.
8. Revert the changes you made in steps 3 and 4: set `Ingestion Type` and `Starting XStream Position` back to their previous values.
This procedure recovers only the changes still available in the source redo logs. If the required redo information is no longer retained on the source, the recovered table can have gaps. In that case, you must take a new snapshot to fully resynchronize the table.
---
title: Openflow Connector for Oracle: Set up Snowflake
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/oracle/setup-snowflake.md
section: Loading & Unloading Data
---
# %oracleofc%: Set up Snowflake
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
The %oracleofc% is also subject to additional terms of service beyond the standard
connector terms of service. For more information, see the
[Openflow Connector for Oracle Addendum](https://www.snowflake.cn/en/legal/optional-offerings/offering-specific-terms/openflow-oracle-terms/).
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
- [About Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/about)
- [Openflow Connector for Oracle: Enable and manage commercial terms](/user-guide/data-integration/openflow/connectors/oracle/manage-commercial-terms)
- [Openflow Connector for Oracle: Configure the Oracle database](/user-guide/data-integration/openflow/connectors/oracle/setup-oracledb)
- [Install and configure the Openflow Connector for Oracle](/user-guide/data-integration/openflow/connectors/oracle/setup-connector)
This topic describes how to set up your Snowflake environment for the
%oracleofc%.
As a Snowflake administrator, perform the following tasks:
1. Create a destination database in Snowflake to store the replicated data:
```sql
CREATE DATABASE ;
```
2. Create a Snowflake [service user](#label-user-type-property):
```sql
CREATE USER
TYPE = SERVICE
COMMENT='Service user for automated access of Openflow';
```
3. Create a Snowflake role for the connector and grant the required
privileges:
```sql
CREATE ROLE ;
GRANT ROLE TO USER ;
GRANT USAGE ON DATABASE TO ROLE ;
GRANT CREATE SCHEMA ON DATABASE
TO ROLE ;
```
Use this role to manage the connector's access to the Snowflake database.
To create objects in the destination database, you must grant the
[USAGE and CREATE SCHEMA privileges](#label-database-privileges)
on the database to the role used to manage access.
4. Create a Snowflake warehouse for the connector and grant the required
privileges:
```sql
CREATE WAREHOUSE WITH
WAREHOUSE_SIZE = 'XSMALL'
AUTO_SUSPEND = 300
AUTO_RESUME = TRUE;
GRANT USAGE, OPERATE ON WAREHOUSE
TO ROLE ;
```
Snowflake recommends starting with a XSMALL warehouse size, then
experimenting with size depending on the number of tables being
replicated and the amount of data transferred. Large numbers of tables
typically scale better with multi-cluster warehouses, rather than a
larger warehouse size. For more information, see
[multi-cluster warehouses](/user-guide/warehouses-multicluster).
5. Set up the public and private keys for key pair authentication:
1. Create a pair of secure keys (public and private).
2. Store the private key for the user in a file to supply to the
connector's configuration.
3. Assign the public key to the Snowflake service user:
```sql
ALTER USER SET RSA_PUBLIC_KEY = 'thekey';
```
For more information, see [Key-pair authentication and key-pair rotation](/user-guide/key-pair-auth).
## Next steps
[Configure the connector](/user-guide/data-integration/openflow/connectors/oracle/setup-connector).
---
title: Openflow Connector for PostgreSQL Maintenance
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/postgres/maintenance.md
section: Loading & Unloading Data
---
# %postgresql% Maintenance
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
- [About Openflow](/user-guide/data-integration/openflow/about)
- [Manage Openflow](/user-guide/data-integration/openflow/manage)
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
- [Set up the Openflow Connector for PostgreSQL](/user-guide/data-integration/openflow/connectors/postgres/setup)
- [Openflow Connector for PostgreSQL: Data mapping](/user-guide/data-integration/openflow/connectors/postgres/data-mapping)
This topic describes important maintenance considerations and best practices for
maintaining the %postgresql% when making changes to the source PostgreSQL database.
In addition, this topic describes how to restart table replication and reinstall the connector.
## Check the replication status of a table
Interim failures, such as connection errors or temporary source unavailability during a high-availability failover, do not prevent table replication. Replicated tables keep their current status and the connector retries on the next polling cycle. However, permanent failures, such as unsupported data types, prevent table replication.
To troubleshoot replication issues or verify that a table has been successfully removed from the replication flow, check the Table State Store:
1. In the Openflow runtime canvas, right-click a processor group and choose **Controller Services**. A table listing controller services displays.
2. Locate the row labeled **Table State Store**, click the **More** %sf-vertical-more-button% button on the right side of the row, and then choose **View State**.
A list of tables and their current states displays. Type in the search box to filter the list by table name. The possible states are:
- **NEW**: The table is scheduled for replication but replication hasn't started.
- **SNAPSHOT_REPLICATION**: The connector is copying existing data. This status displays until all records are stored in the destination table.
- **INCREMENTAL_REPLICATION**: The connector is actively replicating changes. This status displays after snapshot replication ends and continues to display indefinitely until a table is either removed from replication or replication fails.
- **FAILED**: Replication has permanently stopped due to an error.
The Openflow runtime canvas doesn't display table status changes — only the current table status. However, table status changes are recorded in logs when they occur. Look for the following log message:
```text
Replication state for table .. changed from to
```
If a permanent failure prevents table replication, remove the table from replication. After you address the problem that caused the failure, you can add the table back to replication. For more information, see [Restart table replication](#label-of-postgres-restart-table-replication).
## Restart table replication
This procedure re-snapshots the table in place. It requires Runtime Extensions version `2026.6.18.9` or later and connector version `0.55.0` or later. On earlier versions, re-snapshotting a table that already exists in Snowflake fails instead of reloading in place. Upgrade Runtime Extensions first, and then upgrade the connector flow before you use this procedure.
A table in a FAILED state (for example, due to a missing primary key or an unsupported schema change) does not restart automatically. If a table enters a FAILED state or you need to restart replication from scratch, use the following procedure to remove and re-add the table to replication.
If the failure was caused by an issue in the source table such as a missing primary key, resolve that issue in the source database before continuing.
1. Remove the table from replication, using one of the following methods:
- Add the table to the **Re-snapshot Table Exclusions** parameter to temporarily exclude it from replication. This approach is convenient when the table is matched by an **Included Table Regex** that you don't want to change.
- In the **Ingestion Parameters** context, either remove the table from **Included Table Names** or modify the **Included Table Regex** so the table is no longer matched.
2. Verify the table has been removed:
1. In the Openflow runtime canvas, right-click a processor group and choose **Controller Services**.
2. In the table listing controller services, locate the **Table State Store** row, click the three vertical dots on the right side of the row, then choose **View State**.
You must wait until the table's state is fully removed from this list before proceeding. Don't continue until this configuration change has completed.
3. Wait until all queues in the connector are empty before you re-add the table. When all FlowFiles have been processed, the **Queued** value on the connector's processor group becomes zero.
Don't re-add the table while change events that were captured before you removed it are still queued. When you re-add a table, the connector loads the new snapshot in append-only mode, so any leftover change event that merges into the table after the re-snapshot might create duplicate rows in the destination table.
4. Re-add the table by reversing the change you made in the first step: either remove the table from **Re-snapshot Table Exclusions**, or add it back to **Included Table Names** or **Included Table Regex**.
You do not need to drop the destination table first. The connector re-snapshots the table in place: it makes a zero-copy [clone](/sql-reference/sql/create-clone) of the current destination table to an archive table named `_ARCHIVE_`, clears the destination table, and then loads the fresh snapshot into the same destination table. Because the destination table object is preserved, dependent objects such as streams remain attached and continue to work.
The archive table retains a copy of the destination table's contents from immediately before the reload, as a safeguard. The connector does not read from or write to it again, so you can drop it at any time once the backup is no longer needed, typically after you confirm that the re-snapshot completed and the destination data is correct.
5. Verify the restart: Check the **Table State Store** using the instructions given previously. The state of the table should appear with the status NEW, then transition to SNAPSHOT_REPLICATION, and finally to INCREMENTAL_REPLICATION.
## Increase the oversized value limit
By default, the connector replicates individual values up to 16 MB and marks any table that contains a larger value as permanently failed. If your Snowflake account has the `ENABLE_OPENFLOW_CDC_POSTGRES_SSV2` parameter set to `true`, the per-value limit can be raised from 16 MB to **128 MB**.
The 128 MB limit applies in two ways: it's both the maximum size of a single value and the maximum total size of a row. The connector adds metadata columns to every replicated row (`_SNOWFLAKE_UPDATED_AT`, `_SNOWFLAKE_INSERTED_AT`, `_SNOWFLAKE_DELETED`) that count toward the per-row limit, along with all other columns in the row. As a result, a single value can't reach the full 128 MB in practice when the row includes other data.
The increased limit doesn't apply equally to all column types.
In Snowflake, the maximum size for `BINARY` is **64 MB** (`BINARY(67108864)`), even when the increased size limits are enabled. Only `VARCHAR`, `VARIANT`, `ARRAY` and `OBJECT` columns can hold up to 128 MB.
### Check whether the 128 MB limit is available
You may not be able to verify the `ENABLE_OPENFLOW_CDC_POSTGRES_SSV2` parameter value by querying it. To check if it is enabled, see if the FlowFiles flow through **Upload Rows via Snowpipe Streaming 2** processor (not through **Upload Rows via Snowpipe Streaming**).
### Configure the processors
Update the **Oversized Value Limit** property to `128 MB` on both of the following processors:
- **Fetch Table Rows** (in the **Snapshot Load** group)
- **Read PostgreSQL CDC Stream** (in the **Incremental Load** group)
For each processor:
1. Locate the processor in the flow. On the connector canvas, you can use the search box in the top-right corner to find processors by name.
2. Right-click the processor and select **Configure**.
3. Open the **Properties** tab.
4. Set **Oversized Value Limit** to `128 MB`.
5. Apply the change.
For tables that are already being replicated and have destination columns narrower than `VARCHAR(134217728)` or `BINARY(67108864)`, see [](#label-of-postgres-migrate-oversized-value-tables).
### Migrate existing tables
The steps in [](#label-of-postgres-increase-oversized-value-limit) raise the limit for newly created destination tables. If a table is already being replicated and its destination column type is **not** `VARCHAR(134217728)` or `BINARY(67108864)`, but you now want to load values larger than the original 16 MB limit, you must manually widen the column type on **both** the journal and destination tables.
Before you migrate, check the current destination column type, because it can vary depending on when the snapshot replication was performed.
You must stop replication for the affected table before altering its journal or destination tables. Altering these tables while replication is active can corrupt in-flight data.
To migrate a table:
1. Stop replication for the affected table by stopping the topmost processors of the **Snapshot Load** and **Incremental Load** groups until all queues are empty. For the equivalent stop procedure, see the substeps in [](#label-postgres-reinstall-connector).
2. Widen the column on both the journal table and the destination table, according to the column type:
1. For **VARCHAR** columns, run a single `ALTER TABLE ... ALTER COLUMN ... SET DATA TYPE VARCHAR(134217728)` on both the journal and destination tables.
2. For **BINARY** columns, Snowflake doesn't allow widening `BINARY` in place, so do the following on both the journal and destination tables:
1. Add a new column of type `BINARY(67108864)`.
2. Copy data from the original column into the new column.
3. Drop the original column and rename the new column to the original name.
3. Restart replication by re-enabling the processors.
### Performance considerations
Raising the per-value limit increases the amount of data that the connector loads into memory and moves through the flow, which raises the load on both the runtime and the warehouse. Size the runtime and warehouse accordingly.
During both snapshot and incremental replication, the queue in front of the **Upload Rows via Snowpipe Streaming 2** processor can fill with FlowFiles and trigger back pressure, which consumes a large amount of runtime disk space. For larger tables, use a Large runtime to provide additional storage. For guidance on choosing a size, see [Runtime sizing](/user-guide/data-integration/openflow/connectors/cdc-runtime-sizing#label-openflow-cdc-runtime-sizing).
#### Snapshot replication
During snapshot replication, the product of `fetchSize * rowSize * concurrentQueries` can't exceed the heap size of the NiFi runtime, where:
- `fetchSize` is the number of rows fetched per query, set on the **Fetch Table Rows** processor (default: 100).
- `rowSize` is the size of a single row being fetched.
- `concurrentQueries` is the number of concurrent queries, set on the **Fetch Table Rows** processor (default: 2).
This memory requirement applies even when **Oversized Value Strategy** is set to **Set Null**, because the connector must load each oversized value into memory before it can replace the value with `NULL`.
If the source database contains many densely packed oversized values, consider excluding the affected column from replication before you start the snapshot. For example, if a column contains 1 GB values, loading even nine rows (~9 GB) can exhaust the heap and cause an out-of-memory error on a Medium runtime.
To speed up snapshot replication, you can increase the number of channels that the **Upload Rows via Snowpipe Streaming 2** processor uses. The number of channels is set by the processor's **Channel Group** property, which defaults to `${chunk.index:isEmpty():ifElse('1', ${chunk.index:mod(8)})}`.
To increase the number of channels:
1. Locate the **Upload Rows via Snowpipe Streaming 2** processor in the flow.
2. Stop the processor. You must stop the processor before you can change its properties.
3. Right-click the processor and select **Configure**.
4. Open the **Properties** tab.
5. In the **Channel Group** property, increase the value `8` in the expression. For example, change `8` to `16` to double the number of channels.
6. Apply the change.
7. Start the processor.
While a snapshot replication is in progress, only increase the number of channels. Decreasing the number of channels during an active snapshot can cause data loss.
#### Incremental replication
When the source produces frequent changes to rows that contain large values, you might need a Large warehouse. With smaller warehouses, replicating many 8 MB rows can cause an out-of-memory error. By contrast, replicating 128 MB rows with continuous merges completes without warehouse errors, because the connector streams the data file by file through the **Upload Rows via Snowpipe Streaming 2** processor and the merge processes it gradually.
## Enable error logging on an existing schema
When you set the **Error Handling Strategy** parameter to **Log Errors and Continue**, the connector enables error logging automatically only on tables that it creates afterward. Tables that the connector created earlier don't capture rejected rows until you turn on error logging for them. For more information about the error-handling strategies, see [](/user-guide/data-integration/openflow/connectors/postgres/about#label-postgres-error-handling).
Because the connector stores journal tables in the same schema as the destination tables, you can turn on error logging for a whole destination schema at once. Run the following stored procedure once per destination schema. Replace `my_database` with your destination database and `my_schema` with the destination schema.
The schema name is passed as a quoted identifier (for example, `'"my_schema"'`) so it matches the exact, case-sensitive name that the connector created. For more information about how the connector names destination schemas, see [](/user-guide/data-integration/openflow/connectors/postgres/setup#label-of-postgres-destination-parameters).
```sql
USE DATABASE my_database;
WITH enable_error_logging AS PROCEDURE (schema_name STRING)
RETURNS STRING
LANGUAGE SQL
AS
$$
DECLARE
tables RESULTSET;
table_count NUMBER DEFAULT 0;
BEGIN
SHOW TABLES IN SCHEMA IDENTIFIER(:schema_name);
-- Assign AFTER SHOW TABLES so LAST_QUERY_ID() refers to that result
tables := (
SELECT "database_name", "schema_name", "name"
FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()))
WHERE "kind" = 'TABLE'
);
FOR t IN tables DO
-- Double-quote each identifier so names with special characters are handled safely
EXECUTE IMMEDIATE
'ALTER TABLE "' || REPLACE(t."database_name", '"', '""') || '".' ||
'"' || REPLACE(t."schema_name", '"', '""') || '".' ||
'"' || REPLACE(t."name", '"', '""') || '" ' ||
'SET ERROR_LOGGING = TRUE';
table_count := table_count + 1;
END FOR;
RETURN 'Enabled ERROR_LOGGING on ' || table_count || ' table(s) in schema ' || :schema_name;
END;
$$
CALL enable_error_logging('"my_schema"');
```
### Verify that error logging is enabled
To confirm that error logging is enabled on every table in a schema, run the following procedure. It reports how many tables have error logging enabled and how many don't.
```sql
USE DATABASE my_database;
WITH verify_error_logging AS PROCEDURE (schema_name STRING)
RETURNS STRING
LANGUAGE SQL
AS
$$
DECLARE
tables RESULTSET;
probe RESULTSET;
total_tables NUMBER DEFAULT 0;
logging_enabled NUMBER DEFAULT 0;
disabled_or_invisible NUMBER DEFAULT 0;
BEGIN
SHOW TABLES IN SCHEMA IDENTIFIER(:schema_name);
-- Assign AFTER SHOW TABLES so LAST_QUERY_ID() refers to that result
tables := (
SELECT "database_name", "schema_name", "name"
FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()))
WHERE "kind" = 'TABLE'
);
FOR t IN tables DO
total_tables := total_tables + 1;
-- Probe ERROR_TABLE(): it succeeds only when error logging is enabled and visible
BEGIN
probe := (
EXECUTE IMMEDIATE
'SELECT 1 FROM ERROR_TABLE(' ||
'"' || REPLACE(t."database_name", '"', '""') || '".' ||
'"' || REPLACE(t."schema_name", '"', '""') || '".' ||
'"' || REPLACE(t."name", '"', '""') || '"' ||
') LIMIT 1'
);
logging_enabled := logging_enabled + 1;
EXCEPTION
WHEN STATEMENT_ERROR THEN
disabled_or_invisible := disabled_or_invisible + 1;
END;
END FOR;
RETURN 'schema=' || :schema_name ||
', total_tables=' || total_tables ||
', error_logging_enabled=' || logging_enabled ||
', error_logging_disabled_or_not_visible=' || disabled_or_invisible;
END;
$$
CALL verify_error_logging('"my_schema"');
```
## Upgrading PostgreSQL
Upgrading the connector requires a different approach depending on whether PostgreSQL is being upgraded to the next minor or major version.
Minor version upgrades
- Are data-safe.
- Require no special treatment.
- Require stopping the connector for the duration of the upgrade to avoid reporting connectivity issues.
- Continue replicating, after the upgrade, with no data loss.
Major version upgrades
- Require the PostgreSQL server to drop replication slots, including any used by the connector.
- Cannot preserve or migrate replication slots to the new version. See also [](#label-postgres-upgrade-note).
- Require restarting replication of all tables from the snapshot phase, unless you can stop all writes to the source database for the duration of the upgrade. In that case, you can keep the replicated data and resume with incremental replication only. For more information, see [](#label-postgres-upgrade-incremental-only).
To perform a minor version upgrade, do the following:
1. Stop the connector, including all Processors and Controller Services.
2. Upgrade PostgreSQL.
3. Restart the connector.
To perform a major version upgrade, do the following:
1. Remove all tables from replication in the connector by clearing the **Included Table Names** and **Included Table Regex** parameters.
2. Wait until all queues in the connector are empty, so that every captured change is merged into the destination tables.
3. Stop the connector, including all Processors and Controller Services.
4. Open the **Incremental Load** group in the connector.
5. Clear the state of the CDC processor:
1. Open the **Incremental Load** group in the connector.
2. Right-click the top Processor in the group, **Read PostgreSQL CDC Stream**, and select **View state**.
3. Click **Clear state**.
4. Click **Close**.
6. Upgrade PostgreSQL.
7. Restart the connector. A new replication slot will be created.
8. Re-add all tables to the **Included Table Names** or **Included Table Regex** parameters.
You don't need to drop or rename the destination tables before you re-add the tables. The connector re-snapshots each table in place, which preserves the destination table object along with dependent objects such as streams. Before it loads the fresh snapshot, the connector saves a copy of the previous contents in an archive table named `_ARCHIVE_`, which you can drop once you've confirmed that the re-snapshot completed. For more information, see [](#label-of-postgres-restart-table-replication).
### Upgrade without re-snapshotting tables
When an upgrade drops the replication slot, you can avoid re-snapshotting every table if you can stop all writes to the source database while you upgrade. This applies to major version upgrades and to any upgrade to PostgreSQL 17.0 from version 16 or earlier. The connector keeps the data it already replicated and continues with incremental replication only.
No writes of any kind, whether DML or DDL, can reach the replicated database from the moment you stop the connector until all tables are back in incremental replication. The new replication slot starts at the current write-ahead log position, so any change made during that window is lost and there's no way to recover it without a new snapshot.
To upgrade without re-snapshotting tables, do the following:
1. Remove all tables from replication in the connector by clearing the **Included Table Names** and **Included Table Regex** parameters.
2. Wait until all queues in the connector are empty, so that every captured change is merged into the destination tables.
3. Stop the connector, including all Processors and Controller Services.
4. Stop all writes to the source database, and keep them stopped for the rest of this procedure.
5. Clear the state of the CDC processor:
1. Open the **Incremental Load** group in the connector.
2. Right-click the top Processor in the group, **Read PostgreSQL CDC Stream**, and select **View state**.
3. Click **Clear state**.
4. Click **Close**.
6. Upgrade PostgreSQL.
7. In the `PostgreSQL Ingestion Parameters` context, set the `Ingestion Type` parameter to `incremental`. For more information, see [Openflow Connector for PostgreSQL: Set up incremental replication without snapshots](/user-guide/data-integration/openflow/connectors/postgres/incremental-replication).
8. Restart the connector. A new replication slot is created.
9. Re-add all tables to the **Included Table Names** or **Included Table Regex** parameters. The tables bypass the snapshot phase and replicate incrementally into the existing destination tables.
10. Confirm that every table reaches the status INCREMENTAL_REPLICATION in the **Table State Store** controller service. For instructions on viewing table state, see [](#label-of-postgres-check-table-replication-status).
11. Resume writes to the source database.
12. Set the `Ingestion Type` parameter back to `full`, so that tables you add later still get a snapshot.
### PostgreSQL 17 and later versions upgrades
PostgreSQL 17 improved upgrading such that it no longer requires dropping replication slots when upgrading to later versions such as 17.1 %ra% 18.0.
Upgrading to PostgreSQL 17.0 or later from prior versions (16 and earlier) drops replication slots and should be treated as a major upgrade.
Future versions of PostgreSQL may also improve the upgrade process further.
If the connector is using failover slot support, ensure the slot is caught up and not conflicting before starting the upgrade. See [Additional step when running pg_upgrade](/user-guide/data-integration/openflow/connectors/postgres/failover#label-postgres-failover-pg-upgrade).
## Reclaim journal table storage
Journal tables hold every change to a replicated table. The connector never drops them, but it only
reads the latest journal for each replicated source table, using append-only streams on top of the
journals. To reclaim storage, you can:
- Truncate all journal tables at any time.
- Drop the journal tables related to source tables that were removed from replication.
- Drop all but the latest generation journal tables for actively replicated tables.
For example, if your connector is set to actively replicate source table `orders`, and you have
earlier removed table `customers` from replication, you may have the following journal tables. In
this case you can drop all of them *except* `orders_5678_2`.
```text
customers_1234_1
customers_1234_2
orders_5678_1
orders_5678_2
```
## Stop or delete the connector
When stopping or removing the connector, you have to consider the replication slot (https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION-SLOTS) that the connector uses.
The connector creates its own replication slot with a name starting with
`snowflake_connector_` followed by a random suffix. As the connector reads the replication stream,
it advances the slot, so that PostgreSQL can trim its WAL log and free up disk space.
When the connector is paused, the slot isn't advanced, and changes to the source database keep increasing the WAL
log size. You should not keep the connector paused for extended periods of time, especially on high-traffic databases.
When the connector is removed, whether by dropping it with `DROP OPENFLOW CONNECTOR` (gen 2), deleting it from the Openflow canvas,
or any other means, such as deleting the whole Openflow instance, the replication slot remains in place, and must be dropped manually.
If you have multiple connector instances replicating from the same PostgreSQL database,
each instance will create its own uniquely named replication slot. When dropping a replication slot manually, make sure
it's the right one. You can see which replication slot is used by a given connector instance by checking the state of the `CaptureChangePostgreSQL` processor.
## Reinstall the connector
This section describes how to reinstall the connector.
It covers situations where the new connector is installed in the same runtime, or where it is moved to a new runtime.
Reinstall is often used in conjunction with [Incremental replication without snapshots](/user-guide/data-integration/openflow/connectors/postgres/incremental-replication).
For the connector to be able to continue replicating from the same CDC stream position where it stopped before reinstallation,
the source database must retain the WAL long enough to cover the time between when the old connector stops and the new connector starts.
Ensure the `max_wal_size` parameter of the PostgreSQL server is high enough, depending on your traffic, and keep the reinstallation time to a minimum.
### Prerequisites
Review and note connector parameter context values.
If you're reinstalling the connector in the same runtime, you can reuse the existing context.
If the new instance will be located in a different runtime, you will have to re-enter all parameters.
To reinstall the connector:
1. Finish processing all in-flight FlowFiles in the existing connector, and then stop the connector.
1. Sign in to %sf-web-interface-link%.
2. In the navigation menu, select **Ingestion** %raa% **Openflow**.
3. Select **Launch Openflow**.
4. In the **Openflow** pane select the **Runtimes** tab.
5. Select the runtime containing the connector.
6. Select the connector.
7. Stop the topmost processor **Set Tables for Replication** in the **Snapshot Load** group.
8. Stop the topmost processor **Read PostgreSQL CDC Stream** in the **Incremental Load** group.
9. If you changed the value of the **Merge Task Schedule CRON** parameter, return it to `* * * * * ?`. Otherwise, queues won't be emptied until the next scheduled run.
Wait until all FlowFiles in the connector have been processed, and all queues are empty.
When all FlowFiles have been processed, the **Queued** value on the connector's processor group becomes zero.
If there are any items left in the original connector's queues, there may be data gaps when the new connector starts.
10. Stop all Processors and Controller Services in the connector.
2. Find and copy the name of the replication slot used by the original connector,
by viewing the state of the topmost processor in the `Incremental Load` group with name `Read PostgreSQL CDC Stream`.
The replication slot name is stored under the key `replication.slot.name`.
Copy the value of the key to a text editor.
3. Create a new instance of the connector. If you're using the same runtime as the original connector, you can choose to keep the existing parameter contexts, and reuse the settings.
The existing connector can remain in the runtime and doesn't interfere with the new instance, as long as it remains stopped.
4. If you're installing into a different runtime, or you deleted the previous parameter contexts, enter all the configuration settings into the new parameter contexts,
including the table names and patterns as described in [Set up the Openflow Connector for PostgreSQL](/user-guide/data-integration/openflow/connectors/postgres/setup).
5. Open the `PostgreSQL Ingestion Parameters` context, and set `Ingestion Type` parameter to `incremental`.
For more information, see [](#label-postgres-incremental-replication).
6. Open the `PostgreSQL Source Parameters` context, and set the `Replication Slot Name` parameter to the value you copied earlier.
7. Start the new connector.
### Usage notes
The new connector will use the same existing destination tables that were created by the original connector, but will create new journal tables.
---
title: Openflow Connector for PostgreSQL: Data mapping
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/postgres/data-mapping.md
section: Loading & Unloading Data
---
# %postgresql%: Data mapping
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
- [About Openflow Connector for PostgreSQL](/user-guide/data-integration/openflow/connectors/postgres/about)
- [Set up the Openflow Connector for PostgreSQL](/user-guide/data-integration/openflow/connectors/postgres/setup)
This topic describes how PostgreSQL data types are mapped
to Snowflake data types.
## PostgreSQL to Snowflake data type mapping
The following table shows how PostgreSQL data types are mapped to Snowflake data types
when replicating data.
| PostgreSQL type |
Snowflake type |
Notes |
| SMALLINT / INT2 |
INT |
|
| INTEGER / INT / INT4 |
INT |
|
| BIGINT / INT8 |
INT |
|
| SMALLSERIAL / SERIAL2 |
INT |
|
| SERIAL / SERIAL4 |
INT |
|
| BIGSERIAL / SERIAL8 |
INT |
|
| NUMERIC / DECIMAL |
NUMBER |
Scale and precision are preserved within Snowflake limitations. Negative scale is converted to scale 0 with adjusted precision. |
| REAL / FLOAT4 |
FLOAT |
|
| DOUBLE PRECISION / FLOAT8 |
FLOAT |
|
| MONEY |
FLOAT |
|
| BOOLEAN / BOOL |
BOOLEAN |
|
| CHARACTER / CHAR / BPCHAR |
TEXT |
|
| CHARACTER VARYING / VARCHAR |
TEXT |
Supported by default up to 16 MB. |
| TEXT |
TEXT |
Supported by default up to 16 MB. |
| BYTEA |
BINARY |
Supported by default up to 8 MB. |
| DATE |
DATE |
|
| TIME / TIME WITHOUT TIME ZONE |
TIME |
|
| TIME WITH TIME ZONE / TIMETZ |
TIMESTAMP_TZ |
|
| TIMESTAMP / TIMESTAMP WITHOUT TIME ZONE |
TIMESTAMP_NTZ |
|
| TIMESTAMP WITH TIME ZONE / TIMESTAMPTZ |
TIMESTAMP_LTZ |
|
| INTERVAL |
TEXT |
|
| JSON |
VARIANT |
Supported by default up to 16 MB. |
| JSONB |
VARIANT |
Supported by default up to 16 MB. |
| UUID |
TEXT |
|
| XML |
TEXT |
Supported by default up to 16 MB. |
| BIT |
TEXT |
|
| BIT VARYING / VARBIT |
TEXT |
|
| POINT |
TEXT |
|
| LINE |
TEXT |
|
| LSEG |
TEXT |
|
| BOX |
TEXT |
|
| PATH |
TEXT |
|
| POLYGON |
TEXT |
|
| CIRCLE |
TEXT |
|
| CIDR |
TEXT |
|
| INET |
TEXT |
|
| MACADDR |
TEXT |
|
| MACADDR8 |
TEXT |
|
| TSVECTOR |
TEXT |
|
| TSQUERY |
TEXT |
|
| PG_LSN |
TEXT |
|
For types with default size limits (8 MB / 16 MB) in this table, it is possible to raise these limits. For details, see [Oversized values](/user-guide/data-integration/openflow/connectors/postgres/about#label-postgres-oversized-values).
Any PostgreSQL data types not listed in this table are mapped to TEXT by default.
---
title: Openflow Connector for PostgreSQL: Iceberg table destinations
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/postgres/iceberg.md
section: Loading & Unloading Data
---
# Openflow Connector for PostgreSQL: Iceberg table destinations
Available to all accounts.
- [About Openflow Connector for PostgreSQL](/user-guide/data-integration/openflow/connectors/postgres/about)
- [Set up the Openflow Connector for PostgreSQL](/user-guide/data-integration/openflow/connectors/postgres/setup)
- [Openflow Connector for PostgreSQL: Data mapping](/user-guide/data-integration/openflow/connectors/postgres/data-mapping)
- [Data types for Apache Iceberg™ tables](/user-guide/tables-iceberg-data-types)
- [Snowflake storage for Apache Iceberg™ tables](/user-guide/tables-iceberg-internal-storage)
- [CREATE EXTERNAL VOLUME](/sql-reference/sql/create-external-volume)
The Openflow Connector for PostgreSQL supports writing to Snowflake-managed Apache %iceberg-tm% tables
as an opt-in destination format. Iceberg v2 and v3 are both supported. Setting **Table Storage Format** = `ICEBERG`
and choosing an **Iceberg Version** are the only connector-level changes required. The external volume,
catalog, and serialization policy are inherited from the Snowflake destination database defaults.
The Iceberg specification version is set via the **Iceberg Version** connector parameter, which
defaults to `3` for both Gen2 (Openflow UI wizard) and Gen1 (parameter context) connectors.
Storage can be either [Snowflake storage for Apache %iceberg-tm% tables](/user-guide/tables-iceberg-internal-storage)
(`EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'`) or an external volume in your cloud storage. When you use
Snowflake storage, no external cloud storage or IAM grants are required.
Existing connectors using standard tables aren't affected.
## Prerequisites
- **Openflow runtime**: An existing runtime to host the connector.
- **PostgreSQL source configured for CDC**: Logical replication enabled (`wal_level = logical`),
a publication created, and a user with replication privileges. For details, see
[Set up the Openflow Connector for PostgreSQL](/user-guide/data-integration/openflow/connectors/postgres/setup).
- **External volume in your cloud storage**: An external volume configured for Iceberg storage,
with USAGE granted to the connector's Snowflake role. See
[CREATE EXTERNAL VOLUME](/sql-reference/sql/create-external-volume). Not required when using
Snowflake storage (`EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'`).
- **Snowflake destination database**: An existing database configured with Iceberg parameters
(next section).
## Step 1: Configure the Snowflake destination database
Set the Iceberg defaults on the destination database. The connector reads these defaults at runtime
for external volume and serialization policy. The Iceberg specification version is configured
per-connector via the **Iceberg Version** parameter (see Step 3), not solely via the database-level
`ICEBERG_VERSION_DEFAULT`.
### Option A: Snowflake storage
When you use Snowflake storage, Snowflake stores and manages the Iceberg table files for you.
No external cloud storage or IAM grants are required.
```sql
CREATE DATABASE
EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'
STORAGE_SERIALIZATION_POLICY = ;
```
To configure an existing database:
```sql
ALTER DATABASE SET
EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'
STORAGE_SERIALIZATION_POLICY = ;
```
### Option B: External volume in your cloud storage
If you need to keep table files in your own cloud storage, configure the database with your
external volume:
```sql
CREATE DATABASE
EXTERNAL_VOLUME = ''
STORAGE_SERIALIZATION_POLICY = ;
```
To configure an existing database:
```sql
ALTER DATABASE SET
EXTERNAL_VOLUME = ''
STORAGE_SERIALIZATION_POLICY = ;
```
| Parameter |
Required |
Notes |
| EXTERNAL_VOLUME |
Yes |
The external volume for Iceberg file storage. |
| ICEBERG_VERSION_DEFAULT |
No |
`2` or `3`. Legacy fallback for older connector flows where the **Iceberg Version** parameter is
unset. New connectors set the version via the connector parameter (Step 3) and do not require this
database setting.
|
| STORAGE_SERIALIZATION_POLICY |
Yes |
`COMPATIBLE` produces Parquet files readable by external engines. `OPTIMIZED` enables
Snowflake-specific query optimizations. Choose based on your data query needs. For more information,
see [STORAGE_SERIALIZATION_POLICY](/sql-reference/parameters#storage-serialization-policy).
|
`CATALOG = 'SNOWFLAKE'` is set automatically by the connector on each CREATE ICEBERG TABLE
statement. Don't set it at the database level.
The base location for each table is auto-derived using the
[flat layout](/user-guide/tables-iceberg-managing-external-volumes#label-tables-iceberg-snowflake-managed-flat-layout):
`STORAGE_BASE_URL/database/schema/table_name.randomId/[data | metadata]/`.
No user configuration is needed.
If using an external volume in your cloud storage (Option B), grant the connector's Snowflake role
USAGE on the external volume:
```sql
GRANT USAGE ON EXTERNAL VOLUME TO ROLE ;
```
This step is not required for Snowflake storage.
## Step 2: Set Table Storage Format in the connector's parameter context
Set the **Table Storage Format** parameter to `ICEBERG` in the connector's destination parameter context.
The default is `STANDARD`.
For the full connector creation and configuration workflow, see
[Set up the Openflow Connector for PostgreSQL](/user-guide/data-integration/openflow/connectors/postgres/setup).
## Step 3: Set the Iceberg version
Set the **Iceberg Version** connector parameter to `2` or `3`. This controls the Iceberg specification
version used for type mapping (for example, JSON/JSONB maps to `variant` on v3 versus `string` on v2)
and the `ICEBERG_VERSION=` clause in CREATE ICEBERG TABLE DDL.
- **Gen2 (Openflow UI wizard)**: **Iceberg Version** is a required field when **Table Storage Format** =
`ICEBERG`, defaulting to `3`. This setting is immutable after the connector configuration is first
applied.
- **Gen1 (parameter context)**: The **Iceberg Version** parameter defaults to `3`. Review and change
to `2` if needed before starting the connector. Do not change this value after ingestion begins.
## Step 4: Start and verify
Start the connector as usual. After the initial snapshot completes, verify the destination tables
are Iceberg:
```sql
-- Confirm the table is Iceberg
SELECT GET_DDL('TABLE', '..');
-- Confirm the Iceberg version on the database
SHOW PARAMETERS LIKE 'ICEBERG_VERSION_DEFAULT' IN DATABASE ;
```
## Known limitations
- **Tri-Secret Secure accounts and Snowflake storage**: Accounts with Tri-Secret Secure
(TSS) enabled may be unable to create new Snowflake-managed Iceberg tables that use
[Snowflake storage for Apache %iceberg-tm% tables](/user-guide/tables-iceberg-internal-storage).
For details, see [Encryption](/user-guide/tables-iceberg-internal-storage#encryption).
- **Incompatible type change:** When the source column type changes to a type that maps to a
different Iceberg type, the table is marked as failed and requires a resnapshot. See
[Type mapping reference](#type-mapping-reference) for the complete source-to-Iceberg type
mapping.
- **Parameter change within the same Iceberg type:** The connector doesn't recognize parameter
changes within the same Iceberg type (for example, changing `decimal(10,2)` to `decimal(20,2)`).
The column retains its current Iceberg type.
- **TIMETZ offset not preserved**: Iceberg `timestamptz` stores only the UTC instant. PostgreSQL
TIMETZ values lose the original timezone offset when written to Iceberg tables.
- **Do not change Table Storage Format or Iceberg Version after the connector starts**:
The connector's **Table Storage Format** and **Iceberg Version** parameter should not be modified
after ingestion begins. Gen2 connectors enforce this by making **Iceberg Version** immutable after
first apply. Mixing settings across destination tables is not supported. To switch, follow the
steps in [Switching table storage format or Iceberg version](#switching-table-storage-format-or-iceberg-version).
## Type mapping reference
The following table shows how PostgreSQL types map to Snowflake standard and Iceberg destination types:
| PostgreSQL type |
Snowflake (Standard) |
Iceberg v3 |
Iceberg v2 |
| SMALLINT / INTEGER |
INT |
`long` |
`long` |
| BIGINT |
INT |
`long` |
`long` |
| REAL |
FLOAT |
`double` |
`double` |
| DOUBLE PRECISION |
FLOAT |
`double` |
`double` |
| NUMERIC(P,S) |
NUMBER(P,S) |
`decimal(P,S)` |
`decimal(P,S)` |
| BOOLEAN |
BOOLEAN |
`boolean` |
`boolean` |
| DATE |
DATE |
`date` |
`date` |
| TIME |
TIME |
`time` |
`time` |
| TIMESTAMP |
TIMESTAMP_NTZ |
`timestamp` |
`timestamp` |
| TIMESTAMPTZ |
TIMESTAMP_LTZ |
`timestamptz` |
`timestamptz` |
| TIMETZ |
TIMESTAMP_TZ |
`timestamptz` |
`timestamptz` |
| TEXT / VARCHAR / CHAR |
TEXT |
`string` |
`string` |
| BYTEA |
BINARY |
`binary` |
`binary` |
| JSON / JSONB |
VARIANT |
`variant` |
`string` |
| UUID |
TEXT |
`string` |
`string` |
Source types not listed in the table are mapped to TEXT on standard tables and `string` on Iceberg
tables.
## Switching table storage format or Iceberg version
Switching between Standard and Iceberg, or between Iceberg v2 and v3, requires recreating the
connector. Follow these steps:
1. Stop the connector.
2. Delete the process group in Openflow.
3. Manually clean up the destination database (drop the replicated schemas/tables, or use a new
database).
4. Reimport the connector with the new **Table Storage Format** and select the target **Iceberg Version**
when configuring the connector.
This ensures all connector state is correctly cleaned up within Openflow. The new connector performs
a fresh snapshot into the destination.
## Upgrading an existing connector to use Iceberg Version pinning
Gen2 connector version `2026.7.21` and Gen1 connector version `0.60.0` introduce the
**Iceberg Version** parameter. If you are upgrading from an earlier connector version (for example,
Gen1 `0.56.0` to `0.60.0` or later), a new **Iceberg Version** field appears that you must configure
to match your existing destination tables.
1. Stop the connector.
2. [Upgrade the runtime](/user-guide/data-integration/openflow/manage#label-openflow-upgrading-a-runtime)
to version `2026.7.21` or later.
3. [Upgrade the connector](/user-guide/data-integration/openflow/manage#upgrade-a-connector)
in place (Gen2: to version `2026.7.21` or later; Gen1: to version `0.60.0` or later).
4. Set the **Iceberg Version** parameter to match your existing destination tables:
- **Gen2 (Openflow UI wizard)**: After upgrading, open the connector configuration wizard.
The **Destination details** step now includes a required **Iceberg Version** field, defaulting
to `3`. If your existing destination tables are Iceberg v2, change it to `2` before applying.
This choice is locked after first apply and cannot be changed later.
- **Gen1 (parameter context)**: The **Iceberg Version** parameter defaults to `3` after the flow
upgrade. If your existing destination tables are Iceberg v2, change it to `2` before starting
the connector.
5. Start the connector.
Selecting an **Iceberg Version** that doesn't match your existing destination tables can cause
type-mapping errors or DDL failures. Always verify the version of your existing tables before
choosing a value.
## References
- [CREATE EXTERNAL VOLUME](/sql-reference/sql/create-external-volume)
- [Data types for Apache Iceberg tables](/user-guide/tables-iceberg-data-types)
- [ALTER DATABASE](/sql-reference/sql/alter-database)
- [STORAGE_SERIALIZATION_POLICY](/sql-reference/parameters#storage-serialization-policy)
- [Set up the Openflow Connector for PostgreSQL](/user-guide/data-integration/openflow/connectors/postgres/setup)
---
title: Openflow Connector for PostgreSQL: PostgreSQL 17+ failover slot support
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/postgres/failover.md
section: Loading & Unloading Data
---
# %postgresql%: PostgreSQL 17+ failover slot support
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
- [About Openflow Connector for PostgreSQL](/user-guide/data-integration/openflow/connectors/postgres/about)
- [Set up the Openflow Connector for PostgreSQL](/user-guide/data-integration/openflow/connectors/postgres/setup)
- [Openflow Connector for PostgreSQL Maintenance](/user-guide/data-integration/openflow/connectors/postgres/maintenance)
Requires runtime-extensions 2026.7.16.14 or later. Applies to PostgreSQL 17 and later only.
On PostgreSQL 17 and later, the connector creates its logical replication slot with `failover=true`.
The slot is synchronized to standbys by PostgreSQL and survives:
- A primary failover: replication resumes from the new primary.
- A major-version upgrade through `pg_upgrade` (for example, 17 → 18), provided the slot is caught up and not conflicting or invalidated at upgrade time.
The connector must be pointed at the primary to get failover protection.
PostgreSQL does not allow `failover=true` on slots created against a standby.
If the JDBC URL resolves to a read replica, the slot on that node will not have failover protection.
Version is auto-detected. No new connector property is required.
## Limitations
- PostgreSQL 16 and earlier versions are unchanged. Failover protection is not available on those versions as this is a native PostgreSQL feature that does not exist before PostgreSQL 17.
- Upgrading the connector alone does not retrofit failover onto an existing slot. The connector only sets `failover=true` at slot creation time. To gain failover protection on an existing deployment, see [Retrofit failover protection on an existing deployment](#label-retrofit-failover-protection-on-an-existing-deployment).
- Delivery is at-least-once across a failover. The connector de-duplicates on reconnect.
- The connector does not validate the PostgreSQL configuration described in this topic. If it is missing, `failover=true` is inert and the slot will not survive a failover.
## Required PostgreSQL configuration
The following configuration must be set by the user. The connector can't set these.
- Point the JDBC URL at the primary's writer endpoint so the connector reconnects to the new primary after a failover.
To confirm the connector is connected to the primary, run the following against the connected database.
The result must be `false` (false = primary, true = standby):
```sql
SELECT pg_is_in_recovery();
```
- On the primary, set `synchronized_standby_slots` to list the physical replication slot names of the failover-candidate standbys.
Without it, the failover slot may advance faster than the standby receives WAL:
```ini
synchronized_standby_slots = ''
```
- On each failover-candidate standby, set the following:
- `wal_level = logical` (already required on the primary): set this on standbys so it takes effect on promotion.
- `sync_replication_slots = on`
- `hot_standby_feedback = on`
- `primary_conninfo` must include `dbname=` (PostgreSQL 17 requirement).
- `primary_slot_name` must reference a physical slot listed in `synchronized_standby_slots` on the primary.
## Retrofit failover protection on an existing deployment
To gain failover protection on an existing deployment, there are two paths available: with re-snapshot or without re-snapshot. Follow the procedure for the path you choose.
### Retrofit failover protection without re-snapshotting
Process all existing FlowFiles in the existing connector before you start the new one. The existing connector should have no FlowFiles in queue and should have all processors stopped. Any writes between stopping the existing connector and starting the new connector will not be delivered to Snowflake. Once a slot is dropped, PostgreSQL can't replay old WAL. If you can't guarantee a write-free window, use [Retrofit failover protection with re-snapshot](#label-retrofit-with-re-snapshot) instead.
Do not add new tables while the connector is in incremental mode. New tables will not be snapshotted.
1. Ensure the connector is fully caught up before proceeding. The `confirmed_flush_lsn` should match `pg_current_wal_lsn()`:
```sql
SELECT slot_name, confirmed_flush_lsn, pg_current_wal_lsn() FROM pg_replication_slots;
```
2. Pause writes on the source tables by running the following against your PostgreSQL source database. Replace `` and `` with your values:
```sql
REVOKE INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA FROM ;
```
3. Wait until all queues in the connector are empty.
4. Stop the connector and disable all Controller Services.
5. Drop the replication slot. The slot name is in the `CaptureChangePostgreSQL` processor state under `replication.slot.name`, or in `pg_replication_slots`:
```sql
SELECT pg_drop_replication_slot('');
```
6. In the parameter context, set **Ingestion Type** to `incremental`. To set this, go to **Ingestion Parameters** → **Ingestion Type** → edit the value to `incremental`.
7. Install a new connector. In the top menu, drag down **Import from registry**. Select **postgresql** from **Flow**, select **Keep existing Parameter contexts** to reuse the same parameter context as the existing connector, then click **Import**.
8. Start the connector fully, including all Processors and Controller Services.
9. Verify the new slot was created with `failover=true` by running the following against your PostgreSQL source database:
```sql
SELECT slot_name, failover FROM pg_replication_slots;
```
10. Resume writes:
```sql
GRANT INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA TO ;
```
11. Optionally, delete the existing connector to prevent accidentally resuming it.
Once all existing tables are in **Incremental Replication**, change **Ingestion Type** from `incremental` to `full`. To verify, in Snowsight go to **Ingestion** → **Openflow** → **Connector Observability**, select your connector, and confirm all tables show **Incremental Replication** in the **Replication Phase** column.
### Retrofit failover protection with re-snapshot
Individual change events that occur on the source between dropping the slot and completing the re-snapshot are not delivered as CDC events. They are absorbed into the snapshot's final row values.
1. In the connector, remove all tables from replication by clearing the **Included Table Names** and **Included Table Regex** parameters.
2. Wait until all queues in the connector are empty.
3. Stop the connector, including all Processors and Controller Services.
4. Drop the replication slot. The slot name is in the `CaptureChangePostgreSQL` processor state under `replication.slot.name`, or in `pg_replication_slots`:
```sql
SELECT pg_drop_replication_slot('');
```
5. Install a new connector. In the top menu, drag down **Import from registry**. Select **postgresql** from **Flow**, select **Keep existing Parameter contexts** to reuse the same parameter context as the existing connector, then click **Import**.
6. Re-add tables to **Included Table Names**.
7. Start the new connector fully, including all Processors and Controller Services. The connector creates a new replication slot with `failover=true` and snapshots the tables fresh.
Verify the new slot has `failover=true`:
```sql
SELECT slot_name, failover FROM pg_replication_slots;
```
## Additional step when running pg_upgrade
This step is only needed during a major-version upgrade. It is not part of the general setup.
Ensure the connector is caught up and the slot is not conflicting or invalidated before starting `pg_upgrade`:
```sql
SELECT slot_name, conflicting, invalidation_reason FROM pg_replication_slots;
```
---
title: Openflow Connector for PostgreSQL: Set up incremental replication without snapshots
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/postgres/incremental-replication.md
section: Loading & Unloading Data
---
# %postgresql%: Set up incremental replication without snapshots
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
- [About Openflow](/user-guide/data-integration/openflow/about)
- [Manage Openflow](/user-guide/data-integration/openflow/manage)
- [Openflow connectors](/user-guide/data-integration/openflow/connectors/about-openflow-connectors)
- [Set up the Openflow Connector for PostgreSQL](/user-guide/data-integration/openflow/connectors/postgres/setup)
- [Openflow Connector for PostgreSQL: Data mapping](/user-guide/data-integration/openflow/connectors/postgres/data-mapping)
You can configure the %postgresql% connector to immediately replicate incremental changes for newly added tables, bypassing snapshots. Use incremental load to continue replication without snapshotting every table again when you reinstall the connector over previously replicated data.
To enable incremental replication in a new connector instance:
1. Set up the connector as described in [Set up the Openflow Connector for PostgreSQL](/user-guide/data-integration/openflow/connectors/postgres/setup).
2. In the `PostgreSQL Ingestion Parameters` context, set the `Ingestion Type` parameter to `incremental`.
## Enable incremental replication without snapshots
To enable incremental replication on an existing connector:
1. sign in to %sf-web-interface-link%.
2. in the navigation menu, select **Ingestion** %raa% **Openflow**.
3. In the **Openflow** pane select the **Runtimes** tab.
4. Select the runtime containing the connector.
5. Select the connector.
6. In the `Ingestion Parameters` context, specify `Ingestion Type` = `incremental`.
7. Add new replication tables. These tables immediately switch to their incremental load.
To return to replicating tables with the snapshot load, change **Ingestion Type** from `incremental` to `full`.
# Usage notes
- Changing the value of **Ingestion Type** does not impact any tables that have begun replicating data.
Tables currently in the snapshot phase continue until the snapshot load is complete.
- While **Ingestion Type** is set to `incremental`, new tables added to the list of replicated tables bypass the snapshot phase.
This includes new tables added to the source database that match the `Included Table Regex` parameter.
Ensure that the ingestion type is set to `incremental` to bypass the snapshot phase.
Connectors should only remain in `incremental` mode as long as required as it bypasses snapshots.
Once customer needs for incremental updates have been satisfied the connector should be returned to `full` mode.
- For tables that bypass snapshot load, the connector creates a destination table in Snowflake,
by executing `CREATE TABLE IF NOT EXISTS`, only if no destination table already exists.
Tables going through the snapshot require that no destination table exist.
---
title: Openflow Connector for Salesforce Bulk API: Configure the connector
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/configure-connector.md
section: Loading & Unloading Data
---
# %salesforcebulkapiof%: Configure the connector
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
- [About the Openflow Connector for Salesforce Bulk API](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/about)
- [Openflow Connector for Salesforce Bulk API: Set up Snowflake](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/setup-snowflake)
- [Openflow Connector for Salesforce Bulk API: Set up Salesforce](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/setup-salesforce)
- [Openflow Connector for Salesforce Bulk API: Iceberg table destinations](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/iceberg)
- [Openflow Connector for Salesforce Bulk API: Salesforce formula fields](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/formula-fields)
- [Monitor the Openflow Connector for Salesforce Bulk API](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/monitor)
- [Troubleshooting the Openflow Connector for Salesforce Bulk API](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/troubleshoot)
This topic describes the steps to configure the %salesforcebulkapiof%.
## Install the connector
Follow these steps to install the %salesforcebulkapiof% in an Openflow runtime:
1. Navigate to the Openflow **Overview** page. In the **Featured connectors** section, select **View more connectors**.
2. On the Openflow connectors page, find **Openflow connector for Salesforce Bulk API** and select **Install**.
3. In the **Select runtime** dialog, select your runtime from the **Available runtimes** drop-down.
The Openflow canvas appears with the connector process group added to it.
## Configure the connector
To configure the connector, perform the following steps:
1. Right-click on the imported process group and select **Parameters**.
2. Populate the required parameter values as described in the table below.
| Parameter |
Description |
| Column Removal Strategy |
Defines the strategy to adopt when a column should be removed in the destination table based on the latest received schema. Three possible values: `Drop Column`, `Rename Column`, `Ignore Column`.
- `Drop Column`: Drop the column from the Snowflake table.
- `Rename Column`: Rename the column in the Snowflake table.
- `Ignore Column`: Ignore the column, leaving it as is in the Snowflake table.
|
| Connected App Key |
The private key used for JWT Bearer Flow authentication with Salesforce. Copy-paste the content of the `private.key` file generated during the [Salesforce setup](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/setup-salesforce). This private key must correspond to the public certificate (`public.crt`) uploaded to the external client app in Salesforce. You can also use the next parameter to upload the private key file instead. |
| Connected App Key File |
Upload the `private.key` file by selecting the **Reference asset** checkbox, then upload the file as an asset and select the asset as the value for the parameter. This is an alternative to pasting the key content in the **Connected App Key** parameter. |
| Connected App Key Password |
Password set on the private key file during the [Salesforce setup](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/setup-salesforce) steps. |
| Destination Database |
Name of the database in Snowflake where the Salesforce data will be replicated. The database must exist before starting the connector. |
| Destination Schema |
Name of the schema, in the database above, into which the connector will create tables for the Salesforce data to be added. The schema must exist before starting the connector. |
| Enable Capture Blob Fields |
If set to `true`, fields of type `base64` (binary fields such as `Attachment.Body` and `ContentVersion.VersionData`) are fetched by the connector. The objects that contain blob fields must be listed in the **Special Objects Filter** parameter (Non-Bulk API path). Default: `false`. See [Configure blob field ingestion](#blob-fields) for details. |
| Enable Journal Tables |
If set to `true`, a `JOURNAL_` table is created for each synced object that has a `SystemModstamp` or `LastModifiedDate` field. All changes are appended to the journal table, providing a full history of modifications. This is in addition to the main table that contains the merged data for the object. If a full reload occurs for a given object type, its journal table is also recreated. Default: `false`. |
| Enable Merge Metrics |
If set to `true`, the connector runs an additional query to count records that are added, updated, deleted, or restored during replication. The additional query uses the **Snowflake Warehouse** and applies only to objects that include the `IsDeleted` field. The connector writes the counts to logs in the event table. Default: `false`. See [Monitor the Openflow Connector for Salesforce Bulk API](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/monitor). |
| Enable Views Creation |
If set to `true`, a view named `_FORMULA_VW` is created for each synced object that contains formula fields. The view translates supported Salesforce formula expressions into Snowflake SQL, allowing you to query formula results directly without replicating formula field values from Salesforce. See [](#salesforce-formula-fields) for details. Default: `false`. |
| Filter |
Comma-separated list of objects to replicate from Salesforce, or regular expression to apply against all existing objects. The filter is case-insensitive, meaning that a filter set to `account` would match the object type `Account`. Example: `Account, Opportunity, Contact`.
If left empty, all objects will be replicated. This is not recommended as there are usually thousands of objects in a Salesforce instance.
|
| Incremental Offload |
Whether the processor should perform incremental offload. If `true`, the processor will only fetch the records that have been modified since the last query job submission by using a `WHERE` clause on the appropriate timestamp field. If `false`, all records will be fetched at every execution of the connector. |
| Initial Load Chunking |
If set to a value other than `NONE`, the initial data load will be split into multiple jobs based on this interval. On the first run for an object, the connector will query Salesforce to find the oldest record and use that as the starting point. Each subsequent job will query the next time chunk until caught up to the current time. Set to one of: `NONE`, `MONTHLY`, `QUARTERLY`, `YEARLY`.
This is useful for large datasets where loading all historical data in a single query may time out, exceed API limits, or exceed the storage size of the content repository of the runtime. After catching up, the processor continues with normal incremental offload behavior.
|
| Iceberg Version |
Only applicable when **Table Storage Format** is set to `ICEBERG` (preview). Specifies the Iceberg version for the destination Iceberg table. Supported values are `2` and `3`. Default: `3`. Don't change this value after ingestion begins. For setup instructions, see [Openflow Connector for Salesforce Bulk API: Iceberg table destinations](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/iceberg). |
| OAuth2 Audience |
Audience to set in the JWT token. Set to `https://login.salesforce.com` for production environments or `https://test.salesforce.com` for sandboxes and test environments. |
| OAuth2 Client ID |
Should be set to the **Consumer Key** value retrieved during the Salesforce Setup steps. |
| OAuth2 Subject |
Should be set to the username of an admin-approved user on whose behalf the application interacts with Salesforce APIs. |
| OAuth2 Token Endpoint URL |
Endpoint to negotiate tokens via the JWT Bearer Flow. Example: `https://myCompany.my.salesforce.com/services/oauth2/token`. |
| Object Fields Filter JSON |
A JSON specifying which fields and field patterns should be included or excluded, per Salesforce object. Takes the form of an array with one item per object.
Example 1: This will include all fields that end with 'name' in the 'Account' Salesforce object:
`[ {"objectType":"Account", "includedPattern":".*name"} ]`
Example 2: This will include the fields Id, Name, and Revenue in the 'Account' Salesforce object:
`[ {"objectType":"Account", "included": ["Id", "Name", "Revenue"]} ]`
`excluded` and `excludedPattern` are also available for configuring the filters.
|
| Object Identifier Resolution |
Determines whether schema/table/column names are treated as case-sensitive or case-insensitive. One of: `CASE_INSENSITIVE` / `CASE_SENSITIVE`.
Changing this parameter value will require clearing the state and doing a full reload of all objects.
|
| Removed Column Name Suffix |
Suffix added to the column name when the parameter **Column Removal Strategy** is set to `Rename Column`. Default: `__deleted`. |
| Run Schedule |
Frequency at which the connector will check for updates in Salesforce for configured objects via the **Filter** parameter. Default: `15 minutes`. |
| Salesforce Instance |
Hostname of the Salesforce instance including the domain name. Do not include the protocol prefix (`https://`). For example, use `myCompany.my.salesforce.com`. |
| Snowflake Account Identifier |
Snowflake account name formatted as `[organization-name]-[account-name]` where data will be persisted. Example: `PM-CONNECTORS`. |
| Snowflake Username |
The name of the service user that the connector uses to connect to Snowflake. The service user is required only when using the `KEY_PAIR` authentication strategy (Openflow BYOC only). |
| Snowflake Private Key |
The RSA Private Key that the connector uses for authentication to Snowflake, formatted according to PKCS8 standards and including standard PEM headers and footers. The header line starts with `-----BEGIN PRIVATE`. This is required only when using the `KEY_PAIR` authentication strategy (Openflow BYOC only).
You may also use the next parameter to upload the private key to the Openflow runtime instead.
|
| Snowflake Private Key File |
The file containing the RSA Private Key that the connector uses for authentication to Snowflake, formatted according to PKCS8 standards and including standard PEM headers and footers. The header line starts with `-----BEGIN PRIVATE`. Required only when using the `KEY_PAIR` authentication strategy (Openflow BYOC only).
Select the **Reference asset** checkbox to upload the private key file and store it securely in the Openflow runtime.
|
| Snowflake Private Key Password |
The password associated with the Snowflake Private Key File (if encrypted). This is required only when using the `KEY_PAIR` authentication strategy (Openflow BYOC only). |
| Snowflake Role |
Name of the execute-as role used during query execution. When using `SNOWFLAKE_MANAGED`, this is the execute-as role for Openflow runtimes. When using `KEY_PAIR` (Openflow BYOC only), this is the role assigned to the specified Snowflake username. |
| Snowflake Authentication Strategy |
Authentication strategy for the connector to connect to Snowflake.
Using `SNOWFLAKE_MANAGED` (default) uses the Snowflake managed token associated with the runtime's execute-as role. If using Openflow BYOC, you can also use `KEY_PAIR` to specify a specific user and role via a custom Key Pair.
|
| Snowflake Warehouse |
The Snowflake warehouse used to run queries. |
| Special Objects Filter |
Comma-separated list of objects to offload from Salesforce (using direct API access), or regular expression to apply against all existing objects. The filter is case-insensitive, meaning that a filter set to `account` would match the object type `Account`.
This filter should only be used for objects that are **not** supported by the Salesforce Bulk API, such as knowledge data. This parameter should not overlap with the parameter **Filter**.
Example: `Knowledge.*`
|
| Table Storage Format |
The storage format of the destination Snowflake table. Use `STANDARD` for standard Snowflake tables. The `ICEBERG` option, which writes to Apache Iceberg tables, is a preview feature. Default: `STANDARD`. Don't change this value after ingestion begins. For setup instructions, see [Openflow Connector for Salesforce Bulk API: Iceberg table destinations](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/iceberg). |
## Verify the Salesforce connection
Before enabling and starting the connector, Snowflake recommends verifying that the Salesforce authentication is properly configured. The **Verification** feature on controller services lets you test the connection without starting the full connector flow.
The **JWT Bearer OAuth2 Access Token Provider** controller service depends on two other controller services that must be enabled first: the **Salesforce Private Key Service** and the **Web Client Service Provider**.
1. Double-click the connector process group to open it.
2. Right-click on an empty area of the canvas and select **Controller Services**.
3. Enable the **Salesforce Private Key Service** and the **Web Client Service Provider** services.
4. Locate the **JWT Bearer OAuth2 Access Token Provider** service in the list.
5. Click the **Verification** button for the service. A dialog opens where you can provide property overrides. You can ignore this and click **Verify** directly.
6. If everything is configured properly, the **Acquire token** step shows a green checkmark indicating success. This confirms the connector can authenticate with Salesforce and obtain an access token. You can proceed to the next step to run the connector.
7. If verification fails, review the error message and check the following:
- The **OAuth2 Client ID** parameter matches the **Consumer Key** from the external client app in Salesforce.
- The private key corresponds to the certificate uploaded to the external client app.
- The **OAuth2 Subject** user is authorized for the external client app (see [](#salesforce-approve-client-app)).
- The **OAuth2 Token Endpoint URL** uses the correct Salesforce instance hostname.
- The **OAuth2 Audience** is set to the correct value: `https://login.salesforce.com` for production or `https://test.salesforce.com` for sandboxes.
For detailed troubleshooting, see [Troubleshooting the Openflow Connector for Salesforce Bulk API](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/troubleshoot).
## Run the connector
Follow these steps to start the connector and begin replicating data from Salesforce to Snowflake:
1. Right-click on an empty area in the canvas and select **Enable all Controller Services**.
2. Right-click on the connector process group and select **Start**.
## Manage object replication
After the connector has been started and objects have been replicated, you can add new objects or remove existing objects from replication.
### Add new objects to replication
To add a new object to replication, update the **Filter** parameter (or **Special Objects Filter** parameter, if applicable) with the new object names. You do not need to stop the connector. The new object is replicated at the next scheduled execution.
For example, if the current **Filter** value is `Account, Opportunity` and you want to add the `Contact` object, change the value to `Account, Opportunity, Contact`.
### Remove objects from replication
Removing an object from replication requires stopping the connector and cleaning up both the connector state and the destination table in Snowflake:
1. Stop all processors in the flow by right-clicking on the connector process group and selecting **Stop**.
2. Ensure that no in-flight FlowFiles are being processed.
3. Right-click on the canvas and select **Parameters**, then remove the object name from the **Filter** parameter (or the **Special Objects Filter** parameter, if applicable).
4. Right-click on the canvas and select **Disable all controller services**.
5. Go to **Controller services** and open the state of the controller service named **Salesforce Bulk Jobs State**.
6. Select the trash icon next to the object type you removed to delete its state entry.
7. Right-click on the canvas and select **Enable all controller services**, then start all processors to resume the connector.
8. If applicable, drop the corresponding table from the Snowflake destination database to clean up the previously replicated data. For example:
```sql
DROP TABLE ..;
```
## Configure blob field ingestion
The Salesforce Bulk API 2.0 does not support binary (base64-encoded) fields. The
connector handles these fields through a dedicated Non-Bulk API path that uses the
Salesforce REST Query API. Objects with blob fields must be listed in the **Special
Objects Filter** parameter so they are routed to this path.
To enable blob field ingestion, set **Enable Capture Blob Fields** to `true` in the
connector parameters.
## Next steps
- To monitor replication activity and merge metrics, see [Monitor the Openflow Connector for Salesforce Bulk API](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/monitor).
- To diagnose connector issues, see [Troubleshooting the Openflow Connector for Salesforce Bulk API](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/troubleshoot).
---
title: Openflow Connector for Salesforce Bulk API: Iceberg table destinations
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/iceberg.md
section: Loading & Unloading Data
---
# %salesforcebulkapiof%: Iceberg table destinations
Available to all accounts.
- [About the Openflow Connector for Salesforce Bulk API](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/about)
- [Openflow Connector for Salesforce Bulk API: Set up Snowflake](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/setup-snowflake)
- [Openflow Connector for Salesforce Bulk API: Configure the connector](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/configure-connector)
- [Openflow Connector for Salesforce Bulk API: Salesforce formula fields](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/formula-fields)
- [Snowflake storage for Apache Iceberg™ tables](/user-guide/tables-iceberg-internal-storage)
- [CREATE EXTERNAL VOLUME](/sql-reference/sql/create-external-volume)
The %salesforcebulkapiof% supports writing to Snowflake-managed Apache %iceberg-tm% tables as an opt-in destination format. The connector supports Iceberg v2 and v3. Existing connectors that write to standard Snowflake tables aren't affected.
To use Iceberg table destinations, configure the Snowflake destination database with Iceberg defaults, then set the connector's **Table Storage Format** parameter to `ICEBERG` and choose an **Iceberg Version** before the connector creates destination tables.
## Prerequisites
Before you begin:
- Complete the steps in [Openflow Connector for Salesforce Bulk API: Set up Snowflake](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/setup-snowflake).
- Choose either Snowflake storage or an external volume in your cloud storage for the Iceberg table files.
## Configure the destination database
Configure the destination database with Iceberg defaults before starting the connector. The connector reads the database defaults for the external volume and storage serialization policy when it creates Iceberg tables.
### Option A: Snowflake storage
To use [Snowflake storage for Apache %iceberg-tm% tables](/user-guide/tables-iceberg-internal-storage), configure the database with `EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'`:
```sql
ALTER DATABASE SET
EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'
STORAGE_SERIALIZATION_POLICY = ;
```
With Snowflake storage, Snowflake stores and manages the Iceberg table files. You don't need to configure external cloud storage or grant access to an external volume.
### Option B: External volume in your cloud storage
To store the Iceberg table files in your cloud storage, configure the database with the external volume:
```sql
ALTER DATABASE SET
EXTERNAL_VOLUME = ''
STORAGE_SERIALIZATION_POLICY = ;
```
`COMPATIBLE` produces Parquet files readable by external engines. `OPTIMIZED` enables Snowflake-specific query optimizations. Choose based on your data query needs. For more information, see [STORAGE_SERIALIZATION_POLICY](/sql-reference/parameters#storage-serialization-policy).
Grant the connector role `USAGE` on the external volume:
```sql
GRANT USAGE ON EXTERNAL VOLUME TO ROLE ;
```
This grant isn't required when you use Snowflake storage (`EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'`).
`CATALOG = 'SNOWFLAKE'` is set automatically by the connector when it creates Iceberg tables. Don't set it at the database level.
## Configure the connector
Set the following destination parameters before the connector creates destination tables:
- **Table Storage Format**: Set to `ICEBERG`. The default is `STANDARD`.
- **Iceberg Version**: Set to `2` or `3`. The default is `3`.
Don't change **Table Storage Format** or **Iceberg Version** after ingestion begins. To switch between standard and Iceberg destinations, or between Iceberg v2 and v3, recreate the connector and perform a fresh load into new destination tables.
For the full connector configuration workflow, see [Openflow Connector for Salesforce Bulk API: Configure the connector](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/configure-connector).
## Verify Iceberg table creation
After the initial load completes, verify that the destination tables were created as Iceberg tables:
```sql
SELECT GET_DDL('TABLE', '..');
```
The returned DDL should include `ICEBERG TABLE`.
You can also verify the database-level Iceberg defaults:
```sql
SHOW PARAMETERS LIKE 'EXTERNAL_VOLUME' IN DATABASE ;
SHOW PARAMETERS LIKE 'STORAGE_SERIALIZATION_POLICY' IN DATABASE ;
```
## Limitations and behavior
- **Tri-Secret Secure accounts and Snowflake storage**: Accounts with Tri-Secret Secure (TSS) enabled may be unable to create new Snowflake-managed Iceberg tables that use [Snowflake storage for Apache %iceberg-tm% tables](/user-guide/tables-iceberg-internal-storage). For details, see [Encryption](/user-guide/tables-iceberg-internal-storage#encryption).
- **Collation isn't supported on Iceberg tables**: When **Table Storage Format** is set to `ICEBERG`, the connector doesn't include `DEFAULT_DDL_COLLATION` in table creation parameters.
- **Salesforce time fields use microsecond precision**: Salesforce fields of type `time` are created as `TIME(6)` on Iceberg tables.
- **Formula views can be created over Iceberg tables**: When **Enable Views Creation** is set to `true`, the connector can create formula views over Iceberg base tables. The formula field limitations described in [Openflow Connector for Salesforce Bulk API: Salesforce formula fields](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/formula-fields) still apply.
- **Hard deletes aren't supported**: The connector still represents deleted Salesforce records with the `isDeleted` column rather than hard-deleting rows from the destination table.
---
title: Openflow Connector for Salesforce Bulk API: Salesforce formula fields
source: https://docs.snowflake.cn/en/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/formula-fields.md
section: Loading & Unloading Data
---
# %salesforcebulkapiof%: Salesforce formula fields
This feature is not available in the People's Republic of China.
Snowflake connectors are supported in every region where Snowflake Openflow is available.
[Openflow Snowflake deployments](/user-guide/data-integration/openflow/about-spcs) are available to all accounts in AWS, Azure, and GCP Commercial Regions.
[Snowflake Openflow on BYOC deployments](/user-guide/data-integration/openflow/about-byoc) are available to all accounts in AWS Commercial Regions only ([](#label-na-general-regions)).
This connector is subject to the [Snowflake Connector Terms](https://www.snowflake.cn/legal/snowflake-connector-terms/).
- [About the Openflow Connector for Salesforce Bulk API](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/about)
- [Openflow Connector for Salesforce Bulk API: Configure the connector](/user-guide/data-integration/openflow/connectors/salesforce-bulk-api/configure-connector)
This topic describes how the %salesforcebulkapiof% translates Salesforce formula
fields into Snowflake SQL views, including supported functions and limitations.
## How formula views work
When **Enable Views Creation** is set to `true`, the connector performs the following for each object that has formula fields:
1. Retrieves the formula expressions from the Salesforce object metadata via the Describe API.
2. Parses each formula expression and translates it into equivalent Snowflake SQL.
3. Generates a `CREATE OR REPLACE VIEW` statement that combines non-formula columns from the base table with the translated formula expressions as computed columns.
4. Runs the DDL against Snowflake to create or update the view.
The resulting view is named `_FORMULA_VW`. For example, the `Account` object produces a view named `ACCOUNT_FORMULA_VW`. You can query this view to obtain formula field values alongside the replicated data.
The view is automatically updated whenever the connector detects schema changes in the source object, ensuring that formula definitions stay in sync with Salesforce.
## Cross-object formula fields
Salesforce formulas can reference fields from related objects using relationship traversal (for example, `Account.Owner.Name`). The connector supports these cross-object references by generating `LEFT JOIN` clauses in the view definition. Each relationship traversal produces a join to the corresponding related table in Snowflake.
For cross-object formulas to work correctly, the related objects must also be replicated by the connector. If a related object is not being synced, the formula columns that reference it are replaced with a typed `NULL` in the view. The remaining formula columns continue to compute normally. The affected columns have a `LOOKUP_NOT_SYNCED` comment. Once the referenced objects are added to replication and synced, the view is automatically rebuilt on the next connector run.
## Chained formula fields
Formula fields that reference other formula fields (chained formulas) are supported. The connector resolves the dependency graph before SQL generation and expands each referenced formula field's expression into the referencing formula's AST. Multi-hop chains are fully expanded. If a dependency cannot be translated, the dependent field also returns `NULL` and the column comment indicates `FORMULA_CHAIN_NOT_SUPPORTED`.
## Formula view column comments
Each formula column in the generated view includes a SQL `COMMENT` annotation:
- For successfully translated formulas, the comment contains the original Salesforce formula expression.
- For formulas that could not be translated, the comment contains the failure reason code followed by the original Salesforce formula expression, separated by a colon (for example, `FUNCTION_NOT_SUPPORTED: IMAGE(url, 'alt')`).
You can inspect these comments by running `DESCRIBE VIEW |