CaptureChangePostgreSQL 2025.3.28.13-SNAPSHOT¶
BUNDLE¶
com.snowflake.openflow.runtime | runtime-database-cdc-processors-nar
DESCRIPTION¶
Reads CDC events from a PostgreSQL database.The processor continuously reads events arriving in the stream, filtering for those related to tables providedby the TableStateService, and discarding the rest. After the current batch of events is processed, the processorconfirms the replication slot position back to PostgreSQL, letting it trim the WAL.The processor outputs two types of flow files: DDLs, containing the initial schema of a table, and then everytime its schema changes, and DMLs, with records representing changes to data in the table.One flow file always represents data related to a single table.The DDL with the schema is written to the FlowFile content as a JSON object, in a form such as:{ “columns”: [ { “name”: “<columnName>”, “type”: “<snowflakeType>”, “nullable”: <true|false>, “scale”: <scale>, “precision”: <precision> }, … ], “primaryKeys”: [“<primaryKey1>”, “<primaryKey2>”, …]}The DML records are structured as:{ “primaryKeys”: { “<column>”: <value>, … }, “payload”: { “<column>”: <value>, … }, “metadata”: { “<column>”: <value>, … }}
INPUT REQUIREMENT¶
FORBIDDEN
Supports Sensitive Dynamic Properties¶
false
PROPERTIES¶
Property |
Description |
|---|---|
Column Filter Store |
Service storing per-table column filtering settings. |
JDBC Driver Location |
Comma-separated list of files/folders and/or URLs containing the driver JAR and its dependencies (if any). For example ‘/var/tmp/postgresql-java-client-42.7.5.jar’ |
JDBC URL |
JDBC URL of the database connection, ie. jdbc:postgresql://localhost:5432/postgres |
Max Batch Size |
The maximum number of records to process in a single iteration |
Max Batch Wait Time |
The maximum time to wait for data to appear in the CDC stream. |
Password |
Password to access the PostgreSQL database |
Publication Name |
The name of the CDC publication to read from. |
Record Writer |
The Record Writer is used for serializing DML events |
SSL Context Service |
SSL Context Service supporting encrypted socket communication |
SSL Mode |
Whether to use and enforce SSL when connecting to PostgreSQL |
Table State Store |
The shared store holding the state of replicated tables. |
Username |
Username to access the PostgreSQL database |
STATE MANAGEMENT¶
SCOPES |
DESCRIPTION |
|---|---|
CLUSTER |
Information such as a ‘pointer’ to the current CDC event in the database is stored by this processor, such that it can continue from the same location if restarted, and the name of the replication slot created in PostgreSQL. |
RELATIONSHIPS¶
NAME |
DESCRIPTION |
|---|---|
success |
Successfully created FlowFile from CDC stream events |
WRITES ATTRIBUTES¶
NAME |
DESCRIPTION |
|---|---|
source.schema.name |
Name of the schema of the table from which an event originated |
source.table.name |
Name of the table from which an event originated |
cdc.event.type |
Type of event carried by the flow file: ddl or dml |
cdc.most.significant.position |
Ddl’s most significant position in cdc stream |
cdc.least.significant.position |
Ddl’s least significant position in cdc stream |
cdc.event.seen.at |
Timestamp from time when ddl event has been read by the processor |