CaptureChangeMySQL 2025.3.28.13-SNAPSHOT¶
BUNDLE¶
com.snowflake.openflow.runtime | runtime-database-cdc-processors-nar
DESCRIPTION¶
Reads CDC events from a MySQL database.The processor continuously reads events from binary log files, filtering those related to the tables providedby the TableStateService, and discarding the rest.The processor outputs two types of flow files:- DDLs containing the schema of a table (the initial schema and a new schema on every schema change).- DMLs with records representing changes to the data in the table.One flow file always represents data related to a single table.The DDL with the schema is written to the FlowFile content as a JSON object:{ “columns”: [ { “name”: “<columnName>”, “type”: “<snowflakeType>”, “nullable”: <true|false>, “scale”: <scale>, “precision”: <precision> }, … ], “primaryKeys”: [“<primaryKey1>”, “<primaryKey2>”, …]}Structure of the FlowFiles containing the DML records:{ “primaryKeys”: { “<column>”: <value>, … }, “payload”: { “<column>”: <value>, … }, “metadata”: { “<column>”: <value>, … }}
INPUT REQUIREMENT¶
FORBIDDEN
Supports Sensitive Dynamic Properties¶
false
PROPERTIES¶
Property |
Description |
|---|---|
Column Filter Store |
Service storing per-table column filtering settings. |
Connection Timeout |
Connection to source database timeout |
JDBC Driver Location |
Comma-separated list of files/folders and/or URLs containing the driver JAR and its dependencies (if any). For example ‘/var/tmp/mariadb-java-client-3.4.1.jar’ |
JDBC URL |
JDBC URL of the database connection, ie. jdbc:mariadb://localhost:3306/mysql |
Max Batch Size |
The maximum number of records to process in a single iteration. The number of records may exceed the maximum batch size when the last binlog event contains more than one row. |
Max Batch Wait Time |
The maximum time to wait for data to appear in the binlog. |
Max Queue Size |
The maximum number of elements read from binlog until reader thread will wait for onTrigger |
Password |
Password to access the MySQL database |
Record Writer |
The Record Writer is used for serializing DML events |
SSL Context Service |
SSL Context Service supporting encrypted socket communication |
SSL Mode |
SSL Mode used when SSL Context Service configured supporting certificate verification options |
Table State Store |
The shared store holding the state of replicated tables. |
Username |
Username to access the MySQL database |
STATE MANAGEMENT¶
SCOPES |
DESCRIPTION |
|---|---|
CLUSTER |
Information such as a ‘pointer’ to the current CDC event in the database is stored by this processor, such that it can continue from the same location if restarted. |
RELATIONSHIPS¶
NAME |
DESCRIPTION |
|---|---|
success |
Successfully created FlowFile from CDC stream events |
WRITES ATTRIBUTES¶
NAME |
DESCRIPTION |
|---|---|
source.schema.name |
Name of the schema of the table from which an event originated |
source.table.name |
Name of the table from which an event originated |
cdc.event.type |
Type of event carried by the flow file: ddl or dml |
cdc.most.significant.position |
Ddl’s most significant position in cdc stream |
cdc.least.significant.position |
Ddl’s least significant position in cdc stream |
cdc.event.seen.at |
Timestamp from time when ddl event has been read by the processor |