Openflow Connector for PostgreSQL 维护

备注

This connector is subject to the Snowflake Connector Terms.

This topic describes important maintenance considerations and best practices for maintaining the Openflow Connector for PostgreSQL when making changes to the source PostgreSQL database. In addition this topic describes how to reinstall the connector.

升级 PostgreSQL

升级该连接器需要不同的方法，具体取决于是要将 PostgreSQL 升级到下一个次要版本还是主要版本。

次要版本升级

  • 数据是否安全。

  • 无需特殊处理。

  • 在升级期间需要停止连接器，以避免报告连接问题。

  • 升级后继续复制，不会丢失数据。

主要版本升级

  • 需要 PostgreSQL 服务器删除复制槽，包括连接器使用的任何复制槽。

  • 无法保留复制槽或将其迁移到新版本。另请参阅 PostgresSQL 17 及更高版本升级。

  • 重新开始复制前一快照阶段的所有表。

要执行次要版本升级，请执行以下操作：

  1. 停止连接器，包括所有处理器和控制器服务。

  2. 升级 PostgreSQL。

  3. 重新启动连接器。

要执行主要版本升级，请执行以下操作：

  1. 从连接器内的副本中移除所有表。

  2. 等到连接器中的所有队列均为空。

  3. 停止连接器，包括所有处理器和控制器服务。

  4. 在连接器中打开 Incremental Load 群组。

  5. 右键单击组中的顶部处理器，执行 Read PostgreSQL CDC Stream，然后选择 View state。

  6. 点击 Clear state。

  7. 点击 Close。

  8. 升级 PostgreSQL。

  9. 重新启动连接器。系统将创建一个新的复制槽。

  10. 重新添加所有表以开始复制。

PostgresSQL 17 及更高版本升级

PostgreSQL 17 改进了升级，使其在升级到更高版本（如 17.1 » 18.0）时不再需要移除复制槽。从先前版本（16 及更早版本）升级到 PostgreSQL 17.0 或更高版本会删除复制槽，应将其视为主要版本升级。PostgreSQL 的未来版本也可能会进一步改善升级过程。

Reinstall the connector

This section describes how to reinstall the connector. It covers situations where the new connector is installed in the same runtime, or when it is moved to a new runtime. Reinstall is often used in conjunction with Incremental replication with snapshots.

警告

For the connector to be able to continue replicating from the same CDC stream position where it stopped before reinstallation, the source database must retain the WAL long enough to cover the time since the old connector is stopped and the new connector is started. Ensure the max_wal_size parameter of the PostgreSQL server is high enough, depending on your traffic, and keep the reinstallation time to a minimum.

Prerequisites

Review and note connector parameter context values. If you're reinstalling the connector in the same runtime, you can reuse the existing context. If the new instance will be located in a different runtime, you will have to re-enter all parameters.

To reinstall the connector:

  1. Finish processing all in-flight FlowFiles in the existing connector, and then stop the connector.

    1. Sign in to Snowsight.

    2. In the navigation menu, select Ingestion » Openflow.

    3. In the Openflow pane select the Runtimes tab.

    4. Select the runtime containing the connector.

    5. Select the connector.

    6. Stop the topmost processor Set Tables for Replication in the Snapshot Load group.

    7. Stop the topmost processor Read PostgreSQL CDC Stream in the Incremental Load group.

    8. If you changed the value of the Merge Task Schedule CRON parameter, return it to * * * * * ?, otherwise queues will not be emptied until the next scheduled run.

      Wait until all FlowFiles in the connector have been processed, and all queues are empty. When all FlowFiles have been processed, the Queued value on the connector's processor group becomes zero. If there are any items left in the original connector's queues, there may be data gaps when the new connector starts.

    9. Stop all processors and controller services in the connector.

  2. Find and copy the name of the replication slot used by the original connector, by viewing the state of the topmost processor in the Incremental Load group with name Read PostgreSQL CDC Stream. The replication slot name is stored under the key replication.slot.name. Copy the value of the key to a text editor.

  3. Create a new instance of the connector. If you're using the same runtime as the original connector, you can choose to keep the existing parameter contexts, and reuse the settings.

    小心

    The existing connector can remain in the runtime and doesn't interfere with the new instance, as long as it remains stopped.

  4. If you're installing into a different runtime, or you deleted the previous parameter contexts, enter all the configuration settings into the new parameter contexts, including the table names and patterns as described in 设置 Openflow Connector for PostgreSQL.

  5. Open the PostgreSQL Ingestion Parameters context, and set Ingestion Type parameter to incremental. For more information on the concerns see 启用不使用快照的增量复制.

  6. Open the PostgreSQL Source Parameters context, and set the Replication Slot Name parameter to the value you copied earlier.

  7. Start the new connector.

Usage notes

The new connector will use the same, existing destination tables that created by the original connector, but will create new journal tables.

语言: 中文