Monitor Openflow using telemetry data¶

本主题介绍如何监控 Openflow 的状态和解决问题。

访问 Openflow 日志¶

Snowflake sends Openflow logs to the event table you configured when you set up Openflow (BYOC | Snowflake deployment).

Snowflake 建议您在事件表查询的 WHERE 子句中包含时间戳。这一点尤为重要，因为各种 Snowflake 组件可能会生成大量数据。通过应用筛选器，可以检索到较小的数据子集，从而提高查询性能。

To get started quickly with Openflow’s telemetry, see Example Queries below.

Openflow 遥测架构¶

For information about the event table columns, see Event table columns.

以下部分介绍了 Openflow 如何在事件表中构造遥测。

资源属性

Describes the event metadata set by Openflow. For general information on other types of resource attributes see RESOURCE_ATTRIBUTES column in the Event Table columns documentation.

Resource Attributes Example:

Copy codeExpand code block

{
  "application": "openflow",
  "cloud.service.provider": "aws",
  "container.id": "a1b2c3d4e5f6",
  "container.image.name": "example-openflow-prod.registry-internal.snowflakecomputing.cn/openflow/openflow/openflow_repo/runtime-server",
  "container.image.tag": "2026.3.17.13",
  "deployment.version": "1.35.0",
  "k8s.container.name": "pg-dev-server",
  "k8s.container.restart_count": "0",
  "k8s.namespace.name": "runtime-pg-dev",
  "k8s.node.name": "ip-10-10-62-36.us-east-2.compute.internal",
  "k8s.pod.name": "pg-dev-0",
  "k8s.pod.start_time": "2025-04-25T22:14:29Z",
  "k8s.pod.uid": "94610175-1685-4c8f-b0a1-42898d1058e6",
  "openflow.dataplane.id": "abeddb4f-95ae-45aa-95b1-b4752f30c64a"
}

Show lessSee more

Scroll to top

范围

Scope Example:

Copy codeExpand code block

{
  "name": "runtime"
}

Show lessSee more

Scroll to top

记录类型

根据此行所表示的 Openflow 遥测类型，类型将为以下之一：

• LOG
• METRIC

Openflow 不会收集 TRACE 记录，但在 Snowflake 事件表中，该记录也是此列的有效类型。

记录

可选。JSON 对象描述了该行所表示的指标的类型。

Record Example:

Copy codeExpand code block

{
  "metric": {
    "name": "connection.queued.duration.max",
    "unit": "millisecond"
  },
  "metric_type": "gauge",
  "value_type": "INT"
}

Show lessSee more

Scroll to top

记录属性

日志

Record attributes for Logs will typically indicate where this log was sourced. For example, logs from an Openflow Runtime named testruntime could have Record Attributes of:

Copy codeExpand code block

{
  "log.file.path": "/var/log/pods/runtime-testruntime_testruntime-0_66d80cdb-9484-40a4-bdba-f92eb0af14c7/testruntime-server/0.log",
  "log.iostream": "stdout",
  "logtag": "F"
}

Show lessSee more

Scroll to top

系统指标

System metrics like CPU usage will typically not set Record Attributes, so this will be null.

Openflow 应用程序指标¶

Record Attributes for Application or “Flow” metrics provide details about the component in the data pipeline that produced the metric. This will vary based on the type of component. See Application Metrics

Copy codeExpand code block

{
  "component": "PutSnowpipeStreaming",
  "execution.node": "ALL",
  "group.id": "c052f9d7-7f76-3013-a2c5-d3b064fa7326",
  "id": "c69e2913-22a9-36bb-a159-6a5ed1fb9d63",
  "name": "PutSnowpipeStreaming",
  "type": "processor"
}

Show lessSee more

Scroll to top

值

此列包含遥测的原始值。对于指标，这将是一个数值（整数或双精度）。对于日志，这将是一个半结构化的字符串值或格式良好的 JSON 字符串。

Openflow 运行时日志¶

Openflow Runtimes emit most logs as JSON, so applying Snowflake’s TRY_PARSE_JSON to the VALUE column allows you to further break this value into the following structured fields:

{
  "processGroupId": "6dc1d98f-019d-1000-ffff-ffffa3ba8a09",
  "processGroupIdPath": "/58385a8b-019d-1000-2a52-9ef1c34b0e5f/6dc1d98f-019d-1000-ffff-ffffa3ba8a09",
  "processGroupName": "latency targets",
  "processGroupNamePath": "/Openflow/latency targets",
  "registeredFlowIdentifier": "sqlserver-multidatabase",
  "registeredFlowVersion": "0.29.0-ebb7a257"
}

应用程序指标

连接指标

连接记录属性

每个连接指标都包含以下记录属性：

输入端口和输出端口指标

Input Port and Output Ports are technically two separate types of components. For consistency, metrics and attributes for Input and Output Ports are the same, with the exception of the type attribute that indicates whether it is an input port or an output port.

输入端口和输出端口记录属性

每个端口指标都包含以下记录属性：

流程组指标

流程组记录属性

每个流程组指标都包含以下记录属性：

处理器指标

处理器记录属性

每个处理器指标都包含以下记录属性：

计数器的附加属性

In addition to the standard Processor attributes above, processor.counter metrics include the following:

远程流程组指标

远程流程组记录属性

每个远程流程组指标都包含以下记录属性：

JVM 指标¶

JVM 记录属性¶

JVM 指标不提供记录属性。

CPU 指标¶

CPU 记录属性¶

存储指标

存储记录属性

示例查询

以下示例查询旨在帮助您开始使用 Openflow 遥测。

All queries assume that Openflow is configured to send telemetry to the default Event Table of SNOWFLAKE.TELEMETRY.EVENTS. If your Snowflake Account or Openflow Deployment is configured with a different Event Table, substitute that table name where you see SNOWFLAKE.TELEMETRY.EVENTS.

查找卡住的 FlowFiles¶

此查询返回排队时间超过某个阈值的 FlowFiles，这表明它们可能处于卡住状态并需要干预。阈值默认设置为 30 分钟，可根据实际需求调整。

SELECT * FROM (
  SELECT
    resource_attributes:"openflow.dataplane.id" as Deployment_ID,
    resource_attributes:"k8s.namespace.name" as Runtime_Key,
    record_attributes:name as Connection_Name,
    record_attributes:id as Connection_ID,
    MAX(TO_NUMBER(value / 60 / 1000)) as Max_Queued_File_Minutes
  FROM snowflake.telemetry.events
  WHERE true
    AND record_type = 'METRIC'
    AND record:metric:name = 'connection.queued.duration.max'
    AND timestamp > dateadd(minutes, -30, sysdate())
  GROUP BY 1, 2, 3, 4
  ORDER BY Max_Queued_File_Minutes DESC
) WHERE Max_Queued_File_Minutes > 30;

查找 Openflow 运行时的错误日志¶

SELECT
  timestamp,
  Deployment_ID,
  Runtime_Key,
  parsed_log:level as log_level,
  parsed_log:loggerName as logger,
  parsed_log:formattedMessage as message,
  parsed_log
FROM (
  SELECT
    timestamp,
    resource_attributes:"openflow.dataplane.id" as Deployment_ID,
    resource_attributes:"k8s.namespace.name" as Runtime_Key,
    TRY_PARSE_JSON(value) as parsed_log
  FROM snowflake.telemetry.events
  WHERE true
    AND timestamp > dateadd('minutes', -30, sysdate())
    AND record_type = 'LOG'
    AND resource_attributes:"k8s.namespace.name" like 'runtime-%'
  ORDER BY timestamp DESC
) WHERE log_level = 'ERROR';

查找正在运行和未运行的处理器

某些流程要求所有处理器处于“运行”状态，即使它们未主动处理数据。

此查询可帮助您找到任何正在运行或处于其他状态的处理器，例如：

• 已停止
• invalid
• 已禁用

SELECT
  timestamp,
  resource_attributes:"openflow.dataplane.id" as Deployment_ID,
  resource_attributes:"k8s.namespace.name" as Runtime_Key,
  record_attributes:component as Processor,
  record_attributes:id as Processor_ID,
  TO_NUMBER(value) as Running
FROM snowflake.telemetry.events
WHERE true
  AND record:metric:name = 'processor.run.status.running'
  AND record_type = 'METRIC'
  AND timestamp > dateadd(minutes, -30, sysdate());

查找 Openflow 运行时的高 CPU 使用率¶

数据流变慢或吞吐量下降，可能是由于 CPU 出现瓶颈导致。Openflow 运行时会根据您配置的最小和最大节点数量自动向上扩展。

如果 Openflow 运行时正在使用其最大节点数，但 CPU 使用率仍然很高，请考虑：

1. 增加分配给该运行时的最大节点数
2. 排查连接器或数据流，找出瓶颈所在

使用 Snowsight Charts，可以方便地将 CPU 使用率随时间变化的查询结果可视化。

SELECT
  timestamp,
  resource_attributes:"openflow.dataplane.id" as Deployment_ID,
  resource_attributes:"k8s.namespace.name" as Runtime_Key,
  resource_attributes:"k8s.pod.name" as Runtime_Pod,
  TO_NUMBER(value, 10, 3) * 100 as CPU_Usage_Percentage
FROM snowflake.telemetry.events
WHERE true
  AND timestamp > dateadd(minute, -30, sysdate())
  AND record_type = 'METRIC'
  AND record:metric:name ilike 'container.cpu.usage'
  AND resource_attributes:"k8s.namespace.name" ilike 'runtime-%'
  AND resource_attributes:"k8s.container.name" ilike '%-server'
ORDER BY timestamp desc, CPU_Usage_Percentage desc;