Validation and error handling¶

Server-side validation (default)¶

snowflake.validation=server_side

With server-side validation, data is sent to Snowflake without client-side type checking. The Snowflake backend performs validation and schema evolution, aligned with the behavior of COPY and Snowpipe.

• Invalid records are captured in the Snowflake Error Table associated with the target table.

• The DLQ isn’t used for ingestion validation errors in this mode. Use Error Tables instead. However, records that fail to be deserialized by the Kafka converters (before reaching Snowflake) are still routed to the DLQ if configured.

• Delivers maximum throughput by offloading validation from the connector.

• Supports both default pipe and user-defined pipe modes.

When to use server-side validation:

• You want maximum throughput.

• You can manage error tables in Snowflake.

• You’re using user-defined pipes for in-flight transformations.

Client-side validation¶

snowflake.validation=client_side

With client-side validation, the connector validates data types and schema compatibility before sending rows to Snowflake.

• Invalid records are handled according to the errors.tolerance setting (see below).

• Schema evolution DDL is executed by the connector when new fields are detected.

• Supports Dead Letter Queue (DLQ) for routing invalid records.

• Only works with the default pipe ({tableName}-STREAMING). If a user-created pipe exists and client-side validation is enabled, the connector fails with ERROR_5030.

When to use client-side validation:

• You need a Dead Letter Queue for error handling.

• You’re migrating from v3 and want familiar behavior.

• You need the connector to control validation.

Error tolerance¶

The errors.tolerance property controls how the connector responds to errors:

errors.tolerance=none (default)

The connector task fails on the first error. If a DLQ is configured, the failing record is still sent to the DLQ before the task aborts.

errors.tolerance=all

The connector continues ingesting data. Invalid records are routed to the DLQ (if configured) or silently dropped.

Warning

Setting errors.tolerance=all without configuring a DLQ topic causes invalid records to be silently dropped. This is an unsafe configuration that can cause data loss.

Dead Letter Queue (DLQ)¶

To configure a DLQ:

errors.deadletterqueue.topic.name=my_topic_dlq
errors.log.enable=true

The following records are routed to the DLQ (if configured):

• Converter errors (both validation modes): Records that fail to be deserialized by the Kafka converters are sent to the DLQ regardless of the validation mode. This applies with both errors.tolerance=none (record pushed to DLQ, then task fails) and errors.tolerance=all (record pushed to DLQ, task continues).

• Client-side validation errors: Records that fail client-side data conversion or schema validation are sent to the DLQ when snowflake.validation=client_side.

Records that fail Snowflake server-side ingestion aren’t routed to the DLQ. Use Error Tables instead.

Error tables (server-side validation)¶

When using snowflake.validation=server_side, records that fail server-side ingestion (type mismatches, constraint violations) are logged in the Error Table for inspection and replay.

For more information about Error Tables and error handling with Snowpipe Streaming, see Error handling in Snowpipe Streaming high-performance architecture.