Snowpark Container Services: Monitoring Services¶

Using SYSTEM$GET_SERVICE_LOGS¶

You provide the service name, instance ID, container name, and optionally the number of most recent log lines to retrieve. If only one service instance is running, the service instance ID is 0. For example, the following statement command retrieves the trailing 10 lines from the log of a container named echo that belongs to instance 0 of a service named echo_service:

SELECT SYSTEM$GET_SERVICE_LOGS('echo_service', '0', 'echo', 10);

Example output:

+--------------------------------------------------------------------------+
| SYSTEM$GET_SERVICE_LOGS                                                  |
|--------------------------------------------------------------------------|
| 10.16.6.163 - - [11/Apr/2023 21:44:03] "GET /healthcheck HTTP/1.1" 200 - |
| 10.16.6.163 - - [11/Apr/2023 21:44:08] "GET /healthcheck HTTP/1.1" 200 - |
| 10.16.6.163 - - [11/Apr/2023 21:44:13] "GET /healthcheck HTTP/1.1" 200 - |
| 10.16.6.163 - - [11/Apr/2023 21:44:18] "GET /healthcheck HTTP/1.1" 200 - |
+--------------------------------------------------------------------------+
1 Row(s) produced. Time Elapsed: 0.878s

If you don’t have the information about the service that you need to call the function (such as the instance ID or container name), you can first run the SHOW SERVICE CONTAINERS IN SERVICE command to get information about the service instances and containers running in each instance.

The SYSTEM$GET_SERVICE_LOGS function has the following limitations:

• It merges standard output and standard error streams. The function provides no indication of which stream the output came from.

• It reports the captured data for a specific container in a single service instance.

• It only reports logs for a running container. The function cannot fetch logs from a previous container that was restarted or from a container of a service that is stopped or deleted.

• The function returns up to 100 KB of data.

Using event table¶

Snowflake can capture logs sent from containers to the standard output and standard error streams into the event table configured for your account. For more information about configuring an event table, see Logging, tracing, and metrics.

You control which streams are collected (all, standard error only, or none) that you want stored in an event table by using the spec.logExporters field in the service specification file.

You can then query the event table for events. For example, the following SELECT statement retrieves Snowflake service and job events recorded in the past hour:

SELECT TIMESTAMP, RESOURCE_ATTRIBUTES, RECORD_ATTRIBUTES, VALUE
FROM <current-event-table-for-your-account>
WHERE timestamp > dateadd(hour, -1, current_timestamp())
AND RESOURCE_ATTRIBUTES:"snow.service.name" = <service-name>
AND RECORD_TYPE = 'LOG'
ORDER BY timestamp DESC
LIMIT 10;

Snowflake recommends that you include a timestamp in the WHERE clause of event table queries, as shown in this example. This is particularly important because of the potential volume of data generated by various Snowflake components. By applying filters, you can retrieve a smaller subset of data, which improves query performance.

The event table includes the following columns, which provide useful information regarding the logs collected by Snowflake from your container:

• TIMESTAMP: Shows when Snowflake collected the log.

• RESOURCE_ATTRIBUTES: Provides a JSON object that identifies the Snowflake service and the container in the service that generated the log message. For example, it furnishes details such as the service name, container name, and compute pool name that were specified when the service was run.

  {
    "snow.account.name": "SPCSDOCS1",
    "snow.compute_pool.id": 20,
    "snow.compute_pool.name": "TUTORIAL_COMPUTE_POOL",
    "snow.compute_pool.node.id": "a17e8157",
    "snow.compute_pool.node.instance_family": "CPU_X64_XS",
    "snow.database.id": 26,
    "snow.database.name": "TUTORIAL_DB",
    "snow.schema.id": 212,
    "snow.schema.name": "DATA_SCHEMA",
    "snow.service.container.instance": "0",
    "snow.service.container.name": "echo",
    "snow.service.container.run.id": "b30566",
     "snow.service.id": 114,
    "snow.service.name": "ECHO_SERVICE2",
    "snow.service.type": "Service"
  }
  

  Copy

• RECORD_ATTRIBUTES: For a Snowflake service, it identifies an error source (standard output or standard error).

  { "log.iostream": "stdout" }
  

  Copy

• VALUE: Standard output and standard error are broken into lines, and each line generates a record in the event table.

  "echo-service [2023-10-23 17:52:27,429] [DEBUG] Sending response: {'data': [[0, 'Joe said hello!']]}"
  

Accessing event-table service metrics¶

To log metrics from a service into the event table configured for your account, include the following section in your service specification:

platformMonitor:
  metricConfig:
    groups:
    - <group 1>
    - <group 2>
    - ...

Where each group N refers to a predefined metrics group that you are interested in (for example, system, network, or storage). For more information, see the spec.platformMonitor field section in the documentation on the service specification.

While the service is running, Snowflake records these metrics to the event table in your account. You can query your event table to read the metrics. The following query retrieves the service metrics that were recorded in the past hour for the service my_service:

SELECT timestamp, value
  FROM my_event_table_db.my_event_table_schema.my_event_table
  WHERE timestamp > DATEADD(hour, -1, CURRENT_TIMESTAMP())
    AND RESOURCE_ATTRIBUTES:"snow.service.name" = 'MY_SERVICE'
    AND RECORD_TYPE = 'METRIC'
    ORDER BY timestamp DESC
    LIMIT 10;

If you don’t know the name of the active event table for the account, execute the SHOW PARAMETERS command to display the value of the account-level EVENT_TABLE parameter:

SHOW PARAMETERS LIKE 'event_table' IN ACCOUNT;

For more information about event tables, see Using event table.

Example

Follow these steps to create an example service that records metrics to the event table configured for your account.

1. Follow Tutorial 1 to create a service named echo_service, with one change. In step 3, where you create a service, use the following CREATE SERVICE command that add the platformMonitor field in the modified service specification.

   CREATE SERVICE echo_service
     IN COMPUTE POOL tutorial_compute_pool
     FROM SPECIFICATION $$
       spec:
         containers:
         - name: echo
           image: /tutorial_db/data_schema/tutorial_repository/my_echo_service_image:latest
           env:
             SERVER_PORT: 8000
             CHARACTER_NAME: Bob
           readinessProbe:
             port: 8000
             path: /healthcheck
         endpoints:
         - name: echoendpoint
           port: 8000
           public: true
         platformMonitor:
           metricConfig:
             groups:
             - system
             - system_limits
         $$
       MIN_INSTANCES=1
       MAX_INSTANCES=1;
   

   Copy

2. After the service is running, Snowflake starts recording the metrics in the specified metric groups to the event table, which you can then query. The following query retrieves metrics reported in the last hour by the Echo service.

   SELECT timestamp, value
     ROM my_events
     WHERE timestamp > DATEADD(hour, -1, CURRENT_TIMESTAMP())
       AND RESOURCE_ATTRIBUTES:"snow.service.name" = 'ECHO_SERVICE'
       AND RECORD_TYPE = 'METRIC'
       ORDER BY timestamp, group DESC
       LIMIT 10;
   

   Copy

Accessing compute pool metrics¶

Compute pool metrics offer insights into the nodes in the compute pool and the services running on them. Each node reports node-specific metrics, such as the amount of available memory for containers, as well as service metrics, like the memory usage by individual containers. The compute pool metrics provide information from a node’s perspective.

Each node has a metrics publisher that listens on TCP port 9001. Other services can make an HTTP GET request with the path /metrics to port 9001 on the node. To discover the node’s IP address, retrieve SRV records (or A records) from DNS for the discover.monitor.compute_pool_name.snowflakecomputing.internal hostname. Then, create another service in your account that actively polls each node to retrieve the metrics.

The body in the response provides the metrics using the Prometheus format (https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format) as shown in the following example metrics:

# HELP node_memory_capacity Defines SPCS compute pool resource capacity on the node
# TYPE node_memory_capacity gauge
node_memory_capacity{snow_compute_pool_name="MY_POOL",snow_compute_pool_node_instance_family="CPU_X64_S",snow_compute_pool_node_id="10.244.3.8"} 1
node_cpu_capacity{snow_compute_pool_name="MY_POOL",snow_compute_pool_node_instance_family="CPU_X64_S",snow_compute_pool_node_id="10.244.3.8"} 7.21397383168e+09

Note the following:

• The response body starts with # HELP and # TYPE, which provide a short description and the type of the metric. In this example, the node_memory_capacity metric is of type gauge.

• It is then followed by the metric’s name, a list of labels describing a specific resource (data point), and its value. In this example, the metric (named node_memory_capacity) provides memory information, indicating that the node has 7.2 GB available memory. The metric also includes metadata in the form of labels as shown:

  snow_compute_pool_name="MY_POOL",
  snow_compute_pool_node_instance_family="CPU_X64_S",snow_compute_pool_node_id="10.244.3.8"
  

You can process these metrics any way you choose; for example, you might store metrics in a database and use a UI (such as a Grafana dashboard) to display the information.

For a list of available compute pool metrics, see Available platform metrics.

Example

For an example of configuring Prometheus to poll your compute pool for metrics, see the compute pool metrics tutorials (https://github.com/Snowflake-Labs/spcs-templates/tree/main/user-metrics).

Available platform metrics¶

The following is a list of available platform metrics groups and metrics within each group. Note that storage metrics are currently only collected from block storage volumes.