Logging messages in Python¶

You can log messages from a function or procedure handler written in Python by using logging (https://docs.python.org/library/logging.html), the logging module in Python’s standard library. When you’ve set up an event table to store log entries, Snowflake stores log entries generated by your handler code in the table.

For more information about logging levels supported by Python, see the logging levels documentation (https://docs.python.org/3/library/logging.html#levels). Note that Snowflake treats two of the Python logging levels in a particular way:

The Python CRITICAL level will be treated as FATAL.
The Python NOTSET level will be treated as TRACE.

For general information about setting up logging and retrieving messages in Snowflake, see Logging messages from functions and procedures.

Before logging from code, you must:

Set up an event table to collect messages logged from handler code.

For more information, see Event table overview.
Be sure you have the logging level set so that the messages you want are stored in the event table.

For more information, see Setting levels for logging, metrics, and tracing.

Overriding log threshold levels with Python¶

You can use Python handler code to override the log threshold levels that have been log level set with SQL. When you set the log level with Python, log entries will use the logging levels defined by Python (https://docs.python.org/3/library/logging.html#levels).

By setting log levels in Python, you can do the following:

Override the threshold set for the Snowflake session or for objects such as the procedure or UDF.
Set thresholds scoped to specified Python packages.

For example, you can use the logger name you set (and which is stored in the event table) to set a threshold for that logger with Python.

Python code in the following example sets the log level for the Snowpark session package to DEBUG.

session_logger = logging.getLogger('snowflake.snowpark.session')
session_logger.setLevel(logging.DEBUG)

Copy

Using the logger name to set logging level¶

You can use the logger name recorded in the event table to set a threshold for log entries from that logger. This can be useful when you want to set a logger’s threshold so that it filters out unwanted log entries above a particular level.

To do that, you’d first query the event table to discover the logger name associated with the entries for which you want to capture a different logging level. Then, using that logger name, you’d set the log level to the threshold you want.

Code in the following example queries for log entries, including the logger name in the returned data. You can get the name as the value of the Scope column.

SET event_table_name='my_db.public.my_event_table';

SELECT
  TIMESTAMP as time,
  RECORD['severity_text'] as log_level,
  SCOPE['name'] as logger_name,
  VALUE as message
FROM
  IDENTIFIER($event_table_name)
WHERE
  RECORD_TYPE = 'LOG';

Copy

This query might return many entries from several loggers. If, after looking through the results, you decide that you’re getting many INFO messages that you don’t want from the numpy logger, you can use Python to set that logger’s threshold to capture log entries at the ERROR level and above.

numpy_logger = logging.getLogger('numpy_logs')
numpy_logger.setLevel(logging.ERROR)

Copy

For more about querying the event table, see Viewing log messages.

Adding custom attributes¶

When you create a log entry, you can add your own attributes in key-value pairs. Snowflake saves these custom attributes to the event table’s RECORD_ATTRIBUTES column.

To add custom attributes when calling one of the logging level functions — including logger.info, logger.error, and so on — add an extra keyword argument, setting the argument’s value to the key-value pairs to record as custom attributes.

Code in the following example logs a message “Logging with attributes” to the event table’s VALUE column. It also adds two custom attributes to the RECORD_ATTRIBUTES column.

CREATE OR REPLACE PROCEDURE do_logging_python()
RETURNS VARCHAR
LANGUAGE PYTHON
PACKAGES = ('snowflake-snowpark-python')
RUNTIME_VERSION = 3.9
HANDLER = 'do_things'
AS $$
import logging

logger = logging.getLogger("python_logger")

def do_things(session):

  logger.info("Logging with attributes in SP", extra = {'custom1': 'value1', 'custom2': 'value2'})

  return "SUCCESS"
$$;

Copy

Output of the logger.info call appears in the event table as follows. Note that the RECORD_ATTRIBUTES column will include attributes that Snowflake adds automatically.

---------------------------------------------------------------------
| VALUE                        | RECORD_ATTRIBUTES                  |
---------------------------------------------------------------------
| "Logging with attributes in" | {                                  |
|                              |   "code.filepath": "_udf_code.py", |
|                              |   "code.function": "do_things",    |
|                              |   "code.lineno": 10,               |
|                              |   "custom1": "value1",             |
|                              |   "custom2": "value2"              |
|                              | }                                  |
---------------------------------------------------------------------

Python examples¶

The following sections provide examples of adding support for logging from Python code.

Stored procedure example¶

Code in the following example imports the logging module, gets a logger, and logs a message at the INFO level.

For more information about logging levels supported by Python, see the logging levels documentation (https://docs.python.org/3/library/logging.html#levels).

CREATE OR REPLACE PROCEDURE do_logging()
RETURNS VARCHAR
LANGUAGE PYTHON
PACKAGES=('snowflake-snowpark-python')
RUNTIME_VERSION = 3.9
HANDLER='do_things'
AS $$
import logging

logger = logging.getLogger("python_logger")
logger.info("Logging from Python module.")

def do_things(session):
  logger.info("Logging from Python function start.")

  try:
    throw_exception()
  except Exception:
    logger.error("Logging an error from Python handler: ")
    return "ERROR"

  return "SUCCESS"

def throw_exception():
  raise Exception("Something went wrong.")

$$;

Copy

You can access log messages by executing a SELECT command on the event table. For more information, see Viewing log messages.

Code in the following example queries the event table where the log messages are stored. The query reports on the severity and message of each log entry from the handler class.

SET event_table_name='my_db.public.my_event_table';

SELECT
  RECORD['severity_text'] AS SEVERITY,
  VALUE AS MESSAGE
FROM
  IDENTIFIER($event_table_name)
WHERE
  SCOPE['name'] = 'python_logger'
  AND RECORD_TYPE = 'LOG';

Copy

The preceding example generates the following output.

---------------------------------------------------------------------------
| SEVERITY | MESSAGE                                                      |
---------------------------------------------------------------------------
| "INFO"   | "Logging from Python module."                                |
---------------------------------------------------------------------------
| "INFO"   | "Logging from Python function start."                        |
---------------------------------------------------------------------------
| "ERROR"  | "Logging an error from Python handler."                      |
---------------------------------------------------------------------------

Streamlit example¶

Code in the following example imports the logging module, gets a logger, and logs a message at the INFO level.

For more information about logging levels supported by Python, see the logging levels documentation (https://docs.python.org/3/library/logging.html#levels).

import streamlit as st
import logging

logger = logging.getLogger('app_logger')

st.title("Streamlit logging example")

hifives_val = st.slider("Number of high-fives", min_value=0, max_value=90, value=60)

if st.button("Submit"):
    logger.info(f"Submitted with high-fives: {hifives_val}")

Copy