运行实验以比较和选择模型

With Snowflake ML Experiments, you can set up experiments, organized evaluations of the results of model training. This allows you to quickly compare the results of hyperparameter adjustment, different target metrics, and behavior of different model types in an organized fashion in order to select the best model for your needs. Each experiment consists of a series of runs, which are metadata and artifacts from your training. Snowflake is unopinionated about your run artifacts – you can submit anything that’s useful for your model evaluation process.

After you complete an experiment, the results are visible through Snowsight. You can also retrieve run artifacts at any time in Python or SQL.

Note

Snowflake Experiments require snowflake-ml-python version 1.19.0 or later.

访问控制要求

创建实验需要在存储运行工件的架构上拥有 CREATE EXPERIMENT 权限。创建实验需要在父数据库和架构上拥有 USAGE 权限。

创建实验

首先，创建一个实验。这需要一个现有的数据库和架构，用来存储运行信息。

In the navigation menu, select AI & ML » Experiments.
Select New Experiment.
Enter the Name of your experiment.
选择用于存储实验运行工件的数据库和架构。
Select Create to create the experiment, or Cancel to cancel.

开始实验运行

Each run in an experiment has its own set of metrics, parameters, and artifacts. This information is used in Snowsight to provide visualizations and data about your model training and its results.

Start a run with the start_run(name: Optional[str]) method on your ExperimentTracking instance. This returns a new Run, which supports use in a with statement. Snowflake recommends that you use with statements, so that runs are cleanly completed and it’s easier to reason about run scope.

with exp.start_run("my_run"):
  # .. Train your model and log artifacts

自动记录训练信息

You can autolog training information for XGBoost, LightGBM, or Keras models during model training. Autologging is performed by registering a callback which refers to your experiment and information about the model you’re training. Each time a method is called on your Model instance which adjusts a parameter or metric, it’s automatically logged to your experiment for the active run.

以下示例展示了如何为每个受支持的模型训练器配置实验回调，然后开始基本训练运行以记录工件。

# exp: ExperimentTracking

from lightgbm import LGBMClassifier

from snowflake.ml.experiment.callback.lightgbm import SnowflakeLightgbmCallback
from snowflake.ml.model.model_signature import infer_signature

sig = infer_signature(X, y)
callback = SnowflakeLightgbmCallback(
    exp, model_name="name", model_signature=sig
)
model = LGBMClassifier()
with exp.start_run("my_run"):
    model.fit(X, y, eval_set=[(X, y)], callbacks=[callback])

# exp: ExperimentTracking

import keras

from snowflake.ml.experiment.callback.keras import SnowflakeKerasCallback
from snowflake.ml.model.model_signature import infer_signature

sig = infer_signature(X, y)
callback = SnowflakeKerasCallback(
    exp, model_name="name", model_signature=sig
)
model = keras.Sequential()
model.add(keras.layers.Dense(1))
model.compile(
    optimizer=keras.optimizers.RMSprop(learning_rate=0.1),
    loss="mean_squared_error",
    metrics=["mean_absolute_error"],
)
with exp.start_run("my_run"):
    model.fit(X, y, validation_split=0.5, callbacks=[callback])

手动记录训练信息和工件

对于不支持自动记录或已预训练的模型，您可以在 Python 中手动记录实验信息并上传工件。参数是训练模型的固定输入，而指标在模型步骤中进行评估。您可以选择将训练轮次表示为相应的步骤。以下示例展示了如何记录参数、记录指标和上传工件。

Note

The default step value is 0.

# Logging requires an active run for the exp: ExperimentTracker instance.

# Log model parameters with the log_param(...) or log_params(...) methods
exp.log_param("learning_rate", 0.01)
exp.log_params({"optimizer": "adam", "batch_size": 64})

# Log model metrics with the log_metric(...) or log_metrics(...) methods
exp.log_metric("loss", 0.3, step=100)
exp.log_metrics({"loss": 0.4, "accuracy": 0.8}, step=200)

# Log your model to the experiment's model registry with the log_model(...) method.
exp.log_model(model, model_name="my_model", signatures={"predict": model_signature})
exp.log_model(model, model_name="my_model", sample_input_data=data)

# Log local artifacts to an experiment run with the log_artifact(...) method.
exp.log_artifact('/tmp/file.txt', artifact_path='artifacts')

记录 stdout 输出和 stderr 输出¶

当 Snowflake 笔记本或任何 SPCS 工作负载（例如 ML Jobs）上的运行处于活动状态时，您可以将 stdout 输出和 stderr 输出记录为运行的一部分。要启用实时日志记录，请调用以下方法：

experiment.set_live_logging_status(True)

When live logging is enabled, stdout and stderr output from the notebook is written to the Snowflake default event table. To view the captured output, go to the Experiments UI and select the run. The output is displayed in the Logs tab.

请注意，此功能不适用于旧版笔记本。

完成一次运行

Completing a run makes it immutable and presents it as finished in Snowsight.

If you started a run as part of a with statement, the run is automatically completed when exiting scope. Otherwise, you can end a run by calling your experiment’s end_run(name: Optional[str]) method with the name of the run to complete:

experiment.end_run("my_run")

比较实验中的运行

Experiment evaluation is done through Snowsight. In the navigation menu, select AI & ML » Experiments and select your experiment to examine from the list.

The runs list displays Run name, Status, Created date, and a column for each metric. You can also toggle parameters as additional columns. View more details, such as artifacts, metric charts, and linked model versions from the run view and run comparison views.

Note

Viewing linked model versions is part of the Model Lineage feature, which is only available for customers on Enterprise Edition and above.

You can select up to five runs in your experiment. To compare runs, select the Compare button. You’re presented with the comparison view, which displays run metadata, parameters, metrics, and model version information.

Search and filter runs¶

You can programmatically search and filter runs using the list_metrics and list_params methods. Each method returns a Snowpark DataFrame with one row per run: list_metrics includes a run_name column and one float column per logged metric, while list_params includes a run_name column and one string column per logged parameter.

Because the results are Snowpark DataFrames, you can join, filter, and sort them using Snowpark expressions, or convert them to pandas for local analysis.

from snowflake.ml.experiment import ExperimentTracking

exp = ExperimentTracking(session)
exp.set_experiment("my_experiment")

run_names = ["RUN_1", "RUN_2"]

metrics_df = exp.list_metrics().to_pandas()
params_df  = exp.list_params().to_pandas()
runs_df    = metrics_df.merge(params_df, on="run_name")

results = runs_df[
  (runs_df["run_name"].isin(run_names))
  & (runs_df["loss"] > 0.3)
  & (runs_df["f1 score"] < 0.5)
  & (runs_df["model"].str.startswith("GPT"))
]

print(results)

You can also retrieve metrics or parameters for a single run by passing a run_name argument:

single_run_metrics = exp.list_metrics(run_name="RUN_1")

从运行中检索工件

At any time during or after a run, you can retrieve artifacts. The following example shows how to list a run’s available artifacts in the logs path, and download the logs/log0.txt artifact for the run my_run in the experiment my_experiment to the local directory /tmp:

LIST snow://experiment/my_experiment/versions/my_run/logs;
GET snow://experiment/my_experiment/versions/my_run/logs/log0.txt file:///tmp;

删除运行和实验

After finishing an experiment, you can remove it and all of its associated run artifacts. The following example removes the experiment my_experiment:

DROP EXPERIMENT my_experiment;

You can also remove an individual run from an experiment. The following example removes the run my_run from the experiment my_experiment:

ALTER EXPERIMENT my_experiment DROP RUN my_run;

限制

Snowflake Experiments 受以下限制：

每个架构最多允许 500 次实验。
每个实验最多允许 500 次运行。
运行限制为 1000 个唯一参数和 200 个唯一指标。

成本注意事项

Snowflake Experiments 没有任何额外使用成本。它会产生基于 Snowflake 使用量的标准成本。其中包括以下内容：

Cost of storing run artifacts. For general information about storage costs, see Exploring storage cost.
Cost of visualizing data. The charts in the UI are powered by virtual warehouses. For more information, see Viewing credit usage.