从 Snowflake 笔记本运行 Spark 工作负载¶

您可以通过 Snowflake 笔记本以交互方式运行 Spark 工作负载，而无需管理 Spark 集群。工作负载会在 Snowflake 基础设施上运行。

要使用 Snowflake 笔记本作为客户端，来开发在 Snowflake 上运行的 Spark 工作负载，请执行以下操作：

启动 Snowflake 笔记本。
在笔记本中启动 Spark 会话。
Write PySpark code to load, transform, and analyze data—such as to filter high-value customer orders or aggregate revenue.

使用在仓库上运行的 Snowflake 笔记本¶

For more information about Snowflake Notebooks, see Create a notebook.

完成以下步骤，创建 Snowflake 笔记本：
1. Sign in to Snowsight.
2. At the top of the navigation menu, select (Create) » Notebook » New Notebook.
3. In the Create notebook dialog, enter a name, database, and schema for the new notebook.
  
  For more information, see Create a notebook.
4. For Runtime, select Run on warehouse.
5. For Runtime version, select Snowflake Warehouse Runtime 2.0.
  
  When you select version 2.0, you ensure that you have the dependency support you need, including Python 3.10. For more information, see Legacy Notebook runtimes.
6. For Query warehouse and Notebook warehouse, select warehouses for running query code and kernel and Python code, as described in Create a notebook.
7. Select Create.
8. In the notebook you created, under Packages, ensure that you have the following packages listed to support code in your notebook:
  - Python 3.10 或更高版本
  - snowpark-connect，最新版本

如果您需要添加这些包，请按以下步骤操作：

Under Anaconda Packages, type the packages name in the search box.
选中所需包名称。
Select Save.
To connect to the Snowpark Connect for Spark server and test the connection, copy the following code and paste it in the Python cell of the notebook you created:
```
from snowflake import snowpark_connect

spark = snowpark_connect.init_spark_session()
df = spark.sql("show schemas").limit(10)
df.show()
```

使用在工作区中运行的 Snowflake 笔记本¶

For more information about Snowflake Notebooks in Workspaces, see Snowflake Notebooks in Workspaces.

创建 PyPI 外部访问集成。

您必须使用 ACCOUNTADMIN 角色并拥有一个您可以访问的数据库。

在工作区中从某个 SQL 文件运行以下命令。

USE DATABASE mydb;
USE ROLE accountadmin;

CREATE OR REPLACE NETWORK RULE pypi_network_rule
MODE = EGRESS
TYPE = HOST_PORT
VALUE_LIST = ('pypi.org', 'pypi.python.org', 'pythonhosted.org', 'files.pythonhosted.org');

CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION pypi_access_integration
ALLOWED_NETWORK_RULES = (pypi_network_rule)
ENABLED = true;

在笔记本中启用 PyPI 集成。
1. In the notebook, for Service name, select a service.
2. For External access integrations, select the PyPI integration you created.
3. For Python version, select Python 3.11.
4. Select Create.
Install the snowpark_connect package from PyPI in the notebook, using code such as the following:
```
pip install snowpark-connect[jdk]
```
重新启动内核。
- From the Connect button, select Restart kernel.
Start the snowpark_connect server using code such as the following:
```
import snowflake.snowpark_connect

spark = snowflake.snowpark_connect.init_spark_session()
```

运行您的 Spark 代码，如以下示例所示：

from pyspark.sql.connect.functions import *
from pyspark.sql.connect.types import *
from pyspark.sql import Row

# Sample nested data
data = [(1, ("Alice", 30))]
schema = "id INT, info STRUCT<name:STRING, age:INT>"

df = spark.createDataFrame(data, schema=schema)
df.show()

spark.sql("show databases").show()