Snowpark ML FileSystem and FileSet¶

Installation¶

The FileSystem and FileSet APIs are part of the Snowpark ML Python package, snowflake-ml-python. See Using Snowflake ML Locally for installation instructions.

Creating and Using a File System¶

Creating a Snowpark ML file system requires either a Snowflake Connector for Python Connection object or a Snowpark Python Session. See Connecting to Snowflake for instructions.

After you have either a connection or a session, you can create a Snowpark ML SFFileSystem instance through which you can access data in your internal stage.

If you have a Snowflake Connector for Python connection, pass it as the sf_connection argument:

import fsspec
from snowflake.ml.fileset import sfcfs

sf_fs1 = sfcfs.SFFileSystem(sf_connection=sf_connection)

If you have a Snowpark Python session, pass it as the snowpark_session argument:

import fsspec
from snowflake.ml.fileset import sfcfs

sf_fs2 = sfcfs.SFFileSystem(snowpark_session=sp_session)

SFFileSystem inherits many features from fsspec.FileSystem, such as local caching of files. You can enable this and other features by instantiating a Snowflake file system through the fsspec.filesystem factory function, passing target_protocol="sfc" to use the Snowflake FileSystem implementation:

local_cache_path = "/tmp/sf_files/"
cached_fs = fsspec.filesystem("cached", target_protocol="sfc",
                    target_options={"sf_connection": sf_connection,
                                    "cache_types": "bytes",
                                    "block_size": 32 * 2**20},
                    cache_storage=local_cache_path)

The Snowflake file system supports most read-only methods defined for a fsspec FileSystem, including find, info, isdir, isfile, and exists.

print(*cached_fs.ls("@ML_DATASETS.public.my_models/sales_predict/"), end='\n')

path = '@ML_DATASETS.public.my_models/test/data_7_7_3.snappy.parquet'

with sf_fs1.open(path, mode='rb') as f:
    print(f.read(16))

import pyarrow.parquet as pq

table = pq.read_table(path, filesystem=sf_fs1)
table.take([1, 3])

path = "sfc://@ML_DATASETS.public.my_models/dataset.csv.gz"

with cached_fs.open(path, mode='rb', sf_connection=sf_connection) as f:
    g = gzip.GzipFile(fileobj=f)
    for i in range(3):
        print(g.readline())

Creating and Using a FileSet¶

A Snowflake FileSet represents an immutable snapshot of the result of a SQL query in the form of files in an internal server-side encrypted stage. These files can be accessed through a FileSystem to feed data to tools such as PyTorch and TensorFlow so that you can train models at scale and within your existing data governance model. To create a FileSet, use the FileSet.make method.

You need a Snowflake Python connection or a Snowpark session to create a FileSet. See Connecting to Snowflake for instructions. You must also provide the path to an existing internal server-side encrypted stage, or a subdirectory under such a stage, where the FileSet will be stored.

To create a FileSet from a Snowpark DataFrame, construct a DataFrame and pass it to FileSet.make as snowpark_dataframe; do not call the DataFrame’s collect method:

# Snowpark Python equivalent of "SELECT * FROM MYDATA LIMIT 5000000"
df = snowpark_session.table('mydata').limit(5000000)
fileset_df = fileset.FileSet.make(
    target_stage_loc="@ML_DATASETS.public.my_models/",
    name="from_dataframe",
    snowpark_dataframe=df,
    shuffle=True,
)

To create a FileSet using a Snowflake Connector for Python connection, pass the connection to Fileset.make as sf_connection, and pass the SQL query as query:

fileset_sf = fileset.FileSet.make(
    target_stage_loc="@ML_DATASETS.public.my_models/",
    name="from_connector",
    sf_connection=sf_connection,
    query="SELECT * FROM MYDATA LIMIT 5000000",
    shuffle=True,           # see later section about shuffling
)

Use the files method to get a list of the files in the FileSet:

print(*fileset_df.files())

For information about feeding the data in the FileSet to PyTorch or TensorFlow, see Snowpark ML Framework Connectors.