Categories:

String & binary functions (Large Language Model)

COMPLETE (SNOWFLAKE.CORTEX) (multimodal)

Given an image and a prompt, generates a response (completion) using a language model. This function variant supports image models along with text models, and processes images stored in an internal Snowflake stage or an external stage. COMPLETE can be used to process a single image, multiple images in a batch fashion, applying the same or a different prompt to each image, or multiple images in a single operation (for example, comparison).

Syntax

Use one of the following:

SNOWFLAKE.CORTEX.COMPLETE(
    '<model>', '<prompt>', <file_object>)
FROM <table>
Copy
SNOWFLAKE.CORTEX.COMPLETE(
    '<model>', <prompt_object> )
FROM <table>
Copy

Arguments

model

A string specifying the model to be used. Specify one of the following models:

  • claude-3-5-sonnet

  • pixtral-large

Supported models might have different costs and context windows. New models might be added from time to time.

prompt

A string containing a question about the image and optionally specifying an output format, such as JSON. Either this or the prompt_object argument is required.

prompt_object

A SQL OBJECT containing a string prompt with numbered placeholders ({0}, {1}, and so on) and one or more text or FILE valuse that are inserted into the prompt. The PROMPT function function is a convenient way to create an object with the required layout. Either this argument or prompt is required.

file_object

A FILE object that contains an image file to be processed. Use the TO_FILE function to create FILE objects from a stage path. Required when using a string prompt.

FROM table

An optional table containing image paths and an optional prompt for each image, allowing images to be batch-processed in a single call to COMPLETE.

Returns

A string containing the language model’s response.

Usage notes

  • To process multiple images, the prompt must be an object (typically created using the PROMPT function) that specifies a prompt template and the files to be processed.

  • Only text and images are supported. Video and audio files are not supported.

  • Images with filename extensions .jpg, .jpeg, .png, .gif, and .webp are supported. pixtral-large also supports .bmp.

  • Maximum image size is 10 MB for pixtral-large and 5 MB for claude-3-5-sonnet. Additionally, claude-3-35-sonnet does not support images with a resolution greater than 8000x8000.

  • The stage containing the images must have server-side encryption enabled. Client-side encrypted stages are not supported.

  • The function does not support custom network policies.

  • Stage names are not case-sensitive, but paths are.

Examples

The following examples demonstrate the basic capabilities of the COMPLETE function with images.

Visual question answering

A chart of inflation rates is used to answer a question about the data.

Graph of inflation rates in 2023 with estimates for 2024

Comparison between inflation rates in 2023 and in2024 (Statista (https://www.statista.com/))

SELECT SNOWFLAKE.CORTEX.COMPLETE('claude-3-5-sonnet',
    'Which country will observe the largest inflation change in 2024 compared to 2023?',
    TO_FILE('@myimages', 'highest-inflation.png'));
Copy

Response:

Looking at the data, Venezuela will experience the largest change in inflation rates between 2023 and 2024.
The inflation rate in Venezuela is projected to decrease significantly from 337.46% in 2023 to 99.98% in 2024,
representing a reduction of approximately 237.48 percentage points. This is the most dramatic change among
all countries shown in the chart, even though Zimbabwe has higher absolute inflation rates.

Image classification

This example classifies the landmark identified in a single image.

Photograph of the Space Needle in Seattle with a puppy 
SELECT SNOWFLAKE.CORTEX.COMPLETE('claude-3-5-sonnet',
    'Classify the landmark identified in this image. Respond in JSON only with the landmark name.',
    TO_FILE('@myimages', 'Seattle.jpg'));
Copy

Response:

{"landmark": "Space Needle"}

Entity extraction from an image

This example extracts the entities (objects) from an image and returns the results in JSON format.

Photograph of kitchen after remodeling 
SELECT SNOWFLAKE.CORTEX.COMPLETE('claude-3-5-sonnet',
    'Extract the kitchen appliances identified in this image. Respond in JSON only with the identified appliances.',
    TO_FILE('@myimages, 'kitchen.png'));
Copy

Response:

{
    "appliances": [ "microwave","electric stove","oven","refrigerator" ]
}

Batch processing images from a directory or table

For batch processing of multiple images, performing the same operation on each, store the image files in the same stage. Apply the COMPLETE function to each row of the table.

Note

The stage must have a directory table to retrieve the paths to its files.

First, create the table by retrieving the image locations from the directory, convert these to FILE objects, and storing the resulting FILE objects in a column in a table. Use SQL like the following:

CREATE TABLE image_table AS
    (SELECT TO_FILE('@myimages', RELATIVE_PATH) AS img FROM DIRECTORY(@myimages));
Copy

Then, apply the COMPLETE function to the column containing the FILE objects. The following example classifies each image in the table:

SELECT SNOWFLAKE.CORTEX.COMPLETE('claude-3-5-sonnet',
    PROMPT('Classify the input image {0} in no more than 2 words. Respond in JSON', img_file)) AS image_classification
FROM image_table;
Copy

Response:

{ "classification": "Inflation Rates" }
{ "classification": "beverage refrigerator" }
{ "classification": "Space Needle" }
{ "classification": "Modern Kitchen" }
{ "classification": "Pie Chart" }
{ "classification": "Economic Graph" }
{ "classification": "Persian Cat" }
{ "classification": "Labrador Retriever" }
{ "classification": "Jedi Cat" }
{ "classification": "Sleeping cat" }
{ "classification": "Persian Cat" }
{ "classification": "Garden Costume" }
{ "classification": "Floral Fashion" }

If you already have a table with paths to the images, you can use the TO_FILE function to construct the FILE objects within the query:

SELECT SNOWFLAKE.CORTEX.COMPLETE('claude-3-5-sonnet',
    PROMPT('Classify the input image {0} in no more than 2 words. Respond in JSON',
        TO_FILE('@myimages', img_path)) AS image_classification
FROM image_table;
Copy

You can also retrieve the images to be processed directly from a stage’s directory, as shown here:

SELECT SNOWFLAKE.CORTEX.COMPLETE('claude-3-5-sonnet',
    PROMPT('Classify the input image {0} in no more than 2 words. Respond in JSON',
        TO_FILE('@myimages', RELATIVE_PATH))) as image_classification
FROM DIRECTORY(@myimages);
Copy

Providing images and prompts in a table

To perform a different operation on each image in a table, provide the images and their corresponding prompts in a table. In the following example, the table contains the stage path of each image in the img_path column and the prompt in the prompt column.

SNOWFLAKE.CORTEX.COMPLETE('claude-3-5-sonnet',
    PROMPT('Given the input image {0}, {1}. Respond in JSON',
        TO_FILE('@myimages', img_path), prompt) as image_result)
FROM image_table;
Copy
Language: English