EvaluateRagFaithfulness 2025.3.28.13-SNAPSHOT¶

BUNDLE¶

com.snowflake.openflow.runtime | runtime-rag-evaluation-processors-nar

DESCRIPTION¶

Evaluates the faithfulness of generated answers in a Retrieval-Augmented Generation (RAG) system by analyzing responses using an LLM (e.g., OpenAI’s GPT). The processor enriches each FlowFile record with faithfulness metrics and detailed analysis.

TAGS¶

ai, evaluation, faithfulness, llm, nlp, openai, openflow, rag

INPUT REQUIREMENT¶

REQUIRED

Supports Sensitive Dynamic Properties¶

false

PROPERTIES¶

Property	Description
Context Identifier Record Path	The RecordPath to the array of contexts IDs in the record.
Context Record Path	The RecordPath to the array of contexts in the record.
Evaluation Results Record Path	The RecordPath to write the results of the evaluation to.
Generated Answer Record Path	The path to the answer field in the record
LLM Provider Service	The provider service for sending evaluation prompts to LLM
Question Record Path	The RecordPath to the question field in the record.
Record Reader	The Record Reader to use for reading the FlowFile.
Record Writer	The Record Writer to use for writing the results.

RELATIONSHIPS¶

NAME	DESCRIPTION
failure	FlowFiles that cannot be processed are routed to this relationship
success	FlowFiles that are successfully processed are routed to this relationship

WRITES ATTRIBUTES¶

NAME	DESCRIPTION
average.answer.faithfulness	The average faithfulness score computed over all records.
json.parse.failures	Number of JSON parse failures encountered.

USE CASES¶

Use this processor to assess the faithfulness of answers generated by an LLM compared to the provided context. It provides metrics that can be used for monitoring and improving the performance of RAG systems.