设置 Openflow - BYOC¶

本主题介绍设置 Openflow 的步骤。

设置 Openflow 涉及以下步骤：

Create a deployment in your cloud
Create a Runtime environment in your cloud

先决条件

在您的 Snowflake 和 AWS 账户上需要满足的先决条件如下：

Snowflake 账户¶

You’ll need to first define privileges at the Snowflake account level.

Run the following SQL commands to grant the required privileges to the Openflow admin role:
```
USE ROLE ACCOUNTADMIN;

GRANT CREATE OPENFLOW DATA PLANE INTEGRATION ON ACCOUNT TO ROLE $openflow_admin_role;
GRANT CREATE OPENFLOW RUNTIME INTEGRATION ON ACCOUNT TO ROLE $openflow_admin_role;
```
The new privileges are assigned to the ACCOUNTADMIN role as part of the default set of privileges, and that role can grant the privileges to a role of their choosing for the Openflow admin role, denoted as $openflow_admin_role in the code.
Next, set default_secondary_roles to ALL for all Openflow users:
1. 使用您的 ACCOUNTADMIN 为使用 Openflow 而分配的角色登录 Snowflake。
  
  This may not be any of the following roles: ACCOUNTADMIN, ORGADMIN, GLOBALORGADMIN, or SECURITYADMIN.
  
  If you see a blank screen or the error “message: Invalid consent request” when logging into Openflow, change your role to a role that is not one of these listed roles.
  
  For more information, see Prerequisites.
2. Run the following code, replacing $openflow_user for each Openflow user:
```
USE ROLE ACCOUNTADMIN;
ALTER USER $openflow_user SET DEFAULT_SECONDARY_ROLES = ('ALL');
```

此设置是必需的，因为 Openflow 操作授权会使用任何经过身份验证的用户角色，而不仅仅是默认角色。

部署集成权限

部署集成对象表示为了部署一个或多个 Snowflake Openflow 运行时而预置的一组资源。对于自带云资源的组织，部署集成对象代表托管 Kubernetes 集群及其关联节点。

拥有 Snowflake 账户 CREATE DATA PLANE INTEGRATION 权限的用户可以创建和删除部署集成对象。

为支持区分访问权限，可以直接在部署集成对象上定义其他权限。

您可以授予对于部署集成对象的以下权限：

OWNERSHIP：允许完全控制部署操作对象，包括删除部署。
USAGE：允许创建运行时子对象。

运行时权限

The runtime object represents a cluster of one or more Snowflake Openflow runtime servers, provisioned to run flow definitions. For Kubernetes deployments, the runtime object represents a stateful set of Snowflake Openflow runtime containers deployed in a namespace, along with supporting components.

拥有父部署集成对象 OWNERSHIP 权限和 CREATE RUNTIME INTEGRATION 账户级别权限的用户可创建运行时集成对象。为支持区分访问权限，可以直接在运行时集成对象上定义其他权限。

您可以授予对于运行时集成对象的以下权限：

OWNERSHIP：允许完全控制运行时操作，包括删除相关运行时以及修改运行时流程定义。
USAGE: Enables read access to the deployed runtime for observing health and status, without making any changes.

Snowflake role¶

A Snowflake role is a Snowflake role that is associated with a specific Openflow runtime and used for the following tasks:

Grant access to Snowflake resources.
Grant access to connector-specific resources

Snowflake roles are linked to Openflow Snowflake Managed Token, avoiding the need for customers to create separate service users and key pairs for authentication to Snowflake.

Note

<RUNTIMENAME> denotes the name of the associated runtime.

To create a Snowflake role:

Create the required Snowflake role.

USE ROLE ACCOUNTADMIN;
CREATE ROLE IF NOT EXISTS OPENFLOW_RUNTIME_ROLE_<RUNTIMENAME>

Grant the Snowflake role access to a warehouse. Snowflake recommends using a dedicated warehouse for data ingestion. This warehouse should be used when configuring your connectors for runtimes where you will be using this Snowflake role.
```
GRANT USAGE, OPERATE ON WAREHOUSE <OPENFLOW_INGEST_WAREHOUSE> TO ROLE OPENFLOW_RUNTIME_ROLE_<RUNTIMENAME>;
```
Allow the Snowflake role to use, create or otherwise access Snowflake objects.

Note
Depending on the Openflow connector being created the required underlying objects will vary. The example below is for illustration purposes only.
```
GRANT USAGE ON DATABASE <OPENFLOW_DATABASE> TO ROLE OPENFLOW_RUNTIME_ROLE_<RUNTIMENAME>;
GRANT USAGE ON SCHEMA <OPENFLOW_SCHEMA> TO ROLE OPENFLOW_RUNTIME_ROLE_<RUNTIMENAME>;
```
Allow the user to use the Snowflake role
```
GRANT ROLE OPENFLOW_RUNTIME_ROLE_<RUNTIMENAME> TO USER <username>;
```

角色设置示例

考虑应设置以下角色的场景：

accountadmin：Snowflake 开箱提供的角色，它拥有以下两项 CREATE 权限：
- CREATE OPENFLOW DATA PLANE INTEGRATION
- CREATE OPENFLOW RUNTIME INTEGRATION
deployment_manager：可以创建、管理和删除部署。
deployment1_runtime_manager_1: Can create a runtime only within deployment 1. It can modify and delete a runtime that it created within deployment 1, but not a runtime created by deployment1_runtime_manager_2.
deployment1_runtime_manager_2: Can create a runtime only within deployment 1. It can modify and delete a runtime that it created within deployment 1, but not a runtime created by deployment1_runtime_manager_1.
deployment1_runtime_viewer_1: Can view a runtime canvas within deployment 1 that was created by deployment1_runtime_manager_1.
deployment1_runtime_viewer_2: Can view a runtime canvas within deployment 1 that was created by deployment1_runtime_manager_2.
deployment2_runtime_manager: Can create a runtime only within deployment 2.
deployment2_runtime_viewer: Can view a runtime canvas within deployment 2.

要使用这些角色设置 Openflow，请按照以下步骤操作：

创建新角色，并分配相关权限：

use role ACCOUNTADMIN;
create role if not exists deployment_manager;
create role if not exists deployment1_runtime_manager_1;
create role if not exists deployment1_runtime_manager_2;
create role if not exists deployment1_runtime_viewer_1;
create role if not exists deployment1_runtime_viewer_2;
create role if not exists deployment2_runtime_manager;
create role if not exists deployment2_runtime_viewer;


-- Assign create deployment privilege to roles. (This privilege cannot be granted in Openflow UI.)

grant create openflow data plane integration on account to role deployment_manager;


-- Assign create runtime privilege to roles. (This privilege cannot be granted in the Control Pane UI.)

grant create openflow runtime integration on account to role deployment1_runtime_manager_1;
grant create openflow runtime integration on account to role deployment1_runtime_manager_2;
grant create openflow runtime integration on account to role deployment2_runtime_manager;

-- Grant roles to users. (Repeat this step for each user.)

grant role <role name> to user <username>;

要创建部署，请按照以下步骤操作：
1. 以 deployment_manager 身份登录 Snowsight。
2. In the navigation menu, select Ingestion » Openflow.
3. To create deployment 1, select Create a deployment, and grant the USAGE privilege to deployment1_runtime_manager_1 and deployment1_runtime_manager_2.
4. To create deployment 2, select Create a deployment, and grant the USAGE privilege to deployment2_runtime_manager.
要在部署 1 中创建运行时，请执行以下步骤：
1. 以 deployment1_runtime_manager_1 的身份登录。
2. Create a runtime as described in the following sections. deployment1_runtime_manager_1 should be able to create runtimes and manage any runtimes it created within this deployment.
3. 在 Openflow UI 中，选择 deployment1_runtime_viewer_1 并授予其 USAGE 权限。

AWS 账户¶

确保您的 AWS 账户符合以下条件：

您有一个 AWS 账户，并且具备创建 CloudFormation 堆栈所需的权限。
An AWS administrator in your organization can execute CloudFormation script to set up Amazon Elastic Kubernetes Service (EKS) inside a new VPC (created by CloudFormation) or an existing VPC. See Prerequisites for BYO-VPC (existing VPC).

Note

To learn about how the Openflow installation happens in your AWS account and the permissions that are configured by the CloudFormation template, see Installation process.

BYO-VPC（现有 VPC）的先决条件¶

If you want to use an existing VPC and your own subnets, ensure that you have the following:

For Snowflake managed ingress, two public subnets with:
- Different availability zones
- At least /27 CIDR ranges with 32 available IPs.
- Routes for destination 0.0.0.0/0 and target internet gateway or some other egress routing to the internet.
- A tag that allows Openflow to create a load balancer:
  - Key: kubernetes.io/role/elb
  - Value: 1
- If your public subnets are used by other EKS clusters, a tag that allows Openflow to create a load balancer alongside other load balancers:
  - Key: kubernetes.io/cluster/{deployment-key}
  - Value: 1
Note
Managing your own ingress eliminates the need for public subnets, but requires additional configuration in your AWS account. For more information, see Openflow BYOC - Set up custom ingress.
Two private subnets with:
- Different availability zones
- At least /24 CIDR ranges with 255 available IPs. This limits the number and scale of runtimes you can create, so it may be more appropriate to use a larger range for the deployment.
- Connectivity to Snowflake and AWS services from Private Subnet 1 where the Openflow deployment runs.
  - Among many options, you can connect using route tables with a NAT Gateway, a Transit Gateway, or PrivateLink VPC Endpoints.
  - Without this connectivity, the Openflow deployment will not initialize or set up properly and no infrastructure will be provisioned.
- For Snowflake managed ingress, egress connectivity to LetsEncrypt.org (https://letsencrypt.org), which will provision a TLS certificate.

接受 Openflow 服务条款¶

贵组织仅需执行一次此步骤。

以具有 ORGADMIN 角色的用户身份登录 Snowflake。
In the navigation menu, select Ingestion » Openflow.
接受 Openflow 服务条款。

在您的云环境中创建部署

在 Snowflake 账户中配置部署¶

Important

使用您的 ACCOUNTADMIN 为使用 Openflow 而分配的角色登录 Snowflake。

This may not be any of the following roles: ACCOUNTADMIN, ORGADMIN, GLOBALORGADMIN, or SECURITYADMIN.

If you see a blank screen, or the error: “message: Invalid consent request”, when logging into Openflow, change your role to a role that is not one of these listed roles.

For more information, see Prerequisites.

In the navigation menu, select Ingestion » Openflow.
Select Launch Openflow.
In the Openflow UI, select Create a deployment.
On the Deployments tab, select Create a deployment. The Creating a deployment wizard opens.
In the Prerequisites step, ensure that you meet all the requirements, and then select Next.
In the Deployment location step, select Amazon Web Services as the deployment location, enter a name for your deployment, and then select Next.
In the Configuration step, select one of the following options:
- Managed VPC: Choose this option if you want your VPC to be managed by Snowflake.
- Bring your own VPC: Choose this option if you want to use an existing VPC.

In the PrivateLink step, you can select if you want to establish communication with Snowflake over the private link. Enabling this option requires additional setup in your AWS account. For more information, see AWS PrivateLink and Snowflake.
- If the PrivateLink option is enabled, the End user authentication over PrivateLink step displays.
  - If enabled, browser-based authentication redirects use PrivateLink endpoints.
  - If disabled, end-user authentication uses public Snowflake URLs.
  Regardless of this setting, Deployment communications to Snowflake will use PrivateLink.
  
  If you access Snowsight through a PrivateLink URL, ensure it is enabled. If you access Snowsight through a non-PrivateLink URL, leave it disabled.
In the Custom Ingress step, you can choose to manage your own ingress configuration for the Openflow deployment, such as specifying custom security groups, load balancer settings, or other network controls.

Enabling this option requires additional setup in your AWS account. For more information, see Openflow BYOC - Set up custom ingress.
Select Create Deployment.
配置好部署后，将出现一个对话框，允许您下载 CloudFormation 模板，以在您的 AWS 账户中完成设置过程。下载此模板。请注意，Openflow 不支持修改 CloudFormation 模板。下载模板后，除了选择下拉选项外，不要修改任何值。
(Optional) To encrypt EBS volumes for your Openflow BYOC deployment, see Openflow BYOC - Set up encrypted EBS volumes.

在 AWS 账户中，应用 CloudFormation 模板¶

In your AWS account, create a new CloudFormation Stack using the template. After the Openflow deployment agent’s Amazon Elastic Compute Cloud (EC2) instance is created, it completes the rest of the Installation process using infrastructure as code scripts. You can track the installation progress as described in Track the installation progress.

如果您使用现有 VPC，请在上传 CloudFormation 模板后，在下拉列表中为两个专用子网和您的 VPC 选择相应的值。

在 Snowflake 账户中为 Openflow 创建网络规则¶

只有在使用网络策略来控制 Snowflake 访问权限时，您才需要执行此步骤。网络策略是一组规则，用于控制哪些 IP 地址可访问您的 Snowflake 账户。

导航到您的 Snowflake 账户。
识别作为 CloudFormation 堆栈的一部分创建的 NAT 网关公共 IP 地址。您可以通过在 AWS 控制台上搜索 NAT 网关或检查 CloudFormation 堆栈的输出来找到此地址。

NAT 网关负责 Data Plane Agent (DPA) 和 EKS 的 Openflow 出口流量。网关负责和的 Openflow 出口。DPA 和 EKS 都在安装的专用子网 1 中运行。

Create a network rule for Openflow and add it to your existing network policy. Replace {$NAT_GATEWAY_PUBLIC_IP} in the following code snippet with the NAT gateway public IP address that was created as part of the CloudFormation stack.
```
USE ROLE ACCOUNTADMIN;
USE DATABASE {REPLACE_WITH_YOUR_DB_NAME};

CREATE NETWORK RULE allow_openflow_deployment
MODE = INGRESS
TYPE = IPV4
VALUE_LIST = ('{$NAT_GATEWAY_PUBLIC_IP}/32');
```
查找当前处于活动状态的网络策略。
```
SHOW PARAMETERS LIKE 'NETWORK_POLICY' IN ACCOUNT;
```

从输出中复制值列，并使用它来创建网络规则：

ALTER NETWORK POLICY {ENTER_YOUR_ACTIVE_NETWORK_POLICY_NAME} ADD ALLOWED_NETWORK_RULE_LIST = (allow_openflow_deployment);

设置事件表以记录 Openflow 事件（必填）¶

使用以下选项之一设置事件表：

创建特定于 Openflow 的新事件表（推荐）：

USE ROLE ACCOUNTADMIN;

CREATE DATABASE IF NOT EXISTS openflow;
USE openflow;
CREATE SCHEMA IF NOT EXISTS openflow;
USE SCHEMA openflow;

GRANT CREATE EVENT TABLE
  ON SCHEMA openflow.openflow
  TO ROLE $role_of_deployment_owner;
USE ROLE $role_of_deployment_owner;
CREATE EVENT TABLE IF NOT EXISTS openflow.openflow.openflow_events;
-- Find the Data Plane Integrations
SHOW OPENFLOW DATA PLANE INTEGRATIONS;
ALTER OPENFLOW DATA PLANE INTEGRATION
  $openflow_dataplane_name
  SET EVENT_TABLE = 'openflow.openflow.openflow_events';

创建特定于账户的事件表：

USE DATABASE openflow;
CREATE SCHEMA IF NOT EXISTS openflow.telemetry;
CREATE EVENT TABLE IF NOT EXISTS openflow.telemetry.events;
ALTER ACCOUNT SET EVENT_TABLE = openflow.telemetry.events;

使用现有的特定于账户的事件表：

USE ROLE ACCOUNTADMIN;
ALTER ACCOUNT SET EVENT_TABLE = 'existing_database.existing_schema.existing_event_table';

验证部署

In the navigation menu, select Ingestion » Openflow. Creating a deployment takes about 45 minutes on AWS. Once it’s created, you can view your deployment in the Deployments tab of Openflow UI with its state marked as Active.

在您的云环境中创建运行时环境

In Openflow Control Plane, select Create a runtime. The Create Runtime dialog box appears.
From the Deployment drop-down list, choose the deployment in which you want to create a runtime.
输入运行时名称。
Choose a node type from the Node type drop-down list. This specifies the size of your nodes.
In the Min/Max node range selector, select a range. The minimum value specifies the number of nodes that the runtime starts with when idle and the maximum value specifies the number of nodes that the runtime can scale up to, in the event of high data volume or CPU load.
Select Create. The runtime takes a couple of minutes to get created.

Once created, you can view your runtime by navigating to the Runtimes tab of the Openflow control plane. Click the runtime to open the Openflow canvas.

后续步骤

Deploy a connector in a runtime. For a list of connectors available in Openflow, see Openflow connectors.

网络注意事项：Openflow EKS 到源系统¶

对于 BYOC 部署，请注意以下注意事项：

Openflow CloudFormation 堆栈会创建一个 VPC，其中包含两个公共子网和两个专用子网。
公共子网托管 AWS 网络负载均衡器（稍后创建）。专用子网托管 EKS 集群和所有支持节点组的 EC2 实例。Openflow 运行时在专用子网 1 内运行。
NAT 网关目前是 DPA 和 EKS 的出口。DPA 和 EKS 都在安装的专用子网 1 中运行。

对于 BYO-VPC 部署，请注意以下注意事项：

Openflow 要求您输入将运行 Openflow 的两个专用子网和 AWS 负载均衡器的两个公共子网。
您必须提供自己的从这些专用子网到互联网的出口路由，这些子网可以是中央 NAT 网关。
Openflow 没有创建互联网网关。您必须提供适当的公共互联网出口路由。

The network connectivity generally is as follows: An Openflow EC2 Instance (Agent or EKS) runs in a private subnet that requires Route Table entries to send egress traffic to a Transit Gateway, a PrivateLink VPC Endpoint, or a NAT Gateway connected to an Internet Gateway.

示例：使用新 VPC 进行 BYOC 部署，以便在相同账户的不同 VPC 中与 RDS 通信¶

要启用 Openflow EKS 集群与 RDS 实例之间的通信，您需要创建一个新的安全组，将该 EKS 集群安全组作为 RDS 连接入站规则的来源，并将该组连接到 RDS 中。

Find the EKS cluster security group, navigate to EKS and find your deployment key. You can also find it on the Openflow UI by performing the following steps:
1. 登录 Openflow。
2. Go to the Deployments tab.
3. 选择您的部署旁边的“More options”图标。
4. Select View details. The value in the field Key is your deployment key.
找到部署密钥后，您可以使用它按密钥值筛选 AWS 资源。
Create a new security group that allows access from the Openflow EKS cluster using the relevant database port. For PostgreSQL the default port is 5432.
在 RDS 中将其附加为新的安全组。

If you need to troubleshoot, the Reachability Analyzer (https://docs.aws.amazon.com/vpc/latest/reachability/getting-started.html) can be useful. It will give you detailed information about what may be blocking connectivity by using tracing capabilities within the AWS platform.

有关使用 VPC 对等互连和相关的安全组配置访问 DB 实例的信息，请参阅以下 AWS 文档：

Scenarios for accessing a DB instance in a VPC - Amazon Relational Database Service (https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.Scenarios.html#USER_VPC.Scenario3)
Update your security groups to reference peer security groups - Amazon Virtual Private Cloud (https://docs.aws.amazon.com/vpc/latest/peering/vpc-peering-security-groups.html)

在 AWS 中配置 PrivateLink¶

本节介绍如何使用专用连接访问和配置 Openflow。

通过 PrivateLink 访问 Openflow¶

Before starting with the private link configuration, enable PrivateLink for your account as described in AWS PrivateLink and Snowflake.

Using the ACCOUNTADMIN role, call the SYSTEM$GET_PRIVATELINK_CONFIG function in your Snowflake account and identify the value for openflow-privatelink-url. This is the URL for accessing Openflow over PrivateLink.
Create a CNAME record in your DNS to resolve the URL value to your VPC endpoint.
确认 DNS 设置可以解析该值。
确认您可以通过浏览器使用此 URL 连接到 Openflow UI。

使用 PrivateLink 配置新部署¶

Note

Snowflake recommends that you use the Bring your own VPC version of Openflow deployment and create a VPC endpoint in your VPC before applying the CloudFormation template.

Before starting with the PrivateLink configuration, make sure that PrivateLink is enabled for your account as described in AWS PrivateLink and Snowflake.

请执行以下步骤：

检索 Snowflake 的 VPC 端点服务 ID 和 Openflow PrivateLink URLs：
1. Run the following SQL command using the ACCOUNTADMIN role:
  SELECT SYSTEM$GET_PRIVATELINK_CONFIG()
2. 在输出中，识别并保存以下键的值：
  - privatelink-vpce-id
  - openflow-privatelink-url
  - external-telemetry-privatelink-url
使用以下参数创建 VPC 端点：
- Type: PrivateLink Ready partner services
- Service: privatelink-vpce-id value obtained in the previous step.
- VPC：选择使用时默认使用的角色和仓库。您的 Openflow 部署将在其中运行的 VPC。
- 子网：选择您的 Openflow 部署将在其中运行的两个可用区域和专用子网。
使用以下参数设置 Route 53 专用托管区域：
1. Domain: privatelink.snowflakecomputing.cn
2. Type: Private hosted zone
3. 选择您的 Openflow 部署将在其中运行的区域和 VPC。
Add two CNAME records for the URLs identified in the first step:
1. For openflow-privatelink-url
  - Record name: openflow-privatelink-url value obtained in the first step
  - Record type: CNAME
  - 值：VPC 端点的 DNS 名称
2. For external-telemetry-privatelink-url
  - Record name: external-telemetry-privatelink-url value obtained in the first step
  - Record type: CNAME
  - 值：VPC 端点的 DNS 名称
为部署创建专用的安全组，并启用从安全组到 VPC 端点的流量：
1. 打开与 VPC 端点关联的安全组。
2. Add an inbound rule to the security group that allows All traffic from the security group created for your deployment.
Create a new deployment and apply the CloudFormation Stack following the instructions in the Create a deployment in your cloud section and ensure that:
- The PrivateLink option is enabled. The End user authentication over PrivateLink option can be either enabled or disabled.
- 创建 CloudFormation 堆栈时使用为部署创建的安全组。
Wait until the EKS cluster for your deployment is created. To confirm successful creation, navigate to AWS Console under Elastic Kubernetes Service. Verify that a cluster identified as <deployment-key> displays status ACTIVE.
允许流量从 EKS 流向 VPC 端点：
1. 打开与 VPC 端点关联的安全组。
2. Add an inbound rule to the security group that allows All traffic from the security group assigned to your EKS cluster. The EKS cluster’s security group starts with eks-cluster-sg-<deployment-key>-.

在 AWS 中为 S3 配置 VPC 网关端点¶

为 S3 配置 AWS VPC 网关端点是允许专用子网中的代理 EC2 实例私密地访问 Amazon Linux 2023 存储库的主要方法，该实例上不需要互联网网关、NAT 网关或公共 IP 地址。代理 EC2 实例使用此存储库来安装其依赖项，例如 Docker。

要为 S3 配置 VPC 网关端点，请执行以下操作：

打开浏览器以访问 AWS VPC 仪表板。
In the navigation pane, select Endpoints.
Click Create endpoint and create a new VPC endpoint with parameters:
- Type: AWS services
- Service: com.amazonaws.<your-region>.s3 of type Gateway
- VPC：选择部署的 VPC
- 路由表：选择与专用子网关联的路由表
- Policy: Choose Full access

配置专用部署

专用部署是一项功能，允许您在 VPC 中部署 Openflow，而无需公共互联网入口或出口。

要配置专用部署，在创建新部署时需要选择以下选项：

In the Deployment location step, select Amazon Web Services as the deployment location.
In the VPC Configuration step, select Bring your own VPC to use an existing VPC.
In the PrivateLink step, enable the PrivateLink feature. Enabling this option requires additional setup in your AWS account, see Configuring PrivateLink in AWS. The End user authentication over PrivateLink option can be either enabled or disabled.
In the Custom ingress step, enable the custom ingress feature. Enabling this option requires additional setup in your AWS account. For more information, see Openflow BYOC - Set up custom ingress.

专用部署需要现有的 VPC 能够访问以下域：

*.amazonaws.com, a detailed list of services being accessed includes:
- com.amazonaws.iam
- com.amazonaws.<your-region>.s3
- com.amazonaws.<your-region>.ec2
- com.amazonaws.<your-region>.ecr.api
- com.amazonaws.<your-region>.ecr.dkr
- com.amazonaws.<your-region>.secretsmanager
- com.amazonaws.<your-region>.sts
- com.amazonaws.<your-region>.eks
- com.amazonaws.<your-region>.autoscaling
*.privatelink.snowflakecomputing.cn
oidc-eks.<your-region>.api.aws
shield.us-east-1.amazonaws.com

安装过程

在 CloudFormation 堆栈和 Openflow 代理之间，BYOC 部署安装过程管理多个协调的步骤。目标是划分责任：冷启动（通过 CloudFormation 解决）用于为组织提供一种便捷的方法，从而为其 BYOC 部署提供输入；以及部署的配置及其核心软件组件，这需要随着时间推移而更改（通过 Openflow Agent 解决）。

部署代理有助于创建 Openflow 部署基础设施和安装部署软件组件（包括部署服务）。部署代理使用 Snowflake 系统镜像注册表进行身份验证，以获取 Openflow 容器镜像。

步骤如下：

Note

使用 BYO-VPC 时，您将从模板中选择一个 VPC ID 和两个专用子网 IDs，CloudFormation 堆栈将使用选定专用子网，而不是创建步骤 1a、1b 和 1c 中提到的资源。

The CloudFormation template creates the following and configures with the AWS permissions mentioned in Configured AWS permissions:
1. One VPC with two public subnets and two private subnets. Public subnets host the AWS Network Load Balancer (created later). Private Subnets host the EKS cluster and all of the EC2 instances backing the NodeGroups. Openflow runtimes run within a private subnet.
2. 用于从 VPC 出口的互联网网关
3. 专用子网出口的 NAT 网关
4. 用户输入的 OIDC 配置的 AWS Secrets Manager 条目
5. 供 Openflow 代理通过其 EC2 实例使用的 IAM 角色和实例配置文件
6. An EC2 instance for Openflow deployment agent, complete with a UserData script to automatically run the initialization process. This script sets environment variables for the Openflow deployment agent to use, derived from the input CloudFormation parameters.
7. EC2 Instance Connect endpoint for the Openflow deployment agent to upgrade the deployment when needed.
  - When using BYO-VPC, by default the CloudFormation stack will create an EC2 Instance Connect endpoint. However, this default behavior can be modified. When using the managed VPC option, the CloudFormation stack will always create an EC2 Instance Connect endpoint.
  - Instance Connect 端点可在多个 VPCs 之间共享。
  - 如果删除了部署，同时删除了 CloudFormation 堆栈，它还将移除端点。如果端点为共享，这将阻止对其他 BYO VPC 代理的访问。
  - 要添加 EC2 Instance Connect 端点，请在您的 AWS 账户中执行以下步骤：
    1. In the left navigation, navigate to VPC » Endpoints.
    2. Select Create Endpoint.
    3. Choose the endpoint type as EC2 Instance Connect Endpoint.
    4. 选择 VPC。清除所有安全组（即使其成为未选中状态），以使用默认 VPC 安全组。
    5. 选择子网时，使用与 CloudFormation 参数中的专用子网 1 相同的值。
    6. Select Create. It takes approximately 5 minutes for the endpoint to be created.
8. S3 Bucket that stores the Terraform state, logs, and outputs for the Openflow Agent
Openflow 部署代理会创建以下内容：
1. 包含以下元素的 EKS 集群：
- 节点组
- 自动缩放组
- AWS VPC Container Network Interface (CNI) 附加组件
- Amazon Elastic Block Store (EBS) CSI 附加组件
1. PostgreSQL、OAuth 凭据等的密钥管理器记录。
2. IAM policies and roles for various K8s service accounts to retrieve their secrets from AWS Secrets Manager.
3. K8s 组件
- 命名空间
- 集群自动扩缩器
- EBS CSI 可扩展存储
- AWS 负载均衡器控制器，用于创建可公开访问的网络负载均衡器
- Let’s Encrypt 证书颁发机构
- Nginx Ingress，为 Let’s Encrypt 进行了配置
- Metrics Server
- Certificate manager from Jetstack (http://jetstack.io/)
- External secrets operator (http://external-secrets.io/)
- 用于临时、部署服务和 OIDC 的服务账户
- 用于临时服务、部署服务和 OIDC 的密钥存储
- 用于临时和部署服务的外部密钥。OIDC 的外部密钥由运行时操作器创建和管理。
- PostgreSQL
- 临时
- 用于运行时节点之间通信的自签名证书颁发机构和入口配置
- Openflow 运行时操作器
- Openflow 部署服务

默认情况下，所有 AWS 账户在每个区域都有五个弹性 IP 地址的配额，因为公共 (IPv4) 互联网地址是一种稀缺的公共资源。Snowflake 强烈建议您使用弹性 IP 地址，主要是因为在实例出现故障时，它们能够将地址重新映射到另一个实例，并使用 DNS 主机名进行所有其他节点间通信。

追踪安装进度

在 CloudFormation 堆栈进入 CREATE_COMPLETE 状态后，Openflow 代理会自动创建基础设施的其余部分。

其中涉及到几个步骤，每个步骤都可能需要 10-15 分钟完成，例如：

创建 EKS 集群
将 EBS CSI 附加组件安装到 EKS 集群
创建 RDS PostgreSQL 数据库

Openflow 代理的状态报告尚不可用。在此期间，您可以查看 Openflow 代理上的日志，以验证 BYOC 部署是否已做好运行准备。为此，请执行以下步骤：

在 EC2 实例列表中，找到以下两个实例：
- openflow-agent-{data-plane-key}: This is the Openflow agent that you will use to manage runtimes
- {data-plane-key}-mgmt-group: This is a node in the BYOC deployment’s EKS cluster that runs an operator and other core software
Right-click on the openflow-agent-{data-plane-key} instance and select Connect.
Switch from EC2 Instance Connect to Connect using EC2 Instance Connect Endpoint. Leave the default EC2 Instance Connect Endpoint in place.
Click Connect. A new browser tab or window will appear with a command-line interface.
运行以下命令，查看正在配置您的部署的 docker 镜像的安装日志：
```
journalctl -xe -f -n 100 -u docker
```

安装完成后，您将看到以下输出：

{timestamp} - app stack applied successfully
{timestamp} - All resources applied successfully

已配置的 AWS 权限¶

此部分列出了 Openflow BYOC 堆栈根据角色配置的 AWS 权限。

Note

{key} represents the deployment key that uniquely identifies cloud resources created and managed by Openflow for a particular deployment.

管理用户

cloudformation and all of the following permissions.

IAM Role: openflow-agent-role-{key}

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "autoscaling:DescribeTags",
               "ec2:DescribeImages",
               "ec2:DescribeInstances",
               "ec2:DescribeLaunchTemplates",
               "ec2:DescribeLaunchTemplateVersions",
               "ec2:DescribeNetworkInterfaces",
               "ec2:DescribeSecurityGroups",
               "ec2:DescribeSubnets",
               "ec2:DescribeTags",
               "ec2:DescribeVolumes",
               "ec2:DescribeVpcs",
               "ec2:DescribeVpcAttribute",
               "iam:GetRole",
               "iam:GetOpenIDConnectProvider",
               "ecr:GetAuthorizationToken",
               "ec2:RunInstances",
               "ec2:CreateLaunchTemplate",
               "ec2:CreateSecurityGroup",
               "ec2:CreateTags",
               "ec2:DeleteTags"
            ],
            "Resource": "*",
            "Effect": "Allow"
      },
      {
            "Condition": {
               "StringLike": {
                  "aws:ResourceTag/Name": [
                        "{key}-oidc-provider"
                  ]
               }
            },
            "Action": [
               "iam:CreateOpenIDConnectProvider",
               "iam:DeleteOpenIDConnectProvider",
               "iam:TagOpenIDConnectProvider"
            ],
            "Resource": "arn:aws:iam::{Account_ID}:oidc-provider/oidc.eks.{Region}.amazonaws.com/id/*",
            "Effect": "Allow"
      },
      {
            "Action": [
               "iam:DeletePolicy",
               "iam:CreatePolicy",
               "iam:GetPolicy",
               "iam:GetPolicyVersion",
               "iam:ListPolicyVersions"
            ],
            "Resource": [
               "arn:aws:iam::{Account_ID}:policy/dp-service-role-policy-{key}",
               "arn:aws:iam::{Account_ID}:policy/oauth2-role-policy-{key}",
               "arn:aws:iam::{Account_ID}:policy/temporal-service-role-policy-{key}",
               "arn:aws:iam::{Account_ID}:policy/oidc-service-role-policy-{key}",
               "arn:aws:iam::{Account_ID}:policy/dps-temporal-role-policy-{key}"
               "arn:aws:iam::{Account_ID}:policy/dps-postgres-role-policy-{key}"
            ],
            "Effect": "Allow"
      },
      {
            "Action": [
               "iam:UpdateAssumeRolePolicy",
               "iam:PutRolePolicy",
               "iam:ListInstanceProfilesForRole",
               "iam:ListAttachedRolePolicies",
               "iam:ListRolePolicies",
               "iam:GetRolePolicy",
               "iam:CreateRole",
               "iam:AttachRolePolicy",
               "iam:DeleteRole",
               "iam:DeleteRolePolicy",
               "iam:DetachRolePolicy",
               "iam:TagRole"
            ],
            "Resource": [
               "arn:aws:iam::{Account_ID}:role/openflow-agent-role-{key}",
               "arn:aws:iam::{Account_ID}:role/{key}-*",
               "arn:aws:iam::{Account_ID}:role/dps-temporal-role-{key}",
               "arn:aws:iam::{Account_ID}:role/dps-postgres-role-{key}",
               "arn:aws:iam::{Account_ID}:role/dp-service-role-{key}",
               "arn:aws:iam::{Account_ID}:role/oauth2-role-{key}",
               "arn:aws:iam::{Account_ID}:role/oidc-service-role-{key}"
            ],
            "Effect": "Allow"
      },
      {
            "Action": [
               "autoscaling:CreateOrUpdateTags",
               "autoscaling:DeleteTags"
            ],
            "Resource": "arn:aws:autoscaling:{Region}:{Account_ID}:autoScalingGroup:*:autoScalingGroupName/eks-{key}-*",
            "Effect": "Allow"
      },
      {
            "Condition": {
               "StringLike": {
                  "aws:ResourceTag/Name": [
                        "{key}-EC2SecurityGroup-*",
                        "k8s-traffic-{key}-*",
                        "eks-cluster-sg-{key}-*",
                        "{key}-cluster-sg",
                        "postgres-{key}-sg"
                  ]
               }
            },
            "Action": [
               "ec2:AuthorizeSecurityGroupEgress",
               "ec2:AuthorizeSecurityGroupIngress",
               "ec2:RevokeSecurityGroupEgress",
               "ec2:DeleteSecurityGroup",
               "ec2:CreateTags",
               "ec2:DeleteTags",
               "ec2:CreateNetworkInterface",
               "ec2:DeleteNetworkInterface"
            ],
            "Resource": "arn:aws:ec2:{Region}:{Account_ID}:security-group/*",
            "Effect": "Allow"
      },
      {
            "Condition": {
               "StringLike": {
                  "aws:ResourceTag/elbv2.k8s.aws/cluster": "{key}"
               }
            },
            "Action": [
               "ec2:AuthorizeSecurityGroupEgress",
               "ec2:AuthorizeSecurityGroupIngress",
               "ec2:RevokeSecurityGroupEgress",
               "ec2:DeleteSecurityGroup",
               "ec2:CreateTags",
               "ec2:DeleteTags",
               "ec2:CreateNetworkInterface",
               "ec2:DeleteNetworkInterface"
            ],
            "Resource": "arn:aws:ec2:{Region}:{Account_ID}:security-group/*",
            "Effect": "Allow"
      },
      {
            "Action": [
               "ec2:CreateSecurityGroup"
            ],
            "Resource": "arn:aws:ec2:{Region}:{Account_ID}:vpc/vpc-018d2da0fde903de4",
            "Effect": "Allow"
      },
      {
            "Condition": {
               "StringLike": {
                  "ec2:ResourceTag/Name": "openflow-agent-{key}"
               }
            },
            "Action": [
               "ec2:AttachNetworkInterface"
            ],
            "Resource": "arn:aws:ec2:{Region}:{Account_ID}:instance/*",
            "Effect": "Allow"
      },
      {
            "Condition": {
               "StringLike": {
                  "aws:ResourceTag/Name": "{key}-*-group"
               }
            },
            "Action": [
               "ec2:DeleteLaunchTemplate"
            ],
            "Resource": "arn:aws:ec2:{Region}:{Account_ID}:launch-template/*",
            "Effect": "Allow"
      },
      {
            "Action": [
               "eks:CreateCluster",
               "eks:CreateAccessEntry",
               "eks:CreateAddon",
               "eks:CreateNodegroup",
               "eks:DeleteCluster",
               "eks:DescribeCluster",
               "eks:ListClusters",
               "eks:ListNodeGroups",
               "eks:DescribeUpdate",
               "eks:UpdateClusterConfig",
               "eks:TagResource"
            ],
            "Resource": "arn:aws:eks:{Region}:{Account_ID}:cluster/{key}",
            "Effect": "Allow"
      },
      {
            "Action": [
               "eks:DescribeAddon",
               "eks:DescribeAddonVersions",
               "eks:UpdateAddon",
               "eks:DeleteAddon",
               "eks:DescribeUpdate"
            ],
            "Resource": "arn:aws:eks:{Region}:{Account_ID}:addon/{key}/*",
            "Effect": "Allow"
      },
      {
            "Action": [
               "eks:DeleteNodegroup",
               "eks:DescribeNodegroup",
               "eks:ListNodegroups",
               "eks:UpdateNodegroupConfig",
               "eks:TagResource",
               "eks:DescribeUpdate"
            ],
            "Resource": "arn:aws:eks:{Region}:{Account_ID}:nodegroup/{key}/*",
            "Effect": "Allow"
      },
      {
            "Action": [
               "s3:CreateBucket",
               "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::byoc-tf-state-{key}",
            "Effect": "Allow"
      },
      {
            "Action": [
               "s3:DeleteObject",
               "s3:GetObject",
               "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::byoc-tf-state-{key}/*",
            "Effect": "Allow"
      },
      {
            "Action": [
               "secretsmanager:CreateSecret",
               "secretsmanager:DeleteSecret",
               "secretsmanager:DescribeSecret",
               "secretsmanager:GetResourcePolicy",
               "secretsmanager:GetSecretValue",
               "secretsmanager:PutSecretValue",
               "secretsmanager:UpdateSecretVersionStage"
            ],
            "Resource": "arn:aws:secretsmanager:{Region}:{Account_ID}:secret:*-{key}*",
            "Effect": "Allow"
      },
      {
            "Action": [
               "ecr:BatchCheckLayerAvailability",
               "ecr:BatchGetImage",
               "ecr:DescribeImages",
               "ecr:DescribeRepositories",
               "ecr:GetDownloadUrlForLayer",
               "ecr:ListImages"
            ],
            "Resource": "arn:aws:ecr:{Region}:{Account_ID}:*",
            "Effect": "Allow"
      },
      {
            "Action": [
               "ecr:CreateRepository",
               "ecr:CompleteLayerUpload",
               "ecr:InitiateLayerUpload",
               "ecr:PutImage",
               "ecr:UploadLayerPart"
            ],
            "Resource": "arn:aws:ecr:{Region}:{Account_ID}:repository/snowflake-openflow/*",
            "Effect": "Allow"
      },
      {
            "Condition": {
               "StringLike": {
                  "iam:AWSServiceName": "eks.amazonaws.com"
               }
            },
            "Action": [
               "iam:CreateServiceLinkedRole"
            ],
            "Resource": "arn:aws:iam::*:role/aws-service-role/eks.amazonaws.com/AWSServiceRoleForAmazonEKS",
            "Effect": "Allow"
      },
      {
            "Condition": {
               "StringLike": {
                  "iam:AWSServiceName": "eks-nodegroup.amazonaws.com"
               }
            },
            "Action": [
               "iam:CreateServiceLinkedRole"
            ],
            "Resource": "arn:aws:iam::*:role/aws-service-role/eks-nodegroup.amazonaws.com/AWSServiceRoleForAmazonEKSNodegroup",
            "Effect": "Allow"
      },
      {
            "Action": [
               "eks:AssociateAccessPolicy",
               "eks:ListAssociatedAccessPolicies",
               "eks:DisassociateAccessPolicy"
            ],
            "Resource": "arn:aws:eks:{Region}:{Account_ID}:access-entry/{key}/*",
            "Effect": "Allow"
      },
      {
            "Action": "iam:PassRole",
            "Resource": "*",
            "Effect": "Allow"
      }
   ]
}

IAM Role: {key}-cluster-ServiceRole

AWS 托管策略：

AmazonEKSClusterPolicy
AmazonEKSVPCResourceController

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "cloudwatch:PutMetricData"
            ],
            "Effect": "Allow",
            "Resource": "*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "ec2:DescribeAccountAttributes",
               "ec2:DescribeAddresses",
               "ec2:DescribeInternetGateways"
            ],
            "Effect": "Allow",
            "Resource": "*"
      }
   ]
}

IAM Role: {key}-addon-vpc-cni-Role

AWS 托管策略：

AmazonEKS_CNI_Policy

IAM Role: {key}-eks-role

AWS 托管策略：

AmazonEBSCSIDriverPolicy
AmazonEC2ContainerRegistryReadOnly
AmazonEKS_CNI_Policy
AmazonEKSWorkerNodePolicy
AmazonSSMManagedInstanceCore
AutoScalingFullAccess
ElasticLoadBalancingFullAccess

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "ec2:CreateSecurityGroup",
               "ec2:CreateTags"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:ec2:{Region}:{Account_ID}:security-group/*",
               "arn:aws:ec2:{Region}:{Account_ID}:vpc/{VPC_ID}"
            ],
            "Sid": "CreateOpenflowEKSSecurityGroupAndTags"
      },
      {
            "Action": [
               "ec2:AuthorizeSecurityGroupIngress",
               "ec2:DeleteSecurityGroup"
            ],
            "Condition": {
               "StringLike": {
                  "aws:ResourceTag/Name": "eks-cluster-sg-{key}-*"
               }
            },
            "Effect": "Allow",
            "Resource": [
               "arn:aws:ec2:{Region}:{Account_ID}:security-group/*"
            ],
            "Sid": "OpenflowManageEKSSecurityGroup"
      }
   ]
}

Note

{VPC_ID} represents the identifier of the VPC that was either created by BYOC or used by BYO-VPC.

IAM Role: oidc-service-role-{key}

{
   "Statement": [
      {
            "Action": [
               "secretsmanager:GetSecretValue",
               "secretsmanager:DescribeSecret",
               "secretsmanager:GetResourcePolicy",
               "secretsmanager:ListSecretVersionIds"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:secretsmanager:{Region}:{Account_ID}:secret:oidc-{key}*"
            ]
      }
   ],
   "Version": "2012-10-17"
}

IAM Role: dps-postgres-role-{key}

{
   "Statement": [
      {
            "Action": [
               "secretsmanager:GetSecretValue",
               "secretsmanager:DescribeSecret",
               "secretsmanager:GetResourcePolicy",
               "secretsmanager:ListSecretVersionIds"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:secretsmanager:{Region}:{Account_ID}:secret:postgres_creds-{key}*"
            ]
      }
   ],
   "Version": "2012-10-17"
}

IAM Role: dps-temporal-role-{key}

{
   "Statement": [
      {
            "Action": [
               "secretsmanager:GetSecretValue",
               "secretsmanager:DescribeSecret",
               "secretsmanager:GetResourcePolicy",
               "secretsmanager:ListSecretVersionIds"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:secretsmanager:{Region}:{Account_ID}:secret:temporal_creds-{key}*"
            ]
      }
   ],
   "Version": "2012-10-17"
}

IAM Role: dp-service-role-{key}

{
   "Statement": [
      {
            "Action": [
               "secretsmanager:GetSecretValue",
               "secretsmanager:DescribeSecret",
               "secretsmanager:GetResourcePolicy",
               "secretsmanager:ListSecretVersionIds"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:secretsmanager:{Region}:{Account_ID}:secret:dps_creds-{key}*",
               "arn:aws:secretsmanager:{Region}:{Account_ID}:secret:snowflake-oauth2-{key}*"
            ]
      }
   ],
   "Version": "2012-10-17"
}

IAM Role: oauth2-role-{key}

{
   "Statement": [
      {
            "Action": [
               "secretsmanager:GetSecretValue",
               "secretsmanager:DescribeSecret",
               "secretsmanager:GetResourcePolicy",
               "secretsmanager:ListSecretVersionIds"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:secretsmanager:{Region}:{Account_ID}:secret:snowflake-oauth2-{key}*"
            ]
      }
   ],
   "Version": "2012-10-17"
}

IAM Role: {key}-nodegroup-NodeInstanceRole

AWS 托管策略：

AmazonEBSCSIDriverPolicy
AmazonEC2ContainerRegistryReadOnly
AmazonEKS_CNI_Policy
AmazonEKSWorkerNodePolicy
AmazonSSMManagedInstanceCore
AutoScalingFullAccess
ElasticLoadBalancingFullAccess

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "servicediscovery:CreateService",
               "servicediscovery:DeleteService",
               "servicediscovery:GetService",
               "servicediscovery:GetInstance",
               "servicediscovery:RegisterInstance",
               "servicediscovery:DeregisterInstance",
               "servicediscovery:ListInstances",
               "servicediscovery:ListNamespaces",
               "servicediscovery:ListServices",
               "servicediscovery:GetInstancesHealthStatus",
               "servicediscovery:UpdateInstanceCustomHealthStatus",
               "servicediscovery:GetOperation",
               "route53:GetHealthCheck",
               "route53:CreateHealthCheck",
               "route53:UpdateHealthCheck",
               "route53:ChangeResourceRecordSets",
               "route53:DeleteHealthCheck",
               "appmesh:*"
            ],
            "Effect": "Allow",
            "Resource": "*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "autoscaling:DescribeAutoScalingGroups",
               "autoscaling:DescribeAutoScalingInstances",
               "autoscaling:DescribeLaunchConfigurations",
               "autoscaling:DescribeScalingActivities",
               "autoscaling:DescribeTags",
               "ec2:DescribeInstanceTypes",
               "ec2:DescribeLaunchTemplateVersions"
            ],
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "autoscaling:SetDesiredCapacity",
               "autoscaling:TerminateInstanceInAutoScalingGroup",
               "ec2:DescribeImages",
               "ec2:GetInstanceTypesFromInstanceRequirements",
               "eks:DescribeNodegroup"
            ],
            "Effect": "Allow",
            "Resource": "*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "iam:CreateServiceLinkedRole"
            ],
            "Condition": {
               "StringEquals": {
                  "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:DescribeAccountAttributes",
               "ec2:DescribeAddresses",
               "ec2:DescribeAvailabilityZones",
               "ec2:DescribeInternetGateways",
               "ec2:DescribeVpcs",
               "ec2:DescribeVpcPeeringConnections",
               "ec2:DescribeSubnets",
               "ec2:DescribeSecurityGroups",
               "ec2:DescribeInstances",
               "ec2:DescribeNetworkInterfaces",
               "ec2:DescribeTags",
               "ec2:GetCoipPoolUsage",
               "ec2:DescribeCoipPools",
               "elasticloadbalancing:DescribeLoadBalancers",
               "elasticloadbalancing:DescribeLoadBalancerAttributes",
               "elasticloadbalancing:DescribeListeners",
               "elasticloadbalancing:DescribeListenerCertificates",
               "elasticloadbalancing:DescribeSSLPolicies",
               "elasticloadbalancing:DescribeRules",
               "elasticloadbalancing:DescribeTargetGroups",
               "elasticloadbalancing:DescribeTargetGroupAttributes",
               "elasticloadbalancing:DescribeTargetHealth",
               "elasticloadbalancing:DescribeTags"
            ],
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "cognito-idp:DescribeUserPoolClient",
               "acm:ListCertificates",
               "acm:DescribeCertificate",
               "iam:ListServerCertificates",
               "iam:GetServerCertificate",
               "waf-regional:GetWebACL",
               "waf-regional:GetWebACLForResource",
               "waf-regional:AssociateWebACL",
               "waf-regional:DisassociateWebACL",
               "wafv2:GetWebACL",
               "wafv2:GetWebACLForResource",
               "wafv2:AssociateWebACL",
               "wafv2:DisassociateWebACL",
               "shield:GetSubscriptionState",
               "shield:DescribeProtection",
               "shield:CreateProtection",
               "shield:DeleteProtection"
            ],
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:AuthorizeSecurityGroupIngress",
               "ec2:RevokeSecurityGroupIngress"
            ],
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:CreateSecurityGroup"
            ],
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:CreateTags"
            ],
            "Condition": {
               "Null": {
                  "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
               },
               "StringEquals": {
                  "ec2:CreateAction": "CreateSecurityGroup"
               }
            },
            "Effect": "Allow",
            "Resource": "arn:aws:ec2:*:*:security-group/*"
      },
      {
            "Action": [
               "ec2:CreateTags",
               "ec2:DeleteTags"
            ],
            "Condition": {
               "Null": {
                  "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                  "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
               }
            },
            "Effect": "Allow",
            "Resource": "arn:aws:ec2:*:*:security-group/*"
      },
      {
            "Action": [
               "ec2:AuthorizeSecurityGroupIngress",
               "ec2:RevokeSecurityGroupIngress",
               "ec2:DeleteSecurityGroup"
            ],
            "Condition": {
               "Null": {
                  "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "elasticloadbalancing:CreateLoadBalancer",
               "elasticloadbalancing:CreateTargetGroup"
            ],
            "Condition": {
               "Null": {
                  "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "elasticloadbalancing:CreateListener",
               "elasticloadbalancing:DeleteListener",
               "elasticloadbalancing:CreateRule",
               "elasticloadbalancing:DeleteRule"
            ],
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "elasticloadbalancing:AddTags",
               "elasticloadbalancing:RemoveTags"
            ],
            "Condition": {
               "Null": {
                  "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                  "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
               }
            },
            "Effect": "Allow",
            "Resource": [
               "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
               "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
               "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ]
      },
      {
            "Action": [
               "elasticloadbalancing:AddTags",
               "elasticloadbalancing:RemoveTags"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
               "arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
               "arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
               "arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*"
            ]
      },
      {
            "Action": [
               "elasticloadbalancing:ModifyLoadBalancerAttributes",
               "elasticloadbalancing:SetIpAddressType",
               "elasticloadbalancing:SetSecurityGroups",
               "elasticloadbalancing:SetSubnets",
               "elasticloadbalancing:DeleteLoadBalancer",
               "elasticloadbalancing:ModifyTargetGroup",
               "elasticloadbalancing:ModifyTargetGroupAttributes",
               "elasticloadbalancing:DeleteTargetGroup"
            ],
            "Condition": {
               "Null": {
                  "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "elasticloadbalancing:AddTags"
            ],
            "Condition": {
               "Null": {
                  "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
               },
               "StringEquals": {
                  "elasticloadbalancing:CreateAction": [
                        "CreateTargetGroup",
                        "CreateLoadBalancer"
                  ]
               }
            },
            "Effect": "Allow",
            "Resource": [
               "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
               "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
               "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ]
      },
      {
            "Action": [
               "elasticloadbalancing:RegisterTargets",
               "elasticloadbalancing:DeregisterTargets"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*"
      },
      {
            "Action": [
               "elasticloadbalancing:SetWebAcl",
               "elasticloadbalancing:ModifyListener",
               "elasticloadbalancing:AddListenerCertificates",
               "elasticloadbalancing:RemoveListenerCertificates",
               "elasticloadbalancing:ModifyRule"
            ],
            "Effect": "Allow",
            "Resource": "*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "route53:ChangeResourceRecordSets"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:route53:::hostedzone/*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "route53:GetChange"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:route53:::change/*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "route53:ListResourceRecordSets",
               "route53:ListHostedZonesByName"
            ],
            "Effect": "Allow",
            "Resource": "*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "ec2:CreateSnapshot",
               "ec2:AttachVolume",
               "ec2:DetachVolume",
               "ec2:ModifyVolume",
               "ec2:DescribeAvailabilityZones",
               "ec2:DescribeInstances",
               "ec2:DescribeSnapshots",
               "ec2:DescribeTags",
               "ec2:DescribeVolumes",
               "ec2:DescribeVolumesModifications"
            ],
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:CreateTags"
            ],
            "Condition": {
               "StringEquals": {
                  "ec2:CreateAction": [
                        "CreateVolume",
                        "CreateSnapshot"
                  ]
               }
            },
            "Effect": "Allow",
            "Resource": [
               "arn:aws:ec2:*:*:volume/*",
               "arn:aws:ec2:*:*:snapshot/*"
            ]
      },
      {
            "Action": [
               "ec2:DeleteTags"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:ec2:*:*:volume/*",
               "arn:aws:ec2:*:*:snapshot/*"
            ]
      },
      {
            "Action": [
               "ec2:CreateVolume"
            ],
            "Condition": {
               "StringLike": {
                  "aws:RequestTag/ebs.csi.aws.com/cluster": "true"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:CreateVolume"
            ],
            "Condition": {
               "StringLike": {
                  "aws:RequestTag/CSIVolumeName": "*"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:DeleteVolume"
            ],
            "Condition": {
               "StringLike": {
                  "ec2:ResourceTag/ebs.csi.aws.com/cluster": "true"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:DeleteVolume"
            ],
            "Condition": {
               "StringLike": {
                  "ec2:ResourceTag/CSIVolumeName": "*"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:DeleteVolume"
            ],
            "Condition": {
               "StringLike": {
                  "ec2:ResourceTag/kubernetes.io/created-for/pvc/name": "*"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:DeleteSnapshot"
            ],
            "Condition": {
               "StringLike": {
                  "ec2:ResourceTag/CSIVolumeSnapshotName": "*"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:DeleteSnapshot"
            ],
            "Condition": {
               "StringLike": {
                  "ec2:ResourceTag/ebs.csi.aws.com/cluster": "true"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "route53:ChangeResourceRecordSets"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:route53:::hostedzone/*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "route53:ListHostedZones",
               "route53:ListResourceRecordSets",
               "route53:ListTagsForResource"
            ],
            "Effect": "Allow",
            "Resource": "*"
      }
   ]
}