Cloud destinations

After generation, files can be downloaded as a zip or written directly into your data lake. You never give us long-lived storage access keys. For Amazon S3, you create an IAM role that can write under one prefix; we assume that role for the job, then upload parts there.

Azure Blob and Google Cloud Storage destinations are not available yet.

Amazon S3

On the Delivery step, choose Upload to my S3 bucket. Copy Our AWS principal ARN and External ID from the top of that form into the role’s trust policy. Then paste your role ARN, bucket name, AWS region, and key prefix (default synth-data/). After setup, use Test connection; that writes a small probe object at {prefix}.synth-data-probe and does not delete it.

Use the two values shown on the form — not an ARN from your own user, unless you are running a self-hosted worker (in which case the form shows that worker’s caller identity).

The role should allow s3:PutObject (and optionally s3:ListBucket on that prefix) only. Do not grant s3:DeleteObject or broad s3:*. We do not need GetObject to deliver files.

A finished job lands under:

s3://YOUR_BUCKET/synth-data/{job-id}/

That s3:// string is a location, not a browser link. Open the prefix in the S3 console or list it with the AWS CLI.

Set up in the AWS console

In the generator, open Delivery, select Upload to my S3 bucket, and copy Our AWS principal ARN and External ID.
Confirm the destination bucket exists in the region you will type into the form. Default encryption SSE-S3 (AES-256) works. A customer-managed KMS key needs extra grants that this guide does not cover.
Open IAM → Roles → Create role. Choose Custom trust policy and paste the following, replacing the principal ARN and External ID:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/YOUR_SERVICE_PRINCIPAL"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "PASTE_EXTERNAL_ID_FROM_THE_GENERATOR"
        }
      }
    }
  ]
}

Name the role (for example synth-data-delivery) and create it.
Open the role → Add permissions → Create inline policy → JSON. Replace the bucket name if your prefix is not synth-data/:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PutUnderPrefix",
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::YOUR_BUCKET/synth-data/*"
    },
    {
      "Sid": "ListPrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::YOUR_BUCKET",
      "Condition": {
        "StringLike": {
          "s3:prefix": ["synth-data/", "synth-data/*"]
        }
      }
    }
  ]
}

Copy the role ARN (arn:aws:iam::ACCOUNT:role/synth-data-delivery) into the generator. Fill in bucket, region, and prefix, then Test connection. You should see s3://YOUR_BUCKET/synth-data/.synth-data-probe in the bucket.
Generate as usual. Parts appear under synth-data/{job-id}/.

If the test fails, check that the External ID matches exactly, the trust Principal is our service ARN (not yours), the bucket region matches the form, and the prefix in the policy matches the form.

CloudFormation

Save this template and create the stack in the same region as the bucket. The bucket must already exist. After the stack finishes, copy the RoleArn output into the generator.

AWSTemplateFormatVersion: "2010-09-09"
Description: IAM role that DataStandIn assumes to write generated files under one S3 prefix.

Parameters:
  ServicePrincipalArn:
    Type: String
    Description: Our AWS principal ARN from the generator Delivery step.
  ExternalId:
    Type: String
    NoEcho: true
    Description: External ID from the generator Delivery step.
  BucketName:
    Type: String
    Description: Existing destination bucket name (not an s3:// URI).
  Prefix:
    Type: String
    Default: synth-data/
    Description: Key prefix with a trailing slash.
  RoleName:
    Type: String
    Default: synth-data-delivery

Resources:
  DeliveryRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Ref RoleName
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              AWS: !Ref ServicePrincipalArn
            Action: sts:AssumeRole
            Condition:
              StringEquals:
                sts:ExternalId: !Ref ExternalId
      Policies:
        - PolicyName: SynthDataLakeWrite
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Sid: PutUnderPrefix
                Effect: Allow
                Action: s3:PutObject
                Resource: !Sub arn:aws:s3:::${BucketName}/${Prefix}*
              - Sid: ListPrefix
                Effect: Allow
                Action: s3:ListBucket
                Resource: !Sub arn:aws:s3:::${BucketName}
                Condition:
                  StringLike:
                    s3:prefix:
                      - !Ref Prefix
                      - !Sub ${Prefix}*

Outputs:
  RoleArn:
    Description: Paste this ARN into the generator Delivery step.
    Value: !GetAtt DeliveryRole.Arn

Example create (replace the parameter values):

aws cloudformation create-stack \
  --stack-name synth-data-delivery \
  --template-body file://synth-data-delivery.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameters \
    ParameterKey=ServicePrincipalArn,ParameterValue=arn:aws:iam::123456789012:user/YOUR_SERVICE_PRINCIPAL \
    ParameterKey=ExternalId,ParameterValue=PASTE_EXTERNAL_ID \
    ParameterKey=BucketName,ParameterValue=YOUR_BUCKET \
    ParameterKey=Prefix,ParameterValue=synth-data/

Terraform

Use the same four values as parameters. Apply in the bucket’s region, then paste role_arn into the generator.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.0"
    }
  }
}

variable "service_principal_arn" {
  type        = string
  description = "Our AWS principal ARN from the generator Delivery step."
}

variable "external_id" {
  type        = string
  sensitive   = true
  description = "External ID from the generator Delivery step."
}

variable "bucket_name" {
  type        = string
  description = "Existing destination bucket name (not an s3:// URI)."
}

variable "prefix" {
  type        = string
  default     = "synth-data/"
  description = "Key prefix with a trailing slash."
}

variable "role_name" {
  type    = string
  default = "synth-data-delivery"
}

locals {
  prefix = endswith(var.prefix, "/") ? var.prefix : "${var.prefix}/"
}

data "aws_iam_policy_document" "trust" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "AWS"
      identifiers = [var.service_principal_arn]
    }
    condition {
      test     = "StringEquals"
      variable = "sts:ExternalId"
      values   = [var.external_id]
    }
  }
}

data "aws_iam_policy_document" "write" {
  statement {
    sid       = "PutUnderPrefix"
    actions   = ["s3:PutObject"]
    resources = ["arn:aws:s3:::${var.bucket_name}/${local.prefix}*"]
  }
  statement {
    sid       = "ListPrefix"
    actions   = ["s3:ListBucket"]
    resources = ["arn:aws:s3:::${var.bucket_name}"]
    condition {
      test     = "StringLike"
      variable = "s3:prefix"
      values   = [local.prefix, "${local.prefix}*"]
    }
  }
}

resource "aws_iam_role" "delivery" {
  name               = var.role_name
  assume_role_policy = data.aws_iam_policy_document.trust.json
}

resource "aws_iam_role_policy" "write" {
  name   = "SynthDataLakeWrite"
  role   = aws_iam_role.delivery.id
  policy = data.aws_iam_policy_document.write.json
}

output "role_arn" {
  description = "Paste this ARN into the generator Delivery step."
  value       = aws_iam_role.delivery.arn
}

Example variables file:

service_principal_arn = "arn:aws:iam::123456789012:user/YOUR_SERVICE_PRINCIPAL"
external_id           = "PASTE_EXTERNAL_ID"
bucket_name           = "YOUR_BUCKET"
prefix                = "synth-data/"

Cloud destinations

Amazon S3

Set up in the AWS console

CloudFormation

Terraform

Related