Object Storage
Create or update an object storage connector for AWS S3 or Google Cloud Storage (GCS)
Create or update an object storage connector for AWS S3 or Google Cloud Storage (GCS). Generates a Bento pipeline with an aws_s3 or gcp_cloud_storage input that reads objects from a bucket/prefix and streams their contents to Kafka → Firestore. GCS uses ambient GKE workload identity — no credentials needed. AWS S3 credentials are NEVER stored in the config — they are resolved from environment variables at runtime via Secret Manager.
Technical details
Authentication ▾
Contact us for setup details.
Ingestion ▾
Tools exposed (1) ▾
Create or update an object storage connector for AWS S3 or Google Cloud Storage (GCS). Generates a Bento pipeline with an aws_s3 or gcp_cloud_storage input that reads objects from a bucket/prefix and streams their contents to Kafka → Firestore. GCS uses ambient GKE workload identity — no credentials needed. AWS S3 credentials are NEVER stored in the config — they are resolved from environment variables at runtime via Secret Manager.
Inputs ▾
upsert_bento_object_storage
| Name | Type | Required | Description |
|---|---|---|---|
| dashboardId | string | no | Target dashboard/initiative ID. Omit to use the current dashboard. |
| connectorId | string | no | If provided, updates the existing connector instead of creating a new one. |
| connectorName | string | yes | Human-readable name for this connector, e.g. 'S3 — data-lake/exports/'. |
| provider | string | yes | 'aws_s3' for Amazon S3 (or S3-compatible storage), 'gcs' for Google Cloud Storage. GCS uses ambient workload identity on GKE — no credentials required. AWS S3 requires accessKeyIdEnvVar and secretAccessKeyEnvVar. aws_s3 gcs |
| bucket | string | yes | Bucket name, e.g. 'my-data-lake' or 'company-exports'. |
| prefix | string | no | Key/object prefix to filter files, e.g. 'exports/daily/' or 'reports/2024/'. |
| region | string | no | AWS region for S3, e.g. 'us-east-1'. Required for aws_s3. Ignored for gcs. |
| accessKeyIdEnvVar | string | no | Name of the environment variable that holds the AWS access key ID, e.g. 'AWS_ACCESS_KEY_ID'. Required for aws_s3. The value is resolved from Secret Manager at pipeline runtime. |
| secretAccessKeyEnvVar | string | no | Name of the environment variable that holds the AWS secret access key, e.g. 'AWS_SECRET_ACCESS_KEY'. Required for aws_s3. The value is resolved from Secret Manager at pipeline runtime. |
| idExpression | string | no | Optional Bloblang expression for the Firestore document ID. Defaults to the object key, e.g. 'meta("s3_key").string()'. |