Google BigQuery
Create or update a recurring BigQuery connector
Custom Streaming
Create or update a recurring BigQuery connector. Generates a Bento pipeline with a gcp_bigquery_select input that polls a BigQuery table on a configurable interval and streams rows to Kafka → Firestore. Authentication uses the ambient GCP service account (workload identity on GKE) — no credentials need to be stored.
Technical details
Authentication ▾
Contact us for setup details.
Ingestion ▾
Mode: Streaming
Tools exposed (1) ▾
upsert_bento_bigquery
Create or update a recurring BigQuery connector. Generates a Bento pipeline with a gcp_bigquery_select input that polls a BigQuery table on a configurable interval and streams rows to Kafka → Firestore. Authentication uses the ambient GCP service account (workload identity on GKE) — no credentials need to be stored.
Inputs ▾
upsert_bento_bigquery
| Name | Type | Required | Description |
|---|---|---|---|
| dashboardId | string | no | Target dashboard/initiative ID. Omit to use the current dashboard. |
| connectorId | string | no | If provided, updates the existing connector instead of creating a new one. |
| connectorName | string | yes | Human-readable name for this connector, e.g. 'BigQuery — analytics.sessions table'. |
| project | string | yes | GCP project ID that owns the BigQuery dataset, e.g. 'my-gcp-project'. |
| dataset | string | yes | BigQuery dataset ID, e.g. 'analytics' or 'production_data'. |
| table | string | yes | BigQuery table (or view) to query, e.g. 'sessions' or 'daily_summary'. |
| columns | string[] | no | Columns to SELECT. Omit or pass ['*'] to select all columns. |
| where | string | no | SQL WHERE clause (without the WHERE keyword), e.g. "_PARTITIONDATE >= DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY)". |
| pollingInterval | string | yes | How often Bento polls BigQuery, e.g. '30m', '1h', '6h'. Use 'realtime' for the fastest practical rate (1m). Avoid very short intervals to keep BigQuery costs low. |
| idExpression | string | no | Optional Bloblang expression for the Firestore document ID. Provide when the table has a stable unique key, e.g. 'this.session_id.string()' or 'this.row_hash.string()'. |