A connection is a configuration for syncing data between a source and a destination.
Required fields include
destinationId. You can discover these by using the endpoints for List Sources and List Destinations. Alternatively, these can also be found in the address bar of the Airbyte UI.
If no other configuration settings set, then default configuration settings will be used:
- All streams will be set to the sync mode:
- Replication Frequency will be set to
- Destination Namespace will be set to
- Data Residency will be set to
AUTO(most efficient data plane to sync data selected)
- Connection name defaults are formatted as:
source-name <> destination-name
- Non-Breaking Schema Updates Behavior:
For customizing these settings, see below.
For additonal details on what these settings mean or do, see this document.
Scheduling is limited to at maximum ever hour. If you have a use case for under 1 hour frequency, please reach out to our technical support team through the webapp.
Schedules are set by providing cron expressions and will be read as UTC time-zoned.
Some examples of valid cron timings:
0 * * * * ?for the top of every hour
0 0 12 * * ?for 12:00 PM UTC every day
35 26 1 * * 1 ?for every Monday at 1:26:35 UTC
0 0 */4 * 1 ?for at 00:00 on every 4th day-of-month if it's on Monday.
If you want to provide your own configurations, you can get a list of valid stream names, sync modes, and fields by calling the
GET /v1/streams as documented here.
Here are some documentation pages regarding the different sync modes:
- Full Refresh - Overwrite
- Full Refresh - Append
- Incremental Sync - Append
- Incremental Sync - Deduped History
cursorField is required if the sync mode is
incremental_* and a default does not already exist for that stream. More info about cursors here.
primaryKey can only be set if
sourceDefinedCursorField is set to false and is required if the sync mode is
incremental_deduped_history for that stream. More info about primary keys here.
Choose where the data for this connection will be processed. Depending on your network configuration, you may need to add IP addresses to your allowlist. More info can be found here.
The default will be set to
AUTO where Airbyte decides which is the most efficient data plane to process the sync.
Define the location where the data will be stored in the destination. In most cases, namespaces are schemas in the database you're replicating to. If your desired destination doesn't support it, you can ignore this feature.
More information about Namespaces can be found here.
custom_format. Refer here for more information.
Stream names refer to table names in a typical RDBMS. But it can also be the name of an API endpoint, etc. Similarly to the namespace, stream names can be configured to diverge from their names in the source with a "prefix" field. The prefix is prepended to the source stream name in the destination.
When non-breaking schema updates are detected for the source of the connection, this setting will allow users to:
disable_connectionwhich will pause all scheduled syncs
propagate_columnswhich will propagate column changes only
propagate_fullywhich will propagate all changes