Connections

A connection is a configuration for syncing data between a source and a destination.

Required fields

Required fields include sourceId and destinationId. You can discover these by using the endpoints for List Sources and List Destinations. Alternatively, these can also be found in the address bar of the Airbyte UI.

Default Connection Settings

If no other configuration settings set, then default configuration settings will be used:

All streams will be set to the sync mode: full_refresh_overwrite
Replication Frequency will be set to Manual
Destination Namespace will be set to Destination Default
Data Residency will be set to AUTO (most efficient data plane to sync data selected)
Connection name defaults are formatted as: source-name <> destination-name
Non-Breaking Schema Updates Behavior: ignore

For customizing these settings, see below.

Custom Connections Settings

For additonal details on what these settings mean or do, see this document.

Scheduling Customization

Scheduling is limited to at maximum ever hour. If you have a use case for under 1 hour frequency, please reach out to our technical support team through the webapp.

Schedules are set by providing cron expressions and will be read as UTC time-zoned.

Some examples of valid cron timings:

0 * * * * ? for the top of every hour
0 0 12 * * ? for 12:00 PM UTC every day
35 26 1 * * 1 ? for every Monday at 1:26:35 UTC
0 0 */4 * 1 ? for at 00:00 on every 4th day-of-month if it's on Monday.

Stream-specific Customization

If you want to provide your own configurations, you can get a list of valid stream names, sync modes, and fields by calling the GET /v1/streams as documented here.

Here are some documentation pages regarding the different sync modes:

A cursorField is required if the sync mode is incremental_* and a default does not already exist for that stream. More info about cursors here.

Note

A primaryKey can only be set if sourceDefinedCursorField is set to false and is required if the sync mode is incremental_deduped_history for that stream. More info about primary keys here.

Data Residency

Choose where the data for this connection will be processed. Depending on your network configuration, you may need to add IP addresses to your allowlist. More info can be found here.

The default will be set to AUTO where Airbyte decides which is the most efficient data plane to process the sync.

Namespace Definition

Define the location where the data will be stored in the destination. In most cases, namespaces are schemas in the database you're replicating to. If your desired destination doesn't support it, you can ignore this feature.

More information about Namespaces can be found here.

Namespace Format

Used when namespaceDefinition is custom_format. Refer here for more information.

Prefix

Stream names refer to table names in a typical RDBMS. But it can also be the name of an API endpoint, etc. Similarly to the namespace, stream names can be configured to diverge from their names in the source with a "prefix" field. The prefix is prepended to the source stream name in the destination.

Non-Breaking Schema Updates Behavior

When non-breaking schema updates are detected for the source of the connection, this setting will allow users to:

ignore the changes
disable_connection which will pause all scheduled syncs
propagate_columns which will propagate column changes only
propagate_fully which will propagate all changes

Defaults to ignore if setting is not supplied. More information is available on Managing Schema Changes and Managing Syncs.