Connections

A connection is a configuration for syncing data between a source and a destination. For more information regarding all the possible configurations, please refer to this document.

Required fields

The only required fields when creating a connection are:

1. Source ID
2. Destination ID

The sourceId and destinationId can be in the address bar of your workspace. While on the page of the Source you want to use as input, the URL will be https://cloud.airbyte.com/workspaces/<workspace_uuid>/source/<source_uuid>. Similarly for Destinations, the URL will be https://cloud.airbyte.com/workspaces/<workspace_uuid>/destination/<destination_uuid>.

You can also find them by utilizing our /sources and /destinations endpoints.

Configuration Setting Defaults

If no other configuration settings set, then default configuration settings will be used:

• All streams will be set to the sync mode: full_refresh_overwrite
• Replication Frequency will be set to Manual
• Destination Namespace will be set to Destination Default
• Data Residency will be set to AUTO (most efficient data plane to sync data selected)
• Connection name defaults are formatted as: source-name <> destination-name

Configuration Setting Customization

Scheduling Customization

Scheduling is limited to at maximum ever hour. If you have a use case for under 1 hour frequency, please reach out to our technical support team through the webapp.

Schedules are set by providing cron expressions and will be read as UTC time-zoned.

Some examples of valid cron timings:

• 0 * * * * ? for the top of every hour
• 0 0 12 * * ? for 12:00 PM UTC every day
• 35 26 1 * * 1 ? for every Monday at 1:26:35 UTC
• 0 0 */4 * 1 ? for at 00:00 on every 4th day-of-month if it's on Monday.

Stream-specific Customization

If you want to provide your own configurations, you can get a list of valid stream names, sync modes, and fields by calling the GET /v1/streams as documented here.

Here are some documentation pages regarding the different sync modes:

• Full Refresh - Overwrite
• Full Refresh - Append
• Incremental Sync - Append
• Incremental Sync - Deduped History

A cursorField is required if the sync mode is incremental_* and a default does not already exist for that stream. More info about cursors here.

🚧

A primaryKey can only be set if sourceDefinedCursorField is set to false and is required if the sync mode is incremental_deduped_history for that stream. More info about primary keys here.

Data Residency

Choose where the data for this connection will be processed. Depending on your network configuration, you may need to add IP addresses to your allowlist. More info can be found here.

The default will be set to AUTO where Airbyte decides which is the most efficient data plane to process the sync.

Namespace Definition

Define the location where the data will be stored in the destination. In most cases, namespaces are schemas in the database you're replicating to. If your desired destination doesn't support it, you can ignore this feature.

More information about Namespaces can be found here.

Namespace Format

Used when namespaceDefinition is custom_format. Refer here for more information.

Prefix

Stream names refer to table names in a typical RDBMS. But it can also be the name of an API endpoint, etc. Similarly to the namespace, stream names can be configured to diverge from their names in the source with a "prefix" field. The prefix is prepended to the source stream name in the destination.

Non Breaking Schema Updates Behavior

When non-breaking schema updates are detected for the source of the connection, this setting will allow users to either ignore the change or to disable_connection which will stop all scheduled syncs. Default to ignore if setting is not supplied.