๐Ÿš… Data Ingestion with Airbyte

๐Ÿ“˜

This guide will get you oriented in Airbyte

If you run Faros CE locally, you can access the Airbyte UI at http://localhost:8000

Airbyte allows you to define ELT connections between sources (extract) and destinations (load). Those are the main menus you see in the Airbyte UI.

In this case, sources will be your engineering systems (GitHub, Jira, Jenkins etc.), and the destination will be Faros CE.

Sources

Source connectors are the pieces of code that extract data from those systems. They are either part of the Airbyte-maintained catalog, or built by Faros.

Source settings is where most of your input is needed as they usually include:

  • an API token field
  • a domain field (e.g. your Jira domain name)
  • a field containing the list of "streams" (projects, repositories, pipelines etc.) you want to get synced in Faros CE
  • a start date field controlling how far back to sync data

For convenience, we have canned most sources in your Faros CE instance. All that is missing is the configuration above!

35543554

More information: ๐Ÿšฐ Supported Sources.

The Faros Destination

The Faros Destination connector is where Faros takes all the streams of the various sources we want to support and maps them to our tables you can see in Metabase. More information: ๐Ÿš€ Faros Destination.

It is preconfigured in Faros CE; its main setting is the location of the Hasura endpoint of your Faros CE instance.

Connections between Sources and the Faros Destination

You created an Airbyte connection when you link a source and a destination together; that connection also has a few settings:

  • The frequency of the sync
  • The data streams (e.g for GitHub - Users, Pull Requests, Commits etc.) and their sync mode
  • The table prefix, which allows for namespacing before it is handed over to the destination
33263326

For convenience, we have canned those connections in Faros CE. Feel free to set the frequency. The streams are optimally set based on what the data model supports in Faros CE. Do not change the table prefix for proper destination operations.


Did this page help you?