What is dbt and why is it so powerful?
dbt or Data Build Tool is a powerful resource designed to allow users to transform data within their data warehouse using SQL. dbt empowers Data teams to leverage software engineering products for transforming data.
Where does dbt fit in?
In a traditional data team, a Data Engineer would be responsible for the Extract, Transform & Load processes, commonly referred to as ETL. This process would prepare the data that for the rest of the team to utilise. A Data Analyst would then be able to interrogate the data, and work with the various teams within the business to understand what questions needed to be answered. The Engineer would then likely be required to begin the ETL process again in order to reshape the data to business requirement - this could be a time consuming and costly process.
With the advent of Cloud-Based Data Warehouses (essentially combining powerful super computers with databases), the Transformation process can now be completed within the warehouse. This then allows the Engineer to focus on the Extract and Load elements of the process, and a relatively new role of the Data Analytics Engineer to focus on the Transformation aspect of the data preparation. The Cloud-Based Warehouse is flexible, as the scale can increase according to requirement, and the super-computer can be enhanced to become more powerful.
dbt & Transformation
dbt is a tool that sits across the Transformation part of the modern data stack:
dbt models are built using SQL select statements inside of dbt projects - each representing a modular piece of logic that takes the raw data and build it into the final transformed data required. No complex languages are required as dbt automatically generates DDL (Data Definition Language) once the dbt model is committed.
Teams can collaborate
Within dbt, teams can connect to the data source, create a new 'branch,' through which to build, test and document changes ahead of impacting the existing production line. These separate branches allow developers to work on the same code base without impacting production.
dbt breaks down changes into modular pieces of logic
dbt enables teams to break down tasks into individual parts that can be tested, documented, and then joined together to complete a larger function. These smaller models can be used to break down complex changes into smaller component parts that are easier to understand and troubleshoot. These models are then also available to be reused across other transformations.
Connect to GitHub and create full documentation throughout
Once your models are built as required, dbt allows you to 'commit and sync' models to send a snapshot of the changes to GitHub, where the code denoting the changes made are stored. Within dbt, documentation can also be automatically generated using the dbt docs generate function. dbt also provides a DAG (Directed Acyclic Graph) to visualise the relationships and flow from data source (coloured green) through models (coloured blue) and dependencies.
Used correctly, dbt is a powerful tool in the data transformation process.