Data Onboarding and ETL Tools: When to Use What

Broadly speaking, ETL (extract, transform, load) is the process of taking data from one place, cleaning it up, organizing it, and moving it to another place - a data warehouse. It’s a data migration task that businesses of all sizes typically have to deal with. Because these companies’ ETL needs involve a variety of data complexity and scale, they necessitate the use of different types and combinations of tools, costs, and human time and expertise.

Data onboarding automates and streamlines much of this work, saving time, resolving complexity, and providing a user friendly interface that non-experts can use. Think of it as ETL for the masses.

ETL projects and data onboarding

In the ETL landscape, the need for data onboarding shows up in both large and small projects. It’s needed anywhere you have to import customer data into your products or services. That may involve migrating from other software products, importing spreadsheets and other file types like CSVs, and moving offline data online. The customer data is often messy, with errors or formatting issues, and it may need to be consolidated from multiple files.    

Typically, large ETL projects are laborious and demand long hours of hands-on work from a data engineer or team of engineers who use precise tools and their own extensive expertise to clean up and validate that data and get it from point A to point B. Further, because of the complexity of the data work, the SMEs who understand the context and format of the data the best are, ironically, cut out of the ETL process. 

Smaller ETL projects suffer from a related but different problem: Because of limited scale, it’s often cost prohibitive to bring in heavy-hitter data experts, like a data analyst and their expensive ETL solution and services. That effectively means that a customer doesn’t get the soup-to-nuts ETL treatment that they may prefer; instead, they have to spend time cleaning up their own data before handing it off to the business for importing. Even when businesses have a little more data muscle and can assume more of that janitoring work for the customer, the human labor burden isn’t eliminated--it’s just shifted somewhat to the business and off of the customer.

Data variance: when to use an ETL tool

Another ETL challenge involves data variance. Current ETL systems were designed for high-value but low-variance data. As it pertains to data onboarding, low-variance data typically fits nicely into repeatable rule sets and specific formats. And so, even though low-variance data requires time and expertise to initially get through the ETL process, once it’s mapped, the hard part is done. 

By contrast, high-variance data is more heterogeneous. Think of transactional data, for example, which is marked by different file formats and different rules for each input--which may need to be redefined for different markets. You can’t use traditional ETL products in those cases.

The data onboarding sweet spot: how data onboarding works with ETL

Data onboarding smoothes out most of those wrinkles. 

It complements large or complex ETL projects primarily by saving human experts massive amounts of time, even in those cases when the project demands deep expertise and surgical precision. Because the data onboarding process handles so much automatically, it can knock days, weeks, or months off of an ETL project.    

For less intensive projects, data onboarding replaces the bulk of the ETL process. It can bring ETL capabilities to businesses that previously couldn’t afford a standard ETL service, relieving burdens both from them and their customers. Because it handles so many technical aspects of ETL, including high-variance data, a data onboarding tool can give less technical people, like product managers and customer success teams, a familiar spreadsheet-like interface that they can work from so they don’t have to deal with things like SQL queries.

Data onboarding successes

You can probably imagine how data onboarding could benefit your customers, but here are real-life examples of companies that benefited from using Flatfile’s data onboarding tools:

• Customs City

• Inkit

• Tablecloth

To sum it up: You can use expensive ETL tools and human experts to clean up and move your data--and in some cases, you have to. But that model isn’t a good fit, or even a viable option, for every business. Data onboarding automates and streamlines part of the ETL process. This can save human experts weeks or months of time, whether by automatically fixing high-variance data or just speeding up low-variance data work. 

In other cases, data onboarding can reduce the need for businesses to offload preliminary manual data janitoring to their customers and allow less technical people to more effectively manage their ETL process.