Data Hooks® 3.0

Automate the transformation and validation of data using Flatfile Data Hooks.

Overview

Data Hooks® 3.0 are Flatfile’s latest iteration of our Data Hooks feature. This feature is current only available upon request, and may not be available to all users. If Data Hooks® 3.0 has not been turned on for your account, please refer to our Data Hooks 2.0 documentation if you need help.

Data Hooks are one of Flatfile’s most powerful data validation tools. Put simply, Data Hooks are short sections of JavaScript that run while a file is being imported or when edits are made during review. When implemented properly, Data Hooks can re-format, validate, and correct data automatically during a data import. Our customers have used Data Hooks to automatically reformat area codes, country codes, remove special characters, validate emails against external data, and reformat dates.

Creating a Data Hook®

Data Hooks are attached to specific data templates. To create a new Data Hook, first open the data template editor (you can do this by clicking on the template in the Templates screen), click the Data Hooks® tab, and then click the + symbol on the Data Hook Library. The Library stores all of your active and archived Data Hooks. Active Data Hooks in the Library are executed simultaneously and not in any particular order. Archived Data Hooks are not executed, but are saved for your convenience.

updated Data Hooks screenshot

After giving your Data Hook a name and a description, you are ready to add any dependencies you may need to use.

Saving Your Work

When you are finished writing or updating your Data Hook, click the arrow to open a drop-down menu to set the state of your Data Hook. The Data Hook state can be set to

  • Draft: The Data Hook will not run in your data template yet

  • Dismiss: any changes you have made to your Data Hook will be dismissed

  • Deploy: Any changes you have made to your Data Hook will be saved and will run in your data template. In order to test your data hooks (more on that below), you must deploy it first. 

  • Deactivate: your Data Hook will be archived and will no longer run in your data template

DH status drop-down menu

Working with Dependencies

In order to add a dependency to a Data Hook, click the + Add dependencies button. This will bring up a window that allows you to search for npm packages you can leverage in your Data Hook. After finding the dependency you need, you can select a specific version of the npm package from the drop-down menu.

add dependency window

The dependencies you have added will appear in a list under the Dependencies header.

To use one of these packages in a Data Hook, we use the require() function:

Writing a Data Hook®

Data Hooks are written in Javascript in the text editor in the Data Hooks® tab. You can reference the variables you have available to you under the Variables expandable menu. This list also shows you the data types assigned to the variables in your data template. 

The Data Hooks® tab also includes a helpful list of templates, under the Templates expandable menu, for the most common types of Data Hooks. Clicking on one of these templates will populate the text editor with the template that you clicked on, including comments explaining the structure of the template and how it is used. 

DH templates

Let’s look at the Normalize Date Data Hook. Say you want to format every date that appears in your data as YYYY-MM-DD. You could use code that looked like this:

This code is quite simple, and does exactly what we described above, but even in this short example, there are a few things to call out:

First, this Data Hook leverages the date-fns library to reformat dates in your data into an ISO format (in this example YYYY-MM-DD) and returns warnings and messages if it encounters date data that it cannot parse. In order to use date-fns, it needs to be added as a dependency and then called in the hook itself using const dfns = require(‘date-fns’);

Second, all Data Hooks use the same basic structure, starting with module.exports = async ({recordBatch, session, logger}) => {}. Data Hooks are structured this way to give you the option of executing a Data Hook on every row of your data or on all of your data at once, depending on how you structure your code. Since Data Hooks can also interact with APIs or external libraries in order to validate and transform data, this is something to take into consideration as you write your Data Hook.

Writing an Advanced Data Hook®

Let’s take a look at a more complex example of a Data Hook. In the following example, we call an external API and leverage the data in order to verify addresses.

In this example, we can see just how powerful Data Hooks are. Here, we are using a Data Hook to access an API and validate and transform uploaded addresses. 

First, we use our credentials to access an API for SmartyStreets, the address verification service we have decided to use. SmartyStreets is a paid service, and requires an ID, token, and license before it can be used.

Next, we use batchRecords.records.map to leverage the API without making a call to the SmartyStreets API for every individual record in our data. The batchRecords argument allows us to batch our records and reduce the number of calls we are making. Meanwhile, session and logger are arguments that allow us to access the entire session and log any data we might need, respectively. We can see logger being used in this Data Hook to receive log output for testing. 

Once SmartyStreets has validated our addresses, we can use that information in whatever way we see fit. Here, we are setting a longitude and latitude for valid addresses, and returning an error for invalid addresses. We do this by using record.set()

If you ever get stuck, remember that the provided templates on the Data Hooks screen contain helpful examples and commented code samples that can help get you started.

Including Error Messages in Your Data Hook®

In order to display warnings or errors to your customer who will be importing into Flatfile, we have a few different functions available for you to use. They are:

  • addComment: adds a comment the user who is uploading will see

  • addWarning: adds a warning when certain conditions are met, shown to the user upon review

  • addError: adds an error when certain conditions are met, shown to the user upon review

You can call each of these functions with a format like below:

Testing Your Data Hook®

If the feature is enabled, you can also test your Data Hook. Below the text editor where you write your Data Hook is the testing section. You can type (or copy and paste) a batch of sample data into the provided cells, then click Run Test Data. 

v3 Data Hook testing

Testing your Data Hook on some sample data is a great way to see if your Data Hook works in the way that you expect it to. After running a test, you will see a Test result section appear (this can be closed by clicking on the blue X). Here, you can see the output of the Data Hook after executing on your sample data. Under the Output Logs section, you will see any logs you instructed your Data Hook to make using the logger argument. 

Testing your Data Hook here is vital. Once a Data Hook is deployed, you may not be notified of errors or failures that may occur. 

If you would like to activate the Data Hook feature, please contact our Flatfile Support team by emailing support@flatfile.com.