Data Hooks®

Data Hooks® are one of Flatfile’s most powerful data validation tools. Put simply, Data Hooks are short sections of JavaScript that run while a file is being imported. When implemented properly, Data Hooks can re-format, validate, and correct data automatically during a data import. Our customers have used Data Hooks to automatically reformat area codes, country codes, remove special characters, validate emails against external data, and reformat dates.

Example data hooks

Creating a Data Hook

Data Hooks are attached to specific data templates. To create a new Data Hook, first open the data template editor (you can do this by clicking on the template in the Templates screen), click the Data Hooks® tab, and then click + Add Data Hook®.

Data hooks tab screenshot

Give your Data Hook a name and give it a place in the order. Just like normal code, Data Hooks are executed in order, from top to bottom.

New data hook screenshot

Writing a Data Hook

Data Hooks are written in Javascript in the text editor on the right side of the Edit data template screen. You can reference the variables you have available to you in the middle column under the VARIABLES header. This list also shows you the data types assigned to the variables in your data template. 

Let’s look at a simple example of a Data Hook. If you wanted to change the name field of every user with a user ID of 2001 to “James” (just to be clear, we don’t suggest you do this, specifically, but it is a good example that shows what Data Hooks can do), you could use a few lines of JavaScript that looked like this: 

This code is quite simple, and does exactly what we described above. The amazing power of Data Hooks is that they are pieces of code that you control. If you can code it, you can do it.

Even in this short example, though, there are a few things to call out:

First, the only valid key values for the row object are the names of the fields (column keys) on the associated data template. In the above example, if we did not have a field in the data template with a key of "user_id" or "name", the Data Hook would not be functional, as the correct column data would not be referenced.

Second, row and info must be returned in order to transform and validate the data. Instructions for validating data are stored in the info array. 

  • level sets the importance of the validation and has three options: info, warn, and error.

  • field directs us to the appropriate column in the row we are targeting

  • message is an optional message to the user uploading the data, and is usually used to let them know a Data Hook has executed

Third, Data Hooks will execute on every row of your data. This means that if you include an API call in a Data Hook, that request will be made for every row of your data. In most cases, this is not a problem.

But Data Hooks can also interact with APIs or external libraries in order to validate data. Let’s look at a common example: say you want to format every date that appears in your data as YYYY-MM-DD.

The above code shows one way to reformat dates in your data. It leverages the date-fns library to reformat dates the Data Hook finds in your data into an ISO format (in this example YYYY-MM-DD) and returns warnings and messages if it encounters date data that it cannot parse. 

The main principles of this code are the same as in our simpler example:

  1. We can only use row.date because we have a date row already defined in the data that has been uploaded.

  2. row and info must be returned in order for the Data Hook to execute properly.

  3. This Data Hook will run on every row in the data we upload using the data template it is attached to.

If JavaScript can do it, you can do it in a Data Hook.

Working with Dependencies

In order to improve security and reliability in Data Hooks, Flatfile has whitelisted a select group of packages for consumption in Data Hooks:

  • axios version 0.23.0

  • date-fns version 2.25.0

  • image-validator version 1.2.1

  • parse-address version 1.1.2

To import and use one of these packages in a Data Hook, we use the import() function with an await:

If you would like to activate this feature, please contact our Flatfile Support team by emailing support@flatfile.com.