Data Hooks® are one of Flatfile’s most powerful data validation tools. Put simply, Data Hooks are short sections of JavaScript that run while a file is being imported. When implemented properly, Data Hooks can re-format, validate, and correct data automatically during a data import. Our customers have used Data Hooks to automatically reformat area codes, country codes, remove special characters, validate emails against external data, and reformat dates.
To see some working Data Hooks that you can implement right now in your workflow, check out the list below. To learn more about how to create Data Hooks and how they work, keep reading.
Data Hooks are attached to specific data templates. To create a new Data Hook, first open the data template editor (you can do this by clicking on the template in the Templates screen), click the Data Hooks® tab, and then click + Add Data Hook®.
Give your Data Hook a name and give it a place in the order. Just like normal code, Data Hooks are executed in order, from top to bottom.
Data Hooks are written in Javascript in the text editor on the right side of the Edit data template screen. You can reference the variables you have available to you in the middle column under the VARIABLES header. This list also shows you the data types assigned to the variables in your data template.
Let’s look at a simple example of a Data Hook. If you wanted to change the name field of every user with a user ID of 2001 to “James” (just to be clear, we don’t suggest you do this, specifically, but it is a good example that shows what Data Hooks can do), you could use a few lines of JavaScript that looked like this:
This code is quite simple, and does exactly what we described above. The amazing power of Data Hooks is that they are pieces of code that you control. If you can code it, you can do it.
Even in this short example, though, there are a few things to call out:
First, the only valid key values for the row
object are the names of the fields (column keys) on the associated data template. In the above example, if we did not have a field in the data template with a key of "user_id" or "name", the Data Hook would not be functional, as the correct column data would not be referenced.
Second, row
and info
must be returned in order to transform and validate the data. Instructions for validating data are stored in the info
array.
level
sets the importance of the validation and has three options: info
, warn
, and error
.
field
directs us to the appropriate column in the row we are targeting
message
is an optional message to the user uploading the data, and is usually used to let them know a Data Hook has executed
Third, Data Hooks will execute on every row of your data. This means that if you include an API call in a Data Hook, that request will be made for every row of your data. In most cases, this is not a problem.
But Data Hooks can also interact with APIs or external libraries in order to validate data. Let’s look at a common example: say you want to format every date that appears in your data as YYYY-MM-DD.
The above code shows one way to reformat dates in your data. It leverages the date-fns library to reformat dates the Data Hook finds in your data into an ISO format (in this example YYYY-MM-DD) and returns warnings and messages if it encounters date data that it cannot parse.
The main principles of this code are the same as in our simpler example:
We can only use row.date
because we have a date
row already defined in the data that has been uploaded.
row
and info
must be returned in order for the Data Hook to execute properly.
This Data Hook will run on every row in the data we upload using the data template it is attached to.
If JavaScript can do it, you can do it in a Data Hook.
In order to improve security and reliability in Data Hooks, Flatfile has whitelisted a select group of packages for consumption in Data Hooks:
axios version 0.23.0
date-fns version 2.25.0
image-validator version 1.2.1
parse-address version 1.1.2
To import and use one of these packages in a Data Hook, we use the import()
function with an await
:
If you would like to activate this feature, please contact our Flatfile Support team by emailing support@flatfile.com.