Automate with headless

File feeds are one of the most common ways to exchange batches of data on a regular basis. One key challenge for a traditional feed, however, is its inflexibility. If anything changes about the way source data is extracted, or the destination schema is configured, the feed is all but guaranteed to break.

With Flatfile’s powerful headless data import capabilities, you can seamlessly integrate a headless but adaptable data connection into your system.

​

How it works

You can achieve a fully automated data exchange between systems in just three steps:

Automate ingress
Implement a flexible data pipeline that seamlessly uploads files to Flatfile without manual intervention, streamlining the process of data intake.

Flatfile processing
Within the Flatfile Platform, build a comprehensive configuration that enables automated extraction, mapping, validation, and transformation of your data.

Automate egress
Finally, leverage Flatfile’s Event driven system to automatically export the processed and transformed data from Flatfile to the designated destination system, eliminating the need for manual exporting and ensuring a continuous flow of data to the intended location.

​

Automate ingress

Data comes from many sources, and Flatfile can handle them all. Set up automatic feeds via cron job, cloud function, or any other method that works for your business.

All ingress processes will leverage either our API or our SDKs.

First, configure an ingress destination. Our files will be contained in a Space. You can think of a SpaceA micro-application... as a micro-application, each has it’s own database and configurations.

const { FlatfileClient } = require("@flatfile/api");
const api = new FlatfileClient({ token: process.env.FLATFILE_API_KEY });

try {
  api.spaces.create({
    name: "My Space",
    environmentId: "us_env_123",
  });
} catch (error) {
  console.error(error);
}

Next, we’ll upload our file to Flatfile.

const { FlatfileClient } = require("@flatfile/api");
const api = new FlatfileClient({ token: process.env.FLATFILE_API_KEY });

const readStream = fs.createReadStream("/path/to/file");

try {
  api.files.upload(stream, {
    spaceId: "us_spa_123",
    environmentId: "us_env_123",
  });
} catch (error) {
  console.error(error);
}

​

Flatfile Processing

​

Extract incoming data

Incoming CSV files are automatically extracted to a WorkbookA collection of data....

However, if you’re sending a different file type, you’ll need to configure a listener to handle extraction.

​

Automate mapping

Once our files are extracted, we’ll need to automate the mapping of our data according to our schema. Our Automap Plugin will automatically map the data, matching based on configureable options. See our Automap Guide for more information.

​

Validate & Transform

Similarly, validation and transformation operations can run immediately on records:* events with RecordHooks. See our Data Handling Guide for more information.

​

Automate egress

Now that we’ve leveraged the power of the Flatfile platform to extract, map, validate, and transform our data, we can automate the egress of our data to our destination system.

We’ll have our listener filter on the workbook:map job and configure it to run our egress logic on job:completed. Here’s an example where we automate emailing the processed data.

import flatfile from "@flatfile/api";
import nodemailer from "nodemailer";
import util from "util";

export default function (listener) {
  listener.on(
    "job:completed",
    { job: "workbook:map" },
    async (event: FlatfileEvent) => {
      const { data } = await api.workbooks.get(event.context.workbookId);
      const apiKey = await event.secrets("api_key");
      // Automate an egress for your data, integrating with your existing sytems
    }
  );
}

To pull a working demo of a headless integration, check out our Example Projects.

​

Additional paths

To find the additional Flatfile integration paths that may work better for your business, explore our other use cases: