Jobs

In Flatfile, a Job represents a large unit of work performed asynchronously on a resource such as a file, Workbook, or Sheet. The Jobs workflow provides visibility into the status and progress of your Jobs, allowing you to monitor and troubleshoot the data processing pipeline.

​

Types of Jobs

Jobs can be triggered in a number of ways, most commonly in response to user activity. Jobs are then managed via Listeners, which receive Events published by Flatfile in response to activity.

There are three types of Jobs on the Flatfile Platform:

​

Action Based Jobs

Actions are developer-defined operations that can be mounted on a number of domains (including Sheets, Workbooks, Documents, and Files). Mounting an Action means attaching a custom operation to that domain. That operation can then be triggered by a user event (clicking a button or selecting a menu item).

To run an Action based job, two configurations are necessary.

First, create an Action on a domain. Here’s an example of a Workbook containing an Action:

api.workbook.create({
  name: "October Report Workbook",
  actions: [
    {
      label: "Export Data",
      description: "Send data to destination system",
      operation: "export",
      type: "file",
    },
  ],
});

Then, create a Listener to respond to the Action:

listener.on(
  "job:ready",
  { job: "workbook:export" },
  ({ context: { jobId } }, payload) => {
    const { jobId } = event.context;

    try {
        await api.jobs.ack(jobId, {
          info: "Starting submit job...",
          progress: 10,
        });

        // Custom code here

        await api.jobs.complete(jobId, {
          outcome: {
            message: "Submit Job was completed succesfully.",
          },
        });

    } catch (error) {
        await api.jobs.fail(jobId, {
          outcome: {
            message: "This Job failed.",
          },
        });
    }
  }
);

Note that the Listener is listening for the job:ready event, for the workbook:export Job, which was defined in our Workbook.

​

Custom Jobs

Another trigger option is to create a Custom Job via SDK/API. In the SDK, Jobs are created by calling the api.jobs.create() method.

Creating a custom Job in your Listener enables any Event to trigger a Job.

Here’s an example of creating a custom Job in a Listener:

listener.on(
  "commit:created",
  { sheet: "contacts" },
  async ({ context: { workbookId, sheetId } }) => {
    const { data } = await api.jobs.create({
      type: "workbook",
      operation: "myCustomOperation",
      trigger: "immediate",
      source: workbookId
    });
  }
);

Note that the trigger for this Listener is set to immediate, which means that the Job will be created and executed immediately upon the Event firing.

Therefore, we should have our Listener ready to respond to this Job:

listener.on(
  "job:ready",
  { job: "workbook:myCustomOperation" },
  async ({ context: { jobId, workbookId }, payload }) => {
    try {
      await api.jobs.ack(jobId, {
        info: "Starting my custom operation.",
        progress: 10,
      });

      // Custom code here.

      await api.jobs.complete(jobId, {
        outcome: {
          message: "Sucessfully completed my custom operation.",
        },
      });
    } catch {
      await api.jobs.fail(jobId, {
        outcome: {
          message: "Custom operation failed.",
        },
      });
    }
  }
);

​

System Jobs

Internally, Flatfile uses Jobs to power many of the features of the Flatfile Platform, such as extraction, record mutation, and AI Assist. Here are some examples of Jobs that the Flatfile Platform creates and manages on your behalf:

​

The Anatomy of a Job

​

Lifecycle Events

Jobs fire the following Events during their lifecycle. In chronological order, the Job Events are:

You can listen on any of these events in your Listener, but the most common event to listen for is job:ready. For more on managing your Job see Working with Jobs.

​

Required Parameters

​

Optional Parameters

Please see our API Reference for details on all possible values.

​

Working with Jobs

Jobs can be managed via SDK/API. For a complete list of ways to interact with Jobs please see our API Reference . Commonly, Jobs are acknowledged, progressed, and then completed, or failed. Here’s look at those steps.

​

Jobs.Ack

First, acknowledge a Job. This will update the Job’s status to executing.

await api.jobs.ack(jobId, {
  info: "Starting submit job...",
  progress: 10,
});

​

Jobs.Update

Once a Job is acknowledged, you can begin running your custom operation. Jobs were designed to handle large processing loads, but you can easily update your user by updating the Job with a progress value.

await api.jobs.update(jobId, {
  progress: 50,
  estimatedCompletionAt: new Date("Tue Aug 23 2023 16:19:42 GMT-0700"),
});

Progress is a numerical or percentage value indicating the completion status of the work. You may also provide an estimatedCompletionAt value which will display your estimate of the remaining processing time in the foreground Job overlay. Additionally, the Jobs Panel will share visibility into the estimated remaining time for acknowledged jobs.

Finally, when your Job is complete, update the Job’s status to complete. This status update includes an optional outcome message which will be displayed to the user. Use this to provide detail on the outcome of the Job.

​

Jobs.Complete

Once a job is complete, you can display an alert to the end user using outcome.

await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was completed succesfully. ${myData.length} records were processed.`,
    acknowledge: true,
  },
});

​

Outcome.Next

Optionally, you can add a button to the outcome dialog to control the next step in the workflow once job has finished.

Internal Link

Add a button to the dialog that will redirect the user somewhere within a Space using next > Id.

In this code below, we will create a button that says “See all downloads” with this path: space/us_sp_1234/files?mode=export

await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was completed succesfully. ${myData.length} records were processed.`,
    acknowledge: true,
    //Reference: https://platform.flatfile.com/space/{$id}/{path}?{$query}
    next: {
      type: "id",
      id: "dev_sp_1234",
      path: "files",
      query: "mode=export",
      label: "See all downloads",
    },
  },
});

External Url

Add a button to the dialog that will redirect the user to an external link using next > Url.

In this code below, we will create a button that says “Go to Google”. It will open in a new tab.

await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was completed succesfully. ${myData.length} records were processed.`,
    acknowledge: true,
    next: {
      type: "url",
      url: "http://www.google.com",
      label: "Go to Google",
    },
  },
});

Download

Add a button to the dialog that will redirect the user to an external link using next > Url.

In this code below, we will create a button that says “Download this file”.

await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was completed succesfully. ${myData.length} records were processed.`,
    acknowledge: true,
    next: {
      type: "download",
      fileName: "DownloadedFromFlatfile.csv",
      url: "source_of_file.csv",
      label: "Download this file",
    },
  },
});

Snapshot

Add a button to the dialog that will redirect the user a particular snapshot using next > snapshot.

In this code below, we will create a button that says “Go to Snapshot”.

await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was completed succesfully.`,
    acknowledge: true,
    next: {
      type: "snapshot",
      label: "Go to Snapshot",
      snapshotId: ${snapshot.id},
      sheetId: ${sheet.id}
    },
  },
});

Files

Download files hosted on Flatfile by using next > files.

In this code below, we will create a button that says “Download files”.

await api.jobs.complete(jobId, {
  outcome: {
    message: `The files should download automatically`,
    next: {
      type: "files",
      label: "Download files",
      files: [{fileId: "us_fl_123"}],
    },
  },
});

View

Dynamically update the columns visible in a sheet by using next > view.

In this code below, we will create a button that says “Hide columns”.

await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was completed on sheet ${sheet.name}`,
    acknowledge: true,
    next: {
      type: "view",
      label: "Hide columns",
      sheetId: ${sheet.id},
      hiddenColumns: ['age', 'phone', 'middleName'] //a list of field keys on the sheet
    },
  },
});

Retry

Often in the event of a failure, you may want to add a button to the dialog that will retry the Job using next > Retry.

In this code below, we will create a button that says “Retry”.

await api.jobs.complete(jobId, {
  outcome: {
    message: `Operation was not completed succesfully. No records were processed.`,
    acknowledge: true,
    next: {
      type: "retry",
      label: "Try again",
    },
  },
});