In this guide we will introduce you to Pipelines! We will show where to find pipelines, what is needed to use it and how to set up your first task.
What can I do with Pipelines?
Pipelines has a similar function as the services that you can add to a space. They allow you to import or export data and make use of the transformation service. All functionalities are listed when making a task, but we will show you that part later in this guide.
What pipelines also allows you to do is schedule tasks. For example, if you want to import a big set of data, but don't want to do this during working hours, you can schedule it to run during the night instead. Or if you want to run an export every month to make a backup of your data, you can schedule the export to run on a certain day of the month.
Where can I find Pipelines?
The pipelines menu can be found on the taskbar on the left side of the screen (see 1 in the figure below) and requires that the user has write access to the space they want to use pipelines on.
Clicking this menu item will reveal a submenu with two different options:
- Tasks: Item 2 in image above. This will take you to an overview of all existing tasks on the space, these tasks can be completed, upcoming or repeating tasks.
- Create task: Item 3 in image above. This will directly take you to the place where you can add a task to the space.
How do I set up Pipelines?
So let's set up the first task on a space. For this guide we will start from the task page (2 in image above).
In the image above you can see what the tasks page looks like if no tasks have been added to a space. To add a task you can press either 1 or 2 in the image above, both have the same functionality and take you to the same page.
In this image you can see the landing page when adding a new task, both functionality and branch selectors are dropdown menus. For branches the options are limited to the branches that exist on a space, more can be created in the version control menu if so desired. For functionality, it will list all the current profiles available for the transform service and the import export service. In this example we will add an import for a Json file and using the basic profile. When expanded it will look like this:
As you can see here, import Json has 3 profiles available, this step is not there for every profile, but will expand automatically if it is. Also submenus of profiles will have a '>' at the end if more than one option is available. Once a profile is selected we'll see the next part of setting up a task, namely the config for the profile, as seen below:
In this case we have only two options for the configuration of our task, but other profiles can have different options. For any questions about a specific profile please reach out to us with your question.
Here we see 'File' and 'Graph' as options. For 'File' you can either select a file you want to import, or if you have imported a similar file previously, click on the select from previous uploaded file section.
For graph any input can be used and all nodes imported by this task will be in the graph provided here, leave this blank if you want all nodes to be imported into the default graph.
After all of this there is still one thing left, and that is to add a cron to the task. By pressing the 'Add Cron' button as seen in the screenshot above, we have to choose between two types of run, a run once or a recurring type.
If we select Run Once, there will only be a First run option we have to select, as can be seen in the image below:
Here you can either choose a later date and time to start the run, or you can select 'Now' in the bottom left of the date window, selecting now means that the task will run as soon as it is saved.
If we select Recurring there will be a few more options as can be seen in the image below:
First of all you can use either a simplified notation or a cron notation to set the schedule. In this guide we chose the simplified notation. In this case we chose to run the task on the first day of each month at 12:00, but there are plenty of options to choose from.
After we are done with setting up the task, we now press the green 'Create Task' button located in the bottom left corner. Doing this will take us back to the 'Tasks' page we have seen before, but now there is a new entry!
And finally we will look at the remaining options that can be performed on existing tasks.
In this image you can see four things we have not explained yet, the edit button (pencil icon) , the run now button (play icon), the actions menu (three dots on the right of a task) and the jobs log.
The edit button opens up a brief summary of the task and allows you to change the graph or file used if desired, before pressing the new run now button again, at which point the task will start.
The run now button will rerun the task with the same configuration as was previously set up with just one click on the button.
The actions menu will have two actions as seen in the image above, it allows you to edit the config and the assigned crons of a task via the 'Edit' menu and if the task has a recurring cron associated with it, it allows you to disable the recurring cron via the 'Disable' option.
And finally the jobs log will show a list of any time the task has run and allows you to look at the logs of the job and also allows you to download the file that was used for the job. If no job has been completed it will show 'No Jobs' and if more than 5 jobs have been completed, the list will only show the last 5 and will have a button 'Show more' at the bottom to load earlier jobs. For any job you will also see if it was successful or not, when it started and when the job ended.
In the image above, both jobs have been completed successfully, in the case a job will fail, the icon on the left of the jobs list will be a red cross, as seen in the image below:
Clicking 'Show Logs' will open a new window with a log entry of the functionality used in the task. This log will provide information why the job failed to complete and can be used to solve any issues.
For any remaining questions about the functionality of pipelines or how to use it please contact us at [email protected]