In this guide we will introduce you to Pipelines! We will show where to find pipelines, what is needed to use it and how to set up your first task.
What can I do with Pipelines?
Pipelines has a similar function as the services that you can add to a space. They allow you to import or export data and make use of the transformation service. All functionalities are listed when making a task, but we will show you that part later in this guide.
What pipelines also allows you to do is schedule tasks. For example, if you want to import a big set of data, but don't want to do this during working hours, you can schedule it to run during the night instead. Or if you want to run an export every month to make a backup of your data, you can schedule the export to run on a certain day of the month.
Where can I find Pipelines?
The pipelines menu can be found on the taskbar on the left side of the screen (see 1 in the figure below) and requires that the user has write access to the space they want to use pipelines on.
Clicking this menu item will reveal a submenu with two different options:
- Tasks: Item 2 in image above. This will take you to an overview of all existing tasks on the space, these tasks can be completed, upcoming or repeating tasks.
- Create task: Item 3 in image above. This will directly take you to the place where you can add a task to the space.
How do I set up Pipelines?
So let's set up the first task on a space. For this guide we will start from the task page (2 in image above).
In the image above you can see what the tasks page looks like if no tasks have been added to a space. To add a task you can press either 1 or 2 in the image above, both have the same functionality and take you to the same page.
In this image you can see the landing page when adding a new task, both functionality and branch selectors are dropdown menus. For branches the options are limited to the branches that exist on a space, more can be created in the version control menu if so desired. You also have the option to add a name to the task for easy recognition on the overview page.
For functionality, it will list all the current profiles available for the transform service and the import export service. In this example we will add an import for a Json file and using the basic profile. When expanded it will look like this:
As you can see here, import Json has 3 profiles available, this step is not there for every profile, but will expand automatically if it is. Once a profile is selected we'll see the next part of setting up a task, namely the config for the profile, as seen below:
In this case we have only one option for the configuration of our task, but other profiles can have different options. For any questions about a specific profile please reach out to us with your question.
For graph any input can be used and all nodes imported by this task will be in the graph provided here, leave this blank if you want all nodes to be imported into the default graph.
The next tab in the create task page takes us to 'Crons', this is where you can add a schedule to the task you are creating.
By pressing the 'Add Cron' button as seen in the screenshot above, we have to choose between two types of run, a run once or a recurring type.
If we select Run Once, there will only be a First run option we have to select, as can be seen in the image below:
Here you can either choose a later date and time to start the run, or you can select 'Now' in the bottom left of the date window, selecting now means that the task will run as soon as it is saved.
If we select Recurring there will be a few more options as can be seen in the image below:
First of all you can use either a simplified notation or a cron notation to set the schedule. In this guide we chose the simplified notation. In this case we chose to run the task on the first day of each month at 12:00, but there are plenty of options to choose from.
The final tab in the create task menu is 'Parameters'. The content of this menu will change depending on the selected functionality, but for imports this is the place where you can either upload a file, or select a previously used file for this task.
After we are done with setting up the task, we now press the green 'Save' button located in the bottom left corner. Doing this will take us back to the 'Tasks' page we have seen before, but now there is a new entry!
Now we will look at the remaining options that can be performed on existing tasks.
In this image you can see four things we have not explained yet, the run now button (play icon), the last run column, the actions menu (three dots on the right of a task) and the jobs log on the expanded task.
The run now button will rerun the task with the same configuration as was previously set up with just one click on the button.
The last run column will display either a green check mark if the last run of this task was successful, or a red cross icon if the last run of this task had an error.
The actions menu will have two actions as seen in the image above, it allows you to edit the config and the assigned crons of a task via the 'Edit' menu and if the task has a recurring cron associated with it, it allows you to disable the recurring cron via the 'Disable' option.
And finally the jobs log will show a list of any time the task has run and allows you to look at the logs of the job and also allows you to download the file that was used for the job. If no job has been completed it will show 'No Jobs' and if more than 5 jobs have been completed, the list will only show the last 5 and will have a button 'Show more' at the bottom to load earlier jobs. For any job you will also see if it was successful or not, when it started and when the job ended.
In the case a job will fail, the icon on the left of the jobs list will be a red cross, as seen in the image below:
Clicking 'Show Logs' will open a new window with a log entry of the functionality used in the task. This log will provide information why the job failed to complete and can be used to solve any issues.
For any remaining questions about the functionality of pipelines or how to use it please contact us at support@weaverhq.com