A workflow will run faster when less data runs through fewer tools. Therefore the most straightforward way of reducing our run time is to minimise the volume of data and eliminate the tools we don’t need, or find more efficient ways of processing our data. This will only increase our processing speed but will also reduce the size of the workflow.
6 Steps to reducing the speed of your workflow:
Use Filter / Select
- Use the select tool to remove any fields you don’t need, as early as possible in the workflow
- Use the filter tool or the sample tool to limit the amount of data in the workflow.
Browse tools create temporary yxdb files that take time to create and run.
- Remove Browse tools or
- Disable browse tools. Do this by clicking anywhere on the blank space on your workflow. Workflow configuration will pop up on the left. To to “Runtime” and click “disable all browse tools”
- Use the Autofield tool. The Autofield tool will read all of the string records in an input and set the field data type to the smallest size possible. The Autofield tool can be found in the Preparation tool bar (the blue tools).
Use tool containers to disable part of a workflow.
- Pop tools into a container by highlighting selected tools, right clicking and selecting “Add To New Container” before going to the tool configuration and selecting “Disabled”.
Cache and run
Caching the initial part of the flow will store the data from that flow in a temporary yxdb file on disk, so that when we hit run, we use this temporarily stored data instead of hitting the datasources every time we run the flow.
- Highlight the datasource and part of the workflow, right click and select "Cache and Run Workflow".
Note that we can’t cache:
- Tools with multiple outputs
- Tools that operate in “a circle”, ie where the output of a tool is being combined with a different component of the same data stream.
Enable performance profiling to see which parts of the workflow are running slowest, an hence may need rethinking. Enable performance profiling by going to Workflow configuration and selecting “Runtime” as above, then select "Enable Performance Profiling".