The Data School - The Alteryx Multithreading Engine

The AMP (Alteryx Multithreaded Processing) Engine is a feature in Alteryx that enables Workflow to be executed faster. This is done by using multiple threads at the same time to process the data.

To understand how this works, we first have to understand what a thread is. A thread is a part of a process the CPU is performing. Within one thread a clear sequence of operations is performed.

When you open the task manager on your computer, you can see how many processes and threads your computer is currently executing. In Fig.1 we can see that my computer is currently running 324 processes with a total of 5203 threads.

Figure 1: Processes and Threads running currently.

Let’s take a look at Alteryx again. for example an Input Tool. With a disabled AMP Engine, Alteryx would just read in every single record sequentially from left to right and then from top to bottom. This is a simple, but slow process. With Multithreading, the reading process gets more complicated, as Alteryx will read in multiple lines simultaneously.

Figure 2: The Alteryx Engine vs. the Alteryx Multithreading Process (with two threads in this example).

When the AMP Engine is enabled, the rising complexity may cause errors when handled incorrectly. Therefore it is important to identify where errors might occur.

It is important to mention, that with the AMP Engine off, Alteryx won’t just use s single thread to execute the whole workflow, but just one for an individual tool. Normally Alteryx executes multiple tools simultaneously and therefore multiple threads are still used. This works, as each tool is an individual process within Alteryx.

Suppose we have an Input Tool followed by a Formula Tool. Once the Input Tool has read in a few records, those will be ‘transported’ to the Formula Tool, which will act on those, while the Input Tool is still reading the missing records.

Problems with the AMP Engine

Some known implications can occur when the AMP Engine is used, you may want to check the Alteryx Help Site.

When you encounter unexpected behavior in simple tools it might be a good idea to turn the AMP Engine off and re-run the workflow.

Let’s take a look at something that can come up within an Input Tool.

If we observe the string that’s marked (Fig.3), we can see that it is in quotes and has an Enter Sign in it.

Figure 3: Possible String problem with the AMP Engine.

We want this title to be read in one single line, and not in two. Therefore we need to enable “Ignore Delimiter in Quotes”. With the classic Alteryx Engine, this works perfectly fine and we would get the desired result with a header and two lines, and the "Enter" in the specific cell (Fig. 4).

Figure 4: Output when reading in the CSV via the classic Alteryx Engine.

With the AMP Engine, we run into a special problem. Since the records are not read sequentially downwards Alteryx starts reading line 4 before the Quote in line 3 is even detected. Therefore Alteryx will run into an error, resulting in an error message (Fig.5).

Figure 5: Error Message when trying to read in the CSV via the AMP Engine.

Now, since Alteryx is aware of that problem, there is an additional option in the Input Tool.

By checking “AMP Only: Allow Newlines in Quoted Fields” the problem gets resolved. The workflow can be executed. This ensures that the file is read in the classic format. Now, of course, we also run into performance reduction again, since the AMP Engine can’t work like it regularly does.

Figure 6: Additional Options in the Input Tool with the AMP Engine.

There are a few things that make the AMP Engine behave differently, therefore it is important to have a certain knowledge about such topics. Often it is worth asking yourself if you want to focus on workflow performance and go with the AMP Engine, but for some use cases, the classic Alteryx Engine might also be a better idea. Especially when you change the engine option of an already existing workflow, you have to make sure that all steps still work as expected.