Andy Chu — 2018/05/24

What is xargs -P?

• xargs is a Unix command that runs another Unix command multiple times with different inputs.
• -P 4 tells it to run 4 parallel processes.
• It's a quick and easy "one-liner" that's available almost everywhere.
• [demo]

When is it useful?

1. For productionizing data science and research (often CPU-intensive tasks).

   Example: RAPPOR Data Flow

2. To run tests in parallel (especially with different test frameworks).

   [oil spec test demo]
   

3. As a quick and dirty way to speed up anything else, including I/O-bound tasks.

    cat tasks | xargs wget ...
   

Limitations

• Sometimes you need shared state
• When each process has a non-trivial initialization (though there are often easy workarounds)