DZone Snippets is a public source code repository. Easily build up your personal collection of code snippets, categorize them with tags / keywords, and share them with the world

Snippets Manager

Snippets has posted 5883 posts at DZone. View Full User Profile

Launch Parallel Shell Process

08.29.2009

| 5454 views |

        Bash in not specially well suited to run background jobs, but it can be accomplish. This is a simple parallel process launcher:

#!/bin/bash

# Echo to stderr
#
debug() {    
  echo "$@" >&2
}

# check_process PID
#
check_process() { kill -0 $1 2>/dev/null; }

# run_parallel NPROCESS FACTORY
#
# Run N parallel proceses, wait until EOF and all running processes are finished
#
#
# FACTORY is a process/function that returns (using stdout) the command to run
# 
# The status code of FACTORY is interpreted like this:
#
#   0: There is more data
#   != 0: There is no more data to process
#
run_parallel() {
  NPROCESS=$1  
  shift
  
  PIDS=()  
  EOF=0
  while test $EOF = 0 -0 ${#PIDS[*]} -ne 0; do
    if test $EOF = 0 -a ${#PIDS[*]} -lt $NPROCESS; then
      if ! COMMAND=$("$@"); then
        debug "EOF"
        EOF=1
      fi
      if test "$COMMAND"; then 
        $COMMAND &
        PID=$!
        debug "starting process: $@ ($PID)"
        PIDS[$PID]=$PID
      fi
    fi
    
    debug "active processes: ${PIDS[*]}"
    sleep 0.5
     
    for PID in ${PIDS[*]}; do      
      if ! check_process $PID; then
        wait $PID 
        unset PIDS[$PID]
        debug "process finished: $PID"
      fi
    done        
  done
}


An example of usage:

process() {
  read -t1 SEC && { debug "sleep $SEC"; return; } 
  test $? -eq 142 && return || return 1
}

run_parallel 4 process

Tags:

Comments

Snippets Manager replied on Tue, 2010/03/23 - 11:24am

Using Parallel (http://savannah.nongnu.org/projects/parallel/) it is fairly easy to run stuff in parallel: (echo foss.org.my; echo debian.org; echo freenetproject.org) | parallel traceroute or even: (echo traceroute foss.org.my; echo traceroute debian.org; echo traceroute freenetproject.org) | parallel