[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Is there any advantage to turning the following rsync cmd into a paralle
From: |
Alex Muir |
Subject: |
Is there any advantage to turning the following rsync cmd into a parallel cmd? |
Date: |
Thu, 19 Jul 2012 15:33:55 -0400 |
Hi,
I've created a nice script with parallel and then I use rsync to
return output files which are various files, html and zip in several
different directories. My client is wondering if the rsync aspect at
the end can be done with parallel and if there would be an benefit to
doing that and insisted I ask about it.
Looking at the documentation I saw the one can --return files that
were used as input files as well as create another command to return
files generally. It was not clear to me though if one could return
directories recursively as I do with the below rsync commands.
I launch the script with the following command in these case it runs
up to 1000 listed files on three servers
./launchi4EnrichProcess.sh SEC/10k-GHU/2004/
192.168.56.10,192.168.56.11,192.168.56.12 1000
ls /mnt/xslt_volume/i4ContentSource/$1*.zip | head -$3 | parallel
-j-0 --eta --progress --sshlogin $2 --timeout 1500 --transfer --joblog
/mnt/jobs.txt "sh /mnt/config/run-calabash-withParam.sh
/mnt/config/xproc.xpl /mnt/{/.}Log.txt {}" /mnt/output/$1
#I wonder can the below which loops through N CSV severs rsyncing data
from each server be converted into a few parallel statements and if
there is any advantage to doing that?
IFS=","
var="$2"
for v in $var
do if test $v != ':'; then
rsync -r -v -e ssh $v:/mnt/result/$1 /mnt/result/$1;
rsync -r -v -e ssh $v:/mnt/logs/ /mnt/logs/;
fi
done
Regards
--
-
Alex G. Muir
Software Engineering Consultant
Linkedin Profile : http://ca.linkedin.com/pub/alex-muir/36/ab7/125
- Is there any advantage to turning the following rsync cmd into a parallel cmd?,
Alex Muir <=