parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

how to split the files up into ~equal groups when moving them to other s


From: Alex Muir
Subject: how to split the files up into ~equal groups when moving them to other server
Date: Tue, 17 Jul 2012 15:13:28 -0400

Hi,

I've setup the following commands, the first to move files to another
server and the second to distribute a process over the 2 computers.
Currently I'm just testing with a virtual box.

My question though is how to split the input files up into ~equal
groups when moving them so that the process is truly distributed such
that each process runs with different input files. Currently what I
have below would make the process run with all the same files on the
local and virtual box server.



Transferring Files

ls /mnt/xslt_volume/i4ContentSource/578/*.zip | parallel --sshlogin
192.168.56.10 --transfer mv *.zip
/mnt/xslt_volume/i4ContentSource/578/

Distributing Work

sudo ls /mnt/xslt_volume/i4ContentSource/578/*.zip |  parallel -j-1
--eta --progress --sshlogin 192.168.56.10,:  --timeout 1500 "sh
/mnt/xslt_volume/i4EnrichV4/src/enrich/10k/completeMultiDocumentProcess/processExecution/configuration/run-calabash-withParam.sh
/mnt/xslt_volume/i4EnrichV4/src/enrich/10k/completeMultiDocumentProcess/processExecution/configuration/Createi4EnrichMarkup-Template-ConvertedToParameterDrivenFile.xpl
/mnt/xslt_volume/i4EnrichV4/src/enrich/10k/completeMultiDocumentProcess/processExecution/logs/{/.}Log.txt
{}"


Thanks

--
-

Alex G. Muir
Software Engineering Consultant
Linkedin Profile : http://ca.linkedin.com/pub/alex-muir/36/ab7/125



reply via email to

[Prev in Thread] Current Thread [Next in Thread]