Re: parallel + blast + LSF

Hi,

Maybe this time it could be a GNU parallel issue.

I finally managed to run something with the following script:

--------------------------------------------------------------------------------------------------------------------------------------------------------------

#!/bin/bash

#BSUB -J gnuParallel_blast_test # Name of the job.

#BSUB -o %J.out # Appends std output to file %J.out. (%J is the Job ID)

#BSUB -e %J.err # Appends std error to file %J.err.

#BSUB -q cresco3_h144 # Queue name.

#BSUB -n 8 # Number of CPUs.

#BSUB -W 1210 # walltime

module load 4.8.3/ncbi/12.0.0

module load 4.8.3/parallel/20150122

SLOTS=`cat ${LSB_DJOB_HOSTFILE} |wc -l`

SERVER=""

for i in `cat ${LSB_DJOB_HOSTFILE}| sort`

echo "/afs/enea.it/software/bin/blaunch.sh ${i}" >> servers

done

/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000 -query - > /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/all-vs-all-outfmt-6-filtered

cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000.fasta | parallel --no-notice -vv -j ${SLOTS} --tmpdir /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp --wait --slf servers --block 20k --recstart '>' --pipe blastp -evalue 1e-05 -outfmt 6 -db /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000 -query - -out /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_{#}

--------------------------------------------------------------------------------------------------------------------------------------------------------------

The most important changes are

- using --tmpdir options since temporary files must be on a shared filesystem(!!).

- I used a larger data chunk (--block 20k instead of -N 1)

- I used --wait for LSF load easing

- I used {#} instead of {%} to tag result file for better debugging

I finally got some non empty result files. Unfortunately I only got only 8 file of expected 22

these are stdout and stderr:

--------------------------------------------------------------------------------------------------------------------------------------------------------------

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par09c3l.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par09c3l.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par09c3l.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par09c3l.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par09c3l.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x003.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"3\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_3\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par26f1H.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par26f1H.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par26f1H.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par26f1H.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par26f1H.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x020.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"2\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_2\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/partdB11.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/partdB11.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/partdB11.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/partdB11.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/partdB11.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x020.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"5\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_5\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parzRBOV.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parzRBOV.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parzRBOV.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parzRBOV.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parzRBOV.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x020.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"8\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_8\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/pardYnbH.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/pardYnbH.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/pardYnbH.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/pardYnbH.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/pardYnbH.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x003.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"9\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_9\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parpLXXu.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parpLXXu.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parpLXXu.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parpLXXu.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parpLXXu.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x020.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"11\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_11\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parpDyze.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parpDyze.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parpDyze.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parpDyze.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parpDyze.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x003.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"12\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_12\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parska__.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parska__.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parska__.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parska__.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parska__.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x020.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"14\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_14\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par07hkH.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par07hkH.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par07hkH.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par07hkH.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par07hkH.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x003.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"15\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_15\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parwWxw0.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parwWxw0.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parwWxw0.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parwWxw0.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parwWxw0.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x020.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"17\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_17\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/paroNaAJ.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/paroNaAJ.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/paroNaAJ.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/paroNaAJ.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/paroNaAJ.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x003.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"18\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_18\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parchOGP.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parchOGP.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parchOGP.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parchOGP.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parchOGP.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x020.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"20\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_20\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parVhSi8.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parVhSi8.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parVhSi8.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parVhSi8.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parVhSi8.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x003.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"21\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_21\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/pareLiHn.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/pareLiHn.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/pareLiHn.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/pareLiHn.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/pareLiHn.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x003.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"23\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_23\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parse3wL.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parse3wL.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parse3wL.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parse3wL.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parse3wL.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x003.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"6\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_6\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parQh5eJ.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parQh5eJ.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parQh5eJ.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parQh5eJ.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parQh5eJ.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x003.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"27\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_27\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parnuokL.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parnuokL.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parnuokL.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parnuokL.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parnuokL.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x020.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"25\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_25\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parmXJ5N.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parmXJ5N.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parmXJ5N.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parmXJ5N.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parmXJ5N.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x020.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"26\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_26\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parXM76W.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parXM76W.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parXM76W.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parXM76W.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parXM76W.chr; cat - ) | (/afs/enea.it/software/bin/blaunch.sh cresco3x020.portici.enea.it exec perl -e \\\$ENV\\\{\\\"PARALLEL_PID\\\"\\\}=\\\"22663\\\"\\\;\\\$ENV\\\{\\\"PARALLEL_SEQ\\\"\\\}=\\\"24\\\"\\\;\\\$bashfunc\\\ =\\\ \\\"\\\"\\\;@ARGV=\\\"blastp\\\ -evalue\\\ 1e-05\\\ -outfmt\\\ 6\\\ -db\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000\\\ -query\\\ -\\\ -out\\\ /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_24\\\"\\\;\\\$SIG\\\{CHLD\\\}=sub\\\{\\\$done=1\\\;\\\}\\\;\\\$pid=fork\\\;unless\\$\\\$pid\\$\\\{setpgrp\\\;exec\\\$ENV\\\{SHELL\\\},\\\"-c\\\",\\$\\\$bashfunc.\\\"@ARGV\\\"\\$\\\;die\\\"exec:\\\$\\\!\\\\n\\\"\\\;\\\}do\\\{\\\$s=\\\$s\\\<1\\\?0.001+\\\$s\\\*1.03:\\\$s\\\;select\\$undef,undef,undef,\\\$s\\$\\\;\\\}until\\$\\\$done\\\|\\\|getppid==1\\$\\\;kill\\$SIGHUP,-\\\$\\\{pid\\\}\\$unless\\\$done\\\;wait\\\;exit\\$\\\$\\\?\\\&127\\\?128+\\\(\\\$\\\?\\\&127\\$:1+\\\$\\\?\\\>\\\>8\\\););

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/pargNN8b.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/pargNN8b.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/pargNN8b.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/pargNN8b.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/pargNN8b.chr; cat - ) | (blastp -evalue 1e-05 -outfmt 6 -db /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000 -query - -out /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_10);

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parf3mob.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parf3mob.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parf3mob.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parf3mob.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parf3mob.chr; cat - ) | (blastp -evalue 1e-05 -outfmt 6 -db /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000 -query - -out /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_7);

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parGTSb6.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parGTSb6.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parGTSb6.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parGTSb6.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parGTSb6.chr; cat - ) | (blastp -evalue 1e-05 -outfmt 6 -db /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000 -query - -out /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_16);

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parxwGS2.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parxwGS2.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parxwGS2.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parxwGS2.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parxwGS2.chr; cat - ) | (blastp -evalue 1e-05 -outfmt 6 -db /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000 -query - -out /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_4);

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parCrFM_.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parCrFM_.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parCrFM_.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parCrFM_.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parCrFM_.chr; cat - ) | (blastp -evalue 1e-05 -outfmt 6 -db /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000 -query - -out /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_1);

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par7fOlu.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par7fOlu.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par7fOlu.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par7fOlu.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/par7fOlu.chr; cat - ) | (blastp -evalue 1e-05 -outfmt 6 -db /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000 -query - -out /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_19);

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parKlqXO.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parKlqXO.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parKlqXO.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parKlqXO.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parKlqXO.chr; cat - ) | (blastp -evalue 1e-05 -outfmt 6 -db /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000 -query - -out /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_22);

sh -c 'dd bs=1 count=1 of=/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parsuHqk.chr 2>/dev/null'; test ! -s "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parsuHqk.chr" && rm -f "/gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parsuHqk.chr" && exec true; (cat /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parsuHqk.chr; rm /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/tmp/parsuHqk.chr; cat - ) | (blastp -evalue 1e-05 -outfmt 6 -db /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/goodProteins_first_0010000 -query - -out /gporq1_1M/usr/aprea/bio/solanum_melongena/analysis/orthomcl_00/resultd_13);

--------------------------------------------------------------------------------------------------------------------------------------------------------------

Apr 16 17:47:26 2015 22805 3 7.03 lsb_launch(): Failed while executing tasks.

Apr 16 17:47:27 2015 22885 3 7.03 lsb_launch(): Connection refused by server.

Apr 16 17:47:27 2015 22938 3 7.03 lsb_launch(): Connection refused by server.

Apr 16 17:47:27 2015 22939 3 7.03 lsb_launch(): Connection refused by server.

Apr 16 17:47:27 2015 22924 3 7.03 lsb_launch(): Connection refused by server.

Apr 16 17:47:26 2015 22802 3 7.03 lsb_launch(): Failed while executing tasks.

Apr 16 17:47:27 2015 22928 3 7.03 lsb_launch(): Connection refused by server.

Apr 16 17:47:27 2015 22946 3 7.03 lsb_launch(): Connection refused by server.

Apr 16 17:47:27 2015 22950 3 7.03 lsb_launch(): Connection refused by server.

Apr 16 17:47:27 2015 22923 3 7.03 lsb_launch(): Connection refused by server.

Apr 16 17:47:27 2015 22951 3 7.03 lsb_launch(): Connection refused by server.

Apr 16 17:47:27 2015 22952 3 7.03 lsb_launch(): Connection refused by server.

Apr 16 17:47:27 2015 22915 3 7.03 lsb_launch(): Connection refused by server.

Apr 16 17:47:27 2015 23022 3 7.03 lsb_launch(): Connection refused by server.

Apr 16 17:47:27 2015 22787 3 7.03 lsb_launch(): Failed while executing tasks.

Apr 16 17:47:27 2015 23046 3 7.03 lsb_launch(): Connection refused by server.

Apr 16 17:47:27 2015 23054 3 7.03 lsb_launch(): Connection refused by server.

Apr 16 17:47:27 2015 23058 3 7.03 lsb_launch(): Connection refused by server.

Apr 16 17:47:27 2015 23053 3 7.03 lsb_launch(): Connection refused by server.

--------------------------------------------------------------------------------------------------------------------------------------------------------------

the server file is

--------------------------------------------------------------------------------------------------------------------------------------------------------------

/afs/enea.it/software/bin/blaunch.sh cresco3x003.portici.enea.it

/afs/enea.it/software/bin/blaunch.sh cresco3x020.portici.enea.it

--------------------------------------------------------------------------------------------------------------------------------------------------------------

And the master node is cresco3x003

I don't understand the output from -vv options very well but, as you can see (I hope) the last 8 lines from stdout (with indices 1,4,7,10,13,16,19,22) are different and they correspond to the 8 output file I finally retrieve. I would have expected the same lines at least for the first 5 indices (1,2,3,4,5) since PARALLEL eventually could have recognized it was on the local host but that's not the case and I find that strange....

Does anybody have an idea of what's going on?

Ciao,

Giuseppe

PS I remind I am using version 20150122

On Thu, Apr 16, 2015 at 3:49 PM, George Marselis <george@marsel.is> wrote:

You could do that, and it might work, if you are interested in using the LSF API

otherwise, use a batch script and blaunch < batchscript.sh , instead.

Ciao,

George

On Thu, Apr 16, 2015 at 9:24 AM, Giuseppe Aprea <giuseppe.aprea@gmail.com> wrote:
Dear George,

I was not sure you were talking to me because I am submitting 1 job only. Then I am using blaunch as a spawn agent instead of shh because this is the way our cluster works. Mpi jobs with several hundreds slots also use it (orte_rsh_agent MCA parameter).

What do you think if I replace

--recstart '>' -N 1 --pipe

with

--recstart '>' -block 100k --pipe

?

I think I could reduce load on lsb_launch() using a larger data chunk, couldn't I?

cheers,

g

On Wed, Apr 15, 2015 at 10:32 PM, George Marselis <george@marsel.is> wrote:
> You could use parallel to submit jobs, but its a very bad idea, due to the limitations of the software.

and by that, I mean the limitations of the LSF software.

Parallel rocks. Ole rocks.

Next Parallel Release should be named "GNU Terry Pratchett"

> Let me know which option is better for you.
> As with regard to Martin, he should not use parallel for

^-- this is your brain on Perl.

Best Regards,

George

On Wed, Apr 15, 2015 at 11:27 PM, George Marselis <george@marsel.is> wrote:
Giuseppe, I was referring to both of you. My apologies I was not clear, I had my head stuck in Perl while writing the first email.

My suggestion to both of you is that you should not use parallel for your respective topics.

Giuseppe,

You should use an extra script. Your problem is that you are timing out while trying to submit all those jobs. The timeout happens because of the number of jobs you are submitting: LSF cannot write the job descriptions fast enough to disk, times out because the action is not completed and then stays in that state

----------------
Martin,

You could use parallel to submit jobs, but its a very bad idea, due to the limitations of the software. Use batch scripts and job arrays when possible.

----------------

So, as per my suggestion, I think our discussion is offtopic for this list. We could continue here, if Ole and the list puts up with us, but I think we should take this on a personal email or switch this to the Debian Medical email list https://en.wikipedia.org/wiki/Debian-Med .

Let me know which option is better for you.

As with regard to Martin, he should not use parallel for

Ciao,

George

On Wed, Apr 15, 2015 at 10:50 PM, Giuseppe Aprea <giuseppe.aprea@gmail.com> wrote:
Hi George!

I am not sure who you are talking with. Martin or me? I remind the original topic is about using blast under parallel with LSF.
Martin's problem sounds like something offtopic.

You have both sysadmin and bioinformatics experience so I would really appreciate your help!

I am working on a cluster so I must use LSF to get slots and I would prefer using parallel also since it splits input automatically with --recstart (which is quite nice:D otherwise I have to use another script for that). I see I could do better with chunksize (I have 1 record at time in my example) but that's a secondary problem now. First I have the "lsb_launch(): Failed while waiting for tasks to finish." issue to solve.

cheers,

g

On Wed, Apr 15, 2015 at 7:44 PM, George Marselis <george@marsel.is> wrote:
By the way, LSF and GNU parallel do almost the same thing. So using one of the two, defeats the purpose of using the other.

In the same way, you could have used LSF to submit your jobs to LSF:

bsub < script.sh

where script.sh was

bsub -J amoeba -q smalljobs qfasta file1
bsub -J amoeba -q smalljobs qfasta file2
...
bsub -J amoeba -q smalljobs qfasta file2000

On Wed, Apr 15, 2015 at 8:39 PM, George Marselis <george@marsel.is> wrote:
Hi. LSF/Openlava sysadmin in bioinformatics and parallel user here.

I have seen this a couple more times: You are trying to use GNU parallel to submit the jobs to all nodes.

THat's now the way to do things: You should not submit jobs on *all* your nodes. Please don't do that, as bsub was not designed to read large chunks of jobs. bsub writes the jobs to your home directory, so if your storage is not designed for a lot of writes, you are going to blow the cluster out of the water.

What you want to do is look up either:

1. bsub scripts https://rc.fas.harvard.edu/resources/documentation/legacy-lsf/lsf-submit-an-lsf-job/

or

2. job arrays https://rc.fas.harvard.edu/resources/documentation/legacy-lsf/lsf-submitting-lots-of-short-jobs-job-arrays/

Both bsub scripts and job arrays are useful to you: bsub scripts can be submitted as part of a pipeline: you can program the output of the bsub script from your pipeline and then submit it to bsub. So, instead of submitting your job 2000 times as in

bsub job0
bsub job1

....

bsub job1999

you just submit "bsub < scriptname" which contains 2000 lines which describe your jobs and you are done. The rest is done by bsub/LSF

Now, if your jobs are similar in a way that you just increment counter (as in most bioinformatics jobs), use arrays.

bsub -J JOBNAME[0-1999], where JOBNAME is a string you would like to name your job as, eg "fasta files alignment"

These techniques are useful because you can submit all 2000 jobs in less than a second, you can do it from a single node and you will not have to deal with a grumpy sysadmin or grumpy colleagues who cannot use the cluster. Just make sure you use the appropriate queue.

Let me know if you have any questions.

Best Regards,

George Marselis

On Wed, Apr 15, 2015 at 6:48 PM, Martin d'Anjou <martin.danjou14@gmail.com> wrote:

Hi,

Thanks for clarifying. I want to use GNU Parallel to bsub jobs. This way I can use GNU Parallel to throttle the number of jobs that are submitted to LSF, and it is easier than writing a loop.

parallel -j 100 my_script [bsub options] ::: {1..2000}

my_script (pseudo-code):
#!/bin/bash
...
bsub [bsub options] command ...
post-process data

This way I can submit jobs, say 100 at a time. When I submit all 2000 jobs, it gets problematic and I start hitting limits with file descriptors, etc.

Thanks for sharing,
Martin

On 15-04-15 11:35 AM, Giuseppe Aprea wrote:

Hi Martin,

I am not sure I understand. As far as I can see, things work exactly the opposite way: you have an LSF script which launches GNU Parallel on some hosts provided by LSF. Something like:

-------------------------------------------------------------------------------

-------------------------------------------------------------------------------

#!/bin/bash

#BSUB -J gnuParallel_blast_test # Name of the job.

#BSUB -o %J.out # Appends std output to file %J.out. (%J is the Job ID)

#BSUB -e %J.err # Appends std error to file %J.err.

#BSUB -q large # Queue name.

#BSUB -n 30 # Number of CPUs.

module load 4.8.3/ncbi/12.0.0

module load 4.8.3/parallel/20150122

SLOTS=`cat ${LSB_DJOB_HOSTFILE} |wc -l`

SERVER=""

for i in `cat ${LSB_DJOB_HOSTFILE}| sort`

do

echo "/afs/enea.it/software/bin/blaunch.sh ${i}" >> servers

done

cat absolute_path_to_sequences.fasta | parallel --no-notice -vv -j ${SLOTS} --slf servers --plain --recstart '>' -N 1 --pipe blastp -evalue 1e-05 -outfmt 6 -db absolute_path_to_db_file -query - -out absolute_path_to_result_file_{%}

-------------------------------------------------------------------------------

-------------------------------------------------------------------------------

LSF is the one which gives you the execution hosts so if you are launching bsub from GNU parallel how do you know how to set the --slf option?

g

On Wed, Apr 15, 2015 at 4:24 PM, Martin d'Anjou <martin.danjou14@gmail.com> wrote:

On 15-04-15 09:34 AM, Giuseppe Aprea wrote:

Hi all,

I would like to ask you, please, some help in using parallel with blast alignment software.

I am trying to use GNU parallel v. 20150122 with blast for a very large sequences alignment. I am using Parallel on a cluster which uses LSF as queue system.

Hello Giuseppe,

I am an avid LSF user, and I want to use GNU Parallel to dispatch jobs to LSF. Could you please explain a little bit to me how GNU Parallel works with LSF? I do not see it in the on-line tutorials. For example, I would like to understand how to pass "bsub" options like -oo, -q queue_name, etc. to LSF from GNU Parallel.

Thanks,
Martin

From:	Giuseppe Aprea
Subject:	Re: parallel + blast + LSF
Date:	Thu, 16 Apr 2015 18:33:08 +0200