[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Advice on using parallel as a batch queue system
From: |
Tong, Lianheng |
Subject: |
Advice on using parallel as a batch queue system |
Date: |
Tue, 8 Jan 2013 17:22:05 +0000 |
Dear all,
This is my first post and I am very pleased to have discovered gnu parallel
about a month ago. It had saved me a lot hassle while running a lot of serial
simulation jobs. So a big thanks to the developers :-)
I have however encountered some problems when trying to use parallel as a batch
queue, and I am not sure if it is a feature of the code or me not doing things
incorrectly, so any advice would be welcome. The version of parallel I am
using is 20121222
As the first try, I did the following:
true > queue
tail -f queue | parallel -j 4 &
echo "echo 1" >> queue
(nothing happened)
echo "touch foo" >> queue
(nothing happened, and ls shows that foo was not created)
echo "echo 2" >> queue
(nothing happened, foo not created)
echo "echo 3" >> queue
(1 is output in stdout, and file foo has been created.)
echo "echo 4" >> queue
(nothing happened)
echo "echo 5" >> queue
(2 is displayed in stdout)
echo "touch foo2" >> queue
(3 is displayed in stdout, and file foo2 is created.)
Killing parallel: kill %1 gives the following output:
parallel: SIGTERM received. No new jobs will be started.
parallel: Waiting for these 3 jobs to finish. Send SIGTERM again to stop now.
petrof: lt$ parallel: touch foo2
parallel: echo 5
parallel: echo 4
4
5
[1]+ Done tail -f ../queue | parallel -j 4
So it seems that jobs are only run AFTER job-slots number (in this case 4) of
jobs have been submitted. If I have only sent 3 jobs to the queue, then none of
the jobs are executed until I send the SIGTERM to parallel to terminate the
queue.
On the online documentation, it is said that jobs sent to the queue should run
immediately (provided there are free slots). Is this something that has been
changed in the newer revisions? Or have I done something incorrect?
Many thanks,
Lianheng
==========================================================
Lianheng Tong Tel:
+44 20 7679 3302
London Centre For Nanotechnology Fax: +44 20 7679 0595
University College London
lianheng.tong@ucl.ac.uk
17–19 Gordon Street, London WC1H 0AH, U.K.
==========================================================
- Advice on using parallel as a batch queue system,
Tong, Lianheng <=