[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
GNU Parallel 20130630 alpha released
From: |
Ole Tange |
Subject: |
GNU Parallel 20130630 alpha released |
Date: |
Sun, 30 Jun 2013 20:40:54 +0200 |
GNU Parallel 20130630 alpha has been released. It is available for
download at: http://alpha.gnu.org/gnu/parallel/
I have access to a few machines with more than 10 cores. Spawning jobs
on those remotely is a bother because /etc/ssh/sshd_config:MaxStartup
only all allows for 10 simultaneous logins, so GNU Parallel will only
spawn 10 jobs on those machines.
So far this has been patched using --sshdelay so you could tell GNU
Parallel to slow down logging in. But is a bad solution because you
could have servers in your cluster that are local and some that are on
the other side of the globe, thus having very different login times.
A year ago I devised a solution for that, but the coding
infrastructure for that was missing. That has now changed.
So when you now use --filter-hosts GNU Parallel will compute how long
it took to log in, and use a --sshdelay for that host at 1/5th of that
duration. Ideally 1/10th of the duration should also work, but that
fails too often for me.
The speedup is remarkable because more jobs get scheduled:
$ time src/parallel -j1000% -S m,e,l,o,c,f,w -j0 sleep 20\;echo {} ::: {1..500}
real 18m13.651s
user 0m9.713s
sys 0m19.885s
$ time src/parallel --filter-hosts -j1000% -S m,e,l,o,c,f,w -j0 sleep
20\;echo {} ::: {1..500}
real 1m28.680s
user 0m12.733s
sys 0m25.218s
So I am considering making --filter-hosts default, but I am not sure
if anything breaks. Therefore, dear users, could you please:
* Download the alpha release
http://alpha.gnu.org/gnu/parallel/parallel-20130630.tar.bz2
* Put '--filter-hosts' in /etc/parallel/config
If that causes problems please post a bug report.
/Ole
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- GNU Parallel 20130630 alpha released,
Ole Tange <=