[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: parallel: This should not happen. You have found a bug.
From: |
Linda Walsh |
Subject: |
Re: parallel: This should not happen. You have found a bug. |
Date: |
Sun, 11 Aug 2013 14:42:23 -0700 |
User-agent: |
Thunderbird |
Ole Tange wrote:
On Sun, Aug 11, 2013 at 12:28 AM, Linda Walsh <gnu@tlinx.org> wrote:
I was trying to gunzip a bunch of small files ;
tried
for i in *.gz;do
sem -j6 $i
done
You seem to have forgotten gunzip:
for i in *.gz;do
sem -j6 gunzip $i
done
---
oops yup
This ought to work just fine.
Personally I hate introducing an unneeded variable, so I would write it as:
parallel gunzip ::: *gz
----
specialized syntax -- didn't know...
but your solution should work, too.
so then tried putting the sem in background
That would be wrong and can explain why you get fork-failed later: It
needs to be able to block if 6 jobs are already running.
----
I thought they might all use the same sem (probably not easy)... my
problem was
that when I did it the first way, it was periods of 100% cpu usage, with
minor upticks,
one for about 5 seconds up ~ 180%, but no periods of near 600% cpu usage
which
I would see with a parallel make.
In the ProcessHacker, I would see 6 perl's launched at one time,
then see most or all complete in 1 refresh, see 1 running maybe
4-8s, then see some number (<=6) of perl's running. The 600 files...
have taken well over 10 minutes,
whereas I would have expected the same files processed in make or
xargs to take some number of seconds (<60)....
If you get the error on smaller/fewer files, please include those instead.
parallel: This should not happen. You have found a bug.
Please contact <parallel@gnu.org> and include:
* The version number: 20130122
* The bugid: Can't open semaphore file
/home/law/.parallel/semaphores/id-2fdev2fcons1.lock: Permission denied
Why is GNU Parallel not allowed to create
/home/law/.parallel/semaphores/id-2fdev2fcons1.lock?
----
Likely already in use? I.e. It's cygwin on windows... so likely
if it is already open for write, it can't open it a 2nd time for
write...
This is explains what you see and is most likely the reason for the
failure you see.
Can you create the file by hand:
echo > /home/law/.parallel/semaphores/id-2fdev2fcons1.lock
---
Yes...
While it is running (14 minutes and climbing), it seems to be testing
and recreating (if necessary) the semaphores directory under ".parallel".
I created my own dir, and it stayed, but if I removed the 'semaphores'
dir, it immediately re-created it...
It doesn't look like it is using real semaphores...but a file interlock
which might explain it running single threaded?
What OS is Perl running on and what file system is /home/law/.parallel
hosted on?
----
CYGWIN_NT-6.1-WOW64 / NTFS
(thats 32bit cygwin on 64-bit win7-sp1)...
- parallel: This should not happen. You have found a bug., Linda Walsh, 2013/08/10
- Re: parallel: This should not happen. You have found a bug., Ole Tange, 2013/08/11
- Re: parallel: This should not happen. You have found a bug.,
Linda Walsh <=
- Re: parallel: This should not happen. You have found a bug., Ole Tange, 2013/08/11
- Re: parallel: This should not happen. You have found a bug., Linda Walsh, 2013/08/11
- Re: parallel: This should not happen. You have found a bug., Ole Tange, 2013/08/12
- Re: parallel: This should not happen. You have found a bug., Linda Walsh, 2013/08/13
- Re: parallel: This should not happen. You have found a bug., Ole Tange, 2013/08/13
- Re: parallel: This should not happen. You have found a bug., Linda Walsh, 2013/08/13
- Re: parallel: This should not happen. You have found a bug., Ole Tange, 2013/08/13
- Re: parallel: This should not happen. You have found a bug., Linda Walsh, 2013/08/15
Re: parallel: This should not happen. You have found a bug., Linda Walsh, 2013/08/11