[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: parallel: This should not happen. You have found a bug.
From: |
Linda Walsh |
Subject: |
Re: parallel: This should not happen. You have found a bug. |
Date: |
Sun, 11 Aug 2013 16:05:17 -0700 |
User-agent: |
Thunderbird |
When run in --semaphore mode GNU Parallel uses hard links to create a
counting semaphore. If your filesystem does not support hard links the
behaviour is undefined - and that is likely what you see.
NTFS supports hard links. However, file accesses are infinitely slower
than
semaphores in memory.
Ideally, the program could test to see if the system has
semaphores (they work on cygwin and linux and any posix compatible
using the POSIX::RT::Semaphore package.
I wrote the below blob to make it slightly easier to call:
(from the current program I am working on, so I just
copied it from the edit buffer...
)
############################################################
{ package POSIX::Semaphores; #{{{
use POSIX::RT::Semaphore;
use mem; #on CPAN
our (@EXPORT, @EXPORT_OK, @ISA);
sub sem_open ($;$$$) { POSIX::RT::Semaphore->open(@_) }
sub sem_close($) { $_[0]->close }
sub sem_unlink($) { POSIX::RT::Semaphore::unlink($_[0]) }
sub sem_post($) { $_[0]->post }
sub sem_wait($) { $_[0]->wait }
sub sem_getvalue($) { $_[0]->getvalue }
sub sem_timedwait($$) { $_[0]->timedwait }
sub sem_trywait($) { $_[0]->trywait }
sub sem_destroy($) { $_[0]->destroy }
sub sem_name($) { $_[0]->name }
use mem (@EXPORT = qw(sem_open sem_close sem_unlink
sem_post sem_wait
sem_getvalue sem_name ),
@EXPORT_OK = qw(sem_timedwait sem_trywait
sem_destroy ),
@ISA = qw(Exporter));
use Exporter;
}
------------------
Just made them easier to call but other than that there's not much to
the above.
What OS is Perl running on and what file system is /home/law/.parallel
hosted on?
CYGWIN_NT-6.1-WOW64 / NTFS
(thats 32bit cygwin on 64-bit win7-sp1)...
I have no idea whether that configuration supports hard links or not.
It does -- it had to support the POSIX subsystem which required it.
Just that NTFS file accesses are slow.... 20 minutes for ~400 in
parallel using NTFS hardlinks
vs. <2 seconds for ~200 for the xargs...
Using real semaphores or something else in memory is worth it with that type
of performance discrepancy...
- parallel: This should not happen. You have found a bug., Linda Walsh, 2013/08/10
- Re: parallel: This should not happen. You have found a bug., Ole Tange, 2013/08/11
- Re: parallel: This should not happen. You have found a bug., Linda Walsh, 2013/08/11
- Re: parallel: This should not happen. You have found a bug., Ole Tange, 2013/08/11
- Re: parallel: This should not happen. You have found a bug.,
Linda Walsh <=
- Re: parallel: This should not happen. You have found a bug., Ole Tange, 2013/08/12
- Re: parallel: This should not happen. You have found a bug., Linda Walsh, 2013/08/13
- Re: parallel: This should not happen. You have found a bug., Ole Tange, 2013/08/13
- Re: parallel: This should not happen. You have found a bug., Linda Walsh, 2013/08/13
- Re: parallel: This should not happen. You have found a bug., Ole Tange, 2013/08/13
- Re: parallel: This should not happen. You have found a bug., Linda Walsh, 2013/08/15
Re: parallel: This should not happen. You have found a bug., Linda Walsh, 2013/08/11