Re: GNU Parallel seems to drop

[Ole Tange]

On Tue, Sep 25, 2012 at 6:42 AM, Dirk Eddelbuettel <edd@debian.org> wrote:
>
> Here is more concrete example:

You have not given enough information to generate a big inputfile, so
I cannot reproduce your test.

Based on your explanation it sounds as if the awk script opens files A, B and C.

If 2 awk scripts both open A, B and C then the last one wins and all
data written by the first one is lost.

One way to solve that is to instead have the first invocation open A1,
B1 and C1 while the second writes to A2, B2 and C2. You can use {#} or
$PARALLEL_SEQ for that by writing to A{#} or A$PARALLEL_SEQ.

Maybe you were under the impression that GNU Parallel would cache all
output (even data written to files) and write that atomically in the
end, but that is only the case for STDERR and STDOUT.


/Ole