Re: Combining --pipe and --shebang options

[Michel Samia]

On 23.11.2012 23:07, Ole Tange wrote:
> On Mon, Nov 19, 2012 at 8:26 PM, Ole Tange <tange@it-pol.dk> wrote:
> > On Sun, Nov 18, 2012 at 3:13 PM, Ole Tange <ole@tange.dk> wrote:
> > > On Fri, Nov 16, 2012 at 2:13 PM, Michel Samia
> > > <michel.samia@firma.seznam.cz> wrote:
> > >
> > > > is it possible to combine --shebang and --pipe options?
> :
> > Your idea would be useful for any script (Shell, Perl, Python) which
> > can either process only one file or process stuff from stdin. Using
> > GNU Parallel your script can suddenly process many files/blocks of
> > data and in parallel.
> :
> > So it could be something like:
> >
> > #!/usr/bin/parallel --shebang-program --pipe -k -j24 /usr/bin/python
> >
> > (Please come up with a better name than  --shebang-program)
> It is now named --shebang-wrap since it wraps the shebang line.
>
> > This should accept data on stdin which should be chunked and passed to
> > the python program. So the program will be called like:
> >
> >    cat foo bar | my_program
> >
> > or:
> >
> >    my_program foo bar
> >
> > Without the --pipe:
> >
> > #!/usr/bin/parallel --shebang-program -k -j24 /usr/bin/python
> >
> > should work like:
> >
> >    parallel -k -j24 /usr/bin/python my_program {}
> >
> > So:
> >
> >    my_program foo bar
> >    (echo foo; echo bar) | my_program
> >
> > should do the same as:
> >
> >    parallel -k -j24 /usr/bin/python my_program {} ::: foo bar
> These all work now (--shebang-wrap instead of --shebang-program). But
> please test them. Also if you come up with cool examples please post 
> them.
Thank you very much for the new functionality. Unfortunately I wasn't 
able to reproduce the expected behaviour of this script simple.py

#!/usr/local/bin/parallel --shebang-wrap --pipe -k -j24 /usr/bin/python
# should work like cat
import sys

for line in sys.stdin:
    sys.stdout.write(line)

When I run

    echo 'bla' | ./simple.py

sometimes it hangs and my mouse cursor changes its shape to s cross and 
sometimes it prints error messages:

michel@bellatrix:/tmp$ echo bla | ./simple.py
./simple2.py: line 6: syntax error near unexpected token `sys.stdout.write'
./simple2.py: line 6: `    sys.stdout.write(line)'

which probably means that the script is run by bash, not by Python.

some info about my system if it helps:

michel@bellatrix:/tmp$ uname -a
Linux bellatrix.dev 2.6.18-openvz #1 SMP Wed Feb 15 12:34:09 CET 2012 
x86_64 GNU/Linux
michel@bellatrix:/tmp$ cat /etc/issue
Debian GNU/Linux 6.0 \n \l

michel@bellatrix:/tmp$

I tried this with the yesterday's version of the git repository. I also 
tried to run parallel-local21.sh in testsuite/tests-to-run directory and 
the output seemed to be incorrect:

./parallel-local21.sh
### Test basic --shebang-wrap
Warning: unknown mime-type for "Shebang from perl with args @ARGV\n" -- 
using "application/octet-stream"
Error: no such file "Shebang from perl with args @ARGV\n"
### Same as
Shebang from perl with args arg1
Shebang from perl with args arg2
### stdin
Warning: unknown mime-type for "Shebang from perl with args @ARGV\n" -- 
using "application/octet-stream"
Error: no such file "Shebang from perl with args @ARGV\n"
### Same as
Shebang from perl with args arg1
Shebang from perl with args arg2
### Test --shebang-wrap with parser options
Warning: unknown mime-type for "Shebang from perl with args @ARGV\n" -- 
using "application/octet-stream"
Error: no such file "Shebang from perl with args @ARGV\n"
### Same as
Shebang from perl with args
1
Shebang from perl with args
2
Shebang from perl with args
4
Shebang from perl with args
5
### stdin
Warning: unknown mime-type for "Shebang from perl with args @ARGV\n" -- 
using "application/octet-stream"
Error: no such file "Shebang from perl with args @ARGV\n"
### Same as
Shebang from perl with args
1
Shebang from perl with args
2
Shebang from perl with args
4
Shebang from perl with args
5
### Test --shebang-wrap --pipe with parser options
Suboptimal
Warning: unknown mime-type for "Shebang from perl with args @ARGV\n" -- 
using "application/octet-stream"
Error: no such file "Shebang from perl with args @ARGV\n"
Optimal
Warning: unknown mime-type for "Shebang from perl with args @ARGV\n" -- 
using "application/octet-stream"
Error: no such file "Shebang from perl with args @ARGV\n"
### Same as
Shebang from perl with args
1
Shebang from perl with args
2
Shebang from perl with args
4
Shebang from perl with args
5

 Maybe I have to run it another way (from different directory)?
> > We should allow for putting options on the command interpreter. E.g:
> >
> > #!/usr/bin/parallel --shebang-program -k -j24 /usr/bin/perl -p
> >
> > Also without the --pipe the {} should work as expected:
> >
> > #!/usr/bin/parallel --shebang-program -k -j24 /usr/bin/perl -p {}
> > {.}.out > {.}.log
> This, however, does not work. The above should run:
>
> /usr/bin/parallel -k -j24 /usr/bin/perl -p my_program "{} {.}.out > 
> {.}.log"
>
> The problem is that the script file name needs to be put into the
> middle of the command line (above it is put after -p). As I do not see
> a way to find right place to put in the script file name, I do not see
> a way to get that part working.
>
> /Ole
Now I have to continue with my work, but this weekend I will have more 
time to help you resolve this issue.
Michel