Re: Parallel input from growing file?

[xmoon 2000]

On 14 December 2014 at 14:08, Hans Schou <chlor@schou.dk> wrote:
> Start reading this chapter in the man page:
>   EXAMPLE: GNU Parallel as queue system/batch manager
>
> You are not the first who (mis-)use Gnu Parallel as a queue manager.
>
> /hans
>
> 2014-12-14 12:01 GMT+01:00 xmoon 2000 <xmoon2000@googlemail.com>:
>>
>> Hi,
>>
>> Currently I read jobs from a file on my c drive that gets written to
>> by other scripts, and my master script runs each job found in
>> background with &. I want to use parallel instead of my own script.
>>
>> I would like parallel to monitor this file and run any jobs in it with
>> as many cores as are available, or wait until new jobs are appended to
>> the file. Running 24/7.
>>
>> What is the syntax for doing this?
>>
>> My jobFile looks like this:
>>
>>
>> /myApp prm1 parm2 parm2
>> /myApp parm1 parm2
>> /myApp parm1 parm2 parm3 parm4
>>
>> I could do:
>>
>> tail -f jobFile  | parallel
>>
>> Is this the best approach?
>>
>


Hi Hans,

Yes I had read that section before. However, I assume this is a common
request and wondered if any other functionality had been created that
would allow a jobQueue to be consumed? i.e. the jobs in the jobQueue
are deleted as they are processed?

Also a run "run up to x jobs starting now - would be very useful"