I'm using GNU Parallel 20120822, and it is very useful to me when sending large lists to bash scripts that just take $1 command line parameter as the input. e.g.,
(I'm using j0 because my scripts call up programs such as host or wget, which I hope benefit from being run many times in parallel.) But I have large url-lists of around 250mb that I need to pipe. I've tried changing my scripts so they read from stdin using bash 'read'. e.g.,
It works for smaller files, but as soon as I try it on the bigger lists I always get this type of error:
'10879 broken pipeĀ cat url-list |'
I'm assuming this error message is from cat. When I run the above command on smaller lists it appears to work, but perhaps there are special characters -- that I can't notice -- appearing in bigger lists that get interpreted by the shell as a command and break the pipe? I've tried scanning for such characters, but I don't know what to look for; and the files contain well over 370000 lines, so it is like looking for a needle in a haystack.