Re: PMA leak in functions

[arnold]

Hi.

Thank you for the report. We appreciate your taking the time
to stress the persistent memory use case and report issues.

I investigated this a bit yesterday and today.  I thought I'd found the
cause of the problem, but was mistaken.  I currently do not know exactly
where the memory is being lost from, and to be honest, I'm not sure I
can figure it out.

You're seeing a consequence of the approach we took, which was to add
persitence to a program that was never designed for it.  Things mostly
work, but there are (and always will be) leaks in the backing store file,
as well as corner cases that don't work.  I will be adding something about
this in the manual.

I will try to continue working on this, but I am not sure it's fixable.

Thanks,

Arnold

"KochWilfried@t-online.de" <KochWilfried@t-online.de> wrote:

>  
> -----Original-Nachricht-----
> Betreff: PMA leak in functions
> Datum: 2023-04-23T08:53:14+0200
> Von: "KochWilfried@t-online.de" <KochWilfried@t-online.de>
> An: "bug-gawk@gnu.org" <bug-gawk@gnu.org>
>  
>  
>  
>  
> Configuration Information [Automatically generated, do not change]:
> Machine: x86_64
> OS: linux-gnu
> Compiler: gcc
> Compilation CFLAGS: -g -O2 -DNDEBUG
> uname output: Linux katan 6.2.0-20-generic #20-Ubuntu SMP PREEMPT_DYNAMIC 
> Thu Apr 6 07:48:48 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
> Machine Type: x86_64-pc-linux-gnu
> Gawk Version: 5.2.1c
> Attestation 1:
> I have read https://www.gnu.org/software/gawk/manual/html_node/Bugs.html.
> Yes / No [ Choose one. If "No", then why haven't you? ]
> Attestation 2:
> I have not modified the sources before building gawk.
> True / False
> [ Choose one. If "False", then please explain what you did and why. ]
> Description:
> 5.2.1c doesn't have PMA leaks anymore on deleted arrays.
> Thank you for that.
> But there are PMA leaks on functions or their call stack.
>
> [Detailed description of the problem, suggestion, or complaint.]
> the bash script below ##### creates the awk program fun.awk with 1 Mln 
> functions
> that call each other in a row and increase the parameter i by 1.
> The expected output is 1000001.
> I non PMA-Mode ist cyles 50 times with out problems.
> gawk -f fun.awk
> runs about 5 minutes, mostly busy with parsing.
>  
>  
> In PMA Mode the 8GB heap file is exhausted after 15 runs.
> rm heap.pma;truncate -s 8192000000 heap.pma;(export 
> GAWK_PERSIST_FILE=heap.pma && /usr/bin/time gawk -f fun.awk ); for i in 
> $(seq 1 1000 ) ; do (export GAWK_PERSIST_FILE=heap.pma && echo $i && 
> /usr/bin/time gawk 'BEGIN{print @fp_fun1000(1)}') ; rc=$? ; if [[ $rc != 0 
> ]] ; then break ; fi ;done
> 57.29user 1.94system 0:59.31elapsed 99%CPU (0avgtext+0avgdata 
> 2102272maxresident)k
> 0inputs+6418328outputs (16427major+786025minor)pagefaults 0swaps
> 1
> 1000001
> 3.09user 1.54system 0:04.64elapsed 99%CPU (0avgtext+0avgdata 
> 2518272maxresident)k
> 0inputs+2494448outputs (5758major+535259minor)pagefaults 0swaps
> 2
> 1000001
> 2.02user 0.69system 0:02.72elapsed 99%CPU (0avgtext+0avgdata 
> 2518272maxresident)k
> 0inputs+2480464outputs (3067major+443710minor)pagefaults 0swaps
> 3
> 1000001
> 2.14user 0.67system 0:02.81elapsed 100%CPU (0avgtext+0avgdata 
> 2528256maxresident)k
> 0inputs+782832outputs (3066major+446319minor)pagefaults 0swaps
> ...........
> 14
> 1000001
> 2.00user 0.78system 0:02.79elapsed 100%CPU (0avgtext+0avgdata 
> 2529280maxresident)k
> 0inputs+2415624outputs (3065major+446339minor)pagefaults 0swaps
> 15
> gawk: fatal: node.c:1075:more_blocks: freep: cannot allocate 4800 bytes of 
> memory: Invalid or incomplete multibyte or wide character
> Command exited with non-zero status 2
> 0.78user 0.39system 0:01.18elapsed 99%CPU (0avgtext+0avgdata 
> 1897216maxresident)k
> 0inputs+141944outputs (559major+285563minor)pagefaults 0swaps
>  
> #######
> export MX=1000000
> awk '
> BEGIN {
> MX=ENVIRON["MX"]
> fx=length(MX)
> for(i=MX;i>0;i--){
> print "function fun"i"(i){" > "fun.awk"
> print " i=i+1" > "fun.awk"
> if(i!=MX) {
> print " return @fp_fun"i+1"(i)" > "fun.awk"
> }else{
> print " return i" > "fun.awk"
> }
> print "}" >"fun.awk"
> }
> print "BEGIN{" >"fun.awk"
> for(i=MX;i>0;i--){
> print "fp_fun"i"=\"fun"i"\"" > "fun.awk"
> }
> print "if(\"GAWK_PERSIST_FILE\" in ENVIRON){" > "fun.awk"
> print "}else{" > "fun.awk"
> print " for(i=1;i<50;i++){" > "fun.awk"
> print " print ""@fp_fun"1"(1)" > "fun.awk"
> print " }" > "fun.awk"
> print "}" > "fun.awk"
> print "}" > "fun.awk"
> exit(0)
> }
> '
> #######
>  
> Repeat-By:
> [Describe the sequence of events that causes the problem to occur.]
> Fix:
> [Description of how to fix the problem. If you don't know a
> fix for the problem, don't include this section.]
>  
>