bug#27284: Memory leak in 'guix pull' or 'make' in guix source

[Ludovic Courtès]

Hi!

Ricardo Wurmus <address@hidden> skribis:

>> In the meantime, our best workaround to reduce memory consumption is… to
>> split large files into smaller ones.  Per M-x guix-locations, the
>> candidates are:
>>
>>   gnu/packages/python.scm                                   986
>>   gnu/packages/perl.scm                                     401
>>   gnu/packages/haskell.scm                                  348
>
> […]
>
>> I think we could focus on the first two or three files.  FTR, compiling
>> bioinformatics.scm peaks at ~500 MiB resident on x86_64.
>>
>> Ricardo, WDYT?
>
> I was hoping we could avoid this, but whatever: let’s do this :)

Yeah, me too.  The problem we have is that Guix is hardly releasable in
its current state because on systems with 1 GiB of memory you can’t
upgrade, and I think that’s unacceptable.

So what are the options?  If we get a bug-fix for Guile’s compiler
today, does it help?  If we graft it then we can deliver it without
having to wait for a Guile release, which helps a bit?

I think it’s all about time: we could wait (and hack!) some more, and
solve the root problem.  This is the best long-term course of action,
but at the same time it delays the Guix release.

>> If we do this, do we split arbitrarily?  Like the second half of
>> python.scm goes to python-cont.scm (provided there are no cross
>> top-level references)?  Or do you have a better idea?
>
> Ultimately, I’d like to move packages to locations where they could
> permanently live, but that would probably take longer.
>
> Would it make sense to move all the python2-* packages to a new module?

I’m not sure we can do this, because that may lead to top-level
references across these two modules, which is not OK.

> This would make the split rather simple and users wouldn’t have to
> remember which of their packages ended up in which half of the modules.
> It also means that we probably won’t have to mess with the copyright
> headers.
>
> For perl.scm I have no good ideas.  Let’s split it up at an arbitrary
> point.
>
> For haskell.scm I’d begin by moving the following packages away:
>
> - check.scm: ghc-tasty*, ghc-quickcheck*, ghc-test*, ghc-hunit*, hspec*,
>   ghc-hspec*, …
>
> - web.scm: ghc-tagsoup, ghc-cookie, ghc-http*, ghc-wai*, ghc-json,
>   ghc-warp*, ghc-multipart, ghc-aeson*
>
> - crypto.scm: ghc-tf-random, ghc-digest, ghc-cryptonite, ghc-x509*,
>   ghc-asn1*, ghc-pem, ghc-cryptohash*, ghc-entropy, ghc-crypto-api*,
>   ghc-puremd5
>
> - tls.scm: ghc-tls
>
> Maybe that’s enough already.
>
> Does that seem like a good idea?

It does.  Actually, we could do similarly for Perl and Python:
python-web, python-check, python-crypto, etc.

WDYT?

Thanks,
Ludo’.