[sr #110322] Anonymous HTTPS cloning of emacs git repository has been failing

[Bob Proulx]

Update of sr #110322 (project administration):

                  Status:               Confirmed => Done                   
             Assigned to:                    None => rwp                    
             Open/Closed:                    Open => Closed                 

    _______________________________________________________

Follow-up Comment #2:

See also the mailing list discussion.

*
https://lists.gnu.org/archive/html/savannah-hackers-public/2020-09/msg00004.html
*
https://lists.gnu.org/archive/html/savannah-hackers-public/2020-10/msg00000.html

I think I have this problem mostly resolved right now.  I can git
clone the repository now when previously I could not.

    rwp@mgt0:~/src/emacs-stuff$ time git clone
https://git.savannah.gnu.org/git/emacs.git
    Cloning into 'emacs'...
    remote: Counting objects: 903373, done.
    remote: Compressing objects: 100% (160456/160456), done.
    remote: Total 903373 (delta 742468), reused 902659 (delta 741755)
    Receiving objects: 100% (903373/903373), 293.80 MiB | 15.14 MiB/s, done.
    Resolving deltas: 100% (742468/742468), done.
    Checking connectivity... done.
    Checking out files: 100% (4126/4126), done.

    real    5m5.358s
    user    3m36.801s
    sys     0m17.380s

What I did to debug...

I made an rsync copy of the emacs.git repository (not a recommended
method, not atomic, better to use a git method that will copy things
the correct way, but fine for my throwaway test, where I needed a bit
exact copy) to my own server which has the same git-http-backend
configuration.  And also copied grep.git too for a comparison.  And
had exactly the same reproducible behavior there!  Attempting to clone
emacs times out but cloning grep works perfectly.  Exact same problem
on my testing copy.

Therefore the problem was definitely something in the emacs.git
repository.

I performed some experiments on my copy offline.  It seemed that for
whatever reason git-pack-objects was always a heavy cpu use process at
the start of the the clone.  Interesting.  But there was a
difference between the working and non-working protocols.

    git pack-objects --revs --thin --stdout --progress --delta-base-offset

By my stopwatch and thumb this ran for 1m22s at the start of
every git clone.  Using git-http-backend this would timeout due to
some 60s timeout that I have not yet identified.  However using the
git-daemon only 6 seconds would pass before seeing progress feedback
to the terminal.  The two programs handle this differently,
apparently, and git-http-backend is holding all output until after git
pack-objects completes.

Background: Once a month on the 4th all git repositories are garbage
collecte with "git gc".  Today is the 1st.  So it has been almost the
maximum length of time between explicit git gc runs.  I decided to
manually run this now.

    root@vcs0:/net/vcs/git/emacs.git# git gc
    Counting objects: 900046, done.
    Delta compression using up to 2 threads.
    Compressing objects: 100% (161844/161844), done.
    Writing objects: 100% (900046/900046), done.
    Total 900046 (delta 739533), reused 896905 (delta 737041)
    Removing duplicate objects: 100% (256/256), done.
    Checking connectivity: 903487, done.

Before the storage size was 411MB and git gc reduced the size ot 333MB
after the run.  After doing the git gc then the git pack-objects takes
only 33 seconds to run at the start of the clone.  So only 33 seconds
before there is interaction.

And so it seems that for whatever reason when using the
git-http-backend that this feedback prior to the compression step
would take longer than 60 seconds, would take 1m22s, and would hit the
60 second timeout, and fail returning the HTTP 504 Timeout error.  Now
that git gc has been recently run it takes only 33 seconds, is below
the 60 second timeout, and therefore is allowed to continue.

At this time I can git clone the repository.

Since it appears that for emacs at least the once a month git gc is
not enough to keep the incremental git pack-objects happy I have
increased that to four times a month, about weekly.  That should keep
on top of this problem for the indefinite future.  I'll try to figure
out what the 60s timeout is since it is not proxy_read_timeout but
some other timeout.  Because if this gets really worse then we will
need to increase that value.


    _______________________________________________________

Reply to this item at:

  <https://savannah.nongnu.org/support/?110322>

_______________________________________________
  Message sent via Savannah
  https://savannah.nongnu.org/