[GNUnet-SVN] [gnunet] branch master updated (c89e2dcd8 -> 7fef1456b)

[gnunet]

This is an automated email from the git hooks/post-receive script.

grothoff pushed a change to branch master
in repository gnunet.

    from c89e2dcd8 batch NAMESTORE operation also in zonemaster
     new bdf8e76fc remove remarks about obsolete gnunet-gns-import
     new 7fef1456b enable more parallelism with DHT queue, but limit to 1000 
entries, then kill hard

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 doc/documentation/chapters/installation.texi | 47 ++++++++---------------
 doc/man/gnunet-namestore-fcfsd.1             |  2 +-
 src/dht/dht_api.c                            |  4 +-
 src/util/mq.c                                |  4 +-
 src/zonemaster/gnunet-service-zonemaster.c   | 56 +++++++++++++++++++++++++---
 5 files changed, 70 insertions(+), 43 deletions(-)

diff --git a/doc/documentation/chapters/installation.texi 
b/doc/documentation/chapters/installation.texi
index eeb14473b..3a76fb162 100644
--- a/doc/documentation/chapters/installation.texi
+++ b/doc/documentation/chapters/installation.texi
@@ -294,7 +294,7 @@ If you want to be able to enable DEBUG-level log messages, 
add
 @command{./configure} command.
 @code{DEBUG}-level log messages are in English only and
 should only be useful for developers (or for filing
-really detailed bug reports). 
+really detailed bug reports).
 
 Finally, you probably want to compile @command{gnunet-gtk}, which
 includes @command{gnunet-setup} (a graphical tool for
@@ -389,10 +389,9 @@ For those services, systems with more than one user may 
require each user
 to specify a different port number in their personal configuration file.
 
 Finally, the user should perform the basic initial setup for the GNU Name
-System (GNS). This is done by running two commands:
+System (GNS) certificate authority. This is done by running:
 
 @example
-$ gnunet-gns-import.sh
 $ gnunet-gns-proxy-setup-ca
 @end example
 
@@ -420,7 +419,7 @@ hosts: files gns [NOTFOUND=return] mdns4_minimal \
 
 @c FIXME: Document new behavior.
 You might want to make sure that @file{/lib/libnss_gns.so.2} exists on
-your system, it should have been created during the installation. 
+your system, it should have been created during the installation.
 
 @node Build instructions for Ubuntu 12.04 using Git
 @section Build instructions for Ubuntu 12.04 using Git
@@ -802,7 +801,7 @@ $ gpg --keyserver pgp.mit.edu --recv-keys 
A88C8ADD129828D7EAC02E52E22F9BBFEE3485
 @end example
 
 @noindent
-and rerun the gpg --verify command. 
+and rerun the gpg --verify command.
 
 @example
 $ tar xvf gnurl-7.57.0.tar.xz
@@ -1418,7 +1417,7 @@ as a normal user.
 We begin by installing a few Debian packages from stable:
 
 @example
-# apt-get install gcc make python-zbar libltdl-dev libsqlite3-dev \ 
+# apt-get install gcc make python-zbar libltdl-dev libsqlite3-dev \
 libunistring-dev libopus-dev libpulse-dev openssl libglpk-dev texlive \
 libidn11-dev libmysqlclient-dev libpq-dev libarchive-dev libbz2-dev \
 libflac-dev libgif-dev libglib2.0-dev libgtk-3-dev libmpeg2-4-dev \
@@ -1633,7 +1632,7 @@ have to compile it from source:
 @item Unzip the downloaded source tarball using your favourite
 unzipper application In the MSYS shell
 
address@hidden change to the respective directory 
address@hidden change to the respective directory
 
 @item Configure glpk for "i686-pc-mingw32":
 
@@ -1781,7 +1780,7 @@ $ make install
 This document is a @b{DEPRECATED} installation guide for GNUnet on
 Windows.
 It will not work for recent GNUnet versions, but maybe it will be of
-some use if problems arise. 
+some use if problems arise.
 
 The Windows build uses a UNIX emulator for Windows,
 @uref{http://www.mingw.org/, MinGW}, to build the executable modules.
@@ -1799,7 +1798,7 @@ We regret any inconvenience, and if you have problems, 
please report them.
 * Installer::
 * Source::
 @end menu
-     
+
 @node Hardware and OS requirements
 @subsubsection Hardware and OS requirements
 
@@ -1878,7 +1877,7 @@ This batch file opens a shell which is used to invoke the 
build
 processes.
 MinGW's standard shell (@command{msys.bat}) is not suitable
 because it opens a separate console window.
-On Vista, @command{bash.bat} needs to be run as Administrator. 
+On Vista, @command{bash.bat} needs to be run as Administrator.
 
 @item
 Start @command{bash.sh} and rename
@@ -1948,7 +1947,7 @@ directory (@file{c:\mingw\mingw}).
 
 @item @strong{SQLite}@
 GNUnet uses the SQLite database to store data.
-Get the prebuilt binary from here and unpack it to your MinGW directory. 
+Get the prebuilt binary from here and unpack it to your MinGW directory.
 
 @item @strong{MySQL}@
 As an alternative to SQLite, GNUnet also supports MySQL.
@@ -1984,7 +1983,7 @@ dlltool --input-def ../include/libmySQL.def \
 --output-lib libmysqlclient.a -k
 @end example
 
address@hidden  Copy include\* to include\mysql\ 
address@hidden  Copy include\* to include\mysql\
 
 @item  Pass @code{--with-mysql=/c/mysql} to
 @command{./configure} and copy @file{libmysql.dll}
@@ -2082,7 +2081,7 @@ debug information and are quite large. To compile release 
versions
 (small and fast) set the variable @code{CFLAGS}:
 
 @example
-export CFLAGS='-O2 -march=pentium -fomit-frame-pointer' 
+export CFLAGS='-O2 -march=pentium -fomit-frame-pointer'
 ./configure --prefix=$HOME --with-extractor=$HOME
 @end example
 
@@ -2097,7 +2096,7 @@ located in @file{contrib\win} in the GNUnet source tree.
 @subsubsection Source
 
 @c FIXME: URL
-The sources of all dependencies are available here. 
+The sources of all dependencies are available here.
 
 @c @node Portable GNUnet
 @c @section Portable GNUnet
@@ -2372,7 +2371,7 @@ The hostlist client supports the following proxy types at 
the moment:
 @end itemize
 
 In addition authentication at the proxy with username and password can be
-configured. 
+configured.
 
 To configure proxy support for the hostlist client in the
 @command{gnunet-setup} tool, select the "hostlist" tab and select
@@ -2521,7 +2520,7 @@ inconsistencies. Some of the other databases do not 
support repair.
 @item Access mysql as root:
 
 @example
-$ mysql -u root -p 
+$ mysql -u root -p
 @end example
 
 @noindent
@@ -3324,19 +3323,6 @@ steps.
 First of all, GNS needs to be integrated with the operating system. Most
 of this section is about the operating system level integration.
 
-Additionally, each individual user who wants to use the system must also
-initialize their GNS zones. This can be done by running (after starting
-GNUnet)
-
address@hidden
-$ gnunet-gns-import.sh
address@hidden example
-
address@hidden
-after the local GNUnet peer has been started. Note that the namestore (in
-particular the namestore database backend) should not be reconfigured
-afterwards (as records are not automatically migrated between backends).
-
 The remainder of this chapter will detail the various methods for
 configuring the use of GNS with your operating system.
 
@@ -3528,7 +3514,7 @@ gnurl --version
 curl 7.56.0 (x86_64-unknown-linux-gnu) libcurl/7.56.0 \
 GnuTLS/3.5.13 zlib/1.2.11 libidn2/2.0.4
 Release-Date: 2017-10-08
-Protocols: http https 
+Protocols: http https
 Features: AsynchDNS IDN IPv6 Largefile NTLM SSL libz \
 TLS-SRP UnixSockets HTTPS-proxy
 @end example
@@ -4125,4 +4111,3 @@ Furthermore, 'make install' will silently fail to set the 
DNS binaries to
 be owned by group "gnunetdns" unless that group already exists (!).
 An alternative name for the "gnunetdns" group can be specified using the
 @code{--with-gnunetdns=GRPNAME} configure option.
-
diff --git a/doc/man/gnunet-namestore-fcfsd.1 b/doc/man/gnunet-namestore-fcfsd.1
index 3160910d4..8039e796f 100644
--- a/doc/man/gnunet-namestore-fcfsd.1
+++ b/doc/man/gnunet-namestore-fcfsd.1
@@ -13,7 +13,7 @@ Most users will not want to run an FCFS\-zone and thus will 
not need this progra
 
 \fBgnunet\-gns-fcfsd\fP runs a web server where users can register names to be 
mapped to their GNS zone.  Names are made available on a First Come First 
Served basis (hence fcfs).  Registered names do not expire.  The HTTP server is 
run on the port that is specified in the configuration file in section 
"[fcfsd]" under the name "HTTPPORT".  The key of the zone in which the names 
are registered must be specified under the name "ZONEKEY" in the same section.  
It is possible to manage gnunet [...]
 
-An FCFS\-zone is run at http://gnunet.org/fcfs/.  The respective zone key can 
be imported into an individual user's zone using "gnunet-gns-import.sh".  GNS 
users are encouraged to register their zone with the gnunet.org FCFS authority.
+An FCFS\-zone is run at http://gnunet.org/fcfs/.  GNS users are encouraged to 
register their zone with the gnunet.org FCFS authority.
 
 If you want to run your own FCFS registrar, you need to first create a 
pseudonym (using "gnunet\-identity \-C NAME"), and then assign it to be used 
for the "fcfsd" service using "gnunet\-identity \-e NAME \-s fcfsd".  After 
that, you can start the FCFSD service (possibly using gnunet\-arm).
 
diff --git a/src/dht/dht_api.c b/src/dht/dht_api.c
index 42ddc7b60..7a0771de0 100644
--- a/src/dht/dht_api.c
+++ b/src/dht/dht_api.c
@@ -1028,8 +1028,8 @@ GNUNET_DHT_put (struct GNUNET_DHT_Handle *handle,
   put_msg->expiration = GNUNET_TIME_absolute_hton (exp);
   put_msg->key = *key;
   GNUNET_memcpy (&put_msg[1],
-          data,
-          size);
+                 data,
+                 size);
   GNUNET_MQ_send (handle->mq,
                   env);
   return ph;
diff --git a/src/util/mq.c b/src/util/mq.c
index af700836c..0f9ad9a12 100644
--- a/src/util/mq.c
+++ b/src/util/mq.c
@@ -578,11 +578,9 @@ void
 GNUNET_MQ_set_handlers_closure (struct GNUNET_MQ_Handle *mq,
                                 void *handlers_cls)
 {
-  unsigned int i;
-
   if (NULL == mq->handlers)
     return;
-  for (i=0;NULL != mq->handlers[i].cb; i++)
+  for (unsigned int i=0;NULL != mq->handlers[i].cb; i++)
     mq->handlers[i].cls = handlers_cls;
 }
 
diff --git a/src/zonemaster/gnunet-service-zonemaster.c 
b/src/zonemaster/gnunet-service-zonemaster.c
index 5c3356784..b45ed576c 100644
--- a/src/zonemaster/gnunet-service-zonemaster.c
+++ b/src/zonemaster/gnunet-service-zonemaster.c
@@ -50,6 +50,11 @@
 #define NS_BLOCK_SIZE 100
 
 /**
+ * How many pending DHT operations do we allow at most?
+ */
+#define DHT_QUEUE_LIMIT 1000
+
+/**
  * The initial interval in milliseconds btween puts in
  * a zone iteration
  */
@@ -107,6 +112,11 @@ struct DhtPutActivity
    * Handle for the DHT PUT operation.
    */
   struct GNUNET_DHT_PutHandle *ph;
+
+  /**
+   * When was this PUT initiated?
+   */
+  struct GNUNET_TIME_Absolute start_date;
 };
 
 
@@ -161,6 +171,11 @@ static struct DhtPutActivity *it_head;
 static struct DhtPutActivity *it_tail;
 
 /**
+ * Number of entries in the DHT queue.
+ */
+static unsigned int dht_queue_length;
+
+/**
  * Useful for zone update for DHT put
  */
 static unsigned long long num_public_records;
@@ -265,6 +280,15 @@ shutdown_task (void *cls)
                                  ma);
     GNUNET_free (ma);
   }
+  while (NULL != (ma = it_head))
+  {
+    GNUNET_DHT_put_cancel (ma->ph);
+    GNUNET_CONTAINER_DLL_remove (it_head,
+                                 it_tail,
+                                 ma);
+    dht_queue_length--;
+    GNUNET_free (ma);
+  }
   if (NULL != statistics)
   {
     GNUNET_STATISTICS_destroy (statistics,
@@ -362,8 +386,6 @@ check_zone_dht_next ()
 
   if (0 != ns_iteration_left)
     return; /* current NAMESTORE iteration not yet done */
-  if (NULL != it_head)
-    return; /* waiting on DHT */
   delay = GNUNET_TIME_relative_subtract (next_put_interval,
                                          sub_delta);
   /* We delay *once* per #NS_BLOCK_SIZE, so we need to multiply the
@@ -472,6 +494,10 @@ update_velocity ()
     }
   }
   GNUNET_STATISTICS_set (statistics,
+                         "# size of the DHT queue",
+                         dht_queue_length,
+                         GNUNET_NO);
+  GNUNET_STATISTICS_set (statistics,
                          "% speed increase needed for target velocity",
                          pct,
                          GNUNET_NO);
@@ -498,6 +524,7 @@ dht_put_continuation (void *cls,
   GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
               "PUT complete (%s)\n",
               (GNUNET_OK == success) ? "success" : "failure");
+  dht_queue_length--;
   GNUNET_CONTAINER_DLL_remove (it_head,
                                it_tail,
                                ma);
@@ -508,7 +535,6 @@ dht_put_continuation (void *cls,
     if (0 == put_cnt % DELTA_INTERVAL)
       update_velocity ();
   }
-  check_zone_dht_next ();
 }
 
 
@@ -766,6 +792,7 @@ put_gns_record (void *cls,
   GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
               "Starting DHT PUT\n");
   ma = GNUNET_new (struct DhtPutActivity);
+  ma->start_date = GNUNET_TIME_absolute_get ();
   ma->ph = perform_dht_put (key,
                             label,
                             rd_public,
@@ -780,9 +807,24 @@ put_gns_record (void *cls,
     check_zone_dht_next ();
     return;
   }
-  GNUNET_CONTAINER_DLL_insert (it_head,
-                               it_tail,
-                               ma);
+  dht_queue_length++;
+  GNUNET_CONTAINER_DLL_insert_tail (it_head,
+                                    it_tail,
+                                    ma);
+  if (dht_queue_length > DHT_QUEUE_LIMIT)
+  {
+    ma = it_head;
+    GNUNET_CONTAINER_DLL_remove (it_head,
+                                 it_tail,
+                                 ma);
+    GNUNET_DHT_put_cancel (ma->ph);
+    dht_queue_length--;
+    GNUNET_log (GNUNET_ERROR_TYPE_WARNING,
+                "DHT PUT unconfirmed after %s, aborting PUT\n",
+                GNUNET_STRINGS_relative_time_to_string 
(GNUNET_TIME_absolute_get_duration (ma->start_date),
+                                                        GNUNET_YES));
+    GNUNET_free (ma);
+  }
 }
 
 
@@ -815,6 +857,7 @@ publish_zone_dht_start (void *cls)
                                              NULL,
                                              &zone_iteration_finished,
                                              NULL);
+  GNUNET_assert (NULL != namestore_iter);
 }
 
 
@@ -855,6 +898,7 @@ handle_monitor_event (void *cls,
   if (0 == rd_public_count)
     return; /* nothing to do */
   ma = GNUNET_new (struct DhtPutActivity);
+  ma->start_date = GNUNET_TIME_absolute_get ();
   ma->ph = perform_dht_put (zone,
                             label,
                             rd,

-- 
To stop receiving notification emails like this one, please contact
address@hidden