vendor/zasync/1.1

changeset 0:739837ef4cee 1.1

Importing initial vendor 1.1 drop
author rspivak
date Tue, 10 Jan 2006 01:50:35 +0000
parents
children b6ca95992327
files CHANGES.txt COPYRIGHT.txt CREDITS.txt CVS/Entries CVS/Entries.Log CVS/Repository CVS/Root CVS/Tag LICENSE.txt README.txt __init__.py bforests.py bforests.txt bin/CVS/Entries bin/CVS/Repository bin/CVS/Root bin/CVS/Tag bin/runzasync bin/runzasync.bat bin/zasync bin/zasyncctl.py bucketqueue.py client/CVS/Entries client/CVS/Entries.Log client/CVS/Repository client/CVS/Root client/CVS/Tag client/README.txt client/zasync/CVS/Entries client/zasync/CVS/Repository client/zasync/CVS/Root client/zasync/CVS/Tag client/zasync/__init__.py client/zasync/client.py client/zasync/config.py client/zasync/example_start_script client/zasync/plugins.py client/zasync/plugins.txt client/zasync/schema.xml client/zasync/schema27.xml client/zasync/tests.txt client/zasync/zasync.conf interfaces.py manager.py manager.txt permissions.py tests/CVS/Entries tests/CVS/Repository tests/CVS/Root tests/CVS/Tag tests/__init__.py tests/test_bforests.py tests/test_bucketqueue.py tests/test_manager.py tests/zopetestutils.py www/CVS/Entries www/CVS/Repository www/CVS/Root www/CVS/Tag www/analyzeCalls.zpt www/constructAsynchronousCallManagerForm.zpt www/controlAsynchronousCallManagerForm.zpt www/daliclock.gif
diffstat 63 files changed, 6712 insertions(+), 0 deletions(-) [+]
line diff
     1.1 new file mode 100644
     1.2 --- /dev/null
     1.3 +++ b/CHANGES.txt
     1.4 @@ -0,0 +1,148 @@
     1.5 +version 1.1.0 (tag zasync_1_1_0)
     1.6 +
     1.7 +- Add Windows files and rc files from Arno Gross into bin directory.
     1.8 +
     1.9 +- Put more information on diagnostic call analysis page, thanks to Arno Gross.
    1.10 +
    1.11 +- Make zasync work in 2.7 and 2.8.  Note that making it run in 2.7 now requires
    1.12 +  an extra argument to run: see the example files in the bin directory.
    1.13 +  Briefly, here are two example calls, from one of my test installations:
    1.14 +  
    1.15 +  For 2.7:
    1.16 +  ~/zope2/bin/python -c \
    1.17 +  "import zasync; zasync.run('/home/gary/zope2/instance277/etc/zasync.conf', 2.7)"
    1.18 +  
    1.19 +  For 2.8:
    1.20 +  ~/zope2/bin/python -c \
    1.21 +  "import zasync; zasync.run('/home/gary/zope2/instance/etc/zasync.conf')"
    1.22 +
    1.23 +- Add some example text about the zope_exec plugin in plugins.txt.
    1.24 +
    1.25 +version 1.0.7 (tag zasync-1_0_7)
    1.26 +
    1.27 +- Some small changes (like using get_transaction().begin()) driven more by
    1.28 +  superstition and risk reduction than anything else.
    1.29 +
    1.30 +- Make manager include lines that make Python datetime objects acceptable to
    1.31 +  use in restricted python.  This should make it easier for a simple install.
    1.32 +
    1.33 +- Fix edge-case bug: if you have a call manager with new calls that have
    1.34 +  expired, the client would not be working with wrapped deferred objects so
    1.35 +  aq_... attributes would be missing and the new call wouldn't be tossed, and
    1.36 +  new (non-expired) calls wouldn't be processed.
    1.37 +
    1.38 +version 1.0.6 (tag zasync-1_0_6)
    1.39 +
    1.40 +- More fixes for the code path of handling error-raising reprs (grr).
    1.41 +
    1.42 +version 1.0.5 (tag zasync-1_0_5)
    1.43 +
    1.44 +- cleanFailure using customized code because __repr__ can fail in Zope. :-/
    1.45 +
    1.46 +- Fix nasty small bug in client.py that could make zasync effectively into a
    1.47 +  zombie process.  :-(
    1.48 +
    1.49 +version 1.0.4 (tag zasync-1_0_4)
    1.50 +
    1.51 +- Add in ability to pass in form variables to request.
    1.52 +
    1.53 +- Fix bug in rendering error message in client config.
    1.54 +
    1.55 +version 1.0.3 (tag zasync-1_0_3)
    1.56 +
    1.57 +- fix retry code in client to call makeCall correctly
    1.58 +
    1.59 +version 1.0.2 (tag zasync-1_0_2)
    1.60 +
    1.61 +- Try for more and earlier transaction aborts.
    1.62 +
    1.63 +- Reinstate lost diagnostic call analysis page.
    1.64 +
    1.65 +version 1.0.1 (tag zasync-1_0_1)
    1.66 +
    1.67 +- A new diagnostic call analysis page for the ZMI call manager.  Unfortunately
    1.68 +  it was not exposed in the release. :-/
    1.69 +
    1.70 +- Attempt to improve the README file to give people more of a chance to get
    1.71 +  this set up and going.  
    1.72 +
    1.73 +- A clean up of the two example configuration scripts to remove references to
    1.74 +  Zope 3, which really probably just confused matters.
    1.75 +
    1.76 +- Manager needs to fix up local roles to get the owner info in.
    1.77 +
    1.78 +version 1.0 (tag zasync-1_0)
    1.79 +
    1.80 +- Add an overview tab that includes information on the plugins and allows you
    1.81 +  to ping the zasync client to see if it is active.
    1.82 +
    1.83 +- Make the verbose traceback optional, defaulting to False.  Incorporate the
    1.84 +  new verbose traceback option in logging output
    1.85 +
    1.86 +- Make the normal Zope event log work.  
    1.87 +
    1.88 +- Go ahead and include temporary storage in schema and example zasync.conf so
    1.89 +  DBTab doesn't complain.
    1.90 +
    1.91 +- Add new "aggregate" plugin.
    1.92 +
    1.93 +- Include exception info in Conflict Error logs.
    1.94 +
    1.95 +- Make a returned failure in a zope exec callback abort the transaction.
    1.96 +
    1.97 +- Bug fixes in plugins, especially zope_exec
    1.98 +
    1.99 +- fix bug: if tool disappeared, then zasync was unable to recover because of a
   1.100 +  number of problems.  Most importantly, it was holding on to connections
   1.101 +  without closing them.  Fixed this bug and similar problems elsewhere.
   1.102 +
   1.103 +- More paranoid about starting up now, and more paranoid about telling the
   1.104 +  manager about the available plugins when the tool has been re-found.
   1.105 +
   1.106 +- tweak backoff interval for assigning tasks; add backoff interval to calling
   1.107 +  zope_exec start_worker so that existing threads have a better chance to grab
   1.108 +  new jobs before we create an unnecessary new thread.
   1.109 +
   1.110 +- Add sanitize method so that calls to (and from zope_exec) do not have
   1.111 +  persistent objects.
   1.112 +
   1.113 +- Remove an opportunity for Conflict Error in the client.
   1.114 +
   1.115 +- Don't raise an error when a deferred is called twice--this will abort the
   1.116 +  transaction, which means that the deferred will never be resolved if it was
   1.117 +  supposed to be.  Therefore, return a failure and log the problem.  
   1.118 +
   1.119 +- Add a 'here' to the context dict for the callbacks because some code wants
   1.120 +  it.
   1.121 +
   1.122 +version 0.2
   1.123 +
   1.124 +- Significantly rewritten to solve the problems of 0.1.
   1.125 +
   1.126 +  * don't keep persistent objects around
   1.127 +  
   1.128 +  * try to handle exceptions better
   1.129 +  
   1.130 +  * close the connection after every transaction
   1.131 +
   1.132 +- remove all CMF dependencies of the call manager
   1.133 +  
   1.134 +- Use a ZConfig-based configuration for zasync
   1.135 +
   1.136 +  * better Zope initialization
   1.137 +  
   1.138 +  * configuration-driven plugin registration
   1.139 +
   1.140 +- saner startup
   1.141 +
   1.142 +  * not reliant on zopectl run (intended to be in preparation for using twistd)
   1.143 +  
   1.144 +  * sh start script
   1.145 +  
   1.146 +- try to remove plugins from the manager's plugin list upon zasync shutdown
   1.147 +
   1.148 +- ZPL
   1.149 +
   1.150 +version 0.1
   1.151 +
   1.152 +Initial version.  Not released.  ZVSL.
     2.1 new file mode 100644
     2.2 --- /dev/null
     2.3 +++ b/COPYRIGHT.txt
     2.4 @@ -0,0 +1,7 @@
     2.5 +This software is Copyright (c) Zope Corporation (tm) and
     2.6 +Contributors. All rights reserved.
     2.7 +
     2.8 +This software consists of contributions made by Zope
     2.9 +Corporation and many individuals on behalf of Zope
    2.10 +Corporation.  Specific attributions are listed in the
    2.11 +accompanying credits file.
     3.1 new file mode 100644
     3.2 --- /dev/null
     3.3 +++ b/CREDITS.txt
     3.4 @@ -0,0 +1,9 @@
     3.5 +=======
     3.6 +Credits
     3.7 +=======
     3.8 +
     3.9 +Author and maintainer: 
    3.10 +    Gary Poster <gary@zope.com>
    3.11 +
    3.12 +Other contributors: 
    3.13 +    Zac Bir (protected ldap plugin)
    3.14 \ No newline at end of file
     4.1 new file mode 100644
     4.2 --- /dev/null
     4.3 +++ b/CVS/Entries
     4.4 @@ -0,0 +1,14 @@
     4.5 +/CHANGES.txt/1.2/Mon Sep 19 01:04:40 2005//Tzasync-1_1_0
     4.6 +/COPYRIGHT.txt/1.1.1.1/Sun Oct 10 23:37:05 2004//Tzasync-1_1_0
     4.7 +/CREDITS.txt/1.1.1.1/Sun Oct 10 23:37:05 2004//Tzasync-1_1_0
     4.8 +/LICENSE.txt/1.1.1.1/Sun Oct 10 23:37:06 2004//Tzasync-1_1_0
     4.9 +/README.txt/1.4/Sun Sep 18 01:13:14 2005//Tzasync-1_1_0
    4.10 +/__init__.py/1.2/Sun Apr  3 20:53:36 2005//Tzasync-1_1_0
    4.11 +/bforests.py/1.2/Sat Nov 13 01:10:52 2004//Tzasync-1_1_0
    4.12 +/bforests.txt/1.1.1.1/Sun Oct 10 23:37:05 2004//Tzasync-1_1_0
    4.13 +/bucketqueue.py/1.2/Sat Sep 17 03:26:06 2005//Tzasync-1_1_0
    4.14 +/interfaces.py/1.1.1.1/Sun Oct 10 23:37:06 2004//Tzasync-1_1_0
    4.15 +/manager.py/1.10/Sat Sep 17 03:26:06 2005//Tzasync-1_1_0
    4.16 +/manager.txt/1.2/Tue Oct 19 16:42:21 2004//Tzasync-1_1_0
    4.17 +/permissions.py/1.1.1.1/Sun Oct 10 23:37:07 2004//Tzasync-1_1_0
    4.18 +D
     5.1 new file mode 100644
     5.2 --- /dev/null
     5.3 +++ b/CVS/Entries.Log
     5.4 @@ -0,0 +1,4 @@
     5.5 +A D/bin////
     5.6 +A D/client////
     5.7 +A D/tests////
     5.8 +A D/www////
     6.1 new file mode 100644
     6.2 --- /dev/null
     6.3 +++ b/CVS/Repository
     6.4 @@ -0,0 +1,1 @@
     6.5 +Packages/zasync
     7.1 new file mode 100644
     7.2 --- /dev/null
     7.3 +++ b/CVS/Root
     7.4 @@ -0,0 +1,1 @@
     7.5 +:pserver:anonymous@cvs.zope.org:/cvs-repository
     8.1 new file mode 100644
     8.2 --- /dev/null
     8.3 +++ b/CVS/Tag
     8.4 @@ -0,0 +1,1 @@
     8.5 +Nzasync-1_1_0
     9.1 new file mode 100644
     9.2 --- /dev/null
     9.3 +++ b/LICENSE.txt
     9.4 @@ -0,0 +1,53 @@
     9.5 +Zope Public License (ZPL) Version 2.1
     9.6 +-----------------------------------------------
     9.7 +
     9.8 +A copyright notice accompanies this license document that
     9.9 +identifies the copyright holders.
    9.10 +
    9.11 +This license has been certified as open source. It has also
    9.12 +been designated as GPL compatible by the Free Software
    9.13 +Foundation (FSF).
    9.14 +
    9.15 +Redistribution and use in source and binary forms, with or
    9.16 +without modification, are permitted provided that the
    9.17 +following conditions are met:
    9.18 +
    9.19 +   1. Redistributions in source code must retain the
    9.20 +      accompanying copyright notice, this list of conditions,
    9.21 +      and the following disclaimer.
    9.22 +
    9.23 +   2. Redistributions in binary form must reproduce the
    9.24 +      accompanying copyright notice, this list of conditions,
    9.25 +      and the following disclaimer in the documentation and/or
    9.26 +      other materials provided with the distribution.
    9.27 +
    9.28 +   3. Names of the copyright holders must not be used to
    9.29 +      endorse or promote products derived from this software
    9.30 +      without prior written permission from the copyright
    9.31 +      holders.
    9.32 +
    9.33 +   4. The right to distribute this software or to use it for
    9.34 +      any purpose does not give you the right to use
    9.35 +      Servicemarks (sm) or Trademarks (tm) of the copyright
    9.36 +      holders. Use of them is covered by separate agreement
    9.37 +      with the copyright holders.
    9.38 +
    9.39 +   5. If any files are modified, you must cause the modified
    9.40 +      files to carry prominent notices stating that you changed the
    9.41 +      files and the date of any change.
    9.42 +
    9.43 +Disclaimer
    9.44 +
    9.45 +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS''
    9.46 +AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    9.47 +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
    9.48 +FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
    9.49 +SHALL THE COPYRIGHT HOLDERS BE LIABLE FOR ANY DIRECT,
    9.50 +INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
    9.51 +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
    9.52 +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
    9.53 +OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
    9.54 +LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
    9.55 +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
    9.56 +THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY
    9.57 +OF SUCH DAMAGE.
    10.1 new file mode 100644
    10.2 --- /dev/null
    10.3 +++ b/README.txt
    10.4 @@ -0,0 +1,127 @@
    10.5 +ZASYNC
    10.6 +
    10.7 +ZAsync is a Zope 2 product that allows Twisted to asynchronously
    10.8 +perform tasks in a separate process over ZEO on behalf of Zope
    10.9 +requests. Intended use cases (only the first two are realized) include
   10.10 +performing large numbers of long-running external database queries
   10.11 +without tying up Zope; allowing "fire and forget" Zope transactions
   10.12 +that return control to the app server while scheduling long-running
   10.13 +Zope transactions to be performed in the background and potentially on
   10.14 +another physical machine; connecting over a long-running socket with
   10.15 +other asynchronous processes such as instant message servers; and
   10.16 +performing intelligent agent types of tasks.
   10.17 +
   10.18 +STATUS
   10.19 +
   10.20 +Production.
   10.21 +
   10.22 +PREREQUISITES
   10.23 +
   10.24 +Requires that you run a ZEO server as the database source for your 
   10.25 +Zope application.
   10.26 +
   10.27 +Requires Twisted.  
   10.28 +
   10.29 +Tested with Twisted-1.1.1 and Twisted 2.x, and Zope 2.7 and 2.8.
   10.30 +
   10.31 +Of the five plugins provided in client/zasync/plugins.py,
   10.32 +two require python-ldap.  They are tested with 
   10.33 +python-ldap-2.0.0pre18.  They are not installed in the default 
   10.34 +configuration; see SETUP below for instructions in installing them.
   10.35 +
   10.36 +The bin directory contains some start scripts for Windows and *nix for the 
   10.37 +client.  All scripts need to be updated for your Zope installation.
   10.38 +
   10.39 +SETUP
   10.40 +
   10.41 +The two basic pieces that zasync provides are a Zope 2 tool,
   10.42 +manager.py in this directory, and the Twisted-based ZEO client in
   10.43 +the client directory (client/zasync/client.py).  To use zasync, you
   10.44 +must set up both pieces.  Setting up the zasync client is the majority 
   10.45 +of the work.  These instructions assume you have Twisted available to
   10.46 +the Python that runs Zope, as described in "PREREQUISITES" above.
   10.47 +
   10.48 +The manager is typically installed as a singleton in the root of your
   10.49 +Zope via the ZMI: add an Asynchronous Call Manager from the product
   10.50 +dropdown.  The add page suggests "asynchronous_call_manager" as an id,
   10.51 +which is what the default client configuration expects.  Once you have
   10.52 +installed the manager, you are done with that side of the setup.
   10.53 +
   10.54 +Installing the client requires more work.  There are several approaches.
   10.55 +Here is one.
   10.56 +
   10.57 +- If you want to run zasync on another machine, install an identical 
   10.58 +  Zope software stack on the other machine.  All following instructions
   10.59 +  will apply to this second machine.  (If you want to run it on the same
   10.60 +  machine, no special action is needed.)
   10.61 +  
   10.62 +- zasync uses a ZConfig-based configuration file that should look
   10.63 +  largely familiar to you if you are familiar with zope.conf.  An example
   10.64 +  configuration file can be found in 
   10.65 +  (zasync package)/client/zasync/zasync.conf.  The scheme that it uses is
   10.66 +  defined in (zasync package)/client/zasync/schema.xml.  I put a copy of 
   10.67 +  the configuration file in the same etc directory as you zope.conf file,
   10.68 +  but you can put it whereever you wish--just make sure that the start
   10.69 +  script, described below, has the correct path.  The default
   10.70 +  configuration file will require some hand editing:
   10.71 +  
   10.72 +  * all of the paths to software must be correct for your installation;
   10.73 +  
   10.74 +  * the ZODB section must be pointing to the correct ZEO server(s); and
   10.75 +  
   10.76 +  * the zasync behavior must be set up as you desire. :-)  In particular,
   10.77 +    if you have python-ldap installed and would like to use the ldap 
   10.78 +    plugins, just remove comment marks that precede those lines.
   10.79 +
   10.80 +- The client directory in this package is not a Python package, but
   10.81 +  the top of a python path.  see the bin directory and also
   10.82 +  client/zasync/example_start_script
   10.83 +  for examples of scripts that can start up zasync.  Notice in particular that 
   10.84 +  you need to tell zasync if you are using Zope 2.7--it defaults to a 
   10.85 +  schema for Zope 2.8.  I suggest you 
   10.86 +  copy this file to the bin directory of your Zope software instance and 
   10.87 +  call it "zasync".  You will almost certainly need to edit it.  Other 
   10.88 +  startup scripts are welcome if they provide a better experience for a
   10.89 +  standard Zope install.
   10.90 +
   10.91 +USAGE
   10.92 +
   10.93 +The asynchronous call manager currently exposes four core methods for regular
   10.94 +usage: putCall, putSessionCall, getDeferred, and getSessionDeferred.  By 
   10.95 +default, authenticated users can make session-based calls and administrators 
   10.96 +can make generic calls, but this can be changed on the manager's permissions
   10.97 +tab ("Make Asynchronous Calls" and "Make Asynchronous Session Calls").
   10.98 +
   10.99 +To put a call into the call manager, use "putCall" or "putSessionCall".  The
  10.100 +first argument is the plugin name, and following arguments are passed through
  10.101 +to the plugin.  References to persistent objects are generally not good 
  10.102 +arguments to use: the code sanitizes them into objects that have a path and 
  10.103 +a way to dereference them.  Currently no validation is performed to see if 
  10.104 +the arguments match the plugin signature.
  10.105 +
  10.106 +"putCall" or "putSessionCall" then return a ZopeDeferred object.  The 
  10.107 +ZopeDeferred object represents the upcoming result to the asynchronous 
  10.108 +call.  It supports two forms of interaction with the result: pull and push.
  10.109 +
  10.110 +To have a result cause the performance of given actions ("push"), use the errback and 
  10.111 +callback API on the deferred.  In a design modelled after the Twisted deferred
  10.112 +approach, users can add TALES expressions that will be called in the event of
  10.113 +a success (callback) or failure (errback).  errbacks receive a Twisted failure
  10.114 +object, which provide a number of useful capabilties including easily
  10.115 +obtaining the traceback.  If an errback returns a failure (or raises another 
  10.116 +error) the next errback, if any, is called; otherwise, the processing switches
  10.117 +to the next callback.  See the manager.txt documentation for in-depth examples
  10.118 +and see Twisted documentation for more information on callback chains.
  10.119 +
  10.120 +To poll for a result, get the key off of the deferred returned by putCall
  10.121 +and putSessionCall and then use the "getDeferred" and "getSessionDeferred" to 
  10.122 +obtain the deferred and "getStatus" to determine if the deferred has resolved,
  10.123 +and "getValue" if it has.
  10.124 +
  10.125 +XXX more needed, editing needed :-)
  10.126 +
  10.127 +TO DO
  10.128 +
  10.129 +- automated tests of client and plugins
  10.130 +
  10.131 +- move to using twistd
    11.1 new file mode 100644
    11.2 --- /dev/null
    11.3 +++ b/__init__.py
    11.4 @@ -0,0 +1,42 @@
    11.5 +##############################################################################
    11.6 +#
    11.7 +# Copyright (c) 2004 Zope Corporation and Contributors. All Rights Reserved.
    11.8 +#
    11.9 +# This software is subject to the provisions of the Zope Public License,
   11.10 +# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
   11.11 +# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
   11.12 +# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
   11.13 +# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
   11.14 +# FOR A PARTICULAR PURPOSE.
   11.15 +#
   11.16 +##############################################################################
   11.17 +"""asynchronous link to Twisted over ZEO
   11.18 +
   11.19 +$Id: __init__.py,v 1.2 2005/04/03 20:53:36 poster Exp $
   11.20 +"""
   11.21 +import manager
   11.22 +import permissions
   11.23 +
   11.24 +zasync_globals = globals()
   11.25 +
   11.26 +def initializeSecurity():
   11.27 +    # this should arguably go in Zope itself
   11.28 +    from AccessControl import allow_module
   11.29 +    from AccessControl.SimpleObjectPolicies import allow_type
   11.30 +    import datetime
   11.31 +    allow_module('datetime')
   11.32 +    allow_module('datetime._datetime')
   11.33 +    for t in (datetime.datetime, datetime.date, datetime.time,
   11.34 +              datetime.tzinfo, datetime.timedelta):
   11.35 +        allow_type(t)
   11.36 +
   11.37 +def initialize(context):
   11.38 +    """Zope product setup"""
   11.39 +    initializeSecurity()
   11.40 +    context.registerClass(
   11.41 +        manager.AsynchronousCallManager,
   11.42 +        icon="www/daliclock.gif",
   11.43 +        permission=permissions.ViewManagementScreens,
   11.44 +        constructors=(manager.constructAsynchronousCallManagerForm,
   11.45 +                      manager.constructAsynchronousCallManager)
   11.46 +        )
    12.1 new file mode 100644
    12.2 --- /dev/null
    12.3 +++ b/bforests.py
    12.4 @@ -0,0 +1,217 @@
    12.5 +##############################################################################
    12.6 +#
    12.7 +# Copyright (c) 2004 Zope Corporation and Contributors. All Rights Reserved.
    12.8 +#
    12.9 +# This software is subject to the provisions of the Zope Public License,
   12.10 +# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
   12.11 +# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
   12.12 +# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
   12.13 +# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
   12.14 +# FOR A PARTICULAR PURPOSE.
   12.15 +#
   12.16 +##############################################################################
   12.17 +"""Composite trees
   12.18 +
   12.19 +$Id: bforests.py,v 1.2 2004/11/13 01:10:52 poster Exp $
   12.20 +"""
   12.21 +
   12.22 +from Globals import Persistent
   12.23 +from BTrees import IOBTree, OOBTree, OIBTree, IIBTree
   12.24 +from ZODB.PersistentList import PersistentList
   12.25 +
   12.26 +class AbstractBForest(Persistent):
   12.27 +
   12.28 +    _treemodule = None # override
   12.29 +    _treeclass = None # override
   12.30 +    _marker = object()
   12.31 +    
   12.32 +    def __init__(self, d=None, count=2):
   12.33 +        if count < 0:
   12.34 +            raise ValueError("count must be 0 or greater")
   12.35 +        if count == 0:
   12.36 +            if d is not None:
   12.37 +                raise ValueError(
   12.38 +                    "cannot set initial values without a bucket")
   12.39 +            l = PersistentList()
   12.40 +        else:
   12.41 +            Tree = self._treeclass
   12.42 +            if d is not None:
   12.43 +                first = Tree(d)
   12.44 +            else:
   12.45 +                first = Tree()
   12.46 +            l = [Tree() for i in range(count - 1)]
   12.47 +            l.insert(0, first)
   12.48 +            l = PersistentList(l)
   12.49 +        self.buckets = l
   12.50 +    
   12.51 +    def __getitem__(self, key):
   12.52 +        m = self._marker
   12.53 +        res = self.get(key, m)
   12.54 +        if res is m:
   12.55 +            raise KeyError(key)
   12.56 +        return res
   12.57 +    
   12.58 +    def __setitem__(self, key, value):
   12.59 +        self.buckets[0][key] = value
   12.60 +    
   12.61 +    def get(self, key, default=None):
   12.62 +        m = self._marker
   12.63 +        for b in self.buckets:
   12.64 +            res = b.get(key, m)
   12.65 +            if res is not m:
   12.66 +                return res
   12.67 +        return default
   12.68 +    
   12.69 +    def __delitem__(self, key):
   12.70 +        found = False
   12.71 +        for b in self.buckets:
   12.72 +            try:
   12.73 +                del b[key]
   12.74 +            except KeyError:
   12.75 +                pass
   12.76 +            else:
   12.77 +                found = True
   12.78 +        if not found:
   12.79 +            raise KeyError(key)
   12.80 +    
   12.81 +    def update(self, d):
   12.82 +        self.buckets[0].update(d)
   12.83 +    
   12.84 +    def keys(self):
   12.85 +        union = self._treemodule.union
   12.86 +        buckets = self.buckets
   12.87 +        if len(buckets) == 1:
   12.88 +            res = buckets[0].keys()
   12.89 +        else:
   12.90 +            res = union(buckets[0], buckets[1])
   12.91 +            for b in buckets[2:]:
   12.92 +                res = union(res, b)
   12.93 +        return res
   12.94 +    
   12.95 +    def tree(self):
   12.96 +        # convert to a tree; do as much in C as possible.
   12.97 +        buckets = self.buckets
   12.98 +        res = self._treeclass(buckets[-1])
   12.99 +        for b in buckets[-2::-1]:
  12.100 +            res.update(b)
  12.101 +        return res
  12.102 +    
  12.103 +    def items(self):
  12.104 +        return self.tree().items()
  12.105 +
  12.106 +    def values(self):
  12.107 +        return self.tree().values()
  12.108 +    
  12.109 +    def iteritems(self):
  12.110 +        for key in self.keys():
  12.111 +            yield key, self[key]
  12.112 +    
  12.113 +    def iterkeys(self):
  12.114 +        return iter(self.keys())
  12.115 +    
  12.116 +    __iter__ = iterkeys
  12.117 +    
  12.118 +    def itervalues(self):
  12.119 +        for key in self.keys():
  12.120 +            yield self[key]
  12.121 +    
  12.122 +    def has_key(self, key): 
  12.123 +        try:
  12.124 +            self[key]
  12.125 +        except KeyError:
  12.126 +            return False
  12.127 +        else:
  12.128 +            return True
  12.129 +    
  12.130 +    def __cmp__(self, d):
  12.131 +        # Notice this unpleasant bit: a bforest and dict with the same      
  12.132 +        # contents will return True if the comparison is bforest == dict    
  12.133 +        # (above) but false if the comparison is dict == bforest.  This     
  12.134 +        # might be fixable if we could use rich comparisons, but we can't    
  12.135 +        # do that with the old-style ExtensionClass used in pre-Zope 2.8. 
  12.136 +        
  12.137 +        # this is probably not the most efficient approach, but I'm not sure
  12.138 +        # what is, and this is certainly among the simplest:
  12.139 +        return cmp(dict(self), dict(d)) # :-/ eh
  12.140 +    
  12.141 +    def __len__(self):
  12.142 +        return len(self.tree())
  12.143 +    
  12.144 +    def setdefault(self, key, failobj=None):
  12.145 +        try:
  12.146 +            res = self[key]
  12.147 +        except KeyError:
  12.148 +            res = self[key] = failobj
  12.149 +        return res
  12.150 +    
  12.151 +    def pop(self, key, d=_marker):
  12.152 +        try:
  12.153 +            res = self[key]
  12.154 +        except KeyError:
  12.155 +            if d is self._marker:
  12.156 +                raise KeyError(key)
  12.157 +            else:
  12.158 +                return d
  12.159 +        else:
  12.160 +            del self[key]
  12.161 +            return res
  12.162 +    
  12.163 +    def popitem(self):
  12.164 +        for b in self.buckets:
  12.165 +            try:
  12.166 +                key = b.minKey()
  12.167 +            except ValueError:
  12.168 +                pass
  12.169 +            else:
  12.170 +                val = b[key]
  12.171 +                del b[key]
  12.172 +                return key, val
  12.173 +        else:
  12.174 +            raise KeyError('popitem():dictionary is empty')
  12.175 +                        
  12.176 +    def __contains__(self, key):
  12.177 +        for b in self.buckets:
  12.178 +            if b.has_key(key):
  12.179 +                return True
  12.180 +        return False
  12.181 +    
  12.182 +    def copy(self):
  12.183 +        # this makes an exact copy, including the individual state of each 
  12.184 +        # bucket.  If you want a dict, cast it to a dict, or if you want
  12.185 +        # another one of these but with all of the keys in the first bucket,
  12.186 +        # call obj.__class__(obj)
  12.187 +        copy = self.__class__(count=0)
  12.188 +        copy.buckets = [self._treeclass(t) for t in self.buckets]
  12.189 +        return copy
  12.190 +    
  12.191 +    def clear(self):
  12.192 +        for b in self.buckets:
  12.193 +            b.clear()
  12.194 +    
  12.195 +    def __nonzero__(self):
  12.196 +        for b in self.buckets:
  12.197 +            if bool(b):
  12.198 +                return True
  12.199 +        return False
  12.200 +    
  12.201 +    def rotateBucket(self):
  12.202 +        buckets = self.buckets
  12.203 +        b = buckets.pop()
  12.204 +        b.clear()
  12.205 +        buckets.insert(0, b)
  12.206 +
  12.207 +class IOBForest(AbstractBForest):
  12.208 +    _treemodule = IOBTree
  12.209 +    _treeclass = IOBTree.IOBTree
  12.210 +
  12.211 +class OIBForest(AbstractBForest):
  12.212 +    _treemodule = OIBTree
  12.213 +    _treeclass = OIBTree.OIBTree
  12.214 +
  12.215 +class OOBForest(AbstractBForest):
  12.216 +    _treemodule = OOBTree
  12.217 +    _treeclass = OOBTree.OOBTree
  12.218 +
  12.219 +class IIBForest(AbstractBForest):
  12.220 +    _treemodule = IIBTree
  12.221 +    _treeclass = IIBTree.IIBTree
    13.1 new file mode 100644
    13.2 --- /dev/null
    13.3 +++ b/bforests.txt
    13.4 @@ -0,0 +1,207 @@
    13.5 +bforests are dictionary-like objects that use multiple BTrees for a backend and
    13.6 +support rotation of the composite trees.  This supports various implementations 
    13.7 +of timed member expirations, enabling caches and semi-persistent storage.  A
    13.8 +useful and simple subclass would be to promote a key-value pair to the
    13.9 +first (newest) bucket whenever the key is accessed, for instance.
   13.10 +
   13.11 +Like btrees, bforests come in four flavors: Integer-Integer (IIBForest), 
   13.12 +Integer-Object (IOBForest), Object-Integer (OIBForest), and Object-Object
   13.13 +(OOBForest).  The examples here will deal with them in the abstract: we will
   13.14 +create classes from the imaginary and representative BForest class, and
   13.15 +generate keys from KeyGenerator and values from ValueGenerator.  From the 
   13.16 +examples you should be able to extrapolate usage of all four types.
   13.17 +
   13.18 +First let's instantiate a bforest and look at an empty example.  By default,
   13.19 +a new bforest creates two composite btree buckets.
   13.20 +
   13.21 +>>> d = BForest()
   13.22 +>>> list(d.keys())
   13.23 +[]
   13.24 +>>> list(d.values())
   13.25 +[]
   13.26 +>>> len(d.buckets)
   13.27 +2
   13.28 +>>> dummy_key = KeyGenerator()
   13.29 +>>> d.get(dummy_key)
   13.30 +>>> d.get(dummy_key, 42)
   13.31 +42
   13.32 +
   13.33 +Now we'll populate it.  We'll first create a dictionary we'll use to compare.
   13.34 +
   13.35 +>>> original = {}
   13.36 +>>> for i in range(10):
   13.37 +...     original[KeyGenerator()] = ValueGenerator()
   13.38 +... 
   13.39 +>>> d.update(original)
   13.40 +>>> d == original
   13.41 +True
   13.42 +>>> d_keys = list(d.keys())
   13.43 +>>> d_keys.sort()
   13.44 +>>> o_keys = original.keys()
   13.45 +>>> o_keys.sort()
   13.46 +>>> d_keys == o_keys
   13.47 +True
   13.48 +>>> d_values = list(d.values())
   13.49 +>>> d_values.sort()
   13.50 +>>> o_values = original.values()
   13.51 +>>> o_values.sort()
   13.52 +>>> o_values == d_values
   13.53 +True
   13.54 +>>> d_items = list(d.items())
   13.55 +>>> d_items.sort()
   13.56 +>>> o_items = original.items()
   13.57 +>>> o_items.sort()
   13.58 +>>> o_items == d_items
   13.59 +True
   13.60 +>>> key, value = d.popitem()
   13.61 +>>> value == original.pop(key)
   13.62 +True
   13.63 +>>> key, value = original.popitem()
   13.64 +>>> value == d.pop(key)
   13.65 +True
   13.66 +>>> len(d) == len(original)
   13.67 +True
   13.68 +
   13.69 +Now let's rotate the buckets.
   13.70 +
   13.71 +>>> d.rotateBucket()
   13.72 +
   13.73 +...and we'll do the exact same test as above, first.
   13.74 +
   13.75 +>>> d == original
   13.76 +True
   13.77 +>>> d_keys = list(d.keys())
   13.78 +>>> d_keys.sort()
   13.79 +>>> o_keys = original.keys()
   13.80 +>>> o_keys.sort()
   13.81 +>>> d_keys == o_keys
   13.82 +True
   13.83 +>>> d_values = list(d.values())
   13.84 +>>> d_values.sort()
   13.85 +>>> o_values = original.values()
   13.86 +>>> o_values.sort()
   13.87 +>>> o_values == d_values
   13.88 +True
   13.89 +>>> d_items = list(d.items())
   13.90 +>>> d_items.sort()
   13.91 +>>> o_items = original.items()
   13.92 +>>> o_items.sort()
   13.93 +>>> o_items == d_items
   13.94 +True
   13.95 +>>> key, value = d.popitem()
   13.96 +>>> value == original.pop(key)
   13.97 +True
   13.98 +>>> key, value = original.popitem()
   13.99 +>>> value == d.pop(key)
  13.100 +True
  13.101 +>>> len(d) == len(original)
  13.102 +True
  13.103 +
  13.104 +Now we'll make a new dictionary to represent changes made after the bucket
  13.105 +rotation.
  13.106 +
  13.107 +>>> second = {}
  13.108 +>>> for i in range(10):
  13.109 +...     key = KeyGenerator()
  13.110 +...     value = ValueGenerator()
  13.111 +...     second[key] = value
  13.112 +...     d[key] = value
  13.113 +... 
  13.114 +>>> original.update(second)
  13.115 +
  13.116 +...and we'll do almost the exact same test as above, first.
  13.117 +
  13.118 +>>> d == original
  13.119 +True
  13.120 +>>> d_keys = list(d.keys())
  13.121 +>>> d_keys.sort()
  13.122 +>>> o_keys = original.keys()
  13.123 +>>> o_keys.sort()
  13.124 +>>> d_keys == o_keys
  13.125 +True
  13.126 +>>> d_values = list(d.values())
  13.127 +>>> d_values.sort()
  13.128 +>>> o_values = original.values()
  13.129 +>>> o_values.sort()
  13.130 +>>> o_values == d_values
  13.131 +True
  13.132 +>>> d_items = list(d.items())
  13.133 +>>> d_items.sort()
  13.134 +>>> o_items = original.items()
  13.135 +>>> o_items.sort()
  13.136 +>>> o_items == d_items
  13.137 +True
  13.138 +>>> key, value = d.popitem()
  13.139 +>>> ignore = second.pop(key, None) # keep second up-to-date
  13.140 +>>> value == original.pop(key)
  13.141 +True
  13.142 +>>> key, value = original.popitem()
  13.143 +>>> ignore = second.pop(key, None) # keep second up-to-date
  13.144 +>>> value == d.pop(key)
  13.145 +True
  13.146 +>>> len(d) == len(original)
  13.147 +True
  13.148 +
  13.149 +Now if we rotate the buckets, the first set of items will be gone, but the 
  13.150 +second will remain.
  13.151 +
  13.152 +>>> d.rotateBucket()
  13.153 +>>> d == original
  13.154 +False
  13.155 +>>> d == second
  13.156 +True
  13.157 +
  13.158 +Let's set a value, check the copy behavior,  and then rotate it one more time.
  13.159 +
  13.160 +>>> third = {KeyGenerator(): ValueGenerator()}
  13.161 +>>> d.update(third)
  13.162 +>>> copy = d.copy()
  13.163 +>>> copy == d
  13.164 +True
  13.165 +>>> copy != second # because second doesn't have the values of third
  13.166 +True
  13.167 +>>> list(copy.buckets[0].items()) == list(d.buckets[0].items())
  13.168 +True
  13.169 +>>> list(copy.buckets[1].items()) == list(d.buckets[1].items())
  13.170 +True
  13.171 +>>> copy[KeyGenerator()] = ValueGenerator()
  13.172 +>>> copy == d
  13.173 +False
  13.174 +>>> d.rotateBucket()
  13.175 +>>> d == third
  13.176 +True
  13.177 +>>> d.clear()
  13.178 +>>> d == BForest() == {}
  13.179 +True
  13.180 +
  13.181 +>>> d.update(second)
  13.182 +
  13.183 +We'll make a value in one bucket that we'll override in another.
  13.184 +
  13.185 +>>> d[third.keys()[0]] = ValueGenerator()
  13.186 +>>> d.rotateBucket()
  13.187 +>>> d.update(third)
  13.188 +>>> second.update(third)
  13.189 +>>> d == second
  13.190 +True
  13.191 +
  13.192 +Notice this unpleasant bit: a bforest and dict with the same contents will 
  13.193 +return True if the comparison is bforest == dict (above) but false if the
  13.194 +comparison is dict == bforest.  This might be fixable if we could use rich
  13.195 +comparisons, but we can't do that with the old-style ExtensionClass used in 
  13.196 +pre-Zope 2.8.
  13.197 +
  13.198 +>>> second == d
  13.199 +False
  13.200 +
  13.201 +The tree method converts the bforest to a btree as efficiently as I know how
  13.202 +for a common case of more items in buckets than buckets.
  13.203 +
  13.204 +>>> tree = d.tree()
  13.205 +>>> d_items = list(d.items())
  13.206 +>>> d_items.sort()
  13.207 +>>> t_items = list(tree.items())
  13.208 +>>> t_items.sort()
  13.209 +>>> t_items == d_items
  13.210 +True
  13.211 +
    14.1 new file mode 100644
    14.2 --- /dev/null
    14.3 +++ b/bin/CVS/Entries
    14.4 @@ -0,0 +1,5 @@
    14.5 +/runzasync/1.2/Sun Sep 18 01:13:14 2005//Tzasync-1_1_0
    14.6 +/runzasync.bat/1.2/Sun Sep 18 01:13:14 2005//Tzasync-1_1_0
    14.7 +/zasync/1.2/Sun Sep 18 01:13:14 2005//Tzasync-1_1_0
    14.8 +/zasyncctl.py/1.2/Sun Sep 18 01:13:14 2005//Tzasync-1_1_0
    14.9 +D
    15.1 new file mode 100644
    15.2 --- /dev/null
    15.3 +++ b/bin/CVS/Repository
    15.4 @@ -0,0 +1,1 @@
    15.5 +Packages/zasync/bin
    16.1 new file mode 100644
    16.2 --- /dev/null
    16.3 +++ b/bin/CVS/Root
    16.4 @@ -0,0 +1,1 @@
    16.5 +:pserver:anonymous@cvs.zope.org:/cvs-repository
    17.1 new file mode 100644
    17.2 --- /dev/null
    17.3 +++ b/bin/CVS/Tag
    17.4 @@ -0,0 +1,1 @@
    17.5 +Nzasync-1_1_0
    18.1 new file mode 100755
    18.2 --- /dev/null
    18.3 +++ b/bin/runzasync
    18.4 @@ -0,0 +1,25 @@
    18.5 +#! /bin/sh
    18.6 +
    18.7 +# change these paths to match your installation
    18.8 +ZOPE_HOME="/home2/FIBU/Zope2.7"
    18.9 +PYTHON="${ZOPE_HOME}/bin/python"
   18.10 +INSTANCE_HOME="${ZOPE_HOME}/instances"
   18.11 +
   18.12 +CONFIG_FILE="${INSTANCE_HOME}/etc/zasync.conf"
   18.13 +# only needed if you have a second ACM
   18.14 +CONFIG_FILE2="${INSTANCE_HOME}/etc/zasync2.conf"
   18.15 +
   18.16 +SOFTWARE_HOME="${ZOPE_HOME}/lib/python"
   18.17 +CLIENT_HOME="${INSTANCE_HOME}/Products/zasync/client"
   18.18 +PYTHONPATH="${INSTANCE}/bin:${CLIENT_HOME}:${SOFTWARE_HOME}"
   18.19 +export PYTHONPATH INSTANCE_HOME 
   18.20 +
   18.21 +# use this line if you have only one ACM
   18.22 +# if using Zope 2.8, call run('$CONFIG_FILE')
   18.23 +#exec $PYTHON -c "from zasyncctl import run;run('$CONFIG_FILE', 2.7)" 
   18.24 +
   18.25 +# use these lines if you have two ACMs configured
   18.26 +# if using Zope 2.8, call run('$CONFIG_FILE')
   18.27 +$PYTHON -c "from zasyncctl import run;run('$CONFIG_FILE', 2.7)" 
   18.28 +$PYTHON -c "from zasyncctl import run;run('$CONFIG_FILE2', 2.7)" 
   18.29 +
    19.1 new file mode 100755
    19.2 --- /dev/null
    19.3 +++ b/bin/runzasync.bat
    19.4 @@ -0,0 +1,12 @@
    19.5 +@set ZOPE_HOME=C:\packages\Zope\Zope-2.7.4-0
    19.6 +@set INSTANCE_HOME=C:\Zope-Instances
    19.7 +@set PYTHON=%ZOPE_HOME%\bin\python.exe
    19.8 +@set CONFIG_FILE=%INSTANCE_HOME%\FiBu\etc\zasync.conf
    19.9 +@set SOFTWARE_HOME=%ZOPE_HOME%\lib\python
   19.10 +@set CLIENT_HOME=%INSTANCE_HOME%\FiBu\Products\zasync\client
   19.11 +@set PYTHONPATH=%CLIENT_HOME%
   19.12 +
   19.13 +"%PYTHON%" "-c" "from zasync import run;run('%CONFIG_FILE%', 2.7)"
   19.14 +
   19.15 +
   19.16 +
    20.1 new file mode 100755
    20.2 --- /dev/null
    20.3 +++ b/bin/zasync
    20.4 @@ -0,0 +1,27 @@
    20.5 +#! /bin/sh
    20.6 +
    20.7 +# Script suitable for adding to /etc/rc.d/init.d
    20.8 +return=$rc_done
    20.9 +case "$1" in
   20.10 +    start)
   20.11 +	echo -n "Starting service for Zope Asynchronous Call Manager"
   20.12 +	## the echo return value is set appropriately.
   20.13 +	( /home2/FIBU/Zope2.7/instances/bin/startzasync & ) || return=$rc_failed
   20.14 +	echo -e "$return"
   20.15 +	;;
   20.16 +    stop)
   20.17 +	echo -n "Shutting down service for Zope Asynchronous Call Manager"
   20.18 +	echo -n "This can take a while."
   20.19 +	## the echo return value is set appropriately.
   20.20 +	/home2/FIBU/Zope2.7/instances/bin/stopzasync || return=$rc_failed
   20.21 +	echo -e "$return"
   20.22 +	;;
   20.23 +    *)
   20.24 +	echo "Usage: $0 {start|stop}"
   20.25 +	exit 1
   20.26 +	;;
   20.27 +esac
   20.28 +
   20.29 +# Inform the caller not only verbosely and set an exit status.
   20.30 +test "$return" = "$rc_done" || exit 1
   20.31 +exit 0
    21.1 new file mode 100755
    21.2 --- /dev/null
    21.3 +++ b/bin/zasyncctl.py
    21.4 @@ -0,0 +1,30 @@
    21.5 +import os
    21.6 +
    21.7 +def makePidFile(cfg):
    21.8 +    # write the pid into the pidfile if possible
    21.9 +    try:
   21.10 +         if os.path.exists(cfg.pid_filename):
   21.11 +            os.unlink(cfg.pid_filename)
   21.12 +         f = open(cfg.pid_filename, 'w')
   21.13 +         f.write(str(os.getpid()))
   21.14 +         f.close()
   21.15 +    except IOError:
   21.16 +         pass
   21.17 +
   21.18 +def kill(config_file=None, version=2.8):
   21.19 +    from zasync import config
   21.20 +    import signal
   21.21 +    cfg = config.initialize(config_file, version)
   21.22 +    pid_filename = cfg.pid_filename
   21.23 +    f = open(pid_filename, 'r')
   21.24 +    pid = int(f.read())
   21.25 +    f.close()
   21.26 +    os.kill(pid, signal.SIGKILL)
   21.27 +
   21.28 +# Taken from zasync/client/zasync/__init__.py
   21.29 +def run(config_file=None, version=2.8):
   21.30 +    from zasync import config
   21.31 +    from zasync import client
   21.32 +    conf = config.initialize(config_file, version)
   21.33 +    makePidFile(conf)
   21.34 +    client.run(conf.target_path)
    22.1 new file mode 100644
    22.2 --- /dev/null
    22.3 +++ b/bucketqueue.py
    22.4 @@ -0,0 +1,269 @@
    22.5 +"""multi-producer, multi-consumer queue with items optionally in "buckets";
    22.6 +only allows one item from a given bucket to be obtained at a time.
    22.7 +A subclass of the standard Python library queue.
    22.8 +
    22.9 +$Id: bucketqueue.py,v 1.2 2005/09/17 03:26:06 poster Exp $"""
   22.10 +
   22.11 +from time import time as _time, sleep as _sleep
   22.12 +import sets
   22.13 +try:
   22.14 +    import thread
   22.15 +except ImportError:
   22.16 +    import dummy_thread as thread
   22.17 +from Queue import Empty, Full, Queue
   22.18 +
   22.19 +__all__ = ['BucketQueue']
   22.20 +
   22.21 +class BucketQueue(Queue):
   22.22 +    """A queue that only allows a single item from a given bucket to be 
   22.23 +    obtained at a time.  Within a bucket, order is guaranteed.  Between 
   22.24 +    unblocked buckets, order is also honored.
   22.25 +    """
   22.26 +    def __init__(self, maxsize=0):
   22.27 +        # in Queue, fsema presumably stands for (not) full semaphore, and esema
   22.28 +        # stands for (not) empty semaphore.  For us, esema will mean "primed"
   22.29 +        # or "available" semaphore.
   22.30 +        Queue.__init__(self, maxsize)
   22.31 +        self._bucketsema = {} # bucket name to bucket semaphore
   22.32 +        self._threadbucket = {} # thread id to bucket name
   22.33 +
   22.34 +    def primed(self):
   22.35 +        """Return True if the queue is primed, False otherwise (not reliable!).
   22.36 +        """
   22.37 +        self.mutex.acquire()
   22.38 +        try:
   22.39 +            return self._primed()
   22.40 +        finally:
   22.41 +            self.mutex.release()
   22.42 +    
   22.43 +    def available(self, id=None):
   22.44 +        """Return True if there is currently an item available for this 
   22.45 +        particular thread"""
   22.46 +        if id is None:
   22.47 +            id = thread.get_ident()
   22.48 +        self.mutex.acquire()
   22.49 +        try:
   22.50 +            return self._primed(self._threadbucket.get(id))
   22.51 +        finally:
   22.52 +            self.mutex.release()
   22.53 +    
   22.54 +    def makeBucket(self, name, silent=False):
   22.55 +        if name is None:
   22.56 +            raise ValueError("Bucket name cannot be None")
   22.57 +        if self._bucketsema.has_key(name):
   22.58 +            if not silent:
   22.59 +                raise ValueError("Bucket already exists", name)
   22.60 +        else:
   22.61 +            self._bucketsema[name] = thread.allocate_lock()
   22.62 +
   22.63 +    def put(self, item, block=True, timeout=None, bucket=None):
   22.64 +        """Put an item into the given bucket of the queue.
   22.65 +
   22.66 +        If optional args 'block' is true and 'timeout' is None (the default),
   22.67 +        block if necessary until a free slot is available. If 'timeout' is
   22.68 +        a positive number, it blocks at most 'timeout' seconds and raises
   22.69 +        the Full exception if no free slot was available within that time.
   22.70 +        Otherwise ('block' is false), put an item on the queue if a free slot
   22.71 +        is immediately available, else raise the Full exception ('timeout'
   22.72 +        is ignored in that case).
   22.73 +        """
   22.74 +        if bucket is not None and self._bucketsema.get(bucket) is None:
   22.75 +            raise ValueError("bucket does not exist; create with makeBucket", 
   22.76 +                             bucket)
   22.77 +        if block:
   22.78 +            if timeout is None:
   22.79 +                # blocking, w/o timeout, i.e. forever
   22.80 +                self.fsema.acquire()
   22.81 +            elif timeout >= 0:
   22.82 +                # waiting max. 'timeout' seconds.
   22.83 +                # this code snipped is from threading.py: _Event.wait():
   22.84 +                # Balancing act:  We can't afford a pure busy loop, so we
   22.85 +                # have to sleep; but if we sleep the whole timeout time,
   22.86 +                # we'll be unresponsive.  The scheme here sleeps very
   22.87 +                # little at first, longer as time goes on, but never longer
   22.88 +                # than 20 times per second (or the timeout time remaining).
   22.89 +                delay = 0.0005 # 500 us -> initial delay of 1 ms
   22.90 +                endtime = _time() + timeout
   22.91 +                while True:
   22.92 +                    if self.fsema.acquire(0):
   22.93 +                        break
   22.94 +                    remaining = endtime - _time()
   22.95 +                    if remaining <= 0:  #time is over and no slot was free
   22.96 +                        raise Full
   22.97 +                    delay = min(delay * 2, remaining, .05)
   22.98 +                    _sleep(delay)       #reduce CPU usage by using a sleep
   22.99 +            else:
  22.100 +                raise ValueError("'timeout' must be a positive number")
  22.101 +        elif not self.fsema.acquire(0):
  22.102 +            raise Full
  22.103 +        self.mutex.acquire()
  22.104 +        release_fsema = True
  22.105 +        try:
  22.106 +            was_primed = self._primed()
  22.107 +            self._put((item, bucket))
  22.108 +            # If we fail before here, the empty state has
  22.109 +            # not changed, so we can skip the release of esema
  22.110 +            if not was_primed and self._primed():
  22.111 +                self.esema.release()
  22.112 +            # If we fail before here, the queue can not be full, so
  22.113 +            # release_full_sema remains True
  22.114 +            release_fsema = not self._full()
  22.115 +        finally:
  22.116 +            # Catching system level exceptions here (RecursionDepth,
  22.117 +            # OutOfMemory, etc) - so do as little as possible in terms
  22.118 +            # of Python calls.
  22.119 +            if release_fsema:
  22.120 +                self.fsema.release()
  22.121 +            self.mutex.release()
  22.122 +
  22.123 +    def put_nowait(self, item, bucket=None):
  22.124 +        """Put an item into the queue without blocking.
  22.125 +
  22.126 +        Only enqueue the item if a free slot is immediately available.
  22.127 +        Otherwise raise the Full exception.
  22.128 +        """
  22.129 +        return self.put(item, False, bucket=bucket)
  22.130 +    
  22.131 +    def releaseBucketByThreadId(self, id):
  22.132 +        return self._releaseBucket(id=id)
  22.133 +   
  22.134 +    def releaseBucket(self, bucket=None):
  22.135 +        """Release bucket.  If bucket is None, release the bucket for this
  22.136 +        thread, if any."""
  22.137 +        return self._releaseBucket(bucket)
  22.138 +   
  22.139 +    def _releaseBucket(self, bucket=None, id=None):
  22.140 +        self.mutex.acquire()
  22.141 +        try:
  22.142 +            if bucket is None:
  22.143 +                if id is None:
  22.144 +                    id = thread.get_ident()
  22.145 +                bucket = self._threadbucket.pop(id, None)
  22.146 +            else:
  22.147 +                for tid, b in self._threadbucket.items():
  22.148 +                    if b == bucket:
  22.149 +                        del self._threadbucket[tid]
  22.150 +                        break
  22.151 +                else:
  22.152 +                    bucket = None
  22.153 +            if bucket is not None:
  22.154 +                was_primed = self._primed()
  22.155 +                self._bucketsema[bucket].release()
  22.156 +                if not was_primed and self._primed():
  22.157 +                    self.esema.release()
  22.158 +        finally:
  22.159 +            self.mutex.release()
  22.160 +        return bucket
  22.161 +
  22.162 +    def get(self, block=True, timeout=None):
  22.163 +        """Remove and return an item from the queue.
  22.164 +
  22.165 +        If optional args 'block' is true and 'timeout' is None (the default),
  22.166 +        block if necessary until an item is available. If 'timeout' is
  22.167 +        a positive number, it blocks at most 'timeout' seconds and raises
  22.168 +        the Empty exception if no item was available within that time.
  22.169 +        Otherwise ('block' is false), return an item if one is immediately
  22.170 +        available, else raise the Empty exception ('timeout' is ignored
  22.171 +        in that case).
  22.172 +        """
  22.173 +        self.releaseBucket()
  22.174 +        if block:
  22.175 +            if timeout is None:
  22.176 +                # blocking, w/o timeout, i.e. forever
  22.177 +                self.esema.acquire()
  22.178 +            elif timeout >= 0:
  22.179 +                # waiting max. 'timeout' seconds.
  22.180 +                # this code snipped is from threading.py: _Event.wait():
  22.181 +                # Balancing act:  We can't afford a pure busy loop, so we
  22.182 +                # have to sleep; but if we sleep the whole timeout time,
  22.183 +                # we'll be unresponsive.  The scheme here sleeps very
  22.184 +                # little at first, longer as time goes on, but never longer
  22.185 +                # than 20 times per second (or the timeout time remaining).
  22.186 +                delay = 0.0005 # 500 us -> initial delay of 1 ms
  22.187 +                endtime = _time() + timeout
  22.188 +                while 1:
  22.189 +                    if self.esema.acquire(0):
  22.190 +                        break
  22.191 +                    remaining = endtime - _time()
  22.192 +                    if remaining <= 0:  #time is over and no element arrived
  22.193 +                        raise Empty # XXX or Unavailable if not primed?
  22.194 +                    delay = min(delay * 2, remaining, .05)
  22.195 +                    _sleep(delay)       #reduce CPU usage by using a sleep
  22.196 +            else:
  22.197 +                raise ValueError("'timeout' must be a positive number")
  22.198 +        elif not self.esema.acquire(0):
  22.199 +            raise Empty # XXX or Unavailable if not primed?
  22.200 +        self.mutex.acquire()
  22.201 +        release_esema = True
  22.202 +        try:
  22.203 +            was_full = self._full()
  22.204 +            ix, bucket, bucketlock, item = self._findNext()
  22.205 +            try:
  22.206 +                if bucket is not None:
  22.207 +                    bucketlock.acquire()
  22.208 +                    self._threadbucket[thread.get_ident()] = bucket
  22.209 +                self._removeIx(ix)
  22.210 +            except: # we need to clean up
  22.211 +                if bucketlock is not None:
  22.212 +                    bucketlock.release()
  22.213 +                    self._threadbucket.pop(thread.get_ident(), None)
  22.214 +                raise
  22.215 +            # If we fail before here, the full state has
  22.216 +            # not changed, so we can skip the release of fsema
  22.217 +            if was_full:
  22.218 +                self.fsema.release()
  22.219 +            # Failure means empty state also unchanged - release_esema
  22.220 +            # remains True.
  22.221 +            release_esema = self._primed()
  22.222 +        finally:
  22.223 +            if release_esema:
  22.224 +                self.esema.release()
  22.225 +            self.mutex.release()
  22.226 +        return item
  22.227 +    
  22.228 +    # Override if desired (but others are more useful, below).  These are
  22.229 +    # only called with the appropriate locks held.
  22.230 +    
  22.231 +    def _findNext(self, current=None):
  22.232 +        """return ix, bucket lock, and item of next open, where ix is the value
  22.233 +        that _removeIx needs to remove the item from the queue.  Raise 
  22.234 +        LookupError if no next item is available."""
  22.235 +        buckets = sets.Set()
  22.236 +        for ix, bucket, item in self._enumerateItems():
  22.237 +            if bucket is None:
  22.238 +                return ix, bucket, None, item
  22.239 +            if bucket not in buckets:
  22.240 +                lock = self._bucketsema[bucket]
  22.241 +                if current==bucket or not lock.locked():
  22.242 +                    return ix, bucket, lock, item
  22.243 +                buckets.add(bucket)
  22.244 +        raise LookupError("No next value available")
  22.245 +    
  22.246 +    def _primed(self, current=None):
  22.247 +        try:
  22.248 +            self._findNext(current)
  22.249 +        except LookupError:
  22.250 +            return False
  22.251 +        else:
  22.252 +            return True
  22.253 +
  22.254 +    # Override these methods to implement other queue organizations
  22.255 +    # These will only be called with appropriate locks held
  22.256 +
  22.257 +    def _enumerateItems(self):
  22.258 +        """enumerate ix, bucket name, and item of each item in the queue in
  22.259 +        preferred order (defaults to FIFO).  ix is whatever value that 
  22.260 +        _removeIx needs to remove the item from the queue."""
  22.261 +        for ix, (item, bucket) in enumerate(self.queue):
  22.262 +            yield ix, bucket, item
  22.263 +    
  22.264 +    def _removeIx(self, ix):
  22.265 +        """Given an ix of an item provided by _enumerateItems or _findNext,
  22.266 +        remove the associated entry from the queue"""
  22.267 +        del self.queue[ix]
  22.268 +    
  22.269 +    # _get, from the original Queue implementation, is not used: see 
  22.270 +    # _findNext, _enumerateItems, and _removeIx for methods that perform
  22.271 +    # elements of the original task of _get.
  22.272 +    def _get(self):
  22.273 +        pass
    23.1 new file mode 100644
    23.2 --- /dev/null
    23.3 +++ b/client/CVS/Entries
    23.4 @@ -0,0 +1,2 @@
    23.5 +/README.txt/1.1.1.1/Sun Oct 10 23:37:07 2004//Tzasync-1_1_0
    23.6 +D
    24.1 new file mode 100644
    24.2 --- /dev/null
    24.3 +++ b/client/CVS/Entries.Log
    24.4 @@ -0,0 +1,1 @@
    24.5 +A D/zasync////
    25.1 new file mode 100644
    25.2 --- /dev/null
    25.3 +++ b/client/CVS/Repository
    25.4 @@ -0,0 +1,1 @@
    25.5 +Packages/zasync/client
    26.1 new file mode 100644
    26.2 --- /dev/null
    26.3 +++ b/client/CVS/Root
    26.4 @@ -0,0 +1,1 @@
    26.5 +:pserver:anonymous@cvs.zope.org:/cvs-repository
    27.1 new file mode 100644
    27.2 --- /dev/null
    27.3 +++ b/client/CVS/Tag
    27.4 @@ -0,0 +1,1 @@
    27.5 +Nzasync-1_1_0
    28.1 new file mode 100644
    28.2 --- /dev/null
    28.3 +++ b/client/README.txt
    28.4 @@ -0,0 +1,3 @@
    28.5 +This directory is not a python package, and should not include an 
    28.6 +__init__.py.  It is the directory that should be inserted into the
    28.7 +PYTHONPATH in the process that runs the zasync client.
    29.1 new file mode 100644
    29.2 --- /dev/null
    29.3 +++ b/client/zasync/CVS/Entries
    29.4 @@ -0,0 +1,11 @@
    29.5 +/__init__.py/1.3/Sat Sep 17 03:26:07 2005//Tzasync-1_1_0
    29.6 +/client.py/1.12/Sat Sep 17 04:27:25 2005//Tzasync-1_1_0
    29.7 +/config.py/1.7/Sat Sep 17 04:27:25 2005//Tzasync-1_1_0
    29.8 +/example_start_script/1.3/Sat Nov 13 01:10:52 2004//Tzasync-1_1_0
    29.9 +/plugins.py/1.13/Sat Sep 17 04:27:25 2005//Tzasync-1_1_0
   29.10 +/plugins.txt/1.1/Mon Aug 29 14:36:31 2005//Tzasync-1_1_0
   29.11 +/schema.xml/1.5/Sat Sep 17 04:27:25 2005//Tzasync-1_1_0
   29.12 +/schema27.xml/1.1/Sat Sep 17 03:26:07 2005//Tzasync-1_1_0
   29.13 +/tests.txt/1.2/Sat Sep 17 03:26:07 2005//Tzasync-1_1_0
   29.14 +/zasync.conf/1.5/Sat Nov 13 01:10:52 2004//Tzasync-1_1_0
   29.15 +D
    30.1 new file mode 100644
    30.2 --- /dev/null
    30.3 +++ b/client/zasync/CVS/Repository
    30.4 @@ -0,0 +1,1 @@
    30.5 +Packages/zasync/client/zasync
    31.1 new file mode 100644
    31.2 --- /dev/null
    31.3 +++ b/client/zasync/CVS/Root
    31.4 @@ -0,0 +1,1 @@
    31.5 +:pserver:anonymous@cvs.zope.org:/cvs-repository
    32.1 new file mode 100644
    32.2 --- /dev/null
    32.3 +++ b/client/zasync/CVS/Tag
    32.4 @@ -0,0 +1,1 @@
    32.5 +Nzasync-1_1_0
    33.1 new file mode 100644
    33.2 --- /dev/null
    33.3 +++ b/client/zasync/__init__.py
    33.4 @@ -0,0 +1,5 @@
    33.5 +def run(config_file=None, version=2.8):
    33.6 +    from zasync import config
    33.7 +    conf = config.initialize(config_file, version)
    33.8 +    from zasync import client
    33.9 +    client.run(conf.target_path)
    34.1 new file mode 100644
    34.2 --- /dev/null
    34.3 +++ b/client/zasync/client.py
    34.4 @@ -0,0 +1,625 @@
    34.5 +##############################################################################
    34.6 +#
    34.7 +# Copyright (c) 2004 Zope Corporation and Contributors. All Rights Reserved.
    34.8 +#
    34.9 +# This software is subject to the provisions of the Zope Public License,
   34.10 +# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
   34.11 +# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
   34.12 +# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
   34.13 +# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
   34.14 +# FOR A PARTICULAR PURPOSE.
   34.15 +#
   34.16 +##############################################################################
   34.17 +"""Asynchronous call engine.
   34.18 +
   34.19 +$Id: client.py,v 1.12 2005/09/17 04:27:25 poster Exp $
   34.20 +"""
   34.21 +
   34.22 +# To do:
   34.23 +# - on an exit (keyboard or system), try to stop polling but finish
   34.24 +#   processing remaining zope deferreds, if possible, at least just timing 
   34.25 +#   out
   34.26 +# - try to modify to use twistd so tools such as manhole work easily
   34.27 +# - catch POSKeyErrors too; what the heck.
   34.28 +
   34.29 +# Debugging notes:
   34.30 +# strace -p pid on linux and ktrace -p pid (plus kdump) on os x/bsd should
   34.31 +# catch hanging processes.
   34.32 +# if I switch to using twistd then we can use manhole
   34.33 +
   34.34 +import sys, getopt, logging, urlparse, StringIO, datetime
   34.35 +from ZODB.POSException import ConflictError
   34.36 +from ZEO.Exceptions import ClientDisconnected
   34.37 +from ZPublisher.HTTPRequest import HTTPRequest
   34.38 +from ZPublisher.HTTPResponse import HTTPResponse
   34.39 +from ZPublisher.BaseRequest import RequestContainer
   34.40 +try:
   34.41 +    import transaction # Zope 2.8
   34.42 +except ImportError: # Zope 2.7
   34.43 +    class transaction(object):
   34.44 +        def commit(cls):
   34.45 +            get_transaction().commit()
   34.46 +        commit = classmethod(commit)
   34.47 +        def abort(cls):
   34.48 +            get_transaction().abort()
   34.49 +        abort = classmethod(abort)
   34.50 +        def begin(cls):
   34.51 +            get_transaction().begin()
   34.52 +        begin = classmethod(begin)
   34.53 +
   34.54 +from twisted.internet import reactor, defer
   34.55 +from twisted.python import failure
   34.56 +
   34.57 +from Products.zasync.manager import cleanFailure
   34.58 +
   34.59 +### the following values are set or modified by config:
   34.60 +
   34.61 +max_conflict_resolution_attempts = 5
   34.62 +initial_retry_delay = 5
   34.63 +retry_exponential_backoff = 1.1
   34.64 +max_total_retry = 3600 # change to total max retry, 0 means no max
   34.65 +verbose_traceback = False
   34.66 +
   34.67 +tool_path = None # this will hold the path to the Zope tool.  This variable
   34.68 +# is intentionally not used directly in this file, but that may change.
   34.69 +
   34.70 +app = None # this will hold the Zope application getter
   34.71 +
   34.72 +DB = None # this will hold the database
   34.73 +
   34.74 +plugins = {} # the plugins available
   34.75 +
   34.76 +### end configured
   34.77 +
   34.78 +server_retries = [] # calls that need to be resumed once the ZEO server is 
   34.79 +                    # back
   34.80 +
   34.81 +tool_retries = {} # calls that need to be resumed once a tool is back ({tool
   34.82 +                  # path: [calls]})
   34.83 +
   34.84 +active = {} # {zope deferred id: (original timeout, zasync deferred)} of calls 
   34.85 +            # currently in process
   34.86 +
   34.87 +server_retry_start = None
   34.88 +tool_retry_start = None
   34.89 +
   34.90 +chores = [] # list of (function, args, kwargs) to be performed regularly by
   34.91 +# housekeeping
   34.92 +
   34.93 +# helpers
   34.94 +
   34.95 +def housekeeping(interval=4):
   34.96 +    global chores
   34.97 +    log = logging.getLogger('zasync')
   34.98 +    log.debug('housekeeping: performing %d chores', len(chores))
   34.99 +    for (f, args, kwargs) in chores:
  34.100 +        reactor.callLater(0, f, *args, **kwargs)
  34.101 +    reactor.callLater(interval, housekeeping, interval)
  34.102 +
  34.103 +def getRequestApp(application):
  34.104 +    response = HTTPResponse(stdout=sys.stdout)
  34.105 +    request = HTTPRequest(
  34.106 +        sys.stdin,
  34.107 +        {'SERVER_NAME':'zasync_pseudo_server',
  34.108 +         'SERVER_PORT':'80',
  34.109 +         'REQUEST_METHOD':'GET'},
  34.110 +        response)
  34.111 +    return application.__of__(RequestContainer(REQUEST=request))
  34.112 +
  34.113 +def logException(msg='', log=None):
  34.114 +    global verbose_traceback
  34.115 +    if log is None:
  34.116 +        log = logging.getLogger('zasync')
  34.117 +    transaction.abort()
  34.118 +    msg = 'Logged Exception.  Transaction aborted.  %s' % msg
  34.119 +    if verbose_traceback:
  34.120 +        res = failure.Failure()
  34.121 +        out = StringIO.StringIO()
  34.122 +        res.printDetailedTraceback(out)
  34.123 +        out = out.getvalue()
  34.124 +        log.error("%s\n\n%s\n\n%s", msg, out, res.getErrorMessage())
  34.125 +    else:
  34.126 +        log.error(msg, exc_info=True)
  34.127 +
  34.128 +def scheduleServerRetry(call, *args, **kwargs):
  34.129 +    transaction.abort()
  34.130 +    global server_retries, initial_retry_delay, server_retry_start
  34.131 +    server_retries.append((call, args, kwargs))
  34.132 +    if len(server_retries) == 1:
  34.133 +        server_retry_start = datetime.datetime.now()
  34.134 +        logging.getLogger('zasync').critical(
  34.135 +            'disconnected from ZEO server; retrying.')
  34.136 +        # starting up a new backoff
  34.137 +        reactor.callLater(initial_retry_delay, 
  34.138 +                          retryServer, initial_retry_delay)
  34.139 +def is_connected():
  34.140 +    global DB
  34.141 +    storage = DB._storage
  34.142 +    call = getattr(storage, 'is_connected', None)
  34.143 +    if call is not None:
  34.144 +        return call()
  34.145 +    return True # we'll assume that non-ZEO storages (presumably only useful
  34.146 +    # for tests) are always connected
  34.147 +
  34.148 +def retryServer(delay):
  34.149 +    global server_retries, retry_exponential_backoff
  34.150 +    global server_retry_start, max_total_retry
  34.151 +    if is_connected():
  34.152 +        server_retry_start = None
  34.153 +        delay = 0
  34.154 +        interval = 0.05
  34.155 +        while server_retries:
  34.156 +            call, args, kwargs = server_retries.pop(0)
  34.157 +            reactor.callLater(delay, call, *args, **kwargs)
  34.158 +            delay += interval
  34.159 +        logging.getLogger('zasync').info('reconnected to ZEO server')
  34.160 +    else:
  34.161 +        delay = pow(delay, retry_exponential_backoff)
  34.162 +        diff = datetime.datetime.now() - server_retry_start
  34.163 +        seconds = diff.seconds + 86400 * diff.days
  34.164 +        if (max_total_retry and seconds + delay > max_total_retry):
  34.165 +            logging.getLogger('zasync').critical(
  34.166 +                'disconnected from ZEO server for %d seconds, and maximum '
  34.167 +                'total retry reached.  Stopping.', seconds)
  34.168 +            raise SystemExit('ZEO server unreachable')
  34.169 +        else:
  34.170 +            logging.getLogger('zasync').critical(
  34.171 +                'disconnected from ZEO server for %d seconds; retrying.', 
  34.172 +                seconds)
  34.173 +            reactor.callLater(delay, retryServer, delay)
  34.174 +
  34.175 +def scheduleToolRetry(path, call, *args, **kwargs):
  34.176 +    transaction.abort()
  34.177 +    global tool_retries, initial_retry_delay
  34.178 +    global tool_retry_start
  34.179 +    if path not in tool_retries:
  34.180 +        tool_retries[path] = [(call, args, kwargs)]
  34.181 +        logging.getLogger('zasync').critical(
  34.182 +            'can no longer find tool: /%s',
  34.183 +            '/'.join(path))
  34.184 +        tool_retry_start = datetime.datetime.now()
  34.185 +        reactor.callLater(initial_retry_delay,
  34.186 +                          retryTool, path, initial_retry_delay)
  34.187 +    else:
  34.188 +        tool_retries[path].append((call, args, kwargs))
  34.189 +
  34.190 +def retryTool(path, delay):
  34.191 +    global retry_exponential_backoff, max_total_retry
  34.192 +    global tool_retry_start, app, DB
  34.193 +    application = None
  34.194 +    try:
  34.195 +        try:
  34.196 +            try:
  34.197 +                sync = DB._storage.sync # important
  34.198 +            except AttributeError:
  34.199 +                pass
  34.200 +            else:
  34.201 +                sync()
  34.202 +            application = app()
  34.203 +            tool = application.unrestrictedTraverse(path)
  34.204 +            transaction.commit()
  34.205 +        except ConflictError:
  34.206 +            logging.getLogger('zasync').info(
  34.207 +                'Received ConflictError while trying to retry tool; retrying.', 
  34.208 +                exc_info=True)
  34.209 +            transaction.abort()
  34.210 +            reactor.callLater(delay, retryTool, path, delay)
  34.211 +        except ClientDisconnected:
  34.212 +            logging.getLogger('zasync').critical(
  34.213 +                'ZEO server disconnected while trying to retry tool')
  34.214 +            transaction.abort()
  34.215 +            scheduleServerRetry(retryTool, path, delay)
  34.216 +        except (AttributeError, LookupError):
  34.217 +            transaction.abort()
  34.218 +            delay = pow(delay, retry_exponential_backoff)
  34.219 +            diff = datetime.datetime.now() - tool_retry_start
  34.220 +            seconds = diff.seconds + 86400 * diff.days
  34.221 +            if (max_total_retry and seconds + delay > max_total_retry):
  34.222 +                logging.getLogger('zasync').critical(
  34.223 +                    'cannot find tool /%s for %d seconds, and maximum retry '
  34.224 +                    'delay reached.  Stopping.', '/'.join(path), seconds)
  34.225 +                raise SystemExit('tool not found')
  34.226 +                delay = max_retry_delay
  34.227 +            else:
  34.228 +                logging.getLogger('zasync').critical(
  34.229 +                    'cannot find tool for %d seconds; retrying.',
  34.230 +                    seconds)
  34.231 +                reactor.callLater(delay, retryTool, path, delay)
  34.232 +        else:
  34.233 +            global tool_retries
  34.234 +            retries = tool_retries.pop(path)
  34.235 +            delay = 0
  34.236 +            interval = 0.05
  34.237 +            # schedule all the past calls
  34.238 +            found = False
  34.239 +            for call, args, kwargs in retries:
  34.240 +                reactor.callLater(delay, call, *args, **kwargs)
  34.241 +                delay += interval
  34.242 +                if not found and call is _setPlugins:
  34.243 +                    found = True
  34.244 +            if not found:
  34.245 +                # make sure tool (possibly new) knows about my plugins
  34.246 +                setPlugins(path) # returns a deferred, which we ignore
  34.247 +            logging.getLogger('zasync').info(
  34.248 +                'tool /%s found again; recommencing calls.', '/'.join(path))
  34.249 +    finally:
  34.250 +        if application is not None:
  34.251 +            application._p_jar.close()
  34.252 +
  34.253 +def timeoutErrback(deferred):
  34.254 +    if not deferred.called:
  34.255 +        deferred.errback(defer.TimeoutError('Timed out.'))
  34.256 +
  34.257 +def cancelDelayedCall(value, call):
  34.258 +    if call.active():
  34.259 +        call.cancel()
  34.260 +    return value
  34.261 +
  34.262 +# pollZope schedules makeCall which schedules returnResult.  That's it, really.
  34.263 +def pollZope(path):
  34.264 +    "polls Zope to check for new calls, timeout changes, and expired calls"
  34.265 +    global active, app, DB, plugins
  34.266 +    application = None
  34.267 +    try:
  34.268 +        if is_connected():
  34.269 +            log = logging.getLogger('zasync')
  34.270 +            log.debug('polling %s', '/'.join(path))
  34.271 +            try:
  34.272 +                try:
  34.273 +                    sync = DB._storage.sync # important
  34.274 +                except AttributeError:
  34.275 +                    pass
  34.276 +                else:
  34.277 +                    sync()
  34.278 +                application = app()
  34.279 +                try:
  34.280 +                    tool = application.unrestrictedTraverse(path)
  34.281 +                except (AttributeError, LookupError):
  34.282 +                    scheduleToolRetry(path, pollZope, path)
  34.283 +                    return
  34.284 +                poll_interval = tool.poll_interval
  34.285 +                tool.heartbeat()
  34.286 +                delay = 0
  34.287 +                interval = 0.05
  34.288 +                for zopeDeferred in tool.acceptAll():
  34.289 +                    remainingSeconds = zopeDeferred.remainingSeconds()
  34.290 +                    if remainingSeconds <= 0:
  34.291 +                        # don't bother calling if it has already timed out
  34.292 +                        zopeDeferred.__of__(tool).errback(
  34.293 +                            failure.Failure(defer.TimeoutError('Timed out.')))
  34.294 +                        continue
  34.295 +                    name, args, kwargs = zopeDeferred.getSignature()
  34.296 +                    call_info = plugins.get(name)
  34.297 +                    if call_info is None:
  34.298 +                        zopeDeferred.__of__(tool).errback(
  34.299 +                            failure.Failure(KeyError(name)))
  34.300 +                        continue
  34.301 +                    log.debug('got call for plugin %s (args %r; kwargs %r)', 
  34.302 +                              name, args, kwargs)
  34.303 +                    zopeDeferredId = zopeDeferred.key
  34.304 +                    timeout = zopeDeferred.timeout
  34.305 +                    calltimeout = call_info['timeout']
  34.306 +                    if timeout is None or calltimeout < timeout:
  34.307 +                        zopeDeferred.timeout = calltimeout
  34.308 +                    reactor.callLater(
  34.309 +                        delay, makeCall, 
  34.310 +                        path, zopeDeferredId, name, args, kwargs, 
  34.311 +                        remainingSeconds, timeout)
  34.312 +                    delay += interval
  34.313 +                bad = []
  34.314 +                for zopeDeferredId, (oldTimeout, deferred) in active.items():
  34.315 +                    zopeDeferred = tool.getDeferred(zopeDeferredId)
  34.316 +                    if zopeDeferred is None:
  34.317 +                        bad.append(zopeDeferredId)
  34.318 +                    elif (zopeDeferred.timeout < oldTimeout and 
  34.319 +                        deferred.timeoutCall.active()):
  34.320 +                        deferred.timeoutCall.reset(
  34.321 +                            max(zopeDeferred.remainingSeconds(), 0))
  34.322 +                for zopeDeferredId in bad:
  34.323 +                    del active[bad]
  34.324 +                transaction.commit()
  34.325 +            except ClientDisconnected:
  34.326 +                scheduleServerRetry(pollZope, path)
  34.327 +                return
  34.328 +            except ConflictError:
  34.329 +                transaction.abort()
  34.330 +                log.warning('ZODB ConflictError in pollZope.  Never give up.', 
  34.331 +                            exc_info=True)
  34.332 +                reactor.callLater(1, pollZope, path) # never give up
  34.333 +                return
  34.334 +            except (KeyboardInterrupt, SystemExit):
  34.335 +                raise
  34.336 +            except:
  34.337 +                logException(log=log)
  34.338 +            # XXX if we want the heartbeat interval to be more reliable then
  34.339 +            # we need more care
  34.340 +            reactor.callLater(poll_interval, pollZope, path)
  34.341 +        else:
  34.342 +            scheduleServerRetry(pollZope, path)
  34.343 +    finally:
  34.344 +        if application is not None:
  34.345 +            application._p_jar.close()
  34.346 +
  34.347 +def returnResult(value, path, zopeDeferredId, error=False, count=0):
  34.348 +    global active, app
  34.349 +    if isinstance(value, failure.Failure):
  34.350 +        cleanFailure(value)
  34.351 +    log = logging.getLogger('zasync')
  34.352 +    log.debug('returnResult got value for %s (%s):\n\n%r', 
  34.353 +              zopeDeferredId, path, value)
  34.354 +    try:
  34.355 +        del active[zopeDeferredId]
  34.356 +    except KeyError:
  34.357 +        pass
  34.358 +    application = None
  34.359 +    try:
  34.360 +        if is_connected():
  34.361 +            try:
  34.362 +                application = app()
  34.363 +                try:
  34.364 +                    tool = getRequestApp(application).unrestrictedTraverse(path)
  34.365 +                except (AttributeError, LookupError):
  34.366 +                    scheduleToolRetry(
  34.367 +                        path, returnResult, value, path, zopeDeferredId, 
  34.368 +                        error, count)
  34.369 +                    return value
  34.370 +                zopeDeferred = tool.getDeferred(zopeDeferredId)
  34.371 +                if zopeDeferred is None:
  34.372 +                    return value
  34.373 +                if error:
  34.374 +                    call = zopeDeferred.errback
  34.375 +                else:
  34.376 +                    call = zopeDeferred.callback
  34.377 +                res = call(value)
  34.378 +                transaction.commit()
  34.379 +            except ConflictError:
  34.380 +                log.warning('ZODB ConflictError in returnResult', exc_info=True)
  34.381 +                transaction.abort()
  34.382 +                if count < max_conflict_resolution_attempts:
  34.383 +                    reactor.callLater(
  34.384 +                        count, returnResult, value, path, zopeDeferredId, 
  34.385 +                        error, count+1)
  34.386 +                else:
  34.387 +                    res = failure.Failure()
  34.388 +                    out = StringIO.StringIO()
  34.389 +                    res.printDetailedTraceback(out)
  34.390 +                    log.error(
  34.391 +                        'Too many ConflictErrors in returnResult: '
  34.392 +                        'giving up.\n\n%s', out.getvalue())
  34.393 +            except ClientDisconnected:
  34.394 +                scheduleServerRetry(
  34.395 +                    returnResult, value, path, zopeDeferredId, 
  34.396 +                    error, count)
  34.397 +                res = value
  34.398 +            except (KeyboardInterrupt, SystemExit):
  34.399 +                raise
  34.400 +            except: # give up.  Looks like a bug.  Log should help fix it.
  34.401 +                logException('Exception from Zope.', log=log)
  34.402 +                res = failure.Failure()
  34.403 +            return res
  34.404 +        else:
  34.405 +            scheduleServerRetry(
  34.406 +                returnResult, value, path, zopeDeferredId, error, count)
  34.407 +    finally:
  34.408 +        if application is not None:
  34.409 +            application._p_jar.close()
  34.410 +    return res
  34.411 +
  34.412 +def makeCall(path, zopeDeferredId, name, args, kwargs, remainingSeconds,
  34.413 +             timeout, returnResult=returnResult, count=0):
  34.414 +    # zopeDeferredId is an id of a deferred in the asynchronous tool
  34.415 +    global plugins
  34.416 +    log = logging.getLogger('zasync')
  34.417 +    log.debug('makeCall called for %s (%s)',
  34.418 +              zopeDeferredId, path)
  34.419 +    if is_connected():
  34.420 +        try:
  34.421 +            call_info = plugins[name]
  34.422 +            call = call_info['callable']
  34.423 +        except KeyError:
  34.424 +            res = failure.Failure()
  34.425 +            log.error('Called plugin %s does not exist', name)
  34.426 +        else:
  34.427 +            log.debug('called %s', name)
  34.428 +            try:
  34.429 +                if call_info['zope_aware']:
  34.430 +                    res = call((path, zopeDeferredId), *args, **kwargs)
  34.431 +                else:
  34.432 +                    res = call(*args, **kwargs)
  34.433 +                transaction.commit() # just in case plugin touched Zope:
  34.434 +                # please do not!  Use zope_exec, or follow that pattern...
  34.435 +                # If you do, and you start a conflict error, it may throw
  34.436 +                # the timeout check off.  Just don't, and regard this as 
  34.437 +                # paranoia.
  34.438 +            except ConflictError:
  34.439 +                log.warning('ZODB ConflictError in makeCall', exc_info=True)
  34.440 +                transaction.abort()
  34.441 +                if count < max_conflict_resolution_attempts:
  34.442 +                    reactor.callLater(
  34.443 +                        count, makeCall, path, zopeDeferredId, 
  34.444 +                        name, args, kwargs, remainingSeconds, timeout, 
  34.445 +                        returnResult, count+1)
  34.446 +                    return
  34.447 +                else:
  34.448 +                    res = failure.Failure()
  34.449 +                    out = StringIO.StringIO()
  34.450 +                    res.printDetailedTraceback(out)
  34.451 +                    log.error(
  34.452 +                        'Too many ConflictErrors in makeCall: '
  34.453 +                        'giving up.\n\n%s', out.getvalue())
  34.454 +                    # now we use that res (Failure)
  34.455 +            except ClientDisconnected:
  34.456 +                scheduleServerRetry(
  34.457 +                    makeCall, path, zopeDeferredId, name, args, kwargs, 
  34.458 +                    remainingSeconds, timeout, returnResult, count)
  34.459 +                return
  34.460 +            except (KeyboardInterrupt, SystemExit):
  34.461 +                raise
  34.462 +            except:
  34.463 +                transaction.abort()
  34.464 +                res = failure.Failure()
  34.465 +        if isinstance(res, defer.Deferred):
  34.466 +            res.timeoutCall = reactor.callLater(
  34.467 +                max(remainingSeconds, 0), 
  34.468 +                timeoutErrback, res)
  34.469 +            res.addBoth(cancelDelayedCall, res.timeoutCall)
  34.470 +            active[zopeDeferredId] = (timeout, res)
  34.471 +            res.addCallbacks(
  34.472 +                returnResult, returnResult,
  34.473 +                callbackArgs=(path, zopeDeferredId),
  34.474 +                errbackArgs=(path, zopeDeferredId, True))
  34.475 +        elif isinstance(res, failure.Failure):
  34.476 +            cleanFailure(res)
  34.477 +            reactor.callLater(0, returnResult, res, path, zopeDeferredId, True)
  34.478 +        else:
  34.479 +            reactor.callLater(0, returnResult, res, path, zopeDeferredId)
  34.480 +    else:
  34.481 +        scheduleServerRetry(
  34.482 +            makeCall, path, zopeDeferredId, name, args, kwargs, 
  34.483 +            remainingSeconds, timeout, returnResult, count)
  34.484 +
  34.485 +def stop(ignored=None):
  34.486 +    raise SystemExit()
  34.487 +
  34.488 +def setPlugins(path):
  34.489 +    d = defer.Deferred()
  34.490 +    reactor.callLater(0, _setPlugins, d, path)
  34.491 +    return d
  34.492 +
  34.493 +def _setPlugins(deferred, path, count=0):
  34.494 +    global app, plugins, max_conflict_resolution_attempts
  34.495 +    log = logging.getLogger('zasync')
  34.496 +    application = None
  34.497 +    try:
  34.498 +        try:
  34.499 +            application = app()
  34.500 +            try:
  34.501 +                tool = application.unrestrictedTraverse(tool_path)
  34.502 +            except (AttributeError, LookupError):
  34.503 +                scheduleToolRetry(path, _setPlugins, deferred, path, count)
  34.504 +                return
  34.505 +            else:
  34.506 +                tool.setPlugins(
  34.507 +                    [(n, p['description']) for n, p in plugins.items()])
  34.508 +                transaction.commit()
  34.509 +        except ConflictError:
  34.510 +            transaction.abort()
  34.511 +            if count == max_conflict_resolution_attempts-1:
  34.512 +                log.critical(
  34.513 +                    "Too many conflicts trying to setPlugins!", exc_info=True)
  34.514 +                deferred.errback(failure.Failure())
  34.515 +            else:
  34.516 +                count += 1
  34.517 +                log.info("Conflict error %d trying to setPlugins", count)
  34.518 +                reactor.callLater(count, _setPlugins, deferred, path, count)
  34.519 +        except ClientDisconnected:
  34.520 +            scheduleServerRetry(_setPlugins, deferred, path, count)
  34.521 +            return
  34.522 +        except (KeyboardInterrupt, SystemExit):
  34.523 +            raise
  34.524 +        except:
  34.525 +            transaction.abort()
  34.526 +            log.critical("Unexpected error: probably serious", exc_info=True)
  34.527 +            deferred.errback(failure.Failure())
  34.528 +        else:
  34.529 +            deferred.callback(path)
  34.530 +    finally:
  34.531 +        if application is not None:
  34.532 +            application._p_jar.close()
  34.533 +
  34.534 +def handlePastCalls(path, count=0):
  34.535 +    global app
  34.536 +    log = logging.getLogger('zasync')
  34.537 +    application = None
  34.538 +    try:
  34.539 +        try:
  34.540 +            application = app()
  34.541 +            try:
  34.542 +                tool = application.unrestrictedTraverse(path)
  34.543 +            except (AttributeError, LookupError):
  34.544 +                scheduleToolRetry(path, handlePastCalls, path, count)
  34.545 +                return
  34.546 +            else:
  34.547 +                info = [(d.getSignature(), d.key, 
  34.548 +                         d.remainingSeconds(), d.timeout) for d in 
  34.549 +                        tool.getAcceptedCalls()]
  34.550 +                transaction.commit()
  34.551 +        except ConflictError:
  34.552 +            transaction.abort()
  34.553 +            if count==max_conflict_resolution_attempts-1:
  34.554 +                log.critical(
  34.555 +                    "Too many conflicts trying to handlePastCalls!", 
  34.556 +                    exc_info=True)
  34.557 +                raise SystemExit("Too many conflicts trying to handlePastCalls")
  34.558 +            else:
  34.559 +                count+=1
  34.560 +                log.info("Conflict error %d trying to handlePastCalls", count)
  34.561 +                reactor.callLater(count, handlePastCalls, path, count)
  34.562 +        except ClientDisconnected:
  34.563 +            scheduleServerRetry(handlePastCalls, path, count)
  34.564 +            return
  34.565 +        except (KeyboardInterrupt, SystemExit):
  34.566 +            raise
  34.567 +        except:
  34.568 +            transaction.abort()
  34.569 +            log.critical("Unexpected error: probably serious", exc_info=True)
  34.570 +            deferred.errback(failure.Failure())
  34.571 +        else:
  34.572 +            for (name, args, kwargs), key, remainingSeconds, timeout in info:
  34.573 +                try:
  34.574 +                    call_info = plugins[name]
  34.575 +                except KeyError:
  34.576 +                    retry = False
  34.577 +                else:
  34.578 +                    retry = call_info.get('retry', False)
  34.579 +                if retry and remainingSeconds > 0:
  34.580 +                    reactor.callLater(
  34.581 +                        0, makeCall, tool_path, key, name, args, kwargs,
  34.582 +                        remainingSeconds, timeout)
  34.583 +                else:
  34.584 +                    reactor.callLater(
  34.585 +                        0,
  34.586 +                        returnResult, 
  34.587 +                        failure.Failure(
  34.588 +                            defer.TimeoutError(
  34.589 +                                'zasync was disconnected (now reconnected)')),
  34.590 +                        tool_path,
  34.591 +                        key,
  34.592 +                        error=True)
  34.593 +            log.debug('scheduling first pollZope')
  34.594 +            reactor.callLater(0, pollZope, path)
  34.595 +            log.debug('scheduling first housekeeping')
  34.596 +            reactor.callLater(0, housekeeping)
  34.597 +    finally:
  34.598 +        if application is not None:
  34.599 +            application._p_jar.close()
  34.600 +
  34.601 +def run(path=None):
  34.602 +    # to be called after config.initialize
  34.603 +    global tool_path, app
  34.604 +    if path is None:
  34.605 +        path = tool_path
  34.606 +    tool_path = path = tuple(path)
  34.607 +    d = setPlugins(path)
  34.608 +    d.addCallbacks(handlePastCalls, stop)
  34.609 +    d.addErrback(stop)
  34.610 +    log = logging.getLogger('zasync')
  34.611 +    log.debug('beginning reactor')
  34.612 +    try:
  34.613 +        reactor.run()
  34.614 +    finally:
  34.615 +        application = None
  34.616 +        try:
  34.617 +            transaction.abort()
  34.618 +            application = app()
  34.619 +            try:
  34.620 +                tool = application.unrestrictedTraverse(tool_path)
  34.621 +                tool.setPlugins(())
  34.622 +                transaction.commit()
  34.623 +            except (AttributeError, KeyError, ConflictError, ClientDisconnected):
  34.624 +                transaction.abort()
  34.625 +        finally:
  34.626 +            if application is not None:
  34.627 +                application._p_jar.close()
  34.628 +            logging.getLogger('zasync').critical('Shutting down')
  34.629 +            logging.shutdown()
    35.1 new file mode 100644
    35.2 --- /dev/null
    35.3 +++ b/client/zasync/config.py
    35.4 @@ -0,0 +1,309 @@
    35.5 +##############################################################################
    35.6 +#
    35.7 +# Copyright (c) 2004 Zope Corporation and Contributors. All Rights Reserved.
    35.8 +#
    35.9 +# This software is subject to the provisions of the Zope Public License,
   35.10 +# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
   35.11 +# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
   35.12 +# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
   35.13 +# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
   35.14 +# FOR A PARTICULAR PURPOSE.
   35.15 +#
   35.16 +##############################################################################
   35.17 +"""zconfig helpers
   35.18 +
   35.19 +$Id: config.py,v 1.7 2005/09/17 04:27:25 poster Exp $
   35.20 +"""
   35.21 +
   35.22 +import os, sys, types, traceback, re, sets
   35.23 +import ZConfig
   35.24 +
   35.25 +specified_version = None
   35.26 +
   35.27 +def root_handler(config):
   35.28 +    """Set up the environment and generally do almost everything;
   35.29 +    this was combined from datatypes and configs in Zope gradually as
   35.30 +    necessary elements emerged; it should probably be separated again
   35.31 +    at some point in the future."""
   35.32 +    
   35.33 +    # This is in two parts: set up Zope, then set up zasync.  Unsurprisingly,
   35.34 +    # it takes a lot more to set up Zope.
   35.35 +    
   35.36 +    # ...but first we'll set up logging
   35.37 +    import zLOG 
   35.38 +    zLOG._call_initialize = 0
   35.39 +    config.eventlog()
   35.40 +    config.zasync()
   35.41 +    config.zasync_plugins.name="zasync.plugins"
   35.42 +    config.zasync_plugins()
   35.43 +    
   35.44 +    ###############
   35.45 +    # Set up Zope #
   35.46 +    ###############
   35.47 +
   35.48 +    # Set environment variables
   35.49 +    if config.environment:
   35.50 +        for k,v in config.environment.items():
   35.51 +            os.environ[k] = v
   35.52 +    
   35.53 +    # set up Zope as minimally as possible while still getting it to work
   35.54 +    
   35.55 +    # Add directories to the pythonpath; always insert instancehome/lib/python
   35.56 +    instancelib = os.path.join(config.instancehome, 'lib', 'python')
   35.57 +    if instancelib not in config.path:
   35.58 +        config.path.append(instancelib)
   35.59 +    path = config.path[:]
   35.60 +    path.reverse()
   35.61 +    for dir in path:
   35.62 +        sys.path.insert(0, dir)
   35.63 +
   35.64 +    # Add any product directories not already in Products.__path__.
   35.65 +    # Directories are added in the order they are mentioned
   35.66 +    # Always insert instancehome.Products
   35.67 +
   35.68 +    instanceprod = os.path.join(config.instancehome, 'Products')
   35.69 +    if instanceprod not in config.products:
   35.70 +        config.products.append(instanceprod)
   35.71 +    
   35.72 +    import Products
   35.73 +    L = []
   35.74 +    for d in config.products + Products.__path__:
   35.75 +        if d not in L:
   35.76 +            L.append(d)
   35.77 +    Products.__path__[:] = L
   35.78 +    
   35.79 +    import App, App.config
   35.80 +    par = os.path.dirname
   35.81 +    config.softwarehome = par(par(os.path.abspath(App.__file__)))
   35.82 +    config.zopehome = par(par(d))
   35.83 +    config.debug_mode = False # configurable?
   35.84 +    config.enable_product_installation = False
   35.85 +    if getattr(config, 'clienthome', None) is None:
   35.86 +        config.clienthome = os.path.join(config.instancehome, 'var')
   35.87 +    App.config.setConfiguration(config)
   35.88 +    
   35.89 +    import AccessControl
   35.90 +    AccessControl.setImplementation(
   35.91 +        config.security_policy_implementation)
   35.92 +    if hasattr(config, 'verbose_security'): # Zope 2.8
   35.93 +        AccessControl.setDefaultBehaviors(
   35.94 +            not config.skip_ownership_checking,
   35.95 +            not config.skip_authentication_checking,
   35.96 +            config.verbose_security)
   35.97 +    else: # 2.7
   35.98 +        AccessControl.setDefaultBehaviors(
   35.99 +            not config.skip_ownership_checking,
  35.100 +            not config.skip_authentication_checking)
  35.101 +    
  35.102 +    import OFS.Application
  35.103 +    OFS.Application.import_products()
  35.104 +    
  35.105 +    try: # Zope 2.8
  35.106 +        from App import ZApplication
  35.107 +    except ImportError: # 2.7
  35.108 +        from ZODB import ZApplication
  35.109 +    from AccessControl.SecurityManagement import getSecurityManager
  35.110 +    from AccessControl.SecurityManagement import newSecurityManager
  35.111 +    from AccessControl.SecurityManagement import noSecurityManager
  35.112 +    import AccessControl.User
  35.113 +    import Globals
  35.114 +    try: # Zope 2.8
  35.115 +        from Zope2 import ClassFactory
  35.116 +    except ImportError:
  35.117 +        from Zope import ClassFactory
  35.118 +
  35.119 +    Globals.DatabaseVersion='3'
  35.120 +    
  35.121 +    DB = config.dbtab.getDatabase('/', is_root=1)
  35.122 +    try: # Zope 2.7
  35.123 +        DB.setClassFactory(ClassFactory.ClassFactory)
  35.124 +    except AttributeError: # 2.8
  35.125 +        DB.classFactory = ClassFactory.ClassFactory
  35.126 +
  35.127 +    # "Log on" as system user
  35.128 +    newSecurityManager(None, AccessControl.User.system)
  35.129 +
  35.130 +    # Set up the "app" object that automagically opens
  35.131 +    # connections
  35.132 +    app = ZApplication.ZApplicationWrapper(
  35.133 +        DB, 'Application', OFS.Application.Application, (),
  35.134 +        Globals.VersionNameName)
  35.135 +
  35.136 +    # Initialize the app object
  35.137 +    application = app()
  35.138 +    OFS.Application.initialize(application)
  35.139 +    # ...application is closed below
  35.140 +
  35.141 +    # "Log off" as system user
  35.142 +    noSecurityManager()
  35.143 +    
  35.144 +    #################
  35.145 +    # Set up zasync #
  35.146 +    #################
  35.147 +    
  35.148 +    from zasync import client
  35.149 +    
  35.150 +    client.DB = DB
  35.151 +    client.app = app
  35.152 +    
  35.153 +    path = client.tool_path = tuple(filter(None, config.target.split('/')))
  35.154 +    config.target_path = path
  35.155 +    try:
  35.156 +        tool = application.unrestrictedTraverse(path)
  35.157 +    except (AttributeError, LookupError):
  35.158 +        raise ZConfig.ConfigurationError(
  35.159 +            "target could not be found: %s" % config.target)
  35.160 +    
  35.161 +    application._p_jar.close()
  35.162 +    
  35.163 +    # configure retry behavior
  35.164 +    client.max_conflict_resolution_attempts = (
  35.165 +        max(config.max_conflict_resolution_attempts, 0))
  35.166 +    client.initial_retry_delay = (
  35.167 +        max(config.initial_retry_delay, 1))
  35.168 +    client.retry_exponential_backoff = (
  35.169 +        max(config.retry_exponential_backoff, 1))
  35.170 +    client.max_total_retry = max(config.max_total_retry, 0)
  35.171 +    
  35.172 +    # configure traceback verbosity
  35.173 +    client.verbose_traceback = config.verbose_traceback
  35.174 +    
  35.175 +    # load in plugins.  We don't want to calculate the callables until 
  35.176 +    # this root handler because the python environment is not set up until here.
  35.177 +    for plugin in config.plugins:
  35.178 +        # based off of 
  35.179 +        # http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/223972
  35.180 +        fullFuncName = plugin['handler']
  35.181 +        lastDot = fullFuncName.rfind(u".")
  35.182 +        if lastDot==-1:
  35.183 +            raise ZConfig.ConfigurationError(
  35.184 +                'plugin not found (code expects to find a dot in path): %s' %
  35.185 +                fullFuncName)
  35.186 +        funcName = fullFuncName[lastDot + 1:]
  35.187 +        modulePath = fullFuncName[:lastDot]
  35.188 +        try:
  35.189 +            aMod = sys.modules[modulePath]
  35.190 +            if not isinstance(aMod, types.ModuleType):
  35.191 +                raise KeyError
  35.192 +        except KeyError:
  35.193 +            try:
  35.194 +                aMod = __import__(modulePath, globals(), locals(), [''])
  35.195 +            except ImportError:
  35.196 +                tb = ''.join(traceback.format_exception(*sys.exc_info()))
  35.197 +                raise ZConfig.ConfigurationError(
  35.198 +                    'error importing %s: \n%s' % (fullFuncName, tb))
  35.199 +            sys.modules[modulePath] = aMod
  35.200 +        func = getattr(aMod, funcName)
  35.201 +        if not callable(func):
  35.202 +            raise ZConfig.ConfigurationError(
  35.203 +                'identified plugin must be callable: %s' % fullFuncName)
  35.204 +        plugin['callable'] = func
  35.205 +        plugin['description'] = (
  35.206 +            plugin.get('description') or func.__doc__ or func.__name__)
  35.207 +        client.plugins[plugin['name']] = plugin
  35.208 +
  35.209 +# simple datatypes
  35.210 +def plugin(section):
  35.211 +    return {
  35.212 +        'name':section.getSectionName(),
  35.213 +        'handler':section.handler,
  35.214 +        'description':getattr(section, 'description', None),
  35.215 +        'retry':section.retry,
  35.216 +        'timeout':max(getattr(section, 'timeout', sys.maxint), 10),
  35.217 +        'zope_aware':section.zope_aware,
  35.218 +        }
  35.219 +
  35.220 +def root_section(section):
  35.221 +    if not section.databases:
  35.222 +        section.databases = []
  35.223 +    mount_factories = {} # { name -> factory}
  35.224 +    mount_points = {} # { virtual path -> name }
  35.225 +    dup_err = ('Invalid configuration: ZODB databases named "%s" and "%s" are '
  35.226 +               'both configured to use the same mount point, named "%s"')
  35.227 +    for database in section.databases:
  35.228 +        points = database.getVirtualMountPaths()
  35.229 +        name = database.config.getSectionName()
  35.230 +        mount_factories[name] = database
  35.231 +        for point in points:
  35.232 +            if mount_points.has_key(point):
  35.233 +                raise ConfigurationError(dup_err % (mount_points[point],
  35.234 +                                                    name, point))
  35.235 +            mount_points[point] = name
  35.236 +    from DBTab.DBTab import DBTab
  35.237 +    section.dbtab = DBTab(mount_factories, mount_points)
  35.238 +    
  35.239 +    s = sets.Set()
  35.240 +    for plugin in section.plugins:
  35.241 +        name = plugin['name']
  35.242 +        if name in s:
  35.243 +            raise ConfigurationError(
  35.244 +                'plugin name duplicated: %s' % name)
  35.245 +            s.add(name)
  35.246 +    
  35.247 +    return section
  35.248 +
  35.249 +_ident_re = "[_a-zA-Z][_a-zA-Z0-9]*"
  35.250 +class DottedNameFunctionConversion(
  35.251 +    ZConfig.datatypes.RegularExpressionConversion):
  35.252 +    
  35.253 +    def __init__(self):
  35.254 +        ZConfig.datatypes.RegularExpressionConversion.__init__(
  35.255 +            self, r"%s(?:\.%s)+" % (_ident_re, _ident_re))
  35.256 +DottedNameFunctionConversion = DottedNameFunctionConversion()
  35.257 +
  35.258 +class LoggerFactory:
  35.259 +    """
  35.260 +    A factory used to create loggers while delaying actual logger
  35.261 +    instance construction.  We need to do this because we may want to
  35.262 +    reference a logger before actually instantiating it (for example,
  35.263 +    to allow the app time to set an effective user).  An instance of
  35.264 +    this wrapper is a callable which, when called, returns a logger
  35.265 +    object.
  35.266 +    """
  35.267 +    def __init__(self, section):
  35.268 +        self.name = section.getSectionName()
  35.269 +        self.level = section.level
  35.270 +        self.handler_factories = section.handlers
  35.271 +        self.propagate = section.propagate
  35.272 +        self.resolved = None
  35.273 +        self.section = section
  35.274 +
  35.275 +    def __call__(self):
  35.276 +        if self.resolved is None:
  35.277 +            # set the logger up
  35.278 +            import logging
  35.279 +            logger = logging.getLogger(self.name)
  35.280 +            logger.handlers = []
  35.281 +            logger.propagate = self.propagate
  35.282 +            logger.setLevel(self.level)
  35.283 +            for handler_factory in self.handler_factories:
  35.284 +                handler = handler_factory()
  35.285 +                logger.addHandler(handler)
  35.286 +            self.resolved = logger
  35.287 +        return self.resolved
  35.288 +
  35.289 +### the circus master that runs the configuration
  35.290 +
  35.291 +def initialize(conffile=None, version=2.8):
  35.292 +    # I don't think we can sniff the Zope version safely yet.  If I were not in
  35.293 +    # a rush, I might try, but an explicit version argument will do the trick.
  35.294 +    global specified_version
  35.295 +    if version < 2.7 or version >= 2.9:
  35.296 +        raise RuntimeError('zasync only supports Zope 2.7.x and Zope 2.8.x')
  35.297 +    mydir = os.path.dirname(__file__)
  35.298 +    specified_version = version
  35.299 +    if version < 2.8:
  35.300 +        schema = 'schema27.xml'
  35.301 +    else:
  35.302 +        schema = 'schema.xml'
  35.303 +    try:
  35.304 +        schema = ZConfig.loadSchema(os.path.join(mydir, schema))
  35.305 +    except:
  35.306 +        if version>=2.8:
  35.307 +            print ("You might be running Zope 2.7: "
  35.308 +                   "use 'version=2.7' in run arguments")
  35.309 +        raise
  35.310 +    if conffile is None: conffile = sys.argv[1]
  35.311 +    conf, handler = ZConfig.loadConfig(schema, conffile)
  35.312 +    handler({'root_handler':root_handler})
  35.313 +    return conf
    36.1 new file mode 100755
    36.2 --- /dev/null
    36.3 +++ b/client/zasync/example_start_script
    36.4 @@ -0,0 +1,14 @@
    36.5 +#! /bin/sh
    36.6 +
    36.7 +BASE_DIR='/Users/gary/dev/my-project'
    36.8 +PYTHON="${BASE_DIR}/bin/python"
    36.9 +ZOPE_HOME="${BASE_DIR}/opt/Zope2"
   36.10 +CONFIG_FILE="${BASE_DIR}/etc/zasync.conf"
   36.11 +SOFTWARE_HOME="${BASE_DIR}/opt/Zope2/lib/python"
   36.12 +CLIENT_HOME="${BASE_DIR}/opt/zasync/client"
   36.13 +ZOPE_LIB_PYTHON="${BASE_DIR}/var/zope/lib/python"
   36.14 +PYTHONPATH="${CLIENT_HOME}:${ZOPE_LIB_PYTHON}:${SOFTWARE_HOME}"
   36.15 +export PYTHONPATH
   36.16 +
   36.17 +exec $PYTHON -c "from zasync import run;run('$CONFIG_FILE')"
   36.18 +
    37.1 new file mode 100644
    37.2 --- /dev/null
    37.3 +++ b/client/zasync/plugins.py
    37.4 @@ -0,0 +1,778 @@
    37.5 +##############################################################################
    37.6 +#
    37.7 +# Copyright (c) 2004 Zope Corporation and Contributors. All Rights Reserved.
    37.8 +#
    37.9 +# This software is subject to the provisions of the Zope Public License,
   37.10 +# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
   37.11 +# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
   37.12 +# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
   37.13 +# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
   37.14 +# FOR A PARTICULAR PURPOSE.
   37.15 +#
   37.16 +##############################################################################
   37.17 +"""zasync plugins
   37.18 +
   37.19 +$Id: plugins.py,v 1.13 2005/09/17 04:27:25 poster Exp $
   37.20 +"""
   37.21 +
   37.22 +import logging
   37.23 +from twisted.python import failure
   37.24 +from twisted.internet import reactor, defer
   37.25 +from zasync import client
   37.26 +try:
   37.27 +    import transaction
   37.28 +except ImportError:
   37.29 +    from zasync.client import transaction # this is intended to let us write the
   37.30 +    # backwards compatibility hack only once
   37.31 +
   37.32 +#### simple schedule
   37.33 +
   37.34 +def schedule(seconds):
   37.35 +    """proof of concept and "Hello World"; use to fire your callbacks after 
   37.36 +    seconds (approximately).  zope_exec is better for potentially 
   37.37 +    expensive tasks because it is cancellable (can time out) and you can guage
   37.38 +    better if the expensive task has started.  You might schedule a zope_exec 
   37.39 +    within a schedule callback, if you needed a scheduled expensive task."""
   37.40 +    d = defer.Deferred()
   37.41 +    reactor.callLater(seconds, d.callback, seconds)
   37.42 +    return d
   37.43 +
   37.44 +#### aggegate plugins
   37.45 +
   37.46 +def aggregatePlugins(zopeDeferredTuple, *calls):
   37.47 +    # XXX remove timeout feature; use the schedule plugin!
   37.48 +    # XXX also? instead? incorporate easy and introspectable chaining into
   37.49 +    # zopeDeferreds
   37.50 +    """aggregate calls to other plugins.  each call may either be a number
   37.51 +    (integer or float) of seconds to pause, or a tuple of (plugin name, args 
   37.52 +    tuple, kwargs dict).  For example, 
   37.53 +    
   37.54 +    aggregatePlugins(
   37.55 +        ('zope_exec', ('/my_site', 'home/my_script'), {}),
   37.56 +        5,
   37.57 +        ('zope_exec', ('/my_site', 'home/my_other_script'), {}))
   37.58 +    
   37.59 +    would ask zope_exec to call my_script in one transaction, wait 5 seconds,
   37.60 +    and then call my_other_script.
   37.61 +    
   37.62 +    If any of the plugins fail, the failure is returned without proceeding 
   37.63 +    further down the remaining calls.  The failure is annotated with a list of
   37.64 +    the completed calls ('completed_calls'), the call active during the failure
   37.65 +    ('active_call'), and remaining calls('remaining_calls').
   37.66 +    """
   37.67 +    thunkmaker = AggregateThunkMaker(zopeDeferredTuple, *calls)
   37.68 +    reactor.callLater(0, thunkmaker.makeCall)
   37.69 +    return thunkmaker.deferred
   37.70 +
   37.71 +class AggregateThunkMaker(object):
   37.72 +    
   37.73 +    def __init__(self, zopeDeferredTuple, *calls):
   37.74 +        self.deferred = defer.Deferred()
   37.75 +        self.pending = list(calls)
   37.76 +        self.completed = []
   37.77 +        self.begun = None
   37.78 +        self.results = []
   37.79 +        self.path, self.zopeDeferredId = zopeDeferredTuple
   37.80 +
   37.81 +    def makeCall(self):
   37.82 +        if not self.pending:
   37.83 +            self.deferred.callback(self.results)
   37.84 +        else:
   37.85 +            if self.begun is not None:
   37.86 +                self.completed.append(self.begun)
   37.87 +            task = self.pending.pop(0)
   37.88 +            self.begun = task
   37.89 +            if isinstance(task, (int, float)):
   37.90 +                reactor.callLater(task, self.makeCall)
   37.91 +            else:
   37.92 +                try:
   37.93 +                    plugin, args, kwargs = task
   37.94 +                except (ValueError, TypeError):
   37.95 +                    self.deferred.errback(failure.Failure())
   37.96 +                else:
   37.97 +                    try:
   37.98 +                        call_info = client.plugins[plugin]
   37.99 +                    except KeyError:
  37.100 +                        self.deferred.errback(failure.Failure())
  37.101 +                    else:
  37.102 +                        timeout = call_info["timeout"]
  37.103 +                        remainingSeconds = client.active[self.zopeDeferredId][0]
  37.104 +                        client.makeCall(
  37.105 +                            self.path, self.zopeDeferredId, plugin, args, kwargs, 
  37.106 +                            remainingSeconds, timeout, self.returnResult)
  37.107 +            
  37.108 +    def returnResult(self, value, path, zopeDeferredId, error=False):
  37.109 +        if error:
  37.110 +            value.completed_calls = tuple(self.completed)
  37.111 +            value.active_call = self.begun
  37.112 +            value.remaining_calls = tuple(self.pending)
  37.113 +            self.deferred.errback(value)
  37.114 +        else:
  37.115 +            self.results.append(value)
  37.116 +            reactor.callLater(0, self.makeCall)
  37.117 +
  37.118 +#### LDAP, protected (with SSL) and unprotected
  37.119 +
  37.120 +try:
  37.121 +    import ldap
  37.122 +except ImportError:
  37.123 +    pass
  37.124 +else:
  37.125 +    def query_unprotected_ldap(uri, base, scope, filterstr, attrlist=None): 
  37.126 +        """Query unprotected ldap (i.e., not ldaps).  arguments are the URI of
  37.127 +        the ldap server, the base of the query, the scope of the search, and 
  37.128 +        the query string.  The scope should be one of the following three
  37.129 +        strings:
  37.130 +        - 'object' (equivalent to python ldap's SCOPE_BASE constant)
  37.131 +        - 'subtree' (SCOPE_SUBTREE)
  37.132 +        - 'children' (SCOPE_ONELEVEL)
  37.133 +        
  37.134 +        example values:
  37.135 +        'ldap://ldap.example.edu', 'dc=example,dc=edu', 'subtree', query
  37.136 +        
  37.137 +        Failures are exceptions as defined by the python ldap package.
  37.138 +        """
  37.139 +        scope = {
  37.140 +            'object':ldap.SCOPE_BASE,
  37.141 +            'subtree':ldap.SCOPE_SUBTREE, 
  37.142 +            'children':ldap.SCOPE_ONELEVEL}.get(scope)
  37.143 +        if scope is None:
  37.144 +            raise ValueError(
  37.145 +                "scope must be one of 'object', 'subtree', or 'children'")
  37.146 +        l = ldap.initialize(uri)
  37.147 +        # protocol version 3 is preferred (do I really care?  not sure)
  37.148 +        try:
  37.149 +            l.set_option(ldap.OPT_PROTOCOL_VERSION, ldap.VERSION3)
  37.150 +        except ldap.LDAPError:
  37.151 +            l.set_option(ldap.OPT_PROTOCOL_VERSION, ldap.VERSION2)
  37.152 +        msgid = l.search(base, scope, filterstr, attrlist)
  37.153 +        d = defer.Deferred()
  37.154 +        d.addErrback(abandon_query, l, msgid)
  37.155 +        reactor.callLater(2, poll_results, d, l, msgid)
  37.156 +        return d
  37.157 +    
  37.158 +    def query_protected_ldap(uri, username, password, base, scope, filterstr,
  37.159 +                             attrlist=None): 
  37.160 +        """Query protected ldap (i.e., ldaps, over SSL).  arguments are the URI
  37.161 +        of the ldap server, the user name (including ldap qualifiers, as in the 
  37.162 +        example below), the user password, the base of the query, the scope of 
  37.163 +        the search, and the query string.  The scope should be one of the 
  37.164 +        following three strings:
  37.165 +        - 'object' (equivalent to python ldap's SCOPE_BASE constant)
  37.166 +        - 'subtree' (SCOPE_SUBTREE)
  37.167 +        - 'children' (SCOPE_ONELEVEL)
  37.168 +        
  37.169 +        example values ('ldapuser' and 'ldappass' are username and password 
  37.170 +        # values):
  37.171 +        'ldaps://ldap.example.edu:636', 
  37.172 +        'uid=ldapuser,ou=users,ou=special,dc=example,dc=edu', 'ldappass', 
  37.173 +        'dc=example,dc=edu', 'subtree', query
  37.174 +        
  37.175 +        Failures are exceptions as defined by the python ldap package.
  37.176 +        """
  37.177 +        scope = {
  37.178 +            'object':ldap.SCOPE_BASE,
  37.179 +            'subtree':ldap.SCOPE_SUBTREE, 
  37.180 +            'children':ldap.SCOPE_ONELEVEL}.get(scope)
  37.181 +        if scope is None:
  37.182 +            raise ValueError(
  37.183 +                "scope must be one of 'object', 'subtree', or 'children'")
  37.184 +        l = ldap.initialize(uri)
  37.185 +        # protocol version 3 is preferred (do I really care?  not sure)
  37.186 +        try:
  37.187 +            l.set_option(ldap.OPT_PROTOCOL_VERSION, ldap.VERSION3)
  37.188 +        except ldap.LDAPError:
  37.189 +            l.set_option(ldap.OPT_PROTOCOL_VERSION, ldap.VERSION2)
  37.190 +        l.simple_bind(username, password)
  37.191 +        msgid = l.search(base, scope, filterstr, attrlist)
  37.192 +        d = defer.Deferred()
  37.193 +        d.addErrback(abandon_query, l, msgid)
  37.194 +        reactor.callLater(2, poll_results, d, l, msgid)
  37.195 +        return d
  37.196 +    
  37.197 +    def abandon_query(failure, connection, msgid):
  37.198 +        logging.getLogger('zasync.plugins').info('abandoning ldap search')
  37.199 +        connection.abandon(msgid)
  37.200 +        return failure
  37.201 +    
  37.202 +    def poll_results(deferred, connection, msgid):
  37.203 +        if not deferred.called:
  37.204 +            logging.getLogger('zasync.plugins').debug('polling ldap')
  37.205 +            try:
  37.206 +                res = connection.result(msgid, timeout=0) # poll
  37.207 +            except:
  37.208 +                deferred.errback(failure.Failure())
  37.209 +            else:
  37.210 +                if res is None or res == (None, None):
  37.211 +                    reactor.callLater(2, poll_results, deferred, connection, msgid)
  37.212 +                else:
  37.213 +                    deferred.callback(res)
  37.214 +
  37.215 +#### Zope Exec
  37.216 +
  37.217 +import Queue, thread, time, sys, StringIO
  37.218 +from ZODB.POSException import ConflictError
  37.219 +from ZEO.Exceptions import ClientDisconnected
  37.220 +from Acquisition import aq_parent, aq_inner
  37.221 +from AccessControl.SecurityManagement import newSecurityManager
  37.222 +from Products.PageTemplates.Expressions import getEngine, SecureModuleImporter
  37.223 +
  37.224 +from Products.zasync.manager import Expression, sanitize, cleanFailure
  37.225 +from Products.zasync.bucketqueue import BucketQueue
  37.226 +
  37.227 +MAXTHREADPOOL = 1 # a real thread pool doesn't seem to be a problem, but 
  37.228 +# our use doesn't need more than one thread, and this removes one possible
  37.229 +# bug source.  May increase again; should make it configurable in a way
  37.230 +# other than changing this file....
  37.231 +
  37.232 +threadIds = []
  37.233 +threadPoolLock = thread.allocate_lock()
  37.234 +taskQueue = BucketQueue()
  37.235 +taskStatusLock = thread.allocate_lock()
  37.236 +taskStatus = {}
  37.237 +callbacks = Queue.Queue()
  37.238 +serverDown = False
  37.239 +
  37.240 +CANCEL = object()
  37.241 +CALLBACK = 'CALLBACK'
  37.242 +ERRBACK = 'ERRBACK'
  37.243 +
  37.244 +def getTaskStatus(key, del_if_cancel=False):
  37.245 +    taskStatusLock.acquire()
  37.246 +    try:
  37.247 +        val = taskStatus.get(key)
  37.248 +        if del_if_cancel and val is CANCEL:
  37.249 +            try:
  37.250 +                del taskStatus[key]
  37.251 +            except KeyError:
  37.252 +                pass
  37.253 +    finally:
  37.254 +        taskStatusLock.release()
  37.255 +    return val
  37.256 +
  37.257 +def popTaskStatus(key):
  37.258 +    taskStatusLock.acquire()
  37.259 +    try:
  37.260 +        val = taskStatus.get(key)
  37.261 +        try:
  37.262 +            del taskStatus[key]
  37.263 +        except KeyError:
  37.264 +            pass
  37.265 +    finally:
  37.266 +        taskStatusLock.release()
  37.267 +    return val
  37.268 +        
  37.269 +
  37.270 +def setTaskStatus(key, value):
  37.271 +    taskStatusLock.acquire()
  37.272 +    try:
  37.273 +        taskStatus[key] = value
  37.274 +    finally:
  37.275 +        taskStatusLock.release()
  37.276 +    
  37.277 +
  37.278 +# zopeDeferredTuple is (tool path, zope deferred id)
  37.279 +
  37.280 +def legacy_zope_exec(zopeDeferredTuple, homepath, action, *chain):
  37.281 +    """For backwards compatibility with zope exec calls that did not include 
  37.282 +    request data, register this function for the zope exec plugin instead of
  37.283 +    the standard one."""
  37.284 +    return zope_exec(zopeDeferredTuple, homepath, None, action, *chain)
  37.285 +
  37.286 +def zope_exec(zopeDeferredTuple, homepath, formdata, action, *chain):
  37.287 +    """Perform a callback chain asynchronously, beginning with a single
  37.288 +    action, where an action is either a TALES expression or a tuple of (name,
  37.289 +    TALES expression).  If the action is a tuple, then the result of the
  37.290 +    expression is available to following expressions as results[name] (or
  37.291 +    results/$name).  Each link in the optional following callback chain may be
  37.292 +    a dictionary with one or both of the keys CALLBACK or ERRBACK and values
  37.293 +    that are actions, or may be an action, which will be interpreted as a
  37.294 +    CALLBACK.  The result of the final action is the return value to the
  37.295 +    deferred callback.  If a CALLBACK or ERRBACK is missing, it is a no-op: if
  37.296 +    a value for CALLBACK is needed but not provided in a dictionary, it is
  37.297 +    effectively 'result'; if a value for ERRBACK is needed but not provided in
  37.298 +    a dictionary, it is effectively 'failure'.
  37.299 +    
  37.300 +    homepath is a path from the root (which may be a boolean False value such 
  37.301 +    as None, a tuple or list of path elements, or a string, but will be 
  37.302 +    normalized to a string) to an object that will be available as "home" in the
  37.303 +    action expressions, and is also used as a domain so that only a single 
  37.304 +    zope_exec within a given homepath will be executed at a time.  Note that 
  37.305 +    nested domains are effectively completely different domains: a zope_exec
  37.306 +    with a "/" homepath may be attempted at the same time as a zope_exec 
  37.307 +    with a "/cmf_site" homepath.
  37.308 +    
  37.309 +    Actions are performed as (and in the security context of) the user who owns 
  37.310 +    the zope deferred.  A dummy request is available.
  37.311 +    
  37.312 +    The code checks between each action and immediately prior to committing to
  37.313 +    see if the deferred has been cancelled (timed out); if it has been 
  37.314 +    cancelled, the job is aborted.  zope_exec are not called (although errbacks
  37.315 +    on the zope deferred will be called, as usual).
  37.316 +    
  37.317 +    expressions have the following names available to them:
  37.318 +    
  37.319 +    - nothing: None
  37.320 +    - user: the user object (the owner of the zope deferred)
  37.321 +    - home: the object referred to by the homepath
  37.322 +    - userhome: the location of the user's acl_users folder
  37.323 +    - tool: the asynchronous manager
  37.324 +    - deferred: the Zope deferred
  37.325 +    - root: the physical Zope root
  37.326 +    - modules: the standard Zope secure module importer
  37.327 +    - result: the result of the most recent previous successful call 
  37.328 +              (initializes to None)
  37.329 +    - failure: the failure of the most recent previous unsuccessful call
  37.330 +               (initializes to None)
  37.331 +    - results: a dictionary of results returned from past actions, as described
  37.332 +               above.
  37.333 +    - request: the faux request
  37.334 +    
  37.335 +    Example of putting a call:
  37.336 +    
  37.337 +    root.asynchronous_call_manager.putCall(
  37.338 +        "zope_exec", # name of plugin
  37.339 +        "/another_site", # homepath
  37.340 +        'user/getId', # initial action
  37.341 +        'python: foo', # ...following callbacks and errbacks...
  37.342 +        {ERRBACK: 'python: failure.trap(modules["AccessControl"].Unauthorized, '
  37.343 +                  'root.special_script)'},
  37.344 +        {CALLBACK: ('portal', 'root/my_site'),
  37.345 +         ERRBACK: ('portal', 'nothing')}
  37.346 +        )
  37.347 +
  37.348 +    """
  37.349 +    global taskQueue
  37.350 +    # normalize homepath
  37.351 +    if not homepath or homepath=="/":
  37.352 +        homepath = ''
  37.353 +    else:
  37.354 +        # this may raise errors: if so, makeCall will convert to failures
  37.355 +        homepath = normalize_path(homepath)
  37.356 +    taskQueue.makeBucket(homepath, silent=True)
  37.357 +    chain = (action,) + chain
  37.358 +    d = defer.Deferred()
  37.359 +    setTaskStatus(zopeDeferredTuple, d) # == pending
  37.360 +    logger = logging.getLogger('zasync.plugins')
  37.361 +    logger.debug(
  37.362 +        'zope_exec: scheduling zope deferred %r',
  37.363 +        zopeDeferredTuple)
  37.364 +    d.addErrback(abandon_zope_exec, zopeDeferredTuple)
  37.365 +    reactor.callLater(
  37.366 +        0, put_call, (zopeDeferredTuple, homepath, formdata, chain), homepath)
  37.367 +    return d
  37.368 +
  37.369 +def put_call(call, bucket):
  37.370 +    logger = logging.getLogger('zasync.plugins')
  37.371 +    logger.debug(
  37.372 +        'zope_exec: putting call %r in queue for bucket (homepath) %r',
  37.373 +        call, bucket)
  37.374 +    taskQueue.put(call, bucket=bucket) # this must be after all of the work in
  37.375 +    # zope_exec, above, or else the data structures the worker thread expects
  37.376 +    # are not (necessarily) in place yet.  This is in a separate reactor call
  37.377 +    # just to free the reactor up a bit to do other work.
  37.378 +    reactor.callLater(0.05, start_worker)
  37.379 +    # this must be after the put, because start_worker only
  37.380 +    # creates a thread if it can see that one is necessary
  37.381 +
  37.382 +def normalize_path(path):
  37.383 +    if not isinstance(path, (list, tuple)):
  37.384 +        path = path.split('/')
  37.385 +    tmp = []
  37.386 +    for el in path:
  37.387 +        if el=="..":
  37.388 +            if tmp:
  37.389 +                tmp.pop()
  37.390 +        elif el==".":
  37.391 +            pass
  37.392 +        else:
  37.393 +            tmp.append(el)
  37.394 +    if not tmp or tmp[0]:
  37.395 +        tmp.insert(0, '')
  37.396 +    return '/'.join(tmp)
  37.397 +
  37.398 +def start_worker():
  37.399 +    global threadIds
  37.400 +    logger = logging.getLogger('zasync.plugins')
  37.401 +    threadPoolLock.acquire()
  37.402 +    try:
  37.403 +        len_threadIds = len(threadIds)
  37.404 +    finally:
  37.405 +        threadPoolLock.release()
  37.406 +    if len_threadIds < MAXTHREADPOOL and taskQueue.primed():
  37.407 +        new_id = thread.start_new_thread(zope_exec_worker, ())
  37.408 +        threadPoolLock.acquire()
  37.409 +        try:
  37.410 +            threadIds.append(new_id)
  37.411 +        finally:
  37.412 +            threadPoolLock.release()
  37.413 +        logger.debug(
  37.414 +            "zope_exec: added new worker thread  %r to thread pool "
  37.415 +            "(%d worker threads total)",
  37.416 +            new_id, len_threadIds + 1)
  37.417 +        for c in client.chores:
  37.418 +            if c[0] is schedule_mainthread:
  37.419 +                break
  37.420 +        else:
  37.421 +            client.chores.append((schedule_mainthread, (), {}))
  37.422 +            logger.debug(
  37.423 +                'zope_exec: added schedule_mainthread to zasync chore list')
  37.424 +        if len_threadIds + 1 < MAXTHREADPOOL and taskQueue.primed():
  37.425 +            reactor.callLater(0.1, start_worker) # if start_worker is called by 
  37.426 +            # scheduleServerRetry as set up in schedule_mainthread, then it is
  37.427 +            # possible that more threads could be needed.  We check here, and
  37.428 +            # if it seems we could use some more, then we schedule another 
  37.429 +            # thread to possibly start in a tenth of a second (giving active
  37.430 +            # threads a healthy chance to take a stab at the queue before we
  37.431 +            # look again)
  37.432 +    logger.debug('out of start_worker')
  37.433 +
  37.434 +def abandon_zope_exec(failure, zopeDeferredTuple):
  37.435 +    """this is an errback for the twisted deferred that makes sure that a
  37.436 +    possibly-still-in-queue action chain is cancelled."""
  37.437 +    if getTaskStatus(zopeDeferredTuple) is not None:
  37.438 +        setTaskStatus(zopeDeferredTuple, CANCEL)
  37.439 +        logging.getLogger('zasync.plugins').debug(
  37.440 +            'zope_exec: attempting to cancel pending %r for worker thread',
  37.441 +            zopeDeferredTuple)
  37.442 +    return failure
  37.443 +
  37.444 +def schedule_mainthread():
  37.445 +    # do jobs that the zope_exec_worker needs done in the main
  37.446 +    # thread.  Basically needed for letting zasync return failures or results
  37.447 +    # to Zope.  The housekeeping function in the client module kicks this 
  37.448 +    # regularly.
  37.449 +    global callbacks, serverDown
  37.450 +    logger = logging.getLogger('zasync.plugins')
  37.451 +    while 1:
  37.452 +        try:
  37.453 +            deferred, result = callbacks.get(block=False)
  37.454 +        except Queue.Empty:
  37.455 +            break
  37.456 +        else:
  37.457 +            if not deferred.called:
  37.458 +                if isinstance(result, failure.Failure):
  37.459 +                    logger.debug(
  37.460 +                        'zope_exec: scheduling an errback through zasync\n%s',
  37.461 +                        result.getErrorMessage())
  37.462 +                    deferred.errback(result)
  37.463 +                else:
  37.464 +                    logger.debug(
  37.465 +                        'zope_exec: scheduling a callback through zasync\n%s',
  37.466 +                        result)
  37.467 +                    deferred.callback(result)
  37.468 +    if serverDown:
  37.469 +        serverDown = False
  37.470 +        client.scheduleServerRetry(start_worker)
  37.471 +        logging.getLogger('zasync.plugins').debug(
  37.472 +            'zope_exec: scheduling start_worker to resume after the ZEO server '
  37.473 +            'returns')
  37.474 +
  37.475 +def zope_exec_worker():
  37.476 +    """the thread method."""
  37.477 +    # XXX generalize this to not require zope deferred so other plugins can use 
  37.478 +    logger = logging.getLogger('zasync.plugins')
  37.479 +    thread_id = thread.get_ident()
  37.480 +    logger.debug(
  37.481 +        'zope_exec: beginning worker thread %r', thread_id)
  37.482 +    global taskQueue, taskStatus, threadIds, callbacks, serverDown
  37.483 +    from zasync.client import app, max_conflict_resolution_attempts
  37.484 +    application = remembered_failure = None
  37.485 +    try:
  37.486 +        while 1: # keep on looking for tasks
  37.487 +            # blocks:
  37.488 +            zopeDeferredTuple, homepath, formdata, actions = taskQueue.get()
  37.489 +            logger.debug(
  37.490 +                'zope_exec: thread %r got work from zope deferred %r',
  37.491 +                thread_id, zopeDeferredTuple)
  37.492 +            if getTaskStatus(zopeDeferredTuple, True) is CANCEL:
  37.493 +                logger.debug(
  37.494 +                    'zope_exec: zope deferred %r has been cancelled; '
  37.495 +                    'worker thread aborting', zopeDeferredTuple)
  37.496 +                continue
  37.497 +            toolpath, zopeDeferredId = zopeDeferredTuple
  37.498 +            for attempt in range(max_conflict_resolution_attempts):
  37.499 +                remembered_failure = None
  37.500 +                try:
  37.501 +                    try:
  37.502 +                        deferred = None
  37.503 +                        try:
  37.504 +                            sync = client.DB._storage.sync
  37.505 +                        except AttributeError:
  37.506 +                            pass
  37.507 +                        else:
  37.508 +                            sync() # important
  37.509 +                        transaction.begin() # superstition...
  37.510 +                        application = client.app()
  37.511 +                        home = root = client.getRequestApp(application)
  37.512 +                        if formdata:
  37.513 +                            root.REQUEST.form.update(formdata)
  37.514 +                        tool = root.unrestrictedTraverse(toolpath)
  37.515 +                        zopeDeferred = tool.getDeferred(zopeDeferredId)
  37.516 +                        user = zopeDeferred.getWrappedOwner()
  37.517 +                        newSecurityManager(None, user)
  37.518 +                        results = {}
  37.519 +                        if homepath: # may raise unauthorized or others;
  37.520 +                            # should be caught in bare except below
  37.521 +                            home = root.restrictedTraverse(homepath)
  37.522 +                        result = None
  37.523 +                        fail = None
  37.524 +                        success = True
  37.525 +                        context_dict = {
  37.526 +                            'nothing': None, # automatic; here for clarity
  37.527 +                            'user': user,
  37.528 +                            'home': home,
  37.529 +                            'here': home,
  37.530 +                            'userhome': aq_parent(
  37.531 +                                aq_inner(aq_parent(aq_inner(user)))),
  37.532 +                            'tool': aq_parent(aq_inner(zopeDeferred)),
  37.533 +                            'root': root,
  37.534 +                            'deferred': zopeDeferred,
  37.535 +                            'modules': SecureModuleImporter,
  37.536 +                            'request': root.REQUEST,
  37.537 +                            'result': result,
  37.538 +                            'failure': fail,
  37.539 +                            'results': results,}
  37.540 +                        for action in actions:
  37.541 +                            logger.debug(
  37.542 +                                'zope_exec: worker %r processing action %r',
  37.543 +                                thread_id, action)
  37.544 +                            call = name = None
  37.545 +                            if isinstance(action, basestring):
  37.546 +                                if success:
  37.547 +                                    call = action
  37.548 +                            elif isinstance(action, tuple):
  37.549 +                                if success:
  37.550 +                                    name, call = action
  37.551 +                            elif isinstance(action, dict):
  37.552 +                                if success:
  37.553 +                                    action = action.get(CALLBACK)
  37.554 +                                else:
  37.555 +                                    action = action.get(ERRBACK)
  37.556 +                                if isinstance(action, tuple):
  37.557 +                                    name, call = action
  37.558 +                                elif isinstance(action, basestring):
  37.559 +                                    call = action
  37.560 +                                elif action is not None:
  37.561 +                                    raise ValueError(
  37.562 +                                        "an action within a dictionary must be "
  37.563 +                                        "a string or a tuple pair", action)
  37.564 +                            else:
  37.565 +                                raise ValueError(
  37.566 +                                    "each action must be a string, a tuple "
  37.567 +                                    "pair, or a dict", action)
  37.568 +                            if (not call or 
  37.569 +                                success and call=='result' or
  37.570 +                                not success and call=='failure'): # no-ops
  37.571 +                                continue
  37.572 +                            if getTaskStatus(zopeDeferredTuple, True) is CANCEL:
  37.573 +                                logger.debug(
  37.574 +                                    'zope_exec: worker %r got a cancel request: '
  37.575 +                                    'aborting', thread_id)
  37.576 +                                raise defer.TimeoutError('Timed out.')
  37.577 +                            context = getEngine().getContext(context_dict)
  37.578 +                            call = Expression(call)
  37.579 +                            try:
  37.580 +                                res = call(context)
  37.581 +                            except (ConflictError, ClientDisconnected):
  37.582 +                                raise
  37.583 +                            except:
  37.584 +                                res = failure.Failure()
  37.585 +                            else:
  37.586 +                                if name is not None:
  37.587 +                                    results[name] = res
  37.588 +                            if isinstance(res, failure.Failure):
  37.589 +                                cleanFailure(res) # remove object references
  37.590 +                                fail = context_dict["failure"] = res
  37.591 +                                success = False
  37.592 +                            else:
  37.593 +                                result = context_dict["result"] = res
  37.594 +                                success = True
  37.595 +                        deferred = getTaskStatus(zopeDeferredTuple)
  37.596 +                        if deferred is not CANCEL:
  37.597 +                            if success:
  37.598 +                                result = sanitize(result)
  37.599 +                                transaction.commit()
  37.600 +                                logger.debug(
  37.601 +                                    'zope_exec: worker %r committed successful '
  37.602 +                                    'transaction',
  37.603 +                                    thread_id)
  37.604 +                            else:
  37.605 +                                transaction.abort()
  37.606 +                                msg = ('zope_exec: worker %r aborted failed '
  37.607 +                                       'transaction.' % (thread_id,))
  37.608 +                                if client.verbose_traceback:
  37.609 +                                    out = StringIO.StringIO()
  37.610 +                                    res.printDetailedTraceback(out)
  37.611 +                                    logger.debug(
  37.612 +                                        '%s\n\n%s\n\n%s',
  37.613 +                                        msg, out.getvalue(), 
  37.614 +                                        res.getErrorMessage())
  37.615 +                                else:
  37.616 +                                    logger.debug(msg, exc_info=True)
  37.617 +                        else:
  37.618 +                            transaction.abort()
  37.619 +                            logger.debug(
  37.620 +                                'zope_exec: worker %r got a cancel request at '
  37.621 +                                'the last possible moment: aborted.',
  37.622 +                                thread_id)
  37.623 +                            break # the timeout errback is supposed to have
  37.624 +                            # been communicated within the main zasync thread
  37.625 +                    except ConflictError:
  37.626 +                        remembered_failure = cleanFailure(failure.Failure())
  37.627 +                        transaction.abort() 
  37.628 +                        logger.debug(
  37.629 +                            'zope_exec: worker %s got conflict error', 
  37.630 +                            thread_id, exc_info=True)
  37.631 +                        if attempt < max_conflict_resolution_attempts-1:
  37.632 +                            time.sleep(attempt + 1) # XXX better idea?
  37.633 +                        # (and continue, retrying if appropriate)
  37.634 +                    except ClientDisconnected:
  37.635 +                        transaction.abort()
  37.636 +                        logger.debug(
  37.637 +                            "zope_exec: worker %s lost ZEO server!", thread_id)
  37.638 +                        if getTaskStatus(zopeDeferredTuple) is not CANCEL:
  37.639 +                            # retry later
  37.640 +                            logging.debug(
  37.641 +                                'zope_exec: worker %s rescheduling %r',
  37.642 +                                thread_id, zopeDeferredTuple)
  37.643 +                            taskQueue.put(
  37.644 +                                (zopeDeferredTuple, homepath, actions),
  37.645 +                                bucket=taskQueue.releaseBucket())
  37.646 +                            zopeDeferredTuple = None # foils the last finally.
  37.647 +                        # setting this module global communicates to the 
  37.648 +                        # schedule_mainthread chore that the start_worker
  37.649 +                        # function will need to be called once the server is 
  37.650 +                        # back online again
  37.651 +                        serverDown = True
  37.652 +                        raise
  37.653 +                    except:
  37.654 +                        # in other code, we often try to be nice even if the
  37.655 +                        # tool disappears for a moment.  We don't bother here,
  37.656 +                        # which lets the normal zasync code try to communicate 
  37.657 +                        # the failure back to the log (and Zope, if the tool
  37.658 +                        # and the associated deferred happen to magically 
  37.659 +                        # reappear).
  37.660 +                        f = cleanFailure(failure.Failure())
  37.661 +                        transaction.abort()
  37.662 +                        out = None
  37.663 +                        if client.verbose_traceback:
  37.664 +                            out = StringIO.StringIO()
  37.665 +                            f.printDetailedTraceback(out)
  37.666 +                            logger.debug(
  37.667 +                                'zope_exec: worker %s got an exception.'
  37.668 +                                '\n\n%s\n\n%s',
  37.669 +                                thread_id, out.getvalue(), f.getErrorMessage())
  37.670 +                        else:
  37.671 +                            logger.debug(
  37.672 +                                'zope_exec: worker %s got an exception',
  37.673 +                                thread_id, exc_info=True)
  37.674 +                        if deferred is None:
  37.675 +                            deferred = popTaskStatus(zopeDeferredTuple)
  37.676 +                        if deferred is not CANCEL: # one last chance
  37.677 +                            if deferred is None:
  37.678 +                                if out is None:
  37.679 +                                    out = StringIO.StringIO()
  37.680 +                                    f.printDetailedTraceback(out)
  37.681 +                                    out = "\n\n%s\n\n" % (
  37.682 +                                        out.getvalue(), f.getErrorMessage())
  37.683 +                                else:
  37.684 +                                    out = ""
  37.685 +                                logger.error(
  37.686 +                                    'zope_exec: worker %s cannot find the task '
  37.687 +                                    'status for %r, so it cannot schedule '
  37.688 +                                    'zasync to pass a failure back to Zope.  '
  37.689 +                                    'This is a significant problem.  Here is a '
  37.690 +                                    'lot of diagnostic information:\n\n%s',
  37.691 +                                    thread_id, zopeDeferredTuple, out)
  37.692 +                            else:
  37.693 +                                callbacks.put((deferred, f))
  37.694 +                                logging.debug(
  37.695 +                                    'zope_exec: worker %s scheduled zasync to '
  37.696 +                                    'pass a failure back to Zope', thread_id)
  37.697 +                        break
  37.698 +                    else: # yay! it worked
  37.699 +                        # ok, so the the success or failure that we got from
  37.700 +                        # zope_exec here should be passed back to the deferred.
  37.701 +                        # we want to leverage the ConflictError handling in the
  37.702 +                        # main zasync process, so we need to pass the result
  37.703 +                        # back to the main thread for zasync to communicate.
  37.704 +                        # all work in Zope should already be committed, so we
  37.705 +                        # shouldn't have to mess further with the transaction.
  37.706 +                        deferred = popTaskStatus(zopeDeferredTuple)
  37.707 +                        if deferred is None:
  37.708 +                            f = failure.Failure()
  37.709 +                            out = StringIO.StringIO()
  37.710 +                            f.printDetailedTraceback(out)
  37.711 +                            out = out.getvalue()
  37.712 +                            logger.error(
  37.713 +                                'zope_exec: worker %s cannot find the task '
  37.714 +                                'status for %r, so it cannot schedule '
  37.715 +                                'zasync to pass a result back to Zope.  '
  37.716 +                                'This is a significant problem.  Here is a lot '
  37.717 +                                'of diagnostic information:\n\n%s',
  37.718 +                                thread_id, zopeDeferredTuple, out)
  37.719 +                        elif success:
  37.720 +                            logger.debug(
  37.721 +                                'zope_exec: worker %s get final result %r; '
  37.722 +                                'scheduling zope deferred callback', 
  37.723 +                                thread_id, result)
  37.724 +                            callbacks.put((deferred, result))
  37.725 +                        else:
  37.726 +                            logger.debug(
  37.727 +                                'zope_exec: worker %s get final failure (%s); '
  37.728 +                                'scheduling zope deferred errback', 
  37.729 +                                thread_id, fail.getErrorMessage())
  37.730 +                            callbacks.put((deferred, fail))
  37.731 +                        break
  37.732 +                finally:
  37.733 +                    if remembered_failure is None:
  37.734 +                        remembered_failure = cleanFailure(failure.Failure())
  37.735 +                    if application is not None:
  37.736 +                        application._p_jar.close()
  37.737 +                        application = None
  37.738 +            else: # too many conflict resolution attempts
  37.739 +                logger.debug(
  37.740 +                    'zope_exec: worker %s got too many conflict errors: '
  37.741 +                    'giving up.', thread_id)
  37.742 +                deferred = popTaskStatus(zopeDeferredTuple)
  37.743 +                if deferred is None:
  37.744 +                    f = failure.Failure()
  37.745 +                    out = StringIO.StringIO()
  37.746 +                    f.printDetailedTraceback(out)
  37.747 +                    out = out.getvalue()
  37.748 +                    logger.error(
  37.749 +                        'zope_exec: worker %s cannot find the task '
  37.750 +                        'status for %r, so it cannot schedule '
  37.751 +                        'zasync to pass a result back to Zope.  '
  37.752 +                        'This is a significant problem.  Here is a lot '
  37.753 +                        'of diagnostic information:\n\n%s',
  37.754 +                        thread_id, zopeDeferredTuple, out)
  37.755 +                elif deferred is not CANCEL:
  37.756 +                    if remembered_failure is None:
  37.757 +                        remembered_failure = cleanFailure(failure.Failure())
  37.758 +                    logger.debug(
  37.759 +                        'zope_exec: worker %s scheduling failure message with '
  37.760 +                        'zasync', thread_id)
  37.761 +                    callbacks.put(
  37.762 +                        (deferred, remembered_failure))
  37.763 +    finally:
  37.764 +        logger.debug(
  37.765 +            'zope_exec: worker %s cleaning up and going away', thread_id)
  37.766 +        taskQueue.releaseBucket() # make sure I don't lock a bucket forever
  37.767 +        threadPoolLock.acquire()
  37.768 +        try:
  37.769 +            threadIds.remove(thread.get_ident()) # tell start_worker I'm dead
  37.770 +        finally:
  37.771 +            threadPoolLock.release()
  37.772 +        deferred = getTaskStatus(zopeDeferredTuple, True)
  37.773 +        if deferred is not None: # looks like we still need to clean up the 
  37.774 +                                 # deferred we were just working on.
  37.775 +            if deferred is not CANCEL:
  37.776 +                logger.debug(
  37.777 +                    'zope_exec: worker %s scheduling failure message with '
  37.778 +                    'zasync', thread_id)
  37.779 +                if remembered_failure is None:
  37.780 +                    remembered_failure = cleanFailure(failure.Failure())
  37.781 +                callbacks.put(
  37.782 +                    (deferred, remembered_failure))
    38.1 new file mode 100644
    38.2 --- /dev/null
    38.3 +++ b/client/zasync/plugins.txt
    38.4 @@ -0,0 +1,28 @@
    38.5 +This file would ideally be doctests for the default plugins.  For now, it 
    38.6 +merely holds hopefully useful notes for the plugins.
    38.7 +
    38.8 +The docstrings for each of the plugins are intended to be reasonably
    38.9 +comprehensive (and are shown in the asynchronous call manager's main ZMI
   38.10 +page).  This just inclues some additional notes.
   38.11 +
   38.12 +zope_exec
   38.13 +=========
   38.14 +
   38.15 +zope_exec TALES expressions are prime candidates for the python 
   38.16 +namespace.  For instance, if you would like to send zope_exec two arguments
   38.17 +to a method off of the current object, this pattern should work (replace
   38.18 +'yourMethod' with something more appropriate, of course):
   38.19 +
   38.20 +    >>> acm = root.asynchronous_call_manager
   38.21 +    >>> template = "python:root.restrictedTraverse(%r).yourMethod(%r,%r)"
   38.22 +    >>> method = template % ('/'.join(self.getPhysicalPath()), param1, param2)
   38.23 +    >>> deferred = acm.putSessionCall('legacy_zope_exec', '/', method)
   38.24 +
   38.25 +Another approach is to use the 'result' value (see the zope_exec docstring):
   38.26 +
   38.27 +    >>> deferred = acm.putSessionCall(
   38.28 +    ...     'legacy_zope_exec', '/', 
   38.29 +    ...     'root/' + '/'.join(self.getPhysicalPath()),
   38.30 +    ...     'python: result.yourMethod(%r, %r)' % (param1, param2))
   38.31 +
   38.32 +
    39.1 new file mode 100644
    39.2 --- /dev/null
    39.3 +++ b/client/zasync/schema.xml
    39.4 @@ -0,0 +1,324 @@
    39.5 +<schema handler="root_handler" prefix="Zope2.Startup.datatypes" 
    39.6 +        datatype="zasync.config.root_section">
    39.7 +  <description>
    39.8 +    ZAsync configuration schema.
    39.9 +    
   39.10 +    This schema describes the configuration options available to a site 
   39.11 +    administrator via the zasync.conf configuration file.
   39.12 +  </description>
   39.13 +  <!-- for Zope 2.8 -->
   39.14 +  <import package="ZConfig.components.logger" file="handlers.xml"/>
   39.15 +  <import package="ZConfig.components.logger" file="eventlog.xml"/>
   39.16 +  <!-- end for Zope 2.8 -->
   39.17 +  <import package="ZODB"/>
   39.18 +  <import package="tempstorage" />
   39.19 +
   39.20 +  <sectiontype name="plugin" datatype="zasync.config.plugin">
   39.21 +    <description>
   39.22 +    Register zasync plugins.
   39.23 +    </description>
   39.24 +    
   39.25 +    <key name="handler" required="yes" 
   39.26 +         datatype="zasync.config.DottedNameFunctionConversion">
   39.27 +      <description>
   39.28 +      The handler for the plugin.  The arguments to this callable
   39.29 +      are the signature that Zope calls should match.  If a description
   39.30 +      is not provided as a key to this configuration, the handler's
   39.31 +      docstring will be used, if any, as the description.
   39.32 +      </description>
   39.33 +    </key>
   39.34 +    
   39.35 +    <key name="timeout" datatype="time-interval">
   39.36 +      <description>
   39.37 +      The maximum timeout that this plugin allows/
   39.38 +      </description>
   39.39 +    </key>
   39.40 +    
   39.41 +    <key name="description" datatype="string">
   39.42 +      <description>
   39.43 +      The description of this plugin.  If not provided, handler's docstring
   39.44 +      is used, if any.
   39.45 +      </description>
   39.46 +    </key>
   39.47 +  
   39.48 +    <key name="retry" datatype="boolean" default="yes">
   39.49 +      <description>
   39.50 +      Whether the async call should be retried if zasync restarts while the
   39.51 +      call is accepted (in process).
   39.52 +      </description>
   39.53 +    </key>
   39.54 +  
   39.55 +    <key name="zope-aware" datatype="boolean" default="no">
   39.56 +      <description>
   39.57 +      Whether the async call needs to know about the path to the tool and the 
   39.58 +      zope deferred identifier.  If it does, any arguments from manager calls
   39.59 +      are prepended with a tuple (a single argument) of 
   39.60 +      (tool path, zope deferred id).
   39.61 +      </description>
   39.62 +    </key>
   39.63 +  </sectiontype>
   39.64 +
   39.65 +  <!-- for Zope 2.8 -->
   39.66 +  <sectiontype name="logger" datatype="zasync.config.LoggerFactory">
   39.67 +    <description>
   39.68 +      This "logger" type only applies to access and request ("trace")
   39.69 +      logging; event logging is handled by the "logging" package in
   39.70 +      the Python standard library.  The loghandler type used here is
   39.71 +      provided by the "ZConfig.components.logger" package.
   39.72 +    </description>
   39.73 +    <key name="level"
   39.74 +         datatype="ZConfig.components.logger.datatypes.logging_level"
   39.75 +         default="info"/>
   39.76 +    <key name="propagate" datatype="boolean" default="no"/>
   39.77 +    <multisection name="*"
   39.78 +                  type="ZConfig.logger.handler"
   39.79 +                  attribute="handlers"/>
   39.80 +  </sectiontype>
   39.81 +  <!-- end for Zope 2.8 -->
   39.82 +
   39.83 +  <sectiontype name="environment"
   39.84 +               datatype=".cgi_environment"
   39.85 +               keytype="identifier">
   39.86 +    <description>
   39.87 +     A section which allows you to define simple key-value pairs which
   39.88 +     will be used as environment variable settings during startup.  
   39.89 +    </description>
   39.90 +    <key name="+" attribute="environ">
   39.91 +      <description>
   39.92 +        Use any key/value pair, e.g. 'MY_PRODUCT_ENVVAR foo_bar'
   39.93 +      </description>
   39.94 +    </key>
   39.95 +  </sectiontype>
   39.96 +
   39.97 +  <sectiontype name="zodb_db" datatype=".ZopeDatabase"
   39.98 +               implements="ZODB.database" extends="zodb">
   39.99 +
  39.100 +    <description>
  39.101 +      We need to specialize the database configuration section for Zope
  39.102 +      only by including a (required) mount-point argument, which
  39.103 +      is a string.  A Zope ZODB database can have multiple mount points,
  39.104 +      so this is a multikey.
  39.105 +    </description>
  39.106 +    <multikey name="mount-point" required="yes" attribute="mount_points"
  39.107 +              datatype=".mount_point">
  39.108 +      <description>
  39.109 +       The mount point is the slash-separated path to which this database
  39.110 +       will be mounted within the Zope application server.
  39.111 +      </description>
  39.112 +    </multikey>
  39.113 +
  39.114 +    <key name="connection-class" datatype=".importable_name">
  39.115 +      <description>
  39.116 +       Change the connection class a database uses on a per-database basis to
  39.117 +       support different connection policies.  Use a Python dotted-path
  39.118 +       name to specify the connection class.
  39.119 +      </description>
  39.120 +    </key>
  39.121 +
  39.122 +   <key name="class-factory" datatype=".importable_name"
  39.123 +        default="DBTab.ClassFactories.autoClassFactory">
  39.124 +      <description>
  39.125 +       Change the class factory function a database uses on a
  39.126 +       per-database basis to support different class factory policy.
  39.127 +       Use a Python dotted-path name to specify the class factory function.
  39.128 +      </description>
  39.129 +    </key>
  39.130 +
  39.131 +    <key name="container-class" datatype=".python_dotted_path">
  39.132 +      <description>
  39.133 +       Change the contiainer class a (mounted) database uses on a
  39.134 +       per-database basis to support a different container than a plain
  39.135 +       Folder. Use a Python dotted-path name to specify the container class.
  39.136 +      </description>
  39.137 +    </key>
  39.138 +
  39.139 +  </sectiontype>
  39.140 +
  39.141 +  <!-- end sectiontype defs, begin section and key defs -->
  39.142 +
  39.143 +  <section type="environment" attribute="environment" name="*">
  39.144 +    <description>
  39.145 +     A section which allows a user to define arbitrary key-value pairs for
  39.146 +     use as environment variables during Zope's run cycle.  It
  39.147 +     is not recommended to set system-related environment variables such as
  39.148 +     PYTHONPATH within this section.
  39.149 +    </description>
  39.150 +  </section>
  39.151 +
  39.152 +  <key name="instancehome" datatype="existing-directory" required="yes">
  39.153 +    <description>
  39.154 +    The path to the data files, local product files, import directory,
  39.155 +    and Extensions directory used by Zope.
  39.156 +    </description>
  39.157 +  </key>
  39.158 +
  39.159 +  <key name="clienthome" datatype="existing-directory">
  39.160 +    <description>
  39.161 +      The directory used to store the default filestorage file used to
  39.162 +      back the ZODB database, as well as other files used by the
  39.163 +      Zope applications server during runtime.
  39.164 +    </description>
  39.165 +    <metadefault>$instancehome/var</metadefault>
  39.166 +  </key>
  39.167 +  
  39.168 +  <multikey name="products" datatype="existing-directory">
  39.169 +    <description>
  39.170 +    Name of a directory that contains additional Product packages.  This
  39.171 +    directive may be used as many times as needed to add additional
  39.172 +    collections of products.  Each directory identified will be
  39.173 +    added to the __path__ of the Products package.  All Products are
  39.174 +    initialized in ascending alphabetical order by product name.  If
  39.175 +    two products with the same name exist in two Products directories,
  39.176 +    the order in which the packages appear here defines the load
  39.177 +    order.  The master Products directory exists in Zope's software home,
  39.178 +    and cannot be removed from the products path (and should not be added
  39.179 +    to it here).
  39.180 +    </description>
  39.181 +  </multikey>
  39.182 +  
  39.183 +  <multikey name="path" datatype="existing-directory">
  39.184 +    <description>
  39.185 +    Name of a directory which should be inserted into the
  39.186 +    the beginning of Python's module search path.  This directive
  39.187 +    may be specified as many times as needed to insert additional
  39.188 +    directories.  The set of directories specified is inserted into the
  39.189 +    beginning of the module search path in the order which they are specified
  39.190 +    here.  Note that the processing of this directive may happen too late
  39.191 +    under some circumstances; it is recommended that you use the PYTHONPATH
  39.192 +    environment variable if using this directive doesn't work for you.
  39.193 +    </description>
  39.194 +  </multikey>
  39.195 +
  39.196 +  <key name="security-policy-implementation"
  39.197 +       datatype=".security_policy_implementation"
  39.198 +       default="C">
  39.199 +     <description>
  39.200 +     The default Zope "security policy" implementation is written in C.
  39.201 +     Set this key to "PYTHON" to use the Python implementation
  39.202 +     (useful for debugging purposes); set it to "C" to use the C
  39.203 +     implementation.
  39.204 +     </description>
  39.205 +     <metadefault>C</metadefault>
  39.206 +  </key>
  39.207 +
  39.208 +  <key name="skip-authentication-checking" datatype="boolean"
  39.209 +       default="off">
  39.210 +     <description>
  39.211 +     Set this directive to 'on' to cause Zope to prevent Zope from
  39.212 +     attempting to authenticate users during normal operation.
  39.213 +     Potentially dangerous from a security perspective.  Only works if
  39.214 +     security-policy-implementation is set to 'C'.
  39.215 +     </description>
  39.216 +     <metadefault>off</metadefault>
  39.217 +  </key>
  39.218 +
  39.219 +  <key name="skip-ownership-checking" datatype="boolean"
  39.220 +       default="off">
  39.221 +     <description>
  39.222 +     Set this directive to 'on' to cause Zope to ignore ownership checking
  39.223 +     when attempting to execute "through the web" code. By default, this
  39.224 +     directive is off in order to prevent 'trojan horse' security problems
  39.225 +     whereby a user with less privilege can cause a user with more
  39.226 +     privilege to execute code which the less privileged user has written.
  39.227 +     </description>
  39.228 +     <metadefault>off</metadefault>
  39.229 +  </key>
  39.230 +
  39.231 +  <!-- for 2.8 -->
  39.232 +  <key name="verbose-security" datatype="boolean"
  39.233 +       default="off">
  39.234 +     <description>
  39.235 +     Set this directive to 'on' to enable verbose security exceptions.
  39.236 +     This can help you track down the reason for Unauthorized exceptions,
  39.237 +     but it is not suitable for public sites because it may reveal
  39.238 +     unnecessary information about the structure of your site.  Only
  39.239 +     works if security-policy-implementation is set to 'PYTHON'.
  39.240 +     </description>
  39.241 +     <metadefault>off</metadefault>
  39.242 +  </key>
  39.243 +  <!-- end for 2.8 -->
  39.244 +
  39.245 +  <key name="verbose-traceback" datatype="boolean" default="off">
  39.246 +     <description>
  39.247 +     Set this directive to 'on' to cause zasync to produce very verbose
  39.248 +     tracebacks, including the locals and globals for every frame.
  39.249 +     </description>
  39.250 +     <metadefault>off</metadefault>
  39.251 +  </key>
  39.252 +
  39.253 +  <multisection type="ZODB.Database" name="+" attribute="databases">
  39.254 +    <description>
  39.255 +       Zope ZODB databases must have a name, and they are required to be
  39.256 +       referenced via the "zodb_db" database type because it is
  39.257 +       the only kind of database definition that implements
  39.258 +       the required mount-point argument.  There is another
  39.259 +       database sectiontype named "zodb", but it cannot be used
  39.260 +       in the context of a proper Zope configuration (due to
  39.261 +       lack of a mount-point).
  39.262 +    </description>
  39.263 +  </multisection>
  39.264 +
  39.265 +  <key name="target" datatype="string" default="/asynchronous_call_manager">
  39.266 +    <description>
  39.267 +    identify the path to the object that implements the asynchronous call 
  39.268 +    manager interface
  39.269 +    </description>
  39.270 +  </key>
  39.271 +  
  39.272 +  <key name="max-conflict-resolution-attempts" datatype="integer" default="5">
  39.273 +    <description>
  39.274 +    The number of times a transaction with a ConflictError should be retried
  39.275 +    </description>
  39.276 +  </key>
  39.277 +  
  39.278 +  <key name="initial-retry-delay" datatype="time-interval" default="5">
  39.279 +    <description>
  39.280 +    The initial delay in seconds between attempts to resolve an error in 
  39.281 +    finding the ZEO server or the target
  39.282 +    </description>
  39.283 +  </key>
  39.284 +  
  39.285 +  <key name="retry-exponential-backoff" datatype="float" default="1.1">
  39.286 +    <description>
  39.287 +    An exponential backoff of the retry delay.  Set to 1 for no exponential
  39.288 +    increase.  Values less than one are not allowed.
  39.289 +    </description>
  39.290 +  </key>
  39.291 +  
  39.292 +  <key name="max-total-retry" datatype="time-interval" default="1h">
  39.293 +    <description>
  39.294 +    The maximum total retry time since a failure in reaching the ZEO server
  39.295 +    or the target before zasync gives a up.  A value of 0 indicates that 
  39.296 +    zasync should never give up.  Value is in seconds.
  39.297 +    </description>
  39.298 +  </key>
  39.299 +  
  39.300 +  <multisection type="plugin" name="*" attribute="plugins">
  39.301 +    <description>
  39.302 +    Register zasync plugins.
  39.303 +    </description>
  39.304 +  </multisection>
  39.305 +
  39.306 +  <section type="eventlog" name="*" attribute="eventlog">
  39.307 +    <description>
  39.308 +      Describes the logging performed by Zope in the course of zasync
  39.309 +      calls.
  39.310 +    </description>
  39.311 +  </section>
  39.312 +
  39.313 +  <section type="logger" name="zasync">
  39.314 +     <description>
  39.315 +      Describes the logging performed to capture the zasync log messages
  39.316 +    </description>
  39.317 +  </section>
  39.318 +
  39.319 +  <section type="logger" name="plugins" attribute="zasync_plugins">
  39.320 +     <description>
  39.321 +      Describes the logging performed to capture the zasync plugins log,
  39.322 +      which collects log messages from the plugins, if they are log to the
  39.323 +      'plugin' log.  If messages are propagated, they are propagated
  39.324 +      to the zasync log.
  39.325 +    </description>
  39.326 +  </section>
  39.327 +
  39.328 +</schema>
    40.1 new file mode 100755
    40.2 --- /dev/null
    40.3 +++ b/client/zasync/schema27.xml
    40.4 @@ -0,0 +1,304 @@
    40.5 +<schema handler="root_handler" prefix="Zope.Startup.datatypes" 
    40.6 +        datatype="zasync.config.root_section">
    40.7 +  <description>
    40.8 +    ZAsync configuration schema.
    40.9 +    
   40.10 +    This schema describes the configuration options available to a site 
   40.11 +    administrator via the zasync.conf configuration file.
   40.12 +  </description>
   40.13 +  <!-- Zope 2.7 -->
   40.14 +  <import package="zLOG"/>
   40.15 +  <!-- end for Zope 2.7 -->
   40.16 +  <import package="ZODB"/>
   40.17 +  <import package="tempstorage" />
   40.18 +
   40.19 +  <sectiontype name="plugin" datatype="zasync.config.plugin">
   40.20 +    <description>
   40.21 +    Register zasync plugins.
   40.22 +    </description>
   40.23 +    
   40.24 +    <key name="handler" required="yes" 
   40.25 +         datatype="zasync.config.DottedNameFunctionConversion">
   40.26 +      <description>
   40.27 +      The handler for the plugin.  The arguments to this callable
   40.28 +      are the signature that Zope calls should match.  If a description
   40.29 +      is not provided as a key to this configuration, the handler's
   40.30 +      docstring will be used, if any, as the description.
   40.31 +      </description>
   40.32 +    </key>
   40.33 +    
   40.34 +    <key name="timeout" datatype="time-interval">
   40.35 +      <description>
   40.36 +      The maximum timeout that this plugin allows/
   40.37 +      </description>
   40.38 +    </key>
   40.39 +    
   40.40 +    <key name="description" datatype="string">
   40.41 +      <description>
   40.42 +      The description of this plugin.  If not provided, handler's docstring
   40.43 +      is used, if any.
   40.44 +      </description>
   40.45 +    </key>
   40.46 +  
   40.47 +    <key name="retry" datatype="boolean" default="yes">
   40.48 +      <description>
   40.49 +      Whether the async call should be retried if zasync restarts while the
   40.50 +      call is accepted (in process).
   40.51 +      </description>
   40.52 +    </key>
   40.53 +  
   40.54 +    <key name="zope-aware" datatype="boolean" default="no">
   40.55 +      <description>
   40.56 +      Whether the async call needs to know about the path to the tool and the 
   40.57 +      zope deferred identifier.  If it does, any arguments from manager calls
   40.58 +      are prepended with a tuple (a single argument) of 
   40.59 +      (tool path, zope deferred id).
   40.60 +      </description>
   40.61 +    </key>
   40.62 +  </sectiontype>
   40.63 +
   40.64 +  <!-- for Zope 2.7 -->
   40.65 +  <sectiontype name="logger" datatype="zasync.config.LoggerFactory">
   40.66 +    <description>
   40.67 +      This "logger" type only applies to access and request ("trace")
   40.68 +      logging; event logging is handled by the zLOG package, which
   40.69 +      provides the loghandler type used here.
   40.70 +    </description>
   40.71 +    <key name="level" datatype="zLOG.datatypes.logging_level" default="info"/>
   40.72 +    <key name="propagate" datatype="boolean" default="no"/>
   40.73 +    <multisection type="zLOG.loghandler" attribute="handlers" name="*"/>
   40.74 +  </sectiontype>
   40.75 +  <!-- end for Zope 2.7 -->
   40.76 +
   40.77 +  <sectiontype name="environment"
   40.78 +               datatype=".cgi_environment"
   40.79 +               keytype="identifier">
   40.80 +    <description>
   40.81 +     A section which allows you to define simple key-value pairs which
   40.82 +     will be used as environment variable settings during startup.  
   40.83 +    </description>
   40.84 +    <key name="+" attribute="environ">
   40.85 +      <description>
   40.86 +        Use any key/value pair, e.g. 'MY_PRODUCT_ENVVAR foo_bar'
   40.87 +      </description>
   40.88 +    </key>
   40.89 +  </sectiontype>
   40.90 +
   40.91 +  <sectiontype name="zodb_db" datatype=".ZopeDatabase"
   40.92 +               implements="ZODB.database" extends="zodb">
   40.93 +
   40.94 +    <description>
   40.95 +      We need to specialize the database configuration section for Zope
   40.96 +      only by including a (required) mount-point argument, which
   40.97 +      is a string.  A Zope ZODB database can have multiple mount points,
   40.98 +      so this is a multikey.
   40.99 +    </description>
  40.100 +    <multikey name="mount-point" required="yes" attribute="mount_points"
  40.101 +              datatype=".mount_point">
  40.102 +      <description>
  40.103 +       The mount point is the slash-separated path to which this database
  40.104 +       will be mounted within the Zope application server.
  40.105 +      </description>
  40.106 +    </multikey>
  40.107 +
  40.108 +    <key name="connection-class" datatype=".importable_name">
  40.109 +      <description>
  40.110 +       Change the connection class a database uses on a per-database basis to
  40.111 +       support different connection policies.  Use a Python dotted-path
  40.112 +       name to specify the connection class.
  40.113 +      </description>
  40.114 +    </key>
  40.115 +
  40.116 +   <key name="class-factory" datatype=".importable_name"
  40.117 +        default="DBTab.ClassFactories.autoClassFactory">
  40.118 +      <description>
  40.119 +       Change the class factory function a database uses on a
  40.120 +       per-database basis to support different class factory policy.
  40.121 +       Use a Python dotted-path name to specify the class factory function.
  40.122 +      </description>
  40.123 +    </key>
  40.124 +
  40.125 +    <key name="container-class" datatype=".python_dotted_path">
  40.126 +      <description>
  40.127 +       Change the contiainer class a (mounted) database uses on a
  40.128 +       per-database basis to support a different container than a plain
  40.129 +       Folder. Use a Python dotted-path name to specify the container class.
  40.130 +      </description>
  40.131 +    </key>
  40.132 +
  40.133 +  </sectiontype>
  40.134 +
  40.135 +  <!-- end sectiontype defs, begin section and key defs -->
  40.136 +
  40.137 +  <section type="environment" attribute="environment" name="*">
  40.138 +    <description>
  40.139 +     A section which allows a user to define arbitrary key-value pairs for
  40.140 +     use as environment variables during Zope's run cycle.  It
  40.141 +     is not recommended to set system-related environment variables such as
  40.142 +     PYTHONPATH within this section.
  40.143 +    </description>
  40.144 +  </section>
  40.145 +
  40.146 +  <key name="instancehome" datatype="existing-directory" required="yes">
  40.147 +    <description>
  40.148 +    The path to the data files, local product files, import directory,
  40.149 +    and Extensions directory used by Zope.
  40.150 +    </description>
  40.151 +  </key>
  40.152 +
  40.153 +  <key name="clienthome" datatype="existing-directory">
  40.154 +    <description>
  40.155 +      The directory used to store the default filestorage file used to
  40.156 +      back the ZODB database, as well as other files used by the
  40.157 +      Zope applications server during runtime.
  40.158 +    </description>
  40.159 +    <metadefault>$instancehome/var</metadefault>
  40.160 +  </key>
  40.161 +  
  40.162 +  <multikey name="products" datatype="existing-directory">
  40.163 +    <description>
  40.164 +    Name of a directory that contains additional Product packages.  This
  40.165 +    directive may be used as many times as needed to add additional
  40.166 +    collections of products.  Each directory identified will be
  40.167 +    added to the __path__ of the Products package.  All Products are
  40.168 +    initialized in ascending alphabetical order by product name.  If
  40.169 +    two products with the same name exist in two Products directories,
  40.170 +    the order in which the packages appear here defines the load
  40.171 +    order.  The master Products directory exists in Zope's software home,
  40.172 +    and cannot be removed from the products path (and should not be added
  40.173 +    to it here).
  40.174 +    </description>
  40.175 +  </multikey>
  40.176 +  
  40.177 +  <multikey name="path" datatype="existing-directory">
  40.178 +    <description>
  40.179 +    Name of a directory which should be inserted into the
  40.180 +    the beginning of Python's module search path.  This directive
  40.181 +    may be specified as many times as needed to insert additional
  40.182 +    directories.  The set of directories specified is inserted into the
  40.183 +    beginning of the module search path in the order which they are specified
  40.184 +    here.  Note that the processing of this directive may happen too late
  40.185 +    under some circumstances; it is recommended that you use the PYTHONPATH
  40.186 +    environment variable if using this directive doesn't work for you.
  40.187 +    </description>
  40.188 +  </multikey>
  40.189 +
  40.190 +  <key name="security-policy-implementation"
  40.191 +       datatype=".security_policy_implementation"
  40.192 +       default="C">
  40.193 +     <description>
  40.194 +     The default Zope "security policy" implementation is written in C.
  40.195 +     Set this key to "PYTHON" to use the Python implementation
  40.196 +     (useful for debugging purposes); set it to "C" to use the C
  40.197 +     implementation.
  40.198 +     </description>
  40.199 +     <metadefault>C</metadefault>
  40.200 +  </key>
  40.201 +
  40.202 +  <key name="skip-authentication-checking" datatype="boolean"
  40.203 +       default="off">
  40.204 +     <description>
  40.205 +     Set this directive to 'on' to cause Zope to prevent Zope from
  40.206 +     attempting to authenticate users during normal operation.
  40.207 +     Potentially dangerous from a security perspective.  Only works if
  40.208 +     security-policy-implementation is set to 'C'.
  40.209 +     </description>
  40.210 +     <metadefault>off</metadefault>
  40.211 +  </key>
  40.212 +
  40.213 +  <key name="skip-ownership-checking" datatype="boolean"
  40.214 +       default="off">
  40.215 +     <description>
  40.216 +     Set this directive to 'on' to cause Zope to ignore ownership checking
  40.217 +     when attempting to execute "through the web" code. By default, this
  40.218 +     directive is off in order to prevent 'trojan horse' security problems
  40.219 +     whereby a user with less privilege can cause a user with more
  40.220 +     privilege to execute code which the less privileged user has written.
  40.221 +     </description>
  40.222 +     <metadefault>off</metadefault>
  40.223 +  </key>
  40.224 +
  40.225 +  <key name="verbose-traceback" datatype="boolean" default="off">
  40.226 +     <description>
  40.227 +     Set this directive to 'on' to cause zasync to produce very verbose
  40.228 +     tracebacks, including the locals and globals for every frame.
  40.229 +     </description>
  40.230 +     <metadefault>off</metadefault>
  40.231 +  </key>
  40.232 +
  40.233 +  <multisection type="ZODB.Database" name="+" attribute="databases">
  40.234 +    <description>
  40.235 +       Zope ZODB databases must have a name, and they are required to be
  40.236 +       referenced via the "zodb_db" database type because it is
  40.237 +       the only kind of database definition that implements
  40.238 +       the required mount-point argument.  There is another
  40.239 +       database sectiontype named "zodb", but it cannot be used
  40.240 +       in the context of a proper Zope configuration (due to
  40.241 +       lack of a mount-point).
  40.242 +    </description>
  40.243 +  </multisection>
  40.244 +
  40.245 +  <key name="target" datatype="string" default="/asynchronous_call_manager">
  40.246 +    <description>
  40.247 +    identify the path to the object that implements the asynchronous call 
  40.248 +    manager interface
  40.249 +    </description>
  40.250 +  </key>
  40.251 +  
  40.252 +  <key name="max-conflict-resolution-attempts" datatype="integer" default="5">
  40.253 +    <description>
  40.254 +    The number of times a transaction with a ConflictError should be retried
  40.255 +    </description>
  40.256 +  </key>
  40.257 +  
  40.258 +  <key name="initial-retry-delay" datatype="time-interval" default="5">
  40.259 +    <description>
  40.260 +    The initial delay in seconds between attempts to resolve an error in 
  40.261 +    finding the ZEO server or the target
  40.262 +    </description>
  40.263 +  </key>
  40.264 +  
  40.265 +  <key name="retry-exponential-backoff" datatype="float" default="1.1">
  40.266 +    <description>
  40.267 +    An exponential backoff of the retry delay.  Set to 1 for no exponential
  40.268 +    increase.  Values less than one are not allowed.
  40.269 +    </description>
  40.270 +  </key>
  40.271 +  
  40.272 +  <key name="max-total-retry" datatype="time-interval" default="1h">
  40.273 +    <description>
  40.274 +    The maximum total retry time since a failure in reaching the ZEO server
  40.275 +    or the target before zasync gives a up.  A value of 0 indicates that 
  40.276 +    zasync should never give up.  Value is in seconds.
  40.277 +    </description>
  40.278 +  </key>
  40.279 +  
  40.280 +  <multisection type="plugin" name="*" attribute="plugins">
  40.281 +    <description>
  40.282 +    Register zasync plugins.
  40.283 +    </description>
  40.284 +  </multisection>
  40.285 +
  40.286 +  <section type="eventlog" name="*" attribute="eventlog">
  40.287 +    <description>
  40.288 +      Describes the logging performed by Zope in the course of zasync
  40.289 +      calls.
  40.290 +    </description>
  40.291 +  </section>
  40.292 +
  40.293 +  <section type="logger" name="zasync">
  40.294 +     <description>
  40.295 +      Describes the logging performed to capture the zasync log messages
  40.296 +    </description>
  40.297 +  </section>
  40.298 +
  40.299 +  <section type="logger" name="plugins" attribute="zasync_plugins">
  40.300 +     <description>
  40.301 +      Describes the logging performed to capture the zasync plugins log,
  40.302 +      which collects log messages from the plugins, if they are log to the
  40.303 +      'plugin' log.  If messages are propagated, they are propagated
  40.304 +      to the zasync log.
  40.305 +    </description>
  40.306 +  </section>
  40.307 +
  40.308 +</schema>
    41.1 new file mode 100644
    41.2 --- /dev/null
    41.3 +++ b/client/zasync/tests.txt
    41.4 @@ -0,0 +1,36 @@
    41.5 +Automated tests are difficult for the client because of the scaffolding
    41.6 +necessary.  For now, this documents manual tests.  We need automated tests,
    41.7 +especially stress tests.
    41.8 +
    41.9 +This is an example python script that can be used to test some of the 
   41.10 +behavior.
   41.11 +
   41.12 +## Script (Python) "test_zope_exec"
   41.13 +##parameters=rebuild=False
   41.14 +acm = context.asynchronous_call_manager
   41.15 +req = context.REQUEST
   41.16 +sess = req.SESSION
   41.17 +sess_key = 'test_zope_exec'
   41.18 +deferred_key = sess.get(sess_key)
   41.19 +if rebuild or deferred_key is None:
   41.20 +    deferred = acm.putSessionCall(
   41.21 +        'zope_exec', '/', None,
   41.22 +        ('title', 'root/standard_template.pt/title'), 
   41.23 +        'string: hi from ${results/title}')
   41.24 +    deferred.addCallback(
   41.25 +        'python:root["standard_template.pt"].manage_changeProperties('
   41.26 +        'title=result)')
   41.27 +    deferred.addCallback('deferred/original_result')
   41.28 +    sess[sess_key] = deferred.local_key
   41.29 +    msg = "deferred created"
   41.30 +else:
   41.31 +    deferred = acm.getSessionDeferred(deferred_key)
   41.32 +    state = deferred.getState()
   41.33 +    if state == 'success':
   41.34 +        msg = "%s\n%r" % (state, deferred.getValue())
   41.35 +    elif state == 'failure':
   41.36 +        msg = "%s\n%s\n%s" % (
   41.37 +            state, deferred.getValue(), deferred.failure.getTraceback())
   41.38 +    else:
   41.39 +        msg = "still pending"
   41.40 +return msg
    42.1 new file mode 100644
    42.2 --- /dev/null
    42.3 +++ b/client/zasync/zasync.conf
    42.4 @@ -0,0 +1,339 @@
    42.5 +# zasync configuration
    42.6 +#
    42.7 +# This file configures zasync to set up its running environment and define
    42.8 +# its behavior, including plugins, logging, ZEO server address, and retry
    42.9 +# behavior.
   42.10 +
   42.11 +##############################################################################
   42.12 +# defines
   42.13 +##############################################################################
   42.14 +
   42.15 +# ZConfig "defines" used for later textual substitution; ZOPE and INSTANCE
   42.16 +# must be defined for some defaults to work
   42.17 +
   42.18 +%define BASE_DIR /Users/gary/dev/my-project
   42.19 +%define INSTANCE $BASE_DIR/var/zope
   42.20 +
   42.21 +##############################################################################
   42.22 +# Set up the Zope environment so that products are available for import
   42.23 +##############################################################################
   42.24 +
   42.25 +# Directive: environment
   42.26 +#
   42.27 +# Description:
   42.28 +#     A section which can be used to define arbitrary key-value pairs
   42.29 +#     for use as environment variables during Zope's run cycle.  It
   42.30 +#     is not recommended to set system-related environment variables such as
   42.31 +#     PYTHONPATH within this section.
   42.32 +#
   42.33 +# Default: unset
   42.34 +#
   42.35 +# Example:
   42.36 +#
   42.37 +#    <environment>
   42.38 +#       ZOPE3_SITE_ZCML $INSTANCE/etc/site.zcml
   42.39 +#    </environment>
   42.40 +
   42.41 +# Directive: instancehome
   42.42 +#
   42.43 +# Description:
   42.44 +#     The path to the data files, local product files, import directory,
   42.45 +#     and Extensions directory used by Zope.
   42.46 +#
   42.47 +# Example:
   42.48 +#     instancehome $INSTANCE
   42.49 +
   42.50 +# Directive: products
   42.51 +#
   42.52 +# Description:
   42.53 +#     Name of a directory that contains additional Product packages.  This
   42.54 +#     directive may be used as many times as needed to add additional
   42.55 +#     collections of products.  Each directory identified will be
   42.56 +#     added to the __path__ of the Products package.  All Products are
   42.57 +#     initialized in ascending alphabetical order by product name.  If
   42.58 +#     two products with the same name exist in two Products directories,
   42.59 +#     the order in which the packages appear here defines the load
   42.60 +#     order.  The master Products directory exists in Zope's software home,
   42.61 +#     and cannot be removed from the products path (and should not be added
   42.62 +#     to it here).
   42.63 +#
   42.64 +# Example: 
   42.65 +#     products $INSTANCE/Products
   42.66 +
   42.67 +# Directive: path
   42.68 +#
   42.69 +# Description:
   42.70 +#     Name of a directory which should be inserted into the
   42.71 +#     the beginning of Python's module search path.  This directive
   42.72 +#     may be specified as many times as needed to insert additional
   42.73 +#     directories.  The set of directories specified is inserted into the
   42.74 +#     beginning of the module search path in the order which they are specified
   42.75 +#     here.  Note that the processing of this directive may happen too late
   42.76 +#     under some circumstances; it is recommended that you use the PYTHONPATH
   42.77 +#     environment variable if using this directive doesn't work for you.
   42.78 +#
   42.79 +# Default:
   42.80 +#     path $INSTANCE/lib/python
   42.81 +
   42.82 +# Directive: security-policy-implementation
   42.83 +#
   42.84 +# Description:
   42.85 +#     The default Zope security machinery is implemented in C.
   42.86 +#     Change this to "python" to use the Python version of the
   42.87 +#     Zope security machinery.  This impacts performance but
   42.88 +#     is useful for debugging purposes and required by Products such as
   42.89 +#     VerboseSecurity, which need to "monkey-patch" the security
   42.90 +#     machinery.
   42.91 +#
   42.92 +# Default: C
   42.93 +#
   42.94 +# Example:
   42.95 +#
   42.96 +#    security-policy-implementation python
   42.97 +
   42.98 +# Directive: skip-authentication-checking
   42.99 +#
  42.100 +# Description:
  42.101 +#     Set this directive to 'on' to cause Zope to skip checks related
  42.102 +#     to authentication, for servers which serve only anonymous content.
  42.103 +#     Only works if security-policy-implementation is 'C'.
  42.104 +#
  42.105 +# Default: off
  42.106 +#
  42.107 +# Example:
  42.108 +#
  42.109 +#    skip-authentication-checking on
  42.110 +
  42.111 +# Directive: skip-ownership-checking
  42.112 +#
  42.113 +# Description:
  42.114 +#     Set this directive to 'on' to cause Zope to ignore ownership checking
  42.115 +#     when attempting to execute "through the web" code. By default, this
  42.116 +#     directive is off in order to prevent 'trojan horse' security problems
  42.117 +#     whereby a user with less privilege can cause a user with more
  42.118 +#     privilege to execute dangerous code.
  42.119 +#
  42.120 +# Default: off
  42.121 +#
  42.122 +# Example:
  42.123 +#
  42.124 +#    skip-ownership-checking on
  42.125 +
  42.126 +##############################################################################
  42.127 +# Identify the zasync's target: the ZEO server and the path to the 
  42.128 +# asynchronous call manager.
  42.129 +##############################################################################
  42.130 +
  42.131 +# Database (zodb_db) section
  42.132 +#
  42.133 +# Description:
  42.134 +#     A database section allows the definition of custom database and
  42.135 +#     storage types.  For zasync this is presumably always a ZEO server.
  42.136 +#
  42.137 +# ZEO client storage:
  42.138 +
  42.139 +<zodb_db main>
  42.140 +   <zeoclient>
  42.141 +     server localhost:8100
  42.142 +     storage 1
  42.143 +     cache-size 30000000
  42.144 +     name zeostorage
  42.145 +     var $INSTANCE/var
  42.146 +   </zeoclient>
  42.147 +   mount-point /
  42.148 +   cache-size          5000
  42.149 +   pool-size              7
  42.150 +   version-pool-size      3
  42.151 +   version-cache-size   100
  42.152 +</zodb_db>
  42.153 +
  42.154 +# Temporary storage: this is here to satisfy DBTab.  Do not use sessions in 
  42.155 +# zasync!!
  42.156 +<zodb_db temporary>
  42.157 +   <temporarystorage>
  42.158 +     name sessions
  42.159 +   </temporarystorage>
  42.160 +   mount-point /temp_folder
  42.161 +   container-class Products.TemporaryFolder.TemporaryContainer
  42.162 +</zodb_db>
  42.163 +
  42.164 +# Directive: target
  42.165 +#
  42.166 +# Description:
  42.167 +#     identify the path to the object that implements the asynchronous call 
  42.168 +#     manager interface
  42.169 +#
  42.170 +# Default:
  42.171 +#     target /asynchronous_call_manager
  42.172 +
  42.173 +##############################################################################
  42.174 +# Define zasync's retry behavior
  42.175 +##############################################################################
  42.176 +
  42.177 +# Directive: max-conflict-resolution-attempts
  42.178 +#
  42.179 +# Description:
  42.180 +#     The number of times a transaction with a ConflictError should be retried
  42.181 +#
  42.182 +# Default:
  42.183 +#     max-conflict-resolution-attempts 5
  42.184 +
  42.185 +# Directive: initial-retry-delay
  42.186 +#
  42.187 +# Description:
  42.188 +#     The initial delay in seconds between attempts to resolve an error in 
  42.189 +#     finding the ZEO server or the target
  42.190 +#
  42.191 +# Default:
  42.192 +#     initial-retry-delay 5
  42.193 +
  42.194 +# Directive: retry-exponential-backoff
  42.195 +#
  42.196 +# Description:
  42.197 +#     An exponential backoff of the retry delay.  Set to 1 for no exponential
  42.198 +#     increase.  Values less than one are not allowed.
  42.199 +#
  42.200 +# Default:
  42.201 +#     retry-exponential-backoff 1.1
  42.202 +
  42.203 +# Directive: max-total-retry
  42.204 +#
  42.205 +# Description:
  42.206 +#     The maximum total retry time since a failure in reaching the ZEO server
  42.207 +#     or the target before zasync gives a up.  A value of 0 indicates that 
  42.208 +#     zasync should never give up.  Value is in seconds.
  42.209 +#
  42.210 +# Default:
  42.211 +#     max-total-retry 3600
  42.212 +
  42.213 +##############################################################################
  42.214 +# load the plugins
  42.215 +##############################################################################
  42.216 +
  42.217 +# Directive: plugin
  42.218 +#
  42.219 +# Description:
  42.220 +#     Load in zasync plugins.  Keys are the following:
  42.221 +#     - name: 
  42.222 +#         The name of the plugin that Zope will use to call it.
  42.223 +#         Required.
  42.224 +#     - handler:
  42.225 +#         The handler for the plugin.  The arguments to this callable
  42.226 +#         are the signature that Zope calls should match.  Required.
  42.227 +#     - timeout:
  42.228 +#         The maximum timeout that this plugin allows.
  42.229 +#     - zope-aware:
  42.230 +#         Whether the plugin should receive information about the Zope deferred
  42.231 +#         (this should probably be a different directive, rather than a flag)
  42.232 +#
  42.233 +# Default:
  42.234 +#     (no plugins registered)
  42.235 +#
  42.236 +# Example:
  42.237 +#     <plugin>
  42.238 +#       name unprotected_ldap
  42.239 +#       handler Products.zasync.plugins.query_unprotected_ldap
  42.240 +#       timeout 60
  42.241 +#     </plugin>
  42.242 +
  42.243 +# <plugin unprotected_ldap>
  42.244 +#   handler zasync.plugins.query_unprotected_ldap
  42.245 +#   timeout 60
  42.246 +#   # description Query ldap and return the results, if any
  42.247 +#   # retry yes
  42.248 +# </plugin>
  42.249 +
  42.250 +# <plugin protected_ldap>
  42.251 +#   handler zasync.plugins.query_protected_ldap
  42.252 +#   timeout 60
  42.253 +#   # description Query ldaps (over SSL) and return the results, if any
  42.254 +#   # retry yes
  42.255 +# </plugin>
  42.256 +
  42.257 +<plugin zope_exec>
  42.258 +  handler zasync.plugins.zope_exec
  42.259 +  # 14400 seconds is four hours
  42.260 +  timeout 14400
  42.261 +  zope-aware yes
  42.262 +  # description Perform tasks within Zope off the main app server
  42.263 +  # retry yes
  42.264 +</plugin>
  42.265 +
  42.266 +<plugin schedule>
  42.267 +  handler zasync.plugins.schedule
  42.268 +  # 86400 seconds is one day
  42.269 +  timeout 86400
  42.270 +  # description Call the deferred in the specified number of seconds
  42.271 +  # retry yes
  42.272 +</plugin>
  42.273 +
  42.274 +<plugin aggregate>
  42.275 +  handler zasync.plugins.aggregatePlugins
  42.276 +  # 14400 seconds is four hours
  42.277 +  timeout 14400
  42.278 +  zope-aware yes
  42.279 +  # description Aggregate calls to other plugins
  42.280 +  retry no
  42.281 +</plugin>
  42.282 +
  42.283 +##############################################################################
  42.284 +# configure the loggers
  42.285 +##############################################################################
  42.286 +
  42.287 +# Directives: logger
  42.288 +#
  42.289 +# Description:
  42.290 +#     This area should define an event log and a plugin log.  The 
  42.291 +#     "event" logger logs Zope and zasync event
  42.292 +#     information.  The "plugin" logger logs plugin
  42.293 +#     information (or set the propagate key to "yes" to have these 
  42.294 +#     messages appear in the main event log).  Each logger section
  42.295 +#     may contain a "level" name/value pair which indicates the level
  42.296 +#     of logging detail to capture for this logger.  The default level
  42.297 +#     is INFO.  Level may be any of "CRITICAL", 'ERROR", WARN", "INFO",
  42.298 +#     "DEBUG", and "ALL".  Each logger section may additionally contain
  42.299 +#     one or more "handler" sections which indicates a types of log
  42.300 +#     "handlers" (file, syslog, NT event log, etc) to be used for the
  42.301 +#     logger being defined.  There are 5 types of handlers: logfile,
  42.302 +#     syslog, win32-eventlog, http-handler, email-notifier.  Each
  42.303 +#     handler type has its own set of allowable subkeys which define
  42.304 +#     aspects of the handler.  All handler sections also allow for the
  42.305 +#     specification of a "format" (the log message format string), a
  42.306 +#     "dateformat" (the log message format for date strings), and a
  42.307 +#     "level", which has the same semantics of the overall logger
  42.308 +#     level but overrides the logger's level for the handler it's
  42.309 +#     defined upon.
  42.310 +
  42.311 +<eventlog>
  42.312 +  level debug
  42.313 +  <logfile>
  42.314 +    path $INSTANCE/log/zasync_event.log
  42.315 +    level info
  42.316 +  </logfile>
  42.317 +</eventlog>
  42.318 +
  42.319 +<logger zasync>
  42.320 +  level debug
  42.321 +  <logfile>
  42.322 +    path $INSTANCE/log/zasync.log
  42.323 +    level debug
  42.324 +  </logfile>
  42.325 +</logger>
  42.326 +
  42.327 +<logger plugins>
  42.328 +  level debug
  42.329 +  propagate yes
  42.330 +</logger>
  42.331 +
  42.332 +
  42.333 +# Directive: verbose-traceback
  42.334 +#
  42.335 +# Description:
  42.336 +#     Set this directive to 'on' to cause  zasync to produce very verbose
  42.337 +#     tracebacks, including the locals and globals for every frame.
  42.338 +#
  42.339 +# Default: off
  42.340 +#
  42.341 +# Example:
  42.342 +#
  42.343 +#    verbose-traceback on
    43.1 new file mode 100644
    43.2 --- /dev/null
    43.3 +++ b/interfaces.py
    43.4 @@ -0,0 +1,285 @@
    43.5 +##############################################################################
    43.6 +#
    43.7 +# Copyright (c) 2004 Zope Corporation and Contributors. All Rights Reserved.
    43.8 +#
    43.9 +# This software is subject to the provisions of the Zope Public License,
   43.10 +# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
   43.11 +# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
   43.12 +# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
   43.13 +# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
   43.14 +# FOR A PARTICULAR PURPOSE.
   43.15 +#
   43.16 +##############################################################################
   43.17 +"""interfaces (call manager and zope deferred)
   43.18 +
   43.19 +$Id: interfaces.py,v 1.1.1.1 2004/10/10 23:37:06 poster Exp $
   43.20 +"""
   43.21 +
   43.22 +from Interface import Interface, Attribute
   43.23 +
   43.24 +class IZopeDeferred(Interface):
   43.25 +
   43.26 +    result = Attribute(
   43.27 +        """The result returned from the asynchronous worker.  Initializes to 
   43.28 +        None.  Because None is a possible vaild result from a plugin, this does
   43.29 +        not indicate the state of the deferred: use getState or getRawState.
   43.30 +        
   43.31 +        If you would like to also be protected from timeouts caused by the
   43.32 +        asynchronous worker being off-line, use getResult.""")
   43.33 +
   43.34 +    failure = Attribute(
   43.35 +        """The failure returned from the asynchronous worker.  Initializes to 
   43.36 +        None.  
   43.37 +        
   43.38 +        If you would like to also be protected from timeouts caused by the
   43.39 +        asynchronous worker being off-line, use getResult.""")
   43.40 +    
   43.41 +    original_result = Attribute(
   43.42 +        "The result, not affected by callbacks")
   43.43 +    
   43.44 +    original_failure = Attribute(
   43.45 +        "The original failure, not affected by callbacks")
   43.46 +
   43.47 +    creation_date = Attribute(
   43.48 +        """A standard Python datetime.datetime indicating the creation date
   43.49 +        of the deferred""")
   43.50 +    
   43.51 +    key = Attribute(
   43.52 +        """The value that can be used to obtain this deferred from the 
   43.53 +        asynchronous call manager via getDeferred""")
   43.54 +    
   43.55 +    local_key = Attribute(
   43.56 +        """The value that can be used to obtain this deferred from the
   43.57 +        asynchronous call manager via getSessionDeferred""")
   43.58 +    
   43.59 +    def getSignature():
   43.60 +        """Return the signature of the zasync call: a tuple of the zasync 
   43.61 +        plugin name, the ordered arguments, and the keyword arguments."""
   43.62 +    
   43.63 +    def getState():
   43.64 +        """returns 'success', 'failure', or None, indicating whether the 
   43.65 +        deferred has received a success result, a failure result, or no result
   43.66 +        yet.  If the deferred has not received any response for an 
   43.67 +        implementation-specific interval after the deferred's timeout, returns
   43.68 +        'failure'.  If the deferred eventually does receive a response, the
   43.69 +        state may change to reflect the true response."""
   43.70 +    
   43.71 +    def getRawState():
   43.72 +        """Same as getState, except that no timeout check occurs, so the value
   43.73 +        is always exactly what has been received from the asynchronous worker.
   43.74 +        """
   43.75 +    
   43.76 +    def getValue():
   43.77 +        """If deferred has been called with a success, returns result; if 
   43.78 +        deferred has been called with a failure, returns failure; if deferred
   43.79 +        has not yet been called and an implementation-specific interval after
   43.80 +        the deferred's timeout has passed, returns a failure; otherwise raises
   43.81 +        a RuntimeError."""
   43.82 +    
   43.83 +    def getRawValue():
   43.84 +        """If deferred has been called with a success, returns result; if 
   43.85 +        deferred has been called with a failure, returns failure; otherwise 
   43.86 +        raises a RuntimeError."""
   43.87 +    
   43.88 +    def getErrorMessage():
   43.89 +        """If getState returns 'failure', tries to convert the failure to an
   43.90 +        error message using the failure getErrorMessage API.  If not a failure,
   43.91 +        raises RuntimeError."""
   43.92 +    
   43.93 +    def callback(result):
   43.94 +        """stores result, changes state, and fires the first callback of the
   43.95 +        callback/errback chain.  If it has been called before, raises 
   43.96 +        RuntimeError."""
   43.97 +    
   43.98 +    def errback(failure):
   43.99 +        """stores failure, changes state, and fires the first errback of the
  43.100 +        callback/errback chain.  If it has been called before, raises 
  43.101 +        RuntimeError."""
  43.102 +    
  43.103 +    def addCallback(callback):
  43.104 +        """Convenience method for addCallbacks(callback)."""
  43.105 +    
  43.106 +    def addErrback(errback):
  43.107 +        """Convenience method for addCallbacks(errback=errback)."""
  43.108 +
  43.109 +    def addCallbacks(callback=None, errback=None):
  43.110 +        """add a callback, an errback, or both.  Passing neither raises a
  43.111 +        ValueError.  These calls may either be a basestring that will be
  43.112 +        evaluated as a TALES Expression; or a pair (list or tuple) of 
  43.113 +        (basestring, basestring) that indicate, in order, the name that the
  43.114 +        result should be stored as and the expression that should be 
  43.115 +        evaluated as a TALES expression (a parallel to the behavior of 
  43.116 +        tal:define).  Expressions have the following locally defined names 
  43.117 +        available to them:
  43.118 +        
  43.119 +        * 'nothing' is None;
  43.120 +         
  43.121 +        * 'user' is the current user;
  43.122 +         
  43.123 +        * 'userhome' is the container of the user's acl_users;
  43.124 +         
  43.125 +        * 'tool' is the asynchronous call manager;
  43.126 +         
  43.127 +        * 'root' is the Zope "physical" root;
  43.128 +         
  43.129 +        * 'deferred' is the current deferred object;
  43.130 +         
  43.131 +        * 'modules' is the standard Zope secure module loader;
  43.132 +         
  43.133 +        * 'request' is the request object at the time of the callback, which
  43.134 +          is virtually always an "artificial" request, not associated with
  43.135 +          a true browser request;
  43.136 +         
  43.137 +        * 'result' begins as the result of a successful callback, or as None
  43.138 +          if the deferred results in a failure, and then becomes any 
  43.139 +          non-failure result of previous callbacks or errbacks--see 
  43.140 +          deferred.original_result if you want the unambiguous initial value;
  43.141 +         
  43.142 +        * 'failure' begins as the failure of an unsuccessful callback, or as
  43.143 +          None if the deferred succeeds, and then becomes any 
  43.144 +          failure result of previous callbacks or errbacks--see 
  43.145 +          deferred.original_failure if you want the unambiguous initial value;
  43.146 +         
  43.147 +        * 'results' is a dictionary of any value stored using the 
  43.148 +          tal:define-inspired syntax of the pair described above."""
  43.149 +    
  43.150 +    def setTimeout(seconds):
  43.151 +        """set a timeout on a deferred.  This should be regarded as a rough 
  43.152 +        guide.  seconds must be >= 0, and if a timeout has been set for this
  43.153 +        deferred previously, seconds can only be <= the previous timeout.
  43.154 +        The deferred currently includes no way to cancel a timeout.  The 
  43.155 +        timeout value is calculated from the deferred's creation_date.
  43.156 +        seconds will be converted to an integer using standard casting (i.e.,
  43.157 +        "seconds = int(seconds)".
  43.158 +        
  43.159 +        The Twisted deferred API that inspired this variation has deprecated
  43.160 +        the equivalent call (also called "setTimeout").  The Zope deferred
  43.161 +        probably will continue to support this call because of its convenience.
  43.162 +        """
  43.163 +    def remainingSeconds():
  43.164 +        """returns number of seconds until timeout (positive integer), or 
  43.165 +        number of seconds since timeout (negative integer)"""
  43.166 +    
  43.167 +    # below are excerpts from the standard Zope Owned API. Zope deferreds use 
  43.168 +    # the deferred owner to specify the security privileges under which
  43.169 +    # callbacks and errbacks are run.
  43.170 +                                                                            
  43.171 +    def changeOwnership(user, recursive=0):
  43.172 +        """Change the ownership to the given user.  If 'recursive' is
  43.173 +        true then also take ownership of all sub-objects, otherwise
  43.174 +        sub-objects retain their ownership information.  'recursive' is not
  43.175 +        important for Zope deferreds.
  43.176 +        
  43.177 +        A representative method of the standard Zope "Owned" API: see 
  43.178 +        lib/python/AccessControl/Owned.py in the standard Zope distribution
  43.179 +        for the code.  Zope deferreds use the deferred owner to specify the
  43.180 +        security privileges under which callbacks and errbacks are run."""
  43.181 +    
  43.182 +    def getWrappedOwner():
  43.183 +        """Get the owner, as described in getOwnerTuple, modestly wrapped in 
  43.184 +        the user folder.  If the object is not owned, return None.  If the 
  43.185 +        owner's user database doesn't exist, return Nobody.  If the owner ID 
  43.186 +        does not exist in the user database, return Nobody.
  43.187 +        
  43.188 +        A representative method of the standard Zope "Owned" API: see 
  43.189 +        lib/python/AccessControl/Owned.py in the standard Zope distribution
  43.190 +        for the code.  Zope deferreds use the deferred owner to specify the
  43.191 +        security privileges under which callbacks and errbacks are run."""
  43.192 +    
  43.193 +    def getOwnerTuple():
  43.194 +        """Return a tuple, (userdb_path, user_id) for the owner.  Ownership 
  43.195 +        can be acquired, but only from the containment path.  If unowned, 
  43.196 +        return None.
  43.197 +        
  43.198 +        A representative method of the standard Zope "Owned" API: see 
  43.199 +        lib/python/AccessControl/Owned.py in the standard Zope distribution
  43.200 +        for the code.  Zope deferreds use the deferred owner to specify the
  43.201 +        security privileges under which callbacks and errbacks are run."""
  43.202 +
  43.203 +class IAbstractAsynchronousCallManager(Interface):
  43.204 +    """The current zeo-client-backed approach isn't the only one: we might be
  43.205 +    able to do this same API in the same Zope process, if that becomes
  43.206 +    desirable, for instance.  This abstract interface indicates the calls that
  43.207 +    should be maintained."""
  43.208 +    
  43.209 +    def listPlugins():
  43.210 +        """show plugin names available.  key is name, value is description.
  43.211 +        If no plugins, no engine should be expected (i.e., the tool will
  43.212 +        not work).  This part of the API is particularly unstable."""
  43.213 +    
  43.214 +    def putCall(_plugin, *args, **kwargs):
  43.215 +        """Make a call to the plugin named by _plugin, with associated args
  43.216 +        and kwargs.  Currently the only sanity check is to confirm that the 
  43.217 +        plugin name is available: no checks on the signature are performed.
  43.218 +        Importantly, arguments *must not* be persistent objects.  This check
  43.219 +        is also currently not enforced, but is very important.  Returns a
  43.220 +        zope deferred for this call, from which you can get the key and with
  43.221 +        which you can set callbacks, errbacks, and timeouts.
  43.222 +        """
  43.223 +    
  43.224 +    def putSessionCall(_plugin, *args, **kwargs):
  43.225 +        """As with putCall, except the deferred key includes the current user's 
  43.226 +        browser identifier, as defined by the standard Zope session machinery.
  43.227 +        """
  43.228 +    
  43.229 +    def getDeferred(d_id, default=None):
  43.230 +        """Given a deferred key, try to return the associated deferred, or else
  43.231 +        return the default."""
  43.232 +    
  43.233 +    def getSessionDeferred(d_id, default=None):
  43.234 +        """Given a deferred key or local_key, try to return the associated 
  43.235 +        deferred, or else return the default.  If a key is used, the key
  43.236 +        must include the browser id of the current user's session, or else
  43.237 +        raises ValueError."""
  43.238 +    
  43.239 +    def __len__():
  43.240 +        """returns the total number of deferreds currently stored by the tool.
  43.241 +        """
  43.242 +    
  43.243 +    def __nonzero__():
  43.244 +        "returns True, no matter what the len is"
  43.245 +    
  43.246 +    # private to deferreds
  43.247 +    
  43.248 +    def resolve(deferred):
  43.249 +        """Informs the asynchronous tool that the deferred has been resolved,
  43.250 +        and is no longer new or accepted (==pending)."""
  43.251 +
  43.252 +class IZasyncAsynchronousCallManager(IAbstractAsynchronousCallManager):
  43.253 +    
  43.254 +    # public
  43.255 +    
  43.256 +    rotation_period = Attribute(
  43.257 +        """The rotation period in seconds for the resolved deferred cache
  43.258 +        """)
  43.259 +        
  43.260 +    poll_interval = Attribute(
  43.261 +        """The approximate period of polling for the zasync process.
  43.262 +        """)
  43.263 +
  43.264 +    def getNextCacheRotation():
  43.265 +        """Return the next datetime.datetime when the resolved deferred cache 
  43.266 +        will rotate."""
  43.267 +    
  43.268 +    def resetNextCacheRotation(clear=False):
  43.269 +        """Reset the next cache rotation to (now + rotation_period), or if 
  43.270 +        clear is True, to never (until another call is made and the cache
  43.271 +        rotation time is set again)."""
  43.272 +    
  43.273 +    # private to zasync
  43.274 +    
  43.275 +    def setPlugins(plugins):
  43.276 +        """The Zasync engine should pass a sequence of available plugins,
  43.277 +        in which each plugin is represented by a pair of (name, description),
  43.278 +        when it starts; and attempt to set an empty tuple when in stops.
  43.279 +        This part of the API is particularly unstable."""
  43.280 +    
  43.281 +    def acceptAll():
  43.282 +        """Get all newly-added calls and move them to accepted status."""
  43.283 +    
  43.284 +    def getAcceptedCalls():
  43.285 +        """list the accepted calls."""
  43.286 +    
  43.287 +    def heartbeat():
  43.288 +        """Called every poll_interval (approximately).  Do work as needed;
  43.289 +        currently handles cache rotation."""
    44.1 new file mode 100644
    44.2 --- /dev/null
    44.3 +++ b/manager.py
    44.4 @@ -0,0 +1,811 @@
    44.5 +##############################################################################
    44.6 +#
    44.7 +# Copyright (c) 2004 Zope Corporation and Contributors. All Rights Reserved.
    44.8 +#
    44.9 +# This software is subject to the provisions of the Zope Public License,
   44.10 +# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
   44.11 +# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
   44.12 +# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
   44.13 +# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
   44.14 +# FOR A PARTICULAR PURPOSE.
   44.15 +#
   44.16 +##############################################################################
   44.17 +"""Asynchronous call manager
   44.18 +
   44.19 +A place to hang asynchronous calls and get their results
   44.20 +
   44.21 +$Id: manager.py,v 1.10 2005/09/17 03:26:06 poster Exp $
   44.22 +"""
   44.23 +
   44.24 +from types import NoneType
   44.25 +import datetime, random, sys, os, logging, sets, traceback
   44.26 +
   44.27 +from twisted.internet import defer
   44.28 +from twisted.python import failure
   44.29 +
   44.30 +from Persistence import Persistent
   44.31 +from ZODB.POSException import ConflictError
   44.32 +from ZEO.Exceptions import ClientDisconnected
   44.33 +from Globals import InitializeClass, Persistent
   44.34 +from Acquisition import aq_base, aq_inner, aq_parent
   44.35 +from AccessControl.Owned import ownerInfo
   44.36 +from AccessControl import ClassSecurityInfo, Unauthorized, \
   44.37 +    getSecurityManager
   44.38 +from AccessControl.SecurityManagement import newSecurityManager, \
   44.39 +    setSecurityManager
   44.40 +from Persistence import PersistentMapping
   44.41 +from OFS.SimpleItem import SimpleItem
   44.42 +from OFS.PropertyManager import PropertyManager
   44.43 +from BTrees import OOBTree
   44.44 +from Products.PageTemplates.PageTemplateFile import PageTemplateFile
   44.45 +from Products.PageTemplates.Expressions import getEngine, SecureModuleImporter
   44.46 +from Products.Sessions.BrowserIdManager import BROWSERID_MANAGER_NAME
   44.47 +from AccessControl.SecurityInfo import allow_class
   44.48 +
   44.49 +import permissions, bforests, interfaces
   44.50 +
   44.51 +UNCALLED = 0
   44.52 +CALLED = 1
   44.53 +FAILURE = -1
   44.54 +state_name_map = {UNCALLED: None, CALLED: 'success', FAILURE: 'failure'}
   44.55 +
   44.56 +# try to make failure.Failure available to restricted python
   44.57 +allow_class(failure.Failure) # in particular, this should allow access to
   44.58 +# 'getErrorMessage', 'getBriefTraceback', 'getTraceback', and 'trap'.
   44.59 +
   44.60 +def iterable(value):
   44.61 +    try:
   44.62 +        iter(value)
   44.63 +    except TypeError:
   44.64 +        return False
   44.65 +    else:
   44.66 +        return True
   44.67 +
   44.68 +def countable(value):
   44.69 +    try:
   44.70 +        len(value)
   44.71 +    except TypeError:
   44.72 +        return False
   44.73 +    else:
   44.74 +        return True
   44.75 +
   44.76 +class Reference:
   44.77 +    
   44.78 +    def __init__(self, obj):
   44.79 +        try:
   44.80 +            self._path = obj.getPhysicalPath()
   44.81 +        except (AttributeError, KeyError):
   44.82 +            self._path = None
   44.83 +        self._repr = repr(obj)
   44.84 +    
   44.85 +    def dereference(self, context):
   44.86 +        return context.restrictedTraverse(self._path, None)
   44.87 +    
   44.88 +    def __repr__(self):
   44.89 +        return self._repr
   44.90 +allow_class(Reference)
   44.91 +
   44.92 +class StandIn:
   44.93 +    
   44.94 +    data = klass = contents = None
   44.95 +    
   44.96 +    def __init__(self, obj, depth, contents=None):
   44.97 +        d = getattr(obj, '__dict__', None)
   44.98 +        if d is not None:
   44.99 +            self.data = sanitize(d, depth)
  44.100 +        self.klass = obj.__class__
  44.101 +        self.contents = contents
  44.102 +        self._repr = repr(obj)
  44.103 +    
  44.104 +    def __getattr__(self, name):    
  44.105 +        if self.data is not None and self.data is not MAX_DEPTH_MARKER:
  44.106 +            try:
  44.107 +                return self.data[name]
  44.108 +            except KeyError:
  44.109 +                pass
  44.110 +        raise AttributeError(name)
  44.111 +    
  44.112 +    def __repr__(self):
  44.113 +        return self._repr
  44.114 +allow_class(StandIn)
  44.115 +
  44.116 +MAX_DEPTH = 4
  44.117 +
  44.118 +class MAX_DEPTH_MARKER: pass # singleton
  44.119 +
  44.120 +def sanitize(value, depth=0, max_depth=MAX_DEPTH):
  44.121 +    if depth >= max_depth:
  44.122 +        return MAX_DEPTH_MARKER
  44.123 +    depth += 1
  44.124 +    if aq_base(value) is not value or isinstance(value, Persistent):
  44.125 +        if (aq_parent(value) is not None and 
  44.126 +            getattr(aq_base(value), 'getPhysicalPath', None) is not None):
  44.127 +            value = Reference(value)
  44.128 +        else:
  44.129 +            value = StandIn(value, depth) # XXX could theoretically do a better 
  44.130 +            # job for BTrees, but don't think I really want to make sending
  44.131 +            # those across the wire all that easy
  44.132 +    elif getattr(value, '__dict__', None) is not None:
  44.133 +        value = StandIn(value, depth) # XXX not ideal for subclasses of built 
  44.134 +        # in types, but oh well
  44.135 +    elif isinstance(value, (list, tuple, sets.Set)):
  44.136 +        val_type = type(value)
  44.137 +        contents = [sanitize(item, depth) for item in value]
  44.138 +        if val_type is list:
  44.139 +            value = contents
  44.140 +        else:
  44.141 +            try:
  44.142 +                value = val_type(contents)
  44.143 +            except TypeError:
  44.144 +                value = StandIn(value, depth, contents)
  44.145 +    elif isinstance(value, dict):
  44.146 +        contents = [
  44.147 +            (sanitize(k, depth), sanitize(v, depth)) for k, v in value.items()]
  44.148 +        val_type = type(value)
  44.149 +        try:
  44.150 +            value = val_type(contents)
  44.151 +        except TypeError:
  44.152 +            value = StandIn(value, depth, contents)
  44.153 +    elif not isinstance(value, basestring) and iterable(value):
  44.154 +        contents = None
  44.155 +        if countable(value):
  44.156 +            contents = [sanitize(item, depth) for item in value]
  44.157 +        value = StandIn(value, depth, contents)
  44.158 +    return value
  44.159 +
  44.160 +def cleanFailure(failure):
  44.161 +    # used instead of failure.cleanFailure for Zope-based failures.  See comment
  44.162 +    # in safe_repr below.  Could be monkey patched, but want to try and be a
  44.163 +    # good Twisted citizen for other Twisted services (particularly in the 
  44.164 +    # client).
  44.165 +    c = failure.__dict__.copy()
  44.166 +    
  44.167 +    def safe_repr(obj): # this is needed because Zope effectively raises some 
  44.168 +        # exceptions in __repr__ (I found one in Shared/DC/Scripts/Bindings.py, 
  44.169 +        # UnauthorizedBinding.__getattr__). :-(
  44.170 +        try:
  44.171 +            return repr(obj)
  44.172 +        except:
  44.173 +            return traceback.format_exception_only(*sys.exc_info()[:2])[0]
  44.174 +
  44.175 +    c['frames'] = [
  44.176 +        [
  44.177 +            v[0], v[1], v[2],
  44.178 +            [(j[0], safe_repr(j[1])) for j in v[3]],
  44.179 +            [(j[0], safe_repr(j[1])) for j in v[4]]
  44.180 +        ] for v in failure.frames
  44.181 +    ]
  44.182 +
  44.183 +    c['tb'] = None
  44.184 +
  44.185 +    if failure.stack is not None:
  44.186 +        c['stack'] = [
  44.187 +            [
  44.188 +                v[0], v[1], v[2],
  44.189 +                [(j[0], repr(j[1])) for j in v[3]],
  44.190 +                [(j[0], repr(j[1])) for j in v[4]]
  44.191 +            ] for v in failure.stack
  44.192 +        ]
  44.193 +
  44.194 +    c['pickled'] = 1
  44.195 +    failure.__dict__ = c
  44.196 +    return failure
  44.197 +
  44.198 +# Expression class copied from CMFCore/Expression.py
  44.199 +class Expression (Persistent):
  44.200 +    text = ''
  44.201 +    _v_compiled = None
  44.202 +
  44.203 +    security = ClassSecurityInfo()
  44.204 +
  44.205 +    def __init__(self, text):
  44.206 +        self.text = text
  44.207 +        self._v_compiled = getEngine().compile(text)
  44.208 +
  44.209 +    def __call__(self, econtext):
  44.210 +        compiled = self._v_compiled
  44.211 +        if compiled is None:
  44.212 +            compiled = self._v_compiled = getEngine().compile(self.text)
  44.213 +        res = compiled(econtext)
  44.214 +        if isinstance(res, Exception):
  44.215 +            raise res
  44.216 +        return res
  44.217 +
  44.218 +class Deferred(SimpleItem):
  44.219 +    
  44.220 +    result = failure = original_result = original_failure = None
  44.221 +    key = local_key = resolution_date = None
  44.222 +    timeout = 60 * 60 * 24 # one day; zasync plugins generally constrain 
  44.223 +    # timeouts more strictly
  44.224 +    
  44.225 +
  44.226 +    # keep webdav interface off deferreds and tool
  44.227 +    __implements__ = (interfaces.IZopeDeferred,)
  44.228 +    
  44.229 +    security = ClassSecurityInfo()
  44.230 +    
  44.231 +    def __init__(self, plugin, args, kwargs):
  44.232 +        self.creation_date = datetime.datetime.now()
  44.233 +        self.__signature = (plugin, args, kwargs)
  44.234 +        self.__callbacks = ()
  44.235 +        self.raw_state = UNCALLED # raw_state is not reliable.
  44.236 +        # Use getState.
  44.237 +    
  44.238 +    security.declareProtected(permissions.View, 'getSignature')
  44.239 +    def getSignature(self):
  44.240 +        return self.__signature
  44.241 +    
  44.242 +    security.declareProtected(permissions.View, 'getState')
  44.243 +    def getState(self):
  44.244 +        "returns 'success', 'failure', or None"
  44.245 +        state = self.raw_state
  44.246 +        if state==UNCALLED and (
  44.247 +            self.remainingSeconds() < -(self.aq_parent.poll_interval*3)):
  44.248 +            state=FAILURE
  44.249 +        global state_name_map
  44.250 +        return state_name_map[state]
  44.251 +    
  44.252 +    security.declareProtected(permissions.View, 'getRawState')
  44.253 +    def getRawState(self):
  44.254 +        "returns 'success', 'failure', or None"
  44.255 +        global state_name_map
  44.256 +        return state_name_map[self.raw_state]
  44.257 +    
  44.258 +    security.declareProtected(permissions.View, 'getValue')
  44.259 +    def getValue(self):
  44.260 +        state = self.raw_state
  44.261 +        if state==UNCALLED:
  44.262 +            if (self.remainingSeconds() < -(self.aq_parent.poll_interval*3)):
  44.263 +                return failure.Failure(defer.TimeoutError('Timed out.'))
  44.264 +            raise RuntimeError("Deferred still pending")
  44.265 +        if state==CALLED:
  44.266 +            return self.result
  44.267 +        else:
  44.268 +            return self.failure
  44.269 +    
  44.270 +    security.declareProtected(permissions.View, 'getRawValue')
  44.271 +    def getRawValue(self):
  44.272 +        state = self.raw_state
  44.273 +        if state==UNCALLED:
  44.274 +            raise RuntimeError("Deferred still pending")
  44.275 +        if state==CALLED:
  44.276 +            return self.result
  44.277 +        else:
  44.278 +            return self.failure
  44.279 +    
  44.280 +    security.declareProtected(permissions.View, 'getErrorMessage')
  44.281 +    def getErrorMessage(self):
  44.282 +        state = self.getState()
  44.283 +        if state != 'failure':
  44.284 +            raise RuntimeError("No failure state.", state or 'pending')
  44.285 +        value = self.getValue()
  44.286 +        return value and value.getErrorMessage() or 'Unknown error.'
  44.287 +    
  44.288 +    def __repr__(self):
  44.289 +        classname = type(self).__name__
  44.290 +        if aq_parent(self) is not None:
  44.291 +            location = " in %s" % '/'.join(self.getPhysicalPath())
  44.292 +        else:
  44.293 +            location = ""
  44.294 +        return "<%s (key %r)%s at %s>" % (
  44.295 +            classname, self.key, location, id(aq_base(self)))
  44.296 +    
  44.297 +    def __call_all(self):
  44.298 +        callbacks = self.__callbacks
  44.299 +        assert self.raw_state != UNCALLED
  44.300 +        res = (self.raw_state == FAILURE and self.failure or self.result)
  44.301 +        self.resolution_date = datetime.datetime.now()
  44.302 +        if not callbacks:
  44.303 +            return res
  44.304 +        user = self.getWrappedOwner()
  44.305 +        tool = aq_parent(self)
  44.306 +        root = self.getPhysicalRoot()
  44.307 +        results = {}
  44.308 +        old_sm = getSecurityManager()
  44.309 +        newSecurityManager(None, user)
  44.310 +        context_dict = {
  44.311 +            'nothing': None, # provided automatically but here for clarity
  44.312 +            'user': user,
  44.313 +            'userhome': aq_parent(aq_inner(aq_parent(aq_inner(user)))),
  44.314 +            'tool': tool,
  44.315 +            'root': root,
  44.316 +            'deferred': self,
  44.317 +            'here': self,
  44.318 +            'modules': SecureModuleImporter,
  44.319 +            'request': root.REQUEST,
  44.320 +            'result': self.result,
  44.321 +            'failure': self.failure,
  44.322 +            'original': res,
  44.323 +            'results': results,
  44.324 +            }
  44.325 +        try:
  44.326 +            callbacks = list(callbacks)
  44.327 +            self.__callbacks = ()
  44.328 +            while callbacks:
  44.329 +                callback, errback = callbacks.pop(0)
  44.330 +                if self.raw_state == CALLED:
  44.331 +                    call = callback
  44.332 +                else:
  44.333 +                    assert self.raw_state == FAILURE
  44.334 +                    call = errback
  44.335 +                name = None
  44.336 +                if call is None:
  44.337 +                    continue # no-op
  44.338 +                elif isinstance(call, tuple):
  44.339 +                    name, call = call
  44.340 +                context = getEngine().getContext(context_dict) # XXX looks like
  44.341 +                # we need to create a new one of these each time in order to 
  44.342 +                # make the change to result or failure stick.  Before tried
  44.343 +                # context.contexts['result'] = res (and similar for failure)
  44.344 +                # to change, but no success.
  44.345 +                try:
  44.346 +                    res = call(context)
  44.347 +                except ((ConflictError, KeyboardInterrupt, 
  44.348 +                         SystemExit, ClientDisconnected)):
  44.349 +                    raise
  44.350 +                except:
  44.351 +                    res = failure.Failure()
  44.352 +                else:
  44.353 +                    if name is not None:
  44.354 +                        results[name] = res
  44.355 +                if isinstance(res, failure.Failure):
  44.356 +                    cleanFailure(res) # remove object references
  44.357 +                    self.failure = context_dict["failure"] = res
  44.358 +                    self.raw_state = FAILURE
  44.359 +                else:
  44.360 +                    self.result = context_dict["result"] = res
  44.361 +                    self.raw_state = CALLED
  44.362 +        finally:
  44.363 +            setSecurityManager(old_sm)
  44.364 +        return res
  44.365 +        # manage transactions and retries externally (zasync)
  44.366 +    
  44.367 +    security.declarePrivate('callback')
  44.368 +    def callback(self, result):
  44.369 +        self.aq_inner.aq_parent.resolve(self)
  44.370 +        if self.raw_state != UNCALLED:
  44.371 +            # we don't want to abort the transaction because there's a good 
  44.372 +            # chance that we *need* to have the manager resolve this deferred.
  44.373 +            # Therefore, we return a failure rather than raise a transaction.
  44.374 +            # We do not call the callbacks.
  44.375 +            logging.getLogger('zasync').error(
  44.376 +                "Zope deferred %r has already been called" % (self.key,))
  44.377 +            return failure.Failure(
  44.378 +                RuntimeError("Deferred has already been called"))
  44.379 +        self.raw_state = CALLED
  44.380 +        self.original_result = self.result = result
  44.381 +        return self.__call_all()
  44.382 +    
  44.383 +    security.declarePrivate('errback')
  44.384 +    def errback(self, fail):
  44.385 +        self.aq_inner.aq_parent.resolve(self)
  44.386 +        if self.raw_state != UNCALLED:
  44.387 +            # we don't want to abort the transaction because there's a good 
  44.388 +            # chance that we *need* to have the manager resolve this deferred.
  44.389 +            # Therefore, we return a failure rather than raise a transaction.
  44.390 +            # We do not call the errbacks.
  44.391 +            logging.getLogger('zasync').error(
  44.392 +                "Zope deferred %r has already been called" % (self.key,))
  44.393 +            return failure.Failure(
  44.394 +                RuntimeError("Deferred has already been called"))
  44.395 +        self.raw_state = FAILURE
  44.396 +        self.original_failure = self.failure = fail
  44.397 +        return self.__call_all()
  44.398 +    
  44.399 +    security.declareProtected(permissions.ModifyDeferred, 
  44.400 +                              'addCallback')
  44.401 +    def addCallback(self, callback):
  44.402 +        return self.addCallbacks(callback)
  44.403 +    
  44.404 +    security.declareProtected(permissions.ModifyDeferred, 
  44.405 +                              'addErrback')
  44.406 +    def addErrback(self, errback):
  44.407 +        return self.addCallbacks(errback=errback)
  44.408 +        
  44.409 +    def _convertArg(self, arg, default):
  44.410 +        if not arg or arg==default:
  44.411 +            arg = None
  44.412 +        else:
  44.413 +            if isinstance(arg, basestring):
  44.414 +                arg = Expression(arg)
  44.415 +            elif (isinstance(arg, (list, tuple)) and 
  44.416 +                  len(arg)==2 and 
  44.417 +                  isinstance(arg[0], basestring) and
  44.418 +                  isinstance(arg[1], basestring)):
  44.419 +                arg = (arg[0], Expression(arg[1]))
  44.420 +            else:
  44.421 +                raise ValueError(
  44.422 +                    "Must be None, expression string, or pair of (result name, "
  44.423 +                    "expression string)", arg)
  44.424 +        return arg
  44.425 +    
  44.426 +    security.declareProtected(permissions.ModifyDeferred, 
  44.427 +                              'addCallbacks')
  44.428 +    def addCallbacks(self, callback=None, errback=None):
  44.429 +        if errback is None and callback is None:
  44.430 +            raise ValueError("No callback or errback added")
  44.431 +        callback = self._convertArg(callback, 'result')
  44.432 +        errback = self._convertArg(errback, 'failure')
  44.433 +        self.__callbacks += ((callback, errback),)
  44.434 +        # call immediately if called
  44.435 +        if self.raw_state != UNCALLED:
  44.436 +            return self.__call_all()
  44.437 +        return self
  44.438 +    
  44.439 +    # XXX this part of the API may be in flux; Twisted has deprecated it
  44.440 +    security.declareProtected(permissions.ModifyDeferred, 
  44.441 +                              'setTimeout')
  44.442 +    def setTimeout(self, seconds):
  44.443 +        if self.getState() is not None:
  44.444 +            raise RuntimeError(
  44.445 +                "Cannot set timeout on completed deferred")
  44.446 +        seconds = int(seconds)
  44.447 +        if seconds < 0:
  44.448 +            raise ValueError("Seconds must be >= 0")
  44.449 +        if self.timeout < seconds:
  44.450 +            raise ValueError(
  44.451 +                "Cannot set timeout greater than previous value", self.timeout)
  44.452 +        self.timeout = seconds
  44.453 +    
  44.454 +    security.declareProtected(permissions.View, 'remainingSeconds')
  44.455 +    def remainingSeconds(self):
  44.456 +        """returns number of seconds until timeout (positive integer), or 
  44.457 +        number of seconds since timeout (negative integer)"""
  44.458 +        timeout = self.timeout
  44.459 +        end = self.creation_date + datetime.timedelta(seconds = timeout)
  44.460 +        now = datetime.datetime.now()
  44.461 +        delta = end - now
  44.462 +        if delta < datetime.timedelta():
  44.463 +            timeout = -(-delta).seconds # XXX should include day seconds too
  44.464 +        else:
  44.465 +            timeout = delta.seconds
  44.466 +        return timeout
  44.467 +InitializeClass(Deferred)
  44.468 +
  44.469 +def getDeferredInfo(context, dictionary, sort_field=None, reverse=False):
  44.470 +    res = []
  44.471 +    for d in dictionary.values():
  44.472 +        d = d.__of__(context)
  44.473 +        plugin, args, kwargs = d.getSignature()
  44.474 +        state = d.getState()
  44.475 +        value = original_value = original_state = None
  44.476 +        if state is not None:
  44.477 +            value = d.getValue()
  44.478 +            if d.original_failure is None:
  44.479 +                original_value = d.original_result
  44.480 +                original_state = state_name_map[CALLED]
  44.481 +            else:
  44.482 +                original_value = d.original_failure
  44.483 +                original_state = state_name_map[FAILURE]
  44.484 +        info ={
  44.485 +            'key': d.key, 
  44.486 +            'user': d.getOwnerTuple(), 
  44.487 +            'plugin': plugin,
  44.488 +            'args': args,
  44.489 +            'kwargs': kwargs,
  44.490 +            'state': state,
  44.491 +            'value': value,
  44.492 +            'creation_date': d.creation_date,
  44.493 +            'resolution_date': d.resolution_date,
  44.494 +            'original_state': original_state,
  44.495 +            'original_value': original_value}
  44.496 +        if sort_field is None:
  44.497 +            res.append(info)
  44.498 +        else:
  44.499 +            res.append((info.get(sort_field), info))
  44.500 +    if sort_field is not None:
  44.501 +        res.sort()
  44.502 +        if reverse:
  44.503 +            res.reverse()
  44.504 +        res = [val for sort, val in res]
  44.505 +    return res
  44.506 +
  44.507 +class AsynchronousCallManager(PropertyManager, SimpleItem):
  44.508 +    """a tool that holds deferreds for zasync to manipulate"""
  44.509 +    security = ClassSecurityInfo()
  44.510 +    
  44.511 +    manage_options = (
  44.512 +        ({'label':'Overview', 'action':'manage_overview',},
  44.513 +         {'label':'Calls', 'action':'manage_calls',},) +
  44.514 +        PropertyManager.manage_options
  44.515 +        + SimpleItem.manage_options)
  44.516 +    
  44.517 +    _properties = (
  44.518 +        {'id':'rotation_period', 'type': 'int', 'mode':'w',
  44.519 +         'label': 'Resolved cache rotation period in seconds'},
  44.520 +        {'id':'poll_interval', 'type': 'int', 'mode': 'w',
  44.521 +         'label': 'Interval between zasync call polls'},
  44.522 +        )
  44.523 +    
  44.524 +    id = 'asynchronous_call_manager'
  44.525 +    title = meta_type = 'Asynchronous Call Manager'
  44.526 +    icon = "misc_/zasync/tool.gif"
  44.527 +    rotation_period = 60*60*24 # a day
  44.528 +    poll_interval = 5
  44.529 +    __plugins = ()
  44.530 +    _next_rotate = _last_ping = _last_pong = None
  44.531 +    
  44.532 +    # keep webdav interface off deferreds and tool
  44.533 +    __implements__ = (interfaces.IZasyncAsynchronousCallManager,)
  44.534 +    
  44.535 +    def _getBrowserId(self):
  44.536 +        bim = getattr(self, BROWSERID_MANAGER_NAME)
  44.537 +        return bim.getBrowserId()
  44.538 +    
  44.539 +    def __init__(self, id=None):
  44.540 +        if id is not None:
  44.541 +            self.id = id
  44.542 +        # items to be picked up by zasync
  44.543 +        self._new = OOBTree.OOBTree()
  44.544 +        # items collected by zasync from the queue
  44.545 +        self._accepted = OOBTree.OOBTree()
  44.546 +        # long term cache
  44.547 +        self._resolved = bforests.OOBForest()
  44.548 +        self._next_rotate = None
  44.549 +        self._last_ping = None
  44.550 +        self._last_pong = None
  44.551 +    
  44.552 +
  44.553 +    security.declareProtected(permissions.ViewManagementScreens, 
  44.554 +                              'manage_overview')
  44.555 +    manage_overview = PageTemplateFile(
  44.556 +        'www/controlAsynchronousCallManagerForm.zpt', globals(),
  44.557 +        __name__='manage_overview')
  44.558 +    
  44.559 +
  44.560 +    security.declareProtected(permissions.ViewManagementScreens, 
  44.561 +                              'manage_calls')
  44.562 +    manage_calls = PageTemplateFile(
  44.563 +        'www/analyzeCalls.zpt', globals(),
  44.564 +        __name__='manage_calls')
  44.565 +    
  44.566 +    security.declareProtected(permissions.ViewManagementScreens, 
  44.567 +                              'ping')
  44.568 +    def ping(self, REQUEST=None):
  44.569 +        """make a ping request to the zasync client to see if it replies
  44.570 +        in the heartbeat method."""
  44.571 +        self._last_ping = datetime.datetime.now()
  44.572 +        self._last_pong = None
  44.573 +        if REQUEST is not None:
  44.574 +            REQUEST.RESPONSE.redirect(
  44.575 +                '%s/manage_overview' % self.absolute_url())
  44.576 +
  44.577 +    security.declareProtected(permissions.ViewManagementScreens, 
  44.578 +                              'getLastPing')
  44.579 +    def getLastPing(self):
  44.580 +        return self._last_ping
  44.581 +
  44.582 +    security.declareProtected(permissions.ViewManagementScreens, 
  44.583 +                              'getLastPong')
  44.584 +    def getLastPong(self):
  44.585 +        return self._last_pong
  44.586 +    
  44.587 +    # property code copied from CMFCore/utils.py SimpleItemWithProperties >>>
  44.588 +    security.declarePrivate('manage_addProperty')
  44.589 +    security.declarePrivate('manage_delProperties')
  44.590 +    security.declarePrivate('manage_changePropertyTypes')
  44.591 +
  44.592 +    def manage_propertiesForm(self, REQUEST, *args, **kw):
  44.593 +        'An override that makes the schema fixed.'
  44.594 +        my_kw = kw.copy()
  44.595 +        my_kw['property_extensible_schema__'] = 0
  44.596 +        return apply(PropertyManager.manage_propertiesForm,
  44.597 +                     (self, self, REQUEST,) + args, my_kw)
  44.598 +
  44.599 +    security.declarePublic('propertyLabel')
  44.600 +    def propertyLabel(self, id):
  44.601 +        """Return a label for the given property id
  44.602 +        """
  44.603 +        for p in self._properties:
  44.604 +            if p['id'] == id:
  44.605 +                return p.get('label', id)
  44.606 +        return id
  44.607 +    # <<< end copy from CMF
  44.608 +    
  44.609 +    security.declarePrivate('resolve')
  44.610 +    def resolve(self, deferred):
  44.611 +        try:
  44.612 +            del self._accepted[deferred.key]
  44.613 +        except KeyError:
  44.614 +            pass # presumably resolved before
  44.615 +        else:
  44.616 +            self._resolved[deferred.key] = deferred
  44.617 +            if self._next_rotate is None and self.rotation_period:
  44.618 +                self._next_rotate = (
  44.619 +                    datetime.datetime.now() + 
  44.620 +                    datetime.timedelta(seconds=self.rotation_period))
  44.621 +    
  44.622 +    security.declarePublic('getNextCacheRotation')
  44.623 +    def getNextCacheRotation(self):
  44.624 +        return self._next_rotate
  44.625 +    
  44.626 +    security.declareProtected(permissions.ViewManagementScreens, 
  44.627 +                              'resetNextCacheRotation')
  44.628 +    def resetNextCacheRotation(self, clear=False):
  44.629 +        if clear:
  44.630 +            res = self._next_rotate = None
  44.631 +        else:
  44.632 +            res = self._next_rotate = (
  44.633 +                datetime.datetime.now() +
  44.634 +                datetime.timedelta(seconds=self.rotation_period))
  44.635 +        return res
  44.636 +
  44.637 +    security.declarePrivate('setPlugins')
  44.638 +    def setPlugins(self, plugins): # protect plugins by groups or perms?
  44.639 +        "set item list, name: description"
  44.640 +        # XXX it would be good to include a signature validation function
  44.641 +        # XXX it would be good to be able to specify if some plugins are
  44.642 +        # not available to session calls--although that needs to be more
  44.643 +        # general; session call approach may need to be adjusted.  See XXX in
  44.644 +        # getSessionDeferred.
  44.645 +        self.__plugins = tuple(plugins)
  44.646 +
  44.647 +    security.declareProtected(permissions.View, 'listPlugins')
  44.648 +    def listPlugins(self):
  44.649 +        """show plugin names available.  key is name, value is description.
  44.650 +        If no plugins, no engine should be expected (i.e., the tool will
  44.651 +        not work)."""
  44.652 +        return dict(self.__plugins)
  44.653 +    
  44.654 +    security.declarePrivate('_putCall')
  44.655 +    def _putCall(self, _id_prefix, _plugin, *args, **kwargs):
  44.656 +        if _plugin not in self.listPlugins(): # keys of dict
  44.657 +            raise ValueError(
  44.658 +                "Requested plugin is not registered")
  44.659 +        while 1:
  44.660 +            randomizer = key = random.randint(-sys.maxint-1, sys.maxint)
  44.661 +            if _id_prefix is not None:
  44.662 +                key = _id_prefix + (randomizer,)
  44.663 +            if self.getDeferred(key) is None:
  44.664 +                break
  44.665 +        args = sanitize(args)
  44.666 +        kwargs = sanitize(kwargs)
  44.667 +        d = Deferred(_plugin, args, kwargs)
  44.668 +        d.key = key
  44.669 +        d.id = repr(key)
  44.670 +        d.local_key = randomizer
  44.671 +        self._new[key] = d
  44.672 +        wrapped = d.__of__(self)
  44.673 +        wrapped.manage_fixupOwnershipAfterAdd()
  44.674 +        user=getSecurityManager().getUser()
  44.675 +        if user is not None:
  44.676 +            userid=user.getId()
  44.677 +            if userid is not None:
  44.678 +                wrapped.manage_setLocalRoles(userid, ['Owner'])
  44.679 +        return wrapped
  44.680 +    
  44.681 +    security.declareProtected(
  44.682 +        permissions.MakeAsynchronousApplicationCalls, 'putCall')
  44.683 +    def putCall(self, _plugin, *args, **kwargs):
  44.684 +        return self._putCall(
  44.685 +            None, _plugin, *args, **kwargs)
  44.686 +    
  44.687 +    security.declareProtected(
  44.688 +        permissions.MakeAsynchronousSessionCalls, 'putSessionCall')
  44.689 +    def putSessionCall(self, _plugin, *args, **kwargs):
  44.690 +        return self._putCall(
  44.691 +            (self._getBrowserId(),), _plugin, *args, **kwargs)
  44.692 +    
  44.693 +    security.declareProtected(
  44.694 +        permissions.MakeAsynchronousApplicationCalls, 'getDeferred')
  44.695 +    def getDeferred(self, d_id, default=None):
  44.696 +        for src in (self._new, self._accepted, self._resolved):
  44.697 +            res = src.get(d_id)
  44.698 +            if res is not None:
  44.699 +                return res.__of__(self)
  44.700 +        return default
  44.701 +    
  44.702 +    security.declareProtected(
  44.703 +        permissions.MakeAsynchronousSessionCalls, 'getSessionDeferred')
  44.704 +    def getSessionDeferred(self, d_id, default=None):
  44.705 +        # XXX this API doesn't support the use case of users who have limited
  44.706 +        # permissions but still should be able to store away keys for later
  44.707 +        # look-up, irrespective of sessions.  We should either have three 
  44.708 +        # levels of deferred call or let plugins specify permissions--but then
  44.709 +        # in what context?
  44.710 +        bid = self._getBrowserId()
  44.711 +        if isinstance(d_id, tuple):
  44.712 +            if d_id[0] != bid:
  44.713 +                return default
  44.714 +            d_id = d_id[-1]
  44.715 +        return self.getDeferred((bid, d_id), default)
  44.716 +    
  44.717 +    def __len__(self): # potentially expensive
  44.718 +        return len(self._new) + len(self._accepted) + len(self._resolved)
  44.719 +
  44.720 +    def __nonzero__(self):
  44.721 +        return True
  44.722 +    
  44.723 +    security.declareProtected(
  44.724 +        permissions.ViewManagementScreens, 'listNewCalls')
  44.725 +    def listNewCalls(self, sort='creation_date', reverse=True):
  44.726 +        return getDeferredInfo(self, self._new, sort, reverse)
  44.727 +    
  44.728 +    security.declareProtected(
  44.729 +        permissions.ViewManagementScreens, 'listAcceptedCalls')
  44.730 +    def listAcceptedCalls(self, sort='creation_date', reverse=True):
  44.731 +        return getDeferredInfo(self, self._accepted, sort, reverse)
  44.732 +    
  44.733 +    security.declareProtected(
  44.734 +        permissions.ViewManagementScreens, 'listResolvedCalls')
  44.735 +    def listResolvedCalls(self, bucket=None, sort='creation_date', reverse=True):
  44.736 +        if bucket is None:
  44.737 +            d = self._resolved
  44.738 +        else:
  44.739 +            d = self._resolved.buckets[bucket]
  44.740 +        return getDeferredInfo(self, d, sort, reverse)
  44.741 +    
  44.742 +    security.declareProtected(
  44.743 +        permissions.ViewManagementScreens, 'lenNewCalls')
  44.744 +    def lenNewCalls(self):
  44.745 +        return len(self._new)
  44.746 +    
  44.747 +    security.declareProtected(
  44.748 +        permissions.ViewManagementScreens, 'lenAcceptedCalls')
  44.749 +    def lenAcceptedCalls(self):
  44.750 +        return len(self._accepted)
  44.751 +    
  44.752 +    security.declareProtected(
  44.753 +        permissions.ViewManagementScreens, 'lenResolvedCalls')
  44.754 +    def lenResolvedCalls(self, bucket=None):
  44.755 +        if bucket is None:
  44.756 +            d = self._resolved
  44.757 +        else:
  44.758 +            d = self._resolved.buckets[bucket]
  44.759 +        return len(d)
  44.760 +    
  44.761 +    security.declareProtected(
  44.762 +        permissions.ViewManagementScreens, 'lenResolvedBuckets')
  44.763 +    def lenResolvedBuckets(self):
  44.764 +        return len(self._resolved.buckets)
  44.765 +    
  44.766 +    security.declareProtected(
  44.767 +        permissions.ViewManagementScreens, 'nextResolvedBucketRotation')
  44.768 +    def nextResolvedBucketRotation(self):
  44.769 +        return self._next_rotate
  44.770 +    
  44.771 +    security.declarePrivate('acceptAll')
  44.772 +    def acceptAll(self):
  44.773 +        self._accepted.update(self._new)
  44.774 +        res = self._new.values()
  44.775 +        self._new.clear()
  44.776 +        return res
  44.777 +    
  44.778 +    security.declarePrivate('getAcceptedCalls')
  44.779 +    def getAcceptedCalls(self):
  44.780 +        return self._accepted.values()
  44.781 +    
  44.782 +    security.declarePrivate('heartbeat')
  44.783 +    def heartbeat(self):
  44.784 +        next_rotate = self._next_rotate
  44.785 +        now = datetime.datetime.now()
  44.786 +        if next_rotate is not None and next_rotate < now:
  44.787 +            self._resolved.rotateBucket()
  44.788 +            if self._resolved: # i.e., not empty; len is expensive!
  44.789 +                self._next_rotate = (
  44.790 +                    now + 
  44.791 +                    datetime.timedelta(seconds=self.rotation_period))
  44.792 +            else:
  44.793 +                self._next_rotate = None
  44.794 +        if self._last_ping is not None and self._last_pong is None:
  44.795 +            self._last_pong = now
  44.796 +InitializeClass(AsynchronousCallManager)
  44.797 +
  44.798 +constructAsynchronousCallManagerForm = PageTemplateFile(
  44.799 +    'www/constructAsynchronousCallManagerForm.zpt', globals(),
  44.800 +    __name__='manage_addSchedulerForm')
  44.801 +
  44.802 +def constructAsynchronousCallManager(
  44.803 +    dispatcher, id="asynchronous_call_manager", 
  44.804 +    poll_interval=None, rotation_period=None, RESPONSE=None):
  44.805 +    """Construct an AsynchronousCallManager"""
  44.806 +    acm = AsynchronousCallManager(id)
  44.807 +    if poll_interval is not None:
  44.808 +        acm.poll_interval = max(poll_interval, 2)
  44.809 +    if rotation_period is not None:
  44.810 +        acm.rotation_period = abs(rotation_period)
  44.811 +    container = dispatcher.this()
  44.812 +    container._setObject(id, acm)
  44.813 +
  44.814 +    if RESPONSE is not None:
  44.815 +        RESPONSE.redirect(container.absolute_url() + '/manage_main')
    45.1 new file mode 100644
    45.2 --- /dev/null
    45.3 +++ b/manager.txt
    45.4 @@ -0,0 +1,509 @@
    45.5 +The manager.py file contains code for both the asynchronous call manager and the 
    45.6 +zope deferred objects that it manages.  The asynchronous call manager is a
    45.7 +tool that enables a client thread, such as a thread handling a Zope request,
    45.8 +to request that a task be performed asynchronously.  The current version of
    45.9 +this product relies on a ZEO client process (see the client directory) that 
   45.10 +could be running on another machine to perform the asynchronous tasks.
   45.11 +
   45.12 +A call creates a zope Deferred object which represents the call.  Status and 
   45.13 +results can be obtained from it via polling (pull); and callbacks can be 
   45.14 +registered when the result is obtained (push).  Callbacks are TALES expressions
   45.15 +performed in the security context of the owner of the deferred--by default, the
   45.16 +user who created the call.  Calls can be made and obtained for the current
   45.17 +session if a user has the "Make asynchronous session calls" permission, and 
   45.18 +made and obtained generally if a user has the "Make asynchronous application 
   45.19 +calls" permission.
   45.20 +
   45.21 +The design is relatively simple, and is significantly inspired by the Twisted
   45.22 +project's deferred model. Probably the trickiest part of the model is the
   45.23 +callback/errback scheme. See
   45.24 +http://twistedmatrix.com/documents/current/howto/defer for the Twisted model
   45.25 +for extra background.  This design is not identical, but very similar.
   45.26 +
   45.27 +Let's look at an extended example.  This will be run as a bit of an integration
   45.28 +test--the code here actually sets up a dummy Zope application instance and 
   45.29 +manipulates it, emulating some basic browser behaviors.  We'll begin by setting 
   45.30 +up an asynchronous call manager in an actual Zope application object.
   45.31 +
   45.32 +>>> from Products.zasync.tests import zopetestutils
   45.33 +>>> app = zopetestutils.startRequest(zopetestutils.ADMIN)
   45.34 +>>> from Products.zasync.manager import AsynchronousCallManager
   45.35 +>>> acm = AsynchronousCallManager()
   45.36 +>>> acm_id = app._setObject(acm.getId(), acm)
   45.37 +>>> acm = app._getOb(acm_id)
   45.38 +
   45.39 +Now we have an AsynchronousCallManager (acm) installed in the Zope root, using
   45.40 +the default id.  This is the typical configuration.  We'll also create a couple
   45.41 +of sample users.
   45.42 +
   45.43 +>>> uf = app.acl_users
   45.44 +>>> uf.userFolderAddUser('addie', '123', ('Manager',), ()) # = admin
   45.45 +>>> uf.userFolderAddUser('arthur', '123', (), ()) # = authenticated
   45.46 +
   45.47 +Now we'll save all that and log back in as addie the admin.
   45.48 +
   45.49 +>>> zopetestutils.closeRequest()
   45.50 +>>> app = zopetestutils.startRequest(('/acl_users', 'addie'))
   45.51 +
   45.52 +Let's look at some properties of an empty acm.
   45.53 +
   45.54 +>>> acm = app._getOb(acm_id)
   45.55 +>>> acm.rotation_period # a day in seconds by default
   45.56 +86400
   45.57 +>>> acm.poll_interval
   45.58 +5
   45.59 +>>> len(acm)
   45.60 +0
   45.61 +>>> bool(acm)
   45.62 +True
   45.63 +
   45.64 +Cache rotation doesn't begin until there are deferreds that have been resolved.
   45.65 +
   45.66 +>>> acm.getNextCacheRotation() # returns None
   45.67 +
   45.68 +There are no registered plugins yet.  This means that putting a call in will 
   45.69 +raise a value error.
   45.70 +
   45.71 +>>> acm.listPlugins()
   45.72 +{}
   45.73 +>>> acm.putCall("this plugin doesn't exist")
   45.74 +Traceback (most recent call last):
   45.75 +...
   45.76 +ValueError: Requested plugin is not registered
   45.77 +>>> acm.putSessionCall("this plugin doesn't exist")
   45.78 +Traceback (most recent call last):
   45.79 +...
   45.80 +ValueError: Requested plugin is not registered
   45.81 +
   45.82 +There are also no deferreds.  This means that asking for them will not get 
   45.83 +much.
   45.84 +
   45.85 +>>> acm.getDeferred('foo') # returns None, the default "no value"
   45.86 +>>> acm.getDeferred('foo', 42) # now we explicitly provide 42 as the default
   45.87 +42
   45.88 +>>> acm.getSessionDeferred('foo') # returns None
   45.89 +>>> acm.getSessionDeferred('foo', 42)
   45.90 +42
   45.91 +
   45.92 +These calls are ones the zasync client would make to get deferreds.
   45.93 +
   45.94 +>>> list(acm.acceptAll())
   45.95 +[]
   45.96 +>>> list(acm.getAcceptedCalls())
   45.97 +[]
   45.98 +
   45.99 +OK, enough of that.  Let's register some fake plugins and make some calls.
  45.100 +This registration would normally be done by the zasync client, not by a user
  45.101 +account.
  45.102 +
  45.103 +The API to register plugins should be considered very unstable, by the way.
  45.104 +
  45.105 +>>> acm.setPlugins((
  45.106 +...     ('query_ldap', '...do an ldap query...'),
  45.107 +...     ('zope_exec', '...execute a series of actions in Zope...')))
  45.108 +>>> import pprint
  45.109 +>>> pprint.pprint(acm.listPlugins())
  45.110 +{'query_ldap': '...do an ldap query...',
  45.111 + 'zope_exec': '...execute a series of actions in Zope...'}
  45.112 + 
  45.113 +First we'll make an application-level deferred.  When you make a call, it is 
  45.114 +*essential* to realize that you should not pass persistent objects.  This is
  45.115 +not yet enforced, but it should be.  If you pass persistent objects, there's
  45.116 +a reasonable chance that very bad things will happen, like POSKey errors in 
  45.117 +your database.  Bad.  Do not do this.  Argument constraints and signature
  45.118 +checks are a "to do" for the acm.
  45.119 + 
  45.120 +>>> appDeferred = acm.putCall('query_ldap', 'fake', 'arguments', fake=True)
  45.121 +>>> len(acm)
  45.122 +1
  45.123 +>>> from Acquisition import aq_base, aq_parent, aq_inner
  45.124 +>>> aq_base(aq_parent(appDeferred)) is aq_base(acm)
  45.125 +True
  45.126 +>>> res = acm.getDeferred(appDeferred.key)
  45.127 +>>> aq_base(res) is aq_base(appDeferred)
  45.128 +True
  45.129 +>>> aq_base(aq_parent(res)) is aq_base(acm)
  45.130 +True
  45.131 +>>> acm.getSessionDeferred(appDeferred.key) # None
  45.132 +
  45.133 +Let's look at the deferred a little bit.  The call signature is obtained from 
  45.134 +getSignature--it returns a tuple of the plugin name, the ordered arguments, and 
  45.135 +the keyword arguments.
  45.136 +
  45.137 +>>> pprint.pprint(appDeferred.getSignature())
  45.138 +('query_ldap', ('fake', 'arguments'), {'fake': True})
  45.139 +
  45.140 +The status is currently uncalled.
  45.141 +
  45.142 +>>> appDeferred.getState() # None
  45.143 +>>> appDeferred.getRawState() # None
  45.144 +
  45.145 +getValue will raise an error while the deferred has not been called
  45.146 +
  45.147 +>>> appDeferred.getValue()
  45.148 +Traceback (most recent call last):
  45.149 +...
  45.150 +RuntimeError: Deferred still pending
  45.151 +
  45.152 +Deferreds have a maximum timeout of one day, which can be constrained further
  45.153 +by the user or the zasync client calling setTimeout.
  45.154 +
  45.155 +>>> appDeferred.timeout
  45.156 +86400
  45.157 +
  45.158 +The remaining seconds call gives a feel for how many seconds are left until the
  45.159 +deferred will timeout.  This is approximate.
  45.160 +
  45.161 +>>> remaining = appDeferred.remainingSeconds() 
  45.162 +>>> 86398 <= remaining <= 86400 # giving a two second window
  45.163 +True
  45.164 +
  45.165 +The calculation is based on the creation_date, so we can manipulate that for a
  45.166 +few examples.
  45.167 +
  45.168 +>>> import datetime
  45.169 +>>> original_creation_date = appDeferred.creation_date
  45.170 +>>> appDeferred.creation_date -= datetime.timedelta(seconds = 43200)
  45.171 +>>> remaining = appDeferred.remainingSeconds() 
  45.172 +>>> 43198 <= remaining <= 43200 # giving a two second window
  45.173 +True
  45.174 +>>> appDeferred.creation_date -= datetime.timedelta(seconds=remaining+1)
  45.175 +>>> appDeferred.remainingSeconds() <= 0
  45.176 +True
  45.177 +
  45.178 +setTimeout is what you should use to actually modify a deferred's timeout, not
  45.179 +muck with the creation_date as we do here for example and test purposes.
  45.180 +Note that the timeout can only be set to a smaller and smaller value, never 
  45.181 +set larger or cleared.
  45.182 +
  45.183 +>>> appDeferred.creation_date = original_creation_date # undo our changes
  45.184 +>>> appDeferred.setTimeout(43200) # half a day
  45.185 +>>> remaining = appDeferred.remainingSeconds() 
  45.186 +>>> 43198 <= remaining <= 43200 # giving a two second window
  45.187 +True
  45.188 +>>> appDeferred.setTimeout(86400)
  45.189 +Traceback (most recent call last):
  45.190 +...
  45.191 +ValueError: ('Cannot set timeout greater than previous value', 43200)
  45.192 +>>> appDeferred.setTimeout(21600)
  45.193 +>>> remaining = appDeferred.remainingSeconds() 
  45.194 +>>> 21598 <= remaining <= 21600 # giving a two second window
  45.195 +True
  45.196 +
  45.197 +You can set a timeout of 0, but not anything less than 0.
  45.198 +
  45.199 +>>> appDeferred.setTimeout(-1)
  45.200 +Traceback (most recent call last):
  45.201 +...
  45.202 +ValueError: Seconds must be >= 0
  45.203 +
  45.204 +In order to protect users of the asynchronous call manager from a disabled
  45.205 +zasync client that never provides a success, timeout failure, or other 
  45.206 +failure, the deferred state and value calls provide a dynamically generated
  45.207 +failure value if the timeout has exceeded the timeout value plus three times
  45.208 +the poll interval.  This currently does not trigger the errbacks, but this may
  45.209 +be changed in future releases.
  45.210 +
  45.211 +We'll muck with the creation_date to fake this again.
  45.212 +
  45.213 +>>> appDeferred.getState() # None
  45.214 +>>> appDeferred.creation_date = original_creation_date - datetime.timedelta(
  45.215 +... seconds = appDeferred.timeout + 3 * appDeferred.poll_interval + 1)
  45.216 +>>> appDeferred.remainingSeconds() <= -(3 * appDeferred.poll_interval + 1)
  45.217 +True
  45.218 +>>> appDeferred.getState()
  45.219 +'failure'
  45.220 +>>> val = appDeferred.getValue()
  45.221 +>>> from twisted.python import failure
  45.222 +>>> isinstance(val, failure.Failure)
  45.223 +True
  45.224 +>>> from twisted.internet import defer
  45.225 +>>> val.type is defer.TimeoutError
  45.226 +True
  45.227 +>>> val.getErrorMessage()
  45.228 +'Timed out.'
  45.229 +
  45.230 +The raw state and raw value are still uncalled, though.  
  45.231 +
  45.232 +>>> appDeferred.getRawState() # None
  45.233 +>>> appDeferred.getRawValue()
  45.234 +Traceback (most recent call last):
  45.235 +...
  45.236 +RuntimeError: Deferred still pending
  45.237 +
  45.238 +If the deferred is ever called with a callback or an errback then the state
  45.239 +changes then to reflect the actual result.
  45.240 +
  45.241 +Let's move the creation date back again.
  45.242 +
  45.243 +>>> appDeferred.creation_date = original_creation_date
  45.244 +
  45.245 +Another important aspect of the deferred to notice is the owner.  The owner is 
  45.246 +the user who made the call.  This is important because callbacks and errbacks
  45.247 +are run in the security context of the owner.  The API here is the standard 
  45.248 +Owned API in Zope 2's lib/python/AccessControl/Owned.py.
  45.249 +
  45.250 +>>> appDeferred.getOwnerTuple()
  45.251 +(['acl_users'], 'addie')
  45.252 +
  45.253 +The only remaining part of the deferred API we need to explore from a user's
  45.254 +perspective is setting the callbacks and errbacks themselves.  We will postpone
  45.255 +that for a deferred created by arthur, the user who only has Authenticated
  45.256 +privileges, so we can test some of the security checks.  For now, let's log
  45.257 +out as addie the administrator and look at the acm as if we were the zasync 
  45.258 +client, getting some new deferred calls.
  45.259 +
  45.260 +>>> addie_request = app.REQUEST # remember for later
  45.261 +>>> zopetestutils.closeRequest()
  45.262 +>>> app = zopetestutils.startRequest(zopetestutils.ADMIN)
  45.263 +>>> acm = app._getOb(acm_id)
  45.264 +
  45.265 +Right now the acm has one deferred call that has never been accepted by an
  45.266 +asynchronous worker, and no pending or resolved calls.  We'll pretend to be 
  45.267 +an asynchronous worker.  Every poll by the asynchronous worker should get all
  45.268 +of the new calls and then call the acm's heartbeat.
  45.269 +
  45.270 +>>> new = acm.acceptAll()
  45.271 +>>> acm.heartbeat()
  45.272 +
  45.273 +The heartbeat currently only performs cache rotation for resolved deferreds, 
  45.274 +and that only after certain intervals, so it is effectively a no-op in this
  45.275 +case.  We'll explore this behavior at the end of this document.
  45.276 +
  45.277 +acceptAll returned all of the new deferreds and registered all of them as
  45.278 +accepted.
  45.279 +
  45.280 +>>> len(new)
  45.281 +1
  45.282 +>>> new[0] is aq_base(appDeferred)
  45.283 +True
  45.284 +>>> accepted = acm.getAcceptedCalls()
  45.285 +>>> list(accepted) == list(new) # what was new is now accepted
  45.286 +True
  45.287 +>>> len(acm.acceptAll()) # they have all been accepted now
  45.288 +0
  45.289 +
  45.290 +Notice that, unlike getDeferred and getSessionDeferred, acceptAll returns
  45.291 +deferreds without contextual wrapping.
  45.292 +
  45.293 +Now at this point zasync would commit the transaction and then go off and
  45.294 +schedule the deferred call to be performed.  We'll move on as well, for now,
  45.295 +and log in as arthur, the user with only authenticated privileges.  arthur
  45.296 +will create a session call, and then attach a callback for which he does not
  45.297 +have sufficient privileges.
  45.298 +
  45.299 +>>> zopetestutils.closeRequest()
  45.300 +>>> app = zopetestutils.startRequest(('/acl_users', 'arthur'))
  45.301 +>>> acm = app._getOb(acm_id)
  45.302 +>>> sessDeferred = acm.putSessionCall('zope_exec', 'silly', demo=42)
  45.303 +>>> aq_base(sessDeferred) is aq_base(
  45.304 +...     acm.getSessionDeferred(sessDeferred.local_key))
  45.305 +True
  45.306 +>>> aq_base(sessDeferred) is aq_base(
  45.307 +...     acm.getSessionDeferred(sessDeferred.key))
  45.308 +True
  45.309 +>>> aq_base(sessDeferred) is aq_base(
  45.310 +...     acm.getDeferred(sessDeferred.key))
  45.311 +True
  45.312 +
  45.313 +This callback should raise an Unauthorized error:
  45.314 +
  45.315 +>>> d = sessDeferred.addCallback("python: tool.putCall('zope_exec', result)")
  45.316 +
  45.317 +This errback should succeed; since it does not return a failure or raise an
  45.318 +exception, control should proceed to the next callback.  Notice we are saving
  45.319 +the failure message (should be about an Unauthorized) into 'failure_msg'.
  45.320 +
  45.321 +>>> d = sessDeferred.addErrback(('failure_msg', "failure/getErrorMessage"))
  45.322 +
  45.323 +>>> d = sessDeferred.addCallbacks("result/baz", "failure/ignored")
  45.324 +
  45.325 +Finally, we'll add a callback that should transparently pass the failure 
  45.326 +through, and an errback that looks at the new failure and compares it to the
  45.327 +class of the previous failure.
  45.328 +
  45.329 +>>> d = sessDeferred.addCallback("root/ignored")
  45.330 +>>> d = sessDeferred.addErrback(
  45.331 +...     "python: results['failure_msg']==failure.getErrorMessage()")
  45.332 +
  45.333 +That should leave a TypeError in the failure and False in the result.
  45.334 +These examples are contrived; a typical pattern may be to have the TALES
  45.335 +expression call a Python script somewhere.
  45.336 +
  45.337 +Let's switch back to the zasync client and get the new deferred.
  45.338 +
  45.339 +>>> arthur_request = app.REQUEST # remember for later
  45.340 +>>> zopetestutils.closeRequest()
  45.341 +>>> app = zopetestutils.startRequest(zopetestutils.ADMIN)
  45.342 +>>> acm = app._getOb(acm_id)
  45.343 +>>> new = acm.acceptAll()
  45.344 +>>> acm.heartbeat()
  45.345 +
  45.346 +Now new should contain the single new deferred.
  45.347 +
  45.348 +>>> len(new)
  45.349 +1
  45.350 +>>> new[0] is aq_base(sessDeferred)
  45.351 +True
  45.352 +
  45.353 +Both deferreds should be in the accepted bin.
  45.354 +
  45.355 +>>> len(acm.getAcceptedCalls())
  45.356 +2
  45.357 +
  45.358 +Let's return a failure for the first appDeferred, while we're at it.
  45.359 +
  45.360 +>>> res = failure.Failure(defer.TimeoutError('Timed out.'))
  45.361 +>>> out = acm.getDeferred(appDeferred.key).errback(res)
  45.362 +>>> res is out
  45.363 +True
  45.364 +
  45.365 +Note that you may only call errback or callback once.  This actually tries
  45.366 +to send a message to the zasync logger, because it should only be called from
  45.367 +within the zasync client.  So first we'll set up a handler for the zasync log.
  45.368 +
  45.369 +>>> import logging, StringIO
  45.370 +>>> logger = logging.getLogger('zasync')
  45.371 +>>> log_out = StringIO.StringIO()
  45.372 +>>> hdlr = logging.StreamHandler(log_out)
  45.373 +>>> formatter = logging.Formatter('%(levelname)s %(message)s')
  45.374 +>>> hdlr.setFormatter(formatter)
  45.375 +>>> logger.addHandler(hdlr) 
  45.376 +>>> logger.setLevel(logging.DEBUG)
  45.377 +
  45.378 +Now we'll actually test the re-calls. First a callback...
  45.379 +
  45.380 +>>> out = acm.getDeferred(appDeferred.key).callback(42)
  45.381 +>>> isinstance(out, failure.Failure)
  45.382 +True
  45.383 +>>> out.getErrorMessage()
  45.384 +'Deferred has already been called'
  45.385 +>>> log_out.getvalue() == (
  45.386 +...     'ERROR Zope deferred %r has already been called\n' % (appDeferred.key,))
  45.387 +True
  45.388 +
  45.389 +...and now an errback.
  45.390 +
  45.391 +>>> log_out.seek(0) # clear the error log
  45.392 +>>> log_out.truncate()
  45.393 +>>> out = acm.getDeferred(appDeferred.key).errback(res)
  45.394 +>>> isinstance(out, failure.Failure)
  45.395 +True
  45.396 +>>> log_out.getvalue() == (
  45.397 +...     'ERROR Zope deferred %r has already been called\n' % (appDeferred.key,))
  45.398 +True
  45.399 +
  45.400 +Now that a deferred has been resolved, there should only be one accepted call
  45.401 +remaining, and the next cache rotation should be set.
  45.402 +
  45.403 +>>> len(acm.getAcceptedCalls())
  45.404 +1
  45.405 +>>> acm.getNextCacheRotation() > datetime.datetime.now()
  45.406 +True
  45.407 +
  45.408 +There's still two deferreds in the acm though:
  45.409 +
  45.410 +>>> len(acm)
  45.411 +2
  45.412 +
  45.413 +So addie has a deferred waiting for her with a failure result, and, in our
  45.414 +imaginary scenario, the zasync worker is still calculating the result to
  45.415 +arthur's call.  What do we have left to do?  Let's switch back to addie and get
  45.416 +the result; then switch back to the zasync worker to send a result to arthur,
  45.417 +and force a cache rotation; let arthur get his result; and have the zasync
  45.418 +worker force a final cache rotation.  Whew!
  45.419 +
  45.420 +>>> zopetestutils.closeRequest()
  45.421 +>>> app = zopetestutils.startRequest(
  45.422 +...     ('/acl_users', 'addie'), previous=addie_request)
  45.423 +>>> acm = app._getOb(acm_id)
  45.424 +>>> appDeferred = acm.getDeferred(appDeferred.key)
  45.425 +>>> appDeferred.getState()
  45.426 +'failure'
  45.427 +>>> appDeferred.getErrorMessage()
  45.428 +'Timed out.'
  45.429 +>>> appDeferred.result # None
  45.430 +>>> isinstance(appDeferred.failure, failure.Failure)
  45.431 +True
  45.432 +>>> appDeferred.failure is appDeferred.getValue()
  45.433 +True
  45.434 +
  45.435 +addie can't get arthur's session deferred through the session-based API
  45.436 +
  45.437 +>>> acm.getSessionDeferred(sessDeferred.key) # None
  45.438 +>>> acm.getSessionDeferred(sessDeferred.local_key) # None
  45.439 +
  45.440 +But she can get it from the core API.  We aren't checking permissions here,
  45.441 +but by default users with the Manager role can use "putCall" and "getDeferred"
  45.442 +and users with the Authenticated role can use "putSessionCall" and 
  45.443 +"getSessionDeferred", so addie the administrator would be able to do this,
  45.444 +but poor arthur wouldn't.
  45.445 +
  45.446 +>>> aq_base(acm.getDeferred(sessDeferred.key)) is aq_base(sessDeferred)
  45.447 +True
  45.448 +
  45.449 +Now back to the zasync user: we'll force a cache rotation and then send a
  45.450 +result to arthur.
  45.451 +
  45.452 +>>> zopetestutils.closeRequest()
  45.453 +>>> app = zopetestutils.startRequest(zopetestutils.ADMIN)
  45.454 +>>> acm = app._getOb(acm_id)
  45.455 +>>> new = acm.acceptAll()
  45.456 +>>> len(new)
  45.457 +0
  45.458 +
  45.459 +We'll force the cache rotation by mucking with the "next cache rotation" time.
  45.460 +We're reaching into a protected attribute for this.
  45.461 +
  45.462 +>>> now = acm._next_rotate = datetime.datetime.now()
  45.463 +>>> acm.heartbeat() # should rotate cache and set next rotate to None
  45.464 +>>> acm.getNextCacheRotation() >= now + datetime.timedelta(
  45.465 +...     seconds=acm.rotation_period)
  45.466 +True
  45.467 +>>> sessDeferred = acm.getDeferred(sessDeferred.key)
  45.468 +>>> sessDeferred.callback(42) # returns result of the callback/errback chain
  45.469 +False
  45.470 +>>> len(acm.getAcceptedCalls())
  45.471 +0
  45.472 +
  45.473 +Now back to arthur...
  45.474 +
  45.475 +>>> zopetestutils.closeRequest()
  45.476 +>>> app = zopetestutils.startRequest(
  45.477 +...     ('/acl_users', 'arthur'), previous=arthur_request)
  45.478 +>>> acm = app._getOb(acm_id)
  45.479 +>>> sessDeferred = acm.getSessionDeferred(sessDeferred.local_key)
  45.480 +>>> sessDeferred.result
  45.481 +False
  45.482 +>>> sessDeferred.original_result
  45.483 +42
  45.484 +>>> sessDeferred.failure.type is TypeError
  45.485 +True
  45.486 +
  45.487 +...and finally back to zasync again to rotate the appDeferred out of the 
  45.488 +cache...
  45.489 +
  45.490 +>>> zopetestutils.closeRequest()
  45.491 +>>> app = zopetestutils.startRequest(zopetestutils.ADMIN)
  45.492 +>>> acm = app._getOb(acm_id)
  45.493 +>>> new = acm.acceptAll()
  45.494 +>>> len(new)
  45.495 +0
  45.496 +>>> now = acm._next_rotate = datetime.datetime.now() # forcing
  45.497 +>>> acm.heartbeat() # should rotate cache and set next rotate to None
  45.498 +>>> acm.getNextCacheRotation() >= now + datetime.timedelta(
  45.499 +...     seconds=acm.rotation_period)
  45.500 +True
  45.501 +>>> aq_base(sessDeferred) is aq_base(acm.getDeferred(sessDeferred.key))
  45.502 +True
  45.503 +>>> acm.getDeferred(appDeferred.key) is None # rotated out
  45.504 +True
  45.505 +
  45.506 +I should test resetNextCacheRotation too but I just can't make myself.  
  45.507 +
  45.508 +Clean up.
  45.509 +
  45.510 +>>> zopetestutils.closeRequest()
  45.511 +>>> zopetestutils.forget()
  45.512 +
  45.513 +The end.
    46.1 new file mode 100644
    46.2 --- /dev/null
    46.3 +++ b/permissions.py
    46.4 @@ -0,0 +1,63 @@
    46.5 +##############################################################################
    46.6 +#
    46.7 +# Copyright (c) 2004 Zope Corporation and Contributors. All Rights Reserved.
    46.8 +#
    46.9 +# This software is subject to the provisions of the Zope Public License,
   46.10 +# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
   46.11 +# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
   46.12 +# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
   46.13 +# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
   46.14 +# FOR A PARTICULAR PURPOSE.
   46.15 +#
   46.16 +##############################################################################
   46.17 +"""permissions
   46.18 +
   46.19 +much of file is copied and modifed from CMFCore/CMFCorePermissions.py
   46.20 +
   46.21 +$Id: permissions.py,v 1.1.1.1 2004/10/10 23:37:07 poster Exp $
   46.22 +"""
   46.23 +
   46.24 +import Products
   46.25 +from AccessControl import Permissions
   46.26 +from AccessControl.Permission import _registeredPermissions
   46.27 +from AccessControl.Permission import pname
   46.28 +from Globals import ApplicationDefaultPermissions
   46.29 +
   46.30 +# General Zope permissions
   46.31 +View = Permissions.view
   46.32 +AccessContentsInformation = Permissions.access_contents_information
   46.33 +DeleteObjects = Permissions.delete_objects
   46.34 +UndoChanges = Permissions.undo_changes
   46.35 +ChangePermissions = Permissions.change_permissions
   46.36 +ViewManagementScreens = Permissions.view_management_screens
   46.37 +ManageProperties = Permissions.manage_properties
   46.38 +FTPAccess = Permissions.ftp_access
   46.39 +
   46.40 +MakeAsynchronousApplicationCalls = "Make asynchronous application calls"
   46.41 +MakeAsynchronousSessionCalls = "Make asynchronous session calls"
   46.42 +ModifyDeferred = "Modify deferred"
   46.43 +
   46.44 +# copied from CMF
   46.45 +def setDefaultRoles(permission, roles):
   46.46 +    '''
   46.47 +    Sets the defaults roles for a permission.
   46.48 +    '''
   46.49 +    # XXX This ought to be in AccessControl.SecurityInfo.
   46.50 +    registered = _registeredPermissions
   46.51 +    if not registered.has_key(permission):
   46.52 +        registered[permission] = 1
   46.53 +        Products.__ac_permissions__=(
   46.54 +            Products.__ac_permissions__+((permission,(),roles),))
   46.55 +        mangled = pname(permission)
   46.56 +        setattr(ApplicationDefaultPermissions, mangled, roles)
   46.57 +
   46.58 +# Note that we can only use the default Zope roles in calls to
   46.59 +# setDefaultRoles().  The default Zope roles are:
   46.60 +# Anonymous, Manager, and Owner.
   46.61 +
   46.62 +setDefaultRoles(MakeAsynchronousSessionCalls, 
   46.63 +                ('Anonymous', 'Manager',))
   46.64 +setDefaultRoles(MakeAsynchronousApplicationCalls, 
   46.65 +                ('Manager',))
   46.66 +setDefaultRoles(ModifyDeferred, 
   46.67 +                ('Owner', 'Manager',))
   46.68 \ No newline at end of file
    47.1 new file mode 100644
    47.2 --- /dev/null
    47.3 +++ b/tests/CVS/Entries
    47.4 @@ -0,0 +1,6 @@
    47.5 +/__init__.py/1.1.1.1/Sun Oct 10 23:37:13 2004//Tzasync-1_1_0
    47.6 +/test_bforests.py/1.1.1.1/Sun Oct 10 23:37:13 2004//Tzasync-1_1_0
    47.7 +/test_bucketqueue.py/1.3/Sat Sep 17 03:48:16 2005//Tzasync-1_1_0
    47.8 +/test_manager.py/1.1.1.1/Sun Oct 10 23:37:13 2004//Tzasync-1_1_0
    47.9 +/zopetestutils.py/1.1.1.1/Sun Oct 10 23:37:13 2004//Tzasync-1_1_0
   47.10 +D
    48.1 new file mode 100644
    48.2 --- /dev/null
    48.3 +++ b/tests/CVS/Repository
    48.4 @@ -0,0 +1,1 @@
    48.5 +Packages/zasync/tests
    49.1 new file mode 100644
    49.2 --- /dev/null
    49.3 +++ b/tests/CVS/Root
    49.4 @@ -0,0 +1,1 @@
    49.5 +:pserver:anonymous@cvs.zope.org:/cvs-repository
    50.1 new file mode 100644
    50.2 --- /dev/null
    50.3 +++ b/tests/CVS/Tag
    50.4 @@ -0,0 +1,1 @@
    50.5 +Nzasync-1_1_0
    51.1 new file mode 100644
    52.1 new file mode 100644
    52.2 --- /dev/null
    52.3 +++ b/tests/test_bforests.py
    52.4 @@ -0,0 +1,47 @@
    52.5 +##############################################################################
    52.6 +#
    52.7 +# Copyright (c) 2004 Zope Corporation and Contributors. All Rights Reserved.
    52.8 +#
    52.9 +# This software is subject to the provisions of the Zope Public License,
   52.10 +# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
   52.11 +# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
   52.12 +# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
   52.13 +# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
   52.14 +# FOR A PARTICULAR PURPOSE.
   52.15 +#
   52.16 +##############################################################################
   52.17 +"""test bforests
   52.18 +
   52.19 +$Id: test_bforests.py,v 1.1.1.1 2004/10/10 23:37:13 poster Exp $
   52.20 +"""
   52.21 +
   52.22 +import unittest
   52.23 +from Products.zasync.bforests import IOBForest, OIBForest, OOBForest, IIBForest
   52.24 +from Products.zasync.tests.zopetestutils import DocFileSuite, \
   52.25 +    StringGenerator, NumberGenerator
   52.26 +
   52.27 +def test_suite():
   52.28 +    suite = unittest.TestSuite()
   52.29 +    numgen = iter(NumberGenerator()).next
   52.30 +    strgen = iter(StringGenerator()).next
   52.31 +    suite.addTest(
   52.32 +        DocFileSuite(
   52.33 +            '../bforests.txt', 
   52.34 +            BForest=IOBForest, KeyGenerator=numgen, ValueGenerator=strgen))
   52.35 +    suite.addTest(
   52.36 +        DocFileSuite(
   52.37 +            '../bforests.txt', 
   52.38 +            BForest=OIBForest, KeyGenerator=strgen, ValueGenerator=numgen))
   52.39 +    suite.addTest(
   52.40 +        DocFileSuite(
   52.41 +            '../bforests.txt', 
   52.42 +            BForest=IIBForest, KeyGenerator=numgen, ValueGenerator=numgen))
   52.43 +    suite.addTest(
   52.44 +        DocFileSuite(
   52.45 +            '../bforests.txt', 
   52.46 +            BForest=OOBForest, KeyGenerator=strgen, ValueGenerator=strgen))
   52.47 +    return suite
   52.48 +
   52.49 +if __name__ == '__main__': 
   52.50 +    import unittest
   52.51 +    unittest.main(defaultTest='test_suite')
    53.1 new file mode 100644
    53.2 --- /dev/null
    53.3 +++ b/tests/test_bucketqueue.py
    53.4 @@ -0,0 +1,296 @@
    53.5 +# Modifications of the standard library Queue tests to test the           
    53.6 +# bucketqueue; because I'm leveraging the Queue tests already in the      
    53.7 +# library, I'm using their (admittedly antiquated) approach.  Because I'm 
    53.8 +# importing test_queue from the original tests, and it runs automatically 
    53.9 +# on import, running the tests generates the following output (ideally):  
   53.10 +#
   53.11 +#   Simple Queue tests seemed to work
   53.12 +#   Failing Queue tests seemed to work
   53.13 +#   Simple BucketQueue tests seemed to work
   53.14 +#   Failing BucketQueue tests seemed to work
   53.15 +#
   53.16 +# The first two are simply running tests on the original Python Queue.  The 
   53.17 +# last two indicate success for the BucketQueue.
   53.18 +
   53.19 +import threading, time
   53.20 +from test.test_queue import FailingQueueException
   53.21 +from test.test_queue import FailingQueueTest as OriginalFailingQueueTest
   53.22 +from test.test_queue import SimpleQueueTest as OriginalSimpleQueueTest
   53.23 +from test.test_support import verify, TestFailed, verbose
   53.24 +from Products.zasync import bucketqueue
   53.25 +
   53.26 +queue_size = 5
   53.27 +
   53.28 +# Execute a function that blocks, and in a seperate thread, a function that
   53.29 +# triggers the release.  Returns the result of the blocking function.
   53.30 +class _TriggerThread(threading.Thread):
   53.31 +    def __init__(self, fn, args, prefn=None, preargs=()):
   53.32 +        self.fn = fn
   53.33 +        self.args = args
   53.34 +        self.prefn = prefn
   53.35 +        self.preargs = preargs
   53.36 +        self.preEvent = threading.Event()
   53.37 +        self.startedEvent = threading.Event()
   53.38 +        threading.Thread.__init__(self)
   53.39 +    def run(self):
   53.40 +        global thread_res
   53.41 +        if self.prefn is not None:
   53.42 +            self.prefn(*self.preargs)
   53.43 +        self.preEvent.set()
   53.44 +        time.sleep(.1)
   53.45 +        self.startedEvent.set()
   53.46 +        self.result = self.fn(*self.args)
   53.47 +
   53.48 +def _doBlockingTest(block_func, block_args, 
   53.49 +                    trigger_func, trigger_args, 
   53.50 +                    pre_func=None, pre_args=()):
   53.51 +    t = _TriggerThread(trigger_func, trigger_args, pre_func, pre_args)
   53.52 +    t.start()
   53.53 +    t.preEvent.wait()
   53.54 +    try:
   53.55 +        return block_func(*block_args)
   53.56 +    finally:
   53.57 +        # If we unblocked before our thread made the call, we failed!
   53.58 +        if not t.startedEvent.isSet():
   53.59 +            raise TestFailed("blocking function '%r' appeared not to block" %
   53.60 +                             block_func)
   53.61 +        t.join(1) # make sure the thread terminates
   53.62 +        if t.isAlive():
   53.63 +            raise TestFailed("trigger function '%r' appeared to not return" %
   53.64 +                             trigger_func)
   53.65 +
   53.66 +# A BucketQueue subclass that can provoke failure at a moment's notice
   53.67 +
   53.68 +class FailingQueue(bucketqueue.BucketQueue):
   53.69 +    def __init__(self, *args):
   53.70 +        self.fail_next_put = False
   53.71 +        self.fail_next_get = False
   53.72 +        bucketqueue.BucketQueue.__init__(self, *args)
   53.73 +    def _put(self, item):
   53.74 +        if self.fail_next_put:
   53.75 +            self.fail_next_put = False
   53.76 +            raise FailingQueueException, "You Lose"
   53.77 +        return bucketqueue.BucketQueue._put(self, item)
   53.78 +    def _removeIx(self, ix):
   53.79 +        if self.fail_next_get:
   53.80 +            self.fail_next_get = False
   53.81 +            raise FailingQueueException, "You Lose"
   53.82 +        return bucketqueue.BucketQueue._removeIx(self, ix)
   53.83 +
   53.84 +def SeparateBucketFailingQueueTest(q):
   53.85 +    # this test has every item in a separate bucket
   53.86 +    for i in range(queue_size):
   53.87 +        q.makeBucket(i, silent=True)
   53.88 +    if not q.empty():
   53.89 +        raise RuntimeError, "Call this function with an empty queue"
   53.90 +    for i in range(queue_size-1):
   53.91 +        q.put(i, bucket=i)
   53.92 +    next_bucket = i + 1 # queue_size - 1
   53.93 +    # Test a failing non-blocking put.
   53.94 +    q.fail_next_put = True
   53.95 +    try:
   53.96 +        q.put("oops", block=0, bucket=next_bucket)
   53.97 +        raise TestFailed("The queue didn't fail when it should have")
   53.98 +    except FailingQueueException:
   53.99 +        pass
  53.100 +    q.fail_next_put = True
  53.101 +    try:
  53.102 +        q.put("oops", timeout=0.1, bucket=next_bucket)
  53.103 +        raise TestFailed("The queue didn't fail when it should have")
  53.104 +    except FailingQueueException:
  53.105 +        pass
  53.106 +    q.put("last", bucket=next_bucket)
  53.107 +    verify(q.full(), "Queue should be full")
  53.108 +    # Test a failing blocking put
  53.109 +    q.fail_next_put = True
  53.110 +    try:
  53.111 +        _doBlockingTest( q.put, ("full", True, None, next_bucket), q.get, ())
  53.112 +        raise TestFailed("The queue didn't fail when it should have")
  53.113 +    except FailingQueueException:
  53.114 +        pass
  53.115 +    # Check the Queue isn't damaged.
  53.116 +    # put failed, but get succeeded - re-add
  53.117 +    q.put("last", bucket=next_bucket)
  53.118 +    # Test a failing timeout put
  53.119 +    q.fail_next_put = True
  53.120 +    try:
  53.121 +        _doBlockingTest(q.put, ("full", True, 0.2, next_bucket), 
  53.122 +                        q.get, ())
  53.123 +        raise TestFailed("The queue didn't fail when it should have")
  53.124 +    except FailingQueueException:
  53.125 +        pass
  53.126 +    # Check the Queue isn't damaged.
  53.127 +    # put failed, but get succeeded - re-add
  53.128 +    q.put("last", bucket=next_bucket)
  53.129 +    verify(q.full(), "Queue should be full")
  53.130 +    q.get()
  53.131 +    verify(not q.full(), "Queue should not be full")
  53.132 +    q.put("last", bucket=next_bucket)
  53.133 +    verify(q.full(), "Queue should be full")
  53.134 +    # Test a blocking put
  53.135 +    _doBlockingTest( q.put, ("full", True, None, next_bucket), q.get, ())
  53.136 +    # Empty it
  53.137 +    for i in range(queue_size):
  53.138 +        q.get()
  53.139 +    verify(q.empty(), "Queue should be empty")
  53.140 +    q.put(0, "first")
  53.141 +    q.fail_next_get = True
  53.142 +    try:
  53.143 +        q.get()
  53.144 +        raise TestFailed("The queue didn't fail when it should have")
  53.145 +    except FailingQueueException:
  53.146 +        pass
  53.147 +    verify(not q.empty(), "Queue should not be empty")
  53.148 +    q.fail_next_get = True
  53.149 +    try:
  53.150 +        q.get(timeout=0.1)
  53.151 +        raise TestFailed("The queue didn't fail when it should have")
  53.152 +    except FailingQueueException:
  53.153 +        pass
  53.154 +    verify(not q.empty(), "Queue should not be empty")
  53.155 +    q.get()
  53.156 +    verify(q.empty(), "Queue should be empty")
  53.157 +    q.fail_next_get = True
  53.158 +    try:
  53.159 +        _doBlockingTest( q.get, (), q.put, ('empty', True, None, 0))
  53.160 +        raise TestFailed("The queue didn't fail when it should have")
  53.161 +    except FailingQueueException:
  53.162 +        pass
  53.163 +    # put succeeded, but get failed.
  53.164 +    verify(not q.empty(), "Queue should not be empty")
  53.165 +    q.get()
  53.166 +    verify(q.empty(), "Queue should be empty")
  53.167 +
  53.168 +def SeparateBucketSimpleQueueTest(q):
  53.169 +    # this test has every item in a separate bucket
  53.170 +    for i in range(queue_size):
  53.171 +        q.makeBucket(i, silent=True)
  53.172 +    if not q.empty():
  53.173 +        raise RuntimeError, "Call this function with an empty queue"
  53.174 +    # I guess we better check things actually queue correctly a little :)
  53.175 +    q.put(111, bucket=1)
  53.176 +    q.put(222, bucket=2)
  53.177 +    verify(q.get() == 111 and q.get() == 222,
  53.178 +           "Didn't seem to queue the correct data!")
  53.179 +    for i in range(queue_size-1):
  53.180 +        q.put(i, i)
  53.181 +    next_bucket = i + 1
  53.182 +    verify(not q.full(), "Queue should not be full")
  53.183 +    q.put("last", bucket=next_bucket)
  53.184 +    verify(q.full(), "Queue should be full")
  53.185 +    try:
  53.186 +        q.put("full", block=0, bucket=next_bucket)
  53.187 +        raise TestFailed("Didn't appear to block with a full queue")
  53.188 +    except bucketqueue.Full:
  53.189 +        pass
  53.190 +    try:
  53.191 +        q.put("full", timeout=0.1, bucket=next_bucket)
  53.192 +        raise TestFailed("Didn't appear to time-out with a full queue")
  53.193 +    except bucketqueue.Full:
  53.194 +        pass
  53.195 +    # Test a blocking put
  53.196 +    _doBlockingTest( q.put, ("full", True, None, next_bucket), q.get, ())
  53.197 +    _doBlockingTest( q.put, ("full", True, 0.2, next_bucket), q.get, ())
  53.198 +    # Empty it
  53.199 +    for i in range(queue_size):
  53.200 +        q.get()
  53.201 +    verify(q.empty(), "Queue should be empty")
  53.202 +    try:
  53.203 +        q.get(block=0)
  53.204 +        raise TestFailed("Didn't appear to block with an empty queue")
  53.205 +    except bucketqueue.Empty:
  53.206 +        pass
  53.207 +    try:
  53.208 +        q.get(timeout=0.1)
  53.209 +        raise TestFailed("Didn't appear to time-out with an empty queue")
  53.210 +    except bucketqueue.Empty:
  53.211 +        pass
  53.212 +    # Test a blocking get
  53.213 +    _doBlockingTest(q.get, (), q.put, ('empty', True, None, 0))
  53.214 +    _doBlockingTest(q.get, (True, 0.2), q.put, ('empty', True, None, 0))
  53.215 +
  53.216 +def BucketLockSimpleQueueTest(q):
  53.217 +    q.makeBucket(1, silent=True)
  53.218 +    q.makeBucket(2, silent=True)
  53.219 +    q.makeBucket(3, silent=True)
  53.220 +    if not q.empty():
  53.221 +        raise RuntimeError, "Call this function with an empty queue"
  53.222 +    q.put(111, bucket=1)
  53.223 +    verify(q.primed(), "Primed should be True")
  53.224 +    verify(q.available(), "Available should be True")
  53.225 +    q.put(112, bucket=1)
  53.226 +    q.put(113, bucket=1)
  53.227 +    verify(q.get()==111 and q.get()==112 and q.get()==113, 
  53.228 +           "Didn't get the correct queue data")
  53.229 +           # since it is all in the same thread, no blocks
  53.230 +    # now this thread has the bucket lock to 1.
  53.231 +    q.releaseBucket()
  53.232 +    q.put(111, bucket=1)
  53.233 +    q.put(112, bucket=1)
  53.234 +    q.put(113, bucket=1)
  53.235 +    verify(
  53.236 +        _doBlockingTest(
  53.237 +            q.get, (),
  53.238 +            q.put, (221, True, None, 2),
  53.239 +            q.get, ()) == 221, # pre gets 111
  53.240 +        "Didn't get the correct queue data")
  53.241 +    # now this thread holds the bucket lock to 2, and the dead thread holds
  53.242 +    # the bucket lock to 1
  53.243 +    verify(not q.primed(), "Primed should be False")
  53.244 +    verify(not q.empty(), "Empty should be False")
  53.245 +    verify(not q.available(), "Available should be False")
  53.246 +    try:
  53.247 +        q.get(block=0) # this releases the bucket lock on 2, but there's none
  53.248 +        # there
  53.249 +        raise TestFailed("Didn't appear to block with an empty queue")
  53.250 +    except bucketqueue.Empty:
  53.251 +        pass # XXX Queue.Empty isn't quite right anymore :-(
  53.252 +    verify(not q.primed(), "Primed should be False")
  53.253 +    verify(not q.available(), "Available should be False")
  53.254 +    q.put(222, bucket=2)
  53.255 +    verify(q.primed(), "Primed should be True")
  53.256 +    verify(q.available(), "Available should be True")
  53.257 +    verify(q.get()==222, "Didn't get the correct queue data")
  53.258 +    q.put(223, bucket=2)
  53.259 +    verify(not q.primed(), "Primed should be False") # (again) but since this
  53.260 +    # thread holds the lock on bucket 2, we can still get one.
  53.261 +    verify(q.available(), "Available should be True")
  53.262 +    verify(q.get()==223, "Didn't get the correct queue data")
  53.263 +    verify(q.releaseBucket(1)==1,
  53.264 +           "Release of bucket didn't work correctly")
  53.265 +    verify(q.releaseBucket()==2,
  53.266 +           "Release of bucket didn't work correctly")
  53.267 +    verify(
  53.268 +        _doBlockingTest(
  53.269 +            q.get, (),
  53.270 +            q.releaseBucket, (),
  53.271 +            q.get, ()) == 113, # pre gets 112
  53.272 +        "Didn't get the correct queue data")
  53.273 +    verify(q.empty(), "Queue should be empty")
  53.274 +    # thread holds the lock on 1
  53.275 +    verify(q.releaseBucket()==1,
  53.276 +           "Release of bucket didn't work correctly")
  53.277 +    verify(q.empty(), "Queue should be empty")
  53.278 +
  53.279 +def test():
  53.280 +    q=bucketqueue.BucketQueue(queue_size)
  53.281 +    # Do it a couple of times on the same queue
  53.282 +    OriginalSimpleQueueTest(q)
  53.283 +    OriginalSimpleQueueTest(q)
  53.284 +    SeparateBucketSimpleQueueTest(q)
  53.285 +    SeparateBucketSimpleQueueTest(q)
  53.286 +    q = bucketqueue.BucketQueue()
  53.287 +    BucketLockSimpleQueueTest(q)
  53.288 +    BucketLockSimpleQueueTest(q)
  53.289 +    if verbose:
  53.290 +        print "Simple BucketQueue tests seemed to work"
  53.291 +    q = FailingQueue(queue_size)
  53.292 +    OriginalFailingQueueTest(q)
  53.293 +    OriginalFailingQueueTest(q)
  53.294 +    SeparateBucketFailingQueueTest(q)
  53.295 +    SeparateBucketFailingQueueTest(q)
  53.296 +    if verbose:
  53.297 +        print "Failing BucketQueue tests seemed to work"
  53.298 +
  53.299 +if __name__ == "__main__":
  53.300 +    test()
    54.1 new file mode 100644
    54.2 --- /dev/null
    54.3 +++ b/tests/test_manager.py
    54.4 @@ -0,0 +1,28 @@
    54.5 +##############################################################################
    54.6 +#
    54.7 +# Copyright (c) 2004 Zope Corporation and Contributors. All Rights Reserved.
    54.8 +#
    54.9 +# This software is subject to the provisions of the Zope Public License,
   54.10 +# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
   54.11 +# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
   54.12 +# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
   54.13 +# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
   54.14 +# FOR A PARTICULAR PURPOSE.
   54.15 +#
   54.16 +##############################################################################
   54.17 +"""test asynchronous call manager and deferred
   54.18 +
   54.19 +$Id: test_manager.py,v 1.1.1.1 2004/10/10 23:37:13 poster Exp $
   54.20 +"""
   54.21 +
   54.22 +import unittest
   54.23 +from Products.zasync.tests.zopetestutils import DocFileSuite
   54.24 +
   54.25 +def test_suite():
   54.26 +    suite = unittest.TestSuite()
   54.27 +    suite.addTest(DocFileSuite('../manager.txt'))
   54.28 +    return suite
   54.29 +
   54.30 +if __name__ == '__main__': 
   54.31 +    import unittest
   54.32 +    unittest.main(defaultTest='test_suite')
    55.1 new file mode 100644
    55.2 --- /dev/null
    55.3 +++ b/tests/zopetestutils.py
    55.4 @@ -0,0 +1,224 @@
    55.5 +##############################################################################
    55.6 +#
    55.7 +# Copyright (c) 2004 Zope Corporation and Contributors. All Rights Reserved.
    55.8 +#
    55.9 +# This software is subject to the provisions of the Zope Public License,
   55.10 +# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
   55.11 +# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
   55.12 +# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
   55.13 +# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
   55.14 +# FOR A PARTICULAR PURPOSE.
   55.15 +#
   55.16 +##############################################################################
   55.17 +"""zope test utilities
   55.18 +
   55.19 +$Id: zopetestutils.py,v 1.1.1.1 2004/10/10 23:37:13 poster Exp $
   55.20 +"""
   55.21 +import os, doctest, new, unittest, StringIO
   55.22 +import ZODB
   55.23 +from ZODB.DB import DB
   55.24 +from ZEO.ClientStorage import ClientStorage
   55.25 +from ZODB.DemoStorage import DemoStorage
   55.26 +from OFS.Application import Application
   55.27 +from OFS.Application import initialize
   55.28 +from Testing.makerequest import makerequest
   55.29 +from AccessControl.User import system as ADMIN, nobody as ANONYMOUS
   55.30 +from AccessControl import getSecurityManager
   55.31 +from AccessControl.SecurityManagement import newSecurityManager
   55.32 +from AccessControl.SecurityManagement import setSecurityManager
   55.33 +from Acquisition import aq_parent, aq_inner
   55.34 +
   55.35 +manager = None
   55.36 +connection = None
   55.37 +
   55.38 +class CheckpointManager:
   55.39 +
   55.40 +    # XXX TODO: remember past pops so we can skip over unnecessary undos.
   55.41 +
   55.42 +    def __init__(self, connection=None, domain='localhost', port=8100):
   55.43 +        # a checkpoint manager can be instantiated either with a connection
   55.44 +        # (a database object's _p_jar) or a ZEO domain and port
   55.45 +        if connection is None:
   55.46 +            self.db = DB(ClientStorage((domain, port)))
   55.47 +            self.connection = self.db.open()
   55.48 +        else:
   55.49 +            self.connection = connection
   55.50 +            self.db = self.connection.db()
   55.51 +        self.root = self.connection.root()
   55.52 +        self.app = self.root['Application']
   55.53 +        self.undo_stack = []
   55.54 +    
   55.55 +    def push(self):
   55.56 +        get_transaction().commit()
   55.57 +        current_td = self.db.undoInfo(0,-1)[0]
   55.58 +        if self.undo_stack and self.undo_stack[-1]==current_td:
   55.59 +            raise RuntimeError("No change since list begin")
   55.60 +        self.undo_stack.append(current_td)
   55.61 +    
   55.62 +    def pop(self):
   55.63 +        get_transaction().commit() # XXX could be cleaned up
   55.64 +        undosteps = []
   55.65 +        checkpoint = self.undo_stack.pop()
   55.66 +        start = 0
   55.67 +        step = -20
   55.68 +        info = self.db.undoInfo
   55.69 +        while checkpoint is not None:
   55.70 +            history = info(start, start+step)
   55.71 +            if not history:
   55.72 +                raise RuntimeError("Checkpoint not found.")
   55.73 +            for i in history:
   55.74 +                if i['id']==checkpoint['id']:
   55.75 +                    checkpoint = None
   55.76 +                    break
   55.77 +                if i['time'] < checkpoint['time']:
   55.78 +                    raise RuntimeError(
   55.79 +                        "Checkpoint not found.")
   55.80 +                undosteps.append(i['id'])
   55.81 +            start += step
   55.82 +        undo = self.db.undo
   55.83 +        for tid in undosteps:
   55.84 +            undo(tid)
   55.85 +        get_transaction().commit()
   55.86 +        self.connection.sync() # XXX needed?
   55.87 +    
   55.88 +    def close(self):
   55.89 +        self.connection.close()
   55.90 +
   55.91 +def forget():
   55.92 +    global manager
   55.93 +    manager.pop()
   55.94 +
   55.95 +def remember():
   55.96 +    global manager
   55.97 +    manager.push()
   55.98 +
   55.99 +def retry():
  55.100 +    global manager
  55.101 +    manager.pop()
  55.102 +    manager.push()
  55.103 +
  55.104 +def app():
  55.105 +    global connection, manager
  55.106 +    if connection is None:
  55.107 +        db = ZODB.DB(DemoStorage(quota=(1<<20)))
  55.108 +        connection = db.open()
  55.109 +        application = Application()
  55.110 +        connection.root()["Application"] = application
  55.111 +        get_transaction().commit(1)
  55.112 +        responseOut = StringIO.StringIO()
  55.113 +        app = makerequest(application, stdout=responseOut)        
  55.114 +        initialize(app)
  55.115 +        manager = CheckpointManager(connection)
  55.116 +        remember()
  55.117 +    return connection.root()["Application"]
  55.118 +
  55.119 +old_securitymanager = None
  55.120 +
  55.121 +def startRequest(user=ANONYMOUS, previous=None, out=None):
  55.122 +    """pass a user object or a user tuple of (path to user database, user id),
  55.123 +    or defaults to the anonymous user; pass a previous request object to
  55.124 +    carry over cookie information; pass an open file to which the RESPONSE
  55.125 +    should write or a new StringIO will be created (accessible as 
  55.126 +    REQUEST.RESPONSE.stdout)"""
  55.127 +    global old_securitymanager
  55.128 +    if out is None:
  55.129 +        out = StringIO.StringIO()
  55.130 +    application = makerequest(app(), stdout=out)
  55.131 +    if isinstance(user, (tuple, list)):
  55.132 +        user_db, user_id = user
  55.133 +        user_db = application.unrestrictedTraverse(user_db, None)
  55.134 +        if user_db is None:
  55.135 +            user = ANONYMOUS
  55.136 +        else:
  55.137 +            user = user_db.getUserById(user_id, ANONYMOUS)
  55.138 +        user = user.__of__(user_db)
  55.139 +    if aq_parent(user) is None:
  55.140 +        user = user.__of__(application)
  55.141 +    if old_securitymanager is None:
  55.142 +        old_securitymanager = getSecurityManager()
  55.143 +    newSecurityManager(application.REQUEST, user)
  55.144 +    if previous is not None:
  55.145 +        # try to keep the cookies around
  55.146 +        cookies = previous.cookies
  55.147 +        for name, info in previous.RESPONSE.cookies.items():
  55.148 +            if info.get('expires') == "Sun, 10-May-1971 11:59:00 GMT":
  55.149 +                # XXX should really be more careful, checking to see if the
  55.150 +                # date is prior to now
  55.151 +                cookies.pop(name, None)
  55.152 +            else:
  55.153 +                cookies[name] = info.get('value')
  55.154 +        application.REQUEST.cookies.update(cookies)
  55.155 +    return application
  55.156 +
  55.157 +def closeRequest():
  55.158 +    global old_securitymanager, connection
  55.159 +    get_transaction().commit()
  55.160 +    setSecurityManager(old_securitymanager)
  55.161 +    old_securitymanager = None
  55.162 +
  55.163 +def anotherRequest(app, out=None):
  55.164 +    user = getSecurityManager().getUser()
  55.165 +    base_user = aq_parent(user)
  55.166 +    if base_user is ANONYMOUS or base_user is ADMIN:
  55.167 +        user = base_user
  55.168 +    else:
  55.169 +        user = (aq_parent(aq_inner(user)).getPhysicalPath(), user.getId())
  55.170 +    previous = app.REQUEST
  55.171 +    closeRequest()
  55.172 +    return startRequest(user, previous, out)
  55.173 +
  55.174 +# DocFileSuite is slightly modified from original in Zope 3 from March 2004:
  55.175 +
  55.176 +def DocFileSuite(*paths, **globals):
  55.177 +    globals['__name__'] = '__main__'
  55.178 +    t = doctest.Tester(globs=globals)
  55.179 +    suite = unittest.TestSuite()
  55.180 +    dir = os.path.split(__file__)[0]
  55.181 +    for path in paths:
  55.182 +        path = os.path.join(dir, path)
  55.183 +        source = open(path).read()
  55.184 +        def runit(path=path, source=source):
  55.185 +            doctest._utest(t, path, source, path, 1)
  55.186 +        runit = new.function(runit.func_code, runit.func_globals, path,
  55.187 +                             runit.func_defaults, runit.func_closure)
  55.188 +        f = unittest.FunctionTestCase(runit,
  55.189 +                                      description="doctest from %s" % path)
  55.190 +        suite.addTest(f)
  55.191 +    return suite
  55.192 +
  55.193 +def StringGenerator(src='abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'):
  55.194 +    "infinite-ish unique string generator"
  55.195 +    for el in src:
  55.196 +        yield el
  55.197 +    for pre in StringGenerator(src):
  55.198 +        for el in src:
  55.199 +            yield pre + el
  55.200 +
  55.201 +def NumberGenerator(number=0, interval=1):
  55.202 +    "infinite-ish unique number generator"
  55.203 +    while 1:
  55.204 +        yield number
  55.205 +        number += interval
  55.206 +
  55.207 +def test_stuff():
  55.208 +    application = app()
  55.209 +    global manager
  55.210 +    try:
  55.211 +        manager.push()
  55.212 +        application.manage_addFolder('foo')
  55.213 +        get_transaction().commit()
  55.214 +        application.foo.manage_addFile('file')
  55.215 +        get_transaction().commit()
  55.216 +        assert application.foo.file.getId()=='file'
  55.217 +        manager.pop()
  55.218 +        try:
  55.219 +            application.foo
  55.220 +        except AttributeError:
  55.221 +            return "yay"
  55.222 +        else:
  55.223 +            return "boo"
  55.224 +    finally:
  55.225 +        manager.close()
  55.226 +
  55.227 +if __name__=="__main__":
  55.228 +    print test_stuff()
    56.1 new file mode 100644
    56.2 --- /dev/null
    56.3 +++ b/www/CVS/Entries
    56.4 @@ -0,0 +1,5 @@
    56.5 +/analyzeCalls.zpt/1.2/Sat Sep 17 03:26:07 2005//Tzasync-1_1_0
    56.6 +/constructAsynchronousCallManagerForm.zpt/1.1.1.1/Sun Oct 10 23:37:14 2004//Tzasync-1_1_0
    56.7 +/controlAsynchronousCallManagerForm.zpt/1.1/Thu Oct 14 16:35:03 2004//Tzasync-1_1_0
    56.8 +/daliclock.gif/1.1.1.1/Sun Oct 10 23:37:14 2004/-kb/Tzasync-1_1_0
    56.9 +D
    57.1 new file mode 100644
    57.2 --- /dev/null
    57.3 +++ b/www/CVS/Repository
    57.4 @@ -0,0 +1,1 @@
    57.5 +Packages/zasync/www
    58.1 new file mode 100644
    58.2 --- /dev/null
    58.3 +++ b/www/CVS/Root
    58.4 @@ -0,0 +1,1 @@
    58.5 +:pserver:anonymous@cvs.zope.org:/cvs-repository
    59.1 new file mode 100644
    59.2 --- /dev/null
    59.3 +++ b/www/CVS/Tag
    59.4 @@ -0,0 +1,1 @@
    59.5 +Nzasync-1_1_0
    60.1 new file mode 100644
    60.2 --- /dev/null
    60.3 +++ b/www/analyzeCalls.zpt
    60.4 @@ -0,0 +1,331 @@
    60.5 +<tal:block replace="structure here/manage_page_header">Header</tal:block>
    60.6 +<tal:block replace="structure here/manage_tabs">Tabs</tal:block>
    60.7 +<tal:block define="
    60.8 +    batch request/batch|python: 0;
    60.9 +    next_batch python:batch+20;
   60.10 +    prev_batch python:max(0,batch-20);
   60.11 +    is_less python:batch>0;
   60.12 +    ">
   60.13 +<!-- New calls -->
   60.14 +<tal:block define="
   60.15 +    total here/lenNewCalls;
   60.16 +    showNewCalls request/showNewCalls|nothing;
   60.17 +    ">
   60.18 +  <h2>New calls</h2>
   60.19 +  <p class="form-help">
   60.20 +    The manager currently has <strong tal:content="total">42</strong>
   60.21 +    new calls, not yet accepted by the zasync client.
   60.22 +  </p>
   60.23 + <tal:block condition="showNewCalls">
   60.24 +  <tal:block define="
   60.25 +      all_info python:here.listNewCalls();
   60.26 +      batch_info python:all_info[batch:next_batch];
   60.27 +      is_more python:all_info[next_batch:];
   60.28 +      ">
   60.29 +    <table>
   60.30 +      <tr style="background-color: gray; margin: 2px;">
   60.31 +        <metal:block define-macro='table_header'>
   60.32 +          <th>creation date</th>
   60.33 +          <th>resolution date</th>
   60.34 +          <th>time elapsed</th>
   60.35 +          <th>original resolution state</th>
   60.36 +          <th>final resolution state</th>
   60.37 +          <th>key</th>
   60.38 +          <th>user</th>
   60.39 +          <th>plugin</th>
   60.40 +        </metal:block>
   60.41 +      </tr>
   60.42 +      <tal:block repeat="row batch_info">
   60.43 +        <tr tal:define="odd repeat/row/odd;"
   60.44 +            tal:attributes="style python:odd and 'background-color: yellow'">
   60.45 +          <td tal:content="row/creation_date/isoformat">date</td> 
   60.46 +          <td tal:content="row/resolution_date/isoformat" 
   60.47 +            tal:condition="row/resolution_date">date</td>
   60.48 +          <td tal:condition="not:row/resolution_date">Not Available</td>
   60.49 +          <td tal:condition="not:row/resolution_date">Not Available</td>
   60.50 +          <td tal:condition="not:row/original_state" >Not Available</td> 
   60.51 +          <td tal:condition="not:row/state">Not Available</td> 
   60.52 +          <td tal:content="row/key">key</td> 
   60.53 +          <td tal:content="row/user">user</td> 
   60.54 +          <td tal:content="row/plugin">plugin</td> 
   60.55 +        </tr>
   60.56 +        <tr tal:define="
   60.57 +            odd repeat/row/odd;
   60.58 +            "
   60.59 +            tal:attributes="style python:odd and 'background-color: yellow'">
   60.60 +          <td colspan="8">
   60.61 +            <p><strong>Plugin Arguments</strong></p>
   60.62 +            <p tal:content="row/args">args</p> 
   60.63 +            <p tal:content="row/kwargs">kwargs</p> 
   60.64 +          </td>
   60.65 +        </tr>
   60.66 +      </tal:block>
   60.67 +    </table>
   60.68 +    <form method="post" tal:condition="is_less">
   60.69 +      <input type="submit" name="showNewCalls" value="Newer calls">
   60.70 +      <input type="hidden" tal:attributes="value prev_batch"
   60.71 +        name="batch:int" />
   60.72 +      <input type="hidden" tal:attributes="value batchBucket"
   60.73 +        name="analyzeBucket:int" tal:condition="batchBucket" />
   60.74 +    </form>
   60.75 +    <form method="post" tal:condition="is_more">
   60.76 +      <input type="submit" name="showNewCalls" value="Older calls">
   60.77 +      <input type="hidden" tal:attributes="value next_batch"
   60.78 +        name="batch:int" />
   60.79 +      <input type="hidden" tal:attributes="value batchBucket"
   60.80 +        name="analyzeBucket:int" tal:condition="batchBucket" />
   60.81 +    </form>
   60.82 +  </tal:block>
   60.83 + </tal:block>
   60.84 + <tal:block condition="python: not showNewCalls and total">
   60.85 +  <form method="post">
   60.86 +    <input type="submit" name="showNewCalls"
   60.87 +      value="show all new calls" />
   60.88 +    <input type="hidden" name="batch:int" value="0" />
   60.89 +  </form>
   60.90 + </tal:block>
   60.91 +</tal:block>
   60.92 +<!-- New calls -->
   60.93 +
   60.94 +<!--  Accepted calls -->
   60.95 +<tal:block define="
   60.96 +    total here/lenAcceptedCalls;
   60.97 +    showAccepted request/showAccepted|nothing;
   60.98 +    ">
   60.99 +  <h2>Accepted calls</h2>
  60.100 +  <p class="form-help">
  60.101 +   The manager currently has <strong tal:content="total">42</strong>
  60.102 +   accepted calls, not yet resolved by the zasync client.
  60.103 + </p>
  60.104 + <tal:block condition="showAccepted">
  60.105 +  <tal:block define="
  60.106 +      all_info python:here.listAcceptedCalls();
  60.107 +      batch_info python:all_info[batch:next_batch];
  60.108 +      is_more python:all_info[next_batch:];
  60.109 +      ">
  60.110 +    <table>
  60.111 +      <tr style="background-color: gray; margin: 2px;">
  60.112 +        <metal:block use-macro="template/macros/table_header">
  60.113 +        Table Header</metal:block>
  60.114 +      </tr>
  60.115 +      <tal:block repeat="row batch_info">
  60.116 +        <tr tal:define="odd repeat/row/odd;"
  60.117 +            tal:attributes="style python:odd and 'background-color: yellow'">
  60.118 +          <td tal:content="row/creation_date/isoformat">date</td> 
  60.119 +          <td tal:content="row/resolution_date/isoformat" 
  60.120 +            tal:condition="row/resolution_date">date</td>
  60.121 +          <td tal:condition="not:row/resolution_date">Not Available</td>
  60.122 +          <td tal:condition="not:row/resolution_date">Not Available</td>
  60.123 +          <td tal:condition="not:row/original_state" >Not Available</td> 
  60.124 +          <td tal:condition="not:row/state">Not Available</td> 
  60.125 +          <td tal:content="row/key">key</td> 
  60.126 +          <td tal:content="row/user">user</td> 
  60.127 +          <td tal:content="row/plugin">plugin</td> 
  60.128 +        </tr>
  60.129 +        <tr tal:define="
  60.130 +            odd repeat/row/odd;
  60.131 +            "
  60.132 +            tal:attributes="style python:odd and 'background-color: yellow'">
  60.133 +          <td colspan="8">
  60.134 +            <p><strong>Plugin Arguments</strong></p>
  60.135 +            <p tal:content="row/args">args</p> 
  60.136 +            <p tal:content="row/kwargs">kwargs</p> 
  60.137 +          </td>
  60.138 +        </tr>
  60.139 +      </tal:block>
  60.140 +    </table>
  60.141 +    <form method="post" tal:condition="is_less">
  60.142 +      <input type="submit" name="showAccepted" value="Newer calls">
  60.143 +      <input type="hidden" tal:attributes="value prev_batch"
  60.144 +        name="batch:int" />
  60.145 +      <input type="hidden" tal:attributes="value batchBucket"
  60.146 +        name="analyzeBucket:int" tal:condition="batchBucket" />
  60.147 +    </form>
  60.148 +    <form method="post" tal:condition="is_more">
  60.149 +      <input type="submit" name="showAccepted" value="Older calls">
  60.150 +      <input type="hidden" tal:attributes="value next_batch"
  60.151 +        name="batch:int" />
  60.152 +      <input type="hidden" tal:attributes="value batchBucket"
  60.153 +        name="analyzeBucket:int" tal:condition="batchBucket" />
  60.154 +    </form>
  60.155 +  </tal:block>
  60.156 + </tal:block>
  60.157 + <tal:block condition="python: not showAccepted and total">
  60.158 +  <form method="post">
  60.159 +    <input type="submit" name="showAccepted"
  60.160 +      value="show all accepted calls" />
  60.161 +    <input type="hidden" name="batch:int" value="0" />
  60.162 +  </form>
  60.163 + </tal:block>
  60.164 +</tal:block>
  60.165 +<!-- Accepted calls -->
  60.166 +
  60.167 +<!-- Resolved calls -->
  60.168 +<tal:block define="
  60.169 +    total here/lenResolvedCalls;
  60.170 +    rotation_period here/rotation_period;
  60.171 +    buckets here/lenResolvedBuckets;
  60.172 +    min_stay python: rotation_period * (buckets-1);
  60.173 +    max_stay python: rotation_period * buckets;
  60.174 +    next_rotate here/nextResolvedBucketRotation;
  60.175 +    analyze request/analyzeResolved|nothing;
  60.176 +    batchBucket request/analyzeBucket|nothing;
  60.177 +    ">
  60.178 +<h2>Resolved calls</h2>
  60.179 +<p class="form-help">
  60.180 +  The manager currently has <strong tal:content="total">42</strong>
  60.181 +  resolved calls.  These will be kept in the database for no less than 
  60.182 +  <tal:block define="raw_seconds min_stay; days python:raw_seconds//86400;">
  60.183 +    <metal:block define-macro="pretty_duration"><tal:block define="
  60.184 +      hours python:raw_seconds%86400//3600;
  60.185 +      minutes python:raw_seconds%3600//60;
  60.186 +      seconds python:raw_seconds%60;
  60.187 +      one_day python:days==1;
  60.188 +      one_hour python:hours==1;
  60.189 +      one_minute python:minutes==1;
  60.190 +      one_second python:seconds==1;">
  60.191 +      <tal:block condition="python: days">
  60.192 +        <span tal:replace="days"/>
  60.193 +        day<tal:block condition="not:one_day">s</tal:block>,
  60.194 +      </tal:block>
  60.195 +      <tal:block condition="python: days or hours">
  60.196 +        <span tal:replace="hours"/>
  60.197 +        hour<tal:block condition="not:one_hour">s</tal:block>,
  60.198 +      </tal:block>
  60.199 +      <tal:block condition="python: minutes or hours or days">
  60.200 +        <span tal:replace="minutes"/>
  60.201 +        minute<tal:block condition="not:one_minute">s</tal:block>,
  60.202 +      </tal:block>
  60.203 +      <span tal:replace="seconds"/>
  60.204 +        second<tal:block 
  60.205 +        condition="not:one_second">s</tal:block></tal:block></metal:block>;
  60.206 +  </tal:block>
  60.207 +  and, typically, no more than 
  60.208 +  <tal:block define="raw_seconds max_stay; days python:raw_seconds//86400;">
  60.209 +    <metal:block use-macro="template/macros/pretty_duration" />.
  60.210 +  </tal:block>
  60.211 +</p>
  60.212 +<p class="form-help">The resolved calls are an aggregate of 
  60.213 +  <span tal:replace="buckets">2</span> rotating buckets, ordered from newest to
  60.214 +  oldest. <span tal:repeat="bucketIndex python: range(buckets)">Bucket <span
  60.215 +  tal:replace="bucketIndex">0</span> has <span tal:replace="python:
  60.216 +  here.lenResolvedCalls(bucketIndex)">0</span> calls. </span></p>
  60.217 +<p class="form-help">The buckets rotate every 
  60.218 +  <span tal:replace="rotation_period">86400</span> seconds 
  60.219 +  (<span tal:replace="python: rotation_period/3600"/> hours), with the oldest
  60.220 +  bucket discarding its contents.  Change the rotation period on the properties
  60.221 +  tab.<span tal:condition="next_rotate">The buckets will rotate next at
  60.222 +  approximately <span
  60.223 +  tal:replace="next_rotate/isoformat">isoformatDateGoesHere</span>.</span>
  60.224 +  <span class="form-help" tal:condition="not: next_rotate">The buckets are not
  60.225 +  currently scheduled to rotate.</span></p>
  60.226 +<tal:block condition="analyze">
  60.227 +  <tal:block define="
  60.228 +      all_info python:here.listResolvedCalls(batchBucket);
  60.229 +      batch_info python:all_info[batch:next_batch];
  60.230 +      is_more python:all_info[next_batch:];
  60.231 +      ">
  60.232 +    <table>
  60.233 +      <tr style="background-color: gray; margin: 2px;">
  60.234 +        <metal:block use-macro="template/macros/table_header">
  60.235 +        Table Header</metal:block>
  60.236 +      </tr>
  60.237 +      <tal:block repeat="row batch_info">
  60.238 +        <tr tal:define="odd repeat/row/odd;"
  60.239 +            tal:attributes="style python:odd and 'background-color: yellow'">
  60.240 +          <td tal:content="row/creation_date/isoformat">date</td> 
  60.241 +          <td tal:content="row/resolution_date/isoformat" 
  60.242 +            tal:condition="row/resolution_date">date</td>
  60.243 +          <td tal:condition="not:row/resolution_date">Not Available</td>
  60.244 +          <td tal:condition="row/resolution_date">
  60.245 +            <tal:block define="
  60.246 +              diff python:row['resolution_date']-row['creation_date'];
  60.247 +              raw_seconds diff/seconds;
  60.248 +              days diff/days;">
  60.249 +              <metal:block use-macro="template/macros/pretty_duration" />.
  60.250 +            </tal:block>
  60.251 +          </td>
  60.252 +          <td tal:condition="not:row/resolution_date">Not Available</td>
  60.253 +          <td tal:content="row/original_state">success/failure</td> 
  60.254 +          <td tal:content="row/state">success/failure</td> 
  60.255 +          <td tal:content="row/key">key</td> 
  60.256 +          <td tal:content="row/user">user</td> 
  60.257 +          <td tal:content="row/plugin">plugin</td> 
  60.258 +        </tr>
  60.259 +        <tr tal:define="
  60.260 +            odd repeat/row/odd;
  60.261 +            "
  60.262 +            tal:attributes="style python:odd and 'background-color: yellow'">
  60.263 +          <td colspan="8">
  60.264 +            <p><strong>Plugin Arguments</strong></p>
  60.265 +            <p tal:content="row/args">args</p> 
  60.266 +            <p tal:content="row/kwargs">kwargs</p> 
  60.267 +          </td>
  60.268 +        </tr>
  60.269 +        <tr tal:define="
  60.270 +            success python: row['original_state']=='success';
  60.271 +            odd repeat/row/odd;"
  60.272 +            tal:attributes="style python:odd and 'background-color: yellow'">
  60.273 +          <td colspan="8">
  60.274 +            <p><strong>Original Resolution Value</strong></p>
  60.275 +            <pre tal:replace="row/original_value" tal:condition="success">value
  60.276 +            </pre> 
  60.277 +            <p tal:condition="python: success and row['original_value'] is None">
  60.278 +              (None)</p>
  60.279 +            <pre tal:content="row/original_value/getTraceback" 
  60.280 +              tal:condition="not:success">traceback</pre>
  60.281 +          </td>
  60.282 +        </tr>
  60.283 +        <tr tal:define="
  60.284 +            success python: row['state']=='success';
  60.285 +            same python:row['original_value'] is row['value'];
  60.286 +            odd repeat/row/odd;"
  60.287 +            tal:attributes="style python:odd and 'background-color: yellow'">
  60.288 +          <td colspan="8">
  60.289 +            <p><strong>Final Resolution Value (after callback chain)</strong></p>
  60.290 +            <p tal:condition="same">(Same as original resolution)</p>
  60.291 +            <tal:block condition="not:same">
  60.292 +              <pre tal:replace="row/value" tal:condition="success">value</pre> 
  60.293 +              <p tal:condition="python: success and row['value'] is None">
  60.294 +                (None)</p>
  60.295 +              <pre tal:content="row/value/getTraceback" 
  60.296 +                tal:condition="not:success">traceback</pre>
  60.297 +            </tal:block>
  60.298 +          </td>
  60.299 +        </tr>
  60.300 +      </tal:block>
  60.301 +    </table>
  60.302 +    <form method="post" tal:condition="is_less">
  60.303 +      <input type="submit" name="analyzeResolved" value="Newer calls">
  60.304 +      <input type="hidden" tal:attributes="value prev_batch"
  60.305 +        name="batch:int" />
  60.306 +      <input type="hidden" tal:attributes="value batchBucket"
  60.307 +        name="analyzeBucket:int" tal:condition="batchBucket" />
  60.308 +    </form>
  60.309 +    <form method="post" tal:condition="is_more">
  60.310 +      <input type="submit" name="analyzeResolved" value="Older calls">
  60.311 +      <input type="hidden" tal:attributes="value next_batch"
  60.312 +        name="batch:int" />
  60.313 +      <input type="hidden" tal:attributes="value batchBucket"
  60.314 +        name="analyzeBucket:int" tal:condition="batchBucket" />
  60.315 +    </form>
  60.316 +  </tal:block>
  60.317 +</tal:block>
  60.318 +<tal:block condition="python: not analyze and total">
  60.319 +  <form method="post">
  60.320 +    <input type="submit" name="analyzeResolved"
  60.321 +      value="analyze all resolved calls" />
  60.322 +    <input type="hidden" name="batch:int" value="0" />
  60.323 +  </form>
  60.324 +  <form tal:repeat="bucketIndex python: range(buckets)" method="post">
  60.325 +    <input type="submit" name="analyzeResolved" value="analyze bucket calls" 
  60.326 +      tal:attributes="value string:analyze bucket $bucketIndex calls" />
  60.327 +    <input type="hidden" name="batch:int" value="0" />
  60.328 +    <input type="hidden" name="analyzeBucket:int" value="0" 
  60.329 +      tal:attributes="value bucketIndex" />
  60.330 +  </form>
  60.331 +</tal:block>
  60.332 +</tal:block>
  60.333 +<!-- Resolved calls -->
  60.334 +</tal:block>
  60.335 +<tal:block replace="structure here/manage_page_footer">Footer</tal:block>
    61.1 new file mode 100644
    61.2 --- /dev/null
    61.3 +++ b/www/constructAsynchronousCallManagerForm.zpt
    61.4 @@ -0,0 +1,59 @@
    61.5 +<h1 tal:replace="structure here/manage_page_header" />
    61.6 +<h2 tal:replace="structure python: here.manage_form_title(
    61.7 +  form_title='Add Asynchronous Call Manager')" />
    61.8 +
    61.9 +<p class="form-help">
   61.10 +Create an asynchronous call manager.  Requires zasync to poll the manager across
   61.11 +ZEO to do anything useful.  Typically you only need one of these for your whole 
   61.12 +Zope, installed in the root of your site, and the id listed below is the default 
   61.13 +for which zasync looks.
   61.14 +</p>
   61.15 +
   61.16 +<form action="constructAsynchronousCallManager" method="POST">
   61.17 +<table cellspacing="0" cellpadding="2" border="0">
   61.18 +  <tr>
   61.19 +    <td align="left" valign="top">
   61.20 +    <div class="form-label">
   61.21 +      Id
   61.22 +    </div>
   61.23 +    </td>
   61.24 +    <td align="left" valign="top" class="form-label">
   61.25 +    <input type="text" name="id" size="40" value="asynchronous_call_manager" />
   61.26 +    <p tal:condition="here/asynchronous_call_manager|nothing" style="color:red">
   61.27 +    You appear to already have an asynchronous_call_manager installed.</p>
   61.28 +    </td>
   61.29 +  </tr>
   61.30 +  <tr>
   61.31 +    <td align="left" valign="top">
   61.32 +    <div class="form-label">
   61.33 +      Seconds between zasync polling (anything below 2 seconds will normalize to 2 seconds)
   61.34 +    </div>
   61.35 +    </td>
   61.36 +    <td align="left" valign="top" class="form-label">
   61.37 +    <input type="text" name="poll_interval:int" size="4" value="5" />
   61.38 +    </td>
   61.39 +  </tr>
   61.40 +  <tr>
   61.41 +    <td align="left" valign="top">
   61.42 +    <div class="form-label">
   61.43 +      Period of cache rotation for resolved deferreds in seconds (defaults to one day)
   61.44 +    </div>
   61.45 +    </td>
   61.46 +    <td align="left" valign="top" class="form-label">
   61.47 +    <input type="text" name="rotation_period:int" size="10" value="86400" />
   61.48 +    </td>
   61.49 +  </tr>
   61.50 +  <tr>
   61.51 +    <td align="left" valign="top">
   61.52 +    </td>
   61.53 +    <td align="left" valign="top">
   61.54 +    <div class="form-element">
   61.55 +    <input class="form-element" type="submit" name="submit" 
   61.56 +     value=" Create " /> 
   61.57 +    </div>
   61.58 +    </td>
   61.59 +  </tr>
   61.60 +</table>
   61.61 +</form>
   61.62 +
   61.63 +<h1 tal:replace="structure here/manage_page_footer" />
    62.1 new file mode 100644
    62.2 --- /dev/null
    62.3 +++ b/www/controlAsynchronousCallManagerForm.zpt
    62.4 @@ -0,0 +1,55 @@
    62.5 +<tal:block replace="structure here/manage_page_header">Header</tal:block>
    62.6 +<tal:block replace="structure here/manage_tabs">Tabs</tal:block>
    62.7 +
    62.8 +<p class="form-help">The asynchronous call manager is a
    62.9 +tool that enables a client thread, such as a thread handling a Zope request,
   62.10 +to request that a task be performed asynchronously.  The current version of
   62.11 +this product relies on a ZEO client process that 
   62.12 +could be running on another machine to perform the asynchronous tasks.</p>
   62.13 +<h2>Ping Client</h2>
   62.14 +<p class="form-help">The best way to confirm the status of the zasync client
   62.15 +is to look at the client's log (usually on the file system in 
   62.16 +[installation]/var/zope/log/zasync.log, or similar).  However, you can also 
   62.17 +ping the 
   62.18 +zasync client by clicking the button below.  If the zasync client is active,
   62.19 +you should see a response if you refresh the page after the current poll
   62.20 +interval (<span tal:replace="here/poll_interval"/> seconds) plus a few extra 
   62.21 +seconds,
   62.22 +at most (a bit longer if the client is just starting up).</p>
   62.23 +<tal:block tal:define="ping here/getLastPing;pong here/getLastPong">
   62.24 +<form tal:condition="python: pong or not ping"
   62.25 +  action="ping" method="post"><input type="submit" value="PING" /></form>
   62.26 +<ul>
   62.27 +  <li>Last ping: <strong tal:condition="not: ping">never</strong>
   62.28 +  <strong tal:condition="ping" tal:content="ping/isoformat">datetime</strong>
   62.29 +  </li>
   62.30 +  <li>Last pong (response from zasync): 
   62.31 +    <span tal:condition="not: pong"><strong
   62.32 +      tal:attributes="style python: ping and 'color:red'">not yet</strong>
   62.33 +      <span tal:condition="ping">(<a href="" 
   62.34 +      tal:attributes="href string:${here/absolute_url}/manage_overview">
   62.35 +      refresh to check again</a>)</span></span>
   62.36 +    <strong tal:condition="pong" tal:content="pong/isoformat"
   62.37 +      style="color:green">datetime</strong>
   62.38 +  </li>
   62.39 +</ul>
   62.40 +</tal:block>
   62.41 +<h2>Available Plugins</h2>
   62.42 +<tal:block tal:define="plugins here/listPlugins; names plugins/keys">
   62.43 +<p class="form-help">
   62.44 +The helper process provides plugins that determine what calls can be made.
   62.45 +<span tal:condition="names">The following plugins were available at the time
   62.46 +this page was rendered.</span>
   62.47 +</p>
   62.48 +<dl tal:condition="names">
   62.49 +  <tal:block repeat="name names">
   62.50 +    <dt tal:content="name">plugin name</dt>
   62.51 +    <dd><pre tal:content="python: plugins[name]">plugin description</pre></dd>
   62.52 +  </tal:block>
   62.53 +</dl>
   62.54 +<p class="form-help" tal:condition="not: names">
   62.55 +<strong>The helper has not declared any plugins.  This means that the 
   62.56 +asynchronous call manager cannot perform any useful tasks.  It also typically
   62.57 +means that the zasync helper (the ZEO client) is not active.</strong></p>
   62.58 +</tal:block>
   62.59 +<tal:block replace="structure here/manage_page_footer">Footer</tal:block>
    63.1 new file mode 100644
    63.2 index 0000000000000000000000000000000000000000..cbc81883890ac6dd7b6c78639dd5891541e5d9b7
    63.3 GIT binary patch
    63.4 literal 622
    63.5 zc$}Tl+e^~{90l;ROKZ9|CpvXr=B7?p{mRU#vo*D)TV`pqpITb0p-Y$hsihS`pl)8E
    63.6 zh(@7LjU@JvSp;HYAweP*MUeCoK})a;<U=<p66~A5{)EoM;lp`7_Hvu4(4~YDa~%Ff
    63.7 z0Ei-ZWB{2;9*$(mTDL{Zh4E2}G&y3T)kZLe2n`dhG077F%2o?=;^f7#G6ya*d8pON
    63.8 zA}!VRsRpMWid?F&s@hDFjv}$y38M$+ZKAd+Y-`7bN<pXf9P35?ne=P9S4QoYh677u
    63.9 zP6W@YYk~!ntz{4W80}-8`EX}&=b>>+&v5A8b(MFB!k1JhLNs<)H9ZOU6s-m^KcW0M
   63.10 z4!Ws6HA^=pi0)(M0lrL^(F|xtL{AWSNkd@@y{46Sptqm}tS;iytE;pGnwwuH4H1Z>
   63.11 zpT7o)Vu_N8^zT0q(Iz5hIZRV)q*5aLBOaARbwtiLG)vU{f?s-ne5Y?beiHqn|BdcX
   63.12 zN_kgSr`DzA+}<vCleW>@-q^}$UEb54=9aEv=zfa@k?01)-@>1U^|yPveg3}wfqto6
   63.13 z7CGpP2TZlq>F}=$hyx7AYOB~CK2d&70wO%%nz>B2q&v!dq@m>?$JHB8a+~+rQkA)S
   63.14 zRF)POa>#R+>)0)ljd3We+f%?<6|H;KO&UL9iX0icoCdFJ>*2IMWFKd@9~B&{$YNCk
   63.15 wNj3XNIu>+~Hw;S$6t_mK`sWEbER*6BFFp-uAT}n3dWC*P((a2Hu~L}-0zZxv9RL6T
   63.16