Personal tools
You are here: Home Presentations Buildout Potluck Notes

Buildout Potluck Notes

by Veda Williams last modified Feb 17, 2010 01:38 PM

From the Etherpad presentation. Not as good as being there, but still helpful!

Agenda:

version pinning (possibly fulvio, as part of next bullet)    
how to handle production, staging, dev instance (fulvio)    
how to deal with packages under dev - David    
how to get zcml loaded - when it's needed - David    
debugging buildout - Alex    
multiple storages - different mount points
what other things do you put in buildout - reverse proxy, cron  jobs,
log rotations, backups (possibly fulvio)    
setting subversion properties in buildout - Chris H    
running command-line commands with a buildout recipe    

++++++++++++++++++++++++++++++++++++++++++++++

How to handle production, staging, dev instance - (Fulvio)
config folder:
------------------
buildout.cfg.tmpl
base.cfg
devel.cfg
staging.cfg
prod.cfg
versions.cfg
varnish.cfg
zeo.cfg
pound.cfg


buildout.cfg.tmpl
------------------------
[buildout]
extends =
    config/prod.cfg
#    config/staging.cfg
#    config/devel.cfg

## Optional devel.cfg support.  If you're using devel.cfg from above,
## you can uncomment lines below to enable

parts +=

{.....}

NOTES:  Use svn ignore  to not check in buildout.cfg which is made manually) from the buildout.cfg.tmpl

zeo.cfg:
======

[buildout]
extends = base.cfg
parts +=
    zeo
    backup
    backup-daily
    backup-weekly
    logrotate
    logrotate-daily
    supervisor
    supervisor-reboot

{.....}

[backup]
recipe = collective.recipe.backup

[backup-daily]
recipe = z3c.recipe.usercrontab
times = 0 3 * * 0-6
command = ${buildout:directory}/bin/backup

[backup-weekly]
recipe = z3c.recipe.usercrontab
times = 0 3 * * 7
command = ${buildout:directory}/bin/zeopack -p 8000 -d 1 && ${buildout:directory}/bin/backup

[logrotate]
recipe = collective.recipe.template
input = etc/logrotate.conf.tmpl
output = etc/logrotate.conf

[logrotate-daily]
recipe = z3c.recipe.usercrontab
times = 0 6 * * *
command = /usr/sbin/logrotate --state ${buildout:directory}/var/logrotate.status ${buildout:directory}/${logrotate:output}

[supervisor]
recipe = zc.recipe.egg
eggs = supervisor

[supervisor-reboot]
recipe = z3c.recipe.usercrontab
times = @reboot
command = ${buildout:directory}/bin/supervisord -c ${buildout:directory}/etc/supervisord.conf


logrotate.conf.tmpl
==============
rotate 4
weekly
create
compress
delaycompress

${buildout:directory}/var/log/zeo.log {
    postrotate
        ${buildout:bin-directory}/zeo logreopen
    endscript
}

${buildout:directory}/var/log/instance1.log ${buildout:directory}/var/log/instance1-Z2.log {
    sharedscripts
    postrotate
        /bin/kill -USR2 $(cat ${buildout:directory}/var/instance1.pid)
    endscript
}

(USR2) this works only with newer zope .  (not with older zope or zeo or .??) for those, stop and start.




devel.cfg
=======
[buildout]
extends = base.cfg
parts +=
    packages
    zopepy

{.....}

base.cfg
======
[buildout]
extensions +=
    mr.developer
extends =
    versions.cfg

versions = versions

{.....}

versions.cfg
=========
[buildout]
extensions = buildout.dumppickedversions

[versions]
# Buildout infrastructure
buildout.eggtractor = 0.6
plone.recipe.zope2install = 3.2
plone.recipe.zope2instance = 3.4
tl.eggdeps = 0.3.2
setuptools = 0.6c9
zc.buildout = 1.3.1
zc.recipe.egg = 1.2.2
zc.recipe.cmmi = 1.2.1

# Zope
zope2-url = http://www.zope.org/Products/Zope/2.10.9/Zope-2.10.9-final.tgz

# External dependencies
Markdown = 1.7
# Changed from PIL
PILwoTk = 1.1.6.4
elementtree = 1.2.6-20050316
feedparser = 4.1
python-openid = 2.2.4
simplejson = 2.0.9

# Plone release
Plone = 3.3.1
Products.ATContentTypes = 1.3.4
Products.ATReferenceBrowserWidget = 2.0.4

{......}

++++++++++++++++++++++++++++++++++++++++++

PINNING (Alex)

http://pastie.org/821174

allow-picked-versions = false

++++++++++++++++++++++++++++++++++++++++++

DEBUGGING (Alex)

./bin/instance:

#!/opt/local/Library/Frameworks/Python.framework/Versions/2.4/Resources/Python.app/Contents/MacOS/Python

import sys
sys.path[0:0] = [
  '/Users/alex/buildout_sites/cfe_buildout/src/cfe.content',
  '/Users/alex/buildout_sites/cfe_buildout/src/cfe.theme',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/elementtree-1.2.6_20050316-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/PILwoTk-1.1.6.4-py2.4-macosx-10.3-i386.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/Plone-3.3.1-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/Products.CacheSetup-1.2.1-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/webcouturier.dropdownmenu-2.0-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/plone.reload-1.1-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/Products.PDBDebugMode-1.1-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/Products.Clouseau-0.8.4dev-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/Products.DocFinderTab-1.0.4-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/Products.Gloworm-1.0-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/plone.recipe.zope2instance-3.4-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/zc.recipe.egg-1.2.2-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/setuptools-0.6c9-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/zc.buildout-1.3.1-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/archetypes.kss-1.4.3-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/plone.app.customerize-1.1.2-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/collective.monkeypatcher-1.0b2-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/fake-eggs/Zope2',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/Products.CMFCore-2.1.2-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/fake-eggs/Acquisition',
  '/Users/alex/buildout_sites/cfe_buildout/fake-eggs/zope.testing',
  '/Users/alex/buildout_sites/cfe_buildout/fake-eggs/zope.interface',
  '/Users/alex/buildout_sites/cfe_buildout/fake-eggs/zope.component',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/plone.browserlayer-1.0.1-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/Products.PolicyHTTPCacheManager-1.2-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/Products.PageCacheManager-1.2-py2.4.egg',


.installed.cfg

[buildout]
installed_develop_eggs =
parts = roadrunner _mr.developer productdistros zope2 instance1 z3c.offlinepack packages zopepy

[roadrunner]
__buildout_installed__ = /Users/alex/buildout_sites/cfe_buildout/bin/roadrunner
        /Users/alex/buildout_sites/cfe_buildout/parts/roadrunner
__buildout_signature__ = roadrunner-0.2.3.1-py2.4.egg zc.buildout-1.3.1-py2.4.egg zc.recipe.egg-1.2.2-py2.4.egg setuptools-0.6c9-py2.4.egg zc.buildout-1.3.1-py2.4.egg
_b = /Users/alex/buildout_sites/cfe_buildout/bin
_d = /Users/alex/buildout_sites/cfe_buildout/develop-eggs
_e = /Users/alex/buildout_sites/cfe_buildout/eggs
bin-directory = /Users/alex/buildout_sites/cfe_buildout/bin
develop-eggs-directory = /Users/alex/buildout_sites/cfe_buildout/develop-eggs
eggs-directory = /Users/alex/buildout_sites/cfe_buildout/eggs
executable = /opt/local/Library/Frameworks/Python.framework/Versions/2.4/Resources/Python.app/Contents/MacOS/Python
find-links = http://dist.plone.org
        http://dist.plone.org/thirdparty
        http://download.zope.org/ppix/
        http://download.zope.org/distribution/
        http://effbot.org/downloads
packages-under-test = %(__buildout_space_n__)scfe.*
recipe = roadrunner:plone
zope2-instance = instance1

[_mr.developer]
__buildout_installed__ = /Users/alex/buildout_sites/cfe_buildout/bin/develop
__buildout_signature__ = zc.recipe.egg-1.2.2-py2.4.egg setuptools-0.6c9-py2.4.egg zc.buildout-1.3.1-py2.4.egg
_b = /Users/alex/buildout_sites/cfe_buildout/bin
_d = /Users/alex/buildout_sites/cfe_buildout/develop-eggs
_e = /Users/alex/buildout_sites/cfe_buildout/eggs
arguments = auto_checkout=set(['cfe.theme', 'cfe.content']),
        buildout_dir="/Users/alex/buildout_sites/cfe_buildout",
        develeggs={'cfe.content': '/Users/alex/buildout_sites/cfe_buildout/src/cfe.content',
        %(__buildout_space__)s'cfe.theme': '/Users/alex/buildout_sites/cfe_buildout/src/cfe.theme'},
        sources={'cfe.content': {'kind': 'svn',
        %(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space
__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s'name': 'cfe.content',
        %(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space
__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s'path': '/Users/alex/b
uildout_sites/cfe_buildout/src/cfe.content',
        %(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space
__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s'url': 'https://svn.ra
gingweb.com/wc/projects/WallaceCenter/cfe/cfe.content'},
        %(__buildout_space__)s'cfe.theme': {'kind': 'svn',
        %(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space
__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s'name': 'cfe.theme',
        %(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space
__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s'path': '/Users/alex/buildout_sites/cfe_buildout/src/cfe.theme',
        %(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space

..installed.cfg:


bin/buildout
#!/opt/local/Library/Frameworks/Python.framework/Versions/2.4/Resources/Python.app/Contents/MacOS/Python

import sys
sys.path[0:0] = [
  '/Users/alex/buildout_sites/cfe_buildout/eggs/zc.buildout-1.3.1-py2.4.egg',
  '/Users/alex/buildout_sites/cfe_buildout/eggs/setuptools-0.6c9-py2.4.egg',
  ]

import zc.buildout.buildout

if __name__ == '__main__':
    zc.buildout.buildout.main()


         ),
    ) == 0

ws.add_entry(tmpeggs)
ws.require('zc.buildout')
import zc.buildout.buildout
zc.buildout.buildout.main(sys.argv[1:] + ['bootstrap'])
shutil.rmtree(tmpeggs)

++++++++++++++++++++++++++++++++++++++++

Managing packages under development
===============================

Options:
1. "by hand"
2. svn externals + eggtractor
3. mr.developer
4. infrae.subversion

1. "by hand"
Set 'develop' setting so buildout uses a local egg instead of fetching releases from pypi.
(pinnning versions will probably override this)

[buildout]
develop = src/my.package   <— path to where you checked out the package

[instance]
eggs += my.package
zcml += my.package
pros: simple
cons: have to manage the checkout manually; gets tedious

2. svn externals + eggtractor
Automatically populate buildout's 'develop' setting based on scanning the 'src' directory

in src/EXTERNALS.txt:
my.package -r 1234 http://path/to/trunk

svn propset svn:externals -F EXTERNALS.txt .
svn up

[buildout]
extensions = buildout.eggtractor

pros: don't have to edit buildout.cfg when adding a new package
cons: only works with svn. still tedious to switch between a developed package and a release.

3. mr.developer
Keeps a list of package sources within your buildout configuration; lets you easily toggle which ones get developed.

[buildout]
extensions = mr.developer
sources = sources
always-checkout = true

auto-checkout = my.package

[sources]
my.package = svn http://path/to/trunk

pros: supports svn, git, hg, cvs. provides bin/develop command for managing checkouts without running buildout. all configuration contained without buildout .cfg files, which makes it easy to have several variants of a buildout which check out different things (e.g., this is how we managed Plone 4's PLIP process). easy to switch between releases and checkouts
cons: ??

4. infrae.subversion
A recipe that checks out a package and develops it.  It used to be buggy; I think it's better now, but don't see any huge points in favor of it over mr.developer.

===================
How to get ZCML loaded
===================

The simple way:

[instance]
recipe = plone.recipe.zope2instance
zcml = my.package

So what does that do?

Consider what happens when Zope starts up:
  - reads parts/instance/etc/site.zcml:
    - essentially: loads all *-meta.zcml "slugs" from parts/instance/etc/package-includes
    - then loads ZCML for all Products.* packages in the Python environment
    - then loads all *-configure.zcml "slugs" from package-includes
    - then loads any *-overrides.zcml files from package-includes.

davisagli:package-includes davidg$ ls
001-plone.app.skineditor-configure.zcml

Normally, those slugs are place in package-includes because of the zcml = my.package line in buildout.cfg

However!

As of Plone 3.3, a package can include the following in its setup.py:

   entry_points="""
      [z3c.autoinclude.plugin]
      target = plone
      """
and then Plone will recognize the package as an official Plone plugin, and will automatically load its ZCML even if it isn't listed in the buildout.cfg

How do you know if a package supports this?  No good way other than checking setup.py in the source repository.  Someone should lead a push to get this added to all common add-on products.

=========
default.cfg
=========

[buildout]
eggs-directory = /Users/davidg/.buildout/eggs
download-directory = /Users/davidg/.buildout/downloads
download-cache = /Users/davidg/.buildout/downloads
extends-cache = /Users/davidg/.buildout/extends



---------
a note - zcml, eggs, zcml
[instance1]
zcml-eggs =
    webcouturier.dropdownmenu
eggs =
    cfe.content
    cfe.theme
    ${buildout:eggs}
    elementtree
    PILwoTk
    Plone
    ${instance1:zcml-eggs}
zcml =
    ${instance1:zcml-eggs}
products =
    ${buildout:directory}/products
    ${productdistros:location}

++++++++++++++++++++++++++++++++++++++++++++

Chris: UW Marketing: svn properties

For ignores; ignore dynamically created directories: bin, data, var..

#
# applied by: svn propset svn:ignore -F ./IGNORE.txt .
#
.installed.cfg
.mr.developer.cfg
bin
buildout.cfg
develop-eggs
fake-eggs
parts
var
eggs
downloads

Only see what you care about

*.egg-info directory: doesn't need to be in svn either

Document Actions