Buildout Potluck Notes
From the Etherpad presentation. Not as good as being there, but still helpful!
Agenda:
version pinning (possibly fulvio, as part of next bullet)
how to handle production, staging, dev instance (fulvio)
how to deal with packages under dev - David
how to get zcml loaded - when it's needed - David
debugging buildout - Alex
multiple storages - different mount points
what other things do you put in buildout - reverse proxy, cron jobs,
log rotations, backups (possibly fulvio)
setting subversion properties in buildout - Chris H
running command-line commands with a buildout recipe
++++++++++++++++++++++++++++++++++++++++++++++
How to handle production, staging, dev instance - (Fulvio)
config folder:
------------------
buildout.cfg.tmpl
base.cfg
devel.cfg
staging.cfg
prod.cfg
versions.cfg
varnish.cfg
zeo.cfg
pound.cfg
buildout.cfg.tmpl
------------------------
[buildout]
extends =
config/prod.cfg
# config/staging.cfg
# config/devel.cfg
## Optional devel.cfg support. If you're using devel.cfg from above,
## you can uncomment lines below to enable
parts +=
{.....}
NOTES: Use svn ignore to not check in buildout.cfg which is made manually) from the buildout.cfg.tmpl
zeo.cfg:
======
[buildout]
extends = base.cfg
parts +=
zeo
backup
backup-daily
backup-weekly
logrotate
logrotate-daily
supervisor
supervisor-reboot
{.....}
[backup]
recipe = collective.recipe.backup
[backup-daily]
recipe = z3c.recipe.usercrontab
times = 0 3 * * 0-6
command = ${buildout:directory}/bin/backup
[backup-weekly]
recipe = z3c.recipe.usercrontab
times = 0 3 * * 7
command = ${buildout:directory}/bin/zeopack -p 8000 -d 1 && ${buildout:directory}/bin/backup
[logrotate]
recipe = collective.recipe.template
input = etc/logrotate.conf.tmpl
output = etc/logrotate.conf
[logrotate-daily]
recipe = z3c.recipe.usercrontab
times = 0 6 * * *
command = /usr/sbin/logrotate --state ${buildout:directory}/var/logrotate.status ${buildout:directory}/${logrotate:output}
[supervisor]
recipe = zc.recipe.egg
eggs = supervisor
[supervisor-reboot]
recipe = z3c.recipe.usercrontab
times = @reboot
command = ${buildout:directory}/bin/supervisord -c ${buildout:directory}/etc/supervisord.conf
logrotate.conf.tmpl
==============
rotate 4
weekly
create
compress
delaycompress
${buildout:directory}/var/log/zeo.log {
postrotate
${buildout:bin-directory}/zeo logreopen
endscript
}
${buildout:directory}/var/log/instance1.log ${buildout:directory}/var/log/instance1-Z2.log {
sharedscripts
postrotate
/bin/kill -USR2 $(cat ${buildout:directory}/var/instance1.pid)
endscript
}
(USR2) this works only with newer zope . (not with older zope or zeo or .??) for those, stop and start.
devel.cfg
=======
[buildout]
extends = base.cfg
parts +=
packages
zopepy
{.....}
base.cfg
======
[buildout]
extensions +=
mr.developer
extends =
versions.cfg
versions = versions
{.....}
versions.cfg
=========
[buildout]
extensions = buildout.dumppickedversions
[versions]
# Buildout infrastructure
buildout.eggtractor = 0.6
plone.recipe.zope2install = 3.2
plone.recipe.zope2instance = 3.4
tl.eggdeps = 0.3.2
setuptools = 0.6c9
zc.buildout = 1.3.1
zc.recipe.egg = 1.2.2
zc.recipe.cmmi = 1.2.1
# Zope
zope2-url = http://www.zope.org/Products/Zope/2.10.9/Zope-2.10.9-final.tgz
# External dependencies
Markdown = 1.7
# Changed from PIL
PILwoTk = 1.1.6.4
elementtree = 1.2.6-20050316
feedparser = 4.1
python-openid = 2.2.4
simplejson = 2.0.9
# Plone release
Plone = 3.3.1
Products.ATContentTypes = 1.3.4
Products.ATReferenceBrowserWidget = 2.0.4
{......}
++++++++++++++++++++++++++++++++++++++++++
PINNING (Alex)
http://pastie.org/821174
allow-picked-versions = false
++++++++++++++++++++++++++++++++++++++++++
DEBUGGING (Alex)
./bin/instance:
#!/opt/local/Library/Frameworks/Python.framework/Versions/2.4/Resources/Python.app/Contents/MacOS/Python
import sys
sys.path[0:0] = [
'/Users/alex/buildout_sites/cfe_buildout/src/cfe.content',
'/Users/alex/buildout_sites/cfe_buildout/src/cfe.theme',
'/Users/alex/buildout_sites/cfe_buildout/eggs/elementtree-1.2.6_20050316-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/PILwoTk-1.1.6.4-py2.4-macosx-10.3-i386.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/Plone-3.3.1-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/Products.CacheSetup-1.2.1-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/webcouturier.dropdownmenu-2.0-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/plone.reload-1.1-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/Products.PDBDebugMode-1.1-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/Products.Clouseau-0.8.4dev-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/Products.DocFinderTab-1.0.4-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/Products.Gloworm-1.0-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/plone.recipe.zope2instance-3.4-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/zc.recipe.egg-1.2.2-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/setuptools-0.6c9-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/zc.buildout-1.3.1-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/archetypes.kss-1.4.3-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/plone.app.customerize-1.1.2-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/collective.monkeypatcher-1.0b2-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/fake-eggs/Zope2',
'/Users/alex/buildout_sites/cfe_buildout/eggs/Products.CMFCore-2.1.2-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/fake-eggs/Acquisition',
'/Users/alex/buildout_sites/cfe_buildout/fake-eggs/zope.testing',
'/Users/alex/buildout_sites/cfe_buildout/fake-eggs/zope.interface',
'/Users/alex/buildout_sites/cfe_buildout/fake-eggs/zope.component',
'/Users/alex/buildout_sites/cfe_buildout/eggs/plone.browserlayer-1.0.1-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/Products.PolicyHTTPCacheManager-1.2-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/Products.PageCacheManager-1.2-py2.4.egg',
.installed.cfg
[buildout]
installed_develop_eggs =
parts = roadrunner _mr.developer productdistros zope2 instance1 z3c.offlinepack packages zopepy
[roadrunner]
__buildout_installed__ = /Users/alex/buildout_sites/cfe_buildout/bin/roadrunner
/Users/alex/buildout_sites/cfe_buildout/parts/roadrunner
__buildout_signature__ = roadrunner-0.2.3.1-py2.4.egg zc.buildout-1.3.1-py2.4.egg zc.recipe.egg-1.2.2-py2.4.egg setuptools-0.6c9-py2.4.egg zc.buildout-1.3.1-py2.4.egg
_b = /Users/alex/buildout_sites/cfe_buildout/bin
_d = /Users/alex/buildout_sites/cfe_buildout/develop-eggs
_e = /Users/alex/buildout_sites/cfe_buildout/eggs
bin-directory = /Users/alex/buildout_sites/cfe_buildout/bin
develop-eggs-directory = /Users/alex/buildout_sites/cfe_buildout/develop-eggs
eggs-directory = /Users/alex/buildout_sites/cfe_buildout/eggs
executable = /opt/local/Library/Frameworks/Python.framework/Versions/2.4/Resources/Python.app/Contents/MacOS/Python
find-links = http://dist.plone.org
http://dist.plone.org/thirdparty
http://download.zope.org/ppix/
http://download.zope.org/distribution/
http://effbot.org/downloads
packages-under-test = %(__buildout_space_n__)scfe.*
recipe = roadrunner:plone
zope2-instance = instance1
[_mr.developer]
__buildout_installed__ = /Users/alex/buildout_sites/cfe_buildout/bin/develop
__buildout_signature__ = zc.recipe.egg-1.2.2-py2.4.egg setuptools-0.6c9-py2.4.egg zc.buildout-1.3.1-py2.4.egg
_b = /Users/alex/buildout_sites/cfe_buildout/bin
_d = /Users/alex/buildout_sites/cfe_buildout/develop-eggs
_e = /Users/alex/buildout_sites/cfe_buildout/eggs
arguments = auto_checkout=set(['cfe.theme', 'cfe.content']),
buildout_dir="/Users/alex/buildout_sites/cfe_buildout",
develeggs={'cfe.content': '/Users/alex/buildout_sites/cfe_buildout/src/cfe.content',
%(__buildout_space__)s'cfe.theme': '/Users/alex/buildout_sites/cfe_buildout/src/cfe.theme'},
sources={'cfe.content': {'kind': 'svn',
%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space
__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s'name': 'cfe.content',
%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space
__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s'path': '/Users/alex/b
uildout_sites/cfe_buildout/src/cfe.content',
%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space
__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s'url': 'https://svn.ra
gingweb.com/wc/projects/WallaceCenter/cfe/cfe.content'},
%(__buildout_space__)s'cfe.theme': {'kind': 'svn',
%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space
__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s'name': 'cfe.theme',
%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space
__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s'path': '/Users/alex/buildout_sites/cfe_buildout/src/cfe.theme',
%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space__)s%(__buildout_space
..installed.cfg:
bin/buildout
#!/opt/local/Library/Frameworks/Python.framework/Versions/2.4/Resources/Python.app/Contents/MacOS/Python
import sys
sys.path[0:0] = [
'/Users/alex/buildout_sites/cfe_buildout/eggs/zc.buildout-1.3.1-py2.4.egg',
'/Users/alex/buildout_sites/cfe_buildout/eggs/setuptools-0.6c9-py2.4.egg',
]
import zc.buildout.buildout
if __name__ == '__main__':
zc.buildout.buildout.main()
),
) == 0
ws.add_entry(tmpeggs)
ws.require('zc.buildout')
import zc.buildout.buildout
zc.buildout.buildout.main(sys.argv[1:] + ['bootstrap'])
shutil.rmtree(tmpeggs)
++++++++++++++++++++++++++++++++++++++++
Managing packages under development
===============================
Options:
1. "by hand"
2. svn externals + eggtractor
3. mr.developer
4. infrae.subversion
1. "by hand"
Set 'develop' setting so buildout uses a local egg instead of fetching releases from pypi.
(pinnning versions will probably override this)
[buildout]
develop = src/my.package <— path to where you checked out the package
[instance]
eggs += my.package
zcml += my.package
pros: simple
cons: have to manage the checkout manually; gets tedious
2. svn externals + eggtractor
Automatically populate buildout's 'develop' setting based on scanning the 'src' directory
in src/EXTERNALS.txt:
my.package -r 1234 http://path/to/trunk
svn propset svn:externals -F EXTERNALS.txt .
svn up
[buildout]
extensions = buildout.eggtractor
pros: don't have to edit buildout.cfg when adding a new package
cons: only works with svn. still tedious to switch between a developed package and a release.
3. mr.developer
Keeps a list of package sources within your buildout configuration; lets you easily toggle which ones get developed.
[buildout]
extensions = mr.developer
sources = sources
always-checkout = true
auto-checkout = my.package
[sources]
my.package = svn http://path/to/trunk
pros: supports svn, git, hg, cvs. provides bin/develop command for managing checkouts without running buildout. all configuration contained without buildout .cfg files, which makes it easy to have several variants of a buildout which check out different things (e.g., this is how we managed Plone 4's PLIP process). easy to switch between releases and checkouts
cons: ??
4. infrae.subversion
A recipe that checks out a package and develops it. It used to be buggy; I think it's better now, but don't see any huge points in favor of it over mr.developer.
===================
How to get ZCML loaded
===================
The simple way:
[instance]
recipe = plone.recipe.zope2instance
zcml = my.package
So what does that do?
Consider what happens when Zope starts up:
- reads parts/instance/etc/site.zcml:
- essentially: loads all *-meta.zcml "slugs" from parts/instance/etc/package-includes
- then loads ZCML for all Products.* packages in the Python environment
- then loads all *-configure.zcml "slugs" from package-includes
- then loads any *-overrides.zcml files from package-includes.
davisagli:package-includes davidg$ ls
001-plone.app.skineditor-configure.zcml
Normally, those slugs are place in package-includes because of the zcml = my.package line in buildout.cfg
However!
As of Plone 3.3, a package can include the following in its setup.py:
entry_points="""
[z3c.autoinclude.plugin]
target = plone
"""
and then Plone will recognize the package as an official Plone plugin, and will automatically load its ZCML even if it isn't listed in the buildout.cfg
How do you know if a package supports this? No good way other than checking setup.py in the source repository. Someone should lead a push to get this added to all common add-on products.
=========
default.cfg
=========
[buildout]
eggs-directory = /Users/davidg/.buildout/eggs
download-directory = /Users/davidg/.buildout/downloads
download-cache = /Users/davidg/.buildout/downloads
extends-cache = /Users/davidg/.buildout/extends
---------
a note - zcml, eggs, zcml
[instance1]
zcml-eggs =
webcouturier.dropdownmenu
eggs =
cfe.content
cfe.theme
${buildout:eggs}
elementtree
PILwoTk
Plone
${instance1:zcml-eggs}
zcml =
${instance1:zcml-eggs}
products =
${buildout:directory}/products
${productdistros:location}
++++++++++++++++++++++++++++++++++++++++++++
Chris: UW Marketing: svn properties
For ignores; ignore dynamically created directories: bin, data, var..
#
# applied by: svn propset svn:ignore -F ./IGNORE.txt .
#
.installed.cfg
.mr.developer.cfg
bin
buildout.cfg
develop-eggs
fake-eggs
parts
var
eggs
downloads
Only see what you care about
*.egg-info directory: doesn't need to be in svn either
Forgot your password?
