Extending PyScaffold¶
PyScaffold is carefully designed to cover the essentials of authoring and
distributing Python packages. Most of time, tweaking putup
options is
enough to ensure proper configuration of a project.
However, for advanced use cases a deeper level of programmability may be
required and PyScaffold’s extension systems provides this.
PyScaffold can be extended at runtime by other Python packages.
Therefore it is possible to change the behaviour of the putup
command line
tool without changing the PyScaffold code itself. In order to explain how this
mechanism works, the following sections define a few important concepts and
present a comprehensive guide about how to create custom extensions.
Additionally, Cookiecutter templates can also be used but writing a native PyScaffold extension is the preferred way.
Note
A perfect start for your own custom extension is the extension custom_extension
for PyScaffold. Just install it with pip install pyscaffoldext-custom-extension
and then create your own extension template with
putup --custom-extension pyscaffoldext-my-own-extension
.
Project Structure Representation¶
Each Python package project is internally represented by PyScaffold as a tree
data structure, that directly relates to a directory entry in the file system.
This tree is implemented as a simple (and possibly nested) dict
in which
keys indicate the path where files will be generated, while values indicate
their content. For instance, the following dict:
{
'project': {
'folder': {
'file.txt': 'Hello World!',
'another-folder': {
'empty-file.txt': ''
}
}
}
}
represents a project/folder
directory in the file system containing two
entries. The first entry is a file named file.txt
with content Hello World!
while the second entry is a sub-directory named another-folder
. In turn,
another-folder
contains an empty file named empty-file.txt
.
Additionally, tuple values are also allowed in order to specify some useful metadata. In this case, the first element of the tuple is the file content. For example, the dict:
{
'project': {
'namespace': {
'module.py': ('print("Hello World!")', helpers.NO_OVERWRITE)
}
}
}
represents a project/namespace/module.py
file with content
print("Hello World!")
, that will not be overwritten if already exists.
Note
The NO_OVERWRITE
flag is defined in the module
pyscaffold.api.helpers
This tree representation is often referred in this document as project structure or simply structure.
Scaffold Actions¶
PyScaffold organizes the generation of a project into a series of steps with well defined purposes. Each step is called action and is implemented as a simple function that receives two arguments: a project structure and a dict with options (some of them parsed from command line arguments, other from default values).
An action MUST return a tuple also composed by a project structure and a dict with options. The return values, thus, are usually modified versions of the input arguments. Additionally an action can also have side effects, like creating directories or adding files to version control. The following pseudo-code illustrates a basic action:
def action(project_structure, options):
new_struct, new_opts = modify(project_structure, options)
some_side_effect()
return new_struct, new_opts
The output of each action is used as the input of the subsequent action, and
initially the structure argument is just an empty dict. Each action is uniquely
identified by a string in the format <module name>:<function name>
,
similarly to the convention used for a setuptools entry point.
For example, if an action is defined in the action
function of the
extras.py
file that is part of the pyscaffoldext.contrib
project,
the action identifier is pyscaffoldext.contrib.extras:action
.
By default, the sequence of actions taken by PyScaffold is:
pyscaffold.api:get_default_options
pyscaffold.api:verify_options_consistency
pyscaffold.structure:define_structure
pyscaffold.structure:apply_update_rules
pyscaffold.structure:create_structure
pyscaffold.api:init_git
The project structure is usually empty until define_structure. This action just loads the in-memory dict representation, that is only written to disk by the create_structure action.
Note that, this sequence varies according to the command line options.
To retrieve an updated list, please use putup --list-actions
or
putup --dry-run
.
What are Extensions?¶
From the standpoint of PyScaffold, an extension is just an class inheriting
from Extension
overriding and
implementing certain methods. This methods allow inject actions at arbitrary
positions in the aforementioned list. Furthermore, extensions can also remove
actions.
Creating an Extension¶
In order to create an extension it is necessary to write a class that inherits
from Extension
and implements the method
activate
that receives a list of actions,
registers a custom action that will be called later and returns a modified version
of the list of actions:
from ..api import Extension
from ..api import helpers
class MyExtension(Extension):
"""Help text on commandline when running putup -h"""
def activate(self, actions):
"""Activate extension
Args:
actions (list): list of actions to perform
Returns:
list: updated list of actions
"""
actions = helpers.register(actions, self.action, after='create_structure')
actions = helpers.unregister(actions, 'init_git')
return actions
def action(self, struct, opts):
"""Perform some actions that modifies the structure and options
Args:
struct (dict): project representation as (possibly) nested
:obj:`dict`.
opts (dict): given options, see :obj:`create_project` for
an extensive list.
Returns:
struct, opts: updated project representation and options
"""
...
return new_actions, new_opts
Note
The register
and unregister
methods implemented in the module
pyscaffold.api.helpers
basically create modified copies of the
action list by inserting/removing the specified functions, with some
awareness about their execution order.
Action List Helper Methods¶
As implied by the previous example, the helpers
module
provides a series of useful functions and makes it easier to manipulate the
action list, by using register
and
unregister
.
Since the action order is relevant, the first function accepts special keyword
arguments (before
and after
) that should be used to place the extension
actions precisely among the default actions. The value of these arguments can
be presented in 2 different forms:
helpers.register(actions, hook1, before='define_structure')
helpers.register(actions, hook2, after='pyscaffold.structure:create_structure')
The first form uses as a position reference the first action with a matching
name, regardless of the module. Accordingly, the second form tries to find an
action that matches both the given name and module. When no reference is given,
register
assumes as default position
after='pyscaffold.structure:define_structure'
. This position is special
since most extensions are expected to create additional files inside the
project. Therefore, it is possible to easily amend the project structure before
it is materialized by create_structure
.
The unregister
function accepts as second
argument a position reference which can similarly present the module name:
helpers.unregister(actions, 'init_git')
helpers.unregister(actions, 'pyscaffold.api:init_git')
Note
These functions DO NOT modify the actions list, instead they return a new list with the changes applied.
Note
For convenience, the functions register
and
unregister
are aliased as instance methods
of the Extension
class.
Therefore, inside the activate
method, one
could simply call actions = self.register(actions, self.my_action)
.
Structure Helper Methods¶
PyScaffold also provides extra facilities to manipulate the project structure.
The following functions are accessible through the
helpers
module:
The first function can be used to deep merge a dictionary argument with the
current representation of the to-be-generated directory tree, automatically
considering any metadata present in tuple values. On the other hand, the second
and third functions can be used to ensure a single file is present or absent in
the current representation of the project structure, automatically handling
parent directories. Finally, modify
can be used
to change the contents of an existing file in the project structure and/or
its metadata (for example adding NO_OVERWRITE
or
NO_CREATE
flags).
Note
Similarly to the actions list helpers, these functions also DO NOT modify the project structure. Instead they return a new structure with the changes applied.
The following example illustrates the implementation of a AwesomeFiles
extension which defines the define_awesome_files
action:
from pathlib import PurePath
from ..api import Extension
from ..api import helpers
MY_AWESOME_FILE = """\
# -*- coding: utf-8 -*-
__author__ = "{author}"
__copyright__ = "{author}"
__license__ = "{license}"
def awesome():
return "Awesome!"
"""
MY_AWESOME_TEST = """\
import pytest
from {qual_pkg}.awesome import awesome
def test_awesome():
assert awesome() == "Awesome!"
"""
class AwesomeFiles(Extension):
"""Adding some additional awesome files"""
def activate(self, actions):
return helpers.register(actions, self.define_awesome_files)
def define_awesome_files(self, struct, opts):
struct = helpers.merge(struct, {
opts['project']: {
'src': {
opts['package']: {
'awesome.py': MY_AWESOME_FILE.format(**opts)
},
}
'tests': {
'awesome_test.py': (
MY_AWESOME_TEST.format(**opts),
helpers.NO_OVERWRITE
)
}
}
})
struct['.python-version'] = ('3.6.1', helpers.NO_OVERWRITE)
for filename in ['awesome_file1', 'awesome_file2']:
struct = helpers.ensure(
struct,
PurePath(opts['project'], 'src', 'awesome', filename),
content='AWESOME!', update_rule=helpers.NO_CREATE)
# The second argument is the file path, represented by a
# list of file parts or a string.
# Alternatively in this example:
# path = '{project}/src/awesome/{filename}'.format(
# filename=filename, **opts)
# The `reject` can be used to avoid default files being generated.
struct = helpers.reject(
struct, '{project}/src/{package}/skeleton.py'.format(**opts))
# Alternatively in this example:
# path = [opts['project'], 'src', opts['package'], 'skeleton.py'])
# `modify` can be used to change contents in an existing file
struct = helpers.modify(
struct,
PurePath(opts['project'], 'tests', 'awesome_test.py'),
lambda content: 'import pdb\n' + content)
# And/or change the update behavior
struct = helpers.modify(struct, [opts['project'], '.travis.yml'],
update_rule=helpers.NO_CREATE)
# It is import to remember the return values
return struct, opts
Note
The project
and package
options should be used to provide
the correct location of the files relative to the current working
directory.
As shown by the previous example, the helpers
module
also presents constants that can be used as metadata. The NO_OVERWRITE
flag
avoids an existing file to be overwritten when putup
is used in update
mode. Similarly, NO_CREATE
avoids creating a file from template in update
mode, even if it does not exist.
For more sophisticated extensions which need to read and parse their
own command line arguments it is necessary to override
activate
that receives an
argparse.ArgumentParser
argument. This object can then be modified
in order to add custom command line arguments that will later be stored in the
opts
dictionary.
Just remember the convention that after the command line arguments parsing,
the extension function should be stored under the extensions
attribute
(a list) of the argparse
generated object. For reference check out the
implementation of the namespace extension
as well as the pyproject extension which serves as a blueprint for new
extensions.
Activating Extensions¶
PyScaffold extensions are not activated by default. Instead, it is necessary
to add a CLI option to do it.
This is possible by setting up a setuptools entry point under the
pyscaffold.cli
group.
This entry point should point to our extension class, e.g. AwesomeFiles
like defined above. If you for instance use a scaffold generated by PyScaffold
to write a PyScaffold extension (we hope you do ;-), you would add the following
to the options.entry_points
section in setup.cfg
:
[options.entry_points]
pyscaffold.cli =
awesome_files = your_package.your_module:AwesomeFiles
Examples¶
Some options for the putup
command are already implemented as extensions
and can be used as reference implementation:
Conventions for Community Extensions¶
In order to make it easy to find PyScaffold extensions, community packages
should be namespaced as in pyscaffoldext.${EXT_NAME}
(where ${EXT_NAME}
is the name of the extension being developed). Although this naming convention
slightly differs from PEP423, it is close enough and shorter.
Similarly to sphinxcontrib-*
packages, names registered in PyPI should
contain a dash -
, instead of a dot .
. This way, third-party extension
development can be easily bootstrapped with the command:
putup pyscaffoldext-${EXT_NAME} -p ${EXT_NAME} --namespace pyscaffoldext --no-skeleton
If you put your extension code in the module extension.py
then the
options.entry_points
section in setup.cfg
looks like:
[options.entry_points]
pyscaffold.cli =
awesome_files = pyscaffoldext.${EXT_NAME}.extension:AwesomeFiles
In this example, AwesomeFiles
represents the name of the class that
implementes the extension and awesome_files
is the string used to create
the flag for the putup
command (--awesome-files
).
Final Considerations¶
When writing extensions, it is important to be consistent with the default
PyScaffold behavior. In particular, PyScaffold uses a pretend
option to
indicate when the actions should not run but instead just indicate the
expected results to the user, that MUST be respected.
The pretend
option is automatically observed for files registered in
the project structure representation, but complex actions may require
specialized coding. The helpers
module provides a
special logger
object useful in
these situations. Please refer to Cookiecutter Extension for a
practical example.
Other options that should be considered are the update
and force
flags. See pyscaffold.api.create_project
for a list of available
options.