Transformation pipelines
Important
Loki is still under active development and has not yet seen a stable release. Interfaces can change at any time, objects may be renamed, or concepts may be re-thought. Make sure to sync your work to the current release frequently by rebasing feature branches and upstreaming more general applicable work in the form of pull requests.
Transformations
Transformations are the building blocks of a transformation pipeline in Loki.
They encode the workflow of converting a Sourcefile
or an individual
program unit (such as Module
or Subroutine
) to the desired
output format.
A transformation can encode a single modification, combine multiple steps, or call other transformations to create complex changes. If a transformation depends on another transformation, inheritance can be used to combine them.
Every transformation in a pipeline should implement the interface defined by
Transformation
. It provides generic entry points for transforming
different objects and thus allows for batch processing. To implement a new
transformation, only one or all of the relevant methods
Transformation.transform_subroutine
,
Transformation.transform_module
, or Transformation.transform_file
need to be implemented.
Example: A transformation that renames every module and subroutine by appending a given suffix:
class RenameTransformation(Transformation):
def __init__(self, suffix):
super().__init__()
self._suffix = suffix
def _rename(self, item):
item.name += self._suffix
def transform_subroutine(self, routine, **kwargs):
self._rename(routine)
def transform_module(self, module, **kwargs):
self._rename(module)
The transformation can be applied by calling apply()
with
the relevant object.
source = Sourcefile(...) # may contain modules and subroutines
transformation = RenameTransformation(suffix='_example')
transformation.apply(source)
Note that, despite only implementing logic for transforming modules and
subroutines, it works also for sourcefiles. While the Sourcefile
object itself is not modified (because we did not implement
transform_file()
), the dispatch mechanism in the Transformation
base class takes care of calling the relevant method for each member of the
given object.
Typically, transformations should be implemented by users to encode the transformation pipeline for their individual use-case. However, Loki comes with a few built-in transformations for common tasks and we expect this list to grow in the future:
Further transformations are defined for specific use-cases but may prove
useful in a wider context. These are defined in transformations
:
Additionally, a number of tools for common transformation tasks are provided as functions that can be readily used in a step of the transformation pipeline:
Bulk processing large source trees
Transformations can be applied over source trees using the Scheduler
.
It is a work queue manager that automatically discovers source files in a list
of paths. Given the name of an entry routine, it allows to build a call graph
and thus derive the dependencies within this source tree.
Calling Scheduler.process
on a source tree and providing it with a
Transformation
applies this transformation to all modules and routines,
making sure that routines with the relevant CallStatement
are always
processed before their target Subroutine
.
When applying the transformation to an item in the source tree, the scheduler provides certain information about the item to the transformation:
the transformation mode (provided in the scheduler’s config),
the item’s role (e.g., ‘driver’ or ‘kernel’, configurable via the scheduler’s config), and
targets (routines that are called from the item and are included in the scheduler’s tree, i.e., will be processed afterwards).