Transformation pipelines
Important
Loki is still under active development and has not yet seen a stable release. Interfaces can change at any time, objects may be renamed, or concepts may be re-thought. Make sure to sync your work to the current release frequently by rebasing feature branches and upstreaming more general applicable work in the form of pull requests.
Transformations
Transformations are the building blocks of a transformation pipeline in Loki.
They encode the workflow of converting a Sourcefile
or an individual
program unit (such as Module
or Subroutine
) to the desired
output format.
A transformation can encode a single modification, combine multiple steps, or call other transformations to create complex changes. If a transformation depends on another transformation, inheritance can be used to combine them.
Every transformation in a pipeline should implement the interface defined by
Transformation
. It provides generic entry points for transforming
different objects and thus allows for batch processing. To implement a new
transformation, only one or all of the relevant methods
Transformation.transform_subroutine
,
Transformation.transform_module
, or Transformation.transform_file
need to be implemented.
Example: A transformation that renames every module and subroutine by appending a given suffix:
class RenameTransformation(Transformation):
def __init__(self, suffix):
super().__init__()
self._suffix = suffix
def _rename(self, item):
item.name += self._suffix
def transform_subroutine(self, routine, **kwargs):
self._rename(routine)
def transform_module(self, module, **kwargs):
self._rename(module)
The transformation can be applied by calling apply()
with
the relevant object.
source = Sourcefile(...) # may contain modules and subroutines
transformation = RenameTransformation(suffix='_example')
transformation.apply(source)
Note that, despite only implementing logic for transforming modules and
subroutines, it works also for sourcefiles. While the Sourcefile
object itself is not modified (because we did not implement
transform_file()
), the dispatch mechanism in the Transformation
base class takes care of calling the relevant method for each member of the
given object.
Typically, transformations should be implemented by users to encode the transformation pipeline for their individual use-case. However, Loki comes with a few built-in transformations for common tasks and we expect this list to grow in the future:
Base class for source code transformations that manipulate source items like |
|
Generates a list of files to add, remove or replace in a CMake target's list of sources |
|
|
Write out modified source files to a select build directory |
|
Basic |
|
Fortran-to-C transformation that translates the given routine into C and generates the corresponding ISO-C wrappers. |
|
Fortran-to-Maxeler transformation that translates the given routine into .maxj and generates a matching manager and host code with corresponding ISO-C wrappers. |
|
A transformer class to convert Fortran to Python or DaCe. |
|
Base class for the Analysis part of the hoist variables functionality/transformation. |
|
Base class for the Synthesis part of the hoist variables functionality/transformation. |
|
Specialisation for the Analysis part of the hoist variables functionality/transformation, to hoist only temporary arrays and if provided only temporary arrays with specific variables/variable names within the array dimensions. |
|
Specialisation for the Synthesis part of the hoist variables functionality/transformation, to hoist temporary arrays and make them |
|
Parametrise variables with provided values. |
Further transformations are defined for specific use-cases but may prove
useful in a wider context. These are defined in transformations
:
Additionally, a number of tools for common transformation tasks are provided as functions that can be readily used in a step of the transformation pipeline:
|
Shift all array indices to adjust to 0-based indexing conventions (eg. |
|
Invert data/loop accesses from column to row-major |
|
Resolve implicit vector notation by inserting explicit loops |
|
Replace the |
|
Promote a list of variables by inserting new array dimensions of given size and updating all uses of these variables with a given index expression. |
|
Promote multiple variables with potentially non-matching promotion dimensions or index expressions. |
|
Determine promotion dimensions corresponding to the iteration space of a loop nest. |
|
Resolve |
|
Replace instances of variables with known constant values by Literals. |
|
Replaces InlineCall expression to elemental functions with the called functions body. |
Search for loops annotated with the loki loop-interchange pragma and attempt to reorder them. |
|
Search for loops annotated with the loki loop-fusion pragma and attempt to fuse them into a single loop. |
|
Search for |
|
Hoist one or multiple code regions annotated by pragma ranges and insert them at a specified target location. |
|
Convert regions annotated with |
|
|
Converts all variables and symbols in a subroutine to lower-case. |
|
Replace known intrinsic functions and symbols. |
Sanitise imports by removing unused symbols and eliminating imports with empty symbol lists. |
|
|
Find all uses of |
|
Modify/extend variable declarations to |
Bulk processing large source trees
Transformations can be applied over source trees using the Scheduler
.
It is a work queue manager that automatically discovers source files in a list
of paths. Given the name of an entry routine, it allows to build a call graph
and thus derive the dependencies within this source tree.
Calling Scheduler.process
on a source tree and providing it with a
Transformation
applies this transformation to all modules and routines,
making sure that routines with the relevant CallStatement
are always
processed before their target Subroutine
.
When applying the transformation to an item in the source tree, the scheduler provides certain information about the item to the transformation:
the transformation mode (provided in the scheduler’s config),
the item’s role (e.g., ‘driver’ or ‘kernel’, configurable via the scheduler’s config), and
targets (routines that are called from the item and are included in the scheduler’s tree, i.e., will be processed afterwards).
|
Work queue manager to enqueue and process individual Item routines/modules with a given kernel. |
|
Configuration object for the transformation |
|
Base class of a work item that represents a single source routine to be processed |
|
Implementation of |
|
Implementation of |