We would like to get rid of weave dependencies, avoid name conflicts with the support code and have a nicer user interface for the produced module. The proposed new structure is as follows:
We would like to get rid of weave dependencies, avoid name conflicts with the support code and have a nicer user interface for the produced module. The proposed new structure is as follows:
{{{
.. code-block:: c
<imports>
<imports>
struct op1 {
<persistent variables>
struct op1 {
<support code>
<persistent variables>
init() { <initialize persistent fields> }
<support code>
cleanup { <clean up persistent fields> }
init() { <initialize persistent fields> }
run(<inputs>) { <run the computation for op1> }
cleanup { <clean up persistent fields> }
};
run(<inputs>) { <run the computation for op1> }
};
struct op2 { <same> };
...
struct op2 { <same> };
struct opN { <ditto> };
struct driver {
op1 o1; op2 o2; ... opN oN;
<input storage>
<output storage>
init(<storage>) { <initialize ops, storage> }
cleanup() { <free storage?> }
run() {
<extract inputs>
o1.run(input1, input2);
o2.run(o1.output1);
...
...
oN.run(...);
struct opN { <ditto> };
<sync outputs>
}
struct driver {
}
op1 o1; op2 o2; ... opN oN;
<input storage>
PyObject* <name>(PyObject* inputs) {
<output storage>
<init driver, input/output storage>
init(<storage>) { <initialize ops, storage> }
<put inputs in input storage>
cleanup() { <free storage?> }
driver.run()
run() {
<free input storage>
<extract inputs>
<return output storage>
o1.run(input1, input2);
}
o2.run(o1.output1);
...
PyObject* <name>_driver(PyObject* storage) {
oN.run(...);
<init driver with storage>
<sync outputs>
<return driver>
}
}
}
<export <name> and <name>_driver>
PyObject* <name>(PyObject* inputs) {
}}}
<init driver, input/output storage>
<put inputs in input storage>
driver.run()
<free input storage>
<return output storage>
}
PyObject* <name>_driver(PyObject* storage) {
<init driver with storage>
<return driver>
}
<export <name> and <name>_driver>
Gains:
Gains:
* support code can be put inside a struct and become private to the Op
* support code can be put inside a struct and become private to the Op
* we can export several functions that can be used directly, eg {{{z = module.add(1, 2)}}}
* we can export several functions that can be used directly, eg ``z = module.add(1, 2)``
* this won't do filtering like {{{Result.filter}}} so the usefulness is limited by that
* this won't do filtering like ``Result.filter`` so the usefulness is limited by that
* the sequence of operations might be clearer to read
* the sequence of operations might be clearer to read
* we can use more descriptive names in each Op struct representing its input names (if we can find them using the inspect module) without worrying about name conflicts
* we can use more descriptive names in each Op struct representing its input names (if we can find them using the inspect module) without worrying about name conflicts
...
@@ -106,14 +108,15 @@ Losses:
...
@@ -106,14 +108,15 @@ Losses:
* make functions static and inline as much as possible
* make functions static and inline as much as possible
== Caching ==
Caching
=======
The current way of caching is from a hash of the generated code. That is inefficient because code has to be generated each time, which might be a costly process. Furthermore, usage of hashing in sets make it difficult to ensure a consistent ordering of Ops in graphs where several orderings are valid, so the generated C code is potentially different each time. Here is a proposal for a better way to compute the hash:
The current way of caching is from a hash of the generated code. That is inefficient because code has to be generated each time, which might be a costly process. Furthermore, usage of hashing in sets make it difficult to ensure a consistent ordering of Ops in graphs where several orderings are valid, so the generated C code is potentially different each time. Here is a proposal for a better way to compute the hash:
* Result_hash = Result version + Result desc
* Result_hash = Result version + Result desc
* Op_hash = Op version + Op desc + input/output hashes
* Op_hash = Op version + Op desc + input/output hashes
* Env_hash = Env version + combination of the Op hashes and their traversal order wrt a consistent traversal method
* Env_hash = Env version + combination of the Op hashes and their traversal order wrt a consistent traversal method
The version could be set explicitly via a {{{__version__}}} field or it could simply be equal to the file's last modification date. We could also have a {{{__nocache__}}} field indicating that code produced by the Op or Result cannot be cached.
The version could be set explicitly via a ``__version__`` field or it could simply be equal to the file's last modification date. We could also have a ``__nocache__`` field indicating that code produced by the Op or Result cannot be cached.
It should also be easier to bypass the cache (eg an option to CLinker to regenerate the code).
It should also be easier to bypass the cache (eg an option to CLinker to regenerate the code).
