Emissions CSV export

[aaccensi]

Context

As part of the emissions project we want to be able to export the emissions data in a CSV. The existing ConfiguredCSVSerializer serializes graph data to CSV based on a YAML config in ETSource, but it only supported simple query-based columns. The emissions export requires iterating over nodes in a group and reading attributes directly from each node, including support for both the energy and molecules graphs.

Implemented changes

Expanded ConfiguredCSVSerializer to support two new column types:

• node_group: A hidden column whose value names a node group. The config row is expanded into one CSV row per node in that group, combining nodes from both the energy and molecules graphs.
• node_attribute: Evaluates a Ruby expression against node_api for each expanded node. Supports an optional transform: expression (with value bound to the result) for post-processing, e.g. unit conversion or conditional mapping.

Other changes:

• Added an optional period: parameter (:present or :future) to the serializer to select which graph to use for node expansion. Existing column types (present, future, unit, query) are unaffected.
• Fixed label: being silently ignored for non-query column types.
• Added emissions_present and emissions_future CSV endpoints backed by a new emissions_csv.yml config in ETSource.

Related

This effort is in 3 places but only 2 of them have PRs (to merge back into the general emissions branch)

• etengine: Emissions CSV export etengine#1753
• etsource: Emissions CSV export #3443
• etmodel: No PR, created emissions branch that didn't exist yet

Checklist

• I have tested these changes
• I have updated documentation as needed
• I have tagged the relevant people for review

[mabijkerk]

The data export works and the config looks clear enough: the rows are not too much. The data export does generate various numbers of decimals. Perhaps it is good to determine a fixed amount of decimals?

[kndehaan]

The data export does generate various numbers of decimals. Perhaps it is good to determine a fixed amount of decimals?

Discussed with @noracato that we have two approaches to do this:

1. Limit the number of decimals in the transform function in the yml file: transform: "(value * 1e-6).round(3)". This does complicates it a bit more in the yml file however.
2. Apply the same decimals rounding to all data exports that are generated with the serializer and yml files. Then the rounding can be integrated in the serializer.

This is something that we should discuss further.

[kndehaan]

Looking good! Once quintel/etengine#1753 has been reapproved, we can merge this.