Compose a spec from several files#

Build one spec out of files that each say part of it. merge composes fragments: the files of a component library, each owning part of the math. override lays patches over a base: the spec a framework ships, and the change a project makes to it. Each loads every fragment and the base with to_spec before it composes them, and hands back the composed spec loaded the same way. Each takes its files as a list, and the two compose as override(merge([…]), […]).

A library of components#

Write the network as a spec. It balances the injection at a bus, and reads the injection under given: what the components put in is theirs to say. Nothing in it names a component class.

network.yaml

dimensions:
  snapshot: { dtype: int }
  bus: { dtype: str }
given:
  expressions:
    Bus_injection:
      dims: [snapshot, bus]
      description: what the components put into a bus
constraints:
  Bus_balance:
    dims: [snapshot, bus]
    expression: Bus_injection == 0

Write each component file against the network. It declares its own dimension and its own math. It reads Bus_injection too, and says what it puts into a bus as a named expression, a term whose adds_to: names Bus_injection.

generator.yaml

dimensions:
  snapshot: { dtype: int }
  bus: { dtype: str }
  generator: { dtype: str }
relations:
  Generator_bus: { key: generator, values: bus }
parameters:
  Generator_p_nom: { dims: [generator] }
  Generator_marginal_cost: { dims: [generator] }
variables:
  Generator_p: { dims: [snapshot, generator], bounds: { lower: 0, upper: Generator_p_nom } }
given:
  expressions:
    Bus_injection: { dims: [snapshot, bus] }
expressions:
  Generator_injection:
    expression: sum(Generator_p, by=Generator_bus, over=generator, into=bus)
    adds_to: Bus_injection
objective:
  sense: minimize
  expression: sum(Generator_p * Generator_marginal_cost)

load.yaml

dimensions:
  snapshot: { dtype: int }
  bus: { dtype: str }
  load: { dtype: str }
relations:
  Load_bus: { key: load, values: bus }
parameters:
  Load_p_set: { dims: [snapshot, load] }
given:
  expressions:
    Bus_injection: { dims: [snapshot, bus] }
expressions:
  Load_injection:
    expression: -sum(Load_p_set, by=Load_bus, over=load, into=bus)
    adds_to: Bus_injection

Each file loads on its own and prints as math on its own.

Merge the files you need. Give them as a list. A refusal names a file by its path as the list gives it, and a mapping or a loaded spec by its place in the list, such as '#2'.

import mathspec as ms

spec = ms.merge(['network.yaml', 'generator.yaml', 'load.yaml'])

spec defines Bus_injection as Generator_injection + Load_injection, and keeps each term as a named expression. Nothing is left under given:, so spec is fully defined. The objectives of the fragments are summed, each term in parentheses, in the order of the list. The order changes no meaning: canonical writes the same text for every order.

Add a component without touching the network. A new file adds its own term, and network.yaml stays as it is.

store.yaml

dimensions:
  snapshot: { dtype: int }
  bus: { dtype: str }
  store: { dtype: str }
relations:
  Store_bus: { key: store, values: bus }
parameters:
  Store_e_nom: { dims: [store] }
variables:
  Store_p: { dims: [snapshot, store] }
  Store_e: { dims: [snapshot, store], bounds: { lower: 0, upper: Store_e_nom } }
constraints:
  Store_energy_balance:
    dims: [snapshot, store]
    expression: Store_e == shift(Store_e, along=snapshot, offset=1, edge='wrap') - Store_p
given:
  expressions:
    Bus_injection: { dims: [snapshot, bus] }
expressions:
  Store_injection:
    expression: sum(Store_p, by=Store_bus, over=store, into=bus)
    adds_to: Bus_injection

Add the file to the list:

spec = ms.merge(['network.yaml', 'generator.yaml', 'load.yaml', 'store.yaml'])

Bus_injection is Generator_injection + Load_injection + Store_injection. A merged spec takes a further term the same way, so ms.merge([ms.merge(['network.yaml', 'generator.yaml', 'load.yaml']), 'store.yaml']) gives the same sum.

A library can also couple its components through a flow variable per port, which each component pins at its own port. A component library is written that way.

The entry	What happens
a dimension or a relation	every fragment may declare it, and the ones that do say the same thing about it
a `description` on a shared dimension or relation	it is prose rather than a claim, and the first wording in the list is carried
any other declaration	one fragment declares it, and a second is refused
an entry under `given:`	it is checked against the fragment that introduces the name, then folded into it. Its description fills the declaration where the introducer wrote none
a given expression	the definition's body carries no dimension the reader's `dims` do not name
a given entry no fragment introduces	it stays under `given:` until a host model provides it
an expression with `adds_to:`	it adds this expression as a term to the sum it names. The name becomes its definition, where one fragment writes one, followed by every term by its name, in the order of the list. A definition written as `cases:` takes no term, and terms on a name no fragment defines that only their own files read are refused
`objective`	the terms are summed in the order of the list, each in parentheses, and the senses agree. The first description in the list is carried
`version`	every fragment is written against the same one
`description` at the top of a fragment	it is about the fragment and is not carried. Pass the composed spec's as `description=`

A name two fragments declare#

Fragments own their math, so a name two of them declare is refused, both named. Here two files each say what a generator fleet is:

fragments 'gas.yaml' and 'coal.yaml' both declare the parameter 'Generator_p_nom'. Two of the same kind of thing are two rows of a dimension rather than two fragments: merge the fragment once, and let the data carry both. Different math under one spelling is a rename: call one of them something else.

A term is a named expression like any other, so the terms of two fragments need two names. Name each term after its component, such as Generator_injection and Load_injection.

A column read one way and introduced another#

What a fragment states about a column it reads has to agree with the fragment that introduces the column. The reader may say less, such as the frame with no domain, and may not say something else. Here a file that caps emissions reads Generator_p as binary:

emissions.yaml

dimensions:
  snapshot: { dtype: int }
  generator: { dtype: str }
given:
  variables:
    Generator_p: { dims: [snapshot, generator], domain: binary }
parameters:
  Generator_co2: { dims: [generator] }
  co2_cap: { dims: [] }
constraints:
  co2_limit:
    dims: []
    expression: sum(Generator_p * Generator_co2) <= co2_cap

fragment 'emissions.yaml' reads the given variable 'Generator_p' as {'dims': ['snapshot', 'generator'], 'domain': 'binary'}, where 'generator.yaml' introduces it as {'dims': ['snapshot', 'generator'], 'bounds': {'lower': 0.0, 'upper': 'Generator_p_nom'}, 'domain': 'continuous', 'absence': 'undefined'}. A given declaration says the same as the declaration it is folded into, or less: restate the frame as the introducer declares it, or leave the field out.

Two fragments that both only read a column have to read it the same way, and a difference is refused as it is for a dimension.

A fragment that does not load on its own#

A fragment is a whole spec, so merge loads each one before it composes them. A fragment to_spec refuses is refused under its own name, with the refusal to_spec gives. A sibling cannot make it load. Here the generator declares Generator_p and reads it under given: as well:

fragment 'generator.yaml' does not load on its own. A fragment is a whole spec: it declares what it builds, and reads what a sibling builds under 'given:'.
Given variable 'Generator_p' collides with the variable of the same name. Names share one flat namespace — rename one of them.

A base and its patches#

Write the base as a spec, and each patch as the change it makes. A patch names only the fields it changes. A declaration a patch does not name stays as the base wrote it. A named expression the patch writes on one line replaces the body, expression: or cases:, and keeps the other fields, such as adds_to:. The base loads on its own, and override loads it first. A patch is not a spec, so it is laid over as written, and the patched spec is loaded after.

base.yaml

dimensions:
  snapshot: { dtype: int }
  generator: { dtype: str }
parameters:
  capacity: { dims: [generator] }
  cost: { dims: [generator] }
  load: { dims: [snapshot] }
variables:
  dispatch: { dims: [snapshot, generator], bounds: { lower: 0, upper: capacity } }
constraints:
  power_balance:
    dims: [snapshot]
    expression: sum(dispatch, over=generator) == load
objective:
  sense: minimize
  expression: sum(dispatch * cost)

operate.yaml

variables:
  dispatch: { where: "capacity > 0" }

carbon.yaml

parameters:
  emission_rate: { dims: [generator] }
constraints:
  emission_cap:
    dims: []
    expression: sum(dispatch * emission_rate) <= 1000

Lay the patches on the base. Give them as a list. A refusal names a patch by its path, as merge names a fragment.

import mathspec as ms

spec = ms.override('base.yaml', ['carbon.yaml', 'operate.yaml'])

spec declares emission_cap beside power_balance, and dispatch carries the mask capacity > 0.

Remove a declaration with null. A patch that does not mention a declaration leaves it alone, so removal needs a marker of its own.

feasibility.yaml

constraints:
  emission_cap: null
objective: null

A null makes what it names absent. On a declaration, the declaration is removed. On a field, the field takes its default and the declaration stays: dispatch: { where: null } gives that variable no mask, and dispatch: { bounds: { upper: null } } leaves it open above. Higher up, constraints: null is refused, because a section is not a declaration and nulling it removes nothing.

Put a patch that refines another after it. override lays the patches in the order of the list, each on the result of the ones before. Where two patches write one field, the later one wins. A later patch may also edit or remove a declaration an earlier one creates.

spec = ms.override('base.yaml', ['pathway.yaml', 'project.yaml'])

What a patch may say#

The entry	What happens
some fields of a declaration	those fields change, and the rest of the declaration stays
a whole declaration under a new name	it is added
`null` under a declaration's name	it is removed
`null` on a field of a declaration	the field takes its default, and the rest of the declaration stays
`null` under a section's name	it is refused
a dimension or a relation	it is added, or restated word for word as the base declares it
an entry under one kind of `given:`	it is edited, added or removed like any declaration, and the other kinds stay
`version`, `description`	the patch's value replaces the base's
a field an earlier patch writes	the later patch's value replaces it

A partial entry on a missing name#

An entry naming some fields has to land on a declaration the base or an earlier patch has. A mistyped name is refused rather than read as a new declaration:

patch 'project.yaml' edits the constraint 'power_balnce', which its base does not declare. Did you mean 'power_balance'? A patch creates a declaration only by writing it whole, and this one is not: a constraint needs `expression`.

To add a constraint, write the whole constraint. To change one, spell its name as the base spells it.

A dimension redeclared#

A patch may add a dimension or a relation, and may restate one the base declares. The restatement is word for word: half a declaration is a second reading of the same name. Changing one under the expressions already written over it is refused, and so is removing one:

patch 'relabelled.yaml' declares the dimension 'snapshot' as {'dtype': 'str'}, where its base declares {'dtype': 'int'}. A patch adjusts the math, not the coordinate space the math is already written over: restate the declaration word for word, leave it out, or give the patch a dimension of its own under a name of its own.

A section set to `null`#

A null removes the declaration it names. A section holds declarations rather than being one, so nulling a section is refused rather than read as emptying it:

patch 'project.yaml' sets 'constraints' to null, which removes nothing: the removal marker names one declaration, and a section is not one. Remove the declarations one at a time, each under its own name, or leave the section out of the patch.

A stale removal#

A removal says what the base has, so a removal of a declaration the base does not have is refused with the near miss:

patch 'stale.yaml' removes the constraint 'power_balnce', which its base does not declare. A removal is a claim about what is there, so a stale one is a patch that no longer describes the spec it lands on. Did you mean 'power_balance'?