Skip to content

feat(persistent): in-place solver updates (Solver framework + HiGHS/Gurobi/Xpress/Mosek)#718

Open
FabianHofmann wants to merge 22 commits into
masterfrom
feat/solver-update
Open

feat(persistent): in-place solver updates (Solver framework + HiGHS/Gurobi/Xpress/Mosek)#718
FabianHofmann wants to merge 22 commits into
masterfrom
feat/solver-update

Conversation

@FabianHofmann
Copy link
Copy Markdown
Collaborator

@FabianHofmann FabianHofmann commented May 21, 2026

Changes proposed in this Pull Request

Adds an opt-in persistent-update framework so a built solver can be re-solved against a mutated Model without a full rebuild.

Core

  • linopy.persistent: ModelSnapshot, ModelDiff, StructuralKey, VarKind, RebuildReason. ModelDiff stores changes in flat-native arrays (bounds / var-types / RHS / signs / COO coefs / linear objective / sense) plus per-container VarSlice / ConSlice views.
  • ModelDiff.from_snapshot(snap, model) and ModelDiff.from_models(m1, m2) — snapshot-based and snapshot-free diffs.
  • _coef_dirty flag on constraints with RHS-setter short-circuit so RHS-only edits skip the coefficient re-walk.

Solver orchestration

  • Solver gains track_updates, lazy-build (first solve(model, …) builds), apply_update, update, disallow_rebuild, and structured rebuild reasons. Backends without persistent-update support short-circuit to rebuild.
  • Per-backend apply_update:
    • HiGHS — changeColsBounds / changeColsIntegrality / changeRowBounds / changeCoeff / changeColsCost / changeObjectiveSense. Sign change → rebuild.
    • Gurobi — setAttr(LB/UB/VType/RHS/Sense/Obj), chgCoeff, ModelSense. In-place sign change.
    • Xpress — chgbounds / chgcoltype / chgrhs / chgrowtype / chgmcoef / chgobj / chgobjsense. In-place sign change.
    • Mosek — chgvarbound / chgconbound / putvartypelist / putaijlist / putclist / putobjsense. Sign change → rebuild.

Tests

  • New test_persistent_snapshot_diff.py covering all ModelDiff semantics.
  • New parametrized test_persistent_apply_update.py running 9 cases × 4 backends (skipped per backend when license/installation is unavailable).
  • Cross-instance, pickle, and threading coverage in test_persistent_solver_extras.py.

Checklist

  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
  • Unit tests for new features were added (if applicable).
  • A note for the release notes doc/release_notes.rst of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

FabianHofmann and others added 19 commits May 19, 2026 15:19
@FabianHofmann
feat(constraints): add _coef_dirty flag + rhs setter short-circuit
27a3129 
Tracks per-Constraint coefficient mutation via a single boolean slot,
flipped in coeffs/vars/lhs setters. Pure-constant rhs writes now
short-circuit and leave coeffs/vars buffers untouched (by identity),
so rhs-only updates don't trigger expensive coefficient recompare on
the persistent-solver fast path.
@FabianHofmann
feat(persistent): add ModelSnapshot, CoefPattern, StructuralKey
663ab92 
Pure-Python snapshot primitives for the persistent-solver Phase 1.
Deep-copies value-side fields (var_lb/ub, con_rhs/sign, obj_linear),
holds vlabels/clabels by reference, stores canonical CSR
(indptr, indices) per constraint container. No Solver import.
@FabianHofmann
feat(persistent): add ModelDiff and compute_diff
c1da075 
Pure-function diff for the persistent-solver Phase 1. Detects
structural, coord, sparsity, quadratic-objective, value-only var/con,
and objective-linear/sense changes. Supports same_model fast path
via _coef_dirty and cross-model full re-scan. Includes a focused
test suite covering capture, mutation paths, deep-copy invariant,
and the same_model toggle.
@FabianHofmann
feat(solvers): add persistent-update orchestration to Solver
572b426 
- supports_persistent_update class flag (default False)
- snapshot/_rebuilds/_in_place_updates/_last_rebuild_reason fields
- snapshot capture at end of direct _build, _clear_coef_dirty helper
- apply_update stub raising UnsupportedUpdate
- solve(model, assign) dispatcher with diff-or-rebuild path
- update(model, apply=True) primitive returning ModelDiff
- threading.Lock around diff+apply+resnapshot
- __getstate__/__setstate__ drop native handle and snapshot
@FabianHofmann
test(persistent): smoke test Solver orchestrator with fake backend
61b3647
@FabianHofmann
feat(solvers): short-circuit rebuild when backend lacks persistent-up…
ff1ae15 
…date support

Skip diff computation entirely when supports_persistent_update is False
on apply, per plan: 'dispatcher checks flag before calling — if False,
skips diffing entirely and goes to rebuild.'
@FabianHofmann
feat(solvers): Gurobi apply_update for persistent solves
f67cec6
@FabianHofmann
feat(solvers): HiGHS apply_update for persistent solves
8026293
@FabianHofmann
test(persistent): cross-model, pickle, threading, failure-path coverage
b6601e3
@FabianHofmann
refactor(persistent): row-block numpy snapshot/diff
6922c7d 
Replace xarray-based snapshot and CSR pattern compare with per-row
canonicalised numpy buffers; new ContainerVarUpdate / ContainerRowUpdate
payloads. Gurobi/HiGHS apply_update rewritten around batched setAttr /
changeColsBounds / changeColsCost / changeColsIntegrality; coefficient
writes touch only changed cells. Cross-model diff now ~matches same-model
cost for bound/rhs/coef-value sweeps.
@FabianHofmann
feat(persistent): opt-in coord-equality via ignore_dims
0c30688 
compute_diff/Solver.solve/Solver.update grow an ignore_dims kwarg.
None (default) keeps the current no-coord-check behaviour;
any iterable opts into per-container coord-equality on every dim
not in the set, supporting rolling-horizon workflows where e.g.
the snapshot dim is expected to drift.
@FabianHofmann
feat(persistent): lazy-build Solver, ModelDiff constructors, disallow…
8dbb8be 
…_rebuild

- Solver.from_name now accepts model=None; the first solve(m, ...) builds.
- compute_diff folded into ModelDiff.from_snapshot classmethod; new
  ModelDiff.from_models diffs two linopy models directly.
- Solver.solve grows disallow_rebuild=True, which raises
  RebuildRequiredError instead of falling back to a rebuild.
@FabianHofmann
feat(persistent): opt-in update tracking, snapshot-free ModelDiff.fro…
595bab0 
…m_models

- Add `track_updates` flag (default False) to Solver; skip ModelSnapshot
  capture when disabled. Raise UpdatesDisabledError on solve(model)/update()
  if a built solver was constructed without tracking.
- Rewrite ModelDiff.from_models to build directly from two models without
  capturing snapshots; share helpers with from_snapshot.
- Update persistent tests to opt into track_updates=True; add coverage
  for the disabled path.
@FabianHofmann
feat(persistent): wire ModelDiff.from_models for track_updates=False
6707917 
Cross-instance resolves now diff via from_models against the previously
built model, with no snapshot. Same-instance mutation still raises
UpdatesDisabledError. Snapshot recapture is skipped in this mode.
Add cross-instance solve/update tests for the no-snapshot path.
@FabianHofmann
refactor(persistent): cleanups, VarKind enum, fold _clear_coef_dirty
2ae0bef 
Collapse _diff_objective QUAD_OBJ branches; cache n_coef_updates;
short-circuit _canonicalize_rows when rows already sorted; tighten
buffer extraction. Introduce VarKind enum used across snapshot/diff
and HiGHS/Gurobi apply_update; reuse linopy.constants sign tokens.
Move _clear_coef_dirty into ModelSnapshot.capture.
@FabianHofmann
refactor(persistent): CSR-backed ContainerConBuffers
33fc991 
Source con buffers from Constraint.to_matrix_with_rhs, replacing the
dense (n_rows, max_n_term) arrays with CSR (indptr, indices, data).
Sign dtype adopts 'U1' across the persistent layer and apply_update
in HiGHS/Gurobi consumes CSR-slice payloads instead of -1 masks.
Deletes _canonicalize_rows and the _INT64_MAX sentinel.
@FabianHofmann
refactor(persistent): flat-native ModelDiff storage
3bffb6e 
Replace per-container ContainerVarUpdate/ContainerRowUpdate dicts with
flat arrays (var_bounds_*, var_type_*, con_coef_* COO, con_rhs_*,
con_sign_*) plus VarSlice/ConSlice per-container offsets for
diagnostics. Add con_rhs_as_bounds() for ranged-row solvers. Backend
apply_update bodies collapse to flat-array calls; remove duplicated
label->position resolution.
@FabianHofmann
feat(persistent): apply_update for Xpress and Mosek
4995e8b 
Implement in-place model updates for Xpress (chgbounds/chgrhs/chgmcoef/
chgrowtype/chgobj/chgobjsense/chgcoltype) and Mosek (chgvarbound/
chgconbound/putaijlist/putclist/putvartypelist/putobjsense). Mosek
rejects constraint sign change to trigger rebuild. Consolidate
gurobi/highs apply_update tests into a single parametrized file that
also covers xpress and mosek.
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
5c0a369 
for more information, see https://pre-commit.ci
@FabianHofmann FabianHofmann requested review from FBumann and coroa May 21, 2026 15:21
@FabianHofmann
Copy link
Copy Markdown
Collaborator Author

FabianHofmann commented May 21, 2026

@FBumann here we go, if you want to take a first look. docs to come

@FBumann
Copy link
Copy Markdown
Collaborator

FBumann commented May 22, 2026

@FBumann here we go, if you want to take a first look. docs to come

Ill have a look ltoday

@FabianHofmann
fix(persistent): serialize concurrent solves; satisfy mypy
9fd88de 
* hold solver lock through _run_direct so two threads calling
  solve(model) on the same Solver no longer race on the native handle
  (HiGHS returned 0.0 from the second concurrent solve).
* narrow Optional ndarrays in persistent.diff.push_var / push_con and
  in HiGHS/Gurobi/Xpress/Mosek apply_update objective paths.
* widen Constraint.rhs setter to ExpressionLike | VariableLike |
  ConstantLike to match the as_expression call in the body.
* widen Constraints.__getitem__(str) return type to Constraint (the
  dominant case) so tests can set .rhs/.coeffs/.sign without ignores.
* add docs for in-place solver updates.
@FBumann
Copy link
Copy Markdown
Collaborator

FBumann commented May 22, 2026

@FabianHofmann Sorry, I wont be able to review this today.

@FabianHofmann
Copy link
Copy Markdown
Collaborator Author

FabianHofmann commented May 22, 2026

@FabianHofmann Sorry, I wont be able to review this today.

take your time, there is no hurry. I'll do some integration tests anyway

@FBumann
Copy link
Copy Markdown
Collaborator

FBumann commented May 22, 2026

Are there any conflicts with #717? Are we sure we want to merge and publish this before we go v1?

Just making sure...

@MaykThewessen
Copy link
Copy Markdown
Contributor

MaykThewessen commented May 23, 2026 •
edited
Loading

Benched PR #718 ModelDiff vs hand-rolled changeRowsBounds path on a synthetic 69K-var / 160K-row DC-OPF, 8 weekly-style chunks (RHS + objective mutations only, identical structure across chunks). Per-chunk averages, chunk 0 excluded:

route t_build t_diff t_apply t_solve t_total in_place rebuilds
blueprint + changeRowsBounds (hand-rolled) 0.000 0.001 0.006 0.759 0.766 0 0
ModelDiff sweep (fresh Model per chunk) 0.222 0.012 0.039 12.365 12.652 7 0
ModelDiff in-place (mutate model.constraints[name].rhs.values[...]) 0.003 0.010 0.041 4.919 4.982 7 0
ModelDiff in-place + manual solver.solver_model.clearSolver() 0.003 0.010 0.041 0.572 0.637 7 0

Diff compute + apply_update sum to ~50 ms — those are not the bottleneck. The regression is entirely in t_solve: after in-place RHS mutation the persistent path keeps HiGHS's prior basis, and HiGHS then skips presolve ("Solving LP with useful basis so presolve not used"). For LPs with strong presolve reduction (89 % row reduction on our production model, similar on this synthetic) the skipped presolve costs more than the warm basis saves.

Manually calling solver.solver_model.clearSolver() between solver.update() and solver.solve() recovers presolve and brings the persistent path slightly under the hand-rolled baseline. But that defeats the encapsulation — the user reaches past the Solver API to touch the native HiGHS handle.

Suggestion: expose an opt-in on the persistent API so users with presolve-heavy LPs can drop the warm basis without bypassing the encapsulation. Either:

  • a keep_basis: bool = True kwarg on Solver.solve() / Solver.update(), or
  • a Solver.clear_basis() method on the base class with backend-specific implementations (Highs.clearSolver(), Gurobi reset(), etc.).

Zero structural rebuilds observed across all 7 re-solves in both ModelDiff routes (RebuildReason.NONE). The diff correctly classifies these as RHS + coefficient-only changes — the persistent API is doing its job, the missing piece is just the basis-clear hook.

Caveats:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coroa coroa Awaiting requested review from coroa

@FBumann FBumann Awaiting requested review from FBumann

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@FabianHofmann @FBumann @MaykThewessen