molq

Unified job queue — one submission API for local, SLURM, PBS, and LSF

Documentation  ·  Quick start  ·  Ecosystem

molq is a unified job queue for Python workloads that need the same submission API on a laptop, a workstation, or an HPC cluster. A Cluster says where jobs run, a Submitor tracks how they progress — and the same code runs against local subprocesses or remote schedulers over SSH.

Under active development. Public APIs may change between minor releases.

Capabilities

Module	Capability
cluster	Cluster — destination spec: scheduler kind × transport × scheduler options, plus live queue snapshots
submitor	Submitor + JobHandle — single entry point for submitting, tracking, and waiting on jobs
scheduler	Scheduler protocol with Shell, Slurm, PBS, and LSF backends, each routing shell calls through a transport
transport	LocalTransport / SshTransport — runs shell and file ops here or on a remote host via OpenSSH
store	JobStore — SQLite persistence with WAL mode, UUID job identity, schema versioning, and v1 auto-migration
reconciler	JobReconciler — batch-queries schedulers, diffs against the store, syncs job state
monitor	Blocking waits and polling engine driven by pluggable strategies
strategies	Pluggable polling strategies; exponential backoff by default
callbacks	EventBus — synchronous pub/sub for job lifecycle events with handler isolation
models	Job data models — JobRecord, RetryPolicy, RetentionPolicy, JobDependency, SubmitorDefaults
types	Frozen value types — Memory, Duration, Script, JobResources, JobScheduling, JobExecution
options	Per-scheduler frozen option dataclasses (Local, Slurm, PBS, LSF) — no untyped dicts
config	Profile and config loading from molcfg (~/.molcrafts/molq/config/config.toml, or MOLCRAFTS_HOME)
plugin	MolqPlugin host — official builtins + third-party entry points (molq.plugins)
plugins	Official plugins (e.g. nerve → local Nerve menu-bar status; fail-open)
workspace	Workspace / Project — directory handles over a cluster's filesystem (local or remote)
ssh_config	Surfaces ~/.ssh/config hosts as cluster candidates
serde	Serialization helpers for stored requests and config-driven values
errors	Unified MolqError exception hierarchy with typed context
status	JobState enum with terminal-state semantics
merge	Pure function that merges per-submit parameters with Submitor defaults
dashboard	Full-screen terminal dashboard for monitoring runs and jobs
testing	FakeScheduler and make_submitor for tests and runnable examples without a real cluster
cli	Typer + Rich CLI: jobs (submit/list/status/logs/cancel/…), live (watch/monitor/daemon), setup (clusters/workspace/plugins)

Install

pip install molcrafts-molq

Requires Python 3.12+. Depends on typer, rich, molcrafts-mollog, and molcrafts-molcfg.

Quick start

import molq as mq

# Cluster = destination (where to run). Submitor = lifecycle (how jobs are tracked).
cluster = mq.Cluster("devbox", "local")
submitor = mq.Submitor(target=cluster)

handle = submitor.submit_job(
    argv=["python", "train.py"],
    resources=mq.JobResources(
        cpu_count=4,
        memory=mq.Memory.gb(8),
        time_limit=mq.Duration.hours(2),
    ),
)

record = handle.wait()
print(record.state)

Swap to a cluster by changing one line — mq.Cluster("hpc", "slurm", host="user@hpc.example.com") — and the rest of the code is unchanged. See the docs for retries, dependencies, profiles, and the CLI.

Documentation

• Getting Started — installation and your first job
• Concepts — Cluster, Submitor, Scheduler, Transport, Workspace, Project
• Schedulers — scheduler matrix and option classes
• Monitoring — lifecycle, reconciliation, polling, and dashboards
• CLI Reference — command-line usage
• API Reference — exported classes, enums, options, and errors

MolCrafts ecosystem

Project	Role
molpy	Python toolkit — the shared molecular data model & workflow layer
molrs	Rust core — molecular data structures & compute kernels (native + WASM)
molpack	Packmol-grade molecular packing (Rust + Python)
molvis	WebGL molecular visualization & editing
molexp	Workflow & experiment-management platform
molnex	Molecular machine-learning framework
molq	Unified job queue — local / SLURM / PBS / LSF — this repo
molcfg	Layered configuration library
mollog	Structured logging, stdlib-compatible
molhub	Molecular dataset hub
molmcp	MCP server for the ecosystem
molrec	Atomistic record specification

Contributing

Issues and pull requests are welcome — see the docs for development setup.

License

MIT — see LICENSE.

---

_{Crafted with 💚 by MolCrafts}