Improve scheduler actor by TheHeroBrine422 · Pull Request #63 · FyraStack/odorobo

TheHeroBrine422 · 2026-07-10T06:38:40Z

intended changes for this PR

improve actor cache
- currently the actor cache is setup in a way that doesn't really support the scheduler updating it when messages happen. this makes the cache very difficult to use because it can't be relied on for up to date information which means we can barely use it as a cache. a specific issue i hit with this was trying to implement affinity/placement groups. if we rely on the cache data for calculating affinity, we could break the affinity rules due to race conditions.
- i am replacing the generic version I made with one that is just integrated into the scheduler directly, because otherwise I don't really feel comfortable doing direct modifications to the cache in other segments of the scheduler
- on top of this, the vm cache arguably needs slightly different code because we basically need the ability to query by either actor id or vmid, and so it needed some sort of modification to support this either way.
- todo: have messages sent to scheduler directly update the cache
Placement groups #28
improve scheduling algorithm
- i realized that currently the scheduling algorithm doesnt consider the resource requirements of the new vm. so for example if an agent had 15/16 vcpus scheduled, and then a new VM was to be scheduled that needed another 16 vcpus, it could theoretically schedule it to that agent. which would be bad

All that is left is testing and review by another maintainer.

this was done for two reasons: 1) the vm and agent actor caches have slightly different requirements, specifically around keys (actorid vs vmid) 2) the scheduler needs direct control of the cache so the cache can be updated in response to messages it forwards. being possibly up to 1 second out of date in this data could cause problems. In theory the scheduler could update the generic cache, but I don't think using a generic cache if we need to modify it directly anyway really makes sense. Signed-off-by: Caleb Jones <caleb@calebgj.io>

Signed-off-by: Caleb Jones <caleb@calebgj.io>

scheduler affinity Signed-off-by: Caleb Jones <caleb@calebgj.io>

…k/odorobo into caleb/fix-scheduler-actor-cache

Signed-off-by: Caleb Jones <caleb@calebgj.io>

github-advanced-security

clippy found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

this also fixes an issue where if a VM is never linked (ex: it didnt boot) it would never be removed from the caches.

…ctor-cache

halfcyan · 2026-07-16T23:39:26Z

i will look at this when i get back home. have you run it through clippy?

halfcyan · 2026-07-17T02:14:07Z

-    async fn on_update(
+    // this function intentionally only checks against the cache. this has some positives and negatives:
+    // positive: it will never trigger any network requests so its very fast, and having to do network requests for scoring whenever we want to schedule a vm is likely a bad idea
+    // negative: it technically has a delayed view of the cluster, meaning that some things that happened in the future, may not exist yet. so we need to be careful about how this is done so affinity rules are not accidentally broken. mostly this means, if we do anything that could affect the outcome of an affinity rule (ex: network request to an agent), we need to update the cache, before we do the action.


assuming anything later in the cache is referencing anything that changed as a result of this, i think it's fine. so like if you have multiple VM creations get scheduled, as long as each subsequent one references the one before it.

korewaChino · 2026-07-20T07:42:59Z

                    max_vcpus: None,
                    memory: ByteSize::gib(4),
-                    image: "/var/lib/odorobo/f43.raw".to_string(),
+                    image: "/var/lib/odorobo/f43-1.raw".to_string(),


why are we still hardcoding this btw

halfcyan · 2026-07-31T20:20:16Z

+                    ?actor_ref,
+                    "can no longer reach agent actor, stopping updater"
+                );
+                data_cache.remove(&actor_ref.id());


when the updater gives up after six failed polls, it removes the actor from data_cache and returns, but its completed JoinHandle remains in keepalive_tasks. the finder uses contains_key to decide whether an updater exists, so it will see the stale handle and never restart polling when the actor becomes reachable again. that can permanently remove a temporarily disconnected agent/VM from scheduling.

probably remove the handle when the updater exits, or have the finder treat finished handles as stale (e.g. check JoinHandle::is_finished() and replace the entry). same issue applies to both agent_updater_task and vm_updater_task.

halfcyan · 2026-07-31T20:22:19Z

+            if let Ok(data) = actor_ref.ask(&GetAgentStatus).await {
+                if data_cache.contains_key(&actor_ref.id()) {
+                    data_cache.alter(&actor_ref.id(), |_, mut v| {
+                        v.data = data;
+
+                        v.extended_vm_set.extend(v.data.vms.iter());
+
+                        v
+                    });


when an agent update succeeds, this only extends the existing set. it doesn't remove VM IDs that disappeared from AgentStatus.vms. affinity scheduling can then treat a VM like it's still on an agent after shutdown, deletion, migration, etc.

halfcyan · 2026-07-31T20:24:51Z

+        self.vm_data_cache.insert(
+            msg.vmid,
+            CachedVMActor {
+                actor_ref: None,
+                scheduled_agent_id: target_agent.id(),
+                data: GetVMInfoReply {
+                    vmid: msg.vmid,
+                    config: Some(msg.config.clone()),
+                },
+            },
+        );


the placeholder VM that gets inserted here can leak if the actor dies before it has a successful update. basically, if the actor dies before sending GetVMInfo to vm_updater_task, there's no vm_actorid_ulid_map entry. because of that, on_link_died skips cleanup.

also see lines 626-640

halfcyan · 2026-07-31T20:27:08Z

+        let reply = target_agent.ask(&msg).await;
+
+        // remove from caches if the agent rejected the request
+        if reply.is_err() {


this treats all act errors as a rejected create, but not all of them are (reply may be lost as an example)

halfcyan · 2026-07-31T20:27:54Z

+
+                vm_actorid_ulid_map.insert(actor_ref.id(), vmid); // should we be doing this on every loop? idk. but we at least need to do it on the first iteration given we don't know the mapping before that
+
+                data_cache.insert(


vm_data_cache is keyed only by vmid, but migration can temporarily have two VM actors with the same VM ID. the later updater overwrites the earlier cache entry. if the old actor then dies, on_link_died can remove the cache entry belonging to the new actor and remove the VM from the wrong agent’s extended_vm_set

halfcyan · 2026-07-31T20:28:53Z

-// todo: this code is really bad, and we should not have effectively two copies of ths same thing.
-#[derive(Copy, Clone)]
-struct VMActorCacheUpdater;
+fn evaluate_table_value(value_option: Option<&String>, requirement: &AffinityRequirement) -> bool {


can you add some unit tests for the new affinity logic? in particular, tests for In, NotIn, Lt, and Gt, missing metadata keys, inverse rules, empty requirements, required versus preferred rules, and capacity rejection would make these semantics explicit and prevent regressions. the scheduler currently has no tests covering this new behavior

TheHeroBrine422 added 13 commits May 1, 2026 12:04

improve scheduling algorithm to use estimated future values.

09c41f0

Signed-off-by: Caleb Jones <caleb@calebgj.io>

committing before lunch so i dont have code just sitting on my machine

d0dd375

Signed-off-by: Caleb Jones <caleb@calebgj.io>

continue to work on scheduler actor caches and placement groups

4449502

Signed-off-by: Caleb Jones <caleb@calebgj.io>

continue to work on scheduler actor (still not finished)

7a63518

Signed-off-by: Caleb Jones <caleb@calebgj.io>

more work on scheduler affinity stuff. still not finished.

11186cd

Signed-off-by: Caleb Jones <caleb@calebgj.io>

theoretically write scheduler affinity scoring. need to test this.

2202c17

Signed-off-by: Caleb Jones <caleb@calebgj.io>

remove ipam actor

fedb58b

Signed-off-by: Caleb Jones <caleb@calebgj.io>

Merge branch 'main' into caleb/fix-scheduler-actor-cache

e26b7d7

fix bug with getting all agent and vm actors in actor finders

29c2a84

Signed-off-by: Caleb Jones <caleb@calebgj.io>

Merge branch 'main' into caleb/fix-scheduler-actor-cache

99e313f

commiting stuff before cypress rebase. was in the middle of testing

2fde501

scheduler affinity Signed-off-by: Caleb Jones <caleb@calebgj.io>

Merge branch 'caleb/fix-scheduler-actor-cache' of github.com:FyraStac…

da311a3

…k/odorobo into caleb/fix-scheduler-actor-cache

github-advanced-security AI found potential problems Jul 10, 2026

View reviewed changes

TheHeroBrine422 added 2 commits July 10, 2026 02:14

Merge branch 'main' into caleb/fix-scheduler-actor-cache

5274b03

Signed-off-by: Caleb Jones <caleb@calebgj.io>

cargo fmt

674c5a9

Signed-off-by: Caleb Jones <caleb@calebgj.io>

github-advanced-security AI found potential problems Jul 10, 2026

View reviewed changes

TheHeroBrine422 added 2 commits July 10, 2026 02:22

fix bugs created by merge commit

06c7885

cargo fmt

2f1ad4e

github-advanced-security AI found potential problems Jul 10, 2026

View reviewed changes

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

TheHeroBrine422 added 4 commits July 11, 2026 03:13

random debugging stuff

1561884

Merge branch 'main' into caleb/fix-scheduler-actor-cache

a575032

fix: remove unused AffinityDirection enum

188a2b9

feat: remove VMs from actors if unreachacble by updaters.

43b6f56

this also fixes an issue where if a VM is never linked (ex: it didnt boot) it would never be removed from the caches.

github-advanced-security AI found potential problems Jul 16, 2026

View reviewed changes

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

TheHeroBrine422 added 2 commits July 16, 2026 05:37

chore: cargo fmt

a784bab

Merge remote-tracking branch 'origin/main' into caleb/fix-scheduler-a…

e42ee50

…ctor-cache

TheHeroBrine422 changed the title ~~Caleb/fix scheduler actor cache~~ Improve scheduler actor Jul 16, 2026

github-advanced-security AI found potential problems Jul 16, 2026

View reviewed changes

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

fix: better extended_vm_set removal and setup

81d5b9c

github-advanced-security AI found potential problems Jul 16, 2026

View reviewed changes

Comment thread odorobo/src/actors/scheduler_actor.rs Fixed

chore: clippy and fmt

ecbc2e9

TheHeroBrine422 marked this pull request as ready for review July 16, 2026 20:23

TheHeroBrine422 requested review from halfcyan and korewaChino July 16, 2026 20:23

halfcyan previously approved these changes Jul 17, 2026

View reviewed changes

korewaChino previously approved these changes Jul 20, 2026

View reviewed changes

resolve conflicts

2ecbae0

halfcyan dismissed stale reviews from korewaChino and themself via 2ecbae0 July 31, 2026 06:31

fix clippy warnings

7698f8f

halfcyan requested changes Jul 31, 2026

View reviewed changes

halfcyan added 2 commits July 31, 2026 14:44

Merge branch 'main' into caleb/fix-scheduler-actor-cache

f8473e2

Merge branch 'main' into caleb/fix-scheduler-actor-cache

53fcfad


		vm_actorid_ulid_map.insert(actor_ref.id(), vmid); // should we be doing this on every loop? idk. but we at least need to do it on the first iteration given we don't know the mapping before that

		data_cache.insert(

Uh oh!

Conversation

TheHeroBrine422 commented Jul 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-advanced-security AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

halfcyan commented Jul 16, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants