Restructure the scheduler #17

Open
opened 6 years ago by novirium · 1 comments
Owner

At the moment the dependency on APScheduler is primarily for cron-string parsing, it's intended use case (high-throughput server jobs) doesn't really fit our situation (for instance, we need to recreate a file-based job storage, which was actually removed from earlier APScheduler versions due to its low performance).

Probably move to an approach that uses a smaller cron parser library, and set up our own thread/job creation logic. Jobs would have a proper data structure (rather than parsing a string), and would require an hours and minutes field. Seconds would default to "0", and all larger time periods (days, weeks, months, etc) would default to "*". Methods for adding a job would accept these as keywords, or a simple dict with these as keys (to allow direct passing from a config).

A convenience config specification for a job cron-style trigger could be made available somewhere to avoid repeating that all over the place.

Jobs would need a way to define whether they should trigger a wakeup or not from low power. Blocking (stopping lowpower shutdown) should still be a context manager (with) I think, to reduce chance of things holding everything on, though I guess we could also provide that as an option to apply to a whole job.

Would need to replicate some of the APScheduler logic - logging when a job is triggered but is still running from last time, having a grace period to still trigger even if boot from low power took longer than expected, etc. Probably don't need the whole thread pool thing though (shouldn't have many consecutive jobs).

Config for the scheduler should probably be it's own specify_config dict that then gets incorporated into the core config as a dict - so [shepherd.scheduler] in the TOML.

Will need a way to use the same job system to just run a plain thread straight away - to make it easy to run stuff in a plugin indefinitely as soon as setup is finished and other plugins are instanced. "add_job_now", "main_job"?

At the moment the dependency on APScheduler is primarily for cron-string parsing, it's intended use case (high-throughput server jobs) doesn't really fit our situation (for instance, we need to recreate a file-based job storage, which was actually removed from earlier APScheduler versions due to its low performance). Probably move to an approach that uses a smaller cron parser library, and set up our own thread/job creation logic. Jobs would have a proper data structure (rather than parsing a string), and would require an hours and minutes field. Seconds would default to "0", and all larger time periods (days, weeks, months, etc) would default to "*". Methods for adding a job would accept these as keywords, or a simple dict with these as keys (to allow direct passing from a config). A convenience config specification for a job cron-style trigger could be made available somewhere to avoid repeating that all over the place. Jobs would need a way to define whether they should trigger a wakeup or not from low power. Blocking (stopping lowpower shutdown) should still be a context manager (with) I think, to reduce chance of things holding everything on, though I guess we could also provide that as an option to apply to a whole job. Would need to replicate some of the APScheduler logic - logging when a job is triggered but is still running from last time, having a grace period to still trigger even if boot from low power took longer than expected, etc. Probably don't need the whole thread pool thing though (shouldn't have many consecutive jobs). Config for the scheduler should probably be it's own `specify_config` dict that then gets incorporated into the core config as a dict - so `[shepherd.scheduler]` in the TOML. Will need a way to use the same job system to just run a plain thread straight away - to make it easy to run stuff in a plugin indefinitely as soon as setup is finished and other plugins are instanced. "add_job_now", "main_job"?
Poster
Owner

Also potentially change the terminology from "job" to "task"

Also potentially change the terminology from "job" to "task"
novirium added the [status] Review [type] Enhancement labels 6 years ago
Sign in to join this conversation.
Loading…
There is no content yet.