A priority-queue job scheduler with DAG workflows, retry backoff, a dead letter queue, and real-time updates — no Celery, no external broker.
Flint is a background job scheduler built from scratch — no Celery, no RQ, no managed queue. It takes jobs over a REST API, queues them in an in-memory min-heap by priority and schedule, runs them through async worker coroutines, and tracks every state transition in PostgreSQL. It's for engineers who want to see what's actually happening underneath a job queue abstraction, not use one blind.
When a job comes in, the scheduler polls PostgreSQL every second for due jobs and pushes them into a shared heap. Worker coroutines pop from that heap, claim the job atomically in Postgres, and run it. Jobs can depend on other jobs, recur on an interval, retry with backoff, or land in a dead letter queue after exhausting retries. Redis handles exactly two things — SSE pub/sub and worker heartbeats — it is not the queue.
The database claim was the moment this project stopped feeling like a toy. I went in assuming correctness under concurrency would require some kind of external lock — Redis, a semaphore, something explicit. It didn't. Postgres row-level locking on a plain `UPDATE ... WHERE` was already sufficient, and I was about to add a second distributed system to solve a problem the first one already solved. That's a default now: check what your source of truth can already guarantee before reaching for a new coordination layer.
Cooperative cancellation also changed how I think about "stopping" something. My first instinct was to just kill the task. Writing the checkpoint-based version forced me to actually enumerate what state a running job can leave behind and accept that some of it is permanent. Cancellation isn't a hard stop — it's a negotiation with whatever's already in flight.