A Laravel queue driver for Rostam - at-least-once delivery on the key-value engine, claiming jobs with leases rather than destructively, over the native binary TCP protocol.
rostamlabs/rostam-queue-laravel is a Laravel package for a laravel queue driver for rostam - at-least-once delivery on the key-value engine, claiming jobs with leases rather than destructively, over the native binary tcp protocol..
It currently has 0 GitHub stars and 2 downloads on Packagist (latest version v0.2.0).
Install it with composer require rostamlabs/rostam-queue-laravel.
Discover more Laravel packages by rostamlabs
or browse all Laravel packages to compare alternatives.
Last updated
A Laravel queue driver for Rostam, built on its key-value engine over the native binary TCP protocol.
Read this part before the installation instructions.
At-least-once delivery. A job the queue accepted is delivered, including when the worker holding it dies. A worker that is merely slow — slow enough for its lease to lapse — can have the same job handed to somebody else while it is still running, so your handlers must be idempotent. This is the same guarantee the Redis and database drivers give, for the same reason.
A worker dying while it holds a job counts as an attempt, so a job that crashes
its worker every time reaches maxTries and is failed, instead of taking workers
down forever.
retry_after must exceed your longest job. It is the lease: hold a job longer
than that and the queue assumes you died. Set it too low and work runs twice; set it
too high and a genuinely dead worker's job waits that long to come back.
A queue is only as durable as the node never throwing away a record it still holds.
A single-node rostam-server throws them away in two ways, both measured:
Started with -relocating-eviction, the server copies a page's still-live
records forward instead of dropping them with it. The same churn left both keys in
place, and a first-in-first-out backlog churned eight times over held with no
live evictions at a quarter of the budget — while at half it evicted 1,714 live
records along the way.
It is best-effort by design, and the server says so: it never allocates a page,
never triggers another eviction and never fails a write, so a record that does not
fit the room left over is dropped like any other. That is why the flag is one of
three things this driver asks for, not the whole answer — the live set has to stay
well under the budget, and the eviction count is the backstop. The flag does
nothing under -cluster, which this driver does not support anyway. So:
'at_cap_policy' => 'headroom',
declares the one setup this driver runs on, and the connector refuses to start without it:
rostam-server started with -relocating-eviction;max_memory well above everything live on it — the backlog at its worst, the
delayed jobs, and anything else sharing the server (a quarter held in the
measurement above; half did not);The first two cannot be read off the wire — that part is your declaration. What
can be read is whether the node has already evicted live records
(rostam_kv_evictions_live_total). The driver reads it before the first job is
accepted or taken and again every verify_every seconds, and refuses to run on a
node that has evicted any. Be clear about what that is:
on_evictions => 'refuse' means, and it is the default;
'ignore' skips the check entirely — see the configuration below for when that
is a reasonable trade and what it costs.A -cluster refuses writes at capacity instead of evicting, which is what a queue
wants — but it answers a read from whichever replica received it, and this driver
reads a job's absence as that job being finished. A lagging replica would drop a
job. 'at_cap_policy' => 'reject_writes' is refused with that explanation.
Rostam v0.6.0 added a flush op, and it is not Redis's FLUSHDB: it has no unit
smaller than the whole keyspace, so one call destroys every queued job along with
everything else on that server — including this queue's own cursors. Measured
against v0.6.0, a flush sent carrying the key app: still removed session:b.
That is reachable by accident: rostamlabs/rostam-cache-laravel v0.2.0 and newer
can be configured with 'flush' => 'server', and then an ordinary
php artisan cache:clear issues exactly this op. Every job that was waiting — not
yet taken by a worker — disappears, and nothing reports it; a worker already running
a job keeps its copy and finishes it. Keep the queue on a server nothing flushes, or
leave the cache driver on its default generational flush, which does not touch
these keys.
A job's payload is one value, and an entry has to fit in one page of the server's
cache — where the page bounds the key and the value together. The page follows
the PER-SHARD budget: floorPow2(max_memory / shards / 16), clamped to 1 MiB…1 GiB.
On a default server that is the 1 MiB floor, and strlen($key) + strlen($value)
reached 1,048,550 bytes on v0.6.0 and 1,048,546 on v0.7.0-beta6 and beta7,
constant across key lengths; the key here is the queue's own
({prefix}{queue}:job:{id}), so nearly all of it is yours. Measured at other
geometries on beta7: 2,097,122 at 32 MiB on one shard, 4,194,274 at 256 MiB on
four, 8,388,578 at 128 MiB on one — 30 bytes under the page each time on an
in-memory shard, and 16 bytes less again on a persistent one (-data). Most
deployments sit on the floor, so halving the shard count raises the limit where
raising max_memory alone does not. A larger job fails at push with the server's
generic internal error. Keep payloads small — pass ids, not models.
-tcp listener
and -relocating-evictionrostamlabs/rostam-client-php ^0.3rostam-server -tcp 127.0.0.1:7000 -relocating-eviction -config rostam.json
composer require rostamlabs/rostam-queue-laravel
config/queue.php:
'rostam' => [
'driver' => 'rostam',
'connection' => 'default', // an entry under rostam.connections
'queue' => 'default',
'retry_after' => 90, // the lease; longer than your longest job
'at_cap_policy' => 'headroom', // required - see above
'on_evictions' => 'refuse', // or 'ignore' - see below
'after_commit' => false, // Laravel's own option
'verify_every' => 60, // seconds between eviction checks
'sweep_seconds' => 60, // delayed seconds one pop may move
'reclaim_batch' => 32, // ids one pop checks for jobs whose worker died
'tombstone_ttl' => 604800, // how long a killed slot stays killed
],
tombstone_ttl must be longer than retry_after, and the connector refuses a
pair that is not. A killed slot is what stops a push from landing in a slot the
queue has moved past, and a lease — including a dead worker's — lasts
retry_after; a tombstone that expires first lets a slot be re-used while its
lease is still held.
on_evictions decides what the eviction check does when it finds something.
refuse (the default) is the safe answer and an unforgiving one: the count is
node-wide, so a co-tenant's evicted cache key stops this queue too, and a worker
that has refused keeps refusing until it restarts — while the count itself only
resets when the server restarts, which on a node without -data takes the
backlog with it. Set it to ignore on a node whose eviction count is somebody
else's, and accept that this queue will not notice the day it loses a job.
The server itself is described once, under rostam.connections, shared with
rostamlabs/rostam-cache-laravel
if you use it. A host/port given inline here wins over that.
php artisan queue:clear rostam works, and queue names may be backed enums or
forwarded with Laravel's queue routes.
Rostam's key-value engine has no list and no sorted set, so the queue is a dense id space with server-side counters:
{prefix}{queue}:tail allocates the next id to write
{prefix}{queue}:head the next id to read
{prefix}{queue}:job:{id} the payload (no TTL), or a tombstone
{prefix}{queue}:lease:{id} who holds it, with a TTL the engine expires
{prefix}{queue}:reclaim how far redelivery has settled
{prefix}{queue}:d:{second}:tail ids handed out for a delayed second, sealed while it is swept
{prefix}{queue}:d:{second}:job:{id} a delayed job, or a tombstone
{prefix}{queue}:mig:{second}:{id} the lease on moving one delayed job
{prefix}{queue}:swept the last delayed second fully moved
{prefix}{queue}:dgen the delayed generation, moved on by clear()
{prefix}{queue}:delayed:{gen} a gauge of delayed jobs waiting in that generation
Claiming takes a lease, not the job. The payload stays put for as long as a worker holds it, so a worker that dies does not take the work with it:
| payload | lease | means | | --- | --- | --- | | present | present | still being worked on | | present | absent | the worker died — redeliver | | absent or tombstone | — | finished, or a slot that was killed |
The middle row is what every other driver needs a reservation index for, and finding
expired reservations normally needs a scan this engine does not have. It does not
need one: the ids are dense, so walking them is the index. Each pop reads a
window of reclaim_batch ids in two round trips and looks past jobs still
running, so one long job does not hold back the abandoned ones above it within
that window. The cursor stops behind anything unsettled, so work more than
reclaim_batch ids above a job that is still running waits for it to finish.
Nothing is lost by waiting: the payload and its lapsed lease stay as they are.
Every step that decides something is one atomic op. Each job this driver has ever lost was lost in the gap between two round trips, and each gap is closed by deciding with a single op rather than checking and then acting.
set_nx
— a tombstone — and the push writes with set_nx too, so exactly one of them owns
the slot. If the worker won, the push draws another id.release() writes the retry before finishing the original, for the same
reason.clear() overwrites each slot with a tombstone instead of deleting it, so a
push that drew one of those ids finds it killed and lands after the clear, where
it is delivered. It starts at the redelivery cursor rather than the reader, so a
job whose worker died — which sits below the reader, waiting to be handed back
— is cleared and counted like any other, instead of reappearing minutes after the
queue reported itself empty. It is the one place this driver overwrites a payload,
which is what clearing a queue is; a job a worker is running finishes normally,
since that worker already holds it. The walk costs two round trips per 512 ids.Delayed jobs wait in per-second buckets, and each pop moves the seconds that
have come due. The same rules apply there:
clear() moves the delayed generation on: a job stored under an older one is
deleted instead of delivered when its second comes, and the generation is read
under the move lease, so a job delayed after a clear is never taken for one it
cleared.A queue idle for a long time catches up over several pops rather than sweeping a day of seconds in one.
tombstone_ttl (a week by default) between
drawing its id and writing its payload can land in a slot whose tombstone has
expired, behind every cursor. A tombstone that never expired would keep every
killed slot in memory forever; this is the trade, and tombstone_ttl => 0 takes
the other side of it. A clear() passing over such a slot does not re-arm it:
below the reader it kills only slots that hold something, which is what stops
clearing an empty queue from writing a key per id.clear() is running may land on either side of it.head, tail, reclaim, swept, dgen) and
a delayed generation's gauge have no TTL — they are the queue. A worker killed
between marking a delayed second done and deleting its counter leaves that one
small key behind too.$queue->size(); // pending + delayed
$queue->pendingSize(); // between the reader and the writer — an upper bound
$queue->delayedSize(); // delayed jobs waiting, however far out
$queue->reservedSize(); // always 0 — see below
$queue->redeliveredCount(); // jobs this instance handed back after a worker died
$queue->holesSeen(); // empty slots this instance killed and moved past
pendingSize() counts the ids between the reader and the writer: waiting jobs, plus
killed slots the reader has not yet passed. Jobs being worked on are not in it — the
reader has passed them.
delayedSize() is a gauge kept beside the buckets, since the buckets cannot be
enumerated; a worker killed between storing a delayed job and counting it leaves it
off by one.
holesSeen() counts empty slots this instance killed and moved past. A push caught
mid-write leaves one and re-routes itself, so a steady trickle is normal. A run
of 64 in a single pop throws JobVanished: that is records written and then gone.
Losing races to other workers never counts; a pop that loses every race returns
nothing, and the worker polls again.
reservedSize() is always zero, and that is a statement rather than a stub. A
reserved job is one a worker holds but has not finished; the only record of that is a
lease key per job, and nothing on this engine can count keys by pattern. A number
this driver cannot know would be worse than none.
composer install
composer test
Every race this driver has lost jobs to has a test that drives that exact interleaving — a push caught between its id and its payload, a worker killed while moving a delayed job, a clear finishing under a sweep — through a client that can stop the world between any two round trips. Each of those tests fails if its hook never fires, and every fix is checked by reverting it and watching a test go red — 22 of them at the last count. Where two guards overlap by design (a delayed job is both sealed out of a swept second and moved by its own push), reverting one alone leaves the other holding, and the pair is checked together.
Against a real server, a chaos test runs producers and workers as separate processes and kills workers at random instants, then asserts that every job a producer saw accepted was finished by a worker — not merely handed out:
ROSTAM_TEST_SERVER=127.0.0.1:7000 vendor/bin/phpunit tests/Chaos/ChaosTest.php
Apache-2.0, the same licence as Rostam itself — see LICENSE and NOTICE.
Get paid by clients abroad
Wise gives you local bank details in several currencies and converts at the mid-market rate. Useful when your invoices are not in your own currency.
Try WiseNew Laravel packages worth a look, once a month. No spam, one-click unsubscribe.