Laravel Queue Manager

A Laravel package to manage multiple queue workers similar to Horizon, with memory monitoring and graceful shutdown capabilities. This package allows you to run multiple queue workers with different configurations from a single command.

Features

• 🚀 Multiple Worker Processes - Spawn multiple workers with different configurations
• 🧠 Memory Monitoring - Automatically restart workers that exceed memory limits
• 🔄 Auto-Restart - Workers are restarted when they crash or terminate
• 🛑 Graceful Shutdown - Properly handles SIGTERM, SIGINT (Ctrl+C), and SIGHUP signals
• ⚙️ Configurable - Easy configuration file for all worker settings
• 📊 Process Monitoring - Real-time display of worker status and PIDs

Requirements

• PHP 8.1 or higher
• Laravel 10.x or 11.x
• PCNTL extension (for signal handling)

Installation

Install the package via Composer:

composer require randika-srimal/queue-manager

For Local Development

If you're developing this package locally, add it to your Laravel project's composer.json:

{
    "repositories": [
        {
            "type": "path",
            "url": "./packages/queue-manager"
        }
    ],
    "require": {
        "randika-srimal/queue-manager": "@dev"
    }
}

Then run:

composer update randika-srimal/queue-manager

Configuration

Publish the configuration file:

php artisan vendor:publish --tag=queue-manager-config

This will create config/queue-manager.php in your Laravel project. Edit this file to configure your workers:

<?php

return [
    'workers' => [
        [
            'connection' => 'database',
            'queue' => 'default',
            'processes' => 3,
            'memory' => 128,  // Maximum memory in MB before restart
            'timeout' => 60,  // Maximum execution time per job in seconds
            'sleep' => 3,     // Seconds to sleep when no jobs available
            'tries' => 3,     // Number of times to attempt a job
        ],
        [
            'connection' => 'redis',
            'queue' => 'high-priority,default',
            'processes' => 5,
            'memory' => 256,
            'timeout' => 60,
            'sleep' => 1,
            'tries' => 3,
        ],
    ],

    'memory_check_interval' => 30, // How often to check memory (seconds)
    'restart_delay' => 2,          // Delay before restarting a worker (seconds)
];

Usage

Start the queue manager:

php artisan queue:manage

Options

• --no-restart - Disable automatic restart of failed workers

php artisan queue:manage --no-restart

Stopping the Queue Manager

Press Ctrl+C to gracefully shutdown all workers. The manager will:

1. Send SIGTERM to all worker processes
2. Wait up to 10 seconds for workers to finish current jobs
3. Force kill any remaining processes

How It Works

1. Worker Spawning - The command reads the configuration and spawns the specified number of worker processes for each queue configuration
2. Process Monitoring - Continuously monitors all worker processes
3. Memory Checking - Periodically checks memory usage and restarts workers exceeding limits
4. Auto-Restart - Automatically restarts workers that crash or terminate unexpectedly
5. Signal Handling - Catches shutdown signals and gracefully terminates all child processes

Example Configuration

Database Queue with Email Workers

'workers' => [
    [
        'connection' => 'database',
        'queue' => 'default',
        'processes' => 3,
        'memory' => 128,
        'timeout' => 60,
        'sleep' => 3,
        'tries' => 3,
    ],
    [
        'connection' => 'database',
        'queue' => 'emails',
        'processes' => 2,
        'memory' => 128,
        'timeout' => 120,
        'sleep' => 5,
        'tries' => 2,
    ],
],

Redis Queue with Priority

'workers' => [
    [
        'connection' => 'redis',
        'queue' => 'high-priority,default,low-priority',
        'processes' => 5,
        'memory' => 256,
        'timeout' => 60,
        'sleep' => 1,
        'tries' => 3,
    ],
],

Process Supervisor

For production environments, it's recommended to run the queue manager under a process supervisor like systemd or Supervisor.

Systemd Example

Create /etc/systemd/system/queue-manager.service:

[Unit]
Description=Laravel Queue Manager
After=network.target

[Service]
Type=simple
User=www-data
WorkingDirectory=/var/www/html
ExecStart=/usr/bin/php /var/www/html/artisan queue:manage
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl enable queue-manager
sudo systemctl start queue-manager

Supervisor Example

Create /etc/supervisor/conf.d/queue-manager.conf:

[program:queue-manager]
process_name=%(program_name)s
command=php /var/www/html/artisan queue:manage
autostart=true
autorestart=true
user=www-data
redirect_stderr=true
stdout_logfile=/var/www/html/storage/logs/queue-manager.log

Update and start:

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start queue-manager

License

The MIT License (MIT). Please see License File for more information.

Credits

Inspired by Laravel Horizon's process management capabilities.