Queue System

The Volāre application implements a robust queue infrastructure for handling asynchronous tasks using PostgreSQL for development and Amazon SQS for production.

Overview

Key Features:

Database-driven queues for development (PostgreSQL)
Automatic worker management via Docker Compose
Production-ready configuration for Amazon SQS
Built-in monitoring and debugging commands
Graceful restarts and failure handling

Architecture

Components

Application Code -> Queue Connection -> PostgreSQL Jobs Table
                                              |
                                              v
                    Queue Worker Container -> Job Handler -> Success/Failure

Infrastructure

Development Environment:

Queue Driver: PostgreSQL database driver
Worker Process: Dedicated Docker container (queue service)
Management: Docker Compose (no Supervisor needed)
Persistence: Database tables (jobs, failed_jobs, job_batches)

Production Environment:

Queue Driver: Amazon SQS
Worker Process: EC2 instances or ECS containers
Management: AWS infrastructure

Queue Priority

The main worker processes queues in priority order:

aerticket-tickets - Ticket issuance (highest priority)
aerticket-fastlane - Fast-lane booking operations
aerticket-bookings - Booking creation
aerticket-voids - Ticket void operations
pipedrive-sync - Pipedrive CRM sync jobs (Lead/Booking/Client/Passenger/BookingUpsell/BookingNote)
offer-recalculations - Bulk offer flight/price recalculation sweeps
default - General tasks

pipedrive-sync is listed before default so Pipedrive sync jobs aren’t starved when default has a backlog. The queue name is configurable via the PIPEDRIVE_QUEUE env var (default pipedrive-sync, defined in backend/config/pipedrive.php).

flight-searches is not on the main worker — it runs on a dedicated queue-flights service (see Flight Search Worker) so a burst of customer-facing Aerticket searches can’t delay ticketing/booking jobs, and the main worker’s admin sweeps (offer-recalculations) can’t slow customer-driven cache fills.

Quick Start

Dispatching Jobs

use App\Jobs\TestQueueJob;

// Simple dispatch
TestQueueJob::dispatch('Hello, Queue!');

// Delayed dispatch (5 minutes)
TestQueueJob::dispatch('Delayed message')
    ->delay(now()->addMinutes(5));

// Custom queue
TestQueueJob::dispatch('High priority')
    ->onQueue('high-priority');

Monitoring

# Check queue status
./vendor/bin/sail artisan queue:monitor

# Watch logs in real-time
./vendor/bin/sail artisan tail

# Check failed jobs
./vendor/bin/sail artisan queue:failed

Configuration

Environment Variables

# Queue driver: database (dev) or sqs (production)
QUEUE_CONNECTION=database

# Database queue settings
DB_QUEUE_CONNECTION=pgsql
DB_QUEUE_TABLE=jobs
DB_QUEUE=default
DB_QUEUE_RETRY_AFTER=90

# Job retry settings
QUEUE_RETRY_AFTER=90
QUEUE_MAX_ATTEMPTS=3

Docker Compose Configuration

queue:
  build:
    context: './vendor/laravel/sail/runtimes/8.5'
    dockerfile: Dockerfile
  volumes:
    - '.:/var/www/html'
  networks:
    - sail
  depends_on:
    - pgsql
    - redis
  command: php artisan queue:work --queue=aerticket-tickets,aerticket-fastlane,aerticket-bookings,aerticket-voids,pipedrive-sync,offer-recalculations,default --sleep=3 --tries=3 --max-time=3600

Worker Behavior:

Polls every 3 seconds for new jobs
Max 3 retry attempts per job
Max 1 hour runtime before graceful restart
Auto-restart on container failure

Flight Search Worker (`queue-flights`)

A second Compose service runs a dedicated worker for customer-facing flight cache work (SearchFlightCacheJob, CascadeFlightSearchForDateJob), isolated on the flight-searches queue:

queue-flights:
  command: php artisan queue:work --queue=flight-searches --sleep=3 --tries=3 --max-time=3600

Isolating it keeps a burst of Aerticket searches from delaying ticketing/booking jobs, and keeps the main worker’s admin sweeps (offer-recalculations) from slowing customer-driven cache fills.

Source: backend/compose.yaml (queue and queue-flights services)

Production Worker Memory (`queue-worker.sh`)

In production the queue worker runs through the docker/common/queue-worker/queue-worker.sh wrapper, which passes --memory="$QUEUE_WORKER_MEMORY" (default 384 MB) to queue:work and validates that the value is a positive integer. Making the memory limit configurable via the QUEUE_WORKER_MEMORY env var fixed a memory-driven restart loop (#2220). The prod wrapper runs a single worker over the combined queue list (aerticket-tickets,aerticket-fastlane,aerticket-bookings,aerticket-voids,pipedrive-sync,offer-recalculations,flight-searches,default).

Source: backend/docker/common/queue-worker/queue-worker.sh

Common Operations

Restarting Workers

# Signal workers to restart after current job
./vendor/bin/sail artisan queue:restart

# Or restart the Docker container
./vendor/bin/sail restart queue

Handling Failed Jobs

# List all failed jobs
./vendor/bin/sail artisan queue:failed

# Retry a specific failed job
./vendor/bin/sail artisan queue:retry {job_id}

# Retry all failed jobs
./vendor/bin/sail artisan queue:retry all

# Flush all failed jobs
./vendor/bin/sail artisan queue:flush

Monitoring Workers

# Check worker status
docker ps | grep queue

# View worker logs
./vendor/bin/sail logs queue

# Follow worker logs in real-time
./vendor/bin/sail logs -f queue

AerTicket Queue Jobs

Ticket Issuance Queue

Queue Name: aerticket-tickets

Purpose: Process ticket issuance requests with highest priority

Jobs:

AerticketIssueTicketJob - Issues tickets for confirmed bookings

Configuration:

public int $tries = 3;
public int $timeout = 120;
public array $backoff = [30, 60, 180];

Booking Creation Queue

Queue Name: aerticket-bookings

Purpose: Process flight booking creation asynchronously

Jobs:

AerticketCreateBookingJob - Creates flight bookings from admin panel
CreateCheckoutFlightBookingJob - Checkout flight booking job triggered from booking admin actions
AerticketRetrieveBookingJob - Retrieves PNR details

Checkout Flight Booking (Per-Leg):

After successful checkout payment, bookings with flights transition to pending_flight_booking. Admin users trigger CreateCheckoutFlightBookingJob from the booking detail page (Book All Flights / Retry per-leg). One job is dispatched per unbooked leg.

Configuration:

public int $tries = 3;
public int $maxExceptions = 2;
public int $timeout = 420;
public array $backoff = [120];
public int $uniqueFor = 480; // ShouldBeUnique per booking+leg

Features:

Implements ShouldBeUnique with uniqueId = "checkout-flight:{bookingId}:{legIndex}"
Idempotency: Per-leg check via FlightBooking::where(booking_id, leg_index)->exists() with row-level lock
International legs: re-searches round-trip, matches by flight numbers + price
Domestic legs: searches one-way, matches by flight numbers + price
Differentiated retry: noMatchingFare releases with 5-minute delay (fare refresh), other errors use 120s backoff
Notifies admins via Filament on per-leg success and all-legs-complete
Creates FlightBooking with booking_id, leg_index, flight_type
All-legs-booked check: only transitions to flights_confirmed when ALL legs have FlightBooking records

Source: backend/app/Jobs/CreateCheckoutFlightBookingJob.php

Flight Search Queue

Queue Name: flight-searches

Purpose: Process dynamic flight cache searches asynchronously for bulk operations

Jobs:

SearchFlightCacheJob - Executes flight search for a single cache entry

Configuration:

public int $tries = 3;
public int $timeout = 90;
public array $backoff = [30, 60, 120];

Features:

User attribution for activity logging
Automatic failure handling with status update
Structured logging to Grafana Loki
Activity log entries for audit trail

Usage:

use App\Jobs\SearchFlightCacheJob;

// Dispatch with user attribution
SearchFlightCacheJob::dispatch($cacheId, $userId)
    ->onQueue('flight-searches');

// Via service (recommended)
$service = app(DynamicFlightCachePopulatorService::class);
$result = $service->dispatchSearchJobs($entries, $userId);

Source: backend/app/Jobs/SearchFlightCacheJob.php

Related: Dynamic Flight Cache for full documentation

Offer Recalculation Queue

Queue Name: offer-recalculations (RecalculateOfferFlightsJob::QUEUE_NAME)

Purpose: Run bulk offer flight/price recalculation sweeps off the main request path, isolated from customer-facing flight searches.

Jobs:

RecalculateOfferFlightsJob — recalculates an offer’s bound flights and prices

Consumed by the main queue worker (dev/local) and by the combined production worker.

Source: backend/app/Jobs/RecalculateOfferFlightsJob.php

Pipedrive Sync Queue

Queue Name: pipedrive-sync (configurable via PIPEDRIVE_QUEUE)

Purpose: Push CRM-relevant model changes to Pipedrive asynchronously via retryable jobs.

Dispatched by: PipedriveSyncObserver, registered in app/Providers/AppServiceProvider.php for Client, Booking, Passenger, BookingUpsell, BookingNote, and Lead.

The worker must include this queue in its --queue= list, or observer-dispatched jobs will accumulate unprocessed in the jobs table. Newsletter subscriptions are unaffected because NewsletterSubscriptionController calls Pipedrive inline from the HTTP request.

Related: Leads API - Pipedrive Sync for the full sync flow, custom fields, and operational requirements.

Security Best Practices

Validate Job Data

public function __construct(
    public int $userId,
    public string $action
) {
    if (!User::find($userId)) {
        throw new \InvalidArgumentException('Invalid user ID');
    }
}

Encrypt Sensitive Data

use Illuminate\Contracts\Queue\ShouldBeEncrypted;

class ProcessPayment implements ShouldQueue, ShouldBeEncrypted
{
    public function __construct(
        public string $creditCardNumber
    ) {}
}

Rate Limiting

use Illuminate\Queue\Middleware\RateLimited;

public function middleware(): array
{
    return [new RateLimited('api-calls')];
}

Troubleshooting

Worker Not Processing Jobs

Check worker is running:

docker ps | grep queue

Check worker logs:

./vendor/bin/sail logs queue

Restart worker:

./vendor/bin/sail restart queue

Jobs Failing Repeatedly

Check failed job details:

./vendor/bin/sail artisan queue:failed

Review exception:

./vendor/bin/sail artisan tinker
>>> DB::table('failed_jobs')->latest()->first()->exception;

Fix issue and retry:

./vendor/bin/sail artisan queue:retry all

Connection Timeouts

Increase timeout in job:

public $timeout = 120; // 2 minutes

Increase retry_after:

DB_QUEUE_RETRY_AFTER=180

Queue System

Overview

Architecture

Components

Infrastructure

Queue Priority

Quick Start

Dispatching Jobs

Monitoring

Configuration

Environment Variables

Docker Compose Configuration

Flight Search Worker (queue-flights)

Production Worker Memory (queue-worker.sh)

Common Operations

Restarting Workers

Handling Failed Jobs

Monitoring Workers

AerTicket Queue Jobs

Ticket Issuance Queue

Booking Creation Queue

Flight Search Queue

Offer Recalculation Queue

Pipedrive Sync Queue

Security Best Practices

Validate Job Data

Encrypt Sensitive Data

Rate Limiting

Troubleshooting

Worker Not Processing Jobs

Jobs Failing Repeatedly

Connection Timeouts

Flight Search Worker (`queue-flights`)

Production Worker Memory (`queue-worker.sh`)