Queue System

The Volāre application implements a queue infrastructure for asynchronous tasks. The committed Laravel default is the database queue driver, and queue.php documents the production retry budget against the database driver; the SQS block remains present in config but has no retry_after knob and is not the source for the current timing guarantees.

Overview

Key Features:

• Database-driven queues by default (QUEUE_CONNECTION=database)
• Automatic worker management via Docker Compose
• SQS config block present as an optional Laravel driver, but with no retry_after; current retry-after invariants are documented against database/beanstalkd/redis
• Built-in monitoring and debugging commands
• Graceful restarts and failure handling

Architecture

Components

Application Code -> Queue Connection -> PostgreSQL Jobs Table
                                              |
                                              v
                    Queue Worker Container -> Job Handler -> Success/Failure

Infrastructure

Runtime profile documented in code:

• Queue Driver: database by default (backend/config/queue.php), with the database connection carrying DB_QUEUE_RETRY_AFTER
• Worker Process: Dockerized queue worker (queue / queue-flights locally; queue-worker.sh in production image)
• Management: Docker Compose locally; container orchestration in production

Queue Priority

The main worker processes queues in priority order:

1. aerticket-tickets - Ticket issuance (highest priority)
2. aerticket-fastlane - Fast-lane booking operations
3. aerticket-bookings - Booking creation
4. aerticket-voids - Ticket void operations
5. pipedrive-sync - Pipedrive CRM sync jobs (Lead/Booking/Client/Passenger/BookingUpsell/BookingNote/ProductByMarket/NewsletterSubscriber, plus AttachProductsToDealJob)
6. activecampaign-sync - ActiveCampaign newsletter contact delivery
7. offer-recalculations - Bulk offer flight/price recalculation sweeps
8. offer-generation - Auto-offer generation runs queued from the admin “Auto-Generate Offers” actions (GenerateAutoOffersJob), plus Last Seats campaign price syncs (SyncLastSeatsPricesJob)
9. default - General tasks

pipedrive-sync and activecampaign-sync are listed before default so marketing/CRM sync jobs aren’t starved when default has a backlog. The Pipedrive queue name is configurable via the PIPEDRIVE_QUEUE env var (default pipedrive-sync, defined in backend/config/pipedrive.php); activecampaign-sync is a constant on the job (see ActiveCampaign Sync Queue).

flight-searches is not on the main worker — it runs on a dedicated queue-flights service (see Flight Search Worker) so a burst of customer-facing Aerticket searches can’t delay ticketing/booking jobs, and the main worker’s admin sweeps (offer-recalculations) can’t slow customer-driven cache fills.

Quick Start

Dispatching Jobs

use App\Jobs\TestQueueJob;

// Simple dispatch
TestQueueJob::dispatch('Hello, Queue!');

// Delayed dispatch (5 minutes)
TestQueueJob::dispatch('Delayed message')
    ->delay(now()->addMinutes(5));

// Custom queue
TestQueueJob::dispatch('High priority')
    ->onQueue('high-priority');

Monitoring

# Check queue status
./vendor/bin/sail artisan queue:monitor

# Watch logs in real-time
./vendor/bin/sail artisan tail

# Check failed jobs
./vendor/bin/sail artisan queue:failed

Configuration

Environment Variables

# Queue driver (committed default: database)
QUEUE_CONNECTION=database

# Database queue settings
DB_QUEUE_CONNECTION=pgsql
DB_QUEUE_TABLE=jobs
DB_QUEUE=default
DB_QUEUE_RETRY_AFTER=240

# Job retry settings
QUEUE_MAX_ATTEMPTS=3

Docker Compose Configuration

queue:
  build:
    context: './vendor/laravel/sail/runtimes/8.5'
    dockerfile: Dockerfile
  volumes:
    - '.:/var/www/html'
  networks:
    - sail
  depends_on:
    - pgsql
    - redis
  command: php artisan queue:work --queue=aerticket-tickets,aerticket-fastlane,aerticket-bookings,aerticket-voids,pipedrive-sync,activecampaign-sync,offer-recalculations,offer-generation,default --sleep=3 --tries=3 --max-time=3600

Worker Behavior:

• Polls every 3 seconds for new jobs
• Max 3 retry attempts per job
• Max 1 hour runtime before graceful restart
• Auto-restart on container failure

Flight Search Worker (queue-flights)

A second Compose service runs a dedicated worker for customer-facing flight cache work (SearchFlightCacheJob, CascadeFlightSearchForDateJob), isolated on the flight-searches queue:

queue-flights:
  command: php artisan queue:work --queue=flight-searches --sleep=3 --tries=3 --max-time=3600

Isolating it keeps a burst of Aerticket searches from delaying ticketing/booking jobs, and keeps the main worker’s admin sweeps (offer-recalculations) from slowing customer-driven cache fills.

Source: backend/compose.yaml (queue and queue-flights services)

Production Worker Restarts (queue-worker.sh)

In production the queue worker runs through the docker/common/queue-worker/queue-worker.sh wrapper, which passes --memory="$QUEUE_WORKER_MEMORY" (default 384 MB) to queue:work and validates that the value is a positive integer. Making the memory limit configurable via the QUEUE_WORKER_MEMORY env var fixed a memory-driven restart loop (#2220). The prod wrapper runs a single worker over the combined queue list (aerticket-tickets,aerticket-fastlane,aerticket-bookings,aerticket-voids,pipedrive-sync,activecampaign-sync,offer-recalculations,offer-generation,flight-searches,default).

Any job timeout also restarts the container. The wrapper execs queue:work under set -eu, so the worker is PID 1. Laravel’s job-timeout SIGALRM handler exits non-zero, PID 1 dies, and the container restarts — one restart per timed-out job, regardless of memory. This is a second, independent cause of the same restart-loop symptom as #2220, and it is why job $timeout values are treated as an operational constraint rather than a tuning knob (see Flight Search Queue).

Source: backend/docker/common/queue-worker/queue-worker.sh

Common Operations

Restarting Workers

# Signal workers to restart after current job
./vendor/bin/sail artisan queue:restart

# Or restart the Docker container
./vendor/bin/sail restart queue

A worker boots the application once and keeps that code in memory for the life of the process, so a release that changes a queued class needs this signal or the worker keeps running the old code until it exits on its own.

The failure is asymmetric and easy to misread: anything sent with notifyNow() goes out fine, because it renders in the web request on new code, while every queued job fails. Transactional emails are exactly that shape — the customer’s own email is immediate, but the copy recipients configured in System → Emails are queued (TransactionalEmailCopier uses notify() so an operator’s click never blocks on SMTP). A release without queue:restart therefore looks healthy — customers receive their emails — while every internal copy lands in failed_jobs with an error such as Call to undefined method for a method the release added.

Include queue:restart in the deploy steps for any release that touches a Notification, Mailable, Job, or a model those call.

Handling Failed Jobs

# List all failed jobs
./vendor/bin/sail artisan queue:failed

# Retry a specific failed job
./vendor/bin/sail artisan queue:retry {job_id}

# Retry all failed jobs
./vendor/bin/sail artisan queue:retry all

# Flush all failed jobs
./vendor/bin/sail artisan queue:flush

Monitoring Workers

# Check worker status
docker ps | grep queue

# View worker logs
./vendor/bin/sail logs queue

# Follow worker logs in real-time
./vendor/bin/sail logs -f queue

AerTicket Queue Jobs

Ticket Issuance Queue

Queue Name: aerticket-tickets

Purpose: Process ticket issuance requests with highest priority

Jobs:

• AerticketIssueTicketJob - Issues tickets for confirmed bookings

Configuration:

public int $tries = 3;
public int $timeout = 120;
public array $backoff = [30, 60, 180];

Booking Creation Queue

Queue Name: aerticket-bookings

Purpose: Process flight booking creation asynchronously

Jobs:

• AerticketCreateBookingJob - Creates flight bookings from admin panel
• CreateCheckoutFlightBookingJob - Checkout flight booking job triggered from booking admin actions
• AerticketRetrieveBookingJob - Retrieves PNR details

Checkout Flight Booking (Per-Leg):

After successful checkout payment, bookings with flights transition to pending_flight_booking. Admin users trigger CreateCheckoutFlightBookingJob from the booking detail page (Book All Flights / Retry per-leg). One job is dispatched per unbooked leg.

Configuration:

public int $tries = 3;
public int $maxExceptions = 2;
public int $timeout = 420;
public array $backoff = [120];
public int $uniqueFor = 480; // ShouldBeUnique per booking+leg

Features:

• Implements ShouldBeUnique with uniqueId = "checkout-flight:{bookingId}:{legIndex}"
• Idempotency: a cache lock on checkout-flight-booking:{bookingId}:{legIndex} held across the booking call, plus a per-leg FlightBooking::where(booking_id, leg_index)->exists() check inside it. The check alone is not enough — its row lock is released when its transaction commits, before the supplier is called, so two concurrent runs would both find no booking and both book
• International legs: re-searches round-trip, matches by flight numbers + price
• Domestic legs: searches one-way, matches by flight numbers + price
• Differentiated retry: noMatchingFare releases with 5-minute delay (fare refresh), other errors use 120s backoff
• Notifies admins via Filament on per-leg success and all-legs-complete
• Creates FlightBooking with booking_id, leg_index, flight_type
• All-legs-booked check: only transitions to flights_confirmed when ALL legs have FlightBooking records

Source: backend/app/Jobs/CreateCheckoutFlightBookingJob.php

Related: Checkout API - Admin-Driven Flight Booking

Flight Search Queue

Queue Name: flight-searches

Purpose: Process dynamic flight cache searches asynchronously for bulk operations

Jobs:

• SearchFlightCacheJob - Executes flight search for a single cache entry

Configuration:

public int $tries = 3;
public int $timeout = 150;
public array $backoff = [5, 15, 30];

Why the timeout is 150s

The timeout is derived, not chosen: SIZED_FOR_MAX_SHIFTS (4) × AERTICKET_SEARCH_TIMEOUT (30s) + 30s persistence headroom.

Multi-city routes run DynamicFlightCachePopulatorService::executeMultiShiftSearchForEntry(), which issues one serial Aerticket call per observed overnight shift. The job’s wall clock is therefore shifts × search timeout, not a single search timeout. Most multi-city routes carry 3 or more shifts, so the previous ceiling — sized for two searches — timed out routinely, and every timeout restarted the worker container.

SearchFlightCacheJob::SIZED_FOR_MAX_SHIFTS is a production snapshot, not a hard limit: the shift set grows as routes learn new shapes. The populator logs a Multi-shift search exceeds the sized job timeout budget warning when a route exceeds it. That warning is the signal to re-derive $timeout — it arrives before the route starts timing out.

Do not change AERTICKET_SEARCH_TIMEOUT, $timeout, or $backoff in isolation. Three values are coupled, and FlightCacheTimeoutBudgetTest fails the build if the relationship breaks:

Value	Constraint
SearchFlightCacheJob::$timeout (150s)	≥ SIZED_FOR_MAX_SHIFTS × AERTICKET_SEARCH_TIMEOUT plus persistence headroom
DynamicFlightCache::SEARCHING_STALE_AFTER_MINUTES (10)	> worst case timeout × tries + sum(backoff) = 500s ≈ 8.3 min
Connection retry_after (240s)	> $timeout, or the queue re-releases a still-running job and two workers write fares for the same cache entry

AERTICKET_SEARCH_TIMEOUT defaults to 30s in config/aerticket.php (search.timeout); phpunit.xml pins it to the production value so the budget test asserts against production, not a developer’s local .env.

Features:

• User attribution for activity logging
• Automatic failure handling with status update
• Structured logging to Grafana Loki
• Activity log entries for audit trail

Usage:

use App\Jobs\SearchFlightCacheJob;

// Dispatch with user attribution
SearchFlightCacheJob::dispatch($cacheId, $userId)
    ->onQueue('flight-searches');

// Via service (recommended)
$service = app(DynamicFlightCachePopulatorService::class);
$result = $service->dispatchSearchJobs($entries, $userId);

Source: backend/app/Jobs/SearchFlightCacheJob.php

Related: Dynamic Flight Cache for full documentation

Offer Recalculation Queue

Queue Name: offer-recalculations (RecalculateOfferFlightsJob::QUEUE_NAME)

Purpose: Run bulk offer flight/price recalculation sweeps off the main request path, isolated from customer-facing flight searches.

Jobs:

• RecalculateOfferFlightsJob — recalculates an offer’s bound flights and prices

Consumed by the main queue worker (dev/local) and by the combined production worker.

RecalculateSupplierOfferPricesJob — the land re-pricing a signed supplier contract addendum triggers — is not on this queue: it runs on default, one attempt, 600 s timeout, and invokes offers:recalculate-prices in-process (Artisan::call()) for the whole supplier rather than per offer.

Source: backend/app/Jobs/RecalculateOfferFlightsJob.php, backend/app/Jobs/RecalculateSupplierOfferPricesJob.php

Offer Generation Queue

Queue Name: offer-generation (GenerateAutoOffersJob::QUEUE_NAME)

Purpose: Run the auto-offer generator off the request path when an operator triggers it from the admin. A large product (five airports, a year of flight cache) needs minutes, far past the 60 s web ceiling — the inline action used to surface as a bare “Error while loading page” (nginx 504). The 15-minute scheduler does not use this queue: offers:auto-generate runs the generator directly inside the scheduler container.

Jobs:

• GenerateAutoOffersJob(?int $productByMarketId = null, int $maxOffersPerDate = 1, ?int $userId = null) — null product means all eligible products. tries=3, timeout=300, backoff=[30, 60, 120].
• SyncLastSeatsPricesJob(int $productByMarketId, string $departureDate, ?int $departureId = null) — re-persists one Last Seats campaign departure’s offer prices by calling offers:recalculate-prices --apply --force --product --departure-date, then records the discount the offers now hold. It does not take the auto-offers lock.

Concurrency: every run takes the auto-offers cache lock (TTL = job timeout). A run that finds the lock held is released back to the queue with a 120 s delay rather than dropped, so a single-product dispatch arriving during a whole-catalogue run still gets its turn. Three lost attempts exhaust tries and the job fails with MaxAttemptsExceededException; the operator is told the run never started.

Reporting: when a userId is set (admin dispatch), the job sends the run summary — built by AutoOfferGenerationSummary — to that user’s notification bell as a Filament database notification with a “View offers” link; a notification failure is logged and swallowed (a deleted user is skipped silently) so a reporting problem never re-runs a generation whose offers are already written. See Offers → Triggers for the exact wording.

Consumed by the main queue worker (dev/local) and by the combined production worker. As with activecampaign-sync, the queue name is a constant on the job and the --queue= lists in compose.yaml and queue-worker.sh are hardcoded: a job dispatched to a queue no worker consumes sits in jobs forever, which is exactly how offers:auto-generate --queue behaved before offer-generation was added to those lists.

Source: backend/app/Jobs/GenerateAutoOffersJob.php, backend/app/Jobs/SyncLastSeatsPricesJob.php, backend/app/Services/Offers/AutoOfferGenerationSummary.php

Pipedrive Sync Queue

Queue Name: pipedrive-sync (configurable via PIPEDRIVE_QUEUE)

Purpose: Push CRM-relevant model changes to Pipedrive asynchronously via retryable jobs.

Dispatched by:

• PipedriveSyncObserver, registered in app/Providers/AppServiceProvider.php for Client, Booking, Passenger, BookingUpsell, BookingNote, and Lead.
• NewsletterSubscriptionController::store(), which dispatches SyncNewsletterSubscriberToPipedriveJob explicitly — that model has no observer, since the job writes pipedrive_person_id back on success.
• SyncBookingToPipedriveJob, which dispatches SyncProductByMarketToPipedriveJob and AttachProductsToDealJob onto the same queue.
• The Artisan re-dispatchers pipedrive:sync-all and pipedrive:retry-failed.

The worker must include this queue in its --queue= list, or these jobs will accumulate unprocessed in the jobs table. For newsletter signups that means the row persists locally and newsletter_subscribers.pipedrive_person_id stays null; the subscriber’s pipedrive_syncs row is the record of what happened.

Related: Leads API - Pipedrive Sync for the full sync flow, custom fields, and operational requirements. Newsletter Subscriptions API — Pipedrive Sync Behaviour for the newsletter Person mirror and its recovery commands.

ActiveCampaign Sync Queue

Queue Name: activecampaign-sync (SyncNewsletterSubscriberToActiveCampaignJob::QUEUE_NAME)

Purpose: Mirror newsletter subscribers into ActiveCampaign so marketing can run campaigns off them. List membership is what triggers marketing’s welcome automation; the market tag is what lets campaigns segment the single list.

Jobs:

• SyncNewsletterSubscriberToActiveCampaignJob — initial delivery upserts one subscriber, adds it to the newsletter list, and tags its market; phone-only delivery fills an existing contact’s empty phone without changing subscription state

Dispatched by:

• NewsletterSubscriptionController::store(), for newly created subscribers and, in phone-only mode, when a repeated signup fills the first local phone.
• activecampaign:sync-subscribers, the operator-invoked backfill/repair command — it dispatches one initial-delivery job per subscriber whose activecampaign_contact_id is still null (--limit=500, --dry-run), staggered one second apart. It excludes successfully mirrored rows, not necessarily every pre-existing remote contact; see the re-delivery caution. It does not repair phone-only failures on linked contacts. Not scheduled.

Initial-delivery API calls — 3 in steady state without a phone:

Call	Payload
POST /contact/sync	contact wrapper: email, first/last name, and fieldValues for the three GDPR consent fields
POST /contactLists	configured list + contact id + status: 1
POST /contactTags	contact id + the market tag’s id — skipped when the row has no market

When a phone is supplied, initial delivery also reads /contacts/{id} and updates only an empty phone. Phone-only delivery performs that guarded read/update without any list, tag or consent writes; it searches by exact email first if the local contact id is missing. It never creates a contact, marks partial initial delivery complete, or reactivates an unsubscribed contact. Existing CRM phones are preserved in both modes.

The consent fields and the market tag are addressed by numeric, account-specific ids, resolved from their name/title by ActiveCampaignSchemaResolver (GET /fields, GET /tags, creating the object when missing). Those lookups are cached for 7 days in the default cache store, so they amortize to roughly one per field/tag per week rather than adding calls per subscriber.

Configuration:

public int $tries = 3;
public int $timeout = 120;
public array $backoff = [30, 60, 180];

Unlike pipedrive-sync, the queue name is not env-configurable — it is a constant on the job. The worker --queue= lists in compose.yaml and queue-worker.sh are hardcoded, so an env override would silently route jobs to a queue no worker consumes.

Consumed by the main queue worker (dev/local) and by the combined production worker. If the queue is missing from the --queue= list, subscribers still persist locally but never reach ActiveCampaign — newsletter_subscribers.activecampaign_contact_id stays null, which is the marker for an unmirrored row and exactly what activecampaign:sync-subscribers re-dispatches once the queue is fixed.

In environments without ActiveCampaign credentials (local, CI) the job checks ActiveCampaignClient::isConfigured(), logs at info, and returns, so it never accumulates failed jobs. The backfill command applies the same check before dispatching anything: with credentials missing and rows pending it errors and exits 1 instead of queueing jobs that would each skip.

Source: backend/app/Jobs/ActiveCampaign/SyncNewsletterSubscriberToActiveCampaignJob.php, backend/app/Services/ActiveCampaign/ActiveCampaignSchemaResolver.php, backend/app/Console/Commands/ActiveCampaignSyncSubscribersCommand.php

Related: Newsletter Subscriptions API — ActiveCampaign Sync Behaviour for endpoint contract, retry semantics, and configuration.

Security Best Practices

Validate Job Data

public function __construct(
    public int $userId,
    public string $action
) {
    if (!User::find($userId)) {
        throw new \InvalidArgumentException('Invalid user ID');
    }
}

Encrypt Sensitive Data

use Illuminate\Contracts\Queue\ShouldBeEncrypted;

class ProcessPayment implements ShouldQueue, ShouldBeEncrypted
{
    public function __construct(
        public string $creditCardNumber
    ) {}
}

Rate Limiting

use Illuminate\Queue\Middleware\RateLimited;

public function middleware(): array
{
    return [new RateLimited('api-calls')];
}

Troubleshooting

Worker Not Processing Jobs

1. Check worker is running:

docker ps | grep queue

2. Check worker logs:

./vendor/bin/sail logs queue

3. Restart worker:

./vendor/bin/sail restart queue

Jobs Failing Repeatedly

1. Check failed job details:

./vendor/bin/sail artisan queue:failed

2. Review exception:

./vendor/bin/sail artisan tinker
>>> DB::table('failed_jobs')->latest()->first()->exception;

3. Fix issue and retry:

./vendor/bin/sail artisan queue:retry all

Connection Timeouts

A job’s $timeout must stay below the connection’s retry_after, or the queue re-releases the still-running job to a second worker and it executes twice. Raise $timeout and retry_after together, and check the job’s own docblock first — several timeouts (notably SearchFlightCacheJob) are derived from an upstream API budget rather than picked, so raising them without re-deriving hides a real problem.

retry_after defaults to 240s on the database, beanstalkd and redis connections (DB_QUEUE_RETRY_AFTER, BEANSTALKD_QUEUE_RETRY_AFTER, REDIS_QUEUE_RETRY_AFTER). Production runs the database driver on PostgreSQL. The sqs driver exposes no retry_after — its AWS-side visibility timeout is the equivalent — but that connection is an unwired placeholder here.

Six jobs run longer than retry_after, and are guarded for it

240s does not cover every job, and cannot: CascadeFlightSearchForDateJob (600s), CreateCheckoutFlightBookingJob (420s), AerticketCreateBookingJob (330s), AerticketFastlaneTicketingJob (320s), AerticketVoidBookingJob and GenerateAutoOffersJob (300s) each have a ceiling set by a supplier call budgeted at ~300s, or — for the cascade — by measured search times. Cutting those ceilings would kill legitimate work, and raising retry_after past 600s would delay recovery for every job on the connection when a worker really does crash.

So the queue re-releasing them mid-run is expected, and each one is made safe to receive that second delivery:

• Four take an atomic cache lock on their unit of work — PNR, fare, booking leg, product — held across the supplier call. A lock released before the call, or replaced by an “is it already done?” lookup, would not help: both concurrent runs would pass the check and both would call the supplier. The loser returns rather than waiting, except in GenerateAutoOffersJob, which releases back to the queue: it locks on one key for every run, so a single-product dispatch arriving during a whole-catalogue run is not a duplicate of it and must not be dropped.
• The lock only covers deliveries that overlap. Sequential re-execution is routine here — the worker runs with --max-time=3600 --memory=384 and a SIGALRM at $timeout, so it can die mid-call and the same payload is picked up again once it restarts, with the first lock already expired. Each of the four therefore re-reads its state inside the lock: AerticketVoidBookingJob the ticket status, CreateCheckoutFlightBookingJob the leg’s FlightBooking, GenerateAutoOffersJob the per-date cap, and AerticketCreateBookingJob the persisted fare_id (added for this — it was being passed to FlightBooking::create() while being neither a column nor fillable, so Eloquent discarded it and there was nothing to check). That last check is bounded to 24h, because the same fareId can be re-emitted for a fare-family on a later search and an unbounded check would refuse a legitimate sale. It also ignores reservations that no longer stand, so rebooking after a void is not turned away — but a live ticket (issued, ticketing) holds the fare whatever the booking status says, since cancelling a booking does not void its ticket.
• One window stays open for all four: if the supplier acts and the process dies before the row is written, nothing is persisted and the next delivery acts again. Closing it needs an idempotency key on the supplier call, which the Aerticket API does not offer.
• AerticketFastlaneTicketingJob has $tries = 1, so the second delivery is failed by the worker before it reaches the handler. Its guard is therefore in failed(), which refuses to overwrite a ticket_status that says issuance was already requested — otherwise it would record a failure for a ticket being issued.
• CascadeFlightSearchForDateJob likewise never runs twice ($tries = 1), and now has a failed() so an unfinished cascade is reported instead of silently leaving that departure date without fresh flight cache.

JobTimeoutBudgetTest walks every job in app/Jobs/ and fails the build on a new $timeout above retry_after that is not listed with the guard that makes it safe. JobGuardLockSemanticsTest pins the lock semantics against the database cache store, which is what production locks on.

One thing remains open: the cascade will keep producing a spurious MaxAttemptsExceededException (~2/month). Nothing is duplicated and the first run usually completes, but it only stops by moving flight-searches to a connection with a higher retry_after — new workers, so a deployment decision.