Your Laravel Horizon's unique jobs may never run

Laravel's queueing system has the option for unique jobs. These are jobs that can be dispatched only once on the queue.

Implementing these is simple, you just add the ShouldBeUnique interface to your job:

<?php
 
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Contracts\Queue\ShouldBeUnique;
 
class ProcessVideo implements ShouldQueue, ShouldBeUnique
{
    ...
}

The job will get a cache key which is used to acquire a lock

In Illuminate/Foundation/Bus/PendingDispatch.php

    protected function shouldDispatch()
    {
        if (! $this->job instanceof ShouldBeUnique) {
            return true;
        }

        return (new UniqueLock(Container::getInstance()->make(Cache::class)))
                    ->acquire($this->job);
    }

When we look in Redis we see the following entry:

127.0.0.1:6379> keys *
1) "laravel_database_queues:default"
2) "laravel_database_laravel_cache_:laravel_unique_job:App\\Jobs\\ProcessVideo"
3) "laravel_horizon:recent_jobs"
4) "laravel_horizon:7310d76a-de44-4cdf-8f2e-4c09b1805263"

When we try to retrieve the TTL of the cache lock we get -2 as response:

127.0.0.1:6379> ttl laravel_database_laravel_cache_:laravel_unique_job:App\\Jobs\\ProcessVideo
(integer) -2

This means that there is no TTL on the cache lock, source.

But when we retrieve the TTL of the job it is set to 3441 seconds.

127.0.0.1:6379> ttl laravel_horizon:7310d76a-de44-4cdf-8f2e-4c09b1805263
(integer) 3441

What would happen if the TTL of the job expires? The job will never dispatch again until the cache is cleared!

So where does the TTL of the job come from?

In Horizon's config file, config/horizon.php there is a trim section

   /*
    |--------------------------------------------------------------------------
    | Job Trimming Times
    |--------------------------------------------------------------------------
    |
    | Here you can configure for how long (in minutes) you desire Horizon to
    | persist the recent and failed jobs. Typically, recent jobs are kept
    | for one hour while all failed jobs are stored for an entire week.
    |
    */

    'trim' => [
        'recent' => 60,
        'pending' => 60,
        'completed' => 60,
        'recent_failed' => 10080,
        'failed' => 10080,
        'monitored' => 10080,
    ],

This means that jobs have 60 minutes to execute before being deleted.

We can test this out by setting the trim to one minute and then dispatching the job. If we look in Redis and keep track of the TTL and the keys we see that our job disappears after one minute but our cache lock stays.

127.0.0.1:6379> keys *
1) "laravel_database_queues:default"
2) "laravel_database_laravel_cache_:laravel_unique_job:App\\Jobs\\ProcessVideo"
3) "laravel_horizon:recent_jobs"
4) "laravel_horizon:pending_jobs"
5) "laravel_database_queues:default:notify"
6) "laravel_horizon:d7c26c51-94ac-4883-8f8d-bdaa02e7fa7c"
127.0.0.1:6379> ttl laravel_horizon:d7c26c51-94ac-4883-8f8d-bdaa02e7fa7c
(integer) 50
127.0.0.1:6379> ttl laravel_horizon:d7c26c51-94ac-4883-8f8d-bdaa02e7fa7c
(integer) 24
127.0.0.1:6379> ttl laravel_horizon:d7c26c51-94ac-4883-8f8d-bdaa02e7fa7c
(integer) -2
127.0.0.1:6379> keys *
1) "laravel_database_queues:default"
2) "laravel_database_laravel_cache_:laravel_unique_job:App\\Jobs\\ProcessVideo"
3) "laravel_horizon:recent_jobs"
4) "laravel_horizon:pending_jobs"
5) "laravel_database_queues:default:notify"

When we try to dispatch our job after this and list all the keys in Redis we are missing our job because of the lock

127.0.0.1:6379> keys *
1) "laravel_database_queues:default"
2) "laravel_database_laravel_cache_:laravel_unique_job:App\\Jobs\\ProcessVideo"
3) "laravel_horizon:recent_jobs"
4) "laravel_horizon:pending_jobs"
5) "laravel_database_queues:default:notify"

Only after flushing the Redis DB we can dispatch our job again.

Is this an issue? No, this is intended behaviour. But it can give some strange results if you do now know what is happening.

The Fix

There are a few things you can do to fix this:

• Rewrite your logic so that all jobs execute within the trim time by for example chunking a part of the jobs every X minutes

• Increase the Horizon workers so that more jobs get processed

• Increase the trim time