Michael Raypold

Influential Books

Thu, 15 Jul 2021 16:20:58 -0700

The following are a few of the books which I’ve found impactful in my day-to-day thinking, work and career development since graduating from university.

Measure What Matters: How Google, Bono, and the Gates Foundation Rock the World with OKRs

Takeaways:

OKRs are incredibly valuable in their ability to focus an organization or team, but poorly constructed OKRs can do more damage than good.
Clear goals and measurable results are a focus mechanism that help an organization scale out.
Use OKRs as an opportunity to set stretch goals and aspirational targets instead of sticking with the status quo.
OKRs are an opportunity to communicate the why alongside the what.

The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

Of all the books in this list, The DevOps Handbook is the single most influential book and has been a guiding hand in how I attempt to organize engineering processes and teams.

Takeaways:

Optimize delivery for rapid and small releases to make it easier to debug issues.
Structure your teams such that they own and maintain the code you ship to production.
You don’t “hand off to ops” after you’ve built the feature.
Closely follow and optimize for cycle time and lead time.
DevOps is a culture not a job title.

An Elegant Puzzle: Systems of Engineering Management

This is a quick read full of useful learnings over Will Larson’s career.

I would recommend it for new managers.

Takeaways:

Always be recruiting and have a hiring funnel.
Inbound recruiting can be difficult without a strong company brand.
You should build up referral networks over the course of your career.
Learn to manage and present upwards.
Finish fully staffing one team before attempting to build a second half staffed team.

Powerful: Building a Culture of Freedom and Responsibility

I genuinely love this book and revisit it often to get an idea for how Patty McCord helped recruit, retain and motivate talent at Netflix.

“A business leader’s job is to create great teams that do amazing work on time. That’s it. That’s the job of management”.
You need to be great at hiring to also be great at letting people go.
Employees are always looking for challenging opportunities and interesting problems to solve.
Like the above, people want to learn. Don’t optimize for office perks.
Challenge your team(s).
Everyone should understand the business.
Be as transparent as possible about the success and failures of the business.
Annual performance bonuses won’t necessarily make people work harder.
Whenever possible remove structure, policies, and process to move the authority to the team. This is very similar to the advice in Turn the Ship Around.

The Phoenix Project

A fun quick read.

Takeaways:

Interpersonal communication skills are as important as coding.
Small changes can snowball across an organization, so chaining together a lot of small wins using a bottom-up approach can be very effective.

Turn the Ship Around! A True Story of Turning Followers Into Leaders

Another quick read with parallels to engineering leadership or leadership more generally.

Takeaways:

To grant authority to mid-organization teams you need to provide them with clarity and competence.
Speak with intent instead of asking for permission.
Move from a leader-follower model to a leader-leader model.

Inspired

A good reference when you are asking yourself what to do as new product owner.

Takeaways:

Product roadmaps are not necessarily the most effective way of communicating product strategy or long-term needs.

The Hard Thing about Hard Things

One of the earliest books that I read when first promoted to CTO.

Ben’s new book What You Do Is Who You Are: How to Create Your Business Culture is also very good.

Takeaways:

Peacetime CEO vs Wartime CEO
Building tech companies is not as romanticized as it’s made out to be.

Laravel Queues in Production

Sat, 20 Apr 2019 16:58:21 -0700

The Laravel queue and event system is one of the most powerful features of the framework and indispensable if you want to build a modern web application.

The queue provides a performant means to run background tasks that don’t block your HTTP lifecycle, allowing you to serve up a responsive application to your customers. Typically, you would use the queue to send emails, collect metrics or generate reports, amongst a number of other tasks that are either time consuming, prone to failure or can be done in an asynchronous manner that don’t impede the user’s workflow.

Having now used the Laravel event system to queue up listeners as well as background jobs for the last few years, I’ve written up some gotchas and best practices for others to learn from.

Use Multiple Queues and Prioritize Them

By default, Laravel assigns all jobs and event listeners to the default queue. This is typically fine for a low volume web application and quick jobs or event listeners.

However, sometimes you will need to prioritize jobs on the queue so that they are executed before others.

As an example, let’s consider a project planning board like Jira or Asana. When a user is logged into the application, we want to push events down the websocket using the queue. Similarly, the queue is also used to send emails to notify users of changes to the project card/issue.

In the above example, the websocket queue should take priority over emails, as the user is currently logged into the application and would like real time updates while they are interacting with the project planning board, instead of jumping back and forth between their email client to find out every time a co-worker has made an edit or an update.

There are a couple ways to accomplish this in Laravel. You can dispatch the job like so

WebsocketMessenger::dispatch()->onQueue(‘websocket’);

Or update the class itself with a ‘queue’ variable.

class WebsocketMessenger implements ShouldQueue
{

 /**
 * The name of the queue the job should be sent to.
 *
 * @var string|null
 */
 public $queue = 'websocket';
}

I would strongly recommend using enums for both of the above examples so that you can easily rename queues in the future with a single line change.

/**
 * Class Queues
 *
 * References which queue a queueable a class belongs on.
 *
 * @package App\Utils\Enums
 */
abstract class Queues
{

 /**
 * The queue for websocket events/jobs.

 * @var string
 */
 const WEBSOCKET = 'websocket';
}

WebsocketMessenger::dispatch()->onQueue(Queues::WEBSOCKET);

You then prioritize the queue:work command like so.

php artisan queue:work --queue=websocket,email,default

The queue will process everything in the websocket queue before attempting to process the email queue and finally anything on the default queue.

Starvation

The above code example has a potential problem with resource starvation if your websocket queue is always processing events/jobs, preventing the emails from ever sending because it cannot get a free executor slot.

This can be difficult to spot in production without correct monitoring and may take the form of your Redis instance eventually exhausting all memory and throwing an error in your logs like follows.

Laravel-queue-worker INFO OOM command not allowed when used memory > 'maxmemory'.

As an aside, if you don’t have production alarms set up for Redis memory utilization, you should.

If you chose not to use Laravel Horizon, which will prevent the starvation problem, or some other appropriate solution that will automatically scale your queue runners based on queue size, you can work around this problem through editing your supervisor config like follows to guarantee there is always at least one exectuor running for each queue.

[program:laravel-worker-queue]
process_name=%(program_name)s_%(process_num)02d
command=php artisan queue:work --queue=websocket,email,default --daemon
numprocs=4

[program:laravel-worker-websocket]
process_name=%(program_name)s_%(process_num)02d
command=php artisan queue:work --queue=websocket --daemon
numprocs=1

[program:laravel-worker-email]
process_name=%(program_name)s_%(process_num)02d
command=php artisan queue:work --queue=email --daemon
numprocs=1

[program:laravel-worker-default]
process_name=%(program_name)s_%(process_num)02d
command=php artisan queue:work --queue=default --daemon
numprocs=1

I’ve ignored additional configuration parameters in the supervisor config as those will be specific to your system.

Supervisor

If you are using supervisor to wrap php artisan queue:work, as suggested in the Laravel documentation, you need to be careful to monitor your production system for supervisor failures.

A simple way to do this is to have your logging services send out a notification anytime it finds a string like gave up or entered FATAL state, too many start retries too quickly.

Failure to catch this error can result in your job queue not being processed.

Admittedly this is an exceptionally rare occurrence, but I’ve seen it happen when our multi availability zone database had to swap over from one zone to the other, resulting in all Laravel database connections being dropped for a couple seconds. During these few seconds, supervisor attempted to restart the process and eventually gave up. The database was offline for less than 5 seconds, an almost inconsequential amount of time, but we didn’t detect the supervisor issues for another couple hours due to poor job queue telemetry (since rectified).

To save yourself a similar headache, you can wrap your job queue in a bash script that delays the restarts.

Docker

It might be tempting to install supervisor within your docker image and have it manage the Laravel queue.

Don’t do that.

Docker works best when it runs a single isolated process. It’s much more difficult to catch misbehaving supervisor process that fan out multiple queue workers when they are within a docker container.

Your container orchestration system will bring the container back up if the queue runner exits, and you have the added benefit of being able to precisely specify the memory and CPU allocation for each job queue process within Kubernetes to better solve the bin packing problem.

Instead of using supervisor, you are better off setting up a CMD instruction in your Dockerfile like follows.

CMD php artisan queue:work --queue=default --sleep=1 --tries=5 –-daemon

Furthermore, I would strongly recommend doing some profiling on your jobs so you can better set memory and CPU limits on your kubernetes resources.

Having statsd collect memory_get_peak_usage(true) over time will allow you make better utilization of your server resources.

If you are using Redis for both your cache and queue, you should run them on separate connections and instances.

The first reason is partially outlined above. If you get into a scenario where you have process starvation on the job queue, you may run out of Redis memory, causing both the cache and queue to become inoperable.

The second reason is that for some unforeseen reason, you may one day need to flush your application cache. Perhaps there are some keys that are not expiring which are causing issues, or you have a bug in your code which is causing extreme writes to the cache. You can now safely flush the cache without losing unprocessed jobs.

In short, you will get better isolation and performance.

Config Changes

If you are running a multi-tenant system, you might be dynamically loading different config values within the job queue classes that you are currently executing.

This can be a huge source of unexpected bugs because Laravel wont reboot the framework while running the queue in --daemon mode, thus not resetting the config to the default configured values like would occur during a normal HTTP request/response lifecycle.

You have two solutions here:

Manually reset the config before the job ends. However you have to be careful about exceptions being thrown prior to the “reset”, thus never hitting your reset block and causing further state issues.
Don’t run the queue in --daemon mode which incurs a performance hit.

Logging

Here is a simple hack that I find makes it easier to debug system issues.

Laravel by default will store failed jobs in the database, however you may find it easier to dig through the logs for this information, so you can place the following code in your AppServiceProvider.

/**
 * Log failed queue events so that we have more system visibility.
 *
 * @link https://laravel.com/docs/5.8/queues#dealing-with-failed-jobs
 */
Queue::failing(function (JobFailed $event) {
 \Log::error('Queue Failure', [
 'connectionName' => $event->connectionName,
 'queue' => $event->job->getQueue(),
 'exceptionMessage' => $event->exception->getMessage(),
 'exception' => $event->exception->getTrace(),
 ]);
});

Be careful about logging personally identifiable information with the above solution; especially if you are forwarding your logs to an external logging provider.

Conclusion

I would encourage everyone to give the Laravel Queue and Horizon documentation a thorough read.

The queue system is surprisingly powerful and can be used in high throughput environments.

Reach out if you have any questions.

Laravel Distributed Maintenance Mode

Sat, 07 Oct 2017 06:50:24 -0700

The Laravel maintainers have begun to invest more time in providing features and tooling for managing Laravel in a distributed environment.

Laravel 5.4 modified the withoutOverlapping() option for kernel tasks (PR #16196) to use the cache to create a lock and prevent overlapping scheduled tasks across instances.

Laravel Horizon makes it easier to monitor and maintain Laravel powered Redis queues in both a distributed and single instance deployment.

Laravel 5.5 includes TrustedProxy so that you can more easily place your application behind a load balancer.

What is still missing, is the ability to put the application into maintenance mode when running multiple Laravel instances behind a load balancer.

If you don’t need to put your app in maintenance mode, but simply need to prevent people from accessing the site for a short time, the easiest option might be to point your domain to a static maintenance page for a while. For example, you could use Route53 to redirect traffic to an s3 bucket containing a maintenance page.

Why Do We Need Maintenance Mode

Maintenance mode displays a maintenance page to visitors as well as prevents queued jobs from running.

Therefore, if you are using the Laravel job queue or scheduled tasks, simply pointing your domain to a static maintenance page is not an ideal solution, as background jobs will continue to run and interact with your database and cache.

If you have unexpected long running migration, or you need to perform emergency maintenance on a Laravel deployment that has background tasks, re-routing traffic to a static maintenance page is not an ideal solution.

How Does Maintenance Mode Work

The built in php artisan down command that puts the app into maintenance mode, places a temporary file in the app’s storage directory at /framework/down. You can find the code for the DownCommand code here.

When a request is processed by Laravel, it is either handled by the HTTP or console kernel.

The HTTP kernel is generally the bulk of your application requests and passes through the global HTTP middleware and middleware groups before being sent to your routing files.

The console kernel is used for any scheduled tasks you define.

Both kernels will pass through a CheckForMaintenanceMode middleware that checks if the application is currently in maintenance mode, which then throws an exception that is handled appropriately in the app\Excpetions\Handler.php file.

/**
 * Handle an incoming request.
 *
 * @param \Illuminate\Http\Request $request
 * @param \Closure $next
 * @return mixed
 *
 * @throws \Symfony\Component\HttpKernel\Exception\HttpException
 */
 public function handle($request, Closure $next)
 {
 if ($this->app->isDownForMaintenance()) {
 $data = json_decode(file_get_contents($this->app->storagePath().'/framework/down'), true);
 throw new MaintenanceModeException($data['time'], $data['retry'], $data['message']);
 }
 return $next($request);
 }

source CheckForMaintenanceMode.php

Similar checks are made in the Worker.php and the QueueManager.php files.

Finally, the isDownForMaintenance() command is handled in your application container.

/**
 * Determine if the application is currently down for maintenance.
 *
 * @return bool
 */
 public function isDownForMaintenance()
 {
 return file_exists($this->storagePath().'/framework/down');
 }

source Application.php

Why the Current Solution Doesnt’t Work in a Distributed Environment

If your storage directory is shared across all your deployed instances, the current solution is fine.

For many deployments, we don’t share storage directories across all instances because we don’t handle file uploads.

It is also possible that your job queue is managed on a different server than that which is serving your webapp.

Solving Distributed Maintenance Mode.

Thankfully, we can follow a similar approach outlined in PR #16196 where a lock is checked to confirm that the kernel command is not already running elsewhere.

By using the cache, we can set a maintenance mode lock in Redis that is checked by all deployed instances before continuing the request lifecycle.

Plan of Attack

Create a maintenance mode enum.
Extend the php artisan down command and override the handle function.
Extend the php artisan up command and override the handle function.
Register the commands in the service provider and kernel.
Extend the application and override the isDownForMaintenance function.
Extend the CheckForMaintenanceMode middleware and override the handle function.

Creating the Enums

The first thing that we will do is create some enums for common string based cache keys that will be used throughout the maintenance mode code.

The reason we’re using enums is so that we can easily update the cache key without having to hunt down and potentially miss references throughout the application if we ever want to update the keys in the future.

Here, I’m just using an abstract class, but you could alternatively put these in a config file.

I find this approach cleaner and use it frequently.

<?php

namespace App\Utils\Enums;

abstract class CacheKeys
{
 /**
 * @var string A cache accessor for maintenance mode
 */
 const MAINTENANCE_PAYLOAD = 'laravel-maintenance-mode-payload';
}

Extending Artisan Down

Next, extend Laravel’s down command to use the cache instead of the filesystem.

Notice that CacheKeys::MAINTENANCE_PAYLOAD references the enum we created above. This will be used in all the extensions that are created.

<?php

namespace App\Console\Commands;

use App\Utils\Enums\CacheKeys;
use Cache;
use Illuminate\Foundation\Console\DownCommand as Down;

class DownCommand extends Down
{
 /**
 * Execute the console command.
 *
 * Overrides Laravel's DownCommand to use the cache instead of the
 * filesystem so that maintenance mode is propagated across all
 * servers and job queues.
 *
 * @return mixed
 */
 public function handle()
 {
 Cache::forever(CacheKeys::MAINTENANCE_PAYLOAD, json_encode($this->getDownFilePayload()));
 $this->comment('Application is now in maintenance mode.');
 }
}

The $this->getDownFilePayload serializes the options passed into the Artisan command at runtime and is in the original Down class.

/**
 * Get the payload to be placed in the "down" file.
 *
 * @return array
 */
protected function getDownFilePayload()
{
 return [
 'time' => $this->currentTime(),
 'message' => $this->option('message'),
 'retry' => $this->getRetryTime(),
 ];
}

If you wish, you can use the retry field to have the cache automatically purge the key/value after the given period of time and take the application out of maintenance mode.

Extending Artisan Up

Now, extend Laravel’s up command to purge the cache key that is being used as a lock, thus taking the application out of maintenance mode.

<?php

namespace App\Console\Commands;

use App\Utils\Enums\CacheKeys;
use Cache;
use Illuminate\Foundation\Console\UpCommand as Up;

class UpCommand extends Up
{
 /**
 * Execute the console command.
 *
 * Overrides Laravel's UpCommand to use the cache instead of the
 * filesystem so that maintenance mode is propagated across all
 * servers and job queues.
 *
 * @return mixed
 */
 public function handle()
 {
 Cache::forget(CacheKeys::MAINTENANCE_PAYLOAD);
 $this->info('Application is now live.');
 }
}

Register the New Commands

Now that we have extended the necessary core Laravel commands, we need to tell the application that when it is bootstrapped, it should use the modified commands instead of the Illuminate commands that ship with the framework.

This can easily be accomplished in the AppServiceProvider boot method.

/**
 * Bootstrap any application services.
 *
 * @return void
 */
 public function boot()
 {
 /**
 * Override Laravel's "php artisan down" command to put the application in maintenance mode
 * using our custom Redis based lock.
 */
 $this->app->extend('command.down', function () {
 return new DownCommand();
 });

 /**
 * Override Laravel's "php artisan up" command to bring the application out of maintenance mode
 * using our custom Redis based lock.
 */
 $this->app->extend('command.up', function () {
 return new UpCommand();
 });
}

Remember to import the commands.

use App\Console\Commands\DownCommand;
use App\Console\Commands\UpCommand;

You will also need to add the commands to the console kernel like you do all other commands.

<?php

namespace App\Console;

use App\Console\Commands\DownCommand;
use App\Console\Commands\UpCommand;
use Illuminate\Foundation\Console\Kernel as ConsoleKernel;

class Kernel extends ConsoleKernel
{
 /**
 * The Artisan commands provided by your application.
 *
 * @var array
 */
 protected $commands = [
 DownCommand::class,
 UpCommand::class,
 ];

 /**
 * Register the Closure based commands for the application.
 *
 * @return void
 */
 protected function commands()
 {
 require base_path('routes/console.php');
 }
}

Extend Laravel’s Application Class

Now that we are using the cache instead of the filesystem to store the maintenance mode lock, the framework needs to be directed to the new location so that it knows when the application is in maintenance mode.

First, extend the core Application class so that we can override Laravel’s isDownForMaintenance method that is currently checking the filesystem for the maintenance lock.

I’ve created an App\Extensions\Illuminate\Foundation directory in an attempt to closely mirror the framework.

<?php

namespace App\Extensions\Illuminate\Foundation;

use App\Utils\Enums\CacheKeys;
use Cache;
use Illuminate\Foundation\Application as App;

class Application extends App
{
 /**
 * Returns whether the application is down for maintenance.
 *
 * Laravel maintenance mode is overridden to use the cache instead of the filesystem,
 * allowing us to treat Redis as a global distributed lock so that
 * maintenance mode is propagated to multiple servers.
 *
 * @return bool Whether the app is in maintenance mod.
 */
 public function isDownForMaintenance()
 {
 return Cache::has(CacheKeys::MAINTENANCE_PAYLOAD);
 }
}

This next step that needs to be taken when extending the application is to change how the web application is bootstrapped.

You will find the necessary file to modify in bootstrap/app.php.

The $app variable is now using our extended Application class created above, and thus anytime isDownForMaintenance is called, the cache will be checked for the existence of the maintenance mode lock instead of the storage directory.

/*
|--------------------------------------------------------------------------
| Create The Application
|--------------------------------------------------------------------------
|
| The first thing we will do is create a new Laravel application instance
| which serves as the "glue" for all the components of Laravel, and is
| the IoC container for the system binding all of the various parts.
|
*/

$app = new App\Extensions\Illuminate\Foundation\Application(
 realpath(__DIR__.'/../')
);

Extending Maintenance Mode Middleware

The final step that needs to be taken is to extend the maintenance mode middleware that ships with the framework.

This middleware is checked every time a request comes into the application. We want it to now query the cache lock that we have created instead of the storage directory.

<?php

namespace App\Http\Middleware;

use App\Utils\Enums\CacheKeys;
use Cache;
use Closure;
use Illuminate\Foundation\Http\Exceptions\MaintenanceModeException;
use Illuminate\Foundation\Http\Middleware\CheckForMaintenanceMode as MaintenanceMode;

class CheckForMaintenanceMode extends MaintenanceMode
{

 /**
 * Handle an incoming request.
 *
 * If the application is in maintenance mode, we retrieve the payload from the cache containing
 * the user set message and time to display on the error page when the MaintenanceModeException
 * is thrown.
 *
 * @param \Illuminate\Http\Request $request
 * @param \Closure $next
 * @return mixed
 */
 public function handle($request, Closure $next)
 {
 if ($this->app->isDownForMaintenance()) {
 $data = json_decode(Cache::get(CacheKeys::MAINTENANCE_PAYLOAD), true);

 throw new MaintenanceModeException($data['time'], $data['retry'], $data['message']);
 }

 return $next($request);
 }
}

Laravel uses the \Illuminate\Foundation\Http\Middleware\CheckForMaintenanceMode::class in the HTTP kernel global middleware grouping which needs to be replaced with our new custom middleware as such.

<?php

use App\Http\Middleware\CheckForMaintenanceMode;

class Kernel extends HttpKernel
{
 /**
 * The application's global HTTP middleware stack.
 *
 * These middleware are run during every request to your application.
 *
 * @var array
 */
 protected $middleware = [
 \Fideloper\Proxy\TrustProxies::class,
 CheckForMaintenanceMode::class,
 \Illuminate\Foundation\Http\Middleware\ValidatePostSize::class,
 \App\Http\Middleware\TrimStrings::class,
 ];
}

You should now be able to call php artisan down and php artisan up to bring the application in and out of maintenance mode!

Tests

Finally, its always a good idea to write tests for your code so that you can refactor in the future with more confidence.

I tend to write tests as I code, instead of before, to verify that something is working before I go manually test in the browser.

There are a couple different ways you can approach testing this feature.

The first is a straight forward integration test where you put the application into maintenance mode and then assert that the user cannot access the home route. That might look like the following.

public function test_home_should_not_be_accessible_in_maintenance_mode()
{
 // Arrange
 Artisan::call('down');

 $user = factory(User::class)->create();

 // Act && Assert
 $this
 ->actingAs($user)
 ->get(route('home'))
 ->assertStatus(503);
}

These next tests are arguably less valuable, but might give you more confidence when working with the codebase in the future.

public function test_isDownForMaintenance_returns_true_when_in_maintenance_mode()
{
 // Arrange
 Cache::forever(CacheKeys::MAINTENANCE_PAYLOAD, true);

 // Act && Assert
 $this->assertTrue($this->app->isDownForMaintenance());
}

public function test_isDownForMaintenance_returns_false_when_not_in_maintence_mode()
{
 // Arrange
 Cache::forget(CacheKeys::MAINTENANCE_PAYLOAD);

 // Act && Assert
 $this->assertFalse($this->app->isDownForMaintenance());
}

There are many more things you could test. For example, the middleware is passing through to the next middleware in the chain when the application is not in maintenance mode or that the Up and Down commands are setting the appropriate fields in the cache.

Production Environment Performance Difference

Unfortunately, I can’t give a true apples to apples comparison as this was not tested in a controlled environment. The deployment of this upgrade also enabled the OPcache. If you aren’t already using the OPcache, I highly recommend doing some reading on it and enabling it on your production servers.

That being said, our production servers (m4.large) currently have a latency of around 80ms. Your performance will differ based on where your cache is located, the size of your server, php version, etc…

What You Should Be Aware Of

Finally, be aware that if you are using AWS Elastic Load Balancer with health checks, or a similar service, putting your app into maintenace mode will likely cause your health checks to fail.

You should always dry run these type of changes in a staging environment, which thankfully we did.

Shortly after putting the application into maintenance mode, health checks to our Laravel status endpoints began to return a 503 status code, causing the ELB to take the instances out of the rotation.

In hindsight, this should have been obvious.

A solution is to exempt your status check endpoints from the CheckForMaintenanceMode middleware, which gets run on every request.

To do this, you will want to follow a similar approach to how Laravel’s VerifyCsrfToken middleware works.

class VerifyCsrfToken
{
 /**
 * The URIs that should be excluded from CSRF verification.
 *
 * @var array
 */
 protected $except = [];

 /**
 * Handle an incoming request.
 *
 * @param \Illuminate\Http\Request $request
 * @param \Closure $next
 * @return mixed
 *
 * @throws \Illuminate\Session\TokenMismatchException
 */
 public function handle($request, Closure $next)
 {
 if (
 $this->isReading($request) ||
 $this->runningUnitTests() ||
 $this->inExceptArray($request) ||
 $this->tokensMatch($request)
 ) {
 return $this->addCookieToResponse($request, $next($request));
 }

 throw new TokenMismatchException;
 }

 /**
 * Determine if the request has a URI that should pass through CSRF verification.
 *
 * @param \Illuminate\Http\Request $request
 * @return bool
 */
 protected function inExceptArray($request)
 {
 foreach ($this->except as $except) {
 if ($except !== '/') {
 $except = trim($except, '/');
 }

 if ($request->is($except)) {
 return true;
 }
 }

 return false;
 }
}

Copy the inExceptArray() function to your CheckForMaintenaceMode middleware, which you can then check in the handle() function in a similar fashion to the VerifyCsrfToken middleware above.

Conclusion

We moved from a filesystem based maintenance mode lock to one stored in a Redis cache.

If you aren’t using Redis or a distributed cache, you can still follow the approach outlined above, just be aware that maintenance mode won’t propogate across all your servers.

Whether you should use this approach, over pointing your load balancer to a static maintenance page, I leave up to you.

Michael Raypold

Influential Books

Measure What Matters: How Google, Bono, and the Gates Foundation Rock the World with OKRs

The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

An Elegant Puzzle: Systems of Engineering Management

Powerful: Building a Culture of Freedom and Responsibility

The Phoenix Project

Turn the Ship Around! A True Story of Turning Followers Into Leaders

Inspired

The Hard Thing about Hard Things

Laravel Queues in Production

Use Multiple Queues and Prioritize Them

Starvation

Supervisor

Docker

Sharing Cache and Queue Instances (Don’t)

Config Changes

Logging

Conclusion

Laravel Distributed Maintenance Mode

Why Do We Need Maintenance Mode

How Does Maintenance Mode Work

Why the Current Solution Doesnt’t Work in a Distributed Environment

Solving Distributed Maintenance Mode.

Plan of Attack

Creating the Enums

Extending Artisan Down

Extending Artisan Up

Register the New Commands

Extend Laravel’s Application Class

Extending Maintenance Mode Middleware

Tests

Production Environment Performance Difference

What You Should Be Aware Of

Conclusion