opdavies/talks

Fork 0

Oliver Davies 14781d3d6f Restructure files

2017-10-30 01:08:00 +00:00

16 KiB

Raw Blame History

autoscale: true build-lists: true theme: next, 9

[fit] Deploying Drupal
with Fabric

Oliver Davies

bit.ly/deploying-drupal-fabric

[.build-lists: false]

What is Fabric and what do I use it for?
How to write and organise Fabric scripts
Task examples

[.build-lists: false]

Senior Developer at Microserve
Part-time freelance Developer & System Administrator
Drupal Bristol, PHPSW, DrupalCamp Bristol organiser
Sticker collector, herder of elePHPants
@opdavies
oliverdavies.uk

^ Drupal, Symfony, Silex, Laravel, Sculpin Not a Python Developer

[fit] What is
Fabric?

What is Fabric?

Fabric is a Python (2.5-2.7) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.

What is Fabric?

It provides a basic suite of operations for executing local or remote shell commands (normally or via sudo) and uploading/downloading files, as well as auxiliary functionality such as prompting the running user for input, or aborting execution.

I use Fabric to...

Simplify my build process
Deploy code directly to different environments
Act as an intermediate step

[.build-lists: false]

Why Fabric?

Powerful
Flexible
Easier to read and write than bash

^ - Can be used for different languages, frameworks

Can be used for small, simple deployments or large, complicated ones
Not very opinioned

Installing Fabric

$ pip install fabric

# macOS
$ brew install fabric

# Debian, Ubuntu
$ apt-get install fabric
$ apt-get install python-fabric

[fit] Writing your
first fabfile

# fabfile.py

from fabric.api import env, run, cd, local

env.hosts = ['example.com']

# Do stuff...

^ Callables

# fabfile.py

from fabric.api import *

env.hosts = ['example.com']

# Do stuff...

Operations

cd, lcd - change directory
run, sudo, local - run a command
get - download files
put - upload files

[.footer: http://docs.fabfile.org/en/1.13/api/core/operations.html]

Utils

warn: print warning message
abort: abort execution, exit with error status
error: call func with given error message
puts: alias for print whose output is managed by Fabric's output controls

[.footer: http://docs.fabfile.org/en/1.13/api/core/utils.html]

File management

from fabric.contrib.files import *

exists - check if path exists
contains - check if file contains text/matches regex
sed - run search and replace on a file
upload_template - render and upload a template to remote host

[.footer: http://docs.fabfile.org/en/1.13/api/contrib/files.html#fabric.contrib.files.append]

^ Allows for jinja2 templates

Tasks


def main():
  with cd('/var/www/html'):
    run('git pull')
    run('composer install')

Task arguments

def main(run_composer=True):
  with cd('/var/www/html'):
    run('git pull')

    if run_composer:
      run('composer install')

Task arguments

def main(run_composer=True, env='prod', build_type):
  with cd('/var/www/html'):
    run('git pull')

    if run_composer:
      if env == 'prod':
        run('composer install --no-dev')
      else:
        run('composer install')

      if build_type == 'drupal':
        ...
      elif build_type == 'symfony':
        ...
      elif build_type == 'sculpin':
        ...

Calling other tasks

@task
def main():
  with cd('/var/www/html'):
    build()
    post_install()

def build():
  run('git pull')
  run('composer install')

def post_install():
  with prefix('drush'):
    run('updatedb -y')
    run('entity-updates -y')
    run('cache-rebuild')

^ Better organised code Not everything in one long Easier to read and comprehend Single responsibilty principle

Running Tasks

fab --list

fab <task>

fab <task>:build_number=$BUILD_ID,build_type=drupal

^ fabfile.py in the current directory is found automatically.

[production] Executing task 'main'
[production] run: git pull
[production] out: Already up-to-date.
[production] out:

[production] run: composer install
...
[production] out: Generating autoload files
[production] out:

Done.
Disconnecting from production... done.

Downsides

Running build tasks on production

[.build-lists: false]

Not Building on Prod

Build locally and deploy.

Local tasks

# Runs remotely.

from fabric.api import run

run('git pull')
run('composer install')

# Runs locally.

from fabric.api import local

local('git pull')
local('composer install')

Local tasks

# Remote.

from fabric.api import cd

with cd('themes/custom/drupalbristol'):
  ...

# Runs locally.

from fabric.api import lcd

with lcd('themes/custom/drupalbristol'):
  ...

rsync

from fabric.contrib.project import rsync_project

...

def deploy():
  rsync_project(
    local_dir='./',
    remote_dir='/var/www/html'
    default_opts='-vzcrSLh',
    exclude=('.git', 'node_modules/', '.sass-cache/')
  )

[production] Executing task 'main'
[localhost] local: git pull
Current branch master is up to date.
[localhost] local: composer install
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Nothing to install or update
Generating autoload files

Done.

^ - The risky steps have been run separate to the live code.

Any issues will not affect the live site.

[.build-lists: false]

Not Building on Prod

~~Build locally and deploy.~~
Build in a separate directory and switch after build.

^ Capistrano

Deploying into a different directory

from fabric.api import *
from time import time

project_dir = '/var/www/html'
next_release = "%(time).0f" % { 'time': time() } # Current timestamp

def init():
  if not exists(project_dir):
    run('mkdir -p %s/backups' % project_dir)
    run('mkdir -p %s/shared' % project_dir)
    run('mkdir -p %s/releases' % project_dir)

Deploying into a different directory

current_release = '%s/%s' % (releases_dir, next_release)

run('git clone %s %s' % (git_repo, current_release))

def build():
  with cd(current_release):
    pre_tasks()
    build_site()
    post_tasks()

^ - Clone the repository into a different directory

Run any "risky" tasks away from the production code

Deploying into a different directory

def pre_build(build_number):
  with cd('current'):
    print '==> Dumping the DB (just in case)...'
    backup_database()

def backup_database():
  cd('drush sql-dump --gzip > ../backups/%s.sql.gz' % build_number)

Deploying into a different directory

def update_symlinks():
  run('ln -nfs %s/releases/%s %s/current'
    % (project_dir, next_release, project_dir))

# /var/www/html/current

[production] Executing task 'main'
[production] run: git clone https://github.com/opdavies/oliverdavies.uk.git
  /var/www/html/releases/1505865600
Installing Composer dependencies...
[production] run: composer install --no-dev
Update the symlink to the new release...
[production] run: ln -nfs /var/www/html/releases/1505865600
  /var/www/html/current

Done.

# /var/www/html

shared # settings.local.php, sites.php, files etc.
releases/1502323200
releases/1505692800
releases/1505696400
releases/1505865600
current -> releases/1505865600 # symlink

[.build-lists: false]

Positives

Errors happen away from production

Downsides

Lots of release directories

^ If the build fails, then your live site is not affected.

Removing old builds

def main(builds_to_keep=3):
  with cd('%s/releases' % project_dir):
    run("ls -1tr | head -n -%d | xargs -d '\\n' rm -fr"
      % builds_to_keep)

^ - Find directory names

1tr not ltr
Remove the directories
Additional tasks, removing DB etc

[fit] Is the site
still running?

Checking for failures

run(command).return_code == 0:
  # Pass

run(command).return_code == 1:
  # Fail

run(command).failed:
  # Fail

^ Works for local and remote.

print 'Checking the site is alive...'
if run('drush status | egrep "Connected|Successful"').failed:
  # Revert back to previous build.

^ egrep is an acronym for "Extended Global Regular Expressions Print". It is a program which scans a specified file line by line, returning lines that contain a pattern matching a given regular expression.

$ drush status

Drupal version                  :  8.3.7
Site URI                        :  http://default
Database driver                 :  mysql
Database hostname               :  db
Database username               :  user
Database name                   :  default
Database                        :  Connected
Drupal bootstrap                :  Successful
Drupal user                     :
Default theme                   :  bartik
Administration theme            :  seven
PHP configuration               :  /etc/php5/cli/php.ini
...

^ Successful

$ drush status

Drupal version                  :  8.3.7
Site URI                        :  http://default
Database driver                 :  mysql
Database hostname               :  db
Database username               :  user
Database name                   :  default
PHP configuration               :  /etc/php5/cli/php.ini
...

^ Failed. "Database" to "PHP configuration" missing if cannot connect or DB is empty.

[fit] Does the code still
merge cleanly?

^ Pre-task

def check_for_merge_conflicts(target_branch):
  with settings(warn_only=True):
    print('Ensuring that this can be merged into the main branch.')

    if local('git fetch && git merge --no-ff origin/%s'
      % target_branch).failed:
      abort('Cannot merge into target branch.')

^ Define target branch in Jenkins/when Fabric is run.

[fit] Do our tests
still pass?

[.footer: http://nashvillephp.org/images/testing-workshop-banner-1600x800.jpg]

with settings(warn_only=True):
  with lcd('%s/docroot/core' % project_dir):
    if local('../../vendor/bin/phpunit ../modules/custom').failed:
      abort('Tests failed!')

[localhost] run: ../../vendor/bin/phpunit ../modules/custom
[localhost] out: PHPUnit 4.8.35 by Sebastian Bergmann and contributors.
[localhost] out:
[localhost] out: .......
[localhost] out:
[localhost] out: Time: 1.59 minutes, Memory: 6.00MB
[localhost] out:
[localhost] out: OK (7 tests, 42 assertions)
[localhost] out:


Done.

[localhost] run: ../../vendor/bin/phpunit ../modules/custom
[localhost] out: PHPUnit 4.8.35 by Sebastian Bergmann and contributors.
[localhost] out:
[localhost] out: E
[localhost] out:
[localhost] out: Time: 18.67 seconds, Memory: 6.00MB
[localhost] out:
[localhost] out: There was 1 error:
[localhost] out:
[localhost] out: 1) Drupal\Tests\broadbean\Functional\AddJobTest::testNodesAreCreated
[localhost] out: Behat\Mink\Exception\ExpectationException: Current response status code is 200, but 201 expected.
[localhost] out:
[localhost] out: /var/www/html/vendor/behat/mink/src/WebAssert.php:770
[localhost] out: /var/www/html/vendor/behat/mink/src/WebAssert.php:130
[localhost] out: /var/www/html/docroot/modules/custom/broadbean/tests/src/Functional/AddJobTest.php:66
[localhost] out:
[localhost] out: FAILURES!
[localhost] out: Tests: 1, Assertions: 6, Errors: 1.
[localhost] out:

Warning: run() received nonzero return code 2 while executing '../../vendor/bin/phpunit ../modules/custom/broadbean'!

Fatal error: Tests failed!

Aborting.

[fit] Making Fabric
Smarter

Conditional variables

drupal_version = None

if exists('composer.json') and exists('core'):
  drupal_version = 8
else:
  drupal_version = 7

Conditional tasks

if exists('composer.json'):
  run('composer install')

with cd('themes/custom/example'):
  if exists('package.json') and not exists('node_modules'):
    run('yarn --pure-lockfile')

  if exists('gulpfile.js'):
    run('node_modules/.bin/gulp --production')
  elif exists('gruntfile.js'):
    run('node_modules/.bin/grunt build')

Project settings file

# app.yml

drupal:
  version: 8
  root: web
  config:
    import: yes
    name: sync
    cmi_tools: no
  tests:
    simpletest: false
    phpunit: true
  theme:
    path: 'themes/custom/drupalbristol'
    build:
      type: gulp
      npm: no
      yarn: yes

composer:
  install: true

Project settings file

# fabfile.py

from fabric.api import *
import yaml

config = []

if exists('app.yml'):
  with open('app.yml', 'r') as file:
    config = yaml.load(file.read())

Project settings file

# fabfile.py

if config['composer']['install'] == True:
  local('composer install')

Project settings file

# fabfile.py

if build_type == 'drupal':
  drupal = config['drupal']

Project settings file

# fabfile.py

if build_type == 'drupal':
  drupal = config['drupal']

  with cd(drupal['root']):
    if drupal['version'] == 8:

    if drupal['version'] == 7:

Project settings file

# fabfile.py

if build_type == 'drupal':
  drupal = config['drupal']

  with cd(drupal['root']):
    if drupal['version'] == 8:
      if drupal['config']['import'] == True:
        # Import the staged configuration.
        run('drush cim -y %s' % drupal['config']['name'])

Project settings file

# fabfile.py

if build_type == 'drupal':
  drupal = config['drupal']

  with cd(drupal['root']):
    if drupal['version'] == 8:
      if drupal['config']['import'] == True:
        if drupal['config']['cmi_tools'] == True:
          # Use Drush CMI Tools.
          run('drush cimy -y %s' % drupal['config']['name'])
        else:
          # Use core.
          run('drush cim -y %s' % drupal['config']['name'])

^ - Less hard-coded values

More flexible
No need to use different files for different versions or frameworks
No forked fabfiles per project or lots of conditionals based on the project

Project settings file

# fabfile.py

theme = config['theme']

with cd(theme['path']):
  if theme['build']['gulp'] == True:
    if env == 'prod':
      run('node_modules/.bin/gulp --production')
    else:
      run('node_modules/.bin/gulp')

Project settings file v2

# app.yml

commands:
  build: |
    cd web/themes/custom/drupalbristol
    yarn --pure-lockfile
    npm run prod    

  deploy: |
    cd web
    drush updatedb -y
    drush cache-rebuild -y

^ How do we know when to run Composer or npm? How do we know what commands to run? Can't tell this just by the presence of files.

Project settings file v2

# fabfile.py

# Run build commands locally.
for hook in config['commands'].get('build', '').split("\n"):
  run(hook)

...

# Run deploy commands remotely.
for hook in config['commands'].get('deploy', '').split("\n"):
  run(hook)

^ "run" won't work locally

Other things

Run Drush commands
Run automated tests
Verify file permissions
Restart services
Anything you can do on the command line...

16 KiB Raw Blame History

[fit] Deploying Drupal with Fabric

Oliver Davies

bit.ly/deploying-drupal-fabric

[fit] What is Fabric?

What is Fabric?

What is Fabric?

I use Fabric to...

Why Fabric?

Installing Fabric

[fit] Writing your first fabfile

Operations

Utils

File management

Tasks

Task arguments

Task arguments

Calling other tasks

Running Tasks

Downsides

Not Building on Prod

Local tasks

Local tasks

rsync

Not Building on Prod

Deploying into a different directory

Deploying into a different directory

Deploying into a different directory

Deploying into a different directory

Positives

Downsides

Removing old builds

[fit] Is the site still running?

Checking for failures

[fit] Does the code still merge cleanly?

[fit] Do our tests still pass?

[fit] Making Fabric Smarter

Conditional variables

Conditional tasks

Project settings file

Project settings file

Project settings file

Project settings file

Project settings file

Project settings file

Project settings file

Project settings file

Project settings file v2

Project settings file v2

Other things

Fabric has...

Thanks!

Questions?

@opdavies

oliverdavies.uk

16 KiB

Raw Blame History

[fit] Deploying Drupal
with Fabric

[fit] What is
Fabric?

[fit] Writing your
first fabfile

[fit] Is the site
still running?

[fit] Does the code still
merge cleanly?

[fit] Do our tests
still pass?

[fit] Making Fabric
Smarter