908 lines
16 KiB
Markdown
908 lines
16 KiB
Markdown
autoscale: true
|
|
build-lists: true
|
|
theme: next, 9
|
|
|
|
# [fit] Deploying Drupal <br>with Fabric
|
|
### Oliver Davies
|
|
### bit.ly/deploying-drupal-fabric
|
|
|
|
---
|
|
|
|
[.build-lists: false]
|
|
|
|
- What is Fabric and what do I use it for?
|
|
- How to write and organise Fabric scripts
|
|
- Task examples
|
|
|
|
---
|
|
|
|
[.build-lists: false]
|
|
|
|
|
|
|
|
- Senior Developer at Microserve
|
|
- Part-time freelance Developer & System Administrator
|
|
- Drupal Bristol, PHPSW, DrupalCamp Bristol organiser
|
|
- Sticker collector, herder of elePHPants
|
|
- @opdavies
|
|
- oliverdavies.uk
|
|
|
|
|
|
^ Drupal, Symfony, Silex, Laravel, Sculpin
|
|
Not a Python Developer
|
|
|
|
---
|
|
|
|
|
|
|
|
## [fit] What is <br>Fabric?
|
|
|
|
---
|
|
|
|
## What is Fabric?
|
|
|
|
Fabric is a Python (2.5-2.7) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.
|
|
|
|
---
|
|
|
|
## What is Fabric?
|
|
|
|
It provides a basic suite of operations for executing local or remote shell commands (normally or via sudo) and uploading/downloading files, as well as auxiliary functionality such as prompting the running user for input, or aborting execution.
|
|
|
|
---
|
|
|
|
## I use Fabric to...
|
|
|
|
- Simplify my build process
|
|
- Deploy code directly to different environments
|
|
- Act as an intermediate step
|
|
|
|
---
|
|
|
|
|
|
|
|
---
|
|
|
|
[.build-lists: false]
|
|
|
|
## Why Fabric?
|
|
|
|
- Powerful
|
|
- Flexible
|
|
- Easier to read and write than bash
|
|
|
|
^ - Can be used for different languages, frameworks
|
|
- Can be used for small, simple deployments or large, complicated ones
|
|
- Not very opinioned
|
|
|
|
---
|
|
|
|
## Installing Fabric
|
|
|
|
```bash
|
|
$ pip install fabric
|
|
|
|
# macOS
|
|
$ brew install fabric
|
|
|
|
# Debian, Ubuntu
|
|
$ apt-get install fabric
|
|
$ apt-get install python-fabric
|
|
```
|
|
|
|
---
|
|
|
|
## [fit] Writing your <br>first fabfile
|
|
|
|
---
|
|
|
|
```python
|
|
# fabfile.py
|
|
|
|
from fabric.api import env, run, cd, local
|
|
|
|
env.hosts = ['example.com']
|
|
|
|
# Do stuff...
|
|
```
|
|
|
|
^ Callables
|
|
|
|
---
|
|
|
|
```python
|
|
# fabfile.py
|
|
|
|
from fabric.api import *
|
|
|
|
env.hosts = ['example.com']
|
|
|
|
# Do stuff...
|
|
```
|
|
|
|
---
|
|
|
|
## Operations
|
|
|
|
- cd, lcd - *change directory*
|
|
- run, sudo, local - *run a command*
|
|
- get - *download files*
|
|
- put - *upload files*
|
|
|
|
[.footer: http://docs.fabfile.org/en/1.13/api/core/operations.html]
|
|
|
|
---
|
|
|
|
## Utils
|
|
|
|
- warn: *print warning message*
|
|
- abort: *abort execution, exit with error status*
|
|
- error: *call func with given error message*
|
|
- puts: *alias for print whose output is managed by Fabric's output controls*
|
|
|
|
[.footer: http://docs.fabfile.org/en/1.13/api/core/utils.html]
|
|
|
|
---
|
|
|
|
## File management
|
|
|
|
```python
|
|
from fabric.contrib.files import *
|
|
```
|
|
|
|
- exists - *check if path exists*
|
|
- contains - *check if file contains text/matches regex*
|
|
- sed - *run search and replace on a file*
|
|
- upload_template - *render and upload a template to remote host*
|
|
|
|
[.footer: http://docs.fabfile.org/en/1.13/api/contrib/files.html#fabric.contrib.files.append]
|
|
|
|
^ Allows for jinja2 templates
|
|
|
|
---
|
|
|
|
## Tasks
|
|
|
|
```python
|
|
|
|
def main():
|
|
with cd('/var/www/html'):
|
|
run('git pull')
|
|
run('composer install')
|
|
```
|
|
|
|
---
|
|
|
|
## Task arguments
|
|
|
|
```python, [.highlight: 1, 4-6]
|
|
def main(run_composer=True):
|
|
with cd('/var/www/html'):
|
|
run('git pull')
|
|
|
|
if run_composer:
|
|
run('composer install')
|
|
```
|
|
|
|
---
|
|
|
|
## Task arguments
|
|
|
|
```python, [.highlight: 1, 4-15]
|
|
def main(run_composer=True, env='prod', build_type):
|
|
with cd('/var/www/html'):
|
|
run('git pull')
|
|
|
|
if run_composer:
|
|
if env == 'prod':
|
|
run('composer install --no-dev')
|
|
else:
|
|
run('composer install')
|
|
|
|
if build_type == 'drupal':
|
|
...
|
|
elif build_type == 'symfony':
|
|
...
|
|
elif build_type == 'sculpin':
|
|
...
|
|
```
|
|
|
|
---
|
|
|
|
## Calling other tasks
|
|
|
|
```python, [.highlight: 4-15]
|
|
@task
|
|
def main():
|
|
with cd('/var/www/html'):
|
|
build()
|
|
post_install()
|
|
|
|
def build():
|
|
run('git pull')
|
|
run('composer install')
|
|
|
|
def post_install():
|
|
with prefix('drush'):
|
|
run('updatedb -y')
|
|
run('entity-updates -y')
|
|
run('cache-rebuild')
|
|
```
|
|
|
|
^ Better organised code
|
|
Not everything in one long
|
|
Easier to read and comprehend
|
|
Single responsibilty principle
|
|
|
|
---
|
|
|
|
## Running Tasks
|
|
|
|
```bash
|
|
fab --list
|
|
|
|
fab <task>
|
|
|
|
fab <task>:build_number=$BUILD_ID,build_type=drupal
|
|
```
|
|
|
|
^ fabfile.py in the current directory is found automatically.
|
|
|
|
---
|
|
|
|
```
|
|
[production] Executing task 'main'
|
|
[production] run: git pull
|
|
[production] out: Already up-to-date.
|
|
[production] out:
|
|
|
|
[production] run: composer install
|
|
...
|
|
[production] out: Generating autoload files
|
|
[production] out:
|
|
|
|
Done.
|
|
Disconnecting from production... done.
|
|
```
|
|
|
|
---
|
|
|
|
## Downsides
|
|
|
|
- Running build tasks on production
|
|
|
|
---
|
|
|
|
[.build-lists: false]
|
|
|
|
## Not Building on Prod
|
|
|
|
1. Build locally and deploy.
|
|
|
|
---
|
|
|
|
## Local tasks
|
|
|
|
```python
|
|
# Runs remotely.
|
|
|
|
from fabric.api import run
|
|
|
|
run('git pull')
|
|
run('composer install')
|
|
|
|
# Runs locally.
|
|
|
|
from fabric.api import local
|
|
|
|
local('git pull')
|
|
local('composer install')
|
|
```
|
|
|
|
---
|
|
|
|
## Local tasks
|
|
|
|
```python
|
|
# Remote.
|
|
|
|
from fabric.api import cd
|
|
|
|
with cd('themes/custom/drupalbristol'):
|
|
...
|
|
|
|
# Runs locally.
|
|
|
|
from fabric.api import lcd
|
|
|
|
with lcd('themes/custom/drupalbristol'):
|
|
...
|
|
```
|
|
|
|
---
|
|
|
|
## rsync
|
|
|
|
```python, [.highlight: 1, 5-11]
|
|
from fabric.contrib.project import rsync_project
|
|
|
|
...
|
|
|
|
def deploy():
|
|
rsync_project(
|
|
local_dir='./',
|
|
remote_dir='/var/www/html'
|
|
default_opts='-vzcrSLh',
|
|
exclude=('.git', 'node_modules/', '.sass-cache/')
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
```
|
|
[production] Executing task 'main'
|
|
[localhost] local: git pull
|
|
Current branch master is up to date.
|
|
[localhost] local: composer install
|
|
Loading composer repositories with package information
|
|
Installing dependencies (including require-dev) from lock file
|
|
Nothing to install or update
|
|
Generating autoload files
|
|
|
|
Done.
|
|
```
|
|
|
|
^ - The risky steps have been run separate to the live code.
|
|
- Any issues will not affect the live site.
|
|
|
|
---
|
|
|
|
[.build-lists: false]
|
|
|
|
## Not Building on Prod
|
|
|
|
1. ~~Build locally and deploy.~~
|
|
1. Build in a separate directory and switch after build.
|
|
|
|
^ Capistrano
|
|
|
|
---
|
|
|
|
## Deploying into a different directory
|
|
|
|
```python
|
|
from fabric.api import *
|
|
from time import time
|
|
|
|
project_dir = '/var/www/html'
|
|
next_release = "%(time).0f" % { 'time': time() } # Current timestamp
|
|
|
|
def init():
|
|
if not exists(project_dir):
|
|
run('mkdir -p %s/backups' % project_dir)
|
|
run('mkdir -p %s/shared' % project_dir)
|
|
run('mkdir -p %s/releases' % project_dir)
|
|
```
|
|
|
|
---
|
|
|
|
## Deploying into a different directory
|
|
|
|
```python
|
|
current_release = '%s/%s' % (releases_dir, next_release)
|
|
|
|
run('git clone %s %s' % (git_repo, current_release))
|
|
|
|
def build():
|
|
with cd(current_release):
|
|
pre_tasks()
|
|
build_site()
|
|
post_tasks()
|
|
```
|
|
|
|
^ - Clone the repository into a different directory
|
|
- Run any "risky" tasks away from the production code
|
|
|
|
---
|
|
|
|
## Deploying into a different directory
|
|
|
|
```python
|
|
def pre_build(build_number):
|
|
with cd('current'):
|
|
print '==> Dumping the DB (just in case)...'
|
|
backup_database()
|
|
|
|
def backup_database():
|
|
cd('drush sql-dump --gzip > ../backups/%s.sql.gz' % build_number)
|
|
```
|
|
|
|
---
|
|
|
|
## Deploying into a different directory
|
|
|
|
```python
|
|
def update_symlinks():
|
|
run('ln -nfs %s/releases/%s %s/current'
|
|
% (project_dir, next_release, project_dir))
|
|
|
|
# /var/www/html/current
|
|
```
|
|
|
|
---
|
|
|
|
```
|
|
[production] Executing task 'main'
|
|
[production] run: git clone https://github.com/opdavies/oliverdavies.uk.git
|
|
/var/www/html/releases/1505865600
|
|
Installing Composer dependencies...
|
|
[production] run: composer install --no-dev
|
|
Update the symlink to the new release...
|
|
[production] run: ln -nfs /var/www/html/releases/1505865600
|
|
/var/www/html/current
|
|
|
|
Done.
|
|
```
|
|
|
|
---
|
|
|
|
```bash
|
|
# /var/www/html
|
|
|
|
shared # settings.local.php, sites.php, files etc.
|
|
releases/1502323200
|
|
releases/1505692800
|
|
releases/1505696400
|
|
releases/1505865600
|
|
current -> releases/1505865600 # symlink
|
|
```
|
|
|
|
---
|
|
|
|
[.build-lists: false]
|
|
|
|
## Positives
|
|
|
|
- Errors happen away from production
|
|
|
|
## Downsides
|
|
|
|
- Lots of release directories
|
|
|
|
^ If the build fails, then your live site is not affected.
|
|
|
|
---
|
|
|
|
## Removing old builds
|
|
|
|
```python
|
|
def main(builds_to_keep=3):
|
|
with cd('%s/releases' % project_dir):
|
|
run("ls -1tr | head -n -%d | xargs -d '\\n' rm -fr"
|
|
% builds_to_keep)
|
|
```
|
|
|
|
^ - Find directory names
|
|
- 1tr not ltr
|
|
- Remove the directories
|
|
- Additional tasks, removing DB etc
|
|
|
|
---
|
|
|
|
|
|
|
|
## [fit] Is the site <br>still running?
|
|
|
|
---
|
|
|
|
## Checking for failures
|
|
|
|
```python
|
|
run(command).return_code == 0:
|
|
# Pass
|
|
|
|
run(command).return_code == 1:
|
|
# Fail
|
|
|
|
run(command).failed:
|
|
# Fail
|
|
```
|
|
|
|
^ Works for local and remote.
|
|
|
|
---
|
|
|
|
```python
|
|
print 'Checking the site is alive...'
|
|
if run('drush status | egrep "Connected|Successful"').failed:
|
|
# Revert back to previous build.
|
|
```
|
|
|
|
^ egrep is an acronym for "Extended Global Regular Expressions Print". It is a program which scans a specified file line by line, returning lines that contain a pattern matching a given regular expression.
|
|
|
|
---
|
|
|
|
```bash, [.highlight 3]
|
|
$ drush status
|
|
|
|
Drupal version : 8.3.7
|
|
Site URI : http://default
|
|
Database driver : mysql
|
|
Database hostname : db
|
|
Database username : user
|
|
Database name : default
|
|
Database : Connected
|
|
Drupal bootstrap : Successful
|
|
Drupal user :
|
|
Default theme : bartik
|
|
Administration theme : seven
|
|
PHP configuration : /etc/php5/cli/php.ini
|
|
...
|
|
```
|
|
|
|
^ Successful
|
|
|
|
---
|
|
|
|
```bash, [.highlight 3]
|
|
$ drush status
|
|
|
|
Drupal version : 8.3.7
|
|
Site URI : http://default
|
|
Database driver : mysql
|
|
Database hostname : db
|
|
Database username : user
|
|
Database name : default
|
|
PHP configuration : /etc/php5/cli/php.ini
|
|
...
|
|
```
|
|
|
|
^ Failed.
|
|
"Database" to "PHP configuration" missing if cannot connect or DB is empty.
|
|
|
|
---
|
|
|
|
## [fit] Does the code still <br>merge cleanly?
|
|
|
|
^ Pre-task
|
|
|
|
---
|
|
|
|
```python
|
|
def check_for_merge_conflicts(target_branch):
|
|
with settings(warn_only=True):
|
|
print('Ensuring that this can be merged into the main branch.')
|
|
|
|
if local('git fetch && git merge --no-ff origin/%s'
|
|
% target_branch).failed:
|
|
abort('Cannot merge into target branch.')
|
|
```
|
|
|
|
^ Define target branch in Jenkins/when Fabric is run.
|
|
|
|
---
|
|
|
|
|
|
|
|
## [fit] Do our tests <br>still pass?
|
|
|
|
[.footer: http://nashvillephp.org/images/testing-workshop-banner-1600x800.jpg]
|
|
|
|
---
|
|
|
|
```python
|
|
with settings(warn_only=True):
|
|
with lcd('%s/docroot/core' % project_dir):
|
|
if local('../../vendor/bin/phpunit ../modules/custom').failed:
|
|
abort('Tests failed!')
|
|
```
|
|
|
|
---
|
|
|
|
```
|
|
[localhost] run: ../../vendor/bin/phpunit ../modules/custom
|
|
[localhost] out: PHPUnit 4.8.35 by Sebastian Bergmann and contributors.
|
|
[localhost] out:
|
|
[localhost] out: .......
|
|
[localhost] out:
|
|
[localhost] out: Time: 1.59 minutes, Memory: 6.00MB
|
|
[localhost] out:
|
|
[localhost] out: OK (7 tests, 42 assertions)
|
|
[localhost] out:
|
|
|
|
|
|
Done.
|
|
```
|
|
|
|
---
|
|
|
|
```
|
|
[localhost] run: ../../vendor/bin/phpunit ../modules/custom
|
|
[localhost] out: PHPUnit 4.8.35 by Sebastian Bergmann and contributors.
|
|
[localhost] out:
|
|
[localhost] out: E
|
|
[localhost] out:
|
|
[localhost] out: Time: 18.67 seconds, Memory: 6.00MB
|
|
[localhost] out:
|
|
[localhost] out: There was 1 error:
|
|
[localhost] out:
|
|
[localhost] out: 1) Drupal\Tests\broadbean\Functional\AddJobTest::testNodesAreCreated
|
|
[localhost] out: Behat\Mink\Exception\ExpectationException: Current response status code is 200, but 201 expected.
|
|
[localhost] out:
|
|
[localhost] out: /var/www/html/vendor/behat/mink/src/WebAssert.php:770
|
|
[localhost] out: /var/www/html/vendor/behat/mink/src/WebAssert.php:130
|
|
[localhost] out: /var/www/html/docroot/modules/custom/broadbean/tests/src/Functional/AddJobTest.php:66
|
|
[localhost] out:
|
|
[localhost] out: FAILURES!
|
|
[localhost] out: Tests: 1, Assertions: 6, Errors: 1.
|
|
[localhost] out:
|
|
|
|
Warning: run() received nonzero return code 2 while executing '../../vendor/bin/phpunit ../modules/custom/broadbean'!
|
|
|
|
Fatal error: Tests failed!
|
|
|
|
Aborting.
|
|
```
|
|
|
|
---
|
|
|
|
|
|
|
|
## [fit] Making Fabric <br>Smarter
|
|
|
|
---
|
|
|
|
## Conditional variables
|
|
|
|
```python
|
|
drupal_version = None
|
|
|
|
if exists('composer.json') and exists('core'):
|
|
drupal_version = 8
|
|
else:
|
|
drupal_version = 7
|
|
```
|
|
|
|
---
|
|
|
|
## Conditional tasks
|
|
|
|
```python
|
|
if exists('composer.json'):
|
|
run('composer install')
|
|
|
|
with cd('themes/custom/example'):
|
|
if exists('package.json') and not exists('node_modules'):
|
|
run('yarn --pure-lockfile')
|
|
|
|
if exists('gulpfile.js'):
|
|
run('node_modules/.bin/gulp --production')
|
|
elif exists('gruntfile.js'):
|
|
run('node_modules/.bin/grunt build')
|
|
```
|
|
|
|
---
|
|
|
|
## Project settings file
|
|
|
|
```yml
|
|
# app.yml
|
|
|
|
drupal:
|
|
version: 8
|
|
root: web
|
|
config:
|
|
import: yes
|
|
name: sync
|
|
cmi_tools: no
|
|
tests:
|
|
simpletest: false
|
|
phpunit: true
|
|
theme:
|
|
path: 'themes/custom/drupalbristol'
|
|
build:
|
|
type: gulp
|
|
npm: no
|
|
yarn: yes
|
|
|
|
composer:
|
|
install: true
|
|
```
|
|
|
|
---
|
|
|
|
## Project settings file
|
|
|
|
```python, [.highlight 3]
|
|
# fabfile.py
|
|
|
|
from fabric.api import *
|
|
import yaml
|
|
|
|
config = []
|
|
|
|
if exists('app.yml'):
|
|
with open('app.yml', 'r') as file:
|
|
config = yaml.load(file.read())
|
|
```
|
|
|
|
---
|
|
|
|
## Project settings file
|
|
|
|
```python
|
|
# fabfile.py
|
|
|
|
if config['composer']['install'] == True:
|
|
local('composer install')
|
|
```
|
|
|
|
---
|
|
|
|
## Project settings file
|
|
|
|
```python
|
|
# fabfile.py
|
|
|
|
if build_type == 'drupal':
|
|
drupal = config['drupal']
|
|
```
|
|
|
|
---
|
|
|
|
## Project settings file
|
|
|
|
```python, [.highlight: 6-9]
|
|
# fabfile.py
|
|
|
|
if build_type == 'drupal':
|
|
drupal = config['drupal']
|
|
|
|
with cd(drupal['root']):
|
|
if drupal['version'] == 8:
|
|
|
|
if drupal['version'] == 7:
|
|
```
|
|
|
|
---
|
|
|
|
## Project settings file
|
|
|
|
```python, [.highlight: 8-10]
|
|
# fabfile.py
|
|
|
|
if build_type == 'drupal':
|
|
drupal = config['drupal']
|
|
|
|
with cd(drupal['root']):
|
|
if drupal['version'] == 8:
|
|
if drupal['config']['import'] == True:
|
|
# Import the staged configuration.
|
|
run('drush cim -y %s' % drupal['config']['name'])
|
|
```
|
|
|
|
---
|
|
|
|
## Project settings file
|
|
|
|
```python, [.highlight: 9-15]
|
|
# fabfile.py
|
|
|
|
if build_type == 'drupal':
|
|
drupal = config['drupal']
|
|
|
|
with cd(drupal['root']):
|
|
if drupal['version'] == 8:
|
|
if drupal['config']['import'] == True:
|
|
if drupal['config']['cmi_tools'] == True:
|
|
# Use Drush CMI Tools.
|
|
run('drush cimy -y %s' % drupal['config']['name'])
|
|
else:
|
|
# Use core.
|
|
run('drush cim -y %s' % drupal['config']['name'])
|
|
```
|
|
|
|
^ - Less hard-coded values
|
|
- More flexible
|
|
- No need to use different files for different versions or frameworks
|
|
- No forked fabfiles per project or lots of conditionals based on the project
|
|
|
|
---
|
|
|
|
## Project settings file
|
|
|
|
```python
|
|
# fabfile.py
|
|
|
|
theme = config['theme']
|
|
|
|
with cd(theme['path']):
|
|
if theme['build']['gulp'] == True:
|
|
if env == 'prod':
|
|
run('node_modules/.bin/gulp --production')
|
|
else:
|
|
run('node_modules/.bin/gulp')
|
|
```
|
|
|
|
---
|
|
|
|
## Project settings file v2
|
|
|
|
```yml
|
|
# app.yml
|
|
|
|
commands:
|
|
build: |
|
|
cd web/themes/custom/drupalbristol
|
|
yarn --pure-lockfile
|
|
npm run prod
|
|
|
|
deploy: |
|
|
cd web
|
|
drush updatedb -y
|
|
drush cache-rebuild -y
|
|
```
|
|
|
|
^ How do we know when to run Composer or npm?
|
|
How do we know what commands to run?
|
|
Can't tell this just by the presence of files.
|
|
|
|
---
|
|
|
|
## Project settings file v2
|
|
|
|
```python
|
|
# fabfile.py
|
|
|
|
# Run build commands locally.
|
|
for hook in config['commands'].get('build', '').split("\n"):
|
|
run(hook)
|
|
|
|
...
|
|
|
|
# Run deploy commands remotely.
|
|
for hook in config['commands'].get('deploy', '').split("\n"):
|
|
run(hook)
|
|
```
|
|
|
|
^ "run" won't work locally
|
|
|
|
---
|
|
|
|
## Other things
|
|
|
|
- Run Drush commands
|
|
- Run automated tests
|
|
- Verify file permissions
|
|
- Restart services
|
|
- Anything you can do on the command line...
|
|
|
|
---
|
|
|
|
[.build-lists: false]
|
|
|
|
## Fabric has...
|
|
|
|
- Simplified my build process
|
|
- Made my build process more flexible
|
|
- Made my build process more robust
|
|
|
|
---
|
|
|
|
[.build-lists: false]
|
|
|
|
- https://www.oliverdavies.uk/talks/deploying-drupal-fabric
|
|
- http://fabfile.org
|
|
- https://github.com/opdavies/fabric-example-drupal
|
|
- https://github.com/opdavies/fabric-example-sculpin
|
|
- https://deploy.serversforhackers.com (~~$129~~ $79)
|
|
|
|
---
|
|
|
|
## Thanks!
|
|
|
|
# Questions?
|
|
|
|
### @opdavies
|
|
### oliverdavies.uk
|