diff --git a/deploying-drupal-fabric/2017-10-26-drupal-somerset/images/files.png b/deploying-drupal-fabric/2017-10-26-drupal-somerset/images/files.png new file mode 100644 index 0000000..a23b62a Binary files /dev/null and b/deploying-drupal-fabric/2017-10-26-drupal-somerset/images/files.png differ diff --git a/deploying-drupal-fabric/2017-10-26-drupal-somerset/images/fry.jpeg b/deploying-drupal-fabric/2017-10-26-drupal-somerset/images/fry.jpeg new file mode 100644 index 0000000..1e9fe70 Binary files /dev/null and b/deploying-drupal-fabric/2017-10-26-drupal-somerset/images/fry.jpeg differ diff --git a/deploying-drupal-fabric/2017-10-26-drupal-somerset/images/grumpy.jpg b/deploying-drupal-fabric/2017-10-26-drupal-somerset/images/grumpy.jpg new file mode 100644 index 0000000..f44fa63 Binary files /dev/null and b/deploying-drupal-fabric/2017-10-26-drupal-somerset/images/grumpy.jpg differ diff --git a/deploying-drupal-fabric/2017-10-26-drupal-somerset/images/homer-smart.png b/deploying-drupal-fabric/2017-10-26-drupal-somerset/images/homer-smart.png new file mode 100644 index 0000000..50c5705 Binary files /dev/null and b/deploying-drupal-fabric/2017-10-26-drupal-somerset/images/homer-smart.png differ diff --git a/deploying-drupal-fabric/2017-10-26-drupal-somerset/slides.md b/deploying-drupal-fabric/2017-10-26-drupal-somerset/slides.md new file mode 100644 index 0000000..ce23273 --- /dev/null +++ b/deploying-drupal-fabric/2017-10-26-drupal-somerset/slides.md @@ -0,0 +1,907 @@ +autoscale: true +build-lists: true +theme: next, 9 + +# [fit] Deploying Drupal
with Fabric +### Oliver Davies +### bit.ly/deploying-drupal-fabric + +--- + +[.build-lists: false] + +- What is Fabric and what do I use it for? +- How to write and organise Fabric scripts +- Task examples + +--- + +[.build-lists: false] + +![right](../../me-phpnw.png) + +- Senior Developer at Microserve +- Part-time freelance Developer & System Administrator +- Drupal Bristol, PHPSW, DrupalCamp Bristol organiser +- Sticker collector, herder of elePHPants +- @opdavies +- oliverdavies.uk + + +^ Drupal, Symfony, Silex, Laravel, Sculpin +Not a Python Developer + +--- + +![](images/fry.jpeg) + +## [fit] What is
Fabric? + +--- + +## What is Fabric? + +Fabric is a Python (2.5-2.7) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks. + +--- + +## What is Fabric? + +It provides a basic suite of operations for executing local or remote shell commands (normally or via sudo) and uploading/downloading files, as well as auxiliary functionality such as prompting the running user for input, or aborting execution. + +--- + +## I use Fabric to... + +- Simplify my build process +- Deploy code directly to different environments +- Act as an intermediate step + +--- + +![fit](images/files.png) + +--- + +[.build-lists: false] + +## Why Fabric? + +- Powerful +- Flexible +- Easier to read and write than bash + +^ - Can be used for different languages, frameworks +- Can be used for small, simple deployments or large, complicated ones +- Not very opinioned + +--- + +## Installing Fabric + +```bash +$ pip install fabric + +# macOS +$ brew install fabric + +# Debian, Ubuntu +$ apt-get install fabric +$ apt-get install python-fabric +``` + +--- + +## [fit] Writing your
first fabfile + +--- + +```python +# fabfile.py + +from fabric.api import env, run, cd, local + +env.hosts = ['example.com'] + +# Do stuff... +``` + +^ Callables + +--- + +```python +# fabfile.py + +from fabric.api import * + +env.hosts = ['example.com'] + +# Do stuff... +``` + +--- + +## Operations + +- cd, lcd - *change directory* +- run, sudo, local - *run a command* +- get - *download files* +- put - *upload files* + +[.footer: http://docs.fabfile.org/en/1.13/api/core/operations.html] + +--- + +## Utils + +- warn: *print warning message* +- abort: *abort execution, exit with error status* +- error: *call func with given error message* +- puts: *alias for print whose output is managed by Fabric's output controls* + +[.footer: http://docs.fabfile.org/en/1.13/api/core/utils.html] + +--- + +## File management + +```python +from fabric.contrib.files import * +``` + +- exists - *check if path exists* +- contains - *check if file contains text/matches regex* +- sed - *run search and replace on a file* +- upload_template - *render and upload a template to remote host* + +[.footer: http://docs.fabfile.org/en/1.13/api/contrib/files.html#fabric.contrib.files.append] + +^ Allows for jinja2 templates + +--- + +## Tasks + +```python + +def main(): + with cd('/var/www/html'): + run('git pull') + run('composer install') +``` + +--- + +## Task arguments + +```python, [.highlight: 1, 4-6] +def main(run_composer=True): + with cd('/var/www/html'): + run('git pull') + + if run_composer: + run('composer install') +``` + +--- + +## Task arguments + +```python, [.highlight: 1, 4-15] +def main(run_composer=True, env='prod', build_type): + with cd('/var/www/html'): + run('git pull') + + if run_composer: + if env == 'prod': + run('composer install --no-dev') + else: + run('composer install') + + if build_type == 'drupal': + ... + elif build_type == 'symfony': + ... + elif build_type == 'sculpin': + ... +``` + +--- + +## Calling other tasks + +```python, [.highlight: 4-15] +@task +def main(): + with cd('/var/www/html'): + build() + post_install() + +def build(): + run('git pull') + run('composer install') + +def post_install(): + with prefix('drush'): + run('updatedb -y') + run('entity-updates -y') + run('cache-rebuild') +``` + +^ Better organised code +Not everything in one long +Easier to read and comprehend +Single responsibilty principle + +--- + +## Running Tasks + +```bash +fab --list + +fab + +fab :build_number=$BUILD_ID,build_type=drupal +``` + +^ fabfile.py in the current directory is found automatically. + +--- + +``` +[production] Executing task 'main' +[production] run: git pull +[production] out: Already up-to-date. +[production] out: + +[production] run: composer install +... +[production] out: Generating autoload files +[production] out: + +Done. +Disconnecting from production... done. +``` + +--- + +## Downsides + +- Running build tasks on production + +--- + +[.build-lists: false] + +## Not Building on Prod + +1. Build locally and deploy. + +--- + +## Local tasks + +```python +# Runs remotely. + +from fabric.api import run + +run('git pull') +run('composer install') + +# Runs locally. + +from fabric.api import local + +local('git pull') +local('composer install') +``` + +--- + +## Local tasks + +```python +# Remote. + +from fabric.api import cd + +with cd('themes/custom/drupalbristol'): + ... + +# Runs locally. + +from fabric.api import lcd + +with lcd('themes/custom/drupalbristol'): + ... +``` + +--- + +## rsync + +```python, [.highlight: 1, 5-11] +from fabric.contrib.project import rsync_project + +... + +def deploy(): + rsync_project( + local_dir='./', + remote_dir='/var/www/html' + default_opts='-vzcrSLh', + exclude=('.git', 'node_modules/', '.sass-cache/') + ) +``` + +--- + +``` +[production] Executing task 'main' +[localhost] local: git pull +Current branch master is up to date. +[localhost] local: composer install +Loading composer repositories with package information +Installing dependencies (including require-dev) from lock file +Nothing to install or update +Generating autoload files + +Done. +``` + +^ - The risky steps have been run separate to the live code. +- Any issues will not affect the live site. + +--- + +[.build-lists: false] + +## Not Building on Prod + +1. ~~Build locally and deploy.~~ +1. Build in a separate directory and switch after build. + +^ Capistrano + +--- + +## Deploying into a different directory + +```python +from fabric.api import * +from time import time + +project_dir = '/var/www/html' +next_release = "%(time).0f" % { 'time': time() } # Current timestamp + +def init(): + if not exists(project_dir): + run('mkdir -p %s/backups' % project_dir) + run('mkdir -p %s/shared' % project_dir) + run('mkdir -p %s/releases' % project_dir) +``` + +--- + +## Deploying into a different directory + +```python +current_release = '%s/%s' % (releases_dir, next_release) + +run('git clone %s %s' % (git_repo, current_release)) + +def build(): + with cd(current_release): + pre_tasks() + build_site() + post_tasks() +``` + +^ - Clone the repository into a different directory +- Run any "risky" tasks away from the production code + +--- + +## Deploying into a different directory + +```python +def pre_build(build_number): + with cd('current'): + print '==> Dumping the DB (just in case)...' + backup_database() + +def backup_database(): + cd('drush sql-dump --gzip > ../backups/%s.sql.gz' % build_number) +``` + +--- + +## Deploying into a different directory + +```python +def update_symlinks(): + run('ln -nfs %s/releases/%s %s/current' + % (project_dir, next_release, project_dir)) + +# /var/www/html/current +``` + +--- + +``` +[production] Executing task 'main' +[production] run: git clone https://github.com/opdavies/oliverdavies.uk.git + /var/www/html/releases/1505865600 +Installing Composer dependencies... +[production] run: composer install --no-dev +Update the symlink to the new release... +[production] run: ln -nfs /var/www/html/releases/1505865600 + /var/www/html/current + +Done. +``` + +--- + +```bash +# /var/www/html + +shared # settings.local.php, sites.php, files etc. +releases/1502323200 +releases/1505692800 +releases/1505696400 +releases/1505865600 +current -> releases/1505865600 # symlink +``` + +--- + +[.build-lists: false] + +## Positives + +- Errors happen away from production + +## Downsides + +- Lots of release directories + +^ If the build fails, then your live site is not affected. + +--- + +## Removing old builds + +```python +def main(builds_to_keep=3): + with cd('%s/releases' % project_dir): + run("ls -1tr | head -n -%d | xargs -d '\\n' rm -fr" + % builds_to_keep) +``` + +^ - Find directory names +- 1tr not ltr +- Remove the directories +- Additional tasks, removing DB etc + +--- + + + +## [fit] Is the site
still running? + +--- + +## Checking for failures + +```python +run(command).return_code == 0: + # Pass + +run(command).return_code == 1: + # Fail + +run(command).failed: + # Fail +``` + +^ Works for local and remote. + +--- + +```python +print 'Checking the site is alive...' +if run('drush status | egrep "Connected|Successful"').failed: + # Revert back to previous build. +``` + +^ egrep is an acronym for "Extended Global Regular Expressions Print". It is a program which scans a specified file line by line, returning lines that contain a pattern matching a given regular expression. + +--- + +```bash, [.highlight 3] +$ drush status + +Drupal version : 8.3.7 +Site URI : http://default +Database driver : mysql +Database hostname : db +Database username : user +Database name : default +Database : Connected +Drupal bootstrap : Successful +Drupal user : +Default theme : bartik +Administration theme : seven +PHP configuration : /etc/php5/cli/php.ini +... +``` + +^ Successful + +--- + +```bash, [.highlight 3] +$ drush status + +Drupal version : 8.3.7 +Site URI : http://default +Database driver : mysql +Database hostname : db +Database username : user +Database name : default +PHP configuration : /etc/php5/cli/php.ini +... +``` + +^ Failed. +"Database" to "PHP configuration" missing if cannot connect or DB is empty. + +--- + +## [fit] Does the code still
merge cleanly? + +^ Pre-task + +--- + +```python +def check_for_merge_conflicts(target_branch): + with settings(warn_only=True): + print('Ensuring that this can be merged into the main branch.') + + if local('git fetch && git merge --no-ff origin/%s' + % target_branch).failed: + abort('Cannot merge into target branch.') +``` + +^ Define target branch in Jenkins/when Fabric is run. + +--- + +![](images/grumpy.jpg) + +## [fit] Do our tests
still pass? + +[.footer: http://nashvillephp.org/images/testing-workshop-banner-1600x800.jpg] + +--- + +```python +with settings(warn_only=True): + with lcd('%s/docroot/core' % project_dir): + if local('../../vendor/bin/phpunit ../modules/custom').failed: + abort('Tests failed!') +``` + +--- + +``` +[localhost] run: ../../vendor/bin/phpunit ../modules/custom +[localhost] out: PHPUnit 4.8.35 by Sebastian Bergmann and contributors. +[localhost] out: +[localhost] out: ....... +[localhost] out: +[localhost] out: Time: 1.59 minutes, Memory: 6.00MB +[localhost] out: +[localhost] out: OK (7 tests, 42 assertions) +[localhost] out: + + +Done. +``` + +--- + +``` +[localhost] run: ../../vendor/bin/phpunit ../modules/custom +[localhost] out: PHPUnit 4.8.35 by Sebastian Bergmann and contributors. +[localhost] out: +[localhost] out: E +[localhost] out: +[localhost] out: Time: 18.67 seconds, Memory: 6.00MB +[localhost] out: +[localhost] out: There was 1 error: +[localhost] out: +[localhost] out: 1) Drupal\Tests\broadbean\Functional\AddJobTest::testNodesAreCreated +[localhost] out: Behat\Mink\Exception\ExpectationException: Current response status code is 200, but 201 expected. +[localhost] out: +[localhost] out: /var/www/html/vendor/behat/mink/src/WebAssert.php:770 +[localhost] out: /var/www/html/vendor/behat/mink/src/WebAssert.php:130 +[localhost] out: /var/www/html/docroot/modules/custom/broadbean/tests/src/Functional/AddJobTest.php:66 +[localhost] out: +[localhost] out: FAILURES! +[localhost] out: Tests: 1, Assertions: 6, Errors: 1. +[localhost] out: + +Warning: run() received nonzero return code 2 while executing '../../vendor/bin/phpunit ../modules/custom/broadbean'! + +Fatal error: Tests failed! + +Aborting. +``` + +--- + +![](images/homer-smart.png) + +## [fit] Making Fabric
Smarter + +--- + +## Conditional variables + +```python +drupal_version = None + +if exists('composer.json') and exists('core'): + drupal_version = 8 +else: + drupal_version = 7 +``` + +--- + +## Conditional tasks + +```python +if exists('composer.json'): + run('composer install') + +with cd('themes/custom/example'): + if exists('package.json') and not exists('node_modules'): + run('yarn --pure-lockfile') + + if exists('gulpfile.js'): + run('node_modules/.bin/gulp --production') + elif exists('gruntfile.js'): + run('node_modules/.bin/grunt build') +``` + +--- + +## Project settings file + +```yml +# app.yml + +drupal: + version: 8 + root: web + config: + import: yes + name: sync + cmi_tools: no + tests: + simpletest: false + phpunit: true + theme: + path: 'themes/custom/drupalbristol' + build: + type: gulp + npm: no + yarn: yes + +composer: + install: true +``` + +--- + +## Project settings file + +```python, [.highlight 3] +# fabfile.py + +from fabric.api import * +import yaml + +config = [] + +if exists('app.yml'): + with open('app.yml', 'r') as file: + config = yaml.load(file.read()) +``` + +--- + +## Project settings file + +```python +# fabfile.py + +if config['composer']['install'] == True: + local('composer install') +``` + +--- + +## Project settings file + +```python +# fabfile.py + +if build_type == 'drupal': + drupal = config['drupal'] +``` + +--- + +## Project settings file + +```python, [.highlight: 6-9] +# fabfile.py + +if build_type == 'drupal': + drupal = config['drupal'] + + with cd(drupal['root']): + if drupal['version'] == 8: + + if drupal['version'] == 7: +``` + +--- + +## Project settings file + +```python, [.highlight: 8-10] +# fabfile.py + +if build_type == 'drupal': + drupal = config['drupal'] + + with cd(drupal['root']): + if drupal['version'] == 8: + if drupal['config']['import'] == True: + # Import the staged configuration. + run('drush cim -y %s' % drupal['config']['name']) +``` + +--- + +## Project settings file + +```python, [.highlight: 9-15] +# fabfile.py + +if build_type == 'drupal': + drupal = config['drupal'] + + with cd(drupal['root']): + if drupal['version'] == 8: + if drupal['config']['import'] == True: + if drupal['config']['cmi_tools'] == True: + # Use Drush CMI Tools. + run('drush cimy -y %s' % drupal['config']['name']) + else: + # Use core. + run('drush cim -y %s' % drupal['config']['name']) +``` + +^ - Less hard-coded values +- More flexible +- No need to use different files for different versions or frameworks +- No forked fabfiles per project or lots of conditionals based on the project + +--- + +## Project settings file + +```python +# fabfile.py + +theme = config['theme'] + +with cd(theme['path']): + if theme['build']['gulp'] == True: + if env == 'prod': + run('node_modules/.bin/gulp --production') + else: + run('node_modules/.bin/gulp') +``` + +--- + +## Project settings file v2 + +```yml +# app.yml + +commands: + build: | + cd web/themes/custom/drupalbristol + yarn --pure-lockfile + npm run prod + + deploy: | + cd web + drush updatedb -y + drush cache-rebuild -y +``` + +^ How do we know when to run Composer or npm? +How do we know what commands to run? +Can't tell this just by the presence of files. + +--- + +## Project settings file v2 + +```python +# fabfile.py + +# Run build commands locally. +for hook in config['commands'].get('build', '').split("\n"): + run(hook) + +... + +# Run deploy commands remotely. +for hook in config['commands'].get('deploy', '').split("\n"): + run(hook) +``` + +^ "run" won't work locally + +--- + +## Other things + +- Run Drush commands +- Run automated tests +- Verify file permissions +- Restart services +- Anything you can do on the command line... + +--- + +[.build-lists: false] + +## Fabric has... + +- Simplified my build process +- Made my build process more flexible +- Made my build process more robust + +--- + +[.build-lists: false] + +- https://www.oliverdavies.uk/talks/deploying-drupal-fabric +- http://fabfile.org +- https://github.com/opdavies/fabric-example-drupal +- https://github.com/opdavies/fabric-example-sculpin +- https://deploy.serversforhackers.com (~~$129~~ $79) + +--- + +## Thanks! + +# Questions? + +### @opdavies +### oliverdavies.uk diff --git a/deploying-drupal-fabric/2017-10-26-drupal-somerset/slides.pdf b/deploying-drupal-fabric/2017-10-26-drupal-somerset/slides.pdf new file mode 100644 index 0000000..e6f3ba4 Binary files /dev/null and b/deploying-drupal-fabric/2017-10-26-drupal-somerset/slides.pdf differ