Vagrant, Symfony, Rsync (and Assetic)

13 September 2015, Rhodri Pugh

This post is a follow up to one I wrote on my personal blog about a year and a half ago. Since then I was lucky enough to have an extremely talented engineer join Owsy, who wasn’t as easy-going about slow page response times on their development VM as I was ;)

Disclaimer: This is not my work, I’m just the blogger.

Pretty Slow…

As detailed in that other post, we used to run all our development VMs using NFS to mount the host codebase into the VM. This isn’t amazingly fast though, so page reloads of the (large) Symfony application were generally at least a few seconds. And doing things like clearing the cache in the VM or running unit tests and console commands was always sluggish.

Faster Please!

I had tried switching to the built-in rsync support in Vagrant before, and it was too slow to sync changes over. This time though we used the vagrant-gatling-rsync plugin which we hoped would cope with the size of the project.

Below is a commented example of the Vagrantfile we use, as if you’re reading this you probably just want to get on and use it, then I’ll discuss workflow issues afterwards.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
# Vagrantfile

require "yaml"

dir = File.dirname(File.expand_path(__FILE__))

# Read in the (volatile) settings file, or use the (versioned) default one.
if File.exist?("#{dir}/settings.yml")
settings = YAML.load_file("#{dir}/settings.yml")
else
settings = YAML.load_file("#{dir}/settings.yml.dist")
end

Vagrant.configure("2") do |config|

# Some variables that specify where the codebase is on the host, relative to
# this Vagrantfile, and the location of the application inside the VM. This is
# where the codebase will be rsync'd to.
symfony_host = "../../../symfony-application/"
symfony_guest = "/opt/symfony-application"

# General VM config, you can probably ignore this bit as it's not relevant to
# the rsync'ing - I've included it so this is a complete example
config.vm.box = "centos65"
config.vm.network :private_network, ip: '192.168.33.1'
config.vm.provider "virtualbox" do |v|
v.name = "dev-symfony"
v.customize ["modifyvm", :id, "--ioapic", "on"]
v.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
v.customize ["modifyvm", :id, "--natdnsproxy1", "on"]
v.cpus = 2
v.memory = 2048
end

# Check the codebase directory exists on the host, we should probably throw an
# error here if it isn't... something to add.
if File.directory?(symfony_host) then

# Check if the settings file has configured rsync
if settings['sync_type'] == 'rsync' then

# Check the required plugin is installed
unless Vagrant.has_plugin? 'vagrant-gatling-rsync'
raise 'You need to install the "vagrant-gatling-rsync" plugin.'
end

# Here we configure the user/group that the files will be chown'd to when
# they are rsync'd over to the VM. These settings will need to match the
# configuration you have for Apache or PHP-FPM so the permissions work.
guest_user = "user_name"
guest_group = "group_name"

# This is the actual rsync configuration. It specifies where we're copying
# from/to, the user/group to use, then some rsync options (if you're using
# this as a template and don't have any unusual requirements then you can
# probably leave this unmodified)
config.vm.synced_folder symfony_host, symfony_guest, type: "rsync", owner: guest_user, group: guest_group,
rsync__args: ["--verbose", "--archive", "--delete", "-z", "--copy-links"],
rsync__exclude: settings['rsync_exclude']

# This section optionally configures a bindfs share to mount folders *from*
# the guest back to the host. The use-case for this usually for doctrine
# migrations that need to be created on the guest, but then edited and
# versioned on the host.
if Vagrant.has_plugin?("vagrant-bindfs") then
settings['synced_folders'].each do |folder|
config.vm.synced_folder symfony_host + folder, "/vagrant-nfs/" + folder, type: "nfs", nfs_udp: false
config.bindfs.bind_folder "/vagrant-nfs/" + folder, symfony_guest + '/' + folder, 'force-user' => guest_user, 'force-group' => guest_group
end
end

# Some vagrant-gatling-rsync options you can tweak, but these should work.
config.gatling.latency = 0.2
config.gatling.time_format = "%H:%M:%S"

# This setting controls whether vagrant-gatling-rsync will run its watcher
# when the VM boots or not. Some developers boot the VM from the shell and
# want the watcher to start by default, others (like me) use Vagrant
# Manager to boot the VM and then manually start the watcher later.
config.gatling.rsync_on_startup = settings['rsync_watch']
else
# If rsync isn't configured default to NFS, which should "just work"
config.vm.synced_folder symfony_host, symfony_guest, type: "nfs", nfs_udp: false
end
end
end

Next, here’s an example of a settings file (this is the one I use).

1
2
3
4
5
6
7
# settings.yml

---
sync_type: "rsync"
synced_folders: ~ # list of folders (eg. "app/DoctrineMigrations/")
rsync_exclude: [".git/", "app/cache/", "web/js/", "web/css", "web/images", "app/logs/", "vendor/"]
rsync_watch: false

Usage

You then use this as usual with Vagrant…

$> vagrant up

On boot your codebase will be rsync’d over to the VM. If you have the watcher configured to run on boot then it’ll listen for changes, otherwise you now need to start the watcher manually using this command…

1
2
3
4
5
$> vagrant gatling-rsync-auto
==> default: Doing an initial rsync...
==> default: Rsyncing folder: /Users/dev/Code/symfony-application/ => /opt/symfony-application
==> default: - Exclude: [".vagrant/", ".git/", "app/cache/", "web/js/", "web/css", "web/images", "app/logs/", "vendor/"]
==> default: Watching: /Users/dev/Code/symfony-application

The application on the guest VM now uses a local filesystem so it’s much faster than with NFS. For us, page loads dropped from a few seconds (sometimes 5 to 10 seconds), down to about 0.25 seconds. Pretty dramatic, and a massively better development experience.

Workflows

So, now we have a fast development version of our application. But the change in setup (with NFS both host and guest see the same codebase, with rsync there are two copies) means we need to make a few changes to our workflows on common Symfony tasks. The way we solve each problem differs slightly between developers, the big split generally coming from how some developers want to use an interactive debugger in their IDE from the host, and others (mainly me) who want to do everything apart from edit code inside the VM.

Composer Vendors

The running application needs to use the vendor libraries, so these need to exist on the VM. Some developers want to be able to debug the application (including the vendors) in their IDE, so need these to exist on both. Here the solution is to manage the vendors (via composer install etc…) on the host, and then have these rsync’d to the guest. The downside of this is that you need to have PHP and associated dependencies on the host.

If this isn’t a requirement though, then it’s simpler to run Composer on the guest, and add the vendor folder to the rsync exclude list.

Doctrine Migrations

When using the schema tool to generate migration diffs this writes the new migration file to the local filesystem. If you’re running Composer on the host then no problem. If you’re running Composer purely on the guest though you’ll be left with a migration file which does not exist on the host, and may get overwritten/deleted the next time rsync runs.

There are two solutions for this, one being the use of bindfs in the Vagrantfile above (which can mount your local app/DoctrineMigrations folder via NFS to the VM), the other is a short script which can be run on the host, eg…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#!/bin/sh
#
# bin/doctrine-diff
#
# Usage: bin/doctrine-diff
# Generates a migration diff on the guest then scp's it back to the host
#

FILE=`ssh user@vmname -q \
'/opt/symfony-application/app/console doctrine:migrations:diff' \
| sed 's/.*symfony-application\///' \
| sed 's/" from schema differences.//'`

scp -q user@vmname:/opt/symfony-application/$FILE $FILE

echo "Generated: $FILE"

The first of those solutions is probably the neatest.

Building Assets

The subject of my original blog post was to do with how to effectively build assets with Assetic. Here again, how you do this depends on if you’re running Composer on the host or the guest.

If you’re running it on the host then the files can be rsyc’d over, or if you’re running it on the guest then you can use a watcher there to build them. Here’s the one I use…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#!/bin/bash
#
# Usage: bin/inotify-assets
# Run assetic:dump on file changes

function rebuild() {
app/console assetic:dump
app/console assets:install web --symlink
}

rm -rf app/cache/dev

rebuild

while inotifywait -qre create,delete,modify,move src/
do
rebuild
done

This requires inotify-tools be installed.

Conclusion

If you’re currently using NFS and not having any performance problems then I can’t recommend switching to using rsync as it is a more complicated setup. But if you are finding NFS too slow for whatever reason then I seriously recommend giving it a try.

Of the two workflow approaches I discussed, using Composer on the host is probably the most straight-forward. I choose against it though, my reason being I want to isolate all the runtime dependencies and tooling to the VM.

I’ve pushed the Vagrantfile code to Github if anyone finds it useful:

http://github.com/owsy/vagrantfile-symfony