The book Patterns of Enterprise Application Architecture (PoEAA) laid the blueprints for Rails’ architecture. When choosing which enterprise design patterns to encode into the framework, Rails picked, to name a few, Active Record, Template View, Application Controller, etc. By covering these patterns with a sweet coating of convention-over-configuration, Rails simplifies pattern analysis a lot.

These design assumptions were absolutely pragmatic for the type of applications that Rails was targeting at. However, as applications growing more and more complex, developers are starting to realize these default architectural patterns may not scale very well. Typically, four main areas are overloaded in an enterprise application:

1. high coupling between domain model and data source,
2. bloated domain model with a mix of domain logic and application logic,
3. presentation behaviour leaked into views, and
4. high coupling between view data and template

To understand these problems better as well as to figure out possible solutions, I would like to walk you through some enterprise patterns from the same book that Rails’ architecture heavily bases upon.

Note that the patterns I am mentioning here will benefit more for complex applications, for example, an e-commerce web application selling digital goods. There’s clearly some overhead to use these patterns for simple projects where the default Rails patterns shine. Most importantly, you should only apply these patterns where they make sense. I am a strong believer in contextual solution and these patterns are definitely not the only way to go.

Data Mapper

Data Mapper is a layer that transfers data between objects and a database. It typically creates Domain Model objects by populating their attributes from the database.

Assuming we are building an e-commerce platform, we get a store object by going through the store mapper which connects to the database:

# app/controllers/stores_controller.rb

class StoreController < ApplicationController
  def show
    @store = StoreMapper.find(params[:id])
  end
end

The difference between Active Record and Data Mapper is actually that in the Active Record implementation, Domain Model not only encapsulates business logic, but also takes responsibility of database access, while in the Data Mapper implementation, Domain Model is ignorant of database and become Plain Old Ruby Object with business logic. Data Mapper allows database logic and the object model to evolve independently.

Note that the Active Record pattern and the Data Mapper pattern mentioned are not the Ruby libraries these patterns inspire. The active_record gem is an implementation of the Active Record pattern. The data_mapper gem is also a “Active Record”-ish implementation, although it has elements of the Data Mapper pattern. For those who are interested in seeing the implementation difference between these two gems, I created a gist for you.

Here is Martin Fowler on the choice of Active Record or Data Mapper for Domain Model:

An OO domain model will often look similar to a database model, yet it will still have a lot of differences. A Domain Model mingles data and process, has multivalued attributes and a complex web of associations, and uses inheritance.

As a result I see two styles of Domain Model in the field. A simple Domain Model looks very much like the database design with mostly one domain object for each database table. A rich Domain Model can look different from the database design, with inheritance, strategies, and other [Gang of Four] patterns, and complex webs of interconnected objects. A rich Domain Model is better for more complex logic, but is harder to map to the database. A simple Domain Model can use Active Record, whereas a rich Domain Model requires Data Mapper.

In an enterprise Rails application, it’s not rare to see complex domain models stuffed with associations, validations, scopes and business logics. It has become a growing pain to deal with such “fat models”. We need to separate the database concerns out into a new dedicated layer: the data mappers. However, as far as I know, there’s no ORM in Ruby giving us such separation yet, although it’s been said the upcoming 2.0 release of the data_mapper gem will fully implement the Data Mapper pattern. Before we are able to consume the new data_mapper gem, is there a way to mitigate existing overloaded Rails models? Besides, switching an ORM for existing code is not effortless.

[Updated 01/10 2012], Piotr Solnica from the data_mapper gem made an official announcement saying they are actively working on Data Mapper 2.0. Announcement goes here.

As a compromised solution, I would extract database related logic for each ActiveRecord::Base model (e.g., validations and scopes) out into a module and then mix it in:

# app/mappers/store_mapper.rb

module StoreMapper
  extend ActiveSupport::Concern

  included do
    # validations
    validates_presence_of :name
    ...

    # scopes
    scope :disabled, where(:disabled => true)
    ...
  end
end

# app/models/store.rb

class Store < ActiveRecord::Base
  include StoreMapper

  # associations
  has_many :products
  belongs_to :company
  ...

  # business logics
  def calculate_sells(from_date, to_date)
    # calculate sells from date to date
  end
  ...
end

This half-baked solution, although not migrating to the Data Mapper pattern, cleanly isolates the definitions of database logic with the ones of business logic. Of course, it’s recommended to use the Data Mapper pattern where possible.

Service Layer

Layering is one of the most common techniques to break apart a complex software system. The higher layer uses service defined by the lower layer, but the lower layer is unaware of the higher layer. Each layer usually hides its lower layers from the layers above. A Rails project is typically divided into three layers:

1. presentation layer, including views and controllers,
2. domain layer, including models, and
3. data source layer, which is hidden behind models extending from ActiveRecord::Base.

It’s not difficult to understand the Data Mapper pattern is an effort to break down #3 into another layer to lower the complexity of models. However, in the context of enterprise application, models are still overwhelmed by complicated business logic. As Fowler pointed out, business logic can be further divided into “domain logic” and “application logic”, and Service Layer is a pattern to encapsulate model’s “application logic” by establishing a boundary where the presentation layers interact with the application:

… Service Layer is a pattern for organization business logic. Many Designers, including me, like to divide “business logic” into two kinds: “domain logic”, having to do purely with the problem domain (such as strategies for calculating revenue recognition on a contract), and “application logic”, having to do with application responsibilities [Cockburn UC] (such as notifying contract administrators, and integrated applications, of revenue recognition calculations). Application logic is sometimes referred to as “workflow logic”, although different people have different interpretations of “workflow”.

He also pointed out, Service Layer is a good fit for coordinating operations among multiple models, as well as for talking to multiple presentation layers:

The benefit of Service Layer is that it defines a common set of application operations available to many kinds of clients and it coordinates an application’s response in each operation. …

The easier question to answer is probably when not to use it. You probably don’t need a Service Layer if your application’s business logic will only have one kind of client - say, a user interface - and its use case responses don’t involve multiple transactional resources.

To translate this into a code example, let’s assume in an e-commerce platform, we need to email monthly sells report to the store owner. In this example, StoreService acts as a coordinator of multiple models for the “mailing sells report” workflow:

# app/services/store_service.rb

class StoreService
  def mail_sells_report(store_id, from_date, to_date)
    store = StoreMapper.find(store_id)
    store_sells = store.calculate_sells(from_date, to_date)
    store_report = SellsReport.generate(store_sells)
    SellsReportMailer.report_mailer(store.owner, store_report).deliver
  end
end

Another place where Service Layer shines is reusing workflows for different controllers, for example, the same workflow in the above example is used in the Storefront::StoresController for store front and in the Api::StoresController for REST API calls. Service Layer becomes a Facade in this case.

Presentation Model

It’s not uncommon to present multiple models in a view. Most importantly, not all attributes of a model are needed for a certain presentation. In a complex system where there are lots of screens, views become a very busy place for extracting state and behavior from models. The Presentation Model comes to ease the pain:

Presentation Model pulls the state and behavior of the view out into a model class that is part of the presentation. The Presentation Model coordinates with the domain layer and provides an interface to the view that minimizes decision making in the view. The view either stores all its state in the Presentation Model or synchronizes its state with Presentation Model frequently.

As an example, let’s build a view for the scenario “creating a store for a user”:

# app/presenters/create_store_presenter.rb

class CreateStorePresenter
  attr_reader :store, :user

  delegate :name, :to => :store, :prefix => true
  delegate :email, :to => :user, :prefix => true

  def initialize(params)
    @store = Store.new(params[:store])
    @user = @store.build_user(params[:user])
  end

  def valid?
    @user.valid? && @store.valid?
  end

  def save
    @store.save
  end
end

<!-- app/views/stores/create_store.html.erb -->

<h1>Create a store</h1>

<%= form_for(@create_store_presenter) do |f| %>
  <p>
    <%= f.label :store_name %><br/>
    <%= f.text_field :store_name %>
  </p>

  <p>
    <%= f.label :user_email %><br/>
    <%= f.text_field :user_email %>
  </p>

  <p>
    <%= f.submit "Create" %>
  </p>
<% end %>

In the example, we wrap two models (Store and User) into a presenter and extract only the attributes needed (Store#name and User#email) for the “create store” screen. To summarize this pattern, I would like to once again consult Fowler:

Presentation Model is a pattern that pulls presentation behavior from a view. … It’s useful for allowing you to test without the UI, support for some form of multiple view and a separation of concerns which may make it easier to develop the user interface.

… Presentation Model allows you to write logic that is completely independent of the views used for display. You also do not need to rely on the view to store state. …

Two Step Views

Rails comes with a neat templating system that allows you to quickly create dynamic pages. However, this Template View pattern has drawbacks as Fowler pointed out, especially in a situation where the view is very complex:

… the common implementations make it too easy to put complicated logic in the page, thus making it hard to maintain, particularly by nonprogrammers. You need good discipline to keep the page simple and display oriented, putting logic in the helper. …

What’s worse, if the display of a view is based on conditions, for example in a multi-appearance application, you will find logic that determines which template to render leaked into many places in views or controllers. Again, this is fine for a simple Rails application. But it becomes unmanageable as the application growing more complex.

Let’s think about an example: an e-commerce platform supports multiple stores and each store has its own storefront to display a product. It’s common to see the following solution:

<!-- app/views/products/_product.html.erb -->

<% if store.amazon_store? %>
  <%= render :partial => '/amazon/products/_product.html.erb', :object => product %>
<% elsif store.apple_store? %>
  <%= render :partial => '/apple/products/_product.html.erb', :object => product %>
<% else %>
  <%= render :partial => '/default/products/_product.html.erb', :object => product %>
<% end %>

The determination logic for displaying a product based on store is leaked into the product partial. Apparently, to build the whole multi-appearance application, this approach does not scale. Thankfully, Fowler has something good for us - the Two Step View pattern:

… You may also want to make global changes to the appearance of the site easily, but common approaches using Template View or Transform View make this difficult because presentation decisions are often duplicated across multiple pages or transform modules. A global change can force you to change several files.

Two Step View deals with this problem by splitting the transformation into two stages. The first transforms the model data into a logical presentation without any special formatting; the second converts that logical presentation with the actual formatting needed. …

From the above diagram, the multi-storefront example can be reimplemented with the Two Step View pattern. The process is in two steps. The first step is to transform the product data into a logical presentation. The second step is to convert this logical presentation into different HTML.

Note that in the implementation, a gem called cells is used to help define logical presentation. The cells gem is very helpful in this respect although it was originally designed for other purposes.

For the logical presentation, we define it to display name, description, price and reviews of a product for every store:

<!-- app/views/products/_product.html.erb -->

<%= render_cell :product, :name, product.name %>
<%= render_cell :product, :description, product.description %>
<%= render_cell :product, :price, product.price %>
<%= render_cell :product, :reviews, product.reviews %>

We then define three strategies (ProductCell, Amazon::ProductCell, and Apple::ProductCell) to convert the logical presentation to different HTML. We make use of cells’ strategy builder to return strategy class based on current store in session:

# app/cells/product_cell.rb

class ProductCell < Cell::Rails
  # return strategy class based on current store in session
  build { "#{current_store.name}::ProductCell".classify.constantize rescue nil }

  def name(name)
    content_tag(:h1, name)
  end

  ...

  def reviews(reviews)
    # display reviews
  end
end

# app/cells/amazon/product_cell.rb

module Amazon
  class ProductCell < ::ProductCell
    def name(name)
      content_tag(:p, name)
    end

    ...

    def reviews(reviews)
      # display reviews for Amazon store
    end
  end
end

# app/cells/apple/product_cell.rb

module Apple
  class ProductCell < ::ProductCell
    def name(name)
      content_tag(:div, name, :class => 'title')
    end

    ...

    def reviews(reviews)
      # display reviews for Apple store
    end
  end
end

As you may see, the Two Step View pattern makes multi-appearance implementation manageable in a way that different appearance implementations are organized in a set of strategy classes to parse a common logial presentation.

Summary

The default enterprise design patterns encoded into Rails are perfect matches for small/medium size projects. They are light weight and easy to understand. However, as the application growing more mature, these patterns do not scale well due to layers taking too much responsibility. That said, a “fat” layer need to be broken into smaller ones:

1. Data Mapper is an effort to extract out data source layer from Domain Model that implements Active Record,
2. Service Layer is an endeavor to extract out application logic from Domain Model,
3. Presentation Model is an attempt to extract out presentation logic from a single or multiple Domain Model, and
4. Two Step View is a try to break down presentation logic into two processing steps so that a view can be generated in different formats.

In return for breaking apart a system into smaller layers, each layer becomes easier to maintain, reuse, test and scale.

Last but not least, I would like to thank Martin Fowler for his awesome book and would love to hear any feedback for you.

Testing REST web services client has never been easy. It requires a running web server, multiple threads, network conection and complex transaction management.

Ideally, REST web service client test should have the following characteristics:

1. The experience of testing REST resource is similar to that of testing a ActiveRecord model
2. Start up and shut down web server for the purpose of running REST web services
3. Rollback test data after each test
4. Control fixture creation for REST web services
5. All tests are automatic

In this article, I demonstrate solutions to each of those mentioned.

As an example throughout the article, let’s assume we have web services for a model called Task and we are testing its corresponding client code. Here is a sample action in the TasksController of the web server:

# server/app/controllers/tasks_controller.rb

def index
  @tasks = Task.all
  render :status => :ok, :json => @tasks
end

We render @tasks as the JSON format where to_json is called on the object. When you run ”curl http://localhost:3000/tasks.json”, you will get the following result:

$ curl http://localhost:3000/tasks.json
[{"id":1,"name":"Write a blog post","created_at":"2011-07-20T04:05:41Z","updated_at":"2011-07-20T04:05:41Z","ends_at":"2011-08-20T03:15:00Z"}]

ActiveResource

In order to test our REST web services, we need a HTTP client. There are lots of them out there, but I found ActiveResource the most enjoyable to use in a less complex situation. ActiveResource provides ActiveRecord compatible APIs, so when writing web service client tests, we feel like we are writing unit tests for a ActiveRecord model.

To start with, we just need to extend it from ActiveResource::Base and give it the web server URL and representation format. That’s it!

# client/app/models/task.rb

class Task < ActiveResource::Base
  self.site = "http://localhost:3000"
  self.format = :json
end

And we are using it as if you are using an ActiveRecord object:

# client/spec/models/task_spec.rb

describe Task do
  it "should return all the tasks" do
    @tasks = Task.all
    @tasks.size.should == 1
  end
end

Web Server

To maintain a zero-setup test environment, we’ll have our test control the stratup and shutdown of a web server. By having the tests start and stop the web server, they can be easily run with no external dependencies.

To control the startup and shutdown of a web server before and after all suites run, it’s as simple as having something like this:

# client/spec_helper.rb

RSpec.configure do |config|
  config.before :suite do
    @server = Server.new(server_path)
    @server.start
  end

  config.after :suite do
    @server.stop
  end
end

The implementation of Server is also dead simple. Execute “script/rails server -d” to daemonize the server and issue a kill to stop it:

# client/lib/server.rb

Class Server
  def initialize(server_path)
    @server_path = server_path
  end

  def start
    `#{rails_script} server -d -e test`
  end

  def stop
    pid = File.read(pidfile)
    `kill -9 #{pid}`
  end

  private

  def rails_script
    File.join(@server_path, 'script', 'rails')
  end

  def pidfile
    File.join(@server_path, 'tmp', 'pids', 'server.pid')
  end
end

Transaction Rollback

For testing strategies of web services, most people recommend to either truncate test data on each run or to mock out the request and response. These approaches are less ideal since they’re either less effective or they’re not testing full stack of the targeted web services.

Would it be possible to wrap web services calls in a transaction and rollback data after each test, like what Rails’s transactional fixture does?

Of course! But let’s first try to understand why making transaction rollback for web service calls is difficult:

• Tests and web server are running in two separate threads, web server’s transactional boundary can’t expand to tests

• Web service calls may commit its transaction

• Web server doesn’t know when to rollback the test data

To overcome these problems, we’ll need to fully control the lifecycle of web server’s database connection in the client tests. But how are we able to do this in a client-server architecture?

dRuby to rescue!

For those who are not familiar with it, dRuby is as the Remote Method Invocation to Java as to Ruby. It allows methods to be called in one Ruby process upon a Ruby object located in another Ruby process. Here is a good introduction to brush you up.

We’ll make use of dRuby to directly control the lifecycle of web service’s database connection (ActiveRecord::Base.connection) in our web services client tests. To do that, we add the following code to web server’s ”config/environments/test.rb”:

# server/config/environments/test.rb

config.after_initialize do
  ActiveRecord::ConnectionAdapters::ConnectionPool.class_eval do
    alias_method :old_checkout, :checkout

    def checkout
      @cached_connection ||= old_checkout
    end
  end

  require 'drb'
  DRb.start_service("druby://localhost:8000", ActiveRecord::Base)
end

The above code snippet does two things:

1. Patch ActiveRecord::ConnectionAdapters::ConnectionPool#checkout to make sure only one connection is shared across threads
2. Start a dRuby service for ActiveRecord::Base to be used in tests

In case you are wondering why it’s necessary to share one database connection across threads: ActiveRecord creates one database connection for each thread in its connection pool. Our web service client tests run in a separate thread from the server’s so it’s impossible to track which connection to rollback data for the web services calls. What we are doing here is to make sure there is only one connection created and we always rollback data for this connection.

After the aforementioned setup, we are able to expand the transaction boundary to tests:

# client/spec/models/task_spec.rb

describe Task do
  before :all do
    @semaphore = Mutex.new
    DRb.start_service
    @remote_base = DRbObject.new nil, "druby://localhost:8000"
  end

  before :each do
    begin_remote_transaction
  end

  after :each do
    rollback_remote_transaction
  end

  it "creates a task through web serices" do
    task = Task.create(:name => "Write a blog post", :ends_at => Date.tomorrow)
    Task.find(task.id).should == task
  end

  private

  def begin_remote_transaction
    @semaphore.lock
    @remote_base.connection.increment_open_transactions
    @remote_base.connection.transaction_joinable = false
    @remote_base.connection.begin_db_transaction
  end

  def rollback_remote_transaction
    @remote_base.connection.rollback_db_transaction
    @remote_base.connection.decrement_open_transactions
    @remote_base.clear_active_connections!
    @semaphore.unlock
  end
end

Voila! With dRuby, we use begin+rollback to isolate changes of web services calls to the database, instead of having to delete+insert for every test case. A huge performance boost!

Note that the Mutex lock in the code is to make sure that multiple web service client tests can run concurrently, for exmaple, using the parallel_tests gem. Without this lock, while the remote ActiveRecord connection is shared, the tests will behavior strangely in a multi-threads environment. You can ignore those lines if your web service client tests never run concurrently.

We can easily refactor out the begin_remote_transaction method and the rollback_remote_transaction method to spec_helper.rb, so that our web services client tests have little difference from usual ActiveRecord unit tests.

# client/spec_helper.rb

RSpec.configure do |config|
  config.before :all do
    @semaphore = Mutex.new
    DRb.start_service
    @remote_base = DRbObject.new nil, "druby://localhost:8000"
  end

  config.before :each do
    @semaphore.lock
    @remote_base.connection.increment_open_transactions
    @remote_base.connection.transaction_joinable = false
    @remote_base.connection.begin_db_transaction
  end

  config.after :each do
    @remote_base.connection.rollback_db_transaction
    @remote_base.connection.decrement_open_transactions
    @remote_base.clear_active_connections!
    @semaphore.unlock
  end
end

# client/spec/models/task_spec.rb

describe Task do
  it "creates a task through web serices" do
    task = Task.create(:name => "Write a blog post", :ends_at => Date.tomorrow)
    Task.find(task.id).should == task
  end
end

Fixture Creation

Most of the time, we create test fixtures to quickly define prototypes for each of the models and ask for instances with properties that are important to the test at hand. But in the context of REST web services, we can’t create fixtures unless there is a REST API defined. To break this constraint, we use dRuby to open up another channel to directly interact with fixture data on web server.

Assuming we are using the factory_girl gem for fixture creation, We create a dRuby service for port discovery and a dRuby service for each fixture instance:

# server/lib/drb_active_record_instance_factory.rb

require 'factory_girl'

class DRbActiveRecordInstanceFactory
  def get_port_for_fixture_instance(factory_instance)
    port = create_port
    inst = Factory.create(factory_instance)
    DRb.start_service("druby://localhost:#{port}", inst)
    port
  end

  def create_port
    # create a random port
  end
end

DRb.start_service('druby://localhost:9000', DRbActiveRecordInstanceFactory.new)

In tests, we ask for the port of the fixture instance and query its corresponding remote reference:

# client/spec/models/task_spec.rb

describe Task do
  before :all do
    @drb_factory = DRbObject.new(nil, 'druby://localhost:9000')
  end

  before do
    remote_task_port = @drb_factory.get_port_for_fixture_instance(:task)
    @remote_task = DRbObject.new(nil, "druby://localhost:#{remote_task_port}")
  end

  it "should ..."
    # test REST web services calls with @remote_task
  end
end

Summary

Testing REST web services client can be less complex if we have full control over objects on the web server. ActiveResource and dRuby stand out to help! They make writing web service client tests feel like writing local unit tests.

Distributed version control systems open up lots of flexibility and provide lots of efficiency in work-flow than their centralized cousins (e.g. Perforce). There’s often value to use DVCS in the team.

However, the case that I run into frequently is where there is a corporate standard for a deficient VCS, but the team wish to work efficiently by using a more powerful VCS. In that case, there are many successful stories of using multiple VCS: DVCS for local version control and committing to a shared centralized VCS with a DVCS-to-VCS bridge.

Git is particularly good at this aspect which provides tons of bridges to other VCS (That’s another good reason for preferring Git over Mercurial :-)). Lately I have been using the git-p4 bridge to synchronize codes between a centralized Perforce repository and a local Git repository. This post is a tutorial on how to set this up.

Setting up the Perforce command line client

The git-p4 bridge requires the p4 command line client properly set up. There is a lengthy tutorial on Perforce’s documentation website. The following is a short sums-up:

1. Install the p4 command line client (use MacPort or Homebrew if you are a Mac guy :-))

2. Create a .p4settings file in your home directory with the global environment settings for your Perforce repository. And in your .bashrc, export P4CONFIG=/path/to/your/.p4settings. For example, here is what my settings look like:

   P4PORT=repo_url:repo_port
   P4USER=user_name
   P4PASSWD=password
   P4CLIENT=client_workspace_name
   P4EDITOR=vim

3. Run “p4 client” to define the workspace mappings

4. Run “p4 info” to verify the settings

After you have done all these, don’t forget to issue “p4 sync repo_url” to test whether you are able to checkout stuff from your repository.

Installing git-p4 bridge

The git-p4 bridge is a Python script to enable bidirectional operation between a Perforce depot and Git. It doesn’t come with the Git distribution by default. To make it invokable in your system, you need to download the script and put it into your system path. For me, I clone the Git source code from GitHub and make a soft-link to the script:

1. Clone the Git repository to somewhere

   git clone https://github.com/git/git.git git_source_path

2. Create a soft-link to the git-p4 script in one of your system paths

   ln -s git_source_path/contrib/fast-import/git-p4 /usr/bin/git-p4 

Windows users need to include the git-p4.bat file in the system path.

Commands

Four things to remember when using git-p4:

• Instead of using “git push” to push local commits to remote repository, use “git-p4 submit”
• Instead of using “git fetch” to fetch changes from remote repository to local, use “git-p4 sync”
• Instead of using “git pull” to fetch and merge changes from remote repository to local, use “git-p4 rebase”
• Instead of using “git merge” to merge local branches, use “git rebase”

For the last one, the reason is that when you run “git merge”, Git creates an extra commit on top of the stack for the merging. This is not something we wanna show in the remote non-git repository. So we merge code with “git rebase”. Detailed explanation of the difference between git-merge and git-rebase goes here.

Workflow

There is a detailed explanation on the usage of git-p4 in Git’s source. Here is an example almost covering daily usage:

1. Login with “p4 login” and clone a Perforce project:

   git-p4 clone //depot/path/project

2. Make some changes to the project and commit locally to Git:

   git commit changed_file

3. In the meantime somebody in the team submitted changes to the remote Perforce repository. Merge it to your local repository:

   git-p4 rebase

4. Submit your local changes back to Perforce:

   git-p4 submit

Ignoring the .gitignore file

Normally we check in a .gitignore file to ignore files that we don’t want to check into the remote repository for each project. However, since the remote repository is not Git, it’s meaningless to check in this .gitignore file. To ignore this file, simply add an entry to .git/info/exclude.

Under the hook

There is no magic happening with git-p4. What it does is simply invoking the p4 command line tool to download sources to local, and then clone a Git repository out of it. You can simply verify this by typing “git branch -a”:

You may be amazed once again by how flexible the design of Git is which makes it possible to bridge to multiple VCS!

Summary

To quote Martin Fowler’s opinions on dual VCS, “a lot of teams can benefit from this dual-VCS working style, particularly if there’s a lot of corporate ceremony enforced by their corporate VCS. Using dual-VCS can often make both the local development team happier and the corporate controllers happier as their motivations for VCS are often different”.