Beyond Git Submodules: Syncing Codebases with Google's Copybara

Managing split codebases is a common architectural headache. Many engineering organizations face a difficult tension: they want the speed and consolidation of an internal monorepo, but they also need to publish open-source libraries, share code with external partners, or ingest upstream dependencies.

Traditional Git patterns fall short here. Git submodules are notoriously fragile and difficult for developers to navigate. Git subtrees can work, but they quickly become unwieldy when you need to apply complex transformations, such as scrubbing internal configuration files, stripping proprietary comments, or rewriting import paths. Custom bash scripts relying on rsync and sed are the default fallback, but they are highly error-prone and difficult to maintain.

Copybara, an open-source tool developed and used internally by Google, offers a different approach. Rather than acting as a simple repository mirror, Copybara is a programmable transformation engine designed to move code deterministically between repositories while restructuring it on the fly.

The Architecture of Stateless Syncing

Most synchronization tools rely on an external database or a stateful CI/CD runner to track which commits have already been migrated. This introduces a single point of failure and makes local testing difficult.

Copybara solves this by being entirely stateless. It stores the synchronization state directly within the destination repository. When Copybara successfully migrates a change, it appends a metadata label to the destination commit message, typically formatted as GitOrigin-RevId: [commit-hash].

When a workflow runs, Copybara inspects the commit history of the destination repository, extracts the last imported revision ID, and calculates the delta from the source repository. Because the state lives in the git history itself, any developer or automated CI runner can execute the same configuration file and achieve identical, deterministic results.

To prevent split-brain scenarios, Copybara requires you to designate one repository as the authoritative source of truth. However, this does not mean workflows must be strictly one-way. Contributions can still be accepted in non-authoritative repositories (such as a public GitHub repo) and merged back into the authoritative internal repository, with Copybara handling the transformation and path mapping bidirectionally.

Starlark as a Transformation Engine

Copybara configurations are written in Starlark, a dialect of Python originally designed for the Bazel build system. Starlark is deterministic and side-effect free, meaning executing the same configuration twice is guaranteed to produce the same output.

Instead of writing fragile regex scripts, developers define declarative workflows in a copy.bara.sky file. These workflows specify an origin, a destination, the files to include or exclude, and a sequence of transformations.

Here is a practical example of a Copybara workflow that syncs a public repository into an internal monorepo, moving the files into a specific subdirectory and rewriting build targets:

sourceUrl = "ssh://git@github.com/example/public-library.git"
destinationUrl = "ssh://git@github.com/example/internal-monorepo.git"

core.workflow(
    name = "default",
    origin = git.origin(
        url = sourceUrl,
        ref = "master",
    ),
    destination = git.destination(
        url = destinationUrl,
        fetch = "master",
        push = "master",
    ),
    destination_files = glob(["third_party/public_library/**"]),
    authoring = authoring.pass_thru("Copybara <copybara@example.com>"),
    transformations = [
        core.move("", "third_party/public_library"),
        core.replace(
            before = "//src/internal/compat",
            after = "//third_party/public_library/compat",
            paths = glob(["**/BUILD"])
        ),
    ],
)

In this configuration, destination_files uses a glob pattern to ensure Copybara only touches the third_party/public_library directory in the destination monorepo, leaving the rest of the codebase untouched. The transformations block moves the incoming code into that subdirectory and rewrites internal Bazel build paths on the fly.

The Developer Angle: Setup and the First-Run Trap

Adopting Copybara requires a shift in how you build and run your developer tooling. Because Copybara is built with Java and Bazel, there are no simple pre-compiled binaries distributed through standard package managers, though Arch Linux users can find it in the AUR as copybara-git.

To run Copybara locally, you must compile it from source. This requires JDK 11 and Bazel installed on your machine:

git clone https://github.com/google/copybara.git
cd copybara
bazel build //java/com/google/copybara:copybara_deploy.jar

This command compiles an executable "uberjar" at bazel-bin/java/com/google/copybara/copybara_deploy.jar. You can run your configuration using this JAR:

java -jar bazel-bin/java/com/google/copybara/copybara_deploy.jar copy.bara.sky

When running a new Copybara workflow for the first time, you will inevitably hit this error:

ERROR: Cannot find last imported revision. Use --force if you really want to proceed with the migration...

Because the destination repository does not yet contain any commits with the GitOrigin-RevId label, Copybara has no baseline to calculate the diff. To establish the initial sync, you must append the --force flag. This tells Copybara to copy the current state of the source repository wholesale and write the initial tracking label into the destination commit history.

Automating the Contribution Loop with Pull Requests

If you are using Copybara to manage an open-source project, you do not want to commit external changes directly to your internal master branch. Instead, you can configure Copybara to automatically generate pull requests in your destination repository.

By swapping git.destination with git.github_pr_destination, Copybara will package the incoming changes into a new branch and open a PR:

destination = git.github_pr_destination(
    url = destinationUrl,
    destination_ref = "master",
    pr_branch = "from_public_repo",
    title = "PR from external public repo",
    body = "This is an automated pull request generated by Copybara.",
    integrates = [],
)

This allows internal developers to review, run CI checks, and merge external contributions using their standard internal code review workflows.

The Trade-offs: Is Copybara Worth the Overhead?

Copybara is a highly specialized tool, and it is not the right choice for every team.

If your synchronization needs are simple, such as mirroring an entire repository to a backup location without modifying the file structure or scrubbing content, Copybara is over-engineered. A simple GitHub Action running standard git commands will be faster to set up and easier to maintain.

Furthermore, Copybara's developer experience reflects its internal Google heritage. The documentation is sparse, there is no official stable release process (Google recommends building from HEAD or using weekly, untested snapshot releases), and the dependency on Bazel and Java adds significant weight to your toolchain.

However, if you are managing a complex monorepo where you must enforce strict boundaries between proprietary and public code, or if you need to perform programmatic, multi-step code transformations during sync, Copybara is unmatched. Its stateless design makes it incredibly reliable in automated CI/CD pipelines, and its Starlark configuration engine provides the precise control needed to keep split codebases in lockstep.