Data Mirroring & Replication
Beyond robocopy

Robocopy - a hidden Windows gem and indispensable part of many system administrator toolboxes since the XP/2003 days.

Chances are that you either have never heard of it or you have and you love it. In the formidable sea of its 80 something command line options, there's one that stands on its own as it transforms robocopy from a file copying tool into a data replicator.

This is, of course, the robocopy /mir option.

Often a go-to solution for both one-off and routine data migration jobs, it is undeniably useful, but not without its share of quirks and problems. Below we go through the most notable of these and show how they can be resolved by using Bvckup 2, so that you can sit back and replicate in style.

Shadow copying

One robocopy limitation that you are bound to hit pretty quickly is its inability to copy files that are currently in use by other running apps.

For example, an attempt to copy a mailbox file while Outlook is running will result in an error - The process cannot access the file because another process has locked a portion of the file. The only way around this is to use shadow copying. As you probably know, Windows shadow copying service allows any app with sufficient privileges to create a temporary static snapshot of a disk volume and use it to gain full access to every file, locked or not.

To use this with robocopy, a bit of scripting is required to create a snapshot with the vssadmin or vshadow command before doing the actual replication and to dispose of it afterwards.

Things can be simplified to a degree by using 3rd party tools like hobocopy and shadowspan, but these have their own issues. In comparison, Bvckup 2 includes comprehensive out-of-the-box support for shadow copying, including detailed status and error reporting and advanced fine-tuning options.

Block-level updates

Robocopy is really good at bulk copying, but when it comes to updating files, it always re-copies them in full even if the actual change is limited to just few bytes.

This is clearly not an issue for one-time migrations, but for any sort of periodic replication it will cause unnecessary delays that could be avoided by using rsync-style updating. To address this Bvckup 2 defaults to delta copying of larger files, whereby only modified *blocks* of files are copied over to destination. The benefits of delta copying are particularly pronounced when replicating very large slowly changing files, such as VM images and VHD containers, which just happen to be frequent subjects of many replication jobs.

Move detection

When a file is moved or renamed at the source, this change should ideally be replicated by simply moving the corresponding files at the destination.

Robocopy operates by reconciling all differences immediately, as it scans, so for any moved file it ends up deleting and re-copying its destination counterpart in full. Bvckup 2 employs a different approach.

It first does a complete scan of source and destination. Then it compares two resulting trees and generates a list of simple steps that would bring destination in sync with the source. And, finally, it executes these steps.

As a result both file and folder renames/moves are recognized and replicated as you'd expect them to be - with a single move.

Destination snapshots

Another way to speed up replication is to be smart with caching information that doesn't change between the runs. When replication process is the only entity that makes changes to destination, it really has no need to scan destination on every run if it retains a copy of destination contents from the last time. That's what Bvckup 2 does. It defaults to caching results of initial destination scan and then diligently updating resulting snapshot with every change that it makes to destination.

This lets the app to know the exact state of destination at the start of every run, without needing to scan it again.

This may not sound like much of an optimization, but scanning a large disk location can take from several minutes to well into an hour; especially over the network, and with slower NAS devices.

Faster scanning

When a file system scan is needed, it generally doesn't hurt to get through it as quickly as possible.

To that end Bvckup 2 employs parallel file system scanner that speeds up directory scans by traversing multiple subfolders at once.

This gives a slight boost to local scans, but it does an absolute wonder to over-the-network scans as it reduces pointless idling while waiting for remote queries to complete.

File system testing

Once you've used robocopy long enough, you inevitably learn about the /FFT flag.

This typically follows a period of intense head scratching as you stare at unchanged files that yet somehow get re-copied on every run. The reason for this is that file systems trim file timestamps when they store them. For example, FAT rounds up last modification time to 2 seconds, and creation time to 10 ms. Microsoft NTFS uses resolution of 1/10 of microsecond on both, and Samba NTFS can be set to whatever the vendor wants. Speaking of Samba, many Linux-based NAS devices not only use custom timestamp precision, but also don't support setting of creation times or basic file attributes such as System and Hidden. With robocopy you'd typically need to first realize that these issues exists and then work around them using command line flags like /FFT.

In contrast, Bvckup 2 takes care of all these issues automatically and also provides a summary of file system properties.

Email reports

Monitoring is an integral part of any automated process, and email alerts are the go-to option in many setups.

While perfectly feasible with a bit of external scripting, they generally cannot capture the full context of an error or a failure and thus provide only limited visibility into what is actually happening at the location. Email notifications are natively supported by Bvckup 2 and can be customized with over 50 run-time variables to capture all aspects of the job execution.

Better logging

Wading through reams of raw robocopy logs is not the most efficient way to perform root-cause analysis. Bvckup 2 implements simple, but remarkably convenient twist on conventional logs - the hierarchical logging. Log is displayed as a list of collapsible entries, each level down capturing progressively more context that its parent.

With all nodes collapsed, this gives concise summary for a quick inspection, and keeps all the details just a couple of clicks away for a complete drill down. Logs are stored as plain text files, perfectly grep'able, and they are serviced with a size- and age-based log rotation.
It always starts with just robocopy itself.

Then it turns into a small script to add a simple email alert on completion. Then a bit more to capture the logs. Then hook it up to Task Scheduler to run every few hours. Then tack on something else, and before you know it, you looking at something that is just few lines short of being semi-sentient.

Enter Bvckup 2...

Bvckup 2 was built from the ground up to focus data replication and mirroring backups, and these two things only.

It is fast and convenient, with a host of secondary features each supporting its primary function, which is to get data from A to B as quickly as possible.

Here's the app at a glance:
  • Data replication between any two file system locations, local or over the network
  • Full shadow copying support for handling locked and in-use files

  • Very fast bulk copier
  • Delta copying for rsync-like block-level file updates

  • Replication of moves/renames as a single operation
  • Automatic detection of file system properties, including timestamp resolution
  • Removable device tracking

  • Optional archival of deleted items, with automatic archive trimming
  • Support for flexible and powerful inclusion/exclusion filtering rules

  • Comprehensive logging, including log rotation
  • Support for simulated runs

  • Built-in scheduler - periodic, real-time or manual runs
  • Excellent user interface, including superb hierarchical log viewer

  • Email reporting and alerts
  • Support for running as a system service
  • Support for massive backups that weigh well into tens of million items
Give Bvckup 2 a try and see the difference a specialized high-performance replicator can make.
Download Fully-functional trial
Requires Windows XP or newer
Made by Pipemetrics in Switzerland

Dev blog
Miscellanea Press resources
On robocopy

Legal Terms