Data mirroring & replication
Beyond robocopy

Robocopy - a hidden Windows gem and indispensable part of many system administrators' toolboxes since the XP/2003 days.

Chances are that you either have never heard of it or you have and you love it. In the formidable sea of its 80 something command line options, there's one that stands on its own as it transforms robocopy from a file copying tool into a data replicator.

This is, of course, the robocopy /mir option.

Often a go-to solution for both one-off and routine data migration jobs, it is undeniably useful, but is not without its share of quirks and problems. Below we go through the most notable of these and show how they can be resolved by using Bvckup 2, so that you can sit back and replicate in style.

Shadow copying

One robocopy limitation that you are bound to hit pretty quickly is its inability to copy files that are currently in use by other running apps.

For example, an attempt to copy a mailbox file while Outlook is running will result in an error - The process cannot access the file because another process has locked a portion of the file. The only way around this is to use shadow copying. As you probably know, Windows shadow copying service allows any app with sufficient privileges to create a temporary static snapshot of a disk volume and use it to gain full access to every file, locked or not.

To use this with robocopy, a bit of scripting is required to create a snapshot with either the vssadmin or vshadow command before doing the actual replication and to dispose of the snapshot afterwards.

Things can be simplified to a degree using 3rd party tools like hobocopy and shadowspan, but these have their own issues. In comparison, Bvckup 2 includes comprehensive out-of-the-box support for shadow copying, including detailed status and error reporting and advanced fine-tuning options.

Block-level updates

Robocopy is really good at bulk copying, but when it comes to updating files, it always re-copies them in full even if the actual change is limited to just few bytes.

This is clearly not an issue for one-time migrations, but for any sort of periodic replication it will cause unnecessary delays that could be avoided by using rsync-style updating. To address this Bvckup 2 defaults to delta copying of larger files, whereby only modified *blocks* of files are copied over to the destination. The benefits of delta copying are particularly pronounced when replicating very large slowly changing files, such as VM images and VHD containers, which just happen to be frequent subjects of many replication jobs.

Move detection

When a file is moved or renamed at the source, this change should ideally be replicated by simply moving the corresponding files at the destination.

Robocopy operates by reconciling all differences immediately, as it scans, so for any moved file it ends up deleting and re-copying its destination counterpart in full. Bvckup 2 employs a different approach.

It first does a complete scan of the source and destination. Then it compares two resulting trees and generates a list of simple steps that would bring the destination in sync with the source. And, finally, it executes these steps.

As a result both file and folder renames/moves are recognized and replicated as you'd expect them to be - with a single move.

Destination snapshots

Another way to speed up replication is to be smart with caching information that doesn't change between the runs. When the replication process is the only entity that makes changes to the destination, it really has no need to scan the destination on every run if it retains a copy of the destination's contents from the last time. That's what Bvckup 2 does. It defaults to caching the results of the initial destination scan and then diligently updates the resulting snapshot with every change that it makes to destination.

This lets the app know the exact state of the destination at the start of every run, without needing to scan it again.

This may not sound like much of an optimization, but scanning a large disk location can take from several minutes to well into an hour; especially over the network, and with slower NAS devices.

Faster scanning

When a file system scan is needed, it generally doesn't hurt to get through it as quickly as possible.

To that end Bvckup 2 employs a parallel file system scanner that speeds up directory scans by traversing multiple subfolders at once.

This gives a slight boost to local scans, but it does an absolute wonder to over-the-network scans as it reduces pointless idling while waiting for remote queries to complete.

File system testing

Once you've used robocopy for long enough, you inevitably learn about the /FFT switch.

This typically follows a period of intense head scratching as you stare at unchanged files that yet somehow get re-copied on every run. The reason for this is that the file system trims file timestamps when it stores them. For example, FAT rounds up the last modification time to 2 seconds, and the creation time to 10 ms. Microsoft's NTFS uses a resolution of 1/10 of microsecond on both, and Samba's NTFS can be set to whatever the vendor wants. Speaking of Samba, many Linux-based NAS devices not only use custom timestamp precision, but also don't support the setting of creation times or basic file attributes such as System and Hidden. With robocopy you'd typically need to first realize that these issues exists and then work around them using command line switches like /FFT.

In contrast, Bvckup 2 takes care of all these issues automatically and also provides a summary of the file system properties.

Email reports

Monitoring is an integral part of any automated process, and email alerts are the go-to option in many setups.

While perfectly feasible with a bit of external scripting, they generally cannot capture the full context of an error or a failure and thus provide only limited visibility into what is actually happening at the location. Email notifications are natively supported by Bvckup 2 and can be customized with over 50 run-time variables to capture all aspects of the job execution.

Better logging

Wading through reams of raw robocopy logs is not the most efficient way to perform a root-cause analysis. Bvckup 2 implements a simple, but remarkably convenient twist on conventional logs - hierarchical logging. A log is displayed as a list of collapsible entries, each level down capturing progressively more context than its parent.

With all nodes collapsed, this gives a concise summary for a quick inspection, but still keeps all the details just a couple of clicks away for a complete drill down. Logs are stored as plain text files, perfectly grep'able, and they are serviced with a size- and age-based log rotation.
It always starts with just robocopy itself.

Then it turns into a small script to add a simple email alert on completion. Then a bit more to capture the logs. Then you hook it up to Task Scheduler to run every few hours. Then tack on something else, and before you know it, you are looking at something that is just a few lines short of being semi-sentient.

Enter Bvckup 2 ...

The program was built from the ground up with the singular focus on data replication.

It is fast and convenient with a extensive list of secondary features, each supporting its primary function - getting the data from A to B as quickly as possible.

Here it is at a glance:
  • Data replication between any two file system locations, local or over the network
  • Full shadow copying support for handling locked and in-use files

  • Very fast bulk copier
  • Delta copying for rsync-like block-level file updates

  • Replication of moves/renames as a single operation
  • Automatic detection of file system properties, including timestamp resolution
  • Removable device tracking

  • Optional archival of deleted items, with automatic archive trimming
  • Support for flexible and powerful inclusion/exclusion filtering rules

  • Comprehensive logging, including full support for log rotation
  • Support for simulated runs

  • Built-in scheduler - periodic, real-time or manual runs
  • Excellent user interface, including a superb hierarchical log viewer

  • Email reporting and alerts
  • Support for running as a system service
  • Support for massive backups that weigh well into the tens of millions of items
Give Bvckup 2 a try and see the difference a professional file replicator can make.
Made by IO Bureau in Switzerland
Support

Updates
Blog / RSS
Follow Twitter
Reddit
Miscellanea Press kit
Testimonials
Company Imprint

Legal Terms
Privacy