Data mirroring & replication
a hidden Windows gem and indispensable part of many system administrators' toolboxes
since the XP/2003 days.
Chances are that you either have never heard of it or you have and you
In the formidable sea of its 80 something command line options, there's one
that stands on its own as it transforms robocopy from a
file copying tool into a data replicator
This is, of course, the robocopy /mir
Often a go-to solution for both one-off and routine data migration
jobs, it is undeniably useful, but is not without its share of quirks and
problems. Below we go through the most notable
of these and show how they can be resolved by using
so that you can sit back and replicate in style.
One robocopy limitation that you are bound to hit pretty
quickly is its inability to copy files that are currently
in use by other running apps.
For example, an attempt to copy a mailbox file
while Outlook is running will result in an error -
The process cannot access the file because another
process has locked a portion of the file.
The only way around this is to use
As you probably know, Windows shadow copying service allows any
app with sufficient privileges to create a temporary static
snapshot of a disk volume and use it to gain full access
to every file, locked or not.
To use this with robocopy, a bit of scripting is required to
create a snapshot with either the
command before doing the actual replication and to dispose of the
Things can be simplified to a degree using 3rd party tools like
but these have their own issues.
includes comprehensive out-of-the-box support for shadow copying,
including detailed status and error reporting and
advanced fine-tuning options.
Robocopy is really good at bulk copying, but when
it comes to updating
files, it always re-copies
them in full even if the actual change is limited to
just few bytes.
This is clearly not an issue for one-time migrations, but
for any sort of periodic replication it will cause
unnecessary delays that could be avoided by using
To address this Bvckup 2
of larger files,
whereby only modified *blocks* of files are
copied over to the destination.
The benefits of delta copying are particularly pronounced
when replicating very large slowly changing files, such as
which just happen to be frequent subjects of many replication jobs.
When a file is moved or renamed at the source, this
change should ideally be replicated by simply moving
the corresponding files at the destination.
Robocopy operates by reconciling all differences immediately,
as it scans, so for any moved file it ends up deleting and
re-copying its destination counterpart in full.
employs a different approach.
It first does a complete scan of the source and destination.
Then it compares two resulting trees and generates a list
of simple steps that would bring the destination in sync with the
source. And, finally, it executes these steps.
As a result both file and folder renames/moves are
recognized and replicated as you'd expect them to be -
with a single move.
Another way to speed up replication is to be smart with
caching information that doesn't change between the runs.
When the replication process is the only entity that makes
changes to the destination, it really has no need to scan
the destination on every run if it retains a copy of
the destination's contents from the last time.
That's what Bvckup 2 does. It defaults to caching
the results of the initial destination scan and then diligently
updates the resulting snapshot with every change that it makes
This lets the app know the exact state of the destination
at the start of every run, without needing to scan it
This may not sound like much of an optimization, but
scanning a large disk location can take from several
minutes to well into an hour; especially over the
network, and with slower NAS devices.
When a file system scan is needed, it generally doesn't
hurt to get through it as quickly as possible.
To that end Bvckup 2
parallel file system scanner
that speeds up
directory scans by traversing multiple subfolders at once.
This gives a slight boost to local scans, but it does an
absolute wonder to over-the-network scans as it reduces
pointless idling while waiting for remote queries to complete.
File system testing
Once you've used robocopy for long enough, you inevitably
learn about the /FFT switch
This typically follows a period of intense head scratching
as you stare at unchanged files that yet somehow get
re-copied on every run.
The reason for this is that the file system trims file
timestamps when it stores them. For example, FAT
rounds up the last modification time to 2 seconds, and
the creation time to 10 ms. Microsoft's NTFS uses a resolution
of 1/10 of microsecond on both, and Samba's NTFS can be
set to whatever the vendor wants.
Speaking of Samba
, many Linux-based NAS devices not
only use custom timestamp precision, but also don't
support the setting of creation times or basic file
attributes such as System and Hidden.
With robocopy you'd typically need to first realize
that these issues exists and then work around them
using command line switches like /FFT.
In contrast, Bvckup 2
takes care of all these
issues automatically and also provides a
Monitoring is an integral part of any automated
process, and email alerts are the go-to option in many setups.
While perfectly feasible with a bit of external scripting,
they generally cannot capture the full context of an error
or a failure and thus provide only limited visibility into
what is actually happening at the location.
are natively supported by Bvckup 2
and can be customized with over 50 run-time variables to
capture all aspects of the job execution.
Wading through reams of raw robocopy logs is not the
most efficient way to perform a root-cause analysis.
implements a simple, but remarkably
convenient twist on conventional logs -
A log is displayed as a list of collapsible
, each level down capturing progressively more
context than its parent.
With all nodes collapsed, this gives a concise summary for
a quick inspection, but still keeps all the details just a
couple of clicks away for a complete drill down.
Logs are stored as plain text files, perfectly grep'able,
and they are serviced with a size- and age-based log rotation.
It always starts with just robocopy itself.
Then it turns into a small script to add a simple
email alert on completion. Then a bit more to capture
the logs. Then you hook it up to Task Scheduler to run
every few hours. Then tack on something else,
and before you know it, you are looking at something
that is just a few lines short of being semi-sentient.
Enter Bvckup 2 ...
The program was built from the ground up with the singular focus on data replication.
It is fast and convenient with a extensive list of secondary features, each supporting its primary function - getting the data from A to B as quickly as possible.
Here it is at a glance:
- Data replication between any two file system locations, local or over the network
- Full shadow copying support for handling locked and in-use files
- Very fast bulk copier
- Delta copying for rsync-like block-level file updates
- Replication of moves/renames as a single operation
- Automatic detection of file system properties, including timestamp resolution
- Removable device tracking
- Optional archival of deleted items, with automatic archive trimming
- Support for flexible and powerful inclusion/exclusion filtering rules
- Comprehensive logging, including full support for log rotation
- Support for simulated runs
- Built-in scheduler - periodic, real-time or manual runs
- Excellent user interface, including a superb hierarchical log viewer
- Email reporting and alerts
- Support for running as a system service
- Support for massive backups that weigh well into the tens of millions of items
Give Bvckup 2 a try and see the difference a professional file replicator can make.