The support forum

Release 72

Alex Pankratov :

Dec 22, 2014



New  -  Multiple copying speed improvements


This is a result of a couple of weeks worth of performance testing. Tests covered different common setups, including same-drive backups, different drive backups, backing up local to/from USB2, local and USB3, local and a network share over SMB1 and over SMB2, over gigabit links and over WiFi, with warm cache and with cold cache, etc.

The ultimate goal of testing was to determine optimal sizes of read and write requests, the total number of I/O buffers as well as to fine-tune the copying pipeline in general.

As a result the following are the changes in R72:

     a.  The app now takes into account both the type and relative
           disposition of the source and destination when selecting
           I/O buffering strategy. In particular, it is now aware of the
           version of SMB protocol used to access remote shares,
           understands locations residing on the same *physical*
           disk and includes special handling of USB drives

     b.  The app now does NOT explicitly flush the write cache upon
           completing the copying of a file *unless* destination is a
           removable device.

           Previously, it has always flushed the cache and it led to
           significant delays when backing up from the network to a
           local fixed drive.

           The flushing behavior is configurable via .ini and it can be
           overridden to be "flush always" or "flush never".

     c.   Threshold for delta copying a file is now 2 Megs instead of
           128K from before. Meaning that files under 2MB are now
            always copied in full.

            Again, this is configurable via the .ini.

     d.   File that are copied in full no longer use staged copying if
           destination file doesn't yet exist.

           With staged copying, the file is first copied into a temporary
           file at destination, which is then renamed in place. As of R72
           this is done only if destination file exists.

           * Configurable via the .ini.

     e.   The app no longer uses the slow-start reading routine, whereby
            it used to first issue 1 read request, wait for it to complete, then
            issue 2 requests, then 3 and so up to the maximum number of
            I/O buffer.

            This was a homage to TCP/IP slow start. No idea why I thought
            it'd be a good idea in the disk I/O context.

           * Not configurable via the .ini. Surprise.

With all things combined, the bulk throughput in both cold and warm cache cases is marginally better (5-10%) than that of robocopy, which
is pretty much a golden standard of I/O performance on Windows 8
and newer.

The app now also logs I/O throughput, both for per-file and for per-run -
https://bvckup2.com/wip/r72-file-mbps.png
https://bvckup2.com/wip/r72-backup-mbps.png

Dec 22, 2014

New  -  Added support for archiving past versions of modified files


        This is an advanced-use option.
        Use with care.
        Understand what you are doing.
        May void warranty, etc.

It is now possible to set the app to archive existing backup copy of a file before backing it up. In other words, the app will move file's backup copy to the archive and then re-copy the original in full to create new backup copy.

Obviously, this is a form of file versioning. The reason it may void the warranty is that it can chew through available disk space *like that* if the archive trimming is off or if the backup is run to often or if modified files are really large, etc. Quite a few opportunities for getting your backup routine in trouble. Hence the warning.

I would recommend to NOT use this option unless you have to and you operate with small-ish files. Please wait for a proper versioning support in Bvckup 3 as per - https://bvckup2.com/support/forum/topic/158

--

In order to enable this feature, you will need to change "archive_modified" entry in backup's settings.ini file from "0" to "1".

See this post for instructions -
https://bvckup2.com/support/forum/topic/933

You will also need to ensure that the Copying preference for the backup is set to "in full" (and not "delta") and the Deleting is set to "archive".

Dec 22, 2014

New  -  Added support for destintion scan filters


        Advance use option.
        An excellent way to shoot oneself in the foot.
        Please make sure you know what you are doing.

Similarly to how a backup can be configured to exclude/include files and folders when scanning the source location, it is now possible to do the same when scanning the destination.

A recap


When running a backup, the app first scans the source and builds a tree of files and folders. It uses rules from the "Backup What" sections of its configuration to decide which items to include into the tree and which - do not. The resulting tree is exactly what needs to be backed up and nothing else.

The app then scans the destination (or loads its snapshot saved from the last run) and builds a tree of what's already there. It then compares resulting two trees and compiles a list of file/folder operations that is needed to transform destination tree into the source tree.

Prior to Release 72, destination scan was unconditional. It picked up everything that was on the disk.

The change


As of this release each backup now has a separate set of filters that is applied during the destination scan. By default the set is empty, so everything is included, just as before.

However you can now add a rule to exclude certain files or folders and it will ultimately make the app act as if these items don't exist.

This is obviously a *very* fertile ground for creating lots of peculiar issues. Please make sure you know what you are doing.

How to use


Destination scan filters are configured through the UI, but this function is disabled by default. To access configuration window you first need to add the following line to bvckup2-ui.ini -

        edit_dst_filters 1

with the app not running. After this, CTRL-clicking on Details button in the "Backup What" section will bring up the destination filter configuration dialog. It will also double-check with you that you know what you are doing, just in case -
https://bvckup2.com/wip/r72-dst-filters-warning.png

Addendum


As of Release 77.2 it's also possible to tell the scanner to use _source_ filters when scanning _destination_ location (completely ignoring all and any configured destination rules). This is enabled by setting:

        conf.dst.filters.as_source   1

in the backup's settings.ini file.

This is particularly useful when backing up selected set of items from the source into a prepopulated destination. That is, when the source filters are set to "start with a blank list" with just some items included.

Dec 22, 2014


New  -  An option to suppress the "Waiting for device..." state


It is now possible to tell the app to NOT monitor for the availability of a drive or a network share and just assume that they are readily available. This applies to real-time and periodic backups that would, by default, enter and remain in the "waiting for device..." state if either source or destination location is not presently available.

This is primarily intended for cases when src/dst device is in fact unavailable normally and instead getting mounted or mapped by the pre-backup command.

* This option has no effect if the device tracking - by fingerprint or by volume label - is enabled.

How to use


This is a per-backup option. The following entry in the backup's settings.ini controls it for the source device -

        conf.src_force_present 1

and

         conf.dst_force_present 1

for the destination device.

See this topic for general instruction on how to change .ini files -
https://bvckup2.com/support/forum/topic/480/2839

Dec 22, 2014


Smaller changes


1.  Clicking on an empty toolbar area to the right of Go/Stop button now deselects any selected backups, while Ctrl-clicking there selects them all.

2. The issue with very long file/folder names overlapping neighboring fields in the main UI list has been resolved.

3. Backup tiles can now be styled independently. See last section in this post for details - https://www.bvckup2.com/support/forum/topic/481/2842

4. The versioning scheme has been extended to support interim maintenance releases. That is, there's now always one primary release (e.g. Release 72) that may be followed by one or more revisions to cover any required tweaks and bug fixes (e.g. Release 72 Revision 2). Revision information is now shown in the Help/About window when applicable.

Alex Pankratov :

Dec 23, 2014


R72 Rev 1 is out.

Resolves an issue with a spurious error being logged for remote locations on Windows XP - https://bvckup2.com/wip/r72.1-xp-error.png

If you are on XP and see this issue, then update. Otherwise the update is not required.

Alex Pankratov :

Dec 25, 2014


R72 Rev 2 is out.

Resolves a (rare) issue with the backup planning module. More specifically, to handle rather exotic case of file "foo" getting moved into a new subfolder also called "foo", the app uses so-called "staged moving". It first creates a temporary folder, moved "foo" there, then creates the subfolder ("foo") and finally moves file "foo" there from the staging directory. This is all working well, however the UI had an issue where it would panic as creating and removing the staging folder weren't counted as backup steps, so it resulted in a discrepancy between expected and actual step counts. Incidentally, this was triggered in certain cases by enabling the "archive_modified" option (added in this release).

Long story short - if you had "archive_modified" enabled and saw the app panic, then you need to update. Otherwise the update is optional.

Alex Pankratov :

Jan 06, 2015


R72 Rev 3 is out.

1. Resolved the "GetFileInformationByHandleEx" issue on Vista - if you get this error during the Preparation phase of the backup, this fix is for you.

2. Resolved an issue with email notifications - when talking to a mail server, the app would erroneously think that the server does *not* support user logins if this capability was listed last in server's EHLO reply. In particular mail.1und1.de is like that.

3. Resolved an issue with LinkStation NASes - some models don't support 'created' timestamps *and* they return some really crazy values for these when an app queries them. This didn't go down well with the timestamp granularity detection module. No more. As of this release if the 'created' timestamps aren't supported, they are ignored altogether.

4. Improved timestamp granularity detection logic - when running a backup the app now keeps track of the largest difference between timestamp values that it asks the OS to set and the timestamp value that is actually gets set on a file. This is basically the "largest observed timestamp rounding error". The idea here is that if the timestamp granularity was not detected correctly (it was deduced to be lower than it actually is, i.e. 10 ms instead of 100 ms), then it can be automatically corrected through tracking of rounding errors.

5. Reduced per-file copying overhead in the delta copier - eliminated explicit file buffer flushing when closing CRC files. This should speed things up a bit when using delta copier.

6. The archival function now uses two separate tags when moving an item to the archive - one tag is for deleted files and another tag is for modified files if this is enabled (https://www.bvckup2.com/support/forum/topic/502/2958). Both tags now default to " ($date $time)" and they can be changed through the setting.ini file edit.

Alex Pankratov :

Jan 09, 2015


R72 Rev 4 is out.

It includes a fix/workaround for detecting TrueCrypt volumes getting mounted and unmounted when the app is running in service mode.
Original report on the issue - https://bvckup2.com/support/forum/topic/511

The root of the issue is that TC broadcasts WM_DEVICECHANGE event only to the windows in its own session and it doesn't implement respective notifications for the services. To workaround this limitation R72.4 implements optional drive re-enumeration on timer. It basically goes through all drive letters every 10 seconds (the letter list and the interval are configurable through .ini) and checks for new/disappeared drives.

This monitoring is disabled by default *unless* the app is running in service mode *and* the TrueCrypt driver is present and enabled on the system. Meaning, that if you don't use TC or if you are running in the default (application) mode, this change is transparent to you.

Peacecamper :

Jan 09, 2015

Any news on when you will add the monthly scheduling for backups?

AndCycle :

Jan 09, 2015

I got timestamp issue since rev 3, which doesn't happened before,
backup target is a personal linux server with samba 3.6.24 and no additional tuning,

now it always do full backup when file changed,

always report
"Resetting the delta state"
"Destination file timestamps have changed since the last run"

anything I can help?

Alex Pankratov :

Jan 09, 2015

@Peacecamper - Support for monthly backups is scheduled for the next release, i.e. Release 73.

Releases are larger milestones that add features, and revisions are smaller interim releases that address any issues that may surface.

@AndCycle - Ack, I think I know what might be causing this. Let me have a look and I'll ping you for a quick test when I got the fix in place, OK?

Peacecamper :

Jan 11, 2015

Thanks for the update. :)

Alex Pankratov :

Jan 11, 2015


R72 Rev 5 is out.

Resolved an issue with files getting repeatedly backed up on every run after a single change *if* destination device doesn't support 'created' timestamps. The problem was introduced in R72 Rev 3 and it primarily affects people backing up to vanilla Samba-based storage devices.

Kudos to everyone who helped with reporting and troubleshooting the issue. It was a lovely weekend, wasn't it? :)

Alex Pankratov :

Jan 19, 2015


R72 Rev 6 is out.

1. Added support for Drive Extender NTFS reparse points - a simple matter of adding a reparse point of 0x80000005 to the list of RP types that are treated as regular file system entries.

2. Resolved an UI issue with closing of the message bar - in some very rare cases the message bar that floats up at the bottom of the window would end up in a slight overlap with the window contents. The UI would detect this and panic, so that I could take a look at why it happens and put a proper fix in. This is the fix.

3. Resolved an issue with network location monitoring - deleting a backup job that is in the "waiting for the network location" state didn't clear the actual location monitoring task and that caused the app trip a self-consistency check when the task completed. No more.

4. Reworked the code for disabling local file buffering - there's a way to tell Windows to *not* to do any local buffering when reading/writing a file over the network. This is meant to speed up the copying of larger files. The official way as described in MSDN simply doesn't work, so prior to this release the app resorted to using Zw... API as a replacement. Apparently, in some exceedingly rare cases this resulted in the app going down. The issue was further complicated by the fact that when it happened, this produced no stack trace. Long story short, the change was to switch to using Nt... version of the same API as it appears to be in a wide(r) use and it's generally accepted to be stable.

johnseers :

Jan 28, 2017


Hi Alex

I have been interested in this feature for some time and have provided feed back to you to add it into your bvckup. You have promised it will appear in bvckup3 but it has been some time.

I am not sure why you are not very keen on it. Something to do with the disk space it uses?

I am prompted to ask again as I just checked how my backup is operating and found that I am not operating versioning any more. It must have got lost over time somehow. I believed I was doing versioning - lulled into a false sense of security by the "delete archive" option. It does not do what I thought!

So, I have to go back and rediscover how to switch on versioning in the config files. This is not as easy as it should be as the instructions are spread over multiple posts and are not all that clear.

Please could you add it as a checkbox option sometime soon? That would be very kind of you. Oooh, and along with a tool or two to manage the archive ....

OK. I can do without the tools. But just one little checkbox please. You can surround it with all sorts of warnings about disk space etc etc if you like.

Regards

John Seers

Alex Pankratov :

Jan 29, 2017

John,

I am not sure why you are not very keen on it. Something to do with the disk space it uses?


Disk space usage *and* the convenience of recovering a backup from a specific date or specific file version. There are also several performance-related issues, in particular the need to re-copy every modified file *in full* on every backup.

Summed up, it makes current implementation more of a hack than a proper feature so I am not comfortable with formally promoting it to one. It is functional and it is useful, but it's not up to the standards yet.

---

With regards to how it managed to disable itself - is it possible that you've switched Copying to "Delta copying" at some point? This will effectively disable archiving of _modified_ files. Other than this, the app has no code to it that would reset "archive_modified" to 0 in the INI file.

If you'd like, forward me settings.ini for the job and I can have a look.

I'll also try and think of a way to make this option a bit more "sticky" and, for example, log an error if either of pre-conditions is broken.

johnseers :

Jan 31, 2017

Hello Alex

Thanks for the reply.

I do not think there was anything very mysterious about me losing the versioning. I cannot really remember any specific details but I did change how I did my backups once a while ago. I think I saw the delete Archive option and thought that was the versioning option. I was too busy/distracted to check it out and then forgot all about it.  As a compliment to your software it just runs and I rarely need to look at it. Hence two(?) years later I noticed it was not versioning.  :(

So, sorry to hear you will not put the option on a firmer footing yet. If you can do some work to make it less of a hack that would be good.

Regards

John Seers

P.S. Have recommended your software to a friend and I will install and set it up for them at the weekend. I will switch on versioning for them ...



johnseers :

Jan 31, 2017

" ... is it possible that you've switched Copying to "Delta copying" at some point? "

Yes, I think I did try out Delta copying. Perhaps that is what did it.

Alex Pankratov :

Feb 03, 2017

So, sorry to hear you will not put the option on a firmer footing yet. If you can do some work to make it less of a hack that would be good.


Let me see what I can do.

Alex Pankratov :

Feb 09, 2017

Yes, I think I did try out Delta copying. Perhaps that is what did it.


Checked the code just now and the app already logs an error when the archival of modified files is only partially configured:

    The "archive_modified" override is ignored.
    Deletion must be set to "Archive" and Copying - to "In full", not "Delta".

New topic

Create
Made by Pipemetrics in Switzerland
Support


Follow
Twitter
Dev blog
Miscellanea Press resources
Testimonials
On robocopy
Company
Imprint

Legal Terms
Privacy