The support forum

Beta Release 31

Alex Pankratov :

Sep 04, 2013

Significantly improved program start-up time


I reviewed what the app is doing at the launch time and then trimmed, delayed and rearranged a lot of things so that the main program window is now displayed as soon as possible. The speed-up is significant and in many cases it is as high as 10 times if not more.

Incidentally, slow start was a cause of a couple of other problems - the inability to switch to the service mode and the program window taking few minutes to show up when running in service mode. Both should be resolved now.

Added handling of "System Volume Information" folder


The app now ignores "System Volume Information" directory, when the backup location is pointing *at the root* of the drive.

This is done not to mess up drive's original SysVolInfo that contains drive-specific data and that typically should not be touched.

Asynchronous file copier


This is a big one.

I have rewrote the part that handles copying of files. It now works in a fully asynchronous fashion - it can issue multiple read and write requests and then let Windows complete them in the background.

This is highly beneficial when data is copied between two physical disks as the OS can be kept busy reading and writing data at the same time. In particular, this delivers a speedup of 10-15% over robocopy with bulk (non-delta) copying. As a side note, robocopy is actually *really* good performance-wise (but it still appears to be synchronous on the inside).

Delta copier improvements


The asynchronous nature of new copier comes *especially* handy when dealing with delta copying. Delta copier forms a copying pipeline out of reader, hasher and writer modules.

The copying is started by seeding reader with a bunch of read requests. As these requests are completed by the OS, read data is forwarded to the hashing module, which is a purely computational task. Hashing module uses all available CPU cores, so the more you have the faster the hashing will be.

Then, once a block is hashed, it is forwarded to the writer, which compares block hashes to those from the previous run and issues write requests for blocks whose hashes changed.

Then, once a write request is completed, the block is shuffled back into the reader, which uses it to issue another read request. And so this carousel spins until there's nothing more to read, hash and write.

This release also includes highly-optimized versions of the actual hashing code (courtesy of OpenSSL) and this alone speeds up hashing step by 20-30% compared to older versions.

With all things combined, the net effect is that the delta copier now runs as fast as robocopy *despite* the fact that it does quite a bit more than just read/write things.

Folder timestamps


Beta 31 now copies folder timestamps. More specifically, it now correctly replicates "last modified" timestamps - this is done as a post-processing step, once all files are copied and updated.

The overhead of this step is minimal, but it still can be turned off in the configuration if needed.

Percentage indicator for longer copies


Added simpler % indicator as per this - https://bvckup2.com/wip/09022013-2

Interestingly, the app starts with a simple percentage, e.g. 13%, but then looks at effective copying speed and adds decimal points if the copying is slow so to always show some activity with slower copies.

Switched to signing releases with Pipemetrics SA certificate


Yaletown Software Design of Canada is no more. Please welcome Pipemetrics SA of Switzerland. Both are my companies that I use for legal and accounting purposes.

pgfitzgerald :

Sep 04, 2013

I don't think the update process worked right for me. Two things I noticed (there may be more):

1) The update looked like a full install wizard.
2) Bvckup was not closed during the update.

After closing the installer and the program itself, then reopening Bvckup, it was updated to b31.

I'm not sure how to replicate it on this system to take more detailed notes since I don't think I have a b30 standalone installer anywhere. I will try on another computer running b30 when I get home.

Alex Pankratov :

Sep 04, 2013

Yeah, I screwed up a bit in the installer.

I was tidying thing up and moved functions imported directly from kernel32.dll into a separate part of the app. That part needs a one-time initialization, which I added to bvckup2.exe, but not to setup.exe and uninstall.exe.

So, when the setup is run on x64 box, it doesn't find IsWow64Process function, assumes it runs on a 32-bit machine, skips disabling of Wow64 registry redirection and ends up not finding the Uninstaller reg entry -> launches as a new installer.

http://25.media.tumblr.com/tumblr_m090cqVAx31qev01no1_400.jpg

Alex Pankratov :

Sep 04, 2013

*things

pgfitzgerald :

Sep 04, 2013

Glad you figured it out. :-)

Deipotent :

Sep 04, 2013

beta 30 is failing to update to 31 with following error:

2013.09.04 19:40:32.303 (UTC+0) 2 00F94 ui | Checking for updates, reason: manual check ...

2013.09.04 19:40:32.306 (UTC+0) 3 00F94 ui | Sending the request for http://update.bvckup2.com/check?ver=0.30.0.0&nag=...

2013.09.04 19:40:32.735 (UTC+0) 3 00F94 ui | Receiving the reply...

2013.09.04 19:40:32.740 (UTC+0) 0 00F94 ui | Function ParseUpdateInfo failed with 0,

/var/www/bvckup2.com/html/beta/files/bvckup2-setup-0.31.0.0-yaletown.exe -- not found

Alex Pankratov :

Sep 04, 2013

Yep, took it offline, so not to fan the flames. Give me a couple of minutes, I'll wrapping up packaging of the next update.

Alex Pankratov :

Sep 04, 2013

Done. Please update. http://bvckup2.com/beta/setup.exe

pgfitzgerald :

Sep 04, 2013

Hmm. The manual update doesn't seem to be working for me. After downloading and running http://bvckup2.com/beta/setup.exe, About Bvckup still shows b31 and Check for Updates shows b32 is available.

/confused

Deipotent :

Sep 04, 2013

Auto update from b30 to b32 works fine now, for me. Only issue is that UI is not restarted due to AppLocker issue I reported a while back.

Curious about a couple of things:

1) Hashing - is only one hash at-a-time done on multiple cores, or is each stage of the pipeline multi-threaded (ie. multiple hashes, each on a single or on multiple cores) ?

2) Asynchronous file copier - what's the max number of read/write requests in motion at any one-time ?

Alex Pankratov :

Sep 04, 2013

I just checked - I'm getting b32 installer from that link.
Right click, save as, right click, properties, details -> 0.32.0.0

If it's the same in your case, then the ->b31 update might've messed things up. Try removing Bvckup from the Control Panel (it will ask if to remove config data - just say No), then refresh the Add/Remove program list, see "Bvckup 2" is gone. If it's not, then remove it again (it may complain that the uninstaller not found and if to remove the program entry - say Yes). Then install (b32) again.

pgfitzgerald :

Sep 04, 2013

* Yep, setup.exe is 0.32.0.0.
* When run, it says it detects beta 32.
* There was only one Bvckup in the start screen. It was beta 31.
* There were two entries in Programs and Features: beta 31 and beta 32.
* Uninstalling beta 31 was successful, but did not remove the entry from Programs and Features.
* Uninstalling beta 32 was also successful, and also did not remove the entry from Programs and Features.

I think maybe the beta 31 update installed a 32-bit version in addition to the 64-bit version of beta 30 I previously had, causing things to get super confused.

License agreement still shows Yaletown Software Design. Not sure if you intend for that to be updated to Pipemetrics or not.

I'm all setup on beta 32 now. Thanks!

Deipotent :

Sep 04, 2013

Device tracking - Just unplugged one of my external hard drives and got the red error text to the right of the job tile when the job tried to run, which is expected:

failed. Backup error: destination device not found.


However, the job did not auto run when I re-connected the drive. Should it auto-run on device arrival, or is there an option to do this ?

The other thing I noticed is that all existing and new jobs had both source and dest device tracking options set to "Don't track", even though the removable device help dialog (from clicking on eject style button to right of src/dst edit boxes) states that it tracks by ID by default, but this behaviour can be changed under More... button.

So, new jobs didn't track removable devices by ID, and existing jobs were not updated to track by ID.

Deipotent :

Sep 04, 2013

However, the job did not auto run when I re-connected the drive. Should it

auto-run on device arrival, or is there an option to do this ?


To clarify, I only expect it to run on device arrival (or want an option to do so) if a previous auto-run could not occur due to "device not found" error.

Alex Pankratov :

Sep 04, 2013

@Deipotent -

1) Hashing - is only one hash at-a-time done on multiple cores, or is each stage of the pipeline multi-threaded (ie. multiple hashes, each on a single or on multiple cores) ?


By default, delta blocks are 32K each. Each block is hashed as a single operation. Copier reads 32 blocks per request, so it has 32 block hashes to compute and these are distributed across all available cores, less two (but obviously at least one core if there's less than two).

One more core is used to compute running file hash. This is new in v2 and it's a safeguard that detects false negatives. That's the cases when a changed block just happens to end up with the same hash as before. So in v2 if the copier sees that all block hashes remained unchanged, it compares freshly computed file hash against the hash from the previous run. If they match, everything is OK and the file is unchanged. But if they differ, than it's that 1 in metamillion chance of block hash collision and so it scrapes everything and re-copies file in full.

And one more core is reserved for the dispatcher thread that directs buffers between the reader, hasher and writer. It spends most of its time sleeping, but it is also responsible for checking block hashes and deciding which blocks to write.

Delta block size, read/write buffer sizes - all these are in the .ini, but mess with them at your own risk.

2) Asynchronous file copier - what's the max number of read/write requests in motion at any one-time ?


This is determined dynamically based on the copying type (basic or delta) and relative disposition of source and destination. For basic copying, it uses 1 buffer if src/dst are the same volume, 4 - if not. If either is a network drive it uses 16.

For delta copying, it all the same, but it uses 2 buffers if src/dst are the same.

These are overridable through .ini (max_io_buffers), but these are pretty solid numbers. I did quite a bit of testing.

Also, tangentially related


The copier reads in 1Meg blocks, but writes out in 64K chunks. And, again, that's unless either end is over the network, in which case it writes in 1Meg chunks.

Alex Pankratov :

Sep 04, 2013

@pgf -

I think maybe the beta 31 update installed a 32-bit version in addition to the 64-bit version of beta 30 I previously had, causing things to get super confused.


My thoughts exactly.

License agreement still shows Yaletown Software Design. Not sure if you intend for that to be updated to Pipemetrics or not.


Aye. I'm attending to this next. It also says Yaletown in .exe versions.

I'm all setup on beta 32 now. Thanks!


Great, thanks for your patience :)

Alex Pankratov :

Sep 04, 2013

However, the job did not auto run when I re-connected the drive. Should it auto-run on device arrival, or is there an option to do this ?


Was a real-time job? Devices/locations going away and coming back needs another round of polish.

The other thing I noticed is that all existing and new jobs had both source and dest device tracking options set to "Don't track", even though the removable device help dialog (from clicking on eject style button to right of src/dst edit boxes) states that it tracks by ID by default, but this behaviour can be changed under More... button.


Can't reproduce this. I did -
1. Picked "Add new backup" from the menu
2. Typed C:\Foo into Source
3. Typed D:\Bar into Destination, got the "ejectable" icon
4. Clicked on More Options -> Destination was "Track by ID" + Fingerprint

Deipotent :

Sep 04, 2013

Thanks for all the info on how the hashing and async file copier works.

For basic copying, it uses 1 buffer if src/dst are the same volume


I'm a little surprised by this for HDD's (as opposed to SSD's) as I would've thought that this isn't any faster than synchronous copy, and doesn't take advantage of NCQ. However, if your real-world testing shows you different... :)

Deipotent :

Sep 04, 2013

Was a real-time job? Devices/locations going away and coming back needs another round of polish.


Both real-time and "Every ..." jobs - none of them auto started.

Can't reproduce this. I did -

1. Picked "Add new backup" from the menu

2. Typed C:\Foo into Source

3. Typed D:\Bar into Destination, got the "ejectable" icon

4. Clicked on More Options -> Destination was "Track by ID" + Fingerprint


The bug only appears if you use the Browse button to select the removable device, rather than typing it.

Deipotent :

Sep 11, 2013

Further to my device unavailable issue - if the dest is unavailable for several days and I shut down my PC every night, the UI only indicates the job last run time since PC was switched on, and not when job was last successfully run. eg. my dest has been disconnected for 4 days but, after booting up my PC, each failed job is only showing "8 hours ago".

I know there is limited space on the job tiles but, in error cases, the time since last successful backup is a *very* important stat, IMO.

Alex Pankratov :

Sep 12, 2013

Noted, thanks. Let me think about it.

Also, dealing with destination (or source) location going missing is on my list and it should be done in the next release or two. It's basically the only big "core" issue I have at the moment.

New topic

Create
Made by IO Bureau in Switzerland
Support

Updates
Blog / RSS
Follow Twitter
Reddit
Miscellanea Press kit
Testimonials
Company Imprint

Legal Terms
Privacy