Screenshots & sketches RSS feed

Preview 16 ships a smaller UI enhancement that makes life easier when working with very large backups.

The program now displays the progress of the backup planning activity as it moves through all three planning stages.

Backup planning involves looking at the source and backup trees, finding the differences and then generating a list of simple steps that, when executed in correct order, bring the backup in sync with the source.

As you may or may not know, Bvckup 2 creates a backup plan in three stages. The change in this release is that the program now reports how far along each stage it is at any given moment.

This makes little difference for smaller and medium-sized backups (up to 100K files, give or take), because planning is generally very fast. But for larger backups with millions of items this change makes the process a bit more predictable.

Planning stages

The three planning stages, as used by Bvckup 2, are the tree comparision, compilation and packaging.

During the comparison stage the planner ingests source and backup file trees and spits out a list of create/delete steps needed to bring the latter in sync with the former.

This basic plan is then passed through the file/folder move detection. It also has various policies applied to it, e.g. the file archival, forced file updates, etc. This is the compilation phase and it produces the backup plan.

Finally, the plan is trimmed down and stripped of all temporary data that was used during the planning. This comprises the 3rd stage, the packaging.

GIF screencap at the top of the post shows the program go through all three stages.

Keep in mind that the program was artificially slowed down for demo purposes. Regular production version will make a plan of this size in a fraction of a second.
Preview build 13 strikes one the oldest items off the ToDo list - the UI support for archiving file backups.

Internally, the engine supported archiving of modified files for quite some time now. It was first added back in 2014, but as it was deemed "temporary" in its design, the feature could only be enabled with a manual configuration override.

Archiving was originally meant as a safety net for people who click their way through the warnings only to stare in a petrified disbelief at years worth of their precious family jewels vanishing from their one and only removable drive.

The archiving worked well for this purpose. It saved the jewels.

However it worked by appending a timestamp to the file or folder name, so retrieving items from the archive was tedious. It was not a good general-purpose archival approach.

Fast forward 7 years to last week and preview build 12 rolls out support for a much simpler archive structure.

Details are at the link, but the keyword here is "human-friendly".

The second part to the recipe was the new UI mechanism for editing details of a backup job.

Again, the details are at the link, but the gist of it is that the backup configuration now allows for more settings per section, while still remaining compact in size and light in function.

So the "Deleting" section became "Archiving", gained some weight and finally added UI support for the file versioning.

It took a while, but it's finally here.

The caveat

There is always a caveat.

It's in the fact that file versioning prevents delta copying from actually ** delta ** copying.

This copying method gets its speed benefits from being selective with updating an existing copy of a file. But the archiving ensures that there's no existing copy  =>  the file gets copied in full, at full IO cost.

Hence this brand new warning: To explain it differently:

When the archiving is enabled, there will be one extra full copy of a file after every backup run.

But delta copying is all about doing less than a full copy, so it's not possible in principle to mix one with another.

The next preview build of Release 82 will ship with a new mechanic for accessing configuration details in the Backup Settings window.

Instead of expanding configuration sections in place by pushing bottom of the window down, the program will pull details to the center of the window and then de-emphasize the rest of the settings.

The change is not dramatic, but it looks nice (always a plus) and it also allows for larger detail sections... which will come into play shortly.

Stay tuned.
Release 82 is an upcoming major Bvckup 2 update and it is now starting to enter the preview update channel.

First up is a new and important feature - the write verification.

Write verification does exactly what its name implies - for every data block that the program writes out, it reads the block back and checks that it remained intact.

This process helps to safeguard against two types of problems.

1.  Data corruption on the way to the device

This may happen due to the software and firmware bugs, memory bit flips (both in the machine's own RAM and in the controller/device memory) and transport errors if going over the network.

The relative risk of such corruption is exceedingly small, but the backups tend to push a lot of data around, so the absolute risk may grow with time to the non-negligible levels.

2.  Latent storage media problems

Reading data back effectively acts as a form of scrubbing and helps flushing any pending issues with the storage media.

If the media is OK, the read-back will just complete nominally.

If the media is dying, i.e. if the data can still be read, but only after retrying or via the sector ECC recovery, then we will get our data back and trigger the sector relocation, avoiding potential data loss if the media were to degrade further.

If the media is dead, then we'll won't have our data back, but we'll learn about the problem now rather than later.

How it works

The somewhat less obvious aspect of write verification is that it needs to bypass the file system cache and to try and read the data from actual the device.

The program arranges for this by opening the target file for the second time and requesting a so-called unbuffered read access to it.

This doesn't prevent writes from being cached, but ensures that reads are served from the disk.

Going around the cache automatically means that the copying becomes slower.

The degradation is in part mitigated by the asynchronous nature of the ultra copier and it's further reduced by the delta copying if it's in effect, but there is a slow down. No miracle there.

How it looks

Once enabled, the write verification setting will show up in the Configuration section: It will also be mentioned in the completion statement for all non-empty files: Finally, if a write verification fails, the log will show the following: Verification failures are non-retryable, the program will not retrying the operation in an attempt to correct the problem.

This is by design to keep the issue on the surface and to force investigating it.

In prerelease builds the write verification can be enabled for a job via the right-click menu as shown in the opening screenshot.

In production builds it's likely to be the same, except the Preview submenu will have a different name and it will be visible only if the Ctrl held down.

You can get a prerelease build by switching your installation to the "preview" update channel as described here.
Maintenance release 81.7 adds a relatively simple new feature - an option for automatically discarding files' delta copying state based on its age or the number of uses.

When enabled, this will force the re-copying of large files in full as per set thresholds. This may be desirable in some setups, e.g. as a data scrubbing mechanism for flushing out pending storage media issues.
It's been a while since the last sketch, so here is one.

This is for a rework of the web dashboard that aims to make it more compact and better suited for managing a large number of licenses. No doubt, a nice problem to have for everyone involved.

*  Still very much a work in progress.
Maintenance release 81.3 includes a rework of how the program tracks, handles and reports errors encountered during a backup run.

Starting with this release Bvckup 2 will report not only the error count and the number of retries, but also the number of errors actually corrected through retrying.

Corrected error count is needed, because a single backup step may sometimes fail with more than one error, e.g. - When this happens, earlier Bvckup 2 versions would say that the run completed with 2 errors and 1 retry... which was not terribly helpful, even if it was technically correct.

With the new change, the program now tracks the exact number of errors associated with each failure and report how many of them were in fact corrected through retrying.

Another change in 81.3 has to do with the retrying logic itself.

In the presence of multiple errors the program will now look at all errors in order to decide whether the operation should be retried. This is different from earlier versions that looked at the last error only.

It turns out that yanking a network cable from a NAS sometimes results in "An unexpected network error occurred" first, followed by "The file is in use" second. The "in-use" error is non-retryable, so looking at the last error only ended up suppressing a retry.
Another new feature in Release 81 is an option for waking up the computer to run scheduled (periodic) backups.

When enabled, this option will cause the program to ask Windows to wake up the computer around the time of the next scheduled backup run.

If no user interaction is registered while the machine is awake and the backup is running, Windows will put it back to sleep once the backup is done.


One caveat is that the machine should support software-driven wake-ups in general, e.g. they should not disabled at the BIOS level or at the OS level.

Any errors of this sort are reported through the log:

The second caveat is that wake-up timers are imprecise.

Asking for a wake-up at 12:00:00 may translate into computer just starting waking up some 10 to 30 seconds later.

To work around this Bvckup 2 schedules all wake-ups for a bit earlier than the actual backup time and it also tolerates the machine (still) waking up later than needed.

The defaults are 30 and 60 seconds respectively, but, as per usual, they are fully adjustable.
Upcoming Release 81 will introduce an option for locking down UI access when the program is running in service mode.


In service mode, the program splits the backup engine and its user interface into two separate processes. The engine runs autonomously in the background as a system service, under a service user account. The user interface runs as a regular desktop program, under a desktop user account.

This split allows the UI to be started and stopped at will, with the engine running continuously even when no user is logged in.

When the UI is started, it connects to the engine, at which point the engine starts feeding the UI with updates and the UI sends back control and configuration commands as per user's input.

The change in R81 allows to set the engine to ask for a password when the UI first connects to it. Once enabled, it will cause the program to pop up a password prompt when the UI is opened. No password = no UI. This option is primarily meant for preventing non-admin local users from opening the UI and messing with the setup.

Fine print

1. The password is verified by the engine, meaning that the check is isolated from the desktop user context and can't be worked around without having Admin rights.

2. Salted hash of the password is stored in the engine's config file. The password can be cleared by stopping the service, wiping password hash from the config file and restarting the service. This also requires Admin rights.

3. When the program is being updated, the UI knows how to restart and reconnect back to the service without requesting a password.

Due to how updates work, this is one of more complicated (and interesting!) parts, based on single-use authentication tokens that are issued by the engine to the UI specifically for this case. However it's completely invisible in use.

4. The UI does away with dual password entry fields (enter once, enter again to confirm). Instead, just as many modern UIs do, it allows revealing the password if needed. 5. Don't forget your password.
When Pro licenses were introduced last year, with that we also introduced a new support and software update model referred to as Pro Support.

All details on this element of Pro licenses - coverage, eligibility, renewals and costs - are now available here:

Pro Support

* In particular, do note that renewal of the Pro Support while it is still active is offered at a discounted rate.
Preview 12 of Release 81 is out and it adds UI-side support for configuring multi-threaded (parallel) processing.

The contents of Backup Settings window got squished together a bit and a new Processing section was added.

Surprisingly enough, the window actually got smaller as a result, but it does look a bit more crowded now.

There's also a couple of important options behind Edit details:


While we are at it, here is a quick rundown of the design process leading up to the final version.

Two options in the Large files section define how the program handles the copying of "large" files.

The definition of "large" is based on a file size threshold and the question was if this threshold was important enough to have a place in the UI.

Initially, the idea was to both display and to allow editing it.

Take 1

This gives too much prominence to an otherwise minor option and it also invites to mess with it. We can do better.

Take 2

The option is shown as a text, but can be edited by clicking on it.

Furthermore the idea is that it can be edited inline. That is, with the edit box just appearing in place and going away once the edit is done.

This required implementing an auto-sizing edit control, built into a radio button widget. Doable, but hard, and all of this is still for a secondary option.

Take 3

We take this option out and put it into a separate text. Also stick it at the very end of the text. This allows stretching out the edit box to the max and don't bother with dynamically sizing it: We can also add a bit of transition to compensate for edit box not aligning with the option value when it is static. This is not bad. Let's try with a borderless edit box: Not bad either and the slide is just a couple of pixels now. How about we don't slide it at all? Looks and feels quite nice, however...

The "details"

... the issue is with how this contraption behaves in edge cases.

When using it for the first time, it's very easy to discover that typing something and hitting Enter stands for "Accept the changes" and hitting Esc works as "Discard the changes".

But the user clicking outside of the edit box can be ambiguous. Some people will expect this to work as OK, while others will expect it to discard changes. Moreover, one may make more sense than the other depending on the context.

While it's possible to anticipate and to accommodate this sort of nuance, it gets us a bit too far into deep customization territory.

This could've been justified if inline editing were added to the UI arsenal and used a first class element across the entire program. But the way things stand, it will be a random widget used in one obscure corner of the UI.

As such, this option was shelved.

Instead the window uses an in-place hint to show the threshold value and related details.
Introducing a brand new addition to the ever growing family of Bvckup 2 spinoffs and companion products...

Please meet - byenow.

Multi-threaded folder removal utility, it speeds up the deletion by processing multiple subfolders in parallel - something that works especially well for deleting folders off network shares.

Check it out here -
Next maintenance release - 80.9 - will improve how the program estimates and displays completion time for longer file copies.

A brand new version of the copying speed-time estimator is in. It is far more sensitive to the changes in copying conditions while also producing estimates that are far less jittery.

New version also has a visually distinct mode when the program is correcting its estimate due to changes in the copying speed.

Significant copying speed-ups are not uncommon, but the slow-downs are even more routine. They happen few seconds into the copying, when the write cache fills in and the writing speed drops to the device's native value: When this happens, the completion time gets pushed back and the UI reacts to this by highlighting the estimate in red. It also increases its rate of refresh to make changes more obvious.

The GIF at the top captures this in action with a backup that goes from an NVMe drive to an HDD.

The copying speed first spikes to 1600MB/s as the write cache is getting filled, but quickly drops to 100 MB/s, which is the native write speed of the HDD.

When a change in speed like this happens, the estimator enters a correcting state and starts to gradually adjust the ETA.

Once the copying speed stabilizes, the estimator switches to the stable state and sticks to its most recent prediction. That is, it fixes the ETA and keeps it constant until further notice.

As the copying speed fluctuates, the effective ETA may go a bit up or down, but the estimator will ignore this jitter in favour of keeping the countdown steady.

This ensures that the countdown behaves like a clock, without sudden stalls and accelerations like this: The estimator also keeps monitoring effective ETA to make sure it doesn't drift too far away from the fixed estimate. If it does, then the estimator initiates go back into correcting state and the process repeats itself.

So there you have it - our new and shiny stabilized estimator. Coming up in Release 80.9.

*  A broader context here is that even the fastest storage devices will complete the IO unevenly. As a result the raw measurements that go into an estimator will always be noisy and feeding them into an estimator, as-is, will produce noisy estimates.

For example, here is (an otherwise truly excellent) Far Manager running a bulk copy and using an overly simplistic estimator: This is more distracting than it is helpful.

A better, human-friendly estimator needs to de-noise the data and to smooth out the estimates. It also needs to make use of heuristics to further align the result with what a human would expect, e.g. to not re-adjust the ETA every chance it gets.

Long story short, calculating ETA is one of those things that look trivial on the surface, but in practice has quite a bit of depth and complexity. Just something to keep in mind.
Preview 7 of R81 adds support for working with live files.

This is a bit of a rabbit hole, so some background is in order.

If we are to open an existing file, append some data to it and keep it open, then the dir command will keep returning file's original size until we close the file:

The same will also happen to the last-modified timestamp. Even though we "touch" the file, everyone else will be seeing its old timestamp until we release the file reference.

This looks odd, doesn't it?

The explanation to this behavior can be found here and it has to do with the fact that modern file systems such as NTFS and ReFS store file meta data alongside the file itself rather than with its folder entry.

This allows attaching arbitrary meta data to the files (i.e. not just attributes and timestamps), but it also means that a simple directory listing requires combing through each and every file to collect respective bits of meta.

Obviously, this doesn't scale well.

This issue is resolved by caching parts of file's meta in its folder's entry. This speeds up directory listings, but it also comes at a cost of meta being replicated lazily.

For example, if we are to add data to a file, then its new file size will not get propagated to its folder record immediately. Instead, it will done at some point later on.

Depending on the Windows version, this may happen when we flush file buffers, query file size via a handle or close its handle.

All this means that there's no reliable way of detecting actively modified files based on their directory listing alone.

So if we want to identify and back up such live files, we need a workaround.

One workaround is to just forcibly update them on every run, regardless of whether they have actually changed or not. It's crude and wasteful, but in a pinch it works.

A better option, supported by the new preview release, is to double-check directory listing data by re-querying file size and timestamps directly from the files themselves.

The queries aren't cheap, so they are not issued for all files.

Instead, this option works similarly to the forced updates - we specify one or more files of interest and the program rechecks their meta if it runs into them during a scan.

None of this is too complicated, but it takes time to understand why things work they way they do and how to work around that.

*  Kudos to James Kindon for raising the issue and providing original traces to identify it.
Our pre-release program has been formally opened to everyone.

In the weeks leading up to a release of the next major version we will now be posting internal development builds to the preview update channel.

These builds are meant to let anyone interested to kick the tires of what's coming and to allow us to see how the new version behaves outside our test environment.

All pre-releases are of production quality, with no known issues, but they may have some wrinkles and they may also not have all planned features implemented yet.

More information on the program, including instructions on how to obtain a pre-release head over here -

Pre-release builds

Ready when you are.
Next maintenance update - 80.6 - will be adding UI support for configuring program's update channel.

Update channels were first introduced back in 2016 and they got formally documented earlier this year.

However the selection of channels remained limited to newest, stable and, the everyone's favourite, symantec. There was no need for making channels easily configurable. Not until now.

A brand new update channel called preview will go live shortly after the 80.6 release and it will host development builds of upcoming major releases, starting with R81.

A chance for you to see what's coming and a chance for us to crash on machines other than ours. Win-win.

* Update channel configuration is an advanced option, so it will not be visible unless Help menu is opened with Ctrl being held.
Release 81 will include one of the most interesting and important improvements of recent times - the full-scale support for parallel backup processing.

So here's a quick screencap showing it in action.

The test setup is C:\Windows\System32 with about 15K files and 1500 folders being copied from an NVMe to an SSD drive.

R80 (before)17 sec
R81 (after)5.7 sec

More on this to follow.
The second improvement to the handling of cloud-stored files in 80.5 has to do with gracefully tolerating these files being offline.

Call it an opportunistic or best-effort backup if you will.

We will try and back these files up if we can, but we won't make much fuss if we can't.

When run, every backup job goes through 3 main phases:

  • Scanning
  • Planning
  • Executing

Scanning creates file indexes of the source and backup locations.

Planning compares resulting two indexes, finds the differences and compiles a list of basic steps - create folder, copy file, move this, delete that, update third - that, when executed in order, bring the backup in sync with the source.

Executing merely goes through the plan and diligently executes each step.

Having a formal planning phase comes with one very important benefit - it allows extending core backup logic with a minimum amount of changes, making it a very simple, fast and predictable process.

In case of offline files, the opportunistic backup is implemented by simply disabling scheduled file update if the source file has a Recall-On attribute set.

Simple, which is why the change can be rolled out as a part of a smaller update rather than needing to wait for the next major release.

* The same mechanism is already used by the option that disables file updates when the backup copy is newer than the source.

It works the same way - an update is scheduled using regular file comparison logic, and then an additional check is made to compare file times and to disable the planned action if needed.
Next update (80.5) will introduce several improvements to the handling of OneDrive files.

One of these improvements is a new option for excluding and including cloud-stored files/folders based on their local state.

As you probably know, files stored in the OneDrive folder may or may not actually have their data cached locally.

When there's no local copy available, the file is said to be offline and its contents will be retrieved from the cloud storage when the file is accessed. An offline file like this will have either Recall-On-Open or Recall-On-Data-Access attribute set, allowing it to be identified as such.

The Pinned attribute will be set on files that were marked for permanent local storage using "Always keep on this device" - These files will show up with the solid green icon in the Explorer. Conversely, the Unpinned attribute will be set on files that were explicitly purged from the computer using "Free up space" - Finally, there's also an Offline attribute that is set on offline files by older versions of OneDrive.

With release 80.5 it becomes possible to define rules based on these four attributes.

For example, we can exclude all unpinned files from the backup. Once in place, a rule like this will cause Bvckup 2 to remove file's backup when its OneDrive copy is discarded with Free up space.

While not dramatically useful by itself, this change paves the way for another improvement - an option for suppressing file updates if the source file is offline.

See Part 2 for details.
The company behind the best file replicator of the modern age is now known as IO Bureau.

This change took an excruciatingly long time, so here's a bit of background along with some field notes and takeaways.

7 years ago

The original company name - Pipemetrics - was picked in 2012, when the company was formed.

The plan back then was to make a distributed network profiling and monitoring system, tentatively called Notwerk. As quirky as it was, the name quite accurately predicted what happened next - the project did not werk out.

So not few months later we ended up switching to another idea - a proper, production rewrite of the original Bvckup. It was a file backup program that I wrote over a weekend few years prior and that was getting an unexpected amount of traction despite all my efforts to ignore it.

The company name remained though and according to it we were still knee-deep in the pipe measurement business. Even had a logo to match - That's an undersea network cable cross-section... or at least an artist's impression of the same.

In any case, all this was around 7 years ago.

4 years ago

The discrepancy between the product and the company name was one concern. Another concern surfaced a bit later on, when Windows 10 with its pervasive telemetry was released and more people started paying closer attention to the tracking and privacy matters.

And there we were, making an offline software for handling one's private data, and yet we were called ...metrics. What was it that we were merticsing exactly?

Neither of these concerns required an immediate name change, but they lingered. And so about 3 years in we started looking for a better name.


There are only two hard things in computer science:
cache invalidation and naming things.
Now, that is just factually wrong.

Naming things is not hard. It is an utterly frustrating and painful, painful process that drains all fun from life and creative juices even from those who didn't have them to begin with.

Sometimes you can stumble upon a great name in a matter of seconds, but more often you don't. Or you just google it up and realize that you are the N-th heir to the throne, ten years too late.

Say, you want to name something that deals with data. Dat should not be hard. Ding - "dat" is taken, and you haven't even considered it yet.


Pfft, lol. Are you kidding? Taken.

How about Datto?

Quirky, weird. No way anyone else would think it up.

You'd think. Taken.

In fact, taken so long ago that people now complain how much better it was in the good old days, whenever that was.


You know - uranium, unobtanium, pinacoladium. No-pe, taken.

Days go by and all that you can come up with are just OK-ayish bland names, each flawed in more ways than one, and most of them even without a matching dot-com domain.

Inevitably, you start looking for a way to make the process more formal, structured, predictable. To keep track of the "progress" and not to cover the same ground twice.

The good news is that it is doable. The bad news is that genuinely great names are still a matter of luck and random insipiration.

Part 1 - Keywords and associations

The name needs to be unique and memorable, obviously.

The shorter the better. There are exceptions, of course, but the rule of thumb is that it should be 3-4 syllables max.

It needs to be pronounceable and it should not have any negative connotations. Yes, I know, there's an irony in here somewhere.

In fact, looking back this is probably the most important property of a good name - it needs to roll off the tongue and leave a good impression behind.

That impression should be strong and memorable, even more so than the name itself. While people tend to forget specific names, the concepts and feelings stick around for much longer.

To that end good names often have an unusual, but an otherwise obvious association with whatever they are attached to.

An excellent example of this is a distributed database called...



Distributed storage indeed. How ridiculously good.

Naturally, there are great names that don't have any explicit associations.

For example, Akamai does have a very good explanation behind it, but it's not obvious. However the name is simple and it sounds nice, so it is easy to remember just because of that.

Same goes for Skype, originally standing for Sky Peer-to-Peer.

Kodak is yet another example, except that it's a completely made up word. George Eastman, the founder of the company, liked the impression the sound of the letter K made, so he went on to find a name that was "short, easy to pronounce, and not resemble any other name."

The year was 1888.

Finding a good association is a matter of certain luck, but the chances can be improved by formalizing the process.

By brute-forcing it basically.

Start by defining what it is that the company (product, gadget, whatever) does and what it is about. Compile a list of relevant keywords. Then for each keyword add several more - synonyms, related concepts, ideas, references.

Repeat several times.

The end result will be a list comprising both very specific and very abstract things that can be used as latches for an association.

For us, being in a data preservation domain, it went something like this: The list was over 50 entries long, stretching as far as "bastion", "marmalade" and "formaldehyde".

The problem, of course, is that a keyword alone usually doesn't make for an unique name.

Part 2 - Prefixes and suffixes

Lots and lots of names follow certain patterns. These patterns can be identified, distilled and formalized.

One such pattern is adding a trailer qualifier:

  • ... technologies
  • ... software
  • ... systems
  • ... project
  • ... group
  • ... labs
  • ... associates

Similar, but with more "edge":

  • ... initiative
  • ... innovations
  • ... engineering
  • ... committee
  • ... society
  • ... masters
  • ... theory
  • ... method
  • ... logic
  • ... works
  • ... matters

Alternatively, the keyword itself can be equipped with a suffix:

  • -com
  • -sys
  • -ware
  • -soft
  • -tec
  • -tech
  • -tronic
  • -mark
  • -craft

This tends to result in a rather pompous and boring enterprise-y names, which sometimes may actually be desirable:

Infocom, Unisys, Commware, Microsoft, Symantec, Datamark, Netcraft, etc.

Prepending prefix to the word is yet another option:

  • Para-...
  • Meta-...
  • Vari-...
  • Poly-...
  • Inter-...
  • Super-...
  • Hyper-...

InterPlay (of the Descent game fame), ParaData, HyperLoop, etc.

One other option I feel I must mention is that mixing Maxwell into any technical name tends to make it dramatically better.

The association here is that with Maxwell's equations that carry an incredible mix of mathematical elegance and complexity.

"Maxwell" also sounds nice, and it is just 2 syllables long.

As a result, we had Maxwell Storent on the short list. In the end it lost to other options, but I still think it was a great name, full of technical competence and credibility at no extra charge :)

Part 3 - Word mangling

Mangling a keyword is yet another way of turning it into an unique name.

If we were back in the .com era, prepending an e to everything would've sounded like an excellent idea. Still have fond memories working in a place called eTunnels.

Dropping a vowel was an equally acceptable option with a great following - Flickr, Tumblr, etc.

Duplicating letters also worked, e.g. Dribbble or FFFFound.

In fact, there was also Forrst, which boldly went where nobody dared, both dropping and duplicating like there's no tomorrow.

Replacing c's with k's and dropping trailing vowels tends to make words sound vaguely Scandinavian.

Objective becomes Objektiv, which can pass for a noun.
Accurate - Akkurat, etc.

Technik can help adding some German flavor to the name. We would've gone this route except that we are in a French-speaking canton and it didn't seem like the right thing to do.

Replacing trailing -e with an -a tends to create Latin-sounding words, e.g. Apertura, Machina, Textura, etc.

Similarly, tweaking the word to end with -ium or -eum results in a Latin-scientific feel.

The yield

Fast forward 4 years of episodic search spurts and we had a list of names with quite a few candidates on it. Here it is, with just few entries trimmed:

  • Data Mechanics - really liked this one, actually, a good pun
  • Data Matters - similar, also double-meaning
  • Data Associates - only if in hopes of being acquired by CA
  • Data Committee - implied that we don't get any real work done

  • Swiss Data Works
  • Swiss Data Reserve - implied that we are a storage provider

  • Datori
  • Datamata
  • Datamation
  • DataTechnik
  • SuperData
  • Redata

  • Storewell
  • Storent
  • Prestor
  • ByteMasters
  • MetaByters
  • Keepwell

  • Keep Systems
  • Parity Works
  • Zeroloss Systems
  • Data & Loss - the barristers

  • Stray neutrino
    * As per the theory that the bitrot is caused by neutrinos

  • Ashwell
  • Waltron
  • Corelion - as in Coreli-on, not Core-lion
  • Hyper - with "Hyper Abc" as product names
  • Variance
  • Wellmade
  • Struktura - also liked this one a lot, but the .com is squatted

And the winner is...

It's reasonably short.

It rhymes when pronounced.

It's generic enough to cover more than just backup software.

The and @iobureau were available.

It also can be made to look both like IO and 10 when rasterized down to a small size. Not terribly important, but having a simple symbol to go with the name is always a plus. So, ladies and gentlemen, until we decide to rename ourselves again we shall be known as the IO Bureau.

Bonus material

In case you want to try your hand at finding a good sounding, but otherwise random name, here are some pointers:

1. Reverse dictionary
To search for words with specific ending.

2. List of last names
To make your own Bang & Olufsen.

3. Lists of gods
Quite a few solid leads there.

4. List of Intel project codenames
Skylake, Ice Lake, etc. Mostly geographical landmarks.

5. List of Microsoft project codenames
Not as consistent as Intel's, but still worth a read.

6. List of Solar System objects
Including a handy list of moons.

7. List of planets in science fiction
Examples of names made up by other people.

8. List of fantasy worlds
Same here.


Longer reads

The Weird Science of Naming New Products
An NYT article that follows the naming of a very high-profile project. Perhaps with no directly actionable insight, but really interesting nonetheless.

Brand Names
This is a really really deep dive into scientific aspects behind naming and name perception. Fantastic resource... though a bit like trying to learn how to fry a steak by studying metallurgy and proteins :) Be warned that you may end up being even more confused than before.

Feeds  -  RSS  /  Twitter  /  Reddit
Release 80 has been out for over a week now and with it came the change to the licensing model that replaced Professional and Personal options with new Pro and Basic licenses.

To smooth out the transition the change includes a "grace" transition period, during which legacy licenses are still being available for purchase over at /buy/legacy page.

Once the grace period is over, legacy licenses will be removed from sale and only new Pro and Basic licenses will be available.

The grace period is set to end this Wednesday, June 26th.

To emphasize - All existing licenses will continue working just as they do now and new program versions will recognize and support both new and old license types. However it will no longer be possible to buy the old licenses.
Release 80 will be introducing a reworked mechanism for in-program checks for updates.

In particular, the UI will now make a clear distinction between upgrade releases and maintenance patches.

For maintenance releases it will also put more stress on the importance of an update if it happens to resolve a high-impact issue. See "Critical patch" screenshot above.

Additionally, the release notes are now displayed off the main view, giving them a bit more space if and when it's needed:
Release 80 will be making a small but important change to the default behavior of the exclusion/inclusion rules.

Starting with R80 all explicit folder exclusions will be final.

In plain English it means that excluding a folder by clicking on it in the middle pane of the Backup What window will now prevent any further rules from being applied to its contents: For example, if there were an include *.dll rule defined further below, it will now no longer pick up basebrd.dll from Basebrd subfolder.

It is, of course, still possible to re-include any items from an excluded folder just like so:

Prior to this change excluding a folder wasn't "final", meaning that it merely set the default action for all sub-items to exclude, and this action could still be overridden by other rules.

New behavior will be enabled for all backup jobs created with R80 and newer releases. All existing jobs will continue to use their original filtering logic and work exactly as they did before.

There is also a way to tell if a job uses the new or the old logic. Here's an older job: And here's one with the new logic:
Release 80 is nearly here and, coincidentally, so is the Bvckup's 5th anniversary of its first production release back in 2014.

So it's only fitting that the program's logo is getting a refresh.

Alte Haas Grotesk has been replaced by a considerably more expressive Brutal Type, slightly customized and re-kerned.

The symbol has also been redrawn from a bastardized V to a proper and better balanced geometrical construction.

The symbol now comes in three versions. One is shaded suitable for larger sizes, where the gradients do not get lost: Second is the monochrome, which is a reduction of the first version to just black and white: The last one is simple, which is just the shape without any embellishments as used for program icons and such.

While we are at it, it's probably worth sharing several other options that didn't make the cut.

An alternative weave: An alternative shading style: Something with color: Truth be told, the one of the left was my absolute favorite, but it ended up looking like a royal mess at smaller sizes. And smaller sizes are important, because, you know... Windows.
It's easy to do things when they go as planned. It's when they go sideways is what makes everything much more exciting.

In other words, the next maintenance release includes changes to better handle several new edge cases.

When reading a file, the program does that by requesting blocks of data, back to back: Then, as nears the end of a file two things can happen. If the file size is not a multiple of block size, the program will receive back an incomplete block: And if the file is perfectly aligned to the block size, the program will see either a zero-sized block or an explicit "end of file" notification or EOF for short. Upon seeing either of these lovely events, the program will know that it reached the end of the file and there's nothing else left to copy.

There is however a nuance.

In theory, the program may receive a partial block back even if it's not at the end of a file.

It is declared, that if this happens, it indicates a busy device or some other intermittent issue. The program is then welcome to retry the request or to read the remainder with a separate query.

In practice, this means a faulty storage device or a severely stressed system.

Further to that, very few program implement retrying on partial reads and OS vendors fully realize that.

So the overwhelmingly common practice is for the OS to just fail such requests, automatically meaning that a partial read = EOF.

Think different

However "overwhelmingly common" is not 100%.

As it turns out there do exist storage devices that will return partial blocks mid-file if ... the file is corrupted: Technically, it makes some sense - this allows salvaging at least some data from a corrupted block. Practically however this is completely useless unless you are in a data recovery business.

Partial mid-file reads

The first edge case that is now being explicitly handled by Bvckup 2 is that of reading a partial block mid-file.

More specifically, the program will now check that a partial block is the exact tail end of a file and it will raise an alarm if it's not.

Past-EOF reads

As you probably know Bvckup 2 uses async IO and it normally has several read/write requests "in flight".

With the new release, if it detects an EOF of the source file, it will then check any read requests for blocks beyond this EOF location and ensure that they too show the same EOF.

Rapidly growing files

There are cases when being able past EOF marker is actually OK.

If the source file is being actively modified and it is growing very quickly, the program may see an EOF followed by a actual data block, because the latter gets added exactly between the EOF and the second requests.

This is now also detected and reported accordingly.

Rapidly shrinking files

There's also an inverse case - the source file being very quickly shrunk down.

When this happens, Bvckup 2 will see an EOF first. It will realize that EOF is for a smaller file size than it saw when the copying started. So it will re-check the size and the current file size will now be even less than the recorded EOF.

Clearly, this is a mess and it is too now detected and logged.

Will be in release 79.26.
Next maintenance release adds support for indicating if a backup may be in an indeterminate state due to Bvckup 2 not being shut down cleanly on its previous run.

In particular, this is meant to highlight backups that were active when the host computer ran into a blue screen or experienced a sudden power loss.

Prior to this change the engine was already recording whether a backup job was running, used that to detect incomplete runs and to run a bit of house maintenance.

With this release, the engine also records at which exact phase the job was and then uses this information to purge any cached information that may no longer be accurate.

Most importantly this includes the file index snapshot, a.k.a. the "destination snapshot". If the job was already in the active phase, making changes to the backup location, the engine will discard the snapshot and rebuild it from scratch on the next run.

Will be in 79.25.
As you may know, Bvckup 2 can run in a so-called service mode which keeps the backups running even if the program itself is not started or when no one is logged in at the computer.

The program can be switched between the service and desktop modes on the fly and this functionality has been improved for both speed and robustness in the upcoming 79.23 release.


1.  The program can now automatically recover from any issues stemming from being foricibly closed/killed midway through the mode switch process.

This is different from before where any such recovery, while still always possible, needed to be done manually.

2.  Switching back to the desktop mode now displays a % done for the slow part of the process ( see below ).

3.  The slow part itself was reworked for speed and it was also fully parallelized. The more CPU cores you have, the faster it goes. It can still be slow, mind you, but it is less so now.

4.  Finally, it's now possible to cancel the switch if desired.

The slow part

Under the hood Bvckup 2 is essentially a pair of programs sharing a single executable.

One is the backup engine, which is a module that handles scheduling and execution of backup jobs, tracking of storage devices, dispatching of email alerts, etc.

Another is the user interface module - it talks to the engine, retrieves and displays a list of jobs and allows controlling and configuring them by sending requests to the engine.

Both modules operate independently from each other and communicate through a formal messaging protocol. Similarly their configuration data is also not mixed, with each module using their own directory to store all its data.

When the program is switched to the service mode, the engine module is simply shut down and re-launched as a system service.

Switching back to the desktop mode reverses that - it stops the service and starts the engine inside the UI process. Easy.

But there is one nuance. There always is...

When running as a service, the engine runs under a dedicated user account. This account is a member of the Local Admins group, so it has no problem accessing engine's config files even though they were created under a desktop user account.

However, any files it creates are owned by the Local Admins and they are not fully accessible by the original (desktop) process. So, going back to the desktop mode requires another step - we need to reset the owner on all engine config files.

This issue is further compounded by the fact that the number of files in the engine config can be really quite large.

This is because the engine config is where settings for all backup jobs are kept, and this includes the blockhash data of files copied using the delta copier.

The more files are delta-copied, the larger the engine config will be and the longer it will take for the owner reset to complete. So that is the slow part.
Work-in-progress for the new About window for Release 80.

Incidentally, this is what this blog was originally about - posting bits of work, UI sketches, coding details - all the daily grind that goes into making a seemingly simple piece of software and that stays largely hidden once the result gets shipped.

When you see a product that is simple and logical, it's very rarely that it came together that way on its own. More often than not it's a result of a long iterative process of trying this, that and third, going back, tweaking, revising and retrying, and eventually converging to a stable point.

So here's something to acknowledge all other 20-plus versions of the About window that were also really not bad at all, but didn't quite make the cut!   :)
All manually initiated checks for updates will now see the latest available version.

This applies to the case when a recently released update is going through a gradual roll-out phase and it's not immediately visible to all installations.

Previously, both manual and scheduled (automated) checks for new versions were treated the same, so it wasn't possible for an installation to retrieve a freshly released update until its IP address was included in the newest channel coverage.

With this change, gradual roll-out provisions now apply to automated checks only. Manual checks will now always be served through the newest channel. They will also override Symantec-specific processing if one applies.

For detailed description of gradual roll-outs, update channels, Symantec-specific handling and available configuration options see this post.
Preview of the Pro Support option for new Pro licenses.

Pro licenses will come with a year of Pro Support - an option that enables access to the program's upgrade releases (R80, R81, etc.) and priority tech support.

Basic licenses will not have this option, meaning that they can only be used with a program release that was current at the moment the license was issued. Additionally, Basic licenses will be supported through public support forum only.

Maintenance updates ("patches") will be provided for all license types, unconditionally. In other words, the bugfixes are on us.

After a year, Pro Support can be extended for another year. This extension is completely optional and it's not automated in any way. Extension fee is 20% of the original license cost.

See the original announcement for a bit more context.
Release 79.18 ships with a brand new storage tracking module.

Storage tracking module is responsible for maintaining detailed list of storage devices, file system volumes and drives.

The backup engine uses this information for tracking removable drives, running backups in response to drive arrivals, adapting bulk IO profile to the device specifics and for several other tasks.

In earlier releases volume information was queried in an ad-hoc fashion, when it was needed or when it became stale.

This release gathers all this functionality in a single place, adds support for tracking device information and improves detection of changes to network drives.

It also reworks processing of Windows device notifications, which are the "pings" sent by the OS when devices arrive or depart, volumes get mounted, locked, formatted, renamed, etc.

The program is now capable of sourcing these notifications from more than one place, which gives it a fallback option should the default notification source stop working ... at least until you get to reboot that WS2012 instance.


Storage tracking module includes explicit support for TrueCrypt volumes. This is notable because TrueCrypt does not implement all required volume notifications, making it impossible to track volume activity via standard means.

For example, a system service will never receive any alerts for TC volumes being mounted or unmounted.

These quirks are transparently accommodated. It all just works as expected.

Under the hood

New code comes from Diskovery, a storage inspection tool that we first released back in 2016 and have been steadily improving ever since.

In fact, the plan has always been to use Diskovery as a testbed for all things storage-related and then fold them into Bvckup 2.

And so we are now at exactly that point - all hard-earned Diskovery smarts are now being merged into Bvckup 2.

Staged merge

First, with this release the program gets access to raw device information, which is vastly more detailed and accurate than what's available through conventional Windows APIs.

Next, we'll be enabling the monitoring of device SMART status and tracking of SMART attributes. The code that supports this is ready to go, it's now only a matter of pacing the merge through its stages.

We are also on track for adding previously announced UI support for the storage monitor to the next major release, R80.

Stay tuned for updates.
With the next maintenance release it will be possible to quickly reload a backup job after adding manual overrides for program's many hidden options.

* As with other advanced options, the Reload item is shown only if the context menu is opened with Ctrl held down. Will be in 79.15.
Preview of the differences between new Basic and Pro licenses.
Support for new Basic licenses and their respective restrictions, coming up in the next feature release (R80).

Looked a various options for enforcing restrictions of simpler licenses and settled on the same approach currently used for expired trials.

That is, all and any features in any backup job can be used under any license, but if a feature is not covered by the license, then the backup job will be disabled after every run.

It's still perfectly possible to enable it back, but the only way to do this is to manually select the job and click on the Go button.
It is now possible to pre-compute delta copying state for large files when taking over existing backups with Bvckup 2.

The way the delta copying works is that it relies on block hashes saved from the last file update to detect which parts of the file changed since then and copy over only these modified blocks.

This means that the actual delta part of the copying kicks in only on the second update, because when we are copying file for the first time, we don't have the block hashes yet. Once the file is copied, we have the hashes and all set to do the delta updating.

This inability to detect block changes on the first run is not a big deal. Most of the times we are adding new files to a backup and these need to be copied in full anyway, so we do just that and initialize their delta state at the same time.

However, when taking over existing backups, it presents a slight problem.

Consider the case when you spent a day copying a very large file over a slow link to an off-site storage. Then, you created a backup job to handle all further updates.

If you are to touch the file now and run the backup, there'll be no block hashes yet, so the file will be re-copied in full. If only we could somehow initialize the delta copying state without copying a file...

Enter delta state pre-comp

Release 79.13 adds support for pre-computing block hashes for all files in the backup that qualify for delta copying but not yet have the delta state.

That is, if you put an existing backup under Bvckup's control, you can now flip a switch, do a quick run and then have all files ready for delta copying on their first update.

Quick here assumes that your source is local, reads are cheap and writes are expensive, which is how the setup should be for making full use of delta copying benefits.

The screencap above shows taking over a Debian VM backup that is initially created by just copy-pasting VM folder in Windows Explorer.

Running a backup executes delta pre-comp for .vmdk files and patches up "created" times on backup items - all without writing a single byte to destination.
Next release is adding support for displaying absolute date/time of the next backup run.

When the next run falls on today, the UI will show just the time. For runs falling on tomorrow it will say "tomorrow" plus the time. Otherwise it will show both the date and time.

Another change is a reshuffle of the About window:

There's now Check for updates option next to the version info, which is automatically displayed as Newer version is available notice if there was one seen at some point earlier on: The window also got a separate Validity field, which is used for displaying remaining trial time and also for confirming that all production licenses are in fact perpetual.
Next release is adding support for dynamically changing source and backup paths using run-time variables.

In particular, it's now possible to direct backups into a folder that is named after current date, hour, day of the week, etc.

It's been possible to do this using a bit of pre-backup scripting and it will, of course, remain the path of a true Jedi. For the rest of us though there's now a (much) easier way.

If you want to have a day worth of hourly backups, you'd just point them at ...\%HOUR%, and just as if by magic backups will go into ...\00, ...\01, etc. depending on the current hour.

For a year worth of weekly snapshots just use %WEEK%.

%DAY_OF_YEAR% will give you a year of dailies.

%DAY_OF_WEEK% - a week of dailies, and so forth and so on.

One caveat is that when the backup destination switches to a new folder, it will invalidate destination snapshot and require a full destination rescan.

Similarly, it will also invalidate delta copying state, so actively modified files will end up being recopied in full on their next update.

Finally, since no good feature is truly complete without a way to shoot oneself in a foot, be advised that the same set of variables can be used in ther source path as well.
An option for running Bvckup 2 at a low system priority has been around for a long time.

This option is primarily meant for machines that are already running close to their capacity, often due to using newer Windows versions on older hardware.

Enabling it caused the program to set its process priority to so-called Below Normal level and that throttled down both its CPU and disk run-time allowances.

At least, that's the theory.

With an introduction of the ultra copier, this option has become less effective, because the program is now capable of pushing disks to their limits even on a low system priority.

For this reason the next maintenance release (79.6) will extend this option to explicitly throttle down program's disk activity.

Internally, this is done very simply.

When enabled, this option will cause the backup engine to use a much smaller count of memory buffers to move data from A to B.

* It is also perfectly possible to throttle disk load dynamically by looking at the actual disk performance and by yielding IO capacity to other programs when required. This is planned and it will be coming with later releases.
As you might've heard (and more than once), the next release will coincide with a change to our licensing model.

Existing Personal and Professional licenses will be phased out in favor of Basic, Pro and Server tiers.

The principal change is that licensing options will now differ in what the program can do as opposed to what it can be used for. Licenses will remain perpetual and require no paid renewal.

The secondary change is that we'll also start making an explicit difference between feature releases and maintenance updates. Feature releases are major versions that introduce new features and significant improvements - R79, R80, etc.

Maintenance updates are minor releases that patch issues and fix bugs. In the current versioning notation these are indicated by sub-versions - 79.1, 79.2, etc.

The change itself is that we will continue providing maintenance updates for free, while the access to new features will be free for one year and an optional paid extension after that. The one year starts from the moment of licensing an installation. All major releases within one year window are included, each of which coming with a lifetime of maintenance updates. The paid-for option will extend the access to feature releases for another year and will be priced as a smaller percentage of the original license cost.

Basically, the bugfixes are on us. The new features, should you need them, are at small cost after the first year. If you don't find any new features useful, you will be able to keep on using your existing release that will remain fully functional and supported.

Finally, all legacy licenses (of Personal and Professional types) will continue receiving all feature releases free of charge, just as they do now.

Questions or comments? Post them here or send them privately.
Rolled out a web dashboard option for reseller partners to allow assigning purchases to their customers.

This option passes control over purchased activation codes and licenses to a customer, granting them access to the dashboard, but without exposing any transaction details. This also conceals code and license details from the reseller, leaving only purchase details visible. To each their own.

* We are always looking for new distributor partners. If you are a VAR, an IT service provider or a system integrator, drop us a line and let's talk.
Heatmap | More contrast
(Black/blue parts are the unchanged blocks)
Here's a little fun data visualization.

This is a heatmap of a day worth of changes to a large TrueCrypt container file that was backed up with delta copying and all block changes diligently saved as a bitmap. Block changed - its bit is 1, unchanged - the bit is 0.

Container file is 256 GB, the block size is 64 KB, which gives us 4M of blocks to the total of 512 KB when packed into a bitmap.

With the bitmap at hand, we head over to to generate a 2D entropy map using a Hilbert curve fill, overlay with absolute changes, then pass it through some cosmetic filters and, voilà - a simple way to see where the delta copying gets its speed from.

The brighter parts are the 10 Gigs worth of changes, the dark background is 246 GB of unchanged space... meaning that we ended up copying under 4% of the file and spent a fraction of time doing that.

*   For a detailed tour of delta copying see here.
By popular request - an option to suppress tray icon blinking when a periodic backup job misses its scheduled run.

This covers all causes - the computer being off, sleeping or storage devices missing or being inaccessible.

Will be in Release 79.
This is the final part of the ultra copier tour and it covers a much improved support for resuming copying after cancellations and IO errors.

It works just as you'd expect - if a copying of a file is interrupted part-way through, then on the next run the copying will resume from around where it was before.

The most interesting bit here is that this works even if the copy is aborted by disconnecting a drive, in the middle of a write. More often than not this will cause data corruption, but the program is now capable of detecting the extent of such corruption and correcting it without recopying the file in full.

†  Some conditions apply

First, this is enabled only for files that are delta-copied, because all resume-related data is saved with the file's delta state.

Second, the source file must remain unchanged between the attempts (as witnessed by its size and timestamps). Because if the source changes, we don't know where the change was, so we must start from the beginning.

Third, the destination file is, obviously, expected to also stay the same, but that's a more general requirement for the delta copying as a whole. Touching target file between the runs will automatically invalidate its delta state and trigger a re-copying.

* For people lacking certain excitement in their lives it is possible to suppress the last check. Inquire within for details.

In addition to recording how far it went in the source file, the ultra copier also remembers the reason for aborting a copy. Three main reasons are the read errors, write errors and user cancellations.

In case of read errors and user cancellations the copier gets a chance to shut down the copying process in an orderly fashion. This ensures a consistent state of the backup copy, so in these cases the copying is always resumable.

In case of write errors it depends.

Recovering from write errors

If we are to yank out an USB drive while it is writing data, the state of the backup copy will be somewhat uncertain. Ditto for the network copies when the router decides it had enough for the day and dies.

So when we are resuming after a write failure, we need a way to ensure that the state of the destination file is consistent with our (or rather delta copier's) view of it.

Ultra does that by keeping track of file ranges that it successfully updated on the last run in a form of a write log. However since we are resuming after a write failure, some of these writes may actually have never hit the storage media even though they were reported as "completed". This is due to all the caching, lazy-writing and outright lying that modern drives do to improve their performance.

For this reason when resuming after a write failure the ultra will go through the write log, read respective parts of the backup copy and check their hashes against those stored in the delta state. If there's a match, we are in clear. Otherwise, it's a block we need to re-copy.

The earliest non-matching block from the write log gives us an adjusted resume point. Simple... with a hint of elegance.

So there you have it...

1.   Faster bulk copying  #
2.   Faster delta copying  #
3.   Resuming support and error recovery

All courtesy of the new ultra copier. Coming to an update server near you in a few days...
This is a part 2 of a tour of the Bvckup's brand new copying engine dubbed the ultra copier.

Part 1 covered improvements to the engine's bulk copying and in this post we will look at changes to its incremental file updater, a.k.a. the delta copier.

Quick recap

Delta copying is a way of updating files whereby only blocks that changed since the last update are copied over. When files change, more often than not they change only here and there, in an isolated manner. This makes delta copying a godsend for backing up all sorts of large, slowly changing files from large RAW photos, to database files, to VM images and encrypted file containers.

If you are familiar with rsync, it is based on the same idea to derive the same speed benefits. However where Bvckup 2 differs from rsync is how it does change detection.

Detecting changes

The recipe is simple - split the file in large-ish blocks, compute block hashes and then see if the hash changed from the last time we looked at the file. If it did, we got a modified block, so we copy it over.

We still end up reading a file in full, but we save time on being selective with writing, which is usually an expensive thing to do.

Bvckup 2 works by saving block hashes between the runs and then using these as a reference point on the next update.

read block
hash block
load hash from the last time
if (new-hash old-hash) then
    write block
    save new-hash

This requires only basic IO access to source/backup copies and some local space to store the block hashes. That is, the delta copying works with any storage devices that are accessible for reading/writing.

*  In comparison, rsync requires a copy of itself to be running on receiving end (that's when copying over the network). It doesn't require any interim storage, but it ends up reading both source and target files in full. This makes rsync poorly suited for local copying and it also explains the r in its name.

False negatives

It is possible that two different blocks of data will have the same hash. This is a so-called hash collision.

With delta copying if we ever run into a hash collision, we will end up skipping a modified block ... which is something we most certainly don't want to be doing.

Avoiding false negatives comes down to reducing the chances of running into them on a real-world data. This is done by using cryptographic hashing for its avalanche effect - a property that causes a hash to change drastically in response to small changes to the input.

Using longer hashes is another option.

And using multiple hashes computed with different algorithms is yet another one.

Old delta setup

From its very first public release and up to Release 79 the program used the following delta copying setup:

Block size32KB
Block hashesMD5 + a variant of CRC32
Full-file hashSHA-1

This yields 20 bytes of hash per block, meaning that the delta state of a file is about 0.06% of its size.

Full-file hash

In addition to hashing individual blocks, delta copier was also computing a SHA-1 hash of the whole file and storing it with the delta state.

The purpose of this hash was to catch cases when no modified blocks were detected but the full-file hash changed.

That was *not* a very good idea. It added very little in terms of the false negative protection and it created a bottleneck.

See, SHA-1 can't be parallelized. In fact, very few cryptographic hash functions can be and they all expect to be fed blocks in sequence, one by one.

This means that performance of the delta copier was effectively bounded by the speed of SHA-1 on a single CPU core.

For mechanical drives it's usually not an issue, because they are quite slow. But SSDs and NVMe drives can be read faster than we can hash what we get from them:

HDD Seagate ST9500325AS 70.5 MB/s
HDD WDC WD5000LPVX 118.8 MB/s
SHA-1 Intel i3 @ 2.53GHz 332.4 MB/s
SHA-1 Xeon E5 v4 @ 2.10GHz 380.1 MB/s
SSD Samsung T3, USB 3.0 429.7 MB/s
SHA-1 Intel i5 @ 2.70GHz 520.9 MB/s
SSD Samsung 860 Pro 534.8 MB/s
NVMe Samsung MZVPV512HDGL 2157.9 MB/s

*  These are read rates for the drives and bulk rates for SHA-1 using a well-optimized implementation.

So the full-file hash is a bottleneck and it should be eliminated, which is exactly what the ultra copier in delta mode does.

New delta setup

Starting with Release 79 the delta copying uses a larger block size, replaces MD5 with Blake2b and compliments it with two weaker, but very fast hashes called xxHash and spooky.

Block size64KB
Block hashesBlake2b + xxHash + Spooky
Full-file hash-

Per-block hash size is now 32 bytes - 16 from Blake2b, 8 from xxHash and 8 from Spooky, but the blocks are twice as large, so the ratio of delta-state / file-size is actually lower, ~ 0.05%.

Impact on performance

Block hashing is now noticeably faster:

MD5 595.4 MB/s
SHA-1 561.9 MB/s
SHA-256 247.4 MB/s
SHA-512 391.4 MB/s
Blake2b 762.3 MB/s

Non-crypto hashes, xxHash and Spooky, are running at 10.8 GB/s and 11.7 GB/s respectively. That's gigabytes per second, so these hashes effectively come free of charge.

But more importantly, we are no longer constricted by the full-file hash, so we can push these 762 MB/s per core.

*  Numbers are obviously specific to our test machine, but they give a good idea of relative performance of the algorithms.

The net effect of these changes is that the delta copier now can fully saturate an IO of a very fast drive while using just a small handful of hashing threads.

This in turn means that delta copying now comes with virtually no performance impact even when it ends up copying files in full. It basically goes as fast as a regular bulk copier.

An example

Copying a 16 GB file from an NVMe to an SSD drive using the old delta copier vs the new one:

Release 78Release 79
Creating262.8 MB/s547.2 MB/s
Updating272.5 MB/s2026.6 MB/s

Copying file afresh now runs at the write speed of the target drive and updating it afterwards (with little to no changes) runs at the read speed of the source drive.

Needless to say, that's a huge improvement.

Next up is the part 3 of the ultra copier series. It covers resuming of interrupted copies and fast error recovery.
The backup engine now understands OneDrive failures related to files being unpinned and offline.
The next release (R79) will add support for automatically retrying file system operations that fail for network-related reasons.

More specifically, the backup engine will retry an operation for up to 20 times in 15 sec intervals when running into Unexpected network error, Location is unreachable and several other failures.

This should come especially handy when running larger backups over long connections, e.g. inter-branch VPNs or similar.

Prior to this release the engine already maintained a list of so-called fatal errors that causes a backup to be aborted if any of them is encountered.

These are errors like Disk corrupted or IO device error and they signal an unrecoverable condition that will cause all remaining backup steps to fail the same way.

Network-related failures were also on the list, but now they've been moved on a separate list of retryable errors, for which the engine is, basically, "trying harder".

The UI will still show a total error count in the job's summary, but it will render it in black if all errors were recovered from by retrying. See the In action screengrab for details.

The retry count and the pause interval are configurable, but the 20 x 15 sec default should work well in most cases.

It's also possible to add/remove error codes to both fatal and retryable lists, so if you ever need to retry on, say, Disk full, you can certainly do that too.

Ultra copier is an internal name of a brand new bulk and delta copying module, shipping with R79.

A complete rewrite, it builds on what we learned in the past 4 years and implements significant improvements in several key areas.

  • Copying of smaller files
  • Working with very fast drives
  • Faster scalable delta copying
  • First-class support for resuming of copying
What follows is a tour of the first two items. The other two will be covered in a separate post.

Smaller files

Copying lots of small files quickly is a challenge.

The per-file overhead of prep/post work is often comparable to the time needed for the actual copying. This fixed cost is split between program's own overhead and the time spent actually opening and closing files, copying meta info, etc.

With older drives this was not an area worthy of optimization, because the cost of merely opening/creating a file dwarfed that of any prep work that the app itself was doing.

However with newer drives and faster machines it's no longer the case. All that trivial activity like allocating buffers, writing to the log, pre-configuring the IO - it all suddenly adds up and starts to matter.

For this reason the ultra copier now aggressively pre-allocates, caches, recycles and otherwise streamlines the prep/post phases to keep its per-file overhead to an absolute minimum.

The effect of this obviously varies, but it can be as eye-popping as a threefold speed-up, for example, when cloning C:\Windows on an NVMe drive.

Faster drives

Bvckup 2 has been using multi-buffer async IO from its very first release.

The core of the technique is that the program doesn't wait for read/write requests to complete, but it rather just queues them with Windows and later checks if they are done.

That last bit - check if it's done - is where the new code does things differently.

Ultra's IO pipeline is built around IO completion ports (IOCP) which it uses to track, well, the completion of IO requests.

The program issues read/write requests as before, but it also asks Windows to queue a "done" notification once a request is completed.

This queue is called an "IO completion... port", which tends to muddy the waters somewhat, but it is one of more elegant and useful mechanisms of the Windows kernel.

The key point of IOCP is in the last line: With IOCP we no longer need to drag the full list of pending requests across the userspace boundary just to learn which of them might've been completed.

This makes IOCP really quite fast.

But wait, there's more.

IOCP can also accommodate async non-IO operations.

The ultra copier makes a full use of this when delta-copying a file. It feeds a stream of block hashing requests to a pool of worker threads and then receives their completions events via the same port that it uses for IO requests. This allows for uniform handling of all async operations in the IO code. Reading, hashing and writing now become equal parts of the IO pipeline, leading to a simpler code.

Synchronous IO

There are documented cases when async IO requests may complete synchronously. In fact, Microsoft also says that your code should be prepared for this to happen at all times.

We care about synchronous completion, because it makes the pipeline stutter, so it's not good for performance.

When this happens, Windows still queues an IOCP notification, so the simplest thing to do is to ignore how request completes and just wait for an IOCP ping.

This however adds a small delay to the IO flow, because we end up completing a request later than we could've.

You probably see where this is headed and you are correct - the ultra copier suppresses IOCP pings for sync reads/writes and processes them immediately.

If you ever wondered what SetFileCompletionNotificationModes is for - here you go, now you know :)

Locked IO buffers

Among smaller performance tweaks, the ultra copier defaults to using unbuffered IO when reading larger files. Larger files aren't likely to be cached in full, so bypassing the cache has a small, but noticeable effect on the reading speed.

There also happens to be a way to further improve performance by locking IO buffers with SetFileIoOverlappedRange.

    * Some conditions apply

In particular, this requires holding a rather exotic privilege and it just may lead to memory starvation for the rest of the system. Ask me how I know.

But when used with care it does appear to improve bulk IO rate on faster drives.

Next up, the delta copying improvements.
The UI will now show read/write byte counts for longer copies, and it will also display the percentage of data that was written out during delta updates. The lower it is, the better.

* Also, the progress bar got a facelift... which should be of little surprise, because it's hardly the first time it ever happened.

May 2013: Jun 2013: Sep 2013: Nov 2013: Nov 2013: Feb 2014: Aug 2015: Dec 2017: The original plan was to switch to the last style, but after testing it on live data it proved to animate very poorly. When IO buffers are large (which is common) that nice right-to-left sliding animation turns into a jumpy-jerky movement which is plain confusing.

In any case, here's a version that made it into R79... ...though that's certainly not the final word in the progress bar saga. Far from it.
Just a quick pair of screenshots to introduce a little something called the ultra copier.

It's a faster, lighter and altogether more beautiful rewrite of the core copying module which handles both bulk and delta copying.

The numbers are for copying ~100K files from C:\Windows from one location on a fast drive to another, with the OS file cache purged beforehand. The speed up is indeed almost threefold...

More to follow shortly...
This is Disk IO Monitor - a fairly simple utility that samples and displays, in real-time, read/write rates of a physical drive.

Read activity is on the top, write activity is at the bottom.

What's interesting about this graph is that it can display samples as large as gigabytes and still clearly show byte-level differences between them.

To that end, it spreads each sample across several horizontal bands, each displaying a count of full GBs, MBs and KBs, plus "leftover" bytes in each sample.

If we want to display, say, 12345678, then it's 11 MB + 792 KB + 445 bytes, so the band closest to the zero line shows a bar filled to 445/1024, the next band - 792/1024 and the 3rd - 11/1024.

This is basically a mod-1024 factorization of a sample value.

It is also a discrete (rather than continuous) variation of a log scale, which is very commonly used for "squeezing" large values into more manageable ranges.

In practical terms this allows us to quickly see how different any two samples are, even if they are very large.

For example - These two samples are both in the MB range, very close to each other, but looking at the KB band it's immediately obvious which one is larger and approximately by how much.

This sort of analysis is often needed when trying to spot jitter that is order of magnitude smaller than absolute sample values.

In our case, this need surfaces when analyzing the effects of OS file caching on bulk IO performance.

Here's a 16 GB file being copied using Windows Explorer: Here's the same file being copied with a custom, somewhat better optimized code: * Both tests are run with the file cache purged.

Windows Explorer makes uses synchronous IO that goes through the Windows File Cache. This delays writes and it also adds jitter to read operations.

In comparison, the second graph is for the code that uses asynchronous IOCP-based copying engine that bypasses the OS cache. The resulting data flow is far more stable and predictable. It's also substantially faster.


Here's a dev build if you feel like seeing this thing in action:


~ 72KB, no dependencies, signed. Just save somewhere and run.

It defaults to monitoring of the C: drive, but another drive can be selected by clicking on "Disk C:". The sampling rate is 500 ms and it is fixed for now.

The Purge cache option will clear the so-called Windows Standby List, which effectively empties the OS file cache. Use with care, obviously.

Happy graphing :)
Next version of Bvckup 2 will have a proper understanding of machine's entire storage stack - lettered drives, simple, dynamic and virtual volumes, physical disks, their types, models, connection details, health data, etc.

This will allow the backup engine to automatically adapt its IO strategy to specific devices used in any given backup job. It will also let the program to keep an eye on the devices' health, which is not a bad thing to have either.

The storage stack scanner is adapted from another project of ours called Diskovery and the IO profiling logic is based on the data gathered through CCSIO Benchmark.

While the scanner part is anything but trivial, the most time-consuming part, as per usual, was the UI.

There's a lot of information to be displayed - both the properties of volumes and devices, and how they relate to each other.

As you likely know, a device may host multiple volumes and a volume may span multiple devices. So in a most general case it's a full-mesh, which is the reason why we can't display volumes and devices as a simple hierarchy.

So, after going... ... through ... ... some ... ... options ... ... it became obvious that having two lines per entry was a way to go and that volumes and devices should be on separate lists, both visible simultaneously.

After a bit more sketching the lists arranged themselves into two columns and the scroll bar of the left list went to the left of it: * I must say that the idea of placing that damn scroll bar on the left took waaay too long to converge to. For whatever reason it's just a strangely un-obvious thing to do...

In any case, the result is a layout where we can display relations between volumes and devices with literal, drawn links.

After that just needed to take care of smaller details like scrolling slits... out-of-view arrows... and connector points... and, voilà, we now have something that is both usable and reasonably good-looking. Still to be added is a separate pane for displaying further details - device hardware info, SMART data, etc. This pane will sit at the bottom of the window and will switch between showing volume and device details depending on what's selected above.

Stay tuned for the updates :)
You know how some people, when tired of their established ways, tend to move furniture around the house to make the same thing look like something new?

Here's the same idea applied to the world of progress bars.

This is a new and improved progress bar that is making its way into R79. The idea is to use the vacant space on the right-hand side (the "todo" part) to show the sub-progress.

In comparision, here's a lively version we have now: This is a GIF, trust me.

In Bvckup's case a progress bar also needs to accommodate the delta copying. In particular, the bar should be able to display processed file blocks that were copied in full, copied in part or not copied at all.

So here's a short list of options to achieve just that: Will probably go either with the 2nd from the top or the 5th from the bottom. Maybe both.
There are several ways in which a program can query security attributes (Owner, Group and ACLs) of one file and copy them to another file.

In a true Microsoft's way, some are well-documented, but they don't work as advertised, and some work as needed, but they are not documented.

First hit on Google is Get/SetFileSecurity. The name checks out, it accepts a HANDLE, so it should be quick and simple way to do what we want.

Nope, it's not.

It's an older API that doesn't correctly handle inheritable ACLs and its use is not recommended. Not by the API documentation, mind you, but by a random blog post from one of many Microsoft development teams, which you still need to find first. What they recommend instead is Get/SetSecurityInfo.

And you'd still naively think it's the one.

Nope, it's not.

The problem is that SetSecurityInfo will fail with "access denied" if you are trying to set DACL and the file is not opened with READ_CONTROL.

You may wonder why the heck it needs read control... excellent question. That's because it wants to look up the name of the file and then *feed that name into SetNamedSecurityInfo function*. The latter will then re-open the file and start spraying kernel with requests, which is the last thing you need when you are already quite busy with copying a million of files from A to B.

Looking at the trace of SetSecurityInfo with Procmon shows that it generates over a dozen of kernel calls... whereas only one is really needed.

The name of that one call that we want is NtSetSecurityObject, but it's a part of so-called Native API and as such it is officially off-limits for any application use.

However, if we are to comb several pagefuls of relevant APIs, we may spot something called SetKernelObjectSecurity, which looks suspiciously like NtSetSecurityObject.

Long story short, you want to copy security attributes - use Get / SetKernelObjectSecurity. So now you know. Congratulations :)
R78.6 adds support for alternate data streams.

This is a rather obscure, but interesting feature of NTFS that allows storing secondary data blobs under the same file name.

You'd give such blob a name - say, ABC - and then you'd be able to read, write and delete it by referring to it as filename.ext:ABC.

No doubts, some of you may be thinking why though?

The answer is that alt. streams is a generic and more flexible replacement for an older "extended attributes" mechanism. It is meant for the cases when you may want to pin an extra bit of info to a file (or a folder), the info that is application-specific, but that doesn't belong in a file itself.

An example here is a small tag that browsers add to executable files downloaded from the Internet. This tag is used by Windows Explorer to show its "Ugh-oh, something from the Internets" warning when you are trying to run such files. The tag is stored as an alternate stream called Zone.Identifier. You delete this stream - you won't get the warning.

Long story short, Bvckup 2 now can replicate these secondary data streams. This option is in Backup Settings > More Options > Copy Also section and it is Off by default.
New version of Backup Settings > More Options > Copy Also is coming up in Release 78.6.

•   File attributes are out.
•   Owner and Group are now a single multi-state option.
•   DACL is now called Security info.
•   SACL is now in, as Auditing info.
•   Alternate NTFS streams are in as well.

The rationale behind File attributes option removal is that it was the only one from the list that was familiar to non-professional users. That instilled a false sense of confidence when looking at other options, enticing to tick them all on.

From now on the list comprises exclusively... erm... alien-looking stuff, so hopefully this will deter people from randomly enabling options they should not really be enabling. This also lets those in need of a complete replication configure it now directly in the UI.
Here's a way to present two closely-related options with a single checkbox with 4 distinct states.

It is very rarely (if ever) that anyone would need to copy Owner information without also copying file's Group info. However, these are separate bits, so bunching them together doesn't seem like a right thing to do. But giving each of them their own piece of UI real-estate doesn't feel right either.

So how can we pack them up?

First, let's look at how this can be done conventionally. We'll just be verbose about it and list all the options as is: This is OK in terms of usability, but when taken as a part of the whole it makes window look cluttered: And *unnecessary* cluttered at that, because in a vast majority of cases Owner and Group will be either both checked or unchecked. Ditto for Security and Audit options.

So let's try something else.

Windows natively supports something called "3-state" checkboxes that can be in checked, unchecked or indeterminate states. "Indeterminate" is a rather strange term as if the program is not sure in which state the option is. However we can think of it as an in-between or partial state and do something like this: This is not bad, however we lose the overall meaning of the option when it is in a partial state: So let's put the context back: Better, but now we are back to things being too verbose and looking cluttered. Let's simplify it for two most common cases: Better, but now it's off visually: How about with chop the specifics and move them to the side: Almost there, just need to patch up visual hierarchy a bit to help telling principal parts from the secondary ones: And that's it. The result is an UI that looks clean and simple for two most common cases, fully supports marginal ones and packs 4 options in a space of two.
Earlier today we rolled out a new revision of purchase discounts.

Discounts are now dead simple - they are based on the number of licenses that you already have plus the size of your order.

As simple as that.

Also, the purchase form was reworked to display effective pricing if you are logged into your dashboard account (and we basically know who you are).
Licensing server now recognizes a handful of cheat codes. These codes are still refused, but at least in a more respectful manner :)
Also rolled out today is the first revision of Dashboard API.

The API is primarily meant for integrating with external inventory management systems by exporting dashboard data in json/csv formats.

A bit later we'll be expanding it to allow changing things as well, so there's that.

The API is in pre-beta, so it's available on opt-in basis only - if interested, drop us a line and we'll enable it for your account.
Rolled out a couple of improvements to the web dashboard.

One is an option to hide revoked licenses from the list and another is a support for automatically adjusting all displayed dates/times to browser's native time zone, with an option to override if needed. Previously all times were displayed in CET time zone (because that's where our backend servers are).
Release 78 will introduce changes to file/folder filtering system to make it a bit more flexible and logical, while also keeping it as point-and-clicky as possible.

In pre-78 releases if you excluded a folder, you weren't able to then go and include some of its subfolders or files. Similarly, it wasn't possible to exclude some of the items in an explicitly included folder.

In fact, it wasn't possible to exclude anything at all if using the Start with an empty list mode or include any item when using the Include everything option.

While this approach mapped on a simpler code and sort of made sense when it was first coded, in retrospect it wasn't that good. So, R78 will come with a fundamental overhaul of the filtering module... and whatever new and exciting bugs that may come with it.

What the middle pane now does is it allows quickly setting the action (include or exclude) for folder and files by simply clicking on them. For files the action is final - if you exclude it, then it is excluded. End of story.

For folders however it also sets a default action for all its files and subfolders. This action may be overriden for any item by clicking on it, which simply reverses the default. So when bvckup2 looks at what to do with any given file or folder, it will first pass it through this set of filters, yielding a preliminary include or exclude action.

Next, the item will be passed through generic filters, from the bottom pane. First hit on any of these determines the outcome. If there are no hits, then the default action it is.

There are also new, finer controls over the filtering flow. In short - it's now possible to mark any generic filter as final, which will suppress generic filter matching for all folder contents.

Bonus material

If you are to think about all this for as looong as I have, you might realize that the Include everything / Start with an empty list selector at the top, this one - is actually no longer needed!

All it does is merely sets the default action for the top directory - and we may just allow doing that by simply clicking on it in the middle pane.

However I think having it a separate option helps making the UI a bit more approachable and easier to use, so it will stay.
Release 77 will introduce the Errors only view for the logs.

As you might've heard the log viewer happens to be one of the most complex parts of Bvckup 2. It looks simple, but on the back it is anything but.

1. Log files are created by the backup engine, but displayed by the UI module. In desktop mode this is easy enough to handle, but in service mode the engine runs in a process of its own and under a different user account, meaning the logs are simply inaccessible to the UI.

So what ends up happening is that the engine opens logs on behalf of the UI and passes open file handles back to it.

PS. Ever wondered what all those "process" handles are doing on the DuplicateHandle() argument list? Exactly to allow one process to do something privileged on behalf another process.

2. Log files are rotated, so what you see in the UI is actually a seamlessly glued together list of all log entries across all copies of a backup log.

This would've not been a big deal if it weren't for...

Finally, the UI needs to expand/collapse log parts and do it nearly instantaneously. This is a hard problem, especially considering that logs can grow very big.

As the engine writes out the logs, the UI scans them and creates an auxiliary index, which it then uses to quickly understand what should and should not be shown in any given state.

These ndecies cannot be kept in memory, not in full. Instead the UI stores them in a specialized on-disk database, which too needs to be very quick.

However, all this complexity had a nice little side-effect. It made adding the Error view very easy.

All the UI had to do is to build another index and include just the error entries and their immediate context. This way switching to/from the Error view is a simple matter of swapping one index file for another. That's it.

This might not be the most exciting thing ever, but still has some elegance to it, doesn't it?
One of several tweaks coming up in Release 77 - the on-hover extension of long log lines.

Windows comes with native support for this sort of functionality and it's called "in-place tooltips". For example, they are used in Windows Explorer when displaying file and folder names that don't quite fit the allotted space.

But in a true Windows fashion, while it works, it also... flickers!
Blinkety-blink. Here's a slo-mo in all its glory: So after killing several hours on trying to make it behave, it became clear that it's faster and simpler to just redo it from scratch.

Not the first time it happened and, sure as hell, not the last :-|
Today we rolled out a new addition to the /purchase page.

Internally known as Perks for Tweets it allows receiving a bit of a discount in exchange for a Bvckup 2 mention on Twitter.

The main reason we are doing this is to try and reach out to new people, so the fine print is as follows. Your Twitter account should be at least a month old and you should have some number of followers and updates (tweets) on record. A tweet should include a link to and/or an @bvckup mention (but without being directed at @bvckup).

Keep in mind that this is one of them wild marketing experiments, so things are subject to change in the next few weeks.
Starting with 76.9 when the app is switched to run as a service, the engine will now run under a dedicated user account.

The account is called "Bvckup 2 Service". It is automatically added when the app is switched to the service mode and removed when the app is switched back to the desktop mode or uninstalled.

The account is made a member of Administrators and Backup Operators groups, which gives it the same set of privileges as provided by the LocalSystem account.

Ideally, it should've only needed Backup Operators membership, but in some cases this group lacks network access privileges, so the default is to add it to the Administrators group as well.

It is still possible to install the service under the LocalSystem account - click the More button at the bottom left of the Switch Mode dialog and flip the Run service under option.

This change has been long coming as it's an altogether better and tidier setup. It helps isolating Bvckup 2 from the rest of the services, eliminating effects of any custom LocalSystem policies that might be in place and it also allows for a fine-grained control over its permission set. So if another Windows update comes out and (again) breaks things, it should be much easier to hammer it all back in place.
With the next maintenance release (76.8) Bvckup 2 will log device status when a backup job is created, loaded or enabled (after being in a disabled state).

The app was already logging device arrival/deparature events and OK'ays from the share monitoring module, but the messages were lacking consistency and no initial device state was logged.

All tidied up now.
options: 1a | 1b | 1c | 2a | 2b | 2c | 3a | 3b
Reworking the welcome popup that is shown on the first launch. it will now briefly go over the differences between file and system backups, explain how the trial works and what licensing options are available.

This set of sketches shows various styling options for the index on the left hand side of the window.

In comparison, here's the existing version in all its one-page glory:

Rolled out a couple of Customer dashboard improvements.

First is a cosmetic change to how an inline editing is done - click on a Pen icon, start typing, press Enter to save, Escape to cancel (or click on respective button if that's your preference).

Second change is a bit more interesting - it adds support for deactivating licenses via the dashboard.
Please note that this feature is currently reserved for site license and larger resellers only, and it is enabled on a case-by-case basis.
Another change in upcoming 76.5 is a new warning that is shown when copying of owner, group or DACL attributes is enabled for a backup job that goes across the computer boundary .

As you might guess the warning is due to what we are seeing on the support end, when some people enable some backup options merely because they look reasonable, but not because they actually need them.
Added support for pre-seeding backups and adjusting source or backup locations when the on-disk data is being moved around.

Previously, if either source or destination location was changed in backup settings, the app would purge all delta copying data and the snapshot of destination if present. This caused destination to be re-scanned on the next run and all files to be re-copied in full next time they needed an update.

With the next relase (76.5) the app will now ask if the backup state should be purged or not. This way if you are setting up a remote backup job, you may first pre-seed the backup by attaching the drive locally and making a backup on it. They you'd take the device to the remote machine, change Destination path in the backup job, say No to the state reset and - voila - you are back in business as if the drive has always been remote.
Bvckup 2 is getting media and device ejection support.

Media ejection refers to, well, ejecting storage media from an otherwise fixed device such as a tape drive.

Device ejection has to do with "safe removal" of USB drives, the same thing you can do by clicking on a standard Windows "safe remove" icon in the system tray before yanking drive's USB cable out.

However, in a true Windows fashion ejecting a media sometimes also ejects a device and sometimes you can also eject media from an external USB HDD (and, yes, this makes absolutely no sense).

In any case, the next maintenance release (76.5) will get ejection support, but without the UI part. It will be possible to set it up with a good old INI file edit only.

Then, next full release (R77) will get the UI support and above sketches are for that bit. The tricky part is that ejection settings are ultimately per-location preference, so they are needed for both source and destination drives and this takes 2x of screen estate to accommodate.

Secondly, ejection is available only for local drives and not network shares, so the Option 2 series aren't really good. They all allows enabling ejection for non-local drives, meaning that we have to do extra "sanity checks" and error messaging when validating backup settings. That's just... messy.

In comparison, Option 1 simply doesn't show ejection option for non-local drives, just like it doesn't show device tracking (the pushpin icon) in that case. So it's tidy and logical.
The 9th revision of the website is now live.
New tray icon state for the case when all configured jobs are disabled. Will be in the next R76 maintenance update.
This is a development test of new parallel copying feature that may be making its way into the next release.

If you are familiar with robocopy, it's an equivalent of its /mt option, which causes several files to be copied, well, in parallel.

As it's probably obvious from the above screenshot it works really well, especially for smaller files when an overhead of opening and closing file is comparable to the bulk I/O time. Here, for example, C:\Windows\Fonts folder with about 1500+ items is getting copied to a Synology NAS over 1Gbps link.

This feature is going to be available under a Pro license only. More on this to follow.
First stab at the Drive Health Monitor UI for the next Bvckup 2 release.

There's lots of information to be displayed, so the idea is to recycle columns for hosting multiple releated bits of data.

For example, you can toggle between showing drive vendor and its model in the 2nd column, and between disk capacity and its current utilization in the 5th column.

Naturally, none of this is final, but I think for the first pass this is not bad.
A list of storage device vendors as observed by Diskovery to date. I must admit that I haven't even heard of at least 3 of these.

Here's few more notable findings from the aggregated analysis of 200+ logs submitted so far:

  • Top three vendors - Seagate (19%), WD (18%) and Samsung (12%)
  • 219 HDDs and 58 SSDs - 79% and 21% respectively.
  • The only SMART attribute present in all SMART reading was #12 - the power cycle count.
  • All but one device also reported power on hours and reallocated sector count .

    This is good news, because the "reallocated" attribute (#5) is the principal predictor of an imminent drive failure. Once you see it starting to climb up, it's time to change the drive.
  • The temperature attribute (#194) was reported by 71% of HDDs and 51% of SSDs. However some devices report temperature readings in different attributes, so these percentages may be off.
  • Attributes in the 17 to 159 are unused (with an exception of one SSD that reports something #64).
  • Lots and lots undocumented attributes. Some vendors have incomplete specs and others (e.g. Seagate) just don't release SMART specs at all. This means that we can't even label some of the attributes.
  • Lots of inaccuracies in the specs too, meaning that it's hard to reliably interpret the attribute's raw values.

    Raw value is a 6-byte sequence of vendor-specific data, so you *must* have a spec to understand them. And you do want to understand them, because that's where the actually interesting data is.

    Regrettably some specs would say that it's just a 48-bit counter, whereby in reality it'll be a 16-bit something followed by a 32-bit counter. Furthermore, certain vendors (e.g. Kingston) tend to change the format not just between their product lines, but between sibling products that are just a model away from each other. It's really quite nuts.
New release is scheduled for the next week, so stay tuned.
Two years worth of changes from the last version of the beta cycle (April 2014) to the upcoming production release (April 2016).

Just thought I'd share.
Finalized the last change that will go into R75 - full UI support for advanced filtering rules that have been supported internally since last August.

More specifically, the UI can now be used to set up rules for matching on the file size and created/last-modified times. Attribute matching has also been extended to support more exotic values such as Offline and Don't-Index.
Added an option to copy parts of the log to the clipboard - either just a single line or the line and all its children (the block).

If the block is large (over 64K), then the app will also display its size on the right-hand side on the menu. The neat thing is that this information is readily available in the log index, so this requires no log scanning of any kind or no interim memory allocations. Basically, it's as O(1) as it gets.

Will be in R75.
Network Logon configuration has now been reworked to allow configuring a couple of advanced aspects of the share access.

Will be in R75.
Back to Bvckup 2 now and its upcoming Release 75.

With the next release the UI will start adopting a new "progressive refinement" approach for managing advanced and less-frequently used features.

After trying this and that, the design ended up converging to a simple More button at the bottom left corner of the dialog that, when pressed, injects additional settings into the dialog.

This is certainly not something new and revolutionary and it must've been done before, but it does feel a bit more natural compared to a more conventional approach of popping up a separate window with all the advanced extras.

Also, the initial iterations had the More button toggle between "More..." and "Less..." when clicked, but it looked a tad too busy, hence the current revision of a simple "sticky" background highlight when in the expanded state.
Added support for displaying both effective and all supported SATA signaling rates for physical drives. These are also interchangeably knowns as "transfer speeds" and "link rates".

Detecting supported signaling rates is trivial. After all, it is a part of the drive's ATA IDENTIFY block, word 76.

In comparison, drive's effective signaling rate is a bit trickier to determine. In theory, it is also a part of IDENTIFY block, word 77. In practice, this is only available from drives that support ACS-3, which is a more recent revision of the ATA command set.

So for drives that don't indicate their effective signaling rate Diskovery looks at "phy link rate" as reported by CSMI-compliant drive controllers.

But, wait!

What if both rate readings are available and they are different? Not possible, you say? "Ha-ha" says an outdated Intel RST driver that ships with Windows 8.1. In this case Diskovery will default to the drive's version of the value and also issue a warning of the discrepancy.
Diskovery now recognizes commonly used emulated RAM-based storage, like SoftPerfect RAM Disk and Dataram RAMDisk.
The empty space at the bottom of the Diskovery window is now used to show quick status messages. A little opening animation is included.
Diskovery can now also talk to the Virtual Disk Service and extract all sort of interesting information from it, including exact volume type (spanned, striped, mirrored, etc.), its online/offline status and its health.
Diskovery now detects the USB drive type and its effective connection speed. No more USB3 drives sitting on USB2 hubs.
An opening screen for Diskovery, must be read in Jeeves'es voice for full effect. Not long now, almost done tying up all loose ends...
With Diskovery coming together, I wanted to take a moment and show what a UI sketching process for an app looks like.

In a vast majority of cases native UI controls (widgets) work just fine. There's really no point in re-skinning, say, a button unless, of course, your goal is to make it stick out like a sore thumb.

There are however cases when the app needs to display information in a way that is not natively supported by the OS. In other cases, while it might be possible to render the data with a native control, it won't do it well or the result won't look good.

In these cases you have no option but to go with a custom UI, but, luckily, if you start with a need (rather than a quest for bells and whistles) the process is fairly predictable and logical.

Diskovery deals with displaying the bits and pieces of the computer storage stack - from physical drives, to parititions, to volumes, to logical drives and mount points.

These fall naturally into a hierarchy, with each level comprising a list of key-value(s) pairs.

So taking the first stab at the UI, we would quickly arrive at something like this: Primary list of drives and volumes on the left, with details of a selected item on the right, organized into a tree of "key:value" entries.

This is functional, but the right pane is not exactly easy to follow. So the first change will be to tabulate the key-value pairs:
Next, note that items in bold act like section headers, so they don't really need to be collapsible. So we do away with the buttons and connecting lines between top-level items:
This gets us a bit of extra horizontal space and trims a useless, but still actionable UI element.

Next, a couple of tweaks to how we render selected items.

There's no need for any items on the right to be selectable at all, so we won't render a selection highlight in the right pane.

The left pane can use a less jarring selection style, consistent with how Windows itself does it.
Next, we try and improve readability of the data on the right by helping to guide the eye from a key to its value.

The simplest option is the grid lines:
At this point, we take a sip of our cold coffee, squint an eye and assess if we like how it looks.

It's not bad, but we can try something else:
Now, that's not bad either and it has a nice visual consistency with the selection bar in the left pane.

However a minor nit here is that we now have bold items (section headers) paired randomly with row highlights. See how Storage space is on white and Partition table is on light gray?

Moreover, adding or expanding items will cause this pairing to flip-flop, e.g. we add an extra row about Partition table and it now sits on the white background:
That's not a disaster, but we can do better.

The idea is to recognize section headers as such and style them independently:
With this change in place we can just restart the row highlight sequence at the top of each section and all's good.

Next, we apply the same styling to the left pane as it happens to also comprise of sections:
Another sip of coffee, re-squint the eye and the whole thing now looks a bit cluttered and busy.

Easy enough to rectify by adding a bit of vertical padding:
Next, we remove vertical connecting lines leading from section headers to the top-level items.

However, we remove them on the right, but keep them on the left. This is because on the left we have a hierarchy of devices, so connection lines make sense. It also seems to look better this way.
Next, we move the expand/collapse buttons to the right.

In this particular app expandable items contain secondary details that 99% of users will not ever need. These items will also be collapsed by default (even though they are expanded here, but that's just because we are sketching through the details).

This change also helps decreasing the leading pad of top-level items and providing for a more compact horizontal packing of the data.
Alright, so that's about it as far as functional tweaks go. So now it's where it gets subjective. We are going to spruce it up a bit.

It's all about colors, padding and little nuances. If we were designing for the web, it would also involve texturing, but for a desktop app texturing is a sure way to ruin native look and feel.

In any case, moving on - invert the headers from dark-on-light to light-on-dark:
Add a bit of hue to the row highlights, tint their edges a bit and also flatten the look of expand/collapse buttons:
Splice in a scroll bar and make sure our precious design still looks reasonable:
Re-style the left pane to look more like an actual connected hierarchy of disks and volumes. The rationale here is to make it more obvious that items on the left are clickable.
Almost there. Take a final look and adjust vertical padding and spacing to de-clump items a bit more: And here it is. Compare to the starting point and you'd probably agree that there's at least some improvement.

Needless to say, one man's extra vertical padding is another man's complete waste of space, but any customization is always a combination of rational and subjective choices. The former is what gives an app the ease of use and the latter is what give it the personality. Can't have one without the other.
Diskovery is a little companion app that throws a simple UI on top of the new S.M.A.R.T. diagnostics module of Bvckup 2.

The app will go live shortly and its main purpose is to let us estimate how well the diagnostics module fares against the real world hardware. Because there are specs and there's life.

In particular, certain USB drives and older RAID controllers don't always cooperate fully (or at all), and it would help to know what- and how common they are.

Once released, Diskovery will remain publicly available and we'll update and support it. It's a simple tool for quickly inspecting one's storage hardware and I can think of a number of cases where it can come handy.
Having a working backup is great, but knowing if any drives are nearing their lifespan is just as essential.

As you might've heard the next release of Bvckup 2 will be adding support for the drive health monitoring via S.M.A.R.T. diagnostics .

However, first things first, and today I'd like to do a quick review of what's what in the Windows data storage stack.

Read in full here  =>  Drives, Partitions, Volumes
Final remark on general email config improvements in R75.

When connecting to a mail server over SSL/TLS, it's not uncommon for the server to present a certificate that doesn't match its hostname.

In particular, this happens when a custom domain is used with a managed (shared) mail service. You'd grab a domain, set its MX to mx.domain, add CNAME for mx.domain to point at mx.mail-service and - voilà! - you are served with mx.mail-service cert when connecting to mx.domain.

Ultimately, this is a mail server mis-configuration issue - your MX should really be pointing at mx.mail-service. In theory this shouldn't be happening, but in practice it does.

To address this, it's now possible to specify the exact domain the app should be looking for in an SSL certificate:

In the above example, you'd simply set this field to mx.mail-service and that's it.

To make things a bit easier that app now also helps you to pre-populate this field with the actual domain from the server's certificate.

It does that when you send out a test email and the field is left blank. As you can see from an opening screenshot, the app will recognize the certificate mismatch and re-run the test, accepting any and all certificates. It will then make note of the domain in the cert and copy it into the "SSL domain" field for you.

Finally, setting "SSL domain" to * will effectively disable all certificate checks and force the app to accept any certificate that the server sends, even if it's an expired or a self-signed one.
New alert configuration flow - coming up in R75.

GIF is a bit choppy, sorry about that.
R75 is getting an overhaul of the email configuration UI.

Earlier versions had general and alert-specific settings mixed up. For example, "From" doesn't really belong to the alert configuration, while "Include" from General Settings does.

This has been sorted out.

A larger picture here is that the app will be getting additional alert types - e.g. for S.M.A.R.T. diagnostics and backup verification - and this requires a proper structure to support these cleanly.

Also, while at it, the UI got a new framework for handling window transitions between different states.

For example, in the above screencap there's are 5 primary states and 2 intermediate ones. Prior to R75 moving window between these states required quite a bit of hand coding. This is no more, so we can now animate the hell out of any window. Consider yourselves warned :)
Long time, no update.
How about a rant? :)

The year is 2015.

The age of cars trying to drive themselves and rockets trying to land on sea barges. Graphics cards are effortlessly churning photo-realistic images in real-time, mining crypto money and cracking passwords.

And here we have Windows still defiantly refusing to use any of that double-buffering nonsense when re-painting one of its basic common controls. Because who wouldn't want a bit of flickering?

Another iteration over tabbed dialogs, now with a bit of animation.
Tab control for multi-page configuration windows in general, and for the email notification configuration in particular.

Prior to R75 there was one primary email alert, which was sent on backup completion. Now the app is getting other types of alerts, so the UI needs to be adapted to support that.

The idea is that the config window for specific alert type will give access to both the alert details and general email settings that are recycled between all alert types.

So here are several sketches for organizing page index of a multi-page configuration window. Nothing terribly novel here, it's more just a matter of getting styling details right - make sure it's obvious that it's an index and that its items are selectable.

Option Z (right-most) looks OK, but it's not obvious that index items are actionable as they look like static section headers.

Option Y is a bit better, but the arrow actually points at "From" field and you can bet some people will end up in a stupor because of that.

Option X uses standard Windows white-on-blue for selected item, so it's better in the actionability department. But it doesn't look nice.

Option B attempts to fix that.

Option A attempts the same but using app's own styling of selected items, which is already used in several places in the UI.

Thoughts or comments? There is a thread for that!
Minor tweak to the look of the delta copying progress bar - a bit more contrast, both for completed and outstanding parts.

Will be in R74.15
Second pass over backup grouping UI for R75.
I think it's getting there...
First stab at backup grouping UI for the next release (R75).
Thoughts or comments? There is a thread for that!
There's a good guiding principle coming from Paul Graham that says make something people want. It's very simple and it's very effective. It also makes real-world user opinions a critical part of any good product design.

In other words - please take a moment and talk to us! :)
Unicode, as you may know, is a standard that assigns pretty pictograms to numerical codes from the 0 to 0x0010FFFF range.

In simpler cases pictograms represent letters, numbers and symbols. In other cases they are decorative glyph elements, like umlauts, that are meant to be combined with the letters.

There are also Unicode encodings - UTF-8, UTF-16, etc. - these deal with how specific Unicode numerical IDs are translated into sequences of *bytes* for storage, transmission, etc.

The simplest encoding is UTF-32 - it simply stores Unicode ID as a 32-bit number. This is however rather wasteful as at most 21 bit is needed to represent any Unicode ID.

Prior to Windows 2000, Microsoft used a simple 16-bit encoding that mapped every Unicode ID from the 0 to 0x00FFFF range to the exact same 16-bit number. This is called UCS-2 encoding. The Unicode range was smaller back then, so this approach worked.

Then, Unicode range grew and IDs no longer fit into 16 bits. So to address this, UCS-2 was modified to encode symbols from the 0x010000 - 0x10FFFF range using two 16-bit numbers, from the 0xD800 - 0xDFFF range. Exact details of this encoding are not important, but what it meant was that some 16-bit numbers, e.g. 0xD812, were no longer valid Unicode symbols just on their own.

This is called UTF-16 encoding, and this is what Windows uses at the moment.

But wait !

As Apple likes to say after a screw-up - "it turns out" that despite routinely talking about UTF-16 in their API documentation, Windows file system still uses plain 16-bit encoding for the file names!

For example, "\xD812Haha" is a valid file name even though it is *not* a valid UTF-16 string. This in turn means that if the app is converting file names into other encoding, it should go easy on full UTF-16 compliance checks and be ready to digest ill-formed UTF-16 string.

Bvckup 2 keeps all text in UTF-8 format internally, so it does quite a bit of conversion from/to UTF-16 and this leads to some interesting things when the app runs into an malformed UTF-16 name. Like those created by *ahem* Adobe *ahem* After Effects *ahem*, for example.

The solution is simple - try and convert as if we have the UTF-16 encoding, but fallback to UCS-2 when running into invalid UTF-16 sequences. Also, apply the same approach when converting back to UTF-16 and all will be good.

Will be in Release 74.8.

* Kudos to Andrea for reporting this.
Release 74 also implements new delta copying defaults and the results are ... interesting :)

Delta copying, when enabled, has always been used conditionally and the criteria was the size of the file. For smaller files the overhead of setting up the delta copier is larger than any potential savings, so these files are simply copied in full.

The file size threshold was initially set to 128KB and the value was inherited from the original Bvckup version, where it made sense because the I/O was generally slower back then.

Fast forward few years and 128KB is no longer a good default, so as of Release 71 it was increased to 2MB.

With Release 74 the criteria changes again.

First, the threshold goes up to 32MB.

Second, the delta copying will also be used for files between 2MB and 32MB, but only IF they were modified in the last 30 days OR if they were previously delta-copied.

The rationale is pretty simple - we try and not bother with delta copying unless files are large or they change frequently.

This change has two main effects.

1.   Smaller backup state.

Delta copying stores per-block hashes in a backup's configuration directory. This is done for every file that is delta-copied, so the fewer files are, the less data ends up being in the \deltas folder.

The above screenshot shows the \deltas folder stats after backing up \Windows\System32 directory. And here are the same stats after backing up with an older version -
That is, twice the disk usage and nearly 10x more files and folders in the \deltas folder.

2.   Faster overall backups.

For smaller files the overhead of setting up the delta state is comparable to the time needed for the actual copying. We skip the delta setup -> we complete the copying faster. Magic.


That's nearly a 10% speed up. This is obviously specific to the contents of \Windows\System32, but it's not hard to extrapolate from here.
Release 74 will also be adding an option for exporting app's complete configuration as a single .zip file.

Restoring app's config from this backup is a simple matter of shutting down the app and extracting contents of the .zip into %LocalAppData%\Bvckup2\ folder.

But since we can hardly expect anyone to know this location by heart, a short text file called How to restore.txt is also included with each backup zip.
Run-time memory usage is now capped.

Prior to Release 74 app's memory usage was proportional to the number of items in the backup. This was because both the source and destination file trees were kept in memory in their entirety, as was the backup plan. While not a big deal for smaller backups, it does translate into quite a bit of RAM when the file count starts to climb into a million range.

Release 74 implements the off-RAM storage for large data sets. In previous release the run-time arrangement was like this -

With the new release the objects are kept in the app's memory only if they are in active use. The rest is trickled down into a swap file for storage, through a couple of intermediate caches.

The object cache holds fully assembled instances of application data objects and its size is kept at or below preset item count.

When the object cache grows too big, least recently used objects are written down to the swap file. The writes are passed through a typical page cache to coalesce the I/O into larger chunks.

This lunch is obviously not free.

There's a cost to even successful cache lookups and there's most certainly an expense to doing the disk I/O.

The app mitigates this by using aggressive cache optimization (buckets, MRU lookups, etc), raw IO caching and also by allowing keeping swap files in a custom location, and presumably on a different physical drive. But, again, all this starts to really matter only when backups grow into a million item range.

While reasonable defaults are provided, the cache sizes, the trimming policy and other details are fully configurable through the .ini files. The app may also be set up to log cache/swap performance if required.

This change is big, it took nearly two months to implement and it means that Bvckup 2 can now handle backups of unlimited size. To say that it's a very big deal would be an understatement.
Still working on major engine changes to cap its run-time use (and I will be posting data from the test runs shortly), but meanwhile here's something simple that I'm considering adding to the next release.

It's an option for the app to report your backup statistics to one of our servers and then being able to display these stats as a simple PNG served over HTTP from the same stats server.

It's really meant for trying to motivate other people to back up their things as well as to indirectly promote the app as well. And, yes, I lived through the 90s, that's where I got the idea from :)
Currently working on the improvements to the scanning and planning modules of the app. No pretty UI screenshots to show, just some performance numbers.

The ultimate goal of this rework is to put a cap on the run-time memory usage, which in turn is needed to better accommodate very large backups (tens of millions files and up).

The app comes with a formal backup planner, which is a module that accepts two full directory trees (one for the source location and another for the backup one), digests them and produces a list of simple steps that, when executed in order, bring one tree in sync with another. The app is then diligently goes through the list and creates, updates, renames and deletes files and folders as directed.

An alternative to formal planning is to to traverse the trees and make on-disk changes just as you spot the differences. There are several drawback to this approach, including not being able to estimate the amount of work needed, to understand disk space requirements and to detect file/folder moves. It's just an altogether messier and less predictable way to go about the whole thing.

That's how Bvckup 1 worked and that's how robocopy works, for example.

Long story short, having a backup planner is the right thing to do. However, it means that the app needs to have two full file trees readily available for the planner to digest. If these trees are kept entirely in memory, it may get expensive with large backups. At 100 to 300 bytes per item, a million file backup means 200 to 600 megs of RAM just for the trees.

That's quite a bit, but more importantly this memory usage grows linearly with the size of the backup. This is no good.

The solution obviously enough is to not keep full trees in memory. This is a fairly complicated endeavour and it maps onto several large changes to the app inner structure.

One such change is nearly done. Both the scanning planning modules no longer require for the trees to be resident in memory at all times. Instead, they now use them in a piecemeal fashion.

Behind the scenes, trees still reside in memory for the time being, but the underlying rework has some nice side effects. Trees now take less memory, they are faster to populate, faster to save and load to/from a file, they take substantally less space on disk and they compress better with NTFS compression.

This will ship as a part of R74.

After that, R75 will replace in-memory trees with version that can swap out tree parts to the disk, significantly reducint the run-time memory footprint of the scanning/planning process.
Release 73 adds copying speed and completion time estimation for longer file copies. To toggle between the views just click on the text next to the progress bar.

The estimation is calculated in three steps.

First, the app looks at effective copying speed and uses sliding window to average it across last 25 samples taken 650 ms apart.

Second, it uses this average speed and the remaining byte count to approximate the completion time.

Finally, the completion time is averaged with its own sliding window, but using only 4 past samples.

This double-averaging helps mitigating the effects of sudden short-lived I/O bursts and hiccups on the estimation.

The parameters - 25, 650 and 4 - are configurable through the engine .ini file, so the whole thing is open for tuning if needs be.
The initial revision of the auto-update opton is in.

It's a simple extension of existing "check for updates" option that skips showing the "new version found" prompt and just goes straight to the updating part.

There are two caveats, both applicable only when the app is running in service mode and both stemming from the fact the updates are driven by the UI.

First is that closing the UI will effectively disable auto-updates. Second - if the UI is running without admin privileges, running the installer will trigger the UAC prompt.

Both will be resolved shortly by moving all update logic into the engine module. Meanwhile...

The auto-update option will be hidden by default unless the Preferences window is opened with the Control button held down .
Release 73 will also be adding support for monthly backups. It will now be possible to set a backup to run on the N-th of every M-th month.

You surely are thinking "but 31st!", "February!" or perhaps even "every 6 months staring on Jan 31st!"

All fair points. Fear not, the UI got this covered:
A rework of the loggging module, coming up in Release 73.

As you may or may not know, Bvckup 2 uses a dual-module architecture of the inside.

One module is the backup engine. It maintains the list of backup jobs, schedules, executes backups and generally does other useful things that don't require direct user involvement.

The second module is the user interface. This is the part that displays the app's window and allows users to monitor and control whatever the engine module is doing.

The principal point is that the engine and the UI are isolated from each other. They share no run-time data and communicate with each other using a messaging protocol.

This isolation is what enables the app to run in service mode. In this mode the engine runs in its own process, as a system service, and the UI runs as a regular desktop app.

It was all well and beautiful, but there was, however, a caveat.

The caveat was the backup logs.

Logs are inherently shared data - they are written by the engine and they are read and displayed by the UI. Logs tend to be bulky, so it is shared app data of a special kind and it requires special handling.

Prior to R73 the app included a third module (in its dual-module architecture, no less) - the log manager. It was a complicated contraption that abstracted log access for the engine and the UI, and the biggest issue with it was that it behaved completely differently depending on which mode the app was in.

In desktop mode, the manager was writing *and* indexing logs on behalf of the engine and the UI was merely displaying them. However, in service mode, the engine did no indexing, just the writing, while the UI was making copies of all logs for its own use and then it was indexing *them*.

This is exactly the kind of non-uniform conditional processing that the dual-module design was meant to avoid!

Enter R73.

The separate logging module is no more. The engine no longer does any log indexing, it is now entirely the UI's reponsibility. Indexes can now be rebuilt on demand at any time, and finally, the UI no longer creates its own copies of the log files. Instead, it works with the engine to gain access to the engine's copy of the logs, using DuplicateHandle to move open file handles across the process boundary when needed.

The UI now can display logs while it is still catching up with them.

Think -- running in service mode and launching the UI only once a week. On each launch the UI needs to index a week's worth of log entries it hasn't seen before. Prior to R73, the UI would just sit there and show "synching ... XX% done". Now the sync is done in the background (as per the screenshot).

The UI now can also recover gracefully from indexes getting corrupted (e.g. due to a cold power cycle or blue screen).

Just as important, the underlying code is now much simpler, lighter and more uniform, the executable is 8KB smaller and the developer is generally happier. Good news all around.
Running an extended set of tests to fine-tune the performance of the Bvckup 2 bulk copier.

It's an inherently simple matter of determining an optimal size and count of read/write buffers used to ferry data from source to destination. There are however ... erm ... details.

A source can be local or it can be over-the-network. There are slow local devices (e.g. a USB2 drive) and there are fast local devices. There is a warm, there is a cold cache and there are files that are cached only partially.

The writes can go out in lots of smaller chunks, all in parallel, or with a smaller number of large buffers. Again, it can go to a local device or over the network. The app can also explicitly flush all buffers after copying or leave flushing to the system.

The above graph is the I/O throughput of a 0.5 GB file being repeatedly copied from the local HDD to Synology Diskstation over a Gigabit wire. What changes between the runs is the buffering strategy and as you may notice there is a difference.

Still need to run several over-nighters to gather more statistics, but there is already a couple of things that can be changed in the copying code to make it run faster.

Scheduled to appear in the Release 72, coming up shortly.
"Send test email..." is now done.
R71 adds an option of automatically enabling NTFS compression for the backup logs.

Based on what I've seen so far, both .log and .tdb files compress to about 20-30% of their raw sizes.
Reworked part of the More Options page in the backup configuration window.

* The animation here is a bit jerky, but it's just a GIF encoding artefact.
Email template editor. "Show variables" opens up another part that lists all variables that can be used in the form fields.

This is getting a bit too complex for my taste, but it's the part of the UI with little foot traffic, so it's probably OK.
A little primer on the mail server configuration.
Upon even further consideration it looks like the From field is not of a prime interest either, so it too can go into the Details.

This cuts down the number of fields and renders section headers unnecessary, so they too can go.

Basically the idea is to reduce the dialog down to fields that actually need user's input and stash everything else away... I don't know if it's the right thing to do though. Perhaps having a summary of the whole email setup (including the subject, From and the message) is not such a bad thing. Not sure. Need to sleep on it.
Upon further consideration, decided to remove both Subject and Message off this dialog to keep it simple(r) and to the point.
First stab at the email configuration window. Too many options, too little space. As usual.

EHLO there!

Release 71 is getting email notifications, so here's all you always wanted to know about teaching your app speak SMTP.

First of all, let's get MAPI non-sense out of the way. As per usual, it sounds good in theory, but an awful mess in practice. So, no, MAPI is not an option unless your goal is to burn through your support budget or to practice your communication skills with the help of less-than-happy users.

This leaves us with raw SMTP.

SMTP is an inherently simple protocol - send a text command, receive a text response. Repeat. Open a socket, connect, send(), recv(), done. A couple of hours of work at most. Ha.

It starts to get complicated when you throw TLS in a mix to secure the communication with the server. In particular, this is required when you want to send emails in a way that requires logging into a server first.

There are two ways in which SMTP can be combined with TLS.

First is when you take an existing SMTP session and say STARTTLS. The server says OK and then you basically stop talking SMTP over your TCP connection and start talking TLS. Then, once the TLS handshake completes, you go back to SMTP, but now it all goes over the TLS session, nice and encrypted. This is a standard way, that's what every decent SMTP client should do. This is TCP/25 and TCP/587.

Second option is SMTPS. You first establish a TLS session and then start talking SMTP over it. Just like HTTPS. Apparently, this is non-standard and discouraged way, but still widely used. This is TCP/465.

Then, there's a matter of how to add TLS support to the app.

The obvious choice is OpenSSL, but that will cost you 200-300KB in extra binary code, so unless you already have OpenSSL linked in, it's not really an option, not for a *backup* app.

Digging through MSDN you may come across something called SSPI or Security Support Provider Interface. It is an attempt to stuff several security protocols like SSL and Kerberos into a single framework in a faint hope for a miracle of reuse. One of SSPI providers is a mysterious entity called "Secure channel", which, upon closer inspection, turns out to be ... yes, TLS!

So, the next step is try and work through all the abstraction fluff of SSPI to make it actually do the TLS handshake. The idea is actually quite neat - you can create a stateful object (context) that takes in data you get from the peer and spits out some data for the peer. At some point it completes the handshake and becomes available for accepting app's own data, which it encrypts and hands you a blob to send to the peer. You would then also get blobs from the peer, pass it to this context/object and get back plaintext data for the app. Whether it's TLS or ROT13 on the back is completely transparent to the app.

That's the idea. In reality, the documentation is subpar and lacking and there are exactly two examples - one is incomplete and another one is buggy.

The good news is though is that once this bloody contraption gets off the ground, you get full TLS support at the cost of just several kilobytes.

With TLS done, there's still a bunch of smaller things to be taken care of, e.g. lack of asynchronous DNS query API in pre-W8 windows. Not a big deal, nothing a spawning of a worker thread would solve, but still ... a bit here, a bit there and it all adds up.

In other words, it's all coming together, but as per usual it all takes longer than anticipated. Shocking, I know :)
Finalized the UX for auto-configuration of the device tracking. It's essentially the same as animation immediately below except for a brief flash of blue when tracking config is touched by the app.
Another subtle UI change in Release 71.

When configuring or reconfiguring a backup, the app would automatically enable/disable device tracking when a location is changed to use a different drive.

Since this change happens behind the scenes, the app will attract the attention to the fact with this little animation. When hovered over, the tooltip for the pushpin button will explain what's up and suggest reviewing the changes.

My only concern is that this still might be a bit too subtle, but it's better than what's in there now... which is no notification at all.
New UI for configuring device tracking in R71.

This is one of those changes that look almost trivial when done, but behind the scenes take a disproportionate amount of time to actually flesh out.

In this case, the main problem was that the feature needs a detailed explanation and the question, of course, was how one can fit a bunch of text into a reasonably small space so not to have this text take over the actual configuration section.

The first iteration was the good old message box. You click on the "More on this..." and you get :

Nice, but looks like a hackjob, does't it? So, naturally, the next step was to spruce it up a bit - put text into a dedicated window, use RTF to add a bit of in-text formatting and a bit of animation to liven things up :

Nice again, looks better, but feels off. Too much going on. It looks like a conventional help window, but ... crippled. It's also not clear how to act when the user wants to resize the window - allow it, not allow it, restict minimum/maximum size, etc.

In short, what's lacking here is the cohesion. Luckily, the solution is right on the surface:

... whereby the text panel slides out on the "More info" click.

This is better, but now we have a window that sits lopsided on the desktop when the panel is open. It is also becoming wide, meaning that the app needs to mind the desktop edges, re-center the window, etc.

But if we just sleep on it, we might suddenly decide to make it an "either-or" window and then the UI magically becomes
(a) compact
(b) cohesive
(c) balanced
(d) with unlimited text space!

So there you have it - a new mini-help subsystem coming to Bvckup 2 in the Release 71, deployed on as-needed basis across the app.

Don't know about you, but I think it's neat, tidy and pretty damn clever :)
Added support for the reseller licensing to the app. Any license can now be tagged as being sold through a distributor, in which case Help/About will show distributor's name and link to its site.

Additionally, Tech Support and Homepage links from the Help menu may be configured to go to the distributor's pages.

It may not look like much, but it's the most important non-technical feature to be added to the app to date.

More on this to follow...
Another option, as per this comment. It's a really nice concept, but it requires a squint to realize what the icon is. Highly-detailed 16x16 px icons are hard to pull off.
Release 71 is going to reshuffle the contents of the backup configuration window a bit and location fields are getting equipped with a "device tracking" icon as a result.

In other words, for a network location the app will display the "key" icon (see /wip entry immediately below) and for local paths it will show "tracking" icon. Both fronting respective dialog for configuring the details.

The main question, of course, is what the icon should be ...
In a process of adding a way to configure network logon credentials for remote locations. Will be in Release 70.
Release 69 also adds an option of simulated backups or, as they are also known, "dry runs".

The option is accessible via the right-click context menu and, when invoked, it puts the backup through its preparation phase, including snapshot creation, scanning and planning steps, and then dumps compiled backup plan to the log without executing it.

Simulation also executes the archive pruning without making any actual changes to the destination.
Release 69 adds support for optional VSS writers.

VSS writers are the components of the shadow copying framework that are tasked with ensuring consistency of specific files in VSS snapshots. For example, when creating a snapshot, Windows would ask the MSSQL writer to flush any pending database changes in a way that would make on-disk copy of the database self-consistent.

Some of the writers are required and always participate in the snapshot creation process, but some are optional and they need to be explicitly enabled by the initiating app. Starting with R69 Bvckup will be enabling all optional writers and it will also provide a mechanism for excluding specific ones if needed.
It's shipped!
Here it is
Nearly done with the checkout flow on the website. Integrated with Stripe yesterday, now finalizing the visuals.

This is a purchase receipt. It is both sent out and shown on the site when the payment goes through.

PS. All numbers and codes are made up. Don't get any ideas :)
Working on the production version of the website.

Cleaned up overall look and feel a bit, got rid of baby blue theme and anorexic body font, reworked the copy, expanded Features section and so on. Still need to slice and code it. Also asked a couple of people for the feedback, so waiting on that too.
The feature everyone was so waiting for ... the payment form :)

Working through the necessary evil - the app licensing :)

The plan is to have a 14 day trial that doesn't require any explicit licensing. Just install and run. It will pop up a notification half way through the trial period and it will also say something on the last day.

Once the trial period is over and until a production (paid) license is installed, the app will go into an unlicensed mode and it will activate two restrictions (see screenshot). It will remain largely functional, but presumably not convenient enough for regular or unattended use.

You would then hit /purchase page on the website, go through the purchase process and get an activation code in return. The code will then need to be sticked into the activation form on the client (see 2nd screenshot) and it will fetch the license from the server.

If the licensing server is inaccessible for whatever reason, there's a couple of ways to retrieve the license manually (see last screenshot).

For those with Beta licenses, they will be accepted at the checkout as a discount code.

For those looking at large deployments / enterprise setups, there will be an option for automating licensing process. Get in touch for details if you have questions.
Updated the About window a bit. Current design has somehow always felt ... cramped?
Super-final version of the progress bar as seen in Beta 60. Note how colors of some parts are inverted under selection, took me disturbingly long to arrive at this solution :-/
Beta 60   - vs -   Beta 59
Finalized changes to the progress bar - now with more contrast, a bit of depth and beautifully centered splatter for partially copied blocks.

And here is a bunch of runners-up if anyone's interested.
Just waiting for the test build to compile... so here's another take on the delta copier progress bar (and here's existing version for comparison).
Clicking and holding error navigation arrow for 750 ms now activates auto-scrolling. It starts at 10 errors per second and accelerates linearly to 100 err/sec, which is pretty damn fast.

Additionally, holding Ctrl while scrolling with the mouse wheel makes it go in 10 line increments.
Beta 60   - vs -   Beta 59
It is now possible to clear the log view and then revert it back to the full log view.
A smaller UI change.

When hopping through errors in a backup log, log viewer will now briefly highlight next entry if it cannot be pulled directly under the cursor. This basically adds a visual terminator to the interaction sequence, which is a surprsingly useful thing to have.

Otherwise it works as before.
Having thought about it a bit more, I can even remove that slight upward slope from the hump of the blue graph. The usage still climbs just a little bit during the run, but for practical intents and purposes it's flat.
Wrapping up changes to the logging system. Here's a quick graph to demonstrate the effects of the rework. For an initial full run of a 100,000 file backup memory usage is down from about 200 to 50 Megs. For a 1,000,000 file backup the effect is similar - peak memory usage dropped from 2.9GB to 0.58GB .

Previously, the UI would keep logs of last few backup runs in the memory and for larger backups this would eat up a good chunk of memory. This was especially true for the initial backups where all files needed copying and thus for 100K files it would generate close to half a million log entries.

For subsequent backups it wasn't a big deal as they would handle just the changes and generate only a handful of steps and a bit of log data So in the end, the memory usage was capped, but the cap was higher than it could've been.

Beta 60 addresses that.

Only a very small part of the log is now kept in memory, just enough to fill the log viewer panel in the UI. The rest is kept on the disk and read from there on as-needed basis.

However this is easier said than done.

Bvckup logs are hierarchical, meaning that they are generic trees with nodes that may have an arbitrary large number of children. Running a 1 million file backup would yield a "Processing..." entry with 1 million children, each with up to a dozen of descendants.

Displaying this tree would've not been a big deal if all log entries were always visible at all times. When the UI would've wanted to show entries 700,503 through 700,523 entries, it would've simply gone to a respective location in the log index file and plucked the data out. But...

The nodes can be collapsed and expanded. This hides and shows arbitrary chunks of the log tree and then the UI has this issue of finding 700,503-th *visible* item. And this is hard.

This implies quick look-ups in some sort of super-imposed binary search tree of visible items. A tree that changes every time a node is expanded or collapsed and that sits in a disk file, which restricts how you can walk and modify it without getting an I/O penalty.

This is fun. This is what cost me 3 weeks of shuffling through CS books and several complete rewrites of the tree indexing code. Believe it or not, but this is basically an original research and it seems that no one had to deal with this issue before. At least not publicly.

The solution is to use slightly customized version of an AVL tree to store the list of node's children (that's using tree within a tree), use it for the visible item look-ups and then optimize the hell out of the index I/O through aggressive write caching.

When all i's are dotted and t's crossed, this yields a logging system that can push up to 1 mil entries per second on a resonably equipped box. This might not be knock-your-socks-off impressive, but it's plenty enough for the task at hand.
A quick update on what's happening.

Disk-based virtual data structures, that's what is :)

This is a part of an effort to further reduce and cap app's run-time memory usage. More specifically, it is meant to help the app cope gracefully with multi-million file backups that involve millions of steps and generate lots and lots of log data.

At the moment Bvckup's UI keeps backup logs in memory. It puts an effort to trim them to a reasonable size, but this still eats quite a bit of RAM, especially with large backups.

It costs about 120 bytes plus the size of an entry to keep a single log line in memory, whereby these 120 bytes help weave entries into a hierarchy (a tree) and track their state.

If we want to reduce the memory usage, then the most obvious optimization would be to keep the actual entry text in a disk file and load it on demand, when it's actually visible in the log viewer.

This certainly helps, but as the log grows, the memory usage still climbs. A million file backup generates about 4 million log entries. At 120 bytes per entry that's 0.5 Gig in just supporting structures. This is unacceptable.

So it means that the entire tree structure of the log needs to sit in a disk file and to be read from there as needed.

Disk-based tree structures are routinely used in databases and file systems to store indecies of data sets, typically in a form of a B-Tree and its variations. The kicker is that these structures are meant for storing sorted data and they optimize for searching. Trying to adapt them for storing unsorted data would be nothing short of fitting a square peg in a round hole.

Long story short - there are no ready-made code for storing and manipulating generic data trees on a disk, leave alone fast code. This appears to be an esoteric problem that everyone solves on their own.

The screenshot above shows a part of a small "tree database" library that I ended up writing, together with its own caching module, predictive page loader and a pony.

Took almost two weeks.

On a bright side though, this code is perfectly reusable for storing disk snapshots as well and this paves way for making the planner module work almost entirely off the memory.

Once done, Bvckup memory usage will NOT depend on the particulars of a backup at all - this a very big deal and an incredibly improtant feature to have for any robust backup software.

So, stay tuned...
Coming up in beta 59 - automatic archive prunning.
If all goes well, it'll be out tomorrow.
Effects of delta copying as seen in Task Manager.
1.28 TB read, 0.37 TB written out - just the changes!
Ready to go. Launching tomorrow.
Coming up shortly, stay tuned.
Switched delta copying progress bar from varying color of the partially copied blocks to varying their height. This should help with telling apart skipped, partial and fully-copied blocks.
Added an option for cloning an existing backup job.
Website footer work.
In preparation for the launch of the Referral program.


This one - the *nix implementation of Windows Networking. If you have a NAS box the chances are that it's using Samba to support network access from Windows computers.

It is a good software, simpler and much snappier than its native Windows counterpart, but in case of NAS devices it typically runs on Linux. This means that the underlying file system is anything but NTFS. This in turn translates into no support for "created" timestamps and no support for certain attributes, including System and Hidden.

Moreover, the attributes that cannot be set or cleared depend on whether it's a dot file or a regular file. Who would've thought? :-|

From what I understand it *is* possible to set things up so that Samba would record these timestamps and attributes and make things transparent to its Windows clients, but in reality this doesn't seem to be happening.

Long story short - starting with Beta 49 Bvckup will now be explicitly testing destination file system to check whether c-times are supported and which attributes can be set.

Additionally, it will also explicitly test both source and destination file systems to determine the granularity of c-times and m-times. This is again thanks to some NAS devices that are customized to use 1 sec for m-times and 1 usec (micro) for c-times.
By popular request.

When ticked, Minimize button minimizes to the taskbar (as usual) and Close, the (x), minimizes to the systray.

When unticked, Minimize minimizes to the systray and Close exits the app.

Trivial in retrospect, but took me ages to realize that behavior of Close and Minimize can be controlled with just one setting.
Have a mapped drive? Want to run as Admin?
Good luck with that!
Second pass over the same sketch. Trimmed down decorations a bit, reworked content to have a summary, added footer, etc.

Full-length version is here.

What do you think? Let me know - @apankrat or over an email.
Started sketching production version of the website. As much as I personally like the minimalist version that's currently up there, it just doesn't seem to work that well :-/
When periodic backup misses its last scheduled run, it's now possible to tell the app how to handle it.

There are two common cases - (a) the app itself wasn't running and (b) the backup drive is a removable and it wasn't plugged in.

With certain setups it makes sense to run the backup as soon as the app can, to "catch up" with any changes and get them backed up quickly.

In other cases, a backup may be large and heavy and set up to run every midnight so not to disturb anyone. Running it ASAP will generally be not a very good idea.

Hence this option.
By popular request.
Context hints for each section in More options.
Here's the screen-by-screen flow of the licensing process.

It starts with the above window and directs users to the website to pick and purchase a license code, which is given on spot and also sent in the receipt email.

Then copy-paste the code, click on Verify, get this -

If this works (the server is accessible, the license code is valid, etc), then this is it. Thank you very much, click Ok, you are done -
Otherwise, it explains the failure and offers either to retry the activation or to use an alternative -
There are two alternatives, both are simple -
For offline boxes and locked-down setups there's the Copy To Clipboard option. Take the URL to a networked machine, open it in a browser, get a licensing file back and then drop it into the app's config folder.

The URL is exactly the same as the one used by the app's own licensing code and it comprises of the license code and installation ID and the fingerprint .

Prepare the email option creates a new email with default email client and puts the Installation ID, Fingerprint and License Code in the message along with some pleasantries. The license file will be attached to the reply and it will simply need to be dropped into %LocalAppData%\Bvckup2\engine.
A bit of UI animation for the license activation window. This is a animated GIF, so the colors and the timing are all off. It looks tidier and faster in reality.
Licensing support. Yay!

This is one of those things that is rather simple conceptually, but translates into a lot of actual petty work.

When the app is started, it looks at several system properties including the CPU model, Windows' MachineGuid value and others, hashes them together to produce an "Installation ID". For example -


The ID stays the same under normal computer use, but it changes with hardware upgrades or when the OS is reinstalled.

The app also generates an "Installation fingerprint" by hashing each of the properties separately and stringing them together. This yields a longer hash -


Like ID, the fingerprint changes with hardware upgrades and OS reinstalls, but, unlike ID, it changes only in parts. This makes it suitable for detecting incremental changes to the user's computer.

Both ID and Fingerprint identify user's installation, the difference is that the ID is used on the app side and the Fingerprint - on the licensing server side.

Next, ID and fingerprint are submitted to the licensing server, either by the app itself or by a human through a website. When licensing for production use (as opposed to beta or evaluation), the request also includes a license code - a random token that is issued to the user in exchange for payment.

The server responds with a license, which is a short text file that binds together few pieces of information and seals them with an RSA signature of the licensing server.

License code: EVAL-8028-8316-0400
Installation: $c!f2ZoPxOlnpLWraJZ3F1nm7JcA
Expires on:   20/12/2013


For beta and evaluation licenses the "license code" is generated automatically by the licensing server. The large block of gibberish at the bottom of the license is the RSA signature.

When the app starts, it loads the license and verifies it using public key of the licensing server that is embedded in the app.

If the license is valid, the app works as normal. If the license is missing or expired or doesn't match the installation ID, the app disables certain features and periodically nags to do the right thing and what not.

Now, let say I upgraded my box. This screws up the Installation ID and invalidates the license. The app recomputes new ID and Fingerprint and sends them to the licensing server.

This is where Fingerprint comes into play. The server takes old and new fingerprints, chops them into individual hashes and does a "fuzzy" match. If enough hashes match, we can assume an incremental software or hardware change on user's end and re-issue the license for new Installation ID.

If fingerprints are wildly different, then we make fuss and require contacting human tech support.

In conclusion, needless to say that the whole scheme relies on the validation code in the app being trustworthy. If someone pries the app open and replaces selected IFs with NOPs, then obviously no licensing will work.

The answer to that is that the app should implement self-consistency checks and deploy counter-measures when these checks fail. This is a separate and very large and interesting subject ... which I have no plans on covering here :-) A good starting point here would be archives and then follow from there.
Reworking About window a bit to include license information and associated trinkets (not shown). This will probably need another iteration or two...
Meet the log error navigation.

Basically the idea is to show back/forth arrows only on hover and make them pull next error to the spot where the cursor is. So to hop through errors you would just go click-click-click without moving the mouse.

This compliments well existing feature of jumping to the first error of the last run by clicking on "Xxx errors" in backup's main window entry.

As per previous post, this leaves a question of how one would get to the *nearest* error if none is currently visible and the answer is that one would simply use conventional scroll for that.

All in all, having played with this for a bit I think it's a good solution. If it doesn't seem like one to you, give it a try and it will grow on you.
Tying up few loose ends in the UI.

The above screenshot is the first option for navigating between the errors in the log pane -- see two red arrows next to the "x" in the pane header. The second option is this -

This time the next/previous arrows are in the error entry itself.

I like the second option better, because it's contextual and I think it's more obvious. However, first option is superior in functionality because it allows jumping to next or previous error when there's currently no error visible in the pane.

Still deciding...
Tweaked the progress indicator a bit.

For one, replaced the fixed-width byte countdown with a simple "% out of [size]". The countdown didn't work out, it was just... odd.

Also made the % adjust the number of shown decimal digits depending on how fast the copying is going.

For fast copies there are no decimals. For slower ones it shows one, for slower yet - two, and for the slowest (like the one on the screenshot) it goes down to 3 decimal places.

The idea, obviously enough, to always show some movement, but try and avoid being too excessive with it.
Added code to replicate folder timestamps. It's ON by default as it takes only several seconds to go through a hundred thousand folders, but it can be disabled if needed
Final version of the update window. De-emphasized the "View changes" button and tweaked the colors a bit.
Reworked the software update mechanism a little.

First, when the update server is processing an update check from the program, it now takes into account program's current version.

It uses this version to compile a consolidated list of all changes between program's version and the latest version available. It then uses this list to understand if there are any critical or important outstanding changes. This yields an importance indicator of the update, when one is available.

Secondly, both the list and the indicator are now sent back to the program.

On program's end, the indicator translates into the "Update is recommended" line. It is one three options with "optional" and "critical" being the other two.

"Recommended" is a routine update that adds this and fixes that. It would better be installed, but it's not the end of the world if it's not.

"Critical" is critical - "you'd be sorry if you don't install it" kind of update.

An example of "optional" update is a new version that updates program's French translation when the user is running an English version, i.e. an update that can be ignored altogether.

There's now also the "View changes" button that expands the window and shows a neatly formatted (RTF) version of the change list.

The biggest deal, of course, is that it's a consolidated list of all changes between the installed and the latest version, rather than a laundry list of all changes since the dawn of time. It makes the list infinitely more useful compared to the raw version.
Adding support for running external command before and after each backup.

Both commands have a configurable timeout value. Additionally, the pre-backup command can be tagged as critical, in which case bvckup checks the exit code and aborts the backup if the code doesn't match specified value.

The screenshot shows touching a TrueCrypt volume before the backup. This is needed, because TrueCrypt goes out of its way to preserve timestamps on its container files and bvckup sees such files as unmodified.
Reworked the More Options section.

It has already grown quite tall and it still needs to take on few more configuration options, so expanding it in place as it's done for all other sections doesn't work. The window gets too big and simply doesn't fit on a desktop.

Instead, it pushes everything up and out of the window and floods it with its own content. As I add more config options to the section, it will also expand the window down a bit if needed. All this should help keep all configuration within single window and also keep window's height in check.

Took me a while to converge to this solution, but it is a pretty decent way to structure it I think.
Second revision of the same thing.
Sketching up a dedicated device tracking section. Not sure if it's worth it, still have to try few things out...
Coming up - the reordering of list items.
Done with the inclusion/exclusion filters

The tree selector widget is a piece of art even though it doesn't look like one :) It is basically a front of an asynchronous parallel disk scanner that dynamically rearranges its scanning queue to first scan subdirectories that a user might be accessing next.

In other words, if you expand a directory, then the scanner will postpone scanning anything else and focus instead on all its subdirectories. So if you are making your way through a massively populated disk, it will try to pre-load what you are looking at and scan the rest on a lower priority.

Also, since the widget starts showing tree contents right away, it allows quickly navigating to and excluding a desired item without needing to wait for the entire scan to complete. Again, quite a bit of speed up with huge trees.
Advanced filter configuration. The pattern matching is all brand new and much improved over the disaster that it was in Bvckup 1. It also adds an option of matching on item attributes, both set and cleared.
Finishing up the dialog for configuring inclusion/exclusion filters.

In inclusive mode (pictured) the tree view tags excluded items with 'x'. In exclusive mode, when "Start with an empty list" selected, the tree view tags included items with checkmarks.

Currently leaning towards Option 1, and not yet decided if to strike-out excluded items or not.
Added per-phase progress bars. Currently used for scanning and execution phases.
Before/After of the backup config window as of Beta Release 20.
Revising the backup config dialog to accommodate a description field.
Revising the main view. Description and the progress bars are back in.
Well into the beta, just dotting the i's and crossing the t's.

One of the outstanding features is the archival of deleted items and this is the supporting UI for it.

Now deciding between a compact format of the config widget or a verbose one. Compact format has a single descriptive field that is re-used to show the explanation of a currently selected option. Verbose format has descriptions for each option visible at all times - it is likely easier on first time users, but it is also a waste of screen space on all subsequent uses.
What can I say?

Beta. Is. Good. To. Go!!!
A quick fix to the UI.

For very large files (say, 256 Gigs), there was a clear lack of UI acknowledgement of the backup progress. So the completed percentage got a couple of point digits and a byte countdown for the file.
Trying to keep it short and sensible.
Added the first draft of the Miscellanea section. Also renamed More Options button to Rabbit Hole, because that's essentially where it leads to :)
The new and much improved delta copier is in.
( the original version is here )

Significantly improved performance through multi-core, parallel hashing, asynchronous I/O and more aggressive caching of the state data.

Strengthened the change detection algorithm through addition of the whole file hash. When all individual block hashes come back unchanged, the algorithm now uses the file hash to confirm that the file has indeed remained unchanged.

Combined with the original use of two separate hashes per data block, this eliminates the need for a precautionary full-file sync every N copies.

Improved handling of cancelled and aborted delta copies by introducing support for partial updates .

The algorithm now remembers how far along the file it managed to proceed and stashes this information for the next run.

When the copying is cancelled partway through, the file is marked as out-of-sync, but the delta state is updated to capture what has been already done. This comes very handy and speeds things up when updating very large files.

Improved handling of large file counts.

Version 1 maintained an in-memory index of all delta copied files and, surprisingly, this didn't scale well. It also kept all delta state information in a single folder, which too led to some pain and suffering when the file count grew large.

New version eliminates both these problems.

Lastly, all copier parameters - block sizes, hash algorithms, cache buffer sizes, threading parameters, etc. - are now configurable on a per-job basis.

All in all - THIS IS IT.

This is how a simple and efficient delta copying should be done. Can't wait to release it to the real world.
Shadow copying is in, for those pesky locked Outlook mailboxes and other "in use" files.

There are three big things left to weave into the program - file system monitoring, new delta copier and software updates - and the beta is good to go. A matter of few days now.
If your message box includes the "Don't ask again" option, just make sure to disable all but one button, when it checked.

Oddly enough, I don't know a single app that does that.
A system tray icon sprite sheet.

Feeling creative? Make your own version, save as a PNG, drop it in a config directory and - voilà - your systray looks just like you want it to.

There are two sets of icons on the sprite sheet. The first three columns is the XP style, which is a bit puffy and matches well older Windows styling. The last three columns is a flat version that goes better with native W7 tray icon style.

Each icon comes in 3 sizes - 16x16, 20x20, 24x24. This is needed to accommodate 3 common DPI levels, see this earlier entry for details.
"Message pending" indication, whereby a message can be either an error, a warning or a notice and these are differentiated with the ! color.
I realize it's not color-blind friendly, but the goal is to attract the attention and make user bring the app window up to read the message --(therefore)--> reflecting the message type via an icon level is not exactly critical.
W7-ish version as per a this hint on Twitter. Not bad at all.
Working on systray icons...
Revised Preferences window a bit.

Added basic control over backup scheduling and switched from "close to systray" to "minimize to systray" - the (x) at the top right corner will now always act as an Exit, just as it is supposed to.
More of the UI work.

Custom pop-out button that stays nice, flat and out of the way when idle, but acts like a regular button otherwise. Not exactly a UI innovation, but it was nice to be able to use it in a real project :)
Took out the engine mode switch from the Preferences and into a dialog of its own. Still need to understand if it's worth mentioning that there's an engine per se or just stick to more amorphous "application mode"...
Looked at few options for splitter styling. First one is final. Tentatively though... heh.
Done with the split-view mode for the main window. It's one of those things that everyone takes for granted and that, in reality, maps to quite a bit of work if done right.

For example, resizing window by the top edge should resize the top pane and leave the bottom part alone.
Likewise, resizing by the bottom edge should expand/compress the bottom pane, but not the top one.

However, if we are compressing window by the top edge and the top pane is at its minimal height, then we should the bottom pane starting to compress.

All good so far, but, say, we change our mind and start expanding the window back (without releasing the mouse button).

We should see the bottom pane brought back to its original size first, followed by the expansion of the top part... even though we are, technically, dragging by the top edge.

This sort of UX detail is something that few people will notice if it's done right, because it feels natural and aligns well with expected behavior. But cut some corners - and you bet everyone will notice.
Going through styling options for the expand/collapse buttons.

Presently leaning towards C or D, because I'm considering making the text selectable with the mouse for easy copy- pasting. For that to work the buttons need to be as unobtrusive as possible.

Still WiP though...
Working on the backup log viewer.

Basically the idea for the log is to support multiple detail levels, each capturing progressively more context that its parent.

With all details collapsed, the log gives a quick overview of when the job was running and if it completed OK or with errors.

Expanding details allows for a drill-down - to look at the details such as the file and byte counts, the timing and performance data and the details of any errors that have occured.

I'm still going to shuffle things around the window, e.g. push the -/+ buttons to the right, tweak the styling too, but the general hierarchical format will stay. It is a very convenient way of organizing the log info.
Finished a piece of code that determines effective timestamp resolution of a file system.

Every file system keeps track of when a file was created and when it was last modified. These timestamps are the natural properties of a file and they are readily available for inspection in the Windows Explorer or any file manager of your choice.

The less obvious aspect of timestamping is that timestamps have resolution. For example, FAT tracks file modification with a measly 2 second precision. In comparison, NTFS uses the 100 ns (nano-second) precision.

It is very typical for a backup software to rely on timestamps to determine if a file has been modified and requires copying. But if the source file sits on an NTFS volume and the backup goes onto a FAT disk, then comparing timestamp directly simply won't work, because FAT timestamp will be rounded-up to a 2-second mark.

In other words, the timestamp granularity needs to be taken into an account when comparing the timestamps. The question is how to determine what it is exactly.

First of all, the granularity for creation and modification times can be different. FAT has them at 10 ms and 2 s respectively, but NTFS has them both at 100 ns.

To complicate matters, there appears to be NAS devices that report using NTFS, but in reality having the granularity that is not 100 ns.

There are also NAS devices that mangle timestamps so spectacularly that it can't be explained in any rational way. Like rounding them up to 1 sec and then subtracting 1 ms.

So what Bvckup 2 does is it probes the file system and attempts to determine effective resolution of both timestamps. This involves dropping a temporary file and then trying to set its timestamps to this, that and 3rd and see what they end up at. From that it's possible to deduce the resolution.

In cases when such probing fails, the app falls back to guessing resolution by the file system name. It is also possible to override the resolution values via the config file, just in case.

Additionally, the program also keeps track of a maximum observed timestamp difference for the backup volume. That is, when replicating the timestamps, it will make note of the values it is asking to set and values that end up being actually recorded for a file or a folder.

This difference is used as a lower threshold for the probed and estimated resolutions and, over time, it converges to the actual volume resolution.
Done with the backup scheduler. Comes with a couple of neat improvements over v1.

The Go and Stop buttons work almost exactly as in v1. If the backup is running, click Stop once to Pause it, click again to Cancel, click third time to disable the job. If the job is disabled, click once to enable it, click again to manually start it. And so on, you get the idea.

First improvement over v1 are the Go! and Stop! commands that are activated by holding Ctrl down when clicking on the toolbar buttons.

Go! moves the job straight into the running state.
It's the "Run it. NOW!" command.

Stop! stops and disables the job straight away.
It's the "Stop it. NOW!" command.

Second improvement is that multiple jobs can now be run in parallel. Generally, this has a negative impact on the overall performance, but there are certain cases when this is desirable. One example would be running a smaller backup in parallel with a much larger and slower one.

There are three ways to do this:

  • With the Go! command.
  • With a per-backup setting that makes the job always run right away, even if there's another job is running.
  • With an app-level setting that does the same, but for all jobs.

PS. I considered adding something more elaborate, e.g. allowing assigning jobs to "run groups" and then allowing only one active job per group, but that seemed like an overkill, so I dropped that.
Another pass over the main window chrome and layout. The "backup jobs" section is just a dummy header ... and it is confusing, I think.

† This is an app screenshot, not a sketch.
Exploring options for the main window format.

Specifically, I want to get rid of the V1's Summary panel, trim the information contained in it a bit and fold it directly into the main window items.

In conjunction with this I'm also considering having two modes for displaying backup jobs on the list - "terse" one-liners and "extended" multi-liners. What I still need to figure out if this should be a per-item preference or an app-wide setting and how one would switch between these display modes.

Work in progress, obviously. Any thoughts - let me know.
Done with a little animation that and switches main window between a "welcome screen" and the regular backup list view.

The animation and fading effects can be disabled, there's a setting for that.

Windows comes with a standard toolbar control, but it's an overkill if you need just a couple of buttons at the top of the window.

It basically takes longer to figure out how to use and customize it compared to creating a basic window and sticking a couple of custom buttons on it.

Had I have a need for toolbar ribbons, docking, on-the-fly customization, etc., then ToolbarWindow32 makes sense. But if it's something simple, then there are faster options.
Meet the toolbar.

To explain a bit what you see in this masterpiece :) -

The idea is to try and keep the menu/toolbar area simple and flat, but still provide some sort of a visual clue that Go and Start are clickables.

Here are three options compared back to back to make the differences a bit more obvious -

Here are few options that were leading up to the above one:

iOS says Hi

Also, considered disposing off the menu bar and folding whatever few items it has into the drop-down menu. But this just looks weird and too alien on Windows, the app needs a menu bar.

Dropped the idea.
The look of the main window with no backups configured. Tried using a stock button, but it looks heavy and dull. Hopefully, this is a bit more refined.
Trying to understand how to indicate if a selected path points at a removable storage.

The app tracks storage devices being plugged into the computer and it adjusts backup paths if a known device re-appears under a different drive name.

I can, of course, not bother with this at all and simply spit out a notification when (and if) the drive change is detected, but then I think it's an important functionality and it'd better be mentioned during the setup.
"Work-in-progress" buttons for beta releases.
Correct your input.

† Idea is courtesy of the OS X login screen, I think. Though I did experiment with concise error indication quite a bit.
Wrapping up the Periodic Backup configuration window.

Unlike the v1 that always stored the backup interval as a number of seconds, the v2 adds a time unit specifier.

This simplifies quite a few things in the UI and yet it keeps the configuration flexible enough to support 1.5 hour backups by allowing to specify a 90 minute interval.

On an unrelated note, killed few hours trying to utilize native Windows controls for the interval configuration. The amount of work required to make them look reasonably good is as depressing as it is remarkable.

For example, the editbox control doesn't natively support vertically centering the input text. It just does not. It is meant to look like this -

Want "120" aligned to the label on the left? Sure thing, that'd be 2 hours of plowing through documentation only to realize that it must be... drum-roll... a multiline control that you will then need to manually pad the right amount at the top to push the line to the middle.

But that's peanuts compared to trying to beat some sense into the dropdown box. Long story, but making a box taller and keeping the line centered requires taking over the control drawing functions, which in presence of Windows themes is a delightfully unpleasant experience.

In any case, turning the top control line into the bottom one -

costs exactly a day of work, while it really shouldn't. Welcome to Windows.
Finalized the design of inline hint/tip widget for input controls.

Initially, it was meant for the config window that hosts periodic backup settings (as seen in a couple of posts below), but having talked to others and slept on the issue I think that the natural language input (i.e. requiring to enter "1 week" or "15 minutes") is not the best option for this case.

An alternative design is coming up in a bit.
Coded the previous sketch, played with few parameters and it is starting to get there ... I think. Will make another pass over this tomorrow and it should be good to go.
For what it's worth I have also gave a try to the standard Tooltip Control . In fact, that's where I started.

A regular tooltip looks like this -
It adds another layer to the windows stack, breaks the visual flow of the layout and still manages to tuck itself under the mouse cursor.

There's also a balloon version of the same, which looks like this -
It is marginally better, but just look at those lovely rounded cornersi . Eat that Apple. Real artists ship.

tl;dr - the standard Tooltip Control is not very good, so I had to come up with a custom replacement.
Not sure about this one, seems a bit clumsy and a bit too custom.

This is basically a way to explain what is expected as an input for a particular configuration value. There's gotta be a cleaner way of delivering the same info...
Working through the supporting code for the Settings window.

Done with collapsing/expanding mechanics and respective fading and moving animations. Still need to revise the copy for the explanatory bits and massage in the option for archiving deleted files.

Not to pat myself on the back, but I think it came out looking pretty decent.

That's not to say there's no way to turn off all this fancy styling and animations, because, you bet your favourite monochrome flat design, there is.
The Backup Settings window. Final ...for now :)
Another take on Backup Settings window, 4 styling variations.

Further compacted vertically, but now it appears to be somewhat cluttered and lacking comfortable vertical rhythm (to borrow a term from the typography).
Working on the Backup Settings window.
I think the original design from the version 1 got many things right.

It provided a good at-a-glance overview of the backup configuration, split settings logically and tucked away the minute details. It did have one problem - it was tall. So tall, in fact, that I got several reports that it didn't fit on screen :-|

V2 adds several new options to the backup config, so this doesn't make things any easier. After looking at few options, including paged, tabbed and two-column, screenful layouts, I circled back to the v1 layout.

To compress it vertically I've done two things - got rid of the group boxes (frames) and collapsed radio button groups into a single line with the current selection.

This gives us the middle screenshot (pic) that shows the config window in the "overview" state.

Clicking on a "Change ↓" button expands the radio button group and allows changing the setting without leaving the page. Latter is pretty nice to have, because ... well ... because it's simply faster and more convenient.

Expanding a section gives the screenshot on the right (pic). To collapse the section, there's a little button in the section header on the right hand side. I'm not 100% convinced it's really needed nor that it's the best solution, but that's how it is at the moment.

I forgot the "how to handle deleted files" section, but it will be in the beta. Not to worry.
Meet the Preferences.
The Backup Engine section - as simple as it looks, it is fronting a couple of months worth of work and it's easily one of the most complex part of the v2. This is what detaches the user interface from the guts of the program and lets latter run as a separate process.

The Maintenance section - I figured that with the backup being an important piece of software, very few users would be interested in fully automatic updates. Furthermore, quite a few people like to retain even tighter control over the software and, for example, do not like it when the software talks to the Internet whenever it wants. Hence the new option -

Don't check, but remind to check for updates

It's basically a kitchen timer that goes "Ding!" every week or two (configurable) and prompts you to let the app check for a new version. We'll see how it goes, but I think it's a sensible compromise between the privacy and functionality.

The More button transforms the window into a comprehensive list of all available options, close to two dozens now and still counting. The idea however is to keep the defaults reasonable so that there won't be much need to venture into this part of the app.

Lastly, note how the Apply/Close buttons change when the settings are modified. I have no idea why this is not a standard behavior, because it really should be.
A message bar is a unobtrusive way to display one-liner messages that would've otherwise required poping up a message box.

When a program detects that it was just restarted with Admin privileges, it has a limited number of options for communicating this joyous news to its user.

#1 - A message box. Bluntly put - I hate them. They are always too sudden, they steal your attention and stuff whatever they want to say down your throat. They ... lack manners.

#2 - A system tray notification. This is less intrusive, but perhaps it's too out of the way. Also, the systray is a shared area so the program is always competing with other apps for the same screen real estate.

#3 - Something else that is right in your field of view if you are looking at the app, but that doesn't get in a way of whatever you are doing at the moment.

Cue in the message bar, now an integral part of the Bvckup UI. It is used for displaying various status notifications right after the launch, after switching the engine to/from the system service mode, when discovering software updates and few other things.

I have also tried the self-closing version of the bar that dismisses itself on a timeout. Like so -
and so -
but having lived with this for a couple of days, it now seems like a solution in a search of a problem. So I shelved it for now. If there's any a real need for this, it'd be very easy to add later on.
Nailed down the installer flow. A couple of things of note here.

The most unconventional part of the setup is how it handles the configuration of the installation parameters.

The configuration page (see Page 3) is very bare and focuses on how the program is going to be used instead of being a mixed bag of the minute details of the setup.

The target location is made into a secondary setting with its value controlled by the usage preference. There's no field for the Start Menu group name. There's no ever popular "Run the program after the setup completes" option either. Instead, exiting the setup automatically runs the program (see Page 4), unless the installer window is closed through an (x) at the top.

This leads to a simpler and more intuitive setup flow... or at least that's the idea :)

The installer supports both privileged (administrative) and unprivileged (regular user) installations.

If you are logged in as a regular user and have no rights to modify C:\Program Files, there's no reason why you shouldn't still be able to install a software, just for yourself.

While other installers routinely ask if to install the software for everyone or just for one user, it's a superficial question. It merely controls if the Start Menu group is created under All Users or your personal profile. The program files still go into a system directory and this requires administrative privileges.

Bvckup implements true system- and user- deployment, which is akin to deploying the program in /usr/bin or ~/bin in Unix terms.

In fact, on a multi-user system it is perfectly possible to install one instance of Bvckup for each user and they will all happily co-exist with each other.
The first page of the installer.
Working through the details of the update mechanism.

This is one of more difficult parts of the application, even though on the surface it looks uncomplicated. If we run the installer and it detects that the app is already installed, it should just update the installation. How hard could it be?

Let's see.

If the program is not running, then it's indeed a simple matter of replacing the program file with a newer version. The fun starts if the program is running.

On Windows, running a program locks its file, preventing any changes from being made to it. This is not required per se and there are operating systems that don't do this. There are also custom PE loaders for Windows that can take an executable and create a running process out of it without any locking fuss. For example, bo2k could do that back in 1999. See its source code for details if interested.

To complicate matters, there might be more than one process running off the same executable. For example, if several users are logged into a Windows box and each runs a copy of a program.

Therefore, the first order of business for any installer is to shut down all running instances of the program. A well-behaved installer should also restart all these instances after it's done with updating.

The shutting down part is not a big deal. There's more than one way to do this and all of them are fairly straight-forward.

It's the restarting that is contrived.

Not only program instances may be running under different user accounts, they may also be running with custom command-line arguments, as Administrators or system services. Restarting this bundle of joy is not an envious task.

Here's how Bvckup works around this mess.

Every Bvckup executable has a unique Global event associated with it. The event is created by the first instance of the program when it's launched and it is opened by all other instances.

The name of the event includes the file ID of the executable file. This groups running instances by the file they are using, because, after all, the installer is interested in a specific executable and nothing else.

For example, the following is an event name used by the Bvckup installation on my development box -


With this in place, the installer starts off by signaling the event.

In response, each running instance makes a temporary copy of the executable file and starts it in a special "wait for the update to finish" mode, passing it all the context and details required to correctly restart later on.

And then it exits.

This temporary process is referred to as a relauncher and it merely spins there waiting for the Update event to become unsignaled.

Meanwhile the installer waits for the program file to become unlocked. Once all running instances replace themselves with the relaunchers, the executable frees up and the installer proceeds with the update. Then, once it's done, it clears the "update" event and exits.

This releases the relaunchers, they restart program instances (now using the actual executable), clean up after themselves and ta-da - the update is completed.

* Techincally, instead of adding a special "relauncher" mode to the program it's possible just to have a small standalone relauncher program.

It comes down to a matter of taste. Some people don't like to spawn from temporary executables, and I like to keep my file count to a bare minimum. To each his own.
Error #1 | #2 | #3 | #4 | #5 | #6
Dotting all i's and crossing all t's, of which there's plenty now that there's an optional system service that just begs to be stopped, restarted and deleted from under app's feet.

Doh... I missed the disabling of the service. Will add it tomorrow.
Vector icons for the website.

This is my first time not using pixel images to decorate a website. Instead, all icons are saved in the SVG format and converted to a webfont, which is then used to style regular HTML characters into icons.

For example, here's a font with the above icon set - VectorIcons.ttf

The advent of webapps like IcoMoon and Fontello not only made the process of building @font-face kits an absolute no-brainer, it also gave an easy access to hundreds of existing vector icons, many of which are free and of production quality.

When coupled with now virtually universal support for CSS text-shadow, box-shadow and gradients, this almost completely eliminates the need for raster form of smaller decorative and nuancing elements.

This is a very big deal and a great time-saver.
Re-finalized the sign-up form styling and mechanics.
Few styling options for an input control. Leaning towards top right or bottom right, but still need to see what works best in context.
More or less done with the Features section on the new website.

Probably missed few things, but that's easy to add later on. The biggest time sink was trying to inject some visual hierarchy into the list, but without going against overall minimalist look and feel of the site.

Might still revise it after sleeping on it for a couple of days :) Won't be the first time.
Plodding through some grunt work with the new website. An almost equal split between working on a copy, javascript and styling.

As simple as the screencap looks, it came with two hurdles that needed solving.

The header font, Alte Haas Grotesk, didn't display correctly after it was converted to the @font-face kit. Specifically, the lowercase 'o' looked like this:

The font's free, but I couldn't find a copy of its license, so I started tracking down font's author to ask a permission to edit the outlines.

While waiting for a reply, I found another workaround which involved generating the webfont kit from OTFs instead of TTFs. Not a big deal, but it still took over 3 hours to resolve.

The second issue was with the slide-out, fade-in animation. Sometimes the slide-out didn't extend all the way down and sometimes it overshot. This traced back to the fact that slide-out height was computed before the OpenSans was fully loaded and the script fused basic sans-serif to compute the height.

Again, not too complicated, but, again, cost an hour to work through.


Little bit of this, little bit of that - all unplanned and it all adds up.
Nearly done with the new website design. Here's a snippet of how the site's header interacts with scrolling.
First pass over UI styling for the main window.
Released a small playable test of the directory scanning module.

It appears that using as many threads as there are CPU cores and traversing the tree in a depth-first fashion (scan child directories first and sibling directories second) yields the best scanning times.

Test and see for yourself - bvckup2-demo2.exe or 64 bit version.

Also, see here for details.
Updated @bvckup background with V2 branding.
2nd sketch for redesign. Larger version is here.
1st pass over the redesign of Larger version is here.

Current design is almost two years old and it grew steadily disconnected from the simpler aesthetics I am pushing for now. So it needs to be redone.

I will probably simplify this even further. I've been wanting to do a true minimalist site for a software product for a long time now and this seems like a real chance to do just that. Yay.
This is a sprite sheet of all raster images that are going into the run-as-a-service demo.

Interestingly enough, the total size of individual images saved as 32bit PNGs is 63,127 bytes, while the size of this sprite sheet (without checkered backdrop) is 63,829. After factoring in the size of the code that is needed to parse the sprite sheet, it really becomes a no brainer to keep every image in its own file and embed them into .exe's resources that way.
Added a "crash" handling and reporting facility that includes several important changes compared to the original version:

1. The program now differentiates between three types of errors and handles them slightly differently.

First, there are unexpected operating system errors, i.e. when a core Windows function fails in a way that it shouldn't be failing. Classic example (that everyone is aware of) is TrackPopupMenu that is declared as returning boolean, but actually doesn't shy away from returning an integer under certain circuimstances.

Secondly, there are inconsistencies in app's own code, whereby one part assumes certain usage that is not observed by another part. These are so-called "assert" failures. The original version has three critical issues, all of this particular kind.

Lastly, there are straight-forward crashes. This is when a program is trying to operate with data that is no longer there or corrupted. This is the worst of them all as it typically indicates a major screw up on programmer's part, but in some cases it may also be an indication of failing hardware, like a faulty memory stick.

2. The program now generates a minidump - a small-ish file in a standard Windows format that captures the context of a failure. Minidump files dramatically simplify investigation of a problem on the developer's end.

3. Finally, the minidumps can now be uploaded to the support server right from the error notification window with one click. The transfer is based on WinInet API as this is supposed to improve the chances of getting the report through in proxied and otherwise "enterprised" environments.

For cases when the upload fails, the error dialog also offers an option of opening a link in a regular browser and reporting a problem that way.
64bit version is stored inside of the 32bit executable, from where it is extracted and launched to replace the 32bit version if latter detects that it is being run on a 64bit system.

Pro: a single executable to distribute
Con: the exe comes out weighing twice as much... or does it?

The size of a 32-bit executable can be easily reduced with the help of tools called executable packers . They work by turning the exe file into a self-extracting archive that also automatically launches the original exe after the extraction. It may sound simple but in reality it involves bundling a custom PE loader with the packed executable (as it might not be possible to extract the exe into a temporary file before running it).

In any case, exe packing is routinely used by developers who care about the size of their products, with uTorrent being one of most notable examples. Original Bvckup was also compressed, owning its tiny size in part to the magic of UPX .

The problem lies with 64-bit executables. For a variety of reasons there are presently no exe packers that support them.

However if we store 64bit binary inside of the 32bit executable and then pass latter through a compressor, we end up with both being compressed. Ta-da!

But there's more.

When an .exe is compressed, the compressor leaves the primary application icon intact. This is done so that the app would keep its original appearance in Windows Explorer and not sport a generic "program" icon.

Starting with Vista, the app is expected to support a variety of icon sizes that quickly add up. For example, bvckup's icon weighs in at about 110KB. That's almost half as big as the actual code, the code that does quite a few meaningful things on top of just looking great :)

So, if we do the 64-in-32bit packaging, we end up compressing 64bit version in its entirety, including its icon, and arriving at a greater size reduction if we were to compress the exes separately.

A page from the first demo of Bvckup 2.
Working on a shared memory buffer interface, a part of the IPC module, which is what enables the separation of the engine and the UI.
A bit more humane approach to critical error reporting.

The message is a homage to NetPositive, the BeOS browser that pioneered the technique of using haiku in error reporting. In fact, the message in option B is from the NetPoisitive original list .

The icon is that of a Zen circle, which has a plenty of meanings, many of which applicable given the circumstances.
Three-frame animation of engine's heartbeats for the demo.
Final UI styling for the "Run As A Service" demo.
The "Run as an Administrator" icon.
Test of a fading transition, part of a small library that deals with supporting UI animations.

Fading animation dissolves a part of a window into background, in preparation for transitioning the window to another state, e.g. moving to the next page in a wizard or resizing the window into a more compact form.

As simple as it sounds, these sort of animation is fairly complex to implement on top of the Windows API.

Windows includes native support for window transparency in a form of layered windows, but it comes with a list of limitations that aren't exactly practical or easy to work, except for several specific cases.

The simplest working solution involves taking a screenshot of the window, creating an overlay window and then using it as a canvas to render the transition from an empty background to the screenshot. All the while making sure that Windows doesn't sneak in an unsolicited window refresh that would lead to a flicker, which in itself is a very "fun" thing to debug.
Every application icon that is exposed to the operating system must exist in at least nine sizes.

16, 32, 48, 256 are more or less expected. These are used in "small", "icon", "tile" and "extra-large " views in Windows Explorer.

20, 22 and 26 are possible sizes of an icon used in the window caption. The exact size depends on screen's DPI level.

And 40 (along with 32 and 48) is the size of an icon used in Alt-Tab view on W7, again depending on the screen's DPI.

If you don't provide an appropriately sized icon, the OS will grab one that is slightly bigger and shrink it down, ruining the pixel polish.

Lastly, each icon ideally must be saved in 32, 8 and 4 bits per pixel formats, so to be compatible... you know... with EGA cards

It would've not that big of a deal if it weren't for the size of a resulting .ico file. For example, the above icon ends up being ~ 120KB, whereas the actual .exe it is attached to and that does quite a few meaningful things sits at around 96KB.
Similarly, every bit of graphics in the app must be done in at least three different sizes - one for each common screen DPI level.

DPI stands for dots-per-inch and it's a metric of a pixel density on the screen. The smaller the screen, the more pixels are packed into a square inch, assuming the resolution stays the same. The difference in DPI between a large LCD panel and a 12" notebook screen can easily be a factor of two or more.

This means that if a program renders 9px text on both screens, on one it will appear twice as small as on the other one, and might command a need for a good magnifying glass.

So naturally it's a good idea to make the OS aware of the actual DPI of the monitor in use. If it's too high, the OS can do few things to make things visible with a naked eye again. Like increase size of the default system font and make windows generally larger.

This can be done automatically by looking at the monitor .inf files or by letting the user configure the DPI manually.

XP defaults to 96 DPI and with some difficulty it can be changed to 120 DPI. Vista and newer versions are better. They come with four predefined levels - 96, 120, 144 and 192 - and they also support custom settings.

This brings us back to needing multiple versions of every graphic in the app.

Vista is the first Windows version to routinely use monitor-specific DPI values out of the box. As such 96 is no longer a dominant DPI level and the apps are now expected to take the DPI into an account when rendering their content.

The higher the DPI, the larger the graphics and the fonts the app should use. The penalty for not doing this is an automatic stretching of the app window that inevitably leads to a blurry appearance and a host of other lovely issues .

In other words - not complying with OS expectations is not really an option. Hence the need for 3+ differently sized copies of the same pixel art.
The UI sketch for run-as-a-service demo.

Trying to find a balance between using custom styling and not getting to far away from the native OS look and feel. See also 8 other options leading up to this one.
Illustrative icons for the same demo.
First pass over the demo UI.
V2 running as a service, an actual screenshot for a change.
Exploration of measured blending of native and custom UI styling.
An idea for a space-saving tab control. Shelved.
An idea for an in-form buttons.
Exploring full-on customization.
Earlier sketches of heavily customized UI.
Made by IO Bureau in Switzerland

Updates Newsletter
Blog & RSS
Follow Twitter
Miscellanea Press kit
Legal Terms