Work in progress, screenshots and sketches
Still frame | Animation
R75 is getting an overhaul of the email configuration UI.

Earlier versions had general and alert-specific settings mixed up. For example, "From" doesn't really belong to the alert configuration, while "Include" from General Settings does.

This has been sorted out.

A larger picture here is that the app will be getting additional alert types - e.g. for S.M.A.R.T. diagnostics and backup verification - and this requires a proper structure to support these cleanly.

Also, while at it, the UI got a new framework for handling window transitions between different states.

For example, in the above screencap there's are 5 primary states and 2 intermediate ones. Prior to R75 moving window between these states required quite a bit of hand coding. This is no more, so we can now animate the hell out of any window. Consider yourselves warned :)
Long time, no update.
How about a rant? :)
The year is 2015.

The age of cars trying to drive themselves and rockets trying to land on sea barges. Graphics cards are effortlessly churning photo-realistic images in real-time, mining crypto money and cracking passwords.

And here we have Windows still defiantly refusing to use any of that double-buffering nonsense when re-painting one of its basic common controls. Because who wouldn't want a bit of flickering?

Another iteration over tabbed dialogs, now with a bit of animation.
Option A and A-flat | Option B | Option X | Option Y | Option Z
Tab control for multi-page configuration windows in general, and for the email notification configuration in particular.

Prior to R75 there was one primary email alert, which was sent on backup completion. Now the app is getting other types of alerts, so the UI needs to be adapted to support that.

The idea is that the config window for specific alert type will give access to both the alert details and general email settings that are recycled between all alert types.

So here are several sketches for organizing page index of a multi-page configuration window. Nothing terribly novel here, it's more just a matter of getting styling details right - make sure it's obvious that it's an index and that its items are selectable.

Option Z (right-most) looks OK, but it's not obvious that index items are actionable as they look like static section headers.

Option Y is a bit better, but the arrow actually points at "From" field and you can bet some people will end up in a stupor because of that.

Option X uses standard Windows white-on-blue for selected item, so it's better in the actionability department. But it doesn't look nice.

Option B attempts to fix that.

Option A attempts the same but using app's own styling of selected items, which is already used in several places in the UI.

Thoughts or comments? There is a thread for that!
Minor tweak to the look of the delta copying progress bar - a bit more contrast, both for completed and outstanding parts.

Will be in R74.15
Collapsed group | Expanded group
Third pass.
Second pass over backup grouping UI for R75.
I think it's getting there...
Collapsed group | Expanded group
First stab at backup grouping UI for the next release (R75).
Thoughts or comments? There is a thread for that!
There's a good guiding principle coming from Paul Graham that says make something people want. It's very simple and it's very effective. It also makes real-world user opinions a critical part of any good product design.

In other words - please take a moment and talk to us! :)
Unicode, as you may know, is a standard that assigns pretty pictograms to numerical codes from the 0 to 0x0010FFFF range.

In simpler cases pictograms represent letters, numbers and symbols. In other cases they are decorative glyph elements, like umlauts, that are meant to be combined with the letters.

There are also Unicode encodings - UTF-8, UTF-16, etc. - these deal with how specific Unicode numerical IDs are translated into sequences of *bytes* for storage, transmission, etc.

The simplest encoding is UTF-32 - it simply stores Unicode ID as a 32-bit number. This is however rather wasteful as at most 21 bit is needed to represent any Unicode ID.

Prior to Windows 2000, Microsoft used a simple 16-bit encoding that mapped every Unicode ID from the 0 to 0x00FFFF range to the exact same 16-bit number. This is called UCS-2 encoding. The Unicode range was smaller back then, so this approach worked.

Then, Unicode range grew and IDs no longer fit into 16 bits. So to address this, UCS-2 was modified to encode symbols from the 0x010000 - 0x10FFFF range using two 16-bit numbers, from the 0xD800 - 0xDFFF range. Exact details of this encoding are not important, but what it meant was that some 16-bit numbers, e.g. 0xD812, were no longer valid Unicode symbols just on their own.

This is called UTF-16 encoding, and this is what Windows uses at the moment.

But wait !

As Apple likes to say after a screw-up - "it turns out" that despite routinely talking about UTF-16 in their API documentation, Windows file system still uses plain 16-bit encoding for the file names!

For example, "\xD812Haha" is a valid file name even though it is *not* a valid UTF-16 string. This in turn means that if the app is converting file names into other encoding, it should go easy on full UTF-16 compliance checks and be ready to digest ill-formed UTF-16 string.

Bvckup 2 keeps all text in UTF-8 format internally, so it does quite a bit of conversion from/to UTF-16 and this leads to some interesting things when the app runs into an malformed UTF-16 name. Like those created by *ahem* Adobe *ahem* After Effects *ahem*, for example.

The solution is simple - try and convert as if we have the UTF-16 encoding, but fallback to UCS-2 when running into invalid UTF-16 sequences. Also, apply the same approach when converting back to UTF-16 and all will be good.

Will be in Release 74.8.

* Kudos to Andrea for reporting this.
Release 74 also implements new delta copying defaults and the results are ... interesting :)

Delta copying, when enabled, has always been used conditionally and the criteria was the size of the file. For smaller files the overhead of setting up the delta copier is larger than any potential savings, so these files are simply copied in full.

The file size threshold was initially set to 128KB and the value was inherited from the original Bvckup version, where it made sense because the I/O was generally slower back then.

Fast forward few years and 128KB is no longer a good default, so as of Release 71 it was increased to 2MB.

With Release 74 the criteria changes again.

First, the threshold goes up to 32MB.

Second, the delta copying will also be used for files between 2MB and 32MB, but only IF they were modified in the last 30 days OR if they were previously delta-copied.

The rationale is pretty simple - we try and not bother with delta copying unless files are large or they change frequently.

This change has two main effects.

1.   Smaller backup state.

Delta copying stores per-block hashes in a backup's configuration directory. This is done for every file that is delta-copied, so the fewer files are, the less data ends up being in the \deltas folder.

The above screenshot shows the \deltas folder stats after backing up \Windows\System32 directory. And here are the same stats after backing up with an older version -

That is, twice the disk usage and nearly 10x more files and folders in the \deltas folder.

2.   Faster overall backups.

For smaller files the overhead of setting up the delta state is comparable to the time needed for the actual copying. We skip the delta setup -> we complete the copying faster. Magic.


That's nearly a 10% speed up. This is obviously specific to the contents of \Windows\System32, but it's not hard to extrapolate from here.
Release 74 will also be adding an option for exporting app's complete configuration as a single .zip file.

Restoring app's config from this backup is a simple matter of shutting down the app and extracting contents of the .zip into %LocalAppData%\Bvckup2\ folder.

But since we can hardly expect anyone to know this location by heart, a short text file called How to restore.txt is also included with each backup zip.
Run-time memory usage is now capped.

Prior to Release 74 app's memory usage was proportional to the number of items in the backup. This was because both the source and destination file trees were kept in memory in their entirety, as was the backup plan. While not a big deal for smaller backups, it does translate into quite a bit of RAM when the file count starts to climb into a million range.

Release 74 implements the off-RAM storage for large data sets. In previous release the run-time arrangement was like this -

With the new release the objects are kept in the app's memory only if they are in active use. The rest is trickled down into a swap file for storage, through a couple of intermediate caches.

The object cache holds fully assembled instances of application data objects and its size is kept at or below preset item count.

When the object cache grows too big, least recently used objects are written down to the swap file. The writes are passed through a typical page cache to coalesce the I/O into larger chunks.

This lunch is obviously not free.

There's a cost to even successful cache lookups and there's most certainly an expense to doing the disk I/O.

The app mitigates this by using aggressive cache optimization (buckets, MRU lookups, etc), raw IO caching and also by allowing keeping swap files in a custom location, and presumably on a different physical drive. But, again, all this starts to really matter only when backups grow into a million item range.

While reasonable defaults are provided, the cache sizes, the trimming policy and other details are fully configurable through the .ini files. The app may also be set up to log cache/swap performance if required.

This change is big, it took nearly two months to implement and it means that Bvckup 2 can now handle backups of unlimited size. To say that it's a very big deal would be an understatement.
Still working on major engine changes to cap its run-time use (and I will be posting data from the test runs shortly), but meanwhile here's something simple that I'm considering adding to the next release.

It's an option for the app to report your backup statistics to one of our servers and then being able to display these stats as a simple PNG served over HTTP from the same stats server.

It's really meant for trying to motivate other people to back up their things as well as to indirectly promote the app as well. And, yes, I lived through the 90s, that's where I got the idea from :)
Currently working on the improvements to the scanning and planning modules of the app. No pretty UI screenshots to show, just some performance numbers.

The ultimate goal of this rework is to put a cap on the run-time memory usage, which in turn is needed to better accommodate very large backups (tens of millions files and up).

The app comes with a formal backup planner, which is a module that accepts two full directory trees (one for the source location and another for the backup one), digests them and produces a list of simple steps that, when executed in order, bring one tree in sync with another. The app is then diligently goes through the list and creates, updates, renames and deletes files and folders as directed.

An alternative to formal planning is to to traverse the trees and make on-disk changes just as you spot the differences. There are several drawback to this approach, including not being able to estimate the amount of work needed, to understand disk space requirements and to detect file/folder moves. It's just an altogether messier and less predictable way to go about the whole thing.

That's how Bvckup 1 worked and that's how robocopy works, for example.

Long story short, having a backup planner is the right thing to do. However, it means that the app needs to have two full file trees readily available for the planner to digest. If these trees are kept entirely in memory, it may get expensive with large backups. At 100 to 300 bytes per item, a million file backup means 200 to 600 megs of RAM just for the trees.

That's quite a bit, but more importantly this memory usage grows linearly with the size of the backup. This is no good.

The solution obviously enough is to not keep full trees in memory. This is a fairly complicated endeavour and it maps onto several large changes to the app inner structure.

One such change is nearly done. Both the scanning planning modules no longer require for the trees to be resident in memory at all times. Instead, they now use them in a piecemeal fashion.

Behind the scenes, trees still reside in memory for the time being, but the underlying rework has some nice side effects. Trees now take less memory, they are faster to populate, faster to save and load to/from a file, they take substantally less space on disk and they compress better with NTFS compression.

This will ship as a part of R74.

After that, R75 will replace in-memory trees with version that can swap out tree parts to the disk, significantly reducint the run-time memory footprint of the scanning/planning process.
Release 73 adds copying speed and completion time estimation for longer file copies. To toggle between the views just click on the text next to the progress bar.

The estimation is calculated in three steps.

First, the app looks at effective copying speed and uses sliding window to average it across last 25 samples taken 650 ms apart.

Second, it uses this average speed and the remaining byte count to approximate the completion time.

Finally, the completion time is averaged with its own sliding window, but using only 4 past samples.

This double-averaging helps mitigating the effects of sudden short-lived I/O bursts and hiccups on the estimation.

The parameters - 25, 650 and 4 - are configurable through the engine .ini file, so the whole thing is open for tuning if needs be.
The initial revision of the auto-update opton is in.

It's a simple extension of existing "check for updates" option that skips showing the "new version found" prompt and just goes straight to the updating part.

There are two caveats, both applicable only when the app is running in service mode and both stemming from the fact the updates are driven by the UI.

First is that closing the UI will effectively disable auto-updates. Second - if the UI is running without admin privileges, running the installer will trigger the UAC prompt.

Both will be resolved shortly by moving all update logic into the engine module. Meanwhile...

The auto-update option will be hidden by default unless the Preferences window is opened with the Control button held down.
Release 73 will also be adding support for monthly backups. It will now be possible to set a backup to run on the N-th of every M-th month.

You surely are thinking "but 31st!", "February!" or perhaps even "every 6 months staring on Jan 31st!". Good point, the UI got just the thing for this -

A rework of the loggging module, coming up in Release 73.

As you may or may not know, Bvckup 2 uses a dual-module architecture of the inside.
One module is the backup engine. It maintains the list of backup jobs, schedules, executes backups and generally does other useful things that don't require direct user involvement.

The second module is the user interface. This is the part that displays the app's window and allows users to monitor and control whatever the engine module is doing.

The principal point is that the engine and the UI are isolated from each other. They share no run-time data and communicate with each other using a messaging protocol.

This isolation is what enables the app to run in service mode. In this mode the engine runs in its own process, as a system service, and the UI runs as a regular desktop app.

It was all well and beautiful, but there was, however, a caveat.

The caveat was the backup logs.

Logs are inherently shared data - they are written by the engine and they are read and displayed by the UI. Logs tend to be bulky, so it is shared app data of a special kind and it requires special handling.

Prior to R73 the app included a third module (in its dual-module architecture, no less) - the log manager. It was a complicated contraption that abstracted log access for the engine and the UI, and the biggest issue with it was that it behaved completely differently depending on which mode the app was in.

In desktop mode, the manager was writing *and* indexing logs on behalf of the engine and the UI was merely displaying them. However, in service mode, the engine did no indexing, just the writing, while the UI was making copies of all logs for its own use and then it was indexing *them*.

This is exactly the kind of non-uniform conditional processing that the dual-module design was meant to avoid!

Enter R73.

The separate logging module is no more. The engine no longer does any log indexing, it is now entirely the UI's reponsibility. Indexes can now be rebuilt on demand at any time, and finally, the UI no longer creates its own copies of the log files. Instead, it works with the engine to gain access to the engine's copy of the logs, using DuplicateHandle to move open file handles across the process boundary when needed.

The UI now can display logs while it is still catching up with them.

Think -- running in service mode and launching the UI only once a week. On each launch the UI needs to index a week's worth of log entries it hasn't seen before. Prior to R73, the UI would just sit there and show "synching ... XX% done". Now the sync is done in the background (as per the screenshot).

The UI now can also recover gracefully from indexes getting corrupted (e.g. due to a cold power cycle or blue screen).

Just as important, the underlying code is now much simpler, lighter and more uniform, the executable is 8KB smaller and the developer is generally happier. Good news all around.
Running an extended set of tests to fine-tune the performance of the Bvckup 2 bulk copier.

It's an inherently simple matter of determining an optimal size and count of read/write buffers used to ferry data from source to destination. There are however ... erm ... details.

A source can be local or it can be over-the-network. There are slow local devices (e.g. a USB2 drive) and there are fast local devices. There is a warm, there is a cold cache and there are files that are cached only partially.

The writes can go out in lots of smaller chunks, all in parallel, or with a smaller number of large buffers. Again, it can go to a local device or over the network. The app can also explicitly flush all buffers after copying or leave flushing to the system.

The above graph is the I/O throughput of a 0.5 GB file being repeatedly copied from the local HDD to Synology Diskstation over a Gigabit wire. What changes between the runs is the buffering strategy and as you may notice there is a difference.

Still need to run several over-nighters to gather more statistics, but there is already a couple of things that can be changed in the copying code to make it run faster.

Scheduled to appear in the Release 72, coming up shortly.
A quick peek at the Reseller Dashboard. Coming soon...
Completed OK | Failed
"Send test email..." is now done.
R71 adds an option of automatically enabling NTFS compression for the backup logs.

Based on what I've seen so far, both .log and .tdb files compress to about 20-30% of their raw sizes.
Reworked part of the More Options page in the backup configuration window.

* The animation here is a bit jerky, but it's just a GIF encoding artefact.
Email template editor. "Show variables" opens up another part that lists all variables that can be used in the form fields.

This is getting a bit too complex for my taste, but it's the part of the UI with little foot traffic, so it's probably OK.
A little primer on the mail server configuration.
Simple config | Custom config
Upon even further consideration it looks like the From field is not of a prime interest either, so it too can go into the Details.

This cuts down the number of fields and renders section headers unnecessary, so they too can go.

Basically the idea is to reduce the dialog down to fields that actually need user's input and stash everything else away... I don't know if it's the right thing to do though. Perhaps having a summary of the whole email setup (including the subject, From and the message) is not such a bad thing. Not sure. Need to sleep on it.
Upon further consideration, decided to remove both Subject and Message off this dialog to keep it simple(r) and to the point.
Simple config | Custom config
First stab at the email configuration window. Too many options, too little space. As usual.

EHLO there!
Release 71 is getting email notifications, so here's all you always wanted to know about teaching your app speak SMTP.

First of all, let's get MAPI non-sense out of the way. As per usual, it sounds good in theory, but an awful mess in practice. So, no, MAPI is not an option unless your goal is to burn through your support budget and/or practice your communication skills with less-than-happy users.

This leaves us with raw SMTP.

SMTP is an inherently simple protocol - send a text command, receive a text response. Repeat. Open a socket, connect, send(), recv(), done. A couple of hours of work at most. Ha.

It starts to get complicated when you throw TLS in a mix to secure the communication with the server. In particular, this is required when you want to send emails in a way that requires logging into a server first.

There are two ways in which SMTP can be combined with TLS.

First is when you take an existing SMTP session and say STARTTLS. The server says OK and then you basically stop talking SMTP over your TCP connection and start talking TLS. Then, once the TLS handshake completes, you go back to SMTP, but now it all goes over the TLS session, nice and encrypted. This is a standard way, that's what every decent SMTP client should do. This is TCP/25 and TCP/587.

Second option is SMTPS. You first establish a TLS session and then start talking SMTP over it. Just like HTTPS. Apparently, this is non-standard and discouraged way, but still widely used. This is TCP/465.

Then, there's a matter of how to add TLS support to the app.

The obvious choice is OpenSSL, but that will cost you 200-300KB in extra binary code, so unless you already have OpenSSL linked in, it's not really an option, not for a *backup* app.

Digging through MSDN you may come across something called SSPI or Security Support Provider Interface. It is an attempt to stuff several security protocols like SSL and Kerberos into a single framework in a faint hope for a miracle of reuse. One of SSPI providers is a mysterious entity called "Secure channel", which, upon closer inspection, turns out to be ... yes, TLS!

So, the next step is try and work through all the abstraction fluff of SSPI to make it actually do the TLS handshake. The idea is actually quite neat - you can create a stateful object (context) that takes in data you get from the peer and spits out some data for the peer. At some point it completes the handshake and becomes available for accepting app's own data, which it encrypts and hands you a blob to send to the peer. You would then also get blobs from the peer, pass it to this context/object and get back plaintext data for the app. Whether it's TLS or ROT13 on the back is completely transparent to the app.

That's the idea. In reality, the documentation is subpar and lacking and there are exactly two examples - one is incomplete and another one is buggy.

The good news is though is that once this bloody contraption gets off the ground, you get full TLS support at the cost of just several kilobytes.

With TLS done, there's still a bunch of smaller things to be taken care of, e.g. lack of asynchronous DNS query API in pre-W8 windows. Not a big deal, nothing a spawning of a worker thread would solve, but still ... a bit here, a bit there and it all adds up.

In other words, it's all coming together, but as per usual it all takes longer than anticipated. Shocking, I know :)
Finalized the UX for auto-configuration of the device tracking. It's essentially the same as animation immediately below except for a brief flash of blue when tracking config is touched by the app.
Another subtle UI change in Release 71.

When configuring or reconfiguring a backup, the app would automatically enable/disable device tracking when a location is changed to use a different drive.

Since this change happens behind the scenes, the app will attract the attention to the fact with this little animation. When hovered over, the tooltip for the pushpin button will explain what's up and suggest reviewing the changes.

My only concern is that this still might be a bit too subtle, but it's better than what's in there now... which is no notification at all.
New UI for configuring device tracking in R71.

This is one of those changes that look almost trivial when done, but behind the scenes take a disproportionate amount of time to actually flesh out.

In this case, the main problem was that the feature needs a detailed explanation and the question, of course, was how one can fit a bunch of text into a reasonably small space so not to have this text take over the actual configuration section.

The first iteration was the good old message box. You click on the "More on this..." and you get :

Nice, but looks like a hackjob, does't it? So, naturally, the next step was to spruce it up a bit - put text into a dedicated window, use RTF to add a bit of in-text formatting and a bit of animation to liven things up :

Nice again, looks better, but feels off. Too much going on. It looks like a conventional help window, but ... crippled. It's also not clear how to act when the user wants to resize the window - allow it, not allow it, restict minimum/maximum size, etc.

In short, what's lacking here is the cohesion. Luckily, the solution is right on the surface:

... whereby the text panel slides out on the "More info" click.

This is better, but now we have a window that sits lopsided on the desktop when the panel is open. It is also becoming wide, meaning that the app needs to mind the desktop edges, re-center the window, etc.

But if we just sleep on it, we might suddenly decide to make it an "either-or" window and then the UI magically becomes
(a) compact
(b) cohesive
(c) balanced
(d) with unlimited text space!

So there you have it - a new mini-help subsystem coming to Bvckup 2 in the Release 71, deployed on as-needed basis across the app.

Don't know about you, but I think it's neat, tidy and pretty damn clever :)
Added support for the reseller licensing to the app. Any license can now be tagged as being sold through a distributor, in which case Help/About will show distributor's name and link to its site.

Additionally, Tech Support and Homepage links from the Help menu may be configured to go to the distributor's pages.

It may not look like much, but it's the most important non-technical feature to be added to the app to date.

More on this to follow...
Another option, as per this comment. It's a really nice concept, but it requires a squint to realize what the icon is. Highly-detailed 16x16 px icons are hard to pull off.
Release 71 is going to reshuffle the contents of the backup configuration window a bit and location fields are getting equipped with a "device tracking" icon as a result.

In other words, for a network location the app will display the "key" icon (see /wip entry immediately below) and for local paths it will show "tracking" icon. Both fronting respective dialog for configuring the details.

The main question, of course, is what the icon should be ...
In a process of adding a way to configure network logon credentials for remote locations. Will be in Release 70.
The option | The result
Release 69 also adds an option of simulated backups or, as they are also known, "dry runs".

The option is accessible via the right-click context menu and, when invoked, it puts the backup through its preparation phase, including snapshot creation, scanning and planning steps, and then dumps compiled backup plan to the log without executing it.

Simulation also executes the archive pruning without making any actual changes to the destination.
Release 69 adds support for optional VSS writers.

VSS writers are the components of the shadow copying framework that are tasked with ensuring consistency of specific files in VSS snapshots. For example, when creating a snapshot, Windows would ask the MSSQL writer to flush any pending database changes in a way that would make on-disk copy of the database self-consistent.

Some of the writers are required and always participate in the snapshot creation process, but some are optional and they need to be explicitly enabled by the initiating app. Starting with R69 Bvckup will be enabling all optional writers and it will also provide a mechanism for excluding specific ones if needed.
It's shipped!
Here it is
Nearly done with the checkout flow on the website. Integrated with Stripe yesterday, now finalizing the visuals.

This is a purchase receipt. It is both sent out and shown on the site when the payment goes through.

PS. All numbers and codes are made up. Don't get any ideas :)
Working on the production version of the website.

Cleaned up overall look and feel a bit, got rid of baby blue theme and anorexic body font, reworked the copy, expanded Features section and so on. Still need to slice and code it. Also asked a couple of people for the feedback, so waiting on that too.
The feature everyone was so waiting for ... the payment form :)
Just expired | Licensing form | Manual/offline licensing

Working through the necessary evil - the app licensing :)

The plan is to have a 14 day trial that doesn't require any explicit licensing. Just install and run. It will pop up a notification half way through the trial period and it will also say something on the last day.

Once the trial period is over and until a production (paid) license is installed, the app will go into an unlicensed mode and it will activate two restrictions (see screenshot). It will remain largely functional, but presumably not convenient enough for regular or unattended use.

You would then hit /purchase page on the website, go through the purchase process and get an activation code in return. The code will then need to be sticked into the activation form on the client (see 2nd screenshot) and it will fetch the license from the server.

If the licensing server is inaccessible for whatever reason, there's a couple of ways to retrieve the license manually (see last screenshot).

For those with Beta licenses, they will be accepted at the checkout as a discount code.

For those looking at large deployments / enterprise setups, there will be an option for automating licensing process. Get in touch for details if you have questions.
Option A | Option B
Updated the About window a bit. Current design has somehow always felt ... cramped?
Full copy ( selected ), Delta copy ( selected )
Super-final version of the progress bar as seen in Beta 60. Note how colors of some parts are inverted under selection, took me disturbingly long to arrive at this solution :-/
Beta 60   - vs -   Beta 59
Finalized changes to the progress bar - now with more contrast, a bit of depth and beautifully centered splatter for partially copied blocks.

And here is a bunch of runners-up if anyone's interested.
Just waiting for the test build to compile... so here's another take on the delta copier progress bar (and here's existing version for comparison).
Clicking and holding error navigation arrow for 750 ms now activates auto-scrolling. It starts at 10 errors per second and accelerates linearly to 100 err/sec, which is pretty damn fast.

Additionally, holding Ctrl while scrolling with the mouse wheel makes it go in 10 line increments.
Beta 60   - vs -   Beta 59
It is now possible to clear the log view and then revert it back to the full log view.
A smaller UI change.

When hopping through errors in a backup log, log viewer will now briefly highlight next entry if it cannot be pulled directly under the cursor. This basically adds a visual terminator to the interaction sequence, which is a surprsingly useful thing to have.

Otherwise it works as before.
Having thought about it a bit more, I can even remove that slight upward slope from the hump of the blue graph. The usage still climbs just a little bit during the run, but for practical intents and purposes it's flat.
Wrapping up changes to the logging system. Here's a quick graph to demonstrate the effects of the rework. For an initial full run of a 100,000 file backup memory usage is down from about 200 to 50 Megs. For a 1,000,000 file backup the effect is similar - peak memory usage dropped from 2.9GB to 0.58GB .

Previously, the UI would keep logs of last few backup runs in the memory and for larger backups this would eat up a good chunk of memory. This was especially true for the initial backups where all files needed copying and thus for 100K files it would generate close to half a million log entries.

For subsequent backups it wasn't a big deal as they would handle just the changes and generate only a handful of steps and a bit of log data So in the end, the memory usage was capped, but the cap was higher than it could've been.

Beta 60 addresses that.

Only a very small part of the log is now kept in memory, just enough to fill the log viewer panel in the UI. The rest is kept on the disk and read from there on as-needed basis.

However this is easier said than done.

Bvckup logs are hierarchical, meaning that they are generic trees with nodes that may have an arbitrary large number of children. Running a 1 million file backup would yield a "Processing..." entry with 1 million children, each with up to a dozen of descendants.

Displaying this tree would've not been a big deal if all log entries were always visible at all times. When the UI would've wanted to show entries 700,503 through 700,523 entries, it would've simply gone to a respective location in the log index file and plucked the data out. But...

The nodes can be collapsed and expanded. This hides and shows arbitrary chunks of the log tree and then the UI has this issue of finding 700,503-th *visible* item. And this is hard.

This implies quick look-ups in some sort of super-imposed binary search tree of visible items. A tree that changes every time a node is expanded or collapsed and that sits in a disk file, which restricts how you can walk and modify it without getting an I/O penalty.

This is fun. This is what cost me 3 weeks of shuffling through CS books and several complete rewrites of the tree indexing code. Believe it or not, but this is basically an original research and it seems that no one had to deal with this issue before. At least not publicly.

The solution is to use slightly customized version of an AVL tree to store the list of node's children (that's using tree within a tree), use it for the visible item look-ups and then optimize the hell out of the index I/O through aggressive write caching.

When all i's are dotted and t's crossed, this yields a logging system that can push up to 1 mil entries per second on a resonably equipped box. This might not be knock-your-socks-off impressive, but it's plenty enough for the task at hand.
A quick update on what's happening.

Disk-based virtual data structures, that's what is :)

This is a part of an effort to further reduce and cap app's run-time memory usage. More specifically, it is meant to help the app cope gracefully with multi-million file backups that involve millions of steps and generate lots and lots of log data.

At the moment Bvckup's UI keeps backup logs in memory. It puts an effort to trim them to a reasonable size, but this still eats quite a bit of RAM, especially with large backups.

It costs about 120 bytes plus the size of an entry to keep a single log line in memory, whereby these 120 bytes help weave entries into a hierarchy (a tree) and track their state.

If we want to reduce the memory usage, then the most obvious optimization would be to keep the actual entry text in a disk file and load it on demand, when it's actually visible in the log viewer.

This certainly helps, but as the log grows, the memory usage still climbs. A million file backup generates about 4 million log entries. At 120 bytes per entry that's 0.5 Gig in just supporting structures. This is unacceptable.

So it means that the entire tree structure of the log needs to sit in a disk file and to be read from there as needed.

Disk-based tree structures are routinely used in databases and file systems to store indecies of data sets, typically in a form of a B-Tree and its variations. The kicker is that these structures are meant for storing sorted data and they optimize for searching. Trying to adapt them for storing unsorted data would be nothing short of fitting a square peg in a round hole.

Long story short - there are no ready-made code for storing and manipulating generic data trees on a disk, leave alone fast code. This appears to be an esoteric problem that everyone solves on their own.

The screenshot above shows a part of a small "tree database" library that I ended up writing, together with its own caching module, predictive page loader and a pony.

Took almost two weeks.

On a bright side though, this code is perfectly reusable for storing disk snapshots as well and this paves way for making the planner module work almost entirely off the the memory.

Once done, Bvckup memory usage will NOT depend on the particulars of a backup at all - this a very big deal and an incredibly improtant feature to have for any robust backup software.

So, stay tuned...
Coming up in beta 59 - automatic archive prunning.
If all goes well, it'll be out tomorrow.
Effects of delta copying as seen in Task Manager.
1.28 TB read, 0.37 TB written out - just the changes!
Ready to go. Launching tomorrow.
Coming up shortly, stay tuned.
After | Before
Switched delta copying progress bar from varying color of the partially copied blocks to varying their height. This should help with telling apart skipped, partial and fully-copied blocks.
Added an option for cloning an existing backup job.
After | Before
Website footer work.
In preparation for the launch of the Referral program.


This one - the *nix implementation of Windows Networking. If you have a NAS box the chances are that it's using Samba to support network access from Windows computers.

It is a good software, simpler and much snappier than its native Windows counterpart, but in case of NAS devices it typically runs on Linux. This means that the underlying file system is anything but NTFS. This in turn translates into no support for "created" timestamps and no support for certain attributes, including System and Hidden.

Moreover, the attributes that cannot be set or cleared depend on whether it's a dot file or a regular file. Who would've thought? :-|

From what I understand it *is* possible to set things up so that Samba would record these timestamps and attributes and make things transparent to its Windows clients, but in reality this doesn't seem to be happening.

Long story short - starting with Beta 49 Bvckup will now be explicitly testing destination file system to check whether c-times are supported and which attributes can be set.

Additionally, it will also explicitly test both source and destination file systems to determine the granularity of c-times and m-times. This is again thanks to some NAS devices that are customized to use 1 sec for m-times and 1 usec (micro) for c-times.
By popular request.

When ticked, Minimize button minimizes to the taskbar (as usual) and Close, the (x), minimizes to the systray.

When unticked, Minimize minimizes to the systray and Close exits the app.

Trivial in retrospect, but took me ages to realize that behavior of Close and Minimize can be controlled with just one setting.
Have a mapped drive? Want to run as Admin?
Good luck with that!
Second pass over the same sketch. Trimmed down decorations a bit, reworked content to have a summary, added footer, etc.

Full-length version is here.

What do you think? Let me know - @apankrat or over an email.
Started sketching production version of the website. As much as I personally like the minimalist version that's currently up there, it just doesn't seem to work that well :-/
When periodic backup misses its last scheduled run, it's now possible to tell the app how to handle it.

There are two common cases - (a) the app itself wasn't running and (b) the backup drive is a removable and it wasn't plugged in.

With certain setups it makes sense to run the backup as soon as the app can, to "catch up" with any changes and get them backed up quickly.

In other cases, a backup may be large and heavy and set up to run every midnight so not to disturb anyone. Running it ASAP will generally be not a very good idea.

Hence this option.
By popular request.
Hint 1 | Hint 2 | Hint 3 | Hint 4 | Hint 5 | Hint 6
Context hints for each section in More options.
Here's the screen-by-screen flow of the licensing process.

It starts with the above window and directs users to the website to pick and purchase a license code, which is given on spot and also sent in the receipt email.
Then copy-paste the code, click on Verify, get this -

If this works (the server is accessible, the license code is valid, etc), then this is it. Thank you very much, click Ok, you are done -
Otherwise, it explains the failure and offers either to retry the activation or to use an alternative -
There are two alternatives, both are simple -
For offline boxes and locked-down setups there's the Copy To Clipboard option. Take the URL to a networked machine, open it in a browser, get a licensing file back and then drop it into the app's config folder.

The URL is exactly the same as the one used by the app's own licensing code and it comprises of the license code and installation ID and the fingerprint.

Prepare the email option creates a new email with default email client and puts the Installation ID, Fingerprint and License Code in the message along with some pleasantries. The license file will be attached to the reply and it will simply need to be dropped into %LocalAppData%\Bvckup2\engine.
A bit of UI animation for the license activation window. This is a animated GIF, so the colors and the timing are all off. It looks tidier and faster in reality.
Licensing support. Yay!

This is one of those things that is rather simple conceptually, but translates into a lot of actual petty work.

When the app is started, it looks at several system properties including the CPU model, Windows' MachineGuid value and others, hashes them together to produce an "Installation ID". For example -


The ID stays the same under normal computer use, but it changes with hardware upgrades or when the OS is reinstalled.

The app also generates an "Installation fingerprint" by hashing each of the properties separately and stringing them together. This yields a longer hash -


Like ID, the fingerprint changes with hardware upgrades and OS reinstalls, but, unlike ID, it changes only in parts. This makes it suitable for detecting incremental changes to the user's computer.

Both ID and Fingerprint identify user's installation, the difference is that the ID is used on the app side and the Fingerprint - on the licensing server side.

Next, ID and fingerprint are submitted to the licensing server, either by the app itself or by a human through a website. When licensing for production use (as opposed to beta or evaluation), the request also includes a license code - a random token that is issued to the user in exchange for payment.

The server responds with a license, which is a short text file that binds together few pieces of information and seals them with an RSA signature of the licensing server.

License code: EVAL-8028-8316-0400
Installation: $c!f2ZoPxOlnpLWraJZ3F1nm7JcA
Expires on:   20/12/2013


For beta and evaluation licenses the "license code" is generated automatically by the licensing server. The large block of gibberish at the bottom of the license is the RSA signature.

When the app starts, it loads the license and verifies it using public key of the licensing server that is embedded in the app.

If the license is valid, the app works as normal. If the license is missing or expired or doesn't match the installation ID, the app disables certain features and periodically nags to do the right thing and what not.

Now, let say I upgraded my box. This screws up the Installation ID and invalidates the license. The app recomputes new ID and Fingerprint and sends them to the licensing server.

This is where Fingerprint comes into play. The server takes old and new fingerprints, chops them into individual hashes and does a "fuzzy" match. If enough hashes match, we can assume an incremental software or hardware change on user's end and re-issue the license for new Installation ID.

If fingerprints are wildly different, then we make fuss and require contacting human tech support.

In conclusion, needless to say that the whole scheme relies on the validation code in the app being trustworthy. If someone pries the app open and replaces selected IFs with NOPs, then obviously no licensing will work.

The answer to that is that the app should implement self-consistency checks and deploy counter-measures when these checks fail. This is a separate and very large and interesting subject ... which I have no plans on covering here :-) A good starting point here would be archives and then follow from there.
After | Before
Reworking About window a bit to include license information and associated trinkets (not shown). This will probably need another iteration or two...
Meet the log error navigation.

Basically the idea is to show back/forth arrows only on hover and make them pull next error to the spot where the cursor is. So to hop through errors you would just go click-click-click without moving the mouse.

This compliments well existing feature of jumping to the first error of the last run by clicking on "Xxx errors" in backup's main window entry.

As per previous post, this leaves a question of how one would get to the *nearest* error if none is currently visible and the answer is that one would simply use conventional scroll for that.

All in all, having played with this for a bit I think it's a good solution. If it doesn't seem like one to you, give it a try and it will grow on you.
Tying up few loose ends in the UI.

The above screenshot is the first option for navigating between the errors in the log pane -- see two red arrows next to the "x" in the pane header. The second option is this -

This time the next/previous arrows are in the error entry itself.

I like the second option better, because it's contextual and I think it's more obvious. However, first option is superior in functionality because it allows jumping to next or previous error when there's currently no error visible in the pane.

Still deciding...
Tweaked the progress indicator a bit.

For one, replaced the fixed-width byte countdown with a simple "% out of [size]". The countdown didn't work out, it was just... odd.

Also made the % adjust the number of shown decimal digits depending on how fast the copying is going.

For fast copies there are no decimals. For slower ones it shows one, for slower yet - two, and for the slowest (like the one on the screenshot) it goes down to 3 decimal places.

The idea, obviously enough, to always show some movement, but try and avoid being too excessive with it.
Added code to replicate folder timestamps. It's ON by default as it takes only several seconds to go through a hundred thousand folders, but it can be disabled if needed
Changes hidden (default) | Changes shown
Final version of the update window. De-emphasized the "View changes" button and tweaked the colors a bit.
Default view | Changes shown
Reworked the software update mechanism a little.

First, when the update server is processing an update check from the program, it now takes into account program's current version.

It uses this version to compile a consolidated list of all changes between program's version and the latest version available. It then uses this list to understand if there are any critical or important outstanding changes. This yields an importance indicator of the update, when one is available.

Secondly, both the list and the indicator are now sent back to the program.

On program's end, the indicator translates into the "Update is recommended" line. It is one three options with "optional" and "critical" being the other two.

"Recommended" is a routine update that adds this and fixes that. It would better be installed, but it's not the end of the world if it's not.

"Critical" is critical - "you'd be sorry if you don't install it" kind of update.

An example of "optional" update is a new version that updates program's French translation when the user is running an English version, i.e. an update that can be ignored altogether.

There's now also the "View changes" button that expands the window and shows a neatly formatted (RTF) version of the change list.

The biggest deal, of course, is that it's a consolidated list of all changes between the installed and the latest version, rather than a laundry list of all changes since the dawn of time. It makes the list infinitely more useful compared to the raw version.

Here's an older post on this that goes a bit deeper to explain the rationale behind these changes Version numbers don't matter.
Adding support for running external command before and after each backup.

Both commands have a configurable timeout value. Additionally, the pre-backup command can be tagged as critical, in which case bvckup checks the exit code and aborts the backup if the code doesn't match specified value.

The screenshot shows touching a TrueCrypt volume before the backup. This is needed, because TrueCrypt goes out of its way to preserve timestamps on its container files and bvckup sees such files as unmodified.
Reworked the More Options section.

It has already grown quite tall and it still needs to take on few more configuration options, so expanding it in place as it's done for all other sections doesn't work. The window gets too big and simply doesn't fit on a desktop.

Instead, it pushes everything up and out of the window and floods it with its own content. As I add more config options to the section, it will also expand the window down a bit if needed. All this should help keep all configuration within single window and also keep window's height in check.

Took me a while to converge to this solution, but it is a pretty decent way to structure it I think.
Second revision of the same thing.
Sketching up a dedicated device tracking section. Not sure if it's worth it, still have to try few things out...
Coming up - the reordering of list items.
Done with the inclusion/exclusion filters

The tree selector widget is a piece of art even though it doesn't look like one :) It is basically a front of an asynchronous parallel disk scanner that dynamically rearranges its scanning queue to first scan subdirectories that a user might be accessing next.

In other words, if you expand a directory, then the scanner will postpone scanning anything else and focus instead on all its subdirectories. So if you are making your way through a massively populated disk, it will try to pre-load what you are looking at and scan the rest on a lower priority.

Also, since the widget starts showing tree contents right away, it allows quickly navigating to and excluding a desired item without needing to wait for the entire scan to complete. Again, quite a bit of speed up with huge trees.
Advanced filter configuration. The pattern matching is all brand new and much improved over the disaster that it was in Bvckup 1. It also adds an option of matching on item attributes, both set and cleared.
Option 1 | Option 2 | Option 3
Finishing up the dialog for configuring inclusion/exclusion filters.

In inclusive mode (pictured) the tree view tags excluded items with 'x'. In exclusive mode, when "Start with an empty list" selected, the tree view tags included items with checkmarks.

Currently leaning towards Option 1, and not yet decided if to strike-out excluded items or not.
Added per-phase progress bars. Currently used for scanning and execution phases.
After | Before
Before/After of the backup config window as of Beta Release 20.
Collapsed | Expanded
Revising the backup config dialog to accommodate a description field.
Revising the main view. Description and the progress bars are back in.
Compact | Verbose
Well into the beta, just dotting the i's and crossing the t's.

One of the outstanding features is the archival of deleted items and this is the supporting UI for it.

Now deciding between a compact format of the config widget or a verbose one. Compact format has a single descriptive field that is re-used to show the explanation of a currently selected option. Verbose format has descriptions for each option visible at all times - it is likely easier on first time users, but it is also a waste of screen space on all subsequent uses.
What can I say?

Beta. Is. Good. To. Go!!!
A quick fix to the UI.

For very large files (say, 256 Gigs), there was a clear lack of UI acknowledgement of the backup progress. So the completed percentage got a couple of point digits and a byte countdown for the file.
Trying to keep it short and sensible.
Added the first draft of the Miscellanea section. Also renamed More Options button to Rabbit Hole, because that's essentially where it leads to :)
The new and much improved delta copier is in.
( the original version is here )

Significantly improved performance through multi-core, parallel hashing, asynchronous I/O and more aggressive caching of the state data.

Strengthened the change detection algorithm through addition of the whole file hash. When all individual block hashes come back unchanged, the algorithm now uses the file hash to confirm that the file has indeed remained unchanged.

Combined with the original use of two separate hashes per data block, this eliminates the need for a precautionary full-file sync every N copies.

Improved handling of cancelled and aborted delta copies by introducing support for partial updates.

The algorithm now remembers how far along the file it managed to proceed and stashes this information for the next run.

When the copying is cancelled partway through, the file is marked as out-of-sync, but the delta state is updated to capture what has been already done. This comes very handy and speeds things up when updating very large files.

Improved handling of large file counts.

Version 1 maintained an in-memory index of all delta copied files and, surprisingly, this didn't scale well. It also kept all delta state information in a single folder, which too led to some pain and suffering when the file count grew large.

New version eliminates both these problems.

Lastly, all copier parameters - block sizes, hash algorithms, cache buffer sizes, threading parameters, etc. - are now configurable on a per-job basis.

All in all - THIS IS IT.

This is how a simple and efficient delta copying should be done. Can't wait to release it to the real world.
Shadow copying is in, for those pesky locked Outlook mailboxes and other "in use" files.

There are three big things left to weave into the program - file system monitoring, new delta copier and software updates - and the beta is good to go. A matter of few days now.
If your message box includes the "Don't ask again" option, just make sure to disable all but one button, when it checked.

Oddly enough, I don't know a single app that does that.
On light | On dark
A system tray icon sprite sheet.

Feeling creative? Make your own version, save as a PNG, drop it in a config directory and - voilà - your systray looks just like you want it to.
There are two sets of icons on the sprite sheet. The first three columns is the XP style, which is a bit puffy and matches well older Windows styling. The last three columns is a flat version that goes better with native W7 tray icon style.

Each icon comes in 3 sizes - 16x16, 20x20, 24x24. This is needed to accommodate 3 common DPI levels, see this earlier entry for details.
Final | 2-framer | Shaker 1 | Shaker 2
"Message pending" indication, whereby a message can be either an error, a warning or a notice and these are differentiated with the ! color.
I realize it's not color-blind friendly, but the goal is to attract the attention and make user bring the app window up to read the message --(therefore)--> reflecting the message type via an icon level is not exactly critical.
W7-ish version as per a this hint on Twitter. Not bad at all.
Working on systray icons...
Revised Preferences window a bit.

Added basic control over backup scheduling and switched from "close to systray" to "minimize to systray" - the (x) at the top right corner will now always act as an Exit, just as it is supposed to.
More of the UI work.

Custom pop-out button that stays nice, flat and out of the way when idle, but acts like a regular button otherwise. Not exactly a UI innovation, but it was nice to be able to use it in a real project :)
Option 1 | Option 1/b | Option 2 | Option 2/b
Took out the engine mode switch from the Preferences and into a dialog of its own. Still need to understand if it's worth mentioning that there's an engine per se or just stick to more amorphous "application mode"...
Option A | Option B | Option C | Option D
Looked at few options for splitter styling. First one is final. Tentatively though... heh.
Done with the split-view mode for the main window. It's one of those things that everyone takes for granted and that, in reality, maps to quite a bit of work if done right.

For example, resizing window by the top edge should resize the top pane and leave the bottom part alone.

Likewise, resizing by the bottom edge should expand/compress the bottom pane, but not the top one.

However, if we are compressing window by the top edge and the top pane is at its minimal height, then we should the bottom pane starting to compress.

All good so far, but, say, we change our mind and start expanding the window back (without releasing the mouse button).

We should see the bottom pane brought back to its original size first, followed by the expansion of the top part... even though we are, technically, dragging by the top edge.

This sort of UX detail is something that few people will notice if it's done right, because it feels natural and aligns well with expected behavior. But cut some corners - and you bet everyone will notice.
Option A | Option B | Option C | Option D
Going through styling options for the expand/collapse buttons.

Presently leaning towards C or D, because I'm considering making the text selectable with the mouse for easy copy- pasting. For that to work the buttons need to be as unobtrusive as possible.

Still WiP though...
Working on the backup log viewer.

Basically the idea for the log is to support multiple detail levels, each capturing progressively more context that its parent.

With all details collapsed, the log gives a quick overview of when the job was running and if it completed OK or with errors.

Expanding details allows for a drill-down - to look at the details such as the file and byte counts, the timing and performance data and the details of any errors that have occured.

I'm still going to shuffle things around the window, e.g. push the -/+ buttons to the right, tweak the styling too, but the general hierarchical format will stay. It is a very convenient way of organizing the log info.
Finished a piece of code that determines effective timestamp resolution of a file system.

Every file system keeps track of when a file was created and when it was last modified. These timestamps are the natural properties of a file and they are readily available for inspection in the Windows Explorer or any file manager of your choice.

The less obvious aspect of timestamping is that timestamps have resolution. For example, FAT tracks file modification with a measly 2 second precision. In comparison, NTFS uses the 100 ns (nano-second) precision.

It is very typical for a backup software to rely on timestamps to determine if a file has been modified and requires copying. But if the source file sits on an NTFS volume and the backup goes onto a FAT disk, then comparing timestamp directly simply won't work, because FAT timestamp will be rounded-up to a 2-second mark.

In other words, the timestamp granularity needs to be taken into an account when comparing the timestamps. The question is how to determine what it is exactly.

First of all, the granularity for creation and modification times can be different. FAT has them at 10 ms and 2 s respectively, but NTFS has them both at 100 ns.

To complicate matters, there appears to be NAS devices that look like NTFS boxes, but with the granularity that is not 100 ns.

So what I did is added code to probe the file system and determine effective resolution for both timestamps. This involves dropping a temporary file and then trying to set its timestamps to this, that and 3rd and see what they end up at. From that it's possible to deduce the resolution.

In cases when such probing fails, the app falls back to guessing resolution by the file system name. It is also possible to override the resolution values from the config, just in case.
Done with the backup scheduler. Comes with a couple of neat improvements over v1.

The Go and Stop buttons work almost exactly as in v1. If the backup is running, click Stop once to Pause it, click again to Cancel, click third time to disable the job. If the job is disabled, click once to enable it, click again to manually start it. And so on, you get the idea.

First improvement over v1 are the Go! and Stop! commands that are activated by holding Ctrl down when clicking on the toolbar buttons.

Go! moves the job straight into the running state.
It's the "Run it. NOW!" command.

Stop! stops and disables the job straight away.
It's the "Stop it. NOW!" command.

Second improvement is that multiple jobs can now be run in parallel. Generally, this has a negative impact on the overall performance, but there are certain cases when this is desirable. One example would be running a smaller backup in parallel with a much larger and slower one.

There are three ways to do this:

  • With the Go! command.
  • With a per-backup setting that makes the job always run right away, even if there's another job is running.
  • With an app-level setting that does the same, but for all jobs.

PS. I considered adding something more elaborate, e.g. allowing assigning jobs to "run groups" and then allowing only one active job per group, but that seemed like an overkill, so I dropped that.
Another pass over the main window chrome and layout. The "backup jobs" section is just a dummy header ... and it is confusing, I think.

† This is an app screenshot, not a sketch.
Terse | Two-liner | Four-liner
Exploring options for the main window format.

Specifically, I want to get rid of the V1's Summary panel, trim the information contained in it a bit and fold it directly into the main window items.

In conjunction with this I'm also considering having two modes for displaying backup jobs on the list - "terse" one-liners and "extended" multi-liners. What I still need to figure out if this should be a per-item preference or an app-wide setting and how one would switch between these display modes.

Work in progress, obviously. Any thoughts - let me know.
Done with a little animation that and switches main window between a "welcome screen" and the regular backup list view.

The animation and fading effects can be disabled, there's a setting for that.

Windows comes with a standard toolbar control, but it's an overkill if you need just a couple of buttons at the top of the window.

It basically takes longer to figure out how to use and customize it compared to creating a basic window and sticking a couple of custom buttons on it.

Had I have a need for toolbar ribbons, docking, on-the-fly customization, etc., then ToolbarWindow32 makes sense. But if it's something simple, then there are faster options.
Meet the toolbar.

To explain a bit what you see in this masterpiece :) -

The idea is to try and keep the menu/toolbar area simple and flat, but still provide some sort of a visual clue that Go and Start are clickables.

Here are three options compared back to back to make the differences a bit more obvious -

Here are few options that were leading up to the above one:

iOS says Hi

Also, considered disposing off the menu bar and folding whatever few items it has into the drop-down menu. But this just looks weird and too alien on Windows, the app needs a menu bar.

Dropped the idea.
Option A | Option B
The look of the main window with no backups configured. Tried using a stock button, but it looks heavy and dull. Hopefully, this is a bit more refined.
Shorter | Iconic | Longer A | Longer B
Trying to understand how to indicate if a selected path points at a removable storage.

The app tracks storage devices being plugged into the computer and it adjusts backup paths if a known device re-appears under a different drive name.

I can, of course, not bother with this at all and simply spit out a notification when (and if) the drive change is detected, but then I think it's an important functionality and it'd better be mentioned during the setup.
"Work-in-progress" buttons for beta releases.
Correct your input.

† Idea is courtesy of the OS X login screen, I think. Though I did experiment with concise error indication quite a bit.
Wrapping up the Periodic Backup configuration window.

Unlike the v1 that always stored the backup interval as a number of seconds, the v2 adds a time unit specifier.

This simplifies quite a few things in the UI and yet it keeps the configuration flexible enough to support 1.5 hour backups by allowing to specify a 90 minute interval.

On an unrelated note, killed few hours trying to utilize native Windows controls for the interval configuration. The amount of work required to make them look reasonably good is as depressing as it is remarkable.

For example, the editbox control doesn't natively support vertically centering the input text. It just does not. It is meant to look like this -

Want "120" aligned to the label on the left? Sure thing, that'd be 2 hours of plowing through documentation only to realize that it must be... drum-roll... a multiline control that you will then need to manually pad the right amount at the top to push the line to the middle.

But that's peanuts compared to trying to beat some sense into the dropdown box. Long story, but making a box taller and keeping the line centered requires taking over the control drawing functions, which in presence of Windows themes is a delightfully unpleasant experience.

In any case, turning the top control line into the bottom one -

costs exactly a day of work, while it really shouldn't. Welcome to Windows.
Finalized the design of inline hint/tip widget for input controls.

Initially, it was meant for the config window that hosts periodic backup settings (as seen in a couple of posts below), but having talked to others and slept on the issue I think that the natural language input (i.e. requiring to enter "1 week" or "15 minutes") is not the best option for this case.

An alternative design is coming up in a bit.
Coded the previous sketch, played with few parameters and it is starting to get there ... I think. Will make another pass over this tomorrow and it should be good to go.
For what it's worth I have also gave a try to the standard Tooltip Control. In fact, that's where I started.

A regular tooltip looks like this -

It adds another layer to the windows stack, breaks the visual flow of the layout and still manages to tuck itself under the mouse cursor.

There's also a balloon version of the same, which looks like this -

It is marginally better, but just look at those lovely rounded cornersi. Eat that Apple. Real artists ship.

tl;dr - the standard Tooltip Control is not very good, so I had to come up with a custom replacement.
Not sure about this one, seems a bit clumsy and a bit too custom.

This is basically a way to explain what is expected as an input for a particular configuration value. There's gotta be a cleaner way of delivering the same info...
Compact form | Fully expanded
Working through the supporting code for the Settings window.

Done with collapsing/expanding mechanics and respective fading and moving animations. Still need to revise the copy for the explanatory bits and massage in the option for archiving deleted files.

Not to pat myself on the back, but I think it came out looking pretty decent.

That's not to say there's no way to turn off all this fancy styling and animations, because, you bet your favourite monochrome flat design, there is.
The Backup Settings window. Final ...for now :)
Revision 2.a | 2.b | 2.c | 2.d
Another take on Backup Settings window, 4 styling variations.

Further compacted vertically, but now it appears to be somewhat cluttered and lacking comfortable vertical rhythm (to borrow a term from the typography).
Basic | Overview | Expanded section
Working on the Backup Settings window.
I think the original design from the version 1 got many things right.

It provided a good at-a-glance overview of the backup configuration, split settings logically and tucked away the minute details. It did have one problem - it was tall. So tall, in fact, that I got several reports that it didn't fit on screen :-|

V2 adds several new options to the backup config, so this doesn't make things any easier. After looking at few options, including paged, tabbed and two-column, screenful layouts, I circled back to the v1 layout.

To compress it vertically I've done two things - got rid of the group boxes (frames) and collapsed radio button groups into a single line with the current selection.

This gives us the middle screenshot (pic) that shows the config window in the "overview" state.

Clicking on a "Change ↓" button expands the radio button group and allows changing the setting without leaving the page. Latter is pretty nice to have, because ... well ... because it's simply faster and more convenient.

Expanding a section gives the screenshot on the right (pic). To collapse the section, there's a little button in the section header on the right hand side. I'm not 100% convinced it's really needed nor that it's the best solution, but that's how it is at the moment.

I forgot the "how to handle deleted files" section, but it will be in the beta. Not to worry.
Unchanged | Modified
Meet the Preferences.
The Backup Engine section - as simple as it looks, it is fronting a couple of months worth of work and it's easily one of the most complex part of the v2. This is what detaches the user interface from the guts of the program and lets latter run as a separate process.

The Maintenance section - I figured that with the backup being an important piece of software, very few users would be interested in fully automatic updates. Furthermore, quite a few people like to retain even tighter control over the software and, for example, do not like it when the software talks to the Internet whenever it wants. Hence the new option -

Don't check, but remind to check for updates

It's basically a kitchen timer that goes "Ding!" every week or two (configurable) and prompts you to let the app check for a new version. We'll see how it goes, but I think it's a sensible compromise between the privacy and functionality.

The More button transforms the window into a comprehensive list of all available options, close to two dozens now and still counting. The idea however is to keep the defaults reasonable so that there won't be much need to venture into this part of the app.

Lastly, note how the Apply/Close buttons change when the settings are modified. I have no idea why this is not a standard behavior, because it really should be.
A notice | A warning | Alt. dismiss
A message bar is a unobtrusive way to display one-liner messages that would've otherwise required poping up a message box.
When a program detects that it was just restarted with Admin privileges, it has a limited number of options for communicating this joyous news to its user.

#1 - A message box. Bluntly put - I hate them. They are always too sudden, they steal your attention and stuff whatever they want to say down your throat. They ... lack manners.

#2 - A system tray notification. This is less intrusive, but perhaps it's too out of the way. Also, the systray is a shared area so the program is always competing with other apps for the same screen real estate.

#3 - Something else that is right in your field of view if you are looking at the app, but that doesn't get in a way of whatever you are doing at the moment.

Cue in the message bar, now an integral part of the Bvckup UI. It is used for displaying various status notifications right after the launch, after switching the engine to/from the system service mode, when discovering software updates and few other things.

I have also tried the self-closing version of the bar that dismisses itself on a timeout. Like so -
and so -
but having lived with this for a couple of days, it now seems like a solution in a search of a problem. So I shelved it for now. If there's any a real need for this, it'd be very easy to add later on.
Starting up... | Page 1 | Page 2 | Page 3 | Page 4
Nailed down the installer flow. A couple of things of note here.
The most unconventional part of the setup is how it handles the configuration of the installation parameters.

The configuration page (see Page 3) is very bare and focuses on how the program is going to be used instead of being a mixed bag of the minute details of the setup.

The target location is made into a secondary setting with its value controlled by the usage preference. There's no field for the Start Menu group name. There's no ever popular "Run the program after the setup completes" option either. Instead, exiting the setup automatically runs the program (see Page 4), unless the installer window is closed through an (x) at the top.

This leads to a simpler and more intuitive setup flow... or at least that's the idea :)

The installer supports both privileged (administrative) and unprivileged (regular user) installations.

If you are logged in as a regular user and have no rights to modify C:\Program Files, there's no reason why you shouldn't still be able to install a software, just for yourself.

While other installers routinely ask if to install the software for everyone or just for one user, it's a superficial question. It merely controls if the Start Menu group is created under All Users or your personal profile. The program files still go into a system directory and this requires administrative privileges.

Bvckup implements true system- and user- deployment, which is akin to deploying the program in /usr/bin or ~/bin in Unix terms.

In fact, on a multi-user system it is perfectly possible to install one instance of Bvckup for each user and they will all happily co-exist with each other.
The first page of the installer.
Working through the details of the update mechanism.

This is one of more difficult parts of the application, even though on the surface it looks uncomplicated. If we run the installer and it detects that the app is already installed, it should just update the installation. How hard could it be?

Let's see.
If the program is not running, then it's indeed a simple matter of replacing the program file with a newer version. The fun starts if the program is running.

On Windows, running a program locks its file, preventing any changes from being made to it. This is not required per se and there are operating systems that don't do this. There are also custom PE loaders for Windows that can take an executable and create a running process out of it without any locking fuss. For example, bo2k could do that back in 1999. See its source code for details if interested.

To complicate matters, there might be more than one process running off the same executable. For example, if several users are logged into a Windows box and each runs a copy of a program.

Therefore, the first order of business for any installer is to shut down all running instances of the program. A well-behaved installer should also restart all these instances after it's done with updating.

The shutting down part is not a big deal. There's more than one way to do this and all of them are fairly straight-forward.

It's the restarting that is contrived.

Not only program instances may be running under different user accounts, they may also be running with custom command-line arguments, as Administrators or system services. Restarting this cornucopia of awesomeness is not an envious task.

Here's how Bvckup works around this mess.

Every Bvckup executable has a corresponding named event in the Global namespace. The event is created by the first instance of the program when it's launched, and it is opened by all other instances.

The name of the event includes the file ID of the executable file. This groups running instances by the file they are using, because, after all, the installer is interested in a specific executable and nothing else.

For example, the following is an event name used by the Bvckup installation on my development box -


With this in place, the installer starts off by signaling the event.

In response, a running program makes a temporary copy of its executable file and starts it in a special "relauncher" mode. And then it exits.

The "relauncher" process opens the "update" event and simply spins there, waiting for it to become unsignaled.

Meanwhile the installer waits for the program file to become unlocked. Once all running instances replace themselves with the relaunchers, the executable frees up and the installer proceeds with the update. It then clears the "update" event and exits.

This releases the relaunchers, they restart program instances and - ta-da - the update is completed.

Techincally, instead of adding a special "relauncher" mode to the program it's possible just to have a small standalone relauncher program. It comes down to a matter of taste. Some people don't like to spawn from temporary executables, and I like to keep my file count to a bare minimum. To each his own.
Error #1 | #2 | #3 | #4 | #5 | #6
Dotting all i's and crossing all t's, of which there's plenty now that there's an optional system service that just begs to be stopped, restarted and deleted from under app's feet.

Doh... I missed the disabling of the service. Will add it tomorrow.
Vector icons for the website.
This is my first time not using pixel images to decorate a website. Instead, all icons are saved in the SVG format and converted to a webfont, which is then used to style regular HTML characters into icons.

For example, here's a font with the above icon set - VectorIcons.ttf

The advent of webapps like IcoMoon and Fontello not only made the process of building @font-face kits an absolute no-brainer, it also gave an easy access to hundreds of existing vector icons, many of which are free and of production quality.

When coupled with now virtually universal support for CSS text-shadow, box-shadow and gradients, this almost completely eliminates the need for raster form of smaller decorative and nuancing elements.

This is a very big deal and a great time-saver.
Still | Animated | Earlier "final" version
Re-finalized the sign-up form styling and mechanics.
Few styling options for an input control. Leaning towards top right or bottom right, but still need to see what works best in context.
Final design | Starting point
More or less done with the Features section on the new website.

Probably missed few things, but that's easy to add later on. The biggest time sink was trying to inject some visual hierarchy into the list, but without going against overall minimalist look and feel of the site.

Might still revise it after sleeping on it for a couple of days :) Won't be the first time.
Plodding through some grunt work with the new website. An almost equal split between working on a copy, javascript and styling.

As simple as the screencap looks, it came with two hurdles that needed solving.
The header font, Alte Haas Grotesk, didn't display correctly after it was converted to the @font-face kit. Specifically, the lowercase 'o' looked like this:

The font's free, but I couldn't find a copy of its license, so I started tracking down font's author to ask a permission to edit the outlines.

While waiting for a reply, I found another workaround which involved generating the webfont kit from OTFs instead of TTFs. Not a big deal, but it still took over 3 hours to resolve.

The second issue was with the slide-out, fade-in animation. Sometimes the slide-out didn't extend all the way down and sometimes it overshot. This traced back to the fact that slide-out height was computed before the OpenSans was fully loaded and the script fused basic sans-serif to compute the height.

Again, not too complicated, but, again, cost an hour to work through.


Little bit of this, little bit of that - all unplanned and it all adds up.
Nearly done with the new website design. Here's a snippet of how the site's header interacts with scrolling.
Option 1 | Option 2 | Option 3 | V1 (careful)
First pass over UI styling for the main window.
Released a small playable test of the directory scanning module.

It appears that using as many threads as there are CPU cores and traversing the tree in a depth-first fashion (scan child directories first and sibling directories second) yields the best scanning times.

Test and see for yourself - bvckup2-demo2.exe or 64 bit version.

Also, see here for details.
Updated @bvckup background with V2 branding.
2nd sketch for redesign. Larger version is here.
1st pass over the redesign of Larger version is here.
Current design is almost two years old and it grew steadily disconnected from the simpler aesthetics I am pushing for now. So it needs to be redone.

I will probably simplify this even further. I've been wanting to do a true minimalist site for a software product for a long time now and this seems like a real chance to do just that. Yay.
This is a sprite sheet of all raster images that are going into the run-as-a-service demo.
Interestingly enough, the total size of individual images saved as 32bit PNGs is 63,127 bytes, while the size of this sprite sheet (without checkered backdrop) is 63,829. After factoring in the size of the code that is needed to parse the sprite sheet, it really becomes a no brainer to keep every image in its own file and embed them into .exe's resources that way.
Static | Animated
Added a "crash" handling and reporting facility that includes several important changes compared to the original version:
1. The program now differentiates between three types of errors and handles them slightly differently.

First, there are unexpected operating system errors, i.e. when a core Windows function fails in a way that it shouldn't be failing. Classic example (that everyone is aware of) is TrackPopupMenu that is declared as returning boolean, but actually doesn't shy away from returning an integer under certain circuimstances.

Secondly, there are inconsistencies in app's own code, whereby one part assumes certain usage that is not observed by another part. These are so-called "assert" failures. The original version has three critical issues, all of this particular kind.

Lastly, there are straight-forward crashes. This is when a program is trying to operate with data that is no longer there or corrupted. This is the worst of them all as it typically indicates a major screw up on programmer's part, but in some cases it may also be an indication of failing hardware, like a faulty memory stick.

2. The program now generates a minidump - a small-ish file in a standard Windows format that captures the context of a failure. Minidump files dramatically simplify investigation of a problem on the developer's end.

3. Finally, the minidumps can now be uploaded to the support server right from the error notification window with one click. The transfer is based on WinInet API as this is supposed to improve the chances of getting the report through in proxied and otherwise "enterprised" environments.

For cases when the upload fails, the error dialog also offers an option of opening a link in a regular browser and reporting a problem that way.
64bit version is stored inside of the 32bit executable, from where it is extracted and launched to replace the 32bit version if latter detects that it is being run on a 64bit system.
Pro: a single executable to distribute
Con: the exe comes out weighing twice as much... or does it?

The size of a 32-bit executable can be easily reduced with the help of tools called executable packers. They work by turning the exe file into a self-extracting archive that also automatically launches the original exe after the extraction. It may sound simple but in reality it involves bundling a custom PE loader with the packed executable (as it might not be possible to extract the exe into a temporary file before running it).

In any case, exe packing is routinely used by developers who care about the size of their products, with uTorrent being one of most notable examples. Original Bvckup was also compressed, owning its tiny size in part to the magic of UPX.

The problem lies with 64-bit executables. For a variety of reasons there are presently no exe packers that support them.

However if we store 64bit binary inside of the 32bit executable and then pass latter through a compressor, we end up with both being compressed. Ta-da!

But there's more.

When an .exe is compressed, the compressor leaves the primary application icon intact. This is done so that the app would keep its original appearance in Windows Explorer and not sport a generic "program" icon.

Starting with Vista, the app is expected to support a variety of icon sizes that quickly add up. For example, bvckup's icon weighs in at about 110KB. That's almost half as big as the actual code, the code that does quite a few meaningful things on top of just looking great :)

So, if we do the 64-in-32bit packaging, we end up compressing 64bit version in its entirety, including its icon, and arriving at a greater size reduction if we were to compress the exes separately.

A page from the first demo of Bvckup 2.
Working on a shared memory buffer interface, a part of the IPC module, which is what enables the separation of the engine and the UI.
Option A | Option B
A bit more humane approach to critical error reporting.
The message is a homage to NetPositive, the BeOS browser that pioneered the technique of using haiku in error reporting. In fact, the message in option B is from the NetPoisitive original list.

The icon is that of a Zen circle, which has a plenty of meanings, many of which applicable given the circumstances.
Three-frame animation of engine's heartbeats for the demo.
Page 1 | Page 2
Final UI styling for the "Run As A Service" demo.
The "Run as an Administrator" icon.
Test of a fading transition, part of a small library that deals with supporting UI animations.
Fading animation dissolves a part of a window into background, in preparation for transitioning the window to another state, e.g. moving to the next page in a wizard or resizing the window into a more compact form.

As simple as it sounds, these sort of animation is fairly complex to implement on top of the Windows API.

Windows includes native support for window transparency in a form of layered windows, but it comes with a list of limitations that aren't exactly practical or easy to work, except for several specific cases.

The simplest working solution involves taking a screenshot of the window, creating an overlay window and then using it as a canvas to render the transition from an empty background to the screenshot. All the while making sure that Windows doesn't sneak in an unsolicited window refresh that would lead to a flicker, which in itself is a very "fun" thing to debug.
Every application icon that is exposed to the operating system must exist in at least nine sizes.
16, 32, 48, 256 are more or less expected. These are used in "small", "icon", "tile" and "extra-large " views in Windows Explorer.

20, 22 and 26 are possible sizes of an icon used in the window caption. The exact size depends on screen's DPI level.

And 40 (along with 32 and 48) is the size of an icon used in Alt-Tab view on W7, again depending on the screen's DPI.

If you don't provide an appropriately sized icon, the OS will grab one that is slightly bigger and shrink it down, ruining the pixel polish.

Lastly, each icon ideally must be saved in 32, 8 and 4 bits per pixel formats, so to be compatible... you know... with EGA cards

It would've not that big of a deal if it weren't for the size of a resulting .ico file. For example, the above icon ends up being ~ 120KB, whereas the actual .exe it is attached to and that does quite a few meaningful things sits at around 96KB.
Similarly, every bit of graphics in the app must be done in at least three different sizes - one for each common screen DPI level.
DPI stands for dots-per-inch and it's a metric of a pixel density on the screen. The smaller the screen, the more pixels are packed into a square inch, assuming the resolution stays the same. The difference in DPI between a large LCD panel and a 12" notebook screen can easily be a factor of two or more.

This means that if a program renders 9px text on both screens, on one it will appear twice as small as on the other one, and might command a need for a good magnifying glass.

So naturally it's a good idea to make the OS aware of the actual DPI of the monitor in use. If it's too high, the OS can do few things to make things visible with a naked eye again. Like increase size of the default system font and make windows generally larger.

This can be done automatically by looking at the monitor .inf files or by letting the user configure the DPI manually.

XP defaults to 96 DPI and with some difficulty it can be changed to 120 DPI. Vista and newer versions are better. They come with four predefined levels - 96, 120, 144 and 192 - and they also support custom settings.

This brings us back to needing multiple versions of every graphic in the app.

Vista is the first Windows version to routinely use monitor-specific DPI values out of the box. As such 96 is no longer a dominant DPI level and the apps are now expected to take the DPI into an account when rendering their content.

The higher the DPI, the larger the graphics and the fonts the app should use. The penalty for not doing this is an automatic stretching of the app window that inevitably leads to a blurry appearance and a host of other lovely issues.

In other words - not complying with OS expectations is not really an option. Hence the need for 3+ differently sized copies of the same pixel art.
The UI sketch for run-as-a-service demo.

Trying to find a balance between using custom styling and not getting to far away from the native OS look and feel. See also 8 other options leading up to this one.
Illustrative icons for the same demo.
First pass over the demo UI.
V2 running as a service, an actual screenshot for a change.
Option A | Option B | Option C
Exploration of measured blending of native and custom UI styling.
Option A | Option B
An idea for a space-saving tab control. Shelved.
An idea for an in-form buttons.
Option A | Option B
Exploring full-on customization.
Option A | Option B
Earlier sketches of heavily customized UI.