Development blog RSS feed

To that end, it spreads each sample across several horizontal bands, each displaying a count of full GBs, MBs and KBs, plus "leftover" bytes in each sample.

If we want to display, say, 12345678, then it's 11 MB + 792 KB + 445 bytes, so the band closest to the zero line shows a bar filled to 445/1024, the next band - 792/1024 and the 3rd - 11/1024.

This is basically a mod-1024 factorization of a sample value.

It is also a discrete (rather than continuous) variation of a log scale, which is very commonly used for "squeezing" large values into more manageable ranges.

In practical terms this allows us to quickly see how different any two samples are, even if they are very large.

For example - These two samples are both in the MB range, very close to each other, but looking at the KB band it's immediately obvious which one is larger and approximately by how much.

This sort of analysis is often needed when trying to spot jitter that is order of magnitude smaller than absolute sample values.

In our case, this need surfaces when analyzing the effects of OS file caching on bulk IO performance.

Here's a 16 GB file being copied using Windows Explorer: Here's the same file being copied with a custom, somewhat better optimized code: * Both tests are run with the file cache purged.

Windows Explorer makes uses synchronous IO that goes through the Windows File Cache. This delays writes and it also adds jitter to read operations.

In comparison, the second graph is for the code that uses asynchronous IOCP-based copying engine that bypasses the OS cache. The resulting data flow is far more stable and predictable. It's also substantially faster.

Download

Here's a dev build if you feel like seeing this thing in action:

io-monitor.exe
~ 72KB, no dependencies, signed. Just save somewhere and run.

It defaults to monitoring of the C: drive, but another drive can be selected by clicking on "Disk C:". The sampling rate is 500 ms and it is fixed for now.

The Purge cache option will clear the so-called Windows Standby List, which effectively empties the OS file cache. Use with care, obviously.

Happy graphing :)

The storage stack scanner is adapted from another project of ours called Diskovery and the IO profiling logic is based on the data gathered through CCSIO Benchmark.

While the scanner part is anything but trivial, the most time-consuming part, as per usual, was the UI.

There's a lot of information to be displayed - both the properties of volumes and devices, and how they relate to each other.

As you likely know, a device may host multiple volumes and a volume may span multiple devices. So in a most general case it's a full-mesh, which is the reason why we can't display volumes and devices as a simple hierarchy.

So, after going... ... through ... ... some ... ... options ... ... it became obvious that having two lines per entry was a way to go and that volumes and devices should be on separate lists, both visible simultaneously.

After a bit more sketching the lists arranged themselves into two columns and the scroll bar of the left list went to the left of it: * I must say that the idea of placing that damn scroll bar on the left took waaay too long to converge to. For whatever reason it's just a strangely un-obvious thing to do...

In any case, the result is a layout where we can display relations between volumes and devices with literal, drawn links.

After that just needed to take care of smaller details like scrolling slits... out-of-view arrows... and connector points... and, voilà, we now have something that is both usable and reasonably good-looking. Still to be added is a separate pane for displaying further details - device hardware info, SMART data, etc. This pane will sit at the bottom of the window and will switch between showing volume and device details depending on what's selected above.

Stay tuned for the updates :)

This is a new and improved progress bar that is making its way into R79. The idea is to use the vacant space on the right-hand side (the "todo" part) to show the sub-progress.

In comparision, here's a lively version we have now: † This is a GIF, trust me.

In Bvckup's case a progress bar also needs to accommodate the delta copying. In particular, the bar should be able to display processed file blocks that were copied in full, copied in part or not copied at all.

So here's a short list of options to achieve just that: Will probably go either with the 2nd from the top or the 5th from the bottom. Maybe both.

It's an older API that doesn't correctly handle inheritable ACLs and its use is not recommended. Not by the API documentation, mind you, but by a random blog post from one of many Microsoft development teams, which you still need to find first. What they recommend instead is Get/SetSecurityInfo.

And you'd still naively think it's the one.

Nope, it's not.

The problem is that SetSecurityInfo will fail with "access denied" if you are trying to set DACL and the file is not opened with READ_CONTROL.

You may wonder why the heck it needs read control... excellent question. That's because it wants to look up the name of the file and then *feed that name into SetNamedSecurityInfo function*. The latter will then re-open the file and start spraying kernel with requests, which is the last thing you need when you are already quite busy with copying a million of files from A to B.

Looking at the trace of SetSecurityInfo with Procmon shows that it generates over a dozen of kernel calls... whereas only one is really needed.

The name of that one call that we want is NtSetSecurityObject, but it's a part of so-called Native API and as such it is officially off-limits for any application use.

However, if we are to comb several pagefuls of relevant APIs, we may spot something called SetKernelObjectSecurity, which looks suspiciously like NtSetSecurityObject.

Long story short, you want to copy security attributes - use Get / SetKernelObjectSecurity. So now you know. Congratulations :)

The answer is that alt. streams is as a generic and more flexible replacement for an older "extended attributes" mechanism. It is meant for the cases when you may want to pin an extra bit of info to a file (or a folder), the info that is application-specific, but that doesn't belong in a file itself.

The book example is a small tag that browsers add to executable files downloaded from the Internet. This tag is used by Windows Explorer to show its "Ugh-oh, something from the Internets" warning when you are trying to run such files. The tag is stored as an alternate stream called Zone.Identifier. You delete this stream - you won't get the warning.

Long story short, Bvckup 2 now can replicate these secondary data streams. This option is in Backup Settings > More Options > Copy Also section and it is Off by default.

•   File attributes are out.
•   Owner and Group are now a single multi-state option.
•   DACL is now called Security info.
•   SACL is now in, as Auditing info.
•   Alternate NTFS streams are in as well.

The rationale behind File attributes option removal is that it was the only one from the list that was familiar to non-professional users. That instilled a false sense of confidence when looking at other options, enticing to tick them all on.

From now on the list comprises exclusively... erm... alien-looking stuff, so hopefully this will deter people from randomly enabling options they should not really be enabling. This also lets those in need of a complete replication configure it now directly in the UI.

First, let's look at how this can be done conventionally. We'll just be verbose about it and list all the options as is: This is OK in terms of usability, but when taken as a part of the whole it makes window look cluttered: And *unnecessary* cluttered at that, because in a vast majority of cases Owner and Group will be either both checked or unchecked. Ditto for Security and Audit options.

So let's try something else.

Windows natively supports something called "3-state" checkboxes that can be in checked, unchecked or indeterminate states. "Indeterminate" is a rather strange term as if the program is not sure in which state the option is. However we can think of it as an in-between or partial state and do something like this: This is not bad, however we lose the overall meaning of the option when it is in a partial state: So let's put the context back: Better, but now we are back to things being too verbose and looking cluttered. Let's simplify it for two most common cases: Better, but now it's off visually: How about with chop the specifics and move them to the side: Almost there, just need to patch up visual hierarchy a bit to help telling principal parts from the secondary ones: And that's it. The result is an UI that looks clean and simple for two most common cases, fully supports marginal ones and packs 4 options in a space of two.

In pre-78 releases if you excluded a folder, you weren't able to then go and include some of its subfolders or files. Similarly, it wasn't possible to exclude some of the items in an explicitly included folder.

In fact, it wasn't possible to exclude anything at all if using the Start with an empty list mode or include any item when using the Include everything option.

While this approach mapped on a simpler code and sort of made sense when it was first coded, in retrospect it wasn't that good. So, R78 will come with a fundamental overhaul of the filtering module... and whatever new and exciting bugs that may come with it.

What the middle pane now does is it allows quickly setting the action (include or exclude) for folder and files by simply clicking on them. For files the action is final - if you exclude it, then it is excluded. End of story.

For folders however it also sets a default action for all its files and subfolders. This action may be overriden for any item by clicking on it, which simply reverses the default. So when bvckup2 looks at what to do with any given file or folder, it will first pass it through this set of filters, yielding a preliminary include or exclude action.

Next, the item will be passed through generic filters, from the bottom pane. First hit on any of these determines the outcome. If there are no hits, then the default action it is.

There are also new, finer controls over the filtering flow. In short - it's now possible to mark any generic filter as final, which will suppress generic filter matching for all folder contents.

Bonus material

If you are to think about all this for as looong as I have, you might realize that the Include everything / Start with an empty list selector at the top, this one - is actually no longer needed!

All it does is merely sets the default action for the top directory - and we may just allow doing that by simply clicking on it in the middle pane.

However I think having it a separate option helps making the UI a bit more approachable and easier to use, so it will stay.

1. Log files are created by the backup engine, but displayed by the UI module. In desktop mode this is easy enough to handle, but in service mode the engine runs in a process of its own and under a different user account, meaning the logs are simply inaccessible to the UI.

So what ends up happening is that the engine opens logs on behalf of the UI and passes open file handles back to it.

PS. Ever wondered what all those "process" handles are doing on the DuplicateHandle() argument list? Exactly to allow one process to do something privileged on behalf another process.

2. Log files are rotated, so what you see in the UI is actually a seamlessly glued together list of all log entries across all copies of a backup log.

This would've not been a big deal if it weren't for...

Finally, the UI needs to expand/collapse log parts and do it nearly instantaneously. This is a hard problem, especially considering that logs can grow very big.

As the engine writes out the logs, the UI scans them and creates an auxiliary index, which it then uses to quickly understand what should and should not be shown in any given state.

These ndecies cannot be kept in memory, not in full. Instead the UI stores them in a specialized on-disk database, which too needs to be very quick.

However, all this complexity had a nice little side-effect. It made adding the Error view very easy.

All the UI had to do is to build another index and include just the error entries and their immediate context. This way switching to/from the Error view is a simple matter of swapping one index file for another. That's it.

This might not be the most exciting thing ever, but still has some elegance to it, doesn't it?

Blinkety-blink. Here's a slo-mo in all its glory: So after killing several hours on trying to make it behave, it became clear that it's faster and simpler to just redo it from scratch.

Not the first time it happened and, sure as hell, not the last :-|

The account is called "Bvckup 2 Service". It is automatically added when the app is switched to the service mode and removed when the app is switched back to the desktop mode or uninstalled.

The account is made a member of Administrators and Backup Operators groups, which gives it the same set of privileges as provided by the LocalSystem account.

Ideally, it should've only needed Backup Operators membership, but in some cases this group lacks network access privileges, so the default is to add it to the Administrators group as well.

It is still possible to install the service under the LocalSystem account - click the More button at the bottom left of the Switch Mode dialog and flip the Run service under option.

This change has been long coming as it's an altogether better and tidier setup. It helps isolating Bvckup 2 from the rest of the services, eliminating effects of any custom LocalSystem policies that might be in place and it also allows for a fine-grained control over its permission set. So if another Windows update comes out and (again) breaks things, it should be much easier to hammer it all back in place.

In comparison, here's the existing version in all its one-page glory:

Please note that this feature is currently reserved for site license and larger resellers only, and it is enabled on a case-by-case basis.

In any case, the next maintenance release (76.5) will get ejection support, but without the UI part. It will be possible to set it up with a good old INI file edit only.

Then, next full release (R77) will get the UI support and above sketches are for that bit. The tricky part is that ejection settings are ultimately per-location preference, so they are needed for both source and destination drives and this takes 2x of screen estate to accommodate.

Secondly, ejection is available only for local drives and not network shares, so the Option 2 series aren't really good. They all allows enabling ejection for non-local drives, meaning that we have to do extra "sanity checks" and error messaging when validating backup settings. That's just... messy.

In comparison, Option 1 simply doesn't show ejection option for non-local drives, just like it doesn't show device tracking (the pushpin icon) in that case. So it's tidy and logical.

• Top three vendors - Seagate (19%), WD (18%) and Samsung (12%)
• 219 HDDs and 58 SSDs - 79% and 21% respectively.
• The only SMART attribute present in all SMART reading was #12 - the power cycle count.
• All but one device also reported power on hours and reallocated sector count .
  
  This is good news, because the "reallocated" attribute (#5) is the principal predictor of an imminent drive failure. Once you see it starting to climb up, it's time to change the drive.
• The temperature attribute (#194) was reported by 71% of HDDs and 51% of SSDs. However some devices report temperature readings in different attributes, so these percentages may be off.
• Attributes in the 17 to 159 are unused (with an exception of one SSD that reports something #64).
• Lots and lots undocumented attributes. Some vendors have incomplete specs and others (e.g. Seagate) just don't release SMART specs at all. This means that we can't even label some of the attributes.
• Lots of inaccuracies in the specs too, meaning that it's hard to reliably interpret the attribute's raw values.
  
  Raw value is a 6-byte sequence of vendor-specific data, so you *must* have a spec to understand them. And you do want to understand them, because that's where the actually interesting data is.
  
  Regrettably some specs would say that it's just a 48-bit counter, whereby in reality it'll be a 16-bit something followed by a 32-bit counter. Furthermore, certain vendors (e.g. Kingston) tend to change the format not just between their product lines, but between sibling products that are just a model away from each other. It's really quite nuts.

New release is scheduled for the next week, so stay tuned.

After trying this and that, the design ended up converging to a simple More button at the bottom left corner of the dialog that, when pressed, injects additional settings into the dialog.

This is certainly not something new and revolutionary and it must've been done before, but it does feel a bit more natural compared to a more conventional approach of popping up a separate window with all the advanced extras.

Also, the initial iterations had the More button toggle between "More..." and "Less..." when clicked, but it looked a tad too busy, hence the current revision of a simple "sticky" background highlight when in the expanded state.

In a vast majority of cases native UI controls (widgets) work just fine. There's really no point in re-skinning, say, a button unless, of course, your goal is to make it stick out like a sore thumb.

There are however cases when the app needs to display information in a way that is not natively supported by the OS. In other cases, while it might be possible to render the data with a native control, it won't do it well or the result won't look good.

In these cases you have no option but to go with a custom UI, but, luckily, if you start with a need (rather than a quest for bells and whistles) the process is fairly predictable and logical.

Diskovery deals with displaying the bits and pieces of the computer storage stack - from physical drives, to parititions, to volumes, to logical drives and mount points.

These fall naturally into a hierarchy, with each level comprising a list of key-value(s) pairs.

So taking the first stab at the UI, we would quickly arrive at something like this: Primary list of drives and volumes on the left, with details of a selected item on the right, organized into a tree of "key:value" entries.

This is functional, but the right pane is not exactly easy to follow. So the first change will be to tabulate the key-value pairs:
Next, note that items in bold act like section headers, so they don't really need to be collapsible. So we do away with the buttons and connecting lines between top-level items:
This gets us a bit of extra horizontal space and trims a useless, but still actionable UI element.

Next, a couple of tweaks to how we render selected items.

There's no need for any items on the right to be selectable at all, so we won't render a selection highlight in the right pane.

The left pane can use a less jarring selection style, consistent with how Windows itself does it.
Next, we try and improve readability of the data on the right by helping to guide the eye from a key to its value.

The simplest option is the grid lines:
At this point, we take a sip of our cold coffee, squint an eye and assess if we like how it looks.

It's not bad, but we can try something else:
Now, that's not bad either and it has a nice visual consistency with the selection bar in the left pane.

However a minor nit here is that we now have bold items (section headers) paired randomly with row highlights. See how Storage space is on white and Partition table is on light gray?

Moreover, adding or expanding items will cause this pairing to flip-flop, e.g. we add an extra row about Partition table and it now sits on the white background:
That's not a disaster, but we can do better.

The idea is to recognize section headers as such and style them independently:
With this change in place we can just restart the row highlight sequence at the top of each section and all's good.

Next, we apply the same styling to the left pane as it happens to also comprise of sections:
Another sip of coffee, re-squint the eye and the whole thing now looks a bit cluttered and busy.

Easy enough to rectify by adding a bit of vertical padding:
Next, we remove vertical connecting lines leading from section headers to the top-level items.

However, we remove them on the right, but keep them on the left. This is because on the left we have a hierarchy of devices, so connection lines make sense. It also seems to look better this way.
Next, we move the expand/collapse buttons to the right.

In this particular app expandable items contain secondary details that 99% of users will not ever need. These items will also be collapsed by default (even though they are expanded here, but that's just because we are sketching through the details).

This change also helps decreasing the leading pad of top-level items and providing for a more compact horizontal packing of the data.
Alright, so that's about it as far as functional tweaks go. So now it's where it gets subjective. We are going to spruce it up a bit.

It's all about colors, padding and little nuances. If we were designing for the web, it would also involve texturing, but for a desktop app texturing is a sure way to ruin native look and feel.

In any case, moving on - invert the headers from dark-on-light to light-on-dark:
Add a bit of hue to the row highlights, tint their edges a bit and also flatten the look of expand/collapse buttons:
Splice in a scroll bar and make sure our precious design still looks reasonable:
Re-style the left pane to look more like an actual connected hierarchy of disks and volumes. The rationale here is to make it more obvious that items on the left are clickable.
Almost there. Take a final look and adjust vertical padding and spacing to de-clump items a bit more: And here it is. Compare to the starting point and you'd probably agree that there's at least some improvement.

Needless to say, one man's extra vertical padding is another man's complete waste of space, but any customization is always a combination of rational and subjective choices. The former is what gives an app the ease of use and the latter is what give it the personality. Can't have one without the other.

In particular, this happens when a custom domain is used with a managed (shared) mail service. You'd grab a domain, set its MX to mx.domain, add CNAME for mx.domain to point at mx.mail-service and - voilà! - you are served with mx.mail-service cert when connecting to mx.domain.

Ultimately, this is a mail server mis-configuration issue - your MX should really be pointing at mx.mail-service. In theory this shouldn't be happening, but in practice it does.

To address this, it's now possible to specify the exact domain the app should be looking for in an SSL certificate:


In the above example, you'd simply set this field to mx.mail-service and that's it.

To make things a bit easier that app now also helps you to pre-populate this field with the actual domain from the server's certificate.

It does that when you send out a test email and the field is left blank. As you can see from an opening screenshot, the app will recognize the certificate mismatch and re-run the test, accepting any and all certificates. It will then make note of the domain in the cert and copy it into the "SSL domain" field for you.

Finally, setting "SSL domain" to * will effectively disable all certificate checks and force the app to accept any certificate that the server sends, even if it's an expired or a self-signed one.

Earlier versions had general and alert-specific settings mixed up. For example, "From" doesn't really belong to the alert configuration, while "Include" from General Settings does.

This has been sorted out.

A larger picture here is that the app will be getting additional alert types - e.g. for S.M.A.R.T. diagnostics and backup verification - and this requires a proper structure to support these cleanly.

Also, while at it, the UI got a new framework for handling window transitions between different states.

For example, in the above screencap there's are 5 primary states and 2 intermediate ones. Prior to R75 moving window between these states required quite a bit of hand coding. This is no more, so we can now animate the hell out of any window. Consider yourselves warned :)

The year is 2015.

The age of cars trying to drive themselves and rockets trying to land on sea barges. Graphics cards are effortlessly churning photo-realistic images in real-time, mining crypto money and cracking passwords.

And here we have Windows still defiantly refusing to use any of that double-buffering nonsense when re-painting one of its basic common controls. Because who wouldn't want a bit of flickering?

Argh.

The idea is that the config window for specific alert type will give access to both the alert details and general email settings that are recycled between all alert types.

So here are several sketches for organizing page index of a multi-page configuration window. Nothing terribly novel here, it's more just a matter of getting styling details right - make sure it's obvious that it's an index and that its items are selectable.

Option Z (right-most) looks OK, but it's not obvious that index items are actionable as they look like static section headers.

Option Y is a bit better, but the arrow actually points at "From" field and you can bet some people will end up in a stupor because of that.

Option X uses standard Windows white-on-blue for selected item, so it's better in the actionability department. But it doesn't look nice.

Option B attempts to fix that.

Option A attempts the same but using app's own styling of selected items, which is already used in several places in the UI.

Thoughts or comments? There is a thread for that!

All fair points. Fear not, the UI got this covered:

The first iteration was the good old message box. You click on the "More on this..." and you get :


Nice, but looks like a hackjob, does't it? So, naturally, the next step was to spruce it up a bit - put text into a dedicated window, use RTF to add a bit of in-text formatting and a bit of animation to liven things up :


Nice again, looks better, but feels off. Too much going on. It looks like a conventional help window, but ... crippled. It's also not clear how to act when the user wants to resize the window - allow it, not allow it, restict minimum/maximum size, etc.

In short, what's lacking here is the cohesion. Luckily, the solution is right on the surface:


... whereby the text panel slides out on the "More info" click.

This is better, but now we have a window that sits lopsided on the desktop when the panel is open. It is also becoming wide, meaning that the app needs to mind the desktop edges, re-center the window, etc.

But if we just sleep on it, we might suddenly decide to make it an "either-or" window and then the UI magically becomes
(a) compact
(b) cohesive
(c) balanced
  and
(d) with unlimited text space!


So there you have it - a new mini-help subsystem coming to Bvckup 2 in the Release 71, deployed on as-needed basis across the app.

Don't know about you, but I think it's neat, tidy and pretty damn clever :)

VSS writers are the components of the shadow copying framework that are tasked with ensuring consistency of specific files in VSS snapshots. For example, when creating a snapshot, Windows would ask the MSSQL writer to flush any pending database changes in a way that would make on-disk copy of the database self-consistent.

Some of the writers are required and always participate in the snapshot creation process, but some are optional and they need to be explicitly enabled by the initiating app. Starting with R69 Bvckup will be enabling all optional writers and it will also provide a mechanism for excluding specific ones if needed.

Only a very small part of the log is now kept in memory, just enough to fill the log viewer panel in the UI. The rest is kept on the disk and read from there on as-needed basis.

However this is easier said than done.

Bvckup logs are hierarchical, meaning that they are generic trees with nodes that may have an arbitrary large number of children. Running a 1 million file backup would yield a "Processing..." entry with 1 million children, each with up to a dozen of descendants.

Displaying this tree would've not been a big deal if all log entries were always visible at all times. When the UI would've wanted to show entries 700,503 through 700,523 entries, it would've simply gone to a respective location in the log index file and plucked the data out. But...

The nodes can be collapsed and expanded. This hides and shows arbitrary chunks of the log tree and then the UI has this issue of finding 700,503-th *visible* item. And this is hard.

This implies quick look-ups in some sort of super-imposed binary search tree of visible items. A tree that changes every time a node is expanded or collapsed and that sits in a disk file, which restricts how you can walk and modify it without getting an I/O penalty.

This is fun. This is what cost me 3 weeks of shuffling through CS books and several complete rewrites of the tree indexing code. Believe it or not, but this is basically an original research and it seems that no one had to deal with this issue before. At least not publicly.

The solution is to use slightly customized version of an AVL tree to store the list of node's children (that's using tree within a tree), use it for the visible item look-ups and then optimize the hell out of the index I/O through aggressive write caching.

When all i's are dotted and t's crossed, this yields a logging system that can push up to 1 mil entries per second on a resonably equipped box. This might not be knock-your-socks-off impressive, but it's plenty enough for the task at hand.

At the moment Bvckup's UI keeps backup logs in memory. It puts an effort to trim them to a reasonable size, but this still eats quite a bit of RAM, especially with large backups.

It costs about 120 bytes plus the size of an entry to keep a single log line in memory, whereby these 120 bytes help weave entries into a hierarchy (a tree) and track their state.

If we want to reduce the memory usage, then the most obvious optimization would be to keep the actual entry text in a disk file and load it on demand, when it's actually visible in the log viewer.

This certainly helps, but as the log grows, the memory usage still climbs. A million file backup generates about 4 million log entries. At 120 bytes per entry that's 0.5 Gig in just supporting structures. This is unacceptable.

So it means that the entire tree structure of the log needs to sit in a disk file and to be read from there as needed.

Disk-based tree structures are routinely used in databases and file systems to store indecies of data sets, typically in a form of a B-Tree and its variations. The kicker is that these structures are meant for storing sorted data and they optimize for searching. Trying to adapt them for storing unsorted data would be nothing short of fitting a square peg in a round hole.

Long story short - there are no ready-made code for storing and manipulating generic data trees on a disk, leave alone fast code. This appears to be an esoteric problem that everyone solves on their own.

The screenshot above shows a part of a small "tree database" library that I ended up writing, together with its own caching module, predictive page loader and a pony.

Took almost two weeks.

On a bright side though, this code is perfectly reusable for storing disk snapshots as well and this paves way for making the planner module work almost entirely off the memory.

Once done, Bvckup memory usage will NOT depend on the particulars of a backup at all - this a very big deal and an incredibly improtant feature to have for any robust backup software.

So, stay tuned...

Then copy-paste the code, click on Verify, get this -

If this works (the server is accessible, the license code is valid, etc), then this is it. Thank you very much, click Ok, you are done -
Otherwise, it explains the failure and offers either to retry the activation or to use an alternative -
There are two alternatives, both are simple -
For offline boxes and locked-down setups there's the Copy To Clipboard option. Take the URL to a networked machine, open it in a browser, get a licensing file back and then drop it into the app's config folder.

The URL is exactly the same as the one used by the app's own licensing code and it comprises of the license code and installation ID and the fingerprint .

Prepare the email option creates a new email with default email client and puts the Installation ID, Fingerprint and License Code in the message along with some pleasantries. The license file will be attached to the reply and it will simply need to be dropped into %LocalAppData%\Bvckup2\engine.

When the app is started, it looks at several system properties including the CPU model, Windows' MachineGuid value and others, hashes them together to produce an "Installation ID". For example -


The ID stays the same under normal computer use, but it changes with hardware upgrades or when the OS is reinstalled.

The app also generates an "Installation fingerprint" by hashing each of the properties separately and stringing them together. This yields a longer hash -


Like ID, the fingerprint changes with hardware upgrades and OS reinstalls, but, unlike ID, it changes only in parts. This makes it suitable for detecting incremental changes to the user's computer.

Both ID and Fingerprint identify user's installation, the difference is that the ID is used on the app side and the Fingerprint - on the licensing server side.

Next, ID and fingerprint are submitted to the licensing server, either by the app itself or by a human through a website. When licensing for production use (as opposed to beta or evaluation), the request also includes a license code - a random token that is issued to the user in exchange for payment.

The server responds with a license, which is a short text file that binds together few pieces of information and seals them with an RSA signature of the licensing server.


For beta and evaluation licenses the "license code" is generated automatically by the licensing server. The large block of gibberish at the bottom of the license is the RSA signature.

When the app starts, it loads the license and verifies it using public key of the licensing server that is embedded in the app.

If the license is valid, the app works as normal. If the license is missing or expired or doesn't match the installation ID, the app disables certain features and periodically nags to do the right thing and what not.

Now, let say I upgraded my box. This screws up the Installation ID and invalidates the license. The app recomputes new ID and Fingerprint and sends them to the licensing server.

This is where Fingerprint comes into play. The server takes old and new fingerprints, chops them into individual hashes and does a "fuzzy" match. If enough hashes match, we can assume an incremental software or hardware change on user's end and re-issue the license for new Installation ID.

If fingerprints are wildly different, then we make fuss and require contacting human tech support.

In conclusion, needless to say that the whole scheme relies on the validation code in the app being trustworthy. If someone pries the app open and replaces selected IFs with NOPs, then obviously no licensing will work.

The answer to that is that the app should implement self-consistency checks and deploy counter-measures when these checks fail. This is a separate and very large and interesting subject ... which I have no plans on covering here :-) A good starting point here would be fravia.org archives and then follow from there.

This time the next/previous arrows are in the error entry itself.

I like the second option better, because it's contextual and I think it's more obvious. However, first option is superior in functionality because it allows jumping to next or previous error when there's currently no error visible in the pane.

Still deciding...

On program's end, the indicator translates into the "Update is recommended" line. It is one three options with "optional" and "critical" being the other two.

"Recommended" is a routine update that adds this and fixes that. It would better be installed, but it's not the end of the world if it's not.

"Critical" is critical - "you'd be sorry if you don't install it" kind of update.

An example of "optional" update is a new version that updates program's French translation when the user is running an English version, i.e. an update that can be ignored altogether.

There's now also the "View changes" button that expands the window and shows a neatly formatted (RTF) version of the change list.

The biggest deal, of course, is that it's a consolidated list of all changes between the installed and the latest version, rather than a laundry list of all changes since the dawn of time. It makes the list infinitely more useful compared to the raw version.

Significantly improved performance through multi-core, parallel hashing, asynchronous I/O and more aggressive caching of the state data.

Strengthened the change detection algorithm through addition of the whole file hash. When all individual block hashes come back unchanged, the algorithm now uses the file hash to confirm that the file has indeed remained unchanged.

Combined with the original use of two separate hashes per data block, this eliminates the need for a precautionary full-file sync every N copies.

Improved handling of cancelled and aborted delta copies by introducing support for partial updates .

The algorithm now remembers how far along the file it managed to proceed and stashes this information for the next run.

When the copying is cancelled partway through, the file is marked as out-of-sync, but the delta state is updated to capture what has been already done. This comes very handy and speeds things up when updating very large files.

Improved handling of large file counts.

Version 1 maintained an in-memory index of all delta copied files and, surprisingly, this didn't scale well. It also kept all delta state information in a single folder, which too led to some pain and suffering when the file count grew large.

New version eliminates both these problems.

Lastly, all copier parameters - block sizes, hash algorithms, cache buffer sizes, threading parameters, etc. - are now configurable on a per-job basis.

All in all - THIS IS IT.

This is how a simple and efficient delta copying should be done. Can't wait to release it to the real world.

There are two sets of icons on the sprite sheet. The first three columns is the XP style, which is a bit puffy and matches well older Windows styling. The last three columns is a flat version that goes better with native W7 tray icon style.

Each icon comes in 3 sizes - 16x16, 20x20, 24x24. This is needed to accommodate 3 common DPI levels, see this earlier entry for details.

When a program detects that it was just restarted with Admin privileges, it has a limited number of options for communicating this joyous news to its user.

#1 - A message box. Bluntly put - I hate them. They are always too sudden, they steal your attention and stuff whatever they want to say down your throat. They ... lack manners.

#2 - A system tray notification. This is less intrusive, but perhaps it's too out of the way. Also, the systray is a shared area so the program is always competing with other apps for the same screen real estate.

#3 - Something else that is right in your field of view if you are looking at the app, but that doesn't get in a way of whatever you are doing at the moment.

Cue in the message bar, now an integral part of the Bvckup UI. It is used for displaying various status notifications right after the launch, after switching the engine to/from the system service mode, when discovering software updates and few other things.

I have also tried the self-closing version of the bar that dismisses itself on a timeout. Like so -
and so -
but having lived with this for a couple of days, it now seems like a solution in a search of a problem. So I shelved it for now. If there's any a real need for this, it'd be very easy to add later on.

The most unconventional part of the setup is how it handles the configuration of the installation parameters.

The configuration page (see Page 3) is very bare and focuses on how the program is going to be used instead of being a mixed bag of the minute details of the setup.

The target location is made into a secondary setting with its value controlled by the usage preference. There's no field for the Start Menu group name. There's no ever popular "Run the program after the setup completes" option either. Instead, exiting the setup automatically runs the program (see Page 4), unless the installer window is closed through an (x) at the top.

This leads to a simpler and more intuitive setup flow... or at least that's the idea :)

The installer supports both privileged (administrative) and unprivileged (regular user) installations.

If you are logged in as a regular user and have no rights to modify C:\Program Files, there's no reason why you shouldn't still be able to install a software, just for yourself.

While other installers routinely ask if to install the software for everyone or just for one user, it's a superficial question. It merely controls if the Start Menu group is created under All Users or your personal profile. The program files still go into a system directory and this requires administrative privileges.

Bvckup implements true system- and user- deployment, which is akin to deploying the program in /usr/bin or ~/bin in Unix terms.

In fact, on a multi-user system it is perfectly possible to install one instance of Bvckup for each user and they will all happily co-exist with each other.

If the program is not running, then it's indeed a simple matter of replacing the program file with a newer version. The fun starts if the program is running.

On Windows, running a program locks its file, preventing any changes from being made to it. This is not required per se and there are operating systems that don't do this. There are also custom PE loaders for Windows that can take an executable and create a running process out of it without any locking fuss. For example, bo2k could do that back in 1999. See its source code for details if interested.

To complicate matters, there might be more than one process running off the same executable. For example, if several users are logged into a Windows box and each runs a copy of a program.


Therefore, the first order of business for any installer is to shut down all running instances of the program. A well-behaved installer should also restart all these instances after it's done with updating.

The shutting down part is not a big deal. There's more than one way to do this and all of them are fairly straight-forward.

It's the restarting that is contrived.

Not only program instances may be running under different user accounts, they may also be running with custom command-line arguments, as Administrators or system services. Restarting this cornucopia of awesomeness is not an envious task.

Here's how Bvckup works around this mess.

Every Bvckup executable has a corresponding named event in the Global namespace. The event is created by the first instance of the program when it's launched, and it is opened by all other instances.


The name of the event includes the file ID of the executable file. This groups running instances by the file they are using, because, after all, the installer is interested in a specific executable and nothing else.

For example, the following is an event name used by the Bvckup installation on my development box -

Global\Bvckup2.e65a96a8001f000000013cba.Update

With this in place, the installer starts off by signaling the event.

In response, a running program makes a temporary copy of its executable file and starts it in a special "relauncher" mode. And then it exits.

The "relauncher" process opens the "update" event and simply spins there, waiting for it to become unsignaled.

Meanwhile the installer waits for the program file to become unlocked. Once all running instances replace themselves with the relaunchers, the executable frees up and the installer proceeds with the update. It then clears the "update" event and exits.

This releases the relaunchers, they restart program instances and - ta-da - the update is completed.

Techincally, instead of adding a special "relauncher" mode to the program it's possible just to have a small standalone relauncher program. It comes down to a matter of taste. Some people don't like to spawn from temporary executables, and I like to keep my file count to a bare minimum. To each his own.

The header font, Alte Haas Grotesk, didn't display correctly after it was converted to the @font-face kit. Specifically, the lowercase 'o' looked like this:

The font's free, but I couldn't find a copy of its license, so I started tracking down font's author to ask a permission to edit the outlines.

While waiting for a reply, I found another workaround which involved generating the webfont kit from OTFs instead of TTFs. Not a big deal, but it still took over 3 hours to resolve.

The second issue was with the slide-out, fade-in animation. Sometimes the slide-out didn't extend all the way down and sometimes it overshot. This traced back to the fact that slide-out height was computed before the OpenSans was fully loaded and the script fused basic sans-serif to compute the height.

Again, not too complicated, but, again, cost an hour to work through.

:-|
Little bit of this, little bit of that - all unplanned and it all adds up.

Current design is almost two years old and it grew steadily disconnected from the simpler aesthetics I am pushing for now. So it needs to be redone.

I will probably simplify this even further. I've been wanting to do a true minimalist site for a software product for a long time now and this seems like a real chance to do just that. Yay.

Interestingly enough, the total size of individual images saved as 32bit PNGs is 63,127 bytes, while the size of this sprite sheet (without checkered backdrop) is 63,829. After factoring in the size of the code that is needed to parse the sprite sheet, it really becomes a no brainer to keep every image in its own file and embed them into .exe's resources that way.

1. The program now differentiates between three types of errors and handles them slightly differently.

First, there are unexpected operating system errors, i.e. when a core Windows function fails in a way that it shouldn't be failing. Classic example (that everyone is aware of) is TrackPopupMenu that is declared as returning boolean, but actually doesn't shy away from returning an integer under certain circuimstances.

Secondly, there are inconsistencies in app's own code, whereby one part assumes certain usage that is not observed by another part. These are so-called "assert" failures. The original version has three critical issues, all of this particular kind.

Lastly, there are straight-forward crashes. This is when a program is trying to operate with data that is no longer there or corrupted. This is the worst of them all as it typically indicates a major screw up on programmer's part, but in some cases it may also be an indication of failing hardware, like a faulty memory stick.

2. The program now generates a minidump - a small-ish file in a standard Windows format that captures the context of a failure. Minidump files dramatically simplify investigation of a problem on the developer's end.

3. Finally, the minidumps can now be uploaded to the support server right from the error notification window with one click. The transfer is based on WinInet API as this is supposed to improve the chances of getting the report through in proxied and otherwise "enterprised" environments.

For cases when the upload fails, the error dialog also offers an option of opening a link in a regular browser and reporting a problem that way.

Pro: a single executable to distribute
Con: the exe comes out weighing twice as much... or does it?

The size of a 32-bit executable can be easily reduced with the help of tools called executable packers . They work by turning the exe file into a self-extracting archive that also automatically launches the original exe after the extraction. It may sound simple but in reality it involves bundling a custom PE loader with the packed executable (as it might not be possible to extract the exe into a temporary file before running it).

In any case, exe packing is routinely used by developers who care about the size of their products, with uTorrent being one of most notable examples. Original Bvckup was also compressed, owning its tiny size in part to the magic of UPX .

The problem lies with 64-bit executables. For a variety of reasons there are presently no exe packers that support them.

However if we store 64bit binary inside of the 32bit executable and then pass latter through a compressor, we end up with both being compressed. Ta-da!

But there's more.

When an .exe is compressed, the compressor leaves the primary application icon intact. This is done so that the app would keep its original appearance in Windows Explorer and not sport a generic "program" icon.

Starting with Vista, the app is expected to support a variety of icon sizes that quickly add up. For example, bvckup's icon weighs in at about 110KB. That's almost half as big as the actual code, the code that does quite a few meaningful things on top of just looking great :)

So, if we do the 64-in-32bit packaging, we end up compressing 64bit version in its entirety, including its icon, and arriving at a greater size reduction if we were to compress the exes separately.

Win-win.

The message is a homage to NetPositive, the BeOS browser that pioneered the technique of using haiku in error reporting. In fact, the message in option B is from the NetPoisitive original list .

The icon is that of a Zen circle, which has a plenty of meanings, many of which applicable given the circumstances.

Fading animation dissolves a part of a window into background, in preparation for transitioning the window to another state, e.g. moving to the next page in a wizard or resizing the window into a more compact form.

As simple as it sounds, these sort of animation is fairly complex to implement on top of the Windows API.

Windows includes native support for window transparency in a form of layered windows, but it comes with a list of limitations that aren't exactly practical or easy to work, except for several specific cases.

The simplest working solution involves taking a screenshot of the window, creating an overlay window and then using it as a canvas to render the transition from an empty background to the screenshot. All the while making sure that Windows doesn't sneak in an unsolicited window refresh that would lead to a flicker, which in itself is a very "fun" thing to debug.

16, 32, 48, 256 are more or less expected. These are used in "small", "icon", "tile" and "extra-large " views in Windows Explorer.

20, 22 and 26 are possible sizes of an icon used in the window caption. The exact size depends on screen's DPI level.

And 40 (along with 32 and 48) is the size of an icon used in Alt-Tab view on W7, again depending on the screen's DPI.

If you don't provide an appropriately sized icon, the OS will grab one that is slightly bigger and shrink it down, ruining the pixel polish.

Lastly, each icon ideally must be saved in 32, 8 and 4 bits per pixel formats, so to be compatible... you know... with EGA cards

It would've not that big of a deal if it weren't for the size of a resulting .ico file. For example, the above icon ends up being ~ 120KB, whereas the actual .exe it is attached to and that does quite a few meaningful things sits at around 96KB.

DPI stands for dots-per-inch and it's a metric of a pixel density on the screen. The smaller the screen, the more pixels are packed into a square inch, assuming the resolution stays the same. The difference in DPI between a large LCD panel and a 12" notebook screen can easily be a factor of two or more.

This means that if a program renders 9px text on both screens, on one it will appear twice as small as on the other one, and might command a need for a good magnifying glass.

So naturally it's a good idea to make the OS aware of the actual DPI of the monitor in use. If it's too high, the OS can do few things to make things visible with a naked eye again. Like increase size of the default system font and make windows generally larger.

This can be done automatically by looking at the monitor .inf files or by letting the user configure the DPI manually.

XP defaults to 96 DPI and with some difficulty it can be changed to 120 DPI. Vista and newer versions are better. They come with four predefined levels - 96, 120, 144 and 192 - and they also support custom settings.

This brings us back to needing multiple versions of every graphic in the app.

Vista is the first Windows version to routinely use monitor-specific DPI values out of the box. As such 96 is no longer a dominant DPI level and the apps are now expected to take the DPI into an account when rendering their content.

The higher the DPI, the larger the graphics and the fonts the app should use. The penalty for not doing this is an automatic stretching of the app window that inevitably leads to a blurry appearance and a host of other lovely issues .

In other words - not complying with OS expectations is not really an option. Hence the need for 3+ differently sized copies of the same pixel art.