Another change in upcoming 76.5 is a new
warning that is shown when copying of
owner, group or DACL attributes is enabled
for a backup job that goes across the
computer boundary.
As you might guess the warning is due to what we are seeing on the support end, when some people enable some backup options merely because they look reasonable, but not because they actually need them.
As you might guess the warning is due to what we are seeing on the support end, when some people enable some backup options merely because they look reasonable, but not because they actually need them.
Added support for pre-seeding backups and
adjusting source or backup locations when
the on-disk data is being moved around.
Previously, if either source or destination location was changed in backup settings, the app would purge all delta copying data and the snapshot of destination if present. This caused destination to be re-scanned on the next run and all files to be re-copied in full next time they needed an update.
With the next relase (76.5) the app will now ask if the backup state should be purged or not. This way if you are setting up a remote backup job, you may first pre-seed the backup by attaching the drive locally and making a backup on it. They you'd take the device to the remote machine, change Destination path in the backup job, say No to the state reset and - voila - you are back in business as if the drive has always been remote.
Previously, if either source or destination location was changed in backup settings, the app would purge all delta copying data and the snapshot of destination if present. This caused destination to be re-scanned on the next run and all files to be re-copied in full next time they needed an update.
With the next relase (76.5) the app will now ask if the backup state should be purged or not. This way if you are setting up a remote backup job, you may first pre-seed the backup by attaching the drive locally and making a backup on it. They you'd take the device to the remote machine, change Destination path in the backup job, say No to the state reset and - voila - you are back in business as if the drive has always been remote.
Bvckup 2 is getting media and device ejection support.
Media ejection refers to, well, ejecting storage media from an otherwise fixed device such as a tape drive.
Device ejection has to do with "safe removal" of USB drives, the same thing you can do by clicking on a standard Windows "safe remove" icon in the system tray before yanking drive's USB cable out.
However, in a true Windows fashion ejecting a media sometimes also ejects a device and sometimes you can also eject media from an external USB HDD (and, yes, this makes absolutely no sense).
—
In any case, the next maintenance release (76.5) will get ejection support, but without the UI part. It will be possible to set it up with a good old INI file edit only.
Then, next full release (R77) will get the UI support and above sketches are for that bit. The tricky part is that ejection settings are ultimately per-location preference, so they are needed for both source and destination drives and this takes 2x of screen estate to accommodate.
Secondly, ejection is available only for local drives and not network shares, so the Option 2 series aren't really good. They all allows enabling ejection for non-local drives, meaning that we have to do extra "sanity checks" and error messaging when validating backup settings. That's just... messy.
In comparison, Option 1 simply doesn't show ejection option for non-local drives, just like it doesn't show device tracking (the pushpin icon) in that case. So it's tidy and logical.
Media ejection refers to, well, ejecting storage media from an otherwise fixed device such as a tape drive.
Device ejection has to do with "safe removal" of USB drives, the same thing you can do by clicking on a standard Windows "safe remove" icon in the system tray before yanking drive's USB cable out.
However, in a true Windows fashion ejecting a media sometimes also ejects a device and sometimes you can also eject media from an external USB HDD (and, yes, this makes absolutely no sense).
In any case, the next maintenance release (76.5) will get ejection support, but without the UI part. It will be possible to set it up with a good old INI file edit only.
Then, next full release (R77) will get the UI support and above sketches are for that bit. The tricky part is that ejection settings are ultimately per-location preference, so they are needed for both source and destination drives and this takes 2x of screen estate to accommodate.
Secondly, ejection is available only for local drives and not network shares, so the Option 2 series aren't really good. They all allows enabling ejection for non-local drives, meaning that we have to do extra "sanity checks" and error messaging when validating backup settings. That's just... messy.
In comparison, Option 1 simply doesn't show ejection option for non-local drives, just like it doesn't show device tracking (the pushpin icon) in that case. So it's tidy and logical.
The 9th revision of the website is now live.
New tray icon state for the case when all configured
jobs are disabled. Will be in the next R76 maintenance
update.
This is a development test of new parallel copying
feature that may be making its way into the next release.
If you are familiar with robocopy, it's an equivalent of its /mt option, which causes several files to be copied, well, in parallel.
As it's probably obvious from the above screenshot it works really well, especially for smaller files when an overhead of opening and closing file is comparable to the bulk I/O time. Here, for example, C:\Windows\Fonts folder with about 1500+ items is getting copied to a Synology NAS over 1Gbps link.
This feature is going to be available under a Pro license only. More on this to follow.
If you are familiar with robocopy, it's an equivalent of its /mt option, which causes several files to be copied, well, in parallel.
As it's probably obvious from the above screenshot it works really well, especially for smaller files when an overhead of opening and closing file is comparable to the bulk I/O time. Here, for example, C:\Windows\Fonts folder with about 1500+ items is getting copied to a Synology NAS over 1Gbps link.
This feature is going to be available under a Pro license only. More on this to follow.
First stab at the Drive Health Monitor UI
for the next Bvckup 2 release.
There's lots of information to be displayed, so the idea is to recycle columns for hosting multiple releated bits of data.
For example, you can toggle between showing drive vendor and its model in the 2nd column, and between disk capacity and its current utilization in the 5th column.
Naturally, none of this is final, but I think for the first pass this is not bad.
There's lots of information to be displayed, so the idea is to recycle columns for hosting multiple releated bits of data.
For example, you can toggle between showing drive vendor and its model in the 2nd column, and between disk capacity and its current utilization in the 5th column.
Naturally, none of this is final, but I think for the first pass this is not bad.
A list of storage device vendors
as observed by Diskovery to date.
I must admit that I haven't even heard of at least 3 of these.
—
Here's few more notable findings from the aggregated analysis of 200+ logs submitted so far:
I must admit that I haven't even heard of at least 3 of these.
Here's few more notable findings from the aggregated analysis of 200+ logs submitted so far:
- Top three vendors - Seagate (19%), WD (18%) and Samsung (12%)
- 219 HDDs and 58 SSDs - 79% and 21% respectively.
- The only SMART attribute present in all SMART reading was #12 - the power cycle count.
- All but one device also reported power on hours and reallocated
sector count.
This is good news, because the "reallocated" attribute (#5) is the principal predictor of an imminent drive failure. Once you see it starting to climb up, it's time to change the drive. - The temperature attribute (#194) was reported by 71% of HDDs and 51% of SSDs. However some devices report temperature readings in different attributes, so these percentages may be off.
- Attributes in the 17 to 159 are unused (with an exception of one SSD that reports something #64).
- Lots and lots undocumented attributes. Some vendors have incomplete specs and others (e.g. Seagate) just don't release SMART specs at all. This means that we can't even label some of the attributes.
- Lots of inaccuracies in the specs too, meaning that it's hard to reliably
interpret the attribute's raw values.
Raw value is a 6-byte sequence of vendor-specific data, so you *must* have a spec to understand them. And you do want to understand them, because that's where the actually interesting data is.
Regrettably some specs would say that it's just a 48-bit counter, whereby in reality it'll be a 16-bit something followed by a 32-bit counter. Furthermore, certain vendors (e.g. Kingston) tend to change the format not just between their product lines, but between sibling products that are just a model away from each other. It's really quite nuts.
Two years worth of changes from the last version of the
beta cycle (April 2014) to the upcoming production release
(April 2016).
Just thought I'd share.
Just thought I'd share.
Finalized the last change that will go into R75 -
full UI support for advanced filtering rules that
have been supported internally
since last August.
More specifically, the UI can now be used to set up rules for matching on the file size and created/last-modified times. Attribute matching has also been extended to support more exotic values such as Offline and Don't-Index.
More specifically, the UI can now be used to set up rules for matching on the file size and created/last-modified times. Attribute matching has also been extended to support more exotic values such as Offline and Don't-Index.
Added an option to copy parts of the log to the
clipboard - either just a single line or the line
and all its children (the block).
If the block is large (over 64K), then the app will also display its size on the right-hand side on the menu. The neat thing is that this information is readily available in the log index, so this requires no log scanning of any kind or no interim memory allocations. Basically, it's as O(1) as it gets.
Will be in R75.
If the block is large (over 64K), then the app will also display its size on the right-hand side on the menu. The neat thing is that this information is readily available in the log index, so this requires no log scanning of any kind or no interim memory allocations. Basically, it's as O(1) as it gets.
Will be in R75.
Network Logon configuration has now been reworked
to allow configuring a couple of advanced aspects
of the share access.
Will be in R75.
Will be in R75.
Back to Bvckup 2 now and its upcoming Release 75.
With the next release the UI will start adopting a new "progressive refinement" approach for managing advanced and less-frequently used features.
With the next release the UI will start adopting a new "progressive refinement" approach for managing advanced and less-frequently used features.
After trying this and that, the design ended up
converging
to a simple More button at the bottom left
corner of the dialog that, when pressed, injects
additional settings into the dialog.
This is certainly not something new and revolutionary and it must've been done before, but it does feel a bit more natural compared to a more conventional approach of popping up a separate window with all the advanced extras.
Also, the initial iterations had the More button toggle between "More..." and "Less..." when clicked, but it looked a tad too busy, hence the current revision of a simple "sticky" background highlight when in the expanded state.
This is certainly not something new and revolutionary and it must've been done before, but it does feel a bit more natural compared to a more conventional approach of popping up a separate window with all the advanced extras.
Also, the initial iterations had the More button toggle between "More..." and "Less..." when clicked, but it looked a tad too busy, hence the current revision of a simple "sticky" background highlight when in the expanded state.
Added support for displaying both effective and
all supported SATA signaling rates for physical
drives. These are also interchangeably knowns
as "transfer speeds" and "link rates".
Detecting supported signaling rates is trivial. After all, it is a part of the drive's ATA IDENTIFY block, word 76.
In comparison, drive's effective signaling rate is a bit trickier to determine. In theory, it is also a part of IDENTIFY block, word 77. In practice, this is only available from drives that support ACS-3, which is a more recent revision of the ATA command set.
So for drives that don't indicate their effective signaling rate Diskovery looks at "phy link rate" as reported by CSMI-compliant drive controllers.
But, wait!
What if both rate readings are available and they are different? Not possible, you say? "Ha-ha" says an outdated Intel RST driver that ships with Windows 8.1. In this case Diskovery will default to the drive's version of the value and also issue a warning of the discrepancy.
Detecting supported signaling rates is trivial. After all, it is a part of the drive's ATA IDENTIFY block, word 76.
In comparison, drive's effective signaling rate is a bit trickier to determine. In theory, it is also a part of IDENTIFY block, word 77. In practice, this is only available from drives that support ACS-3, which is a more recent revision of the ATA command set.
So for drives that don't indicate their effective signaling rate Diskovery looks at "phy link rate" as reported by CSMI-compliant drive controllers.
But, wait!
What if both rate readings are available and they are different? Not possible, you say? "Ha-ha" says an outdated Intel RST driver that ships with Windows 8.1. In this case Diskovery will default to the drive's version of the value and also issue a warning of the discrepancy.
Diskovery now recognizes commonly used emulated
RAM-based storage, like SoftPerfect RAM Disk
and Dataram RAMDisk.
The empty space at the bottom of the Diskovery
window is now used to show quick status messages.
A little opening animation is included.
Diskovery can now also talk to the
Virtual Disk Service
and extract all sort of interesting information from it,
including exact volume type (spanned, striped, mirrored,
etc.),
its online/offline status and its health.
Diskovery now detects the USB drive type and its effective
connection speed. No more USB3 drives sitting on USB2 hubs.
An opening screen for Diskovery, must be read
in Jeeves'es voice for full effect.
Not long now, almost done tying up all loose ends...
With Diskovery coming together,
I wanted to take a moment and show what a UI sketching
process for an app looks like.
—
In a vast majority of cases native UI controls (widgets) work just fine. There's really no point in re-skinning, say, a button unless, of course, your goal is to make it stick out like a sore thumb.
There are however cases when the app needs to display information in a way that is not natively supported by the OS. In other cases, while it might be possible to render the data with a native control, it won't do it well or the result won't look good.
In these cases you have no option but to go with a custom UI, but, luckily, if you start with a need (rather than a quest for bells and whistles) the process is fairly predictable and logical.
—
Diskovery deals with displaying the bits and pieces of the computer storage stack - from physical drives, to parititions, to volumes, to logical drives and mount points.
These fall naturally into a hierarchy, with each level comprising a list of key-value(s) pairs.
So taking the first stab at the UI, we would quickly arrive at something like this:
Primary list of drives and volumes on the left,
with details of a selected item on the right, organized
into a tree of "key:value" entries.
This is functional, but the right pane is not exactly easy to follow. So the first change will be to tabulate the key-value pairs:
Next, note that items in bold act like section
headers, so they don't really need to be collapsible.
So we do away with the buttons and connecting lines
between top-level items:
This gets us a bit of extra horizontal space and trims
a useless, but still actionable UI element.
Next, a couple of tweaks to how we render selected items.
There's no need for any items on the right to be selectable at all, so we won't render a selection highlight in the right pane.
The left pane can use a less jarring selection style, consistent with how Windows itself does it.
Next, we try and improve readability of the data on the
right by helping to guide the eye from a key to its
value.
The simplest option is the grid lines:
At this point, we take a sip of our cold coffee, squint
an eye and assess if we like how it looks.
It's not bad, but we can try something else:
Now, that's not bad either and it has a nice
visual consistency with the selection bar in the
left pane.
However a minor nit here is that we now have bold items (section headers) paired randomly with row highlights. See how Storage space is on white and Partition table is on light gray?
Moreover, adding or expanding items will cause this pairing to flip-flop, e.g. we add an extra row about Partition table and it now sits on the white background:
That's not a disaster, but we can do better.
The idea is to recognize section headers as such and style them independently:
With this change in place we can just restart the row
highlight sequence at the top of each section and
all's good.
Next, we apply the same styling to the left pane as it happens to also comprise of sections:
Another sip of coffee, re-squint the eye and
the whole thing now looks a bit cluttered and
busy.
Easy enough to rectify by adding a bit of vertical padding:
Next, we remove vertical connecting lines leading
from section headers to the top-level items.
However, we remove them on the right, but keep them on the left. This is because on the left we have a hierarchy of devices, so connection lines make sense. It also seems to look better this way.
Next, we move the expand/collapse buttons to
the right.
In this particular app expandable items contain secondary details that 99% of users will not ever need. These items will also be collapsed by default (even though they are expanded here, but that's just because we are sketching through the details).
This change also helps decreasing the leading pad of top-level items and providing for a more compact horizontal packing of the data.
Alright, so that's about it as far as functional
tweaks go. So now it's where it gets subjective. We are
going to spruce it up a bit.
It's all about colors, padding and little nuances. If we were designing for the web, it would also involve texturing, but for a desktop app texturing is a sure way to ruin native look and feel.
In any case, moving on - invert the headers from dark-on-light to light-on-dark:
Add a bit of hue to the row highlights, tint their
edges a bit and also flatten the look of
expand/collapse buttons:
Splice in a scroll bar and make sure our precious design
still looks reasonable:
Re-style the left pane to look more like an actual connected
hierarchy of disks and volumes. The rationale here is to make
it more obvious that items on the left are clickable.
Almost there. Take a final look and adjust vertical padding
and spacing to de-clump items a bit more:
And here it is. Compare to the
starting point and you'd
probably agree that there's at least some improvement.
—
Needless to say, one man's extra vertical padding is another man's complete waste of space, but any customization is always a combination of rational and subjective choices. The former is what gives an app the ease of use and the latter is what give it the personality. Can't have one without the other.
In a vast majority of cases native UI controls (widgets) work just fine. There's really no point in re-skinning, say, a button unless, of course, your goal is to make it stick out like a sore thumb.
There are however cases when the app needs to display information in a way that is not natively supported by the OS. In other cases, while it might be possible to render the data with a native control, it won't do it well or the result won't look good.
In these cases you have no option but to go with a custom UI, but, luckily, if you start with a need (rather than a quest for bells and whistles) the process is fairly predictable and logical.
Diskovery deals with displaying the bits and pieces of the computer storage stack - from physical drives, to parititions, to volumes, to logical drives and mount points.
These fall naturally into a hierarchy, with each level comprising a list of key-value(s) pairs.
So taking the first stab at the UI, we would quickly arrive at something like this:
Primary list of drives and volumes on the left,
with details of a selected item on the right, organized
into a tree of "key:value" entries.
This is functional, but the right pane is not exactly easy to follow. So the first change will be to tabulate the key-value pairs:
Next, note that items in bold act like section
headers, so they don't really need to be collapsible.
So we do away with the buttons and connecting lines
between top-level items:
This gets us a bit of extra horizontal space and trims
a useless, but still actionable UI element.
Next, a couple of tweaks to how we render selected items.
There's no need for any items on the right to be selectable at all, so we won't render a selection highlight in the right pane.
The left pane can use a less jarring selection style, consistent with how Windows itself does it.
Next, we try and improve readability of the data on the
right by helping to guide the eye from a key to its
value.
The simplest option is the grid lines:
At this point, we take a sip of our cold coffee, squint
an eye and assess if we like how it looks.
It's not bad, but we can try something else:
Now, that's not bad either and it has a nice
visual consistency with the selection bar in the
left pane.
However a minor nit here is that we now have bold items (section headers) paired randomly with row highlights. See how Storage space is on white and Partition table is on light gray?
Moreover, adding or expanding items will cause this pairing to flip-flop, e.g. we add an extra row about Partition table and it now sits on the white background:
That's not a disaster, but we can do better.
The idea is to recognize section headers as such and style them independently:
With this change in place we can just restart the row
highlight sequence at the top of each section and
all's good.
Next, we apply the same styling to the left pane as it happens to also comprise of sections:
Another sip of coffee, re-squint the eye and
the whole thing now looks a bit cluttered and
busy.
Easy enough to rectify by adding a bit of vertical padding:
Next, we remove vertical connecting lines leading
from section headers to the top-level items.
However, we remove them on the right, but keep them on the left. This is because on the left we have a hierarchy of devices, so connection lines make sense. It also seems to look better this way.
Next, we move the expand/collapse buttons to
the right.
In this particular app expandable items contain secondary details that 99% of users will not ever need. These items will also be collapsed by default (even though they are expanded here, but that's just because we are sketching through the details).
This change also helps decreasing the leading pad of top-level items and providing for a more compact horizontal packing of the data.
Alright, so that's about it as far as functional
tweaks go. So now it's where it gets subjective. We are
going to spruce it up a bit.
It's all about colors, padding and little nuances. If we were designing for the web, it would also involve texturing, but for a desktop app texturing is a sure way to ruin native look and feel.
In any case, moving on - invert the headers from dark-on-light to light-on-dark:
Add a bit of hue to the row highlights, tint their
edges a bit and also flatten the look of
expand/collapse buttons:
Splice in a scroll bar and make sure our precious design
still looks reasonable:
Re-style the left pane to look more like an actual connected
hierarchy of disks and volumes. The rationale here is to make
it more obvious that items on the left are clickable.
Almost there. Take a final look and adjust vertical padding
and spacing to de-clump items a bit more:
And here it is. Compare to the
starting point and you'd
probably agree that there's at least some improvement.
Needless to say, one man's extra vertical padding is another man's complete waste of space, but any customization is always a combination of rational and subjective choices. The former is what gives an app the ease of use and the latter is what give it the personality. Can't have one without the other.
Diskovery is a little companion app that throws
a simple UI on top of the new S.M.A.R.T. diagnostics
module of Bvckup 2.
The app will go live shortly and its main purpose is to let us estimate how well the diagnostics module fares against the real world hardware. Because there are specs and there's life.
In particular, certain USB drives and older RAID controllers don't always cooperate fully (or at all), and it would help to know what- and how common they are.
Once released, Diskovery will remain publicly available and we'll update and support it. It's a simple tool for quickly inspecting one's storage hardware and I can think of a number of cases where it can come handy.
The app will go live shortly and its main purpose is to let us estimate how well the diagnostics module fares against the real world hardware. Because there are specs and there's life.
In particular, certain USB drives and older RAID controllers don't always cooperate fully (or at all), and it would help to know what- and how common they are.
Once released, Diskovery will remain publicly available and we'll update and support it. It's a simple tool for quickly inspecting one's storage hardware and I can think of a number of cases where it can come handy.
Having a working backup is great, but knowing
if any drives are nearing their lifespan is
just as essential.
As you might've heard the next release of Bvckup 2 will be adding support for the drive health monitoring via S.M.A.R.T. diagnostics.
However, first things first, and today I'd like to do a quick review of what's what in the Windows data storage stack.
Read in full here => Drives, Partitions, Volumes
As you might've heard the next release of Bvckup 2 will be adding support for the drive health monitoring via S.M.A.R.T. diagnostics.
However, first things first, and today I'd like to do a quick review of what's what in the Windows data storage stack.
Read in full here => Drives, Partitions, Volumes
Final remark on general email config improvements in R75.
When connecting to a mail server over SSL/TLS, it's not uncommon for the server to present a certificate that doesn't match its hostname.
When connecting to a mail server over SSL/TLS, it's not uncommon for the server to present a certificate that doesn't match its hostname.
In particular, this happens when a custom domain is used
with a managed (shared) mail service. You'd grab a domain,
set its MX to mx.domain, add CNAME for mx.domain
to point at mx.mail-service and - voilà! -
you are served with mx.mail-service cert when
connecting to mx.domain.
Ultimately, this is a mail server mis-configuration issue - your MX should really be pointing at mx.mail-service. In theory this shouldn't be happening, but in practice it does.
—
To address this, it's now possible to specify the exact domain the app should be looking for in an SSL certificate:
In the above example, you'd simply set this field to mx.mail-service and that's it.
—
To make things a bit easier that app now also helps you to pre-populate this field with the actual domain from the server's certificate.
It does that when you send out a test email and the field is left blank. As you can see from an opening screenshot, the app will recognize the certificate mismatch and re-run the test, accepting any and all certificates. It will then make note of the domain in the cert and copy it into the "SSL domain" field for you.
—
Finally, setting "SSL domain" to * will effectively disable all certificate checks and force the app to accept any certificate that the server sends, even if it's an expired or a self-signed one.
Ultimately, this is a mail server mis-configuration issue - your MX should really be pointing at mx.mail-service. In theory this shouldn't be happening, but in practice it does.
To address this, it's now possible to specify the exact domain the app should be looking for in an SSL certificate:
In the above example, you'd simply set this field to mx.mail-service and that's it.
To make things a bit easier that app now also helps you to pre-populate this field with the actual domain from the server's certificate.
It does that when you send out a test email and the field is left blank. As you can see from an opening screenshot, the app will recognize the certificate mismatch and re-run the test, accepting any and all certificates. It will then make note of the domain in the cert and copy it into the "SSL domain" field for you.
Finally, setting "SSL domain" to * will effectively disable all certificate checks and force the app to accept any certificate that the server sends, even if it's an expired or a self-signed one.
† GIF is a bit choppy, sorry about that.
Earlier versions had general and alert-specific settings
mixed up. For example, "From" doesn't
really belong to the alert configuration,
while "Include" from
General Settings does.
This has been sorted out.
A larger picture here is that the app will be getting additional alert types - e.g. for S.M.A.R.T. diagnostics and backup verification - and this requires a proper structure to support these cleanly.
—
Also, while at it, the UI got a new framework for handling window transitions between different states.
For example, in the above screencap there's are 5 primary states and 2 intermediate ones. Prior to R75 moving window between these states required quite a bit of hand coding. This is no more, so we can now animate the hell out of any window. Consider yourselves warned :)
This has been sorted out.
A larger picture here is that the app will be getting additional alert types - e.g. for S.M.A.R.T. diagnostics and backup verification - and this requires a proper structure to support these cleanly.
Also, while at it, the UI got a new framework for handling window transitions between different states.
For example, in the above screencap there's are 5 primary states and 2 intermediate ones. Prior to R75 moving window between these states required quite a bit of hand coding. This is no more, so we can now animate the hell out of any window. Consider yourselves warned :)
How about a rant? :)
The year is 2015.
The age of cars trying to drive themselves and rockets trying to land on sea barges. Graphics cards are effortlessly churning photo-realistic images in real-time, mining crypto money and cracking passwords.
And here we have Windows still defiantly refusing to use any of that double-buffering nonsense when re-painting one of its basic common controls. Because who wouldn't want a bit of flickering?
Argh.
The age of cars trying to drive themselves and rockets trying to land on sea barges. Graphics cards are effortlessly churning photo-realistic images in real-time, mining crypto money and cracking passwords.
And here we have Windows still defiantly refusing to use any of that double-buffering nonsense when re-painting one of its basic common controls. Because who wouldn't want a bit of flickering?
Argh.
Another iteration over tabbed dialogs, now with a bit of animation.
Tab control for multi-page configuration windows in general,
and for the email notification configuration in particular.
Prior to R75 there was one primary email alert, which was sent on backup completion. Now the app is getting other types of alerts, so the UI needs to be adapted to support that.
—
The idea is that the config window for specific alert type will give access to both the alert details and general email settings that are recycled between all alert types.
So here are several sketches for organizing page index of a multi-page configuration window. Nothing terribly novel here, it's more just a matter of getting styling details right - make sure it's obvious that it's an index and that its items are selectable.
—
Option Z (right-most) looks OK, but it's not obvious that index items are actionable as they look like static section headers.
Option Y is a bit better, but the arrow actually points at "From" field and you can bet some people will end up in a stupor because of that.
Option X uses standard Windows white-on-blue for selected item, so it's better in the actionability department. But it doesn't look nice.
Option B attempts to fix that.
Option A attempts the same but using app's own styling of selected items, which is already used in several places in the UI.
Thoughts or comments? There is a thread for that!
Prior to R75 there was one primary email alert, which was sent on backup completion. Now the app is getting other types of alerts, so the UI needs to be adapted to support that.
The idea is that the config window for specific alert type will give access to both the alert details and general email settings that are recycled between all alert types.
So here are several sketches for organizing page index of a multi-page configuration window. Nothing terribly novel here, it's more just a matter of getting styling details right - make sure it's obvious that it's an index and that its items are selectable.
Option Z (right-most) looks OK, but it's not obvious that index items are actionable as they look like static section headers.
Option Y is a bit better, but the arrow actually points at "From" field and you can bet some people will end up in a stupor because of that.
Option X uses standard Windows white-on-blue for selected item, so it's better in the actionability department. But it doesn't look nice.
Option B attempts to fix that.
Option A attempts the same but using app's own styling of selected items, which is already used in several places in the UI.
Thoughts or comments? There is a thread for that!
Minor tweak to the look of the delta copying progress bar -
a bit more contrast, both for completed and outstanding
parts.
Will be in R74.15
Will be in R74.15
Third pass.
Second pass over backup grouping UI for R75.
I think it's getting there...
I think it's getting there...
First stab at backup grouping UI for the next release (R75).
Thoughts or comments? There is a thread for that!
Thoughts or comments? There is a thread for that!
There's a good guiding principle coming from Paul Graham that says
make something people want.
It's very simple and it's very effective.
It also makes real-world user opinions a critical part of any
good product design.
In other words - please take a moment and talk to us! :)
In other words - please take a moment and talk to us! :)
Unicode,
as you may know, is a standard that assigns pretty pictograms to
numerical codes from the 0 to 0x0010FFFF range.
In simpler cases pictograms represent letters, numbers and symbols. In other cases they are decorative glyph elements, like umlauts, that are meant to be combined with the letters.
—
There are also Unicode encodings - UTF-8, UTF-16, etc. - these deal with how specific Unicode numerical IDs are translated into sequences of *bytes* for storage, transmission, etc.
The simplest encoding is UTF-32 - it simply stores Unicode ID as a 32-bit number. This is however rather wasteful as at most 21 bit is needed to represent any Unicode ID.
Prior to Windows 2000, Microsoft used a simple 16-bit encoding that mapped every Unicode ID from the 0 to 0x00FFFF range to the exact same 16-bit number. This is called UCS-2 encoding. The Unicode range was smaller back then, so this approach worked.
Then, Unicode range grew and IDs no longer fit into 16 bits. So to address this, UCS-2 was modified to encode symbols from the 0x010000 - 0x10FFFF range using two 16-bit numbers, from the 0xD800 - 0xDFFF range. Exact details of this encoding are not important, but what it meant was that some 16-bit numbers, e.g. 0xD812, were no longer valid Unicode symbols just on their own.
This is called UTF-16 encoding, and this is what Windows uses at the moment.
But wait !
As Apple likes to say after a screw-up - "it turns out" that despite routinely talking about UTF-16 in their API documentation, Windows file system still uses plain 16-bit encoding for the file names!
For example, "\xD812Haha" is a valid file name even though it is *not* a valid UTF-16 string. This in turn means that if the app is converting file names into other encoding, it should go easy on full UTF-16 compliance checks and be ready to digest ill-formed UTF-16 string.
—
Bvckup 2 keeps all text in UTF-8 format internally, so it does quite a bit of conversion from/to UTF-16 and this leads to some interesting things when the app runs into an malformed UTF-16 name. Like those created by *ahem* Adobe *ahem* After Effects *ahem*, for example.
The solution is simple - try and convert as if we have the UTF-16 encoding, but fallback to UCS-2 when running into invalid UTF-16 sequences. Also, apply the same approach when converting back to UTF-16 and all will be good.
Will be in Release 74.8.
* Kudos to Andrea for reporting this.
In simpler cases pictograms represent letters, numbers and symbols. In other cases they are decorative glyph elements, like umlauts, that are meant to be combined with the letters.
There are also Unicode encodings - UTF-8, UTF-16, etc. - these deal with how specific Unicode numerical IDs are translated into sequences of *bytes* for storage, transmission, etc.
The simplest encoding is UTF-32 - it simply stores Unicode ID as a 32-bit number. This is however rather wasteful as at most 21 bit is needed to represent any Unicode ID.
Prior to Windows 2000, Microsoft used a simple 16-bit encoding that mapped every Unicode ID from the 0 to 0x00FFFF range to the exact same 16-bit number. This is called UCS-2 encoding. The Unicode range was smaller back then, so this approach worked.
Then, Unicode range grew and IDs no longer fit into 16 bits. So to address this, UCS-2 was modified to encode symbols from the 0x010000 - 0x10FFFF range using two 16-bit numbers, from the 0xD800 - 0xDFFF range. Exact details of this encoding are not important, but what it meant was that some 16-bit numbers, e.g. 0xD812, were no longer valid Unicode symbols just on their own.
This is called UTF-16 encoding, and this is what Windows uses at the moment.
As Apple likes to say after a screw-up - "it turns out" that despite routinely talking about UTF-16 in their API documentation, Windows file system still uses plain 16-bit encoding for the file names!
For example, "\xD812Haha" is a valid file name even though it is *not* a valid UTF-16 string. This in turn means that if the app is converting file names into other encoding, it should go easy on full UTF-16 compliance checks and be ready to digest ill-formed UTF-16 string.
Bvckup 2 keeps all text in UTF-8 format internally, so it does quite a bit of conversion from/to UTF-16 and this leads to some interesting things when the app runs into an malformed UTF-16 name. Like those created by *ahem* Adobe *ahem* After Effects *ahem*, for example.
The solution is simple - try and convert as if we have the UTF-16 encoding, but fallback to UCS-2 when running into invalid UTF-16 sequences. Also, apply the same approach when converting back to UTF-16 and all will be good.
Will be in Release 74.8.
* Kudos to Andrea for reporting this.
Release 74 also implements new delta copying defaults
and the results are ... interesting :)
—
Delta copying, when enabled, has always been used conditionally and the criteria was the size of the file. For smaller files the overhead of setting up the delta copier is larger than any potential savings, so these files are simply copied in full.
The file size threshold was initially set to 128KB and the value was inherited from the original Bvckup version, where it made sense because the I/O was generally slower back then.
Fast forward few years and 128KB is no longer a good default, so as of Release 71 it was increased to 2MB.
—
With Release 74 the criteria changes again.
First, the threshold goes up to 32MB.
Second, the delta copying will also be used for files between 2MB and 32MB, but only IF they were modified in the last 30 days OR if they were previously delta-copied.
The rationale is pretty simple - we try and not bother with delta copying unless files are large or they change frequently.
—
This change has two main effects.
1. Smaller backup state.
Delta copying stores per-block hashes in a backup's configuration directory. This is done for every file that is delta-copied, so the fewer files are, the less data ends up being in the \deltas folder.
The above screenshot shows the \deltas folder stats after backing up \Windows\System32 directory. And here are the same stats after backing up with an older version -
That is, twice the disk usage and nearly 10x more files and folders in the \deltas folder.
2. Faster overall backups.
For smaller files the overhead of setting up the delta state is comparable to the time needed for the actual copying. We skip the delta setup -> we complete the copying faster. Magic.
Before:
After:
That's nearly a 10% speed up. This is obviously specific to the contents of \Windows\System32, but it's not hard to extrapolate from here.
Delta copying, when enabled, has always been used conditionally and the criteria was the size of the file. For smaller files the overhead of setting up the delta copier is larger than any potential savings, so these files are simply copied in full.
The file size threshold was initially set to 128KB and the value was inherited from the original Bvckup version, where it made sense because the I/O was generally slower back then.
Fast forward few years and 128KB is no longer a good default, so as of Release 71 it was increased to 2MB.
With Release 74 the criteria changes again.
First, the threshold goes up to 32MB.
Second, the delta copying will also be used for files between 2MB and 32MB, but only IF they were modified in the last 30 days OR if they were previously delta-copied.
The rationale is pretty simple - we try and not bother with delta copying unless files are large or they change frequently.
This change has two main effects.
1. Smaller backup state.
Delta copying stores per-block hashes in a backup's configuration directory. This is done for every file that is delta-copied, so the fewer files are, the less data ends up being in the \deltas folder.
The above screenshot shows the \deltas folder stats after backing up \Windows\System32 directory. And here are the same stats after backing up with an older version -
That is, twice the disk usage and nearly 10x more files and folders in the \deltas folder.
2. Faster overall backups.
For smaller files the overhead of setting up the delta state is comparable to the time needed for the actual copying. We skip the delta setup -> we complete the copying faster. Magic.
Before:
After:
That's nearly a 10% speed up. This is obviously specific to the contents of \Windows\System32, but it's not hard to extrapolate from here.
Release 74 will also be adding an option
for exporting app's complete configuration
as a single .zip file.
Restoring app's config from this backup is a simple matter of shutting down the app and extracting contents of the .zip into %LocalAppData%\Bvckup2\ folder.
But since we can hardly expect anyone to know this location by heart, a short text file called How to restore.txt is also included with each backup zip.
Restoring app's config from this backup is a simple matter of shutting down the app and extracting contents of the .zip into %LocalAppData%\Bvckup2\ folder.
But since we can hardly expect anyone to know this location by heart, a short text file called How to restore.txt is also included with each backup zip.
Run-time memory usage is now capped.
Prior to Release 74 app's memory usage was proportional to the number of items in the backup. This was because both the source and destination file trees were kept in memory in their entirety, as was the backup plan. While not a big deal for smaller backups, it does translate into quite a bit of RAM when the file count starts to climb into a million range.
—
Release 74 implements the off-RAM storage for large data sets. In previous release the run-time arrangement was like this -
With the new release the objects are kept in the app's memory only if they are in active use. The rest is trickled down into a swap file for storage, through a couple of intermediate caches.
The object cache holds fully assembled instances of application data objects and its size is kept at or below preset item count.
When the object cache grows too big, least recently used objects are written down to the swap file. The writes are passed through a typical page cache to coalesce the I/O into larger chunks.—
This lunch is obviously not free.
There's a cost to even successful cache lookups and there's most certainly an expense to doing the disk I/O.
The app mitigates this by using aggressive cache optimization (buckets, MRU lookups, etc), raw IO caching and also by allowing keeping swap files in a custom location, and presumably on a different physical drive. But, again, all this starts to really matter only when backups grow into a million item range.
While reasonable defaults are provided, the cache sizes, the trimming policy and other details are fully configurable through the .ini files. The app may also be set up to log cache/swap performance if required.—
This change is big, it took nearly two months to implement and it means that Bvckup 2 can now handle backups of unlimited size. To say that it's a very big deal would be an understatement.
Prior to Release 74 app's memory usage was proportional to the number of items in the backup. This was because both the source and destination file trees were kept in memory in their entirety, as was the backup plan. While not a big deal for smaller backups, it does translate into quite a bit of RAM when the file count starts to climb into a million range.
Release 74 implements the off-RAM storage for large data sets. In previous release the run-time arrangement was like this -
With the new release the objects are kept in the app's memory only if they are in active use. The rest is trickled down into a swap file for storage, through a couple of intermediate caches.
The object cache holds fully assembled instances of application data objects and its size is kept at or below preset item count.
When the object cache grows too big, least recently used objects are written down to the swap file. The writes are passed through a typical page cache to coalesce the I/O into larger chunks.
This lunch is obviously not free.
There's a cost to even successful cache lookups and there's most certainly an expense to doing the disk I/O.
The app mitigates this by using aggressive cache optimization (buckets, MRU lookups, etc), raw IO caching and also by allowing keeping swap files in a custom location, and presumably on a different physical drive. But, again, all this starts to really matter only when backups grow into a million item range.
While reasonable defaults are provided, the cache sizes, the trimming policy and other details are fully configurable through the .ini files. The app may also be set up to log cache/swap performance if required.
This change is big, it took nearly two months to implement and it means that Bvckup 2 can now handle backups of unlimited size. To say that it's a very big deal would be an understatement.
Still working on major engine changes to cap its
run-time use (and I will be posting data from the
test runs shortly), but meanwhile here's something
simple that I'm considering adding to the next
release.
It's an option for the app to report your backup statistics to one of our servers and then being able to display these stats as a simple PNG served over HTTP from the same stats server.
It's really meant for trying to motivate other people to back up their things as well as to indirectly promote the app as well. And, yes, I lived through the 90s, that's where I got the idea from :)
It's an option for the app to report your backup statistics to one of our servers and then being able to display these stats as a simple PNG served over HTTP from the same stats server.
It's really meant for trying to motivate other people to back up their things as well as to indirectly promote the app as well. And, yes, I lived through the 90s, that's where I got the idea from :)
Currently working on the improvements to the
scanning and planning modules of the app. No pretty
UI screenshots to show, just some performance numbers.
—
The ultimate goal of this rework is to put a cap on the run-time memory usage, which in turn is needed to better accommodate very large backups (tens of millions files and up).
—
The app comes with a formal backup planner, which is a module that accepts two full directory trees (one for the source location and another for the backup one), digests them and produces a list of simple steps that, when executed in order, bring one tree in sync with another. The app is then diligently goes through the list and creates, updates, renames and deletes files and folders as directed.
An alternative to formal planning is to to traverse the trees and make on-disk changes just as you spot the differences. There are several drawback to this approach, including not being able to estimate the amount of work needed, to understand disk space requirements and to detect file/folder moves. It's just an altogether messier and less predictable way to go about the whole thing.
That's how Bvckup 1 worked and that's how robocopy works, for example.
—
Long story short, having a backup planner is the right thing to do. However, it means that the app needs to have two full file trees readily available for the planner to digest. If these trees are kept entirely in memory, it may get expensive with large backups. At 100 to 300 bytes per item, a million file backup means 200 to 600 megs of RAM just for the trees.
That's quite a bit, but more importantly this memory usage grows linearly with the size of the backup. This is no good.
—
The solution obviously enough is to not keep full trees in memory. This is a fairly complicated endeavour and it maps onto several large changes to the app inner structure.
One such change is nearly done. Both the scanning planning modules no longer require for the trees to be resident in memory at all times. Instead, they now use them in a piecemeal fashion.
Behind the scenes, trees still reside in memory for the time being, but the underlying rework has some nice side effects. Trees now take less memory, they are faster to populate, faster to save and load to/from a file, they take substantally less space on disk and they compress better with NTFS compression.
This will ship as a part of R74.
After that, R75 will replace in-memory trees with version that can swap out tree parts to the disk, significantly reducint the run-time memory footprint of the scanning/planning process.
The ultimate goal of this rework is to put a cap on the run-time memory usage, which in turn is needed to better accommodate very large backups (tens of millions files and up).
The app comes with a formal backup planner, which is a module that accepts two full directory trees (one for the source location and another for the backup one), digests them and produces a list of simple steps that, when executed in order, bring one tree in sync with another. The app is then diligently goes through the list and creates, updates, renames and deletes files and folders as directed.
An alternative to formal planning is to to traverse the trees and make on-disk changes just as you spot the differences. There are several drawback to this approach, including not being able to estimate the amount of work needed, to understand disk space requirements and to detect file/folder moves. It's just an altogether messier and less predictable way to go about the whole thing.
That's how Bvckup 1 worked and that's how robocopy works, for example.
Long story short, having a backup planner is the right thing to do. However, it means that the app needs to have two full file trees readily available for the planner to digest. If these trees are kept entirely in memory, it may get expensive with large backups. At 100 to 300 bytes per item, a million file backup means 200 to 600 megs of RAM just for the trees.
That's quite a bit, but more importantly this memory usage grows linearly with the size of the backup. This is no good.
The solution obviously enough is to not keep full trees in memory. This is a fairly complicated endeavour and it maps onto several large changes to the app inner structure.
One such change is nearly done. Both the scanning planning modules no longer require for the trees to be resident in memory at all times. Instead, they now use them in a piecemeal fashion.
Behind the scenes, trees still reside in memory for the time being, but the underlying rework has some nice side effects. Trees now take less memory, they are faster to populate, faster to save and load to/from a file, they take substantally less space on disk and they compress better with NTFS compression.
This will ship as a part of R74.
After that, R75 will replace in-memory trees with version that can swap out tree parts to the disk, significantly reducint the run-time memory footprint of the scanning/planning process.
Release 73 adds copying speed and completion
time estimation for longer file copies. To
toggle between the views just click on the
text next to the progress bar.
—
The estimation is calculated in three steps.
First, the app looks at effective copying speed and uses sliding window to average it across last 25 samples taken 650 ms apart.
Second, it uses this average speed and the remaining byte count to approximate the completion time.
Finally, the completion time is averaged with its own sliding window, but using only 4 past samples.
This double-averaging helps mitigating the effects of sudden short-lived I/O bursts and hiccups on the estimation.
The parameters - 25, 650 and 4 - are configurable through the engine .ini file, so the whole thing is open for tuning if needs be.
The estimation is calculated in three steps.
First, the app looks at effective copying speed and uses sliding window to average it across last 25 samples taken 650 ms apart.
Second, it uses this average speed and the remaining byte count to approximate the completion time.
Finally, the completion time is averaged with its own sliding window, but using only 4 past samples.
This double-averaging helps mitigating the effects of sudden short-lived I/O bursts and hiccups on the estimation.
The parameters - 25, 650 and 4 - are configurable through the engine .ini file, so the whole thing is open for tuning if needs be.
The initial revision of the auto-update opton
is in.
It's a simple extension of existing "check for updates" option that skips showing the "new version found" prompt and just goes straight to the updating part.
—
There are two caveats, both applicable only when the app is running in service mode and both stemming from the fact the updates are driven by the UI.
First is that closing the UI will effectively disable auto-updates. Second - if the UI is running without admin privileges, running the installer will trigger the UAC prompt.
Both will be resolved shortly by moving all update logic into the engine module. Meanwhile...
The auto-update option will be hidden by default unless the Preferences window is opened with the Control button held down.
It's a simple extension of existing "check for updates" option that skips showing the "new version found" prompt and just goes straight to the updating part.
There are two caveats, both applicable only when the app is running in service mode and both stemming from the fact the updates are driven by the UI.
First is that closing the UI will effectively disable auto-updates. Second - if the UI is running without admin privileges, running the installer will trigger the UAC prompt.
Both will be resolved shortly by moving all update logic into the engine module. Meanwhile...
The auto-update option will be hidden by default unless the Preferences window is opened with the Control button held down.
Release 73 will also be adding support for monthly
backups. It will now be possible to set a backup
to run on the N-th of every M-th month.
You surely are thinking "but 31st!", "February!" or perhaps even "every 6 months staring on Jan 31st!". Good point, the UI got just the thing for this -
You surely are thinking "but 31st!", "February!" or perhaps even "every 6 months staring on Jan 31st!". Good point, the UI got just the thing for this -
A rework of the loggging module, coming up in Release 73.
—
As you may or may not know, Bvckup 2 uses a dual-module architecture of the inside.
As you may or may not know, Bvckup 2 uses a dual-module architecture of the inside.
One module is the backup engine. It maintains
the list of backup jobs, schedules, executes
backups and generally does other useful
things that don't require direct user involvement.
The second module is the user interface. This is the part that displays the app's window and allows users to monitor and control whatever the engine module is doing.
The principal point is that the engine and the UI are isolated from each other. They share no run-time data and communicate with each other using a messaging protocol.
This isolation is what enables the app to run in service mode. In this mode the engine runs in its own process, as a system service, and the UI runs as a regular desktop app.
It was all well and beautiful, but there was, however, a caveat.
—
The caveat was the backup logs.
Logs are inherently shared data - they are written by the engine and they are read and displayed by the UI. Logs tend to be bulky, so it is shared app data of a special kind and it requires special handling.
Prior to R73 the app included a third module (in its dual-module architecture, no less) - the log manager. It was a complicated contraption that abstracted log access for the engine and the UI, and the biggest issue with it was that it behaved completely differently depending on which mode the app was in.
In desktop mode, the manager was writing *and* indexing logs on behalf of the engine and the UI was merely displaying them. However, in service mode, the engine did no indexing, just the writing, while the UI was making copies of all logs for its own use and then it was indexing *them*.
This is exactly the kind of non-uniform conditional processing that the dual-module design was meant to avoid!
—
Enter R73.
The separate logging module is no more. The engine no longer does any log indexing, it is now entirely the UI's reponsibility. Indexes can now be rebuilt on demand at any time, and finally, the UI no longer creates its own copies of the log files. Instead, it works with the engine to gain access to the engine's copy of the logs, using DuplicateHandle to move open file handles across the process boundary when needed.
The UI now can display logs while it is still catching up with them.
Think -- running in service mode and launching the UI only once a week. On each launch the UI needs to index a week's worth of log entries it hasn't seen before. Prior to R73, the UI would just sit there and show "synching ... XX% done". Now the sync is done in the background (as per the screenshot).
The UI now can also recover gracefully from indexes getting corrupted (e.g. due to a cold power cycle or blue screen).
Just as important, the underlying code is now much simpler, lighter and more uniform, the executable is 8KB smaller and the developer is generally happier. Good news all around.
The second module is the user interface. This is the part that displays the app's window and allows users to monitor and control whatever the engine module is doing.
The principal point is that the engine and the UI are isolated from each other. They share no run-time data and communicate with each other using a messaging protocol.
This isolation is what enables the app to run in service mode. In this mode the engine runs in its own process, as a system service, and the UI runs as a regular desktop app.
It was all well and beautiful, but there was, however, a caveat.
The caveat was the backup logs.
Logs are inherently shared data - they are written by the engine and they are read and displayed by the UI. Logs tend to be bulky, so it is shared app data of a special kind and it requires special handling.
Prior to R73 the app included a third module (in its dual-module architecture, no less) - the log manager. It was a complicated contraption that abstracted log access for the engine and the UI, and the biggest issue with it was that it behaved completely differently depending on which mode the app was in.
In desktop mode, the manager was writing *and* indexing logs on behalf of the engine and the UI was merely displaying them. However, in service mode, the engine did no indexing, just the writing, while the UI was making copies of all logs for its own use and then it was indexing *them*.
This is exactly the kind of non-uniform conditional processing that the dual-module design was meant to avoid!
Enter R73.
The separate logging module is no more. The engine no longer does any log indexing, it is now entirely the UI's reponsibility. Indexes can now be rebuilt on demand at any time, and finally, the UI no longer creates its own copies of the log files. Instead, it works with the engine to gain access to the engine's copy of the logs, using DuplicateHandle to move open file handles across the process boundary when needed.
The UI now can display logs while it is still catching up with them.
Think -- running in service mode and launching the UI only once a week. On each launch the UI needs to index a week's worth of log entries it hasn't seen before. Prior to R73, the UI would just sit there and show "synching ... XX% done". Now the sync is done in the background (as per the screenshot).
The UI now can also recover gracefully from indexes getting corrupted (e.g. due to a cold power cycle or blue screen).
Just as important, the underlying code is now much simpler, lighter and more uniform, the executable is 8KB smaller and the developer is generally happier. Good news all around.
Running an extended set of tests to fine-tune the
performance of the Bvckup 2 bulk copier.
It's an inherently simple matter of determining an optimal size and count of read/write buffers used to ferry data from source to destination. There are however ... erm ... details.
—
A source can be local or it can be over-the-network. There are slow local devices (e.g. a USB2 drive) and there are fast local devices. There is a warm, there is a cold cache and there are files that are cached only partially.
The writes can go out in lots of smaller chunks, all in parallel, or with a smaller number of large buffers. Again, it can go to a local device or over the network. The app can also explicitly flush all buffers after copying or leave flushing to the system.
—
The above graph is the I/O throughput of a 0.5 GB file being repeatedly copied from the local HDD to Synology Diskstation over a Gigabit wire. What changes between the runs is the buffering strategy and as you may notice there is a difference.
Still need to run several over-nighters to gather more statistics, but there is already a couple of things that can be changed in the copying code to make it run faster.
Scheduled to appear in the Release 72, coming up shortly.
It's an inherently simple matter of determining an optimal size and count of read/write buffers used to ferry data from source to destination. There are however ... erm ... details.
A source can be local or it can be over-the-network. There are slow local devices (e.g. a USB2 drive) and there are fast local devices. There is a warm, there is a cold cache and there are files that are cached only partially.
The writes can go out in lots of smaller chunks, all in parallel, or with a smaller number of large buffers. Again, it can go to a local device or over the network. The app can also explicitly flush all buffers after copying or leave flushing to the system.
The above graph is the I/O throughput of a 0.5 GB file being repeatedly copied from the local HDD to Synology Diskstation over a Gigabit wire. What changes between the runs is the buffering strategy and as you may notice there is a difference.
Still need to run several over-nighters to gather more statistics, but there is already a couple of things that can be changed in the copying code to make it run faster.
Scheduled to appear in the Release 72, coming up shortly.
A quick peek at the Reseller Dashboard.
Coming soon...
"Send test email..." is now done.
R71 adds an option of automatically enabling
NTFS compression
for the backup logs.
Based on what I've seen so far, both .log and .tdb files compress to about 20-30% of their raw sizes.
Based on what I've seen so far, both .log and .tdb files compress to about 20-30% of their raw sizes.
Reworked part of the More Options page in
the backup configuration window.
* The animation here is a bit jerky, but it's just a GIF encoding artefact.
* The animation here is a bit jerky, but it's just a GIF encoding artefact.
Email template editor. "Show variables" opens
up another part that lists all variables that
can be used in the form fields.
This is getting a bit too complex for my taste, but it's the part of the UI with little foot traffic, so it's probably OK.
This is getting a bit too complex for my taste, but it's the part of the UI with little foot traffic, so it's probably OK.
A little primer on the mail server configuration.
Upon even further consideration it looks like
the From field is not of a prime interest either,
so it too can go into the Details.
This cuts down the number of fields and renders section headers unnecessary, so they too can go.
Basically the idea is to reduce the dialog down to fields that actually need user's input and stash everything else away... I don't know if it's the right thing to do though. Perhaps having a summary of the whole email setup (including the subject, From and the message) is not such a bad thing. Not sure. Need to sleep on it.
This cuts down the number of fields and renders section headers unnecessary, so they too can go.
Basically the idea is to reduce the dialog down to fields that actually need user's input and stash everything else away... I don't know if it's the right thing to do though. Perhaps having a summary of the whole email setup (including the subject, From and the message) is not such a bad thing. Not sure. Need to sleep on it.
Upon further consideration, decided to remove
both Subject and Message off this dialog to
keep it simple(r) and to the point.
First stab at the email configuration window.
Too many options, too little space. As usual.
EHLO there!
Release 71 is getting email notifications, so
here's all you always wanted to know about teaching
your app speak SMTP.
First of all, let's get MAPI non-sense out of the way. As per usual, it sounds good in theory, but an awful mess in practice. So, no, MAPI is not an option unless your goal is to burn through your support budget and/or practice your communication skills with less-than-happy users.
This leaves us with raw SMTP.
SMTP is an inherently simple protocol - send a text command, receive a text response. Repeat. Open a socket, connect, send(), recv(), done. A couple of hours of work at most. Ha.
—
It starts to get complicated when you throw TLS in a mix to secure the communication with the server. In particular, this is required when you want to send emails in a way that requires logging into a server first.
There are two ways in which SMTP can be combined with TLS.
First is when you take an existing SMTP session and say STARTTLS. The server says OK and then you basically stop talking SMTP over your TCP connection and start talking TLS. Then, once the TLS handshake completes, you go back to SMTP, but now it all goes over the TLS session, nice and encrypted. This is a standard way, that's what every decent SMTP client should do. This is TCP/25 and TCP/587.
Second option is SMTPS. You first establish a TLS session and then start talking SMTP over it. Just like HTTPS. Apparently, this is non-standard and discouraged way, but still widely used. This is TCP/465.
—
Then, there's a matter of how to add TLS support to the app.
The obvious choice is OpenSSL, but that will cost you 200-300KB in extra binary code, so unless you already have OpenSSL linked in, it's not really an option, not for a *backup* app.
Digging through MSDN you may come across something called SSPI or Security Support Provider Interface. It is an attempt to stuff several security protocols like SSL and Kerberos into a single framework in a faint hope for a miracle of reuse. One of SSPI providers is a mysterious entity called "Secure channel", which, upon closer inspection, turns out to be ... yes, TLS!
So, the next step is try and work through all the abstraction fluff of SSPI to make it actually do the TLS handshake. The idea is actually quite neat - you can create a stateful object (context) that takes in data you get from the peer and spits out some data for the peer. At some point it completes the handshake and becomes available for accepting app's own data, which it encrypts and hands you a blob to send to the peer. You would then also get blobs from the peer, pass it to this context/object and get back plaintext data for the app. Whether it's TLS or ROT13 on the back is completely transparent to the app.
That's the idea. In reality, the documentation is subpar and lacking and there are exactly two examples - one is incomplete and another one is buggy.
The good news is though is that once this bloody contraption gets off the ground, you get full TLS support at the cost of just several kilobytes.—
With TLS done, there's still a bunch of smaller things to be taken care of, e.g. lack of asynchronous DNS query API in pre-W8 windows. Not a big deal, nothing a spawning of a worker thread would solve, but still ... a bit here, a bit there and it all adds up.
In other words, it's all coming together, but as per usual it all takes longer than anticipated. Shocking, I know :)
First of all, let's get MAPI non-sense out of the way. As per usual, it sounds good in theory, but an awful mess in practice. So, no, MAPI is not an option unless your goal is to burn through your support budget and/or practice your communication skills with less-than-happy users.
This leaves us with raw SMTP.
SMTP is an inherently simple protocol - send a text command, receive a text response. Repeat. Open a socket, connect, send(), recv(), done. A couple of hours of work at most. Ha.
It starts to get complicated when you throw TLS in a mix to secure the communication with the server. In particular, this is required when you want to send emails in a way that requires logging into a server first.
There are two ways in which SMTP can be combined with TLS.
First is when you take an existing SMTP session and say STARTTLS. The server says OK and then you basically stop talking SMTP over your TCP connection and start talking TLS. Then, once the TLS handshake completes, you go back to SMTP, but now it all goes over the TLS session, nice and encrypted. This is a standard way, that's what every decent SMTP client should do. This is TCP/25 and TCP/587.
Second option is SMTPS. You first establish a TLS session and then start talking SMTP over it. Just like HTTPS. Apparently, this is non-standard and discouraged way, but still widely used. This is TCP/465.
Then, there's a matter of how to add TLS support to the app.
The obvious choice is OpenSSL, but that will cost you 200-300KB in extra binary code, so unless you already have OpenSSL linked in, it's not really an option, not for a *backup* app.
Digging through MSDN you may come across something called SSPI or Security Support Provider Interface. It is an attempt to stuff several security protocols like SSL and Kerberos into a single framework in a faint hope for a miracle of reuse. One of SSPI providers is a mysterious entity called "Secure channel", which, upon closer inspection, turns out to be ... yes, TLS!
So, the next step is try and work through all the abstraction fluff of SSPI to make it actually do the TLS handshake. The idea is actually quite neat - you can create a stateful object (context) that takes in data you get from the peer and spits out some data for the peer. At some point it completes the handshake and becomes available for accepting app's own data, which it encrypts and hands you a blob to send to the peer. You would then also get blobs from the peer, pass it to this context/object and get back plaintext data for the app. Whether it's TLS or ROT13 on the back is completely transparent to the app.
That's the idea. In reality, the documentation is subpar and lacking and there are exactly two examples - one is incomplete and another one is buggy.
The good news is though is that once this bloody contraption gets off the ground, you get full TLS support at the cost of just several kilobytes.
With TLS done, there's still a bunch of smaller things to be taken care of, e.g. lack of asynchronous DNS query API in pre-W8 windows. Not a big deal, nothing a spawning of a worker thread would solve, but still ... a bit here, a bit there and it all adds up.
In other words, it's all coming together, but as per usual it all takes longer than anticipated. Shocking, I know :)
Finalized the UX for auto-configuration of
the device tracking. It's essentially the
same as animation immediately below except
for a brief flash of blue when tracking
config is touched by the app.
Another subtle UI change in Release 71.
When configuring or reconfiguring a backup, the app would automatically enable/disable device tracking when a location is changed to use a different drive.
Since this change happens behind the scenes, the app will attract the attention to the fact with this little animation. When hovered over, the tooltip for the pushpin button will explain what's up and suggest reviewing the changes.
My only concern is that this still might be a bit too subtle, but it's better than what's in there now... which is no notification at all.
When configuring or reconfiguring a backup, the app would automatically enable/disable device tracking when a location is changed to use a different drive.
Since this change happens behind the scenes, the app will attract the attention to the fact with this little animation. When hovered over, the tooltip for the pushpin button will explain what's up and suggest reviewing the changes.
My only concern is that this still might be a bit too subtle, but it's better than what's in there now... which is no notification at all.
New UI for configuring device tracking in R71.
This is one of those changes that look almost trivial when done, but behind the scenes take a disproportionate amount of time to actually flesh out.
In this case, the main problem was that the feature needs a detailed explanation and the question, of course, was how one can fit a bunch of text into a reasonably small space so not to have this text take over the actual configuration section.
—
The first iteration was the good old message box. You click on the "More on this..." and you get :
Nice, but looks like a hackjob, does't it? So, naturally, the next step was to spruce it up a bit - put text into a dedicated window, use RTF to add a bit of in-text formatting and a bit of animation to liven things up :
Nice again, looks better, but feels off. Too much going on. It looks like a conventional help window, but ... crippled. It's also not clear how to act when the user wants to resize the window - allow it, not allow it, restict minimum/maximum size, etc.
In short, what's lacking here is the cohesion. Luckily, the solution is right on the surface:
... whereby the text panel slides out on the "More info" click.
This is better, but now we have a window that sits lopsided on the desktop when the panel is open. It is also becoming wide, meaning that the app needs to mind the desktop edges, re-center the window, etc.
But if we just sleep on it, we might suddenly decide to make it an "either-or" window and then the UI magically becomes
(a) compact
(b) cohesive
(c) balanced
and
(d) with unlimited text space!
So there you have it - a new mini-help subsystem coming to Bvckup 2 in the Release 71, deployed on as-needed basis across the app.
Don't know about you, but I think it's neat, tidy and pretty damn clever :)
This is one of those changes that look almost trivial when done, but behind the scenes take a disproportionate amount of time to actually flesh out.
In this case, the main problem was that the feature needs a detailed explanation and the question, of course, was how one can fit a bunch of text into a reasonably small space so not to have this text take over the actual configuration section.
The first iteration was the good old message box. You click on the "More on this..." and you get :
Nice, but looks like a hackjob, does't it? So, naturally, the next step was to spruce it up a bit - put text into a dedicated window, use RTF to add a bit of in-text formatting and a bit of animation to liven things up :
Nice again, looks better, but feels off. Too much going on. It looks like a conventional help window, but ... crippled. It's also not clear how to act when the user wants to resize the window - allow it, not allow it, restict minimum/maximum size, etc.
In short, what's lacking here is the cohesion. Luckily, the solution is right on the surface:
... whereby the text panel slides out on the "More info" click.
This is better, but now we have a window that sits lopsided on the desktop when the panel is open. It is also becoming wide, meaning that the app needs to mind the desktop edges, re-center the window, etc.
But if we just sleep on it, we might suddenly decide to make it an "either-or" window and then the UI magically becomes
(a) compact
(b) cohesive
(c) balanced
and
(d) with unlimited text space!
So there you have it - a new mini-help subsystem coming to Bvckup 2 in the Release 71, deployed on as-needed basis across the app.
Don't know about you, but I think it's neat, tidy and pretty damn clever :)
Added support for the reseller licensing
to the app. Any license can now be tagged as
being sold through a distributor, in which
case Help/About will show distributor's name
and link to its site.
Additionally, Tech Support and Homepage links from the Help menu may be configured to go to the distributor's pages.
It may not look like much, but it's the most important non-technical feature to be added to the app to date.
More on this to follow...
Additionally, Tech Support and Homepage links from the Help menu may be configured to go to the distributor's pages.
It may not look like much, but it's the most important non-technical feature to be added to the app to date.
More on this to follow...
Another option, as per
this comment.
It's a really nice concept, but it requires a squint
to realize what the icon is. Highly-detailed 16x16 px
icons are hard to pull off.
Release 71 is going to reshuffle the contents of
the backup configuration window a bit and location
fields are getting equipped with a "device tracking"
icon as a result.
In other words, for a network location the app will display the "key" icon (see /wip entry immediately below) and for local paths it will show "tracking" icon. Both fronting respective dialog for configuring the details.
The main question, of course, is what the icon should be ...
In other words, for a network location the app will display the "key" icon (see /wip entry immediately below) and for local paths it will show "tracking" icon. Both fronting respective dialog for configuring the details.
The main question, of course, is what the icon should be ...
In a process of adding a way to configure
network logon credentials for remote locations.
Will be in Release 70.
Release 69 also adds an option of simulated backups
or, as they are also known, "dry runs".
The option is accessible via the right-click context menu and, when invoked, it puts the backup through its preparation phase, including snapshot creation, scanning and planning steps, and then dumps compiled backup plan to the log without executing it.
Simulation also executes the archive pruning without making any actual changes to the destination.
The option is accessible via the right-click context menu and, when invoked, it puts the backup through its preparation phase, including snapshot creation, scanning and planning steps, and then dumps compiled backup plan to the log without executing it.
Simulation also executes the archive pruning without making any actual changes to the destination.
Release 69 adds support for optional VSS writers.
—
VSS writers are the components of the shadow copying framework that are tasked with ensuring consistency of specific files in VSS snapshots. For example, when creating a snapshot, Windows would ask the MSSQL writer to flush any pending database changes in a way that would make on-disk copy of the database self-consistent.
Some of the writers are required and always participate in the snapshot creation process, but some are optional and they need to be explicitly enabled by the initiating app. Starting with R69 Bvckup will be enabling all optional writers and it will also provide a mechanism for excluding specific ones if needed.
VSS writers are the components of the shadow copying framework that are tasked with ensuring consistency of specific files in VSS snapshots. For example, when creating a snapshot, Windows would ask the MSSQL writer to flush any pending database changes in a way that would make on-disk copy of the database self-consistent.
Some of the writers are required and always participate in the snapshot creation process, but some are optional and they need to be explicitly enabled by the initiating app. Starting with R69 Bvckup will be enabling all optional writers and it will also provide a mechanism for excluding specific ones if needed.
It's shipped!
Here it is
Here it is
Nearly done with the checkout flow on the
website. Integrated with Stripe yesterday,
now finalizing the visuals.
This is a purchase receipt. It is both sent out and shown on the site when the payment goes through.
PS. All numbers and codes are made up. Don't get any ideas :)
This is a purchase receipt. It is both sent out and shown on the site when the payment goes through.
PS. All numbers and codes are made up. Don't get any ideas :)
Working on the production version of the website.
Cleaned up overall look and feel a bit, got rid of baby blue theme and anorexic body font, reworked the copy, expanded Features section and so on. Still need to slice and code it. Also asked a couple of people for the feedback, so waiting on that too.
Cleaned up overall look and feel a bit, got rid of baby blue theme and anorexic body font, reworked the copy, expanded Features section and so on. Still need to slice and code it. Also asked a couple of people for the feedback, so waiting on that too.
The feature everyone was so waiting for ... the payment form :)
Working through the necessary evil - the app licensing :)
The plan is to have a 14 day trial that doesn't require any explicit licensing. Just install and run. It will pop up a notification half way through the trial period and it will also say something on the last day.
Once the trial period is over and until a production (paid) license is installed, the app will go into an unlicensed mode and it will activate two restrictions (see screenshot). It will remain largely functional, but presumably not convenient enough for regular or unattended use.
You would then hit /purchase page on the website, go through the purchase process and get an activation code in return. The code will then need to be sticked into the activation form on the client (see 2nd screenshot) and it will fetch the license from the server.
If the licensing server is inaccessible for whatever reason, there's a couple of ways to retrieve the license manually (see last screenshot).
For those with Beta licenses, they will be accepted at the checkout as a discount code.
For those looking at large deployments / enterprise setups, there will be an option for automating licensing process. Get in touch for details if you have questions.
Updated the About window a bit.
Current design has somehow always felt ... cramped?
Super-final version of the progress bar as seen in Beta 60.
Note how colors of some parts are inverted under selection,
took me disturbingly long to arrive at this solution :-/
Finalized changes to the progress bar - now with more
contrast, a bit of depth and beautifully centered
splatter for partially copied blocks.
And here is a bunch of runners-up if anyone's interested.
And here is a bunch of runners-up if anyone's interested.
Just waiting for the test build to compile...
so here's another take on the delta copier
progress bar (and here's
existing version for comparison).
Clicking and holding error navigation arrow for 750 ms
now activates auto-scrolling. It starts at 10 errors
per second and accelerates linearly to 100 err/sec,
which is pretty damn fast.
Additionally, holding Ctrl while scrolling with the mouse wheel makes it go in 10 line increments.
Additionally, holding Ctrl while scrolling with the mouse wheel makes it go in 10 line increments.
It is now possible to clear the log view and
then revert it back to the full log view.
A smaller UI change.
When hopping through errors in a backup log, log viewer will now briefly highlight next entry if it cannot be pulled directly under the cursor. This basically adds a visual terminator to the interaction sequence, which is a surprsingly useful thing to have.
Otherwise it works as before.
When hopping through errors in a backup log, log viewer will now briefly highlight next entry if it cannot be pulled directly under the cursor. This basically adds a visual terminator to the interaction sequence, which is a surprsingly useful thing to have.
Otherwise it works as before.
Having thought about it a bit more, I can even
remove that slight upward slope from the hump
of the blue graph. The usage still climbs just
a little bit during the run, but for practical
intents and purposes it's flat.
Wrapping up changes to the logging system. Here's
a quick graph to demonstrate the effects of the
rework.
For an initial full run of a 100,000 file backup
memory usage is down from about 200 to 50 Megs.
For a 1,000,000 file backup the effect is similar
- peak memory usage dropped
from 2.9GB to 0.58GB
.
—
Previously, the UI would keep logs of last few backup runs in the memory and for larger backups this would eat up a good chunk of memory. This was especially true for the initial backups where all files needed copying and thus for 100K files it would generate close to half a million log entries.
For subsequent backups it wasn't a big deal as they would handle just the changes and generate only a handful of steps and a bit of log data So in the end, the memory usage was capped, but the cap was higher than it could've been.
Beta 60 addresses that.
—
Only a very small part of the log is now kept in memory, just enough to fill the log viewer panel in the UI. The rest is kept on the disk and read from there on as-needed basis.
However this is easier said than done.
Bvckup logs are hierarchical, meaning that they are generic trees with nodes that may have an arbitrary large number of children. Running a 1 million file backup would yield a "Processing..." entry with 1 million children, each with up to a dozen of descendants.
—
Displaying this tree would've not been a big deal if all log entries were always visible at all times. When the UI would've wanted to show entries 700,503 through 700,523 entries, it would've simply gone to a respective location in the log index file and plucked the data out. But...
The nodes can be collapsed and expanded. This hides and shows arbitrary chunks of the log tree and then the UI has this issue of finding 700,503-th *visible* item. And this is hard.
This implies quick look-ups in some sort of super-imposed binary search tree of visible items. A tree that changes every time a node is expanded or collapsed and that sits in a disk file, which restricts how you can walk and modify it without getting an I/O penalty.
This is fun. This is what cost me 3 weeks of shuffling through CS books and several complete rewrites of the tree indexing code. Believe it or not, but this is basically an original research and it seems that no one had to deal with this issue before. At least not publicly.
—
The solution is to use slightly customized version of an AVL tree to store the list of node's children (that's using tree within a tree), use it for the visible item look-ups and then optimize the hell out of the index I/O through aggressive write caching.
When all i's are dotted and t's crossed, this yields a logging system that can push up to 1 mil entries per second on a resonably equipped box. This might not be knock-your-socks-off impressive, but it's plenty enough for the task at hand.
Previously, the UI would keep logs of last few backup runs in the memory and for larger backups this would eat up a good chunk of memory. This was especially true for the initial backups where all files needed copying and thus for 100K files it would generate close to half a million log entries.
For subsequent backups it wasn't a big deal as they would handle just the changes and generate only a handful of steps and a bit of log data So in the end, the memory usage was capped, but the cap was higher than it could've been.
Beta 60 addresses that.
Only a very small part of the log is now kept in memory, just enough to fill the log viewer panel in the UI. The rest is kept on the disk and read from there on as-needed basis.
However this is easier said than done.
Bvckup logs are hierarchical, meaning that they are generic trees with nodes that may have an arbitrary large number of children. Running a 1 million file backup would yield a "Processing..." entry with 1 million children, each with up to a dozen of descendants.
Displaying this tree would've not been a big deal if all log entries were always visible at all times. When the UI would've wanted to show entries 700,503 through 700,523 entries, it would've simply gone to a respective location in the log index file and plucked the data out. But...
The nodes can be collapsed and expanded. This hides and shows arbitrary chunks of the log tree and then the UI has this issue of finding 700,503-th *visible* item. And this is hard.
This implies quick look-ups in some sort of super-imposed binary search tree of visible items. A tree that changes every time a node is expanded or collapsed and that sits in a disk file, which restricts how you can walk and modify it without getting an I/O penalty.
This is fun. This is what cost me 3 weeks of shuffling through CS books and several complete rewrites of the tree indexing code. Believe it or not, but this is basically an original research and it seems that no one had to deal with this issue before. At least not publicly.
The solution is to use slightly customized version of an AVL tree to store the list of node's children (that's using tree within a tree), use it for the visible item look-ups and then optimize the hell out of the index I/O through aggressive write caching.
When all i's are dotted and t's crossed, this yields a logging system that can push up to 1 mil entries per second on a resonably equipped box. This might not be knock-your-socks-off impressive, but it's plenty enough for the task at hand.
A quick update on what's happening.
Disk-based virtual data structures, that's what is :)
This is a part of an effort to further reduce and cap app's run-time memory usage. More specifically, it is meant to help the app cope gracefully with multi-million file backups that involve millions of steps and generate lots and lots of log data.
—
At the moment Bvckup's UI keeps backup logs in memory. It puts an effort to trim them to a reasonable size, but this still eats quite a bit of RAM, especially with large backups.
—
It costs about 120 bytes plus the size of an entry to keep a single log line in memory, whereby these 120 bytes help weave entries into a hierarchy (a tree) and track their state.
If we want to reduce the memory usage, then the most obvious optimization would be to keep the actual entry text in a disk file and load it on demand, when it's actually visible in the log viewer.
This certainly helps, but as the log grows, the memory usage still climbs. A million file backup generates about 4 million log entries. At 120 bytes per entry that's 0.5 Gig in just supporting structures. This is unacceptable.
—
So it means that the entire tree structure of the log needs to sit in a disk file and to be read from there as needed.
Disk-based tree structures are routinely used in databases and file systems to store indecies of data sets, typically in a form of a B-Tree and its variations. The kicker is that these structures are meant for storing sorted data and they optimize for searching. Trying to adapt them for storing unsorted data would be nothing short of fitting a square peg in a round hole.
—
Long story short - there are no ready-made code for storing and manipulating generic data trees on a disk, leave alone fast code. This appears to be an esoteric problem that everyone solves on their own.
The screenshot above shows a part of a small "tree database" library that I ended up writing, together with its own caching module, predictive page loader and a pony.
Took almost two weeks.
—
On a bright side though, this code is perfectly reusable for storing disk snapshots as well and this paves way for making the planner module work almost entirely off the the memory.
Once done, Bvckup memory usage will NOT depend on the particulars of a backup at all - this a very big deal and an incredibly improtant feature to have for any robust backup software.
So, stay tuned...
Disk-based virtual data structures, that's what is :)
This is a part of an effort to further reduce and cap app's run-time memory usage. More specifically, it is meant to help the app cope gracefully with multi-million file backups that involve millions of steps and generate lots and lots of log data.
At the moment Bvckup's UI keeps backup logs in memory. It puts an effort to trim them to a reasonable size, but this still eats quite a bit of RAM, especially with large backups.
It costs about 120 bytes plus the size of an entry to keep a single log line in memory, whereby these 120 bytes help weave entries into a hierarchy (a tree) and track their state.
If we want to reduce the memory usage, then the most obvious optimization would be to keep the actual entry text in a disk file and load it on demand, when it's actually visible in the log viewer.
This certainly helps, but as the log grows, the memory usage still climbs. A million file backup generates about 4 million log entries. At 120 bytes per entry that's 0.5 Gig in just supporting structures. This is unacceptable.
So it means that the entire tree structure of the log needs to sit in a disk file and to be read from there as needed.
Disk-based tree structures are routinely used in databases and file systems to store indecies of data sets, typically in a form of a B-Tree and its variations. The kicker is that these structures are meant for storing sorted data and they optimize for searching. Trying to adapt them for storing unsorted data would be nothing short of fitting a square peg in a round hole.
Long story short - there are no ready-made code for storing and manipulating generic data trees on a disk, leave alone fast code. This appears to be an esoteric problem that everyone solves on their own.
The screenshot above shows a part of a small "tree database" library that I ended up writing, together with its own caching module, predictive page loader and a pony.
Took almost two weeks.
On a bright side though, this code is perfectly reusable for storing disk snapshots as well and this paves way for making the planner module work almost entirely off the the memory.
Once done, Bvckup memory usage will NOT depend on the particulars of a backup at all - this a very big deal and an incredibly improtant feature to have for any robust backup software.
So, stay tuned...
Coming up in beta 59 - automatic archive prunning.
If all goes well, it'll be out tomorrow.
If all goes well, it'll be out tomorrow.
Effects of delta copying as seen in Task Manager.
1.28 TB read, 0.37 TB written out - just the changes!
1.28 TB read, 0.37 TB written out - just the changes!
Ready to go. Launching tomorrow.
Referrals.
Coming up shortly, stay tuned.
Coming up shortly, stay tuned.
Switched delta copying progress bar from varying color
of the partially copied blocks to varying their height.
This should help with telling apart skipped, partial and
fully-copied blocks.
Added an option for cloning an existing backup job.
Website footer work.
In preparation for the launch of the Referral program.
In preparation for the launch of the Referral program.
SAMBA !
This one - the *nix implementation of Windows Networking. If you have a NAS box the chances are that it's using Samba to support network access from Windows computers.
It is a good software, simpler and much snappier than its native Windows counterpart, but in case of NAS devices it typically runs on Linux. This means that the underlying file system is anything but NTFS. This in turn translates into no support for "created" timestamps and no support for certain attributes, including System and Hidden.
Moreover, the attributes that cannot be set or cleared depend on whether it's a dot file or a regular file. Who would've thought? :-|
From what I understand it *is* possible to set things up so that Samba would record these timestamps and attributes and make things transparent to its Windows clients, but in reality this doesn't seem to be happening.
Long story short - starting with Beta 49 Bvckup will now be explicitly testing destination file system to check whether c-times are supported and which attributes can be set.
Additionally, it will also explicitly test both source and destination file systems to determine the granularity of c-times and m-times. This is again thanks to some NAS devices that are customized to use 1 sec for m-times and 1 usec (micro) for c-times.
By popular request.
When ticked, Minimize button minimizes to the taskbar (as usual) and Close, the (x), minimizes to the systray.
When unticked, Minimize minimizes to the systray and Close exits the app.
Trivial in retrospect, but took me ages to realize that behavior of Close and Minimize can be controlled with just one setting.
When ticked, Minimize button minimizes to the taskbar (as usual) and Close, the (x), minimizes to the systray.
When unticked, Minimize minimizes to the systray and Close exits the app.
Trivial in retrospect, but took me ages to realize that behavior of Close and Minimize can be controlled with just one setting.
Have a mapped drive? Want to run as Admin?
Good luck with that!
Good luck with that!
Second pass over the same sketch. Trimmed down
decorations a bit, reworked content to have a
summary, added footer, etc.
Full-length version is here.
What do you think? Let me know - @apankrat or over an email.
Full-length version is here.
What do you think? Let me know - @apankrat or over an email.
Started sketching production version of the website.
As much as I personally like the minimalist version
that's currently up there, it just
doesn't seem to work that well :-/
When periodic backup misses its last scheduled run,
it's now possible to tell the app how to handle it.
There are two common cases - (a) the app itself wasn't running and (b) the backup drive is a removable and it wasn't plugged in.
With certain setups it makes sense to run the backup as soon as the app can, to "catch up" with any changes and get them backed up quickly.
In other cases, a backup may be large and heavy and set up to run every midnight so not to disturb anyone. Running it ASAP will generally be not a very good idea.
Hence this option.
There are two common cases - (a) the app itself wasn't running and (b) the backup drive is a removable and it wasn't plugged in.
With certain setups it makes sense to run the backup as soon as the app can, to "catch up" with any changes and get them backed up quickly.
In other cases, a backup may be large and heavy and set up to run every midnight so not to disturb anyone. Running it ASAP will generally be not a very good idea.
Hence this option.
By popular request.
Context hints for each section in More options.
Here's the screen-by-screen flow of the
licensing process.
It starts with the above window and directs users to the website to pick and purchase a license code, which is given on spot and also sent in the receipt email.
It starts with the above window and directs users to the website to pick and purchase a license code, which is given on spot and also sent in the receipt email.
Then copy-paste the code, click on Verify,
get this -
If this works (the server is accessible, the license code is valid, etc), then this is it. Thank you very much, click Ok, you are done -
Otherwise, it explains the failure and offers either to retry the activation or to use an alternative -
There are two alternatives, both are simple -
For offline boxes and locked-down setups there's the Copy To Clipboard option. Take the URL to a networked machine, open it in a browser, get a licensing file back and then drop it into the app's config folder.
The URL is exactly the same as the one used by the app's own licensing code and it comprises of the license code and installation ID and the fingerprint.
Prepare the email option creates a new email with default email client and puts the Installation ID, Fingerprint and License Code in the message along with some pleasantries. The license file will be attached to the reply and it will simply need to be dropped into %LocalAppData%\Bvckup2\engine.
If this works (the server is accessible, the license code is valid, etc), then this is it. Thank you very much, click Ok, you are done -
Otherwise, it explains the failure and offers either to retry the activation or to use an alternative -
There are two alternatives, both are simple -
For offline boxes and locked-down setups there's the Copy To Clipboard option. Take the URL to a networked machine, open it in a browser, get a licensing file back and then drop it into the app's config folder.
The URL is exactly the same as the one used by the app's own licensing code and it comprises of the license code and installation ID and the fingerprint.
Prepare the email option creates a new email with default email client and puts the Installation ID, Fingerprint and License Code in the message along with some pleasantries. The license file will be attached to the reply and it will simply need to be dropped into %LocalAppData%\Bvckup2\engine.
A bit of UI animation for the license
activation window. This is a animated
GIF, so the colors and the timing are
all off. It looks tidier and faster in
reality.
Licensing support. Yay!
This is one of those things that is rather simple conceptually, but translates into a lot of actual petty work.
—
When the app is started, it looks at several system properties including the CPU model, Windows' MachineGuid value and others, hashes them together to produce an "Installation ID". For example -
The ID stays the same under normal computer use, but it changes with hardware upgrades or when the OS is reinstalled.
The app also generates an "Installation fingerprint" by hashing each of the properties separately and stringing them together. This yields a longer hash -
Like ID, the fingerprint changes with hardware upgrades and OS reinstalls, but, unlike ID, it changes only in parts. This makes it suitable for detecting incremental changes to the user's computer.
Both ID and Fingerprint identify user's installation, the difference is that the ID is used on the app side and the Fingerprint - on the licensing server side.—
Next, ID and fingerprint are submitted to the licensing server, either by the app itself or by a human through a website. When licensing for production use (as opposed to beta or evaluation), the request also includes a license code - a random token that is issued to the user in exchange for payment.
The server responds with a license, which is a short text file that binds together few pieces of information and seals them with an RSA signature of the licensing server.
For beta and evaluation licenses the "license code" is generated automatically by the licensing server. The large block of gibberish at the bottom of the license is the RSA signature.
—
When the app starts, it loads the license and verifies it using public key of the licensing server that is embedded in the app.
If the license is valid, the app works as normal. If the license is missing or expired or doesn't match the installation ID, the app disables certain features and periodically nags to do the right thing and what not.
—
Now, let say I upgraded my box. This screws up the Installation ID and invalidates the license. The app recomputes new ID and Fingerprint and sends them to the licensing server.
This is where Fingerprint comes into play. The server takes old and new fingerprints, chops them into individual hashes and does a "fuzzy" match. If enough hashes match, we can assume an incremental software or hardware change on user's end and re-issue the license for new Installation ID.
If fingerprints are wildly different, then we make fuss and require contacting human tech support.
—
In conclusion, needless to say that the whole scheme relies on the validation code in the app being trustworthy. If someone pries the app open and replaces selected IFs with NOPs, then obviously no licensing will work.
The answer to that is that the app should implement self-consistency checks and deploy counter-measures when these checks fail. This is a separate and very large and interesting subject ... which I have no plans on covering here :-) A good starting point here would be fravia.org archives and then follow from there.
This is one of those things that is rather simple conceptually, but translates into a lot of actual petty work.
When the app is started, it looks at several system properties including the CPU model, Windows' MachineGuid value and others, hashes them together to produce an "Installation ID". For example -
$c!f2ZoPxOlnpLWraJZ3F1nm7JcA
The ID stays the same under normal computer use, but it changes with hardware upgrades or when the OS is reinstalled.
The app also generates an "Installation fingerprint" by hashing each of the properties separately and stringing them together. This yields a longer hash -
YQdAXlZR0FLXZ0KpYxlOk8sJ9Z$xtq19!rp6i3C6P5LiUc0J
x2Ynqe29i41qb!1w5wEzot!q0EH8SsLWfY6gvkyoZPtfn3bW
uYnSwItsG8Wh1FkQFWBCaNqdndeiBiDgkrTde$wJUib9wR58
GJ5bQU1Zr1iewtgyRhLWigHyzKDO3oYztT4hMw6MnP6voupb
J6An5QE4!etkMJx!qzOaYmioIOQ0GiQyapcPQeyYLCi4CN28
CEev0!FZFZz3aa$0dCmNnMRBohZ9z6EZH5NxROhokF8pXWVM
9y$bzQAA
x2Ynqe29i41qb!1w5wEzot!q0EH8SsLWfY6gvkyoZPtfn3bW
uYnSwItsG8Wh1FkQFWBCaNqdndeiBiDgkrTde$wJUib9wR58
GJ5bQU1Zr1iewtgyRhLWigHyzKDO3oYztT4hMw6MnP6voupb
J6An5QE4!etkMJx!qzOaYmioIOQ0GiQyapcPQeyYLCi4CN28
CEev0!FZFZz3aa$0dCmNnMRBohZ9z6EZH5NxROhokF8pXWVM
9y$bzQAA
Like ID, the fingerprint changes with hardware upgrades and OS reinstalls, but, unlike ID, it changes only in parts. This makes it suitable for detecting incremental changes to the user's computer.
Both ID and Fingerprint identify user's installation, the difference is that the ID is used on the app side and the Fingerprint - on the licensing server side.
Next, ID and fingerprint are submitted to the licensing server, either by the app itself or by a human through a website. When licensing for production use (as opposed to beta or evaluation), the request also includes a license code - a random token that is issued to the user in exchange for payment.
The server responds with a license, which is a short text file that binds together few pieces of information and seals them with an RSA signature of the licensing server.
License code: EVAL-8028-8316-0400
Installation: $c!f2ZoPxOlnpLWraJZ3F1nm7JcA
Expires on: 20/12/2013
J/8natmgEo1fFr0NSBtw3fsderaMsv6gwELBa7P4tsjtkPxO
Y/4z930IfpujPKwen6QB9Gb33q7lyXQNkAVcvMV5hlXwKNV+
eifzJUqyOE1GdmYd8Z4Io0OuFr2V4tD5vnDmYTFoOye4gd67
D40G/hoKgcvGtX82w6bjLTae+vzm7u6GwowTGrbEmqqJDPNI
4Qc6un+aGuOEWEY0X25DkPYnFy/VnPusWgLRma1eTCrmvOqD
T6WX27++Bwq2+YH1Aw+Br9xEQF6SZOBp9OR1Xp/K24vYSN52
MRlLjhrBIQXmJTTepLKAHEsnA3pvdinU6pzpYexygfjtciUm
VZlgvxfUICWI3pKBNRJtlOQKc9LIyr9jVXQ1jP4QVdY5JNZW
y+71b8cSEtWgRRkYVGZMyUXeJjU7AoW/PhDNWdKvDPBwBInQ
QuNqA1ncxjWMRdLYogR/VY41yKYs9cPaFTPFb4y1XLPiqVRK
vEOMMUvE1kArZS/IuC5yB7Jqd3mXlZaRMacwuhWVcd+fgbFj
wVbLJvRRePKEoP/Kk0CtDPIQNdYVxlEwdZi/S+cFMdfZLiWU
o8HFpT247WWEyElUaDahfCREgjPbs9Iw5pik82bm42phns4I
1936WVycSATBQZp3JC1KXX8iQeDBfOgwe3QQcnWIrpclI/oI
57Mknd6bxeI=
Installation: $c!f2ZoPxOlnpLWraJZ3F1nm7JcA
Expires on: 20/12/2013
J/8natmgEo1fFr0NSBtw3fsderaMsv6gwELBa7P4tsjtkPxO
Y/4z930IfpujPKwen6QB9Gb33q7lyXQNkAVcvMV5hlXwKNV+
eifzJUqyOE1GdmYd8Z4Io0OuFr2V4tD5vnDmYTFoOye4gd67
D40G/hoKgcvGtX82w6bjLTae+vzm7u6GwowTGrbEmqqJDPNI
4Qc6un+aGuOEWEY0X25DkPYnFy/VnPusWgLRma1eTCrmvOqD
T6WX27++Bwq2+YH1Aw+Br9xEQF6SZOBp9OR1Xp/K24vYSN52
MRlLjhrBIQXmJTTepLKAHEsnA3pvdinU6pzpYexygfjtciUm
VZlgvxfUICWI3pKBNRJtlOQKc9LIyr9jVXQ1jP4QVdY5JNZW
y+71b8cSEtWgRRkYVGZMyUXeJjU7AoW/PhDNWdKvDPBwBInQ
QuNqA1ncxjWMRdLYogR/VY41yKYs9cPaFTPFb4y1XLPiqVRK
vEOMMUvE1kArZS/IuC5yB7Jqd3mXlZaRMacwuhWVcd+fgbFj
wVbLJvRRePKEoP/Kk0CtDPIQNdYVxlEwdZi/S+cFMdfZLiWU
o8HFpT247WWEyElUaDahfCREgjPbs9Iw5pik82bm42phns4I
1936WVycSATBQZp3JC1KXX8iQeDBfOgwe3QQcnWIrpclI/oI
57Mknd6bxeI=
For beta and evaluation licenses the "license code" is generated automatically by the licensing server. The large block of gibberish at the bottom of the license is the RSA signature.
When the app starts, it loads the license and verifies it using public key of the licensing server that is embedded in the app.
If the license is valid, the app works as normal. If the license is missing or expired or doesn't match the installation ID, the app disables certain features and periodically nags to do the right thing and what not.
Now, let say I upgraded my box. This screws up the Installation ID and invalidates the license. The app recomputes new ID and Fingerprint and sends them to the licensing server.
This is where Fingerprint comes into play. The server takes old and new fingerprints, chops them into individual hashes and does a "fuzzy" match. If enough hashes match, we can assume an incremental software or hardware change on user's end and re-issue the license for new Installation ID.
If fingerprints are wildly different, then we make fuss and require contacting human tech support.
In conclusion, needless to say that the whole scheme relies on the validation code in the app being trustworthy. If someone pries the app open and replaces selected IFs with NOPs, then obviously no licensing will work.
The answer to that is that the app should implement self-consistency checks and deploy counter-measures when these checks fail. This is a separate and very large and interesting subject ... which I have no plans on covering here :-) A good starting point here would be fravia.org archives and then follow from there.
Reworking About window a bit to include license
information and associated trinkets (not shown).
This will probably need another iteration or two...
Basically the idea is to show back/forth arrows only on hover and make them pull next error to the spot where the cursor is. So to hop through errors you would just go click-click-click without moving the mouse.
This compliments well existing feature of jumping to the first error of the last run by clicking on "Xxx errors" in backup's main window entry.
As per previous post, this leaves a question of how one would get to the *nearest* error if none is currently visible and the answer is that one would simply use conventional scroll for that.
All in all, having played with this for a bit I think it's a good solution. If it doesn't seem like one to you, give it a try and it will grow on you.
Tying up few loose ends in the UI.
The above screenshot is the first option for navigating between the errors in the log pane -- see two red arrows next to the "x" in the pane header. The second option is this -
This time the next/previous arrows are in the error entry itself.
—
I like the second option better, because it's contextual and I think it's more obvious. However, first option is superior in functionality because it allows jumping to next or previous error when there's currently no error visible in the pane.
Still deciding...
The above screenshot is the first option for navigating between the errors in the log pane -- see two red arrows next to the "x" in the pane header. The second option is this -
This time the next/previous arrows are in the error entry itself.
I like the second option better, because it's contextual and I think it's more obvious. However, first option is superior in functionality because it allows jumping to next or previous error when there's currently no error visible in the pane.
Still deciding...
Tweaked the progress indicator a bit.
For one, replaced the fixed-width byte countdown with a simple "% out of [size]". The countdown didn't work out, it was just... odd.
Also made the % adjust the number of shown decimal digits depending on how fast the copying is going.
For fast copies there are no decimals. For slower ones it shows one, for slower yet - two, and for the slowest (like the one on the screenshot) it goes down to 3 decimal places.
The idea, obviously enough, to always show some movement, but try and avoid being too excessive with it.
For one, replaced the fixed-width byte countdown with a simple "% out of [size]". The countdown didn't work out, it was just... odd.
Also made the % adjust the number of shown decimal digits depending on how fast the copying is going.
For fast copies there are no decimals. For slower ones it shows one, for slower yet - two, and for the slowest (like the one on the screenshot) it goes down to 3 decimal places.
The idea, obviously enough, to always show some movement, but try and avoid being too excessive with it.
Added code to replicate folder timestamps. It's ON
by default as it takes only several seconds to go
through a hundred thousand folders, but it can be
disabled if needed
Final version of the update window. De-emphasized the
"View changes" button and tweaked the colors a bit.
Reworked the software update mechanism a little.
First, when the update server is processing an update check from the program, it now takes into account program's current version.
It uses this version to compile a consolidated list of all changes between program's version and the latest version available. It then uses this list to understand if there are any critical or important outstanding changes. This yields an importance indicator of the update, when one is available.
Secondly, both the list and the indicator are now sent back to the program.
—
On program's end, the indicator translates into the "Update is recommended" line. It is one three options with "optional" and "critical" being the other two.
"Recommended" is a routine update that adds this and fixes that. It would better be installed, but it's not the end of the world if it's not.
"Critical" is critical - "you'd be sorry if you don't install it" kind of update.
An example of "optional" update is a new version that updates program's French translation when the user is running an English version, i.e. an update that can be ignored altogether.
—
There's now also the "View changes" button that expands the window and shows a neatly formatted (RTF) version of the change list.
The biggest deal, of course, is that it's a consolidated list of all changes between the installed and the latest version, rather than a laundry list of all changes since the dawn of time. It makes the list infinitely more useful compared to the raw version.
—
Here's an older post on this that goes a bit deeper to explain the rationale behind these changes Version numbers don't matter.
First, when the update server is processing an update check from the program, it now takes into account program's current version.
It uses this version to compile a consolidated list of all changes between program's version and the latest version available. It then uses this list to understand if there are any critical or important outstanding changes. This yields an importance indicator of the update, when one is available.
Secondly, both the list and the indicator are now sent back to the program.
On program's end, the indicator translates into the "Update is recommended" line. It is one three options with "optional" and "critical" being the other two.
"Recommended" is a routine update that adds this and fixes that. It would better be installed, but it's not the end of the world if it's not.
"Critical" is critical - "you'd be sorry if you don't install it" kind of update.
An example of "optional" update is a new version that updates program's French translation when the user is running an English version, i.e. an update that can be ignored altogether.
There's now also the "View changes" button that expands the window and shows a neatly formatted (RTF) version of the change list.
The biggest deal, of course, is that it's a consolidated list of all changes between the installed and the latest version, rather than a laundry list of all changes since the dawn of time. It makes the list infinitely more useful compared to the raw version.
Here's an older post on this that goes a bit deeper to explain the rationale behind these changes Version numbers don't matter.
Adding support for running external command
before and after each backup.
Both commands have a configurable timeout value. Additionally, the pre-backup command can be tagged as critical, in which case bvckup checks the exit code and aborts the backup if the code doesn't match specified value.
The screenshot shows touching a TrueCrypt volume before the backup. This is needed, because TrueCrypt goes out of its way to preserve timestamps on its container files and bvckup sees such files as unmodified.
Both commands have a configurable timeout value. Additionally, the pre-backup command can be tagged as critical, in which case bvckup checks the exit code and aborts the backup if the code doesn't match specified value.
The screenshot shows touching a TrueCrypt volume before the backup. This is needed, because TrueCrypt goes out of its way to preserve timestamps on its container files and bvckup sees such files as unmodified.
Reworked the More Options section.
It has already grown quite tall and it still needs to take on few more configuration options, so expanding it in place as it's done for all other sections doesn't work. The window gets too big and simply doesn't fit on a desktop.
Instead, it pushes everything up and out of the window and floods it with its own content. As I add more config options to the section, it will also expand the window down a bit if needed. All this should help keep all configuration within single window and also keep window's height in check.
Took me a while to converge to this solution, but it is a pretty decent way to structure it I think.
It has already grown quite tall and it still needs to take on few more configuration options, so expanding it in place as it's done for all other sections doesn't work. The window gets too big and simply doesn't fit on a desktop.
Instead, it pushes everything up and out of the window and floods it with its own content. As I add more config options to the section, it will also expand the window down a bit if needed. All this should help keep all configuration within single window and also keep window's height in check.
Took me a while to converge to this solution, but it is a pretty decent way to structure it I think.
Second revision of the same thing.
Sketching up a dedicated device tracking
section. Not sure if it's worth it, still
have to try few things out...
Coming up - the reordering of list items.
Done with the inclusion/exclusion filters
The tree selector widget is a piece of art even though it doesn't look like one :) It is basically a front of an asynchronous parallel disk scanner that dynamically rearranges its scanning queue to first scan subdirectories that a user might be accessing next.
In other words, if you expand a directory, then the scanner will postpone scanning anything else and focus instead on all its subdirectories. So if you are making your way through a massively populated disk, it will try to pre-load what you are looking at and scan the rest on a lower priority.
Also, since the widget starts showing tree contents right away, it allows quickly navigating to and excluding a desired item without needing to wait for the entire scan to complete. Again, quite a bit of speed up with huge trees.
The tree selector widget is a piece of art even though it doesn't look like one :) It is basically a front of an asynchronous parallel disk scanner that dynamically rearranges its scanning queue to first scan subdirectories that a user might be accessing next.
In other words, if you expand a directory, then the scanner will postpone scanning anything else and focus instead on all its subdirectories. So if you are making your way through a massively populated disk, it will try to pre-load what you are looking at and scan the rest on a lower priority.
Also, since the widget starts showing tree contents right away, it allows quickly navigating to and excluding a desired item without needing to wait for the entire scan to complete. Again, quite a bit of speed up with huge trees.
Advanced filter configuration. The pattern
matching is all brand new and much improved
over the disaster that it was in Bvckup 1.
It also adds an option of matching on item
attributes, both set and cleared.
Finishing up the dialog for configuring
inclusion/exclusion filters.
In inclusive mode (pictured) the tree view tags excluded items with 'x'. In exclusive mode, when "Start with an empty list" selected, the tree view tags included items with checkmarks.
Currently leaning towards Option 1, and not yet decided if to strike-out excluded items or not.
In inclusive mode (pictured) the tree view tags excluded items with 'x'. In exclusive mode, when "Start with an empty list" selected, the tree view tags included items with checkmarks.
Currently leaning towards Option 1, and not yet decided if to strike-out excluded items or not.
Added per-phase progress bars. Currently
used for scanning and execution phases.
Before/After of the backup config window
as of Beta Release 20.
Revising the backup config dialog to accommodate
a description field.
Revising the main view. Description and the
progress bars are back in.
Well into the beta, just dotting the i's and
crossing the t's.
One of the outstanding features is the archival of deleted items and this is the supporting UI for it.
Now deciding between a compact format of the config widget or a verbose one. Compact format has a single descriptive field that is re-used to show the explanation of a currently selected option. Verbose format has descriptions for each option visible at all times - it is likely easier on first time users, but it is also a waste of screen space on all subsequent uses.
One of the outstanding features is the archival of deleted items and this is the supporting UI for it.
Now deciding between a compact format of the config widget or a verbose one. Compact format has a single descriptive field that is re-used to show the explanation of a currently selected option. Verbose format has descriptions for each option visible at all times - it is likely easier on first time users, but it is also a waste of screen space on all subsequent uses.
What can I say?
Beta. Is. Good. To. Go!!!
Beta. Is. Good. To. Go!!!
For very large files (say, 256 Gigs), there was a clear lack of UI acknowledgement of the backup progress. So the completed percentage got a couple of point digits and a byte countdown for the file.
EULA
Trying to keep it short and sensible.
Trying to keep it short and sensible.
Added the first draft of the Miscellanea section.
Also renamed More Options button to
Rabbit Hole, because that's essentially
where it leads to :)
( the original version is here )
Significantly improved performance through multi-core, parallel hashing, asynchronous I/O and more aggressive caching of the state data.
Strengthened the change detection algorithm through addition of the whole file hash. When all individual block hashes come back unchanged, the algorithm now uses the file hash to confirm that the file has indeed remained unchanged.
Combined with the original use of two separate hashes per data block, this eliminates the need for a precautionary full-file sync every N copies.
Improved handling of cancelled and aborted delta copies by introducing support for partial updates.
The algorithm now remembers how far along the file it managed to proceed and stashes this information for the next run.
When the copying is cancelled partway through, the file is marked as out-of-sync, but the delta state is updated to capture what has been already done. This comes very handy and speeds things up when updating very large files.
Improved handling of large file counts.
Version 1 maintained an in-memory index of all delta copied files and, surprisingly, this didn't scale well. It also kept all delta state information in a single folder, which too led to some pain and suffering when the file count grew large.
New version eliminates both these problems.
Lastly, all copier parameters - block sizes, hash algorithms, cache buffer sizes, threading parameters, etc. - are now configurable on a per-job basis.
All in all - THIS IS IT.
This is how a simple and efficient delta copying should be done. Can't wait to release it to the real world.
Shadow copying is in,
for those pesky locked Outlook mailboxes and other "in use" files.
There are three big things left to weave into the program - file system monitoring, new delta copier and software updates - and the beta is good to go. A matter of few days now.
There are three big things left to weave into the program - file system monitoring, new delta copier and software updates - and the beta is good to go. A matter of few days now.
If your message box includes the "Don't ask again"
option, just make sure to disable all but one
button, when it checked.
Oddly enough, I don't know a single app that does that.
Oddly enough, I don't know a single app that does that.
Feeling creative? Make your own version, save as a PNG, drop it in a config directory and - voilà - your systray looks just like you want it to.
There are two sets of icons on the sprite sheet.
The first three columns is the XP style, which
is a bit puffy and matches well older Windows
styling. The last three columns is a flat version
that goes better with native W7 tray icon style.
Each icon comes in 3 sizes - 16x16, 20x20, 24x24. This is needed to accommodate 3 common DPI levels, see this earlier entry for details.
Each icon comes in 3 sizes - 16x16, 20x20, 24x24. This is needed to accommodate 3 common DPI levels, see this earlier entry for details.
"Message pending" indication, whereby a message
can be either an error, a warning or a notice
and these are differentiated with the ! color.
I realize it's not color-blind friendly,
but the goal is to attract the attention and
make user bring the app window up to read the
message --(therefore)--> reflecting the message
type via an icon level is not exactly critical.
W7-ish version as per
a
this hint on Twitter. Not bad at all.
Working on systray icons...
Revised Preferences window a bit.
Added basic control over backup scheduling and switched from "close to systray" to "minimize to systray" - the (x) at the top right corner will now always act as an Exit, just as it is supposed to.
Added basic control over backup scheduling and switched from "close to systray" to "minimize to systray" - the (x) at the top right corner will now always act as an Exit, just as it is supposed to.
More of the UI work.
Custom pop-out button that stays nice, flat and out of the way when idle, but acts like a regular button otherwise. Not exactly a UI innovation, but it was nice to be able to use it in a real project :)
Custom pop-out button that stays nice, flat and out of the way when idle, but acts like a regular button otherwise. Not exactly a UI innovation, but it was nice to be able to use it in a real project :)
Took out the engine mode switch from the Preferences
and into a dialog of its own. Still need to understand
if it's worth mentioning that there's an engine per se
or just stick to more amorphous "application mode"...
Looked at few options for splitter styling. First
one is final. Tentatively though... heh.
Done with the split-view mode for the main window.
It's one of those things that everyone takes for
granted and that, in reality, maps to quite a bit
of work if done right.
For example, resizing window by the top edge should resize the top pane and leave the bottom part alone.
Likewise, resizing by the bottom edge should expand/compress the bottom pane, but not the top one.
However, if we are compressing window by the top edge and the top pane is at its minimal height, then we should the bottom pane starting to compress.
All good so far, but, say, we change our mind and start expanding the window back (without releasing the mouse button).
We should see the bottom pane brought back to its original size first, followed by the expansion of the top part... even though we are, technically, dragging by the top edge.
This sort of UX detail is something that few people will notice if it's done right, because it feels natural and aligns well with expected behavior. But cut some corners - and you bet everyone will notice.
For example, resizing window by the top edge should resize the top pane and leave the bottom part alone.
Likewise, resizing by the bottom edge should expand/compress the bottom pane, but not the top one.
However, if we are compressing window by the top edge and the top pane is at its minimal height, then we should the bottom pane starting to compress.
All good so far, but, say, we change our mind and start expanding the window back (without releasing the mouse button).
We should see the bottom pane brought back to its original size first, followed by the expansion of the top part... even though we are, technically, dragging by the top edge.
This sort of UX detail is something that few people will notice if it's done right, because it feels natural and aligns well with expected behavior. But cut some corners - and you bet everyone will notice.
Going through styling options for the expand/collapse buttons.
Presently leaning towards C or D, because I'm considering making the text selectable with the mouse for easy copy- pasting. For that to work the buttons need to be as unobtrusive as possible.
Still WiP though...
Presently leaning towards C or D, because I'm considering making the text selectable with the mouse for easy copy- pasting. For that to work the buttons need to be as unobtrusive as possible.
Still WiP though...
Working on the backup log viewer.
—
Basically the idea for the log is to support multiple detail levels, each capturing progressively more context that its parent.
With all details collapsed, the log gives a quick overview of when the job was running and if it completed OK or with errors.
Expanding details allows for a drill-down - to look at the details such as the file and byte counts, the timing and performance data and the details of any errors that have occured.
I'm still going to shuffle things around the window, e.g. push the -/+ buttons to the right, tweak the styling too, but the general hierarchical format will stay. It is a very convenient way of organizing the log info.
Basically the idea for the log is to support multiple detail levels, each capturing progressively more context that its parent.
With all details collapsed, the log gives a quick overview of when the job was running and if it completed OK or with errors.
Expanding details allows for a drill-down - to look at the details such as the file and byte counts, the timing and performance data and the details of any errors that have occured.
I'm still going to shuffle things around the window, e.g. push the -/+ buttons to the right, tweak the styling too, but the general hierarchical format will stay. It is a very convenient way of organizing the log info.
Finished a piece of code that determines
effective timestamp resolution of a file
system.
—
Every file system keeps track of when a file was created and when it was last modified. These timestamps are the natural properties of a file and they are readily available for inspection in the Windows Explorer or any file manager of your choice.
The less obvious aspect of timestamping is that timestamps have resolution. For example, FAT tracks file modification with a measly 2 second precision. In comparison, NTFS uses the 100 ns (nano-second) precision.—
It is very typical for a backup software to rely on timestamps to determine if a file has been modified and requires copying. But if the source file sits on an NTFS volume and the backup goes onto a FAT disk, then comparing timestamp directly simply won't work, because FAT timestamp will be rounded-up to a 2-second mark.
In other words, the timestamp granularity needs to be taken into an account when comparing the timestamps. The question is how to determine what it is exactly.—
First of all, the granularity for creation and modification times can be different. FAT has them at 10 ms and 2 s respectively, but NTFS has them both at 100 ns.
To complicate matters, there appears to be NAS devices that look like NTFS boxes, but with the granularity that is not 100 ns.
So what I did is added code to probe the file system and determine effective resolution for both timestamps. This involves dropping a temporary file and then trying to set its timestamps to this, that and 3rd and see what they end up at. From that it's possible to deduce the resolution.
In cases when such probing fails, the app falls back to guessing resolution by the file system name. It is also possible to override the resolution values from the config, just in case.
Every file system keeps track of when a file was created and when it was last modified. These timestamps are the natural properties of a file and they are readily available for inspection in the Windows Explorer or any file manager of your choice.
The less obvious aspect of timestamping is that timestamps have resolution. For example, FAT tracks file modification with a measly 2 second precision. In comparison, NTFS uses the 100 ns (nano-second) precision.
It is very typical for a backup software to rely on timestamps to determine if a file has been modified and requires copying. But if the source file sits on an NTFS volume and the backup goes onto a FAT disk, then comparing timestamp directly simply won't work, because FAT timestamp will be rounded-up to a 2-second mark.
In other words, the timestamp granularity needs to be taken into an account when comparing the timestamps. The question is how to determine what it is exactly.
First of all, the granularity for creation and modification times can be different. FAT has them at 10 ms and 2 s respectively, but NTFS has them both at 100 ns.
To complicate matters, there appears to be NAS devices that look like NTFS boxes, but with the granularity that is not 100 ns.
So what I did is added code to probe the file system and determine effective resolution for both timestamps. This involves dropping a temporary file and then trying to set its timestamps to this, that and 3rd and see what they end up at. From that it's possible to deduce the resolution.
In cases when such probing fails, the app falls back to guessing resolution by the file system name. It is also possible to override the resolution values from the config, just in case.
Done with the backup scheduler. Comes with a
couple of neat improvements over v1.
—
The Go and Stop buttons work almost exactly as in v1. If the backup is running, click Stop once to Pause it, click again to Cancel, click third time to disable the job. If the job is disabled, click once to enable it, click again to manually start it. And so on, you get the idea.
—
First improvement over v1 are the Go! and Stop! commands that are activated by holding Ctrl down when clicking on the toolbar buttons.
Go! moves the job straight into the running state.
It's the "Run it. NOW!" command.
Stop! stops and disables the job straight away.
It's the "Stop it. NOW!" command.
—
Second improvement is that multiple jobs can now be run in parallel. Generally, this has a negative impact on the overall performance, but there are certain cases when this is desirable. One example would be running a smaller backup in parallel with a much larger and slower one.
There are three ways to do this:
—
PS. I considered adding something more elaborate, e.g. allowing assigning jobs to "run groups" and then allowing only one active job per group, but that seemed like an overkill, so I dropped that.
The Go and Stop buttons work almost exactly as in v1. If the backup is running, click Stop once to Pause it, click again to Cancel, click third time to disable the job. If the job is disabled, click once to enable it, click again to manually start it. And so on, you get the idea.
First improvement over v1 are the Go! and Stop! commands that are activated by holding Ctrl down when clicking on the toolbar buttons.
Go! moves the job straight into the running state.
It's the "Run it. NOW!" command.
Stop! stops and disables the job straight away.
It's the "Stop it. NOW!" command.
Second improvement is that multiple jobs can now be run in parallel. Generally, this has a negative impact on the overall performance, but there are certain cases when this is desirable. One example would be running a smaller backup in parallel with a much larger and slower one.
There are three ways to do this:
- With the Go! command.
- With a per-backup setting that makes the job always run right away, even if there's another job is running.
- With an app-level setting that does the same, but for all jobs.
PS. I considered adding something more elaborate, e.g. allowing assigning jobs to "run groups" and then allowing only one active job per group, but that seemed like an overkill, so I dropped that.
Another pass over the main window chrome and
layout. The "backup jobs" section is just a
dummy header ... and it is confusing, I think.
† This is an app screenshot, not a sketch.
† This is an app screenshot, not a sketch.
Exploring options for the main window format.
—
Specifically, I want to get rid of the V1's Summary panel, trim the information contained in it a bit and fold it directly into the main window items.
In conjunction with this I'm also considering having two modes for displaying backup jobs on the list - "terse" one-liners and "extended" multi-liners. What I still need to figure out if this should be a per-item preference or an app-wide setting and how one would switch between these display modes.
Work in progress, obviously. Any thoughts - let me know.
Specifically, I want to get rid of the V1's Summary panel, trim the information contained in it a bit and fold it directly into the main window items.
In conjunction with this I'm also considering having two modes for displaying backup jobs on the list - "terse" one-liners and "extended" multi-liners. What I still need to figure out if this should be a per-item preference or an app-wide setting and how one would switch between these display modes.
Work in progress, obviously. Any thoughts - let me know.
Done with a little animation that and switches
main window between a "welcome screen" and the
regular backup list view.
The animation and fading effects can be disabled, there's a setting for that.
—
Windows comes with a standard toolbar control, but it's an overkill if you need just a couple of buttons at the top of the window.
It basically takes longer to figure out how to use and customize it compared to creating a basic window and sticking a couple of custom buttons on it.
Had I have a need for toolbar ribbons, docking, on-the-fly customization, etc., then ToolbarWindow32 makes sense. But if it's something simple, then there are faster options.
The animation and fading effects can be disabled, there's a setting for that.
Windows comes with a standard toolbar control, but it's an overkill if you need just a couple of buttons at the top of the window.
It basically takes longer to figure out how to use and customize it compared to creating a basic window and sticking a couple of custom buttons on it.
Had I have a need for toolbar ribbons, docking, on-the-fly customization, etc., then ToolbarWindow32 makes sense. But if it's something simple, then there are faster options.
Meet the toolbar.
—
To explain a bit what you see in this masterpiece :) -
The idea is to try and keep the menu/toolbar area simple and flat, but still provide some sort of a visual clue that Go and Start are clickables.
Here are three options compared back to back to make the differences a bit more obvious -
—
Here are few options that were leading up to the above one:
iOS says Hi
Also, considered disposing off the menu bar and folding whatever few items it has into the drop-down menu. But this just looks weird and too alien on Windows, the app needs a menu bar.
Dropped the idea.
To explain a bit what you see in this masterpiece :) -
The idea is to try and keep the menu/toolbar area simple and flat, but still provide some sort of a visual clue that Go and Start are clickables.
Here are three options compared back to back to make the differences a bit more obvious -
Here are few options that were leading up to the above one:
Also, considered disposing off the menu bar and folding whatever few items it has into the drop-down menu. But this just looks weird and too alien on Windows, the app needs a menu bar.
Dropped the idea.
The look of the main window with no backups
configured. Tried using a stock button, but
it looks heavy and dull. Hopefully, this is
a bit more refined.
Trying to understand how to indicate if
a selected path points at a removable storage.
The app tracks storage devices being plugged into the computer and it adjusts backup paths if a known device re-appears under a different drive name.
I can, of course, not bother with this at all and simply spit out a notification when (and if) the drive change is detected, but then I think it's an important functionality and it'd better be mentioned during the setup.
The app tracks storage devices being plugged into the computer and it adjusts backup paths if a known device re-appears under a different drive name.
I can, of course, not bother with this at all and simply spit out a notification when (and if) the drive change is detected, but then I think it's an important functionality and it'd better be mentioned during the setup.
"Work-in-progress" buttons for beta releases.
Correct your input.
† Idea is courtesy of the OS X login screen, I think. Though I did experiment with concise error indication quite a bit.
† Idea is courtesy of the OS X login screen, I think. Though I did experiment with concise error indication quite a bit.
Wrapping up the Periodic Backup configuration
window.
Unlike the v1 that always stored the backup interval as a number of seconds, the v2 adds a time unit specifier.
This simplifies quite a few things in the UI and yet it keeps the configuration flexible enough to support 1.5 hour backups by allowing to specify a 90 minute interval.
—
On an unrelated note, killed few hours trying to utilize native Windows controls for the interval configuration. The amount of work required to make them look reasonably good is as depressing as it is remarkable.
For example, the editbox control doesn't natively support vertically centering the input text. It just does not. It is meant to look like this -
Want "120" aligned to the label on the left? Sure thing, that'd be 2 hours of plowing through documentation only to realize that it must be... drum-roll... a multiline control that you will then need to manually pad the right amount at the top to push the line to the middle.
—
But that's peanuts compared to trying to beat some sense into the dropdown box. Long story, but making a box taller and keeping the line centered requires taking over the control drawing functions, which in presence of Windows themes is a delightfully unpleasant experience.
In any case, turning the top control line into the bottom one -
costs exactly a day of work, while it really shouldn't. Welcome to Windows.
Unlike the v1 that always stored the backup interval as a number of seconds, the v2 adds a time unit specifier.
This simplifies quite a few things in the UI and yet it keeps the configuration flexible enough to support 1.5 hour backups by allowing to specify a 90 minute interval.
On an unrelated note, killed few hours trying to utilize native Windows controls for the interval configuration. The amount of work required to make them look reasonably good is as depressing as it is remarkable.
For example, the editbox control doesn't natively support vertically centering the input text. It just does not. It is meant to look like this -
Want "120" aligned to the label on the left? Sure thing, that'd be 2 hours of plowing through documentation only to realize that it must be... drum-roll... a multiline control that you will then need to manually pad the right amount at the top to push the line to the middle.
But that's peanuts compared to trying to beat some sense into the dropdown box. Long story, but making a box taller and keeping the line centered requires taking over the control drawing functions, which in presence of Windows themes is a delightfully unpleasant experience.
In any case, turning the top control line into the bottom one -
costs exactly a day of work, while it really shouldn't. Welcome to Windows.
Finalized the design of inline hint/tip
widget for input controls.
Initially, it was meant for the config window that hosts periodic backup settings (as seen in a couple of posts below), but having talked to others and slept on the issue I think that the natural language input (i.e. requiring to enter "1 week" or "15 minutes") is not the best option for this case.
An alternative design is coming up in a bit.
Initially, it was meant for the config window that hosts periodic backup settings (as seen in a couple of posts below), but having talked to others and slept on the issue I think that the natural language input (i.e. requiring to enter "1 week" or "15 minutes") is not the best option for this case.
An alternative design is coming up in a bit.
Coded the previous sketch, played with few
parameters and it is starting to get there
... I think. Will make another pass over this
tomorrow and it should be good to go.
For what it's worth I have also gave a try
to the standard
Tooltip Control. In fact,
that's where I started.
—
A regular tooltip looks like this -
It adds another layer to the windows stack, breaks the visual flow of the layout and still manages to tuck itself under the mouse cursor.
There's also a balloon version of the same, which looks like this -
It is marginally better, but just look at those lovely rounded cornersi. Eat that Apple. Real artists ship.
—
tl;dr - the standard Tooltip Control is not very good, so I had to come up with a custom replacement.
A regular tooltip looks like this -
It adds another layer to the windows stack, breaks the visual flow of the layout and still manages to tuck itself under the mouse cursor.
There's also a balloon version of the same, which looks like this -
It is marginally better, but just look at those lovely rounded cornersi. Eat that Apple. Real artists ship.
tl;dr - the standard Tooltip Control is not very good, so I had to come up with a custom replacement.
Not sure about this one, seems a bit
clumsy and a bit too custom.
This is basically a way to explain what is expected as an input for a particular configuration value. There's gotta be a cleaner way of delivering the same info...
This is basically a way to explain what is expected as an input for a particular configuration value. There's gotta be a cleaner way of delivering the same info...
Working through the supporting code for the Settings window.
Done with collapsing/expanding mechanics and respective fading and moving animations. Still need to revise the copy for the explanatory bits and massage in the option for archiving deleted files.
—
Not to pat myself on the back, but I think it came out looking pretty decent.
That's not to say there's no way to turn off all this fancy styling and animations, because, you bet your favourite monochrome flat design, there is.
Done with collapsing/expanding mechanics and respective fading and moving animations. Still need to revise the copy for the explanatory bits and massage in the option for archiving deleted files.
Not to pat myself on the back, but I think it came out looking pretty decent.
That's not to say there's no way to turn off all this fancy styling and animations, because, you bet your favourite monochrome flat design, there is.
The Backup Settings window. Final ...for now :)
Another take on Backup Settings window, 4 styling
variations.
Further compacted vertically, but now it appears to be somewhat cluttered and lacking comfortable vertical rhythm (to borrow a term from the typography).
Further compacted vertically, but now it appears to be somewhat cluttered and lacking comfortable vertical rhythm (to borrow a term from the typography).
Working on the Backup Settings window.
I think the original design from the version 1 got
many things right.
It provided a good at-a-glance overview of the backup configuration, split settings logically and tucked away the minute details. It did have one problem - it was tall. So tall, in fact, that I got several reports that it didn't fit on screen :-|
—
V2 adds several new options to the backup config, so this doesn't make things any easier. After looking at few options, including paged, tabbed and two-column, screenful layouts, I circled back to the v1 layout.
To compress it vertically I've done two things - got rid of the group boxes (frames) and collapsed radio button groups into a single line with the current selection.
This gives us the middle screenshot (pic) that shows the config window in the "overview" state.
—
Clicking on a "Change ↓" button expands the radio button group and allows changing the setting without leaving the page. Latter is pretty nice to have, because ... well ... because it's simply faster and more convenient.
Expanding a section gives the screenshot on the right (pic). To collapse the section, there's a little button in the section header on the right hand side. I'm not 100% convinced it's really needed nor that it's the best solution, but that's how it is at the moment.
—
I forgot the "how to handle deleted files" section, but it will be in the beta. Not to worry.
It provided a good at-a-glance overview of the backup configuration, split settings logically and tucked away the minute details. It did have one problem - it was tall. So tall, in fact, that I got several reports that it didn't fit on screen :-|
V2 adds several new options to the backup config, so this doesn't make things any easier. After looking at few options, including paged, tabbed and two-column, screenful layouts, I circled back to the v1 layout.
To compress it vertically I've done two things - got rid of the group boxes (frames) and collapsed radio button groups into a single line with the current selection.
This gives us the middle screenshot (pic) that shows the config window in the "overview" state.
Clicking on a "Change ↓" button expands the radio button group and allows changing the setting without leaving the page. Latter is pretty nice to have, because ... well ... because it's simply faster and more convenient.
Expanding a section gives the screenshot on the right (pic). To collapse the section, there's a little button in the section header on the right hand side. I'm not 100% convinced it's really needed nor that it's the best solution, but that's how it is at the moment.
I forgot the "how to handle deleted files" section, but it will be in the beta. Not to worry.
Meet the Preferences.
The Backup Engine section - as simple as it looks,
it is fronting a couple of months worth of work
and it's easily one of the most complex part of
the v2. This is what detaches the user interface
from the guts of the program and lets latter run
as a separate process.
—
The Maintenance section - I figured that with the backup being an important piece of software, very few users would be interested in fully automatic updates. Furthermore, quite a few people like to retain even tighter control over the software and, for example, do not like it when the software talks to the Internet whenever it wants. Hence the new option -
Don't check, but remind to check for updates
It's basically a kitchen timer that goes "Ding!" every week or two (configurable) and prompts you to let the app check for a new version. We'll see how it goes, but I think it's a sensible compromise between the privacy and functionality.
—
The More button transforms the window into a comprehensive list of all available options, close to two dozens now and still counting. The idea however is to keep the defaults reasonable so that there won't be much need to venture into this part of the app.
—
Lastly, note how the Apply/Close buttons change when the settings are modified. I have no idea why this is not a standard behavior, because it really should be.
The Maintenance section - I figured that with the backup being an important piece of software, very few users would be interested in fully automatic updates. Furthermore, quite a few people like to retain even tighter control over the software and, for example, do not like it when the software talks to the Internet whenever it wants. Hence the new option -
It's basically a kitchen timer that goes "Ding!" every week or two (configurable) and prompts you to let the app check for a new version. We'll see how it goes, but I think it's a sensible compromise between the privacy and functionality.
The More button transforms the window into a comprehensive list of all available options, close to two dozens now and still counting. The idea however is to keep the defaults reasonable so that there won't be much need to venture into this part of the app.
Lastly, note how the Apply/Close buttons change when the settings are modified. I have no idea why this is not a standard behavior, because it really should be.
A message bar is a unobtrusive way to display
one-liner messages that would've otherwise
required poping up a message box.
When a program detects that it was just restarted
with Admin privileges, it has a limited number of
options for communicating this joyous news to its
user.
#1 - A message box. Bluntly put - I hate them. They are always too sudden, they steal your attention and stuff whatever they want to say down your throat. They ... lack manners.
#2 - A system tray notification. This is less intrusive, but perhaps it's too out of the way. Also, the systray is a shared area so the program is always competing with other apps for the same screen real estate.
#3 - Something else that is right in your field of view if you are looking at the app, but that doesn't get in a way of whatever you are doing at the moment.
—
Cue in the message bar, now an integral part of the Bvckup UI. It is used for displaying various status notifications right after the launch, after switching the engine to/from the system service mode, when discovering software updates and few other things.
I have also tried the self-closing version of the bar that dismisses itself on a timeout. Like so -
and so -
but having lived with this for a couple of days, it now seems like a solution in a search of a problem. So I shelved it for now. If there's any a real need for this, it'd be very easy to add later on.
#1 - A message box. Bluntly put - I hate them. They are always too sudden, they steal your attention and stuff whatever they want to say down your throat. They ... lack manners.
#2 - A system tray notification. This is less intrusive, but perhaps it's too out of the way. Also, the systray is a shared area so the program is always competing with other apps for the same screen real estate.
#3 - Something else that is right in your field of view if you are looking at the app, but that doesn't get in a way of whatever you are doing at the moment.
Cue in the message bar, now an integral part of the Bvckup UI. It is used for displaying various status notifications right after the launch, after switching the engine to/from the system service mode, when discovering software updates and few other things.
I have also tried the self-closing version of the bar that dismisses itself on a timeout. Like so -
and so -
but having lived with this for a couple of days, it now seems like a solution in a search of a problem. So I shelved it for now. If there's any a real need for this, it'd be very easy to add later on.
Nailed down the installer flow. A couple of
things of note here.
The most unconventional part of the setup
is how it handles the configuration of the
installation parameters.
The configuration page (see Page 3) is very bare and focuses on how the program is going to be used instead of being a mixed bag of the minute details of the setup.
The target location is made into a secondary setting with its value controlled by the usage preference. There's no field for the Start Menu group name. There's no ever popular "Run the program after the setup completes" option either. Instead, exiting the setup automatically runs the program (see Page 4), unless the installer window is closed through an (x) at the top.
This leads to a simpler and more intuitive setup flow... or at least that's the idea :)
—
The installer supports both privileged (administrative) and unprivileged (regular user) installations.
If you are logged in as a regular user and have no rights to modify C:\Program Files, there's no reason why you shouldn't still be able to install a software, just for yourself.
While other installers routinely ask if to install the software for everyone or just for one user, it's a superficial question. It merely controls if the Start Menu group is created under All Users or your personal profile. The program files still go into a system directory and this requires administrative privileges.
Bvckup implements true system- and user- deployment, which is akin to deploying the program in /usr/bin or ~/bin in Unix terms.
In fact, on a multi-user system it is perfectly possible to install one instance of Bvckup for each user and they will all happily co-exist with each other.
The configuration page (see Page 3) is very bare and focuses on how the program is going to be used instead of being a mixed bag of the minute details of the setup.
The target location is made into a secondary setting with its value controlled by the usage preference. There's no field for the Start Menu group name. There's no ever popular "Run the program after the setup completes" option either. Instead, exiting the setup automatically runs the program (see Page 4), unless the installer window is closed through an (x) at the top.
This leads to a simpler and more intuitive setup flow... or at least that's the idea :)
The installer supports both privileged (administrative) and unprivileged (regular user) installations.
If you are logged in as a regular user and have no rights to modify C:\Program Files, there's no reason why you shouldn't still be able to install a software, just for yourself.
While other installers routinely ask if to install the software for everyone or just for one user, it's a superficial question. It merely controls if the Start Menu group is created under All Users or your personal profile. The program files still go into a system directory and this requires administrative privileges.
Bvckup implements true system- and user- deployment, which is akin to deploying the program in /usr/bin or ~/bin in Unix terms.
In fact, on a multi-user system it is perfectly possible to install one instance of Bvckup for each user and they will all happily co-exist with each other.
The first page of the installer.
Working through the details of the update mechanism.
This is one of more difficult parts of the application, even though on the surface it looks uncomplicated. If we run the installer and it detects that the app is already installed, it should just update the installation. How hard could it be?
Let's see.
This is one of more difficult parts of the application, even though on the surface it looks uncomplicated. If we run the installer and it detects that the app is already installed, it should just update the installation. How hard could it be?
Let's see.
If the program is not running, then it's indeed a simple
matter of replacing the program file with a newer version.
The fun starts if the program is running.
On Windows, running a program locks its file, preventing any changes from being made to it. This is not required per se and there are operating systems that don't do this. There are also custom PE loaders for Windows that can take an executable and create a running process out of it without any locking fuss. For example, bo2k could do that back in 1999. See its source code for details if interested.
—
To complicate matters, there might be more than one process running off the same executable. For example, if several users are logged into a Windows box and each runs a copy of a program.
Therefore, the first order of business for any installer is to shut down all running instances of the program. A well-behaved installer should also restart all these instances after it's done with updating.
—
The shutting down part is not a big deal. There's more than one way to do this and all of them are fairly straight-forward.
It's the restarting that is contrived.
Not only program instances may be running under different user accounts, they may also be running with custom command-line arguments, as Administrators or system services. Restarting this cornucopia of awesomeness is not an envious task.
—
Here's how Bvckup works around this mess.
Every Bvckup executable has a corresponding named event in the Global namespace. The event is created by the first instance of the program when it's launched, and it is opened by all other instances.
The name of the event includes the file ID of the executable file. This groups running instances by the file they are using, because, after all, the installer is interested in a specific executable and nothing else.
For example, the following is an event name used by the Bvckup installation on my development box -
Global\Bvckup2.e65a96a8001f000000013cba.Update
—
With this in place, the installer starts off by signaling the event.
In response, a running program makes a temporary copy of its executable file and starts it in a special "relauncher" mode. And then it exits.
The "relauncher" process opens the "update" event and simply spins there, waiting for it to become unsignaled.
Meanwhile the installer waits for the program file to become unlocked. Once all running instances replace themselves with the relaunchers, the executable frees up and the installer proceeds with the update. It then clears the "update" event and exits.
This releases the relaunchers, they restart program instances and - ta-da - the update is completed.
—
Techincally, instead of adding a special "relauncher" mode to the program it's possible just to have a small standalone relauncher program. It comes down to a matter of taste. Some people don't like to spawn from temporary executables, and I like to keep my file count to a bare minimum. To each his own.
On Windows, running a program locks its file, preventing any changes from being made to it. This is not required per se and there are operating systems that don't do this. There are also custom PE loaders for Windows that can take an executable and create a running process out of it without any locking fuss. For example, bo2k could do that back in 1999. See its source code for details if interested.
To complicate matters, there might be more than one process running off the same executable. For example, if several users are logged into a Windows box and each runs a copy of a program.
Therefore, the first order of business for any installer is to shut down all running instances of the program. A well-behaved installer should also restart all these instances after it's done with updating.
The shutting down part is not a big deal. There's more than one way to do this and all of them are fairly straight-forward.
It's the restarting that is contrived.
Not only program instances may be running under different user accounts, they may also be running with custom command-line arguments, as Administrators or system services. Restarting this cornucopia of awesomeness is not an envious task.
Here's how Bvckup works around this mess.
Every Bvckup executable has a corresponding named event in the Global namespace. The event is created by the first instance of the program when it's launched, and it is opened by all other instances.
The name of the event includes the file ID of the executable file. This groups running instances by the file they are using, because, after all, the installer is interested in a specific executable and nothing else.
For example, the following is an event name used by the Bvckup installation on my development box -
With this in place, the installer starts off by signaling the event.
In response, a running program makes a temporary copy of its executable file and starts it in a special "relauncher" mode. And then it exits.
The "relauncher" process opens the "update" event and simply spins there, waiting for it to become unsignaled.
Meanwhile the installer waits for the program file to become unlocked. Once all running instances replace themselves with the relaunchers, the executable frees up and the installer proceeds with the update. It then clears the "update" event and exits.
This releases the relaunchers, they restart program instances and - ta-da - the update is completed.
Techincally, instead of adding a special "relauncher" mode to the program it's possible just to have a small standalone relauncher program. It comes down to a matter of taste. Some people don't like to spawn from temporary executables, and I like to keep my file count to a bare minimum. To each his own.
Dotting all i's and crossing all t's, of which there's
plenty now that there's an optional system service
that just begs to be stopped, restarted and deleted
from under app's feet.
Doh... I missed the disabling of the service. Will add it tomorrow.
Doh... I missed the disabling of the service. Will add it tomorrow.
Redesigned.
Vector icons for the website.
This is my first time not using pixel images to
decorate a website. Instead, all icons are saved
in the
SVG
format and converted to a webfont, which is then
used to style regular HTML characters into icons.
For example, here's a font with the above icon set - VectorIcons.ttf
The advent of webapps like IcoMoon and Fontello not only made the process of building @font-face kits an absolute no-brainer, it also gave an easy access to hundreds of existing vector icons, many of which are free and of production quality.
When coupled with now virtually universal support for CSS text-shadow, box-shadow and gradients, this almost completely eliminates the need for raster form of smaller decorative and nuancing elements.
This is a very big deal and a great time-saver.
For example, here's a font with the above icon set - VectorIcons.ttf
The advent of webapps like IcoMoon and Fontello not only made the process of building @font-face kits an absolute no-brainer, it also gave an easy access to hundreds of existing vector icons, many of which are free and of production quality.
When coupled with now virtually universal support for CSS text-shadow, box-shadow and gradients, this almost completely eliminates the need for raster form of smaller decorative and nuancing elements.
This is a very big deal and a great time-saver.
Re-finalized the sign-up form styling and mechanics.
Few styling options for an input control.
Leaning towards top right or bottom right,
but still need to see what works best in
context.
More or less done with the Features section
on the new website.
Probably missed few things, but that's easy to add later on. The biggest time sink was trying to inject some visual hierarchy into the list, but without going against overall minimalist look and feel of the site.
Might still revise it after sleeping on it for a couple of days :) Won't be the first time.
Probably missed few things, but that's easy to add later on. The biggest time sink was trying to inject some visual hierarchy into the list, but without going against overall minimalist look and feel of the site.
Might still revise it after sleeping on it for a couple of days :) Won't be the first time.
Plodding through some grunt work with the
new website. An almost equal split between
working on a copy, javascript and styling.
As simple as the screencap looks, it came with two hurdles that needed solving.
As simple as the screencap looks, it came with two hurdles that needed solving.
The header font, Alte Haas Grotesk, didn't
display correctly after it was
converted
to the @font-face kit. Specifically, the
lowercase 'o' looked like this:
The font's free, but I couldn't find a copy of its license, so I started tracking down font's author to ask a permission to edit the outlines.
While waiting for a reply, I found another workaround which involved generating the webfont kit from OTFs instead of TTFs. Not a big deal, but it still took over 3 hours to resolve.
—
The second issue was with the slide-out, fade-in animation. Sometimes the slide-out didn't extend all the way down and sometimes it overshot. This traced back to the fact that slide-out height was computed before the OpenSans was fully loaded and the script fused basic sans-serif to compute the height.
Again, not too complicated, but, again, cost an hour to work through.
:-|
Little bit of this, little bit of that - all unplanned and it all adds up.
The font's free, but I couldn't find a copy of its license, so I started tracking down font's author to ask a permission to edit the outlines.
While waiting for a reply, I found another workaround which involved generating the webfont kit from OTFs instead of TTFs. Not a big deal, but it still took over 3 hours to resolve.
The second issue was with the slide-out, fade-in animation. Sometimes the slide-out didn't extend all the way down and sometimes it overshot. This traced back to the fact that slide-out height was computed before the OpenSans was fully loaded and the script fused basic sans-serif to compute the height.
Again, not too complicated, but, again, cost an hour to work through.
Little bit of this, little bit of that - all unplanned and it all adds up.
Nearly done with the new website design. Here's
a snippet of how the site's header interacts with
scrolling.
First pass over UI styling for the main window.
Released a small playable test of the directory scanning module.
It appears that using as many threads as there are CPU cores and traversing the tree in a depth-first fashion (scan child directories first and sibling directories second) yields the best scanning times.
Test and see for yourself - bvckup2-demo2.exe or 64 bit version.
Also, see here for details.
It appears that using as many threads as there are CPU cores and traversing the tree in a depth-first fashion (scan child directories first and sibling directories second) yields the best scanning times.
Test and see for yourself - bvckup2-demo2.exe or 64 bit version.
Also, see here for details.
Updated
@bvckup
background with V2 branding.
2nd sketch for
bvckup2.com redesign.
Larger version is
here.
1st pass over the redesign of
bvckup2.com.
Larger version is
here.
Current design is almost two years old and
it grew steadily disconnected from the simpler
aesthetics I am pushing for now. So it needs
to be redone.
I will probably simplify this even further. I've been wanting to do a true minimalist site for a software product for a long time now and this seems like a real chance to do just that. Yay.
I will probably simplify this even further. I've been wanting to do a true minimalist site for a software product for a long time now and this seems like a real chance to do just that. Yay.
This is a
sprite
sheet of all raster images that are going into the
run-as-a-service demo.
Interestingly enough, the total size of individual
images saved as 32bit PNGs is 63,127 bytes, while
the size of this sprite sheet (without checkered
backdrop) is 63,829. After factoring in the size
of the code that is needed to parse the sprite sheet,
it really becomes a no brainer to keep every image
in its own file and embed them into .exe's resources
that way.
Added a "crash" handling and reporting facility
that includes several important changes compared to
the original version:
1. The program now differentiates between three types of
errors and handles them slightly differently.
First, there are unexpected operating system errors, i.e. when a core Windows function fails in a way that it shouldn't be failing. Classic example (that everyone is aware of) is TrackPopupMenu that is declared as returning boolean, but actually doesn't shy away from returning an integer under certain circuimstances.
Secondly, there are inconsistencies in app's own code, whereby one part assumes certain usage that is not observed by another part. These are so-called "assert" failures. The original version has three critical issues, all of this particular kind.
Lastly, there are straight-forward crashes. This is when a program is trying to operate with data that is no longer there or corrupted. This is the worst of them all as it typically indicates a major screw up on programmer's part, but in some cases it may also be an indication of failing hardware, like a faulty memory stick.
—
2. The program now generates a minidump - a small-ish file in a standard Windows format that captures the context of a failure. Minidump files dramatically simplify investigation of a problem on the developer's end.
—
3. Finally, the minidumps can now be uploaded to the support server right from the error notification window with one click. The transfer is based on WinInet API as this is supposed to improve the chances of getting the report through in proxied and otherwise "enterprised" environments.
For cases when the upload fails, the error dialog also offers an option of opening a link in a regular browser and reporting a problem that way.
First, there are unexpected operating system errors, i.e. when a core Windows function fails in a way that it shouldn't be failing. Classic example (that everyone is aware of) is TrackPopupMenu that is declared as returning boolean, but actually doesn't shy away from returning an integer under certain circuimstances.
Secondly, there are inconsistencies in app's own code, whereby one part assumes certain usage that is not observed by another part. These are so-called "assert" failures. The original version has three critical issues, all of this particular kind.
Lastly, there are straight-forward crashes. This is when a program is trying to operate with data that is no longer there or corrupted. This is the worst of them all as it typically indicates a major screw up on programmer's part, but in some cases it may also be an indication of failing hardware, like a faulty memory stick.
2. The program now generates a minidump - a small-ish file in a standard Windows format that captures the context of a failure. Minidump files dramatically simplify investigation of a problem on the developer's end.
3. Finally, the minidumps can now be uploaded to the support server right from the error notification window with one click. The transfer is based on WinInet API as this is supposed to improve the chances of getting the report through in proxied and otherwise "enterprised" environments.
For cases when the upload fails, the error dialog also offers an option of opening a link in a regular browser and reporting a problem that way.
64bit version is stored inside of the 32bit
executable, from where it is extracted and
launched to replace the 32bit version if
latter detects that it is being run on a
64bit system.
Pro:
a single executable to distribute
Con: the exe comes out weighing twice as much... or does it?
The size of a 32-bit executable can be easily reduced with the help of tools called executable packers. They work by turning the exe file into a self-extracting archive that also automatically launches the original exe after the extraction. It may sound simple but in reality it involves bundling a custom PE loader with the packed executable (as it might not be possible to extract the exe into a temporary file before running it).
In any case, exe packing is routinely used by developers who care about the size of their products, with uTorrent being one of most notable examples. Original Bvckup was also compressed, owning its tiny size in part to the magic of UPX.
—
The problem lies with 64-bit executables. For a variety of reasons there are presently no exe packers that support them.
However if we store 64bit binary inside of the 32bit executable and then pass latter through a compressor, we end up with both being compressed. Ta-da!
But there's more.
—
When an .exe is compressed, the compressor leaves the primary application icon intact. This is done so that the app would keep its original appearance in Windows Explorer and not sport a generic "program" icon.
Starting with Vista, the app is expected to support a variety of icon sizes that quickly add up. For example, bvckup's icon weighs in at about 110KB. That's almost half as big as the actual code, the code that does quite a few meaningful things on top of just looking great :)
So, if we do the 64-in-32bit packaging, we end up compressing 64bit version in its entirety, including its icon, and arriving at a greater size reduction if we were to compress the exes separately.
Win-win.
Con: the exe comes out weighing twice as much... or does it?
The size of a 32-bit executable can be easily reduced with the help of tools called executable packers. They work by turning the exe file into a self-extracting archive that also automatically launches the original exe after the extraction. It may sound simple but in reality it involves bundling a custom PE loader with the packed executable (as it might not be possible to extract the exe into a temporary file before running it).
In any case, exe packing is routinely used by developers who care about the size of their products, with uTorrent being one of most notable examples. Original Bvckup was also compressed, owning its tiny size in part to the magic of UPX.
The problem lies with 64-bit executables. For a variety of reasons there are presently no exe packers that support them.
However if we store 64bit binary inside of the 32bit executable and then pass latter through a compressor, we end up with both being compressed. Ta-da!
But there's more.
When an .exe is compressed, the compressor leaves the primary application icon intact. This is done so that the app would keep its original appearance in Windows Explorer and not sport a generic "program" icon.
Starting with Vista, the app is expected to support a variety of icon sizes that quickly add up. For example, bvckup's icon weighs in at about 110KB. That's almost half as big as the actual code, the code that does quite a few meaningful things on top of just looking great :)
So, if we do the 64-in-32bit packaging, we end up compressing 64bit version in its entirety, including its icon, and arriving at a greater size reduction if we were to compress the exes separately.
Win-win.
A page from the first demo of Bvckup 2.
Working on a shared memory buffer interface,
a part of the
IPC
module, which is what enables
the separation of the engine and the UI.
A bit more humane approach to critical error reporting.
The message is a homage to
NetPositive,
the BeOS browser that pioneered the technique of using
haiku
in error reporting. In fact, the message in option B is from
the NetPoisitive
original list.
The icon is that of a Zen circle, which has a plenty of meanings, many of which applicable given the circumstances.
The icon is that of a Zen circle, which has a plenty of meanings, many of which applicable given the circumstances.
Three-frame animation of engine's heartbeats for the demo.
Final UI styling for the "Run As A Service" demo.
The "Run as an Administrator" icon.
Test of a fading transition, part of a small library
that deals with supporting UI animations.
Fading animation dissolves a part of a window into
background, in preparation for transitioning the
window to another state, e.g. moving to the next
page in a wizard or resizing the window into a
more compact form.
As simple as it sounds, these sort of animation is fairly complex to implement on top of the Windows API.
Windows includes native support for window transparency in a form of layered windows, but it comes with a list of limitations that aren't exactly practical or easy to work, except for several specific cases.
The simplest working solution involves taking a screenshot of the window, creating an overlay window and then using it as a canvas to render the transition from an empty background to the screenshot. All the while making sure that Windows doesn't sneak in an unsolicited window refresh that would lead to a flicker, which in itself is a very "fun" thing to debug.
As simple as it sounds, these sort of animation is fairly complex to implement on top of the Windows API.
Windows includes native support for window transparency in a form of layered windows, but it comes with a list of limitations that aren't exactly practical or easy to work, except for several specific cases.
The simplest working solution involves taking a screenshot of the window, creating an overlay window and then using it as a canvas to render the transition from an empty background to the screenshot. All the while making sure that Windows doesn't sneak in an unsolicited window refresh that would lead to a flicker, which in itself is a very "fun" thing to debug.
Every application icon that is exposed to the
operating system must exist in at least nine
sizes.
16, 32, 48, 256 are more or less expected.
These are used in "small", "icon", "tile"
and "extra-large " views in Windows Explorer.
20, 22 and 26 are possible sizes of an icon used in the window caption. The exact size depends on screen's DPI level.
And 40 (along with 32 and 48) is the size of an icon used in Alt-Tab view on W7, again depending on the screen's DPI.
If you don't provide an appropriately sized icon, the OS will grab one that is slightly bigger and shrink it down, ruining the pixel polish.
Lastly, each icon ideally must be saved in 32, 8 and 4 bits per pixel formats, so to be compatible... you know... with EGA cards
It would've not that big of a deal if it weren't for the size of a resulting .ico file. For example, the above icon ends up being ~ 120KB, whereas the actual .exe it is attached to and that does quite a few meaningful things sits at around 96KB.
20, 22 and 26 are possible sizes of an icon used in the window caption. The exact size depends on screen's DPI level.
And 40 (along with 32 and 48) is the size of an icon used in Alt-Tab view on W7, again depending on the screen's DPI.
If you don't provide an appropriately sized icon, the OS will grab one that is slightly bigger and shrink it down, ruining the pixel polish.
Lastly, each icon ideally must be saved in 32, 8 and 4 bits per pixel formats, so to be compatible... you know... with EGA cards
It would've not that big of a deal if it weren't for the size of a resulting .ico file. For example, the above icon ends up being ~ 120KB, whereas the actual .exe it is attached to and that does quite a few meaningful things sits at around 96KB.
Similarly, every bit of graphics in the app
must be done in at least three different sizes
- one for each common screen DPI level.
DPI stands for dots-per-inch and it's a metric
of a pixel density on the screen. The smaller
the screen, the more pixels are packed into a
square inch, assuming the resolution stays the
same. The difference in DPI between a large
LCD panel and a 12" notebook screen can easily
be a factor of two or more.
This means that if a program renders 9px text on both screens, on one it will appear twice as small as on the other one, and might command a need for a good magnifying glass.
So naturally it's a good idea to make the OS aware of the actual DPI of the monitor in use. If it's too high, the OS can do few things to make things visible with a naked eye again. Like increase size of the default system font and make windows generally larger.
This can be done automatically by looking at the monitor .inf files or by letting the user configure the DPI manually.
—
XP defaults to 96 DPI and with some difficulty it can be changed to 120 DPI. Vista and newer versions are better. They come with four predefined levels - 96, 120, 144 and 192 - and they also support custom settings.
This brings us back to needing multiple versions of every graphic in the app.
—
Vista is the first Windows version to routinely use monitor-specific DPI values out of the box. As such 96 is no longer a dominant DPI level and the apps are now expected to take the DPI into an account when rendering their content.
The higher the DPI, the larger the graphics and the fonts the app should use. The penalty for not doing this is an automatic stretching of the app window that inevitably leads to a blurry appearance and a host of other lovely issues.
In other words - not complying with OS expectations is not really an option. Hence the need for 3+ differently sized copies of the same pixel art.
This means that if a program renders 9px text on both screens, on one it will appear twice as small as on the other one, and might command a need for a good magnifying glass.
So naturally it's a good idea to make the OS aware of the actual DPI of the monitor in use. If it's too high, the OS can do few things to make things visible with a naked eye again. Like increase size of the default system font and make windows generally larger.
This can be done automatically by looking at the monitor .inf files or by letting the user configure the DPI manually.
XP defaults to 96 DPI and with some difficulty it can be changed to 120 DPI. Vista and newer versions are better. They come with four predefined levels - 96, 120, 144 and 192 - and they also support custom settings.
This brings us back to needing multiple versions of every graphic in the app.
Vista is the first Windows version to routinely use monitor-specific DPI values out of the box. As such 96 is no longer a dominant DPI level and the apps are now expected to take the DPI into an account when rendering their content.
The higher the DPI, the larger the graphics and the fonts the app should use. The penalty for not doing this is an automatic stretching of the app window that inevitably leads to a blurry appearance and a host of other lovely issues.
In other words - not complying with OS expectations is not really an option. Hence the need for 3+ differently sized copies of the same pixel art.
Trying to find a balance between using custom styling and not getting to far away from the native OS look and feel. See also 8 other options leading up to this one.
Illustrative icons for the same demo.
First pass over the demo UI.
V2 running as a service,
an actual screenshot for a change.
Exploration of measured blending
of native and custom UI styling.
An idea for a space-saving tab control. Shelved.
An idea for an in-form buttons.
Exploring full-on customization.
Earlier sketches of heavily customized UI.
First is a cosmetic change to how an inline editing is done - click on a Pen icon, start typing, press Enter to save, Escape to cancel (or click on respective button if that's your preference).
Second change is a bit more interesting - it adds support for deactivating licenses via the dashboard:
Please note that this feature is currently reserved for site license and larger resellers only, and it is enabled on a case-by-case basis.