The initial revision of the auto-update opton
is in.
It's a simple extension of existing "check for updates" option that skips showing the "new version found" prompt and just goes straight to the updating part.
—
There are two caveats, both applicable only when the app is running in service mode and both stemming from the fact the updates are driven by the UI.
First is that closing the UI will effectively disable auto-updates. Second - if the UI is running without admin privileges, running the installer will trigger the UAC prompt.
Both will be resolved shortly by moving all update logic into the engine module. Meanwhile...
The auto-update option will be hidden by default unless the Preferences window is opened with the Control button held down.
It's a simple extension of existing "check for updates" option that skips showing the "new version found" prompt and just goes straight to the updating part.
There are two caveats, both applicable only when the app is running in service mode and both stemming from the fact the updates are driven by the UI.
First is that closing the UI will effectively disable auto-updates. Second - if the UI is running without admin privileges, running the installer will trigger the UAC prompt.
Both will be resolved shortly by moving all update logic into the engine module. Meanwhile...
The auto-update option will be hidden by default unless the Preferences window is opened with the Control button held down.
Release 73 will also be adding support for monthly
backups. It will now be possible to set a backup
to run on the N-th of every M-th month.
You surely are thinking "but 31st!", "February!" or perhaps even "every 6 months staring on Jan 31st!". Good point, the UI got just the thing for this -
You surely are thinking "but 31st!", "February!" or perhaps even "every 6 months staring on Jan 31st!". Good point, the UI got just the thing for this -
A rework of the loggging module, coming up in Release 73.
—
As you may or may not know, Bvckup 2 uses a dual-module architecture of the inside.
As you may or may not know, Bvckup 2 uses a dual-module architecture of the inside.
One module is the backup engine. It maintains
the list of backup jobs, schedules, executes
backups and generally does other useful
things that don't require direct user involvement.
The second module is the user interface. This is the part that displays the app's window and allows users to monitor and control whatever the engine module is doing.
The principal point is that the engine and the UI are isolated from each other. They share no run-time data and communicate with each other using a messaging protocol.
This isolation is what enables the app to run in service mode. In this mode the engine runs in its own process, as a system service, and the UI runs as a regular desktop app.
It was all well and beautiful, but there was, however, a caveat.
—
The caveat was the backup logs.
Logs are inherently shared data - they are written by the engine and they are read and displayed by the UI. Logs tend to be bulky, so it is shared app data of a special kind and it requires special handling.
Prior to R73 the app included a third module (in its dual-module architecture, no less) - the log manager. It was a complicated contraption that abstracted log access for the engine and the UI, and the biggest issue with it was that it behaved completely differently depending on which mode the app was in.
In desktop mode, the manager was writing *and* indexing logs on behalf of the engine and the UI was merely displaying them. However, in service mode, the engine did no indexing, just the writing, while the UI was making copies of all logs for its own use and then it was indexing *them*.
This is exactly the kind of non-uniform conditional processing that the dual-module design was meant to avoid!
—
Enter R73.
The separate logging module is no more. The engine no longer does any log indexing, it is now entirely the UI's reponsibility. Indexes can now be rebuilt on demand at any time, and finally, the UI no longer creates its own copies of the log files. Instead, it works with the engine to gain access to the engine's copy of the logs, using DuplicateHandle to move open file handles across the process boundary when needed.
The UI now can display logs while it is still catching up with them.
Think -- running in service mode and launching the UI only once a week. On each launch the UI needs to index a week's worth of log entries it hasn't seen before. Prior to R73, the UI would just sit there and show "synching ... XX% done". Now the sync is done in the background (as per the screenshot).
The UI now can also recover gracefully from indexes getting corrupted (e.g. due to a cold power cycle or blue screen).
Just as important, the underlying code is now much simpler, lighter and more uniform, the executable is 8KB smaller and the developer is generally happier. Good news all around.
The second module is the user interface. This is the part that displays the app's window and allows users to monitor and control whatever the engine module is doing.
The principal point is that the engine and the UI are isolated from each other. They share no run-time data and communicate with each other using a messaging protocol.
This isolation is what enables the app to run in service mode. In this mode the engine runs in its own process, as a system service, and the UI runs as a regular desktop app.
It was all well and beautiful, but there was, however, a caveat.
The caveat was the backup logs.
Logs are inherently shared data - they are written by the engine and they are read and displayed by the UI. Logs tend to be bulky, so it is shared app data of a special kind and it requires special handling.
Prior to R73 the app included a third module (in its dual-module architecture, no less) - the log manager. It was a complicated contraption that abstracted log access for the engine and the UI, and the biggest issue with it was that it behaved completely differently depending on which mode the app was in.
In desktop mode, the manager was writing *and* indexing logs on behalf of the engine and the UI was merely displaying them. However, in service mode, the engine did no indexing, just the writing, while the UI was making copies of all logs for its own use and then it was indexing *them*.
This is exactly the kind of non-uniform conditional processing that the dual-module design was meant to avoid!
Enter R73.
The separate logging module is no more. The engine no longer does any log indexing, it is now entirely the UI's reponsibility. Indexes can now be rebuilt on demand at any time, and finally, the UI no longer creates its own copies of the log files. Instead, it works with the engine to gain access to the engine's copy of the logs, using DuplicateHandle to move open file handles across the process boundary when needed.
The UI now can display logs while it is still catching up with them.
Think -- running in service mode and launching the UI only once a week. On each launch the UI needs to index a week's worth of log entries it hasn't seen before. Prior to R73, the UI would just sit there and show "synching ... XX% done". Now the sync is done in the background (as per the screenshot).
The UI now can also recover gracefully from indexes getting corrupted (e.g. due to a cold power cycle or blue screen).
Just as important, the underlying code is now much simpler, lighter and more uniform, the executable is 8KB smaller and the developer is generally happier. Good news all around.
Running an extended set of tests to fine-tune the
performance of the Bvckup 2 bulk copier.
It's an inherently simple matter of determining an optimal size and count of read/write buffers used to ferry data from source to destination. There are however ... erm ... details.
—
A source can be local or it can be over-the-network. There are slow local devices (e.g. a USB2 drive) and there are fast local devices. There is a warm, there is a cold cache and there are files that are cached only partially.
The writes can go out in lots of smaller chunks, all in parallel, or with a smaller number of large buffers. Again, it can go to a local device or over the network. The app can also explicitly flush all buffers after copying or leave flushing to the system.
—
The above graph is the I/O throughput of a 0.5 GB file being repeatedly copied from the local HDD to Synology Diskstation over a Gigabit wire. What changes between the runs is the buffering strategy and as you may notice there is a difference.
Still need to run several over-nighters to gather more statistics, but there is already a couple of things that can be changed in the copying code to make it run faster.
Scheduled to appear in the Release 72, coming up shortly.
It's an inherently simple matter of determining an optimal size and count of read/write buffers used to ferry data from source to destination. There are however ... erm ... details.
A source can be local or it can be over-the-network. There are slow local devices (e.g. a USB2 drive) and there are fast local devices. There is a warm, there is a cold cache and there are files that are cached only partially.
The writes can go out in lots of smaller chunks, all in parallel, or with a smaller number of large buffers. Again, it can go to a local device or over the network. The app can also explicitly flush all buffers after copying or leave flushing to the system.
The above graph is the I/O throughput of a 0.5 GB file being repeatedly copied from the local HDD to Synology Diskstation over a Gigabit wire. What changes between the runs is the buffering strategy and as you may notice there is a difference.
Still need to run several over-nighters to gather more statistics, but there is already a couple of things that can be changed in the copying code to make it run faster.
Scheduled to appear in the Release 72, coming up shortly.
A quick peek at the Reseller Dashboard.
Coming soon...
"Send test email..." is now done.
R71 adds an option of automatically enabling
NTFS compression
for the backup logs.
Based on what I've seen so far, both .log and .tdb files compress to about 20-30% of their raw sizes.
Based on what I've seen so far, both .log and .tdb files compress to about 20-30% of their raw sizes.
Reworked part of the More Options page in
the backup configuration window.
* The animation here is a bit jerky, but it's just a GIF encoding artefact.
* The animation here is a bit jerky, but it's just a GIF encoding artefact.
Email template editor. "Show variables" opens
up another part that lists all variables that
can be used in the form fields.
This is getting a bit too complex for my taste, but it's the part of the UI with little foot traffic, so it's probably OK.
This is getting a bit too complex for my taste, but it's the part of the UI with little foot traffic, so it's probably OK.
A little primer on the mail server configuration.
Upon even further consideration it looks like
the From field is not of a prime interest either,
so it too can go into the Details.
This cuts down the number of fields and renders section headers unnecessary, so they too can go.
Basically the idea is to reduce the dialog down to fields that actually need user's input and stash everything else away... I don't know if it's the right thing to do though. Perhaps having a summary of the whole email setup (including the subject, From and the message) is not such a bad thing. Not sure. Need to sleep on it.
This cuts down the number of fields and renders section headers unnecessary, so they too can go.
Basically the idea is to reduce the dialog down to fields that actually need user's input and stash everything else away... I don't know if it's the right thing to do though. Perhaps having a summary of the whole email setup (including the subject, From and the message) is not such a bad thing. Not sure. Need to sleep on it.
Upon further consideration, decided to remove
both Subject and Message off this dialog to
keep it simple(r) and to the point.
First stab at the email configuration window.
Too many options, too little space. As usual.
EHLO there!
Release 71 is getting email notifications, so
here's all you always wanted to know about teaching
your app speak SMTP.
First of all, let's get MAPI non-sense out of the way. As per usual, it sounds good in theory, but an awful mess in practice. So, no, MAPI is not an option unless your goal is to burn through your support budget and/or practice your communication skills with less-than-happy users.
This leaves us with raw SMTP.
SMTP is an inherently simple protocol - send a text command, receive a text response. Repeat. Open a socket, connect, send(), recv(), done. A couple of hours of work at most. Ha.
—
It starts to get complicated when you throw TLS in a mix to secure the communication with the server. In particular, this is required when you want to send emails in a way that requires logging into a server first.
There are two ways in which SMTP can be combined with TLS.
First is when you take an existing SMTP session and say STARTTLS. The server says OK and then you basically stop talking SMTP over your TCP connection and start talking TLS. Then, once the TLS handshake completes, you go back to SMTP, but now it all goes over the TLS session, nice and encrypted. This is a standard way, that's what every decent SMTP client should do. This is TCP/25 and TCP/587.
Second option is SMTPS. You first establish a TLS session and then start talking SMTP over it. Just like HTTPS. Apparently, this is non-standard and discouraged way, but still widely used. This is TCP/465.
—
Then, there's a matter of how to add TLS support to the app.
The obvious choice is OpenSSL, but that will cost you 200-300KB in extra binary code, so unless you already have OpenSSL linked in, it's not really an option, not for a *backup* app.
Digging through MSDN you may come across something called SSPI or Security Support Provider Interface. It is an attempt to stuff several security protocols like SSL and Kerberos into a single framework in a faint hope for a miracle of reuse. One of SSPI providers is a mysterious entity called "Secure channel", which, upon closer inspection, turns out to be ... yes, TLS!
So, the next step is try and work through all the abstraction fluff of SSPI to make it actually do the TLS handshake. The idea is actually quite neat - you can create a stateful object (context) that takes in data you get from the peer and spits out some data for the peer. At some point it completes the handshake and becomes available for accepting app's own data, which it encrypts and hands you a blob to send to the peer. You would then also get blobs from the peer, pass it to this context/object and get back plaintext data for the app. Whether it's TLS or ROT13 on the back is completely transparent to the app.
That's the idea. In reality, the documentation is subpar and lacking and there are exactly two examples - one is incomplete and another one is buggy.
The good news is though is that once this bloody contraption gets off the ground, you get full TLS support at the cost of just several kilobytes.—
With TLS done, there's still a bunch of smaller things to be taken care of, e.g. lack of asynchronous DNS query API in pre-W8 windows. Not a big deal, nothing a spawning of a worker thread would solve, but still ... a bit here, a bit there and it all adds up.
In other words, it's all coming together, but as per usual it all takes longer than anticipated. Shocking, I know :)
First of all, let's get MAPI non-sense out of the way. As per usual, it sounds good in theory, but an awful mess in practice. So, no, MAPI is not an option unless your goal is to burn through your support budget and/or practice your communication skills with less-than-happy users.
This leaves us with raw SMTP.
SMTP is an inherently simple protocol - send a text command, receive a text response. Repeat. Open a socket, connect, send(), recv(), done. A couple of hours of work at most. Ha.
It starts to get complicated when you throw TLS in a mix to secure the communication with the server. In particular, this is required when you want to send emails in a way that requires logging into a server first.
There are two ways in which SMTP can be combined with TLS.
First is when you take an existing SMTP session and say STARTTLS. The server says OK and then you basically stop talking SMTP over your TCP connection and start talking TLS. Then, once the TLS handshake completes, you go back to SMTP, but now it all goes over the TLS session, nice and encrypted. This is a standard way, that's what every decent SMTP client should do. This is TCP/25 and TCP/587.
Second option is SMTPS. You first establish a TLS session and then start talking SMTP over it. Just like HTTPS. Apparently, this is non-standard and discouraged way, but still widely used. This is TCP/465.
Then, there's a matter of how to add TLS support to the app.
The obvious choice is OpenSSL, but that will cost you 200-300KB in extra binary code, so unless you already have OpenSSL linked in, it's not really an option, not for a *backup* app.
Digging through MSDN you may come across something called SSPI or Security Support Provider Interface. It is an attempt to stuff several security protocols like SSL and Kerberos into a single framework in a faint hope for a miracle of reuse. One of SSPI providers is a mysterious entity called "Secure channel", which, upon closer inspection, turns out to be ... yes, TLS!
So, the next step is try and work through all the abstraction fluff of SSPI to make it actually do the TLS handshake. The idea is actually quite neat - you can create a stateful object (context) that takes in data you get from the peer and spits out some data for the peer. At some point it completes the handshake and becomes available for accepting app's own data, which it encrypts and hands you a blob to send to the peer. You would then also get blobs from the peer, pass it to this context/object and get back plaintext data for the app. Whether it's TLS or ROT13 on the back is completely transparent to the app.
That's the idea. In reality, the documentation is subpar and lacking and there are exactly two examples - one is incomplete and another one is buggy.
The good news is though is that once this bloody contraption gets off the ground, you get full TLS support at the cost of just several kilobytes.
With TLS done, there's still a bunch of smaller things to be taken care of, e.g. lack of asynchronous DNS query API in pre-W8 windows. Not a big deal, nothing a spawning of a worker thread would solve, but still ... a bit here, a bit there and it all adds up.
In other words, it's all coming together, but as per usual it all takes longer than anticipated. Shocking, I know :)
Finalized the UX for auto-configuration of
the device tracking. It's essentially the
same as animation immediately below except
for a brief flash of blue when tracking
config is touched by the app.
Another subtle UI change in Release 71.
When configuring or reconfiguring a backup, the app would automatically enable/disable device tracking when a location is changed to use a different drive.
Since this change happens behind the scenes, the app will attract the attention to the fact with this little animation. When hovered over, the tooltip for the pushpin button will explain what's up and suggest reviewing the changes.
My only concern is that this still might be a bit too subtle, but it's better than what's in there now... which is no notification at all.
When configuring or reconfiguring a backup, the app would automatically enable/disable device tracking when a location is changed to use a different drive.
Since this change happens behind the scenes, the app will attract the attention to the fact with this little animation. When hovered over, the tooltip for the pushpin button will explain what's up and suggest reviewing the changes.
My only concern is that this still might be a bit too subtle, but it's better than what's in there now... which is no notification at all.
New UI for configuring device tracking in R71.
This is one of those changes that look almost trivial when done, but behind the scenes take a disproportionate amount of time to actually flesh out.
In this case, the main problem was that the feature needs a detailed explanation and the question, of course, was how one can fit a bunch of text into a reasonably small space so not to have this text take over the actual configuration section.
—
The first iteration was the good old message box. You click on the "More on this..." and you get :
Nice, but looks like a hackjob, does't it? So, naturally, the next step was to spruce it up a bit - put text into a dedicated window, use RTF to add a bit of in-text formatting and a bit of animation to liven things up :
Nice again, looks better, but feels off. Too much going on. It looks like a conventional help window, but ... crippled. It's also not clear how to act when the user wants to resize the window - allow it, not allow it, restict minimum/maximum size, etc.
In short, what's lacking here is the cohesion. Luckily, the solution is right on the surface:
... whereby the text panel slides out on the "More info" click.
This is better, but now we have a window that sits lopsided on the desktop when the panel is open. It is also becoming wide, meaning that the app needs to mind the desktop edges, re-center the window, etc.
But if we just sleep on it, we might suddenly decide to make it an "either-or" window and then the UI magically becomes
(a) compact
(b) cohesive
(c) balanced
and
(d) with unlimited text space!
So there you have it - a new mini-help subsystem coming to Bvckup 2 in the Release 71, deployed on as-needed basis across the app.
Don't know about you, but I think it's neat, tidy and pretty damn clever :)
This is one of those changes that look almost trivial when done, but behind the scenes take a disproportionate amount of time to actually flesh out.
In this case, the main problem was that the feature needs a detailed explanation and the question, of course, was how one can fit a bunch of text into a reasonably small space so not to have this text take over the actual configuration section.
The first iteration was the good old message box. You click on the "More on this..." and you get :
Nice, but looks like a hackjob, does't it? So, naturally, the next step was to spruce it up a bit - put text into a dedicated window, use RTF to add a bit of in-text formatting and a bit of animation to liven things up :
Nice again, looks better, but feels off. Too much going on. It looks like a conventional help window, but ... crippled. It's also not clear how to act when the user wants to resize the window - allow it, not allow it, restict minimum/maximum size, etc.
In short, what's lacking here is the cohesion. Luckily, the solution is right on the surface:
... whereby the text panel slides out on the "More info" click.
This is better, but now we have a window that sits lopsided on the desktop when the panel is open. It is also becoming wide, meaning that the app needs to mind the desktop edges, re-center the window, etc.
But if we just sleep on it, we might suddenly decide to make it an "either-or" window and then the UI magically becomes
(a) compact
(b) cohesive
(c) balanced
and
(d) with unlimited text space!
So there you have it - a new mini-help subsystem coming to Bvckup 2 in the Release 71, deployed on as-needed basis across the app.
Don't know about you, but I think it's neat, tidy and pretty damn clever :)
Added support for the reseller licensing
to the app. Any license can now be tagged as
being sold through a distributor, in which
case Help/About will show distributor's name
and link to its site.
Additionally, Tech Support and Homepage links from the Help menu may be configured to go to the distributor's pages.
It may not look like much, but it's the most important non-technical feature to be added to the app to date.
More on this to follow...
Additionally, Tech Support and Homepage links from the Help menu may be configured to go to the distributor's pages.
It may not look like much, but it's the most important non-technical feature to be added to the app to date.
More on this to follow...
Another option, as per
this comment.
It's a really nice concept, but it requires a squint
to realize what the icon is. Highly-detailed 16x16 px
icons are hard to pull off.
Release 71 is going to reshuffle the contents of
the backup configuration window a bit and location
fields are getting equipped with a "device tracking"
icon as a result.
In other words, for a network location the app will display the "key" icon (see /wip entry immediately below) and for local paths it will show "tracking" icon. Both fronting respective dialog for configuring the details.
The main question, of course, is what the icon should be ...
In other words, for a network location the app will display the "key" icon (see /wip entry immediately below) and for local paths it will show "tracking" icon. Both fronting respective dialog for configuring the details.
The main question, of course, is what the icon should be ...
In a process of adding a way to configure
network logon credentials for remote locations.
Will be in Release 70.
Release 69 also adds an option of simulated backups
or, as they are also known, "dry runs".
The option is accessible via the right-click context menu and, when invoked, it puts the backup through its preparation phase, including snapshot creation, scanning and planning steps, and then dumps compiled backup plan to the log without executing it.
Simulation also executes the archive pruning without making any actual changes to the destination.
The option is accessible via the right-click context menu and, when invoked, it puts the backup through its preparation phase, including snapshot creation, scanning and planning steps, and then dumps compiled backup plan to the log without executing it.
Simulation also executes the archive pruning without making any actual changes to the destination.
Release 69 adds support for optional VSS writers.
—
VSS writers are the components of the shadow copying framework that are tasked with ensuring consistency of specific files in VSS snapshots. For example, when creating a snapshot, Windows would ask the MSSQL writer to flush any pending database changes in a way that would make on-disk copy of the database self-consistent.
Some of the writers are required and always participate in the snapshot creation process, but some are optional and they need to be explicitly enabled by the initiating app. Starting with R69 Bvckup will be enabling all optional writers and it will also provide a mechanism for excluding specific ones if needed.
VSS writers are the components of the shadow copying framework that are tasked with ensuring consistency of specific files in VSS snapshots. For example, when creating a snapshot, Windows would ask the MSSQL writer to flush any pending database changes in a way that would make on-disk copy of the database self-consistent.
Some of the writers are required and always participate in the snapshot creation process, but some are optional and they need to be explicitly enabled by the initiating app. Starting with R69 Bvckup will be enabling all optional writers and it will also provide a mechanism for excluding specific ones if needed.
It's shipped!
Here it is
Here it is
Nearly done with the checkout flow on the
website. Integrated with Stripe yesterday,
now finalizing the visuals.
This is a purchase receipt. It is both sent out and shown on the site when the payment goes through.
PS. All numbers and codes are made up. Just in case. Don't get any ideas :)
This is a purchase receipt. It is both sent out and shown on the site when the payment goes through.
PS. All numbers and codes are made up. Just in case. Don't get any ideas :)
Working on the production version of the website.
Cleaned up overall look and feel a bit, got rid of baby blue theme and anorexic body font, reworked the copy, expanded Features section and so on. Still need to slice and code it. Also asked a couple of people for the feedback, so waiting on that too.
Cleaned up overall look and feel a bit, got rid of baby blue theme and anorexic body font, reworked the copy, expanded Features section and so on. Still need to slice and code it. Also asked a couple of people for the feedback, so waiting on that too.
The feature everyone was so waiting for ... the payment form :)
Working through the necessary evil - the app licensing :)
The plan is to have a 14 day trial that doesn't require any explicit licensing. Just install and run. It will pop up a notification half way through the trial period and it will also say something on the last day.
Once the trial period is over and until a production (paid) license is installed, the app will go into an unlicensed mode and it will activate two restrictions (see screenshot). It will remain largely functional, but presumably not convenient enough for regular or unattended use.
You would then hit /purchase page on the website, go through the purchase process and get an activation code in return. The code will then need to be sticked into the activation form on the client (see 2nd screenshot) and it will fetch the license from the server.
If the licensing server is inaccessible for whatever reason, there's a couple of ways to retrieve the license manually (see last screenshot).
For those with Beta licenses, they will be accepted at the checkout as a discount code.
For those looking at large deployments / enterprise setups, there will be an option for automating licensing process. Get in touch for details if you have questions.
Updated the About window a bit.
Current design has somehow always felt ... cramped?
Super-final version of the progress bar as seen in Beta 60.
Note how colors of some parts are inverted under selection,
took me disturbingly long to arrive at this solution :-/
Finalized changes to the progress bar - now with more
contrast, a bit of depth and beautifully centered
splatter for partially copied blocks.
And here is a bunch of runners-up if anyone's interested.
And here is a bunch of runners-up if anyone's interested.
Just waiting for the test build to compile...
so here's another take on the delta copier
progress bar (and here's
existing version for comparison).
Clicking and holding error navigation arrow for 750 ms
now activates auto-scrolling. It starts at 10 errors
per second and accelerates linearly to 100 err/sec,
which is pretty damn fast.
Additionally, holding Ctrl while scrolling with the mouse wheel makes it go in 10 line increments.
Additionally, holding Ctrl while scrolling with the mouse wheel makes it go in 10 line increments.
It is now possible to clear the log view and
then revert it back to the full log view.
A smaller UI change.
When hopping through errors in a backup log, log viewer will now briefly highlight next entry if it cannot be pulled directly under the cursor. This basically adds a visual terminator to the interaction sequence, which is a surprsingly useful thing to have.
Otherwise it works as before.
When hopping through errors in a backup log, log viewer will now briefly highlight next entry if it cannot be pulled directly under the cursor. This basically adds a visual terminator to the interaction sequence, which is a surprsingly useful thing to have.
Otherwise it works as before.
Having thought about it a bit more, I can even
remove that slight upward slope from the hump
of the blue graph. The usage still climbs just
a little bit during the run, but for practical
intents and purposes it's flat.
Wrapping up changes to the logging system. Here's
a quick graph to demonstrate the effects of the
rework.
For an initial full run of a 100,000 file backup
memory usage is down from about 200 to 50 Megs.
For a 1,000,000 file backup the effect is similar
- peak memory usage dropped
from 2.9GB to 0.58GB
.
—
Previously, the UI would keep logs of last few backup runs in the memory and for larger backups this would eat up a good chunk of memory. This was especially true for the initial backups where all files needed copying and thus for 100K files it would generate close to half a million log entries.
For subsequent backups it wasn't a big deal as they would handle just the changes and generate only a handful of steps and a bit of log data So in the end, the memory usage was capped, but the cap was higher than it could've been.
Beta 60 addresses that.
—
Only a very small part of the log is now kept in memory, just enough to fill the log viewer panel in the UI. The rest is kept on the disk and read from there on as-needed basis.
However this is easier said than done.
Bvckup logs are hierarchical, meaning that they are generic trees with nodes that may have an arbitrary large number of children. Running a 1 million file backup would yield a "Processing..." entry with 1 million children, each with up to a dozen of descendants.
—
Displaying this tree would've not been a big deal if all log entries were always visible at all times. When the UI would've wanted to show entries 700,503 through 700,523 entries, it would've simply gone to a respective location in the log index file and plucked the data out. But...
The nodes can be collapsed and expanded. This hides and shows arbitrary chunks of the log tree and then the UI has this issue of finding 700,503-th *visible* item. And this is hard.
This implies quick look-ups in some sort of super-imposed binary search tree of visible items. A tree that changes every time a node is expanded or collapsed and that sits in a disk file, which restricts how you can walk and modify it without getting an I/O penalty.
This is fun. This is what cost me 3 weeks of shuffling through CS books and several complete rewrites of the tree indexing code. Believe it or not, but this is basically an original research and it seems that no one had to deal with this issue before. At least not publicly.
—
The solution is to use slightly customized version of an AVL tree to store the list of node's children (that's using tree within a tree), use it for the visible item look-ups and then optimize the hell out of the index I/O through aggressive write caching.
When all i's are dotted and t's crossed, this yields a logging system that can push up to 1 mil entries per second on a resonably equipped box. This might not be knock-your-socks-off impressive, but it's plenty enough for the task at hand.
Previously, the UI would keep logs of last few backup runs in the memory and for larger backups this would eat up a good chunk of memory. This was especially true for the initial backups where all files needed copying and thus for 100K files it would generate close to half a million log entries.
For subsequent backups it wasn't a big deal as they would handle just the changes and generate only a handful of steps and a bit of log data So in the end, the memory usage was capped, but the cap was higher than it could've been.
Beta 60 addresses that.
Only a very small part of the log is now kept in memory, just enough to fill the log viewer panel in the UI. The rest is kept on the disk and read from there on as-needed basis.
However this is easier said than done.
Bvckup logs are hierarchical, meaning that they are generic trees with nodes that may have an arbitrary large number of children. Running a 1 million file backup would yield a "Processing..." entry with 1 million children, each with up to a dozen of descendants.
Displaying this tree would've not been a big deal if all log entries were always visible at all times. When the UI would've wanted to show entries 700,503 through 700,523 entries, it would've simply gone to a respective location in the log index file and plucked the data out. But...
The nodes can be collapsed and expanded. This hides and shows arbitrary chunks of the log tree and then the UI has this issue of finding 700,503-th *visible* item. And this is hard.
This implies quick look-ups in some sort of super-imposed binary search tree of visible items. A tree that changes every time a node is expanded or collapsed and that sits in a disk file, which restricts how you can walk and modify it without getting an I/O penalty.
This is fun. This is what cost me 3 weeks of shuffling through CS books and several complete rewrites of the tree indexing code. Believe it or not, but this is basically an original research and it seems that no one had to deal with this issue before. At least not publicly.
The solution is to use slightly customized version of an AVL tree to store the list of node's children (that's using tree within a tree), use it for the visible item look-ups and then optimize the hell out of the index I/O through aggressive write caching.
When all i's are dotted and t's crossed, this yields a logging system that can push up to 1 mil entries per second on a resonably equipped box. This might not be knock-your-socks-off impressive, but it's plenty enough for the task at hand.
A quick update on what's happening.
Disk-based virtual data structures, that's what is :)
This is a part of an effort to further reduce and cap app's run-time memory usage. More specifically, it is meant to help the app cope gracefully with multi-million file backups that involve millions of steps and generate lots and lots of log data.
—
At the moment Bvckup's UI keeps backup logs in memory. It puts an effort to trim them to a reasonable size, but this still eats quite a bit of RAM, especially with large backups.
—
It costs about 120 bytes plus the size of an entry to keep a single log line in memory, whereby these 120 bytes help weave entries into a hierarchy (a tree) and track their state.
If we want to reduce the memory usage, then the most obvious optimization would be to keep the actual entry text in a disk file and load it on demand, when it's actually visible in the log viewer.
This certainly helps, but as the log grows, the memory usage still climbs. A million file backup generates about 4 million log entries. At 120 bytes per entry that's 0.5 Gig in just supporting structures. This is unacceptable.
—
So it means that the entire tree structure of the log needs to sit in a disk file and to be read from there as needed.
Disk-based tree structures are routinely used in databases and file systems to store indecies of data sets, typically in a form of a B-Tree and its variations. The kicker is that these structures are meant for storing sorted data and they optimize for searching. Trying to adapt them for storing unsorted data would be nothing short of fitting a square peg in a round hole.
—
Long story short - there are no ready-made code for storing and manipulating generic data trees on a disk, leave alone fast code. This appears to be an esoteric problem that everyone solves on their own.
The screenshot above shows a part of a small "tree database" library that I ended up writing, together with its own caching module, predictive page loader and a pony.
Took almost two weeks.
—
On a bright side though, this code is perfectly reusable for storing disk snapshots as well and this paves way for making the planner module work almost entirely off the the memory.
Once done, Bvckup memory usage will NOT depend on the particulars of a backup at all - this a very big deal and an incredibly improtant feature to have for any robust backup software.
So, stay tuned...
Disk-based virtual data structures, that's what is :)
This is a part of an effort to further reduce and cap app's run-time memory usage. More specifically, it is meant to help the app cope gracefully with multi-million file backups that involve millions of steps and generate lots and lots of log data.
At the moment Bvckup's UI keeps backup logs in memory. It puts an effort to trim them to a reasonable size, but this still eats quite a bit of RAM, especially with large backups.
It costs about 120 bytes plus the size of an entry to keep a single log line in memory, whereby these 120 bytes help weave entries into a hierarchy (a tree) and track their state.
If we want to reduce the memory usage, then the most obvious optimization would be to keep the actual entry text in a disk file and load it on demand, when it's actually visible in the log viewer.
This certainly helps, but as the log grows, the memory usage still climbs. A million file backup generates about 4 million log entries. At 120 bytes per entry that's 0.5 Gig in just supporting structures. This is unacceptable.
So it means that the entire tree structure of the log needs to sit in a disk file and to be read from there as needed.
Disk-based tree structures are routinely used in databases and file systems to store indecies of data sets, typically in a form of a B-Tree and its variations. The kicker is that these structures are meant for storing sorted data and they optimize for searching. Trying to adapt them for storing unsorted data would be nothing short of fitting a square peg in a round hole.
Long story short - there are no ready-made code for storing and manipulating generic data trees on a disk, leave alone fast code. This appears to be an esoteric problem that everyone solves on their own.
The screenshot above shows a part of a small "tree database" library that I ended up writing, together with its own caching module, predictive page loader and a pony.
Took almost two weeks.
On a bright side though, this code is perfectly reusable for storing disk snapshots as well and this paves way for making the planner module work almost entirely off the the memory.
Once done, Bvckup memory usage will NOT depend on the particulars of a backup at all - this a very big deal and an incredibly improtant feature to have for any robust backup software.
So, stay tuned...
Coming up in beta 59 - automatic archive prunning.
If all goes well, it'll be out tomorrow.
If all goes well, it'll be out tomorrow.
Effects of delta copying as seen in Task Manager.
1.28 TB read, 0.37 TB written out - just the changes!
1.28 TB read, 0.37 TB written out - just the changes!
Ready to go. Launching tomorrow.
Referrals.
Coming up shortly, stay tuned.
Coming up shortly, stay tuned.
Switched delta copying progress bar from varying color
of the partially copied blocks to varying their height.
This should help with telling apart skipped, partial and
fully-copied blocks.
Added an option for cloning an existing backup job.
Website footer work.
In preparation for the launch of the Referral program.
In preparation for the launch of the Referral program.
SAMBA !
This one - the *nix implementation of Windows Networking. If you have a NAS box the chances are that it's using Samba to support network access from Windows computers.
It is a good software, simpler and much snappier than its native Windows counterpart, but in case of NAS devices it typically runs on Linux. This means that the underlying file system is anything but NTFS. This in turn translates into no support for "created" timestamps and no support for certain attributes, including System and Hidden.
Moreover, the attributes that cannot be set or cleared depend on whether it's a dot file or a regular file. Who would've thought? :-|
From what I understand it *is* possible to set things up so that Samba would record these timestamps and attributes and make things transparent to its Windows clients, but in reality this doesn't seem to be happening.
Long story short - starting with Beta 49 Bvckup will now be explicitly testing destination file system to check whether c-times are supported and which attributes can be set.
Additionally, it will also explicitly test both source and destination file systems to determine the granularity of c-times and m-times. This is again thanks to some NAS devices that are customized to use 1 sec for m-times and 1 usec (micro) for c-times.
By popular request.
When ticked, Minimize button minimizes to the taskbar (as usual) and Close, the (x), minimizes to the systray.
When unticked, Minimize minimizes to the systray and Close exits the app.
Trivial in retrospect, but took me ages to realize that behavior of Close and Minimize can be controlled with just one setting.
When ticked, Minimize button minimizes to the taskbar (as usual) and Close, the (x), minimizes to the systray.
When unticked, Minimize minimizes to the systray and Close exits the app.
Trivial in retrospect, but took me ages to realize that behavior of Close and Minimize can be controlled with just one setting.
Have a mapped drive? Want to run as Admin?
Good luck with that!
Good luck with that!
Second pass over the same sketch. Trimmed down
decorations a bit, reworked content to have a
summary, added footer, etc.
Full-length version is here.
What do you think? Let me know - @apankrat or over an email.
Full-length version is here.
What do you think? Let me know - @apankrat or over an email.
Started sketching production version of the website.
As much as I personally like the minimalist version
that's currently up there, it just
doesn't seem to work that well :-/
When periodic backup misses its last scheduled run,
it's now possible to tell the app how to handle it.
There are two common cases - (a) the app itself wasn't running and (b) the backup drive is a removable and it wasn't plugged in.
With certain setups it makes sense to run the backup as soon as the app can, to "catch up" with any changes and get them backed up quickly.
In other cases, a backup may be large and heavy and set up to run every midnight so not to disturb anyone. Running it ASAP will generally be not a very good idea.
Hence this option.
There are two common cases - (a) the app itself wasn't running and (b) the backup drive is a removable and it wasn't plugged in.
With certain setups it makes sense to run the backup as soon as the app can, to "catch up" with any changes and get them backed up quickly.
In other cases, a backup may be large and heavy and set up to run every midnight so not to disturb anyone. Running it ASAP will generally be not a very good idea.
Hence this option.
By popular request.
Context hints for each section in More options.
Here's the screen-by-screen flow of the
licensing process.
It starts with the above window and directs users to the website to pick and purchase a license code, which is given on spot and also sent in the receipt email.
It starts with the above window and directs users to the website to pick and purchase a license code, which is given on spot and also sent in the receipt email.
Then copy-paste the code, click on Verify,
get this -
If this works (the server is accessible, the license code is valid, etc), then this is it. Thank you very much, click Ok, you are done -
Otherwise, it explains the failure and offers either to retry the activation or to use an alternative -
There are two alternatives, both are simple -
For offline boxes and locked-down setups there's the Copy To Clipboard option. Take the URL to a networked machine, open it in a browser, get a licensing file back and then drop it into the app's config folder.
The URL is exactly the same as the one used by the app's own licensing code and it comprises of the license code and installation ID and the fingerprint.
Prepare the email option creates a new email with default email client and puts the Installation ID, Fingerprint and License Code in the message along with some pleasantries. The license file will be attached to the reply and it will simply need to be dropped into %LocalAppData%\Bvckup2\engine.
If this works (the server is accessible, the license code is valid, etc), then this is it. Thank you very much, click Ok, you are done -
Otherwise, it explains the failure and offers either to retry the activation or to use an alternative -
There are two alternatives, both are simple -
For offline boxes and locked-down setups there's the Copy To Clipboard option. Take the URL to a networked machine, open it in a browser, get a licensing file back and then drop it into the app's config folder.
The URL is exactly the same as the one used by the app's own licensing code and it comprises of the license code and installation ID and the fingerprint.
Prepare the email option creates a new email with default email client and puts the Installation ID, Fingerprint and License Code in the message along with some pleasantries. The license file will be attached to the reply and it will simply need to be dropped into %LocalAppData%\Bvckup2\engine.
A bit of UI animation for the license
activation window. This is a animated
GIF, so the colors and the timing are
all off. It looks tidier and faster in
reality.
Licensing support. Yay!
This is one of those things that is rather simple conceptually, but translates into a lot of actual petty work.
—
When the app is started, it looks at several system properties including the CPU model, Windows' MachineGuid value and others, hashes them together to produce an "Installation ID". For example -
The ID stays the same under normal computer use, but it changes with hardware upgrades or when the OS is reinstalled.
The app also generates an "Installation fingerprint" by hashing each of the properties separately and stringing them together. This yields a longer hash -
Like ID, the fingerprint changes with hardware upgrades and OS reinstalls, but, unlike ID, it changes only in parts. This makes it suitable for detecting incremental changes to the user's computer.
Both ID and Fingerprint identify user's installation, the difference is that the ID is used on the app side and the Fingerprint - on the licensing server side.—
Next, ID and fingerprint are submitted to the licensing server, either by the app itself or by a human through a website. When licensing for production use (as opposed to beta or evaluation), the request also includes a license code - a random token that is issued to the user in exchange for payment.
The server responds with a license, which is a short text file that binds together few pieces of information and seals them with an RSA signature of the licensing server.
For beta and evaluation licenses the "license code" is generated automatically by the licensing server. The large block of gibberish at the bottom of the license is the RSA signature.
—
When the app starts, it loads the license and verifies it using public key of the licensing server that is embedded in the app.
If the license is valid, the app works as normal. If the license is missing or expired or doesn't match the installation ID, the app disables certain features and periodically nags to do the right thing and what not.
—
Now, let say I upgraded my box. This screws up the Installation ID and invalidates the license. The app recomputes new ID and Fingerprint and sends them to the licensing server.
This is where Fingerprint comes into play. The server takes old and new fingerprints, chops them into individual hashes and does a "fuzzy" match. If enough hashes match, we can assume an incremental software or hardware change on user's end and re-issue the license for new Installation ID.
If fingerprints are wildly different, then we make fuss and require contacting human tech support.
—
In conclusion, needless to say that the whole scheme relies on the validation code in the app being trustworthy. If someone pries the app open and replaces selected IFs with NOPs, then obviously no licensing will work.
The answer to that is that the app should implement self-consistency checks and deploy counter-measures when these checks fail. This is a separate and very large and interesting subject ... which I have no plans on covering here :-) A good starting point here would be fravia.org archives and then follow from there.
This is one of those things that is rather simple conceptually, but translates into a lot of actual petty work.
When the app is started, it looks at several system properties including the CPU model, Windows' MachineGuid value and others, hashes them together to produce an "Installation ID". For example -
$c!f2ZoPxOlnpLWraJZ3F1nm7JcA
The ID stays the same under normal computer use, but it changes with hardware upgrades or when the OS is reinstalled.
The app also generates an "Installation fingerprint" by hashing each of the properties separately and stringing them together. This yields a longer hash -
YQdAXlZR0FLXZ0KpYxlOk8sJ9Z$xtq19!rp6i3C6P5LiUc0J
x2Ynqe29i41qb!1w5wEzot!q0EH8SsLWfY6gvkyoZPtfn3bW
uYnSwItsG8Wh1FkQFWBCaNqdndeiBiDgkrTde$wJUib9wR58
GJ5bQU1Zr1iewtgyRhLWigHyzKDO3oYztT4hMw6MnP6voupb
J6An5QE4!etkMJx!qzOaYmioIOQ0GiQyapcPQeyYLCi4CN28
CEev0!FZFZz3aa$0dCmNnMRBohZ9z6EZH5NxROhokF8pXWVM
9y$bzQAA
x2Ynqe29i41qb!1w5wEzot!q0EH8SsLWfY6gvkyoZPtfn3bW
uYnSwItsG8Wh1FkQFWBCaNqdndeiBiDgkrTde$wJUib9wR58
GJ5bQU1Zr1iewtgyRhLWigHyzKDO3oYztT4hMw6MnP6voupb
J6An5QE4!etkMJx!qzOaYmioIOQ0GiQyapcPQeyYLCi4CN28
CEev0!FZFZz3aa$0dCmNnMRBohZ9z6EZH5NxROhokF8pXWVM
9y$bzQAA
Like ID, the fingerprint changes with hardware upgrades and OS reinstalls, but, unlike ID, it changes only in parts. This makes it suitable for detecting incremental changes to the user's computer.
Both ID and Fingerprint identify user's installation, the difference is that the ID is used on the app side and the Fingerprint - on the licensing server side.
Next, ID and fingerprint are submitted to the licensing server, either by the app itself or by a human through a website. When licensing for production use (as opposed to beta or evaluation), the request also includes a license code - a random token that is issued to the user in exchange for payment.
The server responds with a license, which is a short text file that binds together few pieces of information and seals them with an RSA signature of the licensing server.
License code: EVAL-8028-8316-0400
Installation: $c!f2ZoPxOlnpLWraJZ3F1nm7JcA
Expires on: 20/12/2013
J/8natmgEo1fFr0NSBtw3fsderaMsv6gwELBa7P4tsjtkPxO
Y/4z930IfpujPKwen6QB9Gb33q7lyXQNkAVcvMV5hlXwKNV+
eifzJUqyOE1GdmYd8Z4Io0OuFr2V4tD5vnDmYTFoOye4gd67
D40G/hoKgcvGtX82w6bjLTae+vzm7u6GwowTGrbEmqqJDPNI
4Qc6un+aGuOEWEY0X25DkPYnFy/VnPusWgLRma1eTCrmvOqD
T6WX27++Bwq2+YH1Aw+Br9xEQF6SZOBp9OR1Xp/K24vYSN52
MRlLjhrBIQXmJTTepLKAHEsnA3pvdinU6pzpYexygfjtciUm
VZlgvxfUICWI3pKBNRJtlOQKc9LIyr9jVXQ1jP4QVdY5JNZW
y+71b8cSEtWgRRkYVGZMyUXeJjU7AoW/PhDNWdKvDPBwBInQ
QuNqA1ncxjWMRdLYogR/VY41yKYs9cPaFTPFb4y1XLPiqVRK
vEOMMUvE1kArZS/IuC5yB7Jqd3mXlZaRMacwuhWVcd+fgbFj
wVbLJvRRePKEoP/Kk0CtDPIQNdYVxlEwdZi/S+cFMdfZLiWU
o8HFpT247WWEyElUaDahfCREgjPbs9Iw5pik82bm42phns4I
1936WVycSATBQZp3JC1KXX8iQeDBfOgwe3QQcnWIrpclI/oI
57Mknd6bxeI=
Installation: $c!f2ZoPxOlnpLWraJZ3F1nm7JcA
Expires on: 20/12/2013
J/8natmgEo1fFr0NSBtw3fsderaMsv6gwELBa7P4tsjtkPxO
Y/4z930IfpujPKwen6QB9Gb33q7lyXQNkAVcvMV5hlXwKNV+
eifzJUqyOE1GdmYd8Z4Io0OuFr2V4tD5vnDmYTFoOye4gd67
D40G/hoKgcvGtX82w6bjLTae+vzm7u6GwowTGrbEmqqJDPNI
4Qc6un+aGuOEWEY0X25DkPYnFy/VnPusWgLRma1eTCrmvOqD
T6WX27++Bwq2+YH1Aw+Br9xEQF6SZOBp9OR1Xp/K24vYSN52
MRlLjhrBIQXmJTTepLKAHEsnA3pvdinU6pzpYexygfjtciUm
VZlgvxfUICWI3pKBNRJtlOQKc9LIyr9jVXQ1jP4QVdY5JNZW
y+71b8cSEtWgRRkYVGZMyUXeJjU7AoW/PhDNWdKvDPBwBInQ
QuNqA1ncxjWMRdLYogR/VY41yKYs9cPaFTPFb4y1XLPiqVRK
vEOMMUvE1kArZS/IuC5yB7Jqd3mXlZaRMacwuhWVcd+fgbFj
wVbLJvRRePKEoP/Kk0CtDPIQNdYVxlEwdZi/S+cFMdfZLiWU
o8HFpT247WWEyElUaDahfCREgjPbs9Iw5pik82bm42phns4I
1936WVycSATBQZp3JC1KXX8iQeDBfOgwe3QQcnWIrpclI/oI
57Mknd6bxeI=
For beta and evaluation licenses the "license code" is generated automatically by the licensing server. The large block of gibberish at the bottom of the license is the RSA signature.
When the app starts, it loads the license and verifies it using public key of the licensing server that is embedded in the app.
If the license is valid, the app works as normal. If the license is missing or expired or doesn't match the installation ID, the app disables certain features and periodically nags to do the right thing and what not.
Now, let say I upgraded my box. This screws up the Installation ID and invalidates the license. The app recomputes new ID and Fingerprint and sends them to the licensing server.
This is where Fingerprint comes into play. The server takes old and new fingerprints, chops them into individual hashes and does a "fuzzy" match. If enough hashes match, we can assume an incremental software or hardware change on user's end and re-issue the license for new Installation ID.
If fingerprints are wildly different, then we make fuss and require contacting human tech support.
In conclusion, needless to say that the whole scheme relies on the validation code in the app being trustworthy. If someone pries the app open and replaces selected IFs with NOPs, then obviously no licensing will work.
The answer to that is that the app should implement self-consistency checks and deploy counter-measures when these checks fail. This is a separate and very large and interesting subject ... which I have no plans on covering here :-) A good starting point here would be fravia.org archives and then follow from there.
Reworking About window a bit to include license
information and associated trinkets (not shown).
This will probably need another iteration or two...
Basically the idea is to show back/forth arrows only on hover and make them pull next error to the spot where the cursor is. So to hop through errors you would just go click-click-click without moving the mouse.
This compliments well existing feature of jumping to the first error of the last run by clicking on "Xxx errors" in backup's main window entry.
As per previous post, this leaves a question of how one would get to the *nearest* error if none is currently visible and the answer is that one would simply use conventional scroll for that.
All in all, having played with this for a bit I think it's a good solution. If it doesn't seem like one to you, give it a try and it will grow on you.
Tying up few loose ends in the UI.
The above screenshot is the first option for navigating between the errors in the log pane -- see two red arrows next to the "x" in the pane header. The second option is this -
This time the next/previous arrows are in the error entry itself.
—
I like the second option better, because it's contextual and I think it's more obvious. However, first option is superior in functionality because it allows jumping to next or previous error when there's currently no error visible in the pane.
Still deciding...
The above screenshot is the first option for navigating between the errors in the log pane -- see two red arrows next to the "x" in the pane header. The second option is this -
This time the next/previous arrows are in the error entry itself.
I like the second option better, because it's contextual and I think it's more obvious. However, first option is superior in functionality because it allows jumping to next or previous error when there's currently no error visible in the pane.
Still deciding...
Tweaked the progress indicator a bit.
For one, replaced the fixed-width byte countdown with a simple "% out of [size]". The countdown didn't work out, it was just... odd.
Also made the % adjust the number of shown decimal digits depending on how fast the copying is going.
For fast copies there are no decimals. For slower ones it shows one, for slower yet - two, and for the slowest (like the one on the screenshot) it goes down to 3 decimal places.
The idea, obviously enough, to always show some movement, but try and avoid being too excessive with it.
For one, replaced the fixed-width byte countdown with a simple "% out of [size]". The countdown didn't work out, it was just... odd.
Also made the % adjust the number of shown decimal digits depending on how fast the copying is going.
For fast copies there are no decimals. For slower ones it shows one, for slower yet - two, and for the slowest (like the one on the screenshot) it goes down to 3 decimal places.
The idea, obviously enough, to always show some movement, but try and avoid being too excessive with it.
Added code to replicate folder timestamps. It's ON
by default as it takes only several seconds to go
through a hundred thousand folders, but it can be
disabled if needed
Final version of the update window. De-emphasized the
"View changes" button and tweaked the colors a bit.
Reworked the software update mechanism a little.
First, when the update server is processing an update check from the program, it now takes into account program's current version.
It uses this version to compile a consolidated list of all changes between program's version and the latest version available. It then uses this list to understand if there are any critical or important outstanding changes. This yields an importance indicator of the update, when one is available.
Secondly, both the list and the indicator are now sent back to the program.
—
On program's end, the indicator translates into the "Update is recommended" line. It is one three options with "optional" and "critical" being the other two.
"Recommended" is a routine update that adds this and fixes that. It would better be installed, but it's not the end of the world if it's not.
"Critical" is critical - "you'd be sorry if you don't install it" kind of update.
An example of "optional" update is a new version that updates program's French translation when the user is running an English version, i.e. an update that can be ignored altogether.
—
There's now also the "View changes" button that expands the window and shows a neatly formatted (RTF) version of the change list.
The biggest deal, of course, is that it's a consolidated list of all changes between the installed and the latest version, rather than a laundry list of all changes since the dawn of time. It makes the list infinitely more useful compared to the raw version.
—
Here's an older post on this that goes a bit deeper to explain the rationale behind these changes Version numbers don't matter.
First, when the update server is processing an update check from the program, it now takes into account program's current version.
It uses this version to compile a consolidated list of all changes between program's version and the latest version available. It then uses this list to understand if there are any critical or important outstanding changes. This yields an importance indicator of the update, when one is available.
Secondly, both the list and the indicator are now sent back to the program.
On program's end, the indicator translates into the "Update is recommended" line. It is one three options with "optional" and "critical" being the other two.
"Recommended" is a routine update that adds this and fixes that. It would better be installed, but it's not the end of the world if it's not.
"Critical" is critical - "you'd be sorry if you don't install it" kind of update.
An example of "optional" update is a new version that updates program's French translation when the user is running an English version, i.e. an update that can be ignored altogether.
There's now also the "View changes" button that expands the window and shows a neatly formatted (RTF) version of the change list.
The biggest deal, of course, is that it's a consolidated list of all changes between the installed and the latest version, rather than a laundry list of all changes since the dawn of time. It makes the list infinitely more useful compared to the raw version.
Here's an older post on this that goes a bit deeper to explain the rationale behind these changes Version numbers don't matter.
Adding support for running external command
before and after each backup.
Both commands have a configurable timeout value. Additionally, the pre-backup command can be tagged as critical, in which case bvckup checks the exit code and aborts the backup if the code doesn't match specified value.
The screenshot shows touching a TrueCrypt volume before the backup. This is needed, because TrueCrypt goes out of its way to preserve timestamps on its container files and bvckup sees such files as unmodified.
Both commands have a configurable timeout value. Additionally, the pre-backup command can be tagged as critical, in which case bvckup checks the exit code and aborts the backup if the code doesn't match specified value.
The screenshot shows touching a TrueCrypt volume before the backup. This is needed, because TrueCrypt goes out of its way to preserve timestamps on its container files and bvckup sees such files as unmodified.
Reworked the More Options section.
It has already grown quite tall and it still needs to take on few more configuration options, so expanding it in place as it's done for all other sections doesn't work. The window gets too big and simply doesn't fit on a desktop.
Instead, it pushes everything up and out of the window and floods it with its own content. As I add more config options to the section, it will also expand the window down a bit if needed. All this should help keep all configuration within single window and also keep window's height in check.
Took me a while to converge to this solution, but it is a pretty decent way to structure it I think.
It has already grown quite tall and it still needs to take on few more configuration options, so expanding it in place as it's done for all other sections doesn't work. The window gets too big and simply doesn't fit on a desktop.
Instead, it pushes everything up and out of the window and floods it with its own content. As I add more config options to the section, it will also expand the window down a bit if needed. All this should help keep all configuration within single window and also keep window's height in check.
Took me a while to converge to this solution, but it is a pretty decent way to structure it I think.
Second revision of the same thing.
Sketching up a dedicated device tracking
section. Not sure if it's worth it, still
have to try few things out...
Coming up - the reordering of list items.
Done with the inclusion/exclusion filters
The tree selector widget is a piece of art even though it doesn't look like one :) It is basically a front of an asynchronous parallel disk scanner that dynamically rearranges its scanning queue to first scan subdirectories that a user might be accessing next.
In other words, if you expand a directory, then the scanner will postpone scanning anything else and focus instead on all its subdirectories. So if you are making your way through a massively populated disk, it will try to pre-load what you are looking at and scan the rest on a lower priority.
Also, since the widget starts showing tree contents right away, it allows quickly navigating to and excluding a desired item without needing to wait for the entire scan to complete. Again, quite a bit of speed up with huge trees.
The tree selector widget is a piece of art even though it doesn't look like one :) It is basically a front of an asynchronous parallel disk scanner that dynamically rearranges its scanning queue to first scan subdirectories that a user might be accessing next.
In other words, if you expand a directory, then the scanner will postpone scanning anything else and focus instead on all its subdirectories. So if you are making your way through a massively populated disk, it will try to pre-load what you are looking at and scan the rest on a lower priority.
Also, since the widget starts showing tree contents right away, it allows quickly navigating to and excluding a desired item without needing to wait for the entire scan to complete. Again, quite a bit of speed up with huge trees.
Advanced filter configuration. The pattern
matching is all brand new and much improved
over the disaster that it was in Bvckup 1.
It also adds an option of matching on item
attributes, both set and cleared.
Finishing up the dialog for configuring
inclusion/exclusion filters.
In inclusive mode (pictured) the tree view tags excluded items with 'x'. In exclusive mode, when "Start with an empty list" selected, the tree view tags included items with checkmarks.
Currently leaning towards Option 1, and not yet decided if to strike-out excluded items or not.
In inclusive mode (pictured) the tree view tags excluded items with 'x'. In exclusive mode, when "Start with an empty list" selected, the tree view tags included items with checkmarks.
Currently leaning towards Option 1, and not yet decided if to strike-out excluded items or not.
Added per-phase progress bars. Currently
used for scanning and execution phases.
Before/After of the backup config window
as of Beta Release 20.
Revising the backup config dialog to accommodate
a description field.
Revising the main view. Description and the
progress bars are back in.
Well into the beta, just dotting the i's and
crossing the t's.
One of the outstanding features is the archival of deleted items and this is the supporting UI for it.
Now deciding between a compact format of the config widget or a verbose one. Compact format has a single descriptive field that is re-used to show the explanation of a currently selected option. Verbose format has descriptions for each option visible at all times - it is likely easier on first time users, but it is also a waste of screen space on all subsequent uses.
One of the outstanding features is the archival of deleted items and this is the supporting UI for it.
Now deciding between a compact format of the config widget or a verbose one. Compact format has a single descriptive field that is re-used to show the explanation of a currently selected option. Verbose format has descriptions for each option visible at all times - it is likely easier on first time users, but it is also a waste of screen space on all subsequent uses.
What can I say?
Beta. Is. Good. To. Go!!!
Beta. Is. Good. To. Go!!!
For very large files (say, 256 Gigs), there was a clear lack of UI acknowledgement of the backup progress. So the completed percentage got a couple of point digits and a byte countdown for the file.
EULA
Trying to keep it short and sensible.
Trying to keep it short and sensible.
Added the first draft of the Miscellanea section.
Also renamed More Options button to
Rabbit Hole, because that's essentially
where it leads to :)
( the original version is here )
Significantly improved performance through multi-core, parallel hashing, asynchronous I/O and more aggressive caching of the state data.
Strengthened the change detection algorithm through addition of the whole file hash. When all individual block hashes come back unchanged, the algorithm now uses the file hash to confirm that the file has indeed remained unchanged.
Combined with the original use of two separate hashes per data block, this eliminates the need for a precautionary full-file sync every N copies.
Improved handling of cancelled and aborted delta copies by introducing support for partial updates.
The algorithm now remembers how far along the file it managed to proceed and stashes this information for the next run.
When the copying is cancelled partway through, the file is marked as out-of-sync, but the delta state is updated to capture what has been already done. This comes very handy and speeds things up when updating very large files.
Improved handling of large file counts.
Version 1 maintained an in-memory index of all delta copied files and, surprisingly, this didn't scale well. It also kept all delta state information in a single folder, which too led to some pain and suffering when the file count grew large.
New version eliminates both these problems.
Lastly, all copier parameters - block sizes, hash algorithms, cache buffer sizes, threading parameters, etc. - are now configurable on a per-job basis.
All in all - THIS IS IT.
This is how a simple and efficient delta copying should be done. Can't wait to release it to the real world.
Shadow copying is in,
for those pesky locked Outlook mailboxes and other "in use" files.
There are three big things left to weave into the program - file system monitoring, new delta copier and software updates - and the beta is good to go. A matter of few days now.
There are three big things left to weave into the program - file system monitoring, new delta copier and software updates - and the beta is good to go. A matter of few days now.
If your message box includes the "Don't ask again"
option, just make sure to disable all but one
button, when it checked.
Oddly enough, I don't know a single app that does that.
Oddly enough, I don't know a single app that does that.
Feeling creative? Make your own version, save as a PNG, drop it in a config directory and - voilà - your systray looks just like you want it to.
There are two sets of icons on the sprite sheet.
The first three columns is the XP style, which
is a bit puffy and matches well older Windows
styling. The last three columns is a flat version
that goes better with native W7 tray icon style.
Each icon comes in 3 sizes - 16x16, 20x20, 24x24. This is needed to accommodate 3 common DPI levels, see this earlier entry for details.
Each icon comes in 3 sizes - 16x16, 20x20, 24x24. This is needed to accommodate 3 common DPI levels, see this earlier entry for details.
"Message pending" indication, whereby a message
can be either an error, a warning or a notice
and these are differentiated with the ! color.
I realize it's not color-blind friendly,
but the goal is to attract the attention and
make user bring the app window up to read the
message --(therefore)--> reflecting the message
type via an icon level is not exactly critical.
W7-ish version as per
a
this hint on Twitter. Not bad at all.
Working on systray icons...
Revised Preferences window a bit.
Added basic control over backup scheduling and switched from "close to systray" to "minimize to systray" - the (x) at the top right corner will now always act as an Exit, just as it is supposed to.
Added basic control over backup scheduling and switched from "close to systray" to "minimize to systray" - the (x) at the top right corner will now always act as an Exit, just as it is supposed to.
More of the UI work.
Custom pop-out button that stays nice, flat and out of the way when idle, but acts like a regular button otherwise. Not exactly a UI innovation, but it was nice to be able to use it in a real project :)
Custom pop-out button that stays nice, flat and out of the way when idle, but acts like a regular button otherwise. Not exactly a UI innovation, but it was nice to be able to use it in a real project :)
Took out the engine mode switch from the Preferences
and into a dialog of its own. Still need to understand
if it's worth mentioning that there's an engine per se
or just stick to more amorphous "application mode"...
Looked at few options for splitter styling. First
one is final. Tentatively though... heh.
Done with the split-view mode for the main window.
It's one of those things that everyone takes for
granted and that, in reality, maps to quite a bit
of work if done right.
For example, resizing window by the top edge should resize the top pane and leave the bottom part alone.
Likewise, resizing by the bottom edge should expand/compress the bottom pane, but not the top one.
However, if we are compressing window by the top edge and the top pane is at its minimal height, then we should the bottom pane starting to compress.
All good so far, but, say, we change our mind and start expanding the window back (without releasing the mouse button).
We should see the bottom pane brought back to its original size first, followed by the expansion of the top part... even though we are, technically, dragging by the top edge.
This sort of UX detail is something that few people will notice if it's done right, because it feels natural and aligns well with expected behavior. But cut some corners - and you bet everyone will notice.
For example, resizing window by the top edge should resize the top pane and leave the bottom part alone.
Likewise, resizing by the bottom edge should expand/compress the bottom pane, but not the top one.
However, if we are compressing window by the top edge and the top pane is at its minimal height, then we should the bottom pane starting to compress.
All good so far, but, say, we change our mind and start expanding the window back (without releasing the mouse button).
We should see the bottom pane brought back to its original size first, followed by the expansion of the top part... even though we are, technically, dragging by the top edge.
This sort of UX detail is something that few people will notice if it's done right, because it feels natural and aligns well with expected behavior. But cut some corners - and you bet everyone will notice.
Going through styling options for the expand/collapse buttons.
Presently leaning towards C or D, because I'm considering making the text selectable with the mouse for easy copy- pasting. For that to work the buttons need to be as unobtrusive as possible.
Still WiP though...
Presently leaning towards C or D, because I'm considering making the text selectable with the mouse for easy copy- pasting. For that to work the buttons need to be as unobtrusive as possible.
Still WiP though...
Working on the backup log viewer.
—
Basically the idea for the log is to support multiple detail levels, each capturing progressively more context that its parent.
With all details collapsed, the log gives a quick overview of when the job was running and if it completed OK or with errors.
Expanding details allows for a drill-down - to look at the details such as the file and byte counts, the timing and performance data and the details of any errors that have occured.
I'm still going to shuffle things around the window, e.g. push the -/+ buttons to the right, tweak the styling too, but the general hierarchical format will stay. It is a very convenient way of organizing the log info.
Basically the idea for the log is to support multiple detail levels, each capturing progressively more context that its parent.
With all details collapsed, the log gives a quick overview of when the job was running and if it completed OK or with errors.
Expanding details allows for a drill-down - to look at the details such as the file and byte counts, the timing and performance data and the details of any errors that have occured.
I'm still going to shuffle things around the window, e.g. push the -/+ buttons to the right, tweak the styling too, but the general hierarchical format will stay. It is a very convenient way of organizing the log info.
Finished a piece of code that determines
effective timestamp resolution of a file
system.
—
Every file system keeps track of when a file was created and when it was last modified. These timestamps are the natural properties of a file and they are readily available for inspection in the Windows Explorer or any file manager of your choice.
The less obvious aspect of timestamping is that timestamps have resolution. For example, FAT tracks file modification with a measly 2 second precision. In comparison, NTFS uses the 100 ns (nano-second) precision.—
It is very typical for a backup software to rely on timestamps to determine if a file has been modified and requires copying. But if the source file sits on an NTFS volume and the backup goes onto a FAT disk, then comparing timestamp directly simply won't work, because FAT timestamp will be rounded-up to a 2-second mark.
In other words, the timestamp granularity needs to be taken into an account when comparing the timestamps. The question is how to determine what it is exactly.—
First of all, the granularity for creation and modification times can be different. FAT has them at 10 ms and 2 s respectively, but NTFS has them both at 100 ns.
To complicate matters, there appears to be NAS devices that look like NTFS boxes, but with the granularity that is not 100 ns.
So what I did is added code to probe the file system and determine effective resolution for both timestamps. This involves dropping a temporary file and then trying to set its timestamps to this, that and 3rd and see what they end up at. From that it's possible to deduce the resolution.
In cases when such probing fails, the app falls back to guessing resolution by the file system name. It is also possible to override the resolution values from the config, just in case.
Every file system keeps track of when a file was created and when it was last modified. These timestamps are the natural properties of a file and they are readily available for inspection in the Windows Explorer or any file manager of your choice.
The less obvious aspect of timestamping is that timestamps have resolution. For example, FAT tracks file modification with a measly 2 second precision. In comparison, NTFS uses the 100 ns (nano-second) precision.
It is very typical for a backup software to rely on timestamps to determine if a file has been modified and requires copying. But if the source file sits on an NTFS volume and the backup goes onto a FAT disk, then comparing timestamp directly simply won't work, because FAT timestamp will be rounded-up to a 2-second mark.
In other words, the timestamp granularity needs to be taken into an account when comparing the timestamps. The question is how to determine what it is exactly.
First of all, the granularity for creation and modification times can be different. FAT has them at 10 ms and 2 s respectively, but NTFS has them both at 100 ns.
To complicate matters, there appears to be NAS devices that look like NTFS boxes, but with the granularity that is not 100 ns.
So what I did is added code to probe the file system and determine effective resolution for both timestamps. This involves dropping a temporary file and then trying to set its timestamps to this, that and 3rd and see what they end up at. From that it's possible to deduce the resolution.
In cases when such probing fails, the app falls back to guessing resolution by the file system name. It is also possible to override the resolution values from the config, just in case.
Done with the backup scheduler. Comes with a
couple of neat improvements over v1.
—
The Go and Stop buttons work almost exactly as in v1. If the backup is running, click Stop once to Pause it, click again to Cancel, click third time to disable the job. If the job is disabled, click once to enable it, click again to manually start it. And so on, you get the idea.
—
First improvement over v1 are the Go! and Stop! commands that are activated by holding Ctrl down when clicking on the toolbar buttons.
Go! moves the job straight into the running state.
It's the "Run it. NOW!" command.
Stop! stops and disables the job straight away.
It's the "Stop it. NOW!" command.
—
Second improvement is that multiple jobs can now be run in parallel. Generally, this has a negative impact on the overall performance, but there are certain cases when this is desirable. One example would be running a smaller backup in parallel with a much larger and slower one.
There are three ways to do this:
—
PS. I considered adding something more elaborate, e.g. allowing assigning jobs to "run groups" and then allowing only one active job per group, but that seemed like an overkill, so I dropped that.
The Go and Stop buttons work almost exactly as in v1. If the backup is running, click Stop once to Pause it, click again to Cancel, click third time to disable the job. If the job is disabled, click once to enable it, click again to manually start it. And so on, you get the idea.
First improvement over v1 are the Go! and Stop! commands that are activated by holding Ctrl down when clicking on the toolbar buttons.
Go! moves the job straight into the running state.
It's the "Run it. NOW!" command.
Stop! stops and disables the job straight away.
It's the "Stop it. NOW!" command.
Second improvement is that multiple jobs can now be run in parallel. Generally, this has a negative impact on the overall performance, but there are certain cases when this is desirable. One example would be running a smaller backup in parallel with a much larger and slower one.
There are three ways to do this:
- With the Go! command.
- With a per-backup setting that makes the job always run right away, even if there's another job is running.
- With an app-level setting that does the same, but for all jobs.
PS. I considered adding something more elaborate, e.g. allowing assigning jobs to "run groups" and then allowing only one active job per group, but that seemed like an overkill, so I dropped that.
Another pass over the main window chrome and
layout. The "backup jobs" section is just a
dummy header ... and it is confusing, I think.
† This is an app screenshot, not a sketch.
† This is an app screenshot, not a sketch.
Exploring options for the main window format.
—
Specifically, I want to get rid of the V1's Summary panel, trim the information contained in it a bit and fold it directly into the main window items.
In conjunction with this I'm also considering having two modes for displaying backup jobs on the list - "terse" one-liners and "extended" multi-liners. What I still need to figure out if this should be a per-item preference or an app-wide setting and how one would switch between these display modes.
Work in progress, obviously. Any thoughts - let me know.
Specifically, I want to get rid of the V1's Summary panel, trim the information contained in it a bit and fold it directly into the main window items.
In conjunction with this I'm also considering having two modes for displaying backup jobs on the list - "terse" one-liners and "extended" multi-liners. What I still need to figure out if this should be a per-item preference or an app-wide setting and how one would switch between these display modes.
Work in progress, obviously. Any thoughts - let me know.
Done with a little animation that and switches
main window between a "welcome screen" and the
regular backup list view.
The animation and fading effects can be disabled, there's a setting for that.
—
Windows comes with a standard toolbar control, but it's an overkill if you need just a couple of buttons at the top of the window.
It basically takes longer to figure out how to use and customize it compared to creating a basic window and sticking a couple of custom buttons on it.
Had I have a need for toolbar ribbons, docking, on-the-fly customization, etc., then ToolbarWindow32 makes sense. But if it's something simple, then there are faster options.
The animation and fading effects can be disabled, there's a setting for that.
Windows comes with a standard toolbar control, but it's an overkill if you need just a couple of buttons at the top of the window.
It basically takes longer to figure out how to use and customize it compared to creating a basic window and sticking a couple of custom buttons on it.
Had I have a need for toolbar ribbons, docking, on-the-fly customization, etc., then ToolbarWindow32 makes sense. But if it's something simple, then there are faster options.
Meet the toolbar.
—
To explain a bit what you see in this masterpiece :) -
The idea is to try and keep the menu/toolbar area simple and flat, but still provide some sort of a visual clue that Go and Start are clickables.
Here are three options compared back to back to make the differences a bit more obvious -
—
Here are few options that were leading up to the above one:
iOS says Hi
Also, considered disposing off the menu bar and folding whatever few items it has into the drop-down menu. But this just looks weird and too alien on Windows, the app needs a menu bar.
Dropped the idea.
To explain a bit what you see in this masterpiece :) -
The idea is to try and keep the menu/toolbar area simple and flat, but still provide some sort of a visual clue that Go and Start are clickables.
Here are three options compared back to back to make the differences a bit more obvious -
Here are few options that were leading up to the above one:
Also, considered disposing off the menu bar and folding whatever few items it has into the drop-down menu. But this just looks weird and too alien on Windows, the app needs a menu bar.
Dropped the idea.
The look of the main window with no backups
configured. Tried using a stock button, but
it looks heavy and dull. Hopefully, this is
a bit more refined.
Trying to understand how to indicate if
a selected path points at a removable storage.
The app tracks storage devices being plugged into the computer and it adjusts backup paths if a known device re-appears under a different drive name.
I can, of course, not bother with this at all and simply spit out a notification when (and if) the drive change is detected, but then I think it's an important functionality and it'd better be mentioned during the setup.
The app tracks storage devices being plugged into the computer and it adjusts backup paths if a known device re-appears under a different drive name.
I can, of course, not bother with this at all and simply spit out a notification when (and if) the drive change is detected, but then I think it's an important functionality and it'd better be mentioned during the setup.
"Work-in-progress" buttons for beta releases.
Correct your input.
† Idea is courtesy of the OS X login screen, I think. Though I did experiment with concise error indication quite a bit.
† Idea is courtesy of the OS X login screen, I think. Though I did experiment with concise error indication quite a bit.
Wrapping up the Periodic Backup configuration
window.
Unlike the v1 that always stored the backup interval as a number of seconds, the v2 adds a time unit specifier.
This simplifies quite a few things in the UI and yet it keeps the configuration flexible enough to support 1.5 hour backups by allowing to specify a 90 minute interval.
—
On an unrelated note, killed few hours trying to utilize native Windows controls for the interval configuration. The amount of work required to make them look reasonably good is as depressing as it is remarkable.
For example, the editbox control doesn't natively support vertically centering the input text. It just does not. It is meant to look like this -
Want "120" aligned to the label on the left? Sure thing, that'd be 2 hours of plowing through documentation only to realize that it must be... drum-roll... a multiline control that you will then need to manually pad the right amount at the top to push the line to the middle.
—
But that's peanuts compared to trying to beat some sense into the dropdown box. Long story, but making a box taller and keeping the line centered requires taking over the control drawing functions, which in presence of Windows themes is a delightfully unpleasant experience.
In any case, turning the top control line into the bottom one -
costs exactly a day of work, while it really shouldn't. Welcome to Windows.
Unlike the v1 that always stored the backup interval as a number of seconds, the v2 adds a time unit specifier.
This simplifies quite a few things in the UI and yet it keeps the configuration flexible enough to support 1.5 hour backups by allowing to specify a 90 minute interval.
On an unrelated note, killed few hours trying to utilize native Windows controls for the interval configuration. The amount of work required to make them look reasonably good is as depressing as it is remarkable.
For example, the editbox control doesn't natively support vertically centering the input text. It just does not. It is meant to look like this -
Want "120" aligned to the label on the left? Sure thing, that'd be 2 hours of plowing through documentation only to realize that it must be... drum-roll... a multiline control that you will then need to manually pad the right amount at the top to push the line to the middle.
But that's peanuts compared to trying to beat some sense into the dropdown box. Long story, but making a box taller and keeping the line centered requires taking over the control drawing functions, which in presence of Windows themes is a delightfully unpleasant experience.
In any case, turning the top control line into the bottom one -
costs exactly a day of work, while it really shouldn't. Welcome to Windows.
Finalized the design of inline hint/tip
widget for input controls.
Initially, it was meant for the config window that hosts periodic backup settings (as seen in a couple of posts below), but having talked to others and slept on the issue I think that the natural language input (i.e. requiring to enter "1 week" or "15 minutes") is not the best option for this case.
An alternative design is coming up in a bit.
Initially, it was meant for the config window that hosts periodic backup settings (as seen in a couple of posts below), but having talked to others and slept on the issue I think that the natural language input (i.e. requiring to enter "1 week" or "15 minutes") is not the best option for this case.
An alternative design is coming up in a bit.
Coded the previous sketch, played with few
parameters and it is starting to get there
... I think. Will make another pass over this
tomorrow and it should be good to go.
For what it's worth I have also gave a try
to the standard
Tooltip Control. In fact,
that's where I started.
—
A regular tooltip looks like this -
It adds another layer to the windows stack, breaks the visual flow of the layout and still manages to tuck itself under the mouse cursor.
There's also a balloon version of the same, which looks like this -
It is marginally better, but just look at those lovely rounded cornersi. Eat that Apple. Real artists ship.
—
tl;dr - the standard Tooltip Control is not very good, so I had to come up with a custom replacement.
A regular tooltip looks like this -
It adds another layer to the windows stack, breaks the visual flow of the layout and still manages to tuck itself under the mouse cursor.
There's also a balloon version of the same, which looks like this -
It is marginally better, but just look at those lovely rounded cornersi. Eat that Apple. Real artists ship.
tl;dr - the standard Tooltip Control is not very good, so I had to come up with a custom replacement.
Not sure about this one, seems a bit
clumsy and a bit too custom.
This is basically a way to explain what is expected as an input for a particular configuration value. There's gotta be a cleaner way of delivering the same info...
This is basically a way to explain what is expected as an input for a particular configuration value. There's gotta be a cleaner way of delivering the same info...
Working through the supporting code for the Settings window.
Done with collapsing/expanding mechanics and respective fading and moving animations. Still need to revise the copy for the explanatory bits and massage in the option for archiving deleted files.
—
Not to pat myself on the back, but I think it came out looking pretty decent.
That's not to say there's no way to turn off all this fancy styling and animations, because, you bet your favourite monochrome flat design, there is.
Done with collapsing/expanding mechanics and respective fading and moving animations. Still need to revise the copy for the explanatory bits and massage in the option for archiving deleted files.
Not to pat myself on the back, but I think it came out looking pretty decent.
That's not to say there's no way to turn off all this fancy styling and animations, because, you bet your favourite monochrome flat design, there is.
The Backup Settings window. Final ...for now :)
Another take on Backup Settings window, 4 styling
variations.
Further compacted vertically, but now it appears to be somewhat cluttered and lacking comfortable vertical rhythm (to borrow a term from the typography).
Further compacted vertically, but now it appears to be somewhat cluttered and lacking comfortable vertical rhythm (to borrow a term from the typography).
Working on the Backup Settings window.
I think the original design from the version 1 got
many things right.
It provided a good at-a-glance overview of the backup configuration, split settings logically and tucked away the minute details. It did have one problem - it was tall. So tall, in fact, that I got several reports that it didn't fit on screen :-|
—
V2 adds several new options to the backup config, so this doesn't make things any easier. After looking at few options, including paged, tabbed and two-column, screenful layouts, I circled back to the v1 layout.
To compress it vertically I've done two things - got rid of the group boxes (frames) and collapsed radio button groups into a single line with the current selection.
This gives us the middle screenshot (pic) that shows the config window in the "overview" state.
—
Clicking on a "Change ↓" button expands the radio button group and allows changing the setting without leaving the page. Latter is pretty nice to have, because ... well ... because it's simply faster and more convenient.
Expanding a section gives the screenshot on the right (pic). To collapse the section, there's a little button in the section header on the right hand side. I'm not 100% convinced it's really needed nor that it's the best solution, but that's how it is at the moment.
—
I forgot the "how to handle deleted files" section, but it will be in the beta. Not to worry.
It provided a good at-a-glance overview of the backup configuration, split settings logically and tucked away the minute details. It did have one problem - it was tall. So tall, in fact, that I got several reports that it didn't fit on screen :-|
V2 adds several new options to the backup config, so this doesn't make things any easier. After looking at few options, including paged, tabbed and two-column, screenful layouts, I circled back to the v1 layout.
To compress it vertically I've done two things - got rid of the group boxes (frames) and collapsed radio button groups into a single line with the current selection.
This gives us the middle screenshot (pic) that shows the config window in the "overview" state.
Clicking on a "Change ↓" button expands the radio button group and allows changing the setting without leaving the page. Latter is pretty nice to have, because ... well ... because it's simply faster and more convenient.
Expanding a section gives the screenshot on the right (pic). To collapse the section, there's a little button in the section header on the right hand side. I'm not 100% convinced it's really needed nor that it's the best solution, but that's how it is at the moment.
I forgot the "how to handle deleted files" section, but it will be in the beta. Not to worry.
Meet the Preferences.
The Backup Engine section - as simple as it looks,
it is fronting a couple of months worth of work
and it's easily one of the most complex part of
the v2. This is what detaches the user interface
from the guts of the program and lets latter run
as a separate process.
—
The Maintenance section - I figured that with the backup being an important piece of software, very few users would be interested in fully automatic updates. Furthermore, quite a few people like to retain even tighter control over the software and, for example, do not like it when the software talks to the Internet whenever it wants. Hence the new option -
Don't check, but remind to check for updates
It's basically a kitchen timer that goes "Ding!" every week or two (configurable) and prompts you to let the app check for a new version. We'll see how it goes, but I think it's a sensible compromise between the privacy and functionality.
—
The More button transforms the window into a comprehensive list of all available options, close to two dozens now and still counting. The idea however is to keep the defaults reasonable so that there won't be much need to venture into this part of the app.
—
Lastly, note how the Apply/Close buttons change when the settings are modified. I have no idea why this is not a standard behavior, because it really should be.
The Maintenance section - I figured that with the backup being an important piece of software, very few users would be interested in fully automatic updates. Furthermore, quite a few people like to retain even tighter control over the software and, for example, do not like it when the software talks to the Internet whenever it wants. Hence the new option -
It's basically a kitchen timer that goes "Ding!" every week or two (configurable) and prompts you to let the app check for a new version. We'll see how it goes, but I think it's a sensible compromise between the privacy and functionality.
The More button transforms the window into a comprehensive list of all available options, close to two dozens now and still counting. The idea however is to keep the defaults reasonable so that there won't be much need to venture into this part of the app.
Lastly, note how the Apply/Close buttons change when the settings are modified. I have no idea why this is not a standard behavior, because it really should be.
A message bar is a unobtrusive way to display
one-liner messages that would've otherwise
required poping up a message box.
When a program detects that it was just restarted
with Admin privileges, it has a limited number of
options for communicating this joyous news to its
user.
#1 - A message box. Bluntly put - I hate them. They are always too sudden, they steal your attention and stuff whatever they want to say down your throat. They ... lack manners.
#2 - A system tray notification. This is less intrusive, but perhaps it's too out of the way. Also, the systray is a shared area so the program is always competing with other apps for the same screen real estate.
#3 - Something else that is right in your field of view if you are looking at the app, but that doesn't get in a way of whatever you are doing at the moment.
—
Cue in the message bar, now an integral part of the Bvckup UI. It is used for displaying various status notifications right after the launch, after switching the engine to/from the system service mode, when discovering software updates and few other things.
I have also tried the self-closing version of the bar that dismisses itself on a timeout. Like so -
and so -
but having lived with this for a couple of days, it now seems like a solution in a search of a problem. So I shelved it for now. If there's any a real need for this, it'd be very easy to add later on.
#1 - A message box. Bluntly put - I hate them. They are always too sudden, they steal your attention and stuff whatever they want to say down your throat. They ... lack manners.
#2 - A system tray notification. This is less intrusive, but perhaps it's too out of the way. Also, the systray is a shared area so the program is always competing with other apps for the same screen real estate.
#3 - Something else that is right in your field of view if you are looking at the app, but that doesn't get in a way of whatever you are doing at the moment.
Cue in the message bar, now an integral part of the Bvckup UI. It is used for displaying various status notifications right after the launch, after switching the engine to/from the system service mode, when discovering software updates and few other things.
I have also tried the self-closing version of the bar that dismisses itself on a timeout. Like so -
and so -
but having lived with this for a couple of days, it now seems like a solution in a search of a problem. So I shelved it for now. If there's any a real need for this, it'd be very easy to add later on.
Nailed down the installer flow. A couple of
things of note here.
The most unconventional part of the setup
is how it handles the configuration of the
installation parameters.
The configuration page (see Page 3) is very bare and focuses on how the program is going to be used instead of being a mixed bag of the minute details of the setup.
The target location is made into a secondary setting with its value controlled by the usage preference. There's no field for the Start Menu group name. There's no ever popular "Run the program after the setup completes" option either. Instead, exiting the setup automatically runs the program (see Page 4), unless the installer window is closed through an (x) at the top.
This leads to a simpler and more intuitive setup flow... or at least that's the idea :)
—
The installer supports both privileged (administrative) and unprivileged (regular user) installations.
If you are logged in as a regular user and have no rights to modify C:\Program Files, there's no reason why you shouldn't still be able to install a software, just for yourself.
While other installers routinely ask if to install the software for everyone or just for one user, it's a superficial question. It merely controls if the Start Menu group is created under All Users or your personal profile. The program files still go into a system directory and this requires administrative privileges.
Bvckup implements true system- and user- deployment, which is akin to deploying the program in /usr/bin or ~/bin in Unix terms.
In fact, on a multi-user system it is perfectly possible to install one instance of Bvckup for each user and they will all happily co-exist with each other.
The configuration page (see Page 3) is very bare and focuses on how the program is going to be used instead of being a mixed bag of the minute details of the setup.
The target location is made into a secondary setting with its value controlled by the usage preference. There's no field for the Start Menu group name. There's no ever popular "Run the program after the setup completes" option either. Instead, exiting the setup automatically runs the program (see Page 4), unless the installer window is closed through an (x) at the top.
This leads to a simpler and more intuitive setup flow... or at least that's the idea :)
The installer supports both privileged (administrative) and unprivileged (regular user) installations.
If you are logged in as a regular user and have no rights to modify C:\Program Files, there's no reason why you shouldn't still be able to install a software, just for yourself.
While other installers routinely ask if to install the software for everyone or just for one user, it's a superficial question. It merely controls if the Start Menu group is created under All Users or your personal profile. The program files still go into a system directory and this requires administrative privileges.
Bvckup implements true system- and user- deployment, which is akin to deploying the program in /usr/bin or ~/bin in Unix terms.
In fact, on a multi-user system it is perfectly possible to install one instance of Bvckup for each user and they will all happily co-exist with each other.
The first page of the installer.
Working through the details of the update mechanism.
This is one of more difficult parts of the application, even though on the surface it looks uncomplicated. If we run the installer and it detects that the app is already installed, it should just update the installation. How hard could it be?
Let's see.
This is one of more difficult parts of the application, even though on the surface it looks uncomplicated. If we run the installer and it detects that the app is already installed, it should just update the installation. How hard could it be?
Let's see.
If the program is not running, then it's indeed a simple
matter of replacing the program file with a newer version.
The fun starts if the program is running.
On Windows, running a program locks its file, preventing any changes from being made to it. This is not required per se and there are operating systems that don't do this. There are also custom PE loaders for Windows that can take an executable and create a running process out of it without any locking fuss. For example, bo2k could do that back in 1999. See its source code for details if interested.
—
To complicate matters, there might be more than one process running off the same executable. For example, if several users are logged into a Windows box and each runs a copy of a program.
Therefore, the first order of business for any installer is to shut down all running instances of the program. A well-behaved installer should also restart all these instances after it's done with updating.
—
The shutting down part is not a big deal. There's more than one way to do this and all of them are fairly straight-forward.
It's the restarting that is contrived.
Not only program instances may be running under different user accounts, they may also be running with custom command-line arguments, as Administrators or system services. Restarting this cornucopia of awesomeness is not an envious task.
—
Here's how Bvckup works around this mess.
Every Bvckup executable has a corresponding named event in the Global namespace. The event is created by the first instance of the program when it's launched, and it is opened by all other instances.
The name of the event includes the file ID of the executable file. This groups running instances by the file they are using, because, after all, the installer is interested in a specific executable and nothing else.
For example, the following is an event name used by the Bvckup installation on my development box -
Global\Bvckup2.e65a96a8001f000000013cba.Update
—
With this in place, the installer starts off by signaling the event.
In response, a running program makes a temporary copy of its executable file and starts it in a special "relauncher" mode. And then it exits.
The "relauncher" process opens the "update" event and simply spins there, waiting for it to become unsignaled.
Meanwhile the installer waits for the program file to become unlocked. Once all running instances replace themselves with the relaunchers, the executable frees up and the installer proceeds with the update. It then clears the "update" event and exits.
This releases the relaunchers, they restart program instances and - ta-da - the update is completed.
—
Techincally, instead of adding a special "relauncher" mode to the program it's possible just to have a small standalone relauncher program. It comes down to a matter of taste. Some people don't like to spawn from temporary executables, and I like to keep my file count to a bare minimum. To each his own.
On Windows, running a program locks its file, preventing any changes from being made to it. This is not required per se and there are operating systems that don't do this. There are also custom PE loaders for Windows that can take an executable and create a running process out of it without any locking fuss. For example, bo2k could do that back in 1999. See its source code for details if interested.
To complicate matters, there might be more than one process running off the same executable. For example, if several users are logged into a Windows box and each runs a copy of a program.
Therefore, the first order of business for any installer is to shut down all running instances of the program. A well-behaved installer should also restart all these instances after it's done with updating.
The shutting down part is not a big deal. There's more than one way to do this and all of them are fairly straight-forward.
It's the restarting that is contrived.
Not only program instances may be running under different user accounts, they may also be running with custom command-line arguments, as Administrators or system services. Restarting this cornucopia of awesomeness is not an envious task.
Here's how Bvckup works around this mess.
Every Bvckup executable has a corresponding named event in the Global namespace. The event is created by the first instance of the program when it's launched, and it is opened by all other instances.
The name of the event includes the file ID of the executable file. This groups running instances by the file they are using, because, after all, the installer is interested in a specific executable and nothing else.
For example, the following is an event name used by the Bvckup installation on my development box -
With this in place, the installer starts off by signaling the event.
In response, a running program makes a temporary copy of its executable file and starts it in a special "relauncher" mode. And then it exits.
The "relauncher" process opens the "update" event and simply spins there, waiting for it to become unsignaled.
Meanwhile the installer waits for the program file to become unlocked. Once all running instances replace themselves with the relaunchers, the executable frees up and the installer proceeds with the update. It then clears the "update" event and exits.
This releases the relaunchers, they restart program instances and - ta-da - the update is completed.
Techincally, instead of adding a special "relauncher" mode to the program it's possible just to have a small standalone relauncher program. It comes down to a matter of taste. Some people don't like to spawn from temporary executables, and I like to keep my file count to a bare minimum. To each his own.
Dotting all i's and crossing all t's, of which there's
plenty now that there's an optional system service
that just begs to be stopped, restarted and deleted
from under app's feet.
Doh... I missed the disabling of the service. Will add it tomorrow.
Doh... I missed the disabling of the service. Will add it tomorrow.
Redesigned.
Vector icons for the website.
This is my first time not using pixel images to
decorate a website. Instead, all icons are saved
in the
SVG
format and converted to a webfont, which is then
used to style regular HTML characters into icons.
For example, here's a font with the above icon set - VectorIcons.ttf
The advent of webapps like IcoMoon and Fontello not only made the process of building @font-face kits an absolute no-brainer, it also gave an easy access to hundreds of existing vector icons, many of which are free and of production quality.
When coupled with now virtually universal support for CSS text-shadow, box-shadow and gradients, this almost completely eliminates the need for raster form of smaller decorative and nuancing elements.
This is a very big deal and a great time-saver.
For example, here's a font with the above icon set - VectorIcons.ttf
The advent of webapps like IcoMoon and Fontello not only made the process of building @font-face kits an absolute no-brainer, it also gave an easy access to hundreds of existing vector icons, many of which are free and of production quality.
When coupled with now virtually universal support for CSS text-shadow, box-shadow and gradients, this almost completely eliminates the need for raster form of smaller decorative and nuancing elements.
This is a very big deal and a great time-saver.
Re-finalized the sign-up form styling and mechanics.
Few styling options for an input control.
Leaning towards top right or bottom right,
but still need to see what works best in
context.
More or less done with the Features section
on the new website.
Probably missed few things, but that's easy to add later on. The biggest time sink was trying to inject some visual hierarchy into the list, but without going against overall minimalist look and feel of the site.
Might still revise it after sleeping on it for a couple of days :) Won't be the first time.
Probably missed few things, but that's easy to add later on. The biggest time sink was trying to inject some visual hierarchy into the list, but without going against overall minimalist look and feel of the site.
Might still revise it after sleeping on it for a couple of days :) Won't be the first time.
Plodding through some grunt work with the
new website. An almost equal split between
working on a copy, javascript and styling.
As simple as the screencap looks, it came with two hurdles that needed solving.
As simple as the screencap looks, it came with two hurdles that needed solving.
The header font, Alte Haas Grotesk, didn't
display correctly after it was
converted
to the @font-face kit. Specifically, the
lowercase 'o' looked like this:
The font's free, but I couldn't find a copy of its license, so I started tracking down font's author to ask a permission to edit the outlines.
While waiting for a reply, I found another workaround which involved generating the webfont kit from OTFs instead of TTFs. Not a big deal, but it still took over 3 hours to resolve.
—
The second issue was with the slide-out, fade-in animation. Sometimes the slide-out didn't extend all the way down and sometimes it overshot. This traced back to the fact that slide-out height was computed before the OpenSans was fully loaded and the script fused basic sans-serif to compute the height.
Again, not too complicated, but, again, cost an hour to work through.
:-|
Little bit of this, little bit of that - all unplanned and it all adds up.
The font's free, but I couldn't find a copy of its license, so I started tracking down font's author to ask a permission to edit the outlines.
While waiting for a reply, I found another workaround which involved generating the webfont kit from OTFs instead of TTFs. Not a big deal, but it still took over 3 hours to resolve.
The second issue was with the slide-out, fade-in animation. Sometimes the slide-out didn't extend all the way down and sometimes it overshot. This traced back to the fact that slide-out height was computed before the OpenSans was fully loaded and the script fused basic sans-serif to compute the height.
Again, not too complicated, but, again, cost an hour to work through.
Little bit of this, little bit of that - all unplanned and it all adds up.
Nearly done with the new website design. Here's
a snippet of how the site's header interacts with
scrolling.
First pass over UI styling for the main window.
Released a small playable test of the directory scanning module.
It appears that using as many threads as there are CPU cores and traversing the tree in a depth-first fashion (scan child directories first and sibling directories second) yields the best scanning times.
Test and see for yourself - bvckup2-demo2.exe or 64 bit version.
Also, see here for details.
It appears that using as many threads as there are CPU cores and traversing the tree in a depth-first fashion (scan child directories first and sibling directories second) yields the best scanning times.
Test and see for yourself - bvckup2-demo2.exe or 64 bit version.
Also, see here for details.
Updated
@bvckup
background with V2 branding.
2nd sketch for
bvckup2.com redesign.
Larger version is
here.
1st pass over the redesign of
bvckup2.com.
Larger version is
here.
Current design is almost two years old and
it grew steadily disconnected from the simpler
aesthetics I am pushing for now. So it needs
to be redone.
I will probably simplify this even further. I've been wanting to do a true minimalist site for a software product for a long time now and this seems like a real chance to do just that. Yay.
I will probably simplify this even further. I've been wanting to do a true minimalist site for a software product for a long time now and this seems like a real chance to do just that. Yay.
This is a
sprite
sheet of all raster images that are going into the
run-as-a-service demo.
Interestingly enough, the total size of individual
images saved as 32bit PNGs is 63,127 bytes, while
the size of this sprite sheet (without checkered
backdrop) is 63,829. After factoring in the size
of the code that is needed to parse the sprite sheet,
it really becomes a no brainer to keep every image
in its own file and embed them into .exe's resources
that way.
Added a "crash" handling and reporting facility
that includes several important changes compared to
the original version:
1. The program now differentiates between three types of
errors and handles them slightly differently.
First, there are unexpected operating system errors, i.e. when a core Windows function fails in a way that it shouldn't be failing. Classic example (that everyone is aware of) is TrackPopupMenu that is declared as returning boolean, but actually doesn't shy away from returning an integer under certain circuimstances.
Secondly, there are inconsistencies in app's own code, whereby one part assumes certain usage that is not observed by another part. These are so-called "assert" failures. The original version has three critical issues, all of this particular kind.
Lastly, there are straight-forward crashes. This is when a program is trying to operate with data that is no longer there or corrupted. This is the worst of them all as it typically indicates a major screw up on programmer's part, but in some cases it may also be an indication of failing hardware, like a faulty memory stick.
—
2. The program now generates a minidump - a small-ish file in a standard Windows format that captures the context of a failure. Minidump files dramatically simplify investigation of a problem on the developer's end.
—
3. Finally, the minidumps can now be uploaded to the support server right from the error notification window with one click. The transfer is based on WinInet API as this is supposed to improve the chances of getting the report through in proxied and otherwise "enterprised" environments.
For cases when the upload fails, the error dialog also offers an option of opening a link in a regular browser and reporting a problem that way.
First, there are unexpected operating system errors, i.e. when a core Windows function fails in a way that it shouldn't be failing. Classic example (that everyone is aware of) is TrackPopupMenu that is declared as returning boolean, but actually doesn't shy away from returning an integer under certain circuimstances.
Secondly, there are inconsistencies in app's own code, whereby one part assumes certain usage that is not observed by another part. These are so-called "assert" failures. The original version has three critical issues, all of this particular kind.
Lastly, there are straight-forward crashes. This is when a program is trying to operate with data that is no longer there or corrupted. This is the worst of them all as it typically indicates a major screw up on programmer's part, but in some cases it may also be an indication of failing hardware, like a faulty memory stick.
2. The program now generates a minidump - a small-ish file in a standard Windows format that captures the context of a failure. Minidump files dramatically simplify investigation of a problem on the developer's end.
3. Finally, the minidumps can now be uploaded to the support server right from the error notification window with one click. The transfer is based on WinInet API as this is supposed to improve the chances of getting the report through in proxied and otherwise "enterprised" environments.
For cases when the upload fails, the error dialog also offers an option of opening a link in a regular browser and reporting a problem that way.
64bit version is stored inside of the 32bit
executable, from where it is extracted and
launched to replace the 32bit version if
latter detects that it is being run on a
64bit system.
Pro:
a single executable to distribute
Con: the exe comes out weighing twice as much... or does it?
The size of a 32-bit executable can be easily reduced with the help of tools called executable packers. They work by turning the exe file into a self-extracting archive that also automatically launches the original exe after the extraction. It may sound simple but in reality it involves bundling a custom PE loader with the packed executable (as it might not be possible to extract the exe into a temporary file before running it).
In any case, exe packing is routinely used by developers who care about the size of their products, with uTorrent being one of most notable examples. Original Bvckup was also compressed, owning its tiny size in part to the magic of UPX.
—
The problem lies with 64-bit executables. For a variety of reasons there are presently no exe packers that support them.
However if we store 64bit binary inside of the 32bit executable and then pass latter through a compressor, we end up with both being compressed. Ta-da!
But there's more.
—
When an .exe is compressed, the compressor leaves the primary application icon intact. This is done so that the app would keep its original appearance in Windows Explorer and not sport a generic "program" icon.
Starting with Vista, the app is expected to support a variety of icon sizes that quickly add up. For example, bvckup's icon weighs in at about 110KB. That's almost half as big as the actual code, the code that does quite a few meaningful things on top of just looking great :)
So, if we do the 64-in-32bit packaging, we end up compressing 64bit version in its entirety, including its icon, and arriving at a greater size reduction if we were to compress the exes separately.
Win-win.
Con: the exe comes out weighing twice as much... or does it?
The size of a 32-bit executable can be easily reduced with the help of tools called executable packers. They work by turning the exe file into a self-extracting archive that also automatically launches the original exe after the extraction. It may sound simple but in reality it involves bundling a custom PE loader with the packed executable (as it might not be possible to extract the exe into a temporary file before running it).
In any case, exe packing is routinely used by developers who care about the size of their products, with uTorrent being one of most notable examples. Original Bvckup was also compressed, owning its tiny size in part to the magic of UPX.
The problem lies with 64-bit executables. For a variety of reasons there are presently no exe packers that support them.
However if we store 64bit binary inside of the 32bit executable and then pass latter through a compressor, we end up with both being compressed. Ta-da!
But there's more.
When an .exe is compressed, the compressor leaves the primary application icon intact. This is done so that the app would keep its original appearance in Windows Explorer and not sport a generic "program" icon.
Starting with Vista, the app is expected to support a variety of icon sizes that quickly add up. For example, bvckup's icon weighs in at about 110KB. That's almost half as big as the actual code, the code that does quite a few meaningful things on top of just looking great :)
So, if we do the 64-in-32bit packaging, we end up compressing 64bit version in its entirety, including its icon, and arriving at a greater size reduction if we were to compress the exes separately.
Win-win.
A page from the first demo of Bvckup 2.
Working on a shared memory buffer interface,
a part of the
IPC
module, which is what enables
the separation of the engine and the UI.
A bit more humane approach to critical error reporting.
The message is a homage to
NetPositive,
the BeOS browser that pioneered the technique of using
haiku
in error reporting. In fact, the message in option B is from
the NetPoisitive
original list.
The icon is that of a Zen circle, which has a plenty of meanings, many of which applicable given the circumstances.
The icon is that of a Zen circle, which has a plenty of meanings, many of which applicable given the circumstances.
Three-frame animation of engine's heartbeats for the demo.
Final UI styling for the "Run As A Service" demo.
The "Run as an Administrator" icon.
Test of a fading transition, part of a small library
that deals with supporting UI animations.
Fading animation dissolves a part of a window into
background, in preparation for transitioning the
window to another state, e.g. moving to the next
page in a wizard or resizing the window into a
more compact form.
As simple as it sounds, these sort of animation is fairly complex to implement on top of the Windows API.
Windows includes native support for window transparency in a form of layered windows, but it comes with a list of limitations that aren't exactly practical or easy to work, except for several specific cases.
The simplest working solution involves taking a screenshot of the window, creating an overlay window and then using it as a canvas to render the transition from an empty background to the screenshot. All the while making sure that Windows doesn't sneak in an unsolicited window refresh that would lead to a flicker, which in itself is a very "fun" thing to debug.
As simple as it sounds, these sort of animation is fairly complex to implement on top of the Windows API.
Windows includes native support for window transparency in a form of layered windows, but it comes with a list of limitations that aren't exactly practical or easy to work, except for several specific cases.
The simplest working solution involves taking a screenshot of the window, creating an overlay window and then using it as a canvas to render the transition from an empty background to the screenshot. All the while making sure that Windows doesn't sneak in an unsolicited window refresh that would lead to a flicker, which in itself is a very "fun" thing to debug.
Every application icon that is exposed to the
operating system must exist in at least nine
sizes.
16, 32, 48, 256 are more or less expected.
These are used in "small", "icon", "tile"
and "extra-large " views in Windows Explorer.
20, 22 and 26 are possible sizes of an icon used in the window caption. The exact size depends on screen's DPI level.
And 40 (along with 32 and 48) is the size of an icon used in Alt-Tab view on W7, again depending on the screen's DPI.
If you don't provide an appropriately sized icon, the OS will grab one that is slightly bigger and shrink it down, ruining the pixel polish.
Lastly, each icon ideally must be saved in 32, 8 and 4 bits per pixel formats, so to be compatible... you know... with EGA cards
It would've not that big of a deal if it weren't for the size of a resulting .ico file. For example, the above icon ends up being ~ 120KB, whereas the actual .exe it is attached to and that does quite a few meaningful things sits at around 96KB.
20, 22 and 26 are possible sizes of an icon used in the window caption. The exact size depends on screen's DPI level.
And 40 (along with 32 and 48) is the size of an icon used in Alt-Tab view on W7, again depending on the screen's DPI.
If you don't provide an appropriately sized icon, the OS will grab one that is slightly bigger and shrink it down, ruining the pixel polish.
Lastly, each icon ideally must be saved in 32, 8 and 4 bits per pixel formats, so to be compatible... you know... with EGA cards
It would've not that big of a deal if it weren't for the size of a resulting .ico file. For example, the above icon ends up being ~ 120KB, whereas the actual .exe it is attached to and that does quite a few meaningful things sits at around 96KB.
Similarly, every bit of graphics in the app
must be done in at least three different sizes
- one for each common screen DPI level.
DPI stands for dots-per-inch and it's a metric
of a pixel density on the screen. The smaller
the screen, the more pixels are packed into a
square inch, assuming the resolution stays the
same. The difference in DPI between a large
LCD panel and a 12" notebook screen can easily
be a factor of two or more.
This means that if a program renders 9px text on both screens, on one it will appear twice as small as on the other one, and might command a need for a good magnifying glass.
So naturally it's a good idea to make the OS aware of the actual DPI of the monitor in use. If it's too high, the OS can do few things to make things visible with a naked eye again. Like increase size of the default system font and make windows generally larger.
This can be done automatically by looking at the monitor .inf files or by letting the user configure the DPI manually.
—
XP defaults to 96 DPI and with some difficulty it can be changed to 120 DPI. Vista and newer versions are better. They come with four predefined levels - 96, 120, 144 and 192 - and they also support custom settings.
This brings us back to needing multiple versions of every graphic in the app.
—
Vista is the first Windows version to routinely use monitor-specific DPI values out of the box. As such 96 is no longer a dominant DPI level and the apps are now expected to take the DPI into an account when rendering their content.
The higher the DPI, the larger the graphics and the fonts the app should use. The penalty for not doing this is an automatic stretching of the app window that inevitably leads to a blurry appearance and a host of other lovely issues.
In other words - not complying with OS expectations is not really an option. Hence the need for 3+ differently sized copies of the same pixel art.
This means that if a program renders 9px text on both screens, on one it will appear twice as small as on the other one, and might command a need for a good magnifying glass.
So naturally it's a good idea to make the OS aware of the actual DPI of the monitor in use. If it's too high, the OS can do few things to make things visible with a naked eye again. Like increase size of the default system font and make windows generally larger.
This can be done automatically by looking at the monitor .inf files or by letting the user configure the DPI manually.
XP defaults to 96 DPI and with some difficulty it can be changed to 120 DPI. Vista and newer versions are better. They come with four predefined levels - 96, 120, 144 and 192 - and they also support custom settings.
This brings us back to needing multiple versions of every graphic in the app.
Vista is the first Windows version to routinely use monitor-specific DPI values out of the box. As such 96 is no longer a dominant DPI level and the apps are now expected to take the DPI into an account when rendering their content.
The higher the DPI, the larger the graphics and the fonts the app should use. The penalty for not doing this is an automatic stretching of the app window that inevitably leads to a blurry appearance and a host of other lovely issues.
In other words - not complying with OS expectations is not really an option. Hence the need for 3+ differently sized copies of the same pixel art.
Trying to find a balance between using custom styling and not getting to far away from the native OS look and feel. See also 8 other options leading up to this one.
Illustrative icons for the same demo.
First pass over the demo UI.
V2 running as a service,
an actual screenshot for a change.
Exploration of measured blending
of native and custom UI styling.
An idea for a space-saving tab control. Shelved.
An idea for an in-form buttons.
Exploring full-on customization.
Earlier sketches of heavily customized UI.
The estimation is calculated in three steps.
First, the app looks at effective copying speed and uses sliding window to average it across last 25 samples taken 650 ms apart.
Second, it uses this average speed and the remaining byte count to approximate the completion time.
Finally, the completion time is averaged with its own sliding window, but using only 4 past samples.
This double-averaging helps mitigating the effects of sudden short-lived I/O bursts and hiccups on the estimation.
The parameters - 25, 650 and 4 - are configurable through the engine .ini file, so the whole thing is open for tuning if needs be.