R81 pre-release / Parallel execution
Nov 04, 2019
The central new feature in Release 81 is its support for parallel backup execution - an ability to execute multiple backup steps at once.
Prior to R81, the file system scan and file ID retrieval were already done
in a parallel fashion, using several threads.
Release 81 extends this behavior to all phases of backup, allowing the program to execute all suitable operations in parallel with each other.
If you are familiar with the /mt switch of robocopy, it's a similar idea, but with automatic thread count management and several other important improvements.
In practical terms, parallel execution results in a *very* noticeable speed increase in the following cases:
⦁ Backups with a lot of small- to medium-sized files
⦁ Backups that require creating a large number of folder
⦁ Backups that require deleting a large number of folders or files
⦁ Backups that "touch" a lot of folders
For a bit of anecdotal data see here - https://bvckup2.com/wip/03102019
Parallel execution is enabled by default with the thread count set to the number of CPUs for local backups and to the double of that for network backups.
To set the thread count to some fixed custom value use the following override -
for how to do this.
Pre-release 220.127.116.11 and newer.
Nov 11, 2019
⦁ Fixed an issue with handling failures when _retrying_ a backup of a
More specifically, if a new file appears at source, the backup is run and
it fails with a retryable error mid-copying. For example, a network
connection goes away. This will cause the program to retry the copying
after a pause. If this retry would also fail with an error, then bvckup2
would pop up the "something is wrong" message and shut down.
Nov 13, 2019
⦁ Fixed an issue with the retrying mechanism panicking when the last
attempt fails. There was an invalid self-consistency check (an assert),
which is as ironic as it gets. A bug in a bug checking code.
Nov 14, 2019
⦁ Fixed a crashing issue with the archive trimming code that surfaced
every time an archive scan would identify and remove expired items.
Dec 03, 2019
⦁ Reworked retrying logic that is used to re-execute failed steps in a
presence of transient errors (e.g. network disconnects). This also
fixed an issue that caused self-inflicted crashes in cases when
copying of a _new_ file succeeded after several attempts.
⦁ Reworked logging module to report step info as it's being executed
IF this step happens to be the only one active and no other steps are
being scheduled until it completes.
This happens when bvckup2 is backing up a very large file, at which
point it will finish all other active steps, but won't queue any new ones.
This is done to allow the bulk copying to proceed in peace if you will.
Prior to 99.5 all step-specific info was recorded into a backup log only
after the step finished. However if there's exactly one step in progress
we can dump its progress into log as we go - which is exactly what 99.5
Dec 05, 2019
⦁ Fixed an invalid self-consistency check in the logging module that
caused 99.5 to shut down with "Something happened" message.
This was caused by change #2 in 99.5.
Dec 18, 2019
⦁ Added support for working with live files. See here for details -
This feature is configured with the following per-job override:
will force the program to double-check size and timestamps for
files with .txt extension and files with names starting with 123.
for how to add
⦁ Fixed a rare logging issue that caused the program to exit with the
"Something happened" message.
Dec 29, 2019
⦁ Fixed another issue with handling a successful backup of a new file
after running into a retryable error.
Jan 20, 2020
⦁ Fixed an issue with an incorrect logging level when failing to update
meta of a directory (the "Updating directory" operation)
⦁ + All changes from the mainstream 80.9 release
Feb 24, 2020
⦁ Another pass over the retrying-on-errors logic. Apparently, a network
hiccup may not necessarily kill _all_ active operations. Sometimes it can
cause just some of them to error out, while the rest will keep on going.
⦁ + All changes from the mainstream 80.10 and .11 releases
There was no 80.99.10 release.
Mar 02, 2020
⦁ Added UI support for configuring multi-threaded backup execution.
New Processing section:
The same expanded:
⦁ Reworked all UI animations to use FPS-based sequencing. Previously,
animation used the "run for X ms using Y steps". X and Y were hand-
coded and because of that the frames-per-second value (FPS) was all
over the place. With this release there's now a single global FPS value
for all animations and another for all fade-ins and fade-outs.
⦁ Resolved (hopefully the last) issue with the retrying logic - there was a
race condition in the code that preemptively cancelled queued, but not
yet executed steps when the program ran into a retryable error.
Mar 09, 2020
⦁ Fixed an issue with program closing down when needing to trim the
$Archive. Trimming is also done in multi-threaded (parallel) fashion
now, but it was done _before_ the parallel execution was configured.
No config => no way to trim => whoops.
Kudos to Greg for the traces.
⦁ Fixed an issue with overly strict self-consistency check in the retrying
code. Had to do with retrying the backup of _new_ files. If this sort of
operation was queued and then cancelled due to other steps failing
(e.g. because the network connection went down), then in some cases
the program would panic by seeing a new-copy step in a retry state.
Topic is locked.