The pre-release forum

R81 pre-release / Parallel execution

Nov 04, 2019

Overview


The central new feature in Release 81 is its support for parallel backup execution - an ability to execute multiple backup steps at once.

Parallel execution


Prior to R81, the file system scan and file ID retrieval were already done
in a parallel fashion, using several threads.

Release 81 extends this behavior to all phases of backup, allowing the program to execute all suitable operations in parallel with each other.

If you are familiar with the /mt switch of robocopy, it's a similar idea, but with automatic thread count management and several other important improvements.

Use cases


In practical terms, parallel execution results in a *very* noticeable speed increase in the following cases:

⦁    Backups with a lot of small- to medium-sized files
⦁    Backups that require creating a large number of folder
⦁    Backups that require deleting a large number of folders or files
⦁    Backups that "touch" a lot of folders

For a bit of anecdotal data see here - https://bvckup2.com/wip/03102019

Configuration


Parallel execution is enabled by default with the thread count set to the number of CPUs for local backups and to the double of that for network backups.

To set the thread count to some fixed custom value use the following override -

        conf.exec.threads   <thread-count>

See https://bvckup2.com/support/forum/topic/1140 for how to do this.

Versions


Pre-release 1.80.99.1 and newer.

Nov 11, 2019

Pre-release 80.99.2


⦁    Fixed an issue with handling failures when _retrying_ a backup of a
      _new_ file.

      More specifically, if a new file appears at source, the backup is run and
      it fails with a retryable error mid-copying. For example, a network
      connection goes away. This will cause the program to retry the copying
      after a pause. If this retry would also fail with an error, then bvckup2
      would pop up the "something is wrong" message and shut down.

Nov 13, 2019

Pre-release 80.99.3


⦁    Fixed an issue with the retrying mechanism panicking when the last
      attempt fails. There was an invalid self-consistency check (an assert),
      which is as ironic as it gets. A bug in a bug checking code.

Nov 14, 2019

Pre-release 80.99.4


⦁    Fixed a crashing issue with the archive trimming code that surfaced
      every time an archive scan would identify and remove expired items.

Dec 03, 2019

Pre-release 80.99.5


⦁    Reworked retrying logic that is used to re-execute failed steps in a
      presence of transient errors (e.g. network disconnects). This also
      fixed an issue that caused self-inflicted crashes in cases when
      copying of a _new_ file succeeded after several attempts.

⦁    Reworked logging module to report step info as it's being executed
      IF this step happens to be the only one active and no other steps are
      being scheduled until it completes.

      This happens when bvckup2 is backing up a very large file, at which
      point it will finish all other active steps, but won't queue any new ones.
      This is done to allow the bulk copying to proceed in peace if you will.

      Prior to 99.5 all step-specific info was recorded into a backup log only
      after the step finished. However if there's exactly one step in progress
      we can dump its progress into log as we go - which is exactly what 99.5
      implements.

Dec 05, 2019

Pre-release 80.99.6


⦁    Fixed an invalid self-consistency check in the logging module that
      caused 99.5 to shut down with "Something happened" message.

      This was caused by change #2 in 99.5.
Topic is locked.

New topic

Create
Made by IO Bureau in Switzerland
Support

Updates
Blog / RSS
Follow Twitter
Reddit
Miscellanea Press kit
Testimonials
Company Imprint

Legal Terms
Privacy