The support forum

Including/excluding files and folders

Aug 29, 2017

Overview of the backup process


Every backup run consists of 3 main phases - scanning, planning and execution.

In the scanning phase the program looks at the source and backup locations and builds two lists of files and folders. These are referred to as source and backup "file trees".

In the planning phase the program compares these two trees and compiles a list of simple steps that, when executed in sequence, bring backup location in sync with the source. The steps include basic file system operations like "create directory", "copy file", "move file", etc. The resulting list is referred to as a "backup plan".

Finally, in the execution phase Bvckup 2 merely goes through the backup plan and carries out operations as prescribed by each step.

Exclusion/inclusion filters


Exclusion/inclusion filters (or "rules") are configured in the "Backup What" section of the backup settings and they are used to decide which files or folders should be included in the file tree.

        https://bvckup2.com/support/data/backup-what-window.png

For each item (file or folder) the program first checks if it is explicitly included or excluded in the middle pane. If there's no match, then it goes through the rules from the bottom pane and checks them as well.

If there is NO match, then the item is treated the same way as its parent folder - if the folder is included, the item is in; if the folder is excluded, the item is out.

If there is a match, then the item is included/excluded as prescribed by the rule.

-- And that's it --

The option at the very top of the window merely controls the action assigned to the top location. With "Include everything" it's "include", with "start with an empty list" it's "exclude".

For the details on how *name* matching works in filtering rules see this topic - https://bvckup2.com/support/forum/topic/499/2946

Aug 29, 2017

Verbose and generic rules


Explicit selections in the middle pane of the configuration window are referred to as "verbose" rules. Additional rules specified in the bottom pane are called "generic" rules.

        https://bvckup2.com/support/data/verbose-generic.png

Verbose rules are always checked first. If one applies, then this it - rule's action determines if an item is included or excluded.

If none of verbose rules applies, the program checks generic rules. If there's a match, it determines the action.

If there's no match, then the item is included/excluded the same way as its parent folder.

The "final" attribute


When we verbosely exclude or include a folder, generic rules are still applied to its contents and they may end up reversing the action.

For example, if we exclude /Foo/ and then add an "include *.jpg" rule, then we will end up with all jpegs from /Foo/ included in the backup.

Enter the "final" rule attribute.

When set to Yes, it suppresses processing of generic rules for the content on a folder, and thus causes the folder to be included or excluded in full.

When set to No, it forces filtering of folder contents through generic rules.

By default the attribute is not set at all, in which case it is assumed to be No. That is, no rules are final.

Overriding "final" behavior


1. It is possible to set "final" attribute for a specific rule.

2. It is possible to set "final" attribute for a class of rules at once -

    Verbose includes
    Verbose excludes
    Generic includes
    Generic excludes

3. It is possible to set "final" attribute for the default action - the case when no rules apply to the top-level items.

Details


The final attribute for a rule is specified by appending ! or ? to the rule's action (! for Yes and ? for No) :

        conf.filters.src.verbose.folder        +!  .\AdvancedInstallers
        conf.filters.src.generic.rule             -?  match \Recycler

The final defaults for rule classes are controlled by:

        conf.filters.src.verbose.include_is_final
        conf.filters.src.verbose.exclude_is_final
        conf.filters.src.generic.include_is_final
        conf.filters.src.generic.exclude_is_final

The final value for the default action is controlled by:

        conf.filters.src.default.is_final

        * This requires R79.5 or newer

Aug 29, 2017

An example


Say, the source location has a large number of folders and we want to back up only *.jpg files from one specific folder. Let's call it Foo\Xyz.

(1)    Start with an empty set
(2)    Verbosely include Foo\Xyz
(3)    Add "include *.jpg" rule
(4)    Add "exclude *" rule

This will however end up pulling *.jpg files from ALL folders.

This is because the exclusion of Xyz's siblings with default action (1) is not final, so the program will descend into each of them and apply rules (3) and (4) to their contents.

To prevent this from happening, we need to make default action final:

     conf.filters.src.default.is_final        1

Once this override is in place, all Foo's siblings will be excluded and their exclusion will be final. The program will then descend into Foo and exclude all Xyz's siblings the same way.

Xyz will be included as per its verbose rule. Verbose inclusions are NOT final, so the program then process its contents while observing generic rules 3 and 4.

#

Now would be a good time to acknowledge that this *is* a bit complicated. The underlying cause of this is that in some cases we may expect generic rules to apply always and in other cases - only selectively.

Moreover, a little variation to verbose rules (like excluding a subfolder in an explicitly included folder) may flip expected behavior 180 degrees, so there appears to be no easy way to accommodate this automatically.

The only solution is to pick reasonable defaults and then allow hand-tuning the filtering logic for edge cases.

Aug 29, 2017

Filtering behavior in older releases


There are 2 differences to how filtering worked in pre-R78 versions:

1. In the middle pane it was not possible to include items in excluded folders and vice verse (see https://bvckup2.com/wip/12072017).

2. The filtering logic in "Include everything" and "Start with a blank list" modes was slightly different.

In particular, in "Include everything" mode all folder exclusions were final, meaning that the program didn't even bother descending into them during the scan.

And this brings us to...

"Upgrade notice - Legacy mode"


When you upgrade from a pre-R78 version, the program may change the global "final" presets so that the filtering logic remains the same as before.

When this happens, you will see the following in the backup log:

        Upgrade notice -
        Legacy mode is enabled for processing of excluded folders.

This means is that the program changed the default for (verbose excluded) and (generic excluded) rules to be "final".

MWorthington :

Sep 09, 2017

Hi Alex,

Nice update!

However, when viewing a selection in "What to Backup", the small tick used on a selected-subfolder (when the parent folder is not selected) is quite hard to see. I'm not sure what to suggest .... maybe a grey, big tick against the parent folder? Or make that small tick red, or blue ...?

Regards,

Mark

MWorthington :

Sep 11, 2017

Alex,

I think it's worth highlighting a significant improvement (I'm not sure if I've missed it in your notes) ....

Up until R78, I could exclude any number of sub-folders for a top-level folder, However, any newly created folders would, by default, be included in a new backup.

With the latest version, I'm in better control. I exclude the top-level and only include the sub-folders I want.

In my case, for the backup strategy I use, that's a BIG improvement, thanks!

Mark

Alex Pankratov :

Sep 11, 2017

the small tick ... is quite hard to see


It's a secondary piece of information, so it doesn't really need to be highly visible. If someone's looking for this information (as in "I wonder if I have any folders excluded below the surface level"), they should be able to find it.

I had another option for this (the one on the right), but I think it takes an effort to notice it and to understand how it works. Though once you know that, it's pretty obvious - https://bvckup2.com/wip/r78-filter-icons.png

With the latest version, I'm in better control. I exclude the top-level and only include the sub-folders I want.


Yep. The new filtering framework is more flexible *and* more uniform at the same time. My main concern with it is that in some cases people might want to exclude a folder, period, and in other cases they may want to exclude everything in a folder by default and then reverse this with wildcard "include" rules. The default is the latter, but we'll have to wait and see if it's a good default or not.


daniel.filipowicz :

Dec 14, 2017

Previous releases (i.e. 76.14)  offer an option to exclude folders in both source and also destination which causes Bvckup to ignore/omit them fully. I could edit destination exclusions via Ctrl-click the Details button next to What to backup. How do I achieve the same behavior in the current release since Ctrl-click on Details doesn't work anymore

Alex Pankratov :

Dec 14, 2017



Daniel,

This is still very well supported. You just need to enable this in the UI settings:

1.  Exit the program
2.  Add the following line to %LocalAppData%\Bvckup2\ui\bvckup2-ui.ini

        edit_dst_filters     1

3.  Start it back up. Then Ctrl-Click on Details will work.

https://bvckup2.com/support/forum/topic/502/2959

daniel.filipowicz :

Dec 15, 2017

Hi Alex,  Thank you for your prompt response.
Problem solved - nice and easy! Thank you again for your help!

gabrielz07 :

Jul 09, 2018

Is there anyway to exclude files without a file extension defined? I have a case where MS Excel creates a bunch of temp files without any extension, how can I exclude them from being backedup?

Alex Pankratov :

Jul 09, 2018

You'll need two rules - one to include anything *with* an extension and another to exclude everything else:

    include file *.*
    exclude file *

gabrielz07 :

Jul 10, 2018

Thank you so much

New topic

Create
Made by Pipemetrics in Switzerland
Support


Follow
Twitter
Blog / RSS
Miscellanea Press resources
Testimonials
On robocopy
Company
Imprint

Legal Terms
Privacy