Including/excluding files and folders
Aug 29, 2017
Overview of the backup process
Every backup run consists of 3 main phases - scanning, planning and execution.
In the scanning phase the program looks at the source and backup locations and builds two lists of files and folders. These are referred to as source and backup "file trees".
In the planning phase the program compares these two trees and compiles a list of simple steps that, when executed in sequence, bring backup location in sync with the source. The steps include basic file system operations like "create directory", "copy file", "move file", etc. The resulting list is referred to as a "backup plan".
Finally, in the execution phase Bvckup 2 merely goes through the backup plan and carries out operations as prescribed by each step.
Exclusion/inclusion filters
Exclusion/inclusion filters (or "rules") are configured in the "Backup What" section of the backup settings and they are used to decide which files or folders should be included in the file tree.
https://bvckup2.com/support/data/backup-what-window.png
For each item (file or folder) the program first checks if it is explicitly included or excluded in the middle pane. If there's no match, then it goes through the rules from the bottom pane and checks them as well.
If there is NO match, then the item is treated the same way as its parent folder - if the folder is included, the item is in; if the folder is excluded, the item is out.
If there is a match, then the item is included/excluded as prescribed by the rule.
-- And that's it --
The option at the very top of the window merely controls the action assigned to the top location. With "Include everything" it's "include", with "start with an empty list" it's "exclude".
For the details on how *name* matching works in filtering rules see this topic -
https://bvckup2.com/support/forum/topic/499/29461
Aug 29, 2017
Verbose and generic rules
Explicit selections in the middle pane of the configuration window are referred to as "verbose" rules. Additional rules specified in the bottom pane are called "generic" rules.
https://bvckup2.com/support/data/verbose-generic.png
Verbose rules are always checked first. If one applies, then this it - rule's action determines if an item is included or excluded.
If none of verbose rules applies, the program checks generic rules. If there's a match, it determines the action.
If there's no match, then the item is included/excluded the same way as its parent folder.
The "final" attribute
When we verbosely exclude or include a folder, generic rules are still applied to its contents and they may end up reversing the action.
For example, if we exclude /Foo/ and then add an "include *.jpg" rule, then we will end up with all jpegs from /Foo/ included in the backup.
Enter the "final" rule attribute.
When set to Yes, it suppresses processing of generic rules for the content on a folder, and thus causes the folder to be included or excluded in full.
When set to No, it forces filtering of folder contents through generic rules.
By default the attribute is not set at all, in which case it inherits the job-scope value. Verbose exclusions are final, all other rules are non-final.
Overriding "final" behavior
1. It is possible to set "final" attribute for a specific rule.
2. It is possible to set "final" attribute for a class of rules at once -
Verbose includes
Verbose excludes
Generic includes
Generic excludes
3. It is possible to set "final" attribute for the default action - the case when no rules apply to the top-level items.
Details
The final attribute for a rule is specified by appending ! or ? to the rule's action (! for Yes and ? for No) :
conf.filters.src.verbose.folder +! .\AdvancedInstallers
conf.filters.src.generic.rule -? match \Recycler
The final defaults for rule classes are controlled by:
conf.filters.src.verbose.include_is_final
conf.filters.src.verbose.exclude_is_final
conf.filters.src.generic.include_is_final
conf.filters.src.generic.exclude_is_final
The final value for the default action is controlled by:
conf.filters.src.default.is_final
* This requires R79.5 or newer
How to add an override
Explained here -
https://bvckup2.com/support/forum/topic/11402
Aug 29, 2017
An example
Say, the source location has a large number of folders and we want to back up only *.jpg files from one specific folder. Let's call it Foo\Xyz.
(1) Start with an empty set
(2) Verbosely include Foo\Xyz
(3) Add "include *.jpg" rule
(4) Add "exclude *" rule
This will however end up pulling *.jpg files from ALL folders.
This is because the exclusion of Xyz's siblings with default action (1) is not final, so the program will descend into each of them and apply rules (3) and (4) to their contents.
To prevent this from happening, we need to make default action final:
conf.filters.src.default.is_final 1
Once this override is in place, all Foo's siblings will be excluded and their exclusion will be final. The program will then descend into Foo and exclude all Xyz's siblings the same way.
Xyz will be included as per its verbose rule. Verbose inclusions are NOT final, so the program then process its contents while observing generic rules 3 and 4.
#
Now would be a good time to acknowledge that this *is* a bit complicated. The underlying cause of this is that in some cases we may expect generic rules to apply always and in other cases - only selectively.
Moreover, a little variation to verbose rules (like excluding a subfolder in an explicitly included folder) may flip expected behavior 180 degrees, so there appears to be no easy way to accommodate this automatically.
The only solution is to pick reasonable defaults and then allow hand-tuning the filtering logic for edge cases.
3
Aug 29, 2017
Filtering behavior in older releases
There are 2 differences to how filtering worked in pre-R78 versions:
1. In the middle pane it was not possible to include items in excluded folders and vice verse (see
https://bvckup2.com/wip/12072017).
2. The filtering logic in "Include everything" and "Start with a blank list" modes was slightly different.
In particular, in "Include everything" mode all folder exclusions were final, meaning that the program didn't even bother descending into them during the scan.
And this brings us to...
"Upgrade notice - Legacy mode"
When you upgrade from a pre-R78 version, the program may change the global "final" presets so that the filtering logic remains the same as before.
When this happens, you will see the following in the backup log:
Upgrade notice -
Legacy mode is enabled for processing of excluded folders.
This means is that the program changed the default for (verbose excluded) and (generic excluded) rules to be "final".
4
MWorthington :
Sep 09, 2017
Hi Alex,
Nice update!
However, when viewing a selection in "What to Backup", the small tick used on a selected-subfolder (when the parent folder is not selected) is quite hard to see. I'm not sure what to suggest .... maybe a grey, big tick against the parent folder? Or make that small tick red, or blue ...?
Regards,
Mark
5
MWorthington :
Sep 11, 2017
Alex,
I think it's worth highlighting a significant improvement (I'm not sure if I've missed it in your notes) ....
Up until R78, I could exclude any number of sub-folders for a top-level folder, However, any newly created folders would, by default, be included in a new backup.
With the latest version, I'm in better control. I exclude the top-level and only include the sub-folders I want.
In my case, for the backup strategy I use, that's a BIG improvement, thanks!
Mark
6
Alex Pankratov :
Sep 11, 2017
the small tick ... is quite hard to see
It's a secondary piece of information, so it doesn't really need to be highly visible. If someone's looking for this information (as in "I wonder if I have any folders excluded below the surface level"), they should be able to find it.
I had another option for this (the one on the right), but I think it takes an effort to notice it and to understand how it works. Though once you know that, it's pretty obvious -
https://bvckup2.com/wip/r78-filter-icons.png
With the latest version, I'm in better control. I exclude the top-level and only include the sub-folders I want.
Yep. The new filtering framework is more flexible *and* more uniform at the same time. My main concern with it is that in some cases people might want to exclude a folder, period, and in other cases they may want to exclude everything in a folder by default and then reverse this with wildcard "include" rules. The default is the latter, but we'll have to wait and see if it's a good default or not.
7
daniel.filipowicz :
Dec 14, 2017
Previous releases (i.e. 76.14) offer an option to exclude folders in both source and also destination which causes Bvckup to ignore/omit them fully. I could edit destination exclusions via Ctrl-click the Details button next to What to backup. How do I achieve the same behavior in the current release since Ctrl-click on Details doesn't work anymore
8
Alex Pankratov :
Dec 14, 2017
Daniel,
This is still very well supported. You just need to enable this in the UI settings:
1. Exit the program
2. Add the following line to %LocalAppData%\Bvckup2\ui\bvckup2-ui.ini
edit_dst_filters 1
3. Start it back up. Then Ctrl-Click on Details will work.
https://bvckup2.com/support/forum/topic/502/29599
daniel.filipowicz :
Dec 15, 2017
Hi Alex, Thank you for your prompt response.
Problem solved - nice and easy! Thank you again for your help!
10
gabrielz07 :
Jul 09, 2018
Is there anyway to exclude files without a file extension defined? I have a case where MS Excel creates a bunch of temp files without any extension, how can I exclude them from being backedup?
11
Alex Pankratov :
Jul 09, 2018
You'll need two rules - one to include anything *with* an extension and another to exclude everything else:
include file *.*
exclude file *
12
gabrielz07 :
Jul 10, 2018
Thank you so much