The support forum

Delta copying discussion

dstdg18 :

Feb 25, 2016



[ Detached from https://bvckup2.com/support/forum/topic/739 ]

Perfect! I love how detailed this post is on the delta copying feature! Thank you kindly!

Blue407 :

Mar 21, 2016

Great level of detail, sets my mind at rest that it should work :)

mhw-at-work :

Dec 20, 2016

"Files smaller than 2MB and files under 32MB that weren't modified within last 30 days are always copied in full."

Let me see if I have this straight: In my source I have a folder tree of 11.2GB in 365,000 files, most under 2mb. Every one of those files will be copied on every bvckup2 run, even when date/time/size are unchanged?

Alex Pankratov :

Dec 20, 2016

No. Delta copying is concerned merely with _how_ modified files are copied. Detection of modified files is an altogether different and unrelated step.

Doequer :

Jan 29, 2017

Hi, the above one was a good explanation about such an useful program's feature, but I'm wondering something about it:

Is there a way one can force the "delta copying" mode, for specific folders/files, which one foresees they should be copied that way, and not in the regular, full mode?

I mean, let's say I have two folders, source and backup; there is an "example_123.x" file in the first one, which I regularly update, but it's filename's will not always be like that, but have some slightly variation with each update, let's say "example_124.x".  Currently, if I want there is the chance Bvckup uses the "delta copying" feature for said updated file (because I'm positively sure it is just an updated version from the same file, and not a totally new one), I have to rename it previously, in order it matches that in the source folder, because otherwise, it is going to handle it as a completely new one, and transfer it fully; besides, if it does copy it in full, I will have to delete the previous "example_123.x" file, which would suppose an additional "deleting" task.

Alex Pankratov :

Jan 29, 2017

It all depends on how exactly example_123.x turns into example_124.x.

If this happens because 123 gets _renamed_ into 124, then bvckup2 can understand that - it will both rename 123's backup copy and preserve its delta copying state, so it will keep on delta-updating the file. This however requires "Detecting Changes" setting in backup settings to be set to "Use snapshot" (and not to "Re-scan on every run") and "Rename detection" enabled for files.

However if 124 is created from scratch and filled with 123's data, then the app won't be able to link these two files and it will indeed re-copy 124 from scratch.

What you are suggesting is effectively a kind of "hint" system to tell bvckup2 that THIS file and THAT file are two versions of the same source file even though it's not obvious. This is not a bad idea, but I strongly suspect that it will get very hairy when it comes to the implementation.

Doequer :

Jan 29, 2017

Yes, I think the only way of avoiding the whole copying process for source files and their equivalent updated ones, would be renaming the source level files with a generic name, and then using that generic name for all the newer files as well, so they match each time.

That's certainly not the more straightforward way of handling the updates for said cases, but will still be more efficient than copying full files each time, principally with larger ones.

What you mention about the "hint" system is kind of what I first thought of, about Bvckup2 being that smart how to detect that some files should be delta copied, even if they have some little differences in their filenames, whether it be through a kind of special index from said files, or some forced option specifically for them.

But of course, I know there should be miles away between just thinking about a kind of workaround like that, and having it implemented. ;)

112233 :

Apr 29, 2017

I apologize if this question is obvious or has already been answered.
What happens if I modify a large file, Bvckup starts backing it up using delta copying, and then my source harddrive dies.
Am I left without a usable copy of the file (assuming I only have the one backup)?

Alex Pankratov :

May 01, 2017

It depends. See the opening post in this topic, caveat #3.

jeffwmiles :

Jul 17, 2017

Is there a way to have Bvckup pre-emptively compute the hash on destination files if they already exist prior to the first run?

I have 4TB of VHDX files from a Hyper-V virtual machine already on the destination, never touched by Bvckup. I want to set up a Bvckup job from the near-identical source VM to these destination files, using delta-copy. Because the delta-copy hash doesn't exist, it begins to copy the files in full.

I'd like to pre-compute that hash to avoid saturating the network for weeks. Perhaps a workaround would be to use Bvckup to make a copy of the destination files locally, and then transport the hash to the source side?

Alex Pankratov :

Jul 17, 2017

Perhaps a workaround would be to use Bvckup to make a copy of the destination files locally, and then transport the hash to the source side?


Exactly right, but there are some per-requisites.

You can indeed pre-create delta state by making a ephemeral backup to a temporary local location. However, delta state for each file stores the exact "created" and "last-modified" timestamps of the backup copy. So for this whole thing to work you will need to have timestamps on these temporary local backup copies match timestamps on their counterparts at your existing destination _exactly_.

For that:

1. Source and destination should be using the exact same file system - this has to do with the fact that NTFS, FAT, etc. they all trim timestamps differently. NTFS has a resolution of 100ns, FAT - 2 seconds and pseudo NTFS on NAS boxes - anything in between.

2. "Last modified" timestamps on source and destination copies of files are the same. This has to do with the fact that created/last-modified timestamps of backup copies are saved in the delta state and they are verified on the next run. If they don't match their live versions, the delta state is assumed to be out-of-sync with the backup copy and it is discarded.

So, to that end -

Create a temporary backup job, set it to manual, point at your source and destination and run a simulated run (via the right-click menu). Now, look at the log and check that all that Bvckup 2 plans to do is update the "created" timestamps for your files. If it plans to copy the data, then you are out of luck.

If it's just timestamps, then run the job and it sync the "created" timestamps and finish quickly.

Now, remove this job. Create another one, set to manual and point it at your source and temporary local folder. Run it and it will copy the files and compute all required delta state. Go back into the job settings and re-point it at your destination.

You will see this - https://bvckup2.com/wip/06092016 - say NO.

And that's it, you will should delta state that is sync'ed with your destination files.

... But ...

I must say that this is hacky. If you happen to have a file that only _appears_ to be unchanged between the source and destination, then it will be fundamentally screwed up by running a job that is set up as per above.

In other words - make sure you know what you are doing.

justbackmeup :

Sep 24, 2017

Now, remove this job. Create another one, set to manual and point it at your source and temporary local folder. Run it and it will copy the files and compute all required delta state. Go back into the job settings and re-point it at your destination.

What if I already have a different backup job which I want to add these items to (ie, currently these items are unselected in the existing job)? If I create my temp job, copy the files so that all the delta states get computed, and then add those files to the existing backup *before* deleting the temp job, will the delta states persist, or will they get deleted when I delete the temp job I was using for computing delta states?

I guess a related question is if delta states are shared across jobs if multiple jobs are working with the same files (and so delta states only get deleted when there are no more relevant jobs existing)?

Of course, the part of this that relates to my seeding my backup will be even easier once we can just compute delta states without running a temp backup job ( https://bvckup2.com/support/forum/topic/965/5385 ). ;)

Alex Pankratov :

Sep 25, 2017

will they get deleted when I delete the temp job I was using for computing delta states?


Delta state is a property of a job. When a job is deleted, the delta state is deleted too.

That said, delta state for a file "xyz.abc" is stored in \engine\backup-00xx\deltas\ directory in a binary file named after md5("path\to\xyz.abc"), e.g.

  \engine\backup-0003\deltas\4d\e8\2a2e3965ed6ecfdc6ebef6473230.dat

whereby 4de82a2e3965ed6ecfdc6ebef6473230 is the md5 hash. So in theory you can move delta state for a file between two jobs if you know the exact file path/name (with path being relative to the job's From folder).

justbackmeup :

Sep 25, 2017

So in theory you can move delta state for a file between two jobs if you know the exact file path/name (with path being relative to the job's From folder).

In my particular case I want all of the deltas for the backup job, so then I'll just copy them all over. :)

Thanks much!

mtc :

Nov 16, 2017

I think there used to be a way to force backup of a very large file, regardless of any date/timestamp/size checks, and also to force delta copying to be used for that file.   Has that capability been removed?  It was useful for container files where the external container file information doesn't ever change (by construction).

Thanks!

Alex Pankratov :

Nov 17, 2017


Have a look here - https://bvckup2.com/support/forum/topic/415
And also here - https://bvckup2.com/support/forum/topic/221

Alex Pankratov :

Oct 30, 2018

Is there a way to have Bvckup pre-emptively compute the hash on destination files if they already exist prior to the first run?


There is now - https://bvckup2.com/support/forum/topic/1131

New topic

Create
Made by Pipemetrics in Switzerland
Support

Follow
Blog / RSS
Social Twitter
Reddit
Miscellanea Press resources
Testimonials
Company
Imprint
Legal Terms
Privacy