The support forum

Blocking system reboots when there are active backups

Dec 04, 2017

Description


As of release 78.14 it's possible to block computer reboots such as those imposed by Windows Updates while a specific backup job is running.

Configuration


This is a per-job option and it is Off by default. In order to turn it on:

    1. Right-click on a backup job in program's main window
    2. Select "Open folder" > "Configuration and Logging"
    3. Use Notepad to create a text file called override.ini there
    4. Put the following line in it

        conf.system.prevent_rebooting   1

    5. Save the ini, exit Notepad, restart Bvckup 2

Jan 08, 2018

Additionally


As of release 78.17 the reboots are blocked by default.

This can be overridden both for all jobs and for a specific job.

A job-level override is the same as above, except the setting should be set to 0 instead of 1 (0 = no, 1 = yes). By default the setting will be at -1, which means "use engine-level default".

To disable reboot blocking at an engine level the magic incantation is as follows:

        prevent_rebooting      0

and it should go into %Bvckup2%\engine\bvckup2-engine.ini file.

Doequer :

Jan 15, 2018

As it would be quite bothersome have to look at the main program's interface to verify no jobs are about to run or already doing it at the same time one needs to reboot, sleep or even shutdown the system, the above feature is very convenient. But I just tried it with a test job, and the system still slept normally, even when there was a backup job already running. I have these parameters in the "engine" file:

prevent_sleeping 1
prevent_rebooting 1

Is there anything else needed to make them work as expected?

Alex Pankratov :

Jan 15, 2018



The way sleep prevention is implemented in Bvckup is that the program sets ES_SYSTEM_REQUIRED flag on a thread that is running a backup [1].

What it appears to be doing is it resets system's internal "idle timer" that is used to put the machine to sleep when there's no user activity [2]. Meaning that this does NOT prevent the sleep that is explicitly initiated by the user.

Now, the question is if in your test you put your machine to sleep or did you let it happen automatically, on inactivity timeout?

What you can try is to start a backup, pause it (using Stop button) and
then wait and see if the machine can go to sleep on its own.

[1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa373233(v=vs.85).aspx

[2] https://msdn.microsoft.com/en-us/library/windows/desktop/aa373208(v=vs.85).aspx

Doequer :

Jan 16, 2018

I see, but I was referring to user initiated sleeps, reboots and shutdowns. My Windows' power options keep the PC always active, with no kind of active idleness timers.

Let's say I want to put to sleep or reboot the machine, in such cases I never stop for a while to see Bvckup's jobs' condition, but rather just do it. But then I thought of this feature for those cases, when a user or even a scheduled third-party program reboots or put the PC to sleep.

Is there a way Bvckup intercepts and delays such kind of requests, so currently active jobs or those with a few seconds to start, don't get abruptly stopped?

I did notice that in the case of "standbys", after turning on the PC again, the interrupted job resumed its activity from its last state; but I doubt whether that will be always the case or a reliable practice. And although I didn't tried it, I think that in the case of reboots and shutdowns, it would get considerably complicated, since I don't know if Bvckup will just resume the jobs next time without any consequent problems.

Alex Pankratov :

Jan 16, 2018

Reboots and sleeps are handled by Windows differently. A sleep is a non-intrusive, state-preserving operation, so it doesn't have a strict veto'ing mechanism like reboots have.

Is there a way Bvckup intercepts and delays such kind of requests, so currently active jobs or those with a few seconds to start, don't get abruptly stopped?


See my links above. Sleep prevention is done by tagging an active execution thread (that's doing a backup) as needing a system to be up. This is a request for Windows, but Windows may or may not act on it.

Reboot prevention is done by creating a similar request, but it is of a stronger nature and it's called a "reboot block". When one is in place, it _will_ block a reboot and if you (or a system) is trying to reboot the machine, you will see a message saying "Program X is blocking a reboot for THIS reason. Would you like to proceed anyway?" or something similar.

I did notice that in the case of "standbys", after turning on the PC again, the interrupted job resumed its activity from its last state; but I doubt whether that will be always the case or a reliable practice.


Doubt not. It will always be the case.

Basically, "sleep" is a non-intrusive system pause. All running programs just freeze until the machine is awoken. "Reboot" is a full system reset, so all programs are shut down with some being restarted after a reboot.

Doequer :

Jan 16, 2018

Alright, thanks for your explanation.

Normally, I just put the system to sleep by myself or via a third-party timer, and if jobs will resume correctly each time, there is no problem. But in case of reboots or shutdowns, I bet I will still need to look at Bvckup's current state before doing it.

I did some additional simple tests to see how Bvckup behaves when some currently active job is interrupted by a restart, a logging off or a shutdown. I noticed that although Bvckup will always resume the interrupted jobs, it will not resume exactly from where it left off, but start from scratch. In my test, I started to copy 3 files, two of 439MB and one from ~1GB, and after every interruption, Bvckup will resume the full job, even if it already had copied fully one of the three files.

By the way, I saw the job's log but there where no signal of what really happened between each newly started copying process, I mean, I didn't see any kind of messages informing about the process being interrupted anytime, neither because what reason (system was logged off, etc.).

Doequer :

Jan 24, 2018

Some remarks about my last comment?

Maybe some improvements could be implemented about the program's behavior while dealing with interrupted jobs.

Alex Pankratov :

Jan 24, 2018


You should see "Cancelling ..." message in the backup log. It is added whenever an active backup is stopped before completion either because of the user's request or because of some external factors such as a system reboot, program exit, external "stop" command, etc.

There are in fact no reason logged and it might indeed be a good idea to do that. Let me see what we can do.

Doequer :

Jan 26, 2018

I see, but take note that the only time when I see that "Cancelling ..." message you mention, is when I manually stop a job through the program's itself; there is nothing in the jobs' log telling me what happened between two different executions, which I'm able to notice by looking at the time lapse gap. I mean, not only there was no interruption's reasons mentioned in the log, but the interruptions themselves aren't mentioned neither.

And what about the jobs' specific status while an interruption affect them? I mean, in my tests, there were sometimes when after a sleep interruption, Bvckup just initiated it again, but from scratch, re-processing the already completed (copied) files. But other times, I saw that it did resume the interrupted job exactly when it was left off, but the files in that test weren't the same ones.

So, I'm wondering which is the expected behavior from Bvckup when dealing with interruptions, according to which one was the interruption, the amount of files, its kind and size , etc., in order it will determine to start from scratch or resume interrupted jobs?

Alex Pankratov :

Jan 26, 2018

I mean, not only there was no interruption's reasons mentioned in the log, but the interruptions themselves aren't mentioned neither.


This means that Bvckup2 process is killed by your operating system rather than being given a chance to exit gracefully. No idea why this is happening, but Windows had "about to shutdown" notifications since Windows NT times if not earlier, and Bvckup2 handles them as expected.

And what about the jobs' specific status while an interruption affect them? I mean, in my tests, there were sometimes when after a sleep interruption, Bvckup just initiated it again, but from scratch, re-processing the already completed (copied) files. But other times, I saw that it did resume the interrupted job exactly when it was left off, but the files in that test weren't the same ones.


I can't comment on this without seeing details. As I explained earlier sleeping/resuming is non-intrusive operation. If there was a backup running, it'll just keep on going after a resume.

So, I'm wondering which is the expected behavior from Bvckup when dealing with interruptions, according to which one was the interruption, the amount of files, its kind and size , etc., in order it will determine to start from scratch or resume interrupted jobs?


On shutdown it will cancel all jobs and exit. It will also remember which ones were active and will start them first IF they are due as per normal scheduling criteria. It will not re-run them simply because they were cancelled by a reboot.

On sleep/resume - no special handling.

Doequer :

Jan 30, 2018

I did some additional tests, you tell me please if Bvckup ran as expected in all cases, or there is rather something incorrect:

https://pastebin.com/ucQtZcm8

As you may see, I started the test job with 5 MKV files, but interrupted it with a "log off" while the second file was still being copied; and there was no other indication of the job being interrupted nor the exact reason, other than a "loaded" (line #92)  and a little time gap.

Then, when I logged in again, Bvckup resumed the job scratch, but delta copying the whole "test file #1" and partially the "test file #2", till it started copying its remaining part (~%75). Once again, I interrupted the job but this time with a "restart", while there was about "%90" of the file copied. After restarting, Bvckup didn't immediately continued the job because the media wasn't present, but once I connected it, it started right away (line #178); but to my surprise, it did it completely from scratch again, so I cancelled it.

Doequer :

Jan 30, 2018

The later case would repeat if there a system shutdown or even worst a power outage. I mean, in such cases I think the best would be if Bvckup could inform in the logs what exactly happened, i.e., if a determined job was interrupted, and because what reason. And then, trying to resume the incomplete processes but right from the point it left off, to avoid repeating the whole copying process unnecessarily, besides gaining some time.

And respect Bvckup not being given the chance of exiting gracefully, I don't know why that happens, since the system already does it with other programs, warning me before restarting/shutting down if there is still some active program/s which is/are preventing those actions to just being executed; in said cases, I can choose to go and wait for such programs to finish what they are doing, closing them manually or simply let the system close them in a forced way.

That situation isn't happening while there is some active Bvckup jobs, which in most cases will just keep working normally after the next system's resume, but I would still prefer to let it finish before sleeping than the contrary; and that's why currently I would have to look in the program's itself before sleeping the PC.

Alex Pankratov :

Jan 31, 2018

https://pastebin.com/ucQtZcm8


There's no "Cancelling" entry in the log. This means that your machine is forcibly killing Bvckup 2 process when it is shutting down instead of letting it to exit gracefully. I too have no idea why this is happening, but this is not a nominal Windows behavior.

I can have a look at the app's central log if you'd like. That'd be %LocalAppData%\Bvckup2\bvckup2.log. Though forward it to me privately, because there might be some information in it that is not worth sharing in public.

And then, trying to resume the incomplete processes but right from the point it left off


It is not set up to resume an interrupted _backup run_ and I don't think it's the right thing to do due to various edge cases where this would lead to spurious errors.

It can however resume copying of larger files in some cases, but it would depend on how exactly the preceding run was cancelled/aborted.

Doequer :

Feb 01, 2018

I repeated the test, which basically consisted in copying a "Test File" from a local hard disk to an USB removable drive. The results:

1- I interrupted it at ~ %25 with a restarting request, and although Windows took the order immediately, it showed me the "Shutting down..." message for a while (~2') before actually restarting. I thought Bvckup was the reason for that delay, since despite Windows didn't warned me about there was still a busy program before restart, it could be waiting for Bvckup to complete the task; but since the copying process started from scratch after rebooting, I don't know the waiting reason.

2- At approximately %40 of the previously restarted task, I shut down the system from the start menu; it was immediate, with no warnings and Bvckup returned the task from scratch at the next system start.

3- I leave the previously restarted task go for a while, and then put the system to sleep; in the next start, Bvckup didn't find the device, which might be normal or not, and there was some errors about that. There was no task resuming, and no percentage indicator but a countdown timer. There were some "out of sync" related error but I'm positive about nothing else interfering with the USB drive; and I even saw a message about the drive being "copy protected", although somehow it isn't present in the log I sent you.

4- Lastly, without leaving the previously started task to finish, I logged off/on the system; the task was restarted from scratch again, this time with a percentage indicator. Then I cancel it.

I have sent you the pertinent logs.

P.S. : After trying with the "service" mode, I started receiving this message:

https://i.imgur.com/ygFCZw4.png

Alex Pankratov :

Feb 01, 2018

Got your logs *and* I was able to reproduce the issue with the program getting forcibly killed by Windows on logoffs and restarts. It appears that something changed in the guts of Windows so it is now dramatically less patient with the programs when tearing down logged-in users' sessions.

The good news is that I ran some tests and there's still a way to ensure a clean(ish) program shutdown even in these cases. The next update will include these fixes and it will also add logging of the backup cancellation cause IF it is external to the program.

Thank you for your persistence with this issue and noticing and reporting it in the first place. I suggest we first check that logoffs/reboots are now detected and handled correctly in your setup, and then have a look at the sleeps/resumes. What you are describing above sounds like the device was not in fact ready, causing a backup to be aborted nearly immediately after the machine gets back from the sleep.

Re: https://i.imgur.com/ygFCZw4.png - this is likely due to having either any folders or files from %LocalAppData%\Bvckup2\engine folder being open with external programs (including Windows Explorer).

Alex Pankratov :

Feb 02, 2018

OK, the fix is now out as a part of 78.20. The update is available over at https://bvckup2.com/update and it should show up in the in-app updates shortly.

Doequer :

Feb 02, 2018

That's good to hear. When I thought about Bvckup dealing with interrupted tasks, it came to my mind what download managers do, i.e., resuming the downloads exactly from the last state they reached before a sudden power outage, manual stops, system halts, etc. But I bet given the Bvckup's more complex nature, it isn't that straightforward neither always possible to do that; or not all times, at least.

As you said, that error message was being caused by Windows explorer "locking" that folder. In such cases, I saw another "Bvckup" folder is created and then deleted from "x:\ProgramData" path. After I unlock the "engine" folder, I can switch normally, but a random "engine_131620394403590000 (backup)" like folder is left at "%LocalAppData%", besides the regular one.

I will do some other tests with the last version and report back.

Doequer :

Feb 07, 2018

On the one hand, Bvckups is resuming correctly the test job which previously failed to do, but just in case of restarts, logoffs and shutdowns; on the other hand, it always starts from scratch after sleeping the system. According to the backup's log, there are some related errors (file not found, path not found, drive not present), so I modified the default "delay_on_resume" with a "4:5" value and activate it through the program's GUI. But to no avail, since those errors still shows up after the system returns from every sleep; and for some reason, the job is never resumed because a forced destination re-scan, this last being caused by a "supposed" folder's modification by another entity.

I think that even when the destination drive isn't present right after the system wakes up, Bvckup should simply wait for it (which it already does), but for some unknown reason it always starts from scratch the previously interrupted job.

I sent you three pairs of logs, in case you need them:

1- Related to the sleep interruption issues.

2- What happened when I shut down Bvckup via the "Windows Task Manager", while a job was being executed. Not that me or any third-party program I use normally will cause this, but I thought off doing it to verify if the interrupted job resumed exactly from where it left off; it didn't. Besides, there was a weird error related to the "swap file", which made the job didn't start by itself, but after a few Bvckup's restarts. And no clues about what happened being present in the log, but a (in this specific case) short time gap.

3- What happened when there was a power outage (without UPS involved), while a job was being executed. The job wasn't resumed afterwards, and again there was no indications of a forced/sudden shutdown as the cause, in the job's log.

Doequer :

Feb 07, 2018

P.S.:

Although not frequents, I think that an abrupt program termination like that I did intentionally in my second test, might occur in case of hardware failures, like a system's disk crash, system's freezes because overheats, etc. So the  result of shutting down the process like that, would be probably the closest to expect as to what will really happen.

Regarding sleeps, after doing some additional tests but with 5 files (previous one was just with 1), I saw that Bvckup does resume the jobs from were it left off, but only does it with already completed files; the one that got interrupted will always start from the beginning next time. It seems for some reason it omits delta states after the system wakes up, and forces re-scans.

I even tried with the "Re-scan destination on every run", while maintaining the  "Use delta copying" option, and curiously enough there was at least one time when I saw the program resumed copying exactly from were it left off after a sleep, but the rest of times it behaves like I commented before.

Alex Pankratov :

Feb 08, 2018

"delay_on_resume"


This option controls if _new_ backups are delayed after reboots/resumes.

I think that even when the destination drive isn't present right after the system wakes up, Bvckup should simply wait for it (which it already does), but for some unknown reason it always starts from scratch the previously interrupted job.


Sleeping and waking up of the computer is largely transparent to all running programs. This is by design (of the OS). When a machine is put to sleep, the program merely ceases to execute. If it were doing ABCD, then its execution may be suspended after A, after B, C or D. It has no control where exactly it would be.

This means that if you sleep your machine when a program is reading a file, yank the drive with this file out and wake machine up, then the read request that was in process WILL fail. There's no way around it.

Moreover, the IO will fail in a way that indicates that there was a fundamental problem with the device, meaning that there's no point in retrying. Yanking a device completely invalidates all state associated with it - all open files, monitoring requests, etc. All of that needs to be re-setup and re-queried from scratch. It's basically impossible to just wait a bit, retry and proceed as if nothing happen.

Conversely, if you are to sleep the machine and then wake it up in the *exact* state it was before the sleep, all backups will just continue as if there were no sleep. This is the whole idea behind sleep/resume function - it is transparent to the program. But it does require things to NOT change while the box is sleeping.

---

With regards to being able to resume interrupted backups from their mid-way points (after killing a process or some such):

This is very hard to do cleanly *AND* this is not something that I think belongs to the backup software in the first place.

My priority for the program is to make consistent backups that behave simply and, therefore, predictably. What you are asking for is, essentially, to trade this for an ability to complete a running backup faster. This will significantly complicate the backup logic and introduce a multitude of edge failure cases. This is not an acceptable trade off.

Doequer :

Feb 15, 2018

My main concern was about the possibility of not interrupting already active jobs, when I put the system to sleep. I saw there are some unrelated programs (video editors, etc.) that already have the feature of "delaying" or directly avoiding the interruption that a "sleep/restart/logoff" request would suppose; and if they are delayed, then they will be processed only right after the active process has ended.

I still find it weird why it doesn't resume exactly from where it left off after a sleeping, but it does it for restarts, logoffs and shutdowns. But in case big files aren't involved, it's not a big deal.

And maybe a kind of generic indication could be added to the logs, telling about a job was interrupted out of nothing, in case of sudden power outages or an unexpected system crash.

New topic

Create
Made by Pipemetrics in Switzerland
Support


Follow
Twitter
Blog
Miscellanea Press resources
Testimonials
On robocopy
Company
Imprint

Legal Terms
Privacy