[Off] Remote backup

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[Off] Remote backup

4D Tech mailing list
Hi Jeff,

This is a great idea, I had not thought of it.  Let me repeat it back to
see if I have it:

[Server#1] Runs 4D Server that users are actively using.  Code in a loop
on the 4D Server calls "New Log File" every 10 minutes.  Code in the loop
then calls LEP to a script to transfer the last journal file to
[Server#2].

[Server#2] Runs 4D Server that users aren't aware of.  Code in a loop on
the 4D Server detects that a complete journal file has arrived (as it will
every ten minutes, my uncertainty is how it knows the file transfer is
complete) and integrates it using INTEGRATE MIRROR LOG FILE.  Code in the
loop then calls LEP to a script to transfer this journal file to
[Server#3].  4D Backup runs on this machine and sends its backup file to
[Server#3]

[Server#3] This is a repository, nothing is actively running on it, it
could be iCloud, Amazon S3, etc.


Do I have this basically correct?

Then some technical questions:

It would seem that Server#1 never runs 4D Backup, because any restoration
would be done using Server#2.  Nevertheless, if Server#1 started up and
somehow choked, a "normal" behavior would be to restore from backup and
integrate the log file.  Since this behavior would be turned off, what is
the procedure you use if Server #1 won't start?  Do you copy the mirror on
Server #2 to Server#1?

How do you handle the timing, so that [Server#2] knows that the file
transfer is complete and it is OK to integrate the log file?  Does
[Server#1] somehow send a 'I'm Done' message to [Server#2] in your
implementation?

What do you do if [Server#1] dies to now point users to [Server#2],
assuming the Server application is a built application so that the server
address and port number are built into the (built) 4D Client?

Many thanks!

Bob Miller
Chomerics, a division of Parker Hannifin Corporation



**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: [Off] Remote backup

4D Tech mailing list
Yes - you have it correct.

> It would seem that Server#1 never runs 4D Backup, because any restoration
> would be done using Server#2.  Nevertheless, if Server#1 started up and
> somehow choked, a "normal" behavior would be to restore from backup and
> integrate the log file.  Since this behavior would be turned off, what is
> the procedure you use if Server #1 won't start?  Do you copy the mirror on
> Server #2 to Server#1?

- Yes, you'll never back up server #1 and you'll disable the automatic restore after crash feature.

- If server 1 crashes and you bring it up, 4D analyzes its current journal file and if any operations are not integrated in the data file, they are automatically integrated.

- If server 1 won't start (i.e. damage detected or the journal won't integrate for some reason), you'll need a recovery procedure.

> How do you handle the timing, so that [Server#2] knows that the file
> transfer is complete and it is OK to integrate the log file?  Does
> [Server#1] somehow send a 'I'm Done' message to [Server#2] in your
> implementation?

You could send a message from #1 to #2. We just use an old-fashioned lock file. Write the lock file, copy the journal file, then delete the lock file. You could also copy the file with a .tmp extension, and when done rename .tmp to .journal (and your code would filter out .tmp files to prevent them from integrating).

In our server, we ship a new journal file every 30 seconds. They are numbered sequentially, so you can sort them by file name and integrate a batch in the correct order.

> What do you do if [Server#1] dies to now point users to [Server#2],
> assuming the Server application is a built application so that the server
> address and port number are built into the (built) 4D Client?

Change the DNS entry, or manually copy the data file from #2 back to #1. We do the latter. It happens so infrequently that 10 minutes to recover isn't a big deal.

Jeff

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: [Off] Remote backup

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Bob Miller writes:

>[Server#1] Runs 4D Server that users are actively using.  Code in a loop on the 4D Server calls
>"New Log File" every 10 minutes.  Code in the loop then calls LEP to a script to transfer the last journal file to [Server#2].
>
>[Server#2] Runs 4D Server that users aren't aware of.  Code in a loop on the 4D Server detects
>that a complete journal file has arrived (as it will every ten minutes, my uncertainty is how it knows the file transfer is
>complete) and integrates it using INTEGRATE MIRROR LOG FILE.  Code in the loop then calls LEP
>to a script to transfer this journal file to [Server#3].  4D Backup runs on this machine and sends its backup file to [Server#3]

>[Server#3] This is a repository, nothing is actively running on it, it could be iCloud, Amazon S3, etc.

>Do I have this basically correct?

Yes.

>Then some technical questions:

>It would seem that Server#1 never runs 4D Backup, because any restoration
>would be done using Server#2.  Nevertheless, if Server#1 started up and
>somehow choked, a "normal" behavior would be to restore from backup and
>integrate the log file.  Since this behavior would be turned off, what is
>the procedure you use if Server #1 won't start?  Do you copy the mirror on
>Server #2 to Server#1?

Yes.

>How do you handle the timing, so that [Server#2] knows that the file
>transfer is complete and it is OK to integrate the log file?  Does
>[Server#1] somehow send a 'I'm Done' message to [Server#2] in your
>implementation?

I think that 4D will complain that the log file is 'busy'. I must confess I've never encountered this case in 10+ years of running multiple mirrors. If I recall correctly, our system copies the file with a 'temp' name, then when the copy is 'done' it changes the name. That limits the exposure considerably. It's pretty much an 'instant' name change, so the mirror never tries to integrate a 'busy' file.

>What do you do if [Server#1] dies to now point users to [Server#2],
>assuming the Server application is a built application so that the server
>address and port number are built into the (built) 4D Client?

There are a number of ways to deal with this on the hostname/DNS level. We've never done this. Surprisingly, we never had any hardware failures when we used dedicated hardware and since moving to SAN/VM there have been a few drive failures, but they never affected operations due to the redundancy and 'auto-move' features of VM.

BTW, there is a 4D Tech Note which includes all the code that you can need to set this up. But it only supports file share or web service as the file transport mechanism. Easy enough to do the LEP RoboCopy or whatever instead though.

Enterprise systems can't afford to not be mirrored. There are lots of white papers available which outline DR scenarios and potentials solutions for enterprise. Less than enterprise systems should seriously look at the very modest cost of deploying a 4D mirror. A few thousand dollar investment will provide a viable DR solution. Business interruptions due to natural disasters are very expensive to recover from. Being able to assure your business managers that the data that they depends on is safe, secure and available to deploy anywhere they need it is extremely compelling. Setting up a 4D Mirror is a simple, low cost way to satisfy that requirement.

I've advocated that 4D should build the mirroring capability into 4D Server, rather than providing a component. That would raise it to higher level of support and more systems would use mirroring. (Maybe they have? I haven't been following the v16 feature set closely, as I'm stuck on v13.x)

HTH,

Tom Benedict
Optum, Inc


This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: [Off] Remote backup

4D Tech mailing list
Don't use 4D's code.  Read the tech note to understand the concepts, but that code is not good for use in the real world.

And don't put your mirror backup code in a component either -- you'll want to prevent your mirror from starting a backup in the middle of an integration, and the only way to do this is with a semaphore. And you can't share a semaphore between the host and the component.

--
Jeffrey Kain
[hidden email]

> On Sep 12, 2017, at 4:01 PM, Benedict, Tom via 4D_Tech <[hidden email]> wrote:
>
> there is a 4D Tech Note which includes all the code that you can need to set this up. But it only supports file share or web service as the file transport mechanism. Easy enough to do the LEP RoboCopy or whatever instead though.

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************