Re: Time Machine on Mac not detecting Data File Changes

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Time Machine on Mac not detecting Data File Changes

Benedict, Tom-2
Dave Nasralla writes:

>I'm not sure I'm following you. Are you saying any type of backup is
>useless (like Backblaze, Carbon Copy Cloner) if 4D Server is running
>at the time of backup?

>How do you backup your 4D Systems?

The only reliable way we have found to back up a 'live' 4D Server database is to set up a 4D Server mirror. Then that mirror can run 4D Backup periodically. That 4D Backup archive can then be included in your enterprise backup system.

I know a lot of folks claim they have no problems running back up software on a 'live' 4D data file. I think they have been lucky.

Tom Benedict | Optum

This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

RE: Time Machine on Mac not detecting Data File Changes

Stephen J. Orth
Dave,

In addition to Tom's experience, which we agree with, it is our experience running any type of non-4D backup software on a live 4D Server will cause data corruption to the live system.  Don't ask me how this happens, but it has been our experience.

Steve

*********************************************
  Stephen J. Orth                                                
  The Aquila Group, Inc.         Office:  (608) 834-9213
  P.O. Box 690                           Mobile:  (608) 347-6447
  Sun Prairie, WI 53590

  E-Mail:  [hidden email]
*********************************************

-----Original Message-----
From: 4D_Tech [mailto:[hidden email]] On Behalf Of Benedict, Tom
Sent: Thursday, July 07, 2016 9:53 AM
To: [hidden email]; [hidden email]
Subject: Re: Time Machine on Mac not detecting Data File Changes


The only reliable way we have found to back up a 'live' 4D Server database is to set up a 4D Server mirror. Then that mirror can run 4D Backup periodically. That 4D Backup archive can then be included in your enterprise backup system.

I know a lot of folks claim they have no problems running back up software on a 'live' 4D data file. I think they have been lucky.

Tom Benedict | Optum

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Time Machine on Mac not detecting Data File Changes

madamov
In reply to this post by Benedict, Tom-2

> On 07.07.2016., at 16.52, Benedict, Tom <[hidden email]> wrote:
>
> Dave Nasralla writes:
>
>> I'm not sure I'm following you. Are you saying any type of backup is
>> useless (like Backblaze, Carbon Copy Cloner) if 4D Server is running
>> at the time of backup?
>
>> How do you backup your 4D Systems?
>
> The only reliable way we have found to back up a 'live' 4D Server database is to set up a 4D Server mirror. Then that mirror can run 4D Backup periodically. That 4D Backup archive can then be included in your enterprise backup system.

No need to do this for reliability reasons. If you can live with “4D Backup is running and you can’t save records” or your applications does not need to run 24/7, there is no need for mirroring, the best way is just to use 4D Backup. Mirroring is complex thing to setup and test, there is code that needs to be written, a reliable method of transferring journal files needs to be developed and tested. Chihab and me had a session about mirroring at Summit, it is avaliable in English and French to Partners, please take a look at it.
>
> I know a lot of folks claim they have no problems running back up software on a 'live' 4D data file. I think they have been lucky.
>
>

Agreed. Very lucky.

Milan


**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Time Machine on Mac not detecting Data File Changes

Charles Miller
In reply to this post by Benedict, Tom-2
On Thu, Jul 7, 2016 at 10:52 AM, Benedict, Tom <[hidden email]>
wrote:

> The only reliable way we have found to back up a 'live' 4D Server database
> is to set up a 4D Server mirror. Then that mirror can run 4D Backup
> periodically. That 4D Backup archive can then be included in your
> enterprise backup system.


I agree 100%. I have not found any backup software that can backup a live
(i.e. running 4D system). One never knows about cache and what as been
written to disk. I had one customer who insisted it would work and so did
not implement 4D backup. They changed their minds when they had a crash and
tried to use the previous time machine backup and it could not be opened. I
wen in was able to write code to get all data put and into a new data file.
I used DataCheck (too bad no longer a product) to export all data it found.
Needless to say they now use 4D backup.

Regards

Chuck


--
-----------------------------------------------------------------------------------------
 Chuck Miller Voice: (617) 739-0306 Fax: (617) 232-1064
 Informed Solutions, Inc.
 Brookline, MA 02446 USA Registered 4D Developer
       Providers of 4D, Sybase & SQL Sever connectivity
          http://www.informed-solutions.com
-----------------------------------------------------------------------------------------
This message and any attached documents contain information which may be
confidential, subject to privilege or exempt from disclosure under
applicable law.  These materials are intended only for the use of the
intended recipient. If you are not the intended recipient of this
transmission, you are hereby notified that any distribution, disclosure,
printing, copying, storage, modification or the taking of any action in
reliance upon this transmission is strictly prohibited.  Delivery of this
message to any person other than the intended recipient shall not
compromise or waive such confidentiality, privilege or exemption
from disclosure as to this communication.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

RE: Time Machine on Mac not detecting Data File Changes

Stephen J. Orth
I will add one point to this, something we hear all the time from our customers.  Let me preface this by saying I'm no expert when it comes to backup software for non-4D systems (i.e. SQL databases).

However, we continually hear how bad our database is because its considered a "locking" backup.  They tell us this is not how it works in the "real world", meaning if we were using any other database system, 3rd party backup software could backup the system live, ensuring no one is locked out.

I have a hard time believing this, but I don't have any evidence to counter their claims.


Steve


-----Original Message-----
From: 4D_Tech [mailto:[hidden email]] On Behalf Of Charles Miller
Sent: Thursday, July 07, 2016 11:17 AM
To: 4D iNug Technical <[hidden email]>
Subject: Re: Time Machine on Mac not detecting Data File Changes


I agree 100%. I have not found any backup software that can backup a live
(i.e. running 4D system). One never knows about cache and what as been
written to disk. I had one customer who insisted it would work and so did
not implement 4D backup. They changed their minds when they had a crash and
tried to use the previous time machine backup and it could not be opened. I
wen in was able to write code to get all data put and into a new data file.
I used DataCheck (too bad no longer a product) to export all data it found.
Needless to say they now use 4D backup.

Regards

Chuck

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

RE: Time Machine on Mac not detecting Data File Changes

Chip Scheide
I have used 2 different piece of software for backups
Retrospect -
Bru -

Both of these software applications had an 'add-on', which cost at
least as much as the original software, to allow these systems to
backup 'live' SQL systems.
 
If I recall correctly - Retrospect multi-SQL server backup was $x000
while the software was $x00

Without these add-ons Windows will not allow the SQL data file to be
backed up -
generating file is Locked or in use error(s)

I do the same thing with my SQL systems, as I do for 4D - I use the
enterprise manager to create a local backup, then backup that copy and
avoid the live system.

On Thu, 7 Jul 2016 11:24:50 -0500, Stephen J. Orth wrote:

> I will add one point to this, something we hear all the time from our
> customers.  Let me preface this by saying I'm no expert when it comes
> to backup software for non-4D systems (i.e. SQL databases).
>
> However, we continually hear how bad our database is because its
> considered a "locking" backup.  They tell us this is not how it works
> in the "real world", meaning if we were using any other database
> system, 3rd party backup software could backup the system live,
> ensuring no one is locked out.
>
> I have a hard time believing this, but I don't have any evidence to
> counter their claims.
>
>
> Steve
>
>
> -----Original Message-----
> From: 4D_Tech [mailto:[hidden email]] On Behalf Of
> Charles Miller
> Sent: Thursday, July 07, 2016 11:17 AM
> To: 4D iNug Technical <[hidden email]>
> Subject: Re: Time Machine on Mac not detecting Data File Changes
>
>
> I agree 100%. I have not found any backup software that can backup a live
> (i.e. running 4D system). One never knows about cache and what as been
> written to disk. I had one customer who insisted it would work and so did
> not implement 4D backup. They changed their minds when they had a crash and
> tried to use the previous time machine backup and it could not be opened. I
> wen in was able to write code to get all data put and into a new data file.
> I used DataCheck (too bad no longer a product) to export all data it found.
> Needless to say they now use 4D backup.
>
> Regards
>
> Chuck
>
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Time Machine on Mac not detecting Data File Changes

Benedict, Tom-2
In reply to this post by Benedict, Tom-2
Milan Adamov writes:

>> On 07.07.2016., at 16.52, Benedict, Tom <[hidden email]<mailto:[hidden email]>> wrote:
>> The only reliable way we have found to back up a 'live' 4D Server database is to set up a 4D Server mirror. Then that mirror can run 4D Backup periodically. That 4D Backup archive can then be included in your enterprise backup system.

>No need to do this for reliability reasons. If you can live with "4D Backup is running and you
>can't save records" or your applications does not need to run 24/7, there is no need for
>mirroring, the best way is just to use 4D Backup.

I agree. If you don't need 24x7 uptime, 4D Backup is the best and easiest choice.

>Mirroring is complex thing to setup and
>test, there is code that needs to be written, a reliable method of transferring journal files
>needs to be developed and tested. Chihab and me had a session about mirroring at Summit,
>it is available in English and French to Partners, please take a look at it.

The session notes are great. That solution works, but, as you say, it takes considerable effort to deploy.

4D should provide a rock solid mirroring solution that is built into the product. We spend way too much time maintaining our mirroring operation. It's very 'touchy' and susceptible to breakage.

Tom Benedict | Optum



This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

RE: Time Machine on Mac not detecting Data File Changes

Benedict, Tom-2
In reply to this post by Benedict, Tom-2
Stephen J. Orth writes:

>However, we continually hear how bad our database is because its considered a "locking" backup.
>They tell us this is not how it works in the "real world", meaning if we were using any other
>database system, 3rd party backup software could backup the system live, ensuring no one is locked out.

We address that through mirroring. Granted, the overhead of having a mirror server running is significant. Another reason for 4D to "build-in" mirror support. Seems like a new product which is a basic 4D daemon service which could be spun up on the same server hardware (or VM) that does nothing but integrate log files from the mother data file into the mirror data file would be a very nice addition to the 4D Product line. It has "enterprise support" written all over it.

Tom Benedict | Optum

This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Time Machine on Mac not detecting Data File Changes

Jeffrey Kain
In reply to this post by Stephen J. Orth
Steve - any reason you haven't set up a mirror? It's very easy and very reliable, and your main system will never be offline for a backup. It greatly simplifies maintenance as well, since you can run your maintenance on the mirror (compact, repair, etc.) and then move the mirror into production with hardly any downtime.

That said, there's no reason one of these 3rd party backup solutions shouldn't work, although you face the potential to lose unflushed data in the cache if your server crashes. This shouldn't happen with a mirror.

> On Jul 7, 2016, at 12:24 PM, Stephen J. Orth <[hidden email]> wrote:
>
> However, we continually hear how bad our database is because its considered a "locking" backup.  They tell us this is not how it works in the "real world", meaning if we were using any other database system, 3rd party backup software could backup the system live, ensuring no one is locked out.

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

RE: Time Machine on Mac not detecting Data File Changes

Stephen J. Orth
Jeff,

No reason other than the Customers are cheap and don't want to setup a second box for the mirror.  We have one new customer coming up that is making a much bigger stink about this than the others, so we will see if they put their money where their mouth is.

As far as 3rd party products, we NEVER WANT a customer using a third part product because (in our experience) they have corrupted the production database.  Like everyone else chiming in, we use 4D backup nightly to perform a full back up, then tell our customers to backup this.

Steve

-----Original Message-----
From: 4D_Tech [mailto:[hidden email]] On Behalf Of Jeffrey Kain
Sent: Thursday, July 07, 2016 12:28 PM
To: 4D iNug Technical <[hidden email]>
Subject: Re: Time Machine on Mac not detecting Data File Changes

Steve - any reason you haven't set up a mirror? It's very easy and very reliable, and your main system will never be offline for a backup. It greatly simplifies maintenance as well, since you can run your maintenance on the mirror (compact, repair, etc.) and then move the mirror into production with hardly any downtime.

That said, there's no reason one of these 3rd party backup solutions shouldn't work, although you face the potential to lose unflushed data in the cache if your server crashes. This shouldn't happen with a mirror.

> On Jul 7, 2016, at 12:24 PM, Stephen J. Orth <[hidden email]> wrote:
>
> However, we continually hear how bad our database is because its considered a "locking" backup.  They tell us this is not how it works in the "real world", meaning if we were using any other database system, 3rd party backup software could backup the system live, ensuring no one is locked out.


**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Time Machine on Mac not detecting Data File Changes

Keith White
In reply to this post by Benedict, Tom-2
Hi

I created a feature request about 2 years ago:-

http://forums.4d.fr/Post/EN/14916413/1/14916414#14916414

For 4D Server to become aware of the need to flush cache and then quiesce while a snapshot happens.  This may require specific Time Machine related code for Mac OS, but my original request was for a Windows VSS Writer.

Best regards

Keith White
Synergist Express Ltd, UK.

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Time Machine on Mac not detecting Data File Changes

Charles Miller
In reply to this post by Stephen J. Orth
There are backup systems that work on the bit level of the drive. I would not trust them for SQL server either. Supposedly every time a bit on HD changes it is moved to backup. These systems are really expensive and I have only seen them on windows boxes. I have one old customer that was looking into this. NO idea if they have used it or ever had to do a restore of DB. It does however work great for document files.

Regards

Chuck
------------------------------------------------------------------------------------------------
 Chuck Miller Voice: (617) 739-0306
 Informed Solutions, Inc. Fax: (617) 232-1064      
mailto:miller.cjay<AT SIGN>gmail.com <http://gmail.com/>
 Brookline, MA 02446 USA Registered 4D Developer                
       Providers of 4D and Sybase connectivity
          http://www.informed-solutions.com 
------------------------------------------------------------------------------------------------


> On Jul 7, 2016, at 12:24 PM, Stephen J. Orth <[hidden email]> wrote:
>
> I will add one point to this, something we hear all the time from our customers.  Let me preface this by saying I'm no expert when it comes to backup software for non-4D systems (i.e. SQL databases).
>
> However, we continually hear how bad our database is because its considered a "locking" backup.  They tell us this is not how it works in the "real world", meaning if we were using any other database system, 3rd party backup software could backup the system live, ensuring no one is locked out.
>
> I have a hard time believing this, but I don't have any evidence to counter their claims.
>
>
> Steve

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Time Machine on Mac not detecting Data File Changes

Benedict, Tom-2
In reply to this post by Benedict, Tom-2
Keith White writes

>I created a feature request about 2 years ago:-
>
>http://forums.4d.fr/Post/EN/14916413/1/14916414#14916414
>
>For 4D Server to become aware of the need to flush cache and then quiesce while a snapshot happens.  This may
>require specific Time Machine related code for Mac OS, but my original request was for a Windows VSS Writer.

Third party back up tools can be used, but they need to be controlled by 4D. Schedule a 4D Backup, then trap On Backup Startup and sublaunch the third party tool. Watch for evidence that the third party tool has 'finished' then pass a 1 to 4D. That will cause 4D Backup to not run and pass execution back to 4D Server so users can save records. In the meantime, users will see the "Backup is running" window.

Tom

This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Time Machine on Mac not detecting Data File Changes

John DeSoi
In reply to this post by Stephen J. Orth
This is completely true for PostgreSQL and probably most databases that support multi version concurrency control (MVCC). You can create consistent backup of the database no matter what is happening, including changes to the table structure. See

https://en.wikipedia.org/wiki/Multiversion_concurrency_control

https://en.wikipedia.org/wiki/List_of_databases_using_MVCC

John DeSoi, Ph.D.


> On Jul 7, 2016, at 11:24 AM, Stephen J. Orth <[hidden email]> wrote:
>
> I will add one point to this, something we hear all the time from our customers.  Let me preface this by saying I'm no expert when it comes to backup software for non-4D systems (i.e. SQL databases).
>
> However, we continually hear how bad our database is because its considered a "locking" backup.  They tell us this is not how it works in the "real world", meaning if we were using any other database system, 3rd party backup software could backup the system live, ensuring no one is locked out.
>
> I have a hard time believing this, but I don't have any evidence to counter their claims.

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

RE: Time Machine on Mac not detecting Data File Changes

Stephen J. Orth
John,

Very interesting....so I guess this begs the question why 4D is not supporting this?  The list of databases using MVCC is rather large.


Steve

-----Original Message-----
From: 4D_Tech [mailto:[hidden email]] On Behalf Of John DeSoi
Sent: Thursday, July 07, 2016 2:32 PM
To: 4D iNug Technical <[hidden email]>
Subject: Re: Time Machine on Mac not detecting Data File Changes

This is completely true for PostgreSQL and probably most databases that support multi version concurrency control (MVCC). You can create consistent backup of the database no matter what is happening, including changes to the table structure. See

https://en.wikipedia.org/wiki/Multiversion_concurrency_control

https://en.wikipedia.org/wiki/List_of_databases_using_MVCC

John DeSoi, Ph.D.

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Time Machine on Mac not detecting Data File Changes

Alan Chan
In reply to this post by Benedict, Tom-2
I can't agree more. 4D built-in mirroring would be the ultimate solution.
Full backup is not long term solution regardless from 4D or third party
due to growing data size.

Alan Chan

4D iNug Technical <[hidden email]> writes:
>4D should provide a rock solid mirroring solution that is built into the
>product. We spend way too much time maintaining our mirroring operation.
>It's very 'touchy' and susceptible to breakage.


**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Time Machine on Mac not detecting Data File Changes

Dave Nasralla
In reply to this post by Stephen J. Orth
Hey All,

I appreciate the discussion. For the record, we do use 4D Backup.
Right now we run it on the weekends, and push the Backup file to the
cloud. We rely on Time Machine and Carbon Copy Cloner during the week.
In theory, you could recover the Backup File and integrate the log.

This keeps me from having to push Gigs of data to the cloud every night.

The problem I am seeing is that the .journal file (Log) is never
closed, so, technically you could have issues.

Something no one has mentioned is to use the New Log File command. In
Summit Sessions it's used in the context of Mirroring. You can also
use it in the context of simply backing up with 3rd party software. I
would think the command would have very little system overhead since
it is simply closing a file, renaming it, and starting a new blank log
file.

In this scenario, the third party backups will easily backup the newly
closed log files. I could even run it every couple of hours during the
busy times and not bother at night. The log files (which only
represent changes) are then sent to the cloud every night.

This is probably the direction I'll head. I could do 4D Backup every
night, but then it complicate the cloud stuff.

Thanks for everyones comments.

dave




On Thu, Jul 7, 2016 at 1:51 PM, Stephen J. Orth
<[hidden email]> wrote:

> John,
>
> Very interesting....so I guess this begs the question why 4D is not supporting this?  The list of databases using MVCC is rather large.
>
>
> Steve
>
> -----Original Message-----
> From: 4D_Tech [mailto:[hidden email]] On Behalf Of John DeSoi
> Sent: Thursday, July 07, 2016 2:32 PM
> To: 4D iNug Technical <[hidden email]>
> Subject: Re: Time Machine on Mac not detecting Data File Changes
>
> This is completely true for PostgreSQL and probably most databases that support multi version concurrency control (MVCC). You can create consistent backup of the database no matter what is happening, including changes to the table structure. See
>
> https://en.wikipedia.org/wiki/Multiversion_concurrency_control
>
> https://en.wikipedia.org/wiki/List_of_databases_using_MVCC
>
> John DeSoi, Ph.D.
>
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************



--
David Nasralla
Clean Air Engineering
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Time Machine on Mac not detecting Data File Changes

Bill Weale
In reply to this post by Benedict, Tom-2
There’s another variation to this approach. I’m quite sure that on backup startup and on backup shutdown are triggered at a time when the datafile is in a backup “state,” meaning no loaded and modified records, no open transactions, cache flushed to disk, etc. During those events one can run pretty much any file manipulation code, including native 4D commands. (There’s alway the issue, particularly on Windows, of copying out to remote volumes, though.)

As for the .journal file always being open. It’s a journal file. Theoretically the events are written to disk in the order they occurred. So, again theoretically, its contents should be good up until the point they aren’t—that point being an incomplete or corrupted journal entry, which the restore program should be able to identify and end the restore.

Theoretically…

Bill


William W. Weale

Business Owners Support, LLC.

Operations Analysis
MIS Advising
Decision Support Systems

> On Jul 7, 2016, at 3:14 PM, Benedict, Tom <[hidden email]> wrote:
>
> Third party back up tools can be used, but they need to be controlled by 4D. Schedule a 4D Backup, then trap On Backup Startup and sublaunch the third party tool. Watch for evidence that the third party tool has 'finished' then pass a 1 to 4D. That will cause 4D Backup to not run and pass execution back to 4D Server so users can save records. In the meantime, users will see the "Backup is running" window.

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Time Machine on Mac not detecting Data File Changes

Mitchell Shiller
In reply to this post by Benedict, Tom-2
Dave,

We have similar size Server setup from you. ( Our data file isn't super big but we use it as a document management system with hundreds of thousands of documents stored on the Server outside the data file.)
This is what we do. We are Mac only environment.

4D Server runs on a Mac mini.
Data file on external Thunderbolt enclosure set up as RAID 1+0.
We use Thunderbay 4 mini from OWC but there are other options that have hardware RAID. A disk less enclosure will run about $350 including SoftRaid software.  Buy prograde SSD's. Since the storage requirements for the data file is relatively small, the smallest SSD (125-250 Gb) will suffice.
Data transfer speeds are extremely fast with Thunderbolt 2.

Yes this is an additional cost. But SSDs do fail.  The cost of the redundancy is less than the headache and downtime of a failed disk.

Another alternative is to swap out the SSDs  on an annual? Or bi-annual basis depending on the grade of SSD.

I put the journal file on a second Thunderbolt drive which is RAID 1.

We use 4D backup to backup nightly to the second drive.
When that is done, we use commercial software like CCC to backup the entire second drive ( .BKs and .BLs) to an encrypted portable disk. We have one for everyday of the week. These are rotated offsite.

We upload the log files and .bks to the cloud daily. Do not use a commercial cloud service, most throttle the upload, are so low to retrieve and work best with small files I.e. Not 4D.  Setup your own cloud and SFTP it.

I even secure the Thunderbolt connectors to the back of the Mac mini with electrical tape. ( Yes, I have been burnt by someone accidentally disconnecting a drive.)

I know this may sound a bit paranoid but over the last 25+ years I have seen all kinds of failures. And for some reason they seem to happen when I am on vacation and after a good bottle of wine. ( Maybe the wine keeps me calm as I try to walk my inexperienced staff through the steps to get us up and going again.

I have been saved a few times by the last redundant step. So to me you can never be too safe.

Good luck.

Mitch

Sent from my iPad
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Time Machine on Mac not detecting Data File Changes

Dave Nasralla
Hi Mitch,

Thanks for sharing your experience. Your system is similar to ours in
that our data file is under 5GB, but I store over 90 GB of files
(mostly .pdfs) outside of the datafile. I had a full size 4 bay
Thunderbay from OWC running a traditional mirrored Raid on two bays.

After 18 months, the second bay failed. The failure was hard to track
down, but I was able to repeat it by trying to copy a 90GB folder from
one bay to another. It would fail every time, so we replaced the
Thunderday and all seems well. (FWIW - did it through a VISA extended
warranty that comes with the card.)

Our SSDs are the Samsung Pro Series drives but not the Intel high end
ones. Remarkably fast.

dave

On Tue, Jul 12, 2016 at 1:35 PM, Mitchell Shiller <[hidden email]> wrote:

> Dave,
>
> We have similar size Server setup from you. ( Our data file isn't super big but we use it as a document management system with hundreds of thousands of documents stored on the Server outside the data file.)
> This is what we do. We are Mac only environment.
>
> 4D Server runs on a Mac mini.
> Data file on external Thunderbolt enclosure set up as RAID 1+0.
> We use Thunderbay 4 mini from OWC but there are other options that have hardware RAID. A disk less enclosure will run about $350 including SoftRaid software.  Buy prograde SSD's. Since the storage requirements for the data file is relatively small, the smallest SSD (125-250 Gb) will suffice.
> Data transfer speeds are extremely fast with Thunderbolt 2.
>
> Yes this is an additional cost. But SSDs do fail.  The cost of the redundancy is less than the headache and downtime of a failed disk.
>
> Another alternative is to swap out the SSDs  on an annual? Or bi-annual basis depending on the grade of SSD.
>
> I put the journal file on a second Thunderbolt drive which is RAID 1.
>
> We use 4D backup to backup nightly to the second drive.
> When that is done, we use commercial software like CCC to backup the entire second drive ( .BKs and .BLs) to an encrypted portable disk. We have one for everyday of the week. These are rotated offsite.
>
> We upload the log files and .bks to the cloud daily. Do not use a commercial cloud service, most throttle the upload, are so low to retrieve and work best with small files I.e. Not 4D.  Setup your own cloud and SFTP it.
>
> I even secure the Thunderbolt connectors to the back of the Mac mini with electrical tape. ( Yes, I have been burnt by someone accidentally disconnecting a drive.)
>
> I know this may sound a bit paranoid but over the last 25+ years I have seen all kinds of failures. And for some reason they seem to happen when I am on vacation and after a good bottle of wine. ( Maybe the wine keeps me calm as I try to walk my inexperienced staff through the steps to get us up and going again.
>
> I have been saved a few times by the last redundant step. So to me you can never be too safe.
>
> Good luck.
>
> Mitch
>
> Sent from my iPad
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************



--
David Nasralla
Clean Air Engineering
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************