4D Backup causes 4D Remote to hang (15.4)

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

4D Backup causes 4D Remote to hang (15.4)

4D Tech mailing list
Changing the thread title of "4D Remote hanging at 3am after upgrading to Hi Sierra. Need logging advice."

4D 15.4
Mac OS High Sierra Client
Windows 2008r2 Server

As noted in previous posts I now suspect that a 5am (formerly 3am) scheduled 4D Backup is causing 4D to 'hang'.
I checked my phone at 5:30 this morning and saw a message that our 4D-based web app was not responding.
Normally, this would be followed by a message that it recovered. There would also be messages if any action was taken to reboot 4D. There were neither.

I remotely logged into the client machine. Finder is frontmost. There are no error dialogs visible.
4D is running. This is an unattended client (web server) so no UI is visible.
I click on the 4D icon in the dock. A "backup is running" dialog appears in the lower left hand corner of the screen. Its status bar is at 1% and not moving.
I dismiss the dialog and check Activity Monitor.
4D is not responding and is consuming 99% of the CPU. Memory usage is NOT abnormal and much lower than many other running apps.
Click 4D again and dialog doesn't appear. Try this a few more times and can't make dialog reappear. I wanted to see if there has been any progress.
Recheck Activity Monitor. 4D is still not responding and is consuming 99% of the CPU. Force quit 4D. Scripts that monitor 4D restart 4D within a minute.

Again, this started when the client was updated to High Sierra 3 weeks ago. That was mandated by their institution's cybersecurity people. 4D has hung almost every night. Controls in place to keep 4D running stopped working after the upgrade.

When I first remembered the backup was running at 3am (now 5am) I ran a backup manually from the server admin console. It ran fine and there were no apparent problems. I even watched the 4D remote. Checks of the data file and indexes show no problems.

What should I try next? Start with a new backup? Reinstall 4D?

Thanks,

Brad Perkins


**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Backup causes 4D Remote to hang (15.4)

4D Tech mailing list
Are there database requests coming in via the web server at the time of the backup? Have you tried turning off the web server during backup? Or at least return an error response if access to the database is needed when backup is running.

John DeSoi, Ph.D.

> On Nov 14, 2017, at 8:40 AM, Perkins, Bradley D via 4D_Tech <[hidden email]> wrote:
>
> What should I try next? Start with a new backup? Reinstall 4D?

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Backup causes 4D Remote to hang (15.4)

4D Tech mailing list
In reply to this post by 4D Tech mailing list
On Nov 14, 2017, at 9:40 AM, Perkins, Bradley D via 4D_Tech <[hidden email]> wrote:

> What should I try next? Start with a new backup? Reinstall 4D?

Personally, I would take the easy way out and change the script that is supposed to quit/restart 4D to something like:

killall -9 4D

which will not be deterred by error alerts or processes that are not responding to normal signals.

If there are problems because 4D was quit by the OS instead of being asked nicely, follow it up with a full system restart, although that will require running the script with root privileges (probably not a problem, just run the killing part using cron as root, and the restarting part using launchd as the 4D user).

Side story: my client was running v15 Server on macOS 10.12, and suffering from mass client disconnects. As the number of users approached 25, 3 or 4 times per day 10-15 users would be disconnected from the server at once, with no consistency among users disconnected. While trouble-shooting, I ran a constant ping from another server to the 4D server, which shows several seconds of dropped packets corresponding to disconnect events. I then ran a ping on the 4D machine to another server, and had no more disconnect events (until someone restarted the computer and didn’t restart the ping command in Terminal). Obviously 4D wasn’t “fixed”; something in 4D server was killing the network stack in some way and all that code was being executed the same way as before. However, the actual problem went away and that’s really all that mattered.

Note to self: perhaps an automated weekly restart of 4D Server is the trick to avoiding the occasional v16x DELAY PROCESS hang.  When a hammer works, might as well use it.

Jim Crate

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Backup causes 4D Remote to hang (15.4)

4D Tech mailing list
I would quit 4D server, delete preferences and recreate using menu commands
re backup. Perhaps that us trhe problem. Also check where the backup is
going to if a HD try a different location

Regards and hiope this helps
Chuck

On Tue, Nov 14, 2017 at 10:25 AM, Jim Crate via 4D_Tech <
[hidden email]> wrote:

> On Nov 14, 2017, at 9:40 AM, Perkins, Bradley D via 4D_Tech <
> [hidden email]> wrote:
>
> > What should I try next? Start with a new backup? Reinstall 4D?
>
> Personally, I would take the easy way out and change the script that is
> supposed to quit/restart 4D to something like:
>
> killall -9 4D
>
> which will not be deterred by error alerts or processes that are not
> responding to normal signals.
>
> If there are problems because 4D was quit by the OS instead of being asked
> nicely, follow it up with a full system restart, although that will require
> running the script with root privileges (probably not a problem, just run
> the killing part using cron as root, and the restarting part using launchd
> as the 4D user).
>
> Side story: my client was running v15 Server on macOS 10.12, and suffering
> from mass client disconnects. As the number of users approached 25, 3 or 4
> times per day 10-15 users would be disconnected from the server at once,
> with no consistency among users disconnected. While trouble-shooting, I ran
> a constant ping from another server to the 4D server, which shows several
> seconds of dropped packets corresponding to disconnect events. I then ran a
> ping on the 4D machine to another server, and had no more disconnect events
> (until someone restarted the computer and didn’t restart the ping command
> in Terminal). Obviously 4D wasn’t “fixed”; something in 4D server was
> killing the network stack in some way and all that code was being executed
> the same way as before. However, the actual problem went away and that’s
> really all that mattered.
>
> Note to self: perhaps an automated weekly restart of 4D Server is the
> trick to avoiding the occasional v16x DELAY PROCESS hang.  When a hammer
> works, might as well use it.
>
> Jim Crate
>
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************
>



--
-----------------------------------------------------------------------------------------
 Chuck Miller Voice: (617) 739-0306 Fax: (617) 232-1064
 Informed Solutions, Inc.
 Brookline, MA 02446 USA Registered 4D Developer
       Providers of 4D, Sybase & SQL Server connectivity
          http://www.informed-solutions.com
-----------------------------------------------------------------------------------------
This message and any attached documents contain information which may be
confidential, subject to privilege or exempt from disclosure under
applicable law.  These materials are intended only for the use of the
intended recipient. If you are not the intended recipient of this
transmission, you are hereby notified that any distribution, disclosure,
printing, copying, storage, modification or the taking of any action in
reliance upon this transmission is strictly prohibited.  Delivery of this
message to any person other than the intended recipient shall not
compromise or waive such confidentiality, privilege or exemption
from disclosure as to this communication.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Backup causes 4D Remote to hang (15.4)

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Jim Crate:

> Note to self: perhaps an automated weekly restart of 4D Server is the trick to avoiding the occasional v16x DELAY PROCESS hang.  When a hammer works, might as well use it.

Sorry, but no.
Our 4D Server shuts down and restarts every night with a script, but one process still hangs every few weeks.
So I guess it´s not the uptime that is causing the hang.

Greetings from Germany,
Bernd Fröhlich
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Backup causes 4D Remote to hang (15.4)

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Thanks Chuck,

I've reset the backup by deleting preferences. I also modified my scripts to use kill -9 instead of killall. Hopefully this will improve the situation. I should know overnight.

-- Brad Perkins

On 11/14/17, 1:00 PM, "4D_Tech on behalf of [hidden email]" <[hidden email] on behalf of [hidden email]> wrote:

    I would quit 4D server, delete preferences and recreate using menu commands
    re backup. Perhaps that us trhe problem. Also check where the backup is
    going to if a HD try a different location
   
    Regards and hiope this helps
    Chuck

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Backup causes 4D Remote to hang (15.4)

4D Tech mailing list
Still having problems. Based on various email notifications, I can see
that the scripts that under past OS versions would reliably restart 4D are
trying to do so, but w/o success. I don't know why they fail, but can see
they are repeatedly trying. That continues until I remotely access the
Mac. At that point the next scheduled run of the script succeeds.

If I try to run these scripts via SSH they don't succeed. I have to launch
Apple Remote Desktop.
I also have a password protected PHP app running on that server that
allows me to kill a hung 4D from my phone. That doesn't work during these
problem periods either.

It seems like this points to a power management issue, but pmset -g would
indicate that everything is set as it should be. I did not mention it
earlier, but this Mac Mini sits in a rack and hooked to a KVM. It normally
doesn't have a display.

Are any others running v15.4 (or v15.5) 4D Remote on Mac OS High Sierra
successfully? Or, if you are running it and having problems, what are they?

Thanks,

Brad Perkins



On 11/15/17, 5:13 PM, "Perkins, Bradley D" <[hidden email]> wrote:

>Thanks Chuck,
>
>I've reset the backup by deleting preferences. I also modified my scripts
>to use kill -9 instead of killall. Hopefully this will improve the
>situation. I should know overnight.
>
>-- Brad Perkins
>
>On 11/14/17, 1:00 PM, "4D_Tech on behalf of [hidden email]"
><[hidden email] on behalf of [hidden email]>
>wrote:
>
>    I would quit 4D server, delete preferences and recreate using menu
>commands
>    re backup. Perhaps that us trhe problem. Also check where the backup
>is
>    going to if a HD try a different location
>    
>    Regards and hiope this helps
>    Chuck
>

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Backup causes 4D Remote to hang (15.4) [Solved?]

4D Tech mailing list
I think this problem is finally solved. As I'd always suspected, it was likely related to power management features.

As previously noted, running the command ...

$ pmset -g

... suggested that all settings were as they should be. Mainly power nap was turned off globally.

About a week ago someone asked another question here about 'app nap'. I'd previously checked 4D.app using Get Info after the Sierra > High Sierra upgrade and verified it was set as needed.

I ran the command to check app nap on 4D.app and there was no value returned. Hmmm?
I then ran the command to enable app nap on 4D.app then check it again. It was now set.

I also toggled all of the machine's energy saver settings. I recorded them. Changed them. Saved them. Then reset them. This may have not helped, but I don't see how it could have hurt.

We've run for almost a week now.

-- Brad


________________________________________
From: Perkins, Bradley D
Sent: Friday, November 17, 2017 9:59 AM
To: [hidden email]
Subject: Re: 4D Backup causes 4D Remote to hang (15.4)

Still having problems. Based on various email notifications, I can see
that the scripts that under past OS versions would reliably restart 4D are
trying to do so, but w/o success. I don't know why they fail, but can see
they are repeatedly trying. That continues until I remotely access the
Mac. At that point the next scheduled run of the script succeeds.

If I try to run these scripts via SSH they don't succeed. I have to launch
Apple Remote Desktop.
I also have a password protected PHP app running on that server that
allows me to kill a hung 4D from my phone. That doesn't work during these
problem periods either.

It seems like this points to a power management issue, but pmset -g would
indicate that everything is set as it should be. I did not mention it
earlier, but this Mac Mini sits in a rack and hooked to a KVM. It normally
doesn't have a display.

Are any others running v15.4 (or v15.5) 4D Remote on Mac OS High Sierra
successfully? Or, if you are running it and having problems, what are they?

Thanks,

Brad Perkins



On 11/15/17, 5:13 PM, "Perkins, Bradley D" <[hidden email]> wrote:

>Thanks Chuck,
>
>I've reset the backup by deleting preferences. I also modified my scripts
>to use kill -9 instead of killall. Hopefully this will improve the
>situation. I should know overnight.
>
>-- Brad Perkins
>
>On 11/14/17, 1:00 PM, "4D_Tech on behalf of [hidden email]"
><[hidden email] on behalf of [hidden email]>
>wrote:
>
>    I would quit 4D server, delete preferences and recreate using menu
>commands
>    re backup. Perhaps that us trhe problem. Also check where the backup
>is
>    going to if a HD try a different location
>
>    Regards and hiope this helps
>    Chuck
>

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************