4D Remote hanging at 3am after upgrading to Hi Sierra.

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

4D Remote hanging at 3am after upgrading to Hi Sierra.

4D Tech mailing list
We had to update a machine to Mac OS X Hi Sierra this week (cybersecurity mandate) on Tuesday of this week.
This machine runs as a web 'client' on 4D 15.4.
I have long standing controls in place to monitor the 4D application and keep it running.
Since the OS update, 4D is hanging at 3am every morning and is not recovering. I know this because my scripts are sending me email every 3 minutes that there is a problem. I have to manually reboot 4D every morning.

From what I can tell Power Nap is off on the machine:

$ pmset -g
System-wide power settings:
Currently in use:
standby              1
Sleep On Power Button 1
womp                 1
autorestart          1
hibernatefile        /var/vm/sleepimage
powernap             0
gpuswitch            2
networkoversleep     0
disksleep            0
sleep                0 (sleep prevented by screensharingd)
autopoweroffdelay    28800
hibernatemode        0
autopoweroff         1
ttyskeepawake        1
displaysleep         30
standbydelay         10800
I have an external job that runs at 3am via launchd, but turned it off yesterday to see if it was causing the problem. We still had the problem this morning. I can also run that job manually while 4D is running and it doesn't cause problems.
When I accessed 4D this morning to reboot, I briefly saw a 4D error dialog that would make me think the source of the problem is in a 4Dprocess that writes a file to disk. That process is running now without problem. Unfortunately the dialog dismissed itself before I could write down details. Therefore, I suspect something is running at ~3 that is not letting 4D write that file.

I've dug through logs and looked at other system settings to see if the system is running something at that time, but haven't found anything.

Any ideas would be appreciated?

-- Brad



**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra.

4D Tech mailing list
since it is at the same time -
There are a couple of possibilities:
- anti virus software (either update, or scan)
- cron job - specifically the daily 'cleanup' routines run by OS X
- some other specific timed action setup by(?) IT, the end user(s)?

Chip

On Fri, 27 Oct 2017 16:22:36 +0000, Perkins, Bradley D via 4D_Tech
wrote:

> We had to update a machine to Mac OS X Hi Sierra this week
> (cybersecurity mandate) on Tuesday of this week.
> This machine runs as a web 'client' on 4D 15.4.
> I have long standing controls in place to monitor the 4D application
> and keep it running.
> Since the OS update, 4D is hanging at 3am every morning and is not
> recovering. I know this because my scripts are sending me email every
> 3 minutes that there is a problem. I have to manually reboot 4D every
> morning.
>
> From what I can tell Power Nap is off on the machine:
>
> $ pmset -g
> System-wide power settings:
> Currently in use:
> standby              1
> Sleep On Power Button 1
> womp                 1
> autorestart          1
> hibernatefile        /var/vm/sleepimage
> powernap             0
> gpuswitch            2
> networkoversleep     0
> disksleep            0
> sleep                0 (sleep prevented by screensharingd)
> autopoweroffdelay    28800
> hibernatemode        0
> autopoweroff         1
> ttyskeepawake        1
> displaysleep         30
> standbydelay         10800
> I have an external job that runs at 3am via launchd, but turned it
> off yesterday to see if it was causing the problem. We still had the
> problem this morning. I can also run that job manually while 4D is
> running and it doesn't cause problems.
> When I accessed 4D this morning to reboot, I briefly saw a 4D error
> dialog that would make me think the source of the problem is in a
> 4Dprocess that writes a file to disk. That process is running now
> without problem. Unfortunately the dialog dismissed itself before I
> could write down details. Therefore, I suspect something is running
> at ~3 that is not letting 4D write that file.
>
> I've dug through logs and looked at other system settings to see if
> the system is running something at that time, but haven't found
> anything.
>
> Any ideas would be appreciated?
>
> -- Brad
>
>
>
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************
---------------
Gas is for washing parts
Alcohol is for drinkin'
Nitromethane is for racing
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra.

4D Tech mailing list
Chip,

Except for the OS Upgrade nothing changed. This problem didn't exist under Sierra.
The machine has  AV installed (IT policy mandates that too) but it has never been a problem in the past.
I'm the only end user of the machine. As noted I do have a 3am job, but that has run fine in the past and doesn't cause problems when manually executed while 4D is running.
What are the 'daily cleanup' routines that OS X runs? I'm assuming previous OS X versions did that too, but want to look into this.

Thanks,

Brad

On 10/27/17, 10:38 AM, "Chip Scheide" <[hidden email]> wrote:

    since it is at the same time -
    There are a couple of possibilities:
    - anti virus software (either update, or scan)
    - cron job - specifically the daily 'cleanup' routines run by OS X
    - some other specific timed action setup by(?) IT, the end user(s)?
   
    Chip
 

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra.

4D Tech mailing list
Try turning on the debug/request log in 4D to see what's firing in your
application at that time, if anything..

Also, look at the Mac Console log to see what's happening at 3 AM, if
there's anything unique running there that's not  happening earlier.



--
Sent from: http://4d.1045681.n5.nabble.com/4D-Tech-f1376241.html
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra.

4D Tech mailing list
In reply to this post by 4D Tech mailing list
here is a link about the cron jobs/scripts
http://thexlab.com/faqs/maintscripts.html

I do not really know what they do, only that they exist, I only quickly
scanned the article.
the article at the above link says they (may) run at 3:15am

of course, the underlying OS changed (Sierra to High Sierra),
existing anti-virus, or other things may interact differently -
especially if you did not update that software with the OS.

EX: -
where I work they run/use Sophos AV
under the OS we were running (10.6 - due to legacy applications),
Sophos thinks - maybe rightly so :) - that MS Office is a virus and
will not let it run. So we have had to get a different AV package
(Intego Virus Barrier)


 
On Fri, 27 Oct 2017 16:46:59 +0000, Perkins, Bradley D wrote:

> Chip,
>
> Except for the OS Upgrade nothing changed. This problem didn't exist
> under Sierra.
> The machine has  AV installed (IT policy mandates that too) but it
> has never been a problem in the past.
> I'm the only end user of the machine. As noted I do have a 3am job,
> but that has run fine in the past and doesn't cause problems when
> manually executed while 4D is running.
> What are the 'daily cleanup' routines that OS X runs? I'm assuming
> previous OS X versions did that too, but want to look into this.
>
> Thanks,
>
> Brad
>
> On 10/27/17, 10:38 AM, "Chip Scheide" <[hidden email]> wrote:
>
>     since it is at the same time -
>     There are a couple of possibilities:
>     - anti virus software (either update, or scan)
>     - cron job - specifically the daily 'cleanup' routines run by OS X
>     - some other specific timed action setup by(?) IT, the end user(s)?
>    
>     Chip
>  
>
---------------
Gas is for washing parts
Alcohol is for drinkin'
Nitromethane is for racing
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra.

4D Tech mailing list
Chip,

/var/log/daily.out exists
Since the OS  is newly updated no weekly or monthlies have run yet.
I can see that the daily OS X cleanup scripts are running at ~22:15 local time.
That rules these out.

Thanks,

Brad Perkins

On 10/27/17, 12:25 PM, "Chip Scheide" <[hidden email]> wrote:

    here is a link about the cron jobs/scripts
    http://thexlab.com/faqs/maintscripts.html
   
    I do not really know what they do, only that they exist, I only quickly
    scanned the article.
    the article at the above link says they (may) run at 3:15am
   
    of course, the underlying OS changed (Sierra to High Sierra),
    existing anti-virus, or other things may interact differently -
    especially if you did not update that software with the OS.
   
    EX: -
    where I work they run/use Sophos AV
    under the OS we were running (10.6 - due to legacy applications),
    Sophos thinks - maybe rightly so :) - that MS Office is a virus and
    will not let it run. So we have had to get a different AV package
    (Intego Virus Barrier)
 

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra.

4D Tech mailing list
In reply to this post by 4D Tech mailing list
I have grep'd everything in /var/log --- the files the the console displays -- and haven't found anything that is running at 3am. That is assuming of course that any system level activity would be logged. The closest entry is at 3:01 and that is when my monitoring script emails me the first of many messages that there is a problem.

I was mainly asking here in case those that are more in tune with current Mac goings on were aware of anything new in High Sierra that might have caused problems with 4D.

I'm not familiar with the use of  4D's debug/request log. Can that be turned on easily, or do I have to recompile the structure to enable it?
I'm mainly interested in what is going on at the client. I'd also guess that whatever is hanging 4D might not get logged to the server.

Thanks,

Brad

> From: JBellos <[hidden email]>

> Try turning on the debug/request log in 4D to see what's firing in your
> application at that time, if anything..

> Also, look at the Mac Console log to see what's happening at 3 AM, if
> there's anything unique running there that's not  happening earlier.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

RE: 4D Remote hanging at 3am after upgrading to Hi Sierra.

4D Tech mailing list
Brad, you wrote in your first posting, you saw a 4D error message very shortly. But you weren't able to read it.
So it would be really helpful if a 4D Server would not show such messages only, they should also be logged into a file (or in system log) regardless of my settings, at least during the starting phase of the server.
The usual message in the (Windows) Event log that 4D is started is not really helpful.

I just filed a feature request, look here: http://forums.4d.com/Post//21398249/1/

This request doesn't help you so much in the present situation, but if this feature would exist already it would be really helpful and you would a step further in your problem solving, I think.


Thanks
Lutz Epperlein

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra.

4D Tech mailing list
In reply to this post by 4D Tech mailing list


One thing you might try is changing the time that the script runs.

If the the script runs successfully at 2:00 AM and the computer hangs at
3:00 AM
then the problem is probably not the script.

If the computer hangs at the new scheduled time for the script and not
3:00AM
then that would lead me to look closer at the script.

We have had issues with writing files to folders where the permissions
would not
allow 4D to access a file in Read Write unless we changed the owner or
permissions
on the folder.

Good Luck.



**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra.

4D Tech mailing list
In reply to this post by 4D Tech mailing list
I think I found the problem. When I upgraded the server OS on Tuesday I had a second GUI login session open for the admin user as I needed to sudo to fix some permissions, etc (aka fast user switching). When I checked on the application early Saturday AM I realized I had left that user logged in with an idle terminal session. I also found that when I remotely logged in to to check on the 'web client' user that 4D was not running, but quickly launched (my login woke the system up). I suspect that power management features were causing the network and/or hard disks to 'sleep' at 3AM. We have not had problems since logging that user out. Therefore, even though you may think that Power Nap is configured to be off, it isn't totally.

Brad Perkins.

On 10/27/17, 3:02 PM, "Perkins, Bradley D" <[hidden email]> wrote:

    I have grep'd everything in /var/log --- the files the the console displays -- and haven't found anything that is running at 3am. That is assuming of course that any system level activity would be logged. The closest entry is at 3:01 and that is when my monitoring script emails me the first of many messages that there is a problem.
   
    I was mainly asking here in case those that are more in tune with current Mac goings on were aware of anything new in High Sierra that might have caused problems with 4D.
   
    I'm not familiar with the use of  4D's debug/request log. Can that be turned on easily, or do I have to recompile the structure to enable it?
    I'm mainly interested in what is going on at the client. I'd also guess that whatever is hanging 4D might not get logged to the server.
   
    Thanks,
   
    Brad
   
    > From: JBellos <[hidden email]>
   
    > Try turning on the debug/request log in 4D to see what's firing in your
    > application at that time, if anything..
   
    > Also, look at the Mac Console log to see what's happening at 3 AM, if
    > there's anything unique running there that's not  happening earlier.

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra. Need logging advice.

4D Tech mailing list
This problem went away for a few days, but restarted a couple of nights ago. It still happens at 3am. I have not been able to observe what happens at that time and suspect that if I try I won't see the problem. I suspect that Apple's evolving power management features are part of the problem and that if a user were interacting with the system at 3am, we wouldn't see this because the system would not be idle. Spending the night on site, which is an hour away, isn't an option, nor is getting up in the middle of the night and remotely logging in in the hope I can observe an obvious problem.

I believe I've done everything I can using the pmset command to prevent sleep, standby etc. It isn't clear if you can totally turn power management off everywhere though. In researching this problem I also discovered the 'caffeinate' command. This is a command-line system utility that is supposed to prevent the computer from sleeping. I haven't tried it yet. Has anyone here used it? If so, can you describe how?

I first deployed this system in 1995 under 3.x and it has provided continuous near 24/7/365 service since then. It has a small, but global user base which is primarily web-based. The machine in question is the web 'client', but it also runs a Apache for static web content and an SFTP server. This system has remained largely unchanged functionality-wise in the last 10 years. The main structure changes have been those required to upgrade major 4D versions. I maintain this system, but haven't implemented many newer 4D features. I've tried to stay aware of how 4D has changed.

While I can appreciate Apple's intentions with the power management features, my experience has been that they increasingly work against running 4D 24/7. This system is deployed on a network at a govt. agency. Their cyber security people force us to run the latest OS. No recent OS upgrade seems to go smoothly 4D-wise. The 4D client in question had always run for weeks or even months w/o problems, but every Mac OS upgrade seems to lessen that. Moving this web system to Windows is not an option. We do run 4D Server on Windows.

I have a number of controls in place to to ensure that the system is running.

1. Auto-launch 4D Remote in web serving mode if it is not running.  I can toggle this on/off. This is done with a combination of shell scripts and AppleScript. Apple's Launchd system manages this.
2. Continuously monitor the 4D web server. Every few minutes an off-site monitoring agent requests a simple dynamically generated page. If the request times out, or if the response contains unexpected content, I get notified. These requests provide an added benefit of keeping 4D busy during light or no usage periods.
3. There is not an easy way to externally determine if 4D, or any Mac OS desktop app has hung. You can easily detect if 4D is running, but not when it is non-responsive. For us this is when 4D has encountered a low-level error that can't be gracefully handled with an error handler. It doesn't help that any displayed dialogs provide little useful information. One solution is to have 4D run a process that writes a 'heartbeat' file every minute and have a system process check for changes to this file. If the file is present and hasn't changed in more than a few minutes, we'll assume that 4D has hung and run a script to kill it. At this point the auto-launcher from (1) above takes over. This is where we seem to be failing currently.

If others have done (3) how do you implement your heartbeat? I write a current timestamp into the file every minute. That assumes the file exists and is writable. It usually is. Is there a better way that is more robust?

I know that 4D has improved logging features, but I've never used them. What would I need to do to enable 4D to log between 2:45 and 3:15 AM?

Thanks,

Brad

________________________________________
From: Perkins, Bradley D
Sent: Monday, October 30, 2017 9:08 AM
To: [hidden email]
Subject: Re: 4D Remote hanging at 3am after upgrading to Hi Sierra.

I think I found the problem. When I upgraded the server OS on Tuesday I had a second GUI login session open for the admin user as I needed to sudo to fix some permissions, etc (aka fast user switching). When I checked on the application early Saturday AM I realized I had left that user logged in with an idle terminal session. I also found that when I remotely logged in to to check on the 'web client' user that 4D was not running, but quickly launched (my login woke the system up). I suspect that power management features were causing the network and/or hard disks to 'sleep' at 3AM. We have not had problems since logging that user out. Therefore, even though you may think that Power Nap is configured to be off, it isn't totally.

Brad Perkins.

On 10/27/17, 3:02 PM, "Perkins, Bradley D" <[hidden email]> wrote:

    I have grep'd everything in /var/log --- the files the the console displays -- and haven't found anything that is running at 3am. That is assuming of course that any system level activity would be logged. The closest entry is at 3:01 and that is when my monitoring script emails me the first of many messages that there is a problem.

    I was mainly asking here in case those that are more in tune with current Mac goings on were aware of anything new in High Sierra that might have caused problems with 4D.

    I'm not familiar with the use of  4D's debug/request log. Can that be turned on easily, or do I have to recompile the structure to enable it?
    I'm mainly interested in what is going on at the client. I'd also guess that whatever is hanging 4D might not get logged to the server.

    Thanks,

    Brad

    > From: JBellos <[hidden email]>

    > Try turning on the debug/request log in 4D to see what's firing in your
    > application at that time, if anything..

    > Also, look at the Mac Console log to see what's happening at 3 AM, if
    > there's anything unique running there that's not  happening earlier.

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra. Need logging advice.

4D Tech mailing list
On Nov 4, 2017, at 3:32 PM, Perkins, Bradley D via 4D_Tech <[hidden email]> wrote:

> I believe I've done everything I can using the pmset command to prevent sleep, standby etc. It isn't clear if you can totally turn power management off everywhere though. In researching this problem I also discovered the 'caffeinate' command. This is a command-line system utility that is supposed to prevent the computer from sleeping. I haven't tried it yet. Has anyone here used it? If so, can you describe how?

I don’t know if this is possible in your situation, but for the Mac Mini running 4D Server on 10.12 for my client I installed Apple’s Server app after having a couple issues suspected to be related to the power management. We had already purchased Server for other machines. If you go this route, you will probably have to configure your Apache web serving through the Server app. If you leave the FTP services turned off it shouldn’t interfere with whatever you’re using to run the SFTP server.

> I first deployed this system in 1995 under 3.x and it has provided continuous near 24/7/365 service since then. It has a small, but global user base which is primarily web-based. The machine in question is the web 'client', but it also runs a Apache for static web content and an SFTP server. This system has remained largely unchanged functionality-wise in the last 10 years. The main structure changes have been those required to upgrade major 4D versions. I maintain this system, but haven't implemented many newer 4D features. I've tried to stay aware of how 4D has changed.

Is Apache still serving static content when 4D Remote stops working?

> 2. Continuously monitor the 4D web server. Every few minutes an off-site monitoring agent requests a simple dynamically generated page. If the request times out, or if the response contains unexpected content, I get notified. These requests provide an added benefit of keeping 4D busy during light or no usage periods.

You’ve only mentioned being notified by the scripts on the machine itself. Does the external service monitor also notify you that the 4D web server has stopped serving?

> 3. There is not an easy way to externally determine if 4D, or any Mac OS desktop app has hung. You can easily detect if 4D is running, but not when it is non-responsive. For us this is when 4D has encountered a low-level error that can't be gracefully handled with an error handler. It doesn't help that any displayed dialogs provide little useful information. One solution is to have 4D run a process that writes a 'heartbeat' file every minute and have a system process check for changes to this file. If the file is present and hasn't changed in more than a few minutes, we'll assume that 4D has hung and run a script to kill it. At this point the auto-launcher from (1) above takes over. This is where we seem to be failing currently.
>
> If others have done (3) how do you implement your heartbeat? I write a current timestamp into the file every minute. That assumes the file exists and is writable. It usually is. Is there a better way that is more robust?

Which part of #3 is failing? Is the script to kill 4D being run and failing? Is it running “kill -9 process_id”? Is the auto-launch failing? Is the heartbeat file writing stopping while the service is actually running? There are numerous reports of problems with file writing and delayed processes. If only the heartbeat file writing is stopping, does the file actually exist and is still writable by that process? Where is the file located?

Does the external service monitor have an API? If so, could you periodically check the external service monitor for your service, and bounce 4D when there is actually a problem with the service?
 

Jim Crate

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra. Need logging advice.

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Hello,

I am an avid user of caffeinate (it's a great name, by the way),
to keep my mac awake.

there are several modes (assertions) to pick and choose,

caffeinate -dimu is the most aggressive,

dprevent display from sleep
iprevent system from sleep
sprevent display from sleep (but not if running on battery)
mprevent disk from sleep
ufake user activity

typically I use it while projecting a presentation,
or when several time consuming downloads are going on and I leave the room,
for example.

> 2017/11/05 4:32、Perkins, Bradley D via 4D_Tech <[hidden email]> のメール:
>
> . In researching this problem I also discovered the 'caffeinate' command. This is a command-line system utility that is supposed to prevent the computer from sleeping. I haven't tried it yet. Has anyone here used it? If so, can you describe how?




**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra. Need logging advice.

4D Tech mailing list
typo, s if for system, not display.

enter "man caffeinate" in terminal for details.

> 2017/11/06 5:00、Keisuke Miyako via 4D_Tech <[hidden email]> のメール:
> sprevent display from sleep (but not if running on battery)



**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra. Need logging advice.

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Jim,

Your observations and questions are helpful.

re: Installing the Mac OS Server App.

        I hadn't considered that but if it is the only way to tame the power
management problems, they might have to try that. There was a time when we
ran this system on Mac OS X Server, but when that OS option was dropped, I
saw no value in running Apple's Server App. Even under OS X Server, I
always ran Apache manually. I found that the Mac OS X Server's UI for
managing web sites would often overwrite our custom configurations, e.g.,
rewrite rules. I wasn't clear earlier but the 4D web server (Active4D)
sits behind Apache and serves on 127.0.0.1 on a high port. All requests go
through Apache.

re: Is Apache still serving when 4D Remote Stops working?

        I believe it is, but haven't actually gotten up and checked at 3am. If I
see that we're down in the morning. I'll check their static site first.


re: You¹ve only mentioned being notified by the scripts on the machine
itself. Does the external service monitor also notify you that the 4D web
server has stopped serving?

        Yes, The external monitor would tell me if the web server was down, or if
the machine isn't reachable (ping). We've seen some of those, but not as
many as the on-machine script notifications. However, I have the external
monitor to stop sending notifications after a few failures. Actually,
given that the machine is sending mail every 3 minutes from 3:01 am until
I restart 4D it isn't totally asleep.

re: Which part of #3 is failing? Is the script to kill 4D being run and
failing? Is it running ³kill -9 process_id²? Is the auto-launch failing?
Is the heartbeat file writing stopping while the service is actually
running? There are numerous reports of problems with file writing and
delayed processes. If only the heartbeat file writing is stopping, does
the file actually exist and is still writable by that process? Where is
the file located?

        I think that this is a big part of the problem. Granted this has run
reliably on past OS versions but I need to thoroughly retest this script.
- It runs "killall 4D", but could be modified to kill the process. I just
verified that  "killall 4D" worked at this time. After doing so, the
auto-launch kicked in and restarted 4D.

- I suspect that the problem might be with the heartbeat file. I'll have
to check if the file is being deleted. We have code in 4D that notifies us
if the heartbeat log can't be created or   is not writable when the
process starts. If that occurs, the client doesn't enable 4D's web server
and it is quickly evident that we have a problem. I have seen no heartbeat
file problem notifications since the OS upgrade, except on 1st 4D launch.
I had to create and set permissions on /var/log/local/4D/, which is where
the heartbeat file is located.

re: Does the external service monitor have an API? If so, could you
periodically check the external service monitor for your service, and
bounce 4D when there is actually a problem with the service?

        I don't believe it does, but if it did I couldn't leverage it. The
monitor unfortunately runs on a machine on a different network that is
behind the institution's firewall. The server in question is on the 'open'
network. The firewalled app can see 'out' but can't be seen from the
server. Moving the monitor isn't really an option. Getting the firewall
open isn't even a consideration.

        Beyond all of the above, I would like to find out why this problem
repeatedly occurs at 3am. If you've followed my earlier posts I do have a
launchd job that runs at that time, but rescheduling it earlier did not
move the problem to the earlier time. I even turned that job off for one
night and that didn't prevent the 3am sleep. I grep'd all of my logs and
could not find anything notable happening between 2:57 and 3:01.

Thanks,

Brad Perkins


>
>------------------------------
>
>Message: 5
>Date: Sun, 5 Nov 2017 12:07:43 -0500
>From: Jim Crate <[hidden email]>
>To: 4D iNug Technical <[hidden email]>
>Subject: Re: 4D Remote hanging at 3am after upgrading to Hi Sierra.
> Need logging advice.
>Message-ID: <[hidden email]>
>Content-Type: text/plain; charset=utf-8
>
>On Nov 4, 2017, at 3:32 PM, Perkins, Bradley D via 4D_Tech
><[hidden email]> wrote:
>
>> I believe I've done everything I can using the pmset command to prevent
>>sleep, standby etc. It isn't clear if you can totally turn power
>>management off everywhere though. In researching this problem I also
>>discovered the 'caffeinate' command. This is a command-line system
>>utility that is supposed to prevent the computer from sleeping. I
>>haven't tried it yet. Has anyone here used it? If so, can you describe
>>how?
>
>I don¹t know if this is possible in your situation, but for the Mac Mini
>running 4D Server on 10.12 for my client I installed Apple¹s Server app
>after having a couple issues suspected to be related to the power
>management. We had already purchased Server for other machines. If you go
>this route, you will probably have to configure your Apache web serving
>through the Server app. If you leave the FTP services turned off it
>shouldn¹t interfere with whatever you¹re using to run the SFTP server.
>
>> I first deployed this system in 1995 under 3.x and it has provided
>>continuous near 24/7/365 service since then. It has a small, but global
>>user base which is primarily web-based. The machine in question is the
>>web 'client', but it also runs a Apache for static web content and an
>>SFTP server. This system has remained largely unchanged
>>functionality-wise in the last 10 years. The main structure changes have
>>been those required to upgrade major 4D versions. I maintain this
>>system, but haven't implemented many newer 4D features. I've tried to
>>stay aware of how 4D has changed.
>
>Is Apache still serving static content when 4D Remote stops working?
>
>> 2. Continuously monitor the 4D web server. Every few minutes an
>>off-site monitoring agent requests a simple dynamically generated page.
>>If the request times out, or if the response contains unexpected
>>content, I get notified. These requests provide an added benefit of
>>keeping 4D busy during light or no usage periods.
>
>You¹ve only mentioned being notified by the scripts on the machine
>itself. Does the external service monitor also notify you that the 4D web
>server has stopped serving?
>
>> 3. There is not an easy way to externally determine if 4D, or any Mac
>>OS desktop app has hung. You can easily detect if 4D is running, but not
>>when it is non-responsive. For us this is when 4D has encountered a
>>low-level error that can't be gracefully handled with an error handler.
>>It doesn't help that any displayed dialogs provide little useful
>>information. One solution is to have 4D run a process that writes a
>>'heartbeat' file every minute and have a system process check for
>>changes to this file. If the file is present and hasn't changed in more
>>than a few minutes, we'll assume that 4D has hung and run a script to
>>kill it. At this point the auto-launcher from (1) above takes over. This
>>is where we seem to be failing currently.
>>
>> If others have done (3) how do you implement your heartbeat? I write a
>>current timestamp into the file every minute. That assumes the file
>>exists and is writable. It usually is. Is there a better way that is
>>more robust?
>
>Which part of #3 is failing? Is the script to kill 4D being run and
>failing? Is it running ³kill -9 process_id²? Is the auto-launch failing?
>Is the heartbeat file writing stopping while the service is actually
>running? There are numerous reports of problems with file writing and
>delayed processes. If only the heartbeat file writing is stopping, does
>the file actually exist and is still writable by that process? Where is
>the file located?
>
>Does the external service monitor have an API? If so, could you
>periodically check the external service monitor for your service, and
>bounce 4D when there is actually a problem with the service?
>
>
>Jim Crate
>
>
>
>------------------------------
>
>Subject: Digest Footer
>
>**********************************************************************
>4D Internet Users Group (4D iNUG)
>FAQ:  http://lists.4d.com/faqnug.html
>Archive:  http://lists.4d.com/archives.html
>Options: http://lists.4d.com/mailman/options/4d_tech
>Unsub:  mailto:[hidden email]
>**********************************************************************

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra. Need logging advice.

4D Tech mailing list
On Nov 5, 2017, at 4:25 PM, Perkins, Bradley D via 4D_Tech <[hidden email]> wrote:

> saw no value in running Apple's Server App. Even under OS X Server, I
> always ran Apache manually. I found that the Mac OS X Server's UI for
> managing web sites would often overwrite our custom configurations, e.g.,
> rewrite rules. I wasn't clear earlier but the 4D web server (Active4D)
> sits behind Apache and serves on 127.0.0.1 on a high port. All requests go
> through Apache.

If you install Server app, you’ll want to look at creating a webapp plist. It’s pretty simple and allows you to hook in your additional configuration without modifying Server-created files that will be overwritten.

> many as the on-machine script notifications. However, I have the external
> monitor to stop sending notifications after a few failures. Actually,
> given that the machine is sending mail every 3 minutes from 3:01 am until
> I restart 4D it isn't totally asleep.

This would seem to indicate it isn’t related to OS sleep (i.e. things controllable with pmset). It could possibly be related to App Nap. Does 4D v15 properl support App Nap? I’m not sure if there is a way to turn it off system-wide.

> I think that this is a big part of the problem. Granted this has run
> reliably on past OS versions but I need to thoroughly retest this script.
> - It runs "killall 4D", but could be modified to kill the process. I just
> verified that  "killall 4D" worked at this time. After doing so, the
> auto-launch kicked in and restarted 4D.

The man page says the default signal sent by killall is TERM (15). You can see all signals by running “man signal”, but the short answer is that TERM will ask 4D to quit, and KILL (9) will tell the OS to kill the process (basically a force quit).


> Beyond all of the above, I would like to find out why this problem
> repeatedly occurs at 3am. If you've followed my earlier posts I do have a
> launchd job that runs at that time, but rescheduling it earlier did not
> move the problem to the earlier time. I even turned that job off for one
> night and that didn't prevent the 3am sleep. I grep'd all of my logs and
> could not find anything notable happening between 2:57 and 3:01.

But not every night at 3am, right? Is there consistency on the days of the week? Days of the month?

This is a 4D client, maybe something is happening to the server or network at that time and the 4D client is losing it’s connection to the server?

Jim Crate

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra. Need logging advice.

4D Tech mailing list
In reply to this post by 4D Tech mailing list
You may have said so earlier, but is the machine set to never sleep in the Energy control panel?

RE: caffeinate

If you launch 4D using caffeinate /path/to/4d, the machine should not go to sleep until 4D quits. See also the -w option where you can have 4D set it up after it has started.

John DeSoi, Ph.D.





> On Nov 5, 2017, at 3:25 PM, Perkins, Bradley D via 4D_Tech <[hidden email]> wrote:
>
> Beyond all of the above, I would like to find out why this problem
> repeatedly occurs at 3am. If you've followed my earlier posts I do have a
> launchd job that runs at that time, but rescheduling it earlier did not
> move the problem to the earlier time. I even turned that job off for one
> night and that didn't prevent the 3am sleep. I grep'd all of my logs and
> could not find anything notable happening between 2:57 and 3:01.

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra. Need logging advice.

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Some more information on my problem.

As I mentioned before I couldn't find anything in the various system logs
on the local machine that would correlate a 3am event. I'd also moved the
one 3am local job to 2am and that didn't move the problem time. Earlier
this week I remembered that 4D Backup runs in the middle of the night. I
checked, and yes it is running at 3am. I rescheduled the backup to 5am and
now the client hang problem starts at 5am. This is a vanilla backup
configured from the server. These backups run quick. I ran one manually
during the day and it did not crash the client. We've also never had this
problem for as long as I can remember (4D at this site since 95). I ran
diagnostics on the server to check the datafile, indexes, etc. No problems
reported.

My remote web app monitor sent email that the 4D web client didn't respond
with an expected web request at 5:02. Since it only checks every 3 minutes
this means the actual problem could have started a few minutes earlier.

The unresponsive 4D Remote has crashed each morning the last 3 mornings
between 5:10 and 5:13. Therefore it is hung for a few minutes before it
crashes. Normally local scripts would reboot it, but they aren't working.

So I have 2 problems:

1. 4D Backup is (possibly, but likely) causing a connected client to hang
then crash. Has anybody seen this happen? Is there a fix?

2. Scripts to reboot hung or not running 4D that have worked for years no
longer do during this time. Again, if I'm logged in remotely via Screen
Sharing I can run these scripts fine. I can even run them via SSH and they
are fine. For the hung script I will modify to use kill -9 though even
though killall works when I test. Using kill -9 may at least allow me to
reboot a 5am hang, but I'd really like for the system to not hang at all.

Thanks,

Brad Perkins

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra. Need logging advice.

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Is this a Mac?

If so, have you tried running a PURGE from the OS in On Backup Shutdown?

http://kb.4d.com/assetid=77831
http://kb.4d.com/assetid=75978

-Tim

Sent from my Verizon, Samsung Galaxy smartphone


**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Remote hanging at 3am after upgrading to Hi Sierra. Need logging advice.

4D Tech mailing list


> On Nov 11, 2017, at 11:26, Timothy Penner via 4D_Tech <[hidden email]> wrote:
>
> Is this a Mac?
>
> If so, have you tried running a PURGE from the OS in On Backup Shutdown?
>
> http://kb.4d.com/assetid=77831
> http://kb.4d.com/assetid=75978
>
> -Tim

Do the backup database methods run on connected clients (4D Remote) as well as on the server?

Brad said the client is disconnecting or hanging when the server runs backup. Even if the server could run purge after finishing the backup, how would that help the remote client which had its connection interrupted?

Jim Crate

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
12