Preemptive mode flaw

classic Classic list List threaded Threaded
31 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Preemptive mode flaw

4D Tech mailing list
I'm looking at preemptive mode again and, again, tried something I wondered
about. "What happens when you inject bad code?" Say you've got a preemptive
thread, it can't use MESSAGE. If you put MESSAGE in the call chain, the
Compiler complains. Go the Compiler! But what happens if you have a method
that calls MESSAGE:

Display_Message

The Compiler complains. Go the Compiler!

Now what if you have this:

CALL WORKER("Preeemptive_Worker";"Display_Message";"Hello world!")

It's illegal - you're injecting code into the preemptive worker that
requires cooperative mode to work. What happens? It really depends. It
isn't consistent, it isn't documented, and it isn't good.

I've put more details into a feature request here:

Provide developer control when a preemptive worker receives non-preemptive
code
http://forums.4d.fr/Post//19391591/1/

Given how many people seem excited about preemptive threads for some
reason, I'm surprised that this hasn't come up. That also makes me
entertain the possibility that I'm being stupid or, more likely, trapped in
some kind of tunnel vision. So, if I've missed the obvious. I'll be
embarrassed, and then I'll get over it. So, don't be shy, feel free to make
me look dumb.

Or, if not, feel free to jump in and vote for my feature requests. I think
we need an incoming-code-for-execution inspection phase anyway.

Note: I'm working from a little scratch database that I've built today. No
frills, very simple code. I'm using 16.1 on OS X and have tried various
things in stand-alone and under 4D Server.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
> CALL WORKER("Preeemptive_Worker";"Display_Message";"Hello world!”)

Not a flaw… don’t do this… you know you are not supposed to, I was told it will produce a runtime error

This is the same as

C_LONGINT(myVar)
EXECUTE FORMULA(“myVar:=\”Hello\””)

It will compile just fine, but throw a runtime error.


--
Neil Dennis
4D Developer since 1990

GreaText - Designing Software for the Way You Work
716 East 1850 N
North Ogden, UT 84414

mailto:[hidden email]
http://www.greatext.com/


**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
In reply to this post by 4D Tech mailing list
is "code injection" even a problem?

preemptive mode is all about multi-tasking.

I would expect minimal communication between those tasks,
literally, they should be "preemptive".
it's true you can start a preemptive process with CALL WORKER,
but that is not required,
you can also use New process or Execute on server.
it's totally fine to have a runaway preemptive task.

workers, on the other hand, is all about messaging.

it's a different way of cooperating compared to cooperative processes,
that is, they is not "manager (scheduler)" or concept of "yield (IDLE)",
nevertheless, process are expected to cooperate and not preempt.

maybe it's better not to consider the two features as a set.

2017/05/07 11:20、David Adams via 4D_Tech <[hidden email]<mailto:[hidden email]>> のメール:

I'm looking at preemptive mode again and, again, tried something I wondered
about. "What happens when you inject bad code?" Say you've got a preemptive
thread, it can't use MESSAGE. If you put MESSAGE in the call chain, the
Compiler complains. Go the Compiler! But what happens if you have a method
that calls MESSAGE:



**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
In reply to this post by 4D Tech mailing list
> Not a flaw… don’t do this… you know you are not supposed to,

I did it on purpose to see what happens. I'm fanatical about checking
inputs and preconditions and wanted to see what sort of error to trap for.
In many cases, there is no error trapped - it just screws up.

This scenario is *way* to easy to produce in the wild to simply leave it up
to developers to screen their own code. I objected to EXECUTE for a very
long time because of this very reason. Unsafe at any speed. I only started
using it under tight regulation once I could check that a method exists
before running the code.

This situation would be 100% avoidable with as lightly different design for
CALL WORKER. Instead of appending code to the workers chain of execution,
pass a message. (CALL WORKER calls are not messages in any normal use of
that word in computing.) Pass a message, let the developer collect it and
decide what to do. At that point, it's on us. As it stands, it's on them as
they haven't provided 1) proper intercept tools to screen the call before
it is executed, 2) a reliable or consistent error handling mechanism to
trap the error.

> I was told it will produce a runtime error

Sometimes it produces a runtime error, sometimes it doesn't. Sometimes it
kills the process that then restarts.

On Sun, May 7, 2017 at 12:37 PM, npdennis <[hidden email]> wrote:

> CALL WORKER("Preeemptive_Worker";"Display_Message";"Hello world!”)
>
>
> Not a flaw… don’t do this… you know you are not supposed to, I was told it
> will produce a runtime error
>
> This is the same as
>
> C_LONGINT(myVar)
> EXECUTE FORMULA(“myVar:=\”Hello\””)
>
> It will compile just fine, but throw a runtime error.
>
>
> --
> Neil Dennis
> 4D Developer since 1990
>
> GreaText - Designing Software for the Way You Work
> 716 East 1850 N
> North Ogden, UT 84414
>
> mailto:[hidden email] <[hidden email]>
> http://www.greatext.com/
>
>
>
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
In reply to this post by 4D Tech mailing list
> On Sun, May 7, 2017 at 1:29 PM, Keisuke Miyako via 4D_Tech <
[hidden email]> wrote:

Hey, thanks for writing! I think the world of your contributions and work,
but we disagree pretty completely on this. I'll offer my thoughts and point
of view on all of this and, hopefully, you can give me a new way of looking
things. (You did manage to shift my thinking on ON ERR CALL in general
recently.) And if anyone else has real-world comments or observations, I'm
all in on hearing them, even if they make me change my opinion. My opinions
are *always* right, I just try to change them a lot ;-)

> is "code injection" even a problem?

Absolutely.

> preemptive mode is all about multi-tasking.
If by that you mean "taking advantage of idle cores", then I agree.

> I would expect minimal communication between those tasks,
> literally, they should be "preemptive".
> it's true you can start a preemptive process with CALL WORKER,
> but that is not required,
> you can also use New process or Execute on server.
> it's totally fine to have a runaway preemptive task.

Good point, thanks for pointing it out. I wasn't thinking about New
process/Execute on server, excellent point.

Just to make it clear, I have very little interest in preemptive
processing. I expect it to be a powerful and useful technique in a *very*
narrow range of problems. To justify the effort of using preemptive
threads, you need a task that is 1) taking a long time, 2) CPU-bound, and
3) doesn't require so much time to prepare and marshall that you don't get
much speed gain, and 4) has a sufficiently large non-cooperative component
to make it work the trouble. That's a lot of hurdles to clear and,
honestly, I don't have a single task in day-to-day use that makes it clear
of all of them. I've worked on systems (and am likely to work on systems
again soon) where such tasks exists, but they're not so commonplace.

> workers, on the other hand, is all about messaging.
Sadly, we *don't* have messages. We have blocks of code that are run.
That's not messaging. It only seems like messaging *if you're 4D.* As a 4D
developer, there's no queue, there's no message. If you're inside of 4D
itself, you can see a queue, etc. But as a 4D developer? You can't. All we
have is a pipe that things pop out of one-at-a-time and then execute. We
don't know if anything is in the pipe, we don't know when something
arrives, we don't have any ability to view/screen/filter/reject/trap that.
There's no event, nothing. Code is being tacked onto the end of a thread
and run blindly. It's an inherently dangerous design, but it _can_ be used
well. It just isn't built to run safely out of the box. Don't get me wrong,
I like the feature - I just don't use it without quite a bit of scaffolding
in place to minimize and/or eliminate risks.

As a practical example a very common, obvious, and good use of a preemptive
thread with CALL PROCESS, consider a log writer. It's a perfect setup.
Let's say you're logging errors/tasks/event/requests/whatever in 8
processes. Should each have its own log file? If not, it's a huge,
time-sucking waste to manage the contention between processes fighting for
a lock on the file. So, the better solution is to pipe all log entries to a
single process for writing. And since writing to disk is a relatively slow
operation, often of low value for logs, why suck up time on the main CPU?

CALL WORKER("LogWriter_Errors";"LogWriter_Start";"Errors")
LogWriter_Errors opens/appends/whatever a log file on disk *and keeps it
open*. Then any number of processes can stream log lines in for writing:

CALL WORKER("LogWriter_Errors";"LogWriter_Write";$error_details)
LogWriter_Errors catches the line, does a SEND PACKET and you're done and
dusted. You could set it up to handle different types of logs (say have
errors and requests both open), use different workers for different types
of logs, etc. None of the data has to go through 4D's data file, you've
just got a pass-through to disk. Sweet. Speed improvement over fighting
over the log file, etc.? 20x is *easy*. That's a huge optimization -
largely because of the design, the preemptive part - I don't even know how
much that helps. But the critical detail is that *the log file is opened
once at the start of the worker and closed only when the worker finishes.*
If that fails, you've got a big fat problem. (As it turns out, 4D currently
has a bug that makes race conditions on files super easy to create. It's in
the bug system somewhere but 16.2 and 16R2 make no difference.) So, yeah,
CALL WORKER is 100% appropriate, a preemptive worker is 100%
appropriate...and the ability to break the whole think with a simple
mistake makes it too dangerous to use.

In fact, that's my conclusion about preemptive mode now: Not ready for
prime time. My hat is off to 4D for trying on this as making a language
multi-threaded isn't a small thing. But I've found it just way too easy to
break so far. (I've tried a few times and ran into problems quickly in each
case.)

I expect that you *can* use preemptive mode today for specific tasks quite
successfully and to great benefit. I'd just be super careful, pick
well-defined stand-alone tasks, and be darn sure you make zero mistakes.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
first up, I myself am still in the process of learning how workers work,
and I totally agree one should probably start with small POCs or relatively minor tasks before going full on.

what I think I leaned so far:

keep in mind a CALL WORKER can create a new worker if the previous one was killed.
for instance. imagine a recursive worker method, using process variables for context,
aborted for whatever reason from the debugger or runtime explorer.
the worker is killed, but a new "call" comes in from the previous context,
starting a new worker that has no idea where the previous worker left off.
that can be a nightmare.

keep in mind a client process can never be preemptive.
most (if not all) published examples are designed for single user,
where some background processes are spawned simultaneously and report to a UI process.
they explain how workers work,
but in client server all the spawned workers will be cooperative,
sharing the same core as the UI process,
not really a show care for preemptive multi-processing.

2017/05/07 12:59、David Adams via 4D_Tech <[hidden email]<mailto:[hidden email]>> のメール:
So, yeah, CALL WORKER is 100% appropriate, a preemptive worker is 100%
appropriate...and the ability to break the who



**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
On Sun, May 7, 2017 at 3:07 PM, Keisuke Miyako via 4D_Tech <
[hidden email]> wrote:

> first up, I myself am still in the process of learning how workers work,
> and I totally agree one should probably start with small POCs or
> relatively minor tasks before going full on.
>

I think many of us here will be very interested in how you end up using
workers, so please share when you find applications that you like!


> keep in mind a CALL WORKER can create a new worker if the previous one was
> killed.
> for instance. imagine a recursive worker method, using process variables
> for context,
> aborted for whatever reason from the debugger or runtime explorer.
> the worker is killed, but a new "call" comes in from the previous context,
> starting a new worker that has no idea where the previous worker left off.
> that can be a nightmare.
>

Yes, exactly! There is no On Worker Start or On Worker Stop event - so it's
*very* important that they not be terminated accidentally.

For what it's worth, I've been working with workers, call message, and call
form intensively for a few months and am pretty deep into the details of
the mechanics. I like the commands and I think 4D did an admirable job of
providing power without complexity. I never would have come up with the
design they did as there are a lot of standard queuing features I would
never have considered dropping the way they did. I've tried to be open
minded about their design choices. But the more I get into it, the more I'm
seeing how different designs wouldn't be much more complex (if any more
complex) without taking away any of the power.

As far as preemptive mode goes, I think that CALL WORKER and preemptive
processing are wedded in people's minds because that's how 4D has been
presenting the subjects. I guess that they added workers to solve the
problem of dropping IP variables for IP communication. I didn't use IP
variables that way (too hard and expensive to use safely, too dangerous and
messy to use any other way)...I just use them for pseudo-constants/cached
items. So, CALL WORKER didn't solve a problem I had...but I love message
queues and have been trying to convert CALL WORKER into an actual message
queue system. (Callbacks and Publish/Subscribe.)

keep in mind a client process can never be preemptive.
>

Yes, I've noticed that. I don't know what the reasoning is there or if it
will change. It's just server-side processes, in compiled mode, using legal
commands, on a multi-core machine, where you've launched the process in
preemptive mode. Whew! Or stand-alone...
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
In reply to this post by 4D Tech mailing list
[JPR]

Hi David,

I'm not sure about this topic... About what do you try to demonstrate...

> I did it on purpose to see what happens. I'm fanatical about checking
> inputs and preconditions and wanted to see what sort of error to trap for.
> In many cases, there is no error trapped - it just screws up.


Is it a crash test? A kind of stress test? Do you really want to find a way to crash 4D? If yes, I can tell you many ways to crash 4D, or your System, or your HD, or even your computer, beyond any hope of recover...

It's like if you test a car, pedal to the metal, and ask what will happen if you close your eyes and turn abruptly the steering wheel. Is it useful?

What I said during this training is basically:

- Except in very exceptional cases, you do NOT need preemptive, so you can keep your interprocess variables and interface tricks.

- If you actually NEED preemptive tasks, then you will develop new programming for that, and then why not do it in a kind of thread-safe programming, just in case...

- Any analysis that concludes into a need to synchronize your thread with others is probably wrong somewhere.

- You can always try to cheat with 4D, if you like it, but in serious programming, don't try to mess with 4D, for I strongly doubt it will do any good...

> I objected to EXECUTE for a very
> long time because of this very reason. Unsafe at any speed.

That's true. But you know, Life is unsafe at any speed.

Why to object to a command? Just don't use it if you do not feel safe with it, and let people willing to use it do it at their own risks. I feel BASE jumping very unsafe, so I don't do it, and I still feel good about people who like it...

My very best,

JPR




> On 06 May 2017, at 21:03, [hidden email] wrote:
> Date: Sun, 7 May 2017 13:38:41 +1000
> From: David Adams <[hidden email]>



**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
On Mon, May 8, 2017 at 01:42 JPR via 4D_Tech <[hidden email]> wrote:

> I'm not sure about this topic... About what do you try to demonstrate...

It's a pretty standard question about exception handling.

> Is it a crash test? A kind of stress test? Do you really want to find a
way to crash 4D?
I'll answer your questions in order: No. No. No. There's a likely scenario
that isn't documented, so I wanted to find out the actual behavior. That's
just a bit of routine, technical due diligence. I'd do this with any
product.

> What I said during this training is basically <snip>

It sounds like you offered some sensible guidelines and such in your class,
but your points don't have much of anything to do with the issue I've
raised.

* I objected to EXECUTE for a very
* long time because of this very reason. Unsafe at any speed.

> That's true. But you know, Life is unsafe at any speed.
? That's just an odd thing to say.

> Why to object to a command? Just don't use it if you do not feel safe
with it,
> and let people willing to use it do it at their own risks. I feel BASE
jumping
> very unsafe, so I don't do it, and I still feel good about people who
like it...

I don't understand your response or how it applies to the issue I've
raised. You're not offering any sort of technical answer, just a confusing
metaphor. I won't follow you in this direction.

I'm talking about a pretty straightforward technical issue, not some kind
of exotic misuse of 4D's commands. For background, where did I get the
notion that CALL WORKER is meant to communicate with preemptive processes?
From watching Laurent Esnault's 2016 Summit presentation. His presentation
was clear, well reasoned, and informative. He explained how CALL WORKER
satisfies a requirement that arose out of the need to communicate with
preemptive workers. In other words, CALL WORKER was *created* to talk to
preemptive workers! So, what I'm talking about is a 100% orthodox use of
the 4D commands as explained by their creator. CALL WORKER has broader
applications, but it sounds like its genesis was to talk to preemptive
workers.

Now take another look at the title of my feature request:

   "Provide developer control when a preemptive worker receives
non-preemptive code"
   http://forums.4d.fr/Post//19391591/1/

That sounds entirely reasonable, no? If a piece of code accepts inputs,
it's basic defensive coding to have a way of dealing with exceptions. We
don't have this now in 4D for preemptive workers that receive code that
cannot be run preemptively. That's bad. 4D should fix this. This is
*entirely* within 4D's powers and absolutely *not* within our powers.

Now, it is easy enough to say (as someone already has) "don't do that!"
Well, of course! But that's as helpful as saying "Don't make mistakes!" or
"Be smarter!" It's worthless advice because there's no way to follow it. If
I could be smarter, I would be. But I can't. I would love to make fewer
mistakes but, by nature, they're something I'm unaware of. That's why
programmers learn to build tools to catch, manage, and prevent mistakes.
I'm not asking 4D to tolerate something illegal, I'm asking it to complain!
But to tell me about that complaint. Right now, there's a whole bunch of
unpredictable/inconsistent/undocumented behavior that could lead to some
seriously messed up outcomes. Since 4D is proud of this new feature (and
why not?), I'd hope that they want to make it more useful, reliable, and
easy to use successfully.

Most people aren't that far into the details on CALL WORKER, etc. yet, but
there's no need to understand those commands to appreciate the basic
quality-oriented coding issues. Back to talking about bad inputs... Take a
simple example. I've got a routine called CalculateRate

   CalculateRate (Longint) : Real

$1 must only be a whole number between 1-5, so the entire set of legal
values is:

1   2   3   4   5
That's it, all other values are illegal. Now, imagine four illegal calls
like these:

C_REAL($rate1;$rate2;$rate3;$rate4)
$rate:=CalculateRate(Pi)
$rate2:=CalculateRate("5")
$rate3:=CalculateRate(Current date)
$rate4:=CalculateRate(0)

Honestly, none of them look wrong per se. Any of those parameters could
make sense, depending on what sort of rate you're trying to calculate. But
in this case $1 has been defined as a longint, so the compiler detects that
the first three calls are illegal. However, the compiler is blind to the
out-of-range value in the $rate4 line. This is because 4D doesn't have any
syntax for declaring an input as allowing only a specific range of values,
etc. 4D's type system is non-extensible, so we're limited to 4D's built-in
types. <skip discussions of abstract types, etc. or of a a custom code
scanner here.> And even if the compiler did see the flaw in the fourth
call, there are other cases that only arise at runtime:

C_LONGINT($rate_code)
$rate_code:=Number_GetRandom(0;5)

This variable gets a longint from the series 0, 1, 2, 3, 4, 5. Ouch! It's
going to generate a bad input roughly 1/6 of the time. You'll only see this
at runtime. That's a silly example, but we've all generated or allowed bad
inputs from good intentions before in less artificial and silly ways.

Now imagine this code:

// CalculateFinalTotal
C_REAL($0;$final_total)
C_REAL($1;$total)
C_REAL($2;$rate_code)

$total:=$1
$rate_code:=$2

C_REAL($rate)
$rate:=CalculateRate ($rate_code)

$final_total:=$total*$rate

$0:=$final_total

That should compile fine, it looks fine...but what if $2 is 0? What happens
then? Imagine this call:

$final_total:=CalculateFinalTotal(100;0)

What is $final_total? What should it be? That depends on how CalculateRate
behaves. Let's assume that CalculateRate is defensive and detects the bad
rate code. (Thus avoiding a zero divide internally or what have you.) So
let's say that it returns 0. Your final total will be 0, the 100 is lost.
And don't divide by that result! Now imagine that CalculateRate returns a
negative value to indicate an error, say -1. Now the final total is -100.
This could also really screw up your calculations. What if that final total
is used in a larger calculation? The bad data spreads and spreads. So what
is the right response to such errors. *There isn't one.* You're getting
garbage, it's a programming flaw and it needs to be fixed. There is no
self-healing here, just detection. That detection alerts you to the problem
before more damage is done to the data and lets you correct the problem.
The problem is in the code, but it poisons the data.

It always makes my heart sink when I talk to developers who have discovered
a hidden bug that has been messing up their data for a long time.
Sometimes, there's just no way to unring the bell. It's horrible. And the
problem I raised about CALL WORKER and preemptive threads absolutely rises
to this level of risk. It's potentially a big deal.

There are lots of ways to deal with the common scenario that I've just
described. But one thing people don't always think through consciously is
how the responsibility is divided. <Skip discussion of Bertrand Meyer's
concept of programming by contract here.> Here is a call chain that
produces bad outcomes:

MyRoutine
    ....
    $final:=CalculateFinalTotal(100;0)
                  CalculateRate(0)

Who is to blame? Who is responsible for detecting the error? Well, the way
that I look at things, the CalculateRate routine is responsible for
*detecting bad inputs*. But the fault lies in "MyRoutine", the one that set
the bad value. It's MyRoutine's job to pass CalculateFinalTotal a good rate
to pass along to CalculateRate. It is CalculateRate's responsibility to
detect the bad input and raise or set an exception. (I use an error stack.)
That's a pretty clear and consistent division of responsibility, at least
in my mind and my code. Breaking it down:

MyRoutine: Responsible for handing in a valid input
CalculateRate: Responsible for detecting the bad input and reporting on it
CalculateFinalTotal: Responsible for checking that no error was generated
by the call to CalculateRate

Put another way:

CalculateRate holds CalculateFinalTotal for passing in good input.
CalculateFinalTotal holds CalculateRate responsible for detecting bad input
If you think things through this way it makes it easier to do good input
validation without endlessly rechecking inputs. In the example above,
CalculateFinalTotal doesn't check the range of input values coming from
MyRoutine. CalculateFinalTotal doesn't know the legal range, CalculateRate
does. So, no redundant (and potentially inconsistent) rules. But the
detection happens *near to the code that manages the input.* But I digress
a bit.

Short story: You get a lot more reliable code this way and it's easier to
think about.
Okay, so how to do that? There are lots of ways, depending on your
situation, requirements, and style. 4D provides the ASSERT command. You can
write your own input validation routines (that's what I do), you can
compile with range checking code injected by the 4D compiler (I do -
although I consider a range error that shows up in runtime as a substantial
flaw on my part), and you can add an ON ERR CALL handler for big exceptions.

CALL WORKER("Preemptive_Worker";"MethodWithCooperativeCode")

It's an input:

     "MethodWithCooperativeCode"

There's no way for us to validate it. ON ERR CALL doesn't work reliably.
The worker may be killed and restarted. You may end up with a runtime error
dialog blocking everything _even_ with an ON ERR CALL running. 4D may start
more and more instances of the flawed worker *with identical names* until
4D crashes or you force it to quit. I've seen all of that and, mostly, I
wasn't looking for any of it.

Now, I've already heard "Don't do that!", as noted. That's utterly
pointless advice. Who makes mistakes on purpose? Apart from due diligence
research, my mistakes are unintentional. As I sometimes say at this
juncture, if I'm of median intelligence and skill in the 4D developer
community and I see something as easy to screw up, then at least half of
all developers will also find it so. If I'm one tick above median, that
means the majority of developers will potentially run into problems. In the
real world, I might make a mistake and other people also might make
mistakes. Doesn't it seem reasonable to have a proper, consistent exception
handler? Yes it does.

Provide developer control when a preemptive worker receives non-preemptive
code
http://forums.4d.fr/Post//19391591/1/

P.S. The other approach to risky features is to say "Don't use it, it's not
worth the trouble." That seems like a pointless shame in this case. But
without reliability, I don't care at all about speed.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
the reason why I think

> CALL WORKER("Preemptive_Worker";"MethodWithCooperativeCode")

or more specifically

> CALL WORKER("Preemptive_Worker";"MethodWithPreemptiveCode")
> CALL WORKER("Preemptive_Worker";"MethodWithCooperativeCode")

is not a problem goes like this:

4D performs 2 checks; checking at compilation time, and checking at runtime.

the first check is well documented. every method with the "run preemptive" property is validated,
that no thread-unsafe commands or methods or inter-process variables or plugins or Begin SQL~End SQL blocks are called.
if the "use preemptive web processes" database property is activated,
web-related database methods, compiler_web, triggers and methods that are public on 4D Mobile are also validated.

then there is the second, runtime (or platform) check.
unicode mode must be active, the app must be 64-bit, compiled, the type must be either server or desktop.
if none of these conditions are met, all processes are automatically cooperative, no error.

and this is where we feel like we are entering uncharted territory; the list of "what if" s.

I try to categorise the potential issues this way:

1. references

any "references", that are "safe" in that they are not interprocess variables (therefore pass the first test),
only work within the same preemptive process that created it, or across cooperative processes.
these include DocRef, DOM ref, and yes, pointers.

http://doc.4d.com/4Dv16R2/4D/16-R2.1620/Preemptive-4D-processes.300-3111846.en.html

the code pass the first test, but breaks in that the references are invalid.
the error itself does not suggest thread-safety per-se,
they are just classic runtime errors due to invalid references.

there is a special provision for objects; an implicit OB Copy is called by CALL WORKER.
http://doc.4d.com/4Dv16R2/4D/16-R2.1620/CALL-WORKER.301-3111543.en.html

2. execution

using interprocess variables inside PROCESS 4D TAGS,
EXECUTE FORMULA, EXECUTE METHOD, etc.
the process has already been started as preemptive,
but an attempt to run cooperative code in the same context has been made.

the error specifically mentions that thread-safety is the problem.

3. calling

I think this is the type of issue we are discussing right now.

CALL WORKER is used to "run cooperative code in an existing preemptive process"
it is similar to issue #2 but different in the sense that it is not the current context that is asked to break thread safety.
when asked to "call worker", 4D already knows that the method is thread unsafe, and so is the target process (worker).
the instruction is simply impossible to execute.
so what happens? nothing (that has been my experience).

incidentally, what I find interesting about CALL WORKER is  that a worker remembers the name of the method that started it off,
and you can invoke that method by passing an empty string.

http://doc.4d.com/4Dv16R2/4D/16-R2.1620/CALL-WORKER.301-3111543.en.html

if we followed that rule and always passed an empty string, we would never have this problem. just saying...

now your feature request, if I understand correctly, talks about detecting, on the receiving end,
if a request to execute cooperative code has been received in a preemptive process.
I think it is not a problem, since there is nothing to receive since CALL WORKER does make such requests.

http://forums.4d.fr/Post//19391591/1/

> What happens when you send an illegal instruction to a worker that's running in preemptive mode?
> I've done my own quick tests and asked around, and here's what I've found:
>
> * I don't get an error.
> * The worker may be restarted in cooperative mode. Which may end up clearing existing variables and so on (?)
> * Other people report getting an error.
> * Sometimes I get an error.
> * Sometimes an error handler catches, sometimes it doesn't.
>
> It's wildly unpredictable.


I agree the documentation could be better, by giving more specific examples of each scenario.
but by and large, I think the results are pretty predictable for each case.

> * I don't get an error.

this must be CALL WORKER asked to do something impossible. (issue #3 above)

> * The worker may be restarted in cooperative mode. Which may end up clearing existing variables and so on (?)

the worker must have been killed or aborted.

> * Other people report getting an error.

some "eval" code must have been executed in the worker (issue #2 above)

> * Sometimes I get an error.
> * Sometimes an error handler catches, sometimes it doesn't.


a mix of all the other problems, just presenting themselves in random order?




**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
On Mon, May 8, 2017 at 11:03 AM, Keisuke Miyako via 4D_Tech <
[hidden email]> wrote:

Another great post, thanks. I really like the way you've broken out the
various scenarios, how they should behave and how you've observed them
behaving. Just to make it really clear, all I'm talking about is raising
errors consistently and not killing/restarting workers as a side-effect. I
in no way have _any_ problem with _any_ of 4D's restrictions on preemptive
processing. None. I looked into it a bit, and it's no simple task to make a
language run on multiple cores. So, they've got my complete respect on
that. Likewise, some of the restrictions in the language are imposed by the
underlying OS. So, it's reality imposing a constraint, not some arbitrary
choice by 4D.

Likewise, I appreciate what the compiler does for detecting errors in
advance. I'm always all in on the compiler doing more and, in this case,
it's a good effort.

But I am concerned (as you figured) about what comes up as #3 in your
taxonomy:

"CALL WORKER is used to "run cooperative code in an existing preemptive
process"

Personally, I think that 4D should *not* run the code. It should throw an
error that we can catch. Every time. It's not 4D's bug, it's my bug. But 4D
needs to tell me about it. This is very realistically the sort of flaw that
could creep into a system of almost any level of complexity from simple on
up.

Bad call: My fault
Scolding me: 4D's job
> and you can invoke that method by passing an empty string.
> http://doc.4d.com/4Dv16R2/4D/16-R2.1620/CALL-WORKER.301-3111543.en.html
> if we followed that rule and always passed an empty string, we would
never have this problem. just saying...

I guess I haven't said clearly, but I'm not primarily thinking of myself.
There is zero chance I'll start tossing unregulated code around for
execution. That's just stupid/insane/unsafe. It's a weak design, although I
can see why 4D did it. I've got a quite involved system of message passing
I'm working on that goes like this:

CallMyMagicThingy($object;{$result_callback_object})

Any method that is going to get a callback has one parameter:

C_OBJECT($1;$message)

That's it. There are formalisms and validators for the contents of the
message. (A header with routing details and action flags, a message
payload, and so on.) This, in fact, is was arguing for comprehensive JSON
scanning features a few months back. Without that, you can't write any sort
of generic validation code. Instead, you need a lot of code that embeds the
rules in the code. Ugh. I'd rather build my own meta-data/schema to
generate, validate, document, and test the formats. Can't. So, I scaled
back the system to code with only much simpler formats and had to write
individual validators. Wasteful and disappointing, but the native command
set doesn't support anything more. Anyway.

> now your feature request, if I understand correctly, talks about
detecting, on the receiving end,
> if a request to execute cooperative code has been received in a
preemptive process.
> I think it is not a problem, since there is nothing to receive since CALL
WORKER does make such requests.

I'm not parsing out what you're saying here, so I'm probably missing your
point. What I'm asking for is that the *recipient* throws an error. The
worker throws an error that I can catch. From there, I've got a lot of
options as how to proceed. There's no call-response with CALL WORKER so
there is not automatic way to report back the flaw to the code that caused
the flaw. But that's another matter:

C_BOOLEN($okay)
$okay:=Call worker("PreemptiveWorker";"BadCode")

If($okay=false)
   ALERT("Aiuto! Something went wrong!")
End if

I could go on about alternatives here all day, but that's not the point. At
the bare minimum, the preemptive worker should not terminate and should
report an error.

> * I don't get an error.
> this must be CALL WORKER asked to do something impossible. (issue #3
above)

Could be. Hard to know. Impossible to count on. The docs don't say. It has
been drilled into us for decades that when something is not in the docs *it
is not a reliable behavior.* There is absolutely no way to distinguish
between "Oh, yeah, we just didn't put that in the docs, good idea" from
"that's an undocumented behavior, don't count on it." They look the same
from the outside since there's nothing to look at!

> * The worker may be restarted in cooperative mode. Which may end up
clearing existing variables and so on (?)

> the worker must have been killed or aborted.

Yes, exactly - but not by my code - by 4D. This is a bug or design flaw.
It's not something I think is easy to defend as a good idea. "Oh, we will
keep this process running until you KILL it or quit. And that's a
guarantee! (+)"

(+) Not an actual guarantee, see the back of your ticket.

> * Other people report getting an error.
> some "eval" code must have been executed in the worker (issue #2 above)

Yeah, I'll buy that.

> * Sometimes I get an error.
> * Sometimes an error handler catches, sometimes it doesn't.

> a mix of all the other problems, just presenting themselves in random
order?

I cant' say for sure as I haven't studied it systemically enough. It was
enough of a dog's breakfast that it struck me as a waste of my time to try
to dig in. (I've poured hundreds of hours into CALL WORKER and CALL FORM
already, but I draw the line at features that just don't look like they
should have come out of the oven yet.) There's no promised behavior from 4D
about errors in case #3, it's easy to get weird results, and the
side-effects are potentially catastrophic. So, yeah, I'm not seeing putting
a bunch more time into it as smart move for me.

I really appreciate you offering a solid, well thought-out technical
answer. You've given me some food for thought and I think given everyone a
broader perspective on the system.

I want to emphasize again that I like CALL WORKER a lot. It's easy to use
and has a classic good 4D economy. Big power, small package. But, it needs
to be used correctly. Personally, I'd much prefer a true messaging system
rather than remote and unregulated code execution scheme. That's a great
approach (and time tested), and would make all of the following better:

* None of the nonesense I've been complaining about.
* Saving the queue for later processing.
* Load-balancing in real-time.
* Message distribution and routing. Easily.
* Message security policies.
* Message/service actions that can be extended to any JSON-capable system
out there. Very easy.
* Publish/subscribe: Easy.
* Networked: Easy

MQs (Message Queues) come out of the middleware world. It's always been all
about interchange amongst heterogeneous systems. That's one reason messages
are normally passed as, well, messages. Passing code locks your system into
a specific interpreter.

Personally, I don't care about preemptive much. I've noticed it seems to be
the new "For loops are faster" and that people get pretty excited. I
figured that I should check it out. I expect it to be suitable for only a
small range of problems. (There's a reason all of the demos show a bunch of
random calculations in a loop...it's because it's hard to find real-world
examples of discrete, time-consuming, CPU-bound, low-dependency tasks.)
When I have a suitable problem, I'll do a very, very, very specific and
tight implementation. Using the feature casually in this context could end
in tears too easily.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Is it still the case that client/server applications are excluded from running preemptive 4D code?

--
Jeffrey Kain
[hidden email]

> On May 7, 2017, at 9:03 PM, Keisuke Miyako via 4D_Tech <[hidden email]> wrote:
>
> then there is the second, runtime (or platform) check.
> unicode mode must be active, the app must be 64-bit, compiled, the type must be either server or desktop.
> if none of these conditions are met, all processes are automatically cooperative, no error.

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

RE: Preemptive mode flaw

4D Tech mailing list
http://doc.4d.com/4Dv16R2/4D/16-R2.1620/Preemptive-4D-processes.300-3111846.en.html

look at "Availability of preemptive mode"

The use of preemptive mode is available in 4D 64-bit versions only. The following execution contexts are currently supported:
Preemptive execution
4D Server X
4D remote -
4D single-user X
Compiled mode X
Interpreted mode -



> -----Original Message-----
> From: 4D_Tech [mailto:[hidden email]] On Behalf Of Jeffrey Kain via
> 4D_Tech
> Sent: Monday, May 8, 2017 2:47 PM
> To: 4D iNug Technical <[hidden email]>
> Cc: Jeffrey Kain <[hidden email]>
> Subject: Re: Preemptive mode flaw
>
> Is it still the case that client/server applications are excluded from running preemptive
> 4D code?
>
> --
> Jeffrey Kain
> [hidden email]
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
Thanks. Nobody ever mentions that when discussing the feature.

> On May 8, 2017, at 8:49 AM, Epperlein, Lutz (agendo) <[hidden email]> wrote:
>
> http://doc.4d.com/4Dv16R2/4D/16-R2.1620/Preemptive-4D-processes.300-3111846.en.html
>
> look at "Availability of preemptive mode"
>
> The use of preemptive mode is available in 4D 64-bit versions only. The following execution contexts are currently supported:
> Preemptive execution
> 4D Server X
> 4D remote -
> 4D single-user X
> Compiled mode X
> Interpreted mode -

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
In reply to this post by 4D Tech mailing list
[JPR]

Hi David,

It seems that we do not have the same way to see Workers. You are right on what you say. But I'm not sure this is the most important, when we talk about new things like Workers. Here are my thoughts:

- Concept of Workers is very new in V16, you know that you can expect improvements in the future. A long time ago, when we got processes in 4D, we had only 2 commands: New process and Kill process. Then we lose Kill process, but we got more commands (Process properties, Get and Set process variable, etc.) in the next versions. You cannot get something at this level of complexity and have all finished at once, in a perfect way.

- The most important regarding Workers is not the way they are actually working at the detail level. IMHO, it's the new way (for 4D Developers) to think their application's programming. We go from a synchronous way of programming, into a totally asynchronous way, from loops to messaging (CALL FORM & CALL WORKER), which is not simple for many developers. This is what I'm trying to explain during Trainings.

- I don't really care the way it works today, as long as I can use it until the next 4D versions, in which these concepts will be expanded.

- I do not concentrate on the missing features in 4D, I prefer to learn how to use the new ones, and to do the preliminary work in order to find the easiest practical way to use it, for developers to not waste time disentangling the different possible paths.

- Thanks to Keisuke and to you for doing those testings, for it will hep us to make a better version in a short future.  

> I don't understand your response or how it applies to the issue I've
> raised. You're not offering any sort of technical answer, just a confusing
> metaphor.
Sorry about that, but we are used to use metaphors to explain how things work. It's like pictures, it saves many words. French way.

>> That's true. But you know, Life is unsafe at any speed.
> ? That's just an odd thing to say.
Yes, probably, but this is the way it works...

My very best,

JPR

> On 08 May 2017, at 00:24, [hidden email] wrote:
>
> Date: Mon, 8 May 2017 08:46:30 +1000
> From: David Adams <[hidden email]>
> To: 4D iNug Technical <[hidden email]>
> Subject: Re: Preemptive mode flaw

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
In reply to this post by 4D Tech mailing list
 > Thanks. Nobody ever mentions that when discussing the feature.

Yes. Has anyone heard if this is an expected feature in a future version?
That would seem like a fair thing.

I'm obviously a lot more into headless and distributed applications than
anything else, so I look at networked 4D systems architecture differently
than some. (But not all.) If I wanted to set up a tolerably stable,
 high-performance 4D Server system, it would look like this:

4D Server
4D Sand-alone merged apps
Some kind of communications layer

In that setup, all of the machines are preemptive capable.

With that said, what I would *really* like to hear are real-world stories
from people getting some advantage out of preemptive mode. And details
about their approach.

Taking a page from Miyako's book, I think that a sensible starting strategy
for a regular 4D Server would be:

* Pick a discrete, time-consuming, CPU-bound task, of course. (Nothing else
could ever be worth the bother.)
* Run the code using Execute on server.

That rules out CALL WORKER, but that's okay. I suspect we're all pretty
careful about what sort of code runs on the server machine!

To communicate with the process on the other CPU, I'd use any number of
old-fashioned tricks. Honestly, I'd be picking jobs that can be described
in either a document or a record. Then the task runner would periodically
poll for new work in either a watch folder or a job table. If we're talking
about large tasks, the polling costs should be low. In fact, if the jobs
are large and rare, just run them on demand in a new process and then let
that process die.

But it would depend on circumstances how best to proceed, of course.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Well, I've answered your various points but, overall, I'm disappointed that
you've again failed to offer any actual technical content. Perhaps Thomas
Maul would contribute? He often does not agree with me, but his arguments
are always well-reasoned and technically interesting. Even if Thomas scolds
someone, it's after providing good information.

I also think it may be too easy to get caught up in how cranky I sound. So
I sound cranky, so what? That's more my problem than anyone else's. Does it
make me or my arguments more convincing when I sound like that guy on the
bus that no one will sit next to? No, not really. I do myself no favor,
your best option is to ignore my frustrated sounds and focus on the
technical merits of what I'm saying. If I'm missing something or being
stupid, I invite correction. And at least I'm still here, the other 4D
people of my generation are quiet...because mostly they are gone. Hey, I
like 4D! We all do. I mean, there's no real need to convince 4D people to
use 4D. We all want it to be better.

Since this whole thing has gotten absurdly long, I'll repeat my basic point
on its own:

CALL WORKER is a remote procedure call system that injects EXECUTE(code)
into a process.
If cooperative code is injected into a preemptive process, bad things
happen.
4D should do less bad things and tell us about the problem via ON ERR call.

That's it. It strikes me as a pretty uncontroversial statement so I don't
understand what the fuss is about. Why wouldn't someone thing it makes
sense to provide an error at this point?
> Concept of Workers is very new in V16, you know that you can expect
improvements in the future.

Of course! Discussion of the subject would be pointless if there were no
chance of change. However, I cannot make any plans based on what *might*
happen in the future. So, speculation is of zero practical value for me.

I'll also note that while "workers" are new in 4D V16, the underling
concepts and tools are neither new nor unfamiliar. They're just new in 4D.
Meaning, there are many, many long-standing implementations that 4D very
likely looked at and considered while developing their design. Like any
other programming team, the crew at 4D must have considered alternative
designs when they came up with what is now CALL WORKER and CALL FORM. I can
see why they picked a remote procedure call scheme, but I can't think of a
single other messaging architecture that works that way. But either way, it
was a design choice made in France, not some sort of discovery and not some
sort of law of nature. Their is nothing sacred about the design.

Now, they had to make choices to deliver a product. Their choices have very
good results and less desirable results. I have been continuously praising
the good aspects. So much so that these features convinced me to move my
own development from V13 to V16. I was also excited enough about these
features arriving in 4D to sign up to do my first 4D presentations in a
decade. So, yeah, I'm a fan.

I'm also willing to say what is a weak spot and what I see as preferable
design choices going forward. Error handling has always been a terrible
weak spot in 4D and my latest "feature request" (it's really about a
bug/design flaw) shows me that 4D still doesn't appreciate the importance
of consistent and reliable exception handling.

> You cannot get something at this level of complexity and have all
finished at once, in a perfect way.

And you cannot expect customers not to say that are unsatisfied by an
incomplete implementation. As it stands, the current design makes a great
many normal messaging patterns difficult or impossible to implement. 4D
could have done this sort of design *without more work* and given us a very
different system:

LAUNCH WORKER("Worker_Name";"Worker_Handler_Name"")
CALL WORKER("Worker_Name";$object)
// Worker_Handler_Name (Object)
C_OBJECT($1;$message)
$message:=$1
// Do whatever you like.

That's the simplest alternate version. At that point, the problem I have
been describing *would be impossible to create.* The flaw doesn't come from
some future feature not being implemented, it's a consequence of the design
itself. 4D can improve the situation by trapping the error when running
EXECUTE inside the worker....but the underlying design is what makes it
possible. The positive side of the design is that it's very easy for people
to reuse existing code without having to mess with C_OBJECT. The downside
is that you can blow up your server.

> IMHO, it's the new way (for 4D Developers) to think their application's
programming.
> We go from a synchronous way of programming, into a totally asynchronous
way, from
> loops to messaging (CALL FORM & CALL WORKER), which is not simple for
many developers.

I can't say. Anyone that's dealt with JavaScript has dealt with a vastly
harder to deal with version of asynchronous programming. In order to
achieve many normal results in 4D you'll need to roll your own versions of
features provided by asynchronous environments. I'm sure you get these
questions all the time:

Q: How do I get an answer from a worker?
A: You don't. A worker can send another message. But only to a worker or a
form.

Q: What form event is triggered by CALL FORM.
A: There isn't one. It's not a form event.

Q: So how does my form method catch CALL FORM?
A: It doesn't. The code is executed in the process that runs the form in
the context of that form. (In case you're using form variables.)

So, there's plenty to learn anyway and some obvious features
(call-response) that aren't implemented.

4D's implementation is only partially asynchronous anyway, but in a good
way. It's not that hard to follow. I don't see how using remote EXECUTE is
easier to understand than sending messages.

> I don't really care the way it works today, as long as I can use
> it until the next 4D versions,  in which these concepts will be expanded.

Just to get back to the original report, I'm talking about a *bug* in the
*current version* that makes these features *unreliable* for use in the
very situation they were supposedly designed for. As far as "future
versions" go. I don't care. How can I? They're completely imaginary. 4D
doesn't publish road maps so I can't even go on something like that. And,
based on history, there's no way to count on what I want or need being
implemented.

A bunch of examples:
-- Improvements to the SQL engine? Nope.
-- Object fields in SQL: Nope. Don't put them in your tables.
-- Proper JSON parsing? Nope. You don't need it. (Every other on earth has
it.)
-- Decent SVG support? No, we have a widgety thing.
-- 4D Chart? No. But we did give you a little graphing toy.
-- Support for later versions of XSLT? Answer: We're dropping all support.
-- 31 frickin characters. Seriously?
-- No structs/rects/classes
-- No data hiding
-- No try/catch
-- No dots in "objects". Actually, that seems to be coming. But if it's
just dot syntax and nothing more, it's only a convenience & I use Cannon's
tool already.

Contrarily, I saw 'contextual mode' in *alpha* and said, more or less "Nuke
it from orbit, it's the only way to be sure!"...but it lingered on for,
what, a decade?

Then again, 4D Write did come back to life and is looking great.

So, no, plans based on future versions aren't a safe bet. And given how
long the future is, I might not have time to wait long enough. 4D might do
something very welcome (4D Write users must be happy), or something very
disappointing (no 4D Chart, no JSON parsing.) Either way, "the future"
hasn't arrived yet so it's not worth a plan.

> I do not concentrate on the missing features in 4D, I prefer to learn how
to use the new ones,
> and to do the preliminary work in order to find the easiest practical way
to use it, for
> developers to not waste time disentangling the different possible paths.

I find it deeply disturbing that a representative of 4D France somehow
thinks that basic error checking is a weird thing to worry about. It's
basic. I need to know and consider what happens when things go wrong.
Otherwise, you can't deploy reliable software. Quality control is a
fundamental part of modern software development. It's not some exotic or
optional idea. When I'm doing due diligence, I try things out to assess if
they are reliable enough to use. Nothing, nothing at all could be more
practical. This is a fundamentally practical concern. In this case, I'm not
talking about a weird condition, an edge case, or anything unlikely. I'm
talking about a very basic problem that the design absolutely allows. It's
a simple fix for 4D and I absolutely reject your implication that I'm
somehow not understanding or doing something wrong. No.

Now, if I were looking for trouble with these new features, I could find
them. I stumbled across a race condition *that should be impossible* having
to do with the file system. I found this (by accident) and put together a
demo to reproduce the problem in about a minute. On Windows, it crashes the
machine. On Mac, it just locks up 4D so badly you have to force quit it. I
submitted everything through official channels and it's in the bug system.
Here's what I've heard from 4D France:



Full quote. I've tried it out again in 16R2 and 16.1, no difference. So,
yeah, I'm concerned about quality. I talk to lots of developers and my
impression is (and has been for years) that people care more about
*reliability* than features. Speed bumps and features are great! But people
most need reliability. When things crash or don't work right, customers get
very upset and it's stressful. That's most developers #1, #2, and #3


>> That's true. But you know, Life is unsafe at any speed.
> ? That's just an odd thing to say.
Yes, probably, but this is the way it works...

I honestly have no idea what you're trying to say. That everything carries
a risk so don't worry about it? That's a frankly frightening attitude
towards risk analysis or software quality. Not all risks are of the same
likelihood, size, or consequence.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
In reply to this post by 4D Tech mailing list
[JPR]

Hi David,

> I find it deeply disturbing that a representative of 4D France somehow
> thinks that basic error checking is a weird thing to worry about

I just would like to correct this:

- I'm not a representative of 4D France, just a contractant.

- I do not talk in the name of 4D in general. I just express my own feeling (or opinion), like you do yourself. This feeling is just based on my own experience.

- I do not pretend to detain the absolute truth, and, during trainings, I always say that I recommend to do so and so, and never say that this is the only way.

- I'm curious to know in which one of my postings you have found the following sentence "that basic error checking is a weird thing to worry about".

I prefer to see the glass half full than half empty, this is my way of thinking, because I think it's a more positive way. But, here also, I just express my opinion, and you're perfectly right to think the contrary. This is the way things are going on, because thread springs light. (De la discussion jaillit la lumière. — Charles Monnard)

And, as my friend Thibault was used to say: "Progressons! - Spinoza"  (Let's get forward!)

My very best,

JPR

> On 08 May 2017, at 19:27, [hidden email] wrote:
> Date: Tue, 9 May 2017 09:04:56 +1000
> From: David Adams <[hidden email]>
> To: 4D iNug Technical <[hidden email]>
>

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
> I'm curious to know in which one of my postings you have found the
following sentence
> "that basic error checking is a weird thing to worry about".

No, not your words - my paraphrase. Since my entire point was about error
trapping and you objected, I didn't know how else to interpret your
opinion. It really wouldn't be hard to agree with me:

"Yes, 4D should trap for bad code coming into a worker and report it in a
way that the developer can manage. 4D should not silently destroy the
worker, lose its context, and restart it without reporting the error."

If you agree, we have no dispute. If you disagree, I guess I don't know
what your position is. If you agree, you (and others) can vote for the
feature request here:

http://forums.4d.fr/Post/EN/19391591/1/19391592

Miyako posted an excellent technical analysis (as he always does!) that may
make it possible to reduce the scope of the error request, at least
internally. I don't care about the details so long as we can catch errors
and prevent 4D from killing the existing worker context, showing error
screens, starting duplicate "unique" workers or any of the rest of it.

On the technical front, this issue is most important with CALL WORKER when
the worker is preemptive. According to the documentation on CALL WORKER:

"...they have been designed mainly for interprocess communication in the
context of preemptive processes...."
http://doc.4d.com/4Dv16/4D/16.1/About-workers.300-3374909.en.html

Since this is the primary reason the feature was created, it seems entirely
reasonable that it work in a predictable, reliable, and
thoroughly documented manner. Likewise, every effort should be made to help
the developer to use these commands correctly, which includes
systematically and consistently catching and reporting on errors. This is
particularly true since both workers and preemptive mode are new features. I

> I prefer to see the glass half full than half empty,

Absolutely! I am also a glass half-full person. But then again, neither of
us has said what we think the glass is half full *of* ;-)
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Preemptive mode flaw

4D Tech mailing list
In reply to this post by 4D Tech mailing list
[JPR]

Hi David,

> "Yes, 4D should trap for bad code coming into a worker and report it in a
> way that the developer can manage. 4D should not silently destroy the
> worker, lose its context, and restart it without reporting the error."
>
> If you agree, we have no dispute.
So let's have no dispute. I agree with you. At least in a perfect world, with perfect products used by perfect people, I agree this is the way it should be. And be sure that 4D is working on it.

Meanwhile I'll try to do defensive programming, then I'll not be trapped by this flaw, and I move forward...

> Absolutely! I am also a glass half-full person. But then again, neither of
> us has said what we think the glass is half full *of* ;-)
For me, it will be half-full (at least) of Puligny-Montrachet 1961 from Maison Faiveley ;-)

My very best,

JPR

> Date: Tue, 9 May 2017 14:53:56 +1000
> From: David Adams <[hidden email]>
> To: 4D iNug Technical <[hidden email]>
> Subject: Re: Preemptive mode flaw


**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
12