Does it matter if you lock an IP object before updating?

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Does it matter if you lock an IP object before updating?

KirkBrooks
This is the sort of thing I think about while driving home...

I see databases using IP variables use a semaphore/locking scheme before
writing to the variable. Something like this:

Lock_object("objectName")
Append to array(<>myArray;$someValue)
Unlock_object("objectName")


The Lock_object method attempts to set a semaphore. If the semaphore is
already set, indicating another process is attempting to update this
variable at the same time, it waits a tick and this continues until it sets
the semaphore.

I wondered how likely it would be for the situation to arise where two
processes would be attempting to access an IP var at the same time and what
4D would do. So I wrote a little test to investigate. The methods are at
the end for anyone who cares to replicate.

The methodology is to start a process which clears an IP text array. Then I
spin off a number (I used 400) other processes. Each of these waits for the
same time (using milliseconds here) then begins a loop that writes some
identifying data to the text array. I ran the test using object locking and
without.

Overview -

There were no lost updates due to collisions in either test case
Setting the semaphore doesn't take much time, but it takes some

Without adding some delay in the drone loops each loop would complete
before the next one began

To introduce some randomness I delay each drone by 1 or 2 clicks


Limitations

The run times are approximate because the delay command waits for 1 tick,
about 17 ms

I didn't look at the effect on writing to multiple, synchronized arrays

My Takeaway - it doesn't seem like much to worry about in terms of locking
IP variables against lost data due to collisions of multiple processes
attempting to write to the same IP var at the same time. My understanding
of why is because this version of 4D is still single threaded. This is why
400 processes are running 'simultaneously' they actually are queued up by
4D and processed sequentially. This is borne out by the fact that as the
number of drones increases the run time increases proportionally in both
cases. If the un-locked version were able to proceed without a successful
write that version would be significantly faster. It's not. So the number
of drones increases the run time due to queuing regardless of setting or
not setting the semaphore.

This situation would be different in the case of multi-threaded execution -
but you can't use an IP var in that case anyway. Probably for just this
reason.

So I'm concluding the whole variable locking fetish is unnecessary.


The test consists of four methods:

Let's call this one "Test":

ARRAY TEXT(<>aTestArray;0)
  // $determine the start time
$delay:=2
$iterations:=20
$nDrones:=400

$n:=Count user processes  //  benchmark

$ms:=Milliseconds+2000  // add 2 secs to now before drones start

  // start a few other processes
For ($i;1;$nDrones)
drone ($i;$ms;$iterations)
End for

  //  wait a bit for everything to run
While ($n<Count user processes)  // some are still running
DELAY PROCESS(Current process;1)
End while

$ms:=Milliseconds-$ms  // approx elapsed time
  // examine the array
$report:="Drones: "+String($nDrones)+"\rIterations:
"+String($iterations)+"\r"
$report:=$report+"Results:\r"
$report:=$report+"Run time: ~"+String($ms)+" ms\r"
$n:=($iterations*$nDrones)-Size of array(<>aTestArray)
$report:=$report+String($n)+" Missing elements\r"
ALERT($report)


​This is the drone method:

​// drone
C_LONGINT($1;$2;$3;$4)
C_LONGINT($id;$i)
C_TEXT($procName)

Case of
: (Count parameters=3)
$procName:="drone_"+String($1)
$id:=New process(Current method name;0;$procName;$1;$2;$3;Current process)

: (Count parameters=4)  //     new process
While (Milliseconds<$2)  // wait for the starting time
DELAY PROCESS(Current process;1)
End while

  // now do the loop
For ($i;1;$3)
APPEND TO ARRAY(<>aTestArray;"Drone "+String($1)+": "+String($i)+"
 "+String(Milliseconds))
If (Milliseconds%2=0)  // introduce some variability
DELAY PROCESS(Current process;1)
Else
DELAY PROCESS(Current process;2)
End if
End for
End case

//droneWithLocking
C_LONGINT($1;$2;$3;$4)
C_LONGINT($id;$i)
C_TEXT($procName)

Case of
: (Count parameters=3)
$procName:="drone_"+String($1)
$id:=New process(Current method name;0;$procName;$1;$2;$3;Current process)

: (Count parameters=4)  //     new process
While (Milliseconds<$2)  // wait for the starting time
DELAY PROCESS(Current process;1)
End while

  // now do the loop
For ($i;1;$3)
Lock_object ("<>aTestArray")
APPEND TO ARRAY(<>aTestArray;"Drone "+String($1)+": "+String($i)+"
 "+String(Milliseconds))
Unlock_object ("<>aTestArray")

If (Milliseconds%2=0)  // introduce some variability
DELAY PROCESS(Current process;1)
Else
DELAY PROCESS(Current process;2)
End if
End for
End case


//Lock_object
While (Semaphore("$"+$1))
IDLE
End while

//Unlock_object

CLEAR SEMAPHORE("$"+$1)


--
Kirk Brooks
San Francisco, CA
=======================
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

David Adams-4
Sure, it's absolutely possible to encounter a race condition if you write
to a shared resource (IP arrays in your case) without a locking mechanism.
It's likelihood increases in compiled mode and as you add more
participants. Even if you can't predict how often you'll encounter a race
condition, you already know that the chance is greater than zero.

A few more points:

* What are you risking? If your data is written incorrectly because of
out-of-expectation sequencing, what happens? Do you crash? Do you get
screwed up calculations? If you don't care about the errors, and they don't
harm you...perhaps it doesn't matter. For example, if you're writing to an
event log and lose a few entries every 10,000, perhaps it doesn't make any
real difference. On the other hand, if an error will cost you, why would
you risk those 2 out of 10,000 (or whatever) errors? Keeping in mind *they
will be very hard or impossible to detect or correct after the fact.* I
don't see the upside.

* Race conditions are, by nature, a bit unpredictable and *very* hard to
diagnose after the fact.

* Hoping that it won't happen isn't a sound strategy. Maybe it won't, maybe
it will. It definitely might.

* If you don't feel like writing your own locking scheme for IP variables,
think about using records instead of arrays. 4D already has an easy to use
locking system for records.

* IP arrays are often used for tasks that would be better served by a
queue. If so, write a real queue using NTK IPC channels, or some other
scheme. (There are several options.)

* If locking your IP arrays is a performance bottleneck, you probably need
an alternative design, not a low-quality implementation of shared resource
management.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

Keisuke Miyako
In reply to this post by KirkBrooks
what if another process removed or changed elements while that process is working its way though the IP array?
would that not become a problem?

but the problem with multi-threaded access in compiled mode is more serious;
you don't want another thread to use the same register while a thread is in the middle of a read/write operation.

p.s.

The "while" loop in your "Lock" method can be an over kill;

why not simply

If (Not (Semaphore ("<>aTestArray";$timeout) ) )

        //..do something..

        CLEAR SEMAPHORE("<>aTestArray")

End if

> 2016/11/01 14:03、Kirk Brooks <[hidden email]> のメール:
> So I'm concluding the whole variable locking fetish is unnecessary.


宮古 啓介
セールス・エンジニア

株式会社フォーディー・ジャパン
〒150-0043
東京都渋谷区道玄坂1-10-2 渋谷THビル6F
Tel: 03-6427-8441
Fax: 03-6427-8449

[hidden email]
www.4D.com/JP

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

Keisuke Miyako
In reply to this post by KirkBrooks
I tend to agree,

in general, semaphores should be used to protect "methods" rather than "objects".
the method may read/write certain documents or access IP variables,
but that is not the most important issue.

some singleton methods may do neither,
but yet require protection.

> 2016/11/01 14:03、Kirk Brooks <[hidden email]> のメール:
> So I'm concluding the whole variable locking fetish is unnecessary.


宮古 啓介
セールス・エンジニア

株式会社フォーディー・ジャパン
〒150-0043
東京都渋谷区道玄坂1-10-2 渋谷THビル6F
Tel: 03-6427-8441
Fax: 03-6427-8449

[hidden email]
www.4D.com/JP

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

KirkBrooks
In reply to this post by David Adams-4
David,
Great points - but, respectfully, does it ever actually happen? Can you
write an example db that does this?

And bear in mind I'm talking about conditions on a client - not the server.
And I'm not talking about situations where an external client might be
attempting to set variables either. Just plain old accessing IP vars from
multiple processes.

On Mon, Oct 31, 2016 at 10:12 PM, David Adams <[hidden email]> wrote:

> Sure, it's absolutely possible to encounter a race condition if you write
> to a shared resource (IP arrays in your case) without a locking mechanism.
> It's likelihood increases in compiled mode and as you add more
> participants. Even if you can't predict how often you'll encounter a race
> condition, you already know that the chance is greater than zero.
> ​...
>
>
> * Hoping that it won't happen isn't a sound strategy. Maybe it won't, maybe
> it will. It definitely might.
>

--
Kirk Brooks
San Francisco, CA
=======================
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

KirkBrooks
In reply to this post by Keisuke Miyako
Miyako,
On Mon, Oct 31, 2016 at 10:16 PM, Keisuke Miyako <[hidden email]>
wrote:

> what if another process removed or changed elements while that process is
> working its way though the IP array?
> would that not become a problem?
>
​What would be the difference between adding and deleting an element? ​

​Granted I didn't test that because I wanted an easy metric to measure but
it seems like the same case - the actions appear to be queued by 4D.​

>
> but the problem with multi-threaded access in compiled mode is more
> serious;
> you don't want another thread to use the same register while a thread is
> in the middle of a read/write operation.
>
​Absolutely, as I noted this is only in respect to a single client without
multi-threading.



> The "while" loop in your "Lock" method can be an over kill;
>
​That's actually what got me thinking about it. I see it in a number of
databases though. ​And if you were to carry this line of thought to it's
logical conclusion such a lock is to ensure the variable will be written
and accept nothing less - which you wouldn't want to do normally. I was
surprised how little overhead that added. On my macBookPro the un-protected
loops took about 2800ms and the protected ones about 3100ms, interpreted.
It's not a significant performance issue in my view.

 --
Kirk Brooks
San Francisco, CA
=======================
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

David Adams-4
In reply to this post by KirkBrooks
> David, Great points - but, respectfully, does it ever actually happen?
Unless 4D prevents it from happening, then it can happen. So, unless 4D
groups multiple lines of code together in a block, you can get different
processes interleaving in overall execution order. Last I heard, can split
execution across processes between lines. (I'm use lines a bit loosely here
as what we write of 1 or 10 lines of code may translate into something else
once compiled.)  That's precisely why you would need a lock. Particularly
compiled.

> Can you write an example db that does this?
Probably, I've seen it done in the past. Would I now? No, I wouldn't bother
trying. Unless 4D *guarantees* the non-standard behavior you're hoping to
exploit, it's risky at best. They don't offer any such reassurance. Also, I
pretty rarely used IP arrays for shared objects for much of anything...I
just don't have a lot of situations where that's the best solution.

> And bear in mind I'm talking about conditions on a client - not the
server.
Doesn't matter. It's a multi-process machine, either way.

> And I'm not talking about situations where an external client might be
> attempting to set variables either.
You mean SET PROCESS VARIABLE, etc.? I forgot about those. I never use
them. While they can be used without trouble if used correctly (and very
carefully), they demonstrate a pretty profound violation of any sensible
concept of scope. I mean, it's bad enough that we don't have private vars
and functions without making it *worse* by giving people a stick to poke
inside of other processes with. Left up to mean, you wouldn't even allow
for public variables, everything would be set through functions. (Can you
tell that I'm reading Bertrand Meyer again?) Those commands just plain
suck. They're like bad globals on steroids. Thank goodness V16 is offering
a very nice alternative for what people (mis)used SET PROCESS VARIABLE for
in the past.

I understand the temptation to think that race conditions won't happen to
you. It's pretty easy to think that way. But since they can, chances are
that they eventually will. That brings you back to what to do about it.
Again, sometimes the situation is harmless (you lose so
low-quality/low-value data), other times it's a big deal (you crash writing
to an element that doesn't exist and/or you scramble key tracking data.)
Like any risk, the first question is "what harm is done if things go
wrong?" It's not worth much effort to prevent a problem that causes no
harm. On the other hand, if the outcome is going to be bad, is it worth a
risk? And if The Bad Thing happens, how will you even know? How will you
recover? Can you even detect it or recover? Again, it depends on what
you're working with. And, for the record, I've seen tons of 4D code that
loads records for writing without checking that their writable, either
because of record locks or the table being in read-write. That's easy to
understand and, in some ways, a less dire problem...but would anyone
suggest it's a strong design or a reliable approach? I hope not. (Again,
unless it's data that doesn't really need to be 100% correct or complete.
There's plenty of that out there in the world.)
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

David Adams-4
In reply to this post by KirkBrooks
> ​What would be the difference between adding and deleting an element? ​

Because you might be writing to an element that doesn't exist. Some
versions/settings, 4D crashes. Other times, it doesn't crash but your array
isn't right. If you've got a group of arrays working together to implement
some kind of data structure, you might end up with 'rows' where the various
columns are about a mixture of different things because of bad luck with
timing. So, you're risking crashing, having bad data, or having meaningless
data. And, again, all very hard to detect and correct after the fact.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

KirkBrooks
In reply to this post by David Adams-4
David,

On Mon, Oct 31, 2016 at 11:34 PM, David Adams <[hidden email]> wrote:

> > Can you write an example db that does this?
> Probably, I've seen it done in the past. Would I now? No, I wouldn't bother
> trying. Unless 4D *guarantees* the non-standard behavior you're hoping to
> exploit, it's risky at best. They don't offer any such reassurance. Also, I
> pretty rarely used IP arrays for shared objects for much of anything...I
> just don't have a lot of situations where that's the best solution.
>

​I think the example I provided establishes the sort of ​situation you
describe. I mean - 400 processes simultaneously writing to the same
variable for 2 seconds.

As I understand it a race condition develops when there is a sequencing
issue - the value Process A will write depends on a value Process B already
wrote. I'm not talking about that either. In this test I don't care what
order the elements are written in (I had to add inconsistency to the delay
periods to force it out of sequence in the first place) or what the value
is - just whether it would be written correctly. Adding the lock wouldn't
prevent that in this case either. If sequence were important you would need
to address it but that wasn't the question I was looking at.

As I mentioned before I didn't look at changing multiple vars at the same
time. So if I have 4 arrays that needed to be updated at the same time the
situation could be different. There's more variability in this case and
setting the lock makes sense.

And I'm not sure 4D guarantees anything. I'm loath too. Buried within
pretty much everyone's EULA is something to the effect of, "we think this
works pretty well but it's up to you to decide if you trust it or not
because we can't guarantee it actually does anything." I may be
paraphrasing. I'm old enough to avoid saying 'never' any more but I'm still
a sucker for strong evidence.


> > And bear in mind I'm talking about conditions on a client - not the
> ​ ​
> server.
> Doesn't matter. It's a multi-process machine, either way.
>
True, but this illustrates that without multi-threading it's still a
sequential processor. Only one process at a time can be executing.



> > And I'm not talking about situations where an external client might be
> > attempting to set variables either.
> You mean SET PROCESS VARIABLE, etc.? I forgot about those. I never use
> ​ ​
> them.

​Me either. I prefer writing records to a messaging table instead. An idea
I believe I got from you. But in this case I suspect the result would be
the same - that 4D doesn't actually execute multiple processes
simultaneously.

The buried lede in this story may actually be how easily 4D handles 400
simultaneous processes.

--
Kirk Brooks
San Francisco, CA
=======================
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

Richard Wright-2
In reply to this post by KirkBrooks
Yes, I’ve seen it happen. 4D language is multi-threaded already - it’s cooperative instead of pre-emptive prior to v16.  A method looping through an interprocess array can easily be paused while another method updates that same array.

------------------------------------------------
Richard Wright
DataDomain
[hidden email]
------------------------------------------------

> Date: Mon, 31 Oct 2016 22:43:48 -0700
> From: Kirk Brooks <[hidden email]>
>
>
> David,
> Great points - but, respectfully, does it ever actually happen? Can you
> write an example db that does this?
>
> And bear in mind I'm talking about conditions on a client - not the server.
> And I'm not talking about situations where an external client might be
> attempting to set variables either. Just plain old accessing IP vars from
> multiple processes.
>
> On Mon, Oct 31, 2016 at 10:12 PM, David Adams <[hidden email]> wrote:
>
>> Sure, it's absolutely possible to encounter a race condition if you write
>> to a shared resource (IP arrays in your case) without a locking mechanism.
>> It's likelihood increases in compiled mode and as you add more
>> participants. Even if you can't predict how often you'll encounter a race
>> condition, you already know that the chance is greater than zero.
>> ​...
>>
>>
>> * Hoping that it won't happen isn't a sound strategy. Maybe it won't, maybe
>> it will. It definitely might.
>>
>
> --
> Kirk Brooks
> San Francisco, CA
> =======================




**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

KirkBrooks
Richard,
That's essentially the type of variance I added to the test by changing the
delay periods. And it's not the question I was looking at.

I may not have stated it clearly enough to begin with: I wanted to know if
competing processes would create collisions attempting to write the same
variable at the same time with the result of lost data because of the
collisions. You and David are talking about protecting the sequence of the
data writes which I'm not concerned with in this case. I'm not saying
that's not an important point, it is, - just not the point I'm looking at.

On Tue, Nov 1, 2016 at 7:49 AM, Richard Wright <
[hidden email]> wrote:

> Yes, I’ve seen it happen. 4D language is multi-threaded already - it’s
> cooperative instead of pre-emptive prior to v16.  A method looping through
> an interprocess array can easily be paused while another method updates
> that same array.
>
> ------------------------------------------------
> Richard Wright
> DataDomain
> [hidden email]
> ------------------------------------------------
>
> > Date: Mon, 31 Oct 2016 22:43:48 -0700
> > From: Kirk Brooks <[hidden email]>
> >
> >
> > David,
> > Great points - but, respectfully, does it ever actually happen? Can you
> > write an example db that does this?
> >
> > And bear in mind I'm talking about conditions on a client - not the
> server.
> > And I'm not talking about situations where an external client might be
> > attempting to set variables either. Just plain old accessing IP vars from
> > multiple processes.
> >
> > On Mon, Oct 31, 2016 at 10:12 PM, David Adams <[hidden email]> wrote:
> >
> >> Sure, it's absolutely possible to encounter a race condition if you
> write
> >> to a shared resource (IP arrays in your case) without a locking
> mechanism.
> >> It's likelihood increases in compiled mode and as you add more
> >> participants. Even if you can't predict how often you'll encounter a
> race
> >> condition, you already know that the chance is greater than zero.
> >> ​...
> >>
> >>
> >> * Hoping that it won't happen isn't a sound strategy. Maybe it won't,
> maybe
> >> it will. It definitely might.
> >>
>
> --
Kirk Brooks
San Francisco, CA
=======================
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

Arnaud de Montard
In reply to this post by KirkBrooks

> Le 1 nov. 2016 à 06:43, Kirk Brooks <[hidden email]> a écrit :
>
> David,
> Great points - but, respectfully, does it ever actually happen? Can you
> write an example db that does this?

In v5 we had the multiprocess for the 1st time. To manage my processes, I used a "process descriptor" made of 5 or 6 arrays, let's say something like this:
   process number | parent process number | form name | table number | record PK
Each time a process was created, ended, modified, these arrays were updated.
4D was much sloooooower than now, and I was quite fast in clicking.
I soon realized that my arrays became unsynchronized because I did not use a semaphore.

--
Arnaud de Montard


**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

Jody Bevan-2
Just to add my 2 cents worth to this.

When one writes a large system with many things happening (potentially) at the same time one needs to code extremely defensively. This means protecting against all kinds of potential problems that have the opportunity to arise. As well, trapping if they do arise and writing a log or other ways that you as the programmer (because users will not tell you) are informed of the issues that you never thought would happen (because they will). Hopefully your logs (or other means) give you enough information so you can track it down and write the code better.

The alternative I lived through many years ago with my own code. As the application grows the strange behaviour grows. Tracking it down becomes near impossible. Therefore we had to go back in and write the code the way we should have written to begin with. Lots of back work, but all of us learned - hopefully.

Therefore creating standards of code, standards of methods, documentation, etc that are ALWAYS done. This is so that the ‘not likely to happen’ just cannot happen as far as you know because of the defensive code you have written. Still though - being notified so you can look back at your code and the data at that site and get that Ahh moment and write the code even better.

We implemented a feature set with an OEM product we had installed at many sites >100 with many 100+ concurrent users. We would collect various ‘problems’ at each site, and each night all the sites ‘called home’ and we recorded them in our local server for this feature. I could then look at these and be proactive in understanding what was happening that was a surprise. It might mean there was no problem, but we could write things better so that the issue didn’t occur. It could mean that there was a problem and we wrote our code to protect that. It could mean speed, convince, better user interface. All in all a great tool, and learned a lot about our code, coding, and user interaction with our system.

I may be paranoid, but I find writing protective code a better way to rest at night, weekends, and holidays knowing that the code is written well, though it takes much longer to get it that way (why I like our shell).

Version 15 and version 16 are giving us a great set of commands so that we can write more defensively than we could before. It is giving us new features that let us write things we had to jump through many hoops to accomplish and ensure got done. Part of it for us is that our shell was based on a version of 4D with version 11/12. Our new shell is taking advantage of the new versions of 4D. An exciting time for us.

Jody

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

Bernd Fröhlich
In reply to this post by KirkBrooks
Kirk Brooks wrote:

> True, but this illustrates that without multi-threading it's still a
> sequential processor. Only one process at a time can be executing.

Yes, BUT...

The thing about semaphores is, that they are atomic commands (the command is guranteed to execute without any other process interrupting it).
And semaphore is the ONLY atomic command, all the other commands can be split in the middle by the systems scheduler.

Your example might work, as you are only adding to the array but it will break, when you also delete array elements.
Consider this:

Process A adds array elements to an IP array and Process B deletes elements.
Works fine, as long as the scheduler does not interrupt one of those commands.

If the array has zero elements and process A tries to create a new one, the sheduler might switch processes when the element is created, but before a value is assigned.
Then process B deletes an element.
Now if it´s processes A time to continue, it tries to write the value to an array element that does not exist anymore. BAD!

That´s why you better protect access to IP arrays/variables with semaphores.

It is very hard to simulate the above since lots of conditions come into play and you MIGHT get lucky that it never happens to you, but if it DOES happen - good luck with finding the error.

Greetings from Germany,
Bernd Fröhlich
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

Tim Nevels-2
In reply to this post by KirkBrooks
On Nov 1, 2016, at 2:00 PM, Bernd Fröhlich wrote:

> Kirk Brooks wrote:
>
>> True, but this illustrates that without multi-threading it's still a
>> sequential processor. Only one process at a time can be executing.
>
> Yes, BUT...
>
> The thing about semaphores is, that they are atomic commands (the command is guranteed to execute without any other process interrupting it).
> And semaphore is the ONLY atomic command, all the other commands can be split in the middle by the systems scheduler.

I’m not sure I agree that the ONLY 4D command in the language that is atomic is “Semaphore”. I’m guessing there are others. Gonna need a 4D engineer like Laurent Esnault that knows the deep internals of 4D’s code to know for sure.

> Your example might work, as you are only adding to the array but it will break, when you also delete array elements.
> Consider this:
>
> Process A adds array elements to an IP array and Process B deletes elements.
> Works fine, as long as the scheduler does not interrupt one of those commands.
>
> If the array has zero elements and process A tries to create a new one, the sheduler might switch processes when the element is created, but before a value is assigned.
> Then process B deletes an element.
> Now if it´s processes A time to continue, it tries to write the value to an array element that does not exist anymore. BAD!
>
> That´s why you better protect access to IP arrays/variables with semaphores.
>
> It is very hard to simulate the above since lots of conditions come into play and you MIGHT get lucky that it never happens to you, but if it DOES happen - good luck with finding the error.

The rule that I have always followed is there is no need to protect simple IP variables like real, longint, text, etc. I can’t see how reading or writing to a single, simple IP variable could cause any issues with multiple processes.

I treat IP arrays differently because usually when you are manipulating an array there are multiple step involved in getting and setting values in the array. You may have to check the array size first, then you APPEND TO ARRAY or INSERT IN ARRAY. Or you do a "Find in array” and then you read or write an element in the array. The point is the read or write to the array requires more than 1 line of 4D code, and I am working under the assumption that a single line of 4D code will run atomically.

Just think of the problems if “<>counter_l := <>counter_l + 1” were to be interrupted in the middle of execution. First you get the value of <>counter_l, then there is a context switch and another process changes <>counter_l, then your process resumes and it takes its own <>counter_l value it had before the context switch, adds 1, and then stores the result in <>counter_l.  

I might be wrong about “one line of 4D code runs atomically”. But and again you need somebody like Laurent Esnault — maybe Miyako or Thomas Maul knows — to tell you how the interpreter works. And also how the 4D compiler works. Could be compiled code acts a bit differently that interpreted at this super low level.  

So if I am going to work on an IP array, I protect that block of code with a semaphore. And of course if I are dealing with a group of related IP arrays it’s absolutely critical to protect them with a semaphore. You want all the operations on all the IP arrays in the group to take place as a single action. There are too many chances for a context switch with so many lines of code.

I think Kirk was asking about the “<>counter_l := <>counter_l + 1” example not IP array updates. I see no reason to protect simple IP variables that are just getting a value set or retrieving a value. IP arrays are another story and the code block does need to be protected with a semaphore.

Tim

********************************************
Tim Nevels
Innovative Solutions
785-749-3444
[hidden email]
********************************************

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

David Adams-4
> I’m not sure I agree that the ONLY 4D command in the language that is
atomic is “Semaphore”.
> I’m guessing there are others. Gonna need a 4D engineer like Laurent
Esnault that knows the
> deep internals of 4D’s code to know for sure.

For what it's worth, I was working at 4D in Cupertino when 4D Server
shipped. We had big arguments in the US office about IP communications and
semaphores. (Most people really don't love semaphores.) Semaphores turned
out to be the one and only reliable mechanism available. Unless something
has changed, that's still going to be true.

I'm sure there are a narrow set of cases where not using semaphores to lock
a shared read-write resource won't cause any harm. It's easy to come up
with cases and I'd bet we all have some of each. Say you have an IP
variable like <>TAB and you assign in Char(13) more than once. No harm. But
for groups of arrays (or even single arrays), it's pretty much not save to
read or write without a lock unless the arrays are static. And it's
*totally* normal to have a static array. Initialize it once at startup (or
on first use) and that's it. No modifications, just lookups. No elements
change position or value, the array never changes size. (Some languages
have a construct for this type of list, we just have arrays and how we use
them.)  No need for a semaphore there either.

I guess the rule is: If you think that you might need a semaphore lock, you
do ;-)
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

Douglas von Roeder
On Tue, Nov 1, 2016 at 2:57 PM, David Adams <[hidden email]> wrote:

> For what it's worth, I was working at 4D in Cupertino when 4D Server
> shipped. We had big arguments in the US office about IP communications and
> semaphores. (Most people really don't love semaphores.) Semaphores turned
> out to be the one and only reliable mechanism available. Unless something
> has changed, that's still going to be true.
>


LR addressed that question directly at a Summit - don't know the year but
it was post-Schaumburg. The only command not susceptible to timeslicing is
Semaphore.




--
Douglas von Roeder
949-336-2902
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

Jeffrey Kain
Well, probably the only command GUARANTEED to be atomic is Semaphore. Other commands may happen to be in the current version, or were in previous versions... but Semaphore always has been atomic and always will be.

> On Nov 1, 2016, at 6:07 PM, Douglas von Roeder <[hidden email]> wrote:
>
> LR addressed that question directly at a Summit - don't know the year but
> it was post-Schaumburg. The only command not susceptible to timeslicing is
> Semaphore.

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

Tom Dillon
In reply to this post by David Adams-4
David Adams wrote:

>Say you have an IP variable like
><>TAB and you assign in Char(13) more than once. No harm.

Unless the name of the variable means what it says. :-7

--
   ------------------------------------------------------------------
   Tom Dillon                                           825 N. 500 W.
   DataCraft                                           Moab, UT 84532
   [hidden email]                           720/209-6502
   ------------------------------------------------------------------
        Maybe we only think we have consciousness. --- Sunastar
   ------------------------------------------------------------------


**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Does it matter if you lock an IP object before updating?

Douglas von Roeder
In reply to this post by Jeffrey Kain
Jeffrey:

That was how I took it.

There was a NUG thread on atomicity some time back. Perhaps there's some
insight in that thread?

Insofar a not Semaphoring shared resources, that'll work fine until it
doesn't. Unfortunately, when it doesn't, tracking it could be, essentially,
impossible.



--
Douglas von Roeder
949-336-2902

On Tue, Nov 1, 2016 at 3:19 PM, Jeffrey Kain <[hidden email]> wrote:

> Well, probably the only command GUARANTEED to be atomic is Semaphore.
> Other commands may happen to be in the current version, or were in previous
> versions... but Semaphore always has been atomic and always will be.
>
> > On Nov 1, 2016, at 6:07 PM, Douglas von Roeder <[hidden email]>
> wrote:
> >
> > LR addressed that question directly at a Summit - don't know the year but
> > it was post-Schaumburg. The only command not susceptible to timeslicing
> is
> > Semaphore.
>
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************
>
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
12