New record numbering system and CALL WORKER

classic Classic list List threaded Threaded
42 messages Options
123
Reply | Threaded
Open this post in threaded view
|

New record numbering system and CALL WORKER

4D Tech mailing list
Using v17 ...

I'm looking at ways to update our ancient record numbering system, and I
think I should be able to do it efficiently with CALL WORKER, but having a
bit of trouble getting my head around  it.

The record numbers are like invoice numbers: each must be unique, and we
don't want to have any gaps in the numbering sequence. So we need to keep
track of any record numbers that are created and then abandoned (eg if a
user creates a new record and then cancels it).
The old system works well but it uses semaphores and SET/GET PROCESS
VARIABLE and DELAY PROCESS. I isn't very efficient, especially when a large
number of records are being imported and each one needs a new record number.
Record numbers must be assigned synchronously - processing must not
continue until the new record number has been assigned. Also, this
frequently needs to be done in a context where there is no active form, so
CALL FORM won't work.

As I see it, we'll need to call a worker, which calls the worker that
assigns the record number.
The first worker has to wait until the second one has created the new
number.
Then the original method, that called the first worker, must wait until the
first worker  gets the new number. It seems that this won't be much better
than the old method.

Am I completely barking up the wrong tree? Is this not an appropriate use
for workers? Or have I misunderstood something?

Pat

--
*************************************************
CatBase - Top Dog in Data Publishing
tel: +44 (0) 207 118 7889
w: http://www.catbase.com
skype: pat.bensky
*************************************************
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list

> On Aug 25, 2018, at 9:10 AM, Pat Bensky via 4D_Tech <[hidden email]> wrote:
>
> As I see it, we'll need to call a worker, which calls the worker that
> assigns the record number.
> The first worker has to wait until the second one has created the new
> number.
> Then the original method, that called the first worker, must wait until the
> first worker  gets the new number. It seems that this won't be much better
> than the old method.
>
> Am I completely barking up the wrong tree? Is this not an appropriate use
> for workers? Or have I misunderstood something?

It would have been really nice if CALL WORKER was implemented like EXECUTE METHOD and could return a value. You can do this on your own, but it is quite a bit  of work to do it in a general way.

For your application, all you need is a shared object (no worker needed). Only one process at a time will be able to modify the counter. Some startup method could initialize the counter to the starting value.

C_LONGINT($0)

Use (Storage.myCounter)
  Storage.myCounter.value:= Storage.myCounter.value+1
  $0:= Storage.myCounter.value
End use


John DeSoi, Ph.D.

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Pat,
A couple of thoughts since it sounds like you're refactoring your code (all
pseudo code, v17):
1) don't use this number as a key field.

2) The counter record:

[COUNTER]

ID

name  - optional but a good idea

_data  - object field: { nextNumber: 0, returned: []}

3) Counter_get_next(text) -> long

$1 is name of counter series

Put [COUNTER] in read/write

Query for record

Wait until record is unlocked

`----

if(COUNTER._data.returned.length>0)` use the first one here

$0:=
COUNTER._data.returned.
shift
()

else

$0:=
COUNTER._data.nextNumber

COUNTER._data.nextNumber:=
COUNTER._data.nextNumber
+1

end if

SAVE RECORD

UNLOAD RECORD


4) Counter_save_unused (text; long)

Same idea except you just push $2 onto 'returned'. You could add a sort
command to make sure the smallest returned value is on top if that's an
issue.


Several years ago I stopped using invoice or PO numbers (or phone numbers
or SSNs or any data value) as key fields. This greatly simplifies the issue
you may have had about the 'unused' numbers that come up when data entry is
canceled. If you're not using that number as a key field you don't need to
assign it until the transaction completes. But if you've got some other
situations that result in 'returned' numbers it's not big deal.

This approach let's 4D handle the record locking and request que and as
long as you don't call it within a transaction just blissfully works.



On Sat, Aug 25, 2018 at 7:11 AM Pat Bensky via 4D_Tech <[hidden email]>
wrote:

> Using v17 ...
>
> I'm looking at ways to update our ancient record numbering system, and I
> think I should be able to do it efficiently with CALL WORKER, but having a
> bit of trouble getting my head around  it.
>
> The record numbers are like invoice numbers: each must be unique, and we
> don't want to have any gaps in the numbering sequence. So we need to keep
> track of any record numbers that are created and then abandoned (eg if a
> user creates a new record and then cancels it).

--
Kirk Brooks
San Francisco, CA
=======================

*We go vote - they go home*
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Thanks John.
But how would that work in a multi-user setting? Whatever method is used
for assigning the new numbers, it must run on the server, otherwise we're
back to the problem of using semaphores, checking record locking, etc.
Also I should point out that there could be a large number of tables for
which the record numbering system needs to be used. We have a RecordNumbers
table which has one record for each table that requires this function. so
we must have ONE process that accesses and updates that table.
PB

On Sat, 25 Aug 2018 at 15:49, John DeSoi via 4D_Tech <[hidden email]>
wrote:

>
> > On Aug 25, 2018, at 9:10 AM, Pat Bensky via 4D_Tech <
> [hidden email]> wrote:
> >
> > As I see it, we'll need to call a worker, which calls the worker that
> > assigns the record number.
> > The first worker has to wait until the second one has created the new
> > number.
> > Then the original method, that called the first worker, must wait until
> the
> > first worker  gets the new number. It seems that this won't be much
> better
> > than the old method.
> >
> > Am I completely barking up the wrong tree? Is this not an appropriate use
> > for workers? Or have I misunderstood something?
>
> It would have been really nice if CALL WORKER was implemented like EXECUTE
> METHOD and could return a value. You can do this on your own, but it is
> quite a bit  of work to do it in a general way.
>
> For your application, all you need is a shared object (no worker needed).
> Only one process at a time will be able to modify the counter. Some startup
> method could initialize the counter to the starting value.
>
> C_LONGINT($0)
>
> Use (Storage.myCounter)
>   Storage.myCounter.value:= Storage.myCounter.value+1
>   $0:= Storage.myCounter.value
> End use
>
>
> John DeSoi, Ph.D.
>
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************



--
*************************************************
CatBase - Top Dog in Data Publishing
tel: +44 (0) 207 118 7889
w: http://www.catbase.com
skype: pat.bensky
*************************************************
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Hi Kirk,
What you say makes sense, but the record number as key field is so integral
to the way the program works, it would be a nightmare to try and change it
:) If I was starting a project from scratch, that would be a different
story.
Pat

On Sat, 25 Aug 2018 at 16:20, Kirk Brooks via 4D_Tech <[hidden email]>
wrote:

> Pat,
> A couple of thoughts since it sounds like you're refactoring your code (all
> pseudo code, v17):
> 1) don't use this number as a key field.
>
> 2) The counter record:
>
> [COUNTER]
>
> ID
>
> name  - optional but a good idea
>
> _data  - object field: { nextNumber: 0, returned: []}
>
> 3) Counter_get_next(text) -> long
>
> $1 is name of counter series
>
> Put [COUNTER] in read/write
>
> Query for record
>
> Wait until record is unlocked
>
> `----
>
> if(COUNTER._data.returned.length>0)` use the first one here
>
> $0:=
> COUNTER._data.returned.
> shift
> ()
>
> else
>
> $0:=
> COUNTER._data.nextNumber
>
> COUNTER._data.nextNumber:=
> COUNTER._data.nextNumber
> +1
>
> end if
>
> SAVE RECORD
>
> UNLOAD RECORD
>
>
> 4) Counter_save_unused (text; long)
>
> Same idea except you just push $2 onto 'returned'. You could add a sort
> command to make sure the smallest returned value is on top if that's an
> issue.
>
>
> Several years ago I stopped using invoice or PO numbers (or phone numbers
> or SSNs or any data value) as key fields. This greatly simplifies the issue
> you may have had about the 'unused' numbers that come up when data entry is
> canceled. If you're not using that number as a key field you don't need to
> assign it until the transaction completes. But if you've got some other
> situations that result in 'returned' numbers it's not big deal.
>
> This approach let's 4D handle the record locking and request que and as
> long as you don't call it within a transaction just blissfully works.
>
>
>
> On Sat, Aug 25, 2018 at 7:11 AM Pat Bensky via 4D_Tech <
> [hidden email]>
> wrote:
>
> > Using v17 ...
> >
> > I'm looking at ways to update our ancient record numbering system, and I
> > think I should be able to do it efficiently with CALL WORKER, but having
> a
> > bit of trouble getting my head around  it.
> >
> > The record numbers are like invoice numbers: each must be unique, and we
> > don't want to have any gaps in the numbering sequence. So we need to keep
> > track of any record numbers that are created and then abandoned (eg if a
> > user creates a new record and then cancels it).
>
> --
> Kirk Brooks
> San Francisco, CA
> =======================
>
> *We go vote - they go home*
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************



--
*************************************************
CatBase - Top Dog in Data Publishing
tel: +44 (0) 207 118 7889
w: http://www.catbase.com
skype: pat.bensky
*************************************************
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
Pat,

On Sat, Aug 25, 2018 at 8:25 AM Pat Bensky via 4D_Tech <[hidden email]>
wrote:

> What you say makes sense, but the record number as key field is so integral
> to the way the program works, it would be a nightmare to try and change it
>

Understood. But this approach still works, you just have more values in the
'returned' collection. Of course I'm assuming you're not going to have
hundreds of returned counters. That could make a difference.

And both methods can be run EOS to speed up the multiuser environment.

Try it out. I know that years ago that much querying and checking record
locking would have been slow. It's not now.

--
Kirk Brooks
San Francisco, CA
=======================

*We go vote - they go home*
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
In reply to this post by 4D Tech mailing list
On Aug 25, 2018, at 10:33 AM, Pat Bensky wrote:

> Thanks John.
> But how would that work in a multi-user setting? Whatever method is used
> for assigning the new numbers, it must run on the server, otherwise we're
> back to the problem of using semaphores, checking record locking, etc.
> Also I should point out that there could be a large number of tables for
> which the record numbering system needs to be used. We have a RecordNumbers
> table which has one record for each table that requires this function. so
> we must have ONE process that accesses and updates that table.

I like John’s idea of using the new shared object “Storage”. This has a built-in, super efficient, locking mechanism using the “Use/End Use” system. My first thought were that this could be the ultimate “next sequence number” system if you have already used the “Sequence number” attribute for a table (which is already the ultimate “next sequence number” system for a table in terms of speed and reliability in a multi-user and transactional system.

So here is an idea for a basic sequence number system based on shared “Storage” object. The goals are:

- Make it as fast as possible
- Make it work on single-user and client/server with the least amount of code
- Use built-in 4D features wherever you can
- Works anywhere and everywhere, no special cases to deal with
   - any process cooperative or preemptive
   - in triggers
   - inside or outside transactions
   - from a form or from a process with no UI
- Don’t care about “lost numbers” (holes in the sequence numbers are OK)

Sorry Pat, you’ll have to complicate the system to make it work for your specific needs, but that should be doable with a more complex storage object. I just wanted to present the concept I have here in the simplest terms for easy digestion.

1. Initialization - this needs to be done once for a database application. If you are running single-user, do this in the “On Startup” method. If you are running client/server do this in “On Server Startup”

Use (Storage)
   Storage.myNextNumber:=1
End use

2. GetNextNumber - this is a method that will return the next number from the shared storage. Set the method property for “Execute on server”. This will make it work in single-user or client/server with no code changes.

// Method: GetNextNumber

C_LONGINT($0)

Use (Storage)
   $0:=Storage.myNextNumber
   Storage.myNextNumber:=$0+1 // increment sequence
End use

3. Permanent storage of the next number - If you want this to maintain it’s value across launches of the application, use a table in the database. So in the “initialization” part above instead of setting "Storage.myNextNumber:=1” you would query a table and get the next number value from a field in a record. When the database quits you update that value so it is available for the next “On Startup” or “On Server Startup” You would put this code in the “On Exit” or “On Server Shutdown”.

4. What if my database crashes and my code does not get a chance to do a proper save of the next number to the database? A valid consideration and one I think should be addressed. We all know that 4D will crash sometimes, so best to be prepared and handle it as best we can.

How about in the “initialization” part you also start a new process whose job is to save the next number from the storage object to a database record. Have this process run as often as you are comfortable with. If you are journaling your data file — and you should be — the record update will be in the journal file so that if 4D crashes, on the next startup it will integrate the change into the database and you’ve not lost anything.

You could bypass the whole shared storage approach and just do database operations on records, but I would speculate that using shared storage is just a tad bit faster than doing a query, wait for record to be unlocked, increment, save record, unload record operation. Might not be, I’ve not done any testing to see which is faster. Just seems like less overhead to not call the database engine every time you need a next number. But it does introduce a "cache within a cache” situation. And the potential risk of losing that last sequence number before your process “flushes” it to the database cache and writes to the journal file. But one of my above stated goals was maximum speed, so I think this will buy you a tiny bit of extra speed.

Here is some example code for saving the next number storage value to the database. Put this in a method that is called in either “On Startup” or “On Server Startup”. To make is super simple, [MyNextNumber] table only contains 1 record and 1 field. And no other process is ever allowed to do any write operations to table [MyNextNumber]. The only process to modify it is “SaveMyNextNumber” then you don’t have to worry about possible record locking issues.

$processID_l:=New Process(“SaveMyNextNumber”;0)

// Method: SaveMyNextNumber

ALL RECORDS([MyNextNumber])
if (Records in selection([MyNextNumber])=0)
   CREATE RECORD([MyNextNumber])
End if

Use (Storage)
   [MyNextNumber]NextNumber:=Storage.myNextNumber
End use

SAVE RECORD([MyNextNumber])
UNLOAD RECORD([MyNextNumber])

Great topic for discussion Pat, glad you brought it up now that we have v17 and all the new features.

And great idea from John about using Storage that got me thinking about this age old next sequence number issue and how to best implement it.

I wonder what other developers think about this crazy idea?

Tim

*****************************************
Tim Nevels
Innovative Solutions
785-749-3444
[hidden email]
*****************************************

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
> Use (Storage)
>   [MyNextNumber]NextNumber:=Storage.myNextNumber
> End use

This is not necessary. For just reading you don’t need to use „use“.
For generating sequence numbers, you have to use „auto increment“. Nothing else.
For invoice numbers, you now can save the number to the database within transactions with SUSPEND/RESUME TRANSACTION (of course, locking and waiting if necessary…)
--

Grüße/Regards,
[heubach-media] | Christian Sakowski
[hidden email]
iChat/AIM: SakowskiF
Tel: +49/(0)40/52 10 59-23



> Am 25.08.2018 um 21:10 schrieb Tim Nevels via 4D_Tech <[hidden email]>:
>
> On Aug 25, 2018, at 10:33 AM, Pat Bensky wrote:
>
>> Thanks John.
>> But how would that work in a multi-user setting? Whatever method is used
>> for assigning the new numbers, it must run on the server, otherwise we're
>> back to the problem of using semaphores, checking record locking, etc.
>> Also I should point out that there could be a large number of tables for
>> which the record numbering system needs to be used. We have a RecordNumbers
>> table which has one record for each table that requires this function. so
>> we must have ONE process that accesses and updates that table.
>
> I like John’s idea of using the new shared object “Storage”. This has a built-in, super efficient, locking mechanism using the “Use/End Use” system. My first thought were that this could be the ultimate “next sequence number” system if you have already used the “Sequence number” attribute for a table (which is already the ultimate “next sequence number” system for a table in terms of speed and reliability in a multi-user and transactional system.
>
> So here is an idea for a basic sequence number system based on shared “Storage” object. The goals are:
>
> - Make it as fast as possible
> - Make it work on single-user and client/server with the least amount of code
> - Use built-in 4D features wherever you can
> - Works anywhere and everywhere, no special cases to deal with
>   - any process cooperative or preemptive
>   - in triggers
>   - inside or outside transactions
>   - from a form or from a process with no UI
> - Don’t care about “lost numbers” (holes in the sequence numbers are OK)
>
> Sorry Pat, you’ll have to complicate the system to make it work for your specific needs, but that should be doable with a more complex storage object. I just wanted to present the concept I have here in the simplest terms for easy digestion.
>
> 1. Initialization - this needs to be done once for a database application. If you are running single-user, do this in the “On Startup” method. If you are running client/server do this in “On Server Startup”
>
> Use (Storage)
>   Storage.myNextNumber:=1
> End use
>
> 2. GetNextNumber - this is a method that will return the next number from the shared storage. Set the method property for “Execute on server”. This will make it work in single-user or client/server with no code changes.
>
> // Method: GetNextNumber
>
> C_LONGINT($0)
>
> Use (Storage)
>   $0:=Storage.myNextNumber
>   Storage.myNextNumber:=$0+1 // increment sequence
> End use
>
> 3. Permanent storage of the next number - If you want this to maintain it’s value across launches of the application, use a table in the database. So in the “initialization” part above instead of setting "Storage.myNextNumber:=1” you would query a table and get the next number value from a field in a record. When the database quits you update that value so it is available for the next “On Startup” or “On Server Startup” You would put this code in the “On Exit” or “On Server Shutdown”.
>
> 4. What if my database crashes and my code does not get a chance to do a proper save of the next number to the database? A valid consideration and one I think should be addressed. We all know that 4D will crash sometimes, so best to be prepared and handle it as best we can.
>
> How about in the “initialization” part you also start a new process whose job is to save the next number from the storage object to a database record. Have this process run as often as you are comfortable with. If you are journaling your data file — and you should be — the record update will be in the journal file so that if 4D crashes, on the next startup it will integrate the change into the database and you’ve not lost anything.
>
> You could bypass the whole shared storage approach and just do database operations on records, but I would speculate that using shared storage is just a tad bit faster than doing a query, wait for record to be unlocked, increment, save record, unload record operation. Might not be, I’ve not done any testing to see which is faster. Just seems like less overhead to not call the database engine every time you need a next number. But it does introduce a "cache within a cache” situation. And the potential risk of losing that last sequence number before your process “flushes” it to the database cache and writes to the journal file. But one of my above stated goals was maximum speed, so I think this will buy you a tiny bit of extra speed.
>
> Here is some example code for saving the next number storage value to the database. Put this in a method that is called in either “On Startup” or “On Server Startup”. To make is super simple, [MyNextNumber] table only contains 1 record and 1 field. And no other process is ever allowed to do any write operations to table [MyNextNumber]. The only process to modify it is “SaveMyNextNumber” then you don’t have to worry about possible record locking issues.
>
> $processID_l:=New Process(“SaveMyNextNumber”;0)
>
> // Method: SaveMyNextNumber
>
> ALL RECORDS([MyNextNumber])
> if (Records in selection([MyNextNumber])=0)
>   CREATE RECORD([MyNextNumber])
> End if
>
> Use (Storage)
>   [MyNextNumber]NextNumber:=Storage.myNextNumber
> End use
>
> SAVE RECORD([MyNextNumber])
> UNLOAD RECORD([MyNextNumber])
>
> Great topic for discussion Pat, glad you brought it up now that we have v17 and all the new features.
>
> And great idea from John about using Storage that got me thinking about this age old next sequence number issue and how to best implement it.
>
> I wonder what other developers think about this crazy idea?
>
> Tim
>
> *****************************************
> Tim Nevels
> Innovative Solutions
> 785-749-3444
> [hidden email]
> *****************************************
>
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************


--
heubach media
Osterfeldstr. 12-14 | Haus 1 | Eingang Nord
22529 Hamburg
tel: 040 / 52 10 59 - 10 | fax: -99
mail: [hidden email]
home: www.heubach-media.de
Geschäftsführer|CEO: Matthias Heubach

Mieten Sie Ihre Computer, iPads & Drucker für Ihre Events bei:
http://www.milo-rental.com

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.
 
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the
material in this e-mail is strictly forbidden.
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
On Aug 25, 2018, at 2:25 PM, Christian Sakowski <[hidden email]> wrote:

> This is not necessary. For just reading you don’t need to use „use“.
> For generating sequence numbers, you have to use „auto increment“. Nothing else.
> For invoice numbers, you now can save the number to the database within transactions with SUSPEND/RESUME TRANSACTION (of course, locking and waiting if necessary…)

Hi Christian,

You are correct, it is not necessary to “Use” a shared object if you are only going to read it. I’m still getting used to this new 4D feature. Thanks for reminding me of that.

Regarding sequence numbers and using the “auto increment” or the “Sequence number” command, I always use that. I agree it’s the best, period.

But this is really a discussion of how one might use shared storage. Rightly or wrongly. Trying to understand how to use this new 4D feature and gather some ideas on possible ways of using it. I’m still learning and trying to understand it.

Tim

*****************************************
Tim Nevels
Innovative Solutions
785-749-3444
[hidden email]
*****************************************

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
Tim,

i mainly use Storage as „constants“. I store global attributes and user specific options and access rights. Those are in fact mostly read only.

--
Grüße/Regards,
[heubach-media] | Christian Sakowski
[hidden email]
iChat/AIM: SakowskiF
Tel: +49/(0)40/52 10 59-23



> Am 25.08.2018 um 22:57 schrieb Tim Nevels <[hidden email]>:
>
> On Aug 25, 2018, at 2:25 PM, Christian Sakowski <[hidden email]> wrote:
>
>> This is not necessary. For just reading you don’t need to use „use“.
>> For generating sequence numbers, you have to use „auto increment“. Nothing else.
>> For invoice numbers, you now can save the number to the database within transactions with SUSPEND/RESUME TRANSACTION (of course, locking and waiting if necessary…)
>
> Hi Christian,
>
> You are correct, it is not necessary to “Use” a shared object if you are only going to read it. I’m still getting used to this new 4D feature. Thanks for reminding me of that.
>
> Regarding sequence numbers and using the “auto increment” or the “Sequence number” command, I always use that. I agree it’s the best, period.
>
> But this is really a discussion of how one might use shared storage. Rightly or wrongly. Trying to understand how to use this new 4D feature and gather some ideas on possible ways of using it. I’m still learning and trying to understand it.
>
> Tim
>
> *****************************************
> Tim Nevels
> Innovative Solutions
> 785-749-3444
> [hidden email]
> *****************************************
>


--
heubach media
Osterfeldstr. 12-14 | Haus 1 | Eingang Nord
22529 Hamburg
tel: 040 / 52 10 59 - 10 | fax: -99
mail: [hidden email]
home: www.heubach-media.de
Geschäftsführer|CEO: Matthias Heubach

Mieten Sie Ihre Computer, iPads & Drucker für Ihre Events bei:
http://www.milo-rental.com

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.
 
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the
material in this e-mail is strictly forbidden.
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
On Aug 25, 2018, at 4:33 PM, Christian Sakowski <[hidden email]> wrote:

> i mainly use Storage as „constants“. I store global attributes and user specific options and access rights. Those are in fact mostly read only.

Hi Christian,

And that is a great use of “Storage”. I’ll be doing the same thing in new databases I create.

But lets explore other possibilities. What could “Storage” be use for in other cases? Is it a good use of “Storage”?

One of the great things about 4D is that features designed and intended to be use for one situation turn out to be very useful in other situations. And I love the “big brains” on the iNUG exposing these new uses.

Tim

*****************************************
Tim Nevels
Innovative Solutions
785-749-3444
[hidden email]
*****************************************

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Hi Pat,

Set the method property to "Execute on Server".

This shared object/collection could also have a collection of numbers that need to be reused if you require that there are no missing numbers in your sequence.

Of course, you'll have to get the database involved at some level to handle startup and shutdown and the case where the server crashed without writing the last sequence number used.

John DeSoi, Ph.D.


> On Aug 25, 2018, at 10:21 AM, Pat Bensky via 4D_Tech <[hidden email]> wrote:
>
> Thanks John.
> But how would that work in a multi-user setting? Whatever method is used
> for assigning the new numbers, it must run on the server, otherwise we're
> back to the problem of using semaphores, checking record locking, etc.
> Also I should point out that there could be a large number of tables for
> which the record numbering system needs to be used. We have a RecordNumbers
> table which has one record for each table that requires this function. so
> we must have ONE process that accesses and updates that table.
> PB

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
In reply to this post by 4D Tech mailing list
> But lets explore other possibilities.

Hopefully not. You should avoid Storage where you can. They are slower and can produce (if not well programmed) dead locks.
--

Grüße/Regards,
[heubach-media] | Christian Sakowski
[hidden email]
iChat/AIM: SakowskiF
Tel: +49/(0)40/52 10 59-23



> Am 25.08.2018 um 23:44 schrieb Tim Nevels via 4D_Tech <[hidden email]>:
>
> On Aug 25, 2018, at 4:33 PM, Christian Sakowski <[hidden email]> wrote:
>
>> i mainly use Storage as „constants“. I store global attributes and user specific options and access rights. Those are in fact mostly read only.
>
> Hi Christian,
>
> And that is a great use of “Storage”. I’ll be doing the same thing in new databases I create.
>
> But lets explore other possibilities. What could “Storage” be use for in other cases? Is it a good use of “Storage”?
>
> One of the great things about 4D is that features designed and intended to be use for one situation turn out to be very useful in other situations. And I love the “big brains” on the iNUG exposing these new uses.
>
> Tim
>
> *****************************************
> Tim Nevels
> Innovative Solutions
> 785-749-3444
> [hidden email]
> *****************************************
>
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************


--
heubach media
Osterfeldstr. 12-14 | Haus 1 | Eingang Nord
22529 Hamburg
tel: 040 / 52 10 59 - 10 | fax: -99
mail: [hidden email]
home: www.heubach-media.de
Geschäftsführer|CEO: Matthias Heubach

Mieten Sie Ihre Computer, iPads & Drucker für Ihre Events bei:
http://www.milo-rental.com

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.
 
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the
material in this e-mail is strictly forbidden.
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
Humm... interesting comment. Why do you say Storage is slower and can produce deadlocks? Tell us why you believe this.

Was my example an example of “not well programmed”? Be honest. Give me real, solid,  constructive criticism. This is a forum for exchanging ideas and learning. I’ve been wrong many times in the past. I may be wrong now. I am interested in learning and improving and understanding more.

Please elaborate on why “you should avoid Storage where you can”.  

I thought Storage was a wonderful new 4D feature. But you say it should be avoided. Why?

Tim

Sent from my iPad

> On Aug 26, 2018, at 3:03 AM, Christian Sakowski <[hidden email]> wrote:
>
> Hopefully not. You should avoid Storage where you can. They are slower and can produce (if not well programmed) dead locks.
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
I totally agree with Christian, in fact,
the point (slow, should be avoided if possible) was right there in the Summit keynote.

on the point of "slow"
of course it is a relative thing and perhaps not a big deal depending on the context,
but you only have to run code with Storage and the speed difference is quite palpable.
it is not a 4D specific phenomena, it is intrinsic to how blocking based on mutex works.

on the point of "good coding"
4D forces you to forfeit exclusive access with the "Use/End use" code block,
so unless you implicitly use storage in a stacked subroutine (super bad idea),
it is hard to violate the FILO tule unintentionally.

a more subtle gotcha is to make the mistake of
using objects and collections as if they were values, not references.
it's not limited to storage, but it can really bite you when you make the mistake with storage.

2018/08/26 17:26、Tim Nevels via 4D_Tech <[hidden email]<mailto:[hidden email]>>のメール:

Humm... interesting comment. Why do you say Storage is slower and can produce deadlocks? Tell us why you believe this.



**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Storage is the best option for interprocess communications if you want to use preemptive processes. I have not done any real benchmarking but I have not noticed that anything seems slow. By setting up return values on a per-process basis you can avoid locking contention on storage.

For example, I have a wrapper for CALL FORM which allows a return value. I'm using this to provide services to non-GUI (possibly preemptive) processes that need to call form related features. One place I'm using this is to call a Javascript auto-layout library (http://ijzerenhein.github.io/autolayout.js) to generate a form layout from a constraint specification. This approach allows me to load the layout library in one (hidden) process and use it from any other process. The call looks like this:


//Internal method to execute the auto layout request in the web/javascript process.

  //$0 - Object with layout specification.
  //$1 - Collection of VFL specification strings.
  //$2 - View with.
  //$3 - View height.
  //$4 - (flag) Layout window, provided by this method only.


C_OBJECT($0;$oResult)
C_COLLECTION($1;$cSpec)
C_LONGINT($2;$width;$3;$height)
C_LONGINT($4;$window)

$cSpec:=$1
$width:=$2
$height:=$3

If (Count parameters<4)
        $window:=FD_Layout_window
        PM_CALL_IN_FORM_WINDOW (Current method name;->$oResult;0;->$cSpec;->$width;->$height;->$window)
Else
        WA EXECUTE JAVASCRIPT FUNCTION(fd_waLayout;"autoLayoutD5";$oResult;$cSpec;$width;$height)
End if

$0:=$oResult


John DeSoi, Ph.D.



> On Aug 25, 2018, at 4:44 PM, Tim Nevels via 4D_Tech <[hidden email]> wrote:
>
> But lets explore other possibilities. What could “Storage” be use for in other cases? Is it a good use of “Storage”?

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
In reply to this post by 4D Tech mailing list
On Aug 26, 2018, at 2:00 PM, Keisuke Miyako wrote:

> I totally agree with Christian, in fact,
> the point (slow, should be avoided if possible) was right there in the Summit keynote.
>
> on the point of "slow"
> of course it is a relative thing and perhaps not a big deal depending on the context,
> but you only have to run code with Storage and the speed difference is quite palpable.
> it is not a 4D specific phenomena, it is intrinsic to how blocking based on mutex works.
>
> on the point of "good coding"
> 4D forces you to forfeit exclusive access with the "Use/End use" code block,
> so unless you implicitly use storage in a stacked subroutine (super bad idea),
> it is hard to violate the FILO tule unintentionally.
>
> a more subtle gotcha is to make the mistake of
> using objects and collections as if they were values, not references.
> it's not limited to storage, but it can really bite you when you make the mistake with storage.


Good to know. John and my idea of using storage for sequence numbering is a bad idea. That’s why I posted here to get comments on if it was a good idea, or a bad idea. Now I know it’s a bad idea.

Storage is slow, so avoid using it if you can. But if you need it, it’s there to be used as a tool. But only in certain circumstances.

I’ll watch and see what others do with storage to learn when it is a good thing to use.

Tim
 
Tim Nevels
[hidden email] <mailto:[hidden email]>
Innovative Solutions
785-749-3444


**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
In reply to this post by 4D Tech mailing list
OK i have done this many times with a table. You have multiple records in the table, one for each id you need. I usually make that records key name ([table]field). If you have to keep track of released numbers, you can add a blob field. I would not be adding object fields as you can not access using SQL. I also have never figured out why you need a semaphore. I think in olden days (v1, 1.5 etc) you needed to slow this down to some extent.

You have be cognizant of whether you are inside a transaction or not. these queries and updates are blazing since you always have relatively few records in the table. If you have 500 keys you will have 500 records. Run method. Query for key. Load record if locked wait retry load. See if you have any returned numbers otherwise set next number, save record, unload record return it.

Pat when you say you need to use all numbers what happens in the following scenario. You create a record and it gets ID 100, for the next months you create 10,000 more records. You then decide to delete 1000, would you reuse that number. or only reuse if cancelling out of creation. If the later why not simply assign right before you save

Regards


Chuck
------------------------------------------------------------------------------------------------
 Chuck Miller Voice: (617) 739-0306
 Informed Solutions, Inc. Fax: (617) 232-1064      
 mailto:cjmiller<AT SIGN>informed-solutions.com
 Brookline, MA 02446 USA Registered 4D Developer                
       Providers of 4D and Sybase connectivity
          http://www.informed-solutions.com 
------------------------------------------------------------------------------------------------
This message and any attached documents contain information which may be confidential, subject to privilege or exempt from disclosure under applicable law.  These materials are intended only for the use of the intended recipient. If you are not the intended recipient of this transmission, you are hereby notified that any distribution, disclosure, printing, copying, storage, modification or the taking of any action in reliance upon this transmission is strictly prohibited.  Delivery of this message to any person other than the intended recipient shall not compromise or waive such confidentiality, privilege or exemption from disclosure as to this communication.

> On Aug 25, 2018, at 10:10 AM, Pat Bensky via 4D_Tech <[hidden email]> wrote:
>
> Using v17 ...
>
> I'm looking at ways to update our ancient record numbering system, and I
> think I should be able to do it efficiently with CALL WORKER, but having a
> bit of trouble getting my head around  it.
>
> The record numbers are like invoice numbers: each must be unique, and we
> don't want to have any gaps in the numbering sequence. So we need to keep
> track of any record numbers that are created and then abandoned (eg if a
> user creates a new record and then cancels it).
> The old system works well but it uses semaphores and SET/GET PROCESS
> VARIABLE and DELAY PROCESS. I isn't very efficient, especially when a large
> number of records are being imported and each one needs a new record number.
> Record numbers must be assigned synchronously - processing must not
> continue until the new record number has been assigned. Also, this
> frequently needs to be done in a context where there is no active form, so
> CALL FORM won't work.
>
> As I see it, we'll need to call a worker, which calls the worker that
> assigns the record number.
> The first worker has to wait until the second one has created the new
> number.
> Then the original method, that called the first worker, must wait until the
> first worker  gets the new number. It seems that this won't be much better
> than the old method.
>
> Am I completely barking up the wrong tree? Is this not an appropriate use
> for workers? Or have I misunderstood something?
>
> Pat

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
Chuck:

That is exactly what we do for our code. Has served us well for all our sites. We can have 200+ current users on the system, importing records from labs (thus needing lots of counters quickly), and never noticed a problem with the system. For those counters we do not wanted to have one go missing we have a BLOB to hold for use next time.

The reason we use our own counters rather than 4D’s auto assigned number is that we are often importing records from previous systems. We can easily roll back the counters when and where needed. We also like to put a good block of unused numbers between what was imported and what has been created by the system. That way looking at the unique numbers the Admins and us can know quickly where the record came from. This is specifically important when the system is first used.

Jody

Jody Bevan
Developer

Argus Productions Inc. <https://www.facebook.com/ArgusProductions/>
+1 587-487-6120



> On Aug 26, 2018, at 7:37 PM, Chuck Miller via 4D_Tech <[hidden email]> wrote:
>
> OK i have done this many times with a table. You have multiple records in the table, one for each id you need. I usually make that records key name ([table]field). If you have to keep track of released numbers, you can add a blob field. I would not be adding object fields as you can not access using SQL. I also have never figured out why you need a semaphore. I think in olden days (v1, 1.5 etc) you needed to slow this down to some extent.
>
> You have be cognizant of whether you are inside a transaction or not. these queries and updates are blazing since you always have relatively few records in the table. If you have 500 keys you will have 500 records. Run method. Query for key. Load record if locked wait retry load. See if you have any returned numbers otherwise set next number, save record, unload record return it.
>
> Pat when you say you need to use all numbers what happens in the following scenario. You create a record and it gets ID 100, for the next months you create 10,000 more records. You then decide to delete 1000, would you reuse that number. or only reuse if cancelling out of creation. If the later why not simply assign right before you save
>
> Regards
>
>
> Chuck
> ------------------------------------------------------------------------------------------------
>

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: New record numbering system and CALL WORKER

4D Tech mailing list
In reply to this post by 4D Tech mailing list
there is  the catch :)

working from inside a transaction requires (as far as I can figure out)
stepping outside the transaction process and generating the value(s)
then passing the value(s) back into the transaction.

Returning requires the same thing-in reverse.

Chip

On Sat, 25 Aug 2018 08:20:37 -0700, Kirk Brooks via 4D_Tech wrote:
>
> This approach let's 4D handle the record locking and request que and as
> long as you don't call it within a transaction just blissfully works.
---------------
Gas is for washing parts
Alcohol is for drinkin'
Nitromethane is for racing
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
123