Why does a converted v13 db limit text vars to 32k?

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Why does a converted v13 db limit text vars to 32k?

KirkBrooks
Working with an old project that's currently at v13. Text vars are
truncating to 32k. I initially thought this would be because of the Unicode
setting but even after enabling and restarting with Unicode they still
truncate.

What's the dang deal here?

Thanks

--
Kirk Brooks
San Francisco, CA
=======================
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Why does a converted v13 db limit text vars to 32k?

Jeffrey Kain
Are you using an old codec in BLOB TO TEXT or Convert Text with these variables - anything with "Mac" in their name? Like "Mac text without length" or "Mac C string"?

--
Jeffrey Kain
[hidden email]

> On Jan 5, 2017, at 5:08 PM, Kirk Brooks <[hidden email]> wrote:
>
> Working with an old project that's currently at v13. Text vars are
> truncating to 32k. I initially thought this would be because of the Unicode
> setting but even after enabling and restarting with Unicode they still
> truncate.
>
> What's the dang deal here?

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Why does a converted v13 db limit text vars to 32k?

KirkBrooks
Hi Jeff,
I first noticed it using the json component of NTK. But to test I tried
this:

C_TEXT($text)
$text:="x"*34000
$x:=Length($text)

$x is 32000. I tried some tricks like using BLOBs but none of that seems to
matter.


On Thu, Jan 5, 2017 at 2:14 PM, Jeffrey Kain <[hidden email]> wrote:

> Are you using an old codec in BLOB TO TEXT or Convert Text with these
> variables - anything with "Mac" in their name? Like "Mac text without
> length" or "Mac C string"?
>
> --
> Jeffrey Kain
> [hidden email]
>
> > On Jan 5, 2017, at 5:08 PM, Kirk Brooks <[hidden email]> wrote:
> >
> > Working with an old project that's currently at v13. Text vars are
> > truncating to 32k. I initially thought this would be because of the
> Unicode
> > setting but even after enabling and restarting with Unicode they still
> > truncate.
> >
> > What's the dang deal here?
>
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************




--
Kirk Brooks
San Francisco, CA
=======================
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Why does a converted v13 db limit text vars to 32k?

KirkBrooks
In reply to this post by KirkBrooks
On Thu, Jan 5, 2017 at 2:08 PM, Kirk Brooks <[hidden email]> wrote:

> What's the dang deal here?
>

​Well, I'm not really sure what the deal is but I resolved it by running
MSC, correcting a few other, unrelated problems, and then running 'repair'.
Verified with:

​C_TEXT($text)
$text:="x"*50000
$x:=Length($text)

​and $x = 50k.

--
Kirk Brooks
San Francisco, CA
=======================
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Why does a converted v13 db limit text vars to 32k?

Keisuke Miyako
I remember there was a bug where if you upgrade from v11 straight to v13
unicode mode was deactivated yet the compatibility dialog told otherwise.

> 2017/01/07 2:26、Kirk Brooks <[hidden email]> のメール:
> ​Well, I'm not really sure what the deal is but I resolved it by running
> MSC, correcting a few other, unrelated problems, and then running 'repair'.


宮古 啓介
セールス・エンジニア

株式会社フォーディー・ジャパン
〒150-0043
東京都渋谷区道玄坂1-10-2 渋谷THビル6F
Tel: 03-6427-8441
Fax: 03-6427-8449

[hidden email]
www.4D.com/JP

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Why does a converted v13 db limit text vars to 32k?

KirkBrooks
Hi Miyako,
That may have been the issue. I wasn't part of the project at that time but
it's possible. It would be consistent with the behaviour I observed.

The good news is MSC/repair seems to fix it.

On Fri, Jan 6, 2017 at 4:29 PM, Keisuke Miyako <[hidden email]>
wrote:

> I remember there was a bug where if you upgrade from v11 straight to v13
> unicode mode was deactivated yet the compatibility dialog told otherwise.
>

--
Kirk Brooks
San Francisco, CA
=======================
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Why does a converted v13 db limit text vars to 32k?

Tim Nevels-2
In reply to this post by KirkBrooks
On Jan 6, 2017, at 7:18 PM, Keisuke Miyako wrote:

> I remember there was a bug where if you upgrade from v11 straight to v13
> unicode mode was deactivated yet the compatibility dialog told otherwise.

Miyako, you are the king of 4D secrets and details. You are a human 4D wikipedia!

OK, there may be others that know more about 4D. The initials JPR comes to mind. But they don’t regularly contribute their 4D knowledge on the iNUG. So they don’t really count… now do they?  Just teasing JPR. :)

I’m sure JPR has an email filter that flags messages that includes “JPR” so he’s reading this.

On a serious note, assuming JPR is reading this message, are you going to be doing the 4D World Tour this year? I enjoyed your presentation last time.

I’m wondering if you’ve got some more cool v16 multiprocess, preemptive CALL WORKER demos and ideas to share with the 4D community. It’s a new area and we need ideas on how to best utilize it.

Tim

********************************************
Tim Nevels
Innovative Solutions
785-749-3444
[hidden email]
********************************************

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Why does a converted v13 db limit text vars to 32k?

David Adams-4
Will the initials DPA serve? I don't have insights into the 4D internals.
I've always approached 4D as a black box and only believe what I can
measure. That's served me very well down the years.

Below are some thoughts on CALL WORKER and CALL FORM. For what it's worth,
I consider these the most important new commands in the 4D language since,
well, I can't even remember. They are potentially revolutionary for 4D
developers and are very, very welcome indeed. They're what have convinced
me to look seriously at 4D again and to move to V15/V16. I'll also again
encourage everyone to check out Thomas Mauls *excellent* presentation on
the subject. Look for it here and scroll ahead to around minute 36+:

https://4dmethod.com/

> I’m wondering if you’ve got some more cool v16 multiprocess, preemptive
CALL WORKER demos and
> ideas to share with the 4D community. It’s a new area and we need ideas
on how to best utilize it.

Use it as a non-persistent, first-in-first-out message/task queue. That's
what they've given us. The cores business is a side-effect, or at least I'd
encourage you to think of them that way. EVERYONE can benefit from CALL
WORKER/CALL PROCESS, but not everyone has a problem that immediately
benefits from multiple cores. If you do, so much the better. It would be a
real shame if people didn't look at these commands because they think that
they're for problems they don't have. While they do solve some problems
that most people don't have, they solve more problems that all of us have.
Get in there early and experiment! (I'm working on getting myself numbers
organized so that I can actually try the 4D implementation. It looks sweet
to me.)

Here's what to use it for - just a couple of ideas off the top of my head:

Anything computationally expensive that's naturally suitable for
parallelization.
Not all problems are easy to parallelize. Dividing the work and marshaling
the results can end up being more trouble than it's worth. Then again, when
it's good...it's great.

IP messaging
Finally, we've got a native message queue. At last. These commands should
have been introduced with New process and then we could have skipped ~20
years of people messing around with 4D's Rohypnol commands.
Message-oriented architectures are, well, the wave of the present. It
promotes loose coupling and better reusability. In my coding religion, it's
one of the pillars of the creed. Think of everything as messages.

Moving towards a native MVC-like GUI control system
It's becoming more plausible to do something MVC-like in 4D now with
objects, form variables, and CALL FORM. Important. This is a huge subject,
but something that everyone should try and get their heads around. Right
away.

Logging!
If you write  a custom log file, it's a total pain to have to manage writes
from multiple processes. Don't. Everything calls into CALL PROCESS and only
that process has write access to the log. This is how servers do logs, and
now so can we. (In the past, you could do the same with NTK's IPC tools.
This is how to write a Web server log, as an example.) Going out to the
file system is slow and lock/unlocks are unexpectedly expensive. Having
lived through a switchover of this description some years back, I'll say
that the speed gains can be *massive.*

Note: As far as I can see, the 4D message queues die with the server or at
least their contents are not preserved. That kind of makes sense as they
operate in a context that would be hard to restore. So, you might need to
address that if you need a 100% perfect log. Skipping many thoughts on
solutions here...that's a big thread of its own. But a good one...there are
some pretty understandable and interesting design choices to make there.

Unique access
Logging is a specific example of managing a shared resource, but generalize
it in your mind to other tasks where you are fighting for access to a
shared or unique resource. Some of them will do quite well in a worker
process. *Reducing contention is a big win.* It can also make your code a
whole lot simpler and more reliable.

Cores?
Instead of focusing on cores, focus on messaging. I'll argue that it's very
likely that most developers do not have a wide range of problems that will
meaningfully benefit from multiple cores. Many should have a few problems
where it's a *massive* benefit, but probably not day to day. 4D Server/4D
themselves absolutely, absolutely do. Us? Not so much. I'm happy to be
wrong about this, but I've been working with message and task queues since
I wrote one at 4D, Inc. in the 1980's and they're part of my toolkit for
any project. That's not the same thing as CALL FORM/CALL WORKER, but you
end up with a lot of the same questions on a design level. In all of that
time, I haven't ended up finding a huge range of tasks that customers had
that could really be split up in real time for massive benefit. Keep in
mind that if more cores are what you need, a multi-machine system gives you
that in a big fat hurry. The CALL WORKER/CALL FORM system is for a single
machine, but what if you have stack of machines sitting around? Those are
extra cores. So, anyone that's ever written a batch processor or
nightly/cron-type system, think about those tasks too.

P.S. The cool kids want to push tasks out to the GPU. Standard cores are so
2014!
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Why does a converted v13 db limit text vars to 32k?

David Adams-4
> Anything computationally expensive that's naturally suitable for
> parallelization. Not all problems are easy to parallelize. Dividing the
work
> and marshaling the results can end up being more trouble than it's worth.
Then
> again, when it's good...it's great.

Well, a few ideas...keeping in mind that this is all actually a bit harder
to implement. Why? Because if you split one task into several parts but
still need a single result, you need to figure out correctly how to divide
the task, how to send the instructions to process each part, how you return
the final results, and how you stitch them together and use them. The
payoff can be huge, in the right situation, but the startup costs are
higher than some other uses of CALL PROCESS/CALL FORM. Anyway, a few
examples:

* Long searches/calculations:
These can be divided up by some kind of condition (ID ranges, date ranges,
record numbers, etc.) and then the final results can be pushed to a
controller for assembly and final disposition.

* Fuzzy matching:
The problem with the better fuzzy match algorithms is that they tend to be
too slow to run in real time. Perhaps that can be changed.

* Burst mode:
Say you've got a surge of SMS messages (or whatever) to send out. I think
that you can use 4DIC from within a worker. (I couldn't see this after a
quick look at the docs - but I think I remember Thomas saying that in his
presentation?) Fine. Start a bunch of named workers and push the tasks out
to them. This could be a big win.

* Analysis:
Lots and lots of interesting information may be buried in your systems'
data...but it's not information until you extract and refine it. Who knows
what sorts of interesting results you'll get from scanning for interesting
trends? If you can use idle cores, why not?

I'm sure other people have nice, specific examples...

On Sat, Jan 7, 2017 at 1:27 PM, David Adams <[hidden email]> wrote:

> Will the initials DPA serve? I don't have insights into the 4D internals.
> I've always approached 4D as a black box and only believe what I can
> measure. That's served me very well down the years.
>
> Below are some thoughts on CALL WORKER and CALL FORM. For what it's worth,
> I consider these the most important new commands in the 4D language since,
> well, I can't even remember. They are potentially revolutionary for 4D
> developers and are very, very welcome indeed. They're what have convinced
> me to look seriously at 4D again and to move to V15/V16. I'll also again
> encourage everyone to check out Thomas Mauls *excellent* presentation on
> the subject. Look for it here and scroll ahead to around minute 36+:
>
> https://4dmethod.com/
>
> > I’m wondering if you’ve got some more cool v16 multiprocess, preemptive
> CALL WORKER demos and
> > ideas to share with the 4D community. It’s a new area and we need ideas
> on how to best utilize it.
>
> Use it as a non-persistent, first-in-first-out message/task queue. That's
> what they've given us. The cores business is a side-effect, or at least I'd
> encourage you to think of them that way. EVERYONE can benefit from CALL
> WORKER/CALL PROCESS, but not everyone has a problem that immediately
> benefits from multiple cores. If you do, so much the better. It would be a
> real shame if people didn't look at these commands because they think that
> they're for problems they don't have. While they do solve some problems
> that most people don't have, they solve more problems that all of us have.
> Get in there early and experiment! (I'm working on getting myself numbers
> organized so that I can actually try the 4D implementation. It looks sweet
> to me.)
>
> Here's what to use it for - just a couple of ideas off the top of my head:
>
> Anything computationally expensive that's naturally suitable for
> parallelization.
> Not all problems are easy to parallelize. Dividing the work and marshaling
> the results can end up being more trouble than it's worth. Then again, when
> it's good...it's great.
>
> IP messaging
> Finally, we've got a native message queue. At last. These commands should
> have been introduced with New process and then we could have skipped ~20
> years of people messing around with 4D's Rohypnol commands.
> Message-oriented architectures are, well, the wave of the present. It
> promotes loose coupling and better reusability. In my coding religion, it's
> one of the pillars of the creed. Think of everything as messages.
>
> Moving towards a native MVC-like GUI control system
> It's becoming more plausible to do something MVC-like in 4D now with
> objects, form variables, and CALL FORM. Important. This is a huge subject,
> but something that everyone should try and get their heads around. Right
> away.
>
> Logging!
> If you write  a custom log file, it's a total pain to have to manage
> writes from multiple processes. Don't. Everything calls into CALL PROCESS
> and only that process has write access to the log. This is how servers do
> logs, and now so can we. (In the past, you could do the same with NTK's IPC
> tools. This is how to write a Web server log, as an example.) Going out to
> the file system is slow and lock/unlocks are unexpectedly expensive. Having
> lived through a switchover of this description some years back, I'll say
> that the speed gains can be *massive.*
>
> Note: As far as I can see, the 4D message queues die with the server or at
> least their contents are not preserved. That kind of makes sense as they
> operate in a context that would be hard to restore. So, you might need to
> address that if you need a 100% perfect log. Skipping many thoughts on
> solutions here...that's a big thread of its own. But a good one...there are
> some pretty understandable and interesting design choices to make there.
>
> Unique access
> Logging is a specific example of managing a shared resource, but
> generalize it in your mind to other tasks where you are fighting for access
> to a shared or unique resource. Some of them will do quite well in a worker
> process. *Reducing contention is a big win.* It can also make your code a
> whole lot simpler and more reliable.
>
> Cores?
> Instead of focusing on cores, focus on messaging. I'll argue that it's
> very likely that most developers do not have a wide range of problems that
> will meaningfully benefit from multiple cores. Many should have a few
> problems where it's a *massive* benefit, but probably not day to day. 4D
> Server/4D themselves absolutely, absolutely do. Us? Not so much. I'm happy
> to be wrong about this, but I've been working with message and task queues
> since I wrote one at 4D, Inc. in the 1980's and they're part of my toolkit
> for any project. That's not the same thing as CALL FORM/CALL WORKER, but
> you end up with a lot of the same questions on a design level. In all of
> that time, I haven't ended up finding a huge range of tasks that customers
> had that could really be split up in real time for massive benefit. Keep in
> mind that if more cores are what you need, a multi-machine system gives you
> that in a big fat hurry. The CALL WORKER/CALL FORM system is for a single
> machine, but what if you have stack of machines sitting around? Those are
> extra cores. So, anyone that's ever written a batch processor or
> nightly/cron-type system, think about those tasks too.
>
> P.S. The cool kids want to push tasks out to the GPU. Standard cores are
> so 2014!
>
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Why does a converted v13 db limit text vars to 32k?

Tim Nevels-2
In reply to this post by KirkBrooks
On Jan 6, 2017, at 8:28 PM, David Adams wrote:

> Will the initials DPA serve? I don't have insights into the 4D internals.
> I've always approached 4D as a black box and only believe what I can
> measure. That's served me very well down the years.

Now that you mention it, I think DPA should definitely be included as another human 4D wikipedia. People forget that when 4D included the web server you were the go-to-guy for how to do web stuff with 4D. I went to many seminars you gave on that topic. You wrote a book or two on the subject as well.

And you do provide a unique perspective on using 4D to tackle some common data processing tasks.

Didn’t mean to slight you. And you definitely do contribute more than JPR on the iNUG now.

Come to think of it, it would be great to have David Adams once again being a presenter of new 4D information. Any chance you could be dragged from Australia to wander America once again doing seminars for something like the 4D World Tour?

> Logging!
> If you write  a custom log file, it's a total pain to have to manage writes
> from multiple processes. Don't. Everything calls into CALL PROCESS and only
> that process has write access to the log. This is how servers do logs, and
> now so can we. (In the past, you could do the same with NTK's IPC tools.
> This is how to write a Web server log, as an example.) Going out to the
> file system is slow and lock/unlocks are unexpectedly expensive. Having
> lived through a switchover of this description some years back, I'll say
> that the speed gains can be *massive.*

Yes this is a great use of CALL WORKER. Eliminates the need for semaphores and IP variables. Just CALL WORKER with some parameters and let it do it’s thing. Automatic message queuing. It scales effortlessly to hundreds of processes and hundreds of calls per second.  Super efficient. 4D handles all the hard stuff.

Another side note. What ever happened to Dan Beckett? Has he left the 4D community?

Wasn’t he part of the Foresight Technology after Rich Gay sold it? Are they still around and doing 4D work?

Tim

********************************************
Tim Nevels
Innovative Solutions
785-749-3444
[hidden email]
********************************************

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Why does a converted v13 db limit text vars to 32k?

David Adams-4
> On Sat, Jan 7, 2017 at 2:21 PM, Tim Nevels <[hidden email]> wrote:

> Didn’t mean to slight you. And you definitely do contribute more than JPR
on
> the iNUG now.

D'oh!  I came across like a whiney prima donna. Sorry for that, not my
intention. Hey, JPR - let me join in in asking for some ideas from you.
You've got an interesting perspective, a long history, have been presenting
about this for some time and usually have really cool demos. So, chime in!
I hope that Thomas does this time, too and that he does more videos.

> Another side note. What ever happened to Dan Beckett? Has he left the 4D
> community?

Dan is still around, using 4D, and working hard, no fear.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
JPR
Reply | Threaded
Open this post in threaded view
|

Re: Why does a converted v13 db limit text vars to 32k?

JPR
In reply to this post by KirkBrooks
[JPR]

> I’m sure JPR has an email filter that flags messages that includes “JPR” so he’s reading this.
Gosh! Tim, you've caught red-handed!

> On a serious note, assuming JPR is reading this message, are you going to be doing the 4D World Tour this year? I enjoyed your presentation last time.
Yes I will be in 10 cities, including 6 cities in USA. It will be 2 days in each city, 1st for presentation of many practical examples, and the second one will be a training day with my friend Add in USA.

> I’m wondering if you’ve got some more cool v16 multiprocess, preemptive CALL WORKER demos and ideas to share with the 4D community. It’s a new area and we need ideas on how to best utilize it.
I'm finishing my program for the 2nd day, and I can tell you that yes, it will be some cool stuff and explanations regarding v16 multiprocess, preemptive CALL WORKER and CALL FORM, and obviously the results of my tests. New ideas are not enough, explaining how it works internally is not enough either, what I would like to share with the 4D community is how to use these new ideas in practical programming. And because I've tried many ways, and because most of these ways could be hard or no-thru, I would like to share with all of you which one is the easiest, safest, most opened to the future, according to my own experience.

So I look forward to meeting you, and it will be with a great pleasure!

My very best,

JPR

> On 07 Jan 2017, at 03:28, [hidden email] wrote:
>
> Subject: Re: Why does a converted v13 db limit text vars to 32k?
> Message-ID: <[hidden email]>
> Content-Type: text/plain; charset=windows-1252
>
> On Jan 6, 2017, at 7:18 PM, Keisuke Miyako wrote:
>
>> I remember there was a bug where if you upgrade from v11 straight to v13
>> unicode mode was deactivated yet the compatibility dialog told otherwise.
>
> Miyako, you are the king of 4D secrets and details. You are a human 4D wikipedia!
>
> OK, there may be others that know more about 4D. The initials JPR comes to mind. But they don’t regularly contribute their 4D knowledge on the iNUG. So they don’t really count… now do they?  Just teasing JPR. :)
>
> I’m sure JPR has an email filter that flags messages that includes “JPR” so he’s reading this.
>
> On a serious note, assuming JPR is reading this message, are you going to be doing the 4D World Tour this year? I enjoyed your presentation last time.
>
> I’m wondering if you’ve got some more cool v16 multiprocess, preemptive CALL WORKER demos and ideas to share with the 4D community. It’s a new area and we need ideas on how to best utilize it.
>
> Tim

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Why does a converted v13 db limit text vars to 32k?

Jeffrey Kain
In reply to this post by David Adams-4
On Jan 6, 2017, at 9:27 PM, David Adams <[hidden email]> wrote:

> Logging!
> If you write  a custom log file, it's a total pain to have to manage writes
> from multiple processes. Don't. Everything calls into CALL PROCESS

Did you mean CALL WORKER here? You mention CALL PROCESS a few times in these last few posts but I don't see how that command applies. Or maybe I'm completely misunderstanding things.


> EVERYONE can benefit from CALL
> WORKER/CALL PROCESS, but not everyone has a problem that immediately
> benefits from multiple cores.

CALL FORM here?

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Why does a converted v13 db limit text vars to 32k?

David Adams-4
Jeff,

Thanks for the proofreading. Right, I was never meaning CALL PROCESS. Just
CALL WORKER and CALL FORM (which is more like CALL FORM INSTANCE WINDOW, or
something.)

You've worked with very large-scale 4D systems for as long as anyone, so
any comments and thoughts that you might have on these new commands would
be of interest...
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Why does a converted v13 db limit text vars to 32k?

Tim Nevels-2
In reply to this post by KirkBrooks
On Jan 7, 2017, at 2:00 PM, JPR wrote:

>> I’m sure JPR has an email filter that flags messages that includes “JPR” so he’s reading this.
> Gosh! Tim, you've caught red-handed!

You’re not the only one with email filters. I’ve also got one that looks for “JPR” and “David Adams” in iNUG messages. :)


>> On a serious note, assuming JPR is reading this message, are you going to be doing the 4D World Tour this year? I enjoyed your presentation last time.
> Yes I will be in 10 cities, including 6 cities in USA. It will be 2 days in each city, 1st for presentation of many practical examples, and the second one will be a training day with my friend Add in USA.

That is good news. I’ll be attending the Denver Colorado meeting. Look forward to seeing you there and your presentation on new v16 features.

Tim

********************************************
Tim Nevels
Innovative Solutions
785-749-3444
[hidden email]
********************************************

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
JPR
Reply | Threaded
Open this post in threaded view
|

Re: Why does a converted v13 db limit text vars to 32k?

JPR
In reply to this post by KirkBrooks
[JPR]

Hi David, Tim, and others,

It seems that January is the time for flooding the street with flowers ;-)

Let's try to make a long story short: When I came back to 4D, I met (for the 1st time) Keisuke in Vegas. I was very happy to meet him, for he is an absolute Wizard in programming, and I was so happy to read his postings on the NUG, always enlightened by the simplicity and elegance of his code. He came to me and told me : "You know, you're a legend". If it is probably very positive to be a legend when you're dead (in fact, I don't know, I've to ask: Hey Walt ? How do you feel being a dead legend?) it is extremely difficult to be an alive legend, for you can only disappoint your fans...

In fact, I think my role on the NUG is not to jump into every questions with my solutions. I see the NUG as a place where everyone can expose their problems and solutions. And there is no such a thing as a stupid solution. An answer that look stupid may be a prefect solution to a problem under different conditions. I've often observed that many developers do not dare to expose their solution for being afraid to be ashamed by a far better one, like a tenderfoot wouldn't draw in front of a sharpshooter. I consider that my job is to help you to understand, not to solve your problems. I will not give you fishes, but I'm happy to explain where and how is the best fishing. Most of the time, as soon as you've got a perfect understanding of the underlying mechanism, you get easily to the proper solution.

So when you want explanations about the internal mechanism of a command in 4D, just put JPR in the mail, and I will do my best to explain how it works.

Let's take a simple example: A recent post is "Best way to find something in an array of objects". Let's analyse this topic. Do I know the Best Way? Do I have the answer to this question? Yes, of course! Do you want to know it? Trivial! The answer is "It depends..."

It depends on many things, and there are many questions to answer to before being close to the solution.

- What do you mean by "Best way"? The easiest one? The fastest one to program? The most efficient one? The most efficient is probably to mimic the way LR has programmed the Object Indexes. But this is NOT the easiest one, I can guarantee you...

- What means 'fast'? I don't know what means Fast or Slow. The only remark I got during my life was "Too slow". (Funny, but I never got a "Too fast" comment...) If it works in an acceptable way, then the best way will probably be the easiest one.

- Where is the original Data, the one used to fill the array? If it is in a Table, it's probably easier to Query the table than to "Find in object array". Also, you can get the content of each path with the DISTINCT ATTRIBUTE PATH and DISTINCT ATTRIBUTE VALUES commands, which may help to reduce the Query field.

- The only way to optimise the Find in array is to sort the array. What sorting algorithm will you use? Is there a prefect one? Quicksort? Or Merge sort, Heapsort, Bubble sort, Shellsort, Comb sort, Bucket sort, Radix sort? Do you think that if one was clearly better than others, it would still be so many algorithms? Probably not. Just because, here also, it depends...

- You can see an Object as a Key-Value Store (which it is, in fact). But where do you want to find? In any place? In a specific property? Including the property name? Case-sensitive or not? Using Fuzzy logic or not? Is each Object in your Object Array consistent with the others or not?

- Can you sort your array? Sometimes, the index of an element in an array codes for some information (record number, grid line, etc.) and you can't sort it. Or you have to create an array with the index of the elements and sort 2 arrays in parallel. But in this case, what about memory? Or sorting time? I had to deal with 20-million element array, did you try one to sort such a big one?

- Is it not better to use another object as an associative array? For instance, if you want to get instantly a Product Name from a Product Code, the traditional 4D way is to create the array with a loop...

$k:=Find in array($arCodes;$myCode)
If ($k>0)
     $arNames{$k}:=$myName
Else
     APPEND TO ARRAY($arCodes;$myCode)
     APPEND TO ARRAY($arNames;$myName)
End if

 ...and to find with:

$k:=Find in array($arCodes;$myCode)
If ($k>0)
     $myName:=$arNames{$k}
Else
     $myName:=""
End if

 What about an Object used as an associative array:

C_OBJECT($myArray)
You create the array with a loop...

OB SET($myArray;myCode;$myName)

...and you find with:

$myName:=OB Get($myArray;$myCode}

This is much faster. Why? Because 4D uses internally a Hash table for the property names, so finding a property in a big object is not a problem of speed.

- Another good question to ask is regarding the unicity of the key. The solution above is possible only because the Product Code is unique.

- When you have to deal with huge unsorted arrays, you may think to use a Hash function in order to divide your monster array into an array of smaller arrays (i.e. a 2D array in 4D). For example, I put my 20-million element array into a 10,000 * 2,000 2-D array. The mechanism is simple: To add an element...

$colNb:=MyHashFunction ($myElement;10000)
APPEND TO ARRAY(MyArray{$colNb};$myElement)

...and to find an Element:

$found:=Find in array(myArray{MyHashFunction ($myElement;10000)};$value2Find)
(A more optimised version of this code is available during the trainings, and I will be happy to sent it to anyone who may need it)

The Hash Function can be anything from a simple Modulo to a CRC16, as long as it returns a number between 0 and 10,000.

- And many other questions like this...

As you can see, there is no Single Perfect Magic Answer. For me, an answer is good as long as it works and makes a Happy User.

So, Guys, I will be always happy to answer to any questions you may have, provided that you begin by searching a solution by yourself and inside the NUG, and I will answer everytimes you will put JPR somewhere in your post.

And I will continue to bring deep explanations during trainings, as you will see during the Tour 2017. A big part of my job is (because I'm lucky enough to work very closely with L.R., L.E., and the rest of the Team) is :

- to break some new programming ground and to find the easiest path for you to adapt, as safely and painlessly as possible, your actual code and the new tools.
- to try to give you a good orientation to move in a way that will be the most adapted to the future versions.
- to help you to solve the problems you may have by explaining the way it works internally in 4D.

And this is enough to fill my time schedule to the upper limit...

My very best,

JPR

> On 07 Jan 2017, at 21:00, [hidden email] wrote:
>
>  Hey, JPR - let me join in in asking for some ideas from you.
> You've got an interesting perspective, a long history, have been presenting
> about this for some time and usually have really cool demos. So, chime in!
> I hope that Thomas does this time, too and that he does more videos.

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: Why does a converted v13 db limit text vars to 32k?

David Adams-4
In reply to this post by Tim Nevels-2
Thanks for the post. You should get a T-Shirt printed for your tour in
black with white writing

   It depends.

;-)

And, of course, it really does depend. Operations at scale are different
than operations on small collections. They're different problems. Yet
another reason it's helpful to have a testing framework so that you can
find how operation perform in different environments. Something that works
great for 1000 items might be terrible for 100,000 - and something that's a
huge help at 100,000 might add pointless overhead at 1000. There are some
very, very large scale 4D databases out there but, mostly, they're not so
big. (At least, that's my experience.) So, if I have to guess, I guess
people are talking about general-case, small-medium situations, unless they
say otherwise.

> What do you mean by "Best way"? The easiest one? The fastest one to
program? The most efficient
> one? The most efficient is probably to mimic the way LR has programmed
the Object Indexes. But this is
> NOT the easiest one, I can guarantee you...

Yes, it would be understand how LR programmed object indexes as it provides
a good way to develop theories about good strategies and to then *test*
them. The only way to know what is better in some situation is to test it
out. (Obviously, algorithms can be compared abstractly but we're
usually talking implantation on this list.) How the hash tables are
implemented for memory-only objects is also important to know about.

Can you say how the hash tables and object indexes are implemented in 4D?
It ought to be explained in some form of documentation, but doesn't seem to
be. Sadly, many of 4D's features are not adequately documented. I haven't
found much of anything on subforms widgets or the dual-process business on
4D Server. So far, I've found 4D's objects to be remarkably fast to access
- far faster than I expected and it would help to know more about the
underlying data structure(s) and algorithm(s).

As an example of when and why it's helpful to understand underlying data
structures and implementations, I can say it in a word: XML. 4D didn't have
a meaningful tree structure before XML and a lot of 4D programmers haven't
had to deal with trees directly. When I see slow DOM XML code, it's
typically not hard to cut the processing time in half. How? By replacing
4D's DOM find commands with a comprehensive tree walk that visits each node
once and only once. You traverse the entire tree, but with no waste. Using
the wrapper commands in the language, you're presumably asking 4D to walk
the whole tree over and over until you find what you want. You can
construct unlikely examples where the difference makes a *massive* speed
difference. But, in general practice, I figure a 2 to 1 improvement. Just a
rule of thumb, the real difference absolutely depends. This is also one
reason that I use XML Tools as it provides a super easy way to get good
performance without hassling with a tree walk.

Anyway, I don't know how the XML library 4D embeds works internally, but
understanding that it's a tree and has to do a tree walk, it's not hard to
see where the inefficiency comes from. Unless there's some kind of magic
index (and I wouldn't expect or even want one), doing 'find in DOM' means
lots of redundant traversals.

I'm hoping to hear a thumbnail explanation of that sort for 4D's hash
table/object implementations. That would be great!

And while it's really, really great that 4D has being doing these road
shows while also continuing to put on Summits, it's not possible for people
to attend.  It's better if you can attend but, if not, the information
ought still be available in a reliable form, I'd say. It's better for 4D as
more people understand how to succeed with the product.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************