4D server v13 or v14: anything which greatly improves scalability and speed?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

4D server v13 or v14: anything which greatly improves scalability and speed?

Paul Lovejoy-2
Hi,

I have been having a look at upgrading a customer from v12 to v13. I was wondering if anything specific has been done in 4D Server v13 (or v14) to improve scalability and performance on a large database with lots of users.

In particular, I was wondering if something has been done to get more performance out of multiple processors with lots of cores.

I believe there was some talk about changing the paradigm for the cooperative thread on 4D Server. But I haven't found any white papers or tech notes on this subject. I did find something in the KB regarding the fact that the application process doesn't need to run on the server unless necessary. That's all I could find.

A customer has a pretty big database with lots of users (180 simultaneous) and lots of processes which is currently running on a 3 year old xServe with 2 quad-core processors clocked at 2.26ghz, 12 gb of RAM and an SSD. Performance is already quite reasonable.

They are wondering if moving up to something with, say, 32 faster cores will give them a major improvement in performance. This would require them to move from Mac to Windows and buy some very expensive hardware, so they really want it to be worth it.

My experience running 4D Server v12 on a more powerful Windows 64 bit box wasn't really that much faster than the above-mentioned xServe configuration.

Thanks for sharing any info or experiences you might have.



Paul
**********************************************************************
4D v13 is available now - with more than 200 new features to make
your applications richer and faster
http://www.4d.com/products/new.html

4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D server v13 or v14: anything which greatly improves scalability and speed?

Jeffrey Kain
Paul,

I guess it depends on what you mean by “pretty big” and “lots of users” (and don’t forget number of processes, which is a more valuable metric than number of users). Moving to a 64-bit Windows server (and increasing the amount of memory allocated to the cache) is probably the biggest performance improvement you can give to a database application. But you need to know if you need that kind of horsepower or not, which means monitoring how your database uses the cache memory.

As a start, you should search this forum and any other forum for posts by Thomas Maul to learn how to do this. It’s a lot to type here, and most of it has been discussed already.

More cores doesn’t mean a faster database, either. It all depends on how your application is written. You need to look at how your application currently uses the cores it has available, keeping in mind that if you use a lot of triggers, a lot of execute on server, and a lot of server-side 4D code that it may not make ANY use of extra cores. It all depends.

For a simple answer, moving to a 64-bit Windows server with lots of RAM (32+ GB) and setting your memory settings to use most of that for the database cache (70%-80%) is the biggest boost you can give to a server. SSD is a close second, but may be the #1 improvement if your application is constantly accessing the disk. Maybe a close look at how you manage your indexes is all you need for great performance on a 32-bit system.

Also, 4D v13 introduces a process timeout feature that suspends inactive processes and removes them from memory when they go idle. It then instantly restores them when needed if the process wakes up. If you have a lot of idle processes, this could make a big difference. In my personal observation of large systems, I’ve never actually seen this happen, so there must be something about this new feature that I don’t understand. But in theory it should let a v13 server support more users/processes than earlier versions.

Jeff

On Dec 27, 2013, at 8:35 AM, Paul Lovejoy <[hidden email]> wrote:

> Hi,
>
> I have been having a look at upgrading a customer from v12 to v13. I was wondering if anything specific has been done in 4D Server v13 (or v14) to improve scalability and performance on a large database with lots of users.


**********************************************************************
4D v13 is available now - with more than 200 new features to make
your applications richer and faster
http://www.4d.com/products/new.html

4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D server v13 or v14: anything which greatly improves scalability and speed?

Thomas Maul-3
In reply to this post by Paul Lovejoy-2
Impossible to say if faster or not - without detailed analyzes...

About versions:
v13 has a better text index and code to move large blobs outside of the .4DD file, both usually helps for heavy used systems. Also better process handling, removing memory needs.
v14 has a better memory manager, optimized for new OS (don't support XP or Win2003 anymore)

Without analyzing, we can only guess.

Cores:
32 cores usually don't bring much.
You see a major benefit from 2 to 4, lower benefit from 4 to 8 and in some cases a benefit from 8 to 16. Sometimes 24 helps. But that all depends of your code, structure, usage.
And easy to analyze. Either watch Task Manager or use Microsoft tools to record CPU usage and check this.
If 17 clients runs in parallel a sequential search and all data is in cache - a 24 core system is faster than 16. As soon just one of them needs to access the hard disk, everything slows down.
I don't know your installation. A 180 user call center software has a different usage as 180 users in government...

RAM:
on Mac 4D uses up to 4 GB. On Windows as much as you have. 180 users? I guess 1000+ processes? I would use 32 or even 48 GB Ram. That allows for 20-36 GB Cache.
Enough? No idea. Depends of the size of your data, and usage of your data.
I guess you a partner? Check the "Info Component" on Partnercentral and let it run for some days, it gives you an idea about cache usage. Only a rough idea.
To get details, you need detailed analyzes.
But 32 GB should be a good start. Again, maybe 16 GB is enough, maybe you need more, an Analyze would say so.

SSD:
if the xServe is 3 years old, the SSD is 3 years old?
If your RAM (see above) is not big enough and it needs to flush often, the SSD is the bottleneck (but good that you don't use a hard disk or even SAN)
Take a look on modern Server/Enterprise disk and compare their IOPS and Latency with yours.
There are good tools for that, my favorite is MS SQLIO, but that is not available for Mac.
Example: http://www.intel.com/content/www/us/en/solid-state-drives/solid-state-drives-910-series.html
Again, an analyzes would show if RAM is big enough for all reads and if Disk is fast enough that flush don't slow down.

That analyzes could be done using Microsoft tools (like monitoring system data like hard disk waiting queue) as well as from 4D experts, contact your local technical account manager for that. Using Tools such as "Cache Recorder/Analyzer" allows to do that very detailed.


I don't know what you mean with very expensive hardware. The only need I see for very expensive hardware is when you want/need to virtualize.
To replace a "local" $500 SSD drive needs 10 times the money as virtualized SAN system - to get the same speed. Of course you get other benefits, but here we just talk about speed.

Without analyze I don't know if you need a 2 socket server system at all, or if a single XEON will do it. Big RAM and modern SSD (and Windows Server 2012 or 2012r2) should have a more visible impact, without needing a lot of money. In general I prefer a middle price for hardware and replacing it every 24-36 months - and not spending a fortune and then use the system 5-8 years (as we sometime see).

hope that helps
Thomas Maul








**********************************************************************
4D v13 is available now - with more than 200 new features to make
your applications richer and faster
http://www.4d.com/products/new.html

4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D server v13 or v14: anything which greatly improves scalability and speed?

Benedict, Tom
In reply to this post by Paul Lovejoy-2
Paul Lovejoy

>A customer has a pretty big database with lots of users (180 simultaneous)
>and lots of processes which is currently running on a 3 year old
>xServe with 2 quad-core processors clocked at 2.26ghz, 12 gb of
>RAM and an SSD. Performance is already quite reasonable.

>They are wondering if moving up to something with, say, 32 faster cores
>will give them a major improvement in performance. This would require
>them to move from Mac to Windows and buy some very expensive hardware,
>so they really want it to be worth it.

>My experience running 4D Server v12 on a more powerful Windows 64
>bit box wasn't really that much faster than the above-mentioned xServe configuration.

Jeff Kain's remarks are pretty thorough. I will add that over the years (nearly 20 now) the single best place to devote resources to is hardware. High end, enterprise class server, Windows 64bit OS, enterprise class SSDs and lots (24GB or more) RAM. My guess is that the xServe is running MacOS, which besides not being 64bit is generally a lot slower than WindowsOS. Can you run Windows on that xServe? If so, that might be a lower cost, relatively low risk interim solution.

Before you do any of this, use either the Info Report or Cache Logging/Analysis components (check with your 4D Account Manager to obtain them) to get a picture of how your cache is performing today. All IO goes through the cache, so the state of the cache is a very good measure of performance.

BTW, we have not noticed a correlation between many cores and better 4D Server performance.

On the app side, if you are doing a lot of reporting from the database you might consider offloading that to a different server. About 10 years ago we moved all reporting to a 'mirror' running on MS SQL Server which is 'synced' every 5 minutes from 4D. There we use enterprise level reporting tools to generate all reports. That took a very big load off of 4D.

Code optimization can certainly help, but that is generally a big and challenging exercise.

Make sure you have some performance benchmarks which you can track over time. We track 4 key application specific metrics which give us a performance profile which matches the user experience.

HTH,

Tom Benedict
Alere Wellbeing, Inc

**********************************************************************
4D v13 is available now - with more than 200 new features to make
your applications richer and faster
http://www.4d.com/products/new.html

4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D server v13 or v14: anything which greatly improves scalability and speed?

Scott Gelvin
In reply to this post by Paul Lovejoy-2
Paul,

Others alluded to other factors to consider, but I did want to offer our experience with 64-bit server applications.

Building your server application for 64-bits simply gives you:
* access to a larger data cache
* access to more memory for the server to manage connected users and processe

In one of our client’s systems, we were straining to support load under 32-bit servers, and while it performed fine under light or medium loads, under heavy loads would cause performance to fall precipitously. Switching to 64-bit (both the OS and the server application) helped support those heavy loads with a reasonable level of system performance. In this specific instance, it was the access to a larger data cache (>4GB) that was the main factor.

In general, you cannot expect a 64-bit Server Application to perform faster *for a given function* compared to its 32-bit counterpart. In fact, you may find some functions will benchmark slightly slower. However, what you *can* expect is to be able to support a greater load for the overall system. In other words, in a 32-bit environment, performance will drop off rapidly at a certain load, but a 64-bit environment can generally retain its performance levels for a greater load.

Since many details will affect overall application performance, I’m only speaking in generalities. You may find a particular detail of your application that bottlenecks the system regardless of whether you are operating 32-bit or 64-bit server application.


Scott Gelvin
Footprints, Inc.
4D specialists since 1987



On Dec 27, 2013, at 6:35 AM, Paul Lovejoy <[hidden email]> wrote:

> Hi,
>
> I have been having a look at upgrading a customer from v12 to v13. I was wondering if anything specific has been done in 4D Server v13 (or v14) to improve scalability and performance on a large database with lots of users.
>
> In particular, I was wondering if something has been done to get more performance out of multiple processors with lots of cores.
>
> I believe there was some talk about changing the paradigm for the cooperative thread on 4D Server. But I haven't found any white papers or tech notes on this subject. I did find something in the KB regarding the fact that the application process doesn't need to run on the server unless necessary. That's all I could find.
>
> A customer has a pretty big database with lots of users (180 simultaneous) and lots of processes which is currently running on a 3 year old xServe with 2 quad-core processors clocked at 2.26ghz, 12 gb of RAM and an SSD. Performance is already quite reasonable.
>
> They are wondering if moving up to something with, say, 32 faster cores will give them a major improvement in performance. This would require them to move from Mac to Windows and buy some very expensive hardware, so they really want it to be worth it.
>
> My experience running 4D Server v12 on a more powerful Windows 64 bit box wasn't really that much faster than the above-mentioned xServe configuration.
>
> Thanks for sharing any info or experiences you might have.
>
>
>
> Paul
> **********************************************************************
> 4D v13 is available now - with more than 200 new features to make
> your applications richer and faster
> http://www.4d.com/products/new.html
>
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************


**********************************************************************
4D v13 is available now - with more than 200 new features to make
your applications richer and faster
http://www.4d.com/products/new.html

4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

RE: 4D server v13 or v14: anything which greatly improves scalability and speed?

Joshua Fletcher
In reply to this post by Jeffrey Kain
Just a small correction: the DB4D process timeout was added in v12:

http://kb.4d.com/assetid=76793

4D v13 added "as needed" or "on demand" Application Server processes.

Kind regards,

Josh

--
Josh Fletcher
Technical Account Manager
4D, Inc


>



-----Original Message-----
> Also, 4D v13 introduces a process timeout feature that suspends inactive
> processes and removes them from memory when they go idle.



**********************************************************************
4D v13 is available now - with more than 200 new features to make
your applications richer and faster
http://www.4d.com/products/new.html

4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

RE: 4D server v13 or v14: anything which greatly improves scalability and speed?

Joshua Fletcher
In reply to this post by Paul Lovejoy-2
Hi Paul.

>



-----Original Message-----
> I believe there was some talk about changing the paradigm for the
> cooperative thread on 4D Server. But I haven't found any white papers or
> tech notes on this subject. I did find something in the KB regarding the
> fact that the application process doesn't need to run on the server unless
> necessary. That's all I could find.

That's the major change in 4D v13 in terms of scalability.  Planned/possible changes to 4D cooperative processes haven't been released.  It's still on the radar but definitely not in v14.0.

> A customer has a pretty big database with lots of users (180 simultaneous)
> and lots of processes which is currently running on a 3 year old xServe
> with 2 quad-core processors clocked at 2.26ghz, 12 gb of RAM and an SSD.
> Performance is already quite reasonable.

This is what I would call "medium+" for 4D, if that helps you get a context.  We have deployments that are double this size.

> They are wondering if moving up to something with, say, 32 faster cores
> will give them a major improvement in performance. This would require them
> to move from Mac to Windows and buy some very expensive hardware, so they
> really want it to be worth it.

If there is no user-perceived performance issues then, no, it won't be obviously worth it to the customer.  Note that "users" includes administrators.  Do they do backups or verifies?  Maintenance operations will be quantifiably faster; you can demonstrate to the customer how much *time* they will save, and time is money.

You have PLENTY of headroom for hardware upgrades as others have already pointed out.  But you must analyze the benefits as well (as others have also pointed out).

> My experience running 4D Server v12 on a more powerful Windows 64 bit box
> wasn't really that much faster than the above-mentioned xServe
> configuration.

My suggestion: invest in a development box for yourself that has 64-bit Windows, several cores, 16+GB RAM, but most importantly SATA III with some fast SSD's.  You can build something like this pretty cheap.  Use this yourself on your customer systems so that you can do direct performance comparison benchmarks.  You need to be able to tell the customer, "hey look this is x% faster and that is y% faster...it will save you z hours a month", IMHO.  I think most customers (especially bean counters) are looking for quantifiable benefits.

Kind regards,

Josh

--
Josh Fletcher
Technical Account Manager
4D, Inc


**********************************************************************
4D v13 is available now - with more than 200 new features to make
your applications richer and faster
http://www.4d.com/products/new.html

4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************