4D Server Hosting - The Future ?

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

4D Server Hosting - The Future ?

4D Tech mailing list
Hi !

I just finished a 1.5 hour phonecall with a support services manager at the technical services company who supply one of my customers with all their hardware/software/maintenance services.

He basically brought me up to date on “how things work” today which is essentially that everything to do with platforms is now virtualised to allow them to ‘tune’ resources to demand in realtime and provide seamless, no downtime backup. Basically, my 4d Server is now a “cloud service” without me even being aware of it, it’s just that the hardware involved happens to be located on the preises.

In particular we discussed backup configurations for 4D server and this was interesting because, while I requested independent drives for logfile (“journal”) and datafile purposes, he essentially told me to just stick everything on the same drive because it was virtual anyway and had multiple redundancy protection via raid, 15-minute snapshotting etc. He offered to “create” a C: and a D: drive to make me feel better, but pointed out that they’re not much more independent than 2 folders would have been.

CONCLUSON
I now realise that the “WAN” / “LAN” distinction is disappearing. He said the only reason the “cloud” solution wasn’t hosted off-site was that they had measured the bandwidth that the customer used and calculated that the cost would be astronomical if it was on AWS or something like that, but in all other respects it was a cloud solution.

I was wondering, how do other major 4D server deployers optimise their deployment strategies to take advantage of this ? It seems a great thing that we are being “floated out to the cloud” without actually having to do extra significant work, but what about things like the backup strategy ? I don’t really like the idea that the log file has the same redundancy system as the main datafile because the whole idea is that the corruption doesn’t get replicated (which is what a RAID system does) and it’s independent at the logical level.

We seem one step away from being able to supply server solutions where “our” customer doesn’t have to host the database server on premises. Is anybody doing this at an advanced level ? (e.g. connecting with 4D client native to a 4D server that’s 3rd-party hosted).

Regards

Peter

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
Hi Peter,

Indeed, it is a brave new world, isn’t it? And like you said, it all happened transparently.

At my last employer everything was virtualized. The SAN ‘guarenteed’ data uptime and redundancy. “Disks” (HDD or SSD) were ‘auto-healing’ and hardware failover was automatic. It is really quite amazing. And it worked very well. Once the VMs were optimized it was as fast or faster than dedicated hardware. Moving 4D Server to the “external cloud" is another question. For optimum (or maybe even just tolerable) performance, your app will need to be designed with a very lightweight front end. Or abandon 4D client and use a web front end.

>I don’t really like the idea that the log file has the same redundancy system as the main datafile because the whole idea is that the corruption doesn’t get replicated (which is what a RAID system does) and it’s independent at the logical level.
>
A SAN is really a lot more that just a RAID. As for a backup strategy, at this point we’re really talking about a disaster recovery plan, since the SAN is ‘guarenteed'. You might consider mirroring to another ‘cloud service'. Since "spinning up" a VM is so easy now, all you need is another 4D server license to set up a 4D mirror server. Have it integrate the log file every minute or two and you have a near real time backup, ready for (manual) failover. Since only the log files sent, and they are tiny, the bandwidth costs would be extremely small.

Tom benedict

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Peter,
I am in the process of moving a database from our own hardware to and AWS
instance. It's true that the most expensive part of setting it up, at this
point, is getting the appropriate amount of band width and throughput speed.

The other thing about VM vs metal is the whole pre-emptive process
benefit basically goes away. Thomas Maul has shown this at the Summit.
Having n+ virtual cores doesn't do anything to actually increase processing
speed because the VM is running on whatever is allocated to it.
Theoretically you could have a VM with 4 cores running an instance with 32
cores. So preemptive threading is looking to be mainly a benefit for
companies that run their own hardware and for desktop apps.

On Thu, Oct 10, 2019 at 5:25 AM Peter Jakobsson via 4D_Tech <
[hidden email]> wrote:

> Hi !
>
> I just finished a 1.5 hour phonecall with a support services manager at
> the technical services company who supply one of my customers with all
> their hardware/software/maintenance services.
>
> He basically brought me up to date on “how things work” today which is
> essentially that everything to do with platforms is now virtualised to
> allow them to ‘tune’ resources to demand in realtime and provide seamless,
> no downtime backup. Basically, my 4d Server is now a “cloud service”
> without me even being aware of it, it’s just that the hardware involved
> happens to be located on the preises.
>
> In particular we discussed backup configurations for 4D server and this
> was interesting because, while I requested independent drives for logfile
> (“journal”) and datafile purposes, he essentially told me to just stick
> everything on the same drive because it was virtual anyway and had multiple
> redundancy protection via raid, 15-minute snapshotting etc. He offered to
> “create” a C: and a D: drive to make me feel better, but pointed out that
> they’re not much more independent than 2 folders would have been.
>
> CONCLUSON
> I now realise that the “WAN” / “LAN” distinction is disappearing. He said
> the only reason the “cloud” solution wasn’t hosted off-site was that they
> had measured the bandwidth that the customer used and calculated that the
> cost would be astronomical if it was on AWS or something like that, but in
> all other respects it was a cloud solution.
>
> I was wondering, how do other major 4D server deployers optimise their
> deployment strategies to take advantage of this ? It seems a great thing
> that we are being “floated out to the cloud” without actually having to do
> extra significant work, but what about things like the backup strategy ? I
> don’t really like the idea that the log file has the same redundancy system
> as the main datafile because the whole idea is that the corruption doesn’t
> get replicated (which is what a RAID system does) and it’s independent at
> the logical level.
>
> We seem one step away from being able to supply server solutions where
> “our” customer doesn’t have to host the database server on premises. Is
> anybody doing this at an advanced level ? (e.g. connecting with 4D client
> native to a 4D server that’s 3rd-party hosted).
>
> Regards
>
> Peter
>
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************



--
Kirk Brooks
San Francisco, CA
=======================

What can be said, can be said clearly,
and what you can’t say, you should shut up about

*Wittgenstein and the Computer *
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
Thanks for your post Kirk.

The guy I spoke to seemed to have it the other way around - the VM’s had 4-12 cores and the “metal” about 32.

He also was of the categorical opinion that the only way to really keep applications “isolated” from each other (i.e. not bring everything else down when they crashed) was to give each mission critical application or service its own VM.

When I put to him “what about the natural O/S level multi-threading” he felt there were too many vulnerabilities and mentioned especially the “crypto viruses” and the Intel multi-threading bug. His approach was basically - if your VM needs ore resources then we can simply allocated more. He wasn’t really bothered by the idea of applications that were multi-threaded internally because it’s all the one big bucket and if something inside the bucket needs more power then just make the bucket bigger.

Peter


> On 10 Oct 2019, at 16:09, Kirk Brooks via 4D_Tech <[hidden email]>
> wrote:
>
> The other thing about VM vs metal is the whole pre-emptive process
> benefit basically goes away. Thomas Maul has shown this at the Summit.
> Having n+ virtual cores doesn't do anything to actually increase processing
> speed because the VM is running on whatever is allocated to it.
> Theoretically you could have a VM with 4 cores running an instance with 32
> cores

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
that all is great until the hardware running the 4,382,619 VMs crashes
:/

Chip
On Thu, 10 Oct 2019 16:23:18 +0100, Peter Jakobsson via 4D_Tech wrote:
>
> He also was of the categorical opinion that the only way to really
> keep applications “isolated” from each other (i.e. not bring
> everything else down when they crashed) was to give each mission
> critical application or service its own VM.
We have done so much, with so little, for so long;
We are now qualified to anything with nothing <sigh>
  - unknown
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Peter,
I had a discussion with Tim Nevels on another channel about this last week
too. He's a big fan and maybe will weigh in here. I will not miss having to
make a trip to the colo to change an SSD that died. (Hint: mirrored SSDs so
you don't have to do that in the middle of the night.) It is kind of
interesting that we are moving back to the topology the industry started
with - what's the difference between a 'main frame computer' and 'cloud
computing'?

On Thu, Oct 10, 2019 at 8:23 AM Peter Jakobsson via 4D_Tech <
[hidden email]> wrote:

> Thanks for your post Kirk.
>
> The guy I spoke to seemed to have it the other way around - the VM’s had
> 4-12 cores and the “metal” about 32.
>
> He also was of the categorical opinion that the only way to really keep
> applications “isolated” from each other (i.e. not bring everything else
> down when they crashed) was to give each mission critical application or
> service its own VM.
>
> When I put to him “what about the natural O/S level multi-threading” he
> felt there were too many vulnerabilities and mentioned especially the
> “crypto viruses” and the Intel multi-threading bug. His approach was
> basically - if your VM needs ore resources then we can simply allocated
> more. He wasn’t really bothered by the idea of applications that were
> multi-threaded internally because it’s all the one big bucket and if
> something inside the bucket needs more power then just make the bucket
> bigger.
>
> Peter
>
>
> > On 10 Oct 2019, at 16:09, Kirk Brooks via 4D_Tech <[hidden email]>
> > wrote:
> >
> > The other thing about VM vs metal is the whole pre-emptive process
> > benefit basically goes away. Thomas Maul has shown this at the Summit.
> > Having n+ virtual cores doesn't do anything to actually increase
> processing
> > speed because the VM is running on whatever is allocated to it.
> > Theoretically you could have a VM with 4 cores running an instance with
> 32
> > cores
>
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************



--
Kirk Brooks
San Francisco, CA
=======================

What can be said, can be said clearly,
and what you can’t say, you should shut up about

*Wittgenstein and the Computer *
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
In reply to this post by 4D Tech mailing list


> On 10 Oct 2019, at 16:58, Chip Scheide <[hidden email]> wrote:
>
> that all is great until the hardware running the 4,382,619 VMs crashes

According to “the guy”, that’s all taken care of. Even if the “metal” melts down, the VM’s just “seamlessly” migrate themselves onto other metal without the users even knowing !

Don’t ask me quite how that works but he kept saying it wasn't a problem.

-P

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
In reply to this post by 4D Tech mailing list
it is a repeating cycle.

change the paradigm to make more money.
Once everyone that needs one has a mainframe switch to desktops, now
everyone has a desktop
switch to the 'cloud'.
eventually it will swing back to personal/business owned devices as
data security becomes a (bigger) issue

BTW - when using a 'cloud' service, depending on your personal/business
outlook you might want to think about who has access to your data, and
what are they doing with it. This applies not just to colo and 4D but
to everything in 'the cloud'.

What is Apple doing with your playlists and other purchasing habits
(and what does that say about you), who else is looking at the files in
your dropbox account, and nobody should have to wonder about what
google, amazon, Facebook twitter etc are doing with all the information
you maybe giving them.


Just because your are paranoid does NOT mean that they are not *really*
out there...  :)

On Thu, 10 Oct 2019 08:59:48 -0700, Kirk Brooks via 4D_Tech wrote:
> It is kind of
> interesting that we are moving back to the topology the industry started
> with - what's the difference between a 'main frame computer' and 'cloud
> computing'?
We have done so much, with so little, for so long;
We are now qualified to anything with nothing <sigh>
  - unknown
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
In reply to this post by 4D Tech mailing list
The key to getting good 4D performance out of VMs is dedicated resources. Once we got the VM ‘locked’ so it wouldn’t be ’smart’ about reallocating ‘idle’ resources (disk space, cache memory etc) to other VMs performance was very good. I don’t know much about VMs, but I’m surprised that you can’t dedicate cores to a specific VM like you can other resources.

Tom Benedict

> On Oct 10, 2019, at 08:23, Peter Jakobsson via 4D_Tech <[hidden email]> wrote:
>
> Thanks for your post Kirk.
>
> The guy I spoke to seemed to have it the other way around - the VM’s had 4-12 cores and the “metal” about 32.
>
> He also was of the categorical opinion that the only way to really keep applications “isolated” from each other (i.e. not bring everything else down when they crashed) was to give each mission critical application or service its own VM.
>
> When I put to him “what about the natural O/S level multi-threading” he felt there were too many vulnerabilities and mentioned especially the “crypto viruses” and the Intel multi-threading bug. His approach was basically - if your VM needs ore resources then we can simply allocated more. He wasn’t really bothered by the idea of applications that were multi-threaded internally because it’s all the one big bucket and if something inside the bucket needs more power then just make the bucket bigger.
>
> Peter
>
>
>> On 10 Oct 2019, at 16:09, Kirk Brooks via 4D_Tech <[hidden email]>
>> wrote:
>>
>> The other thing about VM vs metal is the whole pre-emptive process
>> benefit basically goes away. Thomas Maul has shown this at the Summit.
>> Having n+ virtual cores doesn't do anything to actually increase processing
>> speed because the VM is running on whatever is allocated to it.
>> Theoretically you could have a VM with 4 cores running an instance with 32
>> cores

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Indeed, that is the magic of VM and SAN. They can change the truck’s engine while its traveling down the road at 60mph. I was a skeptic until I saw it in action. Hot swapping of disks, memory, even ‘blades’ with CPUs on them happens transparently.

Keep in mind, though, that this is not done to save money. It might save money, but what it really does is make systems more reliable and faster to deploy.

Tom Benedict

> On Oct 10, 2019, at 09:12, Peter Jakobsson via 4D_Tech <[hidden email]> wrote:
>
>> On 10 Oct 2019, at 16:58, Chip Scheide <[hidden email]> wrote:
>>
>> that all is great until the hardware running the 4,382,619 VMs crashes
>
> According to “the guy”, that’s all taken care of. Even if the “metal” melts down, the VM’s just “seamlessly” migrate themselves onto other metal without the users even knowing !

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
In reply to this post by 4D Tech mailing list
>BTW - when using a 'cloud' service, depending on your personal/business
>
> outlook you might want to think about who has access to your data, and
> what are they doing with it. This applies not just to colo and 4D but
> to everything in 'the cloud’.

That’s why you encrypt your data. https://blog.4d.com/introduction-to-data-encryption-in-4d/ <https://blog.4d.com/introduction-to-data-encryption-in-4d/>

Tom Benedict
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
In reply to this post by 4D Tech mailing list
I don't see how "cloud" hosting can be the future of 4D when LAN execution is great but WAN execution is horribly slow for the same application. Yes, you can rewrite in various ways (harder to write, understand, and maintain) but it is not an easy task.

John DeSoi, Ph.D.


> On Oct 10, 2019, at 6:24 AM, Peter Jakobsson via 4D_Tech <[hidden email]> wrote:
>
> CONCLUSON
> I now realise that the “WAN” / “LAN” distinction is disappearing. He said the only reason the “cloud” solution wasn’t hosted off-site was that they had measured the bandwidth that the customer used and calculated that the cost would be astronomical if it was on AWS or something like that, but in all other respects it was a cloud solution.

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
We host both the Server and the Client in the cloud. End users connect into the "client cloud" using Remote Desktop.

Robb


--
Robb Detlefs | Director, West Coast Operations & Strategic Initiatives
[hidden email]<mailto:[hidden email]> | 510.652.8950 x233 or 646.733.2239 x233
GallerySystems | www.gallerysystems.com<http://www.gallerysystems.com/>
3200 College Ave. Suite 6; Berkeley, CA 94705




On Oct 10, 2019, at 10:02 AM, John DeSoi via 4D_Tech <[hidden email]<mailto:[hidden email]>> wrote:

EXTERNAL: Do not click links or open attachments if you do not recognize the sender.

I don't see how "cloud" hosting can be the future of 4D when LAN execution is great but WAN execution is horribly slow for the same application. Yes, you can rewrite in various ways (harder to write, understand, and maintain) but it is not an easy task.

John DeSoi, Ph.D.


On Oct 10, 2019, at 6:24 AM, Peter Jakobsson via 4D_Tech <[hidden email]<mailto:[hidden email]>> wrote:

CONCLUSON
I now realise that the “WAN” / “LAN” distinction is disappearing. He said the only reason the “cloud” solution wasn’t hosted off-site was that they had measured the bandwidth that the customer used and calculated that the cost would be astronomical if it was on AWS or something like that, but in all other respects it was a cloud solution.

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  https://urldefense.com/v3/__http://lists.4d.com/archives.html__;!09EJrJkydGw!Xytu_B2U0mQcyAmuZQJ4z6tA7e_ofPTL3OqMZxKUuOz4k9HZHFqY7PGYVS6H8-KlhuLYcF6BBQ$
Options: https://urldefense.com/v3/__https://lists.4d.com/mailman/options/4d_tech__;!09EJrJkydGw!Xytu_B2U0mQcyAmuZQJ4z6tA7e_ofPTL3OqMZxKUuOz4k9HZHFqY7PGYVS6H8-KlhuLpyeegqA$
Unsub:  mailto:[hidden email]
**********************************************************************

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
In reply to this post by 4D Tech mailing list
On Oct 10, 2019, at 10:25 AM, Kirk Brooks wrote:

> The other thing about VM vs metal is the whole pre-emptive process
> benefit basically goes away. Thomas Maul has shown this at the Summit.
> Having n+ virtual cores doesn't do anything to actually increase processing
> speed because the VM is running on whatever is allocated to it.
> Theoretically you could have a VM with 4 cores running an instance with 32
> cores. So preemptive threading is looking to be mainly a benefit for
> companies that run their own hardware and for desktop apps.

I was not aware of this. So what you are saying is that if you are deploying 4D Server in a VM there is not need to try and use preemptive mode. You get no benefit from doing that. It’s a waste of time? Sounds like a giant bummer. Hope you are wrong. I’ll put in the secret distress call code “JPR" in this message and see if JPR catches this and knows anything about this and can comment.

If not, I’m going to make it a point to ask one of the Laurents about this at 4D Summit so I’m clear on when you get preemptive benefits. Do you get any when running 4D Server or 4D Client in a VM?

As you know I love running 4D Server in a VM environment that is properly configured on great hardware. Performance is great for 4D Server and for 4D Client instances. And when you need more disk space you can get it in a few minutes by just altering some VM parameters. Same for adding more RAM, just takes a few minutes and a reboot.

And since the disks are all allocated from a SAN that is RAID “whatever” you don’t have to worry about a disk failure and losing data. RAID will protect you from that. And putting the .4dd on a separate physical disk than the .journal file can’t be done and is not a consideration for providing additional protection from hardware failure.

The last part of fault tolerance and recover from failures is to implement “Volume Shadow Copy” and get a snapshot of the machine. I have another client that does this every hour. So if the VM goes completely bad you can restore it back to what it was up to 1 hour ago. That brings drive “C:\” back to where it was. And if your data file is on “D:\” it is stored on the SAN “hard drives” so it is suppose be accurate and good up to the last operation. This is what I am told.

That last thing I tell myself when running 4D in a VM environment is that server hardware failure and disaster recover is not longer my problem. It’s the IT department’s problem. I’m just providing a 4D Server application that runs on their “machine”. I didn’t set up or configure the “machine”. I didn’t implement the backup plan or strategy. Everything depends on IT doing it right and providing an environment that can recover from hardware failures.

In the past I was very involved with server hardware setup and configuration and backups, and so this was another of my “worries” or considerations. Did we buy enough RAM. Did we get enough hard drive space and are we using the drives the right way. Is the backup software running and working. And if there was a problem or failure I was the first one called to help fix it. That’s all gone when 4D Server is running in a VM. It’s all someone else’s problem. All I have to say is “when you get the VM restored and online again, let me know. I’ll check 4D Server and see if we lost any data.” I like it that way.

Tim

*****************************************
Tim Nevels
Innovative Solutions
785-749-3444
[hidden email]
*****************************************

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

RE: 4D Server Hosting - The Future ?

4D Tech mailing list
> The other thing about VM vs metal is the whole pre-emptive process
> benefit basically goes away.

... can go away...

If you have a physical machine with 32 cores and your VM run 8 shared with 4 other VMs running 8 cores, then you may really get 8 cores. If you run 32 VMs all wanting 8 virtual, then of course you are not getting it. I think the point was that virtual cores do not always mean physical cores. It will depend on the number of physical resources allocated to the VM.

Neil








Privacy Disclaimer: This message contains confidential information and is intended only for the named addressee. If you are not the named addressee you should not disseminate, distribute or copy this email. Please delete this email from your system and notify the sender immediately by replying to this email.  If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

The Alternative Investments division of UMB Fund Services provides a full range of services to hedge funds, funds of funds and private equity funds.  Any tax advice in this communication is not intended to be used, and cannot be used, by a client or any other person or entity for the purpose of (a) avoiding penalties that may be imposed on any taxpayer or (b) promoting, marketing, or recommending to another party any matter addressed herein.
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Hey Tim,

On Thu, Oct 10, 2019 at 10:43 AM Tim Nevels via 4D_Tech <
[hidden email]> wrote:

> So what you are saying is that if you are deploying 4D Server in a VM
> there is not need to try and use preemptive mode. You get no benefit from
> doing that. It’s a waste of time?

That's what I got. Thomas showed a demo app he wrote deployed on AWS or the
like and using a lot of cores (32?). But it did not improve the overall
processing time. The reason being, as I understand it, the whole benefit of
preemptive mode is to allow 4D to assign a process to a core. But in the
case of a VM the actual, physical cores are managed by the VM. It really
couldn't be otherwise and support all the cool stuff that makes the VM
attractive.

I don't know that makes preemptive a waste of time but it does mean you
need to think about where your db will be running to decide if it's worth
the effort to implement it.

On the other hand this means that in a VM environment CALL WORKER is pretty
much as useful and a lot less hassle.

It would be interesting to know if preemptive is something that can be
called for a local process on a client machine. Now _that_ would be pretty
handy. Ask JPR when you chat with him.

--
Kirk Brooks
San Francisco, CA
=======================

What can be said, can be said clearly,
and what you can’t say, you should shut up about

*Wittgenstein and the Computer *
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
In reply to this post by 4D Tech mailing list
Hi Kirk,

I am very interested on your progress with AWS. Can you tell me a little bit more about the kind of 4D DB you are running on it?
Are you running client/server or using SQL? Do you notice speed differences between ORDA / Classic 4D?


I’m asking because I am currently using AWS myself. The response is quite ok, but still way slower than LAN client server.
I am thinking of putting more 4D databases on AWS.


Regards,

Rudy Mortier
Two Way Communications bvba



> On 10 Oct 2019, at 17:09, Kirk Brooks via 4D_Tech <[hidden email]> wrote:
>
> Peter,
> I am in the process of moving a database from our own hardware to and AWS
> instance. It's true that the most expensive part of setting it up, at this
> point, is getting the appropriate amount of band width and throughput speed.
>
> The other thing about VM vs metal is the whole pre-emptive process
> benefit basically goes away. Thomas Maul has shown this at the Summit.
> Having n+ virtual cores doesn't do anything to actually increase processing
> speed because the VM is running on whatever is allocated to it.
> Theoretically you could have a VM with 4 cores running an instance with 32
> cores. So preemptive threading is looking to be mainly a benefit for
> companies that run their own hardware and for desktop apps.
>
> On Thu, Oct 10, 2019 at 5:25 AM Peter Jakobsson via 4D_Tech <
> [hidden email]> wrote:
>
>> Hi !
>>
>> I just finished a 1.5 hour phonecall with a support services manager at
>> the technical services company who supply one of my customers with all
>> their hardware/software/maintenance services.
>>
>> He basically brought me up to date on “how things work” today which is
>> essentially that everything to do with platforms is now virtualised to
>> allow them to ‘tune’ resources to demand in realtime and provide seamless,
>> no downtime backup. Basically, my 4d Server is now a “cloud service”
>> without me even being aware of it, it’s just that the hardware involved
>> happens to be located on the preises.
>>
>> In particular we discussed backup configurations for 4D server and this
>> was interesting because, while I requested independent drives for logfile
>> (“journal”) and datafile purposes, he essentially told me to just stick
>> everything on the same drive because it was virtual anyway and had multiple
>> redundancy protection via raid, 15-minute snapshotting etc. He offered to
>> “create” a C: and a D: drive to make me feel better, but pointed out that
>> they’re not much more independent than 2 folders would have been.
>>
>> CONCLUSON
>> I now realise that the “WAN” / “LAN” distinction is disappearing. He said
>> the only reason the “cloud” solution wasn’t hosted off-site was that they
>> had measured the bandwidth that the customer used and calculated that the
>> cost would be astronomical if it was on AWS or something like that, but in
>> all other respects it was a cloud solution.
>>
>> I was wondering, how do other major 4D server deployers optimise their
>> deployment strategies to take advantage of this ? It seems a great thing
>> that we are being “floated out to the cloud” without actually having to do
>> extra significant work, but what about things like the backup strategy ? I
>> don’t really like the idea that the log file has the same redundancy system
>> as the main datafile because the whole idea is that the corruption doesn’t
>> get replicated (which is what a RAID system does) and it’s independent at
>> the logical level.
>>
>> We seem one step away from being able to supply server solutions where
>> “our” customer doesn’t have to host the database server on premises. Is
>> anybody doing this at an advanced level ? (e.g. connecting with 4D client
>> native to a 4D server that’s 3rd-party hosted).
>>
>> Regards
>>
>> Peter
>>
>> **********************************************************************
>> 4D Internet Users Group (4D iNUG)
>> Archive:  http://lists.4d.com/archives.html
>> Options: https://lists.4d.com/mailman/options/4d_tech
>> Unsub:  mailto:[hidden email]
>> **********************************************************************
>
>
>
> --
> Kirk Brooks
> San Francisco, CA
> =======================
>
> What can be said, can be said clearly,
> and what you can’t say, you should shut up about
>
> *Wittgenstein and the Computer *
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
Hey Rudy,
I made a post yesterday on the thread about preemptive processes and how it
works on VM. Ping me if you can't find it. You may find it useful because I
posted some actual data from running a demo you can download. I ran it on
my laptop and an AWS instance. The general specs of the AWS instance are
there too. It shows the sort of differences I've been seeing. It seems the
issue with AWS is partly the sheer horsepower you sign up for in terms of
cores and such but equally important is the amount of bandwidth you commit
to. Impacts the performance and cost quickly.

I'm hoping someone with more expertise in this might join the conversation.
(Balinder? you out there?) It's almost good for directly connecting. I
think if you are running 4D to power a web server it's quite good. Also, I
think if you deploy there optimizations you can make in code to
accommodate the network will become apparent. I quickly saw that places
where I move data from the server to the client and manipulate it on the
client are very speedy. ORDA is probably going to help with that. I'm also
curious if you can set up a situation that cost effective where you run an
instance for the server and then some others to support clients using RDP
or the like.

It is certainly appealing.

On Sat, Oct 12, 2019 at 2:28 PM Two Way Communications via 4D_Tech <
[hidden email]> wrote:

> Hi Kirk,
>
> I am very interested on your progress with AWS. Can you tell me a little
> bit more about the kind of 4D DB you are running on it?
> Are you running client/server or using SQL? Do you notice speed
> differences between ORDA / Classic 4D?
>
>
> I’m asking because I am currently using AWS myself. The response is quite
> ok, but still way slower than LAN client server.
> I am thinking of putting more 4D databases on AWS.
>
>
> Regards,
>
> Rudy Mortier
> Two Way Communications bvba
>
>
>
> > On 10 Oct 2019, at 17:09, Kirk Brooks via 4D_Tech <[hidden email]>
> wrote:
> >
> > Peter,
> > I am in the process of moving a database from our own hardware to and AWS
> > instance. It's true that the most expensive part of setting it up, at
> this
> > point, is getting the appropriate amount of band width and throughput
> speed.
> >
> > The other thing about VM vs metal is the whole pre-emptive process
> > benefit basically goes away. Thomas Maul has shown this at the Summit.
> > Having n+ virtual cores doesn't do anything to actually increase
> processing
> > speed because the VM is running on whatever is allocated to it.
> > Theoretically you could have a VM with 4 cores running an instance with
> 32
> > cores. So preemptive threading is looking to be mainly a benefit for
> > companies that run their own hardware and for desktop apps.
> >
> > On Thu, Oct 10, 2019 at 5:25 AM Peter Jakobsson via 4D_Tech <
> > [hidden email]> wrote:
> >
> >> Hi !
> >>
> >> I just finished a 1.5 hour phonecall with a support services manager at
> >> the technical services company who supply one of my customers with all
> >> their hardware/software/maintenance services.
> >>
> >> He basically brought me up to date on “how things work” today which is
> >> essentially that everything to do with platforms is now virtualised to
> >> allow them to ‘tune’ resources to demand in realtime and provide
> seamless,
> >> no downtime backup. Basically, my 4d Server is now a “cloud service”
> >> without me even being aware of it, it’s just that the hardware involved
> >> happens to be located on the preises.
> >>
> >> In particular we discussed backup configurations for 4D server and this
> >> was interesting because, while I requested independent drives for
> logfile
> >> (“journal”) and datafile purposes, he essentially told me to just stick
> >> everything on the same drive because it was virtual anyway and had
> multiple
> >> redundancy protection via raid, 15-minute snapshotting etc. He offered
> to
> >> “create” a C: and a D: drive to make me feel better, but pointed out
> that
> >> they’re not much more independent than 2 folders would have been.
> >>
> >> CONCLUSON
> >> I now realise that the “WAN” / “LAN” distinction is disappearing. He
> said
> >> the only reason the “cloud” solution wasn’t hosted off-site was that
> they
> >> had measured the bandwidth that the customer used and calculated that
> the
> >> cost would be astronomical if it was on AWS or something like that, but
> in
> >> all other respects it was a cloud solution.
> >>
> >> I was wondering, how do other major 4D server deployers optimise their
> >> deployment strategies to take advantage of this ? It seems a great thing
> >> that we are being “floated out to the cloud” without actually having to
> do
> >> extra significant work, but what about things like the backup strategy
> ? I
> >> don’t really like the idea that the log file has the same redundancy
> system
> >> as the main datafile because the whole idea is that the corruption
> doesn’t
> >> get replicated (which is what a RAID system does) and it’s independent
> at
> >> the logical level.
> >>
> >> We seem one step away from being able to supply server solutions where
> >> “our” customer doesn’t have to host the database server on premises. Is
> >> anybody doing this at an advanced level ? (e.g. connecting with 4D
> client
> >> native to a 4D server that’s 3rd-party hosted).
> >>
> >> Regards
> >>
> >> Peter
> >>
> >> **********************************************************************
> >> 4D Internet Users Group (4D iNUG)
> >> Archive:  http://lists.4d.com/archives.html
> >> Options: https://lists.4d.com/mailman/options/4d_tech
> >> Unsub:  mailto:[hidden email]
> >> **********************************************************************
> >
> >
> >
> > --
> > Kirk Brooks
> > San Francisco, CA
> > =======================
> >
> > What can be said, can be said clearly,
> > and what you can’t say, you should shut up about
> >
> > *Wittgenstein and the Computer *
> > **********************************************************************
> > 4D Internet Users Group (4D iNUG)
> > Archive:  http://lists.4d.com/archives.html
> > Options: https://lists.4d.com/mailman/options/4d_tech
> > Unsub:  mailto:[hidden email]
> > **********************************************************************
>
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************



--
Kirk Brooks
San Francisco, CA
=======================

What can be said, can be said clearly,
and what you can’t say, you should shut up about

*Wittgenstein and the Computer *
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: 4D Server Hosting - The Future ?

4D Tech mailing list
Hi Kirk,

Very interesting.

Here are my experiences on running a 4D v17R5 server, client on Mac and windows:

If you remember my presentation on the 2018 summit (with the SVG charts for temperature mapping), that is the one running on it.

First setup: t2.micro on AWS, smallest possible and free ;-)
Pros:
Very reliable
good for prototyping
handling: quite ok, which has surprised me since it has only 1GB RAM ( ! )
Selection to array / array to selection: tested with + 100.000 records, not to bad (seconds rather than minutes)
Lists are displayed in listboxes, using fields. Display is rather immediate.
Detail forms load within 2 to 10 seconds, depending on how many queries in it

Cons:
bandwidth not always sufficient, which drops handling to a crawl, and also causes big delays when accessing the instance with RDP

2nd setup: m5.large, 8GB RAM, 2 cores. 160$/month
Pros:
Very reliable
much better handling due to guaranteed bandwidth
a *lot* faster than the t2.micro, very workable

I also put 2 RDP users on it, because the server license includes 2 CALS. Works even better, even on a Mac!

I studied what it would take to have more RDP users, but there, I get lost a little bit.
From what I understood, I would need an RDP gateway, so another AWS instance, and then buy additional CALS for RDP.
That sounds quite expensive…

The other thing I tried recently is S3, because one of my applications manages a lot (+ 1,000,000) of documents.

There is a 4Dsummit example that manages all the S3 stuff: very impressive!
Storing and retrieving documents has never been so easy and fast. Cost is affordable (by EU standards ;-) )

I would love to here other AWS experiences.

Regards,

Rudy Mortier
Two Way Communications bvba



> On 13 Oct 2019, at 00:51, Kirk Brooks via 4D_Tech <[hidden email]> wrote:
>
> Hey Rudy,
> I made a post yesterday on the thread about preemptive processes and how it
> works on VM. Ping me if you can't find it. You may find it useful because I
> posted some actual data from running a demo you can download. I ran it on
> my laptop and an AWS instance. The general specs of the AWS instance are
> there too. It shows the sort of differences I've been seeing. It seems the
> issue with AWS is partly the sheer horsepower you sign up for in terms of
> cores and such but equally important is the amount of bandwidth you commit
> to. Impacts the performance and cost quickly.
>
> I'm hoping someone with more expertise in this might join the conversation.
> (Balinder? you out there?) It's almost good for directly connecting. I
> think if you are running 4D to power a web server it's quite good. Also, I
> think if you deploy there optimizations you can make in code to
> accommodate the network will become apparent. I quickly saw that places
> where I move data from the server to the client and manipulate it on the
> client are very speedy. ORDA is probably going to help with that. I'm also
> curious if you can set up a situation that cost effective where you run an
> instance for the server and then some others to support clients using RDP
> or the like.
>
> It is certainly appealing.
>
> On Sat, Oct 12, 2019 at 2:28 PM Two Way Communications via 4D_Tech <
> [hidden email]> wrote:
>
>> Hi Kirk,
>>
>> I am very interested on your progress with AWS. Can you tell me a little
>> bit more about the kind of 4D DB you are running on it?
>> Are you running client/server or using SQL? Do you notice speed
>> differences between ORDA / Classic 4D?
>>
>>
>> I’m asking because I am currently using AWS myself. The response is quite
>> ok, but still way slower than LAN client server.
>> I am thinking of putting more 4D databases on AWS.
>>
>>
>> Regards,
>>
>> Rudy Mortier
>> Two Way Communications bvba
>>
>>
>>
>>> On 10 Oct 2019, at 17:09, Kirk Brooks via 4D_Tech <[hidden email]>
>> wrote:
>>>
>>> Peter,
>>> I am in the process of moving a database from our own hardware to and AWS
>>> instance. It's true that the most expensive part of setting it up, at
>> this
>>> point, is getting the appropriate amount of band width and throughput
>> speed.
>>>
>>> The other thing about VM vs metal is the whole pre-emptive process
>>> benefit basically goes away. Thomas Maul has shown this at the Summit.
>>> Having n+ virtual cores doesn't do anything to actually increase
>> processing
>>> speed because the VM is running on whatever is allocated to it.
>>> Theoretically you could have a VM with 4 cores running an instance with
>> 32
>>> cores. So preemptive threading is looking to be mainly a benefit for
>>> companies that run their own hardware and for desktop apps.
>>>
>>> On Thu, Oct 10, 2019 at 5:25 AM Peter Jakobsson via 4D_Tech <
>>> [hidden email]> wrote:
>>>
>>>> Hi !
>>>>
>>>> I just finished a 1.5 hour phonecall with a support services manager at
>>>> the technical services company who supply one of my customers with all
>>>> their hardware/software/maintenance services.
>>>>
>>>> He basically brought me up to date on “how things work” today which is
>>>> essentially that everything to do with platforms is now virtualised to
>>>> allow them to ‘tune’ resources to demand in realtime and provide
>> seamless,
>>>> no downtime backup. Basically, my 4d Server is now a “cloud service”
>>>> without me even being aware of it, it’s just that the hardware involved
>>>> happens to be located on the preises.
>>>>
>>>> In particular we discussed backup configurations for 4D server and this
>>>> was interesting because, while I requested independent drives for
>> logfile
>>>> (“journal”) and datafile purposes, he essentially told me to just stick
>>>> everything on the same drive because it was virtual anyway and had
>> multiple
>>>> redundancy protection via raid, 15-minute snapshotting etc. He offered
>> to
>>>> “create” a C: and a D: drive to make me feel better, but pointed out
>> that
>>>> they’re not much more independent than 2 folders would have been.
>>>>
>>>> CONCLUSON
>>>> I now realise that the “WAN” / “LAN” distinction is disappearing. He
>> said
>>>> the only reason the “cloud” solution wasn’t hosted off-site was that
>> they
>>>> had measured the bandwidth that the customer used and calculated that
>> the
>>>> cost would be astronomical if it was on AWS or something like that, but
>> in
>>>> all other respects it was a cloud solution.
>>>>
>>>> I was wondering, how do other major 4D server deployers optimise their
>>>> deployment strategies to take advantage of this ? It seems a great thing
>>>> that we are being “floated out to the cloud” without actually having to
>> do
>>>> extra significant work, but what about things like the backup strategy
>> ? I
>>>> don’t really like the idea that the log file has the same redundancy
>> system
>>>> as the main datafile because the whole idea is that the corruption
>> doesn’t
>>>> get replicated (which is what a RAID system does) and it’s independent
>> at
>>>> the logical level.
>>>>
>>>> We seem one step away from being able to supply server solutions where
>>>> “our” customer doesn’t have to host the database server on premises. Is
>>>> anybody doing this at an advanced level ? (e.g. connecting with 4D
>> client
>>>> native to a 4D server that’s 3rd-party hosted).
>>>>
>>>> Regards
>>>>
>>>> Peter
>>>>
>>>> **********************************************************************
>>>> 4D Internet Users Group (4D iNUG)
>>>> Archive:  http://lists.4d.com/archives.html
>>>> Options: https://lists.4d.com/mailman/options/4d_tech
>>>> Unsub:  mailto:[hidden email]
>>>> **********************************************************************
>>>
>>>
>>>
>>> --
>>> Kirk Brooks
>>> San Francisco, CA
>>> =======================
>>>
>>> What can be said, can be said clearly,
>>> and what you can’t say, you should shut up about
>>>
>>> *Wittgenstein and the Computer *
>>> **********************************************************************
>>> 4D Internet Users Group (4D iNUG)
>>> Archive:  http://lists.4d.com/archives.html
>>> Options: https://lists.4d.com/mailman/options/4d_tech
>>> Unsub:  mailto:[hidden email]
>>> **********************************************************************
>>
>> **********************************************************************
>> 4D Internet Users Group (4D iNUG)
>> Archive:  http://lists.4d.com/archives.html
>> Options: https://lists.4d.com/mailman/options/4d_tech
>> Unsub:  mailto:[hidden email]
>> **********************************************************************
>
>
>
> --
> Kirk Brooks
> San Francisco, CA
> =======================
>
> What can be said, can be said clearly,
> and what you can’t say, you should shut up about
>
> *Wittgenstein and the Computer *
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:[hidden email]
> **********************************************************************

**********************************************************************
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************