JSON schema and some object format examples for thought/discussion

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

JSON schema and some object format examples for thought/discussion

David Adams-4
I've been in a couple of threads recently regarding 4D's object and JSON
commands. For background, I use JSON extensively outside of 4D and quite a
bit in 4D. Until now, I've used the NTK 2.x JSON commands or rolled some
simple things by hand with templates and PROCESS 4D TAGS. I've been trying
to see how far I can go replacing these with native 4D tools. One thing I'm
after is the ability to pretty blindly or generically review JSON to make
sure that it's structure and contents are complete and correct.  I get the
impression that not everyone immediately understands why you would need or
want to do such a thing. I figured I'd post an explanation and a workable
solution.

First up, I have a *lot* of configuration data stored in JSON format.
Within a single 4D structure, I've got a bunch of virtual databases and a
bunch of code modules and table modules. So, groupings of methods and other
internal resources that belong together. I've got a JSON manifest for each
of these things that describes what should be there, what can be removed
under certain conditions, and how to find things. I've also got external
resources mapped out. For example, I've got 900+ external image files and
because of all of the meta-data I've got stored in JSON (and UIs that take
advantage of that data), I can very easily find whatever image I need and
generate whatever code I need for it. (Use it in a progress window, set it
as a form icon, get the relative or absolute path to the image, etc.)

So, this stuff helps me a lot during development. More than that, it helps
me at build time. Before building, I run a bunch of validation routines
that sweep through the configuration data and figure out if it's correct
and complete. Are my search dialogs logically configured? Are all of the
external resources I expect in place? And so on. After build, my code
automatically goes out to the built database or stand-alone and deletes
unwanted resources from the /Resources folder. I have a few databases that
need a 50MB+ command-line tool, most don't. Nice to clear that out.

So, having data structures for verification that are themselves verifiable
is part and parcel of how I work. It helps me to avoid a *ton* of otherwise
stressful and tedious bugs. I hate tedium and I hate stress. But I'm
obviously gearing up to be 'angry old man shakes fist at cloud' one day ;-)
Hey! Cranky old men aren't born, they have to train, train, train!

Anyway, I'm trying to use 4D's JSON/objects for some of this and, to be
fair, I probably should have accepted immediately that I need to have a
'schema' for describing what each element should/should not/must contain. I
do that all of the time for method parameters (accepts a value, this range,
any value from this series, etc.) I already do this in my existing code so
I guess it's just price of admission to getting what I need using 4D's
commands.

In the case of C_OBJECT, I decided immediately to inject my own type system
into it. So, a naked object looks like this:

{
"header": {
"type_name": "UniqueTypeSignature",
"version_number": 1,
"instance_name": "Optional value"
},
"body": {
}
}


All of my objects use this format. The header belongs to "object" and the
body belongs to whatever module controls a specific object type. For
example, ErrorStack, or TestResult. Eh? Why would you want that? I've
talked about this already so briefly:

* You can now validate any object at two levels. The overall object level
(header and it's elements and that a body exists), and the specific object
type level.

* Division of responsibility is clear. The Object module is responsible for
the overall envelope and the individual module, like ErrorStack, is
responsible for the body.

* In this setup the Object module and ErrorStack module don't need to know
anything about each other.

To make this easier to use and more useful, you can define each object type
in a catalog. in my case, this is called OBRegister. You register the type,
current version name, name of the method to create the object, and
(optionally) the names of methods to convert the object to text or to
validate it. Since there's a default for these tasks up in Object, any
object can be created, turned to text, or validated generically. Ask the
register what routine to run and you get one back. It's either a custom
method (ErrorStack_Validate) or a the generic one (Object_Validate.) Again,
nothing in Object needs to know the details of how a specific module works
and nothing in, say ErrorStack needs to know how Object works.

This is all pretty bread-and-butter basic programming in most of the world,
but in 4D it's pretty easy to get caught up talking about the nuts and
bolts. So here are some nuts and bolts for storing the information needed
to do the validation generically. What you need is some meta-data or a
"schema". In my case, OBRegister needs a routine like

OBRegister_DefineObjectProperty

This method will take in information to describe the object element, it's
type, and any other attributes I might need. For example, here's a made-up
set of entries for the header part of my Object envelope:

[
   {
      "path":"header",
      "type":"object",
      "required":true
   },
   {
      "path":"header.type_name",
      "type":"TypeName",
      "required":true
   },
   {
      "path":"header.version_number",
      "type":"Number",
      "required":true
   },
   {
      "path":"header.instance_name",
      "type":"Text",
      "required":false
   }
]

(I just typed that JSON up, it's not coming from 4D.) So, with this data in
place, you end up with a way of answering a lot of questions about your
objects. Up in "type", a careful reader will notes I've got "TypeName" as a
type. Eh? JSON only has some simple types. Well, it's your schema so you
can do what you want to. In my case, TypeName is a magic signature and
functions as a key. Other helpful abstract data types include MethodName -
so that you know it's something you can validate as a method name, and so
on.

There are a lot of attributes that you can add to a scheme, like more
nuanced rules than "required: true/false." For example:
-- Value is in a range between _this_ and _that_. Like  1-7.
-- Value must be in this series of possible values. Like "Mon", "Tue",
"Wed", etc.
-- Empty value is allowed: true/false. (This is helpful in conjunction with
a range or series definition more often than you might think.)
-- Value must not be exported. So that your save-to-disk routine knows to
scrub that element.

There's tons of useful stuff. And now it should be clear why I include
version_number in the object header. I tend to evolve my formats over time
and it's *super* handy to be able to version test the object. Well, it's a
complete waste of time and effort until you've got more than one version,
at which point you thank yourself for thinking ahead.

Anywone else headed down the road of building and managing custom types
using 4D's native commands? I'll grant that I was being silly to think that
I could get away without adding some kind of schema...I needed to do it the
last time, I need to do it this time.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: JSON schema and some object format examples for thought/discussion

David Adams-4
...in 4D code, something like this for defining the base object's basic
components:

// Object_RegisterRootObject

OBRegister_RegisterObject (Object is
RootObject;1;"Object_New";"Object_ToText")

C_BOOLEAN($required)
C_BOOLEAN($optional)
$required:=True
$optional:=False
OBRegister_RegisterObject (Object is RootObject;"Root.header";Is
object;$required)
OBRegister_RegisterObject (Object is RootObject;"Root.header";Abstract type
is ObjectType;$required)
OBRegister_RegisterObject (Object is RootObject;"Root.version_number";Is
real;$required)
OBRegister_RegisterObject (Object is RootObject;"Root.instance_name";Is
text;$optional)
OBRegister_RegisterObject (Object is RootObject;"Root.body";Is
object;$required)

There are several custom constants in there.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: JSON schema and some object format examples for thought/discussion

David Adams-4
D'oh! Darn kids! The broke into my yard and messed up my code. (Angry old
man shakes fist at cloud...) The previous example is pretty confusing
because I'm calling the wrong routine to register elements. (I haven't
written this bit of code yet.)

OBRegister_RegisterObject (Object is
RootObject;1;"Object_New";"Object_ToText")
C_BOOLEAN($required)
C_BOOLEAN($optional)
$required:=True
$optional:=False
OBRegister_RegisterElement (Object is RootObject;"Root.header";Is
object;$required)
OBRegister_RegisterElement (Object is RootObject;"Root.header";Abstract
type is ObjectType;$required)
OBRegister_RegisterElement (Object is RootObject;"Root.version_number";Is
real;$required)
OBRegister_RegisterElement (Object is RootObject;"Root.instance_name";Is
text;$optional)
OBRegister_RegisterElement (Object is RootObject;"Root.body";Is
object;$required)

So, OBRegister_RegisterObject for the overall type,
OBRegister_RegisterElement for the object type's elements and their rules.
(Again, the rule set is simple here: "type" and required or not.)

Oh, I meant to mention that anyone that's looked at Cannon's object module
will have seen that he very helpful extends basic JSON types to support
things like dates and other useful types. Same idea here - you can *store*
data as text or numbers but then *interpret* them as something better.
That's the point of a type like "ObjectType" or "MethodName" or
"ErrorName", etc.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: JSON schema and some object format examples for thought/discussion

David Adams-4
...and take the prior as a sketch. Still working out the best way to
organize type descriptors in this case. I want to be able to figure out the
raw JSON type, if it's an array, what type it goes into in 4D (most
important for arrays), and if there is any kind of extra type magic
("abstract" type) to do.


Note that with the scheme above (or anything comparable, however
implemented),

ObjectType (unique key for defined objects) + ElementPath (unique key for
elements), you get a clear index into the dictionary of definitions.
Obviously, elements may repeat within the JSON*, but anything at the same
type of path is expected to remain the same. So, you can't have:

[
  {"foo":"bar"},
  {"foo":5}
]

Not a problem in my case.

 * Cardinality rules are something else that you could specify in a
schema-like system, if you have a need.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************
Reply | Threaded
Open this post in threaded view
|

Re: JSON schema and some object format examples for thought/discussion

David Adams-4
And, yes, my manic stream of emails *is* very much the equivalent of an
unboxing video ;-)

"Hey guys! I'm checking out the C_OBJECT commands today, let's open the
docs!"
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:[hidden email]
**********************************************************************