YODEL MetaData

YODEL supports the definition and application of metadata instances which allow you to pre-define attribute values and apply them to multiple data elements. Use of metadata is optional: its only purpose being to shrink the size of the resulting YODEL packet. Using it may result in slower serialization/deserialization operations as the generation and application of metadata requires more processing.

MetaData is only useful when there are multiple elements which would share the same attribute values (type, fields, etc). The classic example is that a of a record set represented as an array of objects: each object (representing rows in the recordset) will have the same fields and type attribute values.

Understanding MetaData

Metadata can be seen as a "mask" for data. When a YODEL parser encounters a <d> tag with a metadate attribute it looks to the corresponding metadata and applies the attributes defined there to the <d> tag before parsing it. In the case of Arrays and Objects multiple, nested <d> tags might be defined in the metadata.

The parser should continue to apply corresponding metadata until no more metadata is available after which, if more data were present it would parse normally. Attribute values defined directly in the <d> tag will supersede any defined in the corresponding metadata.

This behavior can create results that are difficult to work with (since it's not possible to simply "trust" the metadata). However it does offer more flexibility, especially when dealing with sparse data sets. For example it allows an array to define a column as "date" in metadata while still allowing specific values to be set to "null". Any implementation of this behavior should be heavily documented.

Although metadata is rarely useful for simple values a simple example would be helpful in understanding the concept. Consider the following YODEL packet:

<yodel>
     <d type="string">val1</d>
</yodel>

This packet could be rewritten using metadata like so (note that in such a simple example the metadata actually inflates the size of the packet):

<yodel>
     <md name="MyString">
          <d type="string" />
     </md>
     <d metadata="MyString">val1</d>
</yodel>

Both this packet and the first represent the exact same data. With the second packet a YODEL deserializer would, conceptually, take the following steps:

Process and store all of the defined metadata.
Find the <d> tags with metadata attributes.
Apply the attributes from within the metadata (the "type" and "fields" attributes) to the corresponding <d> tags (in this case the attribute "type" is defined in metadata and so will be applied to the corresponding <d>.
Parse the <d> tags normally with the inserted attributes and values.

To keep parsers relatively simple and fast metadata cannot be nested. For example the following will not be parsed correctly:

<yodel>
     <md name="MyBoolean">
          <d type="boolean" />
     </md>
     <md name="MyObject">
          <d type="Object" fields="prop1">
               <d metadata="MyBoolean" /d>
          </d>
     </md>
     <d metadata="MyObject">
          <d>true</d>
     </d>
</yodel>

Parser should ignore the metadata attribute within the struture of the <md> tag. In this case the data would be parsed as a string (the default data type) not as a boolean.

As we'll see metadata makes much more sense when applied to more complex data structures.

MetaData and Objects

MetaData for objects is defined the same way as the objects themselves: using <d> tags which may be nested as deeply as you wish. By way of example consider an object which represents a "person". This object would have three properties: FirstName, LastName, and Children. The property "Children" is an array of person objects.

Such an object could be represented in YODEL like this:

<yodel>
<d type="object" fields="FirstName,LastName,Children">
     <d type="string">Jim</d>
     <d type="string">Davis</d>
     <d type="array">
          <d type="object" fields="FirstName,LastName,Children">
               <d type="string">Paxton</d>
               <d type="string">Davis</d>
               <d type="array"></d> 
          </d>
          <d type="object" fields="FirstName,LastName,Children">
               <d type="string">Matilda</d>
               <d type="string">Davis</d>
               <d type="array"></d> 
          </d>
     </d> 
</d>
</yodel>

Using metadata to predefine the attributes and attribute values for the person object in advance we could represent the same data like this:

<yodel>
<md name="person">
     <d type="object" fields="FirstName,LastName,Children">
          <d type="string" />
          <d type="string" />
          <d type="array" />
     </d>
</md>
<d metadata="person">
     <d>Jim</d>
     <d>Davis</d>
     <d>
          <d metadata="person">
               <d>Paxton</d>
               <d>Davis</d>
               <d></d> 
          </d>
          <d metadata="person">
               <d>Matilda</d>
               <d>Davis</d>
               <d></d> 
          </d>
     </d> 
</d>
</yodel>

It should be clear that the more "person" objects in the packet the more space will be saved by the implementation of the metadata.

MetaData and Arrays

Arrays present a special situation in relation to metadata: the first child <d> tag encountered in an array will be applied to ALL children (elements) of that array. This special behavior has been implemented to honor the fact that in most cases each element of an array is the same datatype. Although doesn't prevent you from creating arrays whose elements are not the same type it does prevent you from constructing metadata to represent them.

A simple example may illustrate this better. Consider a simple array of numeric values:

<yodel>
<d type="array">
     <d type="number">1</d>
     <d type="number">2</d>
     <d type="number">3</d>
     <d type="number">4</d>
     <d type="number">5</d>
</d>
</yodel>

Repeating the type attribute for every element will quickly bloat your packet. However you can define metadata to address the issue like this:

<yodel>
<md name="MyArray">
     <d type="array">
          <d type="number" />
     </d>
</md>
<d metadata="MyArray">
     <d>1</d>
     <d>2</d>
     <d>3</d>
     <d>4</d>
     <d>5</d>
</d>
</yodel>

The metadata defines an array with a single element of type "number". The metadata for that single element however is applied to ALL elements of the array. Here is another, more complex example which defines a two-dimensional array (represented in YODEL as an array of arrays):

<yodel>
<md name="MyArray">
     <d type="array">
          <d type="array">
               <d type="number" />
          </d>
          <d type="array">
               <d type="string" />
          </d>
          <d type="array">
               <d type="boolean" />
          </d>
     </d>
</md>
<d metadata="MyArray">
     <d>
          <d>1</d>
          <d>2</d>
          <d>3</d>
          <d>4</d>
          <d>5</d>
     </d>
     <d>
          <d>A</d>
          <d>B</d>
          <d>C</d>
          <d>D</d>
          <d>E</d>
     </d>
     <d>
          <d>true</d>
          <d>false</d>
          <d>false</d>
          <d>true</d>
          <d>true</d>
     </d>
</d>
</yodel>

It should be obvious that using metadata in this way can drastically reduce the size of your YODEL packets. Finally here is an example using the "person" objects defined in the section on objects above. An array of such objects could be serialized using metadata like this:

<yodel>
<md name="personArray">
     <d type="array">
          <d type="object" fields="FirstName,LastName,Children">
               <d type="string" />
               <d type="string" />
               <d type="array" />
          </d>
     </d>
</md>
<d metadata="personArray">
     <d>
          <d>Jim</d>
          <d>Davis</d>
          <d></d> 
     </d>
     <d>
          <d>Paxton</d>
          <d>Davis</d>
          <d></d> 
     </d>
     <d>
          <d>Matilda</d>
          <d>Davis</d>
          <d></d> 
     </d>
</d>
</yodel>

The metadata for the array defines the first element of the array (a complex object) and the metadata for that element is applied to all children of linked array. The same data could have also been represented using the "person" metadata from the initial example like this:

<yodel>
<md name="person">
     <d type="object" fields="FirstName,LastName,Children">
          <d type="string" />
          <d type="string" />
          <d type="array" />
     </d>
</md>
<d type="array">
     <d metadata="person">
          <d>Jim</d>
          <d>Davis</d>
          <d></d> 
     </d>
     <d metadata="person">
          <d>Paxton</d>
          <d>Davis</d>
          <d></d> 
     </d>
     <d metadata="person">
          <d>Matilda</d>
          <d>Davis</d>
          <d></d> 
     </d>
</d>
</yodel>

While the first example would result in a (slightly) smaller packet size for large arrays some people may find the second easier to comprehend. Which you prefer to use is completely up to you.

Parsers and MetaData

All YODEL compliant parsers should be able to deserialize passed metadata in accordance with the above rules. In technical terms how they do this is inconsequential as long as the rules are followed. Whatever methods seem best for the implementing language or environment should be used.

Serialization of meta data is completely optional. Compliant YODEL serializers need make no special effort to determine if metadata is beneficial to the data they're passed. However there are possibilities where automatic generation of metadata may be practical.

Many languages have the concept, either explicitly or implicitly, of a RecordSet (the information returned from a SQL database query for example). This is a perfect case in which to implement automatic metadata generation (by generating an array of objects and using metadata to minimize packet size). Such a parser may decide to generate metadata for all recordsets or for only those which represent more than one record of data. Similarly implementations in languages where arrays are limited to single data types may also find it easy to automate metadata generation for arrays.

In other, more complex, cases it may be that custom code is written to generate metadata information. The key, of course, is to generate compliant YODEL packets: the methods of that generation are unimportant.

Developer's of serializers may also implement custom methods to generate metadata. For example the serializer might look for "hints" in the objects themselves (properties which may provide information to the serializers but which would not be serialized themselves). Serializers may also examine complex objects and arrays to see if meta data would be appropriate if this is practical. Again, as long as the resulting YODEL packets are compliant any steps taken to automate or enable the serialization of metadata is encouraged but not required.