mmx metadata framework
...the DNA of your data
MMX metadata framework is a lightweight implementation of OMG Metadata Object Facility built on relational database technology. MMX framework
is based on three general concepts:
Metamodel | MMX Metamodel provides a storage mechanism for various knowledge models. The data model underlying the metadata framework is more abstract in nature than metadata models in general. The model consists of only a few abstract entities... see more.
Access layer | Object oriented methods can be exploited using inheritance to derive the whole data access layer from a small set of primitives created in SQL. MMX Metadata Framework provides several diverse methods of data access to fulfill different requirements... see more.
Generic transformation | A large part of relationships between different objects in metadata model are too complex to be described through simple static relations. Instead, universal data transformation concept is put to use enabling definition of transformations, mappings and transitions of any complexity... see more.

Tips and tricks: Fetching a scalar value with XDTL

November 11, 2011 11:11 by Viktor Varlamov

To fetch a scalar value, there are two choises:

  • xdtl:fetch
  • xdtl:query
Obverve:
 

Using fetch

<xdtl:fetch 
      source="SELECT object_type_nm, object_type_ds FROM MD_OBJECT_TYPE WHERE object_type_cd = 0"  
      rowset="my_variable" 
      connection="$conn" />
<xdtl:log 
      msg="Name of this object type is ${my_variable.get(0)[0]}" /> 

Using query

<xdtl:query 
      source="SELECT object_type_nm, object_type_ds FROM MD_OBJECT_TYPE WHERE object_type_cd = 0" 
      target="my_variable" 
      connection="$conn" />
<xdtl:log 
      msg="Name of this object type is $my_variable" />

 

xdtl:query saves value of the first column in the first row to target variable. UPDATE and DELETE queries save the number of affected rows into the same variable.



Metadata materialization, part 1: Materialization choices

November 9, 2011 13:53 by Viktor Varlamov

Why?

 
MMX Framework's schema is simple: just six tables, three to describe metamodels and three to store metadata. But when it comes to accessing the stored data, users, especially novice users get confused. 
 
SELECT 
    object_id, object_nm 
FROM
    MD_OBJECT 
WHERE
    object_type_cd = mmxmd.get_object_type('Core', 'Document')
 
"Why can't we just have a table Documents?" they ask. You tell them (again) about metamodels and how all this is so flexible and stuff. And then they want a list of documents with some properties and gets complicated. And when you step into the MD_RELATION realm, you have usually lost about 2/3 of your followers. Now what?
 
Or there is an application in your system, legacy or not, that needs to access the MMX data and it's developers are not willing to access anything more complex than a view. A typical integration case, as we have encountered.
 
Or you want migrate an old app to MMX and do not wish to update all those integration points you have built for it other the year. 
So you have to build an view on top of MMX tables for clean and the-way-the-things-always-were access of you MMX data.
 

How?

 
There are three obvious choices on how to materialize MMX data:
  • views
  • materialized views or table for high performance need
  • tables with a set of INSTEAD_OF  triggers that gently allow user to update metadata in MMX schema without actually accessing the actual MMX tables.
The last option is obviously something you will never want to implement. There is an application level in Java or .Net or Ruby to handle all those things the they should.
 
In this series we will be operating under following assumption: we build views to read data, not up update it.
 
Whatever the metamodel that you will be materializing; we suggest that you always should use LEFT OUTER JOINs to bind MD_OBJECT, MD_PROPERY and MD_RELATION tables. You should do this at least for these two reasons:
  1. metamodels are build that way - properties and relations are not required
  2. metadata quality does not allow you to presume that the required properties and relations exist
 
Using a lot of LEFT OUTER JOINs means slower performance. If your data is read often, it may become a performance issue. To counter that, physical materialization of the views is suggested. Different database platforms that MMX supports provide different options for physical materialization, just pick your favorite, but bear in mind, that no physical materialization is needed unless the load from your integration point will become noticeable.
 
In the next three parts of these series we will:
  • build object materialization view generator
  • talk about advance object materialization options
  • build many-to-many relations materialization generator.
 


The X Is For eXtensibility

September 11, 2011 19:24 by mmx

XDTL stands for eXtensible Data Transformation Language. Extensibility here means that new language elements can be easily added without having to make changes to XML schema defining the core XDTL language. These extensions can, for example, be coded in XDTL and stored as XDTL packages with task names identifying the extension elements. XDTL Runtime expects to find the extension element libraries in directories listed in extensions.path parameter in xdtlrt.xml configuration file: the pathlist is scanned sequentially until a task with a matching name is found. 

During package execution an extension is provided with a full copy of the current context. An extension gets access to every variable value that is present in the calling context, as well as its attribute values that get converted into variables with names of the attributes. From the extension's point of view all those values are 'read-write', but only those passed as variables retain their values after the extension element finishes. So, considering passing values to an extension, variables can be seen as 'globals' that return values and extension element attributes as 'locals' that get discarded.

XDTL syntax definition (XML schema) includes an any element that allows XDTL language to be extended with elements not specified directly in the schema. any element in XDTL is defined as

<xs:any namespace="##other" processContents="lax"/>

##other means that only elements from namespace other than the namespace of the parent element are allowed. In other words, when parser sees an unknown element it will not complain but assume that it could be defined in some other schema. This prevents ambiguity in XML Schema (Unique Particle Attribution). Setting processControl attribute of an any element to "lax" states that if that 'other' schema cannot be obtained, parser will not generate an error.

 

So how does this work? We assume that our main script is referencing an external XML schema, elements of which are qualified with prefix 'ext':

xmlns:ext="http://xdtl.org/xdtl-ext"

In this external schema a single element, "show" with attribute "text" is defined. Here are some examples of what works and what doesn't.

<ext:show text="sometext"/> works, as the external namespace with element "show" is referenced by the prefix 'ext'.

<show xmlns="http://xdtl.org/xdtl-ext" text="sometext"/> also works, as the namespace reference is 'embedded' in the "show" element.

<show text="sometext"/> does not validate, as the parser looks for element "show" in the current schema (error message Invalid content was found starting with element 'show' is produced).

<ext:show nottext="sometext"/> does not validate either (Attribute 'nottext' is not allowed to appear in element 'ext:show').

<ext:notshow text="sometext"/> validates but still does not work! As the processContents attribute of any element is "lax", although the element is not found the parser ignores this. However, the XDTL Runtime complains as it cannot find element definition in extension pathlist.

 

What if we would want to use extensions without XML schema accompanying it? For that we remove the reference to the external schema from the script header and run the examples once again.

<ext:show text="sometext"/> would not validate any more as the prefix 'ext' is not defined. The same applies to all the other examples with prefix in front of the extension element.

<show text="sometext"/> would not validate either as the parser looks for extension element in the current schema.

<show xmlns="http://xdtl.org/xdtl-ext" text="sometext"/>, however, validates and works! Although the parser cannot find the schema it does not complain due to "lax" processContents attribute. As long as XDTL Runtime is able to find the library package containing the extension in the pathlist everything is fine, otherwise it would give Command 'Extension' failed error.

 

So here's the summary. Extended elements (commands) can be well-defined (having their syntax definitions in form of an XML schema) or undefined (just the package, no XML schema), just as a transformation designer sees fit. In the former case, extended elements will be validated exactly as the core language elements would, in the latter case they will pass without validation. If an undefined and non-validated extension element is executed and does not match its invocation a run-time error would be generated.