Modularization

This tutorial shows you how to define and edit reusable, modular metadata documents that can be linked together as a graph of metadata using AML.

Before you begin

Prerequisites:

  • JVM: version 7 or higher
  • SBT to build the AMF command-line

Download the example

You can download the example from the AML project examples repository in Github.

git clone https://github.com/aml-org/examples.git
cd examples/modular

Build and install the AMF command-line tool

This tutorial assumes you have a working version of the AMF comand-line as described in the quickstart tutorial.

Initial stand-alone AML dialect

In this tutorial, you will split a stand-alone, monolithic AML metadata document into a set of modular documents that can be edited, published and linked independently.

We will start with the initial version of the dialect, found in aml/modular/dialect/database_configuration.yaml, a very simple AML Dialect describing configuration metadata file with information to connect to different databases.

database_configuration.yaml

#%Dialect 1.0

dialect: DB Config
version: 1.0

uses:
  infra: ../vocabulary/infra.yaml
  cfg: ../vocabulary/config.yaml

external:
  schema-org: http://schema.org/
  dbconfig: http://mycompany.com/dialects/dbconfig#


nodeMappings:

  DatabaseConfigNode:
    classTerm: cfg.ConfigurationItem
    mapping:
      server:
        propertyTerm: cfg.configured
        range: DatabaseServerNode
      dbs:
        propertyTerm: dbconfig.dbs
        mapKey: cfg.environment
        range: DatabaseNode

  DatabaseNode:
    classTerm: infra.Database
    mapping:
      environment:
        propertyTerm: cfg.environment
        range: string
        enum:
          - development
          - production
          - testing
          - qa
      name:
        propertyTerm: schema-org.name
        range: string
        mandatory: true
      username:
        propertyTerm: cfg.username
        range: string

  DatabaseServerNode:
    classTerm: infra.DatabaseServer
    mapping:
      host:
        mandatory: true
        propertyTerm: infra.endpoint
        range: string
      port:
        mandatory: true
        propertyTerm: infra.port
        range: integer
      driver:
        mandatory: true
        propertyTerm: infra.driver
        range: string

documents:
  root:
    encodes: DatabaseConfigNode

This AML dialect uses two vocabularies, one to describe infrastructure and the other to describe configurations, located in aml/modular/vocabulary/config.yaml and aml/modular/vocabulary/infra.yaml.

Since this AML dialect defines a single, monolithic type of document, the documents section of the dialect includes only a root entry encoding the root node of the document.

Using this dialect definition, we can edit standalone configuration documents, like the configuration document located at aml/modular/db1.yaml:

db1.yaml

#%DBConfig 1.0

server:
  host: localhost
  port: 5001
  driver: mysql

dbs:
  dev:
    name: accounts_dev
    username: jsmith
  test:
    name: accounts_test
    username: jsmith

Defining reusable fragments

In many situations, entities defined in a document can be reused in multiple documents. AML allows you to define reusable entities in their own independent metadata documents, known as fragments, that can be linked from all these documents. The graph of information parsed from all these documents will point to the same node URI in the generated metadata graph.

To declare a fragment, the documents section of the AML dialect must include a fragments entry with a mapping from the name of the fragment to the node encoded in the fragment.

Version 1.1 of the DB Config dialect introduces a fragment to declare databases servers. It can be found in the aml/modular/dialect/database_configuration_2.yaml.

database_configuration_2.yaml

#%Dialect 1.0

dialect: DB Config
version: 1.1

...

documents:
  root:
    encodes: DatabaseConfigNode
  fragments:
    encodes:
      DatabaseServer: DatabaseServerNode

In this case, we define a fragment named DatabaseServer encoding a DatabaseServerNode.

With this dialect definition, we can create fragments declaring metadata about database servers with the semantics of the Infrastructure vocabulary, for example in the aml/modular/server.yaml:

server.yaml

#%DatabaseServer / DB Config 1.1

host: localhost
port: 5001
driver: mysql

These fragments can be linked from the configuration document, using the !include linking directive:

db2.yaml

#%DB Config 1.1

server: !include server.yaml

dbs:
  dev:
    name: accounts_dev
    username: jsmith
  test:
    name: accounts_test
    username: jsmith

Libraries of reusable entities

Another common use case is to define related collections of metadata entities that can be reused across different documents.

AML supports these modular collections through the definition of metadata libraries. Libraries are defined in the documents section using the libraries property. This node keeps a map of declarations for the library, including name of the declaration and type of declared nodes.

The version 1.2 of the DB Config dialect, located at aml/modular/dialect/database_configuration_3.yaml includes the declaration of libraries of servers:

database_configuration_3.yaml

#%Dialect 1.0

dialect: DB Config
version: 1.2

...

documents:
  root:
    encodes: DatabaseConfigNode
  fragments:
    encodes:
      DatabaseServer: DatabaseServerNode
  library:
    declares:
      servers: DatabaseServerNode

After these dialect definitions, metadata documents containing libraries of servers can be defined, like the one located at aml/modular/servers_library.yaml:

servers_library.yaml

#%Library / DB Config 1.2

servers:

  local_mysql:
    host: localhost
    port: 5001
    driver: mysql

  local_postgres:
    host: localhost
    port: 5432
    driver: postgres

  production_mysql:
    host: db.myapp.com
    port: 9501
    driver: mysql

These libraries can be referenced from configuration document using library reference aliases, as in aml/modular/db3.yaml:

db3.yaml

#%DB Config 1.2

uses:
  servers: servers_library.yaml

server: servers.local_mysql

dbs:
  dev:
    name: accounts_dev
    username: jsmith
  test:
    name: accounts_test
    username: jsmith

Instead of library reference aliases, $ref linking directives can also be used, as shown in this example JSON representation of the same metadata:

db3.json

{
    "$dialect": "DB Config 1.2",
    "server": { "$ref": "servers_library.yaml#/servers/local_mysql"},
    "dbs": {
        "dev": {
            "name": "accounts_dev",
            "username": "jsmith"
        },
        "test": {
            "name": "accounts_test",
            "username": "jsmith"
        }
    }
}

Parsing and resolving with AMF

AMF can be used to parse modular metadata documents containing links. By default, AMF will produce a JSON-LD document encoding a graph that explicitly links documents using the doc:link-target property. Linked documents will be eagerly followed and added to the doc:references list.

Different types of documents in the graph will be marked with different semantics.

  • Fragments will be marked with the doc:Fragment class.
  • Libraries will be marked with the doc:Module class.

For example, to parse version 1.1 of the database document using a fragment, AMF can be used:

examples [master] $ java -jar amf.jar parse -ds file://aml/modular/dialect/database_configuration_2.yaml -in "AML 1.0" -mime-in application/yaml -ctx true file://aml/modular/db2.yaml
[
  {
    "@id": "#",
    "@type": [
      "meta:DialectInstance",
      "doc:Document",
      "doc:Fragment",
      "doc:Module",
      "doc:Unit"
    ],
    "meta:definedBy": [
      {
        "@id": "file://aml/modular/dialect/database_configuration_2.yaml#"
      }
    ],
    "doc:encodes": [
      {
        "@id": "#/",
        "@type": [
          "http://mycompany.com/vocabularies/config#ConfigurationItem",
          "file://aml/modular/dialect/database_configuration_2.yaml#/declarations/DatabaseConfigNode",
          "meta:DialectDomainElement",
          "doc:DomainElement"
        ],
        "http://mycompany.com/vocabularies/config#configured": [
          {
            "@id": "#/server",
            "@type": [
              "http://mycompany.com/vocabularies/infra#DatabaseServer",
              "file://aml/modular/dialect/database_configuration_2.yaml#/declarations/DatabaseServerNode",
              "meta:DialectDomainElement",
              "doc:DomainElement"
            ],
            "doc:link-target": [
              {
                "@id": "file://aml/modular/server.yaml#/"
              }
            ],
            "doc:link-label": [
              {
                "@value": "server.yaml"
              }
            ]
          }
        ],
        "http://mycompany.com/dialects/dbconfig#dbs": [
         ...
        ]
      }
    ],
    "doc:references": [
      {
        "@id": "file://aml/modular/server.yaml#",
        "@type": [
          "meta:DialectInstanceFragment",
          "doc:Fragment",
          "doc:Unit"
        ],
        "meta:definedBy": [
          {
            "@id": "file://aml/modular/dialect/database_configuration_2.yaml#"
          }
        ],
        "doc:encodes": [
          {
            "@id": "file://aml/modular/server.yaml#/",
            "@type": [
              "http://mycompany.com/vocabularies/infra#DatabaseServer",
              "file://aml/modular/dialect/database_configuration_2.yaml#/declarations/DatabaseServerNode",
              "meta:DialectDomainElement",
              "doc:DomainElement"
            ],
            "http://mycompany.com/vocabularies/infra#driver": [
              {
                "@value": "mysql"
              }
            ],
            "http://mycompany.com/vocabularies/infra#endpoint": [
              {
                "@value": "localhost"
              }
            ],
            "http://mycompany.com/vocabularies/infra#port": [
              {
                "@value": 5001
              }
            ]
          }
        ]
      }
    ],
    "@context": {
      "@base": "file://aml/modular/db2.yaml",
      "doc": "http://a.ml/vocabularies/document#",
      "meta": "http://a.ml/vocabularies/meta#",
      "schema-org": "http://schema.org/"
    }
  }
]

Sometimes, we want to work with the final graph of metadata in which all explicit links have been replaced by standard JSON-LD links. In the process, all the doc:references relationships disappear and a single unit is generated. This process is known as resolution in AMF.

You can trigger the resolution process in AMF programmatically or by passing the --resolve true argument to the command line tool:

examples [master] $ java -jar amf.jar parse -ds file://aml/modular/dialect/database_configuration_2.yaml -in "AML 1.0" -mime-in application/yaml -ctx true --resolve true file://aml/modular/db2.yaml
[
  {
    "@id": "#",
    "@type": [
      "meta:DialectInstance",
      "doc:Document",
      "doc:Fragment",
      "doc:Module",
      "doc:Unit"
    ],
    "meta:definedBy": [
      {
        "@id": "file://aml/modular/dialect/database_configuration_2.yaml#"
      }
    ],
    "doc:encodes": [
      {
        "@id": "#/",
        "@type": [
          "http://mycompany.com/vocabularies/config#ConfigurationItem",
          "file://aml/modular/dialect/database_configuration_2.yaml#/declarations/DatabaseConfigNode",
          "meta:DialectDomainElement",
          "doc:DomainElement"
        ],
        "http://mycompany.com/vocabularies/config#configured": [
          {
            "@id": "file://aml/modular/server.yaml#/",
            "@type": [
              "http://mycompany.com/vocabularies/infra#DatabaseServer",
              "file://aml/modular/dialect/database_configuration_2.yaml#/declarations/DatabaseServerNode",
              "meta:DialectDomainElement",
              "doc:DomainElement"
            ],
            "http://mycompany.com/vocabularies/infra#driver": [
              {
                "@value": "mysql"
              }
            ],
            "http://mycompany.com/vocabularies/infra#endpoint": [
              {
                "@value": "localhost"
              }
            ],
            "http://mycompany.com/vocabularies/infra#port": [
              {
                "@value": 5001
              }
            ]
          }
        ],
        "http://mycompany.com/dialects/dbconfig#dbs": [
          {
            "@id": "#/dbs/dev",
            "@type": [
              "http://mycompany.com/vocabularies/infra#Database",
              "file://aml/modular/dialect/database_configuration_2.yaml#/declarations/DatabaseNode",
              "meta:DialectDomainElement",
              "doc:DomainElement"
            ],
            "schema-org:name": [
              {
                "@value": "accounts_dev"
              }
            ],
            "http://mycompany.com/vocabularies/config#username": [
              {
                "@value": "jsmith"
              }
            ],
            "http://mycompany.com/vocabularies/config#environment": [
              {
                "@value": "dev"
              }
            ]
          },
          {
            "@id": "#/dbs/test",
            "@type": [
              "http://mycompany.com/vocabularies/infra#Database",
              "file://aml/modular/dialect/database_configuration_2.yaml#/declarations/DatabaseNode",
              "meta:DialectDomainElement",
              "doc:DomainElement"
            ],
            "schema-org:name": [
              {
                "@value": "accounts_test"
              }
            ],
            "http://mycompany.com/vocabularies/config#username": [
              {
                "@value": "jsmith"
              }
            ],
            "http://mycompany.com/vocabularies/config#environment": [
              {
                "@value": "test"
              }
            ]
          }
        ]
      }
    ],
    "@context": {
      "@base": "file://aml/modular/db2.yaml",
      "doc": "http://a.ml/vocabularies/document#",
      "meta": "http://a.ml/vocabularies/meta#",
      "schema-org": "http://schema.org/"
    }
  }
]

results matching ""

    No results matching ""